ATP

00:00:00 ◼   ► So your tweet about us having thin topics like oh, there's a lot of stuff

00:00:03 ◼   ► We didn't even get the Apple wearables, but after that you're right, then we're out of topics. I

00:00:06 ◼   ► Just have one item. I guess it was a popular item, but just one last week we were talking about

00:00:13 ◼   ► How if you have some sort of service that?

00:00:16 ◼   ► Deals with your photos and lets you share them no matter what cool features you have you got to at least

00:00:21 ◼   ► Also have a way to send a link to somebody they can view in any web browser

00:00:24 ◼   ► and we also talked about

00:00:26 ◼   ► photo stream and iOS integration and stuff like that.

00:00:29 ◼   ► And although we didn't come out and say it directly,

00:00:32 ◼   ► the implication was that PhotoStream did not have a public URL,

00:00:35 ◼   ► but PhotoStream does have public URLs. You can make a little URL

00:00:38 ◼   ► for a website that you can share with somebody that lets them see

00:00:41 ◼   ► the pictures in your PhotoStream.

00:00:44 ◼   ► I use PhotoStream with my family, and my family

00:00:46 ◼   ► all has iOS devices, and the great thing about that is

00:00:48 ◼   ► it notifies you on your phone or on your iPad,

00:00:51 ◼   ► and when you get the notification, you can just swipe and see the pictures immediately.

00:00:54 ◼   ► immediately, it's a much superior interface for viewing pictures, but you've always got

00:00:59 ◼   ► to have that web interface underneath it all.

00:01:01 ◼   ► And this came up in the context of Carousel, which I think, Casey, you said they didn't

00:01:04 ◼   ► have a way for you to just get an easy web link?

00:01:07 ◼   ► That was my experience, that's correct.

00:01:09 ◼   ► So I had emailed what was to Carousel a completely random email address that wasn't associated

00:01:16 ◼   ► with the Dropbox account.

00:01:17 ◼   ► The email address, in fact, was my work email, and when I received the email from personal

00:01:23 ◼   ► and I'm doing that in air quotes to WorkMe, the work email basically said, "Hey, you need to

00:01:28 ◼   ► install Carousel if you want to do anything useful." And I think it did have like six or

00:01:32 ◼   ► seven pictures in it, little thumbnails of the pictures that I'd actually selected. So it wasn't

00:01:36 ◼   ► like stock photography or anything like that. But with that said, it didn't do anything and there

00:01:42 ◼   ► was no web-based link. It basically said, "Go to the app or get the hell out of here."

00:01:46 ◼   ► **Matt Stauffer** So do we have any more follow-up?

00:01:49 ◼   ► And Casey, do you feel like you need to follow up at all about any topics from last week?

00:01:53 ◼   ► Or is that it?

00:01:54 ◼   ► Do you know what you did, Casey?

00:01:56 ◼   ► Do you want to tell us all what you did?

00:01:58 ◼   ► I'm sorry, Daddy.

00:02:01 ◼   ► I don't know.

00:02:03 ◼   ► I'm a little—not bitter—I'm a little frustrated with the feedback from last week.

00:02:10 ◼   ► And I was debating talking about it, and since you've prompted me, I'll go ahead and

00:02:14 ◼   ► do so and just make my world even worse for myself.

00:02:18 ◼   ► And the thing that frustrated me about the feedback we received is that there were a

00:02:23 ◼   ► few instances, and Mark Edwards is from Bajango, Django—I always pronounce it wrong, I'm

00:02:29 ◼   ► so sorry, Mark—but anyway, he wrote a relatively long missive that he put up as a gist, a gist,

00:02:36 ◼   ► whatever. What am I thinking of?

00:02:37 ◼   ► Jest.

00:02:38 ◼   ► Thank you. I doubted myself. Anyways, I'm all out of whack. So he put up a gist, and

00:02:44 ◼   ► He explained as someone who used to do this sort of thing for a living why vinyl is basically

00:02:50 ◼   ► the most evil thing in the world.

00:02:51 ◼   ► And while I don't entirely agree with what he said, I can't really factually debate

00:02:59 ◼   ► it.

00:03:00 ◼   ► And so things like that—and we'll put a link in the show notes—that was actually

00:03:04 ◼   ► welcome and useful.

00:03:06 ◼   ► But I got a lot of feedback—we got a lot of feedback and I got a lot of feedback—which

00:03:10 ◼   ► basically amounted to "I read something on the internet once, it must be true, you're

00:03:16 ◼   ► crazy and stupid."

00:03:18 ◼   ► And that kind of bothers me a little bit because it was oftentimes not really based in fact,

00:03:26 ◼   ► or sometimes it was, but never ever ever based on experience.

00:03:30 ◼   ► And I'm not going to try to rehash the argument, but suffice to say I put a lot more stock

00:03:34 ◼   ► when I receive feedback from someone who has had the experience of listening to vinyl on

00:03:40 ◼   ► a really nice setup over someone who has just read some things or seen some stuff on the

00:03:45 ◼   ► internet and swears that that's true. Now yes there's a lot of science behind it,

00:03:50 ◼   ► yes I understand that, yes I'm probably crazy. All of that is fine, I agree, I understand

00:03:56 ◼   ► it. It was probably silly of me to say that the fidelity of vinyl was better than the

00:04:00 ◼   ► fidelity of CDs. But when it comes down to it, something I said on Twitter, which I actually

00:04:07 ◼   ► was fairly proud of myself, I'm patting myself on the back, is that it's very kind of cliche,

00:04:12 ◼   ► but life is what happens between one and zero in my eyes. And whether or not the accuracy of the

00:04:18 ◼   ► reproduction on vinyl or CD is better, I just happen to prefer the feel and to some degree,

00:04:25 ◼   ► the what do you call it? The tea something? I already forgot. Tea ceremony, Japanese tea

00:04:30 ◼   ► ceremony? Google it.

00:04:31 ◼   ► Yeah, I know what you mean. So I prefer the tea ceremony of it. And the only thing I will

00:04:37 ◼   ► say to defend myself, other than come at me once you've had personal experience, you

00:04:42 ◼   ► damn nerds, the only other thing I'll say is, why is it, Jon, that you choose to drive

00:04:50 ◼   ► a manual transmission in 2014?

00:04:53 ◼   ► That's not even close to a fair comparison.

00:04:55 ◼   ► I was going to let you slide, but you come back at me with this, and you're just like,

00:04:59 ◼   ► What I was going to let slide is that thing with life is what happens between one and

00:05:03 ◼   ► zero.

00:05:04 ◼   ► That vaguely alludes to terrible illogical reasoning behind crazy—you know what I mean?

00:05:11 ◼   ► I know that's not what you're getting at at all, I understand, but the fact that you chose

00:05:15 ◼   ► to use one and zero is like another nudge at the "Come on guys, we all know that digital

00:05:19 ◼   ► stuff is just cold and unfeeling and—you know what I mean?"

00:05:24 ◼   ► I did.

00:05:25 ◼   ► I did.

00:05:26 ◼   ► But then you came at the other thing—I found the feedback frustrating as well, and you

00:05:29 ◼   ► would think. I wouldn't find the feedback frustrating because I think that most of the

00:05:31 ◼   ► feedback agreed with Marco and I. But I found the feedback very frustrating because, I mean,

00:05:36 ◼   ► this is a common phenomenon, feedback for the show. Very often we will get feedback,

00:05:40 ◼   ► no matter what the topic, on past shows and everything, where someone will be—I mean,

00:05:44 ◼   ► this is not bad feedback. It's good because it shows enthusiasm. You know, someone who's

00:05:48 ◼   ► a big fan of the show will write in and in some ways this is the best kind of feedback,

00:05:52 ◼   ► where they'll be like, "Listen to your show. Maybe think about this, and here's what I

00:05:56 ◼   ► think about it, and they'll present a bunch of ideas about the topic. But what I find frustrating

00:06:01 ◼   ► is a lot of the time, the ideas that they present are things that we talked about on the show.

00:06:07 ◼   ► And you don't want to write back and say, "Yeah, we talked about that, this exact thing,

00:06:12 ◼   ► at length on the show." But then it sounds like you're not valuing their feedback, and it's good

00:06:15 ◼   ► that they're enthusiastic about it, but sometimes it's frustrating to see, like, "I obviously failed

00:06:19 ◼   ► to communicate this idea because it keeps coming back to me as if it's information that we are

00:06:24 ◼   ► missing or if it's new information or new insights. And it's like, we talked about it

00:06:27 ◼   ► for 15 minutes. We said almost these exact words. And it's like, sometimes it's people

00:06:33 ◼   ► sending feedback in real time, or they listen, and then I guess they pause it or whatever,

00:06:37 ◼   ► and then they write their three-page missive, and then they start playing again. And then,

00:06:41 ◼   ► you know, because they haven't gotten to the end of the podcast and realize we addressed

00:06:44 ◼   ► the point. And that happens with tweets, you know, obviously, that's fine. Sometimes it

00:06:47 ◼   ► happens with emails too. But a lot of the feedback was like presenting points back to

00:06:52 ◼   ► any of us, me, Marco, you, that I thought I could swear we covered on the show.

00:06:57 ◼   ► Like we talked about these exact things and it's like maybe we're not communicating effectively

00:07:02 ◼   ► or just like we're talking past our audience or something.

00:07:07 ◼   ► And in the same way I saw a lot of arguments directed at both Casey and us that I thought

00:07:12 ◼   ► we talked about on the show and addressed all different sides of coming back at us in,

00:07:16 ◼   ► you know, arguments that I thought I had, you know, refuted and settled down, arguments

00:07:22 ◼   ► that Casey never made, arguments that we never made,

00:07:24 ◼   ► positions that we never staked out,

00:07:26 ◼   ► the whole nine yards.

00:07:27 ◼   ► This is obviously a very fraught issue.

00:07:30 ◼   ► And I think, Casey, the problem you're still having with it,

00:07:32 ◼   ► and the problem that you continue to have with it,

00:07:34 ◼   ► is that you just gotta--

00:07:37 ◼   ► you have to just make it clear what it is.

00:07:40 ◼   ► In the world of internet, there's

00:07:41 ◼   ► moving the goalposts.

00:07:42 ◼   ► You gotta make it clear what everybody's talking about.

00:07:44 ◼   ► Because at this point, I think everybody on this podcast

00:07:46 ◼   ► all agrees.

00:07:47 ◼   ► It's just that you keep wanting to put the goalposts over there

00:07:49 ◼   ► and pretend that's what we were talking about.

00:07:50 ◼   ► And me and Marco keep wanting to put the goalposts over here

00:07:52 ◼   ► and pretend this is what we're talking about. And I'm trying to keep them kind of where the

00:07:57 ◼   ► original thing was, which was the whole idea on IRL Talk where Faith said, "I'm not going to buy

00:08:02 ◼   ► a device that purports to have high-quality music." And you're right, fidelity is a better word. It was

00:08:06 ◼   ► pointed out to us by Dr. Drang. We should have been using that word, which could be part of the

00:08:09 ◼   ► communication problem here. But anyway, "I'm not going to buy a device like the Pono because if I

00:08:14 ◼   ► want high-quality, meaning high-fidelity music, I'll just listen to vinyl." And that, I think,

00:08:18 ◼   ► is a clear implication that we're talking about the pono and the whole big deal with that is that

00:08:25 ◼   ► there's more information in the music. And it's like, well, if I want more information in the

00:08:30 ◼   ► music, if I want higher fidelity, if I want a more accurate reproduction of sound, I will go to vinyl.

00:08:35 ◼   ► And you seem to follow down that trail. And my objection and Marco's objection is that vinyl

00:08:40 ◼   ► is not good at reproducing sound and not as good as CD. And all this other stuff I think were great

00:08:45 ◼   ► for us to talk about it, and it's good to define the boundaries of what we believe in everything,

00:08:49 ◼   ► but you should be happy with that. It's not as if you shouldn't cling to that idea,

00:08:56 ◼   ► like that fidelity thing. If that's not what you're talking about, then fine. Then you agree

00:09:00 ◼   ► with us there, and we agree with you over the tea ceremony, and everybody's happy.

00:09:03 ◼   ► One of the things that keeps getting thrown back in is, I like to reframe a lot of these people's

00:09:12 ◼   ► arguments and the popular one is like if I said look if I'm really interested in

00:09:17 ◼   ► high fidelity images I'll go to Instagram like that's the effectively

00:09:22 ◼   ► the argument they're making like you know I don't need a full-frame camera

00:09:26 ◼   ► with a really great sensor and lots of megapixels if I'm interested in

00:09:30 ◼   ► photographic fidelity I will look at a camera from picture of 1992 on Instagram

00:09:35 ◼   ► like that's the argument that many people are making and unknowingly that

00:09:39 ◼   ► they're making, that they're completely combining what they think looks good, which may be like

00:09:43 ◼   ► a super grainy Instagram filter, and the concept of fidelity, which is a straightforward thing

00:09:48 ◼   ► that can be measured and talked about in objective terms, more or less.

00:09:53 ◼   ► And people coming back with, like, "I love tube amps," and everything like that, you

00:09:56 ◼   ► know, instead of digital amps, like, that's not what we're talking about. It's like, whatever

00:09:59 ◼   ► it is that you think sounds good, all we're talking about is, given that sound — and

00:10:03 ◼   ► I probably made a mistake using live instruments because then I had to deal with all the people

00:10:06 ◼   ► talking about live music. Given some sound that you think is awesome, what is the best

00:10:12 ◼   ► way to transport that sound through space and time to reproduce it elsewhere? That's

00:10:16 ◼   ► all we're talking about. It has nothing to do with whether you like tube amps or solid

00:10:20 ◼   ► state amps, whether you like six string guitars or five string, whether you like someone singing

00:10:23 ◼   ► through a paper bag or not. Whatever the sound is, you've produced a sound, a song, live,

00:10:27 ◼   ► recorded, whatever, you want to get that across space and time to someone else. You want to

00:10:31 ◼   ► put it on something so that that sound that you love can get to them exactly as it is.

00:10:35 ◼   ► it have? Whatever that sound is, master it however you want, do whatever you want. How

00:10:38 ◼   ► do you get that sound? CD versus vinyl. That's what we were talking about. Everything else

00:10:43 ◼   ► is like immaterial. I don't care if you like the sound of someone singing through a paper

00:10:48 ◼   ► bag. No, no, now I'm talking about vinyl. I don't care if you like Tom Waits. I don't

00:10:53 ◼   ► care what kind of music you want. We were saying you have something there and again,

00:10:57 ◼   ► live music was easy because it's like, "Oh, I hear that now." You just want to put that

00:11:01 ◼   ► on something, so it could be played back elsewhere. And if your choices are CD and vinyl, and

00:11:05 ◼   ► you care about the fidelity of that reproduction, you should pick CD instead of vinyl. And that's

00:11:10 ◼   ► it. And everything else people want to talk about, it's so hard for them to maintain focus.

00:11:14 ◼   ► So it's hard for us to maintain focus because we have drifting. But it's a difficult topic

00:11:18 ◼   ► to talk about, I think.

00:11:19 ◼   ► Yeah, and I think you're right. And I think you're right in saying that we all agree that

00:11:23 ◼   ► if fidelity is truly the issue, that I can get behind the science, even though I give

00:11:29 ◼   ► it a little bit of side-eye, nevertheless, I can get behind the science that says I'm

00:11:34 ◼   ► crazy, and that digital mediums are the better, higher fidelity method of reproducing music.

00:11:41 ◼   ► But just as you said, I actually happen to prefer the tea ceremony, the emotion behind

00:11:47 ◼   ► it. Like I was talking to my wife Erin about it, and she said, "Well, you know, you have

00:11:49 ◼   ► to consider you grew up with vinyl, and this was something that had—that carries a lot

00:11:53 ◼   ► of emotional baggage." I think John—or one of you guys might have said that last

00:11:56 ◼   ► week. Anyway, it carries a lot of emotional baggage that for most humans, especially in

00:12:01 ◼   ► this day and age, they don't have that emotional baggage. And I say baggage actually in a good

00:12:05 ◼   ► way in this context. I'm sure you probably think of it as a bad thing. But nevertheless,

00:12:09 ◼   ► I don't know what one thing I learned from this, and I'm really being serious is that

00:12:13 ◼   ► it is very frustrating to say something which admittedly is a bit contrarian. And admittedly,

00:12:20 ◼   ► I'd kind of picked this fight. So admittedly, I kind of made this bed for myself. But man,

00:12:26 ◼   ► it frustrating to say something that you truly believe, and then the whole of the internet

00:12:31 ◼   ► decides to come out and tell you how wrong you are?"

00:12:34 ◼   ► And that in and of itself, I can deal with that. That's fine. I've been wrong plenty

00:12:37 ◼   ► of times. I'll be wrong plenty of times again. But it was very frustrating for me

00:12:42 ◼   ► to receive a whole bunch of feedback from people who had perhaps never even heard vinyl

00:12:46 ◼   ► before in their lives, yet decided to take the time out of their day to explain to me

00:12:52 ◼   ► how wrong I am. And I actually want—I'm twisting this not—I don't want that to

00:12:58 ◼   ► sound like a complaint. What I want it to sound like is I learned a little bit that

00:13:03 ◼   ► I think I do that to people sometimes. And I can't think of a great example other than

00:13:07 ◼   ► maybe snickering at somebody pulling out an Android phone, for example, but—

00:13:10 ◼   ► Let me get that for you.

00:13:12 ◼   ► Well, I don't want to be that guy, and I think I am that guy. And so it was a very

00:13:16 ◼   ► good learning experience for me that I need to stop being that guy. Does that make sense?

00:13:21 ◼   ► Yeah, I think I don't know. I mean, you know, we're always gonna get oh god

00:13:26 ◼   ► I have so much more to say about this, but I'm trying to resist

00:13:28 ◼   ► We're always gonna get

00:13:31 ◼   ► You know people who disagree with what we say we always do every week. I mean we get yep

00:13:36 ◼   ► Yeah, you know to give listener some idea we get probably I would say 20 feedback emails a day these days

00:13:44 ◼   ► It's a lot

00:13:45 ◼   ► I'm shocked how much feedback we get

00:13:48 ◼   ► from this show. And I should very quickly interject and say I can't speak for you two,

00:13:52 ◼   ► but I try very hard to read every single piece and at worst

00:13:57 ◼   ► skim every single piece of feedback that we get. Yeah, me too.

00:14:00 ◼   ► And it really does like when you only get like a couple a week, that's easy to keep up with. Now

00:14:07 ◼   ► it's getting hard to keep up with. Like it's getting—we get so much feedback now that it's kind of difficult

00:14:12 ◼   ► because some of them, you know, some of them are a few lines and that's great.

00:14:16 ◼   ► some of them are like six paragraphs, six dense paragraphs at that, and those like,

00:14:21 ◼   ► it takes some serious time to get through some of these. But anyway, you know, I think we are

00:14:27 ◼   ► expressing our opinions in public. Lots of what we say is going to be argued with, that's just the

00:14:33 ◼   ► nature of putting yourself out there. You know, like I like, I mean, I've been putting myself out

00:14:40 ◼   ► there and my often rash or incorrect or badly stated opinions out there online for the better

00:14:48 ◼   ► part of a decade now pretty consistently and whenever I say something that everyone jumps

00:14:53 ◼   ► down my throat for for being totally wrong or off base or misinformed or just not very

00:15:01 ◼   ► well stated I learn something from that and I appreciate that because most people don't

00:15:07 ◼   ► expose themselves to that much criticism. Most people don't have an audience to tell them when

00:15:14 ◼   ► they're wrong and who actually will tell them when they're wrong and an audience that's big

00:15:18 ◼   ► and diverse and smart enough to notice when they're wrong and to be able to express that in any kind

00:15:23 ◼   ► of useful way. So most people tend to be terrible arguers, have narrow worldviews, etc. etc. and

00:15:30 ◼   ► it's hard to get people to change their mind when they've spent their whole life thinking that

00:15:37 ◼   ► that they're right and no one has really ever challenged them on that.

00:15:40 ◼   ► So I'm very thankful that we have the opportunity, which most people don't have.

00:15:45 ◼   ► We as podcasters, and anyone else who's a podcaster or a blogger or any other way that

00:15:49 ◼   ► you can put yourself out there, we are lucky that we have the opportunity to be criticized

00:15:55 ◼   ► by a massive worldwide audience.

00:15:58 ◼   ► Even if your audience is 25 people, that's still 25 random strangers who don't care as

00:16:03 ◼   ► much about how you feel as your family does and who are willing to tell you when you screw

00:16:07 ◼   ► something up. And as the audience gets larger, it hurts more when you screw up because you

00:16:13 ◼   ► have more people telling you that you're an idiot. But I think it is useful. And so,

00:16:20 ◼   ► in your case, as Jon said, in your case, you have, with the vinyl thing, you expressed

00:16:27 ◼   ► valid opinions there. However, you worded them in a way, and you made a generalization

00:16:33 ◼   ► that you couldn't support.

00:16:35 ◼   ► And so the feedback kept beating that

00:16:37 ◼   ► into your head basically.

00:16:39 ◼   ► (laughing)

00:16:40 ◼   ► - And it's true.

00:16:41 ◼   ► - And it's, you know, it certainly was not,

00:16:43 ◼   ► certainly was not probably easy to read everyone saying

00:16:47 ◼   ► that you were wrong, although by the way,

00:16:48 ◼   ► there were a lot of people who agree with you,

00:16:50 ◼   ► I should point out.

00:16:51 ◼   ► But like your main point was that it sounds better

00:16:57 ◼   ► because it sounded better to you.

00:16:59 ◼   ► A solid argument would have been,

00:17:01 ◼   ► you like the sound better.

00:17:03 ◼   ► - Right, and I think I fell on my face

00:17:06 ◼   ► because I painted it, like John was saying,

00:17:07 ◼   ► and like you're saying now,

00:17:09 ◼   ► I painted it as a universal statement

00:17:11 ◼   ► when I really shouldn't have,

00:17:12 ◼   ► that it was factually better,

00:17:14 ◼   ► that it was superior to CDs.

00:17:17 ◼   ► And while I might prefer the experience

00:17:19 ◼   ► and the quirks of the sound,

00:17:21 ◼   ► I don't think that I should have gone as bold as I did.

00:17:25 ◼   ► - And as one of the feedback things,

00:17:28 ◼   ► actually a couple of people did this,

00:17:29 ◼   ► Like, presented, if we could just take that sound

00:17:31 ◼   ► that you love and record it onto CD

00:17:33 ◼   ► and play it back to you in a way that you didn't know

00:17:36 ◼   ► whether we were playing back the vinyl or not,

00:17:38 ◼   ► you would not be able to tell.

00:17:40 ◼   ► Like, we would be exactly accurate

00:17:41 ◼   ► that you're producing that vinyl sound

00:17:43 ◼   ► that you love with every nuance that's available.

00:17:45 ◼   ► You know what I mean?

00:17:47 ◼   ► And that gets to the heart of the matter.

00:17:49 ◼   ► And it's like, we're all for you liking whatever sound you like.

00:17:52 ◼   ► But like, when we talk about mediums,

00:17:55 ◼   ► when we just say the word vinyl, you're not talking about a song

00:17:57 ◼   ► or a sound.

00:17:58 ◼   ► talking about a medium and that's a different topic. There's so many different ways to try

00:18:02 ◼   ► to come at this, to try to get people to understand it, let's just keep going back to

00:18:05 ◼   ► whichever side they're on, whether like "No way, man, vinyl is awesome" or like "Vinyl is always

00:18:09 ◼   ► evil." People just don't want to define the boundaries of what it is that they're saying,

00:18:14 ◼   ► and it's impossible to come to any sort of agreement or disagreement if no one is defining

00:18:19 ◼   ► the boundaries of what they're saying. A lot of, especially the tweet-length ones, which you can't

00:18:23 ◼   ► blame. It's hard to talk about this in a single tweet. I fell victim to that, both on all sides

00:18:29 ◼   ► of the debate. The best ones where they say—I like tweets on any topic where they'll say,

00:18:33 ◼   ► "I agree with Casey about this. I agree with Marco about this. I disagree with John about this."

00:18:37 ◼   ► And they will ascribe to all three of us opinions that 50 percent of the time we didn't take.

00:18:43 ◼   ► "I agree with Marco that Starbucks coffee is great. I agree with Casey that all cars should

00:18:49 ◼   ► should be black and I agree with John, the pearl is a terrible language, you know, that's

00:18:52 ◼   ► a silly example, but it's like, you put our names in a tweet and you've expressed your

00:18:57 ◼   ► own opinions, but you've by implication described the opposite opinions to us and I don't think

00:19:02 ◼   ► we ever, anyway, it's the excitement of feedback. I agree with Marco, getting this feedback,

00:19:07 ◼   ► as I've said many times, is very valuable, we're lucky to get a lot of it. You too can

00:19:12 ◼   ► get a lot of it as well if you just go to online communities and whatever topic you're

00:19:15 ◼   ► interested in and just start, you know, like I said, a Usenet in my past where it wasn't

00:19:20 ◼   ► as big as audiences we have here. But it doesn't take, you know, it only takes like four or

00:19:23 ◼   ► five mean people to put you in your place for a couple of years and you'll get better.

00:19:27 ◼   ► Great.

00:19:28 ◼   ► Yeah, and again, I don't mean to complain necessarily about getting feedback. I don't

00:19:33 ◼   ► mean to complain about all of the internet or most of the internet thinking I'm wrong.

00:19:38 ◼   ► That actually doesn't bother me. What was frustrating was people who really didn't have,

00:19:44 ◼   ► To be blunt, anything that constructive to add, just coming out of the woodwork and saying,

00:19:48 ◼   ► "Eh, you're wrong."

00:19:49 ◼   ► And, "You're wrong because I read something once."

00:19:52 ◼   ► That's just not helpful.

00:19:53 ◼   ► Whereas something that Mark did, which he said, "Okay, from personal experience, let

00:19:57 ◼   ► me take you through why it is you're wrong."

00:20:01 ◼   ► That is infinitely valuable and extraordinarily valuable, and I'm very thankful for that.

00:20:05 ◼   ► But the other people are expressing their enthusiasm, though.

00:20:08 ◼   ► Even if they're doing it in a way that makes you feel bad, they're saying, "Hey, we listened

00:20:13 ◼   ► to the show, and b) we're enthusiastic enough about it to seek you out on Twitter and send

00:20:17 ◼   ► you something. So there is even good in that, I think.

00:20:19 ◼   ► Yeah, I agree. And that's why even on the emails that I want to reach through the computer

00:20:24 ◼   ► and backhand whoever wrote them, even on those emails, I try always to force myself to end

00:20:30 ◼   ► the email in such a way that when I write, "Thank you for listening and thanks for your

00:20:35 ◼   ► feedback," that it's taken as genuine, because I do mean it genuinely, even if I want to

00:20:40 ◼   ► freaking kill whoever it is I just replied to. So I don't know, maybe this is too inside

00:20:45 ◼   ► baseball, this might be boring, maybe we'll cut it, I don't know, but it just, the point

00:20:49 ◼   ► I was trying to get to was that it made me look at myself with a bit of a critical eye,

00:20:55 ◼   ► which again is even more reason why even the feedback that I'm bemoaning is actually valuable

00:21:01 ◼   ► in its own way.

00:21:03 ◼   ► And to point out also, I actually, so I've been in this high-end audio world for a lot

00:21:10 ◼   ► recently and I've found that, so the reason why I don't like vinyl and a lot of analog

00:21:17 ◼   ► technologies, the same reason why I briefly had a chance to have a tube amp in my house

00:21:23 ◼   ► for a couple of weeks to try out and I didn't like it and I ended up not buying one, the

00:21:29 ◼   ► reason why people like vinyl sound, not everyone, but the reason why the people who do like

00:21:34 ◼   ► it like it, in addition to all the things we talked about last week about the tea ceremony

00:21:38 ◼   ► and the romanticism of it and how hip and awesome it looks and everything is slow and

00:21:43 ◼   ► artisanal and manual, in addition to all that crap, it does sound different. And it sounds

00:21:51 ◼   ► different in various ways that people find pleasing. Now, as, but you know, that wasn't

00:21:58 ◼   ► the argument that John was making against you.

00:22:00 ◼   ► However, that was what most people thought

00:22:03 ◼   ► when, you know, for the argument about vinyl

00:22:05 ◼   ► sounding better than CD or whatever.

00:22:07 ◼   ► You know, in my own purchases, and what I found

00:22:11 ◼   ► in my experiences with high-end audio stuff,

00:22:14 ◼   ► what I like is neutrality in most cases.

00:22:20 ◼   ► So, for instance, I don't use a tube amp

00:22:22 ◼   ► because I use sensitive headphones,

00:22:24 ◼   ► and sensitive headphones, you can very easily

00:22:26 ◼   ► pick up the noise introduced by tube amps. You can also occasionally get

00:22:30 ◼   ► a little pop if a piece of dust hits the wrong way or something like that.

00:22:34 ◼   ► It's almost like records.

00:22:38 ◼   ► It's almost like vinyl. You're introducing distortion.

00:22:42 ◼   ► You're introducing noise. So I try to get very

00:22:46 ◼   ► nice digital stuff. So everything is solid state. There's no moving parts

00:22:50 ◼   ► except the volume knob. There is

00:22:54 ◼   ► as little noise introduced as possible,

00:22:57 ◼   ► and there is as little tonal imbalance

00:23:00 ◼   ► introduced as possible.

00:23:02 ◼   ► And I actually took it a little too far.

00:23:05 ◼   ► So my current setup is I have the Sennheiser HD 800 headphones

00:23:10 ◼   ► plugged into the shit Asgard 2, that is actually its name,

00:23:15 ◼   ► and the shit Bifrost DAC.

00:23:19 ◼   ► And it looks amazing,

00:23:21 ◼   ► these things all look fancy and awesome,

00:23:23 ◼   ► They sound great.

00:23:24 ◼   ► The main reason I like the shit amp and DAC

00:23:28 ◼   ► is because they have like zero noise

00:23:30 ◼   ► that I can hear on anything that I have.

00:23:31 ◼   ► - It's not because they're the shit.

00:23:33 ◼   ► - And the name helps too, and they look pretty awesome,

00:23:35 ◼   ► and they're small and they were reasonably priced.

00:23:37 ◼   ► And I like it because it's super neutral.

00:23:41 ◼   ► Now, the Sennheiser HD 800 headphones,

00:23:44 ◼   ► the best thing I can say about them

00:23:45 ◼   ► is that they are the most neutral headphones I've ever heard.

00:23:48 ◼   ► The downside of this is that they actually sound

00:23:51 ◼   ► little bit boring on certain things. So like certain types of music, I actually like the

00:23:57 ◼   ► distortion by things like amplifying the high end that you hear on some biodynamic models

00:24:03 ◼   ► or the bigger bass I have on my closed headphones, the TH900s. And certain things I like that

00:24:10 ◼   ► distortion. Now throughout all my research in finding these headphones, I tried a few

00:24:16 ◼   ► other models I ended up returning or selling in the meantime, and I found that people's

00:24:20 ◼   ► opinions very dramatically on what they considered sounding good. So one of the most popular

00:24:26 ◼   ► lines, the Sennheiser HD600 and 650, those actually have pretty muted high end. And I

00:24:33 ◼   ► didn't like that sound. A lot of people loved it. And similarly, certain people don't like

00:24:40 ◼   ► the biodynamic super strong high end. And I like it. And what I found is that keeping

00:24:47 ◼   ► everything neutral up until the point of the actual headphones allowing me to make that

00:24:51 ◼   ► choice gives me the flexibility. Like, you can simulate all the other, you can simulate

00:24:55 ◼   ► vinyl with digital plugins and equalizers and stuff. Please email Dan. You can simulate

00:25:03 ◼   ► most of the effects of what people like about these things electronically and artificially

00:25:09 ◼   ► if you want to. Just like Instagram, you know, can simulate the distortions of certain old

00:25:17 ◼   ► processes or things that, or like idealized or enhanced versions of old processes and

00:25:21 ◼   ► old distortions that used to exist in the analog world. And that produces more pleasing-looking

00:25:25 ◼   ► photos just to a lot of people. So it's important to recognize, like, you know, I like everything

00:25:32 ◼   ► to be digital and pure and clean and neutral up until the point where I have my headphones

00:25:36 ◼   ► where I can make that choice and say either I want to listen to something with a lot of

00:25:39 ◼   ► bass with this pair of headphones, or I want to listen to something that's neutral with

00:25:42 ◼   ► this pair of headphones. And I recognize that's a crazy setup, however, that's how I am and

00:25:46 ◼   ► that's how I did it. But none of that, like the argument John was making last week, is

00:25:51 ◼   ► that CDs are better at being neutral. They're better at accurately and neutrally replicating

00:25:58 ◼   ► the recording. And that is correct. That is 100% right. And most people, though, when

00:26:06 ◼   ► they're arguing about what sounds better or worse or what is better or worse, they don't

00:26:11 ◼   ► put it in those terms of personal preference

00:26:13 ◼   ► or of like the difference between neutrality

00:26:16 ◼   ► and pleasing sound.

00:26:18 ◼   ► 'Cause those are very different things.

00:26:20 ◼   ► Perfectly neutral sounds boring to a lot of people.

00:26:23 ◼   ► Like when I edit our podcast,

00:26:24 ◼   ► I edit with perfectly neutral.

00:26:26 ◼   ► However, when I listen to fish,

00:26:28 ◼   ► I like a little bit more bass.

00:26:29 ◼   ► So when I listen to fish, I put on the other headphones.

00:26:31 ◼   ► You know, it's very important to distinguish those things.

00:26:34 ◼   ► Personal preference or more pleasing

00:26:37 ◼   ► versus accurate and neutral.

00:26:39 ◼   ► - Yeah, I think you're right.

00:26:41 ◼   ► but I don't know, this is probably getting a bit old now.

00:26:45 ◼   ► Why don't you tell us about something that's cool?

00:26:46 ◼   ► - I would love to.

00:26:47 ◼   ► It's our friends at PDF Pen.

00:26:49 ◼   ► Once again, this is PDF Pen for iPad.

00:26:53 ◼   ► PDF Pen for iPad lets you edit PDFs anywhere you are,

00:26:56 ◼   ► as long as you have an iPad with you

00:26:57 ◼   ► that happens to be running PDF Pen.

00:26:58 ◼   ► However, anywhere you are with one of those things,

00:27:01 ◼   ► you can edit PDFs.

00:27:02 ◼   ► You can sign contracts, you can fix a typo,

00:27:05 ◼   ► you can actually edit the PDF right there on your device.

00:27:08 ◼   ► You can correct a price list,

00:27:09 ◼   ► You can fill out a form while you're on the go.

00:27:12 ◼   ► And you can take PDF documents with you anytime,

00:27:15 ◼   ► add notes, highlighting, add markup

00:27:17 ◼   ► during your mobile downtime.

00:27:19 ◼   ► You can sync PDFs with PDF pen for OS X

00:27:22 ◼   ► using iCloud or Dropbox,

00:27:24 ◼   ► 'cause PDF pen is for OS X and iPad.

00:27:27 ◼   ► You can grab and save PDFs using Dropbox,

00:27:29 ◼   ► Evernote, Google Drive, Box.

00:27:31 ◼   ► I guess it isn't called box.net anymore.

00:27:33 ◼   ► Is that right?

00:27:34 ◼   ► They dropped the .net, now it's just Box.

00:27:36 ◼   ► Anyway, you can save PDFs to that or from that.

00:27:39 ◼   ► even webDAV and FTP servers.

00:27:42 ◼   ► And anywhere you are with PDF/PEN for iPad,

00:27:45 ◼   ► you have the complete feature-rich

00:27:46 ◼   ► mobile editing power of PDF/PEN.

00:27:49 ◼   ► It's available in the App Store.

00:27:50 ◼   ► Now, a lot of apps these days,

00:27:52 ◼   ► I've been meaning to talk about this,

00:27:53 ◼   ► a lot of apps these days

00:27:54 ◼   ► haven't even been updated for iOS 7 yet.

00:27:56 ◼   ► They're not native, they're kind of ignored.

00:27:59 ◼   ► PDF/PEN is awesome, it is always updated,

00:28:01 ◼   ► it is native to iOS 7,

00:28:03 ◼   ► and there's tons of,

00:28:05 ◼   ► they're always making tons of improvements,

00:28:07 ◼   ► performance improvements, everything else,

00:28:09 ◼   ► they're fantastic. And it syncs all your stuff, once again. They even support a few of the

00:28:15 ◼   ► pressure sensitive pens on the market, the most popular ones. They support the Jot Touch,

00:28:20 ◼   ► the Pogo Connect, and the Jaja or Haja, I'm not sure which one that is, but it's Jaja

00:28:25 ◼   ► that one, which is either Jaja or Haja or maybe Jaja or Haja, who knows. But they support

00:28:30 ◼   ► that one. So check out PDF pen for iPad, go to Smilesoftware.com/ATP, that is Smilesoftware.com/ATP

00:28:38 ◼   ► is smilesoftware.com/ATP. Thanks a lot to PDF/Pen for iPad for sponsoring our show once

00:28:44 ◼   ► again. I'll tell you, I use PDF/Pen and it's fantastic. I use it more on the Mac than the

00:28:50 ◼   ► iPad just because that's where I am most of the time. But it's an amazing piece of software

00:28:53 ◼   ► and it saves my butt pretty frequently. So thanks a lot to PDF/Pen for iPad once again.

00:28:59 ◼   ► We didn't get a chance to talk about last week because we were too busy arguing. We

00:29:05 ◼   ► We didn't get a chance to talk about this trial with Samsung, between Samsung and Apple.

00:29:10 ◼   ► And there's been some interesting things coming out of this, and we've gotten to

00:29:15 ◼   ► look at some of the behind-the-scenes at how Apple does stuff, how they get their jobs

00:29:21 ◼   ► done as a result of a lot of the discovery during the trial.

00:29:26 ◼   ► So I don't know, Jon, do you have any thoughts on this?

00:29:28 ◼   ► Yeah, I picked out in particular not so much the details of the trial or whatever, because

00:29:33 ◼   ► Doesn't this one have to do with patents, too?

00:29:35 ◼   ► I don't even pay attention, but you know how I feel about--

00:29:36 ◼   ► I think so.

00:29:37 ◼   ► --about patents or whatever.

00:29:38 ◼   ► But yeah, because as part of the discovery process,

00:29:41 ◼   ► or some other legal term that I don't know,

00:29:44 ◼   ► they're revealing internal documents from Apple.

00:29:47 ◼   ► And it's great because Apple is-- well, it's great for us,

00:29:50 ◼   ► anyway.

00:29:50 ◼   ► I don't know if it's great for Apple.

00:29:51 ◼   ► That's all the discussion.

00:29:52 ◼   ► But we get so little insight into what goes on in Apple,

00:29:56 ◼   ► we just see their public face.

00:29:57 ◼   ► And their public face is so incredibly controlled.

00:29:59 ◼   ► And I think it's even more controlled now

00:30:01 ◼   ► than it used to be, because as we've said many times before,

00:30:04 ◼   ► in the good old days with Steve Jobs,

00:30:06 ◼   ► yes, Apple's message was controlled,

00:30:07 ◼   ► but Steve Jobs had the ability and often took it

00:30:11 ◼   ► to just go off book and do whatever the hell he wants

00:30:14 ◼   ► and say whatever the hell he wants in whatever way he wants.

00:30:16 ◼   ► And I'm sure he was controlled and calculating as well,

00:30:19 ◼   ► but you never quite knew what he was gonna say.

00:30:20 ◼   ► Tim Cook, you're not gonna make him slip up

00:30:23 ◼   ► and say something.

00:30:23 ◼   ► You're not gonna get something exciting.

00:30:25 ◼   ► So far, it seems like he is very on message

00:30:29 ◼   ► and very controlled, strategically leaking things

00:30:31 ◼   ► and saying things in a certain way

00:30:32 ◼   ► and choosing his words very carefully.

00:30:34 ◼   ► But with Steve Jobs, he felt like there was a chance

00:30:36 ◼   ► he might either act like he's just suddenly caught up

00:30:40 ◼   ► in the moment and saying something spontaneous

00:30:42 ◼   ► or actually do that.

00:30:44 ◼   ► But anyway, these documents are internal Apple documents.

00:30:47 ◼   ► And from the outside, a lot of people say Apple is arrogant,

00:30:50 ◼   ► Apple's out of touch, they don't know what their users

00:30:51 ◼   ► really want or whatever.

00:30:53 ◼   ► But these internal documents make it very, very clear

00:30:55 ◼   ► that that is not the case, that it is merely

00:30:57 ◼   ► their external face that they control so well, but internally.

00:31:01 ◼   ► This is the fiscal year 2014 planning off-site slide deck, and you can see some of these

00:31:07 ◼   ► slides.

00:31:08 ◼   ► If you haven't seen them, we'll put the link in the show notes.

00:31:09 ◼   ► You should check them out, but I'll read off some of the slides here.

00:31:12 ◼   ► So the first slide says, they're talking about iPhone sales, and it says, "Growth rates are

00:31:16 ◼   ► slowing," and they show a graph that shows growth rates going down, down, down, down,

00:31:20 ◼   ► down.

00:31:21 ◼   ► Apple, of course, would never show a slide like this in public, because why are they

00:31:24 ◼   ► going to show something that puts them in a bad light.

00:31:29 ◼   ► And again, something as simple as this--

00:31:31 ◼   ► showing growth rate slowing.

00:31:33 ◼   ► Yes, you'll see stories about Apple about that.

00:31:34 ◼   ► And if someone actually gets in touch with Apple

00:31:36 ◼   ► and asks them about that, they'll

00:31:37 ◼   ► say some blah, blah, blah kind of, well,

00:31:40 ◼   ► we think we're on the right track, blah, blah, blah.

00:31:43 ◼   ► They'll say some stupid PR-y thing.

00:31:44 ◼   ► But here they are saying-- when they're talking to themselves,

00:31:46 ◼   ► they're like, here's the graph.

00:31:48 ◼   ► Growth rate is really slowing.

00:31:49 ◼   ► Next slide, they say, so what's going on?

00:31:51 ◼   ► And they have little sections here that says,

00:31:53 ◼   ► Strongest demand is coming from less expensive and larger screen smartphones.

00:31:57 ◼   ► Carriers have strong interest in capping iPhone due to one or more factors.

00:32:01 ◼   ► High share, the subsidy premium, unfriendly policy, unfriendly is in quotes, lack of alignment.

00:32:06 ◼   ► So they're basically saying, customers want cheaper phones with bigger screens.

00:32:10 ◼   ► Carriers, carriers don't like us that much and they don't like us for good reasons because

00:32:13 ◼   ► we are more demanding of them because we, you know, these unfriendly policies that we have,

00:32:17 ◼   ► our subsidy is really high, lack of alignment with, you know,

00:32:22 ◼   ► our interests aren't aligned with the carriers. Everybody's laying out, yeah, carriers don't like

00:32:27 ◼   ► us that much, and competitors, and they show the little Android icon. Competitors have drastically

00:32:31 ◼   ► improved their hardware and software and their ecosystems, and spending obscene, also in scare

00:32:37 ◼   ► quotes, amounts of money on advertising and carrier channel to gain traction, which is

00:32:41 ◼   ► a Samsung thing. So they're laying out like, "Here are our problems. We're kind of getting our butt

00:32:46 ◼   ► kicked in these areas. Here's what people want." And this is the title of the slide. It's great.

00:32:50 ◼   ► Next slide. Consumers want what we don't have. If you can imagine Apple saying that in public,

00:32:56 ◼   ► what we don't have. And so they show the market share growth and they say, "Where did the growth

00:33:02 ◼   ► come from?" And this, I wish there was a better breakdown and maybe there was, but they're saying,

00:33:07 ◼   ► "Where did the growth in the industry come from?" And it shows the growth and it says,

00:33:10 ◼   ► I can't do the percentages here, but more than half the growth came from phones that are less

00:33:16 ◼   ► than $300 and the rest of the growth came from screens that are more than $300 but have

00:33:21 ◼   ► screens that are bigger than 4 inches.

00:33:24 ◼   ► The objection I have to this graph is, "Okay, great, you've identified that the majority

00:33:28 ◼   ► of the growth is coming from phones that are less than $300.

00:33:31 ◼   ► I think you need to break that down more because if you take that piece and say, 'Okay, well,

00:33:37 ◼   ► out of that growth, actually 99% of that growth is coming from phones that are less than $5,'

00:33:42 ◼   ► That is way different than saying, "Okay, well, most of it's coming from--how much less

00:33:46 ◼   ► than $300?

00:33:47 ◼   ► Break it down.

00:33:48 ◼   ► Is it, you know, what's the average?

00:33:50 ◼   ► What's the distribution in there?"

00:33:52 ◼   ► Without knowing that, you don't know what to do.

00:33:53 ◼   ► But anyway, this is a high-level slide, so I assume they have broken it down internally.

00:33:57 ◼   ► I just, again, I would like to see that broken down.

00:33:59 ◼   ► I didn't.

00:34:01 ◼   ► And all this reminded me of a book I'm reading now, which I would recommend to everybody,

00:34:04 ◼   ► and we'll put a link in the show notes.

00:34:06 ◼   ► It's Ed Catmull's book called "Creativity, Inc."

00:34:10 ◼   ► It's not really a memoir or manifesto.

00:34:13 ◼   ► It's more of just an explanation of the early history of Pixar

00:34:16 ◼   ► and what Ed Catmullen and the rest of the company

00:34:19 ◼   ► has done to make that organization function

00:34:22 ◼   ► and be successful.

00:34:23 ◼   ► And a big focus on the book is being clear-eyed

00:34:27 ◼   ► about your own problems and candid about them internally.

00:34:31 ◼   ► And I know Pixar's not Apple, and I

00:34:32 ◼   ► know Steve Jobs didn't have nearly as much involvement

00:34:35 ◼   ► in Pixar as he did in Apple.

00:34:37 ◼   ► but I can't help but see parallels

00:34:39 ◼   ► between the things that are in this book

00:34:43 ◼   ► and the approach Apple is taking here.

00:34:45 ◼   ► Apple is very clear-eyed in these slides.

00:34:47 ◼   ► They are not sugarcoating it.

00:34:48 ◼   ► They are not like, "Rah, rah, rah,"

00:34:49 ◼   ► like, "Yeah, we may have problems,

00:34:51 ◼   ► but we still have the best phone in this office."

00:34:54 ◼   ► The slides are just, you know, there's not a lot of them,

00:34:56 ◼   ► but they are essentially unrelentingly negative

00:34:58 ◼   ► towards Apple, and that's how they have to be.

00:35:00 ◼   ► That's how Apple improves.

00:35:02 ◼   ► So I felt better about seeing these slides,

00:35:05 ◼   ► and it made me think that at least some

00:35:06 ◼   ► of the culture and strategies described

00:35:10 ◼   ► in that excellent book by Ed Catmull

00:35:12 ◼   ► are alive and well in Apple as well.

00:35:14 ◼   ► - Yeah, I think it's fantastic to see

00:35:19 ◼   ► that Apple knows its own problems,

00:35:21 ◼   ► it knows its own shortcomings.

00:35:23 ◼   ► But I think that's not really that much news necessarily.

00:35:28 ◼   ► To me, seeing all these trial documents

00:35:32 ◼   ► and the emails of Phil Schiller and everything,

00:35:35 ◼   ► It really just says to me that this company is,

00:35:39 ◼   ► by being as private as they usually are,

00:35:42 ◼   ► they are saving us from the boredom

00:35:44 ◼   ► and the relative routineness

00:35:48 ◼   ► that's actually in their company.

00:35:51 ◼   ► These are like, the only big surprise here

00:35:53 ◼   ► is that these people are actually acting somewhat human,

00:35:56 ◼   ► but they're all, they're saying not that surprising things.

00:36:00 ◼   ► They're sending boring business emails

00:36:01 ◼   ► to their advertising agency

00:36:02 ◼   ► that doesn't even capitalize anything,

00:36:04 ◼   ► which is apparently a thing.

00:36:06 ◼   ► - Well, that was a whole other thing.

00:36:07 ◼   ► Like, this article was supposed to be sensational or whatever

00:36:11 ◼   ► and so one of the things they showed was,

00:36:12 ◼   ► look at this email where Phil Schiller's

00:36:14 ◼   ► being angry to his ad agency.

00:36:15 ◼   ► What I took away from that exchange was

00:36:18 ◼   ► how incredibly low the bar is for Apple's ad agency.

00:36:22 ◼   ► Like, you feel like, this is your ad agency?

00:36:24 ◼   ► You're the biggest company in the world

00:36:26 ◼   ► and these jokers are doing your ads?

00:36:28 ◼   ► Boy, that's just depressing.

00:36:30 ◼   ► Not everything can be up to the standard

00:36:33 ◼   ► of Apple and Pixar, I guess,

00:36:34 ◼   ► and it's just, yeah.

00:36:36 ◼   ► - Yeah, I mean, what I really just found

00:36:38 ◼   ► with these documents is like,

00:36:40 ◼   ► wow, this is all really boring stuff that,

00:36:44 ◼   ► like there's very little of newsworthiness here

00:36:48 ◼   ► except that Apple usually does a really good job

00:36:51 ◼   ► of keeping what comes out interesting.

00:36:55 ◼   ► Keeping their public image interesting

00:36:57 ◼   ► and secretive and well edited.

00:37:01 ◼   ► And you see now on the inside that,

00:37:03 ◼   ► yeah, it's a big company, dealing with big company problems

00:37:06 ◼   ► and doing things that big companies do.

00:37:09 ◼   ► - It might not seem that novel to you,

00:37:10 ◼   ► but I compare it to, I mean,

00:37:13 ◼   ► I work for a lot of different companies,

00:37:14 ◼   ► but the one people may have heard of

00:37:15 ◼   ► is that for a time I worked for Palm,

00:37:17 ◼   ► and I wasn't very close to the leadership

00:37:20 ◼   ► of that organization, but internal to the company,

00:37:22 ◼   ► when I worked for Palm, I got presentations from,

00:37:25 ◼   ► you know, some of the big wigs came to visit us

00:37:27 ◼   ► 'cause we were in a different location,

00:37:28 ◼   ► and gave us a presentation on the state of Palm

00:37:30 ◼   ► and what their plans were for the business.

00:37:32 ◼   ► and their slides were not candid and didn't accurately identify the problems they were facing.

00:37:36 ◼   ► They were much more kind of like, "We're actually still doing pretty good in the PDA space, and the

00:37:41 ◼   ► phones are coming on, but we've got these trio things." And it was like, even to me inside the

00:37:46 ◼   ► company, I'm like, "You guys are in denial. If these are the slides, we're showing this

00:37:49 ◼   ► internally. If we are showing these slides to ourselves internally, we are not being honest

00:37:53 ◼   ► with ourselves about the state." I mean, you would hope that they would, you know, we saw it and we

00:37:58 ◼   ► felt it, maybe they felt it, it just didn't feel as honest and straightforward. And like,

00:38:02 ◼   ► if you if you don't think these Apple slides are novel or interesting, like, I don't have that much

00:38:08 ◼   ► experience in big tech companies, but I have seen it from big and small. Most companies are not this

00:38:13 ◼   ► candid with themselves and are not are not this accurate, like, you know, are not able to exactly

00:38:18 ◼   ► pinpoint where their weak points are, and nail them in that way. And I guess that shows, you know,

00:38:23 ◼   ► confidence internally and skill. And if they weren't that skill, they wouldn't be where they

00:38:26 ◼   ► are today. So I think there is something to it. It's not—we shouldn't just accept that

00:38:30 ◼   ► every company does this, because I think many of them don't.

00:38:34 ◼   ► The other thing that we didn't talk about last week, which we could have, is some guy

00:38:39 ◼   ► that apparently I was supposed to know but didn't know is leaving Apple. And that guy

00:38:44 ◼   ► is Greg Christie. And had either of you two heard of him before? I'm assuming, John, that

00:38:49 ◼   ► you had. But Marco, have you heard of this gentleman before last week?

00:38:53 ◼   ► Nope. I still don't even believe this is really a story.

00:38:56 ◼   ► Well, there were stories about him. He was featured in recent stories before the "he was

00:39:01 ◼   ► leaving Apple" story. So even if you weren't aware of him before that, you might have said,

00:39:04 ◼   ► "Did I not see that name before?" And it's like, yes. Like two weeks ago,

00:39:07 ◼   ► you did see that name. I think someone did a story about the early days of the iPhone,

00:39:11 ◼   ► and he was heavily featured. So that's why you might be aware of him.

00:39:14 ◼   ► Yeah, this looks a lot like kind of a gossip story, like people arguing inside Apple,

00:39:19 ◼   ► and then one of them leaves. I mean, that happens all the time in every company.

00:39:23 ◼   ► And then, you know, from some people have—we don't know what's really going on here, but

00:39:28 ◼   ► there's at least one reasonable denial from Matt Panzareno that the details of this big breakup

00:39:33 ◼   ► are not as dramatic as they seem. But the reason I think it's interesting is forget about how he

00:39:39 ◼   ► left or why he left or whatever. Like, who cares, right? What it highlights, what all these stories

00:39:46 ◼   ► highlight is it's time for them to talk about kind of, "Oh, Johnny Ive's organization," because

00:39:51 ◼   ► he is in charge of much more than he used to be in charge of. Anything that happens

00:39:56 ◼   ► in that organization, people coming, people going, or whatever, with him remaining at

00:40:00 ◼   ► the top, in some ways, like, not that it reflects on him, but it makes us focus on him. So what

00:40:05 ◼   ► these stories made me do is think more about how Johnny Ive is doing in his position. And

00:40:11 ◼   ► yes, part of his position is retention or whatever, but not so much in this particular

00:40:15 ◼   ► thing, but what it got me thinking about was what are Johnny Ive's weaknesses in his new

00:40:21 ◼   ► role as software designer. I've had decades to think about what his weaknesses are in

00:40:25 ◼   ► hardware design, and previously it was unknown what is he going to be like in software, but

00:40:30 ◼   ► his organization, I guess, you know, iOS 7 is the first big product of his software organization,

00:40:35 ◼   ► and we've all had a long time to live with it, and I think his weaknesses as a software

00:40:39 ◼   ► designer are more glaring and obvious than they are in hardware, maybe, and in fact they're

00:40:46 ◼   ► very similar. And so anytime anything happens within that organization, it makes me sort

00:40:51 ◼   ► of like dwell more on how he's doing in his new role as software designer. And I think that's a

00:41:00 ◼   ► useful avenue of thought for me. So are you not pleased with how things are going then with Johnny

00:41:06 ◼   ► at the helm? Well, like when somebody leaves like that, even if it's on good terms, bad terms,

00:41:12 ◼   ► or whatever, like you have to think that like it's not I that's leaving, right? People under him are

00:41:17 ◼   ► are going to leave.

00:41:18 ◼   ► And you have to think, maybe there was a--

00:41:20 ◼   ► it's someone who used to be more in charge than they are now.

00:41:24 ◼   ► He was largely responsible for a lot of the early iPhone UI.

00:41:28 ◼   ► And surely his power to make changes and decisions

00:41:31 ◼   ► has decreased now that Johnny Ive has come in above him.

00:41:34 ◼   ► And you know what I mean?

00:41:35 ◼   ► That's the whole thing with the reorg.

00:41:36 ◼   ► Johnny Ive has been given much more power.

00:41:39 ◼   ► And that means the people who used to have the power

00:41:41 ◼   ► have had it taken away.

00:41:42 ◼   ► And if there's any sort of disagreement about what to do,

00:41:45 ◼   ► Johnny Ives is going to win, as he should, because he's in charge.

00:41:48 ◼   ► And what that means is that whatever his strengths and weaknesses are,

00:41:51 ◼   ► are imprinted even more strongly on the organization,

00:41:54 ◼   ► because he is not just one voice anymore, he is the voice.

00:41:57 ◼   ► As I think I tweeted at one point, he's in full Shigeru Miyamoto mode,

00:42:01 ◼   ► where Miyamoto designed Nintendo's flagship games,

00:42:04 ◼   ► like Mario and Donkey Kong and many other things,

00:42:07 ◼   ► and slowly climbed the ranks of the organization until essentially he was

00:42:10 ◼   ► the buck stop with him for all issues related to software design

00:42:14 ◼   ► in all of Nintendo and possibly several aspects of hardware design as well.

00:42:20 ◼   ► Once you are in that position of power, whatever your strengths or weaknesses are are magnified.

00:42:25 ◼   ► I mean, it was true with Steve Jobs.

00:42:27 ◼   ► His strengths and weaknesses were magnified through the whole organization.

00:42:30 ◼   ► So now, that's why I'm dwelling on what I was like as a software designer, because these

00:42:33 ◼   ► other voices, whether you agree with them or not, Scott Foslall, Greg Christie, all

00:42:37 ◼   ► these other people, their voices and their ability to affect change within the organization

00:42:43 ◼   ► are necessarily modified by the larger and larger footprint of this one fallible individual,

00:42:49 ◼   ► for better or for worse.

00:42:50 ◼   ► And so that's what I've got my eye on.

00:42:51 ◼   ► Yeah, that's interesting.

00:42:53 ◼   ► I mean, the problem is, I think you're right that that is something to watch, but I think

00:43:00 ◼   ► everything we've heard about Greg Christie's particular departure makes it sound like maybe

00:43:08 ◼   ► the problem had nothing to do with Jonny Ive.

00:43:11 ◼   ► And we see the crazy rumor sites saying there was some kind of friction there, and then

00:43:17 ◼   ► you see people like John Gruber, who have generally better sources on these kind of

00:43:20 ◼   ► things, saying, "Oh, this was actually in the works for months, and this wasn't even

00:43:27 ◼   ► recent, and there was no bad blood, he just was retiring."

00:43:31 ◼   ► And as the story gets older, we're hearing more that it's not a story than the very

00:43:39 ◼   ► minute that it broke in the news that said it was a story. So that's probably the reality,

00:43:44 ◼   ► it probably is not a story in itself. Now, again, you're right that this is something,

00:43:47 ◼   ► you know, Johnny Ive having a lot more power than he used to and becoming so powerful that

00:43:53 ◼   ► he's able to get bad decisions through, that is something to watch, no question. And iOS

00:44:00 ◼   ► 7 is a great example of some of those things. You know, I think iOS 7 really showed that

00:44:07 ◼   ► Johnny Ive was wanting to make a big change in a lot of very important stuff on iOS and

00:44:14 ◼   ► he did and he went a little too far in a lot of those changes.

00:44:17 ◼   ► But even, as we saw last summer, even from beta 1 to what was released as 7.0, there

00:44:24 ◼   ► were a lot of changes to reign some of that back.

00:44:27 ◼   ► The biggest one being the super thin font was replaced with a more reasonably thick,

00:44:31 ◼   ► thickness one, more reasonably weighted one.

00:44:36 ◼   ► like that, and the lock screen having modifications to make it easier to realize that you have

00:44:41 ◼   ► to swipe right and not up, things like that, there were a lot of changes made. He went

00:44:51 ◼   ► too far and then he dialed it back. I think iOS 7 is great. I think the design of it is

00:44:57 ◼   ► great. It is not perfect, but it is great. It is a massive improvement over 6. This is

00:45:04 ◼   ► going to get into what we want from iOS 8, I guess, but I don't know if we even have

00:45:09 ◼   ► that as a topic. But, again, I think this is something to watch for, you know, watch

00:45:13 ◼   ► for Johnny Ive getting enough power that he can get bad decisions pushed through, and

00:45:19 ◼   ► for him to think that these decisions are good. You know, those things are a problem

00:45:23 ◼   ► together. But other than that, I don't really think this is much of a story.

00:45:27 ◼   ► Like on the spin of the story, what I was getting at is it doesn't matter how or why

00:45:31 ◼   ► Greg Christie left, it only matters that he was somebody who was there for a long time,

00:45:36 ◼   ► who was responsible for a lot of, like, he was a senior person, right? He wasn't just

00:45:40 ◼   ► some other person. He was another voice, whether he agreed with Johnny Ive or not at any point.

00:45:46 ◼   ► If you were like the last, the last, you know, Godzilla standing, like all the other big

00:45:50 ◼   ► guys have left, all the other people who were there from the beginning, who strongly influenced

00:45:55 ◼   ► the original iPhone, you know what I mean? Like, that's what I was saying about, like,

00:45:59 ◼   ► Suguro Miyamoto, you need to have those voices that you will listen to because you believe

00:46:05 ◼   ► that they are valuable and have experience.

00:46:07 ◼   ► And even if you just left because you just felt like retiring, that's a loss.

00:46:12 ◼   ► That's somebody that you don't have in the organization with a wealth of experience for

00:46:15 ◼   ► you to bounce your ideas off of.

00:46:16 ◼   ► You know what I mean?

00:46:17 ◼   ► I don't want to get back to the Ed Catmull book again, but they have this whole chapter

00:46:20 ◼   ► on the brain trust, which maybe we'll put a link to this because I think they did an

00:46:23 ◼   ► excerpt on some website somewhere, where they will get the people, the oldest and wisest

00:46:29 ◼   ► people together in a room and have them all tell each other what's wrong with their projects,

00:46:34 ◼   ► just to bounce ideas over to them, because they all respect each other and they all have a lot

00:46:38 ◼   ► of experience. And any time a senior person leaves in a similar role, like the senior software

00:46:46 ◼   ► designer for iOS in the past, that is a loss for the company. And that's why I get like,

00:46:51 ◼   ► it doesn't matter who cares why he left, he's not there anymore. And that will only serve to magnify

00:46:59 ◼   ► the good and the bad about Ive. And I would rather see him surrounded by more senior people that he

00:47:06 ◼   ► respected to, you know, not to temper his output, but perhaps even to enhance the parts that are

00:47:12 ◼   ► good about it. It's not like they have to disagree with him. Maybe they should more vehemently agree

00:47:15 ◼   ► with him in certain areas where he's doubting himself. That dynamic is what I think is lost when

00:47:20 ◼   ► senior people leave. Yeah, the other thing that I'm a little concerned about,

00:47:25 ◼   ► just on based completely on theory is Johnny getting spread a little thin and I would hope

00:47:32 ◼   ► that he has his trusted minions slash you know advisors or whatever to to take a lot of the

00:47:41 ◼   ► day-to-day off of his plate but based on no facts and just a bunch of assumptions it seems to me

00:47:48 ◼   ► that Johnny is a fairly hands-on guy and if he's a hands-on guy and presumably already had a full

00:47:55 ◼   ► plate with hardware alone, I just can't imagine now adding all of software onto that already

00:48:01 ◼   ► overflowing plate and still being able to be good at your job, not only your existing

00:48:08 ◼   ► job of hardware, but also good at this new job of software, which really, in the strictest

00:48:14 ◼   ► sense, he doesn't have any real experience or education in.

00:48:17 ◼   ► So that kind of concerns me a bit that he'll be spread too thin over the next few years.

00:48:23 ◼   ► Well, see, that distinction, if you were to ask Johnny Ive—I'm going to play Johnny

00:48:27 ◼   ► Ive now because I read a book about him once—we had on the podcast, I bet what he would say

00:48:32 ◼   ► is this distinction between hardware and software is an artificial one that has no bearing on

00:48:37 ◼   ► the experience of using the product.

00:48:38 ◼   ► It's all one and the same part of the same product.

00:48:41 ◼   ► And it makes perfect sense for them to be under the same umbrella, because that separation

00:48:45 ◼   ► and that sort of arguing between them, like, yes, there is a tension technically, but that

00:48:48 ◼   ► doesn't matter to the end product.

00:48:49 ◼   ► It has to be one thing.

00:48:51 ◼   ► I think it's perfectly valid to have that approach, that holistic approach to the product.

00:48:56 ◼   ► And Apple has always had it.

00:48:58 ◼   ► It's just that now it's literally embodied within one person.

00:49:00 ◼   ► And yes, it's a danger of that, and that maybe there was healthy tension between hardware

00:49:03 ◼   ► and software.

00:49:04 ◼   ► But I don't worry about him being spread too thin, because I think he is fulfilling sort

00:49:08 ◼   ► of the Steve Jobs type role of set the direction, be tastemaker, give thumbs up, thumbs down.

00:49:14 ◼   ► He's not in there drawing UIs one at a time or whatever.

00:49:20 ◼   ► He's got a staff.

00:49:21 ◼   ► who do that, but in the end the buck stops with him and if he really wants it to be a super skinny

00:49:24 ◼   ► font on a white background, that's what it's going to be. And like Marco said, if you have to dial it

00:49:31 ◼   ► back, fine. In many ways it's better to go too big and then scale it back than to be too timid,

00:49:35 ◼   ► you know what I mean? So I'm not giving him a bad grade or anything. I'm just watching it because I

00:49:42 ◼   ► would rather see ideas have to fight for their life as hard as possible inside Apple and only

00:49:49 ◼   ► the best ones make it out, then to sort of slide towards a situation where there's not

00:49:53 ◼   ► enough people with skills even close to Johnny I for him to do good work in the software

00:50:01 ◼   ► area.

00:50:02 ◼   ► Like in the hardware area, I don't know what's going on back there, and we don't know those

00:50:04 ◼   ► people as well, and I think he has a trusted team that doesn't have a lot of turnover.

00:50:07 ◼   ► But in the software area, he's lost Scott Forrestal through what we think actually was

00:50:12 ◼   ► a disagreement, and Greg Christie is now gone.

00:50:16 ◼   ► I would be happier if both of those people were still there and working in concert with

00:50:20 ◼   ► Johnny Ive. But what can you do?

00:50:22 ◼   ► Well, you know, maybe the hardware stuff will just be a lot less work now, because you can

00:50:25 ◼   ► just look at what HP and Samsung are making and just do stuff that looks like that, right?

00:50:30 ◼   ► I think it's the opposite. It doesn't work the other way.

00:50:33 ◼   ► You know, another thing that I should point out, and I am not through listening to it

00:50:37 ◼   ► yet, but on Guy Englishes and Rene Ritchie's really great podcast, Debug, on episode 33

00:50:45 ◼   ► we'll put a link to that in the show notes. They had Ken Ferry on who did, apparently was kind of

00:50:50 ◼   ► the primary guy behind Auto Layout. And I'm only about two-thirds of the way through the episode,

00:50:54 ◼   ► but it's really, really good. And one of the things that Ken talks about is kind of what it's

00:51:00 ◼   ► like to get an idea through Apple. You know, so he, I don't, I didn't hear the Auto Layout

00:51:07 ◼   ► specific part yet, but just for the sake of argument, you know, I go in there, I come up with

00:51:13 ◼   ► the idea for auto layout and apparently he basically only has to convince his immediate

00:51:20 ◼   ► supervisor or at least that's what I gleaned from it and I just thought that was really really

00:51:25 ◼   ► interesting that it's not very bureaucratic it's not that you have to convince 3,000 people all the

00:51:29 ◼   ► way up the chain and I'm sure that that's that is the case sometimes but generally for for some

00:51:37 ◼   ► things you can just convince one man or woman and that's all it takes and I just thought that was

00:51:41 ◼   ► - We are also sponsored this week, once again, by New Relic.

00:51:45 ◼   ► New Relic is an all-in-one web application performance management tool.

00:51:49 ◼   ► Apparently there's an acronym for that and it's APM.

00:51:53 ◼   ► New Relic lets you see performance from the end user experience through servers

00:51:57 ◼   ► down to each line of your server-side code. New Relic asks us to take a minute

00:52:01 ◼   ► and say a big thank you to all the data nerds out there building all this great stuff

00:52:05 ◼   ► that we all know and love. They would like to send a shout out to developers,

00:52:09 ◼   ► developers, software geeks, code jockeys, to the brave few who see things differently.

00:52:14 ◼   ► Here's to working nights, to wearing oversize concentration enhancing headphones upon your

00:52:18 ◼   ► furrowed brows. New Relic thanks you, the entire internet thanks you.

00:52:23 ◼   ► Nowadays, if you're in any business, you're in the software business. Software powers

00:52:27 ◼   ► our apps, runs our databases, manages our accounts, and runs e-commerce sites and email

00:52:32 ◼   ► programs. When software breaks, everyone loses. New Relic helps improve your software performance,

00:52:37 ◼   ► so your users have a better experience

00:52:39 ◼   ► and your business is more successful.

00:52:41 ◼   ► How is that for a win-win?

00:52:43 ◼   ► New Relic monitors every move your application makes

00:52:45 ◼   ► across the entire stack.

00:52:47 ◼   ► It can show you what's happening right now in your app.

00:52:50 ◼   ► You can zero in on problems quickly

00:52:52 ◼   ► with transaction tracing, SQL and no SQL,

00:52:55 ◼   ► performance analytics, application topology mapping,

00:52:58 ◼   ► and deployment history markers and comparisons.

00:53:01 ◼   ► Go to newrelic.com/atp

00:53:04 ◼   ► and you can sign up for a free 30-day trial.

00:53:07 ◼   ► then all you gotta do is deploy their agent on your server.

00:53:10 ◼   ► It natively supports Ruby, PHP, Java, .NET, Python,

00:53:13 ◼   ► and even Node.js.

00:53:15 ◼   ► It'll start collecting data immediately,

00:53:17 ◼   ► and then you can quickly see inside your app

00:53:19 ◼   ► to start finding hotspots, fixing issues,

00:53:21 ◼   ► and optimizing performance.

00:53:23 ◼   ► Once again, go to newrelic.com/atp

00:53:26 ◼   ► to start monitoring your web app's performance today

00:53:29 ◼   ► and finding all your issues and fixing them all.

00:53:31 ◼   ► Newrelic.com/atp.

00:53:32 ◼   ► Thanks a lot.

00:53:34 ◼   ► So one quick remaining thought on the Christie thing, which I came up with.

00:53:39 ◼   ► We said Christie had been there since the beginning.

00:53:43 ◼   ► Let's suppose, hypothetically, that he was pretty well aligned or pretty chummy with

00:53:48 ◼   ► Forstall.

00:53:49 ◼   ► Then wouldn't it make sense for him to either choose to leave or be asked to leave?

00:53:59 ◼   ► And the quiet rumblings I've heard are that that's exactly the case, that he was kind

00:54:07 ◼   ► of in forestalls camp, and maybe the person who's replacing him is someone who likes

00:54:14 ◼   ► to argue a little bit more.

00:54:16 ◼   ► And if that's the case, I'd consider that a wonderful thing.

00:54:18 ◼   ► I think it's very smart of people who are in a position of power to have someone around

00:54:25 ◼   ► them that argues because it forces you to really, really vehemently believe in your own opinion,

00:54:33 ◼   ► or change it if the case may be. The best example of that from the Steve Jobs's history is the,

00:54:38 ◼   ► you know, him being adamant that iTunes, was it iTunes? Was it the iPod shouldn't be available

00:54:44 ◼   ► for Windows? iTunes Store shouldn't be available for Windows? Come on, you guys gotta help me out

00:54:47 ◼   ► here, he's not remembering. But anyway, yeah, iTunes for Windows, the chat room says, so I

00:54:51 ◼   ► I will trust them.

00:54:52 ◼   ► But anyway, whatever it was, it's

00:54:53 ◼   ► something that eventually came to pass

00:54:55 ◼   ► and was an enormously positive thing.

00:54:57 ◼   ► And Steve Jobs was dead set against it.

00:54:59 ◼   ► And it was just all of his most trusted lieutenants

00:55:02 ◼   ► were constantly on him, arguing, just like would not give it up.

00:55:05 ◼   ► Like, we've got to do iTunes for Windows.

00:55:06 ◼   ► He said, no, I'm never going to do it.

00:55:07 ◼   ► I mean, eventually they wore him down.

00:55:09 ◼   ► He's like, fine, do friggin' iTunes for Windows,

00:55:11 ◼   ► whatever you want.

00:55:12 ◼   ► That sounds like an unhealthy dynamic.

00:55:14 ◼   ► But I think that is actually a company--

00:55:18 ◼   ► I mean, it's not the healthiest it could be.

00:55:20 ◼   ► It would be better if, again, it was something more

00:55:23 ◼   ► like the Pixar Brain Trust.

00:55:25 ◼   ► But in many ways, it was similar,

00:55:26 ◼   ► because with the Brain Trust thing,

00:55:28 ◼   ► it's a bunch of senior people, I'll

00:55:32 ◼   ► tell you what's wrong with your thing, but none of them

00:55:35 ◼   ► have the power to tell you what you have to do.

00:55:37 ◼   ► Like, they can't make you change your movie

00:55:38 ◼   ► in the case of the Brain.

00:55:39 ◼   ► They just tell you, like, we think this is wrong, that's wrong,

00:55:41 ◼   ► and maybe they suggest solutions,

00:55:43 ◼   ► but they have no authority to make you

00:55:44 ◼   ► do any particular changes.

00:55:46 ◼   ► And in the case of Steve Jobs, his lieutenants

00:55:48 ◼   ► didn't have the authority to make him put iTunes

00:55:50 ◼   ► Windows. But because he respected their opinions and because they were so insistent on it, eventually,

00:55:55 ◼   ► you know, he went against his own better judgment and said, "You know what? What am I even paying

00:55:59 ◼   ► these people for if not to advise me?" And they are so strong and unanimous in their,

00:56:03 ◼   ► in their, you know, insistence that this is a good idea. He was kind of a baby about it and said,

00:56:08 ◼   ► "Fine, whatever. Put your stupid thing on Windows if accounts are to be believed." And that definitely

00:56:13 ◼   ► sounds like him. But in the end, you know, the system worked. Yeah, and that's exactly my point.

00:56:19 ◼   ► And so, I mean, whether or not Christie was forced out, depending on who's replacing him,

00:56:25 ◼   ► it could be a really wonderful thing.

00:56:27 ◼   ► So we'll see.

00:56:27 ◼   ► All right.

00:56:29 ◼   ► So we have a few other things that we can talk about.

00:56:32 ◼   ► And something that, uh, that we've had on the show notes for a while that I don't think I completely understand yet.

00:56:39 ◼   ► And I'd love to hear John, you explain it to me is what is this piece?

00:56:43 ◼   ► L thing all about?

00:56:44 ◼   ► Oh, good.

00:56:45 ◼   ► You know, I get to explain another thing that I don't know the details of, but I do think I know.

00:56:49 ◼   ► I do know enough, I think, to explain the broad strokes.

00:56:54 ◼   ► And the broad strokes are actually interesting.

00:56:56 ◼   ► So P. Sal is from-- Steve Pearlman is his name, I think.

00:57:00 ◼   ► Inventor, one of the original inventors of QuickTime.

00:57:03 ◼   ► He also did the on-live remote gaming service.

00:57:05 ◼   ► And this is another one of his babies.

00:57:07 ◼   ► And he has a little bit of reputation

00:57:08 ◼   ► as kind of a genius crackpot, sort of like Stephen Wolf

00:57:11 ◼   ► for him.

00:57:12 ◼   ► Undeniably incredibly smart and doing amazing things,

00:57:15 ◼   ► but also prone to presenting his creations as the best thing since sliced bread and kind

00:57:21 ◼   ► of magic.

00:57:22 ◼   ► And that's the difficulty with PSEL is, I forgot what the original name for it was,

00:57:26 ◼   ► but it's been rebranded to PSEL.

00:57:28 ◼   ► It's presented in a way that doesn't explain how it works unless you already know how it

00:57:34 ◼   ► works.

00:57:35 ◼   ► And so it seems like magic.

00:57:36 ◼   ► It's like, you have all these problems with cell phone reception because there's so many

00:57:40 ◼   ► people with cell phones, and there's so many towers, and there's so much interference,

00:57:43 ◼   ► and the weak signals and strong signals, and you can't have the cell towers too close.

00:57:46 ◼   ► He's basically describing what we all know about current cell phone reception.

00:57:49 ◼   ► He's like, "Well, what if that could all go away, and you wouldn't have any problem,

00:57:52 ◼   ► and every single cell phone user would have the full bandwidth of the entire tower?"

00:57:56 ◼   ► And you're like, "Well, that's impossible. What are you talking about?"

00:57:59 ◼   ► And it sounds like a perpetual motion machine, and you tend to dismiss him out of hand.

00:58:03 ◼   ► Unless you already know what it is, in which case you would think,

00:58:07 ◼   ► "Oh, well, I know what that is," and then it becomes boring.

00:58:09 ◼   ► And those are the two extremes. Like, "This sounds like magic.

00:58:11 ◼   ► magic. I'm going to be like the only person on earth. It's like as if I have this whole cell

00:58:15 ◼   ► tower to myself. And then you find out how it really works. If you happen to, and if you were

00:58:20 ◼   ► in this field, it's not anything novel or amazing or new. There are certain aspects of it that are

00:58:25 ◼   ► interesting from an engineering perspective. And you're like, "Oh, it's that. Okay, well, now I

00:58:31 ◼   ► think I'm less excited about it." And that was my experience as well. So here's what it is.

00:58:36 ◼   ► In the demonstrations, what he's talking about is having cell towers, and you can't put them

00:58:41 ◼   ► them too close together because they make interference with each other.

00:58:43 ◼   ► And of course, every single cell phone that's trying to communicate with the cell tower

00:58:47 ◼   ► is interfering with the other cell phones that are communicating with the cell tower,

00:58:50 ◼   ► and there's many different strategies that we use to allow multiple cell phones to talk

00:58:54 ◼   ► to the tower.

00:58:55 ◼   ► And you don't need to know the details of all of them, but the most ancient one is like,

00:59:02 ◼   ► let's just all take turns.

00:59:03 ◼   ► There's different ways of multiplexing our signals with frequency division, where we

00:59:08 ◼   ► to carve up and into different channels, different frequencies,

00:59:10 ◼   ► or code division multiplexing, where we send a signal that each thing can

00:59:13 ◼   ► decode to figure out which part of the signal was relevant to it.

00:59:17 ◼   ► But it all has to do with just a big shared space.

00:59:19 ◼   ► And all those techniques-- interference is bad,

00:59:21 ◼   ► because it screws up with the signal each thing is sending out and receiving.

00:59:26 ◼   ► What PSEL is trying to do is make it so that all the cell towers are

00:59:32 ◼   ► aware of where all the cell phones are.

00:59:35 ◼   ► and they're aware of what signal is being put out

00:59:38 ◼   ► by all the phones and all the other towers,

00:59:40 ◼   ► and they do a whole bunch of really complicated math

00:59:42 ◼   ► to figure out, okay, we know what we wanna send

00:59:44 ◼   ► to everybody here.

00:59:46 ◼   ► We're going to output something such that our interference

00:59:49 ◼   ► overlaps with each other in such a way

00:59:51 ◼   ► that it sort of forms a hotspot right in the target area

00:59:56 ◼   ► where all the combination of the interference

00:59:58 ◼   ► from all the signals we're putting out

00:59:59 ◼   ► sum up to exactly what we wanna send to that phone.

01:00:02 ◼   ► And then all the interference from all the signals

01:00:04 ◼   ► putting out sum up to exactly the signal you want to put out to that other phone.

01:00:06 ◼   ► That's why it's like you get all you get all the bandwidth because it's a bunch

01:00:10 ◼   ► of things working in concert to figure out exactly what they need to send out

01:00:13 ◼   ► so that the sum of all their crazy interference exactly equals this the

01:00:17 ◼   ► exact clean signal they want to send to that particular phone. I'm massively

01:00:21 ◼   ► simplifying it obviously but this is this is sort of the upshot of how it

01:00:25 ◼   ► works. And the innovative thing they have is like well how do you do that how can

01:00:29 ◼   ► you have all these cell towers figuring out exactly what they need to put out

01:00:33 ◼   ► sort of in real time, all the signals that are being sent and received to figure out

01:00:38 ◼   ► what they have to put out to make the interference overlap to hit every single little cell phone.

01:00:42 ◼   ► That sounds crazy.

01:00:43 ◼   ► How could you do that?

01:00:44 ◼   ► Well, that's part of the engineering breakthrough in that they claim to have a way to do that

01:00:47 ◼   ► with fiber optic cables or whatever running between the cell towers or slower links if

01:00:51 ◼   ► you're willing to allow for a little more slop in the system.

01:00:55 ◼   ► And to be able to have computing in each one of those locations that scales linearly, which

01:01:01 ◼   ► which is another topic on the topic list that we may get to someday, scaling.

01:01:04 ◼   ► This scales linearly.

01:01:05 ◼   ► So it's like, well, OK, so it's easy to do for two receivers, but is it 10 times as much

01:01:10 ◼   ► to do four?

01:01:11 ◼   ► It is like, how does it scale?

01:01:12 ◼   ► It's like, well, we have a way.

01:01:13 ◼   ► If you want to handle 50 people on a tower, then you need one computer.

01:01:16 ◼   ► If you want to handle 100 people, you need two computers.

01:01:18 ◼   ► You know, 200, you need four computers.

01:01:20 ◼   ► Like, it scales linearly with the number of people.

01:01:23 ◼   ► And that is an interesting engineering breakthrough, assuming it works as advertised.

01:01:27 ◼   ► But there are limitations to the system.

01:01:29 ◼   ► First of all, you do have to have all the towers talking to each other so they can figure

01:01:31 ◼   ► out what the hell to output.

01:01:33 ◼   ► You do need the fast connections between the towers.

01:01:36 ◼   ► And if you make a slower connection between the towers, what you're giving up is essentially

01:01:40 ◼   ► how fast the receivers can move.

01:01:43 ◼   ► So for example, if you're in a car and that car goes over something like 70 miles an hour,

01:01:48 ◼   ► by the time all the cell towers figure out how to constructively overlay all their crazy

01:01:52 ◼   ► overlapping signals to hit where you are, you've moved too far and it won't work as

01:01:56 ◼   ► well.

01:01:57 ◼   ► speed limitations to this, which are not, you know, there are speed limitations that

01:02:01 ◼   ► you moving around really quickly affects everything, but it affects this system more than others.

01:02:05 ◼   ► So if you're in any sort of car trying to use your nav system, whatever, and the system

01:02:09 ◼   ► was in use, if you're going over 70, maybe it wouldn't be able to keep up with you if

01:02:12 ◼   ► you only had microwave links between the towers and everything.

01:02:15 ◼   ► But the idea, and even I think Apple's routers and regular just Wi-Fi routers do this, they

01:02:21 ◼   ► call it beamforming or whatever, the idea is an interesting one, and it's great when

01:02:26 ◼   ► an idea that was worked in labs for years and years suddenly becomes feasible in the

01:02:30 ◼   ► real world due to engineering expertise, which is essentially what they're trying to bring

01:02:34 ◼   ► as a company to this.

01:02:36 ◼   ► And the theoretical benefits are there in that if you could get a system like this,

01:02:40 ◼   ► you can make a much more effective use of bandwidth and you can put small cell tower

01:02:45 ◼   ► type emitters, receivers everywhere.

01:02:48 ◼   ► And don't worry about how they overlap because it'll be just fine.

01:02:51 ◼   ► would make for the ability to support more users in a congested urban environment.

01:03:00 ◼   ► And the other trick they have from engineering perspective is they didn't want to have to

01:03:03 ◼   ► change all our phones.

01:03:04 ◼   ► So they found a way using the existing features of LTE, 4G that are in the existing protocol,

01:03:09 ◼   ► such that an unmodified, plain old iPhone 5S or any other plain old cell phone can participate

01:03:15 ◼   ► in this.

01:03:16 ◼   ► They don't need to have a special chip in the phones or special computing in the phones

01:03:19 ◼   ► or whatever.

01:03:20 ◼   ► again is another engineering type of breakthrough where, yeah, this was fine in theory, but

01:03:23 ◼   ► you can't just go replace all the cell phones in the world with these magic ones that work.

01:03:26 ◼   ► It's like, well, we found a way to get this to work using existing protocols with existing

01:03:30 ◼   ► unmodified phones with all the smarts on the other end.

01:03:33 ◼   ► So I think it is an interesting engineering achievement, assuming it works as advertised.

01:03:38 ◼   ► He still faces go-to-market challenges in terms of getting this technology into all

01:03:42 ◼   ► the cell towers and who's going to be the first person to roll it out and how will it

01:03:44 ◼   ► coexist with other things.

01:03:46 ◼   ► But in the end, it is not magic.

01:03:48 ◼   ► It's just science.

01:03:49 ◼   ► It's not particularly new science, but the engineering is where the interesting parts

01:03:53 ◼   ► are.

01:03:54 ◼   ► There was a really neat video that I watched that showed—I believe it was a bunch of

01:03:59 ◼   ► like laptops or tablets or whatever—that were equipped with LTE radios.

01:04:04 ◼   ► And they'd shown—I'm going to probably get the details wrong—but they'd shown

01:04:09 ◼   ► what happens when you move around, and they turned off the thing where it senses where

01:04:15 ◼   ► you are.

01:04:16 ◼   ► So basically they concentrated this beam of LTE in a specific spot, and just for the purposes

01:04:22 ◼   ► of the demonstration, rather than allow that beam to move around with the device, they

01:04:28 ◼   ► kept it stationary.

01:04:29 ◼   ► And you could see as they moved just a couple of inches that the throughput would just plummet,

01:04:36 ◼   ► and then they would put the device back where it started, and it would come back to full

01:04:40 ◼   ► HD video or something like that.

01:04:42 ◼   ► And then when they had it working as it's designed, which is to say it will follow the

01:04:47 ◼   ► device as it moves, you would get this full HD signal as the dude's like, you know, just

01:04:52 ◼   ► moving these devices around in the room with a reasonable quickness, not 70 miles an hour,

01:04:58 ◼   ► but a reasonable quickness.

01:04:59 ◼   ► And it was fascinating to watch and really, really impressive.

01:05:02 ◼   ► In some ways it's like engineering at its best because what engineers do very well,

01:05:08 ◼   ► As opposed to scientists, like, two sides of the same coin.

01:05:12 ◼   ► The scientists will come up with something, or these theories, or whatever.

01:05:15 ◼   ► But the job of the engineers is to be able to figure out, you know what?

01:05:19 ◼   ► That theory was useless to us 10 years ago, 20 years ago, 50 years ago.

01:05:24 ◼   ► But computing has advanced so much, or technology, or material science, or whatever, has advanced

01:05:28 ◼   ► so much that that crazy theory that no one ever pays attention to, you know, that it's

01:05:32 ◼   ► like, well, that's not feasible.

01:05:33 ◼   ► Obviously, you can never do that.

01:05:35 ◼   ► Starts to enter the realm of feasibility.

01:05:37 ◼   ► And engineers are rewarded by getting the jump at everyone and saying, "You know what?

01:05:40 ◼   ► I think I can do that now.

01:05:42 ◼   ► I think I can do that crazy beamforming thing across huge urban areas to people traveling

01:05:47 ◼   ► up to 70 miles an hour with existing..."

01:05:49 ◼   ► I think I can do that now, to get there first.

01:05:52 ◼   ► Because eventually when technology comes along and it's like, well, now we have...

01:05:55 ◼   ► Everyone will realize we have the computing to do an algorithm that wasn't feasible before,

01:05:59 ◼   ► a compression technique that used to be too computationally expensive but now is trivial

01:06:03 ◼   ► on phones.

01:06:05 ◼   ► And this is an example of that.

01:06:07 ◼   ► He seems to be out ahead of people in terms of, "I think I can do that now, and here's

01:06:11 ◼   ► my proof and here's my implementation."

01:06:14 ◼   ► And whether or not it flies, kind of like on live, like I think I can make people do

01:06:16 ◼   ► video games where the video gaming machine is in the data center and they are miles and

01:06:20 ◼   ► miles away.

01:06:21 ◼   ► I think gaming can work like that.

01:06:24 ◼   ► It did kind of work.

01:06:25 ◼   ► I have on live, I've launched it, I've played it, it's not great, but as a technology proof

01:06:30 ◼   ► of concept, it's interesting.

01:06:32 ◼   ► As with anything else, the engineering is only one small part of the battle of being

01:06:36 ◼   ► successful in the market. But as non-magical as it might be, I do like what he does because

01:06:44 ◼   ► kind of like Elon Musk, he's figuring out what is feasible with current technology and

01:06:50 ◼   ► getting there just slightly ahead of everyone else. And that's great engineering.

01:06:54 ◼   ► Yeah, like I said, it was really impressive to me whether or not it's just the combination

01:07:01 ◼   ► of a bunch of things we've known how to do for a while. It looked really cool.

01:07:06 ◼   ► That's all Apple does too, right?

01:07:07 ◼   ► Take things, you know, we've had touch screens, we've had capacitive touch screens, we've

01:07:11 ◼   ► had user interfaces with buttons, like, you know, bring it all together, figure out that

01:07:15 ◼   ► actually we can make a phone like that now and do it just a little bit ahead of everyone

01:07:18 ◼   ► else.

01:07:19 ◼   ► There's a big reward if you get all the parts right.

01:07:21 ◼   ► Exactly.

01:07:22 ◼   ► We are also sponsored this week by Igloo Software.

01:07:26 ◼   ► Igloo is an intranet you'll actually like.

01:07:29 ◼   ► Now most people think of intranets as, remember this is not internets, this is intranets,

01:07:33 ◼   ► the old boring thing, right?

01:07:35 ◼   ► the old, stale, terrible places that the corporate overlords make you go when you just want to

01:07:40 ◼   ► use Dropbox or WordPress or something easy that helps you get your work done that actually

01:07:43 ◼   ► works. Igloo brings the ease of use and familiarity from consumer software into your corporate

01:07:48 ◼   ► environment by using familiar apps like shared calendars, Twitter-like microblogs, file sharing,

01:07:54 ◼   ► and more. Every piece of content in your Igloo can be social, with comments and like buttons,

01:07:59 ◼   ► and each team in your company can configure their own workspaces.

01:08:03 ◼   ► that's it for your users but what if you're in charge of IT? Well Igloo

01:08:07 ◼   ► handles the security, hosting, and management for you. Igloo is SOC 2 type 2

01:08:11 ◼   ► compliant and they host data securely in SOC 2 type 1 enterprise facilities in

01:08:17 ◼   ► Canada, that's right Canada so everything is polite and CDRs cost too much, on

01:08:21 ◼   ► their own servers. Igloo offers 256-bit SSL and they are not vulnerable to

01:08:27 ◼   ► heartbleed. They offer backups, disaster recovery, single tenant, and shared

01:08:31 ◼   ► environments, integration with many authentication and sync systems, including SAML services

01:08:35 ◼   ► and LDAP, and more. Igloo can even work with HIPAA compliant organizations. You can also

01:08:42 ◼   ► customize everything inside your Igloo with the ability to add CSS and JavaScript globally

01:08:46 ◼   ► across one team space or even on a single page. You can see all this on the Igloo website,

01:08:51 ◼   ► which is actually built on their platform. Learn more at igloosoftware.com/casey.

01:08:55 ◼   ► It's igloosoftware.com/casey, where they made a funny landing pig about why SharePoint sucks

01:09:03 ◼   ► and how igloo is better.

01:09:05 ◼   ► Thanks a lot once again to igloo for sponsoring our show.

01:09:08 ◼   ► I wanted to ask about scaling servers because we've had this in the show notes for a while.

01:09:13 ◼   ► I love this.

01:09:15 ◼   ► So on the topic list, there's just a simple bullet point that just says "scaling servers."

01:09:19 ◼   ► Like you might as well have had a bullet point that said "computers."

01:09:24 ◼   ► I wrote it there, and it was actually a reaction

01:09:26 ◼   ► to a particular blog post that Brent Simmons put up,

01:09:29 ◼   ► where he mentioned scaling and passing.

01:09:32 ◼   ► And I don't want to get into a big thing about it,

01:09:33 ◼   ► but it's just like, every time I see this come up--

01:09:36 ◼   ► and we do it ourselves, and people

01:09:37 ◼   ► are sloppy with terminology-- it's

01:09:39 ◼   ► so important to keep reminding yourself the definition

01:09:44 ◼   ► of terms, like what we should be talking about when

01:09:46 ◼   ► we talk about scaling.

01:09:47 ◼   ► Because we use it as an umbrella term to talk about everything.

01:09:49 ◼   ► And I kind of alluded to it in the PSAL thing,

01:09:51 ◼   ► but my only point is the old saw about the difference

01:09:54 ◼   ► in performance and scaling.

01:09:55 ◼   ► So much of the time when I see people thinking they're blogging about scaling, all they're

01:10:00 ◼   ► blogging about is performance, which is fine and interesting as a topic of itself, but

01:10:04 ◼   ► it's not the same thing as scaling.

01:10:05 ◼   ► And so I just put it in there as a reminder to myself to throw it out.

01:10:09 ◼   ► So in brief, performance is how fast you can make something go.

01:10:12 ◼   ► If you've got a computer and you've got a task to be done, I can do that task in five

01:10:16 ◼   ► minutes.

01:10:17 ◼   ► I need to increase my performance now.

01:10:18 ◼   ► I can do that task in three seconds.

01:10:19 ◼   ► You've increased your performance.

01:10:21 ◼   ► is I've got one task to do and I've got one computer. What if I give you 10 million of

01:10:26 ◼   ► those tasks to do? Can you do them in the same time if you just use 10 million computers?

01:10:31 ◼   ► That's scaling. It doesn't matter how long it takes. Even if it takes your computer a

01:10:34 ◼   ► year to solve the problem, if I give you two of those problems, you can say no problem,

01:10:38 ◼   ► I'll just buy another computer, I can do both of them in a year. Okay, what if I give you

01:10:41 ◼   ► three? I'll buy three computers. This is linear scaling. I'll buy three computers, I can do

01:10:45 ◼   ► all three in a year. What if I give you three million? I'll buy three million computers,

01:10:49 ◼   ► I can do all three million of those problems in a year.

01:10:51 ◼   ► It's not about performance.

01:10:52 ◼   ► Scaling and performance, although they often seem related,

01:10:55 ◼   ► are not the same concept.

01:10:57 ◼   ► The failure to focus on scaling

01:10:59 ◼   ► when you're talking about scaling

01:11:00 ◼   ► and performance when you're talking about performance

01:11:02 ◼   ► leads to all sorts of sadness.

01:11:04 ◼   ► - Oh yeah, I mean, scaling is not how fast your system is,

01:11:09 ◼   ► but how easily you can make it take on more of something.

01:11:14 ◼   ► Usually more traffic, more users, more data,

01:11:17 ◼   ► whatever the case may be, how easy is it to take on more?

01:11:21 ◼   ► And how many changes do you have to make to your software

01:11:24 ◼   ► and to its architecture to add that capacity?

01:11:28 ◼   ► Because with almost everything,

01:11:31 ◼   ► it is not nearly as simple as, oh, just add more servers.

01:11:34 ◼   ► Because you might have like, all right, well, let's say,

01:11:36 ◼   ► you can add more web servers pretty easily up to a point.

01:11:39 ◼   ► Because web servers, you can distribute the calls

01:11:42 ◼   ► to any of them with a load balancer,

01:11:44 ◼   ► as long as you don't use weird shared sessions or anything.

01:11:46 ◼   ► You can distribute calls to any of them.

01:11:48 ◼   ► Any call coming in gets assigned to web server 67.

01:11:52 ◼   ► And then web server 67 does whatever it needs to do

01:11:55 ◼   ► and responds back.

01:11:56 ◼   ► Okay, great.

01:11:57 ◼   ► However, web server 67 probably has to access a database

01:12:01 ◼   ► to do something.

01:12:02 ◼   ► And how do you organize the databases?

01:12:04 ◼   ► And that's usually, it's the data layer,

01:12:07 ◼   ► not the web serving application layer

01:12:09 ◼   ► that's usually the hardest to scale.

01:12:12 ◼   ► Because that's when you have things like

01:12:15 ◼   ► globally shared storage. You have things like limits on disk performance and on how many

01:12:22 ◼   ► writes can a single server even write per second. If you've scaled away all the reads

01:12:30 ◼   ► and given reads to all these replicating servers, that's fine, but can they even keep up with

01:12:35 ◼   ► the writes? How many writes does the master do? And then if you split the roll up like

01:12:39 ◼   ► that, then you get into really hard issues like, "All right, well, how do you cache data?

01:12:44 ◼   ► can you be sure that the data that you have read

01:12:47 ◼   ► off of the read slave is up to date?

01:12:49 ◼   ► Otherwise, you can't really cache it

01:12:51 ◼   ► 'cause you might be clobbering something

01:12:52 ◼   ► in the cache that is newer.

01:12:54 ◼   ► Or you might be caching old data when new data is available.

01:12:57 ◼   ► And there's all these really hard problems

01:12:59 ◼   ► that go along with it that makes scaling

01:13:01 ◼   ► such a complex, interesting topic

01:13:02 ◼   ► and why it's such a hard problem.

01:13:06 ◼   ► - And that's why when I see a blog post

01:13:08 ◼   ► about how to make a data structure smaller

01:13:11 ◼   ► or a query more efficient, and then the word scaling

01:13:13 ◼   ► using that same conversation.

01:13:14 ◼   ► It's like, that's performance.

01:13:16 ◼   ► You're just talking about performance.

01:13:17 ◼   ► And you're talking about the web tier.

01:13:19 ◼   ► Since, oh, it's stateless.

01:13:21 ◼   ► We can scale that out horizontally forever.

01:13:23 ◼   ► Nothing scale-- even the web tier is not like that one-to-one

01:13:27 ◼   ► scaling that I was using in my example.

01:13:29 ◼   ► If you ever get there, you have reached the golden.

01:13:32 ◼   ► You will be a bazillionaire.

01:13:33 ◼   ► Because even just getting there, even on the web tier,

01:13:35 ◼   ► it's like, well, but what about different data centers?

01:13:37 ◼   ► Now we need geographical load balancing.

01:13:39 ◼   ► And what about network connections?

01:13:41 ◼   ► Nothing scales perfectly.

01:13:43 ◼   ► But you're right, the first thing you run into it

01:13:46 ◼   ► is with state.

01:13:46 ◼   ► And you get into CAP theorem of consistency, partition

01:13:50 ◼   ► tolerance, and availability.

01:13:53 ◼   ► This is well-trod territory.

01:13:54 ◼   ► Lots of people will concentrate in scaling.

01:13:56 ◼   ► Google knows probably the most about it

01:13:58 ◼   ► of any company in the entire world,

01:13:59 ◼   ► given the scale they operate at.

01:14:00 ◼   ► Microsoft seems to be learning a lot about it.

01:14:02 ◼   ► Apple, not sure what they know.

01:14:03 ◼   ► But scaling is super hard.

01:14:06 ◼   ► Performance is super hard.

01:14:08 ◼   ► And I mean, from the outside, this

01:14:12 ◼   ► is all sort of esoteric stuff.

01:14:14 ◼   ► But even for people who are just working on a small system--

01:14:17 ◼   ► and it's like, it doesn't matter.

01:14:19 ◼   ► You're not going to have Facebook-level users, right?

01:14:22 ◼   ► But even on a very small system, 10 users, 100 users, 1,000,

01:14:26 ◼   ► 200,000, even then, you should be

01:14:28 ◼   ► thinking about both performance and scaling

01:14:30 ◼   ► and keeping a clear head about which one you're concentrating

01:14:33 ◼   ► on.

01:14:33 ◼   ► And don't concentrate too much on scaling and not

01:14:35 ◼   ► enough on performance, or too much on performance

01:14:36 ◼   ► and not enough scaling.

01:14:38 ◼   ► just keeping a clear head about what the individual tasks are.

01:14:42 ◼   ► And you will eventually find yourself, even in a small system,

01:14:45 ◼   ► reading all those papers about the theory behind it

01:14:47 ◼   ► and making your trade-offs about consistency, availability,

01:14:52 ◼   ► and partition tolerance.

01:14:53 ◼   ► You have to make decisions about it.

01:14:55 ◼   ► If you don't make decisions, the software you choose

01:14:56 ◼   ► will make the decisions for you, and you'll be sad about it.

01:14:59 ◼   ► So that's all part of the learning process.

01:15:01 ◼   ► But I don't know.

01:15:05 ◼   ► I don't want to pick on any particular blog posts.

01:15:07 ◼   ► I love reading those Brent posts.

01:15:08 ◼   ► I love watching him think out loud about what he's doing.

01:15:10 ◼   ► He's a great example of a great software developer,

01:15:12 ◼   ► like showing his work as he works towards a product

01:15:18 ◼   ► and everything like that.

01:15:20 ◼   ► Since I've dealt with server-side software

01:15:22 ◼   ► my entire career, I have a particular itch

01:15:25 ◼   ► when I see the word scaling and it's not really

01:15:28 ◼   ► talking about scaling.

01:15:29 ◼   ► Maybe I'm just being a pedant, right, Casey?

01:15:32 ◼   ► That's right.

01:15:33 ◼   ► That's right.

01:15:34 ◼   ► Well, and there is something to be said for performance optimization to a point.

01:15:39 ◼   ► It's very similar to the premature optimization wisdom in the world, except, and Brent even addressed this in one of his posts on

01:15:45 ◼   ► Inessential.com, which we should mention. He even addressed, when you're talking about server-side design,

01:15:51 ◼   ► for client-side stuff, premature optimization is considered a bad thing because we have such fast hardware.

01:16:01 ◼   ► we have so much memory these days on client-side devices

01:16:04 ◼   ► and desktops and everything.

01:16:05 ◼   ► We have so much, we have so many resources

01:16:07 ◼   ► that it's just not really worth

01:16:11 ◼   ► doing super micro-optimizations everywhere.

01:16:14 ◼   ► However, on servers, things are a little bit different.

01:16:18 ◼   ► On servers, you are paying for what you use basically.

01:16:22 ◼   ► And if your app gets popular at all,

01:16:25 ◼   ► even for a minimal amount of users,

01:16:30 ◼   ► relatively speaking to something like Facebook.

01:16:33 ◼   ► You know, if you get 100,000 users using a web service,

01:16:37 ◼   ► that's substantial needs for the hardware.

01:16:41 ◼   ► And depending on what your service is doing,

01:16:45 ◼   ► that actually could need some serious resources.

01:16:48 ◼   ► And so, you could be talking the difference

01:16:52 ◼   ► between hundreds of dollars a month

01:16:55 ◼   ► or thousands of dollars a month worth of hardware,

01:16:57 ◼   ► or even more than that.

01:17:00 ◼   ► And so it can really make or break your whole economics.

01:17:03 ◼   ► It can make you busier or not as busy.

01:17:06 ◼   ► One of the best things to happen for servers and scaling

01:17:10 ◼   ► in the last decade has been SSDs.

01:17:12 ◼   ► Because for a long time,

01:17:14 ◼   ► for a lot of LAMP-type applications,

01:17:17 ◼   ► one of the biggest limitations was the database's disk speed.

01:17:21 ◼   ► And like at Tumblr, I was able to replace,

01:17:24 ◼   ► I think it was about nine or 12 ReadSlave servers

01:17:28 ◼   ► with one with SSDs.

01:17:31 ◼   ► And nine or 12 servers that had 15K SAS disks in RAID 10

01:17:36 ◼   ► being replaced with one SSD-based server.

01:17:39 ◼   ► And those were old Intel X25e SSDs.

01:17:44 ◼   ► They've come a long way since then.

01:17:46 ◼   ► Now modern SSDs are even faster than those.

01:17:49 ◼   ► And when you can make a big order of magnitude type jump,

01:17:54 ◼   ► If you can go from 10 database servers to one database server,

01:17:59 ◼   ► that kind of difference actually matters and is worth doing.

01:18:03 ◼   ► That is something where performance

01:18:06 ◼   ► has a direct impact, a direct measurable, feelable impact

01:18:09 ◼   ► on scaling.

01:18:12 ◼   ► Optimization for performance, to a point, actually is useful.

01:18:15 ◼   ► Because in practice, most people who make these web services

01:18:21 ◼   ► and make apps and stuff like that,

01:18:23 ◼   ► most people are not going to get as big as Facebook and Google.

01:18:26 ◼   ► Most people are not even going to get as big as smaller things that we've all heard of.

01:18:31 ◼   ► Most people are lucky to get a hundred thousand users to their web service.

01:18:34 ◼   ► And so,

01:18:36 ◼   ► by making decisions

01:18:38 ◼   ► that favor high performance,

01:18:41 ◼   ► you can often keep... that could often be the difference

01:18:45 ◼   ► between

01:18:46 ◼   ► being able to run everything off of a cheap, easy to administer

01:18:49 ◼   ► VPS or a fully managed cloud service at a reasonable price. And running a whole bunch

01:18:55 ◼   ► of dedicated servers or paying tons of money for a cloud service or one of these things,

01:19:00 ◼   ► that could make that big of a difference.

01:19:02 ◼   ► And so it is, or on the scaling side, this could be the difference between needing just

01:19:07 ◼   ► one fast database server and needing a multi-server setup. Or being able to fit all on one server

01:19:13 ◼   ► or having to shard the database, which is really complicated and adds a lot of maintenance.

01:19:18 ◼   ► having to use a different type of tool entirely like Cassandra, which is a world of hurt.

01:19:23 ◼   ► It's just so incredibly not worth using.

01:19:28 ◼   ► Performance does matter, to the point where having a high-performance

01:19:33 ◼   ► system has a very good chance of keeping you from ever needing

01:19:38 ◼   ► a difficult-to-scale system.

01:19:43 ◼   ► Businesses, if they're successful, the actual business, the people who don't understand,

01:19:45 ◼   ► the business people who don't understand technology

01:19:47 ◼   ► and don't want to-- will apply both of these pressures to you.

01:19:49 ◼   ► Because they will apply a pressure for you

01:19:52 ◼   ► to increase performance, because they

01:19:54 ◼   ► don't want to pay a lot of money for servers.

01:19:57 ◼   ► But they will also apply the pressure of you for scaling,

01:19:59 ◼   ► because they will say, my business plan says

01:20:01 ◼   ► we're going to grow 50% year over year for the next five

01:20:04 ◼   ► years.

01:20:04 ◼   ► So you need to be able to say, OK, we have five customers now.

01:20:08 ◼   ► In five years, we're going to have five million customers.

01:20:12 ◼   ► So whatever multiple that is, I don't want to pay--

01:20:16 ◼   ► if you have greater than linear scaling,

01:20:18 ◼   ► it's like, OK, well, one customer costs us $10,

01:20:20 ◼   ► but two customers cost us $100.

01:20:22 ◼   ► You've got a scaling problem immediately.

01:20:23 ◼   ► Because they want the business to grow,

01:20:25 ◼   ► but they want it to grow less than linear.

01:20:27 ◼   ► They want to be able to double their number of customers,

01:20:30 ◼   ► but not double their hardware costs.

01:20:31 ◼   ► And what you're telling them as well,

01:20:33 ◼   ► that's not how scaling works.

01:20:34 ◼   ► If we double our customers, I need

01:20:35 ◼   ► to weigh more than double our hardware costs.

01:20:37 ◼   ► And it's like, how much more?

01:20:38 ◼   ► How close are you to perfect linear scaling?

01:20:41 ◼   ► And so they want both.

01:20:43 ◼   ► It's too expensive.

01:20:44 ◼   ► Every time we get another client,

01:20:45 ◼   ► we need to buy x amount of hardware,

01:20:47 ◼   ► and it costs so much money.

01:20:48 ◼   ► And we're making only a certain amount of money

01:20:49 ◼   ► from each individual client we had.

01:20:51 ◼   ► We can't afford to add that much hardware,

01:20:52 ◼   ► so increase your performance.

01:20:53 ◼   ► But year over year over year, they're like,

01:20:55 ◼   ► "Look, we're gonna quadruple our customer base.

01:20:57 ◼   ► "Can you quadruple our customer base?"

01:20:58 ◼   ► And you're like, "No, we can't.

01:20:59 ◼   ► "We got one database server.

01:21:01 ◼   ► "We're already buying the biggest machine we could possibly,

01:21:03 ◼   ► "that money can buy for our database server.

01:21:05 ◼   ► "We need a strategy for scaling."

01:21:07 ◼   ► And then you end up sharding,

01:21:08 ◼   ► and what about people in different parts of the country?

01:21:10 ◼   ► then you evolve it.

01:21:11 ◼   ► If they're in shards, how do these people

01:21:12 ◼   ► interact with each other?

01:21:14 ◼   ► In some ways, it's easier for software

01:21:16 ◼   ► where it doesn't matter, like Facebook,

01:21:17 ◼   ► where you're like, the message shows up here

01:21:20 ◼   ► a little bit sooner than it shows up there.

01:21:21 ◼   ► It's not a banking system or whatever.

01:21:24 ◼   ► But not everyone's that lucky.

01:21:26 ◼   ► Sometimes it really has to be,

01:21:27 ◼   ► that's where you get into the CAP theorem again,

01:21:29 ◼   ► where you have to pick your trade-offs.

01:21:30 ◼   ► And it's difficult.

01:21:33 ◼   ► And even no matter how bounded your problem is,

01:21:35 ◼   ► even, it just all does is scale everything down.

01:21:38 ◼   ► If you're like, oh, I want to stick to VPSs or something

01:21:42 ◼   ► really cheap.

01:21:42 ◼   ► I don't want to buy dedicated-- like you're

01:21:43 ◼   ► playing in the small leagues.

01:21:45 ◼   ► But it still matters for your company.

01:21:46 ◼   ► When you're going from 10 customers to 10,000,

01:21:49 ◼   ► it's the same exact thing playing out.

01:21:51 ◼   ► It's just a smaller scale versus companies

01:21:53 ◼   ► that start off from day one buying

01:21:54 ◼   ► massive dedicated hardware.

01:21:56 ◼   ► And the only difference in the high end game

01:21:58 ◼   ► is that the high end people are more likely to get

01:22:01 ◼   ► into the situation where they are buying the most

01:22:03 ◼   ► expensive computer equipment that money can buy from anyone

01:22:05 ◼   ► on the entire planet.

01:22:06 ◼   ► When they get to the end of their scaling rope

01:22:08 ◼   ► and they're like, well, we've increased performance

01:22:11 ◼   ► as much as we can go.

01:22:13 ◼   ► No matter how much money we waive at somebody,

01:22:15 ◼   ► we cannot get a bigger single machine to do task X.

01:22:18 ◼   ► It's like, now, if you waited this long

01:22:20 ◼   ► to think about scaling, you have a serious problem,

01:22:22 ◼   ► because you've hit the performance dead end.

01:22:25 ◼   ► I guess you would optimize your software,

01:22:26 ◼   ► but yeah, it's the same story.

01:22:29 ◼   ► It's just a fractal of itself.

01:22:31 ◼   ► It just depends on how many commas there are

01:22:34 ◼   ► before the decimal point in the invoices.

01:22:37 ◼   ► (laughing)

01:22:38 ◼   ► And this is one of the reasons why I really prefer

01:22:42 ◼   ► to do things at a relatively low level

01:22:45 ◼   ► and to do things that are fairly standard

01:22:46 ◼   ► because even if you don't need scaling, in quotes,

01:22:51 ◼   ► even if you don't think you need that today,

01:22:53 ◼   ► knowing how to scale something

01:22:57 ◼   ► or knowing the steps that you take

01:22:59 ◼   ► and the hurdles you will face is useful

01:23:02 ◼   ► when writing anything even before it needs to be scaled.

01:23:05 ◼   ► So for instance, I've said on a number of occasions,

01:23:10 ◼   ► I never rewrite database joins, ever.

01:23:13 ◼   ► Joins are, joins basically put a lot of work

01:23:16 ◼   ► on the database server, which granted,

01:23:17 ◼   ► it is very optimized to do.

01:23:20 ◼   ► However, I prefer to split up the joins into two calls,

01:23:23 ◼   ► like one to fetch the list of IDs

01:23:25 ◼   ► and the second to fetch from the target table,

01:23:28 ◼   ► which ones you get and stuff like that,

01:23:29 ◼   ► and design the schema and the code to do that.

01:23:32 ◼   ► And that's for two major reasons.

01:23:34 ◼   ► One is to give the database a little bit less work.

01:23:38 ◼   ► Well, three major reasons.

01:23:40 ◼   ► One, give the database less work,

01:23:41 ◼   ► because database CPU power,

01:23:43 ◼   ► database resources are expensive and hard to scale.

01:23:46 ◼   ► The second one is caching.

01:23:48 ◼   ► So maybe you can cache that first query,

01:23:51 ◼   ► maybe you can cache the list of IDs you need,

01:23:54 ◼   ► and then only fetch the records, or vice versa,

01:23:56 ◼   ► only fetch the IDs and then fetch the records from cache,

01:23:58 ◼   ► things like that.

01:23:59 ◼   ► And then the third is that then

01:24:01 ◼   ► that also gives me the ability later on,

01:24:03 ◼   ► if the service gets really big to say,

01:24:05 ◼   ► you know what, the users table now has to have

01:24:08 ◼   ► its own database.

01:24:09 ◼   ► This gets hit so often and it's such a drain of performance

01:24:12 ◼   ► that we should split this up into its own database cluster

01:24:15 ◼   ► entirely, so users is now on a different database.

01:24:18 ◼   ► Then you can't do a join,

01:24:20 ◼   ► 'cause it's not even on the same server anymore.

01:24:22 ◼   ► So even with Overcast, I'm not running any joins.

01:24:26 ◼   ► I'm doing it the same way I've always done it,

01:24:27 ◼   ► because it's not that much harder,

01:24:30 ◼   ► And it's just in case in the future,

01:24:34 ◼   ► that'll be fine, I'll be set up for that,

01:24:36 ◼   ► I'll be ready to go.

01:24:37 ◼   ► So many other things, like very careful use of indexing.

01:24:41 ◼   ► One of the most useful books I ever read on scaling

01:24:45 ◼   ► was High Performance MySQL.

01:24:48 ◼   ► And I read the edition before the current one

01:24:51 ◼   ► and I liked it a lot.

01:24:52 ◼   ► The current one got a lot longer

01:24:54 ◼   ► and the reviews weren't as good.

01:24:55 ◼   ► So I'm not sure if the current one is great.

01:24:57 ◼   ► I didn't actually read the whole thing.

01:24:58 ◼   ► But the previous one was awesome.

01:25:00 ◼   ► so maybe give it a shot.

01:25:05 ◼   ► But I think it's important to make those decisions.

01:25:07 ◼   ► Like if you're writing, suppose you're writing C code

01:25:11 ◼   ► for an app, and you're calling something like "sturlan"

01:25:14 ◼   ► in the loop to say like, "Oh, iterate through a string

01:25:18 ◼   ► and calling 'sturlan' 'i' is less than 'sturlan' string."

01:25:21 ◼   ► If you know about performance at all,

01:25:25 ◼   ► you're gonna look at that and say, "Wait a minute,

01:25:27 ◼   ► that's being invoked on every single call."

01:25:26 ◼   ► you get this mentality that makes you make small decisions like that better.

01:25:31 ◼   ► And in the course of a whole project that adds up. And at the end,

01:25:35 ◼   ► even though any one of those calls probably won't matter,

01:25:39 ◼   ► when you make every decision with a certain mindset and a certain sensibility

01:25:42 ◼   ► and a certain wisdom about what will happen in the future,

01:25:45 ◼   ► or what the cost of this might be or might add up to,

01:25:48 ◼   ► when you make all those small decisions the right way,

01:25:52 ◼   ► the sum of all of that is different and does matter.

01:25:56 ◼   ► Same thing applies to scaling.

01:25:57 ◼   ► When you make a bunch of small decisions to say,

01:26:00 ◼   ► oh, you know what, I can put a little more work

01:26:01 ◼   ► on the database here, or, you know,

01:26:03 ◼   ► this doesn't need to be that optimized.

01:26:05 ◼   ► Even, you know, the index isn't quite right

01:26:06 ◼   ► to do this index only, or, you know,

01:26:10 ◼   ► this is gonna have to scan a bunch of rows

01:26:11 ◼   ► to get these results, but that'll be all right.

01:26:14 ◼   ► You know, it's, how often do you really have to do that?

01:26:17 ◼   ► Once you get into the mindset of thinking,

01:26:20 ◼   ► okay, what's going to happen when I have to do this

01:26:23 ◼   ► thousands of times a second?

01:26:25 ◼   ► you'll probably never have to. But if you think about that from the beginning, you can make better,

01:26:30 ◼   ► small decisions along the way that will add up and that will make scaling easier for you.

01:26:34 ◼   ► Can we go back a step? You're saying that you'd rather make round trips to the database than do

01:26:42 ◼   ► something that the database is specifically designed to do? That, I mean—

01:26:46 ◼   ► I think what he's saying is that he doesn't want to use a database.

01:26:48 ◼   ► Right. I mean—

01:26:51 ◼   ► I know you don't like the newfangled key value storage

01:26:53 ◼   ► things or whatever, but if you're not going to do joins--

01:26:56 ◼   ► and I'm not saying that's the right or wrong decision,

01:26:58 ◼   ► but if that's the route you're going,

01:27:01 ◼   ► there are probably places you could store your data that

01:27:05 ◼   ► would give you better performance

01:27:07 ◼   ► for this same kind of usage pattern.

01:27:08 ◼   ► I imagine the reason you wouldn't like them

01:27:10 ◼   ► is because they're newer and less mature

01:27:13 ◼   ► and you're used to the features of being

01:27:15 ◼   ► able to back up MySQL and do easy replication.

01:27:17 ◼   ► And those are all important features

01:27:19 ◼   ► and the reason to stick with it, but you're

01:27:20 ◼   ► kind of using MySQL in sort of a degenerate state. Like, you're using it as a really bad

01:27:25 ◼   ► performing poor man's key value storage with some basic filtering, doing your own joins

01:27:31 ◼   ► client-side, which is fine, because you get the other benefits of it. Your familiarity,

01:27:34 ◼   ► easy backup, easy replication, reliability, and that's why you're avoiding, that's why

01:27:39 ◼   ► you're not another MongoDB nightmare story.

01:27:42 ◼   ► Yeah, I think, obviously I know, as you and as the chat have said, I'm going to get a

01:27:47 ◼   ► lot of email about my no joins stance. But the fact is, first of all, you just said MySQL

01:27:54 ◼   ► is way slower than some other system, and that's probably not the case. Certainly it

01:27:58 ◼   ► is not always the case. The reason I use MySQL, even though I'm not using some of the relational

01:28:05 ◼   ► database type features like joins, I don't use stored procedures either, that's not a

01:28:11 ◼   ► thing where that's just asking for trouble, I don't use triggers, things like that. The

01:28:16 ◼   ► The reason I do this is because MySQL is just frickin' awesome.

01:28:21 ◼   ► There's no other way to say it.

01:28:23 ◼   ► I've heard so much crap about MySQL from people who don't use it and are prejudiced against

01:28:28 ◼   ► it or whatever.

01:28:29 ◼   ► And I don't use Postgres, I don't use Oracle, I don't have any experience with those, so

01:28:32 ◼   ► I can't tell you how it compares to those.

01:28:34 ◼   ► But I can tell you, most of the criticism I've heard about MySQL is wrong, or at best

01:28:39 ◼   ► outdated and only applied to my ISAM and not the more recent InnoDB storage engine, which

01:28:44 ◼   ► at this point is not even recent.

01:28:47 ◼   ► And so if you use MySQL with InnoDB as your storage engine

01:28:51 ◼   ► for all tables, it's amazing, it is awesome.

01:28:55 ◼   ► And I said this in a post recently,

01:28:58 ◼   ► and the gravity of this, I don't want to overstate this,

01:29:01 ◼   ► but it's hard to overstate this.

01:29:04 ◼   ► In all the time at Tumblr, Instapaper,

01:29:06 ◼   ► all that time, now Overcast, I've used MySQL a lot.

01:29:11 ◼   ► With Tumblr, it was under extreme stress

01:29:13 ◼   ► for the entire time I was working there.

01:29:15 ◼   ► It was constantly under extreme stress.

01:29:18 ◼   ► I have never seen MySQL crash once.

01:29:22 ◼   ► I have never seen MySQL corrupt data once.

01:29:25 ◼   ► That's amazing.

01:29:26 ◼   ► Look at any--

01:29:27 ◼   ► - You've never seen it corrupt data?

01:29:30 ◼   ► - I have never seen it corrupt data.

01:29:31 ◼   ► I've never seen it crash.

01:29:33 ◼   ► - MySQL's actual functionality could be argued

01:29:35 ◼   ► to be a corruption of data.

01:29:37 ◼   ► I know it's documented.

01:29:38 ◼   ► I know you're looking up and the doc says

01:29:40 ◼   ► that's exactly what it's gonna do.

01:29:42 ◼   ► It's not wrong.

01:29:43 ◼   ► It's not a bug.

01:29:44 ◼   ► But that's the philosophical difference

01:29:46 ◼   ► many, quote unquote, "real database" people have

01:29:48 ◼   ► with MySQL is the documented behavior.

01:29:51 ◼   ► When it performs that documented behavior,

01:29:53 ◼   ► database people grit their teeth and get angry at it.

01:29:56 ◼   ► And I'm kind of one of them.

01:29:58 ◼   ► I'm not that familiar with exactly what

01:30:00 ◼   ► you're talking about.

01:30:01 ◼   ► However, are you sure you're talking about InnoDB

01:30:03 ◼   ► and modern versions of this?

01:30:06 ◼   ► No, no.

01:30:07 ◼   ► Like weird, weird coercions of data types and cases

01:30:10 ◼   ► where columns that are marked as not null

01:30:12 ◼   ► can actually have nulls and weird things

01:30:14 ◼   ► with empty strings.

01:30:15 ◼   ► They're all documents.

01:30:16 ◼   ► You go to one of those MySQL gotcha pages

01:30:17 ◼   ► and just read through them.

01:30:19 ◼   ► In terms of performance, though, now that I'm saying

01:30:22 ◼   ► you should switch to a different thing or something,

01:30:23 ◼   ► but because most of the things that have massively better

01:30:26 ◼   ► performance have much, much less of the other things

01:30:29 ◼   ► you just described, like stability and reliability.

01:30:31 ◼   ► But performance-wise, if your data model

01:30:34 ◼   ► is kind of simple and never going to have joins,

01:30:38 ◼   ► And if you care about getting more performance, which you may or may not, there are lots of other database systems that do go way faster than MySQL, especially within ODB.

01:30:48 ◼   ► Because the extra overhead of that storage engine...

01:30:52 ◼   ► The new hotness is basically people do almost everything in memory.

01:31:00 ◼   ► It's not like memcache, although there is a persistent memcache thing, but a lot of them are like, "Look, we can't involve disk in the operation of our thing here."

01:31:07 ◼   ► And as you can imagine, when you do almost everything in memory, you go way faster than MySQL trying to do stuff with InnoDB.

01:31:12 ◼   ► Oh, sure. And there's ways you can take advantage of that with caching already.

01:31:20 ◼   ► And by the way, MySQL does already cache a lot in memory.

01:31:24 ◼   ► And of course, the OS will cache a lot of the disk blocks in memory and stuff like that.

01:31:27 ◼   ► But certainly you can use MySQL with Redis or Memcache or any of the cache systems that are out there.

01:31:33 ◼   ► I mean, most people do, it's fine.

01:31:35 ◼   ► You know, that's like people, there's this big,

01:31:39 ◼   ► it's the cool thing to use a different storage engine

01:31:42 ◼   ► these days, but the fact is a storage engine is,

01:31:45 ◼   ► oh boy, here we go.

01:31:46 ◼   ► A storage engine is like a file system.

01:31:48 ◼   ► You know, you don't wanna change that very often

01:31:51 ◼   ► and you want it to be extremely conservative

01:31:53 ◼   ► and extremely focused on data integrity.

01:31:56 ◼   ► And MySQL is so incredibly like battle tested.

01:32:00 ◼   ► It is so solid.

01:32:02 ◼   ► And yeah, I get that it might not have always been that way

01:32:04 ◼   ► back in the olden days with myISAM and all the crap,

01:32:06 ◼   ► but those days are well into the past now.

01:32:09 ◼   ► And modern MySQL, MySQL has been rock solid,

01:32:12 ◼   ► I would say for at least seven or eight years.

01:32:15 ◼   ► And that's why Google use it very heavily.

01:32:18 ◼   ► I don't know if they still do, but they at least did.

01:32:20 ◼   ► I know lots of major, major sites use MySQL.

01:32:23 ◼   ► Twitter use it for a long time, they probably still do.

01:32:27 ◼   ► There's a reason why, it's really, really good.

01:32:30 ◼   ► And there was this great blog post a few years back

01:32:35 ◼   ► when FriendFeed existed, to give you some idea.

01:32:39 ◼   ► And it was a blog post called

01:32:40 ◼   ► How FriendFeed Uses MySQL to Store Schema-less Data.

01:32:43 ◼   ► And I'll have to find it and put it in the show notes.

01:32:45 ◼   ► But there's a lot of good stuff in here.

01:32:47 ◼   ► FriendFeed was a weird project

01:32:50 ◼   ► that was run by incredibly good engineers.

01:32:52 ◼   ► And it basically addresses a lot of this stuff.

01:32:57 ◼   ► Like MySQL, even if you don't use

01:32:59 ◼   ► of the database-y features of it, MySQL is still an amazingly fast, solid, low-needs,

01:33:07 ◼   ► low-maintenance storage engine that has tons of tools, a huge ecosystem, tons of optimization

01:33:14 ◼   ► potential. There's so many reasons to use it. And if you look at one of the newer systems,

01:33:21 ◼   ► it's -- and things like Mongo or all these new NoSQL-type systems, if you look at a lot

01:33:28 ◼   ► of those, yeah, they have some benefits, but if you don't necessarily need those benefits,

01:33:32 ◼   ► and what you really want is for your database to be like a file system and to be basically

01:33:36 ◼   ► bulletproof, MySQL is a great choice. And I feel like it gets ignored because it's old.

01:33:42 ◼   ► This comparison is kind of harsh, so I don't mean it the way it's going to sound, but it's

01:33:47 ◼   ► as close as we can align them for us to align these two things.

01:33:50 ◼   ► Here we go.

01:33:51 ◼   ► In the world of databases, what is the PHP of databases? It's kind of MySQL. It's way

01:33:56 ◼   ► way better than PHP, don't get me wrong.

01:33:57 ◼   ► But if you have to map one to it, that's what ends up.

01:34:00 ◼   ► And what's the Python of databases?

01:34:01 ◼   ► That's probably Postgres.

01:34:04 ◼   ► That's more or less how things line up.

01:34:06 ◼   ► And what's the C, C++ of databases as Oracle?

01:34:11 ◼   ► And what's the Ruby?

01:34:11 ◼   ► It's like VaultDB or something.

01:34:14 ◼   ► But to align things, I say very clearly,

01:34:17 ◼   ► MySQL is the PHP of databases.

01:34:19 ◼   ► And that sounds terrible, I know,

01:34:20 ◼   ► because it is not as bad as PHP.

01:34:22 ◼   ► Do not get me wrong.

01:34:23 ◼   ► PHP is terrible.

01:34:24 ◼   ► But it has many of the same characteristics,

01:34:27 ◼   ► in that it's everywhere, it's a known quantity, it's reliable,

01:34:30 ◼   ► it has idiosyncrasies.

01:34:31 ◼   ► But once you know what those idiosyncrasies are,

01:34:33 ◼   ► they don't bother you that much.

01:34:34 ◼   ► And the thing about using it for schema lists,

01:34:36 ◼   ► that's why people do that.

01:34:37 ◼   ► Because they're like, OK, I have a tool that I know is reliable.

01:34:40 ◼   ► I know what it's capable of.

01:34:41 ◼   ► And I'm going to use it, like I said,

01:34:43 ◼   ► in a degenerate kind of way.

01:34:44 ◼   ► I'm actually going to use it for schema list data.

01:34:46 ◼   ► I know it's not a schema list database system.

01:34:48 ◼   ► I know there actually are schema lists,

01:34:49 ◼   ► but I'm going to define some table with columns

01:34:51 ◼   ► for key and value and version number and some other crap.

01:34:53 ◼   ► And it'll work.

01:34:54 ◼   ► And it's like you were taking advantage

01:34:56 ◼   ► of a system that works in a way that you're comfortable with,

01:35:00 ◼   ► that has all the features you need and is reliable,

01:35:02 ◼   ► and you're building something else on top of that.

01:35:05 ◼   ► And all I was saying with the other database type things

01:35:07 ◼   ► is if you care more about performance,

01:35:09 ◼   ► there is these days much more performance

01:35:11 ◼   ► to be had if you're willing to trade it for less

01:35:13 ◼   ► reliability and more unknowns.

01:35:15 ◼   ► And I'm hoping that that younger section of the data storage

01:35:19 ◼   ► world matures and coalesces, and we

01:35:21 ◼   ► get some sort of equivalent of MySQL, or even best case,

01:35:25 ◼   ► some equivalent of Postgres, where

01:35:26 ◼   ► it's like a tried and true known quantity within that realm.

01:35:29 ◼   ► Right now, we don't have it now.

01:35:30 ◼   ► Right now, it's kind of the Cambrian explosion over there,

01:35:32 ◼   ► and we're not sure how it's going to shake out.

01:35:35 ◼   ► But it's still worth keeping your eye on,

01:35:37 ◼   ► because if 20 years from now, you're still

01:35:40 ◼   ► using MySQL with no joins, there will

01:35:42 ◼   ► be either a great failure of the NoSQL world

01:35:45 ◼   ► to come up with a useful product for the long term,

01:35:48 ◼   ► or a great failure of you to keep an eye on what's

01:35:50 ◼   ► going on over there. Possibly both. I have two questions. Firstly, what is the

01:35:55 ◼   ► pearl of databases, Jon? Yeah, I was trying to think of that. I don't know. Maybe MSQL,

01:36:01 ◼   ► but neither one of you remember that. No, there really is no pearl, but if there is,

01:36:06 ◼   ► I don't know what it is. It's probably one of the newfangled database systems that I'm just not

01:36:11 ◼   ► familiar enough with. Because there's nothing that has the same combination of a precursor to all of

01:36:18 ◼   ► the better known ones now with lots of weird quirks.

01:36:21 ◼   ► I don't know what that is, but it's not MySQL, definitely.

01:36:26 ◼   ► Isn't it just some academic paper that swears that it's academically flawless?

01:36:31 ◼   ► No, that's Perl 6, you're thinking of.

01:36:34 ◼   ► Is there a database that can be written but not read?

01:36:36 ◼   ► PHP looks just like Perl.

01:36:37 ◼   ► You can't even make that comment.

01:36:41 ◼   ► Let's not go down that road.

01:36:42 ◼   ► Can we still back up to, you just spent, Marco, five or ten minutes explaining how MySQL

01:36:47 ◼   ► is amazing and reliable and does what it's supposed to do really well, but you don't

01:36:52 ◼   ► trust it to do a frickin' join? Like, I can't get past that.

01:36:55 ◼   ► No, no, it isn't an issue of trust. It's an issue of leaving yourself open for future

01:37:00 ◼   ► scaling options. It's about being able to say, "You know what? My database server

01:37:04 ◼   ► is going to at some point be split off from my web server once my service has more than

01:37:09 ◼   ► five users, and this is going to have its own server, and it's going to be harder

01:37:13 ◼   ► and more expensive to increase database hardware resources

01:37:17 ◼   ► than to increase application or web hardware resources. And so

01:37:21 ◼   ► you might as well have the application and web stuff do more of the work

01:37:25 ◼   ► and make the database, give the database an easier

01:37:29 ◼   ► job. So things like, you know, reduce the number of queries to it, but the number

01:37:33 ◼   ► of queries that go to it, you know, make them access fewer

01:37:37 ◼   ► rows, make them access fewer tables, make them have to do less CPU work, but it's more

01:37:41 ◼   ► about IO, you know, let the database do only what the database is required to do, and let

01:37:48 ◼   ► your other servers take on as much work as they can.

01:37:51 ◼   ► Okay, but you'd rather ping away at it and do a gazillion round trips to avoid these

01:37:58 ◼   ► joins? Like, I haven't been through a Tumblr, I haven't been through an Instapaper, so I'm

01:38:05 ◼   ► doing that thing which I told you I don't want to do in the beginning of the episode

01:38:08 ◼   ► where I'm saying that seems weird. But genuine question, have you ever done anything to empirically

01:38:14 ◼   ► prove to yourself? Like, have you used a new relic to prove to yourself that this is legitimately

01:38:19 ◼   ► the right way to go? Like, you can do it on paper. I think you could convince yourself, but I'm sure

01:38:24 ◼   ► I'm not going to answer for Marco, but I'm sure the answer to that is yes, because anyone who works

01:38:27 ◼   ► in a service I'd offer quickly realizes, oh, the part that has all the state is the hardest part

01:38:31 ◼   ► to scale. I mean, he basically just laid it out. Like, you can just do it on paper. There's a

01:38:35 ◼   ► certain amount of work that needs to be done. One part of your system you can scale really,

01:38:38 ◼   ► really, really easily.

01:38:39 ◼   ► One part is really hard to scale.

01:38:41 ◼   ► You want the hard to scale part to do the same--

01:38:44 ◼   ► whatever operation it does the most sufficiently, primary key

01:38:47 ◼   ► lookups maybe, have it do that same operation over and over

01:38:50 ◼   ► and over again.

01:38:51 ◼   ► And have everything else you can do

01:38:53 ◼   ► put on the easier to scale parts.

01:38:54 ◼   ► Like, there's lots of ways you can do this with math

01:38:56 ◼   ► to figure out how it works.

01:38:59 ◼   ► But I think even if you just think about that logically

01:39:02 ◼   ► and reason through it--

01:39:04 ◼   ► and that's not even going to get you linear scaling.

01:39:06 ◼   ► We're just trying to keep our heads above water

01:39:08 ◼   ► with this type of strategy.

01:39:09 ◼   ► You always want to push the work out

01:39:13 ◼   ► to the system that's easiest to scale.

01:39:14 ◼   ► And so it's kind of the Google approach with Bigtable,

01:39:17 ◼   ► where they went to the extreme where their data store was

01:39:21 ◼   ► really annoying to deal with and unreliable.

01:39:23 ◼   ► And applications had to retry and figure out

01:39:26 ◼   ► how to resolve conflicts.

01:39:27 ◼   ► And it made it incredibly hard to write applications.

01:39:30 ◼   ► But that was the price of scaling early on.

01:39:32 ◼   ► And they've made strides in that way.

01:39:33 ◼   ► But basically, you're trying to move

01:39:35 ◼   ► work to the systems that are easy to scale, and web servers are way easier to scale. And

01:39:39 ◼   ► not performance, scale. And again, you're confusing performance and like, "Oh, you've

01:39:44 ◼   ► got to take those round trips. You're going across the network. You're making multiple

01:39:47 ◼   ► queries." It's like, that's performance. We're talking about scaling.

01:39:50 ◼   ► Yeah, that's a fair point. And I did that deliberately.

01:39:55 ◼   ► For whatever it's worth, you're putting a lot of weight on these round trips. Round

01:40:00 ◼   ► trips to a database server or to a memcache server that's in the same rack, or at least

01:40:06 ◼   ► in the same data center as the server that you're coming from.

01:40:09 ◼   ► That's a good point.

01:40:10 ◼   ► We're not talking about a lot of time here. A lot of this depends on the kind of application

01:40:16 ◼   ► you're writing. How much data are you actually querying? To build the page or to build the

01:40:20 ◼   ► API response that you're talking about, how many database calls do you actually have to

01:40:24 ◼   ► Is it seven or eight? Is it fifty?

01:40:28 ◼   ► Is it more? What you're doing matters a lot.

01:40:32 ◼   ► Instapaper was fairly easy to scale

01:40:36 ◼   ► because it was never as big as Tumblr and I was using Tumblr style techniques

01:40:40 ◼   ► at Instapaper so of course it was way overkill.

01:40:44 ◼   ► Which was great, which is why I was able to afford to run it

01:40:48 ◼   ► and not go crazy and not go broke. But

01:40:52 ◼   ► At Tumblr we face things like, to give you one example,

01:40:55 ◼   ► how do you display the list of posts

01:40:59 ◼   ► on somebody's dashboard?

01:41:01 ◼   ► So you have to figure out who they follow

01:41:03 ◼   ► and then of everyone they follow,

01:41:05 ◼   ► to find enough posts from those people

01:41:08 ◼   ► to make 20 of them and order them properly.

01:41:10 ◼   ► There are so many different ways to do that.

01:41:12 ◼   ► And there are the naive ways where you do a few joins

01:41:16 ◼   ► and then you do a big sort and it's all fine.

01:41:19 ◼   ► The naive way does not really scale very well

01:41:22 ◼   ► just because you start dealing with hundreds of gigs of data pretty quickly and the database

01:41:27 ◼   ► having to scan millions of rows and that starts performing very badly.

01:41:33 ◼   ► So a lot of times what you think is the most good, naive approach actually in practice

01:41:42 ◼   ► is not useful. It does not scale well enough. It becomes too expensive or becomes completely

01:41:48 ◼   ► impractical or impossible to do at scale. So you start having to do weird little

01:41:53 ◼   ► hacks. And so you know one of the hacks that we did at tumblr early on was

01:41:58 ◼   ► actually something that I learned from Wikipedia what they did on their schema

01:42:01 ◼   ► back and this was probably 2007 or 2008 which is you actually have basically a

01:42:07 ◼   ► separate table that acts as an index because in certain ways it uses memory

01:42:13 ◼   ► more efficiently to do that or in certain ways the query optimizer will

01:42:16 ◼   ► use that better. And so we had a posts reference table that was basically the... It was a special

01:42:23 ◼   ► index of the posts table, and the posts table was this massive thing that was hundreds of

01:42:29 ◼   ► gigs that would only ever be accessed by primary key. Because, as John said, primary key accesses

01:42:33 ◼   ► are very fast and everything's optimized for that. They cache very well, everything else.

01:42:37 ◼   ► So you get the list of post IDs from other calls, and then you could fetch cached posts

01:42:42 ◼   ► or fetch posts from the database or even get them from slaves, and it was mostly all right.

01:42:46 ◼   ► So there's all sorts of things you can do

01:42:48 ◼   ► if you leave yourself options.

01:42:49 ◼   ► That's what I'm saying.

01:42:50 ◼   ► So, and you know, look, I would love,

01:42:53 ◼   ► again, this is like the idea,

01:42:54 ◼   ► this is almost the final discussion.

01:42:56 ◼   ► Academically, I would love to say, yeah,

01:42:59 ◼   ► you know, just use the database as intended

01:43:00 ◼   ► and it works great and that's the easiest and fastest.

01:43:03 ◼   ► In practice, it's not.

01:43:05 ◼   ► In practice, things are more complicated

01:43:07 ◼   ► and you can do the nice simple way for a while,

01:43:10 ◼   ► but eventually you're gonna hit a wall.

01:43:13 ◼   ► And what options you have when you hit that wall

01:43:16 ◼   ► depends on how you've built the system,

01:43:18 ◼   ► and how much effort it is to get over that wall,

01:43:20 ◼   ► and how much it will cost you,

01:43:22 ◼   ► in both things like hardware or service costs,

01:43:25 ◼   ► and in administration time and maintenance over time.

01:43:30 ◼   ► That will all depend on how you've built that system,

01:43:32 ◼   ► and if you've built it to give yourself

01:43:34 ◼   ► good scaling options in mind,

01:43:35 ◼   ► and to do things like be gentle on your databases,

01:43:39 ◼   ► you will have more options,

01:43:41 ◼   ► and it will be cheaper and easier to scale.

01:43:43 ◼   ► - Here's an analogy that I think

01:43:45 ◼   ► case you will be able to identify with.

01:43:47 ◼   ► It's like, you know when you have like-- well, maybe in C++

01:43:49 ◼   ► at least-- when you have code and you want to make it faster,

01:43:52 ◼   ► very often that involves making it longer and uglier?

01:43:55 ◼   ► This is the same type of thing.

01:43:56 ◼   ► You're like, how could it be faster?

01:43:58 ◼   ► I'm adding more and more lines of code.

01:43:59 ◼   ► Like whether you're unrolling a loop

01:44:01 ◼   ► or setting up some big bunch of setup stuff

01:44:03 ◼   ► to do a bunch of SIMD operations on it, it gets longer

01:44:06 ◼   ► and it gets uglier and it looks like you're doing more.

01:44:09 ◼   ► This seems like it's more work.

01:44:10 ◼   ► In the same way, you're like, well,

01:44:12 ◼   ► I had this nice query that got me all the data I wanted

01:44:14 ◼   ► one big self-consistent blob with this nice joint query, and replacing it with these multiple

01:44:18 ◼   ► queries, when I just mesh the data together myself, it's longer. How could that be better?

01:44:24 ◼   ► I guess it's not fully accurate in the C++ case. You're actually increasing performance,

01:44:28 ◼   ► and here, like Marco said, you're protecting yourself for future scaling. Although in some

01:44:33 ◼   ► cases, even in the deep dark world of databases, you've mentioned stored procedure before,

01:44:37 ◼   ► if your query planner is not your friend, like say if you use Informix, speaking from

01:44:41 ◼   ► of experience.

01:44:43 ◼   ► Sometimes, if you want to get a massive amount of data,

01:44:45 ◼   ► it's better to write a stored procedure

01:44:48 ◼   ► and essentially be the query planner yourself.

01:44:49 ◼   ► Do it all in the database, all right on top of the thing,

01:44:52 ◼   ► but say, look, when I give you this join,

01:44:54 ◼   ► you're going to do something dumb.

01:44:55 ◼   ► So let me run this query, this query, this query,

01:44:57 ◼   ► join the mesh results together, put them in a temp table,

01:45:00 ◼   ► index the temp table, join against that temp table

01:45:03 ◼   ► with the second thing.

01:45:03 ◼   ► And you're like, how could that possibly

01:45:05 ◼   ► be faster than just running the query you wanted?

01:45:07 ◼   ► It's like, well, the query planner

01:45:08 ◼   ► made some very unfortunate life choices

01:45:09 ◼   ► when we sent it this query.

01:45:11 ◼   ► And as a matter of fact, doing this crazy stored procedure--

01:45:14 ◼   ► I mean, once you're creating temp tables and indexing them,

01:45:17 ◼   ► you're like, there's no way that could be faster

01:45:18 ◼   ► than running that query.

01:45:19 ◼   ► Well, let me show you.

01:45:21 ◼   ► Databases are funny things.

01:45:22 ◼   ► And the more you can treat them the way

01:45:25 ◼   ► Marco's treating them, the less headaches

01:45:27 ◼   ► you'll have about them.

01:45:28 ◼   ► And I think the more you should think about whether you should

01:45:31 ◼   ► even be using a database.

01:45:32 ◼   ► But that's a conversation another time.

01:45:34 ◼   ► But anyway, the idea that the client side code

01:45:38 ◼   ► the consuming code gets longer and uglier and more complicated, that is not outside

01:45:43 ◼   ► the realm of normal programmer experience when just increasing the performance of your regular compiled code.

01:45:48 ◼   ► Yeah, that makes sense.

01:45:55 ◼   ► And it's a balance. It's a balance you have to strike. Obviously, if you're hitting severe problems and severe scaling challenges,

01:45:56 ◼   ► you're going to have to go more in that direction of more complex code.

01:46:04 ◼   ► And that'll be more bugs. You'll have things like weird caching bugs and things like replication

01:46:08 ◼   ► delay or eventual consistency bugs depending on what kind of structure you're using.

01:46:13 ◼   ► And that's all hard, that's all complicated. And

01:46:18 ◼   ► certainly you shouldn't do that sooner than you have to in most cases. But again,

01:46:23 ◼   ► it's a balance. Just like you shouldn't waste all your memory as much as possible when you're writing a C program,

01:46:28 ◼   ► or any program for that matter, you know, you've got to find the balance.

01:46:33 ◼   ► And with servers and scaling, I would lean a little more,

01:46:37 ◼   ► just because the nature of using someone else's

01:46:39 ◼   ► vast resources on their computer

01:46:41 ◼   ► versus expensive shared server resources,

01:46:44 ◼   ► I would lean a little bit more towards

01:46:46 ◼   ► a little more complicated, but way more scalable.

01:46:50 ◼   ► - That makes sense.

01:46:51 ◼   ► And far be it for me to argue with you, Mr. Tumbler.

01:46:54 ◼   ► - Thanks.

01:46:56 ◼   ► Can I get that on a business card?

01:46:57 ◼   ► All right, thanks a lot for our three sponsors this week,

01:46:59 ◼   ► PDF pen for iPad, New Relic, and Igloo.

01:47:03 ◼   ► And we will see you next week.

01:47:05 ◼   ► (upbeat music)

01:47:08 ◼   ► ♪ Now the show is over ♪

01:47:10 ◼   ► ♪ They didn't even mean to begin ♪

01:47:13 ◼   ► ♪ 'Cause it was accidental ♪

01:47:15 ◼   ► ♪ Oh it was accidental ♪

01:47:18 ◼   ► ♪ John didn't do any research ♪

01:47:20 ◼   ► ♪ Marco and Casey wouldn't let him ♪

01:47:23 ◼   ► ♪ 'Cause it was accidental ♪

01:47:26 ◼   ► It was accidental And you can find the show notes at ATP.FM

01:47:34 ◼   ► And if you're into Twitter You can follow them at

01:47:39 ◼   ► C-A-S-E-Y-L-I-S-S So that's Casey Liss

01:47:44 ◼   ► M-A-R-C-O-A-R-M Aunty Marco Armin

01:47:50 ◼   ► S-I-R-A-C-U-S-A-C-R-A-C-U-S-A

01:47:55 ◼   ► It's accidental, accidental They didn't mean to

01:48:00 ◼   ► Accidental, accidental Tech podcast so long

01:48:08 ◼   ► I'd like to tell you about why I hate my car now.

01:48:10 ◼   ► Wow.

01:48:11 ◼   ► Is it because-- did it get dirty finally?

01:48:13 ◼   ► No, it is actually relatively dirty at the moment.

01:48:18 ◼   ► Aaron and I decided this past Saturday to go to the local drive-in movie theater.

01:48:24 ◼   ► And if you have a local drive-in movie theater, I cannot suggest enough that, assuming you

01:48:29 ◼   ► don't have small children, which I guess definitely eliminates Marco and probably eliminates John

01:48:34 ◼   ► in this particular context, if you don't have small kids, you can escape the kids, go to

01:48:39 ◼   ► a drive-in movie.

01:48:40 ◼   ► It's really cool.

01:48:41 ◼   ► And so the way the Goochland drive-in theater works, which is between Charlottesville and

01:48:46 ◼   ► Richmond is you pull in, you pay something like $8 a person in your car, and you get

01:48:52 ◼   ► a double feature.

01:48:54 ◼   ► And you get the audio for the double feature by tuning to a FM station that the theater

01:49:00 ◼   ► broadcasts.

01:49:01 ◼   ► And it's a really cool experience, especially on a really pretty night like this past Saturday

01:49:06 ◼   ► was.

01:49:07 ◼   ► And we have been several times in the past, although not for a year or two.

01:49:10 ◼   ► And when we've gone in the past, we've taken my car, which at the time was my Subaru.

01:49:15 ◼   ► What color was that?

01:49:17 ◼   ► That was white.

01:49:18 ◼   ► We take an Aaron's car, which was and is still a Mazda6, which is a grayish silver, and we've never had an issue.

01:49:25 ◼   ► Well, this past Saturday, I had just washed and waxed and leather-conditioned and vacuumed Aaron's car,

01:49:32 ◼   ► and so I thought, "Well, my car's dirty," because yes, my car does get dirty, it just takes longer than 40 nanoseconds,

01:49:38 ◼   ► um, unlike Marco's car. So we decided to take the BMW.

01:49:43 ◼   ► The BMW has a push button starter and it has an accessory mode.

01:49:48 ◼   ► And so I figured self, there will be no issue here.

01:49:52 ◼   ► What we'll do is we will go to the drive-in theater.

01:49:56 ◼   ► I will put the car in accessory mode by turning it off and then

01:49:59 ◼   ► pressing the push button once.

01:50:01 ◼   ► And we will listen to the movie on the stereo and it will be wonderful.

01:50:05 ◼   ► I will turn off the iDrive display.

01:50:06 ◼   ► I had to figure out how to do that because I completely forgot.

01:50:09 ◼   ► And I will turn off the iDrive display and everything should be good and right

01:50:13 ◼   ► in the world. So that's what we did. And after 10 minutes, I heard the BMW chime that you

01:50:20 ◼   ► might remember from neutral. And it was telling me that my battery was dying.

01:50:25 ◼   ► What? After 10 minutes of having the radio on.

01:50:28 ◼   ► Doesn't your car have two batteries? Or is that just Marco's car?

01:50:31 ◼   ► I don't think Marco's car does, does it? Not that I know of.