61: Perfectly Neutral

0:00:00So your tweet about us having thin topics like oh, there's a lot of stuff

0:00:03We didn't even get the Apple wearables, but after that you're right, then we're out of topics. I

0:00:06Just have one item. I guess it was a popular item, but just one last week we were talking about

0:00:13How if you have some sort of service that?

0:00:16Deals with your photos and lets you share them no matter what cool features you have you got to at least

0:00:21Also have a way to send a link to somebody they can view in any web browser

0:00:24and we also talked about

0:00:26photo stream and iOS integration and stuff like that.

0:00:29And although we didn't come out and say it directly,

0:00:32the implication was that PhotoStream did not have a public URL,

0:00:35but PhotoStream does have public URLs. You can make a little URL

0:00:38for a website that you can share with somebody that lets them see

0:00:41the pictures in your PhotoStream.

0:00:44I use PhotoStream with my family, and my family

0:00:46all has iOS devices, and the great thing about that is

0:00:48it notifies you on your phone or on your iPad,

0:00:51and when you get the notification, you can just swipe and see the pictures immediately.

0:00:54immediately, it's a much superior interface for viewing pictures, but you've always got

0:00:59to have that web interface underneath it all.

0:01:01And this came up in the context of Carousel, which I think, Casey, you said they didn't

0:01:04have a way for you to just get an easy web link?

0:01:07That was my experience, that's correct.

0:01:09So I had emailed what was to Carousel a completely random email address that wasn't associated

0:01:16with the Dropbox account.

0:01:17The email address, in fact, was my work email, and when I received the email from personal

0:01:23and I'm doing that in air quotes to WorkMe, the work email basically said, "Hey, you need to

0:01:28install Carousel if you want to do anything useful." And I think it did have like six or

0:01:32seven pictures in it, little thumbnails of the pictures that I'd actually selected. So it wasn't

0:01:36like stock photography or anything like that. But with that said, it didn't do anything and there

0:01:42was no web-based link. It basically said, "Go to the app or get the hell out of here."

0:01:46**Matt Stauffer** So do we have any more follow-up?

0:01:49And Casey, do you feel like you need to follow up at all about any topics from last week?

0:01:53Or is that it?

0:01:54Do you know what you did, Casey?

0:01:56Do you want to tell us all what you did?

0:01:58I'm sorry, Daddy.

0:02:01I don't know.

0:02:03I'm a little—not bitter—I'm a little frustrated with the feedback from last week.

0:02:10And I was debating talking about it, and since you've prompted me, I'll go ahead and

0:02:14do so and just make my world even worse for myself.

0:02:18And the thing that frustrated me about the feedback we received is that there were a

0:02:23few instances, and Mark Edwards is from Bajango, Django—I always pronounce it wrong, I'm

0:02:29so sorry, Mark—but anyway, he wrote a relatively long missive that he put up as a gist, a gist,

0:02:36whatever. What am I thinking of?

0:02:37Jest.

0:02:38Thank you. I doubted myself. Anyways, I'm all out of whack. So he put up a gist, and

0:02:44He explained as someone who used to do this sort of thing for a living why vinyl is basically

0:02:50the most evil thing in the world.

0:02:51And while I don't entirely agree with what he said, I can't really factually debate

0:02:59it.

0:03:00And so things like that—and we'll put a link in the show notes—that was actually

0:03:04welcome and useful.

0:03:06But I got a lot of feedback—we got a lot of feedback and I got a lot of feedback—which

0:03:10basically amounted to "I read something on the internet once, it must be true, you're

0:03:16crazy and stupid."

0:03:18And that kind of bothers me a little bit because it was oftentimes not really based in fact,

0:03:26or sometimes it was, but never ever ever based on experience.

0:03:30And I'm not going to try to rehash the argument, but suffice to say I put a lot more stock

0:03:34when I receive feedback from someone who has had the experience of listening to vinyl on

0:03:40a really nice setup over someone who has just read some things or seen some stuff on the

0:03:45internet and swears that that's true. Now yes there's a lot of science behind it,

0:03:50yes I understand that, yes I'm probably crazy. All of that is fine, I agree, I understand

0:03:56it. It was probably silly of me to say that the fidelity of vinyl was better than the

0:04:00fidelity of CDs. But when it comes down to it, something I said on Twitter, which I actually

0:04:07was fairly proud of myself, I'm patting myself on the back, is that it's very kind of cliche,

0:04:12but life is what happens between one and zero in my eyes. And whether or not the accuracy of the

0:04:18reproduction on vinyl or CD is better, I just happen to prefer the feel and to some degree,

0:04:25the what do you call it? The tea something? I already forgot. Tea ceremony, Japanese tea

0:04:30ceremony? Google it.

0:04:31Yeah, I know what you mean. So I prefer the tea ceremony of it. And the only thing I will

0:04:37say to defend myself, other than come at me once you've had personal experience, you

0:04:42damn nerds, the only other thing I'll say is, why is it, Jon, that you choose to drive

0:04:50a manual transmission in 2014?

0:04:53That's not even close to a fair comparison.

0:04:55I was going to let you slide, but you come back at me with this, and you're just like,

0:04:59What I was going to let slide is that thing with life is what happens between one and

0:05:03zero.

0:05:04That vaguely alludes to terrible illogical reasoning behind crazy—you know what I mean?

0:05:11I know that's not what you're getting at at all, I understand, but the fact that you chose

0:05:15to use one and zero is like another nudge at the "Come on guys, we all know that digital

0:05:19stuff is just cold and unfeeling and—you know what I mean?"

0:05:24I did.

0:05:25I did.

0:05:26But then you came at the other thing—I found the feedback frustrating as well, and you

0:05:29would think. I wouldn't find the feedback frustrating because I think that most of the

0:05:31feedback agreed with Marco and I. But I found the feedback very frustrating because, I mean,

0:05:36this is a common phenomenon, feedback for the show. Very often we will get feedback,

0:05:40no matter what the topic, on past shows and everything, where someone will be—I mean,

0:05:44this is not bad feedback. It's good because it shows enthusiasm. You know, someone who's

0:05:48a big fan of the show will write in and in some ways this is the best kind of feedback,

0:05:52where they'll be like, "Listen to your show. Maybe think about this, and here's what I

0:05:56think about it, and they'll present a bunch of ideas about the topic. But what I find frustrating

0:06:01is a lot of the time, the ideas that they present are things that we talked about on the show.

0:06:07And you don't want to write back and say, "Yeah, we talked about that, this exact thing,

0:06:12at length on the show." But then it sounds like you're not valuing their feedback, and it's good

0:06:15that they're enthusiastic about it, but sometimes it's frustrating to see, like, "I obviously failed

0:06:19to communicate this idea because it keeps coming back to me as if it's information that we are

0:06:24missing or if it's new information or new insights. And it's like, we talked about it

0:06:27for 15 minutes. We said almost these exact words. And it's like, sometimes it's people

0:06:33sending feedback in real time, or they listen, and then I guess they pause it or whatever,

0:06:37and then they write their three-page missive, and then they start playing again. And then,

0:06:41you know, because they haven't gotten to the end of the podcast and realize we addressed

0:06:44the point. And that happens with tweets, you know, obviously, that's fine. Sometimes it

0:06:47happens with emails too. But a lot of the feedback was like presenting points back to

0:06:52any of us, me, Marco, you, that I thought I could swear we covered on the show.

0:06:57Like we talked about these exact things and it's like maybe we're not communicating effectively

0:07:02or just like we're talking past our audience or something.

0:07:07And in the same way I saw a lot of arguments directed at both Casey and us that I thought

0:07:12we talked about on the show and addressed all different sides of coming back at us in,

0:07:16you know, arguments that I thought I had, you know, refuted and settled down, arguments

0:07:22that Casey never made, arguments that we never made,

0:07:24positions that we never staked out,

0:07:26the whole nine yards.

0:07:27This is obviously a very fraught issue.

0:07:30And I think, Casey, the problem you're still having with it,

0:07:32and the problem that you continue to have with it,

0:07:34is that you just gotta--

0:07:37you have to just make it clear what it is.

0:07:40In the world of internet, there's

0:07:41moving the goalposts.

0:07:42You gotta make it clear what everybody's talking about.

0:07:44Because at this point, I think everybody on this podcast

0:07:46all agrees.

0:07:47It's just that you keep wanting to put the goalposts over there

0:07:49and pretend that's what we were talking about.

0:07:50And me and Marco keep wanting to put the goalposts over here

0:07:52and pretend this is what we're talking about. And I'm trying to keep them kind of where the

0:07:57original thing was, which was the whole idea on IRL Talk where Faith said, "I'm not going to buy

0:08:02a device that purports to have high-quality music." And you're right, fidelity is a better word. It was

0:08:06pointed out to us by Dr. Drang. We should have been using that word, which could be part of the

0:08:09communication problem here. But anyway, "I'm not going to buy a device like the Pono because if I

0:08:14want high-quality, meaning high-fidelity music, I'll just listen to vinyl." And that, I think,

0:08:18is a clear implication that we're talking about the pono and the whole big deal with that is that

0:08:25there's more information in the music. And it's like, well, if I want more information in the

0:08:30music, if I want higher fidelity, if I want a more accurate reproduction of sound, I will go to vinyl.

0:08:35And you seem to follow down that trail. And my objection and Marco's objection is that vinyl

0:08:40is not good at reproducing sound and not as good as CD. And all this other stuff I think were great

0:08:45for us to talk about it, and it's good to define the boundaries of what we believe in everything,

0:08:49but you should be happy with that. It's not as if you shouldn't cling to that idea,

0:08:56like that fidelity thing. If that's not what you're talking about, then fine. Then you agree

0:09:00with us there, and we agree with you over the tea ceremony, and everybody's happy.

0:09:03One of the things that keeps getting thrown back in is, I like to reframe a lot of these people's

0:09:12arguments and the popular one is like if I said look if I'm really interested in

0:09:17high fidelity images I'll go to Instagram like that's the effectively

0:09:22the argument they're making like you know I don't need a full-frame camera

0:09:26with a really great sensor and lots of megapixels if I'm interested in

0:09:30photographic fidelity I will look at a camera from picture of 1992 on Instagram

0:09:35like that's the argument that many people are making and unknowingly that

0:09:39they're making, that they're completely combining what they think looks good, which may be like

0:09:43a super grainy Instagram filter, and the concept of fidelity, which is a straightforward thing

0:09:48that can be measured and talked about in objective terms, more or less.

0:09:53And people coming back with, like, "I love tube amps," and everything like that, you

0:09:56know, instead of digital amps, like, that's not what we're talking about. It's like, whatever

0:09:59it is that you think sounds good, all we're talking about is, given that sound — and

0:10:03I probably made a mistake using live instruments because then I had to deal with all the people

0:10:06talking about live music. Given some sound that you think is awesome, what is the best

0:10:12way to transport that sound through space and time to reproduce it elsewhere? That's

0:10:16all we're talking about. It has nothing to do with whether you like tube amps or solid

0:10:20state amps, whether you like six string guitars or five string, whether you like someone singing

0:10:23through a paper bag or not. Whatever the sound is, you've produced a sound, a song, live,

0:10:27recorded, whatever, you want to get that across space and time to someone else. You want to

0:10:31put it on something so that that sound that you love can get to them exactly as it is.

0:10:35it have? Whatever that sound is, master it however you want, do whatever you want. How

0:10:38do you get that sound? CD versus vinyl. That's what we were talking about. Everything else

0:10:43is like immaterial. I don't care if you like the sound of someone singing through a paper

0:10:48bag. No, no, now I'm talking about vinyl. I don't care if you like Tom Waits. I don't

0:10:53care what kind of music you want. We were saying you have something there and again,

0:10:57live music was easy because it's like, "Oh, I hear that now." You just want to put that

0:11:01on something, so it could be played back elsewhere. And if your choices are CD and vinyl, and

0:11:05you care about the fidelity of that reproduction, you should pick CD instead of vinyl. And that's

0:11:10it. And everything else people want to talk about, it's so hard for them to maintain focus.

0:11:14So it's hard for us to maintain focus because we have drifting. But it's a difficult topic

0:11:18to talk about, I think.

0:11:19Yeah, and I think you're right. And I think you're right in saying that we all agree that

0:11:23if fidelity is truly the issue, that I can get behind the science, even though I give

0:11:29it a little bit of side-eye, nevertheless, I can get behind the science that says I'm

0:11:34crazy, and that digital mediums are the better, higher fidelity method of reproducing music.

0:11:41But just as you said, I actually happen to prefer the tea ceremony, the emotion behind

0:11:47it. Like I was talking to my wife Erin about it, and she said, "Well, you know, you have

0:11:49to consider you grew up with vinyl, and this was something that had—that carries a lot

0:11:53of emotional baggage." I think John—or one of you guys might have said that last

0:11:56week. Anyway, it carries a lot of emotional baggage that for most humans, especially in

0:12:01this day and age, they don't have that emotional baggage. And I say baggage actually in a good

0:12:05way in this context. I'm sure you probably think of it as a bad thing. But nevertheless,

0:12:09I don't know what one thing I learned from this, and I'm really being serious is that

0:12:13it is very frustrating to say something which admittedly is a bit contrarian. And admittedly,

0:12:20I'd kind of picked this fight. So admittedly, I kind of made this bed for myself. But man,

0:12:26it frustrating to say something that you truly believe, and then the whole of the internet

0:12:31decides to come out and tell you how wrong you are?"

0:12:34And that in and of itself, I can deal with that. That's fine. I've been wrong plenty

0:12:37of times. I'll be wrong plenty of times again. But it was very frustrating for me

0:12:42to receive a whole bunch of feedback from people who had perhaps never even heard vinyl

0:12:46before in their lives, yet decided to take the time out of their day to explain to me

0:12:52how wrong I am. And I actually want—I'm twisting this not—I don't want that to

0:12:58sound like a complaint. What I want it to sound like is I learned a little bit that

0:13:03I think I do that to people sometimes. And I can't think of a great example other than

0:13:07maybe snickering at somebody pulling out an Android phone, for example, but—

0:13:10Let me get that for you.

0:13:12Well, I don't want to be that guy, and I think I am that guy. And so it was a very

0:13:16good learning experience for me that I need to stop being that guy. Does that make sense?

0:13:21Yeah, I think I don't know. I mean, you know, we're always gonna get oh god

0:13:26I have so much more to say about this, but I'm trying to resist

0:13:28We're always gonna get

0:13:31You know people who disagree with what we say we always do every week. I mean we get yep

0:13:36Yeah, you know to give listener some idea we get probably I would say 20 feedback emails a day these days

0:13:44It's a lot

0:13:45I'm shocked how much feedback we get

0:13:48from this show. And I should very quickly interject and say I can't speak for you two,

0:13:52but I try very hard to read every single piece and at worst

0:13:57skim every single piece of feedback that we get. Yeah, me too.

0:14:00And it really does like when you only get like a couple a week, that's easy to keep up with. Now

0:14:07it's getting hard to keep up with. Like it's getting—we get so much feedback now that it's kind of difficult

0:14:12because some of them, you know, some of them are a few lines and that's great.

0:14:16some of them are like six paragraphs, six dense paragraphs at that, and those like,

0:14:21it takes some serious time to get through some of these. But anyway, you know, I think we are

0:14:27expressing our opinions in public. Lots of what we say is going to be argued with, that's just the

0:14:33nature of putting yourself out there. You know, like I like, I mean, I've been putting myself out

0:14:40there and my often rash or incorrect or badly stated opinions out there online for the better

0:14:48part of a decade now pretty consistently and whenever I say something that everyone jumps

0:14:53down my throat for for being totally wrong or off base or misinformed or just not very

0:15:01well stated I learn something from that and I appreciate that because most people don't

0:15:07expose themselves to that much criticism. Most people don't have an audience to tell them when

0:15:14they're wrong and who actually will tell them when they're wrong and an audience that's big

0:15:18and diverse and smart enough to notice when they're wrong and to be able to express that in any kind

0:15:23of useful way. So most people tend to be terrible arguers, have narrow worldviews, etc. etc. and

0:15:30it's hard to get people to change their mind when they've spent their whole life thinking that

0:15:37that they're right and no one has really ever challenged them on that.

0:15:40So I'm very thankful that we have the opportunity, which most people don't have.

0:15:45We as podcasters, and anyone else who's a podcaster or a blogger or any other way that

0:15:49you can put yourself out there, we are lucky that we have the opportunity to be criticized

0:15:55by a massive worldwide audience.

0:15:58Even if your audience is 25 people, that's still 25 random strangers who don't care as

0:16:03much about how you feel as your family does and who are willing to tell you when you screw

0:16:07something up. And as the audience gets larger, it hurts more when you screw up because you

0:16:13have more people telling you that you're an idiot. But I think it is useful. And so,

0:16:20in your case, as Jon said, in your case, you have, with the vinyl thing, you expressed

0:16:27valid opinions there. However, you worded them in a way, and you made a generalization

0:16:33that you couldn't support.

0:16:35And so the feedback kept beating that

0:16:37into your head basically.

0:16:39(laughing)

0:16:40- And it's true.

0:16:41- And it's, you know, it certainly was not,

0:16:43certainly was not probably easy to read everyone saying

0:16:47that you were wrong, although by the way,

0:16:48there were a lot of people who agree with you,

0:16:50I should point out.

0:16:51But like your main point was that it sounds better

0:16:57because it sounded better to you.

0:16:59A solid argument would have been,

0:17:01you like the sound better.

0:17:03- Right, and I think I fell on my face

0:17:06because I painted it, like John was saying,

0:17:07and like you're saying now,

0:17:09I painted it as a universal statement

0:17:11when I really shouldn't have,

0:17:12that it was factually better,

0:17:14that it was superior to CDs.

0:17:17And while I might prefer the experience

0:17:19and the quirks of the sound,

0:17:21I don't think that I should have gone as bold as I did.

0:17:25- And as one of the feedback things,

0:17:28actually a couple of people did this,

0:17:29Like, presented, if we could just take that sound

0:17:31that you love and record it onto CD

0:17:33and play it back to you in a way that you didn't know

0:17:36whether we were playing back the vinyl or not,

0:17:38you would not be able to tell.

0:17:40Like, we would be exactly accurate

0:17:41that you're producing that vinyl sound

0:17:43that you love with every nuance that's available.

0:17:45You know what I mean?

0:17:47And that gets to the heart of the matter.

0:17:49And it's like, we're all for you liking whatever sound you like.

0:17:52But like, when we talk about mediums,

0:17:55when we just say the word vinyl, you're not talking about a song

0:17:57or a sound.

0:17:58talking about a medium and that's a different topic. There's so many different ways to try

0:18:02to come at this, to try to get people to understand it, let's just keep going back to

0:18:05whichever side they're on, whether like "No way, man, vinyl is awesome" or like "Vinyl is always

0:18:09evil." People just don't want to define the boundaries of what it is that they're saying,

0:18:14and it's impossible to come to any sort of agreement or disagreement if no one is defining

0:18:19the boundaries of what they're saying. A lot of, especially the tweet-length ones, which you can't

0:18:23blame. It's hard to talk about this in a single tweet. I fell victim to that, both on all sides

0:18:29of the debate. The best ones where they say—I like tweets on any topic where they'll say,

0:18:33"I agree with Casey about this. I agree with Marco about this. I disagree with John about this."

0:18:37And they will ascribe to all three of us opinions that 50 percent of the time we didn't take.

0:18:43"I agree with Marco that Starbucks coffee is great. I agree with Casey that all cars should

0:18:49should be black and I agree with John, the pearl is a terrible language, you know, that's

0:18:52a silly example, but it's like, you put our names in a tweet and you've expressed your

0:18:57own opinions, but you've by implication described the opposite opinions to us and I don't think

0:19:02we ever, anyway, it's the excitement of feedback. I agree with Marco, getting this feedback,

0:19:07as I've said many times, is very valuable, we're lucky to get a lot of it. You too can

0:19:12get a lot of it as well if you just go to online communities and whatever topic you're

0:19:15interested in and just start, you know, like I said, a Usenet in my past where it wasn't

0:19:20as big as audiences we have here. But it doesn't take, you know, it only takes like four or

0:19:23five mean people to put you in your place for a couple of years and you'll get better.

0:19:27Great.

0:19:28Yeah, and again, I don't mean to complain necessarily about getting feedback. I don't

0:19:33mean to complain about all of the internet or most of the internet thinking I'm wrong.

0:19:38That actually doesn't bother me. What was frustrating was people who really didn't have,

0:19:44To be blunt, anything that constructive to add, just coming out of the woodwork and saying,

0:19:48"Eh, you're wrong."

0:19:49And, "You're wrong because I read something once."

0:19:52That's just not helpful.

0:19:53Whereas something that Mark did, which he said, "Okay, from personal experience, let

0:19:57me take you through why it is you're wrong."

0:20:01That is infinitely valuable and extraordinarily valuable, and I'm very thankful for that.

0:20:05But the other people are expressing their enthusiasm, though.

0:20:08Even if they're doing it in a way that makes you feel bad, they're saying, "Hey, we listened

0:20:13to the show, and b) we're enthusiastic enough about it to seek you out on Twitter and send

0:20:17you something. So there is even good in that, I think.

0:20:19Yeah, I agree. And that's why even on the emails that I want to reach through the computer

0:20:24and backhand whoever wrote them, even on those emails, I try always to force myself to end

0:20:30the email in such a way that when I write, "Thank you for listening and thanks for your

0:20:35feedback," that it's taken as genuine, because I do mean it genuinely, even if I want to

0:20:40freaking kill whoever it is I just replied to. So I don't know, maybe this is too inside

0:20:45baseball, this might be boring, maybe we'll cut it, I don't know, but it just, the point

0:20:49I was trying to get to was that it made me look at myself with a bit of a critical eye,

0:20:55which again is even more reason why even the feedback that I'm bemoaning is actually valuable

0:21:01in its own way.

0:21:03And to point out also, I actually, so I've been in this high-end audio world for a lot

0:21:10recently and I've found that, so the reason why I don't like vinyl and a lot of analog

0:21:17technologies, the same reason why I briefly had a chance to have a tube amp in my house

0:21:23for a couple of weeks to try out and I didn't like it and I ended up not buying one, the

0:21:29reason why people like vinyl sound, not everyone, but the reason why the people who do like

0:21:34it like it, in addition to all the things we talked about last week about the tea ceremony

0:21:38and the romanticism of it and how hip and awesome it looks and everything is slow and

0:21:43artisanal and manual, in addition to all that crap, it does sound different. And it sounds

0:21:51different in various ways that people find pleasing. Now, as, but you know, that wasn't

0:21:58the argument that John was making against you.

0:22:00However, that was what most people thought

0:22:03when, you know, for the argument about vinyl

0:22:05sounding better than CD or whatever.

0:22:07You know, in my own purchases, and what I found

0:22:11in my experiences with high-end audio stuff,

0:22:14what I like is neutrality in most cases.

0:22:20So, for instance, I don't use a tube amp

0:22:22because I use sensitive headphones,

0:22:24and sensitive headphones, you can very easily

0:22:26pick up the noise introduced by tube amps. You can also occasionally get

0:22:30a little pop if a piece of dust hits the wrong way or something like that.

0:22:34It's almost like records.

0:22:38It's almost like vinyl. You're introducing distortion.

0:22:42You're introducing noise. So I try to get very

0:22:46nice digital stuff. So everything is solid state. There's no moving parts

0:22:50except the volume knob. There is

0:22:54as little noise introduced as possible,

0:22:57and there is as little tonal imbalance

0:23:00introduced as possible.

0:23:02And I actually took it a little too far.

0:23:05So my current setup is I have the Sennheiser HD 800 headphones

0:23:10plugged into the shit Asgard 2, that is actually its name,

0:23:15and the shit Bifrost DAC.

0:23:19And it looks amazing,

0:23:21these things all look fancy and awesome,

0:23:23They sound great.

0:23:24The main reason I like the shit amp and DAC

0:23:28is because they have like zero noise

0:23:30that I can hear on anything that I have.

0:23:31- It's not because they're the shit.

0:23:33- And the name helps too, and they look pretty awesome,

0:23:35and they're small and they were reasonably priced.

0:23:37And I like it because it's super neutral.

0:23:41Now, the Sennheiser HD 800 headphones,

0:23:44the best thing I can say about them

0:23:45is that they are the most neutral headphones I've ever heard.

0:23:48The downside of this is that they actually sound

0:23:51little bit boring on certain things. So like certain types of music, I actually like the

0:23:57distortion by things like amplifying the high end that you hear on some biodynamic models

0:24:03or the bigger bass I have on my closed headphones, the TH900s. And certain things I like that

0:24:10distortion. Now throughout all my research in finding these headphones, I tried a few

0:24:16other models I ended up returning or selling in the meantime, and I found that people's

0:24:20opinions very dramatically on what they considered sounding good. So one of the most popular

0:24:26lines, the Sennheiser HD600 and 650, those actually have pretty muted high end. And I

0:24:33didn't like that sound. A lot of people loved it. And similarly, certain people don't like

0:24:40the biodynamic super strong high end. And I like it. And what I found is that keeping

0:24:47everything neutral up until the point of the actual headphones allowing me to make that

0:24:51choice gives me the flexibility. Like, you can simulate all the other, you can simulate

0:24:55vinyl with digital plugins and equalizers and stuff. Please email Dan. You can simulate

0:25:03most of the effects of what people like about these things electronically and artificially

0:25:09if you want to. Just like Instagram, you know, can simulate the distortions of certain old

0:25:17processes or things that, or like idealized or enhanced versions of old processes and

0:25:21old distortions that used to exist in the analog world. And that produces more pleasing-looking

0:25:25photos just to a lot of people. So it's important to recognize, like, you know, I like everything

0:25:32to be digital and pure and clean and neutral up until the point where I have my headphones

0:25:36where I can make that choice and say either I want to listen to something with a lot of

0:25:39bass with this pair of headphones, or I want to listen to something that's neutral with

0:25:42this pair of headphones. And I recognize that's a crazy setup, however, that's how I am and

0:25:46that's how I did it. But none of that, like the argument John was making last week, is

0:25:51that CDs are better at being neutral. They're better at accurately and neutrally replicating

0:25:58the recording. And that is correct. That is 100% right. And most people, though, when

0:26:06they're arguing about what sounds better or worse or what is better or worse, they don't

0:26:11put it in those terms of personal preference

0:26:13or of like the difference between neutrality

0:26:16and pleasing sound.

0:26:18'Cause those are very different things.

0:26:20Perfectly neutral sounds boring to a lot of people.

0:26:23Like when I edit our podcast,

0:26:24I edit with perfectly neutral.

0:26:26However, when I listen to fish,

0:26:28I like a little bit more bass.

0:26:29So when I listen to fish, I put on the other headphones.

0:26:31You know, it's very important to distinguish those things.

0:26:34Personal preference or more pleasing

0:26:37versus accurate and neutral.

0:26:39- Yeah, I think you're right.

0:26:41but I don't know, this is probably getting a bit old now.

0:26:45Why don't you tell us about something that's cool?

0:26:46- I would love to.

0:26:47It's our friends at PDF Pen.

0:26:49Once again, this is PDF Pen for iPad.

0:26:53PDF Pen for iPad lets you edit PDFs anywhere you are,

0:26:56as long as you have an iPad with you

0:26:57that happens to be running PDF Pen.

0:26:58However, anywhere you are with one of those things,

0:27:01you can edit PDFs.

0:27:02You can sign contracts, you can fix a typo,

0:27:05you can actually edit the PDF right there on your device.

0:27:08You can correct a price list,

0:27:09You can fill out a form while you're on the go.

0:27:12And you can take PDF documents with you anytime,

0:27:15add notes, highlighting, add markup

0:27:17during your mobile downtime.

0:27:19You can sync PDFs with PDF pen for OS X

0:27:22using iCloud or Dropbox,

0:27:24'cause PDF pen is for OS X and iPad.

0:27:27You can grab and save PDFs using Dropbox,

0:27:29Evernote, Google Drive, Box.

0:27:31I guess it isn't called box.net anymore.

0:27:33Is that right?

0:27:34They dropped the .net, now it's just Box.

0:27:36Anyway, you can save PDFs to that or from that.

0:27:39even webDAV and FTP servers.

0:27:42And anywhere you are with PDF/PEN for iPad,

0:27:45you have the complete feature-rich

0:27:46mobile editing power of PDF/PEN.

0:27:49It's available in the App Store.

0:27:50Now, a lot of apps these days,

0:27:52I've been meaning to talk about this,

0:27:53a lot of apps these days

0:27:54haven't even been updated for iOS 7 yet.

0:27:56They're not native, they're kind of ignored.

0:27:59PDF/PEN is awesome, it is always updated,

0:28:01it is native to iOS 7,

0:28:03and there's tons of,

0:28:05they're always making tons of improvements,

0:28:07performance improvements, everything else,

0:28:09they're fantastic. And it syncs all your stuff, once again. They even support a few of the

0:28:15pressure sensitive pens on the market, the most popular ones. They support the Jot Touch,

0:28:20the Pogo Connect, and the Jaja or Haja, I'm not sure which one that is, but it's Jaja

0:28:25that one, which is either Jaja or Haja or maybe Jaja or Haja, who knows. But they support

0:28:30that one. So check out PDF pen for iPad, go to Smilesoftware.com/ATP, that is Smilesoftware.com/ATP

0:28:38is smilesoftware.com/ATP. Thanks a lot to PDF/Pen for iPad for sponsoring our show once

0:28:44again. I'll tell you, I use PDF/Pen and it's fantastic. I use it more on the Mac than the

0:28:50iPad just because that's where I am most of the time. But it's an amazing piece of software

0:28:53and it saves my butt pretty frequently. So thanks a lot to PDF/Pen for iPad once again.

0:28:59We didn't get a chance to talk about last week because we were too busy arguing. We

0:29:05We didn't get a chance to talk about this trial with Samsung, between Samsung and Apple.

0:29:10And there's been some interesting things coming out of this, and we've gotten to

0:29:15look at some of the behind-the-scenes at how Apple does stuff, how they get their jobs

0:29:21done as a result of a lot of the discovery during the trial.

0:29:26So I don't know, Jon, do you have any thoughts on this?

0:29:28Yeah, I picked out in particular not so much the details of the trial or whatever, because

0:29:33Doesn't this one have to do with patents, too?

0:29:35I don't even pay attention, but you know how I feel about--

0:29:36I think so.

0:29:37--about patents or whatever.

0:29:38But yeah, because as part of the discovery process,

0:29:41or some other legal term that I don't know,

0:29:44they're revealing internal documents from Apple.

0:29:47And it's great because Apple is-- well, it's great for us,

0:29:50anyway.

0:29:50I don't know if it's great for Apple.

0:29:51That's all the discussion.

0:29:52But we get so little insight into what goes on in Apple,

0:29:56we just see their public face.

0:29:57And their public face is so incredibly controlled.

0:29:59And I think it's even more controlled now

0:30:01than it used to be, because as we've said many times before,

0:30:04in the good old days with Steve Jobs,

0:30:06yes, Apple's message was controlled,

0:30:07but Steve Jobs had the ability and often took it

0:30:11to just go off book and do whatever the hell he wants

0:30:14and say whatever the hell he wants in whatever way he wants.

0:30:16And I'm sure he was controlled and calculating as well,

0:30:19but you never quite knew what he was gonna say.

0:30:20Tim Cook, you're not gonna make him slip up

0:30:23and say something.

0:30:23You're not gonna get something exciting.

0:30:25So far, it seems like he is very on message

0:30:29and very controlled, strategically leaking things

0:30:31and saying things in a certain way

0:30:32and choosing his words very carefully.

0:30:34But with Steve Jobs, he felt like there was a chance

0:30:36he might either act like he's just suddenly caught up

0:30:40in the moment and saying something spontaneous

0:30:42or actually do that.

0:30:44But anyway, these documents are internal Apple documents.

0:30:47And from the outside, a lot of people say Apple is arrogant,

0:30:50Apple's out of touch, they don't know what their users

0:30:51really want or whatever.

0:30:53But these internal documents make it very, very clear

0:30:55that that is not the case, that it is merely

0:30:57their external face that they control so well, but internally.

0:31:01This is the fiscal year 2014 planning off-site slide deck, and you can see some of these

0:31:07slides.

0:31:08If you haven't seen them, we'll put the link in the show notes.

0:31:09You should check them out, but I'll read off some of the slides here.

0:31:12So the first slide says, they're talking about iPhone sales, and it says, "Growth rates are

0:31:16slowing," and they show a graph that shows growth rates going down, down, down, down,

0:31:20down.

0:31:21Apple, of course, would never show a slide like this in public, because why are they

0:31:24going to show something that puts them in a bad light.

0:31:29And again, something as simple as this--

0:31:31showing growth rate slowing.

0:31:33Yes, you'll see stories about Apple about that.

0:31:34And if someone actually gets in touch with Apple

0:31:36and asks them about that, they'll

0:31:37say some blah, blah, blah kind of, well,

0:31:40we think we're on the right track, blah, blah, blah.

0:31:43They'll say some stupid PR-y thing.

0:31:44But here they are saying-- when they're talking to themselves,

0:31:46they're like, here's the graph.

0:31:48Growth rate is really slowing.

0:31:49Next slide, they say, so what's going on?

0:31:51And they have little sections here that says,

0:31:53Strongest demand is coming from less expensive and larger screen smartphones.

0:31:57Carriers have strong interest in capping iPhone due to one or more factors.

0:32:01High share, the subsidy premium, unfriendly policy, unfriendly is in quotes, lack of alignment.

0:32:06So they're basically saying, customers want cheaper phones with bigger screens.

0:32:10Carriers, carriers don't like us that much and they don't like us for good reasons because

0:32:13we are more demanding of them because we, you know, these unfriendly policies that we have,

0:32:17our subsidy is really high, lack of alignment with, you know,

0:32:22our interests aren't aligned with the carriers. Everybody's laying out, yeah, carriers don't like

0:32:27us that much, and competitors, and they show the little Android icon. Competitors have drastically

0:32:31improved their hardware and software and their ecosystems, and spending obscene, also in scare

0:32:37quotes, amounts of money on advertising and carrier channel to gain traction, which is

0:32:41a Samsung thing. So they're laying out like, "Here are our problems. We're kind of getting our butt

0:32:46kicked in these areas. Here's what people want." And this is the title of the slide. It's great.

0:32:50Next slide. Consumers want what we don't have. If you can imagine Apple saying that in public,

0:32:56what we don't have. And so they show the market share growth and they say, "Where did the growth

0:33:02come from?" And this, I wish there was a better breakdown and maybe there was, but they're saying,

0:33:07"Where did the growth in the industry come from?" And it shows the growth and it says,

0:33:10I can't do the percentages here, but more than half the growth came from phones that are less

0:33:16than $300 and the rest of the growth came from screens that are more than $300 but have

0:33:21screens that are bigger than 4 inches.

0:33:24The objection I have to this graph is, "Okay, great, you've identified that the majority

0:33:28of the growth is coming from phones that are less than $300.

0:33:31I think you need to break that down more because if you take that piece and say, 'Okay, well,

0:33:37out of that growth, actually 99% of that growth is coming from phones that are less than $5,'

0:33:42That is way different than saying, "Okay, well, most of it's coming from--how much less

0:33:46than $300?

0:33:47Break it down.

0:33:48Is it, you know, what's the average?

0:33:50What's the distribution in there?"

0:33:52Without knowing that, you don't know what to do.

0:33:53But anyway, this is a high-level slide, so I assume they have broken it down internally.

0:33:57I just, again, I would like to see that broken down.

0:33:59I didn't.

0:34:01And all this reminded me of a book I'm reading now, which I would recommend to everybody,

0:34:04and we'll put a link in the show notes.

0:34:06It's Ed Catmull's book called "Creativity, Inc."

0:34:10It's not really a memoir or manifesto.

0:34:13It's more of just an explanation of the early history of Pixar

0:34:16and what Ed Catmullen and the rest of the company

0:34:19has done to make that organization function

0:34:22and be successful.

0:34:23And a big focus on the book is being clear-eyed

0:34:27about your own problems and candid about them internally.

0:34:31And I know Pixar's not Apple, and I

0:34:32know Steve Jobs didn't have nearly as much involvement

0:34:35in Pixar as he did in Apple.

0:34:37but I can't help but see parallels

0:34:39between the things that are in this book

0:34:43and the approach Apple is taking here.

0:34:45Apple is very clear-eyed in these slides.

0:34:47They are not sugarcoating it.

0:34:48They are not like, "Rah, rah, rah,"

0:34:49like, "Yeah, we may have problems,

0:34:51but we still have the best phone in this office."

0:34:54The slides are just, you know, there's not a lot of them,

0:34:56but they are essentially unrelentingly negative

0:34:58towards Apple, and that's how they have to be.

0:35:00That's how Apple improves.

0:35:02So I felt better about seeing these slides,

0:35:05and it made me think that at least some

0:35:06of the culture and strategies described

0:35:10in that excellent book by Ed Catmull

0:35:12are alive and well in Apple as well.

0:35:14- Yeah, I think it's fantastic to see

0:35:19that Apple knows its own problems,

0:35:21it knows its own shortcomings.

0:35:23But I think that's not really that much news necessarily.

0:35:28To me, seeing all these trial documents

0:35:32and the emails of Phil Schiller and everything,

0:35:35It really just says to me that this company is,

0:35:39by being as private as they usually are,

0:35:42they are saving us from the boredom

0:35:44and the relative routineness

0:35:48that's actually in their company.

0:35:51These are like, the only big surprise here

0:35:53is that these people are actually acting somewhat human,

0:35:56but they're all, they're saying not that surprising things.

0:36:00They're sending boring business emails

0:36:01to their advertising agency

0:36:02that doesn't even capitalize anything,

0:36:04which is apparently a thing.

0:36:06- Well, that was a whole other thing.

0:36:07Like, this article was supposed to be sensational or whatever

0:36:11and so one of the things they showed was,

0:36:12look at this email where Phil Schiller's

0:36:14being angry to his ad agency.

0:36:15What I took away from that exchange was

0:36:18how incredibly low the bar is for Apple's ad agency.

0:36:22Like, you feel like, this is your ad agency?

0:36:24You're the biggest company in the world

0:36:26and these jokers are doing your ads?

0:36:28Boy, that's just depressing.

0:36:30Not everything can be up to the standard

0:36:33of Apple and Pixar, I guess,

0:36:34and it's just, yeah.

0:36:36- Yeah, I mean, what I really just found

0:36:38with these documents is like,

0:36:40wow, this is all really boring stuff that,

0:36:44like there's very little of newsworthiness here

0:36:48except that Apple usually does a really good job

0:36:51of keeping what comes out interesting.

0:36:55Keeping their public image interesting

0:36:57and secretive and well edited.

0:37:01And you see now on the inside that,

0:37:03yeah, it's a big company, dealing with big company problems

0:37:06and doing things that big companies do.

0:37:09- It might not seem that novel to you,

0:37:10but I compare it to, I mean,

0:37:13I work for a lot of different companies,

0:37:14but the one people may have heard of

0:37:15is that for a time I worked for Palm,

0:37:17and I wasn't very close to the leadership

0:37:20of that organization, but internal to the company,

0:37:22when I worked for Palm, I got presentations from,

0:37:25you know, some of the big wigs came to visit us

0:37:27'cause we were in a different location,

0:37:28and gave us a presentation on the state of Palm

0:37:30and what their plans were for the business.

0:37:32and their slides were not candid and didn't accurately identify the problems they were facing.

0:37:36They were much more kind of like, "We're actually still doing pretty good in the PDA space, and the

0:37:41phones are coming on, but we've got these trio things." And it was like, even to me inside the

0:37:46company, I'm like, "You guys are in denial. If these are the slides, we're showing this

0:37:49internally. If we are showing these slides to ourselves internally, we are not being honest

0:37:53with ourselves about the state." I mean, you would hope that they would, you know, we saw it and we

0:37:58felt it, maybe they felt it, it just didn't feel as honest and straightforward. And like,

0:38:02if you if you don't think these Apple slides are novel or interesting, like, I don't have that much

0:38:08experience in big tech companies, but I have seen it from big and small. Most companies are not this

0:38:13candid with themselves and are not are not this accurate, like, you know, are not able to exactly

0:38:18pinpoint where their weak points are, and nail them in that way. And I guess that shows, you know,

0:38:23confidence internally and skill. And if they weren't that skill, they wouldn't be where they

0:38:26are today. So I think there is something to it. It's not—we shouldn't just accept that

0:38:30every company does this, because I think many of them don't.

0:38:34The other thing that we didn't talk about last week, which we could have, is some guy

0:38:39that apparently I was supposed to know but didn't know is leaving Apple. And that guy

0:38:44is Greg Christie. And had either of you two heard of him before? I'm assuming, John, that

0:38:49you had. But Marco, have you heard of this gentleman before last week?

0:38:53Nope. I still don't even believe this is really a story.

0:38:56Well, there were stories about him. He was featured in recent stories before the "he was

0:39:01leaving Apple" story. So even if you weren't aware of him before that, you might have said,

0:39:04"Did I not see that name before?" And it's like, yes. Like two weeks ago,

0:39:07you did see that name. I think someone did a story about the early days of the iPhone,

0:39:11and he was heavily featured. So that's why you might be aware of him.

0:39:14Yeah, this looks a lot like kind of a gossip story, like people arguing inside Apple,

0:39:19and then one of them leaves. I mean, that happens all the time in every company.

0:39:23And then, you know, from some people have—we don't know what's really going on here, but

0:39:28there's at least one reasonable denial from Matt Panzareno that the details of this big breakup

0:39:33are not as dramatic as they seem. But the reason I think it's interesting is forget about how he

0:39:39left or why he left or whatever. Like, who cares, right? What it highlights, what all these stories

0:39:46highlight is it's time for them to talk about kind of, "Oh, Johnny Ive's organization," because

0:39:51he is in charge of much more than he used to be in charge of. Anything that happens

0:39:56in that organization, people coming, people going, or whatever, with him remaining at

0:40:00the top, in some ways, like, not that it reflects on him, but it makes us focus on him. So what

0:40:05these stories made me do is think more about how Johnny Ive is doing in his position. And

0:40:11yes, part of his position is retention or whatever, but not so much in this particular

0:40:15thing, but what it got me thinking about was what are Johnny Ive's weaknesses in his new

0:40:21role as software designer. I've had decades to think about what his weaknesses are in

0:40:25hardware design, and previously it was unknown what is he going to be like in software, but

0:40:30his organization, I guess, you know, iOS 7 is the first big product of his software organization,

0:40:35and we've all had a long time to live with it, and I think his weaknesses as a software

0:40:39designer are more glaring and obvious than they are in hardware, maybe, and in fact they're

0:40:46very similar. And so anytime anything happens within that organization, it makes me sort

0:40:51of like dwell more on how he's doing in his new role as software designer. And I think that's a

0:41:00useful avenue of thought for me. So are you not pleased with how things are going then with Johnny

0:41:06at the helm? Well, like when somebody leaves like that, even if it's on good terms, bad terms,

0:41:12or whatever, like you have to think that like it's not I that's leaving, right? People under him are

0:41:17are going to leave.

0:41:18And you have to think, maybe there was a--

0:41:20it's someone who used to be more in charge than they are now.

0:41:24He was largely responsible for a lot of the early iPhone UI.

0:41:28And surely his power to make changes and decisions

0:41:31has decreased now that Johnny Ive has come in above him.

0:41:34And you know what I mean?

0:41:35That's the whole thing with the reorg.

0:41:36Johnny Ive has been given much more power.

0:41:39And that means the people who used to have the power

0:41:41have had it taken away.

0:41:42And if there's any sort of disagreement about what to do,

0:41:45Johnny Ives is going to win, as he should, because he's in charge.

0:41:48And what that means is that whatever his strengths and weaknesses are,

0:41:51are imprinted even more strongly on the organization,

0:41:54because he is not just one voice anymore, he is the voice.

0:41:57As I think I tweeted at one point, he's in full Shigeru Miyamoto mode,

0:42:01where Miyamoto designed Nintendo's flagship games,

0:42:04like Mario and Donkey Kong and many other things,

0:42:07and slowly climbed the ranks of the organization until essentially he was

0:42:10the buck stop with him for all issues related to software design

0:42:14in all of Nintendo and possibly several aspects of hardware design as well.

0:42:20Once you are in that position of power, whatever your strengths or weaknesses are are magnified.

0:42:25I mean, it was true with Steve Jobs.

0:42:27His strengths and weaknesses were magnified through the whole organization.

0:42:30So now, that's why I'm dwelling on what I was like as a software designer, because these

0:42:33other voices, whether you agree with them or not, Scott Foslall, Greg Christie, all

0:42:37these other people, their voices and their ability to affect change within the organization

0:42:43are necessarily modified by the larger and larger footprint of this one fallible individual,

0:42:49for better or for worse.

0:42:50And so that's what I've got my eye on.

0:42:51Yeah, that's interesting.

0:42:53I mean, the problem is, I think you're right that that is something to watch, but I think

0:43:00everything we've heard about Greg Christie's particular departure makes it sound like maybe

0:43:08the problem had nothing to do with Jonny Ive.

0:43:11And we see the crazy rumor sites saying there was some kind of friction there, and then

0:43:17you see people like John Gruber, who have generally better sources on these kind of

0:43:20things, saying, "Oh, this was actually in the works for months, and this wasn't even

0:43:27recent, and there was no bad blood, he just was retiring."

0:43:31And as the story gets older, we're hearing more that it's not a story than the very

0:43:39minute that it broke in the news that said it was a story. So that's probably the reality,

0:43:44it probably is not a story in itself. Now, again, you're right that this is something,

0:43:47you know, Johnny Ive having a lot more power than he used to and becoming so powerful that

0:43:53he's able to get bad decisions through, that is something to watch, no question. And iOS

0:44:007 is a great example of some of those things. You know, I think iOS 7 really showed that

0:44:07Johnny Ive was wanting to make a big change in a lot of very important stuff on iOS and

0:44:14he did and he went a little too far in a lot of those changes.

0:44:17But even, as we saw last summer, even from beta 1 to what was released as 7.0, there

0:44:24were a lot of changes to reign some of that back.

0:44:27The biggest one being the super thin font was replaced with a more reasonably thick,

0:44:31thickness one, more reasonably weighted one.

0:44:36like that, and the lock screen having modifications to make it easier to realize that you have

0:44:41to swipe right and not up, things like that, there were a lot of changes made. He went

0:44:51too far and then he dialed it back. I think iOS 7 is great. I think the design of it is

0:44:57great. It is not perfect, but it is great. It is a massive improvement over 6. This is

0:45:04going to get into what we want from iOS 8, I guess, but I don't know if we even have

0:45:09that as a topic. But, again, I think this is something to watch for, you know, watch

0:45:13for Johnny Ive getting enough power that he can get bad decisions pushed through, and

0:45:19for him to think that these decisions are good. You know, those things are a problem

0:45:23together. But other than that, I don't really think this is much of a story.

0:45:27Like on the spin of the story, what I was getting at is it doesn't matter how or why

0:45:31Greg Christie left, it only matters that he was somebody who was there for a long time,

0:45:36who was responsible for a lot of, like, he was a senior person, right? He wasn't just

0:45:40some other person. He was another voice, whether he agreed with Johnny Ive or not at any point.

0:45:46If you were like the last, the last, you know, Godzilla standing, like all the other big

0:45:50guys have left, all the other people who were there from the beginning, who strongly influenced

0:45:55the original iPhone, you know what I mean? Like, that's what I was saying about, like,

0:45:59Suguro Miyamoto, you need to have those voices that you will listen to because you believe

0:46:05that they are valuable and have experience.

0:46:07And even if you just left because you just felt like retiring, that's a loss.

0:46:12That's somebody that you don't have in the organization with a wealth of experience for

0:46:15you to bounce your ideas off of.

0:46:16You know what I mean?

0:46:17I don't want to get back to the Ed Catmull book again, but they have this whole chapter

0:46:20on the brain trust, which maybe we'll put a link to this because I think they did an

0:46:23excerpt on some website somewhere, where they will get the people, the oldest and wisest

0:46:29people together in a room and have them all tell each other what's wrong with their projects,

0:46:34just to bounce ideas over to them, because they all respect each other and they all have a lot

0:46:38of experience. And any time a senior person leaves in a similar role, like the senior software

0:46:46designer for iOS in the past, that is a loss for the company. And that's why I get like,

0:46:51it doesn't matter who cares why he left, he's not there anymore. And that will only serve to magnify

0:46:59the good and the bad about Ive. And I would rather see him surrounded by more senior people that he

0:47:06respected to, you know, not to temper his output, but perhaps even to enhance the parts that are

0:47:12good about it. It's not like they have to disagree with him. Maybe they should more vehemently agree

0:47:15with him in certain areas where he's doubting himself. That dynamic is what I think is lost when

0:47:20senior people leave. Yeah, the other thing that I'm a little concerned about,

0:47:25just on based completely on theory is Johnny getting spread a little thin and I would hope

0:47:32that he has his trusted minions slash you know advisors or whatever to to take a lot of the

0:47:41day-to-day off of his plate but based on no facts and just a bunch of assumptions it seems to me

0:47:48that Johnny is a fairly hands-on guy and if he's a hands-on guy and presumably already had a full

0:47:55plate with hardware alone, I just can't imagine now adding all of software onto that already

0:48:01overflowing plate and still being able to be good at your job, not only your existing

0:48:08job of hardware, but also good at this new job of software, which really, in the strictest

0:48:14sense, he doesn't have any real experience or education in.

0:48:17So that kind of concerns me a bit that he'll be spread too thin over the next few years.

0:48:23Well, see, that distinction, if you were to ask Johnny Ive—I'm going to play Johnny

0:48:27Ive now because I read a book about him once—we had on the podcast, I bet what he would say

0:48:32is this distinction between hardware and software is an artificial one that has no bearing on

0:48:37the experience of using the product.

0:48:38It's all one and the same part of the same product.

0:48:41And it makes perfect sense for them to be under the same umbrella, because that separation

0:48:45and that sort of arguing between them, like, yes, there is a tension technically, but that

0:48:48doesn't matter to the end product.

0:48:49It has to be one thing.

0:48:51I think it's perfectly valid to have that approach, that holistic approach to the product.

0:48:56And Apple has always had it.

0:48:58It's just that now it's literally embodied within one person.

0:49:00And yes, it's a danger of that, and that maybe there was healthy tension between hardware

0:49:03and software.

0:49:04But I don't worry about him being spread too thin, because I think he is fulfilling sort

0:49:08of the Steve Jobs type role of set the direction, be tastemaker, give thumbs up, thumbs down.

0:49:14He's not in there drawing UIs one at a time or whatever.

0:49:20He's got a staff.

0:49:21who do that, but in the end the buck stops with him and if he really wants it to be a super skinny

0:49:24font on a white background, that's what it's going to be. And like Marco said, if you have to dial it

0:49:31back, fine. In many ways it's better to go too big and then scale it back than to be too timid,

0:49:35you know what I mean? So I'm not giving him a bad grade or anything. I'm just watching it because I

0:49:42would rather see ideas have to fight for their life as hard as possible inside Apple and only

0:49:49the best ones make it out, then to sort of slide towards a situation where there's not

0:49:53enough people with skills even close to Johnny I for him to do good work in the software

0:50:01area.

0:50:02Like in the hardware area, I don't know what's going on back there, and we don't know those

0:50:04people as well, and I think he has a trusted team that doesn't have a lot of turnover.

0:50:07But in the software area, he's lost Scott Forrestal through what we think actually was

0:50:12a disagreement, and Greg Christie is now gone.

0:50:16I would be happier if both of those people were still there and working in concert with

0:50:20Johnny Ive. But what can you do?

0:50:22Well, you know, maybe the hardware stuff will just be a lot less work now, because you can

0:50:25just look at what HP and Samsung are making and just do stuff that looks like that, right?

0:50:30I think it's the opposite. It doesn't work the other way.

0:50:33You know, another thing that I should point out, and I am not through listening to it

0:50:37yet, but on Guy Englishes and Rene Ritchie's really great podcast, Debug, on episode 33

0:50:45we'll put a link to that in the show notes. They had Ken Ferry on who did, apparently was kind of

0:50:50the primary guy behind Auto Layout. And I'm only about two-thirds of the way through the episode,

0:50:54but it's really, really good. And one of the things that Ken talks about is kind of what it's

0:51:00like to get an idea through Apple. You know, so he, I don't, I didn't hear the Auto Layout

0:51:07specific part yet, but just for the sake of argument, you know, I go in there, I come up with

0:51:13the idea for auto layout and apparently he basically only has to convince his immediate

0:51:20supervisor or at least that's what I gleaned from it and I just thought that was really really

0:51:25interesting that it's not very bureaucratic it's not that you have to convince 3,000 people all the

0:51:29way up the chain and I'm sure that that's that is the case sometimes but generally for for some

0:51:37things you can just convince one man or woman and that's all it takes and I just thought that was

0:51:41- We are also sponsored this week, once again, by New Relic.

0:51:45New Relic is an all-in-one web application performance management tool.

0:51:49Apparently there's an acronym for that and it's APM.

0:51:53New Relic lets you see performance from the end user experience through servers

0:51:57down to each line of your server-side code. New Relic asks us to take a minute

0:52:01and say a big thank you to all the data nerds out there building all this great stuff

0:52:05that we all know and love. They would like to send a shout out to developers,

0:52:09developers, software geeks, code jockeys, to the brave few who see things differently.

0:52:14Here's to working nights, to wearing oversize concentration enhancing headphones upon your

0:52:18furrowed brows. New Relic thanks you, the entire internet thanks you.

0:52:23Nowadays, if you're in any business, you're in the software business. Software powers

0:52:27our apps, runs our databases, manages our accounts, and runs e-commerce sites and email

0:52:32programs. When software breaks, everyone loses. New Relic helps improve your software performance,

0:52:37so your users have a better experience

0:52:39and your business is more successful.

0:52:41How is that for a win-win?

0:52:43New Relic monitors every move your application makes

0:52:45across the entire stack.

0:52:47It can show you what's happening right now in your app.

0:52:50You can zero in on problems quickly

0:52:52with transaction tracing, SQL and no SQL,

0:52:55performance analytics, application topology mapping,

0:52:58and deployment history markers and comparisons.

0:53:01Go to newrelic.com/atp

0:53:04and you can sign up for a free 30-day trial.

0:53:07then all you gotta do is deploy their agent on your server.

0:53:10It natively supports Ruby, PHP, Java, .NET, Python,

0:53:13and even Node.js.

0:53:15It'll start collecting data immediately,

0:53:17and then you can quickly see inside your app

0:53:19to start finding hotspots, fixing issues,

0:53:21and optimizing performance.

0:53:23Once again, go to newrelic.com/atp

0:53:26to start monitoring your web app's performance today

0:53:29and finding all your issues and fixing them all.

0:53:31Newrelic.com/atp.

0:53:32Thanks a lot.

0:53:34So one quick remaining thought on the Christie thing, which I came up with.

0:53:39We said Christie had been there since the beginning.

0:53:43Let's suppose, hypothetically, that he was pretty well aligned or pretty chummy with

0:53:48Forstall.

0:53:49Then wouldn't it make sense for him to either choose to leave or be asked to leave?

0:53:59And the quiet rumblings I've heard are that that's exactly the case, that he was kind

0:54:07of in forestalls camp, and maybe the person who's replacing him is someone who likes

0:54:14to argue a little bit more.

0:54:16And if that's the case, I'd consider that a wonderful thing.

0:54:18I think it's very smart of people who are in a position of power to have someone around

0:54:25them that argues because it forces you to really, really vehemently believe in your own opinion,

0:54:33or change it if the case may be. The best example of that from the Steve Jobs's history is the,

0:54:38you know, him being adamant that iTunes, was it iTunes? Was it the iPod shouldn't be available

0:54:44for Windows? iTunes Store shouldn't be available for Windows? Come on, you guys gotta help me out

0:54:47here, he's not remembering. But anyway, yeah, iTunes for Windows, the chat room says, so I

0:54:51I will trust them.

0:54:52But anyway, whatever it was, it's

0:54:53something that eventually came to pass

0:54:55and was an enormously positive thing.

0:54:57And Steve Jobs was dead set against it.

0:54:59And it was just all of his most trusted lieutenants

0:55:02were constantly on him, arguing, just like would not give it up.

0:55:05Like, we've got to do iTunes for Windows.

0:55:06He said, no, I'm never going to do it.

0:55:07I mean, eventually they wore him down.

0:55:09He's like, fine, do friggin' iTunes for Windows,

0:55:11whatever you want.

0:55:12That sounds like an unhealthy dynamic.

0:55:14But I think that is actually a company--

0:55:18I mean, it's not the healthiest it could be.

0:55:20It would be better if, again, it was something more

0:55:23like the Pixar Brain Trust.

0:55:25But in many ways, it was similar,

0:55:26because with the Brain Trust thing,

0:55:28it's a bunch of senior people, I'll

0:55:32tell you what's wrong with your thing, but none of them

0:55:35have the power to tell you what you have to do.

0:55:37Like, they can't make you change your movie

0:55:38in the case of the Brain.

0:55:39They just tell you, like, we think this is wrong, that's wrong,

0:55:41and maybe they suggest solutions,

0:55:43but they have no authority to make you

0:55:44do any particular changes.

0:55:46And in the case of Steve Jobs, his lieutenants

0:55:48didn't have the authority to make him put iTunes

0:55:50Windows. But because he respected their opinions and because they were so insistent on it, eventually,

0:55:55you know, he went against his own better judgment and said, "You know what? What am I even paying

0:55:59these people for if not to advise me?" And they are so strong and unanimous in their,

0:56:03in their, you know, insistence that this is a good idea. He was kind of a baby about it and said,

0:56:08"Fine, whatever. Put your stupid thing on Windows if accounts are to be believed." And that definitely

0:56:13sounds like him. But in the end, you know, the system worked. Yeah, and that's exactly my point.

0:56:19And so, I mean, whether or not Christie was forced out, depending on who's replacing him,

0:56:25it could be a really wonderful thing.

0:56:27So we'll see.

0:56:27All right.

0:56:29So we have a few other things that we can talk about.

0:56:32And something that, uh, that we've had on the show notes for a while that I don't think I completely understand yet.

0:56:39And I'd love to hear John, you explain it to me is what is this piece?

0:56:43L thing all about?

0:56:44Oh, good.

0:56:45You know, I get to explain another thing that I don't know the details of, but I do think I know.

0:56:49I do know enough, I think, to explain the broad strokes.

0:56:54And the broad strokes are actually interesting.

0:56:56So P. Sal is from-- Steve Pearlman is his name, I think.

0:57:00Inventor, one of the original inventors of QuickTime.

0:57:03He also did the on-live remote gaming service.

0:57:05And this is another one of his babies.

0:57:07And he has a little bit of reputation

0:57:08as kind of a genius crackpot, sort of like Stephen Wolf

0:57:11for him.

0:57:12Undeniably incredibly smart and doing amazing things,

0:57:15but also prone to presenting his creations as the best thing since sliced bread and kind

0:57:21of magic.

0:57:22And that's the difficulty with PSEL is, I forgot what the original name for it was,

0:57:26but it's been rebranded to PSEL.

0:57:28It's presented in a way that doesn't explain how it works unless you already know how it

0:57:34works.

0:57:35And so it seems like magic.

0:57:36It's like, you have all these problems with cell phone reception because there's so many

0:57:40people with cell phones, and there's so many towers, and there's so much interference,

0:57:43and the weak signals and strong signals, and you can't have the cell towers too close.

0:57:46He's basically describing what we all know about current cell phone reception.

0:57:49He's like, "Well, what if that could all go away, and you wouldn't have any problem,

0:57:52and every single cell phone user would have the full bandwidth of the entire tower?"

0:57:56And you're like, "Well, that's impossible. What are you talking about?"

0:57:59And it sounds like a perpetual motion machine, and you tend to dismiss him out of hand.

0:58:03Unless you already know what it is, in which case you would think,

0:58:07"Oh, well, I know what that is," and then it becomes boring.

0:58:09And those are the two extremes. Like, "This sounds like magic.

0:58:11magic. I'm going to be like the only person on earth. It's like as if I have this whole cell

0:58:15tower to myself. And then you find out how it really works. If you happen to, and if you were

0:58:20in this field, it's not anything novel or amazing or new. There are certain aspects of it that are

0:58:25interesting from an engineering perspective. And you're like, "Oh, it's that. Okay, well, now I

0:58:31think I'm less excited about it." And that was my experience as well. So here's what it is.

0:58:36In the demonstrations, what he's talking about is having cell towers, and you can't put them

0:58:41them too close together because they make interference with each other.

0:58:43And of course, every single cell phone that's trying to communicate with the cell tower

0:58:47is interfering with the other cell phones that are communicating with the cell tower,

0:58:50and there's many different strategies that we use to allow multiple cell phones to talk

0:58:54to the tower.

0:58:55And you don't need to know the details of all of them, but the most ancient one is like,

0:59:02let's just all take turns.

0:59:03There's different ways of multiplexing our signals with frequency division, where we

0:59:08to carve up and into different channels, different frequencies,

0:59:10or code division multiplexing, where we send a signal that each thing can

0:59:13decode to figure out which part of the signal was relevant to it.

0:59:17But it all has to do with just a big shared space.

0:59:19And all those techniques-- interference is bad,

0:59:21because it screws up with the signal each thing is sending out and receiving.

0:59:26What PSEL is trying to do is make it so that all the cell towers are

0:59:32aware of where all the cell phones are.

0:59:35and they're aware of what signal is being put out

0:59:38by all the phones and all the other towers,

0:59:40and they do a whole bunch of really complicated math

0:59:42to figure out, okay, we know what we wanna send

0:59:44to everybody here.

0:59:46We're going to output something such that our interference

0:59:49overlaps with each other in such a way

0:59:51that it sort of forms a hotspot right in the target area

0:59:56where all the combination of the interference

0:59:58from all the signals we're putting out

0:59:59sum up to exactly what we wanna send to that phone.

1:00:02And then all the interference from all the signals

1:00:04putting out sum up to exactly the signal you want to put out to that other phone.

1:00:06That's why it's like you get all you get all the bandwidth because it's a bunch

1:00:10of things working in concert to figure out exactly what they need to send out

1:00:13so that the sum of all their crazy interference exactly equals this the

1:00:17exact clean signal they want to send to that particular phone. I'm massively

1:00:21simplifying it obviously but this is this is sort of the upshot of how it

1:00:25works. And the innovative thing they have is like well how do you do that how can

1:00:29you have all these cell towers figuring out exactly what they need to put out

1:00:33sort of in real time, all the signals that are being sent and received to figure out

1:00:38what they have to put out to make the interference overlap to hit every single little cell phone.

1:00:42That sounds crazy.

1:00:43How could you do that?

1:00:44Well, that's part of the engineering breakthrough in that they claim to have a way to do that

1:00:47with fiber optic cables or whatever running between the cell towers or slower links if

1:00:51you're willing to allow for a little more slop in the system.

1:00:55And to be able to have computing in each one of those locations that scales linearly, which

1:01:01which is another topic on the topic list that we may get to someday, scaling.

1:01:04This scales linearly.

1:01:05So it's like, well, OK, so it's easy to do for two receivers, but is it 10 times as much

1:01:10to do four?

1:01:11It is like, how does it scale?

1:01:12It's like, well, we have a way.

1:01:13If you want to handle 50 people on a tower, then you need one computer.

1:01:16If you want to handle 100 people, you need two computers.

1:01:18You know, 200, you need four computers.

1:01:20Like, it scales linearly with the number of people.

1:01:23And that is an interesting engineering breakthrough, assuming it works as advertised.

1:01:27But there are limitations to the system.

1:01:29First of all, you do have to have all the towers talking to each other so they can figure

1:01:31out what the hell to output.

1:01:33You do need the fast connections between the towers.

1:01:36And if you make a slower connection between the towers, what you're giving up is essentially

1:01:40how fast the receivers can move.

1:01:43So for example, if you're in a car and that car goes over something like 70 miles an hour,

1:01:48by the time all the cell towers figure out how to constructively overlay all their crazy

1:01:52overlapping signals to hit where you are, you've moved too far and it won't work as

1:01:56well.

1:01:57speed limitations to this, which are not, you know, there are speed limitations that

1:02:01you moving around really quickly affects everything, but it affects this system more than others.

1:02:05So if you're in any sort of car trying to use your nav system, whatever, and the system

1:02:09was in use, if you're going over 70, maybe it wouldn't be able to keep up with you if

1:02:12you only had microwave links between the towers and everything.

1:02:15But the idea, and even I think Apple's routers and regular just Wi-Fi routers do this, they

1:02:21call it beamforming or whatever, the idea is an interesting one, and it's great when

1:02:26an idea that was worked in labs for years and years suddenly becomes feasible in the

1:02:30real world due to engineering expertise, which is essentially what they're trying to bring

1:02:34as a company to this.

1:02:36And the theoretical benefits are there in that if you could get a system like this,

1:02:40you can make a much more effective use of bandwidth and you can put small cell tower

1:02:45type emitters, receivers everywhere.

1:02:48And don't worry about how they overlap because it'll be just fine.

1:02:51would make for the ability to support more users in a congested urban environment.

1:03:00And the other trick they have from engineering perspective is they didn't want to have to

1:03:03change all our phones.

1:03:04So they found a way using the existing features of LTE, 4G that are in the existing protocol,

1:03:09such that an unmodified, plain old iPhone 5S or any other plain old cell phone can participate

1:03:15in this.

1:03:16They don't need to have a special chip in the phones or special computing in the phones

1:03:19or whatever.

1:03:20again is another engineering type of breakthrough where, yeah, this was fine in theory, but

1:03:23you can't just go replace all the cell phones in the world with these magic ones that work.

1:03:26It's like, well, we found a way to get this to work using existing protocols with existing

1:03:30unmodified phones with all the smarts on the other end.

1:03:33So I think it is an interesting engineering achievement, assuming it works as advertised.

1:03:38He still faces go-to-market challenges in terms of getting this technology into all

1:03:42the cell towers and who's going to be the first person to roll it out and how will it

1:03:44coexist with other things.

1:03:46But in the end, it is not magic.

1:03:48It's just science.

1:03:49It's not particularly new science, but the engineering is where the interesting parts

1:03:53are.

1:03:54There was a really neat video that I watched that showed—I believe it was a bunch of

1:03:59like laptops or tablets or whatever—that were equipped with LTE radios.

1:04:04And they'd shown—I'm going to probably get the details wrong—but they'd shown

1:04:09what happens when you move around, and they turned off the thing where it senses where

1:04:15you are.

1:04:16So basically they concentrated this beam of LTE in a specific spot, and just for the purposes

1:04:22of the demonstration, rather than allow that beam to move around with the device, they

1:04:28kept it stationary.

1:04:29And you could see as they moved just a couple of inches that the throughput would just plummet,

1:04:36and then they would put the device back where it started, and it would come back to full

1:04:40HD video or something like that.

1:04:42And then when they had it working as it's designed, which is to say it will follow the

1:04:47device as it moves, you would get this full HD signal as the dude's like, you know, just

1:04:52moving these devices around in the room with a reasonable quickness, not 70 miles an hour,

1:04:58but a reasonable quickness.

1:04:59And it was fascinating to watch and really, really impressive.

1:05:02In some ways it's like engineering at its best because what engineers do very well,

1:05:08As opposed to scientists, like, two sides of the same coin.

1:05:12The scientists will come up with something, or these theories, or whatever.

1:05:15But the job of the engineers is to be able to figure out, you know what?

1:05:19That theory was useless to us 10 years ago, 20 years ago, 50 years ago.

1:05:24But computing has advanced so much, or technology, or material science, or whatever, has advanced

1:05:28so much that that crazy theory that no one ever pays attention to, you know, that it's

1:05:32like, well, that's not feasible.

1:05:33Obviously, you can never do that.

1:05:35Starts to enter the realm of feasibility.

1:05:37And engineers are rewarded by getting the jump at everyone and saying, "You know what?

1:05:40I think I can do that now.

1:05:42I think I can do that crazy beamforming thing across huge urban areas to people traveling

1:05:47up to 70 miles an hour with existing..."

1:05:49I think I can do that now, to get there first.

1:05:52Because eventually when technology comes along and it's like, well, now we have...

1:05:55Everyone will realize we have the computing to do an algorithm that wasn't feasible before,

1:05:59a compression technique that used to be too computationally expensive but now is trivial

1:06:03on phones.

1:06:05And this is an example of that.

1:06:07He seems to be out ahead of people in terms of, "I think I can do that now, and here's

1:06:11my proof and here's my implementation."

1:06:14And whether or not it flies, kind of like on live, like I think I can make people do

1:06:16video games where the video gaming machine is in the data center and they are miles and

1:06:20miles away.

1:06:21I think gaming can work like that.

1:06:24It did kind of work.

1:06:25I have on live, I've launched it, I've played it, it's not great, but as a technology proof

1:06:30of concept, it's interesting.

1:06:32As with anything else, the engineering is only one small part of the battle of being

1:06:36successful in the market. But as non-magical as it might be, I do like what he does because

1:06:44kind of like Elon Musk, he's figuring out what is feasible with current technology and

1:06:50getting there just slightly ahead of everyone else. And that's great engineering.

1:06:54Yeah, like I said, it was really impressive to me whether or not it's just the combination

1:07:01of a bunch of things we've known how to do for a while. It looked really cool.

1:07:06That's all Apple does too, right?

1:07:07Take things, you know, we've had touch screens, we've had capacitive touch screens, we've

1:07:11had user interfaces with buttons, like, you know, bring it all together, figure out that

1:07:15actually we can make a phone like that now and do it just a little bit ahead of everyone

1:07:18else.

1:07:19There's a big reward if you get all the parts right.

1:07:21Exactly.

1:07:22We are also sponsored this week by Igloo Software.

1:07:26Igloo is an intranet you'll actually like.

1:07:29Now most people think of intranets as, remember this is not internets, this is intranets,

1:07:33the old boring thing, right?

1:07:35the old, stale, terrible places that the corporate overlords make you go when you just want to

1:07:40use Dropbox or WordPress or something easy that helps you get your work done that actually

1:07:43works. Igloo brings the ease of use and familiarity from consumer software into your corporate

1:07:48environment by using familiar apps like shared calendars, Twitter-like microblogs, file sharing,

1:07:54and more. Every piece of content in your Igloo can be social, with comments and like buttons,

1:07:59and each team in your company can configure their own workspaces.

1:08:03that's it for your users but what if you're in charge of IT? Well Igloo

1:08:07handles the security, hosting, and management for you. Igloo is SOC 2 type 2

1:08:11compliant and they host data securely in SOC 2 type 1 enterprise facilities in

1:08:17Canada, that's right Canada so everything is polite and CDRs cost too much, on

1:08:21their own servers. Igloo offers 256-bit SSL and they are not vulnerable to

1:08:27heartbleed. They offer backups, disaster recovery, single tenant, and shared

1:08:31environments, integration with many authentication and sync systems, including SAML services

1:08:35and LDAP, and more. Igloo can even work with HIPAA compliant organizations. You can also

1:08:42customize everything inside your Igloo with the ability to add CSS and JavaScript globally

1:08:46across one team space or even on a single page. You can see all this on the Igloo website,

1:08:51which is actually built on their platform. Learn more at igloosoftware.com/casey.

1:08:55It's igloosoftware.com/casey, where they made a funny landing pig about why SharePoint sucks

1:09:03and how igloo is better.

1:09:05Thanks a lot once again to igloo for sponsoring our show.

1:09:08I wanted to ask about scaling servers because we've had this in the show notes for a while.

1:09:13I love this.

1:09:15So on the topic list, there's just a simple bullet point that just says "scaling servers."

1:09:19Like you might as well have had a bullet point that said "computers."

1:09:24I wrote it there, and it was actually a reaction

1:09:26to a particular blog post that Brent Simmons put up,

1:09:29where he mentioned scaling and passing.

1:09:32And I don't want to get into a big thing about it,

1:09:33but it's just like, every time I see this come up--

1:09:36and we do it ourselves, and people

1:09:37are sloppy with terminology-- it's

1:09:39so important to keep reminding yourself the definition

1:09:44of terms, like what we should be talking about when

1:09:46we talk about scaling.

1:09:47Because we use it as an umbrella term to talk about everything.

1:09:49And I kind of alluded to it in the PSAL thing,

1:09:51but my only point is the old saw about the difference

1:09:54in performance and scaling.

1:09:55So much of the time when I see people thinking they're blogging about scaling, all they're

1:10:00blogging about is performance, which is fine and interesting as a topic of itself, but

1:10:04it's not the same thing as scaling.

1:10:05And so I just put it in there as a reminder to myself to throw it out.

1:10:09So in brief, performance is how fast you can make something go.

1:10:12If you've got a computer and you've got a task to be done, I can do that task in five

1:10:16minutes.

1:10:17I need to increase my performance now.

1:10:18I can do that task in three seconds.

1:10:19You've increased your performance.

1:10:21is I've got one task to do and I've got one computer. What if I give you 10 million of

1:10:26those tasks to do? Can you do them in the same time if you just use 10 million computers?

1:10:31That's scaling. It doesn't matter how long it takes. Even if it takes your computer a

1:10:34year to solve the problem, if I give you two of those problems, you can say no problem,

1:10:38I'll just buy another computer, I can do both of them in a year. Okay, what if I give you

1:10:41three? I'll buy three computers. This is linear scaling. I'll buy three computers, I can do

1:10:45all three in a year. What if I give you three million? I'll buy three million computers,

1:10:49I can do all three million of those problems in a year.

1:10:51It's not about performance.

1:10:52Scaling and performance, although they often seem related,

1:10:55are not the same concept.

1:10:57The failure to focus on scaling

1:10:59when you're talking about scaling

1:11:00and performance when you're talking about performance

1:11:02leads to all sorts of sadness.

1:11:04- Oh yeah, I mean, scaling is not how fast your system is,

1:11:09but how easily you can make it take on more of something.

1:11:14Usually more traffic, more users, more data,

1:11:17whatever the case may be, how easy is it to take on more?

1:11:21And how many changes do you have to make to your software

1:11:24and to its architecture to add that capacity?

1:11:28Because with almost everything,

1:11:31it is not nearly as simple as, oh, just add more servers.

1:11:34Because you might have like, all right, well, let's say,

1:11:36you can add more web servers pretty easily up to a point.

1:11:39Because web servers, you can distribute the calls

1:11:42to any of them with a load balancer,

1:11:44as long as you don't use weird shared sessions or anything.

1:11:46You can distribute calls to any of them.

1:11:48Any call coming in gets assigned to web server 67.

1:11:52And then web server 67 does whatever it needs to do

1:11:55and responds back.

1:11:56Okay, great.

1:11:57However, web server 67 probably has to access a database

1:12:01to do something.

1:12:02And how do you organize the databases?

1:12:04And that's usually, it's the data layer,

1:12:07not the web serving application layer

1:12:09that's usually the hardest to scale.

1:12:12Because that's when you have things like

1:12:15globally shared storage. You have things like limits on disk performance and on how many

1:12:22writes can a single server even write per second. If you've scaled away all the reads

1:12:30and given reads to all these replicating servers, that's fine, but can they even keep up with

1:12:35the writes? How many writes does the master do? And then if you split the roll up like

1:12:39that, then you get into really hard issues like, "All right, well, how do you cache data?

1:12:44can you be sure that the data that you have read

1:12:47off of the read slave is up to date?

1:12:49Otherwise, you can't really cache it

1:12:51'cause you might be clobbering something

1:12:52in the cache that is newer.

1:12:54Or you might be caching old data when new data is available.

1:12:57And there's all these really hard problems

1:12:59that go along with it that makes scaling

1:13:01such a complex, interesting topic

1:13:02and why it's such a hard problem.

1:13:06- And that's why when I see a blog post

1:13:08about how to make a data structure smaller

1:13:11or a query more efficient, and then the word scaling

1:13:13using that same conversation.

1:13:14It's like, that's performance.

1:13:16You're just talking about performance.

1:13:17And you're talking about the web tier.

1:13:19Since, oh, it's stateless.

1:13:21We can scale that out horizontally forever.

1:13:23Nothing scale-- even the web tier is not like that one-to-one

1:13:27scaling that I was using in my example.

1:13:29If you ever get there, you have reached the golden.

1:13:32You will be a bazillionaire.

1:13:33Because even just getting there, even on the web tier,

1:13:35it's like, well, but what about different data centers?

1:13:37Now we need geographical load balancing.

1:13:39And what about network connections?

1:13:41Nothing scales perfectly.

1:13:43But you're right, the first thing you run into it

1:13:46is with state.

1:13:46And you get into CAP theorem of consistency, partition

1:13:50tolerance, and availability.

1:13:53This is well-trod territory.

1:13:54Lots of people will concentrate in scaling.

1:13:56Google knows probably the most about it

1:13:58of any company in the entire world,

1:13:59given the scale they operate at.

1:14:00Microsoft seems to be learning a lot about it.

1:14:02Apple, not sure what they know.

1:14:03But scaling is super hard.

1:14:06Performance is super hard.

1:14:08And I mean, from the outside, this

1:14:12is all sort of esoteric stuff.

1:14:14But even for people who are just working on a small system--

1:14:17and it's like, it doesn't matter.

1:14:19You're not going to have Facebook-level users, right?

1:14:22But even on a very small system, 10 users, 100 users, 1,000,

1:14:26200,000, even then, you should be

1:14:28thinking about both performance and scaling

1:14:30and keeping a clear head about which one you're concentrating

1:14:33on.

1:14:33And don't concentrate too much on scaling and not

1:14:35enough on performance, or too much on performance

1:14:36and not enough scaling.

1:14:38just keeping a clear head about what the individual tasks are.

1:14:42And you will eventually find yourself, even in a small system,

1:14:45reading all those papers about the theory behind it

1:14:47and making your trade-offs about consistency, availability,

1:14:52and partition tolerance.

1:14:53You have to make decisions about it.

1:14:55If you don't make decisions, the software you choose

1:14:56will make the decisions for you, and you'll be sad about it.

1:14:59So that's all part of the learning process.

1:15:01But I don't know.

1:15:05I don't want to pick on any particular blog posts.

1:15:07I love reading those Brent posts.

1:15:08I love watching him think out loud about what he's doing.

1:15:10He's a great example of a great software developer,

1:15:12like showing his work as he works towards a product

1:15:18and everything like that.

1:15:20Since I've dealt with server-side software

1:15:22my entire career, I have a particular itch

1:15:25when I see the word scaling and it's not really

1:15:28talking about scaling.

1:15:29Maybe I'm just being a pedant, right, Casey?

1:15:32That's right.

1:15:33That's right.

1:15:34Well, and there is something to be said for performance optimization to a point.

1:15:39It's very similar to the premature optimization wisdom in the world, except, and Brent even addressed this in one of his posts on

1:15:45Inessential.com, which we should mention. He even addressed, when you're talking about server-side design,

1:15:51for client-side stuff, premature optimization is considered a bad thing because we have such fast hardware.

1:16:01we have so much memory these days on client-side devices

1:16:04and desktops and everything.

1:16:05We have so much, we have so many resources

1:16:07that it's just not really worth

1:16:11doing super micro-optimizations everywhere.

1:16:14However, on servers, things are a little bit different.

1:16:18On servers, you are paying for what you use basically.

1:16:22And if your app gets popular at all,

1:16:25even for a minimal amount of users,

1:16:30relatively speaking to something like Facebook.

1:16:33You know, if you get 100,000 users using a web service,

1:16:37that's substantial needs for the hardware.

1:16:41And depending on what your service is doing,

1:16:45that actually could need some serious resources.

1:16:48And so, you could be talking the difference

1:16:52between hundreds of dollars a month

1:16:55or thousands of dollars a month worth of hardware,

1:16:57or even more than that.

1:17:00And so it can really make or break your whole economics.

1:17:03It can make you busier or not as busy.

1:17:06One of the best things to happen for servers and scaling

1:17:10in the last decade has been SSDs.

1:17:12Because for a long time,

1:17:14for a lot of LAMP-type applications,

1:17:17one of the biggest limitations was the database's disk speed.

1:17:21And like at Tumblr, I was able to replace,

1:17:24I think it was about nine or 12 ReadSlave servers

1:17:28with one with SSDs.

1:17:31And nine or 12 servers that had 15K SAS disks in RAID 10

1:17:36being replaced with one SSD-based server.

1:17:39And those were old Intel X25e SSDs.

1:17:44They've come a long way since then.

1:17:46Now modern SSDs are even faster than those.

1:17:49And when you can make a big order of magnitude type jump,

1:17:54If you can go from 10 database servers to one database server,

1:17:59that kind of difference actually matters and is worth doing.

1:18:03That is something where performance

1:18:06has a direct impact, a direct measurable, feelable impact

1:18:09on scaling.

1:18:12Optimization for performance, to a point, actually is useful.

1:18:15Because in practice, most people who make these web services

1:18:21and make apps and stuff like that,

1:18:23most people are not going to get as big as Facebook and Google.

1:18:26Most people are not even going to get as big as smaller things that we've all heard of.

1:18:31Most people are lucky to get a hundred thousand users to their web service.

1:18:34And so,

1:18:36by making decisions

1:18:38that favor high performance,

1:18:41you can often keep... that could often be the difference

1:18:45between

1:18:46being able to run everything off of a cheap, easy to administer

1:18:49VPS or a fully managed cloud service at a reasonable price. And running a whole bunch

1:18:55of dedicated servers or paying tons of money for a cloud service or one of these things,

1:19:00that could make that big of a difference.

1:19:02And so it is, or on the scaling side, this could be the difference between needing just

1:19:07one fast database server and needing a multi-server setup. Or being able to fit all on one server

1:19:13or having to shard the database, which is really complicated and adds a lot of maintenance.

1:19:18having to use a different type of tool entirely like Cassandra, which is a world of hurt.

1:19:23It's just so incredibly not worth using.

1:19:28Performance does matter, to the point where having a high-performance

1:19:33system has a very good chance of keeping you from ever needing

1:19:38a difficult-to-scale system.

1:19:43Businesses, if they're successful, the actual business, the people who don't understand,

1:19:45the business people who don't understand technology

1:19:47and don't want to-- will apply both of these pressures to you.

1:19:49Because they will apply a pressure for you

1:19:52to increase performance, because they

1:19:54don't want to pay a lot of money for servers.

1:19:57But they will also apply the pressure of you for scaling,

1:19:59because they will say, my business plan says

1:20:01we're going to grow 50% year over year for the next five

1:20:04years.

1:20:04So you need to be able to say, OK, we have five customers now.

1:20:08In five years, we're going to have five million customers.

1:20:12So whatever multiple that is, I don't want to pay--

1:20:16if you have greater than linear scaling,

1:20:18it's like, OK, well, one customer costs us $10,

1:20:20but two customers cost us $100.

1:20:22You've got a scaling problem immediately.

1:20:23Because they want the business to grow,

1:20:25but they want it to grow less than linear.

1:20:27They want to be able to double their number of customers,

1:20:30but not double their hardware costs.

1:20:31And what you're telling them as well,

1:20:33that's not how scaling works.

1:20:34If we double our customers, I need

1:20:35to weigh more than double our hardware costs.

1:20:37And it's like, how much more?

1:20:38How close are you to perfect linear scaling?

1:20:41And so they want both.

1:20:43It's too expensive.

1:20:44Every time we get another client,

1:20:45we need to buy x amount of hardware,

1:20:47and it costs so much money.

1:20:48And we're making only a certain amount of money

1:20:49from each individual client we had.

1:20:51We can't afford to add that much hardware,

1:20:52so increase your performance.

1:20:53But year over year over year, they're like,

1:20:55"Look, we're gonna quadruple our customer base.

1:20:57"Can you quadruple our customer base?"

1:20:58And you're like, "No, we can't.

1:20:59"We got one database server.

1:21:01"We're already buying the biggest machine we could possibly,

1:21:03"that money can buy for our database server.

1:21:05"We need a strategy for scaling."

1:21:07And then you end up sharding,

1:21:08and what about people in different parts of the country?

1:21:10then you evolve it.

1:21:11If they're in shards, how do these people

1:21:12interact with each other?

1:21:14In some ways, it's easier for software

1:21:16where it doesn't matter, like Facebook,

1:21:17where you're like, the message shows up here

1:21:20a little bit sooner than it shows up there.

1:21:21It's not a banking system or whatever.

1:21:24But not everyone's that lucky.

1:21:26Sometimes it really has to be,

1:21:27that's where you get into the CAP theorem again,

1:21:29where you have to pick your trade-offs.

1:21:30And it's difficult.

1:21:33And even no matter how bounded your problem is,

1:21:35even, it just all does is scale everything down.

1:21:38If you're like, oh, I want to stick to VPSs or something

1:21:42really cheap.

1:21:42I don't want to buy dedicated-- like you're

1:21:43playing in the small leagues.

1:21:45But it still matters for your company.

1:21:46When you're going from 10 customers to 10,000,

1:21:49it's the same exact thing playing out.

1:21:51It's just a smaller scale versus companies

1:21:53that start off from day one buying

1:21:54massive dedicated hardware.

1:21:56And the only difference in the high end game

1:21:58is that the high end people are more likely to get

1:22:01into the situation where they are buying the most

1:22:03expensive computer equipment that money can buy from anyone

1:22:05on the entire planet.

1:22:06When they get to the end of their scaling rope

1:22:08and they're like, well, we've increased performance

1:22:11as much as we can go.

1:22:13No matter how much money we waive at somebody,

1:22:15we cannot get a bigger single machine to do task X.

1:22:18It's like, now, if you waited this long

1:22:20to think about scaling, you have a serious problem,

1:22:22because you've hit the performance dead end.

1:22:25I guess you would optimize your software,

1:22:26but yeah, it's the same story.

1:22:29It's just a fractal of itself.

1:22:31It just depends on how many commas there are

1:22:34before the decimal point in the invoices.

1:22:37(laughing)

1:22:38And this is one of the reasons why I really prefer

1:22:42to do things at a relatively low level

1:22:45and to do things that are fairly standard

1:22:46because even if you don't need scaling, in quotes,

1:22:51even if you don't think you need that today,

1:22:53knowing how to scale something

1:22:57or knowing the steps that you take

1:22:59and the hurdles you will face is useful

1:23:02when writing anything even before it needs to be scaled.

1:23:05So for instance, I've said on a number of occasions,

1:23:10I never rewrite database joins, ever.

1:23:13Joins are, joins basically put a lot of work

1:23:16on the database server, which granted,

1:23:17it is very optimized to do.

1:23:20However, I prefer to split up the joins into two calls,

1:23:23like one to fetch the list of IDs

1:23:25and the second to fetch from the target table,

1:23:28which ones you get and stuff like that,

1:23:29and design the schema and the code to do that.

1:23:32And that's for two major reasons.

1:23:34One is to give the database a little bit less work.

1:23:38Well, three major reasons.

1:23:40One, give the database less work,

1:23:41because database CPU power,

1:23:43database resources are expensive and hard to scale.

1:23:46The second one is caching.

1:23:48So maybe you can cache that first query,

1:23:51maybe you can cache the list of IDs you need,

1:23:54and then only fetch the records, or vice versa,

1:23:56only fetch the IDs and then fetch the records from cache,

1:23:58things like that.

1:23:59And then the third is that then

1:24:01that also gives me the ability later on,

1:24:03if the service gets really big to say,

1:24:05you know what, the users table now has to have

1:24:08its own database.

1:24:09This gets hit so often and it's such a drain of performance

1:24:12that we should split this up into its own database cluster

1:24:15entirely, so users is now on a different database.

1:24:18Then you can't do a join,

1:24:20'cause it's not even on the same server anymore.

1:24:22So even with Overcast, I'm not running any joins.

1:24:26I'm doing it the same way I've always done it,

1:24:27because it's not that much harder,

1:24:30And it's just in case in the future,

1:24:34that'll be fine, I'll be set up for that,

1:24:36I'll be ready to go.

1:24:37So many other things, like very careful use of indexing.

1:24:41One of the most useful books I ever read on scaling

1:24:45was High Performance MySQL.

1:24:48And I read the edition before the current one

1:24:51and I liked it a lot.

1:24:52The current one got a lot longer

1:24:54and the reviews weren't as good.

1:24:55So I'm not sure if the current one is great.

1:24:57I didn't actually read the whole thing.

1:24:58But the previous one was awesome.

1:25:00so maybe give it a shot.

1:25:05But I think it's important to make those decisions.

1:25:07Like if you're writing, suppose you're writing C code

1:25:11for an app, and you're calling something like "sturlan"

1:25:14in the loop to say like, "Oh, iterate through a string

1:25:18and calling 'sturlan' 'i' is less than 'sturlan' string."

1:25:21If you know about performance at all,

1:25:25you're gonna look at that and say, "Wait a minute,

1:25:27that's being invoked on every single call."

1:25:26you get this mentality that makes you make small decisions like that better.

1:25:31And in the course of a whole project that adds up. And at the end,

1:25:35even though any one of those calls probably won't matter,

1:25:39when you make every decision with a certain mindset and a certain sensibility

1:25:42and a certain wisdom about what will happen in the future,

1:25:45or what the cost of this might be or might add up to,

1:25:48when you make all those small decisions the right way,

1:25:52the sum of all of that is different and does matter.

1:25:56Same thing applies to scaling.

1:25:57When you make a bunch of small decisions to say,

1:26:00oh, you know what, I can put a little more work

1:26:01on the database here, or, you know,

1:26:03this doesn't need to be that optimized.

1:26:05Even, you know, the index isn't quite right

1:26:06to do this index only, or, you know,

1:26:10this is gonna have to scan a bunch of rows

1:26:11to get these results, but that'll be all right.

1:26:14You know, it's, how often do you really have to do that?

1:26:17Once you get into the mindset of thinking,

1:26:20okay, what's going to happen when I have to do this

1:26:23thousands of times a second?

1:26:25you'll probably never have to. But if you think about that from the beginning, you can make better,

1:26:30small decisions along the way that will add up and that will make scaling easier for you.

1:26:34Can we go back a step? You're saying that you'd rather make round trips to the database than do

1:26:42something that the database is specifically designed to do? That, I mean—

1:26:46I think what he's saying is that he doesn't want to use a database.

1:26:48Right. I mean—

1:26:51I know you don't like the newfangled key value storage

1:26:53things or whatever, but if you're not going to do joins--

1:26:56and I'm not saying that's the right or wrong decision,

1:26:58but if that's the route you're going,

1:27:01there are probably places you could store your data that

1:27:05would give you better performance

1:27:07for this same kind of usage pattern.

1:27:08I imagine the reason you wouldn't like them

1:27:10is because they're newer and less mature

1:27:13and you're used to the features of being

1:27:15able to back up MySQL and do easy replication.

1:27:17And those are all important features

1:27:19and the reason to stick with it, but you're

1:27:20kind of using MySQL in sort of a degenerate state. Like, you're using it as a really bad

1:27:25performing poor man's key value storage with some basic filtering, doing your own joins

1:27:31client-side, which is fine, because you get the other benefits of it. Your familiarity,

1:27:34easy backup, easy replication, reliability, and that's why you're avoiding, that's why

1:27:39you're not another MongoDB nightmare story.

1:27:42Yeah, I think, obviously I know, as you and as the chat have said, I'm going to get a

1:27:47lot of email about my no joins stance. But the fact is, first of all, you just said MySQL

1:27:54is way slower than some other system, and that's probably not the case. Certainly it

1:27:58is not always the case. The reason I use MySQL, even though I'm not using some of the relational

1:28:05database type features like joins, I don't use stored procedures either, that's not a

1:28:11thing where that's just asking for trouble, I don't use triggers, things like that. The

1:28:16The reason I do this is because MySQL is just frickin' awesome.

1:28:21There's no other way to say it.

1:28:23I've heard so much crap about MySQL from people who don't use it and are prejudiced against

1:28:28it or whatever.

1:28:29And I don't use Postgres, I don't use Oracle, I don't have any experience with those, so

1:28:32I can't tell you how it compares to those.

1:28:34But I can tell you, most of the criticism I've heard about MySQL is wrong, or at best

1:28:39outdated and only applied to my ISAM and not the more recent InnoDB storage engine, which

1:28:44at this point is not even recent.

1:28:47And so if you use MySQL with InnoDB as your storage engine

1:28:51for all tables, it's amazing, it is awesome.

1:28:55And I said this in a post recently,

1:28:58and the gravity of this, I don't want to overstate this,

1:29:01but it's hard to overstate this.

1:29:04In all the time at Tumblr, Instapaper,

1:29:06all that time, now Overcast, I've used MySQL a lot.

1:29:11With Tumblr, it was under extreme stress

1:29:13for the entire time I was working there.

1:29:15It was constantly under extreme stress.

1:29:18I have never seen MySQL crash once.

1:29:22I have never seen MySQL corrupt data once.

1:29:25That's amazing.

1:29:26Look at any--

1:29:27- You've never seen it corrupt data?

1:29:30- I have never seen it corrupt data.

1:29:31I've never seen it crash.

1:29:33- MySQL's actual functionality could be argued

1:29:35to be a corruption of data.

1:29:37I know it's documented.

1:29:38I know you're looking up and the doc says

1:29:40that's exactly what it's gonna do.

1:29:42It's not wrong.

1:29:43It's not a bug.

1:29:44But that's the philosophical difference

1:29:46many, quote unquote, "real database" people have

1:29:48with MySQL is the documented behavior.

1:29:51When it performs that documented behavior,

1:29:53database people grit their teeth and get angry at it.

1:29:56And I'm kind of one of them.

1:29:58I'm not that familiar with exactly what

1:30:00you're talking about.

1:30:01However, are you sure you're talking about InnoDB

1:30:03and modern versions of this?

1:30:06No, no.

1:30:07Like weird, weird coercions of data types and cases

1:30:10where columns that are marked as not null

1:30:12can actually have nulls and weird things

1:30:14with empty strings.

1:30:15They're all documents.

1:30:16You go to one of those MySQL gotcha pages

1:30:17and just read through them.

1:30:19In terms of performance, though, now that I'm saying

1:30:22you should switch to a different thing or something,

1:30:23but because most of the things that have massively better

1:30:26performance have much, much less of the other things

1:30:29you just described, like stability and reliability.

1:30:31But performance-wise, if your data model

1:30:34is kind of simple and never going to have joins,

1:30:38And if you care about getting more performance, which you may or may not, there are lots of other database systems that do go way faster than MySQL, especially within ODB.

1:30:48Because the extra overhead of that storage engine...

1:30:52The new hotness is basically people do almost everything in memory.

1:31:00It's not like memcache, although there is a persistent memcache thing, but a lot of them are like, "Look, we can't involve disk in the operation of our thing here."

1:31:07And as you can imagine, when you do almost everything in memory, you go way faster than MySQL trying to do stuff with InnoDB.

1:31:12Oh, sure. And there's ways you can take advantage of that with caching already.

1:31:20And by the way, MySQL does already cache a lot in memory.

1:31:24And of course, the OS will cache a lot of the disk blocks in memory and stuff like that.

1:31:27But certainly you can use MySQL with Redis or Memcache or any of the cache systems that are out there.

1:31:33I mean, most people do, it's fine.

1:31:35You know, that's like people, there's this big,

1:31:39it's the cool thing to use a different storage engine

1:31:42these days, but the fact is a storage engine is,

1:31:45oh boy, here we go.

1:31:46A storage engine is like a file system.

1:31:48You know, you don't wanna change that very often

1:31:51and you want it to be extremely conservative

1:31:53and extremely focused on data integrity.

1:31:56And MySQL is so incredibly like battle tested.

1:32:00It is so solid.

1:32:02And yeah, I get that it might not have always been that way

1:32:04back in the olden days with myISAM and all the crap,

1:32:06but those days are well into the past now.

1:32:09And modern MySQL, MySQL has been rock solid,

1:32:12I would say for at least seven or eight years.

1:32:15And that's why Google use it very heavily.

1:32:18I don't know if they still do, but they at least did.

1:32:20I know lots of major, major sites use MySQL.

1:32:23Twitter use it for a long time, they probably still do.

1:32:27There's a reason why, it's really, really good.

1:32:30And there was this great blog post a few years back

1:32:35when FriendFeed existed, to give you some idea.

1:32:39And it was a blog post called

1:32:40How FriendFeed Uses MySQL to Store Schema-less Data.

1:32:43And I'll have to find it and put it in the show notes.

1:32:45But there's a lot of good stuff in here.

1:32:47FriendFeed was a weird project

1:32:50that was run by incredibly good engineers.

1:32:52And it basically addresses a lot of this stuff.

1:32:57Like MySQL, even if you don't use

1:32:59of the database-y features of it, MySQL is still an amazingly fast, solid, low-needs,

1:33:07low-maintenance storage engine that has tons of tools, a huge ecosystem, tons of optimization

1:33:14potential. There's so many reasons to use it. And if you look at one of the newer systems,

1:33:21it's -- and things like Mongo or all these new NoSQL-type systems, if you look at a lot

1:33:28of those, yeah, they have some benefits, but if you don't necessarily need those benefits,

1:33:32and what you really want is for your database to be like a file system and to be basically

1:33:36bulletproof, MySQL is a great choice. And I feel like it gets ignored because it's old.

1:33:42This comparison is kind of harsh, so I don't mean it the way it's going to sound, but it's

1:33:47as close as we can align them for us to align these two things.

1:33:50Here we go.

1:33:51In the world of databases, what is the PHP of databases? It's kind of MySQL. It's way

1:33:56way better than PHP, don't get me wrong.

1:33:57But if you have to map one to it, that's what ends up.

1:34:00And what's the Python of databases?

1:34:01That's probably Postgres.

1:34:04That's more or less how things line up.

1:34:06And what's the C, C++ of databases as Oracle?

1:34:11And what's the Ruby?

1:34:11It's like VaultDB or something.

1:34:14But to align things, I say very clearly,

1:34:17MySQL is the PHP of databases.

1:34:19And that sounds terrible, I know,

1:34:20because it is not as bad as PHP.

1:34:22Do not get me wrong.

1:34:23PHP is terrible.

1:34:24But it has many of the same characteristics,

1:34:27in that it's everywhere, it's a known quantity, it's reliable,

1:34:30it has idiosyncrasies.

1:34:31But once you know what those idiosyncrasies are,

1:34:33they don't bother you that much.

1:34:34And the thing about using it for schema lists,

1:34:36that's why people do that.

1:34:37Because they're like, OK, I have a tool that I know is reliable.

1:34:40I know what it's capable of.

1:34:41And I'm going to use it, like I said,

1:34:43in a degenerate kind of way.

1:34:44I'm actually going to use it for schema list data.

1:34:46I know it's not a schema list database system.

1:34:48I know there actually are schema lists,

1:34:49but I'm going to define some table with columns

1:34:51for key and value and version number and some other crap.

1:34:53And it'll work.

1:34:54And it's like you were taking advantage

1:34:56of a system that works in a way that you're comfortable with,

1:35:00that has all the features you need and is reliable,

1:35:02and you're building something else on top of that.

1:35:05And all I was saying with the other database type things

1:35:07is if you care more about performance,

1:35:09there is these days much more performance

1:35:11to be had if you're willing to trade it for less

1:35:13reliability and more unknowns.

1:35:15And I'm hoping that that younger section of the data storage

1:35:19world matures and coalesces, and we

1:35:21get some sort of equivalent of MySQL, or even best case,

1:35:25some equivalent of Postgres, where

1:35:26it's like a tried and true known quantity within that realm.

1:35:29Right now, we don't have it now.

1:35:30Right now, it's kind of the Cambrian explosion over there,

1:35:32and we're not sure how it's going to shake out.

1:35:35But it's still worth keeping your eye on,

1:35:37because if 20 years from now, you're still

1:35:40using MySQL with no joins, there will

1:35:42be either a great failure of the NoSQL world

1:35:45to come up with a useful product for the long term,

1:35:48or a great failure of you to keep an eye on what's

1:35:50going on over there. Possibly both. I have two questions. Firstly, what is the

1:35:55pearl of databases, Jon? Yeah, I was trying to think of that. I don't know. Maybe MSQL,

1:36:01but neither one of you remember that. No, there really is no pearl, but if there is,

1:36:06I don't know what it is. It's probably one of the newfangled database systems that I'm just not

1:36:11familiar enough with. Because there's nothing that has the same combination of a precursor to all of

1:36:18the better known ones now with lots of weird quirks.

1:36:21I don't know what that is, but it's not MySQL, definitely.

1:36:26Isn't it just some academic paper that swears that it's academically flawless?

1:36:31No, that's Perl 6, you're thinking of.

1:36:34Is there a database that can be written but not read?

1:36:36PHP looks just like Perl.

1:36:37You can't even make that comment.

1:36:41Let's not go down that road.

1:36:42Can we still back up to, you just spent, Marco, five or ten minutes explaining how MySQL

1:36:47is amazing and reliable and does what it's supposed to do really well, but you don't

1:36:52trust it to do a frickin' join? Like, I can't get past that.

1:36:55No, no, it isn't an issue of trust. It's an issue of leaving yourself open for future

1:37:00scaling options. It's about being able to say, "You know what? My database server

1:37:04is going to at some point be split off from my web server once my service has more than

1:37:09five users, and this is going to have its own server, and it's going to be harder

1:37:13and more expensive to increase database hardware resources

1:37:17than to increase application or web hardware resources. And so

1:37:21you might as well have the application and web stuff do more of the work

1:37:25and make the database, give the database an easier

1:37:29job. So things like, you know, reduce the number of queries to it, but the number

1:37:33of queries that go to it, you know, make them access fewer

1:37:37rows, make them access fewer tables, make them have to do less CPU work, but it's more

1:37:41about IO, you know, let the database do only what the database is required to do, and let

1:37:48your other servers take on as much work as they can.

1:37:51Okay, but you'd rather ping away at it and do a gazillion round trips to avoid these

1:37:58joins? Like, I haven't been through a Tumblr, I haven't been through an Instapaper, so I'm

1:38:05doing that thing which I told you I don't want to do in the beginning of the episode

1:38:08where I'm saying that seems weird. But genuine question, have you ever done anything to empirically

1:38:14prove to yourself? Like, have you used a new relic to prove to yourself that this is legitimately

1:38:19the right way to go? Like, you can do it on paper. I think you could convince yourself, but I'm sure

1:38:24I'm not going to answer for Marco, but I'm sure the answer to that is yes, because anyone who works

1:38:27in a service I'd offer quickly realizes, oh, the part that has all the state is the hardest part

1:38:31to scale. I mean, he basically just laid it out. Like, you can just do it on paper. There's a

1:38:35certain amount of work that needs to be done. One part of your system you can scale really,

1:38:38really, really easily.

1:38:39One part is really hard to scale.

1:38:41You want the hard to scale part to do the same--

1:38:44whatever operation it does the most sufficiently, primary key

1:38:47lookups maybe, have it do that same operation over and over

1:38:50and over again.

1:38:51And have everything else you can do

1:38:53put on the easier to scale parts.

1:38:54Like, there's lots of ways you can do this with math

1:38:56to figure out how it works.

1:38:59But I think even if you just think about that logically

1:39:02and reason through it--

1:39:04and that's not even going to get you linear scaling.

1:39:06We're just trying to keep our heads above water

1:39:08with this type of strategy.

1:39:09You always want to push the work out

1:39:13to the system that's easiest to scale.

1:39:14And so it's kind of the Google approach with Bigtable,

1:39:17where they went to the extreme where their data store was

1:39:21really annoying to deal with and unreliable.

1:39:23And applications had to retry and figure out

1:39:26how to resolve conflicts.

1:39:27And it made it incredibly hard to write applications.

1:39:30But that was the price of scaling early on.

1:39:32And they've made strides in that way.

1:39:33But basically, you're trying to move

1:39:35work to the systems that are easy to scale, and web servers are way easier to scale. And

1:39:39not performance, scale. And again, you're confusing performance and like, "Oh, you've

1:39:44got to take those round trips. You're going across the network. You're making multiple

1:39:47queries." It's like, that's performance. We're talking about scaling.

1:39:50Yeah, that's a fair point. And I did that deliberately.

1:39:55For whatever it's worth, you're putting a lot of weight on these round trips. Round

1:40:00trips to a database server or to a memcache server that's in the same rack, or at least

1:40:06in the same data center as the server that you're coming from.

1:40:09That's a good point.

1:40:10We're not talking about a lot of time here. A lot of this depends on the kind of application

1:40:16you're writing. How much data are you actually querying? To build the page or to build the

1:40:20API response that you're talking about, how many database calls do you actually have to

1:40:24Is it seven or eight? Is it fifty?

1:40:28Is it more? What you're doing matters a lot.

1:40:32Instapaper was fairly easy to scale

1:40:36because it was never as big as Tumblr and I was using Tumblr style techniques

1:40:40at Instapaper so of course it was way overkill.

1:40:44Which was great, which is why I was able to afford to run it

1:40:48and not go crazy and not go broke. But

1:40:52At Tumblr we face things like, to give you one example,

1:40:55how do you display the list of posts

1:40:59on somebody's dashboard?

1:41:01So you have to figure out who they follow

1:41:03and then of everyone they follow,

1:41:05to find enough posts from those people

1:41:08to make 20 of them and order them properly.

1:41:10There are so many different ways to do that.

1:41:12And there are the naive ways where you do a few joins

1:41:16and then you do a big sort and it's all fine.

1:41:19The naive way does not really scale very well

1:41:22just because you start dealing with hundreds of gigs of data pretty quickly and the database

1:41:27having to scan millions of rows and that starts performing very badly.

1:41:33So a lot of times what you think is the most good, naive approach actually in practice

1:41:42is not useful. It does not scale well enough. It becomes too expensive or becomes completely

1:41:48impractical or impossible to do at scale. So you start having to do weird little

1:41:53hacks. And so you know one of the hacks that we did at tumblr early on was

1:41:58actually something that I learned from Wikipedia what they did on their schema

1:42:01back and this was probably 2007 or 2008 which is you actually have basically a

1:42:07separate table that acts as an index because in certain ways it uses memory

1:42:13more efficiently to do that or in certain ways the query optimizer will

1:42:16use that better. And so we had a posts reference table that was basically the... It was a special

1:42:23index of the posts table, and the posts table was this massive thing that was hundreds of

1:42:29gigs that would only ever be accessed by primary key. Because, as John said, primary key accesses

1:42:33are very fast and everything's optimized for that. They cache very well, everything else.

1:42:37So you get the list of post IDs from other calls, and then you could fetch cached posts

1:42:42or fetch posts from the database or even get them from slaves, and it was mostly all right.

1:42:46So there's all sorts of things you can do

1:42:48if you leave yourself options.

1:42:49That's what I'm saying.

1:42:50So, and you know, look, I would love,

1:42:53again, this is like the idea,

1:42:54this is almost the final discussion.

1:42:56Academically, I would love to say, yeah,

1:42:59you know, just use the database as intended

1:43:00and it works great and that's the easiest and fastest.

1:43:03In practice, it's not.

1:43:05In practice, things are more complicated

1:43:07and you can do the nice simple way for a while,

1:43:10but eventually you're gonna hit a wall.

1:43:13And what options you have when you hit that wall

1:43:16depends on how you've built the system,

1:43:18and how much effort it is to get over that wall,

1:43:20and how much it will cost you,

1:43:22in both things like hardware or service costs,

1:43:25and in administration time and maintenance over time.

1:43:30That will all depend on how you've built that system,

1:43:32and if you've built it to give yourself

1:43:34good scaling options in mind,

1:43:35and to do things like be gentle on your databases,

1:43:39you will have more options,

1:43:41and it will be cheaper and easier to scale.

1:43:43- Here's an analogy that I think

1:43:45case you will be able to identify with.

1:43:47It's like, you know when you have like-- well, maybe in C++

1:43:49at least-- when you have code and you want to make it faster,

1:43:52very often that involves making it longer and uglier?

1:43:55This is the same type of thing.

1:43:56You're like, how could it be faster?

1:43:58I'm adding more and more lines of code.

1:43:59Like whether you're unrolling a loop

1:44:01or setting up some big bunch of setup stuff

1:44:03to do a bunch of SIMD operations on it, it gets longer

1:44:06and it gets uglier and it looks like you're doing more.

1:44:09This seems like it's more work.

1:44:10In the same way, you're like, well,

1:44:12I had this nice query that got me all the data I wanted

1:44:14one big self-consistent blob with this nice joint query, and replacing it with these multiple

1:44:18queries, when I just mesh the data together myself, it's longer. How could that be better?

1:44:24I guess it's not fully accurate in the C++ case. You're actually increasing performance,

1:44:28and here, like Marco said, you're protecting yourself for future scaling. Although in some

1:44:33cases, even in the deep dark world of databases, you've mentioned stored procedure before,

1:44:37if your query planner is not your friend, like say if you use Informix, speaking from

1:44:41of experience.

1:44:43Sometimes, if you want to get a massive amount of data,

1:44:45it's better to write a stored procedure

1:44:48and essentially be the query planner yourself.

1:44:49Do it all in the database, all right on top of the thing,

1:44:52but say, look, when I give you this join,

1:44:54you're going to do something dumb.

1:44:55So let me run this query, this query, this query,

1:44:57join the mesh results together, put them in a temp table,

1:45:00index the temp table, join against that temp table

1:45:03with the second thing.

1:45:03And you're like, how could that possibly

1:45:05be faster than just running the query you wanted?

1:45:07It's like, well, the query planner

1:45:08made some very unfortunate life choices

1:45:09when we sent it this query.

1:45:11And as a matter of fact, doing this crazy stored procedure--

1:45:14I mean, once you're creating temp tables and indexing them,

1:45:17you're like, there's no way that could be faster

1:45:18than running that query.

1:45:19Well, let me show you.

1:45:21Databases are funny things.

1:45:22And the more you can treat them the way

1:45:25Marco's treating them, the less headaches

1:45:27you'll have about them.

1:45:28And I think the more you should think about whether you should

1:45:31even be using a database.

1:45:32But that's a conversation another time.

1:45:34But anyway, the idea that the client side code

1:45:38the consuming code gets longer and uglier and more complicated, that is not outside

1:45:43the realm of normal programmer experience when just increasing the performance of your regular compiled code.

1:45:48Yeah, that makes sense.

1:45:55And it's a balance. It's a balance you have to strike. Obviously, if you're hitting severe problems and severe scaling challenges,

1:45:56you're going to have to go more in that direction of more complex code.

1:46:04And that'll be more bugs. You'll have things like weird caching bugs and things like replication

1:46:08delay or eventual consistency bugs depending on what kind of structure you're using.

1:46:13And that's all hard, that's all complicated. And

1:46:18certainly you shouldn't do that sooner than you have to in most cases. But again,

1:46:23it's a balance. Just like you shouldn't waste all your memory as much as possible when you're writing a C program,

1:46:28or any program for that matter, you know, you've got to find the balance.

1:46:33And with servers and scaling, I would lean a little more,

1:46:37just because the nature of using someone else's

1:46:39vast resources on their computer

1:46:41versus expensive shared server resources,

1:46:44I would lean a little bit more towards

1:46:46a little more complicated, but way more scalable.

1:46:50- That makes sense.

1:46:51And far be it for me to argue with you, Mr. Tumbler.

1:46:54- Thanks.

1:46:56Can I get that on a business card?

1:46:57All right, thanks a lot for our three sponsors this week,

1:46:59PDF pen for iPad, New Relic, and Igloo.

1:47:03And we will see you next week.

1:47:05(upbeat music)

1:47:08♪ Now the show is over ♪

1:47:10♪ They didn't even mean to begin ♪

1:47:13♪ 'Cause it was accidental ♪

1:47:15♪ Oh it was accidental ♪

1:47:18♪ John didn't do any research ♪

1:47:20♪ Marco and Casey wouldn't let him ♪

1:47:23♪ 'Cause it was accidental ♪

1:47:26It was accidental And you can find the show notes at ATP.FM

1:47:34And if you're into Twitter You can follow them at

1:47:39C-A-S-E-Y-L-I-S-S So that's Casey Liss

1:47:44M-A-R-C-O-A-R-M Aunty Marco Armin

1:47:50S-I-R-A-C-U-S-A-C-R-A-C-U-S-A

1:47:55It's accidental, accidental They didn't mean to

1:48:00Accidental, accidental Tech podcast so long

1:48:08I'd like to tell you about why I hate my car now.

1:48:10Wow.

1:48:11Is it because-- did it get dirty finally?

1:48:13No, it is actually relatively dirty at the moment.

1:48:18Aaron and I decided this past Saturday to go to the local drive-in movie theater.

1:48:24And if you have a local drive-in movie theater, I cannot suggest enough that, assuming you

1:48:29don't have small children, which I guess definitely eliminates Marco and probably eliminates John

1:48:34in this particular context, if you don't have small kids, you can escape the kids, go to

1:48:39a drive-in movie.

1:48:40It's really cool.

1:48:41And so the way the Goochland drive-in theater works, which is between Charlottesville and

1:48:46Richmond is you pull in, you pay something like $8 a person in your car, and you get

1:48:52a double feature.

1:48:54And you get the audio for the double feature by tuning to a FM station that the theater

1:49:00broadcasts.

1:49:01And it's a really cool experience, especially on a really pretty night like this past Saturday

1:49:06was.

1:49:07And we have been several times in the past, although not for a year or two.

1:49:10And when we've gone in the past, we've taken my car, which at the time was my Subaru.

1:49:15What color was that?

1:49:17That was white.

1:49:18We take an Aaron's car, which was and is still a Mazda6, which is a grayish silver, and we've never had an issue.

1:49:25Well, this past Saturday, I had just washed and waxed and leather-conditioned and vacuumed Aaron's car,

1:49:32and so I thought, "Well, my car's dirty," because yes, my car does get dirty, it just takes longer than 40 nanoseconds,

1:49:38um, unlike Marco's car. So we decided to take the BMW.

1:49:43The BMW has a push button starter and it has an accessory mode.

1:49:48And so I figured self, there will be no issue here.

1:49:52What we'll do is we will go to the drive-in theater.

1:49:56I will put the car in accessory mode by turning it off and then

1:49:59pressing the push button once.

1:50:01And we will listen to the movie on the stereo and it will be wonderful.

1:50:05I will turn off the iDrive display.

1:50:06I had to figure out how to do that because I completely forgot.

1:50:09And I will turn off the iDrive display and everything should be good and right

1:50:13in the world. So that's what we did. And after 10 minutes, I heard the BMW chime that you

1:50:20might remember from neutral. And it was telling me that my battery was dying.

1:50:25What? After 10 minutes of having the radio on.

1:50:28Doesn't your car have two batteries? Or is that just Marco's car?

1:50:31I don't think Marco's car does, does it? Not that I know of.

1:50:34Marco's car has like 10 batteries, right? I thought it only had one.

1:50:38You couldn't find the one battery that one time, so we don't know how many batteries

1:50:41are in this car. This car could be filled with batteries.

1:50:43I thought they're always on the passenger side in the trunk.

1:50:46No, we went through this.

1:50:47Anyway, I thought his car had two batteries, and I thought that was a common BMW thing,

1:50:50but what do I know?

1:50:51I buy Honda's.

1:50:52It may be, for all I know.

1:50:54But anyway, now, it could be that the battery in my car is original.

1:50:59I believe it is, and it was purchased originally in, I think, December 2010.

1:51:04So we're getting to the point that maybe this is all a battery issue that I'm misconstruing

1:51:08to be a car issue.

1:51:10But what I can tell you is that the radio at least once turned itself off because it

1:51:15felt like it was tired of being on, and it had thought that I'd left it on not deliberately.

1:51:21And unlike in a keyed car where you physically put the key into accessory mode, in a push

1:51:27button car it's just being told, "Well, yeah, go put yourself in accessory and I guess

1:51:31we'll hope that you don't turn yourself off."

1:51:34And it turned itself off.

1:51:35And then I turned it back on and like I said after like another 10 minutes or something

1:51:38like that, it started digging away about how the battery was dying. So we ended up watching

1:51:42only the first of the double feature by lowering the windows, which by the way used a whole

1:51:45lot of juice because, you know, moving a motor is a heck of a lot harder than having an FM

1:51:52radio on. But I did that, we lowered the windows and we listened through everyone else's FM

1:51:56radios because my damn car wouldn't stay running and I was afraid I wouldn't be able to crank

1:52:01it. And the other really interesting thing is even though they have little jump starting

1:52:06boxes at the theater, I was so scared that it would get so dead and the jump-starting boxes would take so long to trickle-charge it

1:52:14that I wouldn't even be able to push start the car

1:52:16because it wouldn't be able to engage the push-button ignition into run mode to get the thing so I could frickin' push start it.

1:52:22So it sort of ruined our entire movie-going experience and granted we're talking about a sum total of $16,

1:52:28but it was really really really annoying.

1:52:30So now if we take my car to the movie theater again, if for no other reason than the auto-disable,

1:52:35We're gonna have to take a frickin stereo for like a boombox with us and keep it quiet so we can listen to the stupid movie

1:52:42Yeah, it's a good thing you already got her to marry you because you were not impressing your date on this night

1:52:47Okay, two things one the fact that you can solve this problem by getting like a $20 boombox makes it a lot less of a problem

1:52:57true

1:52:58Two, do you think anybody who designed this whole like, you know, electronic push-button start and accessory mode and automatic turn-off

1:53:07Do you think any of those people involved in those decisions have ever been to a drive-in movie theater?

1:53:12Oh, certainly not, but it's still very annoying

1:53:16You know some of the recent iPods had FN tuners in them like the old iPod Nanos you have to do those hanging around

1:53:22Did they have FN tuners?

1:53:23Oh yeah, we could do that with the headphone splitter actually. That would be totally ridiculous,

1:53:28but it could work.

1:53:29And also you should just get a new battery for your car because that's ridiculous. And

1:53:32yes, batteries do go bad sometimes suddenly after many years, so you do have to replace

1:53:36them.

1:53:37Yeah, and actually, the more I think about it, the more I think that may be the issue

1:53:39in terms of the warning about, "Oh god, there's actually, there's something that pops up on

1:53:44the iDrive." It's increased battery discharge because it gets very upset that the battery

1:53:49got as low as it did. And so the next time I take the car in, which will probably be

1:53:53for the N55 recall that everyone just real—or that BMW just announced, I will certainly

1:53:59ask them to either replace it or double, triple check that it is as healthy as they claim

1:54:04it is. But man, what a stupid problem to have. I mean, maybe this is the biggest first world

1:54:11problem ever, it probably is, but what a frustrating, silly problem to have. Because in Aaron's

1:54:15which has a key like most normal cars, none of this would have been an issue.

1:54:19And I also wonder if part of the problem was because the iDrive is a whole frickin' computer,

1:54:25I wonder if the whole damn computer was turned on with a little 12 or 15 gig hard drive spinning and so on and so forth,

1:54:33even though the display was off. In other words, there's not like a

1:54:36short circuit, if you will, just for the FM radio to keep that on, and the whole freaking nav and

1:54:42everything was powered on just so I could listen to the radio.

1:54:45Do you think you could have brought the soundtrack to the movie on vinyl and maybe played it on a gramophone?

1:54:50Then maybe you don't need any power at all

1:54:52Are we done here

1:54:55Are we done I hate you

1:54:58[BLANK_AUDIO]