372: Things Are Degraded


00:00:00   Do people still buy Xboxes? Is this fight even still on?

00:00:04   Yeah, all three competitors are viable.

00:00:08   You know, Sony destroyed everybody in the last generation, but Xbox is viable.

00:00:12   They have enough money to make a second one, you know, and Nintendo as we know

00:00:16   is definitely viable. So the market can sustain three.

00:00:20   Each generation, maybe the winner will swap and Nintendo won

00:00:24   the Wii generation, Xbox won the PS3 generation.

00:00:28   See what happens this time. Sega did not win.

00:00:32   It hasn't won anything lately. Sorry.

00:00:36   Sega won the Genesis generation and that was it. No, it did not win the Genesis generation. Only in your mind.

00:00:40   No, I think... It depends on how you measure, but

00:00:44   they did very well. Go look up the numbers. It did fine. It was viable.

00:00:48   Viable? I think they were very, very

00:00:52   successful during that generation. Sega didn't win, but they were very, very, very

00:00:56   strong. But I still think Nintendo won. They won Marco's heart.

00:01:00   They did win my heart, dammit. Sega was awesome for that one generation.

00:01:04   I've said this many times on the show. I peddled so many Dreamcasts when I worked at Babbage's

00:01:08   and that just did not work out well. No. The Dreamcast was not viable.

00:01:12   It's sad, because the Dreamcast was a good system with good games,

00:01:16   but Sega just blew it so hard after the Genesis

00:01:20   that nobody trusted them anymore. I'm not going to say it was a good system. Dreamcast had

00:01:24   good games. The system? Oh, come on. It had the little

00:01:28   Game Boy memory card. What did they call that? The controller was awful.

00:01:32   The visual memory thing was dumb. There it is. Yeah, visual memory unit, the VMU,

00:01:36   something like that. It did have a fishing controller, but still. No, that fishing game was amazing.

00:01:40   Yeah, actually, this is not for this week, but I have

00:01:44   a whole retro gaming setup now, and I'm super into this world, and

00:01:48   one of these days I will talk about it on the show. It's

00:01:52   been quite a ride. Aren't there Sega games on Switch?

00:01:56   You can buy them in a virtual console? Yeah, yeah. Sega became just like, you know, software

00:02:00   for everybody now. Yeah, I feel like that's probably your best bet to get all your Sega

00:02:04   hopes and dreams on Nintendo consoles. Oh, no. I don't need to.

00:02:08   Well, I know you can do MAME. Yeah, yeah. No, I have my

00:02:12   Genesis running. You'll see.

00:02:16   I don't have time to talk about this week, so we can do it next week if you want.

00:02:20   Is that less work than just running them in an emulator? I don't know. Oh, it's way more work, but

00:02:24   it's way cooler. Alright, it has a warmer

00:02:28   feel. I'll get there. It's all about the

00:02:32   ritual. I don't have time this week. Save it for the show. Save it for the show.

00:02:36   John, you guessed right about the A12Z. I'm

00:02:40   very impressed. Yeah, we were talking about that when the new iPad had just

00:02:44   come out and we weren't sure what the deal with the system on a chip

00:02:48   was except that it was one letter different from the previous one and it turns out that one letter

00:02:52   stood for something that I speculated about on the show, which is that

00:02:56   all the parts on it are working. The A12X had seven GPU cores

00:03:00   as an insurance policy to increase yields because if one core is a dud

00:03:04   then you can still use it. Apparently now they've gotten good enough at manufacturing

00:03:08   this particular chip or they just saved all the high-bend ones for this particular thing, something like that.

00:03:12   But anyway, this is a common practice in the industry. That's what the A12Z is. It's an

00:03:16   A12X without any boo-boos.

00:03:20   I wanted to very quickly explain what this is because

00:03:24   I bet there are some listeners out there who don't know about this process and it kind of affects a lot of things

00:03:28   about chip making. If you ever hear people talk about a chip having low

00:03:32   yields or things like this or binning, roughly what

00:03:36   this is, and please correct me if I'm getting any parts of this grossly wrong and to any experts in the field, I'm

00:03:40   sorry because I only have a passing knowledge of it, but I just wanted to explain this. So basically

00:03:44   when you make silicon chips, like CPUs, GPUs and everything, they're made on

00:03:48   these giant silicon wafers. You've probably seen them. The big circular cool looking shimmery

00:03:52   silicon wafer and they're cut out. There's some rate

00:03:56   of flaws of either a flaw in the wafer itself or a flaw in the

00:04:00   process that is printing the chip on the wafer.

00:04:04   There is some rate of flaws and it's like, okay, well you might have a couple of flaws per

00:04:08   square inch or whatever the rate is. And so as you're cutting out

00:04:12   those chip dies from that giant wafer, some of them are going to have

00:04:16   bad flaws and you won't be able to use them. And yield is literally like

00:04:20   the percentage of how many that you make can be used. That's one of the

00:04:24   reasons why the larger a chip is, the more expensive it tends to

00:04:28   be because you can fit fewer of them per wafer and

00:04:32   if there's an imperfection, there's a higher chance of each

00:04:36   chip having an imperfection because it has more surface area basically.

00:04:40   And so if you think about if there's going to be a certain number of flaws on each wafer

00:04:44   there's a higher chance that fewer chips on that wafer will contain

00:04:48   no flaws. One of the ways you can deal with some flaws is

00:04:52   alright, we're going to manufacture a chip that has eight GPU cores on it.

00:04:56   But we're only going to actually spec it to have seven. And so we can

00:05:00   crank these out more so and for lower prices because

00:05:04   if there's a flaw and it happens to land on one of these GPU cores, we just

00:05:08   mark that one as the disabled one. And we enable the other seven

00:05:12   and it's fine. And we can still use that chip, we don't have to toss it in the garbage.

00:05:16   And so what they've done here basically is, it used to be that. It used to be

00:05:20   a chip that had eight physical cores on it for the GPU, which takes up a lot of

00:05:24   chip real estate. One of them was just disabled. And which one

00:05:28   didn't really matter. So if there happened to be a flaw in that area of the chip

00:05:32   they could disable that one and still use the chip and still ship it and

00:05:36   make their money. And so all that has changed with the A12X versus Z

00:05:40   appears to be that they're now shipping all eight cores enabled. So they've gotten better

00:05:44   enough at this manufacturing process or it's gotten cheaper or whatever it is

00:05:48   the economics are working out better so that now they don't have to

00:05:52   leave that reserve where they don't have to increase the yield

00:05:56   artificially by having this one disabled area that they can move around as needed. They can just

00:06:00   ship only the ones that have all eight that are flawless.

00:06:04   So they saved all the ones that had all eight working. I don't know what the economics of that is

00:06:08   but these are small items. If this was a premeditated strategy

00:06:12   and they really had awful yields they could have said everyone with eight cores is working, just put that over there

00:06:16   we're going to use them later. That seems less likely than what you described but it is

00:06:20   a possibility. Yeah, definitely. And then the other thing I wanted to touch on while we're talking about this because it's

00:06:24   a very similar thing is the idea of binning. A lot of people don't

00:06:28   know that when you get a CPU that has the same CPU

00:06:32   available in three different clock speeds. That's the same CPU

00:06:36   they're not making, like, okay now we're going to make the 2.5

00:06:40   GHz ones and then next week we're going to make the 2.2 GHz ones

00:06:44   they're making one chip and certain ones

00:06:48   will have whatever flaws or whatever better or worse

00:06:52   outcomes of their manufacturing, certain ones will be able to run at higher clock speeds

00:06:56   without having flaws. And so they basically test them

00:07:00   and they bin them, they sort them into like, this one passed the highest test

00:07:04   so we're going to sell this one as the highest clock speed. The ones that

00:07:08   couldn't quite run at that speed without errors or flaws but could run at a lower clock speed

00:07:12   will bin those down here, will set them by whatever method

00:07:16   like burning in certain things and whatever, will fix them to this one clock speed

00:07:20   that they tested okay at and so on. And as you can imagine

00:07:24   there are fewer that can run at the highest clock speeds. So the highest clock speeds are more

00:07:28   expensive because they don't have as many of them. And of all the ones they make, they can make a whole

00:07:32   bunch more that run at the lower clock speeds than the few that run at the highest clock speeds.

00:07:36   So that's, if you ever hear the term binning, that's what that's about. And it's all

00:07:40   between binning and this weird partially disabled chip thing

00:07:44   it's all about improving yields when you're making these chips and it's directly

00:07:48   related to cost and feasibility and everything else.

00:07:52   And game consoles is one area where obviously it's very price sensitive, you're trying to

00:07:56   sell what is essentially an entire gaming system for less than the cost of a really good PC

00:08:00   video card. So in the past several generations it's been

00:08:04   common practice to plan ahead and just assume that you're never going to get perfect

00:08:08   chips for your game console and just assume that one or more cores are going to be bad

00:08:12   and GPU cores, there's usually more of them and they're more sort of like

00:08:16   regular and equivalent to each other, like GPUs in general are more, if you look at them like

00:08:20   in the wafer view they have a more regular pattern, it's just repeats

00:08:24   of the same block over and over and over again. So it's the perfect, A, it's the

00:08:28   area that usually takes up most of the square millimeters on the chip and B, it's the

00:08:32   perfect place to say, okay, well we're going to allow one or two of these things to be duds and we're

00:08:36   still going to ship it because that's the only way we're going to be able to sell people this thing for $400

00:08:40   taking a minimal loss on it or whatever.

00:08:44   All right, and we also have learned in the last week that

00:08:48   all 2020 iPad Pros have 6 gigs of RAM. So on the 2018s

00:08:52   which is what I have and I think one or both of you guys have, those

00:08:56   only the 1 terabyte models had 6 gigs of RAM for reasons

00:09:00   that I don't think were ever clearly explained, but one way or another, apparently in the

00:09:04   2020 iPad Pros all of them have 6 gigs of RAM, which is a small but

00:09:08   potentially significant difference and if you're really scratching at the bottom of the barrel to

00:09:12   find a recent upgrade, Marco, then maybe that could be your get a

00:09:16   jail free card. I forget the supposed U1 chip that's in there as well.

00:09:20   I've heard since that people are saying no, that's not there after all.

00:09:24   I am so happy that these iPads are really

00:09:28   good and they're the best iPads we've ever made and all that stuff. I have no interest in this

00:09:32   update and that's fine. I use my 11

00:09:36   inch 2018 model every day around the house and

00:09:40   it's wonderful and there's nothing wrong with it and the one thing that we all

00:09:44   want to play with, which is the new iPad Magic Keyboard, isn't out yet

00:09:48   and the reviewers don't have it yet and so, and it's compatible

00:09:52   with mine, with the 2018 models, which is awesome.

00:09:56   I don't know yet, I know that Gruber was talking about this in the talk show, I don't know

00:10:00   yet if the magnet arrangement might have been improved on the new one

00:10:04   like maybe the Magic Keyboard attached us to the old one but it's not as secure

00:10:08   or it's not as good or it's more cumbersome in some way. I don't know, but it appears

00:10:12   that the 2020 iPad is almost identical

00:10:16   to the 2018 iPad and iPad Pro specifically

00:10:20   and that's wonderful because on one level it's

00:10:24   a little concerning in the sense that wow, they had almost two years

00:10:28   and this is all they really changed and I'm

00:10:32   minimizing this. In the world of AR, I think the LIDAR sensor is actually a really big deal

00:10:36   because it makes it so much faster to pair or to

00:10:40   establish the AR room arrangement and it makes it more stable and everything else.

00:10:44   That's wonderful. That's a feature I've never once used my iPad to do

00:10:48   so it's not going to be a thing that affects my life at all

00:10:52   and so the fact that there's literally like nothing else that's different except a RAM

00:10:56   upgrade which I also won't use because I don't multitask almost ever on my iPad.

00:11:00   If you had an earlier iPad model and you want what was great about

00:11:04   the 2018, great, this is a good time to upgrade. There's a slight improvement. Otherwise

00:11:08   though, I see no reason and that's totally fine. Man, what's wrong with me?

00:11:12   I don't have the new iPad and don't want it. I don't have a Mac Pro and

00:11:16   don't want it. Am I okay? I'm not so sure. I'm getting worried.

00:11:20   I mean you're going to get that keyboard when it comes out so don't worry. You'll be able to buy something soon.

00:11:24   Thinking about this update though,

00:11:28   last show I think we characterized it as a speed bump and it is as we've described but

00:11:32   it makes you think

00:11:36   Apple has been content to not update the iPads for a long time

00:11:40   like with big gaps between them and it never seemed like they were in a rush

00:11:44   to get out of speed bump. So I'm thinking like why

00:11:48   this time? Why be in a hurry to put any change?

00:11:52   Why not just make us wait until the A14X sporting

00:11:56   the next big jump? Could be that the next big jump

00:12:00   is not coming for a while. It could be

00:12:04   that they always wanted to have a big jump now and they couldn't hit that date so they wanted to

00:12:08   get something out and they had a bunch of these fast A12s hanging around but this is actually the new

00:12:12   camera system from the one that will have the A14 in it. I don't know

00:12:16   but it is actually a little bit concerning to me. Despite the fact that this is a good

00:12:20   iPad and if you want an iPad you should totally get this one. It's really good. But if you've got the previous one

00:12:24   it's not that compelling and it does make me wonder what's happening.

00:12:28   I forget who was sending around the rumors. The internal part numbers of

00:12:32   the supposed next generation chip. I think it was the A14. They have

00:12:36   all these internal part numbers and they've gone through the sequence where they've been increasing

00:12:40   the number by one with each thing and supposedly

00:12:44   again these are all just rumors. Supposedly the next major chip

00:12:48   does not increase the number by one of the previous internal

00:12:52   codename model part number thing. It's just an entirely different part number. Like it starts over

00:12:56   it's like 101 again where they're up to 108 or 9 or something like that.

00:13:00   And that's making people think wow this is like a big

00:13:04   change. This next chip is not just an iteration of the previous one

00:13:08   or maybe it's the big chip that's going to power all the ARM powered Macs and

00:13:12   all sorts of stuff like that. If that ends up being in remotely true it makes me

00:13:16   think that this model is a stop gap because the next iPad is not going

00:13:20   to be out anytime soon because the next iPad is a big leap because it uses

00:13:24   again speculating on these rumors. Maybe it uses a totally new

00:13:28   chip that's suitable for use in Mac laptops and as well as

00:13:32   iPads and it's a new generation and yada yada yada. And

00:13:36   that could be coming out according to the ARM rumors now. What are they saying like 2021 or something?

00:13:40   And who knows if that will be delayed by all the virus stuff.

00:13:44   So if they waited until 2021 to update

00:13:48   the iPad people would be getting a little bit cranky.

00:13:52   So that makes this A12Z make more sense. They need something

00:13:56   they couldn't wait that long because it would be too long. So give them

00:14:00   a better iPad now because we know the next one is not coming along anytime soon.

00:14:04   Yeah I think that's an important point that in years past

00:14:08   when there were multiple years between iPad

00:14:12   updates and this was also exacerbated by the fact that I don't think they were really touching software either

00:14:16   but nevertheless Apple would go a couple of years between iPad updates of memory

00:14:20   serves and everyone would be like "Dudes! What's going on?"

00:14:24   Like can we get a little love for the iPad somewhere, someway, somehow? And yeah this

00:14:28   is not a very interesting update in terms of the iPad itself although I am so

00:14:32   amped for that keyboard I can't even tell you. But I think the alternative

00:14:36   is all of us talking heads start raking Apple over the coals for not even

00:14:40   doing a speed bump. So yeah the speed bump may not be the most exciting thing in the world but I do think

00:14:44   it's important to recognize that they're giving us what we want. They're showing us that

00:14:48   yeah even if they don't have something magical that changes our lives to offer

00:14:52   they're still doing something. Admittedly this is very little something but

00:14:56   I will take something, some sign that something is happening rather than

00:15:00   having the iPad somehow befall the Mac mini timeline

00:15:04   where it doesn't get updated like you said John for two or three years. So I'm glad that

00:15:08   Apple is doing something even if the something they're doing is a little bit of a snooze to me.

00:15:12   You want the speed bumps to be like

00:15:16   sooner. Speed bumps are good, it's better than nothing but you'd like to see

00:15:20   a big release and then the next one would be a speed bump and then a big release. Some kind of iteration

00:15:24   like that whereas this was the normal gap that we would get for a big release and then instead

00:15:28   we got a speed bump. That said the last time they did a big update it was the change

00:15:32   to these flat sided iPads which everybody loves and they're great right so

00:15:36   they deliver when the big ones come down the line. By the way someone in the chat

00:15:40   found that thread that I was thinking of, Steve Trout and Smith, these are the

00:15:44   are they code numbers? Part names or code numbers? Anyway

00:15:48   A10 was the T-8010, A11 was

00:15:52   the T-8015 so I guess they didn't go by ones. Anyway

00:15:56   the A12 was the T-82X and the A13

00:16:00   was the T-83X and the next one is

00:16:04   T-8101. So it's the T-80, forget about the T-80s, this is the

00:16:08   first in the T-81 line. Does that mean anything?

00:16:12   Who knows? Like just, you know, it's all just tea leaf reading

00:16:16   from rumors and so on and so forth but it is

00:16:20   excitement is building in the world of ARM

00:16:24   on the Mac and apparently on the iPad too because you would imagine that whatever

00:16:28   the work they're doing to make CPUs for laptops it's

00:16:32   basically 100% overlap with the iPads which are already faster than their laptops in

00:16:36   most regards so fingers crossed for 2021.

00:16:40   We are sponsored this week by Squarespace. Start building your website

00:16:44   today at squarespace.com/ATP and enter offer code

00:16:48   TPA check out to get 10% off. Make your next move with a beautiful website

00:16:52   from Squarespace. Squarespace makes it so easy to make

00:16:56   a website and your website with Squarespace can look incredibly nice

00:17:00   with pretty much no effort from you and you can customize it to your heart's content

00:17:04   you can leave things the way they are and you have a great site right there or you can customize

00:17:08   it, you can move stuff around, you can add and remove stuff, you can change the fonts and

00:17:12   the colors and everything else to make it really look like yours and your site and your

00:17:16   brand and whatever else you might need it to look like. And you have amazing functionality

00:17:20   built in with Squarespace. Obviously the basics, you can make a website with

00:17:24   all these kind of content pages, blogs and everything else, portfolios, you can also do fancier

00:17:28   stuff. You can do things like host podcasts there, you can host storefronts

00:17:32   you can have advanced galleries and event calendars and all sorts of dynamic functionality

00:17:36   all built in all with Squarespace and all of this is done with

00:17:40   easy to use visual tools. You don't need to be a developer or

00:17:44   a designer, there is no coding required, you can do this no matter what your skill level

00:17:48   is and if you sign up for a whole year up front you can get a free domain

00:17:52   name with that as well. All of this is backed by incredible support

00:17:56   and you don't need to worry about things like upgrades or patches or

00:18:00   server uptime, they take care of all of that for you at Squarespace.

00:18:04   See for yourself by starting a free trial, whatever you need to build, try it there

00:18:08   first, give it like an hour, I'm telling you, you're going to be amazed what you can get done and

00:18:12   how great the result will look and work. See for yourself at squarespace.com/ATP

00:18:16   to start that free trial and to really test it out for yourself

00:18:20   really build something with it. When you decide to sign up after that, and I know you will, make sure

00:18:24   to go back there to squarespace.com/ATP and use offer code ATP

00:18:28   to get 10% off your first purchase. That's squarespace.com/ATP

00:18:32   code ATP for 10% off your first purchase. Thank you

00:18:36   so much to Squarespace for sponsoring our show, make your next move with Squarespace.

00:18:40   [Music]

00:18:44   May I tell you gentlemen a story? Is it a happy story?

00:18:48   Is it a story where awesome things happen and you're smiling

00:18:52   throughout and it's just joy? No. That's the best

00:18:56   kind. This kind of story still hasn't ended

00:19:00   and it's been almost a week. So Casey, what computer are you using?

00:19:04   [Laughter] Well done. I wish I could

00:19:08   tell you that was a real deep cut, but it's not. No, I'm still on the iMac Pro.

00:19:12   Oh thank god. The iMac Pro seems

00:19:16   to be fine. My phone seems to be fine, although I will say

00:19:20   that there is a very frustrating, I'm going to say gouge,

00:19:24   but that's way over selling it. Like nick in the screen right where I

00:19:28   scroll that's driving me batty. This is by the way the one that I

00:19:32   paid $100 to get. You know, the refurb that I got after I shattered the back

00:19:36   like a moron. But anyways, that's also not the problem here. So how did you

00:19:40   gouge the screen by the way, side mark? I genuinely don't have the famous idea.

00:19:44   Obviously I did something. How do you not know when you put a...

00:19:48   I'd sound like I put the phone in the same... You put it in a pocket with your keys? I mean what's going on?

00:19:52   I was just about to say it's not like I put my phone in the same pocket as my keys. We're missing the point.

00:19:56   We're missing the point. It doesn't matter. It doesn't matter. Remind me never to let Casey use my phone.

00:20:00   [Laughter] He doesn't remember how he put a gouge in the screen. It's so big that it's

00:20:04   annoying his thumb. Yeah, that's very true. Alright, so I gotta tell you

00:20:08   a tale of woe about my sonology.

00:20:12   Oh no! Things are not good

00:20:16   at the Liz household right now. Didn't hear me telling you that you should basically kill it

00:20:20   and get upset? Alright, let me set some ground rules

00:20:24   for you guys and the listeners. The ground rules are

00:20:28   I am not interested in ways of getting rid of the sonology. We've covered

00:20:32   that ad nauseam. I'm not interested. Things I should have or

00:20:36   could have done differently, don't want to know. It's too late. Doesn't matter.

00:20:40   So don't even bother. Don't care.

00:20:44   This gets real bad real quick. You're gonna hear about things you could have done differently.

00:20:48   I know you're trying to set down rules, but you're gonna... I mean from us.

00:20:52   Oh no! You guys all allow it. What do you think this show is? Exactly.

00:20:56   Alright, so let me set the stage. In 2013, some very nice

00:21:00   individuals at Sonology sent all three of us DS1813+

00:21:04   That is the particular model name of our sonologies

00:21:08   and it basically means that they were eight

00:21:12   bay sonologies in the, I believe, small business line

00:21:16   from 2013. And at the time they sent us

00:21:20   they were filled with eight

00:21:24   three terabyte, what are they, Seagate Red's, I think, hard drives?

00:21:28   Western Digital Red. Sorry, yes, what you said. I apologize.

00:21:32   In the last seven-ish years, I don't recall exactly when

00:21:36   in 2013 we got them, I think I've had to replace

00:21:40   one, maybe two drives. The way I have set up

00:21:44   my sonology, which I think was Marco's idea, maybe it was, it must have been John's, because I always

00:21:48   think it's Marco and it's always John. Somebody told me, "Hey, take two drives, make them time

00:21:52   machine, take the other six and make them one humongous volume for everything else." This time this is actually

00:21:56   me. Okay, see, there you go, I actually remembered something. Look at me go.

00:22:00   So, anyway, so drives, not drives, I'm gonna go

00:22:04   one based, so one and two, our time machine, three, four, five, six,

00:22:08   seven, eight are all lumped into one big volume using

00:22:12   sonology hybrid rate, or I think it's hybrid rate, SHR, which basically

00:22:16   means I can lose one of them and everything's fine. And

00:22:20   I noticed that all of these drives,

00:22:24   four, five, six, and eight were all OGs from 2013.

00:22:28   So these are seven-year-old hard drives that have been running non-stop

00:22:32   for seven years. I just want to interject here that

00:22:36   I am using all of those same drives, those eight

00:22:40   three terabyte drives that I got seven years ago, all working perfectly.

00:22:44   Well, I hope so, and I hope you have some wood to knock on.

00:22:48   I'm just saying, I'm not gonna blame it on the fact that you put it in

00:22:52   the same room with you, but that's probably why. Well, yeah, yours is in the basement

00:22:56   or something, isn't it, right? Yeah, climate control, dark,

00:23:00   nobody disturbs it. Mine is in my un-temperature controlled

00:23:04   garage, and it's fine, I've never lost a drive. I cannot believe that.

00:23:08   I cannot believe that yours is still working, but anyways, so be that as it may,

00:23:12   I have four, five, six, and eight that are all the originals from 2013.

00:23:16   Three and seven have been replaced with

00:23:20   ten terabyte Western Digital Reds. I'm slowly going

00:23:24   to replace all these, put these ten terabyte drives in, and again

00:23:28   four, five, six, eight. I would like to replace one of these

00:23:32   in order to try to get ahead of all of these eventually failing.

00:23:36   This is the first part where we'll talk about what Casey could have done differently. What you're

00:23:40   doing now is taking a thing that ain't broke and you're trying to fix it.

00:23:44   Yes. Right? Like you did not have any bad drives, but you decided preemptively

00:23:48   I'm going to take my working Synology and I'm going

00:23:52   to endeavor to upgrade it to stave off

00:23:56   what I think is the imminent failure of my seven year old drives.

00:24:00   Right. So, why do you think I would have made that choice?

00:24:04   Do you get the little emails from Synology that tells you how many bad

00:24:08   sectors it's finding on your disks? Do you get those? Mm-hmm. Have those numbers been going up

00:24:12   lately? No. Are any of those numbers non-zero? I don't believe they were

00:24:16   non-zero now. So, in theory, just to be clear,

00:24:20   in theory drives four, five, six, and eight were all fine.

00:24:24   In theory. And by the way, I know we all talk about our Synology love, but the thing I just described

00:24:28   is a real thing. The Synology has a feature where it will email you weekly

00:24:32   or monthly or whatever updates on your drive health and it will tell you if it has found any bad

00:24:36   sectors. And if you get those emails and it starts saying that the bad sector count is increasing,

00:24:40   your drive's probably going bad and you should do something about it. That happened to me on my

00:24:44   other Synology, which, it says bad sector's found or whatever, and it does a consistency check.

00:24:48   My other Synology, which does not have Western Digital Reds, one of my drives

00:24:52   said I found one bad sector and the next week it found two and the next week it found like

00:24:56   37 and I was like, "Okay! Time to replace that sucker." So,

00:25:00   another cool feature of this NAS. Yeah, it also tells you

00:25:04   how many smart, you know, like the acronym SMART, how many smart errors

00:25:08   and disconnects there have been. Generally speaking, when it comes to this kind of thing

00:25:12   from a hard drive, it's kind of like, you know, I was going down a hill with my car

00:25:16   and I pressed the brakes and once they didn't work. Anything besides

00:25:20   zero on these numbers means replace it immediately.

00:25:24   Well, that's the Marco approach, but let me give you another story.

00:25:28   One of my drives in my Synology has had one bad sector

00:25:32   since like the first week I got it. That number is always one. It's been one for seven years.

00:25:36   Like, you're right that there is usually a cascade, but drives have bad sectors

00:25:40   and usually they're found and mapped out before you get the drive, but it is conceivable

00:25:44   that you get one out of the box and it finds a bad sector and it maps it out.

00:25:48   And if that number never goes up for seven years, like, I mean it may go up someday, right?

00:25:52   But, you know, if you see one and it doesn't change for months and months,

00:25:56   fine. But you do have to watch them. Like, if it shows from one to two to five to thirty-seven

00:26:00   to a hundred and forty-five, like, replace the drive. I don't think I've ever had a bad

00:26:04   sector on a hard drive that didn't then die.

00:26:08   I've got seven years with one bad sector. One of my drives did. I just got an email

00:26:12   the other day. It's still there. It's not going to go away. It's not going to go back to zero.

00:26:16   So why would I try to do this preemptively? Why would I do that?

00:26:20   Because you're home all day and you've got nothing to do and you're going to break your working computer.

00:26:24   Go ahead. That's part of it. Maybe it's because you were worried that, you know,

00:26:28   because it's kind of like, so when you have to replace a drive in a RAID

00:26:32   array, and yes, SHR is not exactly RAID. It's kind of like what Drobos do.

00:26:36   It's like kind of software expandable kind of thing. But, you know, it's close enough to RAID.

00:26:40   So when you have to do a RAID rebuild, one of the common problems is

00:26:44   like, a RAID rebuild requires a lot of disk activity

00:26:48   because you have to read the entire rest of every other disk

00:26:52   to write what is necessary onto the replacement disk. So actually

00:26:56   one thing that can happen is kind of like the, you know, dead cat box thing of like

00:27:00   you might think everything's fine. Shringer's cat? Is that what you're trying to say?

00:27:04   Dead cat box is not what it's called. Go ahead.

00:27:08   But like, so you might think everything's fine, but then the second you replace

00:27:12   a disk, it has to way more

00:27:16   than usual stress the other disks that were in the system

00:27:20   to rebuild the array onto the new disk. So you actually kind of risk a bad

00:27:24   cascade, which I hope this isn't what happened to you, where like you might think everything's fine

00:27:28   replace one disk and then another disk or two

00:27:32   die during the rebuild because it's stressing them out so much.

00:27:36   So, on Thursday, I say to myself, I've got 4, 5,

00:27:40   6, and 8 that I'd like to replace. I'm going to take 5

00:27:44   because it is in the middle of, physically in the middle of 4 and 6.

00:27:48   Sure, let's go with that one. Completely arbitrary seems like a reasonable approach, right? Why not?

00:27:52   I replace drive 5, I tell the Synology, okay rebuild yourself, I go to sleep.

00:27:56   I wake up on Friday, the 27th of March

00:28:00   and the volume has crashed, which is Synology speak for you have lost

00:28:04   everything. Drive 8, during the restore

00:28:08   had one or more IO errors and died.

00:28:12   So, I replaced drive 5 in the process of

00:28:16   fixing everything and putting everything back to where it was or where it should be

00:28:20   drive 8 sh*ts the bed. Well, now I've got problems.

00:28:24   So... No wait, quick question. Can you just put

00:28:28   the old drive 5 back in? That's a good question

00:28:32   but John, I'd like to hear your thoughts before I answer it.

00:28:36   This is not shocking and what Marco suggested is definitely what

00:28:40   I would have tried first, but in the list of things you could do

00:28:44   differently in general because if you're worried

00:28:48   about your drives because, you know, whatever, I forget what the word is, the old raid

00:28:52   word used to be called resilvering, I forget what the hell they call it, but anyway, because doing what you're doing, replacing

00:28:56   a drive is so taxing on the other drives. If the reason you're doing it is you're worried

00:29:00   about all the drives in your thing, in general it's safer to try to

00:29:04   do a one time copy of everything off to another volume

00:29:08   like, you know, if you insisted on doing this, if you're in a situation where

00:29:12   you're like, I feel bad, I feel like these drives are going to go at any minute, what

00:29:16   can I do? If you don't already have a complete backup, which obviously you should, but if you don't

00:29:20   already have a complete backup, what you'd want to do is get something

00:29:24   that can receive all the data and then copy all the data off all the drives and put it onto this

00:29:28   new place. Then, after you've successfully done that and you take that safe copy

00:29:32   of all your data and you put it somewhere and you unplug it and you just sit in the corner of the room, then you go try

00:29:36   to do this whole, you know, swap out drives thing, so then when it blows up like this

00:29:40   you're like, oh well, I tried, and now you've got your data in the other place. But anyway,

00:29:44   continue. What did you actually do? Did you try Marco's thing of putting back in the five and

00:29:48   swapping out the eight? Not exactly, and I have reasons. Let me

00:29:52   explain my thought process. You may not agree, but I have a thought process here.

00:29:56   So drive eight dies, I walk in and I am on the verge of tears

00:30:00   and that is no exaggeration because everything in theory is gone.

00:30:04   Now I do have backup, I know I have a couple of backups

00:30:08   of my photos, that is not up for grabs, and that is the one thing that I

00:30:12   cannot lose. I would be, and in this moment in the story,

00:30:16   I am devastated that I've lost other things. But

00:30:20   I would be beyond repair as a human being

00:30:24   if I lost all my pictures. Did you really lose other things? I thought you had

00:30:28   didn't you, did you have online backup? Did you have like your parents

00:30:32   backup? Like what was your backup situation? Gentlemen, this is my story, I will tell it in my own time.

00:30:36   Oh no. So you were devastated, we're up to the point

00:30:40   where you're devastated because you think you've lost stuff. That means at this point in the story, you think

00:30:44   you don't have some of that data in another location, right? I don't

00:30:48   I am not confident that everything is the way I want it

00:30:52   to be in terms of backups. So let me take you through my situation. I know

00:30:56   I have my photos on an external hard drive that normally is at mom and dad's

00:31:00   but because of the quarantine is actually sitting in the garage. But I know

00:31:04   they're there, I have confidence that I'm okay.

00:31:08   That doesn't happen to be the drive you put into Bay 5, is it? No, no, no, no, no, no.

00:31:12   This is an external, this is an external sitting off to the side.

00:31:16   And I don't think I've updated it in a month because I usually do that at the top

00:31:20   of the month. But I mean, I don't, maybe I lost a week or two of photos.

00:31:24   And even if that's the case, like okay. It's not desirable but fine.

00:31:28   I would be okay with that. I still have the last 20 years of photos, whatever it's been.

00:31:32   But everything else, I'm not sure.

00:31:36   So I've been awake for 30 minutes at this point. It's Friday morning.

00:31:40   I walk into my office and I go, "Oh God, oh God, oh God, oh God, oh God, oh God."

00:31:44   And I look at Backblaze, which I've been running on my iMac and I know hadn't quite finished

00:31:48   uploading everything but I thought it was really close. And it turns out

00:31:52   Backblaze is not as close to done as I thought for my own, because of dumb

00:31:56   things I've done that I'm not going to get into right now. So okay, next thing.

00:32:00   I need to interject again. You're in the middle of running a gigantic Backblaze backup, right?

00:32:04   Yep. This is a good time to rebuild my array. You have the itch, you have the itch

00:32:08   to do this disk thing. Why didn't you wait for the Backblaze thing to finish? I agree.

00:32:12   No argument. That absolutely was a mistake on my part. Absolutely a mistake

00:32:16   on my part. No argument. There's nothing I can say to defend myself. This is like one of the situations where, you know, when they have

00:32:20   like an analysis of catastrophes, it's never like one thing went wrong. It has to

00:32:24   if you do one thing wrong, like usually the system or whatever recovers

00:32:28   but it has to be a series of small errors that build on each other. You know what I mean?

00:32:32   Mm-hmm. All right. Go on. All right. So crash plan, I

00:32:36   haven't run in at least a month, maybe two months. And I

00:32:40   have had a reminder for myself to cancel it for at least a week, maybe two weeks.

00:32:44   But I didn't. So. Finally,

00:32:48   procrastination pays off. Exactly. So in an absolute

00:32:52   desperation scenario, I would lose a couple of months worth of other

00:32:56   things. Again, photos, I'm confident I'm okay. But everything else, I'd lose a

00:33:00   couple months of that, but I'd be okay. Now, restoring from crash plan would be a friggin' nightmare

00:33:04   because they don't do that awesome thing that Backblaze does where they'll just send you a physical hard drive

00:33:08   and then you can send it back and no harm, no foul. I would have to actually download

00:33:12   everything from crash plan, which would stink even on a gigabit connection. But

00:33:16   I could get it. That's all that matters. I could get it. So

00:33:20   I know my pictures are safe, but at this point I think all of these

00:33:24   TV shows that I've acquired by gray means, like

00:33:28   using YouTube DL to download stuff off of NBC's website, for example.

00:33:32   I'm not signing into anything to get it. It's available for anyone. You can do it on

00:33:36   any one of you could do it, but it's work I've put in to download all these shows and sometimes they take

00:33:40   them off the internet and I might want to watch them again sometime. And another great example

00:33:44   is concerts. I really enjoy watching concerts with Aaron. And

00:33:48   I will take the time to use an app called Subler to like go through and add chapters

00:33:52   for all the different songs so I can skip around if I want to. You can think that's crazy.

00:33:56   That's fine. This is me. This is the way I like to do things. All that stuff.

00:34:00   Gone. So I think, "Oh God, oh God, oh God, what do I do?"

00:34:04   The thought did occur to me to turn the

00:34:08   Synology back off, put Drive 5 back in, and just

00:34:12   go from there. Turn it on whistling.

00:34:16   Right, exactly. Nothing to see here.

00:34:20   So anyway, I didn't do that. And before you jump all over me, let me tell you why.

00:34:24   What I thought was, "Okay, the first step is let's restart this thing

00:34:28   as is and just see what the state of the world is."

00:34:32   So I left Drive 5 with this new 10 terabyte drive, and

00:34:36   I reboot it. And it comes up and it says,

00:34:40   "Things are in..." I forget the term for it on the Synology. It's degraded.

00:34:44   That's it. Things are degraded, but they're there.

00:34:48   Drive 8's hiccup is just a hiccup.

00:34:52   Everything is still there. Now there is no Drive 5, according to Synology, because it's empty.

00:34:56   Like that makes perfect sense. But it's there. And at this point I have

00:35:00   to make a choice. I'm not saying I made the right choice, but what I decided was,

00:35:04   "Okay, at this moment in time, Drive 8

00:35:08   still works." And it has already failed once.

00:35:12   I don't know if breathing on this thing

00:35:16   is going to cause it to fail again. So the very first order of

00:35:20   operations, as far as I'm concerned, is get all this data off this

00:35:24   thing as quickly as I possibly can. And that's

00:35:28   what I did. So what I did, and I will give you a chance to respond.

00:35:32   Hold on, wait, wait, wait. Just hold on. Hold on. Just give me a chance.

00:35:36   Just give me a chance. You have to make it through the gauntlet of telling us this story.

00:35:40   At various points I haven't told you the different things that I would have done.

00:35:44   No, not yet. Just give me 30 seconds. I have the talking stick.

00:35:48   So I decide to... Now, mind you, we are in big time

00:35:52   lockdown quarantine mode. We are the worst friends. I look at Amazon to see

00:35:56   what can I do? Is there a drive big enough that I can prime now

00:36:00   on Amazon to get all this stuff off the Synology? The answer is no.

00:36:04   But Best Buy is delivering to the

00:36:08   parking lot of Best Buy, and they happen to have

00:36:12   literally one 12 terabyte external available for purchase.

00:36:16   And so that's what I did. And then I connected it to my Mac Mini

00:36:20   and started r-syncing the entirety of my Synology to this

00:36:24   external drive. And that is where I will pause and allow you to beat the hell out of me.

00:36:28   So I understand your thing. You want to get the data off, right?

00:36:32   It comes back up. Drive 5 is empty. Drive 8, you're scared, is going to break.

00:36:36   Again, why didn't you take out the empty drive 5 and put back the full drive 5?

00:36:40   So that way, if 8 breaks, you still have all your data.

00:36:44   I 100% agree with you. And in retrospect, maybe I should have, but at this

00:36:48   point, I was so nervous that touching anything

00:36:52   would screw it up that I knew at this one moment,

00:36:56   at this one moment, everything was working. So I have to capitalize on it

00:37:00   while I still can. But you didn't know that. Half your data on Drive 8 could be corrupt.

00:37:04   You have no idea what's going on with Drive 8. You know it had some kind of I/O error. You're assuming you can get

00:37:08   all the data off it. Anyway, that's what I would have done differently at that point. As soon as, if it came back up

00:37:12   and if you wanted to get your data off of it, which I suggested earlier would have been the first step,

00:37:16   but fine, you're going to do it now, I would have put the 5 back in. So that way the 8

00:37:20   could die and I'd still have all my data. Well, one minor complicating factor here, if the array

00:37:24   was online at the time, it could have been written to in the meantime.

00:37:28   And so we will come to a point where I will answer this question and I think

00:37:32   that is exactly what happened. And if that is the case, then your old Drive

00:37:36   5 is now out of date and is basically corrupt. Yep.

00:37:40   Yeah, there was no way to, I mean, what's being

00:37:44   written to it? Do you have like jobs running in the background pulling down things from feeds?

00:37:48   I honestly don't think so, I don't know, but. Well, and regardless of like

00:37:52   no matter what it is, even if it's one byte somewhere, if the RAID controller

00:37:56   like keeps track of this, which it should, then it will say, "Sorry,

00:38:00   this disk, this old disk 5 is now out of date, we can't use it." And so

00:38:04   I will cut to the almost end of the story very quickly and say once

00:38:08   I got all 12 terabytes off of that, or well it was like a 10 and a half

00:38:12   terabytes off the Synology, onto the external, which seems as far as I can tell

00:38:16   to have gone just fine. I did at that point, mind you this is

00:38:20   admittedly literally four or five days later, I did plug in Drive 5

00:38:24   because at this point I thought, I don't want to lose everything, but if I did

00:38:28   I now have a complete backup so it's okay. And I did

00:38:32   try plugging in Drive 5 and it absolutely said this drive is useless to me, do you

00:38:36   want to, you know, bring it back up as though it's new? Well, that was five days later, I

00:38:40   can imagine stuff would have happened in five days. Agreed, agreed, but

00:38:44   again, my thought process was, the only thing I care

00:38:48   about is getting this data off this box, that's all I care about. And as

00:38:52   long as I can do that, anything else is secondary. So

00:38:56   I am in this degraded state, I ordered this 12 terabyte from

00:39:00   Best Buy, and within a couple hours I have it, within an hour of that it's

00:39:04   hooked up to the Mac Mini, the Mac Minis are syncing literally 2.2 million files

00:39:08   and something like 10 and a half terabytes to this 12 terabyte drive.

00:39:12   So, at the same time, because I'm a moron but I'm really freaking out now,

00:39:16   I'm also manually r-syncing a handful of things that I just really don't want to lose.

00:39:20   So I'm getting a third copy of my photos, a second copy

00:39:24   or third copy, I'm losing count now, of some of the kid shows that

00:39:28   the kids adore that I've spent a long time amassing. I got

00:39:32   another copy of Top Gear, you know, literally 25 seasons of Top Gear that I have

00:39:36   and I do occasionally go back and watch some of all of these concerts and

00:39:40   things like that, and just some other essentials that I really don't want to lose. So that was

00:39:44   started on Friday, this past Tuesday, yesterday,

00:39:48   the 11-ish terabytes and 2.2 million files are complete.

00:39:52   So what do I do? And I already told you, I thought, alright, let me just try throwing in that

00:39:56   old drive 5, obviously it didn't work, I already told you it didn't work, but

00:40:00   let me try it. So now, what do I do?

00:40:04   Well, the only real option I have

00:40:08   other than just nuking it all from orbit and starting anew, like you were saying, John,

00:40:12   is, well, I take that 10 terabyte that's been sitting in drive 5

00:40:16   all this time, and just try again, and hope that 8 holds on

00:40:20   long enough. So, I thought to myself, what can I do to make

00:40:24   this work, or what can I do to improve my odds? And although

00:40:28   this room, outside of when I'm recording and the door is closed, this room

00:40:32   doesn't get very hot, I thought to myself, well, heat is the enemy

00:40:36   of everything, so I got a box fan and sat it on the filing cabinet that's like

00:40:40   2 or 3 feet away from the Synology, and I have been blasting this box fan up until this recording

00:40:44   on the Synology for a day or so

00:40:48   while it's rebuilding itself, hoping, hoping, against all odds, that drive 8

00:40:52   will hold on long enough for it to repair itself. And then, thankfully

00:40:56   I actually have another 10 terabyte that's brand new waiting

00:41:00   just in case one of these dies, and I would put that in drive 8

00:41:04   and then hopefully, what did I say, the other ones were 4, 4 and 6

00:41:08   don't die in the same way as I'm replacing drive 8. So

00:41:12   it takes about a day for my particular Synology to put everything back together

00:41:16   and I'm watching this like I'm watching the school clock

00:41:20   at the end of the day on Friday, I'm watching it like a hawk. And just

00:41:24   a couple hours ago as we record this, I see it tick, well, a few hours

00:41:28   ago I see it tick past 50%, which is I know roughly where it was when drive 8 died.

00:41:32   I see it get past 60, 70, 80, 90, I even took a

00:41:36   screenshot at 99.5% thinking, this is when it's going to die

00:41:40   and that will make for a hell of a great story, but I'm going to be miserable. So I want

00:41:44   to get a screenshot so we can put it in the show notes, so when it dies at 99.5%

00:41:48   at least we'll all get a good laugh out of it. During this process

00:41:52   however, I realized, oh, today is April 1st.

00:41:56   Guess what happens on April 1st, John?

00:42:00   It's time for a smart test on all of my drives.

00:42:04   The monthly, yeah. Indeed.

00:42:08   So while I'm doing this restore, using

00:42:12   a drive that is on the edge of death as far as I'm concerned, the Synology says to itself

00:42:16   you know what would be a great idea right now? Let's do a smart test. And

00:42:20   as we're recording, which is in the evening on Wednesday,

00:42:24   let's see, the smart test has only been completed on five of the eight drives.

00:42:28   The other three are sitting at 90%. So naturally that's slowing

00:42:32   everything down, it's further hammering these drives that are probably on the verge of

00:42:36   catching on fire. And so as I'm watching this tick up, finally

00:42:40   right around bedtime for the kids I reach 100%, I get my email saying

00:42:44   volume two, which is the volume I'm talking about,

00:42:48   the consistency check of storage pool two on disk station

00:42:52   has ended. No abnormality has been found. And at this point I expect to

00:42:56   see the degraded or repairing become

00:43:00   normal, and everything should be good, right? Well, no.

00:43:04   Apparently, for reasons that I don't understand, it has decided to

00:43:08   start its consistency check over at 0%. So

00:43:12   as I sit here right now, it has gone from 0 to 100, back

00:43:16   to repairing, checking parity consistency 2.01%

00:43:20   as I record right now. So maybe in another day it'll

00:43:24   work this time? I don't even know. But I am so freaking

00:43:28   miserable and annoyed and upset, and this is like the most ridiculous

00:43:32   just inconsequential problem in the world, but I'm about to rip all of my hair out

00:43:36   and go crazy. Is the box fan off now? The box fan's

00:43:40   off now. Well, here, I can turn it on for you. That's alright.

00:43:44   How loud would it be? Because you're basically, you could be sacrificing your data to podcast.

00:43:48   There it is. Do you hear it? It's on now.

00:43:52   Your data's fine now, right? Well, the data is fine in the

00:43:56   - I gotta turn this box fan back off before Marco kills me. - Thank you.

00:44:00   The data is hypothetically fine, insofar as the

00:44:04   volume on the Synology is functional at this moment.

00:44:08   Like, it's not happy, but I can pull data off of it. I guess hypothetically

00:44:12   I could put data on it, although I'm trying my darndest not to. It is theoretically functional.

00:44:16   Additionally, that 12 terabyte I got from Best Buy does,

00:44:20   as far as I can tell, have a complete duplicate of everything on the Synology.

00:44:24   So, what file system did you use? APFS.

00:44:28   No, I'm not on that. On the Synology. Oh, on the Synology? I don't know.

00:44:32   Does it offer ZFS that would actually make this useful?

00:44:36   As BTRFS and EXT4?

00:44:40   I think it's EXT4. Here's the thing about you wondering about the state

00:44:44   of all your data. Like, oh, I got that copy off under the 12 gigabyte driver. You know

00:44:48   you have drives that are having some kind of problem. If you're using a file system like most file systems

00:44:52   that doesn't do any kind of consistency check of the data,

00:44:56   it's like, I asked the drive to read it, these are the bits that came off the disk, here you go.

00:45:00   And you successfully copied those bits to another place, and another place dutifully stored them.

00:45:04   Are those the right bits? That's why file systems

00:45:08   with data integrity checks like ZFS are handy, because it can tell when

00:45:12   the data right off the disk is not the data that was originally written there, and there are other applications

00:45:16   that can store checksums off to the side, so on and so forth. EXT4, which I think

00:45:20   was the default when we all got our Synologies, does not have any features like that.

00:45:24   So, if your data went bad, I mean, the good thing is with media files,

00:45:28   so you got a few bad blocks here and there, you'll see a weird glitch, or maybe you won't even

00:45:32   see a weird glitch in the video, because it's a lossy compressed thing, and it can tolerate

00:45:36   errors, and it's not like it's an executable program where it might

00:45:40   not even launch if the wrong part of it is corrupt. So for media files, it's probably not

00:45:44   actually that bad, but not knowing, you know, and propagating

00:45:48   bitrod is always a problem, which is why it's better to do these things

00:45:52   before things go wrong, or to go whole hog and have some

00:45:56   kind of checksumming system. Checksumming is more reasonable when you have a small

00:46:00   number of very large files, but it sounds like you have a large number

00:46:04   of large files, like 2.2 million files is a lot. It's nothing compared, like

00:46:08   I have way more than that on my boot drive on my Mac, but they're all tiny.

00:46:12   Media files tend to be big, which is usually pretty good for dealing with files, because big files

00:46:16   means long sequential reads, means less metadata to shuffle around,

00:46:20   and if you're going to store a bunch of checksums offline using some other third-party program,

00:46:24   the fewer checksums you have to store, the better. But it sounds like you didn't have

00:46:28   any of that stuff there, so you're just kind of crossing

00:46:32   your fingers, hoping that nothing was corrupt, because otherwise

00:46:36   you just successfully copied the corrupt data to your Best Buy drive.

00:46:40   Indeed, but I don't feel like I have a whole lot of other choices. So,

00:46:44   the funny thing about all this is, in the end, after going around

00:46:48   and around for like two or three episodes with you two about what I should do to back up the Synology,

00:46:52   the future, once I get the Synology itself squared away, the future

00:46:56   backup approach is going to be hook up that Best Buy hard

00:47:00   drive to the Mac Mini, and have Backblaze back that up. You just did that, you just

00:47:04   did the backup, right? Well, in theory. Well, yeah, but not to Backblaze. The Backblaze one

00:47:08   is still in some sort of limbo for uninteresting reasons that I'm not going to go into

00:47:12   right now. But as soon as I get the Synology to a good place,

00:47:16   even if that means nuking it all from orbit, I will then be permanently

00:47:20   hooking this external up to the Mac Mini and having that go to Backblaze

00:47:24   and just calling it a day. But, gentlemen, it has been an adventure.

00:47:28   Can I suggest some things? If it involves getting rid of the Synology, no.

00:47:32   Well, okay. First of all, we'll get there.

00:47:36   Step one, go get yourself

00:47:40   a second 12 terabyte drive. I don't care if you want to return it

00:47:44   in a few weeks, just get yourself one. First thing you need to do is copy everything

00:47:48   from this 12 terabyte drive to a second 12 terabyte drive.

00:47:52   After you do that, connect it to a computer that will

00:47:56   finish the online backup and back it up. During that process,

00:48:00   so, okay, so copy it onto the second drive, so you have two new

00:48:04   hard drives that have this. The second one,

00:48:08   remove it from the computer and power it down. The Synology,

00:48:12   turn it off and keep it off until you have an online

00:48:16   backup of that entire dataset. Just because

00:48:20   if things are, I know certain times, sometimes powering things down can actually make them die

00:48:24   and actually turn them on, but for the most part that's rare. Usually things wear with usage

00:48:28   and so, for the love of God, freeze this data in

00:48:32   place, like until it is securely backed up somewhere useful.

00:48:36   After that, I want you to consider, you mentioned

00:48:40   recently that you don't use Dropbox or iCloud Photo

00:48:44   Library anymore. The lack of iCloud Photo Library in your

00:48:48   setup has just been made apparent by some of your descriptions of your photo status here.

00:48:52   Why? I know Apple's

00:48:56   not wonderful at certain services. I know that the plans are not

00:49:00   free to get enough storage. I know that

00:49:04   iCloud Photo Library doesn't have all the features of things like Google or whatever, who cares.

00:49:08   But, in this case, this data is so important to you

00:49:12   that I think whatever cost it would be, which, I mean you could probably

00:49:16   get away with what, the $10 a month one probably, right? Whatever cost it would be

00:49:20   for you to have enough storage to have your entire photo library in iCloud Photo Library

00:49:24   is probably worth it, just for considering it as an

00:49:28   automatic off-site backup alone. Not to mention all the other

00:49:32   convenience features of it and the integrations and everything else. But just to have

00:49:36   that be somewhere else that you know that your photos

00:49:40   have one additional layer of backup safety,

00:49:44   whether it's iCloud or Google, I don't care which one you use, but one of those you should

00:49:48   be using. I personally would recommend the iCloud one because I think Google Photos has a bunch of weird

00:49:52   issues with their local uploader apps, but that's up to you.

00:49:56   You know all this too. None of this is news to you, but I think this should

00:50:00   inform that decision that you should probably have

00:50:04   a photo, an online photo backup service in place

00:50:08   because they exist, they're pretty good, even among Apple's reputation

00:50:12   for weird service problems, most of those problems haven't hit

00:50:16   iCloud Photo Library. It's been pretty good.

00:50:20   For me, it's been perfect, honestly, as far as I know. But I haven't even heard

00:50:24   of major issues from other people with iCloud Photo Library. It seems to work very well.

00:50:28   So that is, I think, whatever you

00:50:32   choose to do, your strategy I think should include one of these online photo

00:50:36   services because they're well integrated, they're automatic, and

00:50:40   for something as important as your family photos, having that

00:50:44   additional backup plus one layer

00:50:48   I think is worth it. So moving on from that,

00:50:52   you are running a whole bunch of seven year old drives inside of a seven year old

00:50:56   RAID enclosure. This entire setup is dead to you.

00:51:00   Now, whether you want to replace it with

00:51:04   a better, with like a new Synology, that's

00:51:08   up to you. I wouldn't, as we've talked about, and I don't think you necessarily

00:51:12   need it, but I buy stuff I don't need all the time because I like it.

00:51:16   You know, you only live once, so sometimes you just buy stuff

00:51:20   because you like it. So I won't follow you if you want to

00:51:24   replace the Synology with another Synology. But

00:51:28   one thing people don't always consider when they're thinking about RAID

00:51:32   or RAID-like things is the RAID controller itself as a

00:51:36   potential point of failure. But this happens. RAID controllers die, or flake out,

00:51:40   or have problems. I'm not sure I would trust any

00:51:44   part of my storage infrastructure that was like

00:51:48   critical primary storage that was seven years old. I think at that

00:51:52   point, you're rolling the dice more than necessary. And if

00:51:56   you're going to really count on something that is that old as something that is not

00:52:00   well backed up, you know, I think that is taking

00:52:04   too much risk. So, A, for God's sake, get

00:52:08   iCloud Photo Library. I know it's not perfect. Suck it up. Whatever problems you have with it, suck up

00:52:12   those problems. If I can live in the same world as Dave Matthews, you can get

00:52:16   out of Photo Library. It'll be fine. And so,

00:52:20   A, do that. B, get yourself a second

00:52:24   hard drive and copy all these files onto it and turn off the Synology

00:52:28   because, C, you need to preserve that until the online

00:52:32   backup is done. And D, after that, you should really retire

00:52:36   the Synology. And what you do to replace it, that's up to you.

00:52:40   But you're playing with fire here. There's no evidence that

00:52:44   the Synology hardware other than the hard drives is bad, right?

00:52:48   Agreed. So, I know you're suspicious of it because it's old, but

00:52:52   as someone who just got through using a 10+ year old computer that worked

00:52:56   fine the whole time, hard drives, they're moving parts.

00:53:00   They go bad, for sure. He's got some bad hard drives in there, sounds like, right?

00:53:04   But for a low-powered CPU that's doing the same task

00:53:08   all the time with no GPU to speak up,

00:53:12   in the absence of any evidence that there's anything wrong with this Synology hardware

00:53:16   other than that it's got a bunch of old, crappy drives in it, I don't think he needs to go on and buy a new Synology.

00:53:20   Now, that said, new Synologies are cool, and I've thought about getting one just because they're cool, exactly

00:53:24   for the reasons you said. There's nothing wrong with my Synology. In fact, all my drives are still working.

00:53:28   There's no reason I should get a new one, but I look at the new ones every once in a while because they're cool.

00:53:32   They have better CPUs and fancier file systems and other neat

00:53:36   features. But, at this point, I'm still just waiting for mine to die. But it sounds like

00:53:40   Casey may be in, may have had enough

00:53:44   pain and anguish that as a reward for himself, as a treat,

00:53:48   he could get himself a new Synology. Sounds like he's already getting a bunch of new drives, right?

00:53:52   So once you're doing that, you can get a new Synology too. And then you can use the old Synology

00:53:56   which I believe will be resurrected, and use it to back up the new Synology.

00:54:00   That actually is not a bad idea. In the money-no-object

00:54:04   perfect world, that is what I would do, is I would get a new one

00:54:08   and I keep saying, "Oh, I need a new 8 bay, I need a new 8 bay." But

00:54:12   now that I'm thinking about it more, and listeners had said this to me, I could

00:54:16   just put bigger drives in like a 2 or 4 bay, probably like a

00:54:20   4 bay. I think 4 is the minimum that you should get

00:54:24   and probably the right number. Yeah, we're saying the same thing though. That I could get like a

00:54:28   4 bay Synology, and that would probably be enough. And then I'll put

00:54:32   more smaller drives in the existing ones, stick that at mom and dad's house,

00:54:36   and then I have a live, or nearly live, duplicate

00:54:40   of the one in the house. And I might do that, I don't know, it depends

00:54:44   on how grumpy I am with all this whenever I come out the other end of it, be that if I have to nuke it from

00:54:48   orbit, be that if it does finally repair itself, I don't know.

00:54:52   Yeah, and speaking of nuking it, by the way, that will be faster than letting it validate

00:54:56   itself again. If you just wipe it clean and re-copy the data onto it from one of your

00:55:00   two 12-derafight backup drives, I think that will take less time than allowing it to

00:55:04   re-verify, based on my experience. Oh, almost certainly. Trying to do re-verification.

00:55:08   Yeah, because if you figure like verification is going to be a very like

00:55:12   you know, a small block read-write-read-write-read-write kind of cycle, whereas

00:55:16   like, you know, re-copying onto it is going to be these like giant block transfers

00:55:20   and it only has to do like one pass with the writing, whereas

00:55:24   like if you're rebuilding, it has to like, you know, to write the

00:55:28   parity back on one new drive, it has to read everything off of all the drives, right? Then

00:55:32   you're going to have to put in another new drive, it's going to do the exact same process again.

00:55:36   And re-read everything again for that next drive, and like, you're going to have to go through the whole process what, four

00:55:40   times? Something like that, yeah. And I

00:55:44   hear you, but at the same time, it took four

00:55:48   days to rsync all of these files from the

00:55:52   Macbenny sitting literally underneath the thing. That's because you had multiple rsync

00:55:56   jobs running at once. Lesson number one for copying large volumes of data off

00:56:00   of a spinning disk, just do one copy at a time. You think you're going faster by having five

00:56:04   copies in parallel, you're not, you're just angering the disk, you're just making the disk controller angry.

00:56:08   I agree, I agree, no no no, I didn't think I was making it go faster by any stretch of the imagination,

00:56:12   what I was doing was getting redundant copies of the things that I really and truly

00:56:16   couldn't miss. Right, but if you wanted to do that, do the small copies first, like do things

00:56:20   and, you know, do the small emergency copies first, one at a time, serially.

00:56:24   Because there's only, there's only one, I'm not going to say there's only one disk head, because there's multiple disk heads,

00:56:28   but there's only one set of disk heads on a single arm, and that arm can only

00:56:32   follow one instruction at once, go left, go right, go up, you know, just, let it

00:56:36   do its, you know. No, I hear you, I hear you, you're absolutely right. And so,

00:56:40   sitting here now, we have moved to 2.14, no no,

00:56:44   2.15% in the time we've been talking, and so I figure in about a day

00:56:48   it will either finish or one of the drives will die, or

00:56:52   it'll finish and start over again, and if it starts over for a third time, then I think that's

00:56:56   probably a sign. But yeah, I don't know what the final

00:57:00   answer is. I don't see my life not having

00:57:04   a Synology in it, but in contrast to the last we spoke about this,

00:57:08   I do see a world where maybe what I end up doing is I get a

00:57:12   smaller Synology, like I said a moment ago, and putting larger drives

00:57:16   in that, having that local, and then sticking this one in, like

00:57:20   Mom and Dad's garage, and syncing to that and calling it a day.

00:57:24   And then additionally, for now anyway, I'm going to

00:57:28   plan on doing some sort of replication onto that 12 terabyte

00:57:32   once everything is squared away. Once everything's squared away, and I know I'm good

00:57:36   again, then I'll do some sort of replication onto the 12 terabyte, that will be backed up to Backblaze

00:57:40   the way it's supposed to be backed up, because it'll be physically connected to the Mac Mini, Backblaze

00:57:44   has no issue with that. Everything will be right as rain once it spends all

00:57:48   the time uploading to Backblaze, which is taking quite a long time in and of itself, but be that as it

00:57:52   may, then in theory I will be in a much better place. And if

00:57:56   Backblaze had already had everything, which is not Backblaze's fault, it's my

00:58:00   fault, if Backblaze already had everything, I could have asked them for like,

00:58:04   I don't remember how big the drives are they offer, but I could have asked them for one or two physical hard drives

00:58:08   to be next day aired to me, and then I could nuke everything, put these hard drives,

00:58:12   attach them to some things, maybe the Synology itself, copied all that data

00:58:16   back, like John was saying a moment ago, and then sent these hard drives back to Backblaze

00:58:20   and I have literally spent no money other than the normal Backblaze fees.

00:58:24   So, I'm feeling better about it now because of that

00:58:28   12 terabyte drive, which by the way, I don't think I said, I did disconnect and put it

00:58:32   in the garage, so it is literally across the house on a different floor, so

00:58:36   God forbid the house goes up in flames in the next 48 hours, it is as far away

00:58:40   from the Synology, so hopefully one of them will survive if

00:58:44   the house literally goes up in flames, and it is not connected to any sort of power source

00:58:48   or anything like that, but it has been, it has been a rollercoaster.

00:58:52   It has been a rollercoaster, mostly of my own doing, and I appreciate you letting me

00:58:56   get this off my chest, and I hope that this time tomorrow everything will

00:59:00   be restored and everything will be in a good place again, and then

00:59:04   I'll have to weigh whether or not I really want to replace drive 8 preemptively, or just

00:59:08   let it sit until it explodes. You keep forgetting about or not mentioning the

00:59:12   thing that you should do that it seems like you're not going to do because you've already forgotten about it, which is what Margot said,

00:59:16   make a second copy of the 12 terabytes. Yeah, well, I should. And you can do it by buying

00:59:20   a drive, like you can do it in such a way that later you can reuse that drive into one of your Synologies

00:59:24   or whatever, and speaking of new Synologies, you should look at, I was looking at

00:59:28   in my many, in my fantasy Synology shopping, there's more

00:59:32   variants in the CPUs these days, and it's not linear with the

00:59:36   sort of price of the device, like you can get a big one with a wimpy

00:59:40   CPU, and you can get a small one with a powerful CPU, so shop around and see

00:59:44   if you can get one that might actually be able to do all your transcoding for you, especially

00:59:48   most of your stuff is H.264, not H.265, it's possible you could get a

00:59:52   4 or 5 bay Synology that can do all your transcoding without breaking a sweat, so check that out

00:59:56   if you're shopping. Yeah, just looking, like the price differences for the bay count

01:00:00   actually aren't that big, like, if you, like, a 5 bay

01:00:04   is $650, and an 8 bay is $931

01:00:08   right now on Amazon, like, of the same line with all the premium processor, the plus line

01:00:12   and everything, so like, you actually, given, if you're gonna, you know

01:00:16   spend another 7 years with this, whatever you might get next, it might

01:00:20   be worth getting something a little bit bigger, but, uh, but you probably don't need

01:00:24   to. Also, the new Synologies let you connect expansion

01:00:28   bays to them, so you can buy one good Synology with a good CPU and 4

01:00:32   bays or something, and if you fill those 4 bays, you can buy another dumb

01:00:36   Synology bay thing, and use it to expand

01:00:40   your existing Synology, which I don't think that feature existed in 2013. It did,

01:00:44   our support it. Really? I didn't know that. I'm almost sure they do, yeah. I think, I'm

01:00:48   pretty darn sure that Marco's right. I love how on their site, uh,

01:00:52   they have, you know, if you go over, go to products, and then, you know, they have all these, like, sections

01:00:56   'cause they're all enterprise-y, and you have to choose whether you are a personal and home user,

01:01:00   or an IT enthusiast, or, you know, small,

01:01:04   small and mid-sized business enterprise, but is Casey an IT enthusiast?

01:01:08   I think so. I sure hope so. The funny thing is, the models it shows you seem to be

01:01:12   identical between whether you pick IT enthusiast... Well, that's the thing. If you

01:01:16   pick the business one, they're not gonna give you the ones with the good

01:01:20   CPUs. Like, it could be that the enterprise-y ones are the wimpier CPUs, that they just expect

01:01:24   to be doling out files to a large number of people, so look at the actual CPU options. They vary

01:01:28   a lot. That's fair. I don't know. So that's my,

01:01:32   that's my tale of woe, and what I, what I really want to do is

01:01:36   throw a whole pile of money at this problem, but I don't want to throw a whole pile of money at this

01:01:40   problem, so I'm trying to do this on the cheap, and that's half the issue right there. But

01:01:44   I will update you next week as to where all this lands, but it's been...

01:01:48   After you, after you copy the 12 terabyte to another 12 terabyte, right? Yes.

01:01:52   Please do that. He keeps not mentioning it. I know. I don't know why.

01:01:56   Like, yeah, 'cause even if you can get back to Best Buy and just get yourself

01:02:00   another, you know, 12 terabyte external from the parking lot, you can use that

01:02:04   even if you don't want an external, just take, take the drive out of it. Like, I've done that before, where like you buy

01:02:08   a drive versus an external. Later on, you want it to be internal. In fact, I think two

01:02:12   of the drives in my Synology, I think, are this exactly, where like, I just

01:02:16   took, I just took a screwdriver, took apart the enclosure that they came with from Seagate or whoever,

01:02:20   and inside of those enclosures is a regular drive with a little USB controller board. You just unplug it

01:02:24   from that. It's a regular SATA drive, and you plug it into whatever you need. Yeah.

01:02:28   Yeah, I probably should. That's why I always buy my, my mechanisms

01:02:32   specifically, and I buy all these crappy external cases, just so I get to pick the mechanism.

01:02:36   Because if you buy some, I mean, obviously what you bought from Best Buy is like, you gotta do what you gotta do. You need to

01:02:40   get drive ASAP, you know, situation, I get it, right? But with more

01:02:44   time to spare, it's always better to pick the mechanism yourself. Even if you pick wrong,

01:02:48   at least you can, you know, skew towards drives

01:02:52   that you think are, have a better reliability reputation.

01:02:56   You can look at the back play stats. I do that all the time. They, they have the hard drive stats of like

01:03:00   their failure rates. The problem is, there are two problems with the back play stats. One, most

01:03:04   people's use case is not like back plays' use case. Yours might be

01:03:08   similar, kind of, but not really. Like, I don't, back plays drives

01:03:12   are running in a different environment than, you know, they're running in a data center, they're running in these

01:03:16   big racks and these machines right next to each other. They don't have a giant box fan pointing

01:03:20   at them. Yeah, they might have better cooling than you, they might have

01:03:24   worse, they might have more activity or less, but it's, it's not a typical

01:03:28   use case of like, oh, occasionally I watch Plex off of my thing. Like, that's,

01:03:32   that's calmer than back plays. And the second thing is, back plays gives you the exact

01:03:36   model numbers. By the time they do their readouts of the exact model, sometimes it's hard

01:03:40   to find those exact models. So it's like, wow, or you can find them, but they

01:03:44   only have like, you know, three months worth of data. And if you look

01:03:48   at the past reports, you can say three months is not a long enough time to know if this thing is going to die

01:03:52   in the first year. When you see one, it's like, we've had these drives for two years and their reliability

01:03:56   has been excellent. You can no longer buy those drives. So it's not a slam

01:04:00   dunk, but at least it can give you a feel for

01:04:04   manufacturers, model lines. It's so hard, because like, it could be, you know, wow,

01:04:08   it looks like, you know, Western Digital Reds are doing really well. But it turns out the 14 terabyte

01:04:12   Western Digital Red is a reliability disaster and you get it based on the reputation

01:04:16   of the smaller ones and you make a terrible mistake. So it's, it's not an

01:04:20   exact science, but I don't know, I always feel better being able to pick the magnets, especially since

01:04:24   if you buy one off the shelf, it could have something in there that isn't even a drive,

01:04:28   you know, a quote unquote consumer drive. Like, it's a drive that's not

01:04:32   even meant for NAS type situations where it's expected to be

01:04:36   on all the time and, you know, maybe it has a little bit, a little bit more

01:04:40   redundancy. Like, that's in theory when you buy a spinning

01:04:44   disk that's intended for use in a data center or intended for use in

01:04:48   a NAS. A, you're paying some stupid premium for their profit margins.

01:04:52   Just live with that. But B, in theory there is something physically different about the drive

01:04:56   that makes it slightly more suitable to this purpose.

01:05:00   Yeah, I gotta rethink my whole world, unfortunately. But one step at a time.

01:05:04   We'll see how this restore goes.

01:05:08   We are sponsored this week by Eero, the WiFi your home

01:05:12   deserves. Now, we've all had bad WiFi experiences.

01:05:16   In our houses, in our relatives and parents and friends' houses, whatever it might be.

01:05:20   So I told you last time about how I had a vacation rental with terrible WiFi and I

01:05:24   got an Eero and I fixed their WiFi and took it with me when I left. It was wonderful.

01:05:28   And where that Eero went after that was to my in-laws house.

01:05:32   They had, like, you know, areas where their WiFi wouldn't reach. They had bad rooms

01:05:36   including the room that we would stay in, which hit me personally.

01:05:40   When we stayed with them, they would just have slow WiFi there. And I thought, I gotta fix this problem.

01:05:44   At least for us, but also for them. So we

01:05:48   moved that Eero there. And their problem was fixed

01:05:52   immediately. They have amazing WiFi now, thanks to

01:05:56   Eero. There's also an all new Eero now, so not only

01:06:00   can you fix your WiFi problems. As soon as tomorrow, which I'll get to in a moment.

01:06:04   But now there's an all new Eero starting at just $99. So you

01:06:08   too can have your entire home blanketed in fast, reliable WiFi.

01:06:12   Get rid of your slow rooms, your dead spots. You get a strong signal

01:06:16   wherever you need it. It's super easy to set up an Eero system. It sets up in just minutes.

01:06:20   You plug it right into your modem or your router, and you manage it from there.

01:06:24   Super easy to use app. This is the easiest router setup I've ever seen.

01:06:28   Single router or multi router setup. Easiest I've ever seen by far with their

01:06:32   awesome app. So there's no more WiFi problems. No more, you know, video buffering

01:06:36   in certain rooms or can't load Instagram, whatever it is. It's just fixed.

01:06:40   Let Eero fix your WiFi problems too. And you can get it

01:06:44   fixed as soon as tomorrow. Go to Eero.com/ATP

01:06:48   That's Eero.com/ATP and use offer code

01:06:52   ATP at checkout to get free overnight shipping with your order.

01:06:56   That's Eero.com/ATP. Code ATP to get your Eero delivered with

01:07:00   free overnight shipping. You must use this URL to receive this offer.

01:07:04   Eero.com/ATP. Code ATP. Thank you so much to Eero

01:07:08   for fixing our WiFi and sponsoring our show.

01:07:12   So there was some news today as we

01:07:16   record, and it surprised me a lot.

01:07:20   Apple has bought Dark Sky. And if you're not familiar, Dark Sky

01:07:24   I don't know, it started something like five to ten years ago

01:07:28   and at the time it was its own weather app

01:07:32   and then they shortly thereafter provided an API. And

01:07:36   especially early on, it was eerie how accurate it was.

01:07:40   It's still very accurate, but I don't personally feel like it's quite as accurate as it used to be.

01:07:44   But it would eerily predict when it was going to start to rain, where you were

01:07:48   sitting. So it would say something like rain in seven minutes. And

01:07:52   five, six years ago whenever it came out, I would look at my watch and it was, I don't know,

01:07:56   ten past ten, and sure enough, seven minutes later, it started raining.

01:08:00   It was bananas. It was magic as far as I'm concerned.

01:08:04   And over time, I stopped using the Dark Sky app, I personally am a

01:08:08   Carrot Weather user, but Carrot Weather, for Americans anyway, still uses

01:08:12   Dark Sky for most if not all of its weather information.

01:08:16   And pretty much all of my favorite weather apps over the years, including

01:08:20   Check the Weather by Underscore, and I forget what else I've used in the past, but almost all

01:08:24   of them use Dark Sky in part if not in whole, for Americans

01:08:28   anyway. And Apple has bought them. And they, I believe they

01:08:32   either killed the Android app or have said it's not going to get updates. Do you guys remember

01:08:36   the 20 dollars? I think it's removed from the store or something like that. But I think it still works.

01:08:40   Okay, and then on top of that, they have announced that the

01:08:44   iOS app is still available, no, it's still available and it's still purchased,

01:08:48   which is a little bit weird, but the API is going away next year.

01:08:52   And I think that's basically the summary of what's happened. Marco, any thoughts

01:08:56   on this as the person who I think is most close to

01:09:00   developing an app that would use, obviously you don't use Dark Sky, but

01:09:04   you're the kind of person that would develop this kind of app, so how do you feel

01:09:08   about this? Well, I do use Dark Sky. That's the thing. I have their app installed,

01:09:12   I also use, my usual weather app is Weatherline, and

01:09:16   Weatherline uses Dark Sky data. It used to use it, I think, for everything.

01:09:20   When they redid the app, they now use a hybrid that

01:09:24   includes Dark Sky for certain things, but uses other data providers for other things.

01:09:28   This, I really don't like this news, honestly. And we'll see what happens.

01:09:32   Maybe over time, I will end up liking whatever Apple has in mind

01:09:36   to do with this here. But I don't love this the way it is now,

01:09:40   because the Dark Sky API has been so important. It really is,

01:09:44   it has powered so many

01:09:48   indie weather apps that I have really liked it. As you mentioned, Carrot,

01:09:52   Weatherline, and including Dark Sky's own app.

01:09:56   And it's been really great. There have been times in the last,

01:10:00   whatever it's been, ten years, there have been times where it is less accurate.

01:10:04   But, you know, it's an algorithmic based thing. It's reliant

01:10:08   on certain data, and I'm sure they've tweaked the algorithms over time. So, you know,

01:10:12   some people try it and it doesn't work out for them, or it works for a while, then it's bad for a while,

01:10:16   then it's good again. It's always been good for me. And I use it all the time, because I want

01:10:20   to know, like, every day, I'm going to go outside, I'm going to walk my dog for the next half hour, is it going to rain?

01:10:24   The whole concept of the app is a great, it's a great app, and

01:10:28   it has really been important to me, you know, both as a user

01:10:32   of Dark Sky directly, and as a user and fan of third party,

01:10:36   especially indie, weather apps. It's been great. I even,

01:10:40   I forget why, but I even had a couple of email exchanges with the

01:10:44   founders forever ago, and I remember them being very nice. Like, it's just, it's a

01:10:48   wonderful app, run by wonderful people. It's also been very

01:10:52   successful. It has been the top paid app, or within the top

01:10:56   few paid apps on the app store for years. So,

01:11:00   you know, they've, whatever they're selling to Apple for, it's probably not

01:11:04   that their sales sucked, like, it's not, it's probably not that they needed the money,

01:11:08   and they also charge for the API, and they make good money from that, I assume.

01:11:12   So, the reason for them going to Apple, I'm sure they're going to do great things,

01:11:16   I'm sure, you know, Apple wants them to, you know, work on the weather team or whatever,

01:11:20   that's great, the Apple weather app could use some love, but

01:11:24   I would feel a lot better about it if they were still going to run the API for other

01:11:28   apps to use as well. And maybe there's a way that they're going to do this. Maybe Apple

01:11:32   will offer, like, a weather kit API for iOS apps, you know,

01:11:36   screw Android, but for iOS apps to use this, maybe.

01:11:40   That would be nice, but I don't know that they will. So,

01:11:44   you know, if the idea is just for Apple to, you know, have their own weather

01:11:48   source, and I guess not have, maybe not have to pay, like, whoever it is, like,

01:11:52   the Weather Channel or Yahoo or wherever they're going to source now, if, whatever Apple's

01:11:56   value is getting out of this, if it's just to, like, save on a license

01:12:00   fee to some other data provider and to have, like, local rain forecasts

01:12:04   in the iOS weather app and screw everybody else and, you know,

01:12:08   no one else is going to have this data anymore, that's no good. But if there's a way

01:12:12   that third-party apps can continue to be good and have access to

01:12:16   this kind of data, I hope they can manage to do that.

01:12:20   But for now, I don't feel good about this, because that's an uncertain future,

01:12:24   and I really do like all of the

01:12:28   benefits that the Dark Sky API has provided. It also, like, they

01:12:32   also have a website. For a while it was forecast.io, but now I think

01:12:36   it's just dark-sky.something. I don't know, it's always auto-completing for me. And what's

01:12:40   great about the website, not only can you see weather anywhere, of course, which is actually surprisingly nice

01:12:44   on something like a Mac where there really aren't a lot of good weather apps on the Mac, but

01:12:48   also you can look up historical weather data, which is really cool.

01:12:52   All that for free on their website. Like, it's just been a wonderful service and a wonderful

01:12:56   app and a wonderful API that has been such a key part of so many people's

01:13:00   daily usage. So to have Apple take it over and

01:13:04   give it a shutdown date is concerning. I always wonder how Apple feels about

01:13:08   these situations where

01:13:12   there's, like, an API or some other thing

01:13:16   that third-party apps build on. From

01:13:20   our perspective as users and even as developers, these all seem like

01:13:24   clear winds. You just mentioned, like, oh, this could go into some kind of weather kit thing or whatever, but

01:13:28   think about cases where Apple has done this either by accident or on purpose.

01:13:32   There's tons of calendar apps for the Mac because Apple provides a bunch

01:13:36   of APIs that allow those apps to access the Apple calendar.

01:13:40   Also, those calendar apps access other calendars, like Google Calendar

01:13:44   and so on and so forth. Apple, you know, was just a super plan for Apple

01:13:48   to say, we want to have tons of awesome calendar apps on iOS and the Mac

01:13:52   for that matter, so let's do this open calendar API. Clearly,

01:13:56   they made whatever the calendar APIs are called with that intention, same thing with the

01:14:00   address book, contacts, all that stuff. They made those APIs with that intention, but

01:14:04   at various points they could have made different decisions. They could have said, oh, great, well, you can

01:14:08   make an iOS app that accesses the shared calendar, but your app

01:14:12   can't access Google Calendar or something like that. That totally sounds like a thing that Apple would entertain.

01:14:16   They do all sorts of stuff like that related to an app purchase and other scenarios where they're trying

01:14:20   to sort of steer the market through restrictions because they control the whole

01:14:24   store. API stuff, CloudKit and all that stuff.

01:14:28   They make a lot of essentially faceless services. The

01:14:32   only reason they exist, aside from Apple using them themselves, is, hey, if you want

01:14:36   to make an app for iOS that has an online component but you're not a

01:14:40   server-side developer, we have a bunch of libraries that talk to our

01:14:44   servers with a fairly attractive business model of

01:14:48   you don't have to worry about the service being up or anything like that. We will store your data.

01:14:52   They have this tiered thing where if you have a small number of users, it's not a big deal.

01:14:56   The fees go up as you get more. Those services

01:15:00   exist so that people can write better, cooler apps

01:15:04   with less, not with less effort, but with less people power, with

01:15:08   less expertise. It makes their platform better

01:15:12   for users because they get better apps. It makes it more attractive for developers because smaller

01:15:16   developer teams can do better things. So you look at this scenario.

01:15:20   There's tons of cool weather apps for iOS and

01:15:24   less so for the Mac, as usual. But anyway, why are there all these weather apps?

01:15:28   Is every one of these weather apps backed by some company that owns weather

01:15:32   stations and weather data? No. Sometimes they use commercial APIs that exist out in the world

01:15:36   and the Dark Sky API, I'm assuming, is just a wrapper around a bunch of commercial APIs because I

01:15:40   assume the Dark Sky people don't run a bunch of weather stations. So it's just wrappers on top of wrappers

01:15:44   are just fine. But if you want to make a weather app and your idea is it's going to

01:15:48   be named after a vegetable, it's going to have snarky messages about the weather,

01:15:52   you can implement that without worrying so much about

01:15:56   "Yeah, but how would I get the weather? Do I have to contract with AccuWeather?

01:16:00   How do I do all this stuff?" And if the Dark Sky API is

01:16:04   a path of least resistance to do that, then that makes

01:16:08   more weather apps available. So here Apple is faced with a choice. We want Dark Sky

01:16:12   because basically, I'm assuming, they want to make their weather app better because their weather app has been falling

01:16:16   behind. Great, fine, buy them. It's great for the developers because they presumably get a big pay

01:16:20   day. As has been discussed any time anyone gets acquired, a lot of the motivation

01:16:24   for people going to Apple is often that their work can

01:16:28   be put in front of more people because Apple has a huge audience and as

01:16:32   successful as Dark Sky is, if you are the default weather app on the iPhone,

01:16:36   that is a bigger impact. So if you're looking for what that next step is, boy, I'd like to be

01:16:40   in front of hundreds of millions of people instead of just single digit millions, right?

01:16:44   But anyway, it's all good for them and it's good for Apple to get a better weather

01:16:48   app, but I really hope that Apple looks at what

01:16:52   they've acquired and the fact that it is an API and says, "Regardless of whether

01:16:56   we shut down this API or not, we should do the same thing that we have done, again, either

01:17:00   accidentally or on purpose, with things like CloudKit

01:17:04   and like the calendar and contact APIs."

01:17:08   It doesn't always have to be server-side, but enabling

01:17:12   there to be more competitors for their own apps.

01:17:16   And that's why I started this by saying I wonder what Apple thinks about this because

01:17:20   if I was inside Apple, I'd be like, "Don't you see, in all the places

01:17:24   where we have the most richness in the App Store, it's because we did things

01:17:28   like this?" On the other hand, I can imagine someone saying, "How did our weather

01:17:32   app fall behind?" Well, it fell behind because we allow all these other apps to just have

01:17:36   this ecosystem of sharing this API and we need to nip that in the bud and now we'll have the best

01:17:40   weather app again, which is short-sighted and stupid. And I don't think they think that,

01:17:44   but I also don't see, I never see them come out and sort of

01:17:48   other than in obscure technical WWC sessions, tout the idea

01:17:52   that by making services and APIs

01:17:56   available, we are making a better software ecosystem

01:18:00   on our platform. I know they have a big

01:18:04   services picture. When we say services with the capital S as in these earnings

01:18:08   that we've been talking about for the past several years, it's like, "Oh, I'll pay Apple

01:18:12   so I can watch TV shows. I'll pay Apple so I can store my photos for you. I'll pay Apple

01:18:16   for more disk space. I'll pay Apple for a music service." It's all about just paying Apple,

01:18:20   consumers paying Apple. But this scenario where

01:18:24   there are services but the people who pay for them and use them are developers,

01:18:28   like one degree separated from users, doesn't seem like something that Apple

01:18:32   it's not that they don't like it, they do it enough times, but

01:18:36   it always strikes me as a little bit too accidental, huh?

01:18:40   That they just found themselves in a situation like, "Why is it true for Calendar but

01:18:44   not true for something else?" And the podcasts are a good example. They have this podcast directory

01:18:48   which accidentally or on purpose powered this entire ecosystem of podcast apps,

01:18:52   but was that the plan from the beginning or did it just kind of happen and they were smart enough not to screw it

01:18:56   up? I think they should embrace this model because I think this model has been proven to work.

01:19:00   It gives Apple control. They have control over the CloudKit APIs. They have control

01:19:04   over all the other things they offer. They have control over their libraries.

01:19:08   They could make a weather kit and they could have essentially Dark Sky

01:19:12   API v2 powered by Apple's access through weather kit

01:19:16   and that would enable someone else's great idea of how to make a weather app

01:19:20   to be done or someone's bad idea for a weather app. Either way, it will

01:19:24   attract developers to the platform. It will make it a more

01:19:28   attractive place for customers to shop because they have lots of cool weather apps made by

01:19:32   small teams with a good idea who didn't want to deal with the weather stuff. So my fingers are

01:19:36   crossed for that eventuality, but if they just take the

01:19:40   crew, make their weather app better and stop the API after

01:19:44   a year, it's not the end of the world, but it's not great.

01:19:48   That's where we get into Marco being pessimistic about this. I'm choosing to think

01:19:52   that they're going to do the right thing and if not continue that API

01:19:56   then have a similar API fronted by a library or something so that

01:20:00   we can continue to have tons of weird and ridiculous weather apps on

01:20:04   iOS. Yeah, I totally think and hope that that's the case and

01:20:08   it vaguely reminds me of when Google Maps was the

01:20:12   only mapping software on the iPhone and if you wanted to put a map in your

01:20:16   app, you had to jump through some hoops. I don't remember exactly what they were, but you had to jump

01:20:20   through some hoops to establish a relationship with Google and

01:20:24   I think you might have even had to get a Google SDK in order to do it, which kind of makes sense.

01:20:28   And then Apple starts providing its own maps and granted they

01:20:32   were garbage for a long time, but now I don't think they are for most people. And

01:20:36   adding a map into your app is super simple because the API is right

01:20:40   there. It's part of the system and I feel like this feels as

01:20:44   though it's an analogous sort of event and this is exactly what you're saying John, that maybe

01:20:48   there will be some sort of weather API. Perhaps it could be even part of the MapKit API,

01:20:52   who knows. But some sort of weather API that will allow you to

01:20:56   get weather information for all sorts of different things. I mean if you think about it, weather

01:21:00   information, I'm trying to think off the top of my head, on my phone shows up in Fantastical,

01:21:04   it shows up in the Maps app, it shows up in my weather apps,

01:21:08   there are more places than you would expect. Oh, and day one it shows up, there's more

01:21:12   places than you would expect that have or show weather

01:21:16   data and to be able to... Is there already a weather API? I don't even know, maybe there's already a simple

01:21:20   weather API. There isn't. Yeah, I don't think there isn't. So I don't know, I don't know where

01:21:24   this goes. It certainly could be another

01:21:28   form of lock-in, you know, if the only way you get a really good weather port is if you have

01:21:32   an iPhone. Like I hope that's not the case, but it would be an interesting

01:21:36   and very unusual form of lock-in. Well yeah, I really like that Android phone,

01:21:40   but I can't get my weather app that I rely on because, I don't know, I'm a dog walker or something like that,

01:21:44   like a professional dog walker, or I don't know, I'm a landscaper, you know, and it's

01:21:48   incredibly important, I know exactly. I don't think that's that bad of a situation,

01:21:52   that's basically my optimistic scenario, is that it is, you know,

01:21:56   that Apple uses it as a platform differentiator. Again, Apple, as far as I'm aware,

01:22:00   you never know, doesn't own a million weather stations around the world, like some, the weather

01:22:04   data comes from elsewhere. This is all just about mediating access to that and providing

01:22:08   a nice interface to it, and if it's a differentiating factor

01:22:12   to say, you know, anyone can get like the weather from just some source or whatever, but Dark Sky

01:22:16   is a big thing, as Marco said, is like, it was super, you know, whatever they call it, hyper-local,

01:22:20   like you can get the weather right where you're standing right now,

01:22:24   which was a differentiating feature and the reason it's a popular application. So,

01:22:28   having a weather API is something anybody can do. Having a really good weather

01:22:32   API is harder, and that's, if Apple wants to use that as a differentiating feature for the iPhone,

01:22:36   oh, the iPhone has, you know, they bought Dark Sky, so they have the current best

01:22:40   weather API, whether it's because of the way Dark Sky processes the data that it's

01:22:44   got or the data sources it has or whatever, but it's not like they're locking anybody out

01:22:48   because, you know, the weather is, information is

01:22:52   publicly available, as in we can all see what the weather is, and anybody can build weather stations

01:22:56   or anyone can pay someone who has weather stations, so like, I don't feel that there's

01:23:00   any evil in them, you know, confining the best weather API

01:23:04   and the easiest to use weather API to their platform,

01:23:08   that's exactly what they should be doing. You know, Android and Google

01:23:12   can do exactly the same thing. You know, that's how they should be competing with each other.

01:23:16   Yeah, and so the other angle to this is

01:23:20   that weather apps are famously

01:23:24   frequently bad for privacy. Not the ones we've mentioned, not like the nice indie ones,

01:23:28   but there's a lot of weather apps out there. You know, weather apps, by their nature,

01:23:32   need your location for most convenient features.

01:23:36   You know, you can always not give some of your location and maybe just type in your location or have it just

01:23:40   always display that location or have it to change it manually, but for the most part, most people

01:23:44   give most weather apps their location access so they can show the weather wherever they happen to be.

01:23:48   Because weather apps have location access, they can sell that information.

01:23:52   Location data is very valuable in the ad tracking creepiness business.

01:23:56   Not only do many weather apps, you know, have a lot of

01:24:00   not only do many weather apps just sell this data outright,

01:24:04   if you have a weather app that gets location access, you will be approached, people will email you

01:24:08   from these scam companies offering you money to

01:24:12   integrate their SDK. And this might be more money than you might be making from your

01:24:16   app sales if you're not doing that well in the app store, which is very often the

01:24:20   case for a lot of apps, right? They will offer you money to build in their tracking

01:24:24   SDK because your app has location access, right, by the user.

01:24:28   So they'll give you money to put their SDK in that will measure your

01:24:32   location that the app already has access to and tie it to ad profiles

01:24:36   and then they can sell that and make more money from you. So that's a whole thing that's

01:24:40   going on behind the scenes now, and that might be informing this decision by Apple as well.

01:24:44   There's this whole seedy underbelly of weather apps that because

01:24:48   they by their nature usually get your location permission,

01:24:52   they have this valuable data and many of them are being really creepy with it.

01:24:56   Again, not the ones we've mentioned because we wouldn't use an app that did that, right?

01:25:00   The ones we've mentioned, you know, Carrot, Weatherline, like these are all good apps made by good people. They don't do this kind of crap.

01:25:04   But many apps do. So it's a very big deal

01:25:08   that weather apps are a notorious privacy hole

01:25:12   for your location data. And Apple's very sensitive to that. So maybe

01:25:16   this is part of some kind of play to crack down on that.

01:25:20   I haven't quite thought about how they could do that effectively.

01:25:24   One option would be they no longer allow weather apps to

01:25:28   have your location and they just provide weather data for them.

01:25:32   But that's a really aggressive move. And that would have a lot of other

01:25:36   problems too. Like, for instance, no one

01:25:40   weather data source is good for the entire world. There's different

01:25:44   data sources in different parts of the world and certain ones are better than others for different

01:25:48   regions and everything. So I don't think they would or should do something

01:25:52   as drastic as prohibit any other weather API from having location

01:25:56   access. That would be weird. So I'm just trying to think through, like, is there

01:26:00   some kind of privacy angle on this that would lead to an actual

01:26:04   possible and likely and enforceable outcome

01:26:08   that they could maybe crack down on this incredibly seedy

01:26:12   underbelly of privacy selling weather apps by somehow

01:26:16   buying Dark Sky. Again, I haven't quite figured out how that can work yet, but maybe there's something there.

01:26:20   I was thinking about all the other platforms that Apple has where weather data is

01:26:24   useful. Apple has weather data like this weather. There's a weather widget on your watch,

01:26:28   there's weather thing in the notifications, whatever the hell that thing's called on

01:26:32   the Mac on the right side of the screen, the Today View, that has a weather thing in it, I believe.

01:26:36   And obviously on our phones. And buying Dark Sky will give them

01:26:40   better weather presumably, gives them people who are experienced making

01:26:44   more advanced weather apps than Apple's own and plus all the hyper local stuff with the local

01:26:48   radar to say it's going to rain in five minutes. But, you know,

01:26:52   having your watch nudge you that it's going to be raining on your head in five

01:26:56   minutes is pretty cool. Dark Sky app, I already have a watch app that does

01:27:00   that. But, you know, AR glasses, you can look up in the sky and see which direction

01:27:04   the rain clouds are coming from. There's all sorts of scenarios where even better weather

01:27:08   than they currently have is an attractive thing to Apple. So this

01:27:12   purchase doesn't surprise me in the slightest. I'm just really hoping

01:27:16   that the path they take with this purchase is not simply to give Apple's

01:27:20   platforms better weather stuff, but to

01:27:24   make an API and a service that everybody can use, even if they

01:27:28   discontinue the current one, make a new API, like Margot was saying, perhaps with different privacy

01:27:32   trade-offs. Hopefully not with too many different restrictions, like

01:27:36   what we don't want to see is like, oh, Apple has a cool new weather kit that's very

01:27:40   advanced and has all the features of Dark Sky plus an API and you can use it in the right theory of the reason, and

01:27:44   every other weather service is forbidden on the platform. I don't think they would do that, but

01:27:48   that's the worst case scenario. That's like the in-app purchase scenario where

01:27:52   if you want to sell things through your applications

01:27:56   for this huge swath of stuff where you have to go through Apple's thing and give them 30%

01:28:00   and stuff. So that's at one extreme. And the other extreme is it's a free-for-all

01:28:04   Apple is going to compete in that free-for-all by having a really good weather service, which they can

01:28:08   do now that they bought Dark Sky and I feel like they should just be happy to compete with that, like they do with CloudKit.

01:28:12   You can use CloudKit or whatever, you know, core data

01:28:16   I don't like cloud core data, maybe you don't want to use that, but anyway, Apple has a bunch of services

01:28:20   that you can use in your iOS app, or not. By the way, isn't it a shame that they already

01:28:24   use the name CloudKit? Like wouldn't that have been perfect for this?

01:28:28   Yeah, and they don't know if they wanted to use Dark for Dark Sky, but

01:28:32   like the only scenario where Apple has most recently

01:28:36   forced its hand is in a situation like Marco was alluding to, which is where there's a privacy

01:28:40   thing. So Apple has a thing that you can use for your sign-in, sign-in with Apple,

01:28:44   and there they do mandate that if you have an app that allows you to sign in with Facebook

01:28:48   with Google, with whatever, you have to offer sign-in with Apple. And the reason they're doing

01:28:52   that is privacy related. Part of it is yes, okay we want everyone

01:28:56   to use Apple's things, but honestly Apple has you already. The whole point is, you're on your iPhone

01:29:00   you have an iCloud, you know, Apple's got your Apple ID. Like they're all set on that front.

01:29:04   Sign-in with Apple is, I guess, you know, partially

01:29:08   so they can get their hooks into you one more way, but the reason I think they're

01:29:12   mandating it is not just for adoption, but also because

01:29:16   they have a privacy angle, as evidenced by the way the API looks, where you can use it without

01:29:20   even giving people your email, which is unheard of on the other services.

01:29:24   They want everything from you, right? So if there's

01:29:28   a privacy angle on this, I can see Apple being a little bit more forceful

01:29:32   perhaps not mandating the use of their API, but further restricting

01:29:36   the other APIs to compete on a level privacy playing field with Apple's.

01:29:40   They haven't done that with the sign-in stuff. You sign in with Facebook, Facebook gets all

01:29:44   your everything. You sign in with Google, Google gets all your everything. You sign in with Apple, Apple

01:29:48   is the one voluntarily saying, "Hey, when you use our API, you

01:29:52   have to actually ask people for the email address. You don't get it by default.

01:29:56   We'll make up a random email for you." You know, all that stuff that Apple is doing

01:30:00   is tying their own hands behind their back in terms of the amount of data they get.

01:30:04   Apple doesn't want the data. They want to have a more private

01:30:08   API, but they didn't require everybody else to do the same thing, at least yet anyway.

01:30:12   The weather API, we'll see. This is an interesting thing to watch, not so much because we're

01:30:16   interested in weather APIs and the ecosystem weather apps on the iPhone, but because it shows where Apple's head

01:30:20   is at currently in terms of the range of things that they can do

01:30:24   both policy-wise and tech-wise for their platforms.

01:30:28   We are sponsored this week by Postmates.

01:30:32   From an early morning breakfast to a bottle of wine for movie night, sometimes you just need

01:30:36   what you need, and you need it delivered fast. That is where Postmates

01:30:40   comes in. Postmates delivers food from every restaurant you can think of

01:30:44   right to your door. And they don't just deliver takeout. You can also get deliveries from

01:30:48   grocery stores, convenience stores, clothing stores even, you name it.

01:30:52   This is especially nice to have right now. So, no more trips to the

01:30:56   store, no more late night fast food runs. Just download Postmates

01:31:00   on iOS or Android, find your favorites, and you can get anything you want

01:31:04   delivered within the hour. For a limited time, Postmates is giving

01:31:08   our listeners $100 of free delivery credit for your first

01:31:12   seven days. So, that's right. If you are new to Postmates, you can get

01:31:16   $100 of free delivery credit with no minimum purchase.

01:31:20   So to start your free deliveries, download the app and use code ATP.

01:31:24   That's code ATP for $100 of free delivery

01:31:28   credit in your first seven days. So, once again, to start your free deliveries,

01:31:32   download the Postmates app and use code ATP for

01:31:36   $100 of free delivery credit with no minimum purchase for

01:31:40   your first seven days when you download the Postmates app using code ATP.

01:31:44   Anything you need, anytime you need it, Postmate it.

01:31:48   Thanks to Postmates for sponsoring our show.

01:31:52   Just today,

01:31:56   apparently it's now possible, and I'm very unclear on where

01:32:00   the dividing lines are, but it is possible to rent

01:32:04   or buy or do something with video through

01:32:08   Amazon Prime Video's app, but not using in-app

01:32:12   purchase? Did one of you pay closer attention to this? Because I am deeply confused as to what the situation is.

01:32:16   Well, why don't you read the very, very clear Apple quote that's in the show notes.

01:32:20   Well, I would be happy to, John. Apple has established

01:32:24   program for premium subscription video entertainment providers to offer a variety of customer benefits,

01:32:28   including integration with the Apple TV app, AirPlay 2 support, TVOS apps, Universal Search,

01:32:32   Siri support, and, where applicable, single or zero sign-in.

01:32:36   On qualifying premium video entertainment apps such as Prime Video, Altice One, and

01:32:40   Canal Plus, customers have the option to buy or rent movies and TV shows using the payment method

01:32:44   tied to their existing video subscription.

01:32:48   Wow, look what they snuck in there. That's a lot of words. What does that actually mean?

01:32:52   So this is incredible. So forever, Amazon

01:32:56   and Apple have been battling over in-app purchase rules.

01:33:00   Apple has always, since the introduction of in-app purchase on iOS,

01:33:04   they've always required that apps are not allowed

01:33:08   to collect payment information and have their own payment processing

01:33:12   to buy digital goods in their apps

01:33:16   without going through in-app purchase. Like, Amazon and Apple

01:33:20   has not been allowed to have something like the Kindle app,

01:33:24   where I believe we first saw this, and later on when they launched Prime Video

01:33:28   that Amazon will sell you videos to rent or buy

01:33:32   or whatever, and Apple has not allowed them to do

01:33:36   Amazon's own purchase system inside the app in their iOS app.

01:33:40   They would require them to use Apple's in-app purchase system, but Apple's in-app purchase system

01:33:44   charges 30% for most things, and they would require

01:33:48   Amazon charges 30% for most things, and had other various

01:33:52   limitations that Amazon didn't want to comply with.

01:33:56   There's always been this battle of heads, you know, budding, and they've been going back and forth

01:34:00   kind of quietly in the background for years, and Amazon and Apple

01:34:04   had this, you know, pretty tense relationship for a while.

01:34:08   And last year, I think, or whenever it was, they basically

01:34:12   announced that they had come to some kind of agreement. For the first time in a while,

01:34:16   Amazon would start selling Apple products again, which like,

01:34:20   things had gotten so bad that it was actually tricky to buy

01:34:24   Apple products on Amazon. Like, that's how bad their relationship had gotten.

01:34:28   And there was this whole thing. But the critical,

01:34:32   like the main thing that Amazon didn't like about Apple was

01:34:36   this in-app purchase rule, that Amazon just wanted to offer their own purchases

01:34:40   in their app. And Apple doesn't care if you do that for physical goods,

01:34:44   or for services. That's why you can use your own payment methods

01:34:48   for things like, you know, Lyft rides, or for, you know, Amazon's

01:34:52   shopping website, because that's physical goods. Apple doesn't have

01:34:56   a payment processing system for that, but they do have, for any kind of

01:35:00   digital goods, like buying or renting videos or songs or ebooks,

01:35:04   Apple always has required that that goes through their system.

01:35:08   And Amazon has always said, basically, no. And that's why, and

01:35:12   they have all these other crazy rules, like you aren't even allowed to link out to a browser to do it,

01:35:16   and that's why you have all these crazy things where you have situations like the Amazon,

01:35:20   like the Kindle app, who won't tell you that you can go to

01:35:24   Amazon's site to sign up or buy these things. It just has to kind of assume you'll figure that on your own

01:35:28   somehow. And there's all these crazy rejections. Spotify's been involved, and

01:35:32   you know, HBO and Netflix have done their own things also. So it's been this whole thing.

01:35:36   So what has changed today is that Amazon's

01:35:40   app for Prime Video, not for anything else, but so far only

01:35:44   for Prime Video, and I think there's a reason for that that we'll get to in a moment, now allows

01:35:48   you to buy and rent, you know,

01:35:52   or otherwise pay for video using your

01:35:56   Amazon billing method that's already entered. So you can't type in

01:36:00   a credit card newly into this system. Otherwise, there's some

01:36:04   weird exceptions that you might be able to do it some way, but for the most part, you can't do

01:36:08   that. But if you, you know, most people with Amazon accounts have some kind of

01:36:12   payment method already tied to them, and so if you have that,

01:36:16   now you can use the Prime Video app on iOS

01:36:20   with Apple's blessing, maybe through gritted teeth, but with Apple's blessing

01:36:24   to buy videos that don't go

01:36:28   through Apple's payment processing at all and don't pay Apple's 30%.

01:36:32   There's a whole bunch of complicating factors here. For instance, 30%

01:36:36   has not been 30% for people like Amazon for a while. Large

01:36:40   companies like Netflix, HBO, way before Apple

01:36:44   announced the whole like 15% for years to and up thing for the rest of us,

01:36:48   whenever that was about two years ago, one year ago, long before that, the big companies

01:36:52   were already paying something like 15%. Like Apple had already cut deals with some of the big ones

01:36:56   because they wouldn't do it otherwise, and they eventually made that public. So there is already

01:37:00   a history of Apple having to negotiate with big companies

01:37:04   with exceptions to these rules here and there. But this is a

01:37:08   really big one, and I think the reason why they were willing to do this

01:37:12   for Amazon specifically, and apparently Altus One and Canal Plus,

01:37:16   which I don't follow this business, I've never heard of those, I hear Canal Plus is big

01:37:20   I think in Europe or France or something, but it's fine, whatever, I'm sure they're big somewhere, but

01:37:24   the key part of this, of why Apple capitulated on such a massive

01:37:28   thing they've stood so firm on for so long, I think is in the

01:37:32   previous sentence. Apple has established a program for blah blah blah

01:37:36   video providers specifically, not all apps, video

01:37:40   providers, including integration with the Apple

01:37:44   TV app, and then AirPlay support, TV OS, Universal

01:37:48   Search, blah blah blah blah blah. Okay. That's it right there.

01:37:52   Apple was willing to do this because

01:37:56   they have something else they're trying to get off the ground, the TV app.

01:38:00   Which is really important to Apple's new subscription push services

01:38:04   push, TV Plus, it's very important to Apple that

01:38:08   the TV app become the home of where everyone goes to

01:38:12   start their TV watching it. That's like, they want that to be where

01:38:16   people's TV lives. And they don't have all the buy-in.

01:38:20   The TV app is not new, all the APIs it uses, things like the Universal Search

01:38:24   and the Siri support, all this other stuff they mention, and not to mention just the

01:38:28   integration period with the TV app. Netflix still doesn't do that.

01:38:32   I guess Amazon probably didn't before this, or I'm sure there were holes in the

01:38:36   functionality. And so what Apple needs, like, you know, it was

01:38:40   easy when Apple had all the power in their relationship. Like Apple

01:38:44   has iOS, this amazing platform that tons of people use

01:38:48   and where lots of money flows through, and if Amazon doesn't want to play nice

01:38:52   with iOS, that's mostly Amazon's problem, not Apple's problem.

01:38:56   And so Apple was able to basically dictate the terms of this in-app purchase relationship

01:39:00   for a long time, because they had most of the power in that relationship, you know, from

01:39:04   certain angles. But in the last couple years, Apple needs something from them.

01:39:08   Apple needs these companies, in specifically certain

01:39:12   areas like video, to adopt Apple's APIs and to participate in

01:39:16   Apple's TV app and that ecosystem and that whole integration

01:39:20   and everything. Apple needs them to do this. And understandably, people at Netflix

01:39:24   looked at this and were like, "Why should we do this exactly? Why should we

01:39:28   not have our own app be the place to be? Why should we integrate with Apple's TV app,

01:39:32   which is inferior in a lot of ways to our own app, and

01:39:36   then we lose all these metrics and all this data? Like, why? Why?"

01:39:40   Like, there's no reason, from Netflix's point of view, there's no reason to integrate with the Apple TV app.

01:39:44   People's home for Netflix content is the Netflix app, period, right?

01:39:48   Amazon probably, I haven't followed the Amazon situation closely in the video area, but they probably

01:39:52   had similar feelings. Like, why? What's in it for them, right? Why should they adopt Apple's

01:39:56   thing and go into Apple's playground if they don't need to, if they're powerful enough that they can have their own

01:40:00   app and have all the advantages of that? But the Amazon apps always suck. But anyway,

01:40:04   that's a different story. So finally, Apple needed something

01:40:08   from them. And Amazon had all the power on that relationship and Amazon's like,

01:40:12   "Well, no. Probably, like, why should we integrate your TV

01:40:16   app?" Right? So because Apple finally needed something from someone

01:40:20   else, for something that's very important to Apple, you know, their TV services push

01:40:24   and TV app integration and everything else, that's why I think they finally made

01:40:28   this deal specifically for this area. You know? That's why I don't think

01:40:32   we're going to see the same thing, like, in the Kindle app or Comixology, like, because I don't

01:40:36   think Apple's really trying really hard to make, like, you know, a home of all eBooks on iOS.

01:40:40   Like, I don't think that's going to happen.

01:40:42   I mean, Amazon couldn't negotiate for that because it's always, like, give and take. What does Apple,

01:40:46   how does Apple have to offer, "Hey, we'll let you do sales and rentals through the Prime Video app."

01:40:50   Amazon could say, "And also, please let us do it through the Kindle app." Like, it's another thing that you can

01:40:54   offer Amazon. But, like, that brings up, the Kindle app and the iBook

01:40:58   brings up another point. There's strategy taxes all over this because despite what you said

01:41:02   all being true, Apple also has a service that competes with Prime Video

01:41:06   called Apple TV Plus and that competes with Netflix, right? So Apple

01:41:10   is trying to be a platform and have that TV app,

01:41:14   just like you said, and they need the services that

01:41:18   people want to be in that. Netflix is big enough to still

01:41:22   tell Apple to, you know, go walk off a pier.

01:41:26   Amazon is apparently, you know, weak enough that they're willing to integrate, but

01:41:30   in the end, Apple, you know, Apple may have to decide

01:41:34   do you want to be the app that is the gateway for all of these video services

01:41:38   except for Netflix because they're too big? Or do you want to be

01:41:42   the biggest video service, right? Apple TV Plus

01:41:46   is a sibling to Prime Video,

01:41:50   Canal Plus, or whatever, all these other sort of things, like people that have video

01:41:54   services, Apple TV Plus is competing with them in the same way that the Apple

01:41:58   Bookstore, whatever the hell it's called now, no, it's not the iBook store, competes with

01:42:02   the Amazon store, with the Kindle store, right? And it's

01:42:06   like, I understand this move from a strategic

01:42:10   perspective, especially if you're trying to make Netflix feel bad about being left out, if you can

01:42:14   get everybody but Netflix into the TV app

01:42:18   and maybe give advantages to apps that integrate with the TV app that aren't available

01:42:22   to the Netflix app, like, maybe you can use this as a wedge, but at

01:42:26   a certain point, whether that strategy works or not, you've invited

01:42:30   all the foxes into the henhouse, and how well is your

01:42:34   Apple TV Plus service doing that you're paying billions of dollars to make these, you know, TV shows

01:42:38   for when you've got all these other competitors that you are helping

01:42:42   by, you know, whatever you're trying to make Netflix jealous with,

01:42:46   you're also helping competitors to Apple TV Plus in the process

01:42:50   of doing that. So it's this balance where I

01:42:54   do think that, you know, both efforts could just fizzle and go nowhere and we just get the status quo, right?

01:42:58   But if one of the things starts going really well, there may come a

01:43:02   point where Apple has to decide what's more important, Apple TV Plus becoming the next

01:43:06   Netflix, or Apple becoming the platform gateway

01:43:10   for television viewing. Like, what's the

01:43:14   we have to sacrifice one or the other, we can't have both of them, because I don't see any scenario where

01:43:18   Apple TV Plus dethrones Netflix as the most important streaming service

01:43:22   and all video services go through the Apple TV app and they become

01:43:26   the gateway and platform for all TV watching on Apple platforms. Like, I don't see both

01:43:30   of those happening. I barely see one of them happening, but certainly not both. So

01:43:34   best case scenario Apple has to choose. More likely scenario, they never have to

01:43:38   choose because neither one of them goes that well. Yeah, maybe, but I think the power dynamic

01:43:42   is so different though. Like, you know, you brought up the whole Fox and Hen house

01:43:46   argument, but I think it's more like Apple walked into

01:43:50   a Fox house. It was already an established Fox house and Apple's walking

01:43:54   in as a hen and being like, "Hey, can I hang out here too? You guys, maybe you could do me a favor

01:43:58   also?" Like, Apple had so little power

01:44:02   trying to get this TV app off the ground and

01:44:06   Apple TV Plus is such a small player, really. I mean, I'm sure Apple's going to talk

01:44:10   about numbers and how many people they have using it and everything the next time they

01:44:14   get the opportunity to, but the reality is it's a very minor player in this game right now and

01:44:18   will probably remain one for some time, if not forever. So, like,

01:44:22   Apple had to come to these companies on their terms. And so I'm guessing

01:44:26   that this has been in the works for a while.

01:44:30   Possibly even since before the Apple TV Plus launch.

01:44:34   Well, that's part of the strategy tax things. Because you never know.

01:44:38   Like, the organization is so big that someone's trying to work this deal.

01:44:42   As you said, the TV world is different. You do need to work with not

01:44:46   just streaming services, but also cable providers. There's a whole bunch of parties that Apple has

01:44:50   to deal with to do anything in this space, right? So there's one team working on that.

01:44:54   And elsewhere in this giant organization, there's another team saying, "We should have our own service

01:44:58   because we could charge people for it." And, like, you have to reconcile that somehow. It wouldn't surprise

01:45:02   me at all if there was some tension between these two efforts because they are at odds with

01:45:06   each other in many respects. Anyway, I'm guessing

01:45:10   this was in place for a while. This was negotiated a while ago.

01:45:14   Probably back when Apple and Amazon announced they reached some kind of settlement with everything

01:45:18   two years ago. I'm guessing this has been in the works since then. Because this also

01:45:22   -- so, Guy Rambo and Steve Trouton-Smith have been doing some spelunking

01:45:26   on the Amazon binary, and it has a private entitlement

01:45:30   to get extra data from StoreKit. And so there's

01:45:34   clearly some kind of, like, software side of this that they

01:45:38   had to do in addition. And given that that software side of things

01:45:42   involves the StoreKit API, which is one of

01:45:46   Apple's largest, most important, and most creaky and old

01:45:50   and infamously horrible things to work on web services,

01:45:54   I'm guessing that the software implementation side of this took some

01:45:58   time and required certain rollouts of certain

01:46:02   API changes and blah, blah, blah. So I'm guessing that

01:46:06   this has been in the works for a long time as part of that big agreement

01:46:10   that they made a couple years ago. And that's why it's just coming out now.

01:46:14   But anyway, a lot of people -- this is rubbing the wrong way because it's like

01:46:18   Apple's treating these big companies separately and giving them separate rules

01:46:22   than the rules that the rest of us have to follow within

01:46:26   AppPurchase. And you know what? That's true.

01:46:30   It's wonderful that Apple doesn't do that most of the time.

01:46:34   It's wonderful that for the vast majority of

01:46:38   aspects of the app store, big companies and small companies

01:46:42   play by the same rules. But that's not true for all the rules. That's just

01:46:46   business, you know? Facebook and Uber get away with murder

01:46:50   with what their apps do and the privacy things that they

01:46:54   take. Facebook and Uber, you know, big companies like that, like

01:46:58   they're so important that Apple can't ban them from the app

01:47:02   store. That's business. That's the reality.

01:47:06   Amazon is so important that no matter what Amazon does,

01:47:10   Apple can't kick them out of the app store. Spotify probably

01:47:14   also has this status, right? These big companies

01:47:18   are super important to Apple's platform. Way more important than

01:47:22   Overcast or any other independent app that any of us work

01:47:26   on and run and anything like that. Apple does make business deals

01:47:30   sometimes with big companies behind the scenes that do change the rules for them. Or they

01:47:34   do give them, like in the case of Facebook and Uber, they do give them like

01:47:38   maybe a private heads up from an executive, "Hey, you need to stop doing this."

01:47:42   Rather than just ban them from the app store immediately upon their first offense.

01:47:46   Big companies get special treatment because they're really important to Apple.

01:47:50   And because Apple doesn't have all the power in those relationships. Yes,

01:47:54   Apple made a special deal with a small number of big companies.

01:47:58   It's not the first time, it won't be the last time, and that's just business.

01:48:02   I don't love that as any developer who, you know, I have to follow

01:48:06   like I can't implement my own credit card thing for, you know,

01:48:10   my stuff in Overcast. I have to pay Apple's 30%. But

01:48:14   that's the reality. That's business. Apple can live without

01:48:18   Overcast. Apple can't live without Amazon and Netflix and HBO and

01:48:22   stuff like that. So they have to make deals with big companies sometimes.

01:48:26   That's just the reality. And so from that angle, you know, again,

01:48:30   I don't love this, but I see why they have to do it.

01:48:34   I think this is just, I keep fast forwarding like a year and saying like, "How could this possibly

01:48:38   shake out?" And especially if this was a thing made

01:48:42   before Apple TV+ was announced, before the rumors

01:48:46   of it were solid, like, you know, there's been, there's always a possibility

01:48:50   that Apple could do what they did with LG TV+, but it took for the

01:48:54   recent couple of years for it to really come to a head. So say this deal was negotiated three years ago,

01:48:58   Amazon had to judge. If we do this, what if Apple just comes out

01:49:02   with their own video service and competes with us? And the judgment could be, "Well, it's going to take them a while to do that, so we

01:49:06   should just do this deal anyway, and, you know, we'll get all the get-ins good. We'll

01:49:10   sell stuff to our app. We won't have to pay them 30%." And if they come out with the service,

01:49:14   it'll probably take them a while to get up to speed, and, you know, like, you can always stop

01:49:18   whatever, I don't know what the contracts look like, but I can imagine them doing

01:49:22   this while they can, making money while they can, and re-evaluate in a year or two.

01:49:26   Because even for Amazon, Amazon is also trying to compete to Netflix. They don't want

01:49:30   to be just a little serf in the Apple TV, you know, app kingdom.

01:49:34   Everyone wants to be big enough that you can just rebuff

01:49:38   Apple and say, "No, we're doing our own thing." Right now, Netflix is big enough to do that,

01:49:42   and continues to do that. Netflix took all their purchases out of the

01:49:46   Apple ecosystem, and Netflix doesn't want to be in the TV app

01:49:50   thing that tries to control everything, right? Amazon would love to be

01:49:54   in that power position, but apparently they're not. And also, I think Amazon has more things

01:49:58   they want from Apple, which kind of surprises me that they didn't get more concessions, like, you know, the Kindle

01:50:02   store or whatever, because Amazon isn't just a video company. They have all sorts of things.

01:50:06   They would love to not pay Apple for book purchases, to sell

01:50:10   books on phones and not pay Apple 30%. You can give Apple 0%,

01:50:14   right? But apparently they couldn't negotiate for that.

01:50:18   So, as the balance of power shifts, and interestingly,

01:50:22   in this particular scenario, the balance of power among these video

01:50:26   services is influenced by

01:50:30   creative output, right? If you get good TV

01:50:34   shows, that puts you in a more powerful position. It's unlike many

01:50:38   other scenarios where, like, it's a technical concern, or there's some other

01:50:42   regulatory scenario. The reason Apple

01:50:46   TV+ has been successful as it has been is A, the free trial, and B,

01:50:50   the fact that they made TV shows that weren't terrible, that people

01:50:54   were interested in watching and talked about. That's what made Netflix what it is today.

01:50:58   Netflix started by just giving a bunch of content that they didn't make, and they started making their own content.

01:51:02   In the beginning, they had one or two good ones, and now they make so much stuff that the

01:51:06   percentage of stuff that's good may only be 1%, but that

01:51:10   1% is a lot. That is a weird scenario for

01:51:14   Apple to be in. I mean, we talked about this when they were doing the service

01:51:18   to begin with. Apple has a music store,

01:51:22   but they don't have a bunch of bands that they started. Well, I guess the breakpoints, but they don't have

01:51:26   a bunch of bands that Apple is fielding, but they're making TV

01:51:30   shows. It's weird to see Apple suddenly in a

01:51:34   scenario where you can really influence your odds of success

01:51:38   by doing things that have nothing to do with technology and everything to do with

01:51:42   creative output. It's like, how do you become a powerful movie studio? You make good

01:51:46   movies that people want to buy or people want to watch. Pixar is

01:51:50   Pixar because they made a bunch of good movies. Now, Apple can,

01:51:54   in theory, try to get a leg up on its competitors in the

01:51:58   studio market. Yes, there's the technology stuff going on over here. Yes, there's the TV app and

01:52:02   there's all sorts of angles of control. We control the store and there's all those things that they normally

01:52:06   do. But also, A, they have to make good shows

01:52:10   and B, if they make really, really good shows and a lot of them, it can give them an

01:52:14   advantage, which is a strange situation

01:52:18   for a technology company to be in, but it's a fact of life.

01:52:22   Netflix became Netflix not because their app is awesome. Hell, they were able to

01:52:26   voice stupid auto-playing trailers or whatever on us

01:52:30   before they finally gave us a preference for that. The reason they were able to annoy us to that degree

01:52:34   is Netflix is not Netflix because of their app. They're Netflix

01:52:38   these days because of their original content and that's what Apple is competing with ultimately

01:52:42   in the end. For Apple TV Plus to even be a contender,

01:52:46   it has to have good enough content and if it ever wants to

01:52:50   get bigger and start taking market share, it has to have better content.

01:52:54   Thanks to our sponsors this week, Squarespace, Eero, and Postmates

01:52:58   and we will talk to you next week.

01:53:02   Now the show is over, they didn't even

01:53:06   mean to begin, cause it was accidental

01:53:10   Oh it was accidental

01:53:14   John didn't do any research, Marco and Casey wouldn't let him

01:53:18   Cause it was accidental

01:53:22   And you can find the show notes

01:53:26   at ATP.FM and if you're

01:53:30   into Twitter, you can follow them

01:53:34   @C-A-S-E-Y-L-I-S-S

01:53:38   So that's Casey List M-A-R-C-O-A-R-M

01:53:42   Anti Marco Arment S-I-R

01:53:46   A-C-U-S-A Syracuse

01:53:50   It's accidental, accidental

01:53:54   They didn't mean to accidental, accidental

01:53:58   Tech podcast so long

01:54:02   John do you want to tell me about consoles? Cause I'm really excited to hear about it

01:54:06   Yeah this is console follow up and after show

01:54:10   Do we have after show follow up? That's right, well it's not really follow up

01:54:14   I guess it is, it's stuff that I forgot to mention or didn't know, so last time I talked about

01:54:18   the new Xbox and the PlayStation 5 and I hadn't watched the full Sony presentation

01:54:22   and so now I have, but I did talk about it last time

01:54:26   that Microsoft had decided against the trends in the industry

01:54:30   to lock the clock speed on their

01:54:34   system on a chip, essentially, instead of having a turbo boost

01:54:38   or going faster and slower clock speed depending on load and all the stuff that modern

01:54:42   CPUs do, they had it just stay at a constant clock speed, it's actually two clock

01:54:46   speeds because if you enable symmetric multithreading it goes slower or faster

01:54:50   I forget which, but anyway, the point is you're in a particular mode and it just runs that speed the whole time

01:54:54   no varying while it's running in whatever mode it's running in

01:54:58   which is super weird and requires aggressive cooling

01:55:02   but it makes a very consistent experience, like that was their pitch, we don't want you to have the console

01:55:06   if it's in a place where it can't get a lot of cooling your game starts hitching

01:55:10   Sony took a different approach, in some ways a more traditional

01:55:14   approach, their thing does change clock speed, but they spent a while on their presentation

01:55:18   talking about how it doesn't change clock speed based on temperature

01:55:22   they didn't do a good job explaining, or at least I didn't do a good job absorbing why

01:55:26   they don't want to do it based on temperature, the idea was like, well, temperature is

01:55:30   one thing, but it can be a false signal maybe

01:55:34   like, really, we don't want it to

01:55:38   like, here's the thing, this is the reason I'm confused, you don't want

01:55:42   your chip to melt, like temperature is important, like in the end that's what you're controlling for

01:55:46   you know, if the thing is in the console under your TV

01:55:50   and it's in a cabinet where it doesn't have good airflow, and you're playing a game

01:55:54   and the clock speed is whatever, 2 point something gigahertz

01:55:58   and stuff starts to get so hot, at a certain point

01:56:02   chips stop working, like there's a safe temperature for chips operating in, if they get too hot

01:56:06   they just don't work anymore, and worst case scenario you could even damage them, right?

01:56:10   so in the end that's what you're controlling for, heat is the enemy here

01:56:14   but I guess apparently you can't

01:56:18   get accurate enough or reasonable enough temperature sensing

01:56:22   to just have temperature be the only thing

01:56:26   that you're measuring, or if you did you would end up clocking yourself slower than you technically needed to

01:56:30   so it seems like what they're looking for is a situation where they

01:56:34   run as hot as they possibly can, but not

01:56:38   hotter, and they think the way that they can do that

01:56:42   is instead of controlling for temperature, they have sort of assigned

01:56:46   cost, sort of heat cost to certain

01:56:50   operations, and they have special circuitry to say, if you're doing this

01:56:54   type of operation, it takes this much of the heat budget, and this type of operation takes this much of the heat budget

01:56:58   and they sort of real time measure what the actual CPU

01:57:02   and GPU are doing, and from that come up with

01:57:06   they try to assemble that all together to

01:57:10   come up with a profile of how much energy are we using

01:57:14   and then just, they have sort of a balanced equation that says we can do this amount of work

01:57:18   with, this is the amount of cooling we have, and this is how

01:57:22   our cooling relates to our clock speed and we can do this amount of work with this amount of heat, so they're

01:57:26   trying to, I guess get closer to that ragged edge

01:57:30   don't back off just because this one part's going to get really hot for a second

01:57:34   because we know based on the workload across the entire system on a chip that you'll be fine

01:57:38   because overall we're not doing that much work, so

01:57:42   a very different strategy from Microsoft, which committed fully

01:57:46   to just constant clock speed, again two different clock speeds for whether symmetric multithreading

01:57:50   is enabled or not, and Playstation 5 is saying we're going to be variable but we're going to do

01:57:54   variable clock speed better than anyone else has done it, so good luck to

01:57:58   Sony, it was a little bit scary, I hope it works out better than my

01:58:02   description of it made it sound. And the second weird thing Sony is doing, Sony being Sony

01:58:06   they have this big part of the presentation still about 3D audio, or

01:58:10   I don't know, audio that sounds more interesting in games, having more different audio

01:58:14   sources, I'm fully sold on the idea that this can be more immersive

01:58:18   because sound, you don't appreciate it until you've played

01:58:22   a really good game with sound, but point sources and sound

01:58:26   in games, in first person shooters and stuff you want to hear footsteps of enemies

01:58:30   coming, are they behind me, are they to my left, to my right, sound can be super important

01:58:34   but there's a small number of those sound sources and localizing them is difficult

01:58:38   most interestingly, most

01:58:42   movie based surround sound things

01:58:46   only support a small number of sources, because movie soundtracks when they're mastered

01:58:50   I say a small number of sources, you know they have like

01:58:54   7 speakers or whatever around you, or whatever it is, 8 or 9, there's a lot of speakers around you

01:58:58   but it's not the number of speakers that determines the number of sources, the sources are like

01:59:02   how many different individual point sources of sound can there be in the

01:59:06   scene, and I think the movie ones are like 32 or something, it's a big number

01:59:10   but it's not a huge number, and Sony wants to have a situation where you can have like

01:59:14   5000 point sources of sound, the example they gave was like if you're in the middle

01:59:18   of a rainstorm and each raindrop was its own sound source with a drop hitting the ground, that's what it's like in real life

01:59:22   each raindrop that hits the ground is its own sound source, whereas

01:59:26   in games today, they just record the sound of the rain and put it in a channel, or make it

01:59:30   come from all around you or whatever, but that rain sound is like flattened out

01:59:34   so anyway, Sony's super into the 3D audio, they want to have

01:59:38   this new audio engine, they have dedicated hardware for it, it's very powerful, they've spent time

01:59:42   bragging about it, Marco you would like this part of the presentation, you should watch it, they talked about

01:59:46   ring buffers and all sorts of... I'm a stereo guy

01:59:50   well that's why I think you'll like this part of the presentation, first of all, first they talk about hardware

01:59:54   that's tailored for processing sound, which you must be familiar with, like in terms of

01:59:58   how is sound processing different than general

02:00:02   computing, so they made a compute unit that technically has as much power

02:00:06   as like the entire Playstation 4 CPU

02:00:10   but it's just for sound, and because it's just for sound, you could remove all the caching and everything

02:00:14   because you just want the data to flow through it, right, so it's very small and purpose-built

02:00:18   anyway, yeah, so they have dedicated hardware for the sound and

02:00:22   it allows them to have these thousands of sound sources and stuff like that, and the

02:00:26   they're also focusing on output devices, as you would imagine, and their first output device

02:00:30   that they're focusing on is not a surround sound system or a soundbar or whatever, it is

02:00:34   headphones, because this system is not about "hey, how can we play

02:00:38   sound on a bunch of different speakers?" in fact that makes it more difficult for them, you have

02:00:42   two ears, and they just want the sound to go into your two ears, they can

02:00:46   simulate all these sound sources and figure out the culmination

02:00:50   of these sound sources, what sound hits your ears at what time

02:00:54   and that's their first use case, to make you feel like you're

02:00:58   really there just with headphones, just with stereo sound going directly into your ears

02:01:02   the tricky part about that is how do your ears

02:01:06   tell sound is like exactly to my left, or a little bit behind me

02:01:10   or up or down, like all the sort of fine details of where the sound comes from

02:01:14   if you just try to do that with timing, or when the sound hits your left ear

02:01:18   and your right ear, you can get pretty far with that, but it's

02:01:22   difficult, there is a thing in the world of sound design

02:01:26   that Sony talked about a lot in their presentation, which is called

02:01:30   HRTF, which stands for Head-Related Transfer Function

02:01:34   and it's basically, what does

02:01:38   your head do to the sound that affects how you hear it

02:01:42   and your head being like all of the stuff that's attached to your body

02:01:46   surrounding your eardrum, which senses the sound. In other words, it's the shape of your

02:01:50   head and your ear, and to figure out

02:01:54   every individual person has their own head-related transfer function, which is like when sound comes and it hits your head

02:01:58   how does it bounce off all the little wrinkly weird parts of your ear and stuff

02:02:02   and get inside your head to your eardrum and you hear it

02:02:06   and so what they're trying to do is

02:02:10   come up with a set of head-related transfer functions, kind of like the AirPod exercise

02:02:14   you've got to come up with a thing that fits in everybody's ear, but it can't fit in everybody's ear, so they've got to come up with

02:02:18   a set of head-related transfer functions that is representative of most of the

02:02:22   population, and they're thinking of like, maybe you could have a game where you try each one

02:02:26   and you try to pinpoint where the sound sounds like it's coming from and the one that is

02:02:30   closest to the real virtual sound source is the one that you pick

02:02:34   they even talked about maybe you could take a picture of your ear or a video

02:02:38   of your ear and send it out over the network and it would make

02:02:42   a custom head-related transfer function for your ear. The way they make the real ones

02:02:46   by the way, they showed this in the video, is they put little sensors way deep in your ears

02:02:50   and then they make you sit in a fixed position in this giant sound chamber

02:02:54   and they play point source sounds from all over the place and then they record from

02:02:58   inside your ear what it sounds like to the inside of your ear

02:03:02   right, and every different person has their own head-related transfer function that's

02:03:06   custom to them, so if they can do a custom one for individual people, like they're

02:03:10   not going to put you in a sound chamber, but if they know the shape of your ear they can sort of use

02:03:14   machine learning to simulate the head-related transfer function

02:03:18   who knows if they'll ever do that, I think what they'll just do is ship with say

02:03:22   20 different head-related transfer functions that are representative of a range of human ears

02:03:26   and then have some kind of game-like experience where you pick the one that is closest to your ear

02:03:30   but it's fascinating, they spent so much time in the presentation

02:03:34   on this, it could all come to nothing, but it sounds super

02:03:38   cool, like I'm all ready to be in a rainstorm where every raindrop is its own

02:03:42   sound source and I'm all ready to be able to hear footsteps of approaching people

02:03:46   in a first person shooter with much more accuracy than I can hear it right now

02:03:50   so I once again recommend to all you people look into this, these videos

02:03:54   we'll link them again, if you're not interested in consoles they may sound boring, but check it out, cool tech stuff

02:03:58   yeah, so first of all I think there's two angles of this, I actually, both

02:04:02   of these things I think are actually kind of fascinating, so the

02:04:06   heat management of the processor thing, I think actually based on

02:04:10   your explanation I think I do understand what they're doing, and it's actually kind of ingenious

02:04:14   so if you just fix the clock speed like what Microsoft is doing, you lose

02:04:18   out on potential performance and potential power savings

02:04:22   because you could, like, most

02:04:26   chips have a little bit of headroom, they can run faster for a short

02:04:30   time, so they can do certain things faster for short times

02:04:34   and so if Microsoft is not allowing that because they can't sustain

02:04:38   that in a predictable way, because that's

02:04:42   the whole goal here is, with the game console you want extremely predictable

02:04:46   performance for the developers, that way you can make games that work the exact same

02:04:50   way on every Xbox 17 and every PS5

02:04:54   or whatever, and it's fine, right, and that's one of the great

02:04:58   advantages of working on game consoles, is that you have predictable hardware, which you don't have

02:05:02   in the PC world necessarily, so the way Microsoft is

02:05:06   making theirs predictable sounds like they're just fixing the clock speed, so it always runs at this speed

02:05:10   period and, you know, the dev kit machines that you're going to write the games on

02:05:14   are going to be running at the exact same speed as all the consumer machines out there

02:05:18   well it certainly sounds like they're doing the exact same thing, just in a different way

02:05:22   but they're achieving the same result. What you want is predictability

02:05:26   not necessarily constant clock speed, so as long as, so if they

02:05:30   know that they can boost the CPU up to this rate for this

02:05:34   operation, up to this many times per millisecond or whatever, they

02:05:38   can build that into the CPU's

02:05:42   power management unit or whatever, however this is working at the low level

02:05:46   they can have it be, like run these heuristics and clock itself up and down

02:05:50   instead of being based on temperature, based on what it's doing, so it

02:05:54   achieves many of the benefits of dynamic clock speed

02:05:58   but in a predictable, deterministic way

02:06:02   so that every single PS5 will run this at the exact

02:06:06   in the exact same way with the same performance characteristics.

02:06:10   Yeah, that's exactly what they said in the presentation. Yeah, that sounds awesome. Right, so it does

02:06:14   but as I said, you can pretend that that's true, it's like

02:06:18   well this will always be exactly the same, because it's a deterministic state machine, I guess that probably changed the

02:06:22   profile, but anyway, it's deterministic, but the thing that's your enemy

02:06:26   is heat, and heat doesn't care about the repeatability of your

02:06:30   game thing. If your PlayStation 5 is in a really hot spot

02:06:34   and the power thing says, I'm going to do the same thing I did, the same thing I

02:06:38   always do based on my estimation of the cost of these operations, I believe we can run at this

02:06:42   speed, and it may be wrong, and you make it too hot

02:06:46   and getting your GPU too hot or your CPU too hot

02:06:50   can make bad things happen. That's why people go off temperature, because ultimately that's what you're trying

02:06:54   to control, so I totally get the pitch, right, but it scares

02:06:58   me to think that no matter what temperature, I mean I'm sure there's some temperature cut off on the top

02:07:02   of it, but no matter what environment this chip is in, it is going

02:07:06   to do exactly the same thing it did during development, because whatever

02:07:10   the fixed profile is for the cost of each operation, and that may not be

02:07:14   the appropriate thing to do. Now you could say it's the same situation for a fixed clock speed, because if it's fixed clock

02:07:18   speed, and you put that in a really hot situation, how do you know it's safe to go at that clock speed?

02:07:22   Well, we think our cooling solution can handle yada yada yada, but you might be wrong. So I

02:07:26   suppose in the end it's the same trade-off, it just seems to me that what

02:07:30   Sony wants is, like you said, to be able to use more of the

02:07:34   flight envelope to use aviation terminology. There's a certain range of

02:07:38   operations that are safe at any given time. Can I go a little

02:07:42   faster now? Do I have to go a little slower? They want to fill that envelope from top to bottom,

02:07:46   and Microsoft is the more conservative approach, they're saying we're going to draw a straight line, that's the

02:07:50   envelope, we hope our cooling will always keep you under that line,

02:07:54   but you're just going to be at that speed all the time.

02:07:58   In the end they're both trying to do the same thing, but totally different approaches, and

02:08:02   honestly they're both a little bit scary to me, now that I think about it.

02:08:06   Well, but I'm sure neither one of them would reach the point where it would kill itself, right?

02:08:10   I'm sure every chip... Talk to Xbox owners.

02:08:14   Well, yeah. The red ring of death was exactly a heat management

02:08:18   issue. That's true. But in general, any modern

02:08:22   chip from the last 15, 20 years has a thermal shutdown

02:08:26   protection built in, so that if it's running above some maximum temperature,

02:08:30   usually it's like 100 degrees Celsius, something like that, whatever it is, it has a thermal shutdown

02:08:34   limit where it will just shut itself down. The machine will crash and it'll just

02:08:38   that's it. It will refuse to fry itself, basically.

02:08:42   But that's different from usually from the temperatures that it's going to

02:08:46   throttle itself back clock speed wise at. But anyway, I think this works the same way,

02:08:50   just using different metrics. Either way, when they're designing

02:08:54   an enclosure and a system and everything, they probably have

02:08:58   a certain heat budget of how many watts are we going to dissipate from the cooling

02:09:02   solution? So how many watts can the CPU put out and remain

02:09:06   within the design here? And either way, it's like

02:09:10   if we run this at a constant clock speed, we know it's going to be x watts maximum

02:09:14   no matter what you do to it. And I think what Sony is talking

02:09:18   about sounds like a very similar kind of system. Just they know

02:09:22   the power cost of everything so they can know roughly what wattage

02:09:26   of heat they are generating. So it doesn't

02:09:30   I think it's kind of the same. You get the same output

02:09:34   or rather you have the same design constraint of like you need a cooling solution

02:09:38   that will be able to dissipate x watts of heat in

02:09:42   most conditions to keep the temperature under y.

02:09:46   That's the design no matter, in either case. And if the

02:09:50   Sony chip's power management stuff is intelligent enough to not rely

02:09:54   on measuring the temperature and to just be like, look, I know that if I do

02:09:58   this sequence of operations, I will generate this many watts

02:10:02   of heat output. And that's kind of cool. It's a very clever way to do it.

02:10:06   And in a situation like this where like predictability of performance is

02:10:10   more important than either eking out every last

02:10:14   like 5% of performance at the top end or

02:10:18   having a certain ideal temperature that you always stay at, that actually makes a lot

02:10:22   of sense. It's kind of clever. One other factor to throw in here

02:10:26   this is not the part of the presentation that Sony wanted to emphasize, but Sony has way

02:10:30   fewer GPU cores than the Xbox. They're both using AMD parts. They're both using

02:10:34   very similar, if not exactly the same CPU

02:10:38   and GPU cores with minor tweaks. But Sony has, I forget what the numbers are, but it's like

02:10:42   they have 30 something and the Xbox has like 50 something. So it's interesting

02:10:46   that the machine with just more to cool and

02:10:50   you know, like nearly double the number of GPU cores is the one using

02:10:54   the fixed clock speed. We haven't seen what the PlayStation 5 looks like. The Xbox

02:10:58   cooling solution is no joke. It is a really big heatsink with a

02:11:02   humongous fan in it. So, you know

02:11:06   the PlayStation 5 seems like with this variable thing that they could get away

02:11:10   with a console that has a less gargantuan cooling

02:11:14   solution. But we'll see. I don't think anyone knows what that machine looks like yet.

02:11:18   So, moving on to the audio thing. I haven't watched this presentation

02:11:22   yet, but there's a lot of overlap here with binaural audio.

02:11:26   This is something that's existed for a very long time, decades.

02:11:30   I think people discovered binaural audio like in the 70s or 60s, like it's been around for a long time.

02:11:34   But the idea is, as you mentioned, we have two ears

02:11:38   but we hear in 3D. How can this be when stereo

02:11:42   sound is allegedly only two dimensional? And the answer

02:11:46   is very complicated. And the idea is like

02:11:50   our ears are shaped in a certain way, the sound

02:11:54   travels through them in a certain way, our head has this volume in space

02:11:58   that affects sound that we hear in certain ways. It's all very complicated.

02:12:02   But if you place

02:12:06   two microphones in a space, like suppose you want to

02:12:10   record a concert hall. You put two microphones next to each other

02:12:14   about a head's width apart, and you play that back

02:12:18   on headphones or even on regular speakers, it'll sound roughly like there's an orchestra

02:12:22   playing in front of you. It's not going to be incredibly precise, but it'll sound great.

02:12:26   It'll sound, you know, approximately realistic. But it's not going to

02:12:30   be super precise 3D sound. But

02:12:34   if you have microphones that

02:12:38   basically are in simulated or real ears

02:12:42   on the sides of a simulated or real head, and that have

02:12:46   like the shape of the ear around it, like, because you know,

02:12:50   obviously part of your ear is outside of your head, the whole shape

02:12:54   of your ear affects the shape of the sound waves coming in and it alters the sound.

02:12:58   So if you can have a very like

02:13:02   human ear and human head like recording device,

02:13:06   you can record binaural audio, which is a style of recording,

02:13:10   that like the two microphones are in

02:13:14   these ears, and the idea is then if you play it back

02:13:18   with headphones that are in basically the same spot in your ears,

02:13:22   relaying basically the same sound that hit them, which was partially altered

02:13:26   by the head and ears around the recording microphones, then

02:13:30   it will sound much more precisely 3D. And this is a wonderful

02:13:34   thing, if you just go to YouTube or wherever, search for binaural

02:13:38   audio and you'll find tons of recordings of binaural things,

02:13:42   if you've never experienced this effect, it's

02:13:46   quite something. Like it's kind of mind blowing how incredibly

02:13:50   3D and realistic audio can be when it's recorded in

02:13:54   this way. Now the way this is usually recorded is either

02:13:58   that you can actually buy a fake head with fake ears that have

02:14:02   microphones in it, they're extremely expensive, I think they start at like

02:14:06   $800, like they're very expensive, and

02:14:10   this is actually, the market for these is actually increasing now with VR, there's like 3D microphones for VR

02:14:14   stuff, but anyway, you can buy like a whole fake head, or you can do what

02:14:18   I did to play around with this for like less than a hundred bucks,

02:14:22   you can get what look basically like earbuds.

02:14:26   But, and they are earbuds, but on

02:14:30   the outside of each one is a microphone, facing out, and the inside

02:14:34   is headphones, earbuds facing in, and these are inexpensive because they're basically

02:14:38   just earbuds plus microphones, they're inexpensive, and they use

02:14:42   your own head to make your own head transfer function because you're just having

02:14:46   earbuds in your ears, and so you can, for not that much money, experiment

02:14:50   with this, where you can basically record, you know,

02:14:54   yourself in a room or walking through the

02:14:58   world or whatever, record exactly the way your ears inside

02:15:02   your head are receiving the sound with those microphones that are on the outside of the earbuds.

02:15:06   And then you play it back, wearing those same earbuds, or pretty much

02:15:10   any other headphones, but especially when you wear those same earbuds, and it sounds

02:15:14   remarkable, like you can't believe how much it feels like

02:15:18   you are actually there again, it's spooky how accurate it is.

02:15:22   It's really cool. So some of these things were way inside your ears, because

02:15:26   the earbuds necessarily have to be out of your ears, so they're catching the sound

02:15:30   in your ear, right? So the Sony ones, if you look at the video, are like

02:15:34   these little wires going into the poor person's ear canal, right, so that's what they're

02:15:38   using to build their standard set of 20 head-related transfer

02:15:42   functions. But yeah, you won't have one custom for yourself unless you go into that Sony

02:15:46   chamber, but I think games are an interesting scenario, and you mentioned VR,

02:15:50   it's similar, where the positioning is the most important part.

02:15:54   People aren't really cared about the fidelity

02:15:58   of the footsteps sounds you hear. You want to know exactly where the footsteps are coming from, or even

02:16:02   just like environmental stuff where rain is falling on cobblestones,

02:16:06   car tire going across pavement versus going across wood, someone

02:16:10   crunching on gravel, are the walls made

02:16:14   of wood, are they made of metal, how high is the ceiling in this room?

02:16:18   Those type of things, this is all about both how the sound gets into

02:16:22   your ears, where it's coming from, but also how does the sound bounce around, that's why they have this dedicated

02:16:26   audio engine to have 5,000 sound sources. So if you

02:16:30   get, if you're in an environment and something happens, not only does it

02:16:34   happen in a particular place, but the room affects how it sounds. Games

02:16:38   have had this forever, like even going back to Mario 64 or probably earlier,

02:16:42   or think of any driving game where you go through a tunnel and they change the audio, like they just put some filter

02:16:46   on the audio so it sounds different when you're in a tunnel. Yeah, some reverb filter.

02:16:50   Yeah, we're at the stage now where instead of just having a filter when you go through

02:16:54   a tunnel, like every sound source will be affected by the surrounding

02:16:58   geometry in the game at all times, right? And so the tunnel, you don't have to write any

02:17:02   special code for the tunnel, you just make a tunnel and you drive through it and you get the sound.

02:17:06   And so, like, this part of the presentation, it's hard to do a presentation

02:17:10   about sound, because people just want to see graphics or want to hear you talk about graphics,

02:17:14   but I honestly think that if this stuff works and developers actually use

02:17:18   it, it could be, and this is certainly what Sony hopes,

02:17:22   as big a differentiator for the Sony platform as graphics have been

02:17:26   in the past, and Sony better help, that's the case because their graphic power is

02:17:30   significantly less than the Xbox. Marco's

02:17:34   going to buy a PS5 just so he can hear this audio, and so he can play the head-related transfer

02:17:38   function game, and pick the one that matches his weird ears.

02:17:42   I had such an amazing sound experience with

02:17:46   what's the Mario game for the Switch?

02:17:50   Odyssey? That's it, yeah. I kept thinking Galaxy, I knew that was wrong. Yeah. Mario Odyssey has amazing

02:17:54   sound. And that's without doing any of this fancy stuff. Right. Like, that's just

02:17:58   doing the old way of, like, just programming it so that, oh, when you go into the 8-bit mode, it plays the 8-bit

02:18:02   sound. When you go underwater, it muffles the other sound. It's like, it's just doing the

02:18:06   old way, but man, that is such a great game for sound.

02:18:10   Having this technology is good because Nintendo can afford to do that, like, the old

02:18:14   manual way, essentially, because they have a lot of money and this game's going to sell a million copies.

02:18:18   But if you got, like, basically for free by just placing sound sources

02:18:22   and defining materials, you got that, you know, without having to

02:18:26   hire armies of people to manually set up all your sound, that's, hopefully

02:18:30   will make every game have better sound, not just the ones that can afford to

02:18:34   manually mess with sound in every possible environment, and record

02:18:38   500 different footsteps depending on whether you're walking on these different materials, you know.

02:18:42   Right, like, it's almost like back in the days, like, when 3D was just

02:18:46   getting started, like, if you wanted to make a 3D game, you generally had to write your own 3D

02:18:50   renderer, right? And then eventually, as technology got better, and the

02:18:54   APIs got better, and the hardware got better, you could then eventually just say, alright, just put a

02:18:58   triangle at this geometry, and put the camera at this geometry, and you figure out how to make that

02:19:02   happen, right? And then, of course, eventually now nobody writes their own engine anymore, but, you know, the point

02:19:06   stands, like, there hasn't been things like that for audio that were very

02:19:10   advanced, and so this sounds like a way to do that, which sounds very exciting.

02:19:14   And the differentiator, I think, what Sony hopes is, like, Xbox can't do this, because

02:19:18   if you try to do this sound processing on a general purpose CPU, you will use all your

02:19:22   CPU power, like, so their brag was their audio engine on the PS5,

02:19:26   you couldn't, you'd have to absorb every compute resource on the PS4

02:19:30   to match it, because it is a purpose-built sound processing engine, that's all it does.

02:19:34   And I don't think the hardware exists at all on Xbox, so Xbox can't really compete

02:19:38   with this, because they can't spare the cycles to dedicate

02:19:42   to audio, and they don't have dedicated hardware for it.

02:19:46   (beeping)