0:00:00Good evening, you beautiful geeks! Welcome to San Jose's historic California Theater.
0:00:13We are thrilled to once again be live and in person at Apple's Worldwide Developer Conference
0:00:192024. Now won't you please silence your devices, then open your ears and put your hands together
0:00:30for our host, Jon Gruber.
0:00:37Hello! Welcome to the talk show. I am your host Jon Gruber. Most of you here are probably
0:00:52familiar with me. Welcome! I was told we were backstage and the House Sound guys were like,
0:01:01"This crowd is really loud. What do you do?" It is great. The crowd is always great here.
0:01:11It does feel a little extra energetic and with good reason. I think we're going to have
0:01:16a very good show. But before we get to the meat of the show, I have some very special
0:01:23people to thank. These are our sponsors, without whom we would not be here, trust me. They
0:01:30are amazing sponsors, three of them. The first amazing sponsor is iMazing. iMazing 3 is the
0:01:40all new version of the world's best iPhone manager for both Mac and Windows. Since the
0:01:47iPhone's release way back when, the iMazing team has been delivering the missing features
0:01:53you wish existed working for your iPhone, iPad, or if you still have one, an iPod and
0:01:59your computer. I don't know, Phil Schiller obviously still has an iPod. I know he flies
0:02:05with it. Version 3 of iMazing features a brand new user interface. It is so beautiful. It
0:02:12is just a classic indie Mac app, beautiful, simple, gorgeous, makes you want to use it
0:02:20just by looking at it. Version 3 has this new user interface. It is easier than ever
0:02:24to discover, to explore, to get these features. But iMazing gives both individuals, families,
0:02:31it looks very consumerish, but it is very powerful. Small businesses and even large
0:02:36enterprises can use it to manage their devices and efficiently manage Apple mobile devices.
0:02:43It includes powerful local backups with incremental snapshot support, text message extraction
0:02:51options, whether for compliance or keepsakes. You can tell from that word compliance, probably
0:02:56not about families, a little bit more for the enterprise. But it is seriously, it is
0:03:01one of those tools that scales from you to really big companies. Versatile data extraction,
0:03:09file transfer, it is natively coded for the Mac by a passionate team in Switzerland for
0:03:14an unparalleled no compromise experience, trusted by millions of users all over the
0:03:20world. Check out iMazing 3 today and save 20% by visiting iMazing.com/thetalkshow. That
0:03:31is iMazing.com/thetalkshow. Second, my good friends at Flexibits, the makers of Fantastical
0:03:46and Cardhop. These are, you guys probably have heard of them, these are amazing. Fantastical
0:03:54is a calendar, Cardhop is a contact manager, absolutely gorgeous, previous Apple design
0:04:01award winner and they are for every platform Apple makes. Of course the Mac, iPhone, iPad,
0:04:10they even have right on day one a fantastic version of Fantastical for Apple Vision Pro.
0:04:20Really gorgeous. Now I'm not naming names, but there are other companies that don't have
0:04:25a calendar app that is native for Vision Pro. Fantastical was native on day one and it is
0:04:33gorgeous. How Fantastical for Vision did not win an Apple design award this year, I do
0:04:39not know. I got to find Galenzi in the audience after the show and petition him, I don't know,
0:04:45must be a back story there, but it is beautiful, beautiful apps. They also have Flexibits Premium
0:04:52is their subscription service. You pay for Flexibits Premium and you get it all. You
0:04:57get Cardhop and Fantastical for all the platforms. You don't just pay per platform, you just
0:05:03pay one thing, you get them all. They are great apps, they work together very well and
0:05:09they have been using the fact that it is a subscription service to build out an infrastructure
0:05:15on the web to do things like doing the thing where you pick a time for an event and people
0:05:25coordinate that sort of thing. All super, super secure, they only send the data that
0:05:30they need to the cloud, they don't keep anything they don't need, but to coordinate picking
0:05:34time, stuff like that, you go to the web, you send an invitation, people can say, "Oh,
0:05:39I'm free at 2, I'm free at 2, he's free at 3, everybody's free at 2.30, then we'll do
0:05:45it at 2.30." That sort of stuff, all they're built into Flexibits Premium service, it is
0:05:51fantastic. And serious, serious stuff like integration with Microsoft 365 and totally
0:05:57fun stuff like stickers that you can put in messages. I mean, the whole scale from totally
0:06:02serious enterprise to putting stickers in messages with the fantastic mascot. Really,
0:06:09really great. New and existing customers with this special deal for this show, new customers
0:06:15get 20% off when you start a subscription. Lots of people who watch my show probably
0:06:20already subscribe, what's in it for you? Well, guess what? Go to this code, go to this URL
0:06:25and when your renewal comes up, just by having gone to this, you'll get 20% off when you
0:06:31renew. Everybody gets 20% off, new and existing, by going to flexibits.com/wwdc2024. That's
0:06:39F-L-E-X-I-B-I-T-S flexibits.com/wwdc2024. Third, Flighty. How many people here already
0:06:58use Flighty? I use Flighty, it is fantastic. Flighty, another Apple Design Award winner
0:07:10back in 2023, way back last year. Every download of Flighty, you get the pro features complimentary
0:07:19for your first flight. You do not need to sign up for a subscription and then cancel
0:07:23it or anything. Try Flighty next time you're flying. You get all the pro features, you
0:07:29can see what it's like. I guarantee you, you will sign up after you've flown with Flighty
0:07:34compared to flying without it. Every year, there's a new record for problems with flights.
0:07:41I mean, I don't think I'm making this up here. Flying is getting worse. Last minute delays,
0:07:50more disruptions, plus the airlines are secret and they don't really want to keep you up
0:07:56to date. Flighty hooks into all of the data sources and APIs that the airline industry
0:08:01has and keeps you up to date. I cannot tell you just in the last year how many times Flighty
0:08:05has told me about gate changes. It's not just like, oh, you go from A-10 to A-8, but when
0:08:10you go from gate A-23 to gate D-12. It's like, oh, that's 20 minutes away in the airport.
0:08:18Flighty tells you as soon as it happens. You're not at the wrong end of the airport. It is
0:08:21absolutely fantastic. They've got great features for sharing your flights with friends or friends
0:08:29sharing your flights with you. So if somebody in your family or dear friend is traveling,
0:08:34put their flight into Flighty and you get everything you need to know and of course
0:08:38hooks up to all of the latest stuff that you would want them to. Live activities on the
0:08:43home screen, everything you want. It is so fantastic, so beautiful. Honestly, the airlines
0:08:50just want you to sit quietly and just be uninformed at the gate, get there three hours early and
0:08:54just sit there on your iPhone. No, get Flighty. You will be there when you need to be there
0:09:00without wasting time and you will be informed of everything you need to know. Go to flighty.com/tts.
0:09:10The talk show, TTS, flighty.com/tts. My thanks to all of those sponsors. And now with no
0:09:26further ado, I have two very special guests to introduce. Ladies and gentlemen, let's
0:09:34hear it for Craig Federighi and Greg Joswiak. Craig. >> Fired up? I'm fired up. >> You
0:09:47shrunk. >> I'm not really Craig Federighi. I can never pull off the real hair. Anyway. All
0:09:55right. Let's bring up the real Craig. >> I think they got both Craig Federighi and Greg
0:10:07Joswiak at the same time. >> They did. It's a twofer. >> That was fantastic. >> That's as
0:10:16close as Craig hair I'm ever going to get to. >> All right. >> Going in the opposite
0:10:21direction. >> How many people in the audience saw last year's show? How many people remember
0:10:29the end of last year's show? Mr. Federighi is one hell of a guitar player and did some
0:10:42shredding. We don't have anything like that today. Now, how many people here were at Apple
0:10:50Park for the keynote yesterday? And I'm sure for everybody watching in post on YouTube,
0:11:01I'm sure everybody has watched the keynote. But for those who were in Apple Park the last
0:11:07couple of years since this new format has been out, the keynote starts right at 10 a.m.
0:11:12but around 955, Tim comes out, welcomes the crowd, has Craig come out to also say hello
0:11:18to the crowd, and Craig comes out and fibs to the audience and tells them that there
0:11:28are no gags, there are no -- there's no zany opening movie, and none of that is true. Well,
0:11:38I'm an honest man, and I'm telling you, I swear to you, there is no stunt at the end
0:11:44of the show. We spitballed ideas. There was an idea to maybe have Craig parachute from
0:11:58up there. We have a helmet. >> That helmet is heavy, by the way. >> But I am to understand
0:12:06there was some concern about the helmet and the hair. >> I mean, not especially. >> But
0:12:12I thought it looked especially good on Josh. >> Yeah, I had cool Craig hair. Hey, we do
0:12:17have some gifts for you. You have to declare those on your taxes. They are two patches
0:12:22for WWDC. >> I have one more comment about the opening. I thought the opening movie of
0:12:35the keynote was fantastic. I mean, I think you guys always -- [ Applause ] I mean, it
0:12:48looked realistic. You did not jump out of an airplane. >> Whoa, whoa, whoa, whoa, whoa,
0:12:54what the hell, Gruber? What are you implying? >> But here's my real question. My real question,
0:13:03number one, how great was it to see captain Phil Schiller flying the airplane? [ Applause ]
0:13:12>> I will tell you that stuff wasn't the only take. >> Well, that was my question. Are there
0:13:25alternate takes where you didn't say I'm getting too old for this stuff? >> That was the G-rated
0:13:30version of that take. >> You know Phil, you know there are other takes. >> As usual, I
0:13:38think you guys did a very good job ordering the keynote, and I think we can stick to roughly
0:13:45keynote order as we progress through topics tonight. If you guys -- >> We picked that
0:13:51order for a reason. >> It opened with vision OS 2 and a sort of state of Apple vision pro.
0:14:03Do you find it frustrating that five months in of shipping, four months? >> Four, four,
0:14:11five months. Don't age us. >> That there's widespread sentiment that, well, it's not
0:14:24really as popular as the iPhone. This product is a dud. >> John, we've gotten that with
0:14:31every product we've ever introduced for new products. Truly, I mean, the iPod came out
0:14:36and people are like, oh, my God, who's going to buy this, you know, at the time it was
0:14:42a $399 music player, CD players are $39. There's no future in this. And the reality is it takes
0:14:49time. It took us about two years, right? And iPod, if you remember that worked out pretty
0:14:54well. But in the meantime, everybody was calling for its death. Same thing, by the way, for
0:15:02the iPhone. The iPhone started off and it had a fairly meager start. You may remember
0:15:07even after we were sometime into it, we said, okay, we'd love to get 1% market share of
0:15:14the phone business in the first full year, which in 2008, we just barely made it, right?
0:15:19And that turned out pretty well as well. >> Got past two, I think. >> Got past two. And,
0:15:29you know, I mean, we have all the time. And watch has gotten it. I mean, oh, what a dud,
0:15:33who's buying this? It's working out pretty well. So it takes time to build a business.
0:15:38I think there's a general human tendency. I actually think it, God, I hate to tell you
0:15:42whose quote it was, but somebody who talked about the fact that people tend to overestimate
0:15:49things that can happen in the near term and they underestimate things that happen in the
0:15:53longer term. And I think that product expectations are much like that. It takes a little while
0:15:57to build a business. >> Well, and isn't platform building almost
0:16:04the canonical example of that? That what platform ever debuted as a huge hit? Maybe the web,
0:16:15right? You could sort of say sort of exploded and then you look back to -- >> Yeah, even
0:16:19that. Started slower than you think. >> Right. And then you look back to like -- >> That's
0:16:22worked out pretty well as well. >> When Tim Berners-Lee invented it and it's
0:16:27like, oh, no, there was like a five-year stretch there where I was still using Gopher. >> Originally
0:16:32had to buy a next machine in order to do the web. So the market was contained for a while
0:16:37there. >> But I'll tell you what, we're psyched over how many apps we already have for Vision
0:16:40Pro. We have over 2,000 native apps already. We had over 1,000 just in the first few days,
0:16:46which was faster start than even we had for the original app store. And let's not forget,
0:16:51we have a million and a half compatible apps. So there's a lot of things you could do with
0:16:54it even today, even as we're, as you said, building a platform. So it's pretty exciting.
0:17:07>> One of the themes I thought for this show and talking about Vision OS and the platform
0:17:17is just the word patience. And I think that Apple as a company has institutional patience
0:17:25that other companies in the field don't have. For example, I can think of a company that
0:17:35was also in the AR/VR space and even renamed themselves after, I can't remember the word.
0:17:49What was the word? >> I don't know. I don't say the word, you
0:17:56know that. >> That was a trap. Because you told Joanna
0:18:04Stern when she asked you what you thought of the metaverse, your answer was.
0:18:10>> I would never, ever use that term. >> A word I would never say.
0:18:13>> A word I will never say. And you tried to trick me right here.
0:18:19>> But that, you know, seemingly has lost interest in the metaverse in favor of AI and
0:18:26it's the new, you know, it's the hot new thing, chasing the hot new thing and then another
0:18:30hot new thing and then you chase it. Whereas Apple, you know, there was the sometime in
0:18:39the last decade, a thousand no's forever yes. But that when you guys get to a yes, you stick
0:18:48with it. And I think, you know, four months into a platform is very early to judge the
0:18:55long term prospects of it. >> We agree.
0:19:02>> I thought that the other -- >> That was really well said, John.
0:19:06>> WWDC is a developers conference, but I thought that some of the news yesterday about
0:19:13vision was it's -- if you expand developers to creative people, it was creating content
0:19:22for vision pro with partnerships, with cannon making a new lens with two elements so that
0:19:29you can get stereo right from one lens and a camera system. A partnership with black
0:19:35magic. Again, production tools for creating immersive content. And that to me is where
0:19:44Apple is. Where Apple has always been is creating these platforms for creative people to make
0:19:51the things. So I'm curious what else you see coming soon for vision and vision OS.
0:19:58>> Well, let me just maybe even kind of build on that because that was probably one of the
0:20:01biggest questions we got from creatives and film makers was I want to be able to do this.
0:20:07Because once you see a movie, you know, even a 2D movie, it looks amazing. You see a 3D
0:20:13movie, it looks even better. It's kind of like, you know, you've heard some of these
0:20:18directors saying this is the way they always envision a 3D movie to be experienced is how
0:20:23it was in vision pro. And then when you see the immersive video we did, it's like being
0:20:27there. And of course, the film makers realizing this feels like the future, right? This feels
0:20:32like the way you want to experience those things. They want to go out and create their
0:20:34own stuff. So it was a big push for us to go out and work with, you know, the cannons,
0:20:40the black magics to be able to say let's enable an ecosystem because also as part of building
0:20:44a platform, you know, is an ecosystem, right? An ecosystem of content providers, app providers,
0:20:50the whole works. So that was a big deal for us. So it's coming along.
0:20:53>> All right. Moving on. Time is ticking. iOS opened up with --
0:21:03[ Laughter ]
0:21:05>> We agreed. Yeah.
0:21:06>> Home screen customization. Apparently it takes 18 revisions to get to the point where
0:21:15you can arrange icons. Where you want -- can you speak to the engineering challenges?
0:21:24[ Laughter ]
0:21:25>> You talked earlier, John, about patience. You got to wait, you know, pick your moments.
0:21:38We thought 18 years is about the right moment to move an icon from the top down to the bottom.
0:21:48>> It felt worth it to me. >> We did it.
0:21:53[ Applause ]
0:21:57>> There were actually developers that were born. We had to wait until they could mature
0:22:04and join Apple and finish the project. We got to do it.
0:22:13>> It's for the kids. >> You need to save a win for the next generation.
0:22:20>> I think -- and I know we have very serious, very complicated, very serious computer science
0:22:26topics to talk about tonight. But I do think -- I think it was a very fun and going to
0:22:34be beloved feature in addition to being able to arrange the icons where you want, but being
0:22:40able to customize the look and flipping to dark mode, I never thought of it before, but
0:22:46I thought, boy, these icons -- I know dark mode is super duper popular, but it never
0:22:50really had occurred to me before that the icons don't change in dark mode, and now they
0:22:54do, and they do look really cool. But I think even cooler than that is the customization
0:23:00you can do where you can pick these color tints and you can make it your own. Like the
0:23:04example in the keynote was very, very vibrant, like a vibrant electric yellow, and it themed
0:23:09all the things. My two thoughts on that are it takes me back to being when I first got
0:23:16a Mac, and what did I do in the early '90s when I first got a Mac is I sat there and
0:23:21dicked around in ResEdit customizing my icons and making -- ResEdit.
0:23:26>> ResEdit. We loved ResEdit. >> And remember, folder icon maker, folder
0:23:33icon maker was this great utility where if you wanted to make a folder for all of your
0:23:37Photoshop files, you would drop Photoshop's icon on folder icon maker, and it would spit
0:23:44out a folder icon with Photoshop's icon on the folder. Well, I thought my computer was
0:23:50way better than anybody else's because my files were organized in folders that had the
0:23:55icon of the app. I spent a lot of time on that. But a couple of years ago with shortcuts,
0:24:02what I saw -- and my son really got into it and made this customized home screen where
0:24:07all of his icons -- and it was sort of a three or four step process where you make a shortcut,
0:24:11and the shortcut just opens the app, but then you can assign a custom icon to the shortcut
0:24:16so it looks like you're tapping the app and it just jumps you into the app. You spend
0:24:23some time, make a whole home screen of black and white icons, monochromatic, and then if
0:24:28the mood hits you like, "I wish they were all blue," well, then you got to start all
0:24:32over and do them all over again. Now you've made a feature where, you know, I think that's
0:24:37great that you guys spend time on that because people love to customize.
0:24:41>> I am glad, though, that you were able to pass that legacy on to your son. Had we made
0:24:48it too easy, his critical step in his maturation would have missed it, and now he's stronger
0:24:53for it.
0:24:56>> But do you see that as a people pleasing, like, yes, people are going to love this feature?
0:25:01>> Yeah, 100%. When we shipped iOS 14 with widgets and, you know, things went nuts. I
0:25:08mean, just even the number of people becoming developers to download the beta on that first
0:25:16release because --
0:25:17>> It's true.
0:25:19>> And I think it had really gone viral on social media with people sharing what they
0:25:24were doing to their screens and it created this incredible enthusiasm. So, you know,
0:25:31we waited four more years and then we did something about it.
0:25:35>> Patience.
0:25:36>> Patience. That is one of the nice themes I'm told.
0:25:39>> But, you know, I think that at times, I mean, you guys are -- Apple is very famously
0:25:45known for its design strength and, you know, you buy Apple products because they are well
0:25:53designed and they are designed by Apple designers, but it's not, oh, here's how we want your
0:25:58iPhone to look. Your iPhone is going to look just like the ones in the Apple store with
0:26:03our wallpaper and our arrangement and, no, I mean, this is -- your iPhone can look like
0:26:09your iPhone.
0:26:11>> I think that's true. I think the design team has really wanted to -- I mean, when
0:26:15you build a new product, I think you do want to agree to establish a kind of shared reality
0:26:20and identity for that product and we certainly did that pretty thoroughly with iPhone. But
0:26:27then there becomes a point where you want to let everybody make it their own, but you
0:26:33still want it to be their own iPhone. And I think we've found the balance between your
0:26:38-- even as you do all of this customization of your home screen, of your lock screen,
0:26:43we did a bunch of work there, they are very much your own, but they're still very much
0:26:48iPhone and I think that's where we've tried to find the balance and I think this year
0:26:52we pushed that frontier a little further and I think people are really going to enjoy it.
0:26:58>> I would say the biggest upgrade to control center since control center was added, where
0:27:03control center now is almost like a mini environment within itself with third party APIs for controls,
0:27:18including all the way back to the lock screen where you can -- previously it's like you
0:27:22can get the flashlight or the camera or you can get the flashlight and the camera. And
0:27:28now you can fill those spots with anything you want, including from third party apps
0:27:33who can make their own controls for those spots.
0:27:37>> Do you think it's a member of the control center engineering team out there?
0:27:44>> Either way. We absolutely -- we started to see even with the action button on the
0:27:51iPhone, like a lot of enthusiasm for people to run custom actions, sometimes they were
0:28:01of course using shortcuts as a means to do that. And so the idea of making controls really
0:28:08kind of a universal concept in the system, letting you of course tie them to the action
0:28:12button, but also put them in control center and then that naturally became okay, well
0:28:16you're going to want to customize control center and a lot of ways to do that. You might
0:28:19have more of them, so you might want multiple pages of control center. And then the biggest
0:28:25leap, those two sacred buttons, ever since iPhone 10 down at the bottom of the screen,
0:28:31to allow those to be customized was a big step for us to take. But it opens up a lot
0:28:39of power. And the other thing we did many years ago, you might remember, control center
0:28:43had a brief moment with pagination. There was a time where it was side scrolling and
0:28:50that design had the property that it was guaranteed that every time you went to control center
0:28:53it was on the last page you were on that wasn't the page you wanted to use next. So we backed
0:28:59off from that and created an all in one design. For this one, of course we preserved the ability
0:29:05to have a really full featured main page if you want, but also we created a gesture so
0:29:11you can just pull down and get right to the page you want in one step. And we thought
0:29:16that was super important to make it browsable, but also essentially instantly navigable as
0:29:21you build new pages. And so I think people are going to have a lot of fun with that.
0:29:26And I think developers now just exposing more and more of their capability as actions and
0:29:32giving users so many different ways to tap into that, whether it's from the shortcuts
0:29:35app or from control center or these other action buttons are great. And as we'll talk
0:29:40about later, the intense framework that is actually the way they express actions has
0:29:44other side benefits in other parts of the system.
0:29:47>> You say we'll talk later, but I know you haven't looked at my notes here. You said
0:29:52we were going to talk about the show in order. So I'm just figuring. You're a thorough man.
0:29:57You'll get us there eventually.
0:29:58>> All right. Here's a feature that I think kind of flew by in the keynote, but it really
0:30:02caught my eye, and it's the new setup accessories API.
0:30:06>> Whoa!
0:30:07>> I definitely remember the team.
0:30:08>> We now know where every -- by the end of this we're going to have a map of where everybody
0:30:17works.
0:30:18>> We can call out different groups.
0:30:19>> Webkit.
0:30:20>> But I would say that this -- you -- in the keynote it was presented as solving the
0:30:30following problem. It's just when you buy certain third-party peripherals that say use
0:30:35Bluetooth or want to get on Wi-Fi, and when you're setting them up, they would ask to
0:30:41use Bluetooth, which brings up a system prompt, and it would say this wants to use Bluetooth
0:30:47and the explanation text would be like it wants to connect to certain things or something
0:30:52like that. And it's like allow or don't allow. And it's like, well, I don't even -- I have
0:30:57no idea why this thing I bought wants to connect to Bluetooth. What am I doing? It's very confusing.
0:31:01I guess allow it. I bought the thing. And this solves it by presenting it in a very,
0:31:09you know, pretty much using the framework of the way that you pair AirPods.
0:31:15>> Yeah, I mean, this is one of the cases where providing the better user experience
0:31:21and a more private one were hand in hand. And, you know, it is the case today or prior
0:31:27to yesterday, I guess, that when you set up that accessory, if you felt comfortable proceeding
0:31:35and you set up your drone or whatever it was, that app still had open-ended access to Bluetooth
0:31:41or Wi-Fi or whatever, and so you don't know what kind of signature it's gathering of discovering
0:31:46nearby devices and so forth, and you probably didn't go back afterwards and take that permission
0:31:52away and maybe the app even still needed it. So we've really focused over the years, we've
0:31:57seen this on feature after feature where we've made the granting of access very clear and
0:32:05specific and well scoped. And you saw that this year also with contacts because, you
0:32:10know, today you'd have an app, maybe you want to do some messaging with it or something,
0:32:14and it says, all right, this app wants access to your contacts in order to give you maybe
0:32:18convenient autocomplete as you're typing a name or something. Convenient autocomplete,
0:32:26yes, we found that engineer. So, but what did you do? I mean, you just gave him like
0:32:33the full history of your contacts, present and future, out of which we've seen a history
0:32:38of people kind of hoovering that up and using it to build a social graph on their servers
0:32:42and all sorts of things. And so now that's locked down. But one thing we weren't able
0:32:47to really cover in great detail in the keynote is that we've given apps a great out of process
0:32:54API for if you're typing in that app, trying to type a completion, there's a way that they
0:32:59can surface in their own search UI without seeing the contact they're surfacing, surface
0:33:03a match that we get out of your contacts so then you can grant access one at a time. So
0:33:08this this gives you the same kind of convenience you want without just opening the kimono on
0:33:12all your some of the most sensitive data. Right. So it's out of process. That was a
0:33:16graphical image. kimono closed. And there's also a thing that didn't make the keynote,
0:33:24which is to use the new setup accessories. It's a new API. So peripheral makers, the
0:33:32people who make the apps for those peripherals that are on their phone that asked for the
0:33:36permission have to opt into this. I think they're going to want to because it's such
0:33:39a nicer experience. But if they stick with the older one, those dialogues saying the
0:33:45older one saying this app wants to access Bluetooth have been expanded to show you,
0:33:52oh, well, if you before you say allow or don't allow, here are all the Bluetooth things that
0:33:59are in your network right now that you would be telling this app that you have or if it's
0:34:05Wi Fi, here's all the devices and in my house, it's an awful lot of devices are on the Wi
0:34:10Fi. But that's really, really just just fantastic. Because I think those those dialogues before
0:34:17like I have no idea what it's going to I don't really feel comfortable giving this access
0:34:22to this app. I don't know why it wants us but I don't even know what it would see. Now
0:34:25it shows you what it would see. Yeah, these these dialogues. It was started with location
0:34:31where you need some text in your location, you didn't quite get a visceral sense of what
0:34:35that meant. And so we started showing you on a map what you were giving them. And then
0:34:40if you gave app apps location over the long term, we remind you we still do remind you
0:34:46and kind of show you where you've been right as well to give this sense just at a glance
0:34:50visually like this is what you've been handing over. This is what you're going to be handing
0:34:54over. And I think that's a critical design problem. How do you convey the concept of
0:34:59sharing because a little bit of explanatory explanatory text, you know, doesn't doesn't
0:35:05come close to giving like that picture or that real world sample of all your devices
0:35:10and so forth. And so that that's been a big piece of work for us over the last several
0:35:13years. Now, the other problem I see this solves is the accusation that okay, AirPods get a
0:35:22really slick pairing UI because Apple makes AirPods and Apple makes the iPhone and you
0:35:30get this really beautiful UI. What about third parties? Well, sure. And and the pattern that
0:35:39I see, but here's the pattern that I see there's a problem. The problem is it's really hard
0:35:43to pair wireless ear ear pods with a device. Yeah. To Apple solves the problem with a really
0:35:50clever solution. Three, Apple creates API's using that same thing so that anybody can
0:35:58do the same thing that Apple does with it. Apple sounds pretty great. But in between
0:36:05step two and three is patience. That's one of our themes. But is the virtue. But is that
0:36:13what do you say to someone who says that it should ship as a public API right away? Like
0:36:17the day AirPods dropped, that should have been an API that any other company that makes
0:36:23headphones could have used? Well, I think a lot of where where our ability to to innovate
0:36:30comes is from our ability to rapidly try things out, some of which don't work, some of which
0:36:37we need to get out there and learn and perfect before they become API API is a contract virtually
0:36:49written in blood like people then depend on it. We are then signing up to support it,
0:36:55keep it working well over time and getting to the right API. I think any any developer
0:37:01here who knows, you know, you put something together for use inside your app. That's one
0:37:05thing. Getting exactly the right abstractions, the right contract that you want to live with
0:37:11and support for the decades is a whole different exercise. And so the practice of we have teams
0:37:19internally that, you know, live on if you're any team shipping software with the operating
0:37:24system, you're taking the pain of living on a lot of work in progress that are us different
0:37:32teams at Apple trying out new framework APIs and designs and you're riding that crazy wave
0:37:38and taking in, you know, a lot of the pain of doing that also that we can then produce
0:37:45hopefully an API for others that has been perfected through through that process. And
0:37:51I think that's that's a virtuous cycle for us to be able to perfect it and then and then
0:37:56scale it out. And so do you think people out there need a little more patience? And yes,
0:38:05absolutely. Speaking of patience, speak to me about the engineering challenges of bringing
0:38:15color to tap backs. It was a lot more than color this time. That may not have been an
0:38:31engineering issue. That might be the best answer you've ever ever. There is a lot of
0:38:49cool stuff in messages, but I'm going to move on to photos and photos for both iOS and iPad
0:39:03OS has it's very familiar when you launch it. It is a very it's a significant new version.
0:39:12But when I heard that the tab bar at the bottom was eliminated, I have to admit I wasn't.
0:39:18That seems like the wrong way to go. It seems like, oh, here's a list of photos. And now
0:39:25a new version. We've added a tab bar at the bottom to organize. But then I saw it and
0:39:30I realized and then I took out I saw iOS 18 photos and then I took out my phone still
0:39:36running iOS 17 and I realized I don't really use that tab bar. I'm always in library. And
0:39:43what are the what were those tabs for? They were for things like albums and stuff, which
0:39:48you manually create and put things in or manually defined smart albums on the Mac or something
0:39:53like that. But that you the user sit there and spend time on a Saturday afternoon organizing
0:39:59and not necessarily the replacement. But the next step in iOS 18 and iPad or in all the
0:40:06platforms and photos is using machine learning even more to automatically categorize your
0:40:13photos into things like trips or groups or what other ways is machine learning organizing
0:40:22your tens of thousands of photos for you?
0:40:25>> Yeah, so well, one of my favorites now, you know, last year we did pets, which was
0:40:32pretty great. But now we even do like groups of people. So because if you analyze the graph
0:40:38of photos and who appears with whom, you'll find things like in my library also, you know,
0:40:44me and my spouse, me and my best friend, me and the whole family together, these like
0:40:50collections you're very often looking for exactly those those kinds of photos, right?
0:40:55Not just of a single person, but of groups being together or another big one is trips,
0:41:00right? Which are not just someday or some moment, but a extended period where you were
0:41:07away and certain kind of photos and it's amazing. Just turns out we can do an extremely good
0:41:13job of finding trips, finding the best photos of those trips and it creates a really just
0:41:20compelling way to look back. But I think that design point about the tabs, this was one
0:41:28that we worked on a long time. In fact, this was one where we were building it, you know,
0:41:34design was doing incredible work. We were building prototypes. We were living on them.
0:41:39We had patience. It was a whole year where we were thinking of shipping it. And then
0:41:43we said, you know, we don't quite have it right yet. And we kept we kept iterating on
0:41:47it to get it to get it right. And I think that the team did just amazing work to do
0:41:51that. But part of what we realized is for many users, this is true across a lot of apps
0:41:57on the phone. They have tabs, but you only use one of them, you know, and you forget
0:42:03about the rest. And again, almost like that pagination of the original control center.
0:42:13If you ever left photos on the other tab, you probably next time you came back, it was
0:42:16in the wrong place. And we had this great area called for you. That was a tab that you
0:42:21probably rarely went to unless maybe you clicked on a widget or something and it took you in
0:42:25there. And yet that was where all of this organization actually had been building up
0:42:29and happening over time. And it was like this hidden gem that we didn't have the right way
0:42:34to make it accessible. And we finally found the design that said, if you're trying to
0:42:38get at the grid, your most recent photos, there it is, it's effortless. It's just as
0:42:42it was before. You want to see these organized collections. Well, they're right there to
0:42:47each of them just as accessible as the other. I think it really came together to be interesting
0:42:51to see how much whether third parties pick up this pattern or not for other kinds of
0:42:57apps, because I think it's come together really, really nicely. Off the top of my head, I think
0:43:04photos is the app that just broadly almost almost everybody, it's almost the odd user
0:43:11who doesn't have a large collection of photos in their library. It's for very typical users.
0:43:18And at the scale of your platforms, we're talking what, like a billion people? We have
0:43:25over a billion users. That's correct. Over 1%. Over 1%. It's worked out all right. But
0:43:34like, I have lots and lots of, I don't know, probably 100,000s of emails, most of them
0:43:40unread still. But everybody has lots of email, but most people tend to live at the top of
0:43:46their inbox at the new email. But photos is a thing where the stuff that's in there, you
0:43:52know, if you have 50,000 photos in your library, if you go back 40,000 photos, if anything,
0:43:59they might be more valuable to you, right than the most recent ones. You can take a
0:44:03new picture. If I lost my last week of photos, I can take a picture of my family from the
0:44:08last week and they look the same. I can't lose a photo. But you know what I mean? I
0:44:14mean, I think it's very true. It's the one thing that everybody has in their lives where
0:44:19it's a massive amount of data and what are you going to do with it? And this is where
0:44:24machine learning, I think, it's just a step change in exposing this and changing the way
0:44:31people deal with their photo libraries. Well, and you took the photos for a reason,
0:44:36right? And you want to go back and enjoy them. And I think the team's done an amazing job
0:44:39with everything from the photo widget to this redesign to exposing those things. I mean,
0:44:45how many times do you get those photos and you're like, oh, my God, this is amazing.
0:44:47And you share it, right? You go share it back with your family. It's like, that's why we
0:44:53take pictures. We want to enjoy them. Yeah, I don't know about the Gruber household
0:44:58when you were growing up, but in my household, I don't know, there were maybe four photos
0:45:03taken of us per year. Like every few years, one of them made it up onto the mantle or
0:45:11something like that. And growing up, those are like the only photos. My self-conception
0:45:16is rooted in like handful of couple of photos. And now, my kids live in an entirely different
0:45:23world where not only are we taking photos constantly, but now these, like the widgets
0:45:29on my desktop are constantly picking great photos of them at different times of their
0:45:33lives. And then I click on them and then I'm texting them every day with some photo of
0:45:37them. Look at this, you used to look like this. I don't know what I'm doing to them,
0:45:43but it's different at least. And I'm sure it's good. I'm sure it's all psychologically.
0:45:51But it really couldn't do these things without machine learning.
0:45:54No, no way. Yeah. I mean, if we were just picking photos at random, you'd get lots of
0:45:59screenshots, whiteboards, receipts. It'd be great. Look at this. Yeah.
0:46:04And I mentioned mail. Mail very similarly has a very nice machine learning updates this
0:46:10year with things like categorization of your incoming mail. These are where things like
0:46:16receipts go. These are things you subscribe to a bunch of newsletters. They go somewhere
0:46:20else and it leaves your inbox for actual email from actual people trying to get to you. I
0:46:28thought one of the cleverest little things was the, I think it was almost a throwaway
0:46:32line in the keynote that, hey, maybe the first two lines of an email are not the best way
0:46:38to understand what the email is about. Use machine learning to summarize the email. And
0:46:45if you're only going to have, you know, it's a row of your inbox and you only get two or
0:46:48three lines of text. Let AI give a summary. It's got to be better than the first two lines
0:46:52that like PR people send me, right? It's life. Oh, PR people out there. But not your emails.
0:47:04I'm talking about other PR people. I get these, these messages and they used to say, you know,
0:47:10uh, Craig, you know, I love Apple so much and then you, whatever. And then now it's
0:47:15just for me. Now it just says wants free iPhone. That's right through it. That's also for me.
0:47:30But photos, like I said, I think, I don't know, do you guys know like what is the average
0:47:37photo library size? I don't know if you guys even collect that. I don't know. But we know
0:47:43more pictures are taken with an iPhone than any other camera in the world. Anecdotally,
0:47:48I think, you know, very, very typically tens of thousands, lots, lots and lots, lots of
0:47:53lots. Everybody uses email, right? It is, you know, for better or for worse, it is,
0:47:58you know, it's part of the glue. Lots of people are age choosing. Yeah. I was gonna say it's
0:48:02an age thing. Once, once, once people get jobs, I think, yeah, yeah, yeah. For that,
0:48:09for that, some other apps that I don't, there may be some others. Yes. So you're saying
0:48:13those young engineers who got the home screen to have icons for everyone and don't answer
0:48:17their emails. Not on the home screen. Yes. But these are very practical apps that people
0:48:27use very widely and machine learning is making things they actually do better or making it
0:48:35so they do the things way less time like skipping the wants free iPhone, iPhone email. I mean,
0:48:44that's a, that is probably a theme that I'm glad you picked up from, from our event. I
0:48:50mean, I think it's a theme from Apple over many years now that in many cases you didn't
0:48:58need to think about how we're doing it. Is it machine learning? Why does this camera
0:49:03take a good photo? Why is photos building me a nice memory? How is it surfacing a photo
0:49:11I actually care about? Yeah, there was a lot of powerful machine learning happening behind
0:49:15that all along, but the key is what is it doing for you? Right. Watch OS. I think a
0:49:20flagship feature of the new version of watch OS is the machine learning powered update
0:49:26to the photos face, which is doing similar things like, Hey, not just this is a good
0:49:33photo of a person I think you would like on your watch, but it's also one that's framed
0:49:40well for Apple watch and with machine learning. Here's where we think the numerals for the
0:49:46time could go and would fit and works. And I played with it in a demo and it's like,
0:49:53yeah, this is like, these things look designed, they look like posters. Yeah. But where does
0:49:59that are you guys frustrated or you guys just don't care that these features are in the
0:50:04works with amidst this narrative of Apple is behind on artificial intelligence and all
0:50:11of this whiz bang new intelligent software features. Well, we've been doing machine learning
0:50:18and AI. You know, you've heard me say it for so many years. We didn't even call it ML or
0:50:24AI. We called it proactive. If you remember, we're doing proactive features. We're doing
0:50:29this stuff for a long time. We've been building neural engines for seven years now, I think,
0:50:33to help us do AI and ML on devices. So we've been at this a long time. Well, I mean, that
0:50:40has that's been the funny bit now with the AI PC. It's like someone discovered the idea
0:50:54of a neural engine this year. A neural processing unit. Nothing like what we've done.
0:51:02Yeah. So, so obviously, you know, from, from, you know, phone phones for, for many years
0:51:10and you know, the first M one Mac and we introduced in 2020 and every Mac we've introduced since
0:51:16I guess we missed the boat to name it an AI PC because we've been making great ones this
0:51:21this whole time. But you know, that's not the point. They're great Macs.
0:51:27Starling right from iOS into iPad OS. And I think this year it's particularly blurred
0:51:36distinction because maybe there's some minor exceptions, but it seems like with the tent
0:51:42pole features, it's feature parody on iOS and iPad OS with like, for example, when lock
0:51:49screen customization first came to iPhone, it came to iPad one year later. Yeah. The
0:51:56things we've already been talking about moving icons around, tinting your icons, getting
0:52:01dark mode icons, iPad OS 18, the troll center. Yeah. Yeah. New photos app. Yeah. We were,
0:52:07we were able to do them in sync this year, which is great. And then of course we added
0:52:13on you know, one, one particular app this year.
0:52:20Patience. Patience. Patience. Patience for your theme. Patience. Finally, calculator
0:52:31for iPad. Yes.
0:52:38Was your strategy to hold back on a calculator for iPad so that when you did it, we would
0:52:45get this sort of reaction. We wanted you to really, I mean, could you imagine all
0:52:50the, the adulation we would have given up had we actually shipped it right out the gates?
0:52:54You know, when I first came out, would anyone have cheered? No, no, no. You hold it and
0:52:59hold it and hold it like that dramatic pause. Yeah. I forget if I've previously asked you
0:53:07about a calculator app for iPad. I know in other interviews I've seen you do, you have
0:53:12been asked about it all the time. I mean, truly all the time. We did the big iPad introduction,
0:53:16you know, in early May and I was still getting asked, do you guys do a calculator? You know,
0:53:24I'm like, patience. That must've been hard in May. Oh, it was. It was. But one of your
0:53:31answers when you would be asked your sort of non-answer answer was, well, what would
0:53:36we bring to a calculator on a bigger screen? Turned out to be true. You did bring something
0:53:43with math notes. Yeah. I mean, it was a real answer at the time, which is we, we really,
0:53:52I mean, obviously it was straightforward if we wanted to bring a blown up iPhone style
0:53:57calculator to the iPad. But we really felt that we wanted to do something more and different
0:54:04and uniquely iPad when we did it. And we, you know, we really found the moment for a
0:54:11lot of things to come together this year to do that, to do one that made it meaningful
0:54:17for iPad. And so, and I couldn't be happier, I think, and the team with the reception that
0:54:25all of you gave it at the event. I mean, it was fantastic. And we're obviously really
0:54:31happy with the work. I think one of the really cool things about doing it with, or with the
0:54:36pencil integration, which is not the only way to do it. I know in the keynote it was,
0:54:40it came up in the iPad segment, it's all with the pencil, but it's there on the phone. It's
0:54:46speaking about feature parody. It's on the Mac. And obviously on the Mac there's no pencil.
0:54:52You type, you know, you can type Mac too. But I think, but I thought the really interesting
0:54:59thing about emphasizing with pencil wasn't just finally there's an iPad calculator. It's
0:55:03that for mathematical notation, the pencil is actually far more natural and expressive
0:55:10than typing, right? You can't make a square root thing and with a keyboard, you know,
0:55:16you type SQRT with parentheses. And that's not how math students think of it. Math students
0:55:22think of it with the notation that they're doing in class. And you just draw it. And
0:55:26then you're like, oh, I need parentheses. Well, it's a pencil. So you just draw the
0:55:29pencils around it afterwards, or you draw the parentheses around it afterwards. That's
0:55:32how I learned it. We agree, John. Yep. Sort of in the same lines, but the Smart Script,
0:55:44which is the name that's the name for the feature where you're writing and you're handwriting,
0:55:50and Smart Script will actually make your handwriting better. Yeah. I mean, step one was to learn
0:55:58your handwriting so that we could both create writing that was consistent with your handwriting
0:56:07and that we could even sort of build the neatest version of your handwriting that was consistent
0:56:12with your handwriting style. You know, going back years now, we really had this goal to
0:56:17how there's so many things that are great about writing with a pencil or pen. And but
0:56:25there's also so much that's great that's a disjoint set about typing, which is type text.
0:56:30You know, if you spell check and correct like no problem, it can rewrap that. You want to
0:56:34insert in the middle? No problem. It can reflow it. You want to copy and paste a section?
0:56:40You can do that. But of course, when you want to do that with handwriting, those things
0:56:44aren't really possible unless we can actually generate your handwriting. If you want to
0:56:49copy paste and move all the words around and change it, or we want to spell check, we have
0:56:52to rewrite that word for you. And so SmartScript lets us learn your handwriting. And then we're
0:56:59able to build all these other capabilities to I mean, I don't know if you take notes
0:57:03with a pencil, I sometimes do. And always I realize like there's a word I want to insert
0:57:09in the middle there. And then you know, you put an arrow and you scribble some mess up
0:57:12above it, right? Now you can just move it aside. But of course, that means it has to
0:57:17rewrap. So now we need to understand the flow of the text and rewrap it. So we're really
0:57:20trying to bring the best of pencil and the best of type text together and SmartScript
0:57:25lets us do that. So cool. Yeah. Somewhere out there, there's got to be some very long
0:57:37term Apple employees who worked on the Newton who were like, Yes, this is what we were going
0:57:44for. Right. And including scribble to create the gap to say I don't want that word anymore.
0:57:51scribble, scribble it out, scribble it out, which is one thing I've been missing that
0:57:55I can remember with my Newton that I missed until now. And now I've got it. Yep, you got
0:58:00it. There you go. Patience. Patience. So a couple of years ago at WWDC, I think maybe
0:58:102018 I didn't bother looking at it doesn't matter what year but there I think it was
0:58:13sort of an iconic slide in a keynote was you addressing the sort of conjecture out there
0:58:23that Apple was in the midst of merging Mac OS and iPad OS. And do you remember the slide
0:58:31that was very clear slide very simple. No. I really with I believe an anvil drop and
0:58:39there was I think it was the single biggest biggest letters on a Apple slide. I think
0:58:46at the time, the conjecture that you were addressing was of the mind that the Mac, which
0:58:57came out in 1984 is old, and the iPod iPad is new. And that if that was what Apple was
0:59:06thinking, it was going to be the iPad superseding the Mac is what people thought and you're
0:59:13like, No. I think we've come around in the six years since and you know, with the new
0:59:21iPads that came out a month ago, and in re up this sort of mentality of, oh, you're calling
0:59:30these iPad pros, but I can't do my professional work on them. You should let me do boot my
0:59:38iPad into Mac OS or why don't you bring app kit and Mac OS to iPads. And I kind of feel
0:59:48like we're heading towards another No. But I tried to write about it but looking for
0:59:57the answer. I've pontificated along similar lines of that of my not necessarily frustration,
1:00:06but my personal feeling that I'm far more productive on the Mac than the iPad. But the
1:00:10light bulb that went off to me is well, the way I deal with that is I just do what I feel
1:00:16most productive on the Mac on the Mac and let the iPad be the iPad. That's exactly the
1:00:20right answer. John. They're very different products. They're different design points.
1:00:24And, you know, iPads great strength is the fact that it's you've heard us say it's a
1:00:29cliche at this point, it's a magical sheet of glass becomes anything you want it to be.
1:00:33And it's over a million apps that are designed for the iPad to allow to do incredible things.
1:00:38And I wouldn't tell somebody who's a procreate artist that they're not doing pro things,
1:00:41right? They're doing amazing things. And now we have Final Cut Pro, we have Logic Pro,
1:00:45we have all kinds of incredible apps for it. And the Mac is the Mac. And I don't know,
1:00:51I don't know why people have wanted for the time that these have existed to want to merge
1:00:56them together. That's not our that's not our desire to still big fat No. Yeah, I love my
1:01:01iPad, I probably spend at least as much maybe more time on my iPad doing a whole variety
1:01:08of things than I do on my Mac. I also love my Mac. I do not want them to become the same
1:01:14device. I think you have you've used the phrase in the past, maybe that the, you know, Mac
1:01:21is heavy so that iPad can be light or something like that. And I don't think Mac is heavy.
1:01:27I think the Mac is awesome. But, but I think I think you're right. And I think you're years
1:01:34ago wise man talked about kind of cars and trucks. And and I think I was I was honestly
1:01:41bracing myself for another wave of this, not at this particular forum, but of this sentiment
1:01:48when we were about to release the the iPad with an incredible M4 in it. Because there's
1:01:55a mindset out there that says, Okay, you took this incredible powerful V8 engine that powers
1:02:03this truck that I use to tow my boat, and you put it in this sports car. Why can't I
1:02:11haul lumber and a boat with this sports car? And it's like, no, this engine is awesome
1:02:17in the truck. And it's awesome in the sports car. And and when you use your iPad Pro right
1:02:24now, it's, it's the best iPad experience you can ever imagine. And that's worth a lot to
1:02:31to a lot of people. I mean, that is that is a great experience. And we want to keep making
1:02:35iPad, the best iPad it can be we are not trying to create a Windows 8, PC or whatever.
1:02:50Moving on to Mac OS Sequoia. A I can't every time I see Sequoia, I cannot help but note
1:02:56that it has all five vowels. Wow. Wow. And achievement. I think I play too much wordle.
1:03:10I think it looks like well, that's cool. But big deal. I think it's like a game changer.
1:03:16I'm really like it's like a light bulb went off in my head. Like I think I'm going to
1:03:20use this all the time with the new continuity feature where you can get iPhone mirroring
1:03:23on Mac. Yeah. Is that it? It seems like that's harder than it looks. Great low latency connectivity
1:03:42and discovery and all that is is the magic behind continuity and is always a challenge
1:03:49to make to make great. But when it works, it's magic. And that's that's why continuity
1:03:54is magic. But this this is one that many of us and I certainly wanted for a long, a long
1:04:00time. I use continuity camera all the time. And then my phone is up here. And, you know,
1:04:08I believe it up there sometimes and then I want to get at it. And now now click boom,
1:04:14I can, you know, it's it's right there. But there are just so many times when you want
1:04:18to access your phone, it's on the other side of the house or in your bag. And it's it's
1:04:24great. And we made the trackpad gesture handling just totally smooth. So manipulating the phone
1:04:31is really natural. So I think it's great. Apple operates worldwide and. The world is
1:04:41very wide. And you guys often say Apple complies with the laws around the world as needed on
1:04:54a per country or, you know, whoever the jurisdiction. Yep. The world's gotten wider in the last
1:05:01year and flatter. But with the DMA in the EU now in effect. Really. Best stickers out
1:05:19there. Now we know the attendees from Belgium are sitting. Russell. In general, as Apple
1:05:32has adjusted the rules, the nature, whether they're technical rules or they are policy
1:05:39rules like in the app store to adapt to the changing world, to adapt to changing features.
1:05:45In general, you guys seem to make significant efforts to keep the rules the same everywhere
1:05:55as much as you can. For example, when you guys came to an agreement with the Japanese
1:06:02Fair Trade Commission about something, something with reader apps. OK. And we'll just make
1:06:08this the new rules everywhere. With the DMA compliance, you guys have gone a different
1:06:17way where, OK, this is what we need to do in the EU. Here is an extensive set of frameworks,
1:06:26technologies, policies. Trust me, I read it all. It is super extensive. But where what
1:06:35is your thinking about when to, OK, let's apply these changes everywhere versus, OK,
1:06:43we're going to keep this as the default everywhere, but we'll do that only when needed. Well,
1:06:48John, this has been a first of all, it's been a massive effort. A massive effort by a lot
1:06:55of people at Apple to work to comply to it. As you said, we have to write. We do business
1:07:03in Europe. We have to comply with the laws there. And as the DMA is written, it's been
1:07:08a heavy lift. And especially to do that while trying to balance the safety, security, privacy
1:07:17needs of our users because a lot of these things, as you know, run in opposition to
1:07:22those. So it's been a heavy lift. And it's only been in effect for three months since
1:07:31early March. So it's still early days, but we're working hard to try to do what we have
1:07:37to do there. But it's a tough one. Yeah, I don't think, especially among the technically
1:07:44savvy among us, many of us in this room have an idea that we all used Macs since even the
1:07:52pre-internet era. We use them now. And we feel like we know how to keep ourselves safe.
1:08:04We know there is no battle that is more continuously fought and just a pitched battle than the
1:08:15one we have around keeping our devices secure and creating an ecosystem where my mom, my
1:08:26kids, I feel like go ahead and download whatever you want and enjoy your phone. That's not
1:08:35what we say with the Macs, unfortunately. And that has been such a powerful thing for
1:08:43our users and such an amazing thing, I think, for the overall community, for developers,
1:08:49the opportunity it creates when everyone feels like, you know, I want to try that. Oh, I'll
1:08:53just download it. It works. When we see something that has the threat of imperiling our users
1:09:00and disrupting the ecosystem that has been so valuable to all of the participants in
1:09:06that ecosystem, we're going to, where we can, protect our customers. And we've done lots
1:09:15of very, very hard things in the EU to minimize the damage. But it is no panacea what's being
1:09:24put on users there. And I think it's underappreciated, and I understand on a technical audience,
1:09:30why they might think that. I can tell you and from the point of view of our security
1:09:35teams and those that work on protecting our users, this is a serious issue. And so we're
1:09:43trying to do the best we can for all the users where we can.
1:09:47I think it's technically minded. Thank you. I think it's technically minded critics who
1:10:03don't see things your way, who rolled their eyes and think it's spin and not true, that
1:10:11you guys hear an overwhelming amount of feedback from users who say, oh, I like my iPhone and
1:10:16iPad the way they are. I've decided I trust Apple, and I like the idea that everything
1:10:24going into my phone comes through the App Store, which is vetted by Apple. And I mean,
1:10:31is that true that there really are people that feedback to us by real users is just
1:10:35overwhelming in that regard. And look, I think I think look, it's totally valid. Like if
1:10:40you want an Android phone, go buy it. Right? There is there is choice out there. It's fantastic.
1:10:49They're so there are options. And by the way, in Europe, I mean, more people have Android
1:10:55phones than iPhones. So some people, though, say they like the overall ecosystem that Apple
1:11:04has offered. And our customers there write us extensively. Like, don't let them screw
1:11:08this up. And so we don't want to let them screw it up.
1:11:19There's another big topic I want to talk about. But before we get to it, I think I would be
1:11:23remiss not to mark that this WWDC is the 10 year anniversary of Swift. I guess it's me
1:11:37getting older and the way time compresses. But I still think of Swift as new. But I did
1:11:43the math. And when Mac OS 10 10.0 shipped in 2001, do you remember the code name for
1:11:5010.0? Which cat? Oh, Jaguar. Cheetah. Cheetah. Very good crack marketing team. When Mac OS
1:12:0510.0 shipped, Next was only 13 years old. I know Objective C technically came like 1984,
1:12:13something like that. But it was sort of known for only 13 years. And here we are 10 years
1:12:17into Swift. How is this going? I'm very extraordinarily well, and many dimensions.
1:12:25I mean, when you talk about trying to and you're like, if you can go look around the
1:12:31industry and see different places where companies have tried to create new languages, very,
1:12:36very few of them kind of ultimately go anywhere. And Swift was, it has been just a runaway
1:12:46success for app development on our platform. Right? Just a million app on the App Store.
1:12:52I'm not sure what the number is. It's an extraordinary number of apps that use Swift. There's a whole
1:12:58generation of developers now who've come up programming in it. But also many of us who
1:13:06love and loved Objective C have, yeah, hell yeah, have learned to love and appreciate
1:13:15all that Swift brings as well. So it's been phenomenally successful.
1:13:20When we introduced the language, though, our ambition, step one was clearly a great language
1:13:26for app development on our platform. Right? That was mission number one. But the ambition
1:13:30for the language was as a general purpose systems programming language. And what's happened
1:13:37semi-quietly while Swift has continued to be successful as an app programming language
1:13:44is Apple's adoption and some others as a systems programming language. I mean, now with embedded
1:13:51Swift we're running Swift in like the secure Enclave processor. We're running Swift on
1:13:57servers. We're running Swift just all over the, yes, all over the system. And these are
1:14:06places that historically our only alternative really was C or C++ really. And many of us
1:14:16grew up programming those languages. They are by design effectively unsecurable. Right?
1:14:24They have many interesting properties, but they're operating at a pretty low level of
1:14:28abstraction and are inherently not very safe. And in the world we live in now, having a
1:14:33programming language be safe as well as ideally expressive, incredibly productive are big
1:14:39virtues. We've done so much work on Swift to make it great for interoperability, not
1:14:46just with Objective-C and C, but now also with C++ that we think there's a new era ahead
1:14:54where the world needs a safe, expressive systems programming language. And we think Swift is
1:15:01it. The slide I saw in the State of the Union, I wrote it down, Swift is the best choice
1:15:06to succeed C++. That's the whole slide. And that is the truth. You look at the alternatives
1:15:14out there, nothing, look, Swift is built by the team that built probably the most popular
1:15:21C++ compiler in use today in Clang. And Swift is more native to C++ than any of the alternatives
1:15:30out there. Its ability to call and interoperate with C++. Because this code base, there are
1:15:35code bases out there that aren't going away overnight. And people are going to continue
1:15:39to write a lot of C and C++, but people are also going to add a lot onto it where using
1:15:44a language that can interoperate seamlessly with it and let you grow a safer and more
1:15:51productive code base and also create other things is a unique Swift power. And I see
1:15:57Swift as having an extraordinary future, of course inside of Apple, but across the industry.
1:16:04And I think that's the next 10 years.
1:16:13Obviously we want to talk about Apple intelligence, but I feel if we're going to talk about Apple
1:16:18intelligence, we should have somebody out here who's actually intelligent about it.
1:16:26Ladies and gentlemen, John G. Andrea.
1:16:32Hey guys.
1:16:37Wow.
1:16:39All right.
1:16:44JG.
1:16:46JG.
1:16:47JG.
1:16:49GJ.
1:16:50What's the deal, Craig?
1:16:56Sorry.
1:16:59Sorry for keeping you waiting. It's your colleagues fault for having so much to talk about.
1:17:03It was fun listening to you backstage.
1:17:07Siri.
1:17:09Let me give you a slogan. This time we mean it.
1:17:18I think when I started working with the Siri team, the first instruction I gave them was
1:17:21failure is not an option because a lot of people use Siri a lot of the time. And as
1:17:29it's got better over the years, people just, we see in our data that people just use it
1:17:32more. And the numbers, I'm not going to tell you what the numbers are, but they're huge.
1:17:39A billion and a half requests a day.
1:17:42A billion and a half voice requests a day. But within that, like the mix is shifting
1:17:46because when people use a voice assistant and it works for them, then they use it more.
1:17:50So it's a very shifting target.
1:17:54I'm going to give you a recent example from a friend of mine. This is obviously using
1:17:58iOS 17. What time zone is Las Vegas in? And the answer was I tried it and got the same
1:18:06answer. Sorry, I can't help with that, but you can ask me the time in a specific city.
1:18:12Yes.
1:18:14If you change that from what time zone is Las Vegas in to what time zone is Las Vegas,
1:18:21then it would, it gets a good answer.
1:18:22So this is the NLP brittleness problem, which is you don't want to have to teach your system
1:18:27all of the ways that human beings might ask for something. So one of the things that we
1:18:31showed at WWDC was an example of you correcting yourself and saying two different things and
1:18:37the new models figuring out which thing you meant. And so the good news is that the technology
1:18:42of language models is getting dramatically better and is less likely to make these fragile
1:18:47mistakes because they drive us nuts too.
1:18:51It's good to hear that. Let me clarify something. And I don't think it was unclear in the keynote,
1:18:59but I just think that it's so important that it deserves to be emphasized over and over
1:19:05again is everything we've spoken about heretofore is not Apple intelligence, right? Scene learning
1:19:11and photos, the email categorization, all sorts of stuff that would broadly fall under
1:19:19your domain is not what you're calling Apple intelligence.
1:19:23Some of it is powered by Apple intelligence.
1:19:25Well, so things, things like for instance, the smart script, the math and so forth. We
1:19:33are not, those aren't built on the foundation model on device or in the cloud. And when
1:19:39we're able to bring those to the much broader class of devices because they don't require
1:19:47some of the power that is foundational to running these foundation models. And so the
1:19:55keynote was organized in part both to get that big idea out of Apple intelligence, but
1:20:00also to make kind of clear for people what's for all the devices and what's for the class
1:20:06of devices that can support Apple intelligence.
1:20:08So let's talk about the class of devices that can support Apple intelligence. The cutoff
1:20:13is for iPhone is very recent. It is the iPhone 15 pro with the a 17 pro system on a chip
1:20:22and then any iPad or Mac with an M series chip. What, what is that cutoff? What were,
1:20:32why, why is that the cut?
1:20:34So these models, when you run them at runtime is called inference and the inference of large
1:20:39language models is incredibly competition expensive. And so it's a combination of bandwidth
1:20:45in the device. It's the size of the A and E it's the, it's the, the, the, the, the oomph
1:20:51in the device to actually do these models fast enough to be useful. You could in theory
1:20:56run these models on a very old device, but it would be so slow. It would not be useful.
1:21:01And so it is not a scheme to sell new iPhones.
1:21:11No not at all. Otherwise we've been smart enough just to do recent iPads and Macs too.
1:21:15Wouldn't we?
1:21:16Yes.
1:21:17No, you know, we've, we've had so many, I mean, our, our first move in any ways to figure
1:21:22out how we can bring features back as far as, absolutely. That's, I think you've seen
1:21:28that time and time again with us. What, what JG says is, is right here. We, this is what
1:21:36it takes. This is the hardware, what it takes. I mean, it's a pretty extraordinary thing
1:21:40to run models of this power on an iPhone. And it turned out it, it took us building
1:21:46this iPhone.
1:21:49You guys have mentioned the neural engine and that the A17 Pro is the first A series
1:21:54chip with the neural engine. But is RAM a function of that too? Or is it really primarily
1:22:00neural engine or it's really all of it?
1:22:02Sure. It's many dimensions of the system. Yeah. RAM is one of the, one of the pieces
1:22:06of the total.
1:22:07Yeah. And the A17 Pro is not the first A chip that's good in your engine, but it's got a
1:22:10much bigger neural engine than the chip that came before.
1:22:13Right.
1:22:15One thing that it doesn't seem, I don't even know if they're capable of it, but like I,
1:22:23I find with dealing with other human beings.
1:22:27Difficult, right?
1:22:29Difficult. I do. Yeah. Obviously. Well, you know, you know how many, how many colleagues
1:22:34do I have? I got one podcast on the side with one colleague. Let's see if I can keep, keep
1:22:42that going. No, but I, I find it, or I should say maddening when I meet somebody who never
1:22:52says I don't know. I think that the humility of somebody who hopefully you know most of
1:23:01the questions I'm talking to you about, but if you don't know, I appreciate just saying
1:23:06I don't know or I'll go find out. Seems like LLMs do not have the I don't know.
1:23:13They also have the feature that they double down on what they said. I think we technically
1:23:24call this bullshitting, but so what we have done in the features that we announced is
1:23:34we've been very careful about applying this technology in a very thoughtful way. So we
1:23:39don't have features that will write, you know, a college essay about something about the
1:23:45world, right? We've tried to make this, we've tried to corral this technology to do what
1:23:49it's really good at doing. So a good example of that would be summarization. That used
1:23:53to be an open research problem in the NLP community and now it's essentially solved
1:23:57with guardrails and a whole bunch of careful work, but basically you can summarize an email
1:24:01now and so that that's that for each of the features that we launched, we've had these
1:24:05internal debates about is this technology ready for real users to use.
1:24:10Right. So that you're not going to recommend using glue to stick your cheese on a pizza.
1:24:19We do not have that feature. Just to take an example at random.
1:24:28How broadly, how are you thinking though about protecting against, let's say offensive, wrong
1:24:36or just playing goofy like using glue on pizza responses?
1:24:41Yeah, so there's two problems that you have to address with that alignment. One is the
1:24:47hallucination problem you've alluded to just saying random stuff that's not true. But the
1:24:52other is the research community calls the safety problem, which is saying things that
1:24:55are inappropriate or suggesting things are bad or illegal, for example. And there we
1:25:04have to tread much more carefully because we think that the user is using our devices
1:25:10for their purposes and their creative reasons. And so we don't want to be the arbiter. We
1:25:15don't want to have a huge blog list of words you're not allowed to say or ideas that somehow
1:25:20don't work in the word processor. So we have to find that balance point where we're not
1:25:24amplifying inappropriate uses. And we do have a huge number of what we call guardrails and
1:25:31system techniques to make sure that certain topics are off limits. But it's a very fine
1:25:38balance as we spend a lot of time and a lot of meetings trying to figure out, because
1:25:42this is new for us, this is new for the whole industry, trying to figure out where that
1:25:45balance point should be.
1:25:47And it seems you don't really have, like you said, like you're not writing, Apple Intelligence
1:25:54is not generating a first draft of an essay. So you can't get it, you can't give it a prompt.
1:26:01There is no interface to give it a prompt to say, write me a story about shoplifting
1:26:07from my local Apple store. I'm trying to pick something that is funny and wrong, but you
1:26:17know, would be a headache for Jaws that, oh, everybody's getting the new Apple Intelligence
1:26:24to give them instructions on how to steal stuff from the Apple store.
1:26:27Yeah, an example we use internally and research use all the time is like, tell me how to hot
1:26:30wire a car. It's a factual question. Most large language models go try it, will probably
1:26:37refuse to answer that question because they've detected that it's illegal activity and not
1:26:41going to help with that.
1:26:44So what you could do with Apple Intelligence is you can sit down and write your own story
1:26:51that might be inappropriate or goofy or committing crimes in the story, select it and then say,
1:27:02make this more serious. But that's your story that you started with.
1:27:09That could be a fictional story.
1:27:11Right. It could be fictional, but that's not being generated. You're the person. You're
1:27:15the one who wrote the story. So if now it has more professional sentence structure,
1:27:21that's on you. I feel like this is a very interesting balance you guys have.
1:27:25It is. And I mean, we literally had ethicists involved in our discussions on this. We do
1:27:33have our roots as a personal computing company. This is a tool for you. So if you were writing
1:27:40a story or maybe you are writing something about a very harmful phenomena for the purposes
1:27:51of illustrating why it's bad or to protest against it. And if you're not careful, a model
1:28:00that you say, well, summarize this for me or help proofread it for me, make it more
1:28:04professional, a model that says, I'm sorry, I can't do that. This seems to be about
1:28:10harmful topics. Well, now that's not empowering you as the use of the system. We aren't going
1:28:17to introduce the harm. We're not going to amplify the harm. But we do think it's a tool
1:28:23for you. And so that's the line we're carefully trying to draw.
1:28:29Yeah. And we published a blog post yesterday which goes into some depth about how we built
1:28:37these models and how they work and so on and so forth, some technical behind the scenes
1:28:40stuff. And in there we actually listed some of our values about how we approach our Apple
1:28:47intelligence. And one of those is respect the user's agency.
1:28:59I'm not surprised and I'm glad to hear it, but with rumors that you guys were heading
1:29:08towards, I mean, Jaws kind of spoiled it with his tweet when he announced.
1:29:13Absolutely incredible. Absolutely incredible.
1:29:15Well, you saw right through that one, didn't you?
1:29:20But I'll even admit knowing Apple's sensitivity towards the brand and the brand promise to
1:29:29customers that perhaps Apple would shy away from such things. And I think that's great
1:29:36that you don't want to limit, but you're not going to give them the story, but you'll let
1:29:41them make the story they want to make. Yeah, it's very early days. This technology
1:29:45is very, very nascent. It's very powerful, super exciting, the experiences we've been
1:29:49able to build this year. But it really is the first of a many, many year journey with
1:29:54its technology. Image playgrounds, I guess, is the closest
1:29:59or is the part of it where you're generating the most. Right. And there's three styles.
1:30:07I think you call them animation, illustration and sketch, sketch, sketch, but not among
1:30:15them is photo realistic. Yes. Coincidence.
1:30:22A notable absence. Why? Well, because you don't want to make it easy
1:30:32to make deep fakes. Right.
1:30:34And there's no reason to do that in the first version of the product. And there's lots of
1:30:37other tools that you can still get. I mean, let's face it, there is an ick factor
1:30:47to it. Right. It is. This is kind of gross and uncomfortable and worrisome, really. Right.
1:30:54Yeah, potentially. Yeah. Yeah. I mean, the opportunities for abuse are really high. And
1:31:02we we think a lot of this we're focused on communication. That's fun and expressive.
1:31:09We're not trying to create an alternate reality at all. It's very clear when you see our output,
1:31:14no one's going to go like, oh, my God, JG was on the moon or whatever. Right. Is that
1:31:21alligator really on a surfboard? Is that alligator on a surfboard? We want to create no confusion
1:31:25on these points. But like with the new feature in photos where
1:31:29you can remove unwanted either people or objects in the background, that I'm guessing is generative
1:31:36technology because the fill is not just, you know, checkerboard. It's actually filling
1:31:42in what it is. So obviously, the capability is there in in the technology you have. But
1:31:48there you're not generating reality. You're filling in the reality that was there behind
1:31:54the object or movement. Yeah, it's it's and it's a super it's a super
1:31:58fine line. And, you know, we how aggressive we would pursue, like decluttering a photo
1:32:09is also a point of great debate. We make sure to mark up the metadata of the generated image
1:32:13to indicate that it's been altered in this way. And like you say, we're not. Yeah, it's
1:32:18good to. But but but yeah, we're not trying to generate photo realistic images of people
1:32:27or places or anything like that. And or when you point your iPhone at the moon, for example,
1:32:37or a good example, another random example, just off the top of my head. One of if I can
1:32:47pick one word to summarize the entirety of yesterday's keynote, I would pick this word,
1:32:55trust. And let me give you an example. Hypothetically speaking, let's say there's another company
1:33:05that makes desktop operating systems and they see AI as, you know, the bright future that
1:33:19it really has that this is where and we're on the cusp of it. And maybe they want to
1:33:24power AI PCs and would introduce a feature that could recall everything that was on your
1:33:33screen every five seconds. And then it turns out that the first version they shipped is
1:33:42shipping the database of all of the text that was on your screen all the time in a plain
1:33:49text database on your startup drive. Hypothetically speaking, I think that's the perfect reaction.
1:34:02But is that frustrating to you that that it sort of plays into the consumer's worst fear
1:34:11about this stuff? New stuff is scary. Fingerprint sensor on my phone. Whoa, whoa, whoa. Apple's
1:34:17going to have my fingerprint. That seems scary. Oh, no. You know, and they're just on device
1:34:21and it's not just on the device, but it's in a secure enclave. And we can explain to
1:34:25you how the secure enclave is very clear and would never go to the cloud. And if you get
1:34:28another device, you've got to use the fingerprint sensor again because we never had your fingerprint.
1:34:32Oh, OK. Oh, now you're going to scan your face. Oh, no, no, no. Not my face. Right.
1:34:37But that's I'm exaggerating. But I think it's not really normal. Not really. That was that
1:34:43was the reaction. It's normal that people are like, whoa. And then, OK, let me listen.
1:34:50But like with the Windows recall feature, it literally plays into the worst fears people
1:34:55have. They hear this feature and then hear 10 days later, somebody figures it out. And
1:35:00it's like it's all in plain text there. Is that frustrating to you as you're building
1:35:04out features that you're trying to build trust in that overall are outside? Are we frustrated
1:35:10by the failings of our competitors? The answer is no.
1:35:28Let's right before you came out, we were talking about Swift. And to go back to that, a big
1:35:41part of the news didn't really make the keynote because State of the Union is where developer
1:35:46oriented news goes. That's you know, I always call it the developer keynote. And Xcode has
1:35:53a tremendous amount of new generative A.I. features. There's I think at the first cut
1:36:02is completion, code completion. And the next one is Swift Assist. Am I getting the name
1:36:11right? Because it's Swift only. Right. It's very specific to Swift. Yeah. I hear that,
1:36:20though. And I think trust. And I think there's two trust factors. The first one is can a
1:36:24developer trust the code that's being generated? And then the second one is can the developer
1:36:29trust that their code isn't being misused to train the data for others? Yeah, both are
1:36:36true. Right. So so I mean, it turns out these large language models are very good at generating
1:36:42code if you ground them in code that's either been run and tested or code that came, you
1:36:49know, was legitimate code that was being used. And there's a technique for doing this called
1:36:53retrieval augmented generation. And so that second feature is anchored in, you know, ground
1:37:00truth about what is good working code. And yes, certainly we would not use our developers
1:37:07code to train our models. In fact, we could say something even stronger, which is we don't
1:37:11use any of our users data to train Apple Foundation models. So what what can you say about the
1:37:25training data that you did use to train the code generation features? Where did it come
1:37:30from? If not from users? Well, the foundation models themselves are trained on a wide variety
1:37:41of data, some of it public web data and some of it licensed data. And so for the public
1:37:47web data, these large language models learn to generalize by seeing literally, you know,
1:37:55billions and trillions of tokens. When we do that, with with Apple bot, we let publishers
1:38:01opt out of having their website be included in that. But the web is a very big place.
1:38:07And then for the Swift code specifically, I actually don't know the answer to the question.
1:38:12Oh, look at you like that. He said, I don't know. Yeah. Some of my team's feature, but
1:38:19I do know that he uses rag. So it's grounded in code that's probably been licensed or otherwise
1:38:24acquired. Yeah. And then we do a lot of synthetic data generation as well. When it comes to
1:38:28coding data, we can take things like our own documentation prompt them up. All right. That
1:38:38is someone who works on our document. But but use it to prompt prompt a model to say
1:38:44what what are the kinds of things people would want to do with this API write a program that
1:38:47does that and then create a pipeline that verifies that that's working code, corrects
1:38:53it, et cetera. And so through through these kinds of advanced pipelines, you can generate
1:38:58more and more great examples, which then in turn become great, great training data or
1:39:04great prompts at runtime to answer answer questions. So synthetic data, I think, is
1:39:08going to become more and more of the answer for making these models smarter, especially
1:39:13if you can build a good evaluation feedback loop to establish like this. This generated
1:39:18output is actually correct. So a question very specifically for you, Craig, is are the
1:39:25engineers on your teams using these features only in a pilot sense so far? Because these
1:39:35things have come together. Well, we've been working on training them and getting them
1:39:39into a great shape. They've only been we've only achieved the performance and quality
1:39:43level we wanted very recently. And so I can't claim that, you know, this this release was
1:39:48X percent written by our models, but people who who use them love them. I mean, the acceleration
1:39:59is sometimes it's it's you know, it's kind of a mind reading experience. You know, you
1:40:03know what you want. You're like, wow, OK, I did it. It gave me exactly what I wanted.
1:40:07And so I expect we'll be using it very regularly internally. Well, they will be. So I don't
1:40:12mean to say present tense in terms of I know that this is new and maybe, you know, very
1:40:17early release developer tools aren't being used for production releases of software.
1:40:22But the plan is that they this is not just for others who, oh, OK, you guys want code
1:40:27generation and Xcode internally. It's being it will be dog food. Oh, for sure. This is
1:40:31like a lot of internal demand, a lot of interest in it. And absolutely. And in terms of that
1:40:38fear or just back of your mind suspicion that, oh, I don't want to use a tools that have
1:40:44a server component with my data because I don't know what happens with it. Engineers
1:40:50on your team will be able to use it with Apple intelligence, even when it goes to the private
1:40:57cloud compute. Knowing the secrecy that your teams work with, even inside Apple. Yeah,
1:41:04well, and it's I mean, private cloud compute is a topic I'd love to dig into the the predictive
1:41:11code completion feature you talked about, though, one of the great things about it is
1:41:14it's entirely on device. Right. So we really get. Yeah. Which which I think is relatively
1:41:20novel. I think a lot of the tools out there that do that now are involving sending your
1:41:24code off to somebody servers and not private cloud compute servers, for that matter. But
1:41:30this is all running on device. It's incredibly fast and it has it's going to work when you're
1:41:34on the plane or offline, et cetera. So I think that's a fantastic component. But, yeah, I
1:41:40mean, a huge part. And I think many of us who wanted to use models for a lot a lot of
1:41:50times what you want to use them for are things that involve personal or confidential information.
1:41:55You know, if you're going to like revise an email or fix up some code, odds are that's
1:42:02not code. You just are attempting to give away or confidential information that you
1:42:09were intending to leak into someone else's training set. And so for us, a prerequisite
1:42:16to doing personal intelligence that involves data you actually care about is ironclad privacy.
1:42:24And there. Yes. And on device, of course, a fantastic answer where that works. And we've
1:42:32really been pushing the limits of what is possible to do on device. And part of this
1:42:36is specializing these models for certain high value tasks by building adapters that can
1:42:41achieve really high level performance. But sometimes you hard to beat having tons of
1:42:47compute to throw at the problem in an instance when it comes to these big things. But we
1:42:52don't want that to mean the compromise of giving away your data or losing control of
1:42:57your data in any fashion. And so we absolutely moved mountains across like every component
1:43:05of the stack from having our hardware team build us custom servers to us building a custom
1:43:11OS and inventing a lot of technology for trusted attestation and server management, a bunch
1:43:19of code written in Swift to do that, to create, I think, for the first time in the industry,
1:43:26a an at scale AI inference platform where the operator of the platform and no one else
1:43:35can have any access to the data that's being used for inference. So now you can use this
1:43:40thing and really feel like, oh, I'm not I'm not exposing my data to others. It's existing
1:43:46in that same privacy bubble that has protected my data on my phone. And that's been extended
1:43:52to the cloud. And we think that's an extraordinary engineering achievement. And I think what
1:43:56the future of AI should be. Was this table stakes for Apple that, OK, some of these tasks
1:44:14are going to require cloud compute today. And if they're going to and we want to have
1:44:20these features, therefore, we need to do what Craig just said and build out Apple's own
1:44:27servers, Apple's own everything. Yeah. Yep. Private by design. I have to ask, what is
1:44:40the Apple silicon in the cloud servers saying? No, we didn't specify. I said we didn't specify.
1:44:48Oh, we didn't specify. That's the way we choose not to answer. John, when you say how many
1:44:56of the servers there are a lot. It's very powerful. I had some value up here. John,
1:45:10all right. Let me ask about this with private cloud compute. A lot of people are very concerned
1:45:17with the rise of A.I. in general, which in until yesterday was almost entirely for a
1:45:28large language models and generative in the servers in the cloud. The environmental impact.
1:45:35Very, very serious concerns. What is the story with Apple's private cloud compute and its
1:45:43environmental impact? I mean, there are two reasons why we wanted to build private cloud
1:45:48compute out of Apple Silicon. One is the privacy architecture it gives us, you know, building
1:45:53on the secure or enclave, trusted boot, every other thing about the systems. But the other
1:45:59is the incredible energy efficiency of Apple Silicon, which, you know, is forged in our
1:46:04history of building mobile silicon. This is not the history of your typical A.I. inference
1:46:10silicon, right? But it is ours. And so we just get tremendous efficiency out of our
1:46:18private cloud compute. And this is going to allow us to continue to have our data centers
1:46:22be 100 percent carbon neutral, which renewable energy. More than that. It's more than 100.
1:46:28No, no, it's a it's all renewable energy, right? It's all. Yeah. So it's not just it's
1:46:33not just carbon neutral. It's entirely 100 percent renewable energy. That's right. Yeah.
1:46:40So to clarify, this is not with carbon offsets. No, this is 100 percent renewable energy.
1:46:47So it's a phenomenal environmental story. Not to mention that a ton of this is running
1:46:51on your device, which is inherently very efficient. Right. And not even going to the data centers.
1:46:57But as you know, this has been a big concern for generative A.I. is the amount of energy
1:47:01they they're using off the grid. And we have an incredible Apple kind of story on that.
1:47:09Is it possible that achieving that took a little extra time? Hence the patience. Patience.
1:47:19But that's I do think that there was, you know, with with the rise of A.I. and the you
1:47:24know, a couple of years ago, the hot new thing was cryptocurrency. And I don't think people
1:47:30were waiting for Apple to mint its own cryptocurrency. I think most people expected Apple to let
1:47:37that one pass. And I think A.I., there is undeniably not just a little there, there
1:47:44a lot of there there. And I think there was a consensus of, oh, that 2030 whole company
1:47:51is going to be carbon neutral. Wow. That goes out the door because now there's this monkey
1:47:56wrench thrown in. But no, no, this does not disrupt. This doesn't move it back to 2031.
1:48:03It's day one. Renewable energy. Yeah. I think that that goal is very much alive and well.
1:48:18Can you speak to me a little bit about this? This idea of verifiable images on the cloud
1:48:22compute. It's not source code that you're giving to researchers. They're images. But
1:48:28how does that how does getting an image of the OS let them verify the integrity of the
1:48:35OS in a cloud? A couple of things. So our benchmark really is iPhone. Right. When with
1:48:43your iPhone, we of course, are every time we make a change to the behavior of of iPhone,
1:48:51it's because we've shipped a software update with an image of the software and researchers
1:48:56are able to debug that system and observe its behavior and check its code and so forth.
1:49:04And traditionally in the cloud, that's totally not the case. Right. If you someone's operating
1:49:10in a server, you're sending your request to their server. You have no idea what's running
1:49:15on that server. You have no way to inspect what's on that server. Even if at one moment
1:49:20in time it got audited, you don't know if a minute later it changed its code and a minute
1:49:24later changed it back. I mean, you just can't have any confidence in what's happening there.
1:49:30Very much unlike what's happening on your phone. So we want to make sure we could give
1:49:35that same property. And we do this by saying that the whole trusted boot infrastructure
1:49:45that the SCP, the secure enclave processor on the server is measuring is able to identify
1:49:53exactly the total image of that software. And this OS is also frozen so that no new
1:49:59code can be added. No new code can be generated at runtime. Like the OS literally won't map
1:50:04in any executable code pages that aren't part of the encrypted image. And when you make
1:50:09a request to that server, there's a handshake where the SCP attests and says this was a
1:50:16trusted boot and here is the measurements, the signature of this OS that you would be
1:50:22talking to if you were to send me the request and here are the keys to have that conversation
1:50:27with this thing. Well, separately your phone is going to look in an attestation log, an
1:50:34append only log, brought up crypto, that kind of technology, that Apple had to post that
1:50:42these are known posted public images of this and your phone says is the image I'm talking
1:50:48to a publicly posted image? If it's not, I will not talk to that device. So even if an
1:50:55attacker were to try to sign an image, put it on that server and try to intercept a request,
1:51:03your phone would refuse to talk to it. So there you've got these images that we have
1:51:08to publicly post. We are then also equipping researchers with VMs so they can run those
1:51:15images on a Mac and debug and test them and verify our claims as well, which is even a
1:51:22step up from what they're offered for iPhone. And do the same thing where they can take
1:51:27the image on their local VM that they're testing and check that its signature matches. Exactly.
1:51:33The one that you've posted to this write only chain that they can verify that you didn't
1:51:38ship, you Apple didn't ship the researchers a different image. Exactly. Exactly. And no
1:51:50one's ever done anything like that. Not that I know of. Yeah. It's not a subtle difference,
1:51:57but a complete inversion difference with Apple intelligence in the starting point is that
1:52:02because you were running on the customers devices, you've got all of this personal context
1:52:11to start with as opposed to starting with whatever was just typed in a box. How different
1:52:18is that in terms of are your models along the lines of other models and you just have
1:52:27a different data source or is it an entirely different approach to creating generative
1:52:33models? The core models are so-called foundation models. What's new and novel is that they
1:52:40can operate, the ones on device can operate over their semantic understanding of the user's
1:52:46data. So we have a semantic index of your messages, things on your, Craig was mentioning
1:52:53earlier about relationships in your photos and so on and so forth. And so the model can
1:52:57reason over that private data. And what that gives you essentially is a personalized feature
1:53:04that is personal to you and runs on your device. Whereas these other chatbot like products,
1:53:10they have tremendous world knowledge, they've ingested all of Wikipedia or whatever, but
1:53:14they don't know anything about you. And so they're limited from talking in a completely
1:53:17unpersonalized way. And we think that that's just interesting but less useful than getting
1:53:23done with your data like we demonstrated at DubDub.
1:53:29Right like typing into a chatbot when does my mom's flight arrive?
1:53:34It won't work, right?
1:53:36It couldn't.
1:53:37It'll make something up.
1:53:38Right, right, right, because it's not going to say I don't know. We can try it right now
1:53:43and it'll.
1:53:44But I mean it's even more foundational than that. If you ask a chatbot what time is it?
1:53:47Right.
1:53:48It doesn't know. There's no concept that there's an API call that you should be making.
1:53:53Yeah and I know John you sort of track how different parts of our stack develop over
1:53:57time and what's interesting about the semantic index is for many, many years foundational
1:54:02part even going back to Mac OS X has been Spotlight and apps and with iPhone apps contributing
1:54:10information in the Spotlight so that the user could search them or so that the app could
1:54:14use powerful search from the system index within the apps.
1:54:20Well that same interface, the same way that apps are donating that context to Spotlight,
1:54:25that is the source that the semantic index is then able to take that data and index it
1:54:29not just by keyword and so forth but semantically and that becomes the source for Apple intelligence
1:54:38running on device to look things up.
1:54:40So it's not that there's a whole new, there's no new information being exposed to the system.
1:54:44It's the same information that's been exposed to Spotlight all along and there's no new
1:54:49work for developers. Another good reason for developers though to provide information into
1:54:53Spotlight because now it's making Apple intelligence even more able to help the user and to direct
1:55:00the user back into doing activities inside that app.
1:55:04Probably worth saying it's stored securely on your device as well.
1:55:07Yeah that too.
1:55:08Yeah.
1:55:09Yeah in the same way the data always has been on your iPhone.
1:55:12And the difference in context, like I said, it's very useful over here with world knowledge
1:55:18but entirely different but in many more practical ways with personal context.
1:55:25But in your career really, do you find yourself thinking about how humans deal with context
1:55:35naturally? Craig just addressed me as John a minute or two ago.
1:55:39You knew he was talking to me even though you're John.
1:55:44Does your mind as someone whose career has been like this, do you constantly think about
1:55:50how you knew that that was me John not you John John?
1:55:53Yeah.
1:55:54I have spent a considerable part of my career working on this problem.
1:56:01So I mean the technical answer to this question is the breakthrough with transformer models
1:56:07and LLMs that has caused all of the excitement of the last few years is around an idea called
1:56:14attention which is that not all data is equally relevant when you're considering a time sequence
1:56:19of things.
1:56:20So this is why we use examples like when do I need to leave to go pick mom up at the airport
1:56:24because the model really truly has understood the notion that you have to do one thing before
1:56:28the other and therefore that it needs to find a flight from your mom.
1:56:34So let's go call the semantic index and see where that flight is.
1:56:36It could be in my email, it could be in messages, whatever.
1:56:39But that notion that there is the idea of picking somebody up from the airport is actually
1:56:44really learned by these models.
1:56:48And that has really been the breakthrough for LLMs and why this technology is so important
1:56:52to figure out how to use appropriately.
1:56:55That brings me to the last topic from the keynote and that is the partnership with OpenAI
1:57:00and ChatGPT.
1:57:02Here's my question.
1:57:03Everything we have spoken about in this very long interview to date has all been Apple
1:57:08technology.
1:57:09Yes.
1:57:10Correct.
1:57:11Yeah, 100%.
1:57:12All of it.
1:57:13All of it.
1:57:14And this is a lot and it is extremely useful and it is broad.
1:57:17So why partner with OpenAI?
1:57:21Well, we think there are use cases for these larger world models.
1:57:28They're not the use cases we've been talking about.
1:57:31But we know that people are pretty excited to use these models for all kinds of tasks.
1:57:36And what we think is going to happen over the coming years is there will be many of
1:57:40these large models that are specialized for very specific things.
1:57:44And some of them will have different business models.
1:57:45Some of them will be free.
1:57:46Some of them you'll have to pay for.
1:57:48But there may be legal models or maybe models that understand healthcare really well.
1:57:53And so we think that baking an entry point into these models into the user interface
1:58:00is kind of sort of necessary.
1:58:03Otherwise you're forced with like bringing up a chatbot and cutting and pasting stuff.
1:58:08And we respect what these models can do enough to say we should make it easy to do.
1:58:14And we should give you a choice ultimately of models to use.
1:58:17And so we'll see how users like the integration that we've done.
1:58:22But we think that this technology is not going away anytime soon.
1:58:27And I think of it a bit like the way Safari deals with search engines.
1:58:31Search engines, sure.
1:58:32Right?
1:58:33I mean, it just became a thing that you would type into the URL bar your search term and
1:58:38that's just a good way to do that.
1:58:40Well similarly, if you actually do want to write an essay about blue whales and you want
1:58:43to use ChatGPT to do it, we want to make it as easy as possible.
1:58:46And we want to make it very super clear to you that you're using ChatGPT, that the thing
1:58:49you're getting back might have hallucinations.
1:58:52And so that's why we've done the integration that way.
1:58:55Right.
1:58:56All right.
1:58:57It's probably worth clarifying.
1:58:58We talk about it sort of built in, the integration built into the system.
1:59:06The integration is turned off by default.
1:59:08So when you first, I don't know if that's good, but it's a thing that when, you know,
1:59:15if there were ever a time when the system thought that you might want to use ChatGPT,
1:59:21it's going to say, do you want to set this up?
1:59:23Right.
1:59:24So it's not, there's nothing that's happening automatically or whatever.
1:59:27You have to say, yes, I want to enable this feature.
1:59:31And you know, if someone for some reason just never wants it to be suggested to them at
1:59:35all, they also can just say, you know, never ask, I have no interest in this feature.
1:59:40And so if this isn't somehow, you know, insidiously integrated and pre-enabled or anything like
1:59:46this, it's just available if someone wants to turn it on.
1:59:51And it's free and it doesn't require a subscription.
1:59:54And your IP address is obscured and your requests are not logged, you know, so it was a lot
1:59:59of really cool things.
2:00:00There's a lot we did to make sure that if you were going to use it, that it was as good
2:00:03as it could be.
2:00:06So who's paying whom?
2:00:13I pay Craig.
2:00:14Craig pays Sean.
2:00:16Yeah, sorry, you know, no answer.
2:00:23You knew the answer.
2:00:26Can you tell me, is it a good story?
2:00:29Did I get Eddie up here?
2:00:35And a bottle of tequila.
2:00:38All right, last question.
2:00:41I've seen this floating about on social media, the same sentiment, but something to the fact,
2:00:47but I think it's everywhere because I think it resonates with people.
2:00:53I don't want AI to create songs and write poetry and make paintings so that I can wash
2:01:02my dishes and do my laundry.
2:01:06I want AI to wash my dishes and do my laundry so that I can make music and I can make paintings
2:01:12and I can make art.
2:01:16And I think it speaks to Apple as a tool maker for people who want to do things like that.
2:01:22It dates to the origins of the founding of the company.
2:01:26Do you guys see it that way?
2:01:29That you're making tools for creative people, not tools to replace creative people?
2:01:33Yeah, 100%.
2:01:34Absolutely.
2:01:35But I do think these tools will amplify people's creativity and capability.
2:01:43How many people now, I mean, just to go technology from the 80s could pick up a synthesizer and
2:01:51now make music that maybe previously involved in orchestra, but now they're composing something
2:01:58so much bigger because they have a powerful instrument at their disposal.
2:02:02I think these tools are going to enable people to, a single individual to express their creativity
2:02:09in bigger and bigger ways.
2:02:11I think that's tremendously empowering and that's what it's for for us.
2:02:17And that's what computing has always really been for is to augment human creativity.
2:02:24Good answer.
2:02:27Let me thank my guests, John G. Andrea, J.G., Craig Federighi, Jaws.
2:02:34Let's go with Jaws.
2:02:40I would also like to thank everybody here at the theater.
2:02:44I would like to thank my friends at Sandwich who shot the video, Spatial Jen who shot this
2:02:52and hopefully live streamed it.
2:02:56I would like to thank my wife and my son who are here helping backstage, Amy and Jonas.
2:03:04I would like to thank my sponsors, iMazing, Flexibits, Fantastic Alan Cardhop, and of
2:03:11course Flighty, thank you to them, and thank you to you for coming.
2:03:17This is great.
2:03:18And don't forget the helmet.
2:03:19Good job.
2:03:19[Applause]
2:03:37[End of Audio]