00:04:59
◼►
A little bit about spotting the International Space Station.
00:05:04
◼►
Jonas wrote in to point us to a web app that they built that allows you to check when the space station was historically closest to a location, not just visible from it.
00:05:18
◼►
So, really, like, when it is directly overhead.
00:05:21
◼►
The link to this will be in the show notes.
00:05:22
◼►
It is very cleverly called Astronauts for Dinner, which I think is a very good name for this.
00:05:29
◼►
And you can plug in an address or coordinates and see the last time and all the previous times the International Space Station was overhead.
00:05:40
◼►
So, this was in another anecdote of me listening to the show last week as a listener.
00:05:46
◼►
The moment when you said about, you know, I used to, I like to know when the International Space Station is overhead.
00:05:53
◼►
I also had the same feeling as Federico of, why?
00:05:57
◼►
Like, what are you going to do with that?
00:05:59
◼►
Like, what do you do with that information?
00:10:13
◼►
Like, very few instances on my end of wanting to do something with Siri but not being able to speak to Siri.
00:10:22
◼►
And so in those cases, I actually did the double-tap on the home indicator to type to Siri instead.
00:10:30
◼►
I think, so there's a couple of problems here.
00:10:33
◼►
Obviously, the fact that Siri isn't great doesn't exactly incentivize the use of type to Siri.
00:10:40
◼►
And second, the activation method itself.
00:10:44
◼►
I think from the description that Gurman gave, I think it's going to be much easier to just pull down from the dynamic island, you know, kind of do it very quickly, than it is to precisely tap on the home indicator.
00:10:58
◼►
So the gesture sounds easier, in theory.
00:11:01
◼►
Man, I totally forgot that that double-tap thing even existed.
00:11:05
◼►
But if Siri will be smarter, then you will also be more incentivized to actually use it in a variety of cases.
00:11:12
◼►
I think, for me, there is an issue in having two different ways to invoke Siri, no matter what they are.
00:11:20
◼►
Like, where I feel like I would be happier to, you know, long-press the sleep-wake button and be able to then start typing or talking.
00:11:29
◼►
You know, like, maybe I tap another on-screen button to bring up the keyboard or something.
00:11:33
◼►
But there is, like, a cognitive load of, like, I want to talk to Siri.
00:15:02
◼►
That are the accessibility-focused features because they then get their own time in the sun.
00:15:09
◼►
And so, people that care about these features get, they see a lot of coverage about it.
00:15:14
◼►
But then also, we get, you know, like the media and us on podcasts, we spend time talking about these features where we otherwise may just gloss over them more quickly if it was part of the WWDC keynote because everything kind of gets glossed over quite quickly after the keynote.
00:15:31
◼►
So, there's a bunch of stuff going on here.
00:15:37
◼►
And I think that the overall thing is Apple intelligence coming to a lot of areas of accessibility.
00:15:43
◼►
I think that's one of the key things that is visible across a bunch of this stuff.
00:15:46
◼►
And I think that that's really great of like using models that can understand image and text and do things with those and people to be able to converse with these features a little bit more of allowing people who otherwise have issues with sight or hearing or mobility to be able to navigate the world using their devices.
00:16:41
◼►
And so, both VoiceOver and Voice Control are being kind of enhanced to allow for the system to be more aware of what's going on and allowing you to kind of converse with it.
00:16:54
◼►
So, they're using, for example, Apple Intelligence to better summarize images on the screen.
00:16:58
◼►
And you can ask questions of what you're seeing as well, which is fantastic, right?
00:17:03
◼►
So, you're just like, oh, this is that.
00:17:05
◼►
And you can say, well, what does this mean?
00:17:06
◼►
And similarly, you can now use the camera with VoiceOver so you can hold your phone up and your phone can describe to you what's in front of you.
00:17:15
◼►
And you can also ask questions to your phone backwards and forwards about what it is seeing.
00:17:20
◼►
And it can use some context to understand that.
00:17:22
◼►
And like this is, you know, in the conversations about potential AirPods or glasses with cameras, these are the kinds of features that, in theory, could be enabled by devices that are permanently attached to someone invisible.
00:17:38
◼►
Magnifier, as well, is also using Apple Intelligence to better describe what is on the screen.
00:17:42
◼►
So, this is the app that can essentially use the camera or I believe it works on the screen as well to basically make everything bigger.
00:17:49
◼►
You can ask audibly for Magnifier to zoom in on things or turn on the flashlight.
00:17:55
◼►
So, you can just have conversations with the devices that's doing this stuff.
00:17:59
◼►
And Apple kind of spent time talking about both Magnifier and VoiceOver can be used at the action button.
00:18:05
◼►
And there seems to be kind of like some additional features that the action button will be able to play in bringing up and activating these features.
00:18:12
◼►
I think the one that I looked at, you know, there's always something in the accessibility features that make people go, hang on a minute.
00:18:21
◼►
And it can be either design or functionality.
00:18:25
◼►
And I think voice control this year is the one that is maybe raising the most eyebrows.
00:18:32
◼►
So, I'm going to read from Apple's press release.
00:18:35
◼►
The option to say what you see is a feature they have is great for navigating any app, including those with visual layouts, such as Apple Maps or files.
00:18:45
◼►
Using intuitive language like tap the guide about best restaurants or tap the purple folder, this can help users overcome barriers when elements aren't properly labelled for accessibility.
00:18:55
◼►
So, essentially, a user who maybe has some mobility issues that allow them to control their phone correctly can just speak to the phone and ask it to select stuff.
00:19:07
◼►
Now, that is very interesting because that is agentic operation of iOS, essentially.
00:19:16
◼►
Isn't this the paper they published like a year ago that we talked about, Ferret 2?
00:19:25
◼►
I would not have remembered that as a good poll.
00:19:27
◼►
But, yeah, I think this is very interesting, right?
00:19:30
◼►
And, like, they show a video in the press release of somebody, like, asking, like, select the orange folder.
00:19:39
◼►
There's, like, a green, orange, and yellow folder.
00:19:41
◼►
And then you see the phone select the orange folder.
00:19:44
◼►
And so, like, this kind of stuff is really interesting because it is showing the iPhone and the user being able to have a dialogue and do things on the device.
00:19:57
◼►
Which is kind of, at the moment, I mean, we're going to talk about Google I.O. in a little bit, seems to be one of the big new areas of attack, right?
00:20:06
◼►
Of, like, controlling devices and how that's done.
00:20:11
◼►
And then a couple other things that were, I thought, we'll call accessibility reader can now handle more complex page layouts that may include tables, columns, and images.
00:20:21
◼►
So, it can, this is essentially giving people more ways and tools to be able to change fonts, change sizes, you know, like, maybe, like, for some, like, dyslexic fonts as well, like, fonts for dyslexia.
00:20:33
◼►
They have those in there, so it can kind of adjust text.
00:20:43
◼►
And then one that was just, like, wild, and they have a really good video about this in the first release, is Apple Vision Pro wheelchair control.
00:20:53
◼►
Where, essentially, for wheelchairs that can connect, and Apple's working with a couple of partners about building a connection between the two things, you can use the eye tracking functionality of the Vision Pro to control and move a connected wheelchair.
00:21:11
◼►
So, it shows the UI that the user is seeing, where they're essentially got, like, like a D-pad, I'll say, you know, like video game controls.
00:21:20
◼►
And using the eye tracking, you can, like, look at the go forward, and it will go forward, look at left, return left.
00:21:26
◼►
And the video is astounding, like, to see this in action.
00:21:30
◼►
And it, you know, really made me think, like, the Vision Pro is not the device for this, right?
00:21:36
◼►
And you don't want to be wearing it all day, every day.
00:21:39
◼►
But you can imagine a world in the next few years where this functionality is built into something more wearable, like, all day.
00:21:47
◼►
And it's going to absolutely change the lives of people.
00:21:50
◼►
Because they'll be able to have autonomy in their movement where they otherwise may not have had.
00:21:59
◼►
Like, so yeah, every year this is interesting stuff.
00:22:03
◼►
But I think the combination of what they're doing with Apple Intelligence and then this Vision Pro thing, I think is, like, I think it's super cool.
00:22:12
◼►
Yeah, I mean, Apple does such a thorough job with all this technology.
00:22:16
◼►
And one thing that, you know, we've known from hearing interviews from people who work on this at Apple and our friends who cover it more closely, like, Apple doesn't develop these features in a vacuum.
00:22:27
◼►
Like, they're working with people whose lives will be changed because of this technology.
00:22:34
◼►
And it's very grounded work in a way that I really appreciate.
00:22:38
◼►
And yeah, the Vision Pro thing definitely jumped out at me.
00:22:41
◼►
It's like, that seems a little strange.
00:22:43
◼►
But yes, so much of this is, like, building towards the future.
00:23:01
◼►
Like, all of this stuff builds together.
00:23:04
◼►
These aren't necessarily discrete features or technologies, but, like, they're building these things together in a way that they can be used together in meaningful ways.
00:23:14
◼►
And, like, I always find these posts so exciting and encouraging because Apple really does care about this stuff.
00:23:20
◼►
And I think it shows in the way they go about developing them.
00:23:24
◼►
This episode of Connected is brought to you by Squarespace, the all-in-one website platform designed to help you stand out and succeed online.
00:23:34
◼►
Whether you're starting something new or you're scaling something up, Squarespace gives you everything you need to claim a domain name, showcase your offerings with a professional website, grow your brand, and get paid all in one place.
00:23:50
◼►
One of the things I love about Squarespace is that it can grow with you over time.
00:23:55
◼►
So that you've got a site that you built on Squarespace maybe a couple years ago, and now you've got new branding for your company or your project.
00:24:03
◼►
Well, you don't have to go around page by page and change each color and each font and make sure it all lines up because Squarespace design tools are really easy to use.
00:24:13
◼►
You can make site-wide changes really easily and really quickly.
00:24:17
◼►
It's easy to understand what's going to happen with their design tools.
00:25:44
◼►
Because I find the whole Google I.O. keynote to be rough.
00:25:47
◼►
What I love and wait for is The Verge.
00:25:50
◼►
The Verge posted a 35-minute video, which was all of the big announcements.
00:25:56
◼►
I watched that, and I think I got everything that I needed from that and then reading some articles.
00:26:02
◼►
But the whole Google I.O. keynote, I mean, it suffers from the problem, and I actually think is evident in what we're going to talk about, of too many teams working on similar things who all get their time on stage.
00:26:17
◼►
Which is just, like, a fundamental issue with Google and how it operates as an organization.
00:26:22
◼►
I think Ben Thompson referenced it as, like, the keynote is the structure.
00:29:55
◼►
Yeah, like, I don't know what this does.
00:29:59
◼►
Like, a lot of the other stuff that they're showing, you can see how, and I understand not everybody likes it, and I get it, but you can see how people's lives could be made more productive, potentially, by some of these tools.
00:30:19
◼►
And any worthwhile application outweighing the vast problems that will be created by this technology existing.
00:30:36
◼►
And, like, they follow up, you know, they show, like, oh, look at Gemini Omni, it's amazing.
00:30:40
◼►
And, like, oh, and by the way, like, we're also now building tools that will help people understand if the content that they're seeing is generated by AI.
00:30:50
◼►
It's like, is that supposed to, like, give you an out, right?
00:30:55
◼►
That, like, oh, people could just ask Google if this video or photo is real or not, and either we'll, you know, we'll have the credential check stuff that people have to opt into.
00:31:08
◼►
But I'm still not sure how that's actually supposed to work, but whatever.
00:31:11
◼►
But then they're like, oh, you know, we're going to use a bunch of functionality to be able to detect it.
00:31:15
◼►
But, like, that's all great, but it still requires people to think, to ask.
00:31:21
◼►
And the issue is the people that think to ask are not the people that you need to worry about, right?
00:31:29
◼►
Like, if somebody sees a video or an image and they're like, I'm not sure if that's real, awesome.
00:32:33
◼►
And if you can't see that, like, I don't know what to tell you.
00:32:37
◼►
I would expand your angsty question, though, to more than just image and video generation.
00:32:44
◼►
Although I wholly have it about those things.
00:32:47
◼►
At some point, we're going to talk about some of the, frankly, bonkers things that Google DeepMind CEO said.
00:32:53
◼►
But if you really, like, all of these companies, well, not all of them, many of these companies will say, like, this is very powerful technology.
00:33:05
◼►
And it comes, basically, it's the Spider-Man quote of, like, with great power comes great responsibility.
00:33:12
◼►
But I don't believe any of these companies will be able to exercise the responsibility that they say is needed.
00:35:37
◼►
But my point is, I'm trying to rein this back in, if you really think, if you really think AGI is on the horizon, if you really think, if you're anthropic and you really think that Mythos is too powerful to put into the world because it's going to break all software humanity has ever written.
00:36:13
◼►
Smoking, drugs, using windows, whatever it is.
00:36:17
◼►
And so many times those conversations heartbreakingly end in somebody saying, I know it's bad for me, but I'm not going to stop.
00:36:25
◼►
And I just, the whole time I watched it, I watched the whole Verge thing and a bunch of other stuff.
00:36:30
◼►
And that was the feeling I had the whole time.
00:36:33
◼►
It's like, if you really think, if you really think we're this close or that it's on the horizon or we're on the foothills or whatever analogy you want to use.
00:36:42
◼►
If you really think, oh, we need to build a whole system so people understand that the video they're watching is real or fake, stop making the video creation tools.
00:36:54
◼►
Now, what their answer is to this is, well, no one else is going to stop, but we are the ones in our eyes who are going to be responsible with it.
00:37:02
◼►
So we have, this was how, why OpenA was founded and why, you know, they were initially a nonprofit.
00:37:07
◼►
It was, we're going to build up this technology so we know how to control it and we know how to be put responsible safeguards around it.
00:37:14
◼►
So when other people inevitably get here and do similar things, then we have an understanding of how to manage it.
00:37:42
◼►
And this is being provided to me by the AI overview, which is just like, what are we doing?
00:37:47
◼►
A hypothetical future tipping point where artificial intelligence and technological growth accelerate so rapidly that humanity is irreversibly transformed.
00:37:58
◼►
Now, to try and throw Hassabis a rope here.
00:38:03
◼►
I think what, maybe what he was trying to say is, if we can get to the point where we cure all disease, that is essentially singularity moment.
00:38:17
◼►
Because humanity is irreversibly transformed for the positive at that moment, right?
00:38:42
◼►
And if you just follow that thought through, it's like, yes, okay, if you believe that's true, we are standing in the foothills of the singularity.
00:38:52
◼►
However, I recommend you do not use that language because it sounds unhinged.
00:38:59
◼►
And also, the singularity sounds really scary as a phrase.
00:39:05
◼►
And it's like, are we in a simulation kind of stuff?
00:39:46
◼►
It can build a real-time simulation based on real-world physics.
00:39:52
◼►
That's like the short description of it.
00:39:53
◼►
So if you're thinking, we want to achieve the singularity and we want to be able to cure every disease,
00:40:00
◼►
I think it's obvious that you start from a place of, well, let's teach the AI how physics work.
00:40:08
◼►
So that in a not-so-distant future, when we want to put in a nanocamera inside of your body and the nanocamera needs to analyze blood flow and muscle structure and bone density,
00:40:19
◼►
it starts from a place of understanding how real-world physics work.
00:40:25
◼►
My problem with all of this is that I think if they're not lying about this—so big emphasis on this sentence—if they're not lying about this, that is a noble goal.
00:40:39
◼►
And it makes sense to start from a place of giving the AI knowledge about the world, about physics, about how walking on solid ground works, how gravity works, how the sky looks.
00:40:55
◼►
Like, it all makes sense that you start there.
00:42:39
◼►
But to go from that to, ah, just type in a prompt, hey, make me a fake video of Osama Bin Laden actually being alive and chilling in Paris.
00:43:26
◼►
Like, I could somewhat understand why these companies were pushing so hard on photos and videos when they didn't have an interesting product.
00:52:54
◼►
And I think I'm not, like, I'm not, like, this is not super close to me, but I feel like, you know, I've explained this before.
00:52:59
◼►
If I'm Googling for me, usually, is I have an idea of where I want to go.
00:53:04
◼►
And I use ChatGPT for searching for things that I don't know about yet.
00:53:08
◼►
And so, like, when I'm Googling, I usually have a destination in mind.
00:53:15
◼►
That is very, like, in line with showing me links to stuff.
00:53:21
◼►
And if they're going to make that harder and harder, maybe I just need to find another search engine where I can just find links to stuff, you know?
00:53:29
◼►
Because that's what I'm looking for out of that product.
00:53:32
◼►
Because I don't find Google's AI overviews to be sufficient for the bigger stuff.
00:53:40
◼►
Because they try to serve it too quickly.
00:53:42
◼►
And I think that's, like, the fundamental issue with the AI overview.
00:57:52
◼►
It's basically just a bunch of markdown files from this person, Drew, on GitHub.
00:57:58
◼►
And the skill, I was fascinated by the idea because it was basically a bunch of markdown files that taught CloudCode how to put together shortcuts.
00:58:09
◼►
And this is, like, the core of the whole thing.
00:58:12
◼►
Shortcuts behind the scenes have always been fancy XML files.
00:58:27
◼►
I was going to say, our podcast is an XML file.
00:58:29
◼►
It's always been XML, and realistically, it'll always continue to be.
00:58:36
◼►
So, this has always been the case since the workflow days.
00:58:40
◼►
And even after workflow was acquired by Apple, a shortcut is essentially an XML file that gets signed into a proprietary Apple plist format that itself gets saved as a .shortcut file.
00:58:57
◼►
Right now, you can export any shortcut from your shortcuts app as an XML file.
00:59:04
◼►
That skill that I discovered at the beginning of 2026 essentially contains the information for Cloud to learn how the XML syntax worked behind the scenes for different actions, variables, parameters, conditional blocks, repeat blocks, everything.
00:59:26
◼►
And Cloud code could say, well, okay, I know the syntax, now I'm going to put together a new XML file for a new shortcut.
01:07:06
◼►
And that's, and that's, and that was the other breakthrough.
01:07:09
◼►
Like actually putting the agent in a long running loop for multiple hours at a time.
01:07:14
◼►
And then about a month ago, I thought I had it.
01:07:20
◼►
So I started putting together everything that you see on Mac stories today.
01:07:24
◼►
the redesigned shortcuts, the redesigned shortcuts archive, the dedicated mini website for, for, for shortcuts playground.
01:07:32
◼►
And there's a companion shortcut that's for paying members only.
01:07:38
◼►
That, that's also like the crazy part.
01:07:42
◼►
The companion shortcut is called shortcuts playground remote, because I wanted to convey the idea that it's an optional, like remote experience thing.
01:07:50
◼►
That is a shortcut that creates shortcuts on device.
01:07:57
◼►
You actually run it on your iPhone and iPad and you type in an idea.
01:08:01
◼►
And a few minutes later, you see a new shortcut import window on your iPhone or on your iPad.
01:08:08
◼►
And you may wonder how that's possible.
01:08:10
◼►
It's actually talking behind the scenes to your plugin installed on your Mac via SSH.
01:08:18
◼►
And it's getting the shortcut file back as, drum roll, base 64.
01:08:46
◼►
But since I started from an open source skill, and since I, throughout this process, I drew from a lot of previous open source projects for shortcuts.
01:08:59
◼►
That's why I decided that the plugin should be free and open source for everybody to use.
01:09:06
◼►
There are a lot of nice things about the plugin.
01:09:11
◼►
Like, for example, you can ask for shortcuts to be installed and to have a specific color or to have a specific icon glyph.
01:09:22
◼►
And there's actually support for icon synonyms.
01:09:25
◼►
So it actually supports the official shortcuts glyph library.
01:09:30
◼►
But if you don't want to be using the exact name of an icon that Apple is using, you can ask for a synonym of that icon.
01:09:38
◼►
The shortcut knows how to make health kit actions.
01:09:42
◼►
Sorry, the plugin knows how to make health kit actions.
01:09:46
◼►
It knows how to make complex actions with conditional blocks, API calls.
01:09:53
◼►
Anything you can imagine, you can ask.
01:09:56
◼►
And the final thing I will say is, since it's running in an agent on desktop, this thing is running on macOS.
01:10:08
◼►
So one of the nice things that I've discovered is, if you ask shortcuts playground to, like, can you make me a Todoist shortcut for, I don't know, getting my tasks for Todoist due today?
01:10:22
◼►
And let me choose which ones I want to reschedule to another day?
01:10:26
◼►
The agent will actually test the Todoist API on its own before building the shortcut.
01:10:36
◼►
This was actually one of the best practices that I sort of infused myself into the plugin by telling it,
01:10:47
◼►
anytime the user asks you to make a shortcut that involves a web API, try to test the web API yourself so that you don't hallucinate URLs or API responses.
01:11:16
◼►
Like Mark Garman said, in the updated shortcuts app, users presented a prompt asking what do you want your shortcut to do, along with a text field to describe the request.
01:11:26
◼►
The system then automatically builds and installs the shortcut on device.
01:11:29
◼►
So, sounds pretty good, but you never know, right?
01:11:32
◼►
Like with this stuff, you never know how good it's going to be or not.
01:11:36
◼►
So, maybe there'll still be a case for shortcuts playground even in a 27 release here.
01:11:43
◼►
I think this is something that is definitely very impressive and also hard to get my head around, right?
01:12:00
◼►
But I think what's different about this one is that it's actually useful for other people.
01:12:07
◼►
Whereas in the past, when I did things like the MacPad, for example, it was more, I don't want to say self-referential, but it was always to prove a point to myself and share a story about myself.
01:12:20
◼►
Whereas I think what I'm trying to do now and what I will continue doing is release more free and open source things for other people to use.
01:12:33
◼►
Because I feel like I want to be able to, when people ask me the question of like, yeah, sure, but what have you built with AI that's actually useful or that's actually making you money?
01:12:47
◼►
I want to be able to point at stuff and I'm very happy today because when people, now when some people ask, oh yeah, but what have you built with AI that's actually useful?
01:12:58
◼►
I can point them to something practical, the new shortcuts archive on Mac stories, the plugin, the shortcut for members, like that is real stuff that is helping our business.
01:13:10
◼►
And of course, the backdrop of it all, I learned a lot.
01:13:16
◼►
Like I learned things I had no idea about and it was the first time, this was the real first test for this new way of working that we have a Mac stories where there's me or John and we have our agents.
01:13:34
◼►
And at the end of the process, we have our developer that's review, who's reviewing all the work of these agents and who's actually taking the output of our ideas and putting them live on Mac stories.
01:13:48
◼►
So like it's this new way of like hybrid work.
01:13:52
◼►
I don't know how to describe it, but if I like the truth, like if I didn't have agents, there would be no shortcut archive.
01:14:17
◼►
I mean, this, obviously we know you've been doing this for a while and there's definitely been some conversations about like, is this thing going to last?