PodSearch

Under the Radar

278: Unforeseeable Issues

 

00:00:00   Welcome to Under the Radar, a show about independent iOS app development. I'm Marco Arment.

00:00:05   And I'm David Smith. Under the Radar is usually not longer than 30 minutes, so let's get started.

00:00:09   So it's been a big week. We have the release of iOS 17, even, I believe now, the release of Mac OS Sonoma. Although I never jump on that right away. I always wait until a few versions.

00:00:19   Nope, nope, nope. Don't go anywhere near it.

00:00:21   But it's been a watch OS X, the release of the new phones, the new watches. It's been a pretty big two weeks since we last recorded. How has it been going for you so far?

00:00:35   So I think, so certainly stressful is probably the most honest way to start that discussion.

00:00:41   So one thing that doesn't help is I think for the last two or three years, so like the week between the fall event and the iOS launching, I almost always get a cold.

00:00:55   Just which I think is just a manifestation of the being so stressed and run down and then run up to it. And so that's always a fun thing.

00:01:02   But yeah, it was a rough week in that sense. It's always exciting. It's this weird tension, I feel like, and this weird, like dual feeling of like it's awesome and wonderful and it's terrible and awful.

00:01:13   And it's bouncing back and forth between those two feelings. Like there's nothing quite as fun as you release a new feature and you see someone who's super excited about it.

00:01:22   Like that's great. And then you, nothing quite as frustrating as you release a new feature and someone points out the obvious problem or flaw or bug in that feature that you've been staring at for the last two months, but you just never saw this issue or sort of saw this bug.

00:01:36   Did you make the fine moving cases?

00:01:38   I did not.

00:01:40   Sorry to pick on that whoever made those. It's just, it's the topic du jour.

00:01:44   But I think in some ways it's instructive in that sense of it is so difficult to understand the how, to keep a broad perspective while you're in the middle of developing something is very difficult.

00:01:59   And also understanding that every development process, every cycle that we go through of this is always having to balance constraints.

00:02:06   And they were always trading off, you know, I can make this better now and that will impact negatively this other thing and applies to physical products that applies to the software we make.

00:02:15   And it's just one of those realities. And like I always find it really frustrating. And it's like, it's just part of part of the deal is when someone suggests something that's like I knew I could do or could do better, or they were frustrated by and I knew they were going to be frustrated by but it was just a choice that I had to make to, you know, scope a project in a particular direction.

00:02:35   Or, you know, not add this feature even knowing that it would be desired but if I it's like I see all the other knock on effects of adding that feature and all the other problems that it would cause.

00:02:46   So yeah, it's a, I'd say it's a complicated time but it certainly is an exciting time. And yeah, it's like you hope that it goes well like you're, I think in the case of, you know, those cases they they didn't have a great launch, and thankfully this year I would say

00:03:00   overall things went pretty well for me. You had a few bugs had a few issues had a few things that probably worth talking about as they came up but overall like it seems to go well.

00:03:11   And, you know, at this point in the game I think a lot of that is just, it's funny because I think quality and that kind of thing is something that is, is it can be very amorphous and kind of ephemeral, but I think it's knowing the right things to test

00:03:26   knowing where the right thing where issues are likely to crop up. And having done that issue and I think we've talked about this over the summer of the kind of making sure that you do testing that's on old devices being sure that you do install the old version

00:03:38   on an old device like the most time consuming tests I ever do, or where I intentionally downgrade a phone to the old version, install the version from the App Store, get it set up, install the iOS update using the beta program, and then install the new version.

00:03:55   And I definitely caught a bunch of bugs and issues with that process it's very time consuming because every time you want to cycle through that it takes like two hours but yeah obviously you're not looking at it for two hours, but you can't iterate on it very quickly

00:04:07   but like, because I went through that it wasn't running wasn't running into the issues that I've had over previous years, where if I spend all my time, only working on the new, you know, like upgrade my own phone and my testing phones are all running in this case

00:04:20   I was 17, I can get so used to that and lose track of what's changed but at least for me, that's how it went. How did it go for you?

00:04:27   It was actually it was similar but on a much more compressed timescale unfortunately with much lower stakes. I've had a lot of stuff going on in life about moving trying to sell a house like it's been you know a lot of time consuming stuff and so I've had very little time to work on code

00:04:42   and this has been a very busy fall for me and I just I had done most of the work to get overcast iOS 17 widgets done a month or two ago but I just didn't have the sit down time to like get it out the door I knew that my current beta version had some issues on iOS 16

00:05:01   exactly what you were just talking about, not with the migration itself but just you know I was using all the new widget API's and in the process I had broken some stuff on iOS 16 and I just didn't have time to go back and test it for so long

00:05:12   so I actually notified my customers on Twitter or on Mastodon rather, it looks like Twitter because I'm using an app that's like Tweetbot

00:05:20   anyways I notified my customers on Mastodon like look I'm sorry I'm gonna miss day one I just don't have the time I need a few more weeks and it turns out that like the day before like the night before I started working on those things I had some time

00:05:36   and into the morning of iOS 17's release I actually fixed the problems really quickly like I thought it was gonna take a couple of weeks and it took a couple of hours you know sometimes in development often times it goes the other direction often times you think something's gonna be simple and it ends up being a huge time sink

00:05:53   this was the opposite occasionally you get a nice win like oh wait a minute I just accidentally disabled this thing or commented this out or broke support for this API and I can just go and re-add it you know so it ended up being faster than I expected and I submitted to the app store got approved in like 40 minutes

00:06:08   and was in the app store like right before they basically an hour or two before iOS 17 was released so I actually did make it but I ran into a lot of that you know I think there's a lot to be said and honestly this applies to Apple too

00:06:23   to avoid those surprises of hey I updated and this broke all of a sudden or hey this really obvious thing that you didn't think of why didn't you think of this or I don't understand this feature what do you intend here cause this doesn't make sense

00:06:42   all those kind of like initial gut feeling reactions that are hard to get when you're in it because it's your app and you're familiar with it or you're on the latest version of everything or whatever a huge amount of that value is possible through beta testing

00:06:56   but the problem with beta testing is that you have to constantly cycle through beta testers you know we talked about this a little bit in the past and I have found this to be true like I beta tested Overcast with the new widgets for I think a week or two at least but I wasn't getting all the feedback I needed

00:07:13   because I haven't cycled out my beta group in a long time and when you know there's this process of like I think of it as like beta fade you know people slowly stop paying attention to those test flight builds that come in every few days or whatever

00:07:28   and you slowly stop receiving feedback you know the most valuable feedback you ever get from somebody is the very first beta that they install cause that's the first time they're seeing whatever you're working on and that's when you get those initial gut reactions that you need as a developer who can't see them yourself necessarily

00:07:45   the problem is if you keep sending the same beta to the same group of people over and over again they might have paid attention to that first one they're not going to pay attention to ongoing changes nearly as much as that very first one

00:07:59   and so what you have to do really is make sure that you're constantly getting fresh eyes on what you're working on and that takes constant expansion or replacement of the beta group

00:08:09   and I think sometimes when Apple releases something that doesn't land as well as they would probably have hoped I think a lot of that is that their feedback loop doesn't bring in enough new and diverse viewpoints frequently enough maybe

00:08:23   sometimes it seems like when we as commenters say they misread the room I think that's often the reason is like the same group of people who they always run stuff by in their immense culture of secrecy and everything it's probably I'm sure it's almost entirely just employees of the company

00:08:41   maybe their immediate family is towards the end of a cycle or something but for the most part that feedback loop I think doesn't have enough new perspectives coming in all the time or doesn't have a broad enough view to sometimes get the feedback that they need

00:08:53   and we can fall into the same trap with our apps it's so easy for us to just have the same group of beta testers if we have any at all the same group of beta testers a handful of our friends and maybe a couple of power users that use the app

00:09:07   but who else is looking at that before you release it and inevitably no matter how much beta cycling that you do inevitably when you release it into the app store that's going to way more people with way more diverse backgrounds and way more diverse situations than any test flight beta group ever will have

00:09:25   and you will always find new things then but I think you get surprised a lot less often the more fresh eyes you have coming into your beta throughout the cycle

00:09:34   sure, and I think in that way it's like I think you can decide exactly how you want to gather that feedback and what that's going to look like and I think for me what's the challenging thing that I've run into recently is the sense of like I did a very limited beta testing for any of my updates this year

00:09:49   I think mostly it was mostly the kind of like close friends kind of thing where it's less like the beta testing in so far as finding bugs more like bouncing ideas off someone or getting kind of a design feedback and those kinds of more kind of higher level things but

00:10:07   that's in some ways it's a conscious choice because managing a beta process well, I think, is very useful but very time intensive in a way that you're asking someone to give you, like, to give you their time, and they may be very willing and very excited to do that but I want to be

00:10:27   able to, you know, sort of an appropriate like give it an appropriate amount of attention if someone is going to sit down with an app of mine and test it out and try it and put some effort into it for me to just respond it's like, thanks, and not really and I put that feedback to the side is not particularly great is not good for

00:10:43   my relationships with my beta testers or just, you know, my my mental health, and so in general I don't actually do very wide beta testing and instead the sort of place I've ended up settling to is the first like three days of a major update is in a weird way almost my beta period.

00:11:02   And so like for this updates for iOS 17, I issued several very quick turnaround app updates that were fixing issues or making, you know, clear clarifying points of confusion.

00:11:14   And, you know, I, it's almost, it's a funny thing for me to do but I feel like those first few days are both incredibly busy and incredibly unbusy in this oscillating period where I'll be sitting down and, you know, I'll get an email to the help desk system and I'll read it and then investigate it,

00:11:32   or I'm looking at the crash logs, or I'm listening to you know put people are saying in the press or a mastodon or things and I'm just trying to gather all the little bits and pieces that I can and see how quickly I can kind of put that into place because the reality

00:11:45   with these updates is, if you use the phase rollout and especially a year like this where it's like you know iOS 17 features. The initial period is only going to be going out to a tiny fraction of your users in some ways the same way that you would expect

00:11:58   with the beta period beta group that, you know, it's if at this point I think iOS 17, a week in is something like 20% for me in terms of the user base and so it's still very relatively small group of my users so I have a lot of time to gather that feedback

00:12:14   and respond to it and I think it's important for me to do that. And that is a useful way if a beta processes is sort of something that's difficult both in terms of gathering that audience or in terms of you have stale beta testers or whatever kind of the limits you have

00:12:28   there it's, I can be very useful and this is something I found a lot of success with is, you know, spending those first few days, trying to iterate and taking advantage of the fact that this is for whatever you know app reviews clearly highly staffed up highly engaged

00:12:41   like the number of times where I could submit an update and like I said you said 40 minutes I think I've had some of those I even a couple that would be in the 20 minute range that are very useful to be able to kind of iterate on it, and you're not going to be you're going to be cavalier in this that obviously you're still, you're pushing it out to production and anytime you push code to production you need to be very careful and be very aware of what you're doing and I think there's an element of that of just developing an intuition for what are the kinds of changes that you can make that are safe and relatively low-risk.

00:13:10   And what are the kind of changes that are difficult so like something, for example, that I did a couple of times in Widgetsmith for this update is I added additional help text inside of the app or changed the flow for a few things like one of the features is a new like music widget that's interactive so you can browse through your, you know, your favorite albums on your home screen and the process of adding songs and adding albums and adding playlists to that was slightly confusing.

00:13:39   The way I'd originally structured it. I got some feedback from people who in the in their excitement are saying it's like, I'm trying to do this and it's like, am I doing it right and they're trying to you can if you get that sense of their frustration.

00:13:50   And it's like, I don't want the last thing you need is your most excited engaged people who downloaded the app on day one to use your app to be confused. And if they're confused, and they care, the people who don't care as much are just going to bounce off that feature and so that's terrible.

00:14:03   And so it's like I added a bunch of text, and that's the kind of change that is super safe, relatively speaking, to add to something where I'm not changing some deep, some deep logic that was super complicated and depending on lots of things and part of the engine of the app or something

00:14:18   it's like no I'm just adding some text labels in places to make it a little bit clearer or restructuring things or making a few buttons, a bit bigger, or those kinds of changes. Like, those are great things to do quickly to iterate on and to gather that feedback, whether it's from a beta group or from your initial users to really make your app sort of really shine once it actually actually properly expands out to a broad audience.

00:14:43   We are brought to you this episode by Things, the award winning to-do app. Things has just been updated for all of Apple's new OS's, bringing some great improvements, especially for widgets.

00:14:53   As we know, your widgets are now interactive, which means you don't need to open the app to do things like complete your to-dos. They're also available in all sorts of new places.

00:15:02   Your Mac's desktop on Mac OS Sonoma, your iPhone standby feature on iOS 17, your iPad's lock screen, and watchOS 10's new smart stack. Things was also recently updated with one of its most requested features, adjustable text size.

00:15:16   So you can now scale Things' interface to a size that's right for you. Whether you want to shrink your text to fit a small iPhone or increase the size on a 30 inch display, Things gives you the control.

00:15:27   Of course, it isn't just the text that changes size. Things now uses vector graphics for all icons. I love that. And the layout has been finely tuned so that everything scales together beautifully.

00:15:38   I've used Things myself. I still use Things. It is great. It is my favorite to-do app. It is so, so good.

00:15:46   It doesn't look like just stock components. They do a huge amount of custom UI, but at the same time, somehow they make it feel super native to all the platforms they're on.

00:15:57   It doesn't feel like you're running an iPhone app on your Mac or vice versa. All the apps on all the platforms are first class. They've won awards. They have really, really great design and great features.

00:16:07   Things is an amazingly powerful app. It's built to run natively on all these devices. Extremely, extremely well made. So if you haven't tried Things, you should really check it out.

00:16:16   Just go to the website thingsapp.com. You can learn about all the apps' great features and download a free trial for your Mac there.

00:16:23   Of course, you'll also find it on the App Store. Just search for Things. Whatever it is you want to accomplish in life, Things can help you get there.

00:16:30   Try Things today at thingsapp.com. You will not regret it. Once again, thingsapp.com. Great app. You've got to check it out.

00:16:37   Our thanks to Things for their support of this show and Relay FM.

00:16:41   So I have a little bit of a story that I think would be a useful thing to dive into next about the launch that I think was an instructive capsule for a bunch of the different activities and things that typically come out around a launch.

00:16:54   And I hope that hopefully, in some ways I hope no one else runs into these things, but inevitably you will because this is going to talk about an app crashing and bugs and issues and compatibility problems and dealing with all that.

00:17:05   So, last Friday, so the day that the new iPhone 15s came out, I was going through my support queue, like I've been saying.

00:17:15   I wanted to go through my support queue and kind of keep an eye on what's going on in a way that I will at some point very soon turn off my support queue or at least make it much harder to find in the app. I kind of bury it a little bit.

00:17:26   But at this point, I want sort of maximum exposure.

00:17:29   That's interesting by itself, by the way. Let's talk about that sometime. But go ahead.

00:17:33   Yeah, that is a thing. I provide less support as the app gets older, for better or worse, mostly because I need to focus on things and it's less useful to gather that data. But nevertheless, last Friday I had it open and had it gathering things.

00:17:46   And I'm in this world where I'm responding probably within like 15, 20 minutes to new emails that are coming in.

00:17:54   Like I'm really in there trying to really understand what's going on, take advantage of if someone is seeing something, I want to know it. And someone reached out and said, "Hey, I've noticed something that my photo widgets in Widgetsmith, if I apply a photo filter to them, like a photographic filter, so you could make it black and white or sepia or more contrasty or whatever, those kind of filters, if I apply one to it, it doesn't show up anymore.

00:18:21   But if I turn the filter off, then it does show up." And that's anytime you see an email like that, where someone has been able to create a situation where the bug happens and then when the bug doesn't happen, they've done so much of the work for you. It's amazing. Feedback like that is incredible.

00:18:38   So I get this feedback and I'm like, "Oh, wow, that's weird. That's really strange." So I pull out my iPhone, my new iPhone 15 Pro Max, and I try the same thing. And the same thing happens again. I can reproduce it. It's like there's this issue where if I add a photo filter to a particular kind of widget, then it doesn't load. And if I don't, then it does.

00:19:00   And originally they had said it was related to one of my interactive widgets, where if you tap it, you can have what I call a drawer widget, where you have a photo and you tap on the photo and there's information, another widget behind the photo.

00:19:12   And so at first I thought, "Oh, this is some iOS 17 interactivity problem." It's like, "Oh, okay." So I went down that road and started investigating. And then it's like, "Huh, no, that doesn't seem to be the case."

00:19:21   And very quickly, there's this debugging process. And this is the part that I think is just worth maybe just sharing because getting good at debugging problems is like 80 percent of our job, functionally.

00:19:34   There is a meaningful amount of it that is the building part, and then a just as meaningful, if not more meaningful part is being able to identify problems and find them quickly.

00:19:44   So the first thing I do is try it in a different widget type, just a plain photo widget. Huh, same thing happens.

00:19:52   Then try, okay, is this based on certain kinds of photos? So do I take a big raw photo? Do I take a basic photo? Do I take a PNG file? Do I take a small, like create a little 10 by 10 photo and put it in there and see if it happens to that?

00:20:08   Doesn't seem to matter. It's like, okay, so I've now narrowed it down sort of dramatically in a way of what factors are coming into this.

00:20:15   It seems to have nothing to do with interactivity. It seems to have nothing to do with the type of photo that it is.

00:20:21   It's just something about applying photo filters to images in iOS 17. At this point, that was my working theory is this was an iOS 17 bug.

00:20:30   Then I'm like, huh, that's interesting because I hadn't run into this during the summer at all, but that doesn't mean that I, it wasn't the case because the reality of being a one person shop is that there are inevitably going to be things that I just miss.

00:20:43   That, you know, an app like WidgetSmith has so many screens and so many features that it is, you know, physically impossible for me to have tested and tried everything and then also have made meaningful work towards this progress.

00:20:56   And so it's like, I just assumed it was an iOS 17 bug and I tried it. So I was like, okay, let's try it in the simulator, see if it happens there. Like, nope, the simulator is fine.

00:21:04   So then I pull out one of my other iOS 17 phones. Actually, the first thing, of course, I did is I twilled out an iOS 16 phone, tried it there, and it didn't happen, which confirmed my unfortunate, unfortunately, it confirmed to my bias that it was iOS 17, which, you know, spoiler alert wasn't actually the case, which unfortunately, because I did that, then I was just like, oh, gosh, it's an iOS 17 bug.

00:21:24   So I was like, okay, someone who has this is going to have this issue. It's this big problem thing. Thank you for this person for identifying it. But, you know, what am I going to do? And so I try back and forth a few times to see if it was bad. And then I was like, someone reached out and I got a sort of an indication that it might not happen on iPhone 11s.

00:21:44   And I got a 2x 3x bug. And so then I, you know, because like my phone has a 3x screen, whereas the old the anything that has an LCD screen is still 2x. And so then I pulled out my iPhone 10R, which happened to be running iOS 17. And I tried on there, and it doesn't happen there.

00:22:01   So now in my mind, it's in my mental model, it's iOS 17. But only on 3x displays, maybe it only happens on red phones. Maybe it does like like this is the thing. I love that I had a reproducible bug in the sense of that made this much better. But then it's like now what do I do? I'm just trying to find out what it happens. So then I need to find one of my like, go to my drawer of old iPhones, find a 3x phone that supports iPhone 17.

00:22:30   Install it on there. And try it there. And then it worked. And so as well, so I'm like, huh? Well, this is interesting, because it happens 100% of the time on my 15 Pro Max, and 0% of the time on any other device that I can find. So that's interesting. And now I think it's a bug in the new phones, which in some ways was a small comfort to me, because that's a bug that I cannot in any way prevent happening.

00:22:58   Because I can do all that, you know, it's like Apple is very kind to us, they give us iOS 17 with like three months notice, even for minor bug, you know, typical updates, they often will beta test them that we have access to them. So I have an opportunity to install, like, we will probably have an opportunity to install 17.1, you know, for a few weeks, at least before it goes out to the public, most likely. So I can do testing ahead of time and see those bugs. In this case, turns out there was nothing I could have done because I had unfortunately I do not have access to an iPhone 15 Pro Max ahead of time.

00:23:27   And at this point, as fast as I can understand, this is an issue that relates to the new GPU architecture on the iPhone, I guess the A17 Pro chip. It's at this point, that's as far as I've been able to kind of narrow it down that it only happens on that it doesn't happen on the 15 and 15 Plus, it only happens on the 15 Pro and Pro Max. And so it seems to be something related to that chip.

00:23:50   Great, that's useful. That's very helpful to know. Now what do I do? And that was the sort of this one of these fascinating questions then of, do I pull the feature from those devices? Do I try and find a workaround? Do I change the feature in some way that only on those devices, and end up getting kind of it's always kind of complicated, because none of those scenarios are great.

00:24:12   Pulling a feature off an app in that way is always going to be disruptive. It's going to annoy people, especially because some of the photo filters are paid features inside of the app. And so I'm like taking away a paid feature from some of my users.

00:24:25   But in the end, like I spent the better part of a day trying to find workarounds, trying different options for, you know, core image, trying to force it to only process on the CPU in the case that might help.

00:24:35   And in the end, I don't have a way of, I haven't been able to find a workaround. I had some people who are very knowledgeable in core image take a look, and they weren't able to find anything. So for now, I've just had to pull the feature on those phones.

00:24:47   I was able to do a slight workaround for some of them, where you know, I can pre process some of the images ahead of time in the main app where it's fine.

00:24:55   But it doesn't work if for, you know, my photo album widgets, for example, which load images dynamically and then apply the filters to them because if you apply it, if essentially if you touch a core image filter inside of a widget extension, the widget extension will die because of memory pressure.

00:25:11   So I've just had to pull that feature and now I'm just kind of like waiting and seeing and who knows, I've filed a bug with Apple, I filed a developer tech support issue with this.

00:25:19   And we'll just see like hopefully, you know, from my perspective, hopefully it's one of those things of in the same way that I didn't have the opportunity to test this on the new phone ahead of time.

00:25:29   It's like Apple only sort of has the opportunity to really understand if their, you know, if their new architecture has issues for third party things because while I'm sure they download third party apps and run them, you know, they didn't necessarily try every feature of every app on their new architecture architecture and so fair enough, they didn't discover that applying a core image filter inside of a widget extension.

00:25:48   is problematic on their new architecture. But anyway, so it's in so now that updates been out this boot and pushed out. Generally, I haven't had any particular pushback from customers I think they understand like, if you go to that, you know that part of the app I have a message that says unfortunately if you're running one of the new phones that I've had to disable this feature while I work on a compatibility update, and it kind of went from there but anyway, hopefully that's an interesting story I just feel like these kind of things just are never really going to happen.

00:26:16   And I think it is important to try and keep like keep calm and carry on when that happens. And in this case, it's you know, it's not quite a happy happy ending at this point in the sense of I haven't found the workaround I haven't found a way or both you know, iOS 17.1 hasn't fixed it or something like that.

00:26:32   You know, and at the there were times when I discovered this, that I definitely had that moment of despair. You're like, Oh, no, this is going if when I first thought it was every device running iOS 17, their image filters weren't going to work. And there was no workaround that I could come up with that was going to work around that.

00:26:49   It's like that is really problematic and challenging and disheartening. And while I was able to narrow it down and make that less of a problem because, you know, at this point, point 1% of my user base has an iPhone 15 Pro. So, you know, it's not a massive problem in that respect yet.

00:27:05   But, you know, I think I this is not the first time this kind of issue has happened to me. And because I've experienced it enough times, I know that you'll get through it, it'll be fine. And in the moment, it's hard to do things like turning off a feature that you know someone's going to want.

00:27:20   But the reality is, is with some, you know, perspective, it's almost always been fine. It's the, you know, a few grumpy people is not something that you should certainly let get under your skin. And, you know, part of the job is working through these issues and keeping your keeping a level head around it and just kind of work the problem, try and isolate variables, try and work through it. And the more variables you can isolate, the better.

00:27:45   You know, I like what you said a few minutes ago of like, you know, when the new phones get out there, when the new OSes get out there, and when all the new third party app versions get out there, no matter how much testing you've done, like that combination has not been extensively tested by the world yet.

00:28:01   So, you know, Apple can't test all the apps before the developers have even submitted them. And even then, they can't test all the apps, you know, on the all the new hardware, like they only have so many people and only certain people have access to the pre-release hardware. Things are still moving, like your app is still moving, the OS is still moving, things are still in flux.

00:28:18   So it can be very hard to avoid having any issues when new things all launch at the same time. And so it's better to expect that, yeah, this might happen, like this, every so often there will be some weird problem that you have to deal with, you know, immediately after launch on, you know, in September on, you know, new phone or new OS day.

00:28:37   That's just part of being a developer. And not only is that an unavoidable part of your job, but I think also customers are fairly forgiving of that. Like, you get the occasional one star angry person, but for the most part, like customers know, like day one of something, they know that stuff might work a little bit wonkily for a little bit.

00:28:55   As long as you are, you know, fairly aware of it and plan to update on a reasonable basis, your customers tend to be pretty understanding.

00:29:02   Yeah, exactly. And I think that is a key thing in this experience is understanding that, you know, like the people who was who emailed in and said they were having this issue, and I emailed them back right away and say, Hey, this is the issue. Almost always, they're like, Oh, I'm so sorry, that's happening to you. You know, I look forward to a fix in the future, no problem.

00:29:20   And it's, you know, it's being responsive and being responsible. I think those two things together will go a long way to, you know, kind of making an issue like this. It's never great. It's never nice. But you can make it kind of manageable. And then it's just a thing that you deal with. And then, you know, it'll be behind us at some point, I'm sure.

00:29:38   Thanks for listening, everybody. And we'll talk to you in two weeks.

00:29:41   Bye.

00:29:42   .