Under the Radar

154: Knowingly Shipping Bugs


00:00:00   - Welcome to Under the Radar,

00:00:01   a show about independent iOS app development.

00:00:04   I'm Marco Arment.

00:00:05   - And I'm David Smith.

00:00:06   Under the Radar is never longer than 30 minutes,

00:00:08   so let's get started.

00:00:10   - So I find myself in the probably not unique position

00:00:13   this week of shipping as the app store version of Overcast,

00:00:17   a build that I know still has bugs.

00:00:21   - Ah, bugs in your software.

00:00:23   That's unheard of.

00:00:24   - I am knowingly shipping bugs.

00:00:28   I have a feeling this is not that uncommon,

00:00:32   or at least if we're honest with ourselves.

00:00:35   A lot of times we try to tell ourselves,

00:00:38   okay, I'm pretty sure this one works.

00:00:40   This build has no bugs.

00:00:42   I am shipping this knowing it's totally fine.

00:00:44   But of course that's never really the case.

00:00:47   What I want to talk about this week is

00:00:49   what has gotten me into this position

00:00:51   and kind of why we all do this,

00:00:54   and when it might be okay to ship bugs.

00:00:59   So I think first of all it's worth evaluating

00:01:03   how good we are at shipping software

00:01:06   that is truly bug free.

00:01:09   And I don't mean me and you,

00:01:10   I mean this whole business.

00:01:13   As far as I can tell it's impossible, yes.

00:01:16   Like as soon as your code does anything,

00:01:19   or interacts with any other code

00:01:21   like the operating system and its frameworks,

00:01:24   you introduce the potential for bugs.

00:01:26   And the more, and it doesn't really take,

00:01:29   like you know the more complex your app

00:01:30   the more bug potential there is.

00:01:32   But it doesn't really take a very seemingly complex app

00:01:37   like from the user point of view

00:01:39   in order to have enough complexity behind the scenes

00:01:41   where lots of bugs can possibly exist.

00:01:45   And so I would say, I mean I've,

00:01:49   as far as I know in my entire career so far

00:01:53   I have never written or shipped completely bug free code.

00:01:57   Do you know of any that you have?

00:01:59   - No, I think bugs are one of those things

00:02:03   that's like Zeno's paradox or asymptotic growth

00:02:07   where it's like all you can do is approach zero,

00:02:10   but it is impossible to actually reach zero.

00:02:13   (laughing)

00:02:14   Like and really, and I think probably

00:02:16   the more sobering thing that sort of keeps you up at night

00:02:19   if you think about it too hard,

00:02:21   is the nature of you can be in a position

00:02:24   that you have no known bugs,

00:02:26   but that is very different

00:02:28   than there not actually being bugs.

00:02:31   Like the nature of discovering something

00:02:33   is like half the problem most of the time.

00:02:36   It's like it isn't necessarily that,

00:02:39   like bugs are just like this obvious thing

00:02:43   that Xcode tells you about

00:02:44   or there's some obvious glaring issue.

00:02:48   Often the hardest bugs to track down,

00:02:50   or the worst bugs in many ways,

00:02:51   are the ones that don't like stick their head,

00:02:54   you know, stick their head up and raise their hand

00:02:56   and be like, "Here I am."

00:02:57   You know, like when you do this set of three actions,

00:02:59   this bad thing happens.

00:03:00   Like those are the best.

00:03:02   The ones that you're never gonna get,

00:03:04   like there's the ones that are just lingering there.

00:03:06   And honestly, sometimes I think there's a quite,

00:03:08   like the tricky thing too is that a bug,

00:03:11   like what is a bug?

00:03:11   Like in some ways it is when the application behaves

00:03:15   counter to your intentions maybe.

00:03:18   But the reality is our intentions

00:03:20   are never fully specified.

00:03:22   Like there's gonna be cases that we haven't accounted for

00:03:24   because we couldn't have accounted for them.

00:03:27   Or it's dealing with a new situation,

00:03:29   a new device, a new operating system,

00:03:32   a new user's set of data, a new bit of user input.

00:03:35   Like there are so many things

00:03:37   that are strictly impossible to deal with

00:03:40   that I think it is absolutely impossible

00:03:42   to ever ship bug-free code.

00:03:44   All we can do is hope to sort of approach it over time

00:03:49   and hopefully have like the,

00:03:51   I'm thinking back to my sort of corporate days

00:03:54   where you had like the bug burn down chart,

00:03:56   where it's like you have this graph of like known bugs

00:03:59   and it's like the goal is that that, you know,

00:04:01   that the slope of that line is going down rather than up

00:04:05   as you approach shipping.

00:04:06   But it's definitely something

00:04:08   that never actually hits zero.

00:04:09   It just sort of hits a point where it becomes stable

00:04:12   or the nature of the bugs is such

00:04:14   that they're not worth fixing

00:04:16   or it's an issue that would require way more work

00:04:18   than is actually possible, like is reasonable to do

00:04:20   or is possible to do.

00:04:22   Or in the process of fixing it,

00:04:24   you would actually introduce more bugs

00:04:25   and so it's better to just leave the bug you know

00:04:27   than the one that you don't.

00:04:28   But you'll never get to zero, sadly.

00:04:31   - It's such a wonderful goal to think like,

00:04:34   I'm gonna ship a bug-free version.

00:04:36   I'm gonna track down every single bug,

00:04:38   every single crash, everything.

00:04:39   I'm gonna track this all down

00:04:41   and I'm gonna ship a perfect version.

00:04:43   And of course the reality is like,

00:04:45   bug-free is, as you said, it's an impossible goal to reach.

00:04:51   All you can really hope for is relatively few bugs.

00:04:56   Or like, typical usage of the app won't hit bugs

00:05:00   as far as you know that would be bad enough

00:05:04   that people would actually notice and care.

00:05:06   - Yeah.

00:05:07   - And those are all a bunch of escape clauses.

00:05:09   Like every one of those, like every one of those phrases

00:05:12   there is like, that can let in huge classes of bugs

00:05:15   and have it be pretty much okay.

00:05:17   Because the reality is, bug-free code,

00:05:21   which is again impossible, but like trying to minimize bugs

00:05:26   has severely diminishing returns.

00:05:29   Because you will never complete it,

00:05:32   you have to draw a line somewhere of like,

00:05:34   how far am I willing to go?

00:05:36   Like how much time am I willing to spend fixing bugs?

00:05:38   And therefore, how much am I willing to not work on features,

00:05:43   not ship new versions of the app,

00:05:46   basically like how much of the app development process

00:05:49   am I willing to grind to a halt,

00:05:51   and for how long in order to fix bugs?

00:05:55   And because it's a complicated business,

00:05:58   because there's multiple factors,

00:06:00   because some of these factors matter more than others,

00:06:02   that is not always an easy question to answer,

00:06:05   and a lot of times the answer is not

00:06:07   what you think it should be.

00:06:09   Like ideally, like ideally a bug is a showstopper.

00:06:13   Ideally, like if you're trying to make high quality stuff,

00:06:17   the concept, the idea that something is wrong,

00:06:19   and that you know it's wrong,

00:06:20   and that you aren't fixing it,

00:06:22   that is in theory like morally wrong.

00:06:26   Like you shouldn't do it, you should drop everything

00:06:29   and fix that bug before you do anything else.

00:06:31   But the reality is, people have schedules,

00:06:34   people have limited time.

00:06:36   You know a lot of times if you're working for somebody else,

00:06:39   the decision is not in your hands,

00:06:42   and you just have a certain deadline you have to hit,

00:06:44   and the bosses might even tell you,

00:06:47   you can't spend time fixing that obscure bug

00:06:49   you want to fix right now,

00:06:51   you have to instead do this feature that we just sold,

00:06:53   you know through the sales team to a customer,

00:06:55   and you need to make this.

00:06:56   Like there's all sorts of external pressures like that.

00:06:59   You know we can even look at Apple software,

00:07:01   and you know Apple software's full of bugs,

00:07:04   and part of that's because they're a big company,

00:07:06   they have lots of software, there's lots of things,

00:07:08   and part of that's because Apple is a very

00:07:09   schedule driven company, and they are driven,

00:07:13   they prioritize the schedule above all else,

00:07:16   and secondly they prioritize the marketing feature list,

00:07:20   and so Apple, we hear a lot of times from people

00:07:24   who work at Apple who like,

00:07:26   there are certain like phases throughout the year

00:07:28   that it's more or less pragmatic to try to fix bugs,

00:07:32   and it's more or less likely to get any bug fixes done,

00:07:35   because if they're in like super crunch time,

00:07:38   you know working on whatever the new release is

00:07:40   for WVDC or for the fall,

00:07:42   and you try to report a bug that is really not

00:07:47   that bad of a bug, like it's not affecting very many people,

00:07:50   or it's not causing like data loss,

00:07:52   or you know anything really serious,

00:07:54   then they will prioritize that accordingly,

00:07:57   and it typically won't get fixed,

00:07:59   or at least it won't get fixed for a very long time,

00:08:03   and so like there's, we can recognize in our own indie life,

00:08:07   or corporate life if we're still corporate,

00:08:10   we can recognize that we have those same patterns,

00:08:12   we have those same you know prioritization cycles,

00:08:15   where like sometimes you just gotta get a build out there,

00:08:18   and so all this is building up to say,

00:08:21   what I'm doing with Overcast this week and last week is,

00:08:25   I've been hearing reports from people,

00:08:28   and seeing reports in App Store Analytics,

00:08:30   that the crash rate for Overcast 5

00:08:34   was higher than I thought it should be,

00:08:36   that like it was getting crashes,

00:08:39   that some of which were things

00:08:41   that I don't choose to worry about,

00:08:42   so for instance if I go into the Xcode organizer,

00:08:45   and I look at the crashes or the energy logs,

00:08:47   I get tons of crashes in system frameworks for the watch app,

00:08:52   and I couldn't possibly care less,

00:08:56   because I know that debugging the watch app

00:09:00   in crashes that are in system frameworks,

00:09:04   is simply a lost cause, that's just a time vacuum,

00:09:07   that I know if I actually try to do that,

00:09:10   it's gonna be the biggest waste of my time ever,

00:09:13   because chances are, because of the very little access

00:09:16   we have on the watch as you know,

00:09:18   chances are I probably can't fix those bugs,

00:09:22   and even if I could, it would probably take me

00:09:24   a very long time to actually find them.

00:09:27   - Yeah, and many of them aren't things

00:09:29   that users will ever see, because many of them

00:09:31   are happening in the background.

00:09:32   - Right, and so that's a big thing too,

00:09:34   like what I'm finding is many of my crashes

00:09:37   that are reported as crashes in the organizer,

00:09:40   many of those are background crashes,

00:09:43   and on the watch it's almost always like

00:09:44   somehow in the background I've used my two seconds

00:09:47   of CPU time that I have, and it's killing me,

00:09:50   and if it's not in an obvious way, or in an obvious spot,

00:09:54   that is simply not worth fixing for most of the time,

00:09:57   and you can see, we aren't the only ones who think this way,

00:10:00   you can see because if you open up the device panel

00:10:04   in Xcode and you look at the device logs for like your phone

00:10:08   it'll show you logs for all the apps in your phone

00:10:11   that have crashed or been terminated for resource usage,

00:10:13   and you can see like big popular apps,

00:10:16   almost every app you use there's gonna be

00:10:18   a crash log in that list, and sometimes apps crash

00:10:22   all the time in the background and you don't even know,

00:10:25   because the way iOS backgrounding works,

00:10:28   like if an app crashes in the background,

00:10:29   the only way you could notice is if next time you launch it

00:10:32   it takes a little bit longer to launch

00:10:33   'cause it's launching from scratch

00:10:34   instead of being resumed from suspension,

00:10:36   but even that is like not uncommon in iOS

00:10:39   for lots of other reasons, so it's really,

00:10:42   it's kind of crashing in the background

00:10:44   as long as the app wasn't in use at the time,

00:10:47   like if it wasn't like playing audio,

00:10:48   if it was just in the background and it crashes,

00:10:51   that's not that big of a deal,

00:10:53   and so that should be a very low priority bug.

00:10:56   Similarly, like if, like a lot of the bugs I get are,

00:11:00   or like really weird edge case things,

00:11:03   like a crash in core text or in JavaScript core

00:11:06   from one of my web views, and I'm like,

00:11:08   you know, there's only so much I can do about that,

00:11:10   like especially when it's not happening on a lot of devices,

00:11:13   like there's just not much that I really should do

00:11:16   about that, that's probably a system framework,

00:11:18   Apple has these logs too, and Apple probably looks at them

00:11:22   and they can see all the crash logs for all the apps

00:11:25   that go through their system, and they, I'm sure,

00:11:27   I haven't heard this for sure, but I would assume

00:11:30   that they have processes where they can analyze those

00:11:32   en masse and kind of triangulate like which system frameworks

00:11:35   might have bugs in them and which APIs might be causing bugs

00:11:38   and they probably look at that when they're figuring out

00:11:40   how to prioritize their own efforts.

00:11:42   So some of these things, I look at,

00:11:43   and it's like, you know, if it's some big crash

00:11:45   deep in some system thing, a part of a web view,

00:11:47   I'm like, you know what, that's not mine to fix,

00:11:49   that's Apple's to fix, and you know, it might be my problem,

00:11:54   you know, it isn't my fault, it might be my problem,

00:11:57   but only if it's affecting a large number of people,

00:11:59   and in that case it isn't, right?

00:12:01   And so what I look for when I'm trying to see what I can fix

00:12:05   is I look for like a crash that is in my code,

00:12:09   like ideally where like the stack trace includes

00:12:12   something in Overcast and not all system frameworks,

00:12:15   which is a little tricky sometimes, but you know,

00:12:17   it should include my code, it should be in the phone app,

00:12:20   I don't care about the watch app unless it's something

00:12:22   really obvious, and it should be something that is

00:12:26   affecting a large number of devices.

00:12:27   The great thing about the Xcode Organizer thing

00:12:29   is that it tells you this.

00:12:30   And also yes, I'm fully aware of other crash reporting tools

00:12:33   and things like that, you know, they have very similar

00:12:35   advantages too, I guess like using the Apple one

00:12:37   to keep things simple and keep privacy policies simple

00:12:40   and to be able to catch things like the energy termination

00:12:42   logs that the other ones usually don't get.

00:12:45   Anyway, so there's, every time I look at the crash list,

00:12:49   there's like 50 different crashes in there.

00:12:51   Like not 50 devices, 50 different identified crash sources,

00:12:55   but most of them are like, you know, five devices,

00:12:59   10 devices, it's like, you know,

00:13:00   there's only so much I can do.

00:13:03   - Yeah, and I think I have the same experience

00:13:05   where I look at a lot of my apps.

00:13:06   Like there's always a lot of things in that list.

00:13:09   I mean, I do a lot of watch work, and so like the first 40

00:13:12   of my crashes are just, oh what is it, SP remote,

00:13:16   I think is the name of it, what I like--

00:13:17   - PK service run, I believe is the one I see a lot.

00:13:19   - PK service run, there's a couple of these that are just

00:13:20   like these watch dog processes or these watch things

00:13:24   that you just kind of look at and you're like, yeah,

00:13:26   I've even asked the WDC, like I went to the labs

00:13:29   and talked to some watch kit people, like is this something

00:13:31   I should be worrying about?

00:13:33   And I think their answer was usually it's like,

00:13:35   it's, and honestly I kind of wish that this was better

00:13:38   communicated in Xcode is that there's kind of a different

00:13:41   concept between a crash and a resource limitation

00:13:46   or a system killing you situation, because sometimes

00:13:53   the system killing you is a result of your bad behavior.

00:13:58   But sometimes it isn't, and especially when you look

00:14:02   at the watch logs, sometimes the most comedic ones

00:14:05   are like you're being killed because the CPU is overused,

00:14:08   your contribution to that was like 1%.

00:14:11   And it's like, okay, so I didn't really, like something else

00:14:15   on the system is using a lot of CPU and the OS is going

00:14:19   around and just killing everything off to try and preserve

00:14:23   battery or whatever it's trying to do.

00:14:25   It's like, okay, well that's not really my problem.

00:14:28   It's like I want to look for the ones, yeah, I want to look

00:14:29   for the bugs that people are going to notice that are

00:14:31   actually causing issues for people that are things

00:14:35   that I can apply myself to and that my efforts are going

00:14:40   to yield actual tangible, useful results.

00:14:44   'Cause I think it's easy to, and I think honestly

00:14:47   the hardest thing is the more esoteric the bug is,

00:14:52   the longer it is going to take and the length and complexity

00:14:57   of debugging is absolutely exponential.

00:15:01   There's certain kinds of bugs, like the big obvious,

00:15:04   you hit a button and the app crashes.

00:15:07   Like, okay, some things, like I'm putting a nil value

00:15:12   into a dictionary or I'm de-referencing something

00:15:17   that I shouldn't de-reference, like there's some very obvious

00:15:20   like issue going on.

00:15:21   Happens every time I hit this button, it crashes.

00:15:23   Like absolutely fix those.

00:15:24   And then like it just, from there it just grows,

00:15:27   just impossibly for like in this one situation,

00:15:31   for this one device, in this particular like configuration

00:15:34   when other apps are running at the same time

00:15:36   and the moon is in a particular place

00:15:37   and the sun is shining, then this bug will happen.

00:15:42   And like at a certain point, like those are just like,

00:15:45   okay, I could spend months trying to track that down,

00:15:48   but I'll just ignore that and I will just sort of like

00:15:51   mark it as resolved in my mind and until such time

00:15:54   as it appears to actually hurt something,

00:15:56   I will just ignore it.

00:15:57   - We are brought to you this week by Linode.

00:16:00   Linode gives you a suite of powerful hosting options

00:16:03   with prices starting at just $5 a month.

00:16:06   You can be up and running with your own virtual server

00:16:08   in the Linode cloud in under a minute.

00:16:10   Linode has hundreds of thousands of customers,

00:16:13   including David and me.

00:16:14   And we are all serviced by their friendly 24/7 support team.

00:16:18   You can email them, call them, or even chat over IRC

00:16:21   in the Linode community.

00:16:22   They know how important it is to get the help you want.

00:16:25   And they also have a suite of amazing guides

00:16:27   and support documentation to give you a reference

00:16:30   when you need it.

00:16:30   And this is actually really nice.

00:16:31   Even if you aren't a Linode customer, you can view these.

00:16:33   They're public documents on the web.

00:16:35   And if you search for help on running a Linux server,

00:16:37   you will often find a Linode support document

00:16:40   just as the general help document

00:16:41   for running that kind of server.

00:16:43   They're that good.

00:16:44   And it's really way easier than you think

00:16:46   to get these up and running.

00:16:47   They also have a very intuitive control panel

00:16:49   to allow you to deploy, boot, resize, snapshot,

00:16:52   or clone your virtual servers in just a few clicks.

00:16:55   And they have amazing security features too,

00:16:58   such as two-factor authentication to keep you safe.

00:17:01   So Linode is great for things like database hosting,

00:17:04   mail servers, operating a VPN, Docker containers,

00:17:07   private Git servers, and so much more,

00:17:09   all the way up to major web apps,

00:17:10   like I'm running Overcast on it,

00:17:12   Dave runs his backend stuff on it.

00:17:14   It's great.

00:17:15   Linode has fantastic pricing options available.

00:17:17   Plans start at one gig of RAM for just $5 a month.

00:17:21   And they offer high memory plans starting at 16 gigs of RAM.

00:17:24   Listeners of this show can sign up at linode.com/radar

00:17:28   that will support us,

00:17:29   and you will get $20 towards any Linode plan.

00:17:32   So on that one gig plan, that could be four months for free.

00:17:35   And with a seven day money back guarantee,

00:17:37   there's nothing to lose.

00:17:38   So go to linode.com/radar to learn more, sign up,

00:17:42   and take advantage of that $20 credit,

00:17:43   or use promo code radar2018 at checkout.

00:17:47   Thank you so much to Linode for supporting this show

00:17:50   and Relay FM, and for keeping all of our stuff running.

00:17:53   - So what bug did you have to ship this week?

00:17:55   - So I'm shipping a bug that I'm getting reports

00:18:01   from, you know, so as I said,

00:18:03   I've been getting background crashes here and there.

00:18:05   I've tried to alleviate those with previous updates

00:18:08   'cause it seemed like it was usually for too much CPU usage

00:18:11   over too short of a time.

00:18:13   As far as I can tell, iOS does not limit CPU usage

00:18:16   if you're in the foreground.

00:18:17   But if you're in the background,

00:18:19   you are limited to 80% sustained usage.

00:18:23   And so they'll measure it about once a minute.

00:18:24   Or if you are using multi-core,

00:18:26   if you hit like, you know, if you max out all six cores,

00:18:30   you can blow that limit in like 12 seconds.

00:18:32   So it's surprising like how aggressive it is.

00:18:36   But basically, if you're averaging 80% over a minute,

00:18:39   you will get killed.

00:18:40   And so the problem was, you know, as I mentioned earlier,

00:18:44   like if that happens when the app is not running,

00:18:47   it's not that I don't care,

00:18:48   but it becomes a way lower priority, you know?

00:18:51   Whereas if it's happening when somebody is playing audio,

00:18:55   then it results in the audio stopping for them.

00:18:58   And that's not good.

00:18:59   That becomes a much bigger problem.

00:19:03   Because this brings it from like an academic purity exercise

00:19:07   to something that's actually negatively affecting users

00:19:09   in a real way.

00:19:11   So I've been trying to fix this.

00:19:12   I've been doing tons of test flight builds

00:19:15   over the last couple of weeks,

00:19:17   trying really hard to track these down, fix them.

00:19:19   And every build I set out includes in the release notes

00:19:22   to the testers like,

00:19:23   "All right, this time I think I really got it."

00:19:25   And then it gets out there-- - For real this time.

00:19:27   - Yeah, it's definitely the for realest time.

00:19:28   And it gets out there and then I don't have it.

00:19:30   Or what has fortunately happened is that over time,

00:19:35   I have, the current test flight build,

00:19:38   which I'm gonna submit to the App Store today,

00:19:40   the current test flight build does crash way less

00:19:45   than the current App Store version.

00:19:46   So it's like, I have made a significant improvement here.

00:19:50   The problem is not fixed,

00:19:52   but now there's an external factor.

00:19:54   Now we are a few days away

00:19:56   from the App Store holiday shutdown.

00:19:59   - Yay!

00:20:00   - And what happens with this,

00:20:02   so if you're all new or if,

00:20:05   for the people who listen to the show

00:20:06   who aren't developers and for some reason

00:20:07   you still tolerate all this,

00:20:10   the App Store, the whole backend, the app review,

00:20:14   everything, everything that's involved in

00:20:16   updating your apps, getting apps reviewed,

00:20:18   or changing your pricing to your apps or descriptions,

00:20:20   any of that, that all shuts down for a week

00:20:23   around Christmas every year.

00:20:26   Or about a week, it depends, but usually it's about a week.

00:20:28   And so this year the shutdown is from the 23rd,

00:20:32   so it's basically like Christmas week,

00:20:34   like 23rd to 29th I think.

00:20:37   And so you can't update your app at all during that time,

00:20:40   you can't change anything in the App Store during that time.

00:20:43   So if you wanna get an update in anywhere near Christmas,

00:20:47   you gotta do it basically now.

00:20:49   And that's why if you look at your App Store app

00:20:51   in the updates tab, there's probably a lot of updates

00:20:54   in the last few days because we have to get it all in early.

00:20:57   And the other problem is if you ship a major bug,

00:21:02   like a crasher, you can't fix it for a week.

00:21:06   That's really bad. (laughs)

00:21:09   So if you have some kind of horrible bug

00:21:12   that causes data loss or something,

00:21:14   you're just stuck with that for that whole week.

00:21:17   So what I'm trying to do is ship this version of Overcast

00:21:21   now, which is currently about four days

00:21:25   before the shutdown, five days.

00:21:27   That way I figure App Review is probably a little overloaded,

00:21:30   it's probably gonna take at least two days to get approved.

00:21:33   And then if I have some massive bug,

00:21:36   I have a couple days left before the shutdown

00:21:39   that I can try to rush a fix through.

00:21:41   And so because of that, I don't wanna make

00:21:46   massive mega changes to the app right now.

00:21:48   I wanna basically ship what's been running in TestFlight.

00:21:52   And one of the downsides of trying to fix every crash,

00:21:55   trying to fix every bug, is that your fix to the bug

00:22:00   that you think you're fixing might introduce

00:22:02   its own set of bugs.

00:22:03   Fixes are still writing code.

00:22:06   And I'm sure we've all done this,

00:22:09   where we try to fix a bug and the fix

00:22:11   is actually worse than the bug.

00:22:13   I'm running that risk.

00:22:15   - That was my experience for this last week.

00:22:17   - Oh yeah. (laughs)

00:22:19   - That is exactly what happened with me and pedometer.

00:22:21   I introduced it, I made a change,

00:22:23   then I thought I fixed it, and then I made it worse.

00:22:26   And then I fixed it again, and hopefully it's fixed now.

00:22:28   I'm not sure, more on that later.

00:22:30   - Yeah, I do that all the time.

00:22:31   'Cause a lot of times a fix looks easy.

00:22:34   It's like, oh, I'm inserting this nil value,

00:22:36   all right, well put a little check in there

00:22:38   that if it's nil, either skip it or omit it

00:22:41   or put in NS null null or something like that,

00:22:44   put in something else, put in an empty string,

00:22:45   whatever it is.

00:22:47   And then that itself might cause other weird bugs

00:22:50   to happen that you didn't foresee.

00:22:52   A bug looked simple, but it was actually,

00:22:56   the bug was causing behavior that avoided other bugs,

00:23:01   and by fixing the bug, you're now raining down

00:23:04   all that crap on some other part of your app.

00:23:06   - Or honestly, I think what often happens with me

00:23:08   is it's the issue of, so often bug fixing

00:23:12   is you're parachuting into this bit of code

00:23:16   that you get a stack trace for that says,

00:23:18   there's an issue on line 276 of this particular file.

00:23:23   This is where the crash happens, say.

00:23:25   I go in, I see the thing that could have caused the bug,

00:23:29   and I fix it.

00:23:30   But the difficulty in that is that when I wrote

00:23:33   all that code, I probably wrote it all at once

00:23:35   with a good sense of what's going on before and after

00:23:38   and all the dependencies and all the things

00:23:40   that are going on.

00:23:41   And I think the unintended bug's chain

00:23:45   of suffering comes so often, at least for me,

00:23:49   of where I end up being too myopic on,

00:23:51   oh, obviously, this is the issue.

00:23:54   And I'm diving in, I fix it, I change it.

00:23:57   It's like a two-line change.

00:23:58   So hey, I only changed two lines,

00:24:00   but I've lost that sense of the context

00:24:05   for what's going on there.

00:24:06   And so often, that is where I end up with these,

00:24:08   oh, right, yeah, no.

00:24:10   There's a reason why that value was invalid.

00:24:12   And when that value was invalid,

00:24:14   I can't just ignore it or skip it.

00:24:15   I have to actually deal with it.

00:24:17   Otherwise, all this other stuff is gonna start breaking.

00:24:20   Or now I'm letting some loop continue in an invalid state

00:24:24   and it's just making the problem worse.

00:24:26   Those are the things that I often find,

00:24:28   that making a change, that can sometimes be super dangerous

00:24:30   if you just take that view of, oh,

00:24:33   here's the line I need to fix.

00:24:34   Great, thank you, Xcode.

00:24:35   Let me fix that one line.

00:24:38   - So yeah, so that's basically what I've been dealing with

00:24:40   this past week is I just need to get this update out

00:24:44   and I have not found all the crashes yet.

00:24:46   Every test flight build I've sent out

00:24:48   still has some crashes left.

00:24:50   But they all have fewer crashes than the ones before them.

00:24:54   And so at some point, you just have to say,

00:24:56   you know what, I'm at a deadline.

00:24:59   I'm out of time.

00:25:00   I'm going to ship this.

00:25:01   This is gonna be overcast 5.0.5.

00:25:04   And I know it crashes sometimes for some people,

00:25:07   but it doesn't seem like it's that frequent.

00:25:09   And it's way better than the version

00:25:12   that's in the App Store right now.

00:25:13   So you can never achieve perfection.

00:25:17   So you have to know when to call it.

00:25:19   When to say, all right, this is good enough.

00:25:22   And if I've reached a point where most of the app

00:25:26   works totally fine for most people most of the time,

00:25:30   then that's as good as I can really hope for

00:25:33   because it's a big, complex app.

00:25:36   And the complexity is mostly just inherent to the problem.

00:25:39   It's not a lot of complexity that is just because

00:25:41   I'm being sloppy or over-engineering things.

00:25:44   Most of it's just like, it's a complicated app,

00:25:45   it's complicated things.

00:25:47   And maybe the edge cases where it's crashing

00:25:50   are only happening for users who have a lot of podcasts

00:25:53   or if they're running it on an iPhone SE

00:25:56   and its battery is throttling so it's running

00:25:58   at the performance of an iPhone 1,

00:26:02   and maybe that's why it's using so much CPU power

00:26:04   because the CPU is so slow.

00:26:06   There's so many edge cases that you can never

00:26:10   accommodate for.

00:26:10   You're never gonna write code that never gets terminated,

00:26:13   that never crashes, that never shows bugs.

00:26:16   But when you have a deadline, you just gotta ship something.

00:26:20   And shipping something is better than waiting forever

00:26:23   for perfection.

00:26:24   - Oh sure, and I think especially,

00:26:26   it is a really weird time of year

00:26:29   'cause I have it in the same position.

00:26:30   So like, we talked about over the last couple of episodes,

00:26:33   I have been doing all kinds of data changes

00:26:35   in pedometer++ and I had a situation where

00:26:38   I ended up that I had a much easier fix.

00:26:41   That fix dramatically improved things

00:26:44   for the majority of my users, awesome.

00:26:46   For a few of my users, it introduced an issue

00:26:50   where suddenly their step counts were undercounted.

00:26:53   And then I was like, oh no, that's terrible.

00:26:55   Like I hate it when I take people's steps away,

00:26:57   it breaks their streaks, it was awful.

00:26:58   So then I'm like, okay, let me go in and fix this.

00:27:00   And in fixing it, I introduced a bug

00:27:03   that dramatically overcounted steps

00:27:05   and people would end up with these days

00:27:06   with like 300,000 steps.

00:27:09   Because I was being so cautious about never making sure

00:27:13   that the step counts never went down,

00:27:14   that turns out I was actually, in very certain circumstance,

00:27:18   increasing them dramatically.

00:27:20   So I worked on getting that fix done.

00:27:21   And it's like all this is happening

00:27:23   within the back of my mind.

00:27:24   It's like, I have to have this done by the end of the week.

00:27:27   Because A, it's like Christmas is,

00:27:30   I mean it's not as big as it used to be,

00:27:31   but it's like this time of year is a big time of year

00:27:34   for the app store.

00:27:35   It's a big important time in terms of

00:27:37   people downloading things.

00:27:38   I make Apple Watch apps.

00:27:40   I imagine a lot of people are gonna have Apple Watches

00:27:42   sitting under their Christmas tree.

00:27:43   I have all this in the back of my mind.

00:27:45   And at a certain point, it's just like,

00:27:46   you gotta get it through.

00:27:48   And I'm just trying to get to the point that

00:27:50   I think right now, it's mostly good.

00:27:53   It's hard to know 'cause it's like the kinesia of the bugs

00:27:55   are really hard to reproduce.

00:27:57   But I think to your point, it's getting better.

00:28:00   And at a certain point, there's this magic line I think

00:28:03   where it crosses over, it's like it's good enough.

00:28:05   It affects a small enough group of people.

00:28:08   It's never great when it affects any people,

00:28:10   but it's like you're trying to cross this critical

00:28:12   threshold where it's either, it's an annoyance

00:28:16   for some people or for a large group of people,

00:28:19   or it's only affecting a relatively small percentage

00:28:23   of users and how small that is, it varies dramatically.

00:28:27   But there is this magic point where you're just like,

00:28:30   I think this has to be good enough.

00:28:32   And you kinda just ship it and hope for the best.

00:28:34   And then, yeah, it's like hopefully in the next few days,

00:28:36   things settle down before the App Store closes.

00:28:39   And then you have the one benefit of the App Store

00:28:41   is closed, you can't change it even if you wanted to.

00:28:44   - Yeah, it's true actually.

00:28:45   - Reassuring in a weird way that it's not gonna

00:28:49   destroy your Christmas because there's nothing you can do.

00:28:52   - Well, hope everybody has a wonderful holiday

00:28:54   if you celebrate Christmas, and if not,

00:28:56   at least enjoy a week off of no App Store.

00:28:58   So thank you for listening everybody,

00:29:01   and we'll talk to you next week.

00:29:04   - Bye.

00:29:04   [