00:00:00 ◼ ► Welcome to Under the Radar, a show about independent iOS app development. I'm Marco Arment.
00:00:06 ◼ ► And I'm David Smith. Under the Radar is never longer than 30 minutes, so let's get started.
00:00:10 ◼ ► So today's episode is going to follow, I guess, a story of a very kind of up and down experience
00:00:19 ◼ ► I've been having over the last, unfortunately, couple of months in terms of some changes
00:00:24 ◼ ► I've been trying to make in Podometer++. And you may, if you're a long time listener of
00:00:28 ◼ ► the show, you may remember that back in October 4th, we even did a whole episode where I was
00:00:33 ◼ ► talking about how complicated it is to implement step counting and diving into that. And that
00:00:39 ◼ ► particular episode was motivated because at the time I was in the middle of making a big
00:00:58 ◼ ► pretty fundamental to the app and for it to be useful. And so it was a change that I didn't
00:01:10 ◼ ► out, I started getting these sporadic reports that Podometer++ was undercounting steps compared
00:01:23 ◼ ► at the time I went and did my kind of investigation, I started looking around, and as best I could
00:01:35 ◼ ► I was requesting step samples from Core Motion in the past. So I can ask for up to a week's
00:01:40 ◼ ► worth of data, and when I was asking for data from like four or five days ago, I was getting
00:01:45 ◼ ► undercounted values. And so my initial thought was like, "Oh no, they've changed the way
00:01:55 ◼ ► no way that I can realistically use Core Motion for this anymore. I need to switch to health.
00:02:00 ◼ ► Switching to health was really complicated and deals with a whole host of problems because
00:02:05 ◼ ► the health API is not nearly as sort of performant as the Core Motion API. Sometimes it just
00:02:20 ◼ ► minute delays from when you take a step before it would show up. And so I had to build this
00:02:25 ◼ ► really sort of crazy complicated system to kind of manage that and work through it. And
00:02:33 ◼ ► after probably honestly several weeks, maybe a month's work, I think I got to a place that
00:02:36 ◼ ► I was like, "Oh, maybe this works." But then I kind of got stuck. And this is, I think,
00:02:49 ◼ ► this change because I was sort of scared of making a change, messing everything up, somehow
00:02:56 ◼ ► missing some case in my testing, and fundamentally breaking the app for everybody. Because most
00:03:02 ◼ ► of the changes I make for a particular update, they may be a new feature, they may be a sort
00:03:18 ◼ ► was the case for this kind of a fundamental rewriting of the way that I'm acquiring the
00:03:23 ◼ ► data that is key to the app. And I was stuck. And I think that is an interesting state to
00:03:29 ◼ ► find in and for us to talk about because I feel like often we are going to find ourselves
00:03:33 ◼ ► in a situation where we are making a change. And this change could be, you know, it could
00:03:39 ◼ ► be breaking no matter how much testing you do, no matter how robust your system is, there's
00:03:43 ◼ ► always going to be things that potentially could come up. And sometimes this can be paralyzing.
00:03:53 ◼ ► like ostensibly testing this, but mostly just kind of pushing it down the road because I
00:03:59 ◼ ► was scared of what releasing it would actually be like. No matter how much I tested it, no
00:04:04 ◼ ► matter how sort of much I walked around with five phones in my pocket, like going on long
00:04:13 ◼ ► confident enough that I wanted to release it. And I think that fear is kind of a tricky
00:04:18 ◼ ► thing, because it pushed me into a place that I just wasn't being productive. I wasn't doing
00:04:29 ◼ ► like, maybe let's think about making a parameter plus plus for Android, let's go and get an
00:04:33 ◼ ► Android phone. Let's like trial like crazy, like extreme versions of work avoidance. Just
00:04:39 ◼ ► because the word I could never feel confident enough. And that paralysis, I think is just
00:04:45 ◼ ► like it is inevitable with whenever in time you make a big change. And I think it's just
00:05:07 ◼ ► tackle them eventually, like, you know, and like, I'm currently facing a couple now, like
00:05:18 ◼ ► my transition to airplay to what I really have to do is basically rewrite and, you know,
00:05:29 ◼ ► and a whole different approach. And that is really, really intimidating to do. And it's
00:05:39 ◼ ► so I haven't actually started throwing away the audio engine yet. So but that comes next
00:05:43 ◼ ► like that that is going to be the next step like that. I'm probably going to be starting
00:06:00 ◼ ► it forward into what it needs next for the market or whatever, it needs to totally change
00:06:07 ◼ ► needs to be something totally different. And so I'm facing, you know, a complete rewrite
00:06:22 ◼ ► doing well. Like in this weird list before I found it's like as the app does better and
00:06:28 ◼ ► has a bigger audience like the stakes increase in such a way that like if it breaks, it's
00:06:33 ◼ ► one thing if it like when an app is new and young and if you break something, it's like
00:06:38 ◼ ► maybe it'll affect a few hundred people or a few thousand. But as that number gets bigger
00:06:52 ◼ ► the thought of like, I'm glad that iTunes Connect or App Store Connect now has the feature
00:06:58 ◼ ► where you can roll out changes over seven days, which I think makes me slightly confident,
00:07:19 ◼ ► of inertia behind just keeping things going when they're working well. Like it's really,
00:07:23 ◼ ► really hard to to take a risk like that. You know, not only is it hard to justify the time
00:07:29 ◼ ► to do something like this sometimes, which we've talked about before, but it's also just
00:07:33 ◼ ► like it's a risk. Like like I have this audio engine that's being used by all these people
00:07:36 ◼ ► and it's working fine. And I wrote I wrote most of the core components in 2014, I think,
00:07:48 ◼ ► touch a lot of this stuff. It's almost like like some of the the programmers that I respect
00:07:55 ◼ ► the most in a field I will never ever, ever feel comfortable working in our programmers
00:08:00 ◼ ► who work on financial software. So, you know, what is code that deals with people's money,
00:08:05 ◼ ► whether it's banking or whatever that to me is very intimidating because if you screw up,
00:08:11 ◼ ► you could have massive consequences for you know, you could have legal issues. You definitely
00:08:17 ◼ ► would have accounting issues like you could lose your company a ridiculous amount of money
00:08:28 ◼ ► old like Fortran code. But there's a reason why because nobody wants to touch it because
00:08:32 ◼ ► it works. And if you touch it and if you touch it wrong, you have a big problem on your hands.
00:08:37 ◼ ► So that's kind of how I feel like with a lot of my, you know, like my like foundational
00:08:46 ◼ ► up but it certainly has a lot of that stress of like this is working. It's really important
00:08:58 ◼ ► to rewrite a lot of it to get these to get you know, new technology, new features, etcetera.
00:09:16 ◼ ► worse in some respects for some group of people because I, you know, no matter how big I make
00:09:29 ◼ ► up for somebody and that's a really hard thing to to take on. And, you know, it ultimately,
00:10:01 ◼ ► much more of a confidence that it will break than I used to. Oh, yeah, it always breaks.
00:10:26 ◼ ► enough. And like, that's one of those tricky things, these types of situations where, like,
00:10:31 ◼ ► there is a good lesson in this kind of change that like being like really cognizant of the
00:10:38 ◼ ► scope and the scale of the change and what the likely downsides are. And like that, it's
00:10:51 ◼ ► becoming reckless, like having just in general in life, like, whenever something stops being
00:11:09 ◼ ► that yes, this is, you know, like, this could be bad. I need to have a big beta, I need
00:11:13 ◼ ► to do all like do any kind of quality assurance that I can I need to test it. I need to do
00:11:19 ◼ ► code review, even for even if only for yourself, like something that I've started to do a
00:11:22 ◼ ► lot more is I'll end up like, you know, I do a diff in invert and get from my, you know,
00:11:31 ◼ ► go through all the all the diffs to make sure that they all like seem reasonable in isolation
00:11:35 ◼ ► after like a couple of days of not working on the project. You know, just trying to try
00:11:40 ◼ ► and catch like the weird like, why am I doing that? That seems really weird. Like, there's
00:11:49 ◼ ► the mistake enough. And now that I know I'm like that I have, like 100% confidence that
00:11:54 ◼ ► something is going to go wrong. And that confidence is like good, but can also make you get stuck.
00:12:10 ◼ ► in retrospect for my own situation. But is the like, the the, like the app is not getting
00:12:17 ◼ ► any better for me to sort of getting stuck. Like either the change is worth doing or the
00:12:23 ◼ ► change is not worth doing. And after we after our break, I'll have a bit of a turns out
00:12:28 ◼ ► crazy turn of events for this particular situation. But the ethic if the change is worth making,
00:12:40 ◼ ► But at a certain point, like don't get stuck. And I think that is the best advice that I
00:12:44 ◼ ► can give to myself and probably to anyone listening to this is, yes, it's kind of scary.
00:12:49 ◼ ► And yes, it's going to break and it's going to cause problems. But like either you have
00:12:54 ◼ ► to move forward or not. And if you if you're not going to move forward, like what are you
00:12:58 ◼ ► doing, you're just going to be spinning your wheels forever, which isn't is a disservice
00:13:07 ◼ ► We are brought to you this week by Linode. Linode gives you access to a suite of powerful
00:13:16 ◼ ► with your own virtual server in the Linode cloud in under a minute. Whether you're just
00:13:26 ◼ ► the choice for you. They are for the fastest hardware, the fastest network and the best
00:13:31 ◼ ► customer support backing it all up if you need help. It has never been easier to launch
00:13:47 ◼ ► other additional features beyond straight hosting. They have things like block storage,
00:13:51 ◼ ► load balancing, manage beta or manage backups. It's wonderful. I've been using Linode now
00:14:00 ◼ ► just great. And I have slowly moved everything I host over to Linode because they're just
00:14:14 ◼ ► a good value. The performance is great. I'm a very happy Linode customer and I have been
00:14:24 ◼ ► more specialized like maybe you want to run a mail server or run a VPN or run Docker containers
00:14:34 ◼ ► right now. If you want to maybe work there, go to linode.com/careers. Anyway, Linode has
00:14:39 ◼ ► fantastic pricing options available. Plans start at one gig of RAM for just $5 a month.
00:14:45 ◼ ► And they have wonderful plans that go up from there if you have higher needs than that.
00:14:48 ◼ ► Listeners of this show, go to linode.com/radar to support us and that'll get you $20 towards
00:14:54 ◼ ► any Linode plan. So on the $5 a month plan, that could be four months for free. And with
00:15:03 ◼ ► to learn more, sign up and take advantage of that $20 credit or use promo code radar2018
00:15:14 ◼ ► So if you listened to last week's show, you may remember a little anecdote I told where
00:15:20 ◼ ► I had to make sure that I didn't learn the wrong lesson from finding an old monitor buried
00:15:34 ◼ ► The right lesson is that sometimes hoarding technology will be useful, but hoarding technology
00:15:40 ◼ ► in general is probably not good for you. There's a tremendous selection bias and if you focus
00:15:46 ◼ ► on that selection bias, you'll have problems. So I have to do the same thing for this story
00:15:52 ◼ ► I'm about to tell about not learning the wrong lesson from the way this situation with Podometer
00:15:59 ◼ ► ++ ended up playing out. Because I ended up, like I said, I got stuck, I got paralyzed,
00:16:06 ◼ ► and this went on for months. Without exaggeration, probably at least two or three months. I've
00:16:11 ◼ ► been stuck in this place where I have this code that I think kind of works, I've tested
00:16:20 ◼ ► this time, people, users out in the real world, which is sometimes hard to keep in my mind,
00:16:26 ◼ ► are having this trouble. People are getting undercounted steps, and this is bad, and it's
00:16:30 ◼ ► kind of getting increasingly worse and problematic. And at one point, and this happened just last
00:16:36 ◼ ► week, I got an email to our support from someone who said they're having this undercounting
00:16:42 ◼ ► bug, and they said something in their email that totally changed and reframed what I thought
00:16:49 ◼ ► the actual problem was. Because they mentioned that the today widget was showing the correct
00:16:57 ◼ ► number, but the main app was showing the undercounting number. Which is one of those little things
00:17:05 ◼ ► where you never know when there's going to be a little clue that just completely reframes
00:17:10 ◼ ► the problem for you. Because suddenly, this issue that I thought was old data being underreported
00:17:16 ◼ ► turns out couldn't be the case, A, because this is happening on today's data. Like, the
00:17:21 ◼ ► today widget is strictly showing today's date, or today's step counts. And so, it's something
00:17:28 ◼ ► else, and the way that I query the data between the today widget and the main app is slightly
00:17:35 ◼ ► different. Because the today widget only has a concept of today, it doesn't have to store
00:17:49 ◼ ► asks Core Motion for the steps from midnight to today, or till now. So it just does one
00:17:55 ◼ ► query, whereas the main app had done the thing where, I don't know if you remember from episode
00:17:59 ◼ ► 143, where I dived into this, I query the data every 15 minutes, so I get these 15-minute
00:18:04 ◼ ► chunks of data, and it adds them all up for today. But the today widget doesn't do that,
00:18:10 ◼ ► it just asks for all of them. And so, when he said that, it was one of those, like, "Huh,
00:18:15 ◼ ► that's really weird." I go back to Core Motion, I go back to investigating, and it turns out
00:18:20 ◼ ► that the behavior that I was observing, where old data was being undercounted, was I was
00:18:27 ◼ ► seeing the wrong thing there. What was actually happening is data was always being undercounted,
00:18:43 ◼ ► buckets. I started asking for data in, rather than 15 minutes, into hour buckets, or two-hour
00:18:53 ◼ ► I asked for the data, I got wildly different numbers back. And so the actual bug, and this
00:18:59 ◼ ► is an issue that was introduced in Core Motion in iOS 12, is that when you ask for data in
00:19:07 ◼ ► short chunks, you are more likely to expose yourself to a bug that seems like Core Motion
00:19:13 ◼ ► has a greater than or equals to, that should be a greater than or less than or equal to.
00:19:18 ◼ ► Like, there's some kind of boundary condition issue where it isn't correctly counting steps
00:19:23 ◼ ► that happen across the interval that you're asking for. So if I was asking for steps from
00:19:53 ◼ ► have to deal with all the craziness with health, and mostly, the bug goes away if you ask for
00:19:58 ◼ ► it in hour-long chunks. It doesn't completely go away, but it seems like 80-90% of the bug
00:20:08 ◼ ► two time zones, like Nepal and Northern Australia, where they have weird half-hour or quarter-hour
00:20:20 ◼ ► won't even notice a difference if I switch to hour buckets rather than quarter-hour buckets.
00:20:25 ◼ ► And then I can very easily just ask for the whole data for the whole day, and if the numbers
00:20:29 ◼ ► are slightly different than, like, if I am getting undercounting at the hour level, which
00:20:34 ◼ ► seems pretty rare, if I am, I can take the difference and I can reapply it to the steps
00:20:39 ◼ ► that I saw to make sure that the two numbers match up. Like, super easy, it was about a
00:20:48 ◼ ► I need to not learn the lesson that my paralysis and my fear at not shipping the update previously
00:20:55 ◼ ► for the last two months, hurting my customers for all this time and all this problematic
00:20:58 ◼ ► thing, meant that I ultimately found the correct solution, and instead I need to learn, you
00:21:08 ◼ ► just wait around for this magic-like customer support query with this super clue that magically
00:21:22 ◼ ► - Wow, that, I can't believe that all that was the result of just this crazy little bug
00:21:36 ◼ ► - Yeah, I can imagine. Oh, wow, yeah, so I, one thing that I also want to talk about is,
00:21:43 ◼ ► like, maybe, like, some of these large changes and rewrites and things that maybe are not
00:21:49 ◼ ► worth doing, you know, and it's always hard to tell, you know, when something is new in
00:22:00 ◼ ► in HealthKit or whatever, I'm going to rewrite my whole audio engine to be AirPlay 2 compatible.
00:22:05 ◼ ► Sometimes it's hard to see past, like, 'cause in that moment, you are very excited about
00:22:09 ◼ ► that idea, and so you start, like, oh, man, wouldn't it be great, you're thinking of all
00:22:12 ◼ ► the positive things, wouldn't it be awesome if I rewrote this thing with this, you know,
00:22:17 ◼ ► if I did X, Y, or Z, that's this massive job, because then it'll be justified by having
00:22:22 ◼ ► benefits X, Y, Z, and it's hard at that time to see, like, is this going to be worth, like,
00:22:37 ◼ ► going to have problems to deal with that you otherwise wouldn't have, and it's hard to,
00:22:45 ◼ ► in that period of, like, early project optimism, it's hard to really see and to make a good
00:23:11 ◼ ► Overcast to be where everything about your user data is stored in iCloud and that Overcast
00:23:19 ◼ ► servers know nothing about anybody and that all they're doing is, like, crawling the feeds
00:23:23 ◼ ► and sending push notifications, like, whenever, like, things are new, like, this feed is new,
00:23:29 ◼ ► go and it would, like, notify everybody, you should go check this feed because it's new
00:23:37 ◼ ► I don't want people's data as much as possible, like, I'm trying to get rid of people's data
00:23:41 ◼ ► if I can, and in part because it would make running the servers dramatically simpler and
00:23:52 ◼ ► if I ever got hacked, there would be nothing really to see, if the, you know, if certain
00:23:57 ◼ ► servers go down, you know, it wouldn't be, it wouldn't impact the app that much if, you
00:24:07 ◼ ► have, you know, a way simpler server-side code base, so, like, this is a wonderful fantasy.
00:24:13 ◼ ► The reality of it is that that would require rewriting massive portions of my app and then
00:24:21 ◼ ► I would, and massive portions of the backend that currently, like, achieve certain functionality
00:24:26 ◼ ► by having user data, I have to move that into the app into client-side logic and certain
00:24:31 ◼ ► functionality would just no longer be available because I don't have, I wouldn't have, like,
00:24:35 ◼ ► the, you know, combined user data on the server, so you wouldn't have things like recommendations
00:24:39 ◼ ► be very easy, and then, and then I would have all these, like, all this, like, new code
00:24:58 ◼ ► lost time to get this in the system. Meanwhile, my users in reality would not care at all.
00:25:18 ◼ ► to me long-term, like, I, you know, my server-side would be much simpler and cheaper to run,
00:25:26 ◼ ► rewrite so much of the app in order to achieve a benefit that, if I think about it only,
00:25:33 ◼ ► like, optimistically, academically, it sounds great, but in practice, like, man, all that
00:25:38 ◼ ► bug potential, all that work, all that rewriting, sometimes these big, ambitious ideas aren't
00:25:44 ◼ ► worth doing, and I think that's one of those things that, like, it'll probably never be
00:26:01 ◼ ► Yeah, and I think it's so easy, and to jump to wanting to do it, because I think in the
00:26:10 ◼ ► I think that is, like, of all the things that I'm kind of trying to take from this experience
00:26:25 ◼ ► means that we can do anything, which is great, in the sense that it gives us flexibility,
00:26:31 ◼ ► it gives us the opportunity to be creative, like, there's lots of really cool things that
00:26:35 ◼ ► go into that, but it also means that we can be our own worst enemy and be making changes
00:26:41 ◼ ► and doing work that is ultimately not in line with the broader context of our users, or
00:26:53 ◼ ► it's, like, we only have our own perspective, and I think that's the thing that is probably
00:27:36 ◼ ► where I've worked, there likely would have been somebody who, as soon as I found my first,
00:27:52 ◼ ► Like, and then if I'd spent just, like, whatever it was two months ago, if I'd spent one more
00:27:56 ◼ ► day or one more hour or had a slightly different testing regime to work out what the problem
00:28:02 ◼ ► was, like, rather than being stuck in my kind of, like, very focused, like, you know, sort
00:28:16 ◼ ► trying to look for, like, rather than just being like, "Huh, in my current system, when
00:28:27 ◼ ► And in, like, in the case of what you're describing of, like, there could be these really cool
00:28:34 ◼ ► But if they don't benefit our customers, or they don't make, or like, they're not really
00:28:46 ◼ ► I mean, one area that we probably should save for another episode is, like, you also might
00:28:59 ◼ ► You know, like, that's, you know, we've been talking about code this whole time and, you
00:29:02 ◼ ► know, architectural considerations, but there's also, like, you could want to change your
00:29:05 ◼ ► entire business model up, and that comes with some of the same risks and thoughts and everything.
00:29:10 ◼ ► Obviously, you know, some stuff is different, but that's probably safe for a different episode.