Under the Radar

112: Ideal vs. Pragmatic


00:00:00   Welcome to Under the Radar, a show about independent iOS app development.

00:00:03   I'm Mark Orment.

00:00:04   And I'm David Smith.

00:00:05   Under the Radar is never longer than 30 minutes, so let's get started.

00:00:09   This week we wanted to talk about when you're forced to do something for outside reasons

00:00:16   or market demand or pragmatic reasons that maybe you wish you wouldn't have to do or

00:00:23   you think you shouldn't have to do, but you're forced to do it anyway.

00:00:28   And this was brought about, the reason I wanted to talk about this initially is that I had

00:00:33   a number of things happen recently with Overcast that were of this nature and where I had to

00:00:38   basically adopt to the way people actually are or the way things actually work rather

00:00:44   than hold onto my long-standing belief that it should be a different way or my wish that

00:00:51   it was a different way because it would save me some work, which is often the case.

00:00:58   The big one that kind of drove all this was Overcast recently had to implement feed redirect

00:01:05   support or permanent feed change support.

00:01:08   And I talked about this a little bit on ATP last week, so I'm not going to go too far

00:01:12   into it, but the basic idea here is that podcast feeds change over time.

00:01:21   Podcasters often will move the feed to a different service or to a different host or to a different

00:01:27   ad tracking platform or whatever else.

00:01:30   They just move the feed and they don't, if you move a website, you usually set up a redirect

00:01:35   from the old URLs to the new URLs and you usually leave that redirect there forever.

00:01:40   And even if you give it the permanent redirect code of 301 rather than the temporary one

00:01:45   of 302, you still leave up the redirect forever because you know that there's old links that

00:01:51   point to it, there's old search engine ranking that might point to it, and so you know that

00:01:56   it's a good practice on the web to basically make redirects work forever.

00:02:01   So if you're writing something that keeps URLs in a database and crawls them every so

00:02:06   often, on the web you can pretty much never process permanent redirect codes and be fine

00:02:14   because almost all the redirects will keep working indefinitely into the future.

00:02:20   And when I made Overcast, this is how I set up the database.

00:02:23   And this was, you know, I made the database in 2013.

00:02:26   That's when I made the schema, I started crawling and started building up my database of feeds

00:02:30   and the app launched about a year later.

00:02:33   And so for me, at the time, I made this decision of how to structure my feeds table and my

00:02:39   feed items table and how users interact with feeds and feed items.

00:02:45   I designed this whole schema back then and there was this one big flaw in the schema,

00:02:49   which was that feeds couldn't have their URLs changed.

00:02:54   If a feed was created with a new URL, it would be a whole new feed entry and then there was

00:03:00   no mechanism in place to do things like, you know, if a new URL is created and then one

00:03:06   of the old ones redirects to that one, can I merge those entries?

00:03:10   Do I copy the old one to the new one?

00:03:13   Does the old one have some kind of alias list that also points to the new one?

00:03:18   I had no setup like that in place.

00:03:20   The entire app and everything and the entire database were built up without that consideration

00:03:25   or without that possibility.

00:03:28   And this started to become a problem pretty soon after launch that a couple of web, a

00:03:33   couple of podcasts started switching their feed URLs and for a while I thought, well,

00:03:38   you know, people will just subscribe to the new one.

00:03:40   The old one will still work forever because it will redirect and it will be fine.

00:03:43   There were a number of problems with the setup in reality.

00:03:46   One of the big ones is that when I would do things like count subscribers to do things

00:03:51   like search ranking, the two different entries would count as independent feeds.

00:03:56   I would have to do search filtering, so things like, you know, I filter, I don't show search

00:04:00   results that don't have an iTunes ID associated.

00:04:03   So when the iTunes ID would move, it would basically lose the search ranking and the

00:04:08   entire history of the previous one, et cetera.

00:04:10   So there were all sorts of problems with that.

00:04:12   But it wasn't a big problem because not a lot of podcasts were moving their feeds in

00:04:17   2014.

00:04:19   And then over the following years, a lot of podcasts switched from HTTP to HTTPS and because

00:04:28   as a lot of websites did, as we all kind of learned that that would, we probably should

00:04:31   be doing that.

00:04:33   That created a whole lot of duplicate entries.

00:04:37   Only during that time, certain podcast networks shut down or moved or changed their names,

00:04:43   changed their domains, and that caused even more extra or moved entries.

00:04:49   And then the big one was over the last year or two, a lot of major podcast publishers

00:04:56   have switched to dynamic ad insertion platforms, which host the feed at their own URL for who

00:05:01   knows what reason.

00:05:03   And so tons of major podcasts are now switching their feeds.

00:05:08   If they didn't already do it with HTTPS like two years ago, in the last year, they're now

00:05:12   doing it with ad tracking platforms.

00:05:15   So I've had a, it basically went from a small problem in year one to a medium-sized problem

00:05:22   in year two and a really big problem in year three.

00:05:25   And so I finally had to implement feed redirects.

00:05:29   And I had, you know, all this time I've been thinking like, I can just follow redirects.

00:05:33   I can, like, they'll be fine.

00:05:34   They'll be, they'll be good forever.

00:05:37   And the reality was just different.

00:05:38   The reality was podcasters don't do things the way web web developers do things.

00:05:42   They break redirects.

00:05:44   Like they consider iTunes the authoritative source.

00:05:47   And if iTunes moves the feed for them, which they can do through multiple different means,

00:05:52   either a 301 or putting a special XML tag in the feed that was returned with a 200 response

00:05:58   or going through like an iTunes customer support representative running the podcast directory

00:06:03   and like just like emailing somebody and saying, sorry, we messed up our feed.

00:06:06   Can you fix it?

00:06:08   And none of those things would carry over to Overcast.

00:06:11   So the reality of the market was very, very different from how I wanted it to be.

00:06:14   I really badly wanted it to work in this one way that websites always were, which is like,

00:06:19   you know, set up a redirect and keep it up forever.

00:06:22   But the reality was very different.

00:06:23   The reality was podcasters would move their feeds frequently.

00:06:26   They would break the old URLs and redirects most of the time.

00:06:30   And even if they didn't, I still had this problem of like split feeds in the database.

00:06:35   So that's no good.

00:06:37   So finally over the last couple of weeks, I finally wrote redirect support.

00:06:40   And because of my database schema being so unfriendly to it, it's a huge hack and it's

00:06:46   not perfect.

00:06:47   The way I do it is I basically like copy your old entries onto the new feed and delete your

00:06:53   old ones.

00:06:54   And that's not great.

00:06:55   Like there's a whole lot of downsides to that.

00:06:57   One of the big ones is that the app seeing a new feed item ID for all these episodes,

00:07:03   the app redownloads them all.

00:07:05   That's no good either.

00:07:07   So like it maintains your history, but you have to download, like all of a sudden your

00:07:11   phone is downloading like all your saved episodes of a certain show that you hadn't listened

00:07:14   to yet.

00:07:15   And that's kind of weird to happen unexpectedly.

00:07:18   So that's not a great solution.

00:07:21   But I had to be pragmatic.

00:07:23   I had to address this problem somehow because the market was very different than what I

00:07:30   wanted it to be.

00:07:32   Yeah.

00:07:33   And I think I also hear in what you're saying, it's this thing that is so happens to me.

00:07:38   This happened to me so many times is it's as much as we would like.

00:07:42   I think we would like to be able to predict the future.

00:07:46   And when we're initially building our applications and we're designing them and we're structuring

00:07:50   them, we would make choices that our future selves will thank us for.

00:07:58   And conceptually, that sounds great.

00:08:00   That, okay, yeah, of course, I should structure these things with maximum flexibility and

00:08:07   I should be able to, ahead of time, I should consider all these possibilities.

00:08:12   The reality is I've done this enough to know that you can't.

00:08:17   That is impossible.

00:08:20   Things will change and break and become problems in ways that you could never have predicted.

00:08:26   Maybe you get slightly better at this.

00:08:28   Maybe you don't make the same mistake twice.

00:08:31   Those types of things may be improving.

00:08:33   But I feel like the situation you find yourself in is something that is universal to app development

00:08:38   or just software engineering in general.

00:08:41   That you will at some point be forced to either, you either have to make an assumption or you

00:08:47   have to explode complexity.

00:08:52   One of those two things you have to do at hundreds of points in your application as

00:08:57   you're developing it.

00:08:59   You can make an assumption.

00:09:00   You can say, in this case, you're going to assume that mapping a feed URL to the feed

00:09:06   ID directly will be a stable link.

00:09:10   Or you have to explode complexity.

00:09:12   And you have to do some very complicated things to make that work flexibly.

00:09:19   And in this case, you chose to start with the assumption.

00:09:23   But that's the same thing that we do all the time.

00:09:26   So much of software development is about getting good at making that decision between making

00:09:33   an assumption and expanding complexity.

00:09:35   And you can apply this to so many things.

00:09:37   How complicated is your class hierarchy?

00:09:40   What kind of choices do you do for data persistence?

00:09:45   So many different things come down to this very simple situation.

00:09:48   Are you going to make an assumption or are you going to explode complexity?

00:09:54   And that means that you're stuck.

00:09:56   Now you have to go and be forced to do this thing because your assumption proved to be

00:10:02   wrong or proved to be wrong enough that continuing with that assumption becomes problematic.

00:10:10   And it's so much more complex too because like as I've mentioned this on Twitter here

00:10:13   and there the last couple of weeks, I've gotten a number of people who are suggesting like

00:10:17   oh why didn't you just do this with your database or why didn't you just make this migration

00:10:20   happen?

00:10:21   And a lot of them, a lot of the ideas or suggestions don't really work or aren't as simple as they

00:10:26   sound because Overcast is not just a local app with a local database.

00:10:31   It is a service with a server backend that syncs between multiple copies of your app on

00:10:35   multiple devices.

00:10:36   So any kind of like one way migration either can't happen or is way more complex than you

00:10:42   would think.

00:10:43   And also because it is a server backed app, the complexity of the schema has a direct

00:10:49   relationship to how scalable it is and how efficient it is and how expensive it is to

00:10:55   host.

00:10:57   If every look up to the feeds table has to then make another look up to some kind of

00:11:02   like URL aliases table or something like that to map the URL to the feeds I'm fetching,

00:11:08   that's a pretty significant hit in performance.

00:11:11   That's much more complexity and that could be way more load on the database servers and

00:11:15   that could really make scaling more difficult.

00:11:17   And if you have many of those things in your schema for flexibility and everything, then

00:11:24   you start running into pretty big scaling problems and big challenges that either just

00:11:28   make your life hard or make things cost way more to host and that could break your business

00:11:32   model.

00:11:33   So there's all sorts of other considerations here that make things like this sound like

00:11:38   a hard problem.

00:11:39   Anyway, so that was one problem of many that I've had and I kind of wanted to talk for

00:11:46   the rest of the show about a few more of those.

00:11:48   But before we do, speaking of scaling your web servers, we respond to this week by Linode.

00:11:55   Linode has fast, powerful web hosting options that you can get set up in just seconds.

00:11:59   It's very, very easy to understand all their tools and they can let you choose the resources

00:12:03   and the next distribution that you want, giving you the power and flexibility that you need.

00:12:06   All this starts at just $5 a month.

00:12:08   That gets you a server with one gig of RAM in the Linode cloud.

00:12:12   Linode servers are very, very fast.

00:12:14   They have industry leading performance.

00:12:16   They have native SSD storage on all their servers.

00:12:19   They have a lot of

00:12:21   network

00:12:32   friendly

00:12:56   web hosting options.

00:13:09   So

00:13:33   another

00:13:53   one

00:14:12   I

00:14:34   have

00:14:55   more

00:15:19   and

00:15:37   more

00:15:56   of

00:16:13   a

00:16:32   break

00:16:50   just

00:17:09   to

00:17:23   take

00:17:48   a

00:18:07   break

00:18:33   at

00:18:58   the end

00:19:23   of

00:19:41   the

00:19:53   final

00:20:08   look

00:20:27   at

00:20:49   this

00:21:08   a

00:21:26   talk

00:21:44   about

00:22:02   just

00:22:20   before

00:22:40   color

00:22:58   settings

00:23:16   on

00:23:33   the

00:23:47   new

00:24:09   web

00:24:27   on

00:24:45   the

00:25:04   web

00:25:22   on

00:25:40   the web.

00:25:54   it

00:26:17   like

00:26:37   a

00:26:57   the

00:27:17   web

00:27:34   on

00:27:54   the web.

00:28:08   [ Silence ]