Under the Radar

Under the Radar 112: Ideal vs. Pragmatic


  welcome to under the radar a show about

  independent iOS app development I'm mark

  Worman and I'm David Smith

  under-the-radar is never longer than 30

  minutes so let's get started this week

  we wanted to talk about when you're

  forced to do something for outside

  reasons or market demand or pragmatic

  reasons that may be that like maybe you

  don't you wish you wouldn't have to do

  or you think you shouldn't have to do

  but you're forced to do it anyway and

  this was brought about the reason I was

  talked about this initially is that I I

  had a number of things happen recently

  with overcast that were of this nature

  and where I had to basically adopt to

  the way people actually are or the way

  things actually work rather than hold on

  to my you know long-standing you know

  belief that it should be a different way

  or my wish that it was a different way

  because it would save me some work she's

  often up in the case yeah the first the

  the big one that kind of drove all this

  was overcast recently had to implement

  feed redirect support and/or permanent

  feed change support and I talked about

  this a little bit on ATP last week so

  I'm not gonna go too far into it but the

  basic idea here is that podcast feeds

  change over time and podcasters when

  they the podcast is often will move the

  feed to a different service or to a

  different host or tweeted for an ad

  tracking platform or whatever else they

  just move the feet and and they don't

  like if you move a website you usually

  set up a redirect from the old URL to

  the new URLs and you usually leave that

  redirect there forever and even if you

  give it the the permanent redirect code

  of 301 rather than the temporary one of

  302 you still leave up the redirect

  forever because you know that like

  there's old links the point to it

  there's old search engine ranking that

  might point to it and so you you know

  that like it's a good practice on the

  web to to basically make redirects work

  forever so if you're writing something

  that that keeps URLs in a database and

  and crawls them every so often on the

  web you can pretty much never process

  permanent redirect codes and be fine

  because almost all the redirects will

  keep working indefinitely into the


  and when I made overcast this is how I

  set up the database and and this was you

  know I mean I made the database in 2013

  that's when I made the schema I started

  crawling and and started building up my

  database of feeds and and the app

  launched about a year later and so for

  me at the time I made this decision of

  how to structure my feeds table and my

  feed items table and and how users

  interact with feeds and feed items I

  designed this whole schema back then and

  there was one big flaw in the schema

  which was that feeds couldn't have their

  URLs changed if a if a feed was created

  with a new URL it would be a whole new

  feed entry and then there was no

  mechanism in place to do things like you

  know if if a new URL is created and in

  the and then one of the old ones

  redirects to that one how do I can I

  merge those entries do I copy the old

  one to the new one do what does the newt

  does the old one have some kind of an

  alias list that also points to the new

  one I had no setup like that in place

  and so the entire app and everything and

  the entire database were built up

  without that consideration or without

  that possibility and this started to

  become a problem pretty soon after

  launch that a couple of webs a couple of

  podcaster at switching their feed URLs

  and for awhile I thought well you know

  people just subscribe to the new one the

  old one will still work forever because

  it'll redirect and it'll be fine there

  were a number of problems with the set

  up in reality one of the big ones is

  that when I would do things like count

  subscribers to do things like search

  ranking the two different entries would

  count as independent feeds I would have

  to do search filtering so things like

  you know I filter I don't show search

  results that don't have an iTunes ID

  associated so when the iTunes ID would

  move it would basically lose the search

  ranking and the entire history of the

  previous one etc so there were all sorts

  of problems of that but it wasn't a big

  problem because not a lot of podcasts

  were moving their feeds in 2014 and then

  over the following years a lot of

  podcasts switched from HTTP to HTTPS

  and because a lot of websites did as we

  all kind of learn with that would we

  probably should be doing that that

  created a whole lot of duplicate entries

  additionally during that time a certain

  podcast networks shut down or moved or

  changed their names change their domains

  and that caused even more extra or

  removed entries and then the big one was

  over the last year or two a lot of major

  podcast publishers have switched to

  dynamic ad insertion platforms which

  hosts the feet of their own our URL for

  who knows what reason and so tons of

  major podcasts are now switching their

  feeds if they didn't already do it with

  HTTPS like two years ago in the last

  year they're now doing it with ad

  tracking platforms so I've had a like it

  basically went from a small problem in

  year one to a medium-size problem in

  year two and a really big problem in

  year three and so I finally had to

  implement feed redirects and I had you

  know all this time I've been thinking

  like I can just follow redirects I can

  feel like they'll be fine they'll be

  they'll be good forever and the reality

  was just different the reality was

  podcasters don't do things the way web

  web developers do things they break

  redirects like they consider iTunes the

  authoritative source and if iTunes moves

  the feed for them which they can do

  through multiple different means either

  a 301 or putting a special XML tag in

  the feed that was returned with a 200

  response or going through like an iTunes

  customer support representative learning

  the podcast directory and like just like

  emailing somebody and saying sorry we

  messed up our feet can you fix it and

  none of those things would carry over to

  overcast so the reality of the market

  was very very different from how I

  wanted it to be I really badly wanted it

  to work in this one way that websites

  always work which is like you know set

  up a redirect and keep it up forever but

  the reality was very different the

  reality was podcasters had to move their

  feeds frequently they would break the

  old URLs and redirects most of the time

  and even if they didn't I still had this

  problem of like split feed in the

  database so that's no good so finally

  over the last couple of weeks I finally

  wrote redirect support and be

  cause of my database schema being so

  unfriendly to it it's a huge hack and

  it's not perfect the way I do it is that

  basically like copy your old entries

  onto the new feed and delete your old

  ones and that's not great like there's a

  whole lot of downsides to that one of

  the big ones is that the app seeing a

  new feed item ID for all these episodes

  the APRI downloads them all that's no

  good either so you like you it maintains

  your history but you have to download

  like all of a sudden your phone is

  downloading like all your saved episodes

  of a certain show that you hadn't

  listened to yet and that's kind of weird

  to cut to happen unexpectedly so that's

  not a great solution but I had to be

  pragmatic I had to address this problem

  somehow because the market was very

  different than what I wanted it to be

  yeah and I I think I also hear in what

  you're saying it's this thing that is so

  happens to me this happened to me so

  many times is it's as much as we would

  like I think we would like to be able to

  predict the future and when we're

  initially building our applications and

  we're designing them and we're

  structuring them we would make choices

  that you know our future selves will

  thank us for right and conceptually that

  sounds great that okay yeah of course

  like I should engine you I should struct

  structure these things you know with

  maximum flexibility and I should be able

  to I should a head of time I should

  consider all these possibilities the

  reality is I've done this enough to know

  that you can't like that is impossible

  and things will change and break and

  become problems in ways that you could

  never have predicted maybe you get

  slightly better at this you know maybe

  you don't make the same mistake twice

  like those types of things may be

  improving but I feel like the situation

  you find yourself in is something that

  is universal to app development or just

  you know software engineering in general

  that you will at some point make be

  forced to either you know you either

  have to make an assumption or you have

  to explode complexity like those are the

  two choice one of those two things you

  have to do at

  hundreds of points in your application

  as you're developing it you either have

  so you can make an assumption you can

  say in this case you're going to assume

  you know that mapping a feed URL to the

  fee a ID directly will be a stable link

  or you have to explode complexity and

  like you had build you'd have to do some

  very complicated things to make that

  work you know flexibly and in this case

  you know you chose to make the to start

  with the assumption but like that's the

  same thing that we do all the time

  there's so much so much of you know soft

  development is about getting good at

  making that decision between making an

  assumption and expanding complexity and

  you could apply this stuff so many thing

  it's like how complicated is your class

  hierarchy you know what do kind of

  choices do you do for data no data

  persistence or or so many different

  things come down to this base this very

  simple situation or do you are you going

  to make an assumption or are you going

  to explode complexity and like and that

  means that you're stuck like now you

  have to go and do be forced to do this

  thing because your assumption proved to

  be wrong or proved to be in wrong enough

  that continuing with that assumption

  becomes problematic and it's so much

  more complex to is like like as I've

  mentioned this on Twitter here and there

  the last couple of weeks I've gotten a

  number of people who are suggesting like

  oh why don't you why didn't you just do

  this with your database or why didn't

  you just make this migration happen and

  a lot of them a lot of the the ideas or

  suggestions don't really work or aren't

  as simple as they sound

  because overcast is not just a local app

  with a local database it is a service

  with a server back-end that syncs

  between multiple copies of your app on

  multiple devices so any kind of like

  one-way migration either can't happen or

  is way more complex than you would think

  and also because it is a server back tap

  the the complexity of the schema has a

  direct relationship to how scalable it

  is and how efficient it is and how

  expensive it is to host you know if if

  every look up to the feeds table has to

  then make another look up to some kind

  of like URL aliases table or something

  like that like to to map the URL to the

  feeds arm fetching

  that's a pretty significant hit in

  performance that's that's much more

  complexity and that could be way more

  load on the database servers and that

  could really make scaling more difficult

  and if you have multi if you have many

  of those things in your schema for

  flexibility and everything then you

  start running into pretty big scaling

  problems and big challenges that either

  just make your life hard or make things

  cost way more to hosts and that could

  break your business model so like

  there's there's all sorts of other

  considerations here that make things

  that this sound like a hard problem

  anyway so that was you know one problem

  of many that I've had and I kind of

  wanted to talk for the show about a few

  more of those but before we do speaking

  of scaling your web servers we were a

  sponsor this week by Linode Linode has

  fast powerful web hosting options that

  you can get set up in just seconds it's

  very very easy to understand other tools

  and they could let you choose the

  resources and the next distribution that

  you want giving you the power and

  flexibility that you need all this

  starts at just five dollars a month that

  gets you a server with one gigabyte or

  very very fast if industry-leading

  performance they have native SSD storage

  and all their servers access to a 200

  gigabit network eat intel xeon e5

  processors my favorite line of

  processors these are the fastest

  processors in the cloud market leader

  has over 400,000 customers who are all

  serviced by the friendly 24/7 support

  team you can email them if you need any

  help you can chat over IRC if you need

  any help and they have all sorts of

  wonderful documentation that's all

  public you can see it right now you and

  in fact a lot of times if you search for

  Linux server administration topics a lot

  of times the search results you get from

  Google are just Linode support articles

  because they are so easy to use to

  understand and so helpful so litterin

  has fantastic pricing options available

  again you can get a server with one

  gigabyte us five bucks a month you can

  get a server with 16 gigs of ram for

  just 60 bucks a month and a whole lot of

  levels above and below it in between

  across the board this is twice the

  amount of RAM that you'll get at most

  other places as a listener of the show

  if you sign up at lynda.com a slash of

  radar you'll be supporting us and you

  look at $20 towards any Linode plan and

  with a 7 day money-back guarantee

  there's nothing to lose so go to

  lynda.com slash


  are to learn more sign up and take

  advantage about $20 credit or use promo

  code radar 2017 at checkout thank you so

  much to Linode for running all of my

  servers and supporting the show so

  another one I had with overcast a small

  one before I go to another big one is

  password protected feeds this is one

  that you know podcasts sometimes we'll

  have paid memberships or other kind of

  like you know meant private feeds that

  they protect with some kind of password

  authentication scheme usually you know

  HTTP basic auth it's compatible with

  with a lot of apps and this becomes a

  problem when you have a server-side

  crawling like overcast does and in a

  couple of a couple of competitors do

  because it the question becomes like

  where do you do you retain the password

  and the entry on the server side and

  then do you crawl with someone's

  password do you how do you protect that

  from being used by multiple people and

  then if you have a whole lot of people

  who all have separate usernames and

  passwords they're calling the feed with

  do you have to then crawl the feed from

  like you know a hundred thousand

  different users from like hundred

  thousand different copies of the feed

  like you have to keep all those in your

  database and have those clogging up your

  crawler so it's kind of a big problem so

  when I launched overcast I made a

  decision I'm just not gonna support

  those and I this was kind of like a

  political decision like I don't like

  that I have to support these so I'm just

  gonna see if I can not support them and

  and just hope that nobody uses password

  feeds anymore after a while and that

  turned out not to be the case at all it

  turned out that password feeds are

  getting more and more popular as more

  and more podcasts exist and many of them

  choose to have paid tiers or private

  feeds or something like that and what a

  lot of people figured out also is that

  when I switch my crawlers to use go when

  I switch to it to a go offered crawler

  goes URL library if you supply it a URL

  that has the username and password

  prefixed on it the way you could do like

  username colon password at hostname it

  that will just work correctly in the go

  crawling library and people figured this

  out about overcast a couple years ago

  and so people have been adding user name

  and password feeds to overcast

  anyway and just doing it you know just

  like the nerds who figure that out no to

  just add that manually as a URL and so

  I'm already now having my database

  beefed look with lots of these copies of

  these things and and lots of different

  feeds and and the good thing is that

  like you know concerns like whether it's

  you know kept private or not that just

  kind of works with overcast because I

  filter by weather feeds have iTunes IDs

  and if they don't have iTunes IDs they

  don't show up in search or in many other

  places and so these copies of these

  private fees that have these using the

  passwords prefixed on them don't have

  iTunes entries so they don't show up and

  search so that that part ends up working

  but I am seeing my database slowly

  filled with these anyway and now there's

  so many of them that if I would like

  close that hole and make those not work

  anymore that would be a pretty big

  problem for me I have a lot of very

  angry people and that would probably be

  like you know a pretty nasty move so I'm

  not gonna break these now instead of

  decided you know what actually the right

  move here is to just make an interface

  like on the add URL screen

  I should just detect whether it's given

  me a 401 response and prompt people to

  enter a username and password and just

  store that for them in the URL for the

  feed because again this is like

  pragmatic versus ideal like in the ideal

  world I shouldn't have to deal with

  these in the real world turns out I do

  and so I might as well make it pleasant

  for people who use them and I might as

  well and and you know the reality is I

  just have to design my crawlers and

  scale my crawlers in such a way that

  having a whole bunch of these feeds in

  the database is not going to be a

  problem and that's just the reality

  because the reality is you have to do

  like it's really hard to impose your

  will on your users in that way like if

  people want to you want to do this thing

  and they want to use your app to do it

  they're going to do it mm-hmm like it

  would doesn't really get you very far if

  like when someone put like the

  alternative I suppose is like if someone

  went to the ad URL screen and you

  detected they there was a you know a

  username and password prefix to the URL

  you know you could throw up a big nasty


  a chance a no no no this is not what we

  do and it was with a wagging finger bad

  URL that's really that otherwise this is

  going to happen like people are going to

  realize it is happening yeah and if

  anything you could you view it it's like

  okay well you were able to without

  without really doing any extra work you

  were able to see how big of a market and

  need and interest there is in this so at

  least it's a slightly positive in that

  sense like rather than building this

  feature ahead of time you're able to

  build it now knowing that it will be

  used that you know roughly how many

  people are going to use it and then you

  can you know make changes to the way you

  do crawling or the way that you manage

  these things accordingly to take

  advantage of it and so like it's a good

  thing in that sense but yeah like you


  I mean you this is this is a kind of

  thing that just sort of always happens I

  find that you know you expect people to

  be using your app in one way and so

  often it is the way that you want to use

  it or the way that you you know you have

  it in your mind and then you get a

  support request you get an email from

  somebody and they show you something

  about how they were using your app that

  can kind of sum it's like it's like

  you're doing what exactly what woke okay

  like and sometimes I see those things

  and it's like that's genius like that's

  amazing I need to you know like

  percolate this up into this actual

  proper feature and sometimes you're just

  like um I'm amazed that works it's not

  intended to work I don't know if it will

  work in the future

  good luck but like it is just one of

  those things that you know users are

  shockingly creative about with the ways

  that they'll make something work

  especially I mean inviting thing it's a

  compliment to Overcash that people even

  though this isn't a feature of the app

  they took the effort and did the work to

  make it you know to somehow make it work

  because they want to use the app anyway

  that they don't want to switch to

  another app that mase you know support

  this at a at a official level because

  that you know they like the app and they

  want to stay with it anyway and my final

  example here that I wanted to talk about

  today was when the iPhone 10 came out it

  you know it has this new OLED screen and

  it turns out that completely black

  looks really good on an OLED screen the

  problem is that completely black doesn't

  look so good if you have like a totally

  black background and you know other

  elements that are like you know lighter

  and go totally black does not look as

  good on LCD screens the screens that

  we've had an iPhone since the first

  iPhone before this but true black looks

  really good on OLED and and all those

  like dark Gray's that we've been using

  on LCDs because that looks better there

  don't look good on the iPhone 10 so what

  as soon as Efrain 10 came out there was

  immediate demand from users who want all

  their apps to have true black themes

  available designers haven't been

  accustomed to LCD screens for the last

  forever designers have all have all

  basically said like you know looked it

  doesn't look as good it's hard and it is

  harder in many ways like there's certain

  certain design themes and standards that

  just either don't work or have to be

  severely modified to work with a true

  black background and design ocean argue

  like maybe they don't think it looks as

  good but the reality is a lot of people

  think it looks good and a lot of people

  want it and one of the things they

  started doing when the iPhone 10 shipped

  I got a massive spike in requests for

  supporting the accessibility smart

  invert colors feature because what had

  happened was people seeking more dark

  themes in their apps with iOS apps that

  so often are like bright white they

  there have been numerous articles posted

  online and telling people hey neat trick

  you can turn on this smarter invert

  colors feature in accessibility settings

  and all your apps turn black and so they

  were doing that and overcast didn't

  support it correctly with like it like

  it was overcast was not excluding the

  images from being inverted so all the

  artwork would look weird so I got this

  huge spike in requests when the iPhone

  10 came out to support smart invert

  colors the reality was these were not

  people who needed this for accessibility

  reasons for the most part they were

  people who just wanted a black theme and

  because it looks so it looks so good on

  their on their iPhone 10 and it's better

  on the battery and stuff so even though

  I thought you know I should probably

  like you know true black I thought

  looked kind of you know it's it's it's a

  pretty bold look when you first see it

  I now think it looks good but it took me

  a few hours before I thought it looked

  good after I started using it but the

  reality was the market was making their

  own black-themed even if I wasn't like

  they would just turn on my white theme

  and turn on smart invert colors and

  accessibility and they were making a

  black theme so the correct thing for me

  to do which I did in the next update was

  fix my spar time for colour support to

  make the images look correct and then

  also add a black theme because like

  that's the reality no matter what you

  think of black themes everyone now wants

  them and if you don't give them one they

  will work around that to get themselves

  one with accessibility settings and that

  makes a lot of the system look really

  weird and it's over like if you don't

  need invert colors as an accessibility

  option you shouldn't be using it there's

  lots of weird side effects to it but

  people wanted the black themes so badly

  that they did that so again it's one of

  those situations where like designers

  thought you shouldn't do pure black but

  the market says differently and you just

  have to adopt at that point yeah but it

  is always tricky and I think that it

  comes to mind is I've made the mistake

  in the past of going too far down this

  road of adding things like themes and

  like these kind of more super

  superficial options to an app and then I

  end up regretting it in this way in the

  sense that you now have three themes you

  need to support in your app and you know

  when you're anytime you add a view

  anytime you make a change you're gonna

  have to make sure that it works well now

  in all three of those themes it reminds

  me in a weird way of localization where

  I like and in my early days with

  localization I would get somebody who

  would say hey you know I I would like to

  localize your app into you know whatever

  language they spoke natively and I use

  it all the time I think it'd be great

  and I'll you know be happy to do the

  translations for free just send me the

  strings file and it's like initially I

  used to think that's a great idea like

  sure let me let me know the more the

  better and then you realize now I need

  to support this going forward and every

  time you add a string to your


  you have to have it translated or now

  you when you do screen shots you need a

  different thing to manage there and so

  while it is sometimes it's good to you

  know it's good to be adaptive and good

  to be you know responsive to users that

  it is in the back of my mind I've been

  burned enough by it now that I'm like do

  I really want this theme do I really

  want to have to you know that your your

  great trick where when you're checking

  you know dark mode switches have it

  switched between the themes you know

  every every set for a couple seconds so

  that you can make sure that you want

  your you wanna looks good so I really

  want that to cycle through three well or

  now maybe if someone suggests four or

  five like you it gets to a point where

  you can end up being too responsive to

  your users and so I guess there's a

  certain element where you have to draw

  lines and you have to say like or at

  least to think about do what how willing

  am i to support this you know how

  endured injustice create confusion to

  people now when they look in your

  Settings app and look at the themes area

  and they're like do I want dark or do I

  want black it's kind of like when you

  know Apple is selling the matte black

  and the jet black iPhones interesting to

  be like it now I have this choice I

  don't know like how do I know which

  one's better and you know they're having

  a bit of opinion there may actually be

  helpful but in this case it's like

  you're you're at a certain point you're

  being forced to do it just by the volume

  of requests but it's always yeah it

  makes it makes my developer since this

  tingle a little bit and be like do I

  really want to do this do I understand

  what I'm taking on by adding this it

  seems like just a little option you know

  it's just oh it's just black but you

  know you this is just the way things go

  I suppose I mean honestly I did it on a

  plane flight like that's how fast it

  wants to do because it's almost worse

  yeah it's so easy it's like oh let me

  just do this one thing and then it turns

  out you have to support this one thing

  forever well to be fair like you know

  it's similar to localization going from

  one to two themes is a lot more work

  than going from two to three and you

  know like going from two to three like

  if you if you did you're one to two

  migration like if you added theming to

  your app in a reasonable way adding

  additional themes is actually pretty


  and you are right that that you know

  there is certainly a testing and

  maintenance burden and maybe a screen

  shot bird and if you if you go that way

  for certain things but you know it

  really is not as big of a deal but again

  the reality is if the market is like if

  people are willingly making their

  experience with your app worse and

  possibly their experience with their

  entire phone worse in order to achieve a

  thing that you don't think you have to


  the reality is you probably have to do

  it sure yeah I mean ultimately it's like

  there's always a tension there but you

  have to a certain point you just have to

  relent and it's I suppose it's more just

  a question of making sure we're being

  thoughtful when we were lent that we

  understand what we're taking on that

  these seemingly small choices that we

  you know can make we have to live with

  for a long time and I guess that in some

  ways that's the theme of all three of

  these kind of challenges you've had to

  navigate is that whenever we are making

  choices as we're developing we have to

  understand that you know we're going to

  live with all of these choices going

  forward and so at least trying to be

  thoughtful about them or thinking or

  take a moment to be like do I really

  want to do this or how can this come

  back to bite me in the future you know

  how am I going to hate myself in the

  future because of this choice that I'm

  making and having a little bit of

  thoughtfulness is probably wise just to

  avoid you know getting into situations

  that we may regret later thanks we'll

  see everybody and we'll talk to you next

  week bye