Under the Radar

Under the Radar 4: Unexpected Complexity

 

  welcome back to under the radar a show

  about independent iOS development I'm

  Marco Arment and I'm David Smith under

  the radar is never longer than 30

  minutes so let's get started

  so today I think we wanted to talk about

  probably one of the most common but yet

  challenging parts of being a developer

  at least in my experience and that deals

  with unexpected complexity when you

  start off working on something and you

  have a kind of an expectation of how

  complicated it's going to be to do what

  it's going to look like the effort

  involved the time involved and then you

  sit down you open up Xcode you start

  programming and it turns out that it's a

  complete nightmare and it's going to be

  way more effort and way more work and

  that experience happens so often that I

  think having a good like process and

  mindset about dealing with it is one of

  the very important things of being able

  to ship software in a reasonable time

  and reasonable quality and I think we

  both recently had an experience that

  kind of fell into that category and so

  it was kind of it's this episode I think

  will be more of a case study style in

  terms of talking through our experiences

  and so I believe that is the experience

  that you had when you were working on

  the streaming engine for overcast am i

  right oh yeah I mean streaming as a

  feature you know so overcast as a

  podcast player and the the built-in

  Apple API is for playing media using AV

  player

  most of that stuff supports streaming

  out of the box you don't really have to

  do much to enable it or I mean depending

  on how much you want your interface to

  behave you don't necessarily have to do

  anything so it's it's very it's

  relatively easy to support streaming

  media just as well as you do locally

  played media if you're using AV player

  but one of overcast biggest selling

  features and one of the things that

  makes it most compelling at both to me

  and to many of the users is my custom

  audio engine with a feature called smart

  speed where smart speed dynamically

  adjusts the playback and and the way it

  works is it there's a processing

  function that needs to read in the audio

  samples in the middle of the processing

  stream and then

  output possibly a different number of

  them if it's speeding up the sound so it

  might it might take in 500 samples and

  then the middle of it might be silence

  as it detects silence so it might

  compress that down to 300 samples so you

  need you need the ability to inject it's

  the pressing stage a function that can

  read faster than it can output and that

  is not possible with a B player there is

  no method there's no mechanism to do

  that that I have found at all I mean

  really if anyone knows I'm really

  curious but I have looked many times

  every time there's a new SDK I look I've

  asked Apple engineers at WB DC and in

  labs and via email and it really does

  seem like it was just not possible with

  AV player so the way I do this is I

  actually just wrote the I basically

  wrote my own AV player it's it's a not a

  small task and and it's I'm writing at

  the core audio and audio units levels

  and you know custom decoding of

  everything and when you go lower level

  to enable this they're the easiest way

  to do it is something called the ext G's

  ext audio file API and that is that only

  works for for locally stored files it

  does not work for streams there's a

  separate API the audio file stream API

  that is lower level than that and that

  you can you can supply stream data to

  you still have to manage the fetching of

  the stream yourself and a lot you have

  managed a lot yourself so the the job of

  streaming was to convert from that ext

  audio file API that only does local

  files to the lower-level

  audio file stream API and also then

  build out all the UI behind it build at

  the network fetcher that the actual

  streamer that can stream things and of

  course make it all performant and

  responsible of data usage and memory and

  responsible of the CPU and also handle a

  whole bunch of weird edge cases and I I

  knew it was gonna be a big job going in

  but I didn't quite realize how big how

  many of those edge cases there would be

  how tricky some of them would be to

  solve and the biggest thing like when

  we're talking about unexpected

  complexity the biggest thing for me is

  unexpected time that it takes

  you know this is why our business is

  historically so bad at estimating time

  and chip dates and when you're

  independent I don't think we're any

  better at it you know I think in many

  ways meant be worse

  how do you do you find that you are good

  at estimating time I think the honest

  answer is that I've stopped estimating

  time or even trying to like I have found

  that it doesn't actually help for like I

  have this vague sense of like I would

  like sometimes I have that the only

  deadline I usually have is a new devices

  being shipped or a new OS is being

  shipped like there's something specific

  that like if I want to be there on day

  one of the iOS 9 launch then I need to

  you know be in this habit my app ready

  by the beginning of June of September

  like other than that I try hard to not

  think too much about timelines because

  every time I've tried to do that it

  never works like that's semantics like

  the class is the most like universal

  experience of software engineering is

  you think it's going to take two weeks

  it ends up taking four weeks and that's

  just the reality of like you're all like

  as you I mean I think it sounds a little

  bit like you have the same experience

  like you start off thinking okay maybe I

  can do this with these high-level api's

  like you know these things apples you

  know has put together their tried and

  true whatever and you start using it and

  it's like okay it does most of what I

  need but it doesn't do this one critical

  thing and you start being like how

  critical is that well it's probably

  pretty critical and so you like you step

  you take your neck next step down and

  then you take your next step down and

  then before long you're just like

  reimplemented all kinds of crazy stuff

  and you have no way of knowing

  superficially what that's going to look

  how many love stacks deep how much

  rewriting and rebuilding of stuff you're

  gonna need to end up doing to get what

  you're going to do and so then I was

  just like I don't know I'll just sort of

  hope for the best

  and like other than for ios9 things you

  know an idea for those I just tend to

  say like how much like I'll just work

  until I can't I run out of time and just

  try and structure what I'm working on

  such that I can like you know not not be

  half-finished have like half the things

  fully finished rather than all the

  things half finished but yet estimating

  time is a complete nightmare I find yeah

  and and we're lucky that when we're

  working for ourselves on our own product

  for the most part our time estimates

  aren't that important because for the

  most part you know only the users are

  demanding things if anybody it's it's

  not like you know when you're when

  you're a consultant the pressure is much

  stronger on you to meet the clients

  deadlines or when you're working for a

  company you know when you're working

  directly on your employer's product it's

  you know you have similar pressures from

  that but when you're consulting you if

  you have a whole different aspect of it

  which is that you know the client is

  often directly paying you for the time

  that you are taking and that of course

  makes you even more complex we are both

  lucky that we're not currently in that

  kind of role but certainly I would I

  would say from from just anecdotal

  evidence it seems like the vast majority

  of iOS developers are involved in some

  kind of contract work that that seems to

  be the biggest chunk of the of the iOS

  workers out there and and time estimates

  are such a nightmare there I've just

  we're both lucky that we can't really

  talk much about it because I haven't

  done it in forever and even when I did

  it I didn't do very much of it and you

  it's been a while since you've been

  consulting as well right yeah I mean

  I've the last time I did regular

  consulting was probably four or five

  years ago something like that like I

  started out as a consultant that was

  what I was doing to initially but it's

  been a long time since that made up any

  kind of significant portion of my

  business and I Helen very quickly when I

  was doing consulting though to not

  commit to time estimates and as much as

  I could because you're always going to

  run into that there's always some kind

  of complexity that you're not expecting

  and if you don't plan for it if you

  don't look at a problem and say like

  whatever you it's like the old rule of

  like whatever you think it's going to

  take like multiply it by three and least

  you're probably you're probably well off

  but yeah I try as best like the first

  rule like but if you definitely have

  that experience when you start off

  consulting where you start to say like

  oh it's gonna take such you know that'll

  only be like a week's worth of work and

  you commit to the client to do that and

  then it never ends up being a week's

  worth of work it's just that's just the

  reality I mean even and I mean with with

  streaming I had so many problems with I

  would I started it I think three or four

  times and I would get down a certain

  path or try to try to approach it in a

  certain way and realize that it was just

  it was just never gonna work I was never

  gonna be able to complete it in the way

  I was doing it or it just was not

  working or I had made too many mistakes

  or I had made too many subtle bugs that

  I couldn't find at the very low levels

  and so I would just scrap it and start

  over again and when when you're at a big

  time crunch you don't usually have the

  opportunity to do that and that can be a

  blessing and a curse you know if you do

  that too much then you're wasting a

  whole bunch of time and like you know

  it's never going to be perfect it's

  never going to be exactly the perfect

  design exactly what you want and so if

  you if you have too many opportunities

  to cancel it and start over again you're

  just gonna kill tons of time but if you

  don't have any opportunities to do that

  which is often the case when you're

  under time pressure from a client say I

  feel like it's hard to get good quality

  work because you really do learn a lot

  if you try to do something didn't get a

  chance to throw it away and do it over

  again yeah I think one thing that I was

  my experience like whenever I've

  encountered these kind of problems like

  where I'm trying to deal with something

  that ends up being really much more

  complicated than I think like I I start

  off with a mental model for the problem

  that is straightforward for example the

  thing that I would most recently ran

  into this with is I did a data syncing

  and merging algorithm for pedometer plus

  plus so like it takes the steps from

  your watch and your phone and make gives

  you a unified view of them in semi

  real-time which is at first I was like

  oh the mental model for this is great

  there's health kids right I can just use

  health get and you can't because that's

  not how healthy it works because it's

  not available on the watch in the way

  that shows you phone data so I very

  quickly discovered that like my original

  mental model was completely wrong and so

  then like but I might process tends to

  be I have this really straightforward

  mental model it ends up not working and

  then I end up going down these crazy

  paths of like incredibly complicated

  solutions like all of these really like

  brittle and not really gonna work

  solutions like all kinds of crazy -

  algorithms and merging schemes and way

  of throwing data back and forth and you

  get to the end and the times I'm most

  satisfied with my work though or when I

  end up after all of that with a simple

  solution again that is informed by all

  of those crazy

  paths down there down the way like you

  can learn the lesson and be like okay

  actually I don't need to do all these

  crazy things now that I understand the

  problem fully from like top to bottom I

  can say okay I don't need to worry about

  the data at all these in all these crazy

  ways I can just somehow make all these

  sort of nice simplifying assumptions

  about how it is because I've tested and

  verified that that those are those work

  and then I can build like a solution

  that's almost as simple as the one I

  started off with wanting to build and

  then now that actually deals with all

  that complexity and like whenever I'm

  able to go through that process it's

  like it starts simple

  it gets horrific ly messy and then it

  gets simple again I know I've done doing

  it right

  Hyup if ever it is having to stop in

  that middle state where it's still

  really messy like I know it's gonna

  break I know it's not gonna work like it

  needs to be something that I could just

  describe to myself in just like a few a

  few sentences or something has probably

  gone wrong what kind of system did you

  end up with to address the the watch OS

  2 or watch kit to sync problem because

  you know as as you are very well aware

  and as some of our games might be you

  know with what watch kit 1 apps were

  actually fairly straightforward and

  simple to make because they couldn't do

  that much and because they were running

  on the phone in the app sandbox and the

  same sandbox is the iPhone app so that

  you could have a shared container and

  that and the iPhone app and the watch

  app could literally be reading and

  running from the same data and and

  that's how I structure the overcast

  watch app which is I just moved the the

  database file the SQLite file into the

  the app container and then the the

  communication between the watch app and

  the phone app is limited only to

  basically commands and status updates

  and all the data the watch app reads the

  data right off the database and you know

  the phone just tells the watch when it

  has changed and the watch reads it so

  it's it's actually you know very quite

  simple in that way because there is no

  sync issue if the watch can't reach the

  phone at the given moment you just can't

  use the overcast app because it's all it

  is a remote control for the phone so

  it's great with watch OS 2

  that that shared data container no

  longer quite exists the same way or at

  all really it doesn't exist at all yeah

  right so so with watch OS 2 it is like

  it you are running two separate apps and

  you need to figure out how they're going

  to communicate with each other and there

  you know there's method there's methods

  to do the basics but there is no like

  automatic magical syncing that happens

  so you have to deal with how these

  things synchronize their data yourself

  that's one of the reasons why I haven't

  made an overcast watch OS 2 app why I'm

  just key I just keep having the the OS

  one app first of all because it works

  well enough in a watch I was to improved

  watchkit reliability quite a bit and so

  it the app works well enough for the

  most part but also you know that it

  would be a major undertaking as you say

  as you discovered you know it's a major

  undertaking - all of a sudden tackle

  these two - all of a sudden take

  something that used to be a dumb remote

  and make it a full app with its own

  local storage and the problem of syncing

  both ways and so you know if you if you

  take the phone away and you have the

  watch off by itself somewhere playing

  podcast somehow which is own separate

  bag of hurt of like you know how capable

  that is right now and watch iOS 2.0 but

  if you somehow do that then you have to

  when upon reconnection to the phone you

  have to sync up that data again and

  resolve possible conflicts and it's it's

  a it's a very it's a much larger problem

  than it seems upon first glance because

  while the API is and everything are all

  very similar to what they were and watch

  oh and watch get one the the data model

  is totally different and that's where

  most of the complexity I think would lie

  and I haven't yet decided that's worth

  the investment apps for the Apple watch

  are a whole whole separate topic but I I

  have found generally that that I'm I'm

  very scared to tackle that this is one

  of the few times where I've spotted that

  that complexity ahead of time and and

  possibly wisely I don't know but

  possibly wisely I'm avoiding it yeah

  because I think what I've ended up

  having to do in watch OS 2 is

  essentially you have to build two

  completely independent can run on their

  own apps like the watch app has to be

  able to run entirely on its own without

  the

  able to like phone home to the to the

  phone the phone app obviously needs to

  be able to run without the watch which

  isn't that much of a stretch because

  that's what it does if you don't own a

  watch and then you have to do these

  funny things because the communication

  between the two is not reliable both in

  terms of sometimes the watch is just

  physically not close enough to the phone

  in order to communicate and also

  sometimes the just the api's and things

  like because I think it's because

  they're trying to do so much about power

  management and you know sometimes

  certain types of messages only go

  through if it's a good you know it's a

  good thing from a energy perspective to

  be able to do like the radios are all

  turned up and fired up so hey let's go

  ahead and send it right away if they're

  not maybe we'll wait and see if we are

  going to fire up the radio anyway for

  the next you know in the next second if

  not then maybe we'll do it and so you

  have to build the system that is very

  head makes no assumptions about being

  able to talk to the other part but still

  gives your user a consistent experience

  across both and that's where like from

  for this project that I was working on

  became like the really crazy thing

  because like a pedometer in some ways is

  very simple like both at both devices

  have a motion processor on it that tells

  you how many steps they took in a

  particular time period like that's all

  they do and that's most of what the app

  is doing and so I could run completely

  independently in terms of like if you

  just if I could just showed you the

  watch's steps on the watch app and it

  just showed you the phone steps on the

  phone app it'd be trivial but I want to

  be able to do is I've seen merge those

  together and say like which one is a

  better representation of what you're

  doing and that BK is where all the crazy

  complexity comes in because I had a

  couple of things that I was like oh so

  many of like conflict resolution

  strategies don't really work in this

  scenario because I can't just do simple

  things like who updated most recently or

  things because if I can ever for example

  make someone step count go down well I

  guess I could but conceptually I've

  always had that be like a rule for the

  app that if I ever even if it was like

  slightly an error if I ever showed you a

  step count like for example you hit your

  goal right like you got 10,000 steps

  confetti comes down from the screen

  hooray you did it I don't want you to

  then like sync up with your watch and

  say like

  actually now we think that you did 9000

  steps like that would be terrible

  right to withdraw somebody's confetti

  yeah like I'm sorry your confetti now

  goes back up to the top of the screen

  and animate it in Reverse

  it's like it I think that'd be awful

  right play like a sad trombone yeah and

  so you have to like a lot of my syncing

  stuff is about saying like what's the

  biggest number of steps that you've

  shown the user for the particular time

  period that we're looking at and syncing

  that up so that if I ever needed to show

  as sort of add extra steps I have I know

  what I need to add and also I need to do

  that but like it's not just like as

  simple as like who's shown more because

  the nature of these devices is like

  which one of them one of them is going

  to be showing you a better view of the

  user's activity probably at any one time

  like when I'm walking and pushing my

  daughter in a stroller my phone in my

  pocket is gonna be a better idea of how

  many steps I am because my wrist is

  actually not moving you know it's

  attached to a stroller pushing it along

  but when my phone is like in a backpack

  or in my or like my wife runs into this

  a lot like her phone will be in her

  purse but her watch is on her wrist so

  then her her watch is the better

  indicator of steps and so it's like

  having to work out a scheme that is in

  real-time dynamically working out which

  one based on the information we have

  which we may or may not have all the

  time looking at the two data streams and

  merging them together and like that is

  what you end up having having said like

  that's the the minimum threshold for

  having this thing work is you have to

  have built a system that can do all that

  like that was the complexity for me that

  just became like wow this is a whole

  bigger thing than I would ever imagined

  it would be yeah and this and that's

  like one of the one of the things that

  makes these unexpected complexities so

  so frustrating or so powerful is that

  again they're unexpected you have no

  idea when in a project you're going to

  hit something like this until you hit it

  and head so like you know it the more

  experience you get as a developer or

  just as a person who lives in life then

  the more experience you get it becomes a

  little bit more likely that you'll spot

  them ahead of time and and be able to

  possibly either avoid them or at least

  at least like budget for them but

  they're still

  always like stuff that you hit it's

  unexpected and it can totally throw a

  project off so like you know one thing

  that I do to manage this is if I if I

  can tell something is going to be a big

  hairy mess a lot of times I'll just cut

  the feature I'll just you know I'll just

  avoid it I'll say sorry that's that's

  not worth doing we know with overcast

  I knew streaming was going to be a big

  deal so for version 1.0 I just didn't

  include it i I couldn't ship it in time

  I thought it is tomato that it would be

  a better idea to ship the app without

  streaming than to wait another six

  months to a year to tube add streaming

  that turn out to be correct and in the

  case of my watch app now where I I have

  the watch OS one app the watch OS two

  app would be a ton of work and and would

  be different and might not be better you

  know my it might just be different I

  calculated so far that at the moment it

  isn't worth investing the time in that

  and it might never be I don't know but

  certainly right now it isn't worth

  investing with time in that when I could

  instead be doing something with the same

  amount of time I can instead be doing

  something like the Apple TV app or or a

  major new feature or a new e and even a

  brand new app if I wanted to like I you

  have to really decide what's worth

  syncing all this time into and sometimes

  it isn't the big hairy problem in front

  of you sometimes that problem is best

  off just staying unsolved that feature

  might be better off staying

  unimplemented and or that app that might

  be better off not being made and you

  might be better off spending that time

  in other ways because you can do a lot

  of low-hanging fruit in other parts of

  your app or as other apps in the same at

  a time that you could tackle one big

  hairy problem in one app and it might

  not be worth it our sponsor this week is

  need to go to need addition com need is

  a curated retailer and lifestyle

  publication for the modern gentlemen

  each week need launches new collections

  of exclusive clothing literature

  furniture and more now earlier this

  month coinciding with the company's

  second birthday need luncheon all new

  site expanding its availability into 43

  countries in Europe South America Asia

  and the Middle East

  shipping is flat rate for all

  international orders and free for all

  orders to the US Canada and

  Mexico and all returns are free no

  matter where you are all over the world

  this month Neda's launched three

  collections including their holiday gift

  guide featuring dozens of gift ideas for

  the holidays many of which are under $40

  neat is designed to be simple

  straightforward and uncomplicated

  they include prepaid return labels with

  all shipments there for 24/7 support and

  there aren't any subscriptions or

  stylists or other gimmicks to deal with

  it's just a straightforward retailer

  simply come along to neat addition com

  peruse the latest collections and shop

  or don't your choice for listeners of

  this show use coupon code read our for

  20% off that is coupon code radar at

  neat addition calm for 20% off anything

  thank you so much to our friends at need

  for sponsoring under the radar

  getting back to one thing that you just

  said that I think is probably a good

  place to do binding down this discussion

  is I think the thing that I've learned

  most like this is the like the

  battle-hardened wisdom of dealing with

  complexity is making sure that I'm

  solving the the problem for the right

  reason I think most of us who get into

  software engineering or whatever you

  want to call what we do we like solving

  problems and the problems that we like

  solving most are the most difficult

  interesting tricky problems like if that

  is I think I'm mu I'm in my element the

  most I'm the most engaged I'm the most

  excited about what I'm doing when I'm

  solving a really interesting problem

  like if I'm just writing boilerplate

  code that's not nearly as exciting and

  something I want to really like wrap my

  arms around and get you know sort of get

  up in the morning to dive into and so

  the thing that I though then always have

  to keep in the back of my mind when I

  find a problem like this when I'm

  working along on a task and I find that

  there's this big unexpected bump in

  complexity like you were just saying

  like sometimes you're just worth not

  doing it like I think I know I've had to

  teach myself some discipline with is

  that I see when I see a problem like

  that when I seize an opportunity to

  really like you'd go crazy spelunking

  like dive deep down into some crazy low

  api's and like build this crazy cool

  clever system make sure that I'm not

  just doing it like because it would be

  intellectually fun and interesting like

  sometimes there's a place for that but

  I've definitely caught myself sometimes

  just building things

  because I think it would be fun and it

  wouldn't actually make my customers

  happier it wouldn't actually make my

  products better it would just be fun for

  me to do and I end up going down you

  know these crazy rabbit holes and doing

  all these things and often the reality

  is it isn't actually as fun as I thought

  it was gonna be like it's fun for the

  first few out for the first few hours

  first few days and then you just like

  it's like you're in this point where

  like I will always want to avoid where

  it's like I started working on my car

  and I've taken it completely apart and

  like the NIF I've come to discover like

  the fun part was taking it apart the fun

  part is not putting it back together and

  now I don't have a car anymore

  like that's the part where I have to

  catch myself and be like am I really

  doing this for the right reason is this

  complexity worth the effort and having

  that kind of taking a step back from

  your work is a sort of a habit and a

  discipline that I found has been very

  helpful in me making sure that I'm

  staying on track and especially being

  like a one-man shop like there's no one

  I don't have a supervisor being like huh

  yeah yeah we're just gonna cut that

  feature that's that's too much it has to

  be something that I have the ability to

  not just totally suck down these rabbit

  holes and say like yes of course I

  needed to do this thing because it's

  really interesting and really hard like

  I need to be able to say like will this

  will look well my customers notice if I

  do this and if they won't like am I just

  doing this for myself and if I am like

  that's kind of sometimes fun but that's

  really not a way that you're gonna be

  able to reliably ship quality products

  you're just gonna end up with a lot of

  kind of like crazy half-built things

  that probably will end up making your

  app more complicated and harder to use

  for your customers in the end yeah

  that's a fantastic way to look at it and

  we all face that that dilemma because I

  mean look we've we're all geeks we all

  like interesting things we all like

  keeping our brains interested and try

  new things and approaching problems and

  new ways or doing things in better ways

  than what we did in the past

  it's not that you should never do that

  but that there's a balance to be struck

  and that that doing that has significant

  time costs that may not be worth it

  and and you know doing like a major

  rewrite of something that already works

  might also introduce brand-new bugs in

  fact it almost certainly would brand new

  bugs that you that you didn't have

  before they need to get to go facing

  and so it's just a balance you need to

  keep yourself happy

  and learning and intellectually

  satisfied but you also need to ship

  products and and it's important to find

  that balance and and that's something

  that I think mostly just comes with

  experience and and wisdom over time it's

  it's not that you can never have fun but

  that you should know when to have fun

  and when it and when it's no longer fun

  you need to get back to work yeah and I

  think ideally the best lesson I've had

  about that is it's trying to have

  customer focused development in that way

  it's like the time to do that the time

  to solve interesting problems is to find

  features or find apps or find ideas that

  solving a difficult problem will make a

  customer's life better and like the

  intersection of those two things like a

  really interesting technical problem and

  a real honest need or pain point in

  someone's life like that's where really

  interesting and fun work happens and if

  you can't sort of honestly say that it's

  both interesting to build and useful for

  a customer then you're probably not in a

  place that is actually worth building

  all right thanks life listening

  everybody please tell your friends about

  the show helps help us spread the word

  recommend us an overcast and we'll see

  you next week