Under the Radar

Under the Radar 58: Prerelease Testing


00:00:00   welcome to under the radar a show about

00:00:02   independent iOS app development I'm mark

00:00:04   Ormond and I'm David Smith clear the

00:00:06   radar is never longer than 30 minutes so

00:00:08   let's get started so today we wanted to

00:00:12   talk a little bit about testing about

00:00:15   that process we go through towards the

00:00:17   end of developing something where we try

00:00:19   and take it from mostly working to

00:00:22   actually working or fully working and

00:00:25   this is particularly relevant and

00:00:26   interesting for me at this point because

00:00:28   I am in that very final of phases for my

00:00:33   next step when we thought about whole

00:00:34   bit about last week that I'm in the

00:00:37   phase now where functionally and you

00:00:39   know from a feature perspective I've

00:00:40   kind of drawn the line and said this is

00:00:43   the app this is as far as I'm gonna go

00:00:44   with it any other features or any other

00:00:46   things that come up are gonna be in the

00:00:48   next version you know version 1.1 or

00:00:50   I'll push it down the road all um now

00:00:52   now going to be doing is fixing bugs and

00:00:55   testing and finding all the little

00:00:56   issues and problems and it seemed like

00:00:59   something that would be an interesting

00:01:00   topic to kind of walk through because

00:01:03   ultimately the goal is to make sure that

00:01:06   we ship things that are you know quality

00:01:09   are not you know riddled with bugs or

00:01:12   having all kinds of problems and it's a

00:01:16   tricky thing I think especially at from

00:01:18   an indie perspective and I think we're

00:01:19   definitely fair to say but I'm

00:01:20   definitely talking about this from the

00:01:22   the more of the indie perspective rather

00:01:24   than you know if you are in a large

00:01:27   corporate environment where your app is

00:01:29   being worked on by lots of different

00:01:31   people you may actually have an entire

00:01:33   like QA department whose job it is to

00:01:35   test your application like I used to

00:01:38   when a job I used to work at you know

00:01:40   they had there's an a room full of par

00:01:42   like 20 people who all day just sat

00:01:44   there using the product testing is

00:01:47   trying all the crazy head you know edges

00:01:48   and they had these big test plans that

00:01:50   they had to go through before we did a

00:01:51   build and like that is a very different

00:01:53   world then I think either of us work in

00:01:55   and so we're not talking about that kind

00:01:57   of testing like that is a a discipline

00:02:00   unto itself it is useful and important

00:02:02   but for an independent I think that's

00:02:04   just not the reality both from like a

00:02:07   resources perspective like I don't have

00:02:09   the time

00:02:11   - realistically do that you know to do

00:02:13   20 people times 40 hours or a week of

00:02:17   testing like that's just impossible no

00:02:20   so I'm not sure if it's necessary and so

00:02:23   this is coming into like a lot of

00:02:24   there's a certain category of bug that

00:02:27   I'm less worried about this is sort of

00:02:29   the in more like the integration bug

00:02:31   rather than the like functional bug

00:02:34   where sometimes you just have bugs that

00:02:36   come up because you had five people

00:02:37   working on a product and they're you

00:02:39   know one person changed something here

00:02:41   someone else changed it over there and

00:02:42   obviously myself in different time

00:02:44   domains is different like I can be you

00:02:47   know like two weeks ago me and my

00:02:48   current me can be conflicting but that

00:02:50   happens less often but either way like

00:02:54   being independent it's still something

00:02:55   that I have to do and it's a process

00:02:56   that I go through now and it seems like

00:02:59   kind of a good thing to walk through but

00:03:01   I don't do it in a formal way

00:03:03   marco do you do it any kind of formal

00:03:05   testing or processes that you have

00:03:07   around putting something out or do you

00:03:08   just kind of try it out a bunch and see

00:03:11   if it works I mostly just try out a

00:03:14   bunch I mean and so so we should clarify

00:03:16   also that test-driven development is not

00:03:19   what we're really talking about neither

00:03:21   of us really do that we've talked about

00:03:23   it briefly before and I don't think

00:03:25   either of us really has a strong opinion

00:03:27   on it except just to say that we don't

00:03:29   do it mostly as a choice because of our

00:03:32   resources and our style of coding

00:03:35   you know just being single person shops

00:03:37   it makes it harder to justify the the

00:03:40   additional amount of time it takes to

00:03:42   write all the tests and for test-driven

00:03:44   development so that's not what we're

00:03:46   talking about here today at least so

00:03:49   yeah so when it comes to whatever

00:03:52   testing people would call what we used

00:03:54   to call testing just like you or other

00:03:58   people like actually just using your app

00:04:01   and trying out all the different things

00:04:02   you know trying to turn you know testing

00:04:05   out edge cases you know the things that

00:04:07   a good tester a QA person would do that

00:04:09   is that's what we're talking about today

00:04:11   and so my answer to that is generally I

00:04:15   I mostly rely on my own usage and their

00:04:18   and reports from beta testers in order

00:04:21   to to do this and

00:04:23   this has pluses and minuses I mean

00:04:25   obviously it this is a less disciplined

00:04:28   approach than having like a formal unit

00:04:31   testing process a formal UI testing

00:04:33   process having dedicated testing people

00:04:37   or services I should look more into

00:04:41   those things I haven't to date and and

00:04:44   it mostly has not caused problems

00:04:46   there's and and part of this is because

00:04:48   of the way I developed the app in the

00:04:50   first place I mean step number one is I

00:04:52   only make apps that I use if you are

00:04:55   making an app that you don't use

00:04:57   yourself on a regular basis and there's

00:05:00   lots of reasons why you might want to do

00:05:01   this you know if you're a consultant

00:05:02   obviously that's a big one

00:05:04   but you know a lot of people have these

00:05:05   needs if that's the kind of need that

00:05:07   that you're working with or you don't

00:05:09   use the app I really don't know what to

00:05:10   tell you I think things like unit

00:05:12   testing make more sense there or having

00:05:14   having like dedicated QA people or

00:05:16   services that you can hire to do that

00:05:17   but because I use the app myself and and

00:05:22   I also tend to only implement features

00:05:25   that I will use and and this is not I I

00:05:29   don't do that 100% of the time like

00:05:30   there are certain features like in my

00:05:32   nitpicky details feature about the

00:05:34   showing the number of unplayed unplayed

00:05:36   episode as a red badge on the icon of

00:05:39   the app I think you can tell by the

00:05:41   description text in that interface that

00:05:43   I don't want to use this feature but

00:05:45   enough people requested it and it was a

00:05:47   small enough feature I mean that

00:05:49   features something like four lines of

00:05:50   code it's it's it's a very very very

00:05:52   tiny little feature and it satisfied

00:05:55   large user demand a lot of people

00:05:58   requested that feature always before I

00:05:59   added it and so it satisfies a large

00:06:02   user demand it was not that much work to

00:06:03   do and it's very easy for me to test

00:06:06   every so often even though I don't use

00:06:08   it myself so that's something I added

00:06:09   anyway despite not using it myself but

00:06:12   otherwise I try not to add features that

00:06:15   I won't use myself very often if at all

00:06:17   in part because I don't care and in part

00:06:21   because it makes it harder for me to

00:06:23   maintain the quality of the app because

00:06:24   I won't see problems in day-to-day

00:06:26   testing like one of the one of the

00:06:28   issues that I faced recently is I have a

00:06:30   pretty severe carplay bug in the most

00:06:32   recent build and I'm working on fixing

00:06:34   that now I should have it for the next

00:06:36   one but

00:06:37   I don't have a car play vehicle we don't

00:06:39   own a car play vehicle in our household

00:06:40   and we actually are about to get one in

00:06:42   about a month and that'll probably help

00:06:43   although it won't be my car which is

00:06:45   unfortunate but it's at least my wife

00:06:47   will have one so I will have hardware to

00:06:49   test on sometimes but like car plays

00:06:51   this whole area of the app that I never

00:06:53   see because I don't drive a car that's

00:06:56   car play enabled and so there are going

00:06:59   to be all these different areas where my

00:07:01   own day-to-day usage is not going to

00:07:03   cover it or is not going to reveal

00:07:06   subtle problems at least and so for

00:07:08   those kind of uses I really just rely on

00:07:10   beta testers and reports from users

00:07:13   hopefully you know hopefully before I

00:07:15   release these versions that have these

00:07:16   that have problems but you know

00:07:18   sometimes sometimes they get out there

00:07:20   and so in order to in order to minimize

00:07:22   the damage I do try to at least like at

00:07:26   least cover the the biggest most common

00:07:29   uses of the app in my own formal or

00:07:32   informal testing so for example what I'm

00:07:34   releasing a big version I will almost

00:07:36   always if I remember to which isn't

00:07:39   always the case which is a problem but

00:07:41   actually probably this is again this is

00:07:42   like one of the reasons why it might be

00:07:43   useful to have like a checklist or

00:07:45   something but or you know a more formal

00:07:47   structure of doing these things or an

00:07:48   automated way of doing these things but

00:07:50   one of the things that I do in my app is

00:07:53   to to basically start fresh start from a

00:07:56   brand new user every time I do a release

00:07:59   so like take a device that doesn't have

00:08:01   the app on an install a fresh version of

00:08:03   the app and then try creating a new

00:08:05   account and starting all over again and

00:08:07   this helps for a number of reasons

00:08:09   obviously that's a very important thing

00:08:12   for your app to make sure it works at

00:08:14   all and then secondly it shows me the

00:08:17   onboarding experience it shows me the

00:08:18   first run experience a lot like I see

00:08:21   that whenever I whenever I do we release

00:08:23   I I will see the first run experience

00:08:24   and at least once during that process so

00:08:27   that kind of helps remind me of any

00:08:30   shortcomings in it or it helps you know

00:08:32   helps show me if anything's broken about

00:08:33   it or if if things aren't quite right

00:08:35   you know it's so that's nice you know in

00:08:37   that way as well but this is all a very

00:08:40   long way of saying basically the way I

00:08:42   do testing is I just use the app a lot

00:08:45   myself before I let anybody else use it

00:08:46   and then when it gets close to release I

00:08:49   do beta tests

00:08:51   and I can talk a little bit more about

00:08:53   that maybe a little bit later but the

00:08:55   beta test I use tend to find almost any

00:08:58   other reigning problems yeah I think

00:09:00   it's to is it's important to say like I

00:09:02   mean I think my testing strategy is very

00:09:04   similar to yours it's the I use the app

00:09:06   a lot and I try and sort of use it like

00:09:13   as I get closer I tend to kind of use it

00:09:14   slightly and anger like I use it in a

00:09:16   way that it's kind of trying to break it

00:09:18   like I'm bashing on all the buttons and

00:09:20   like what if I change this setting

00:09:21   really quickly back and forth back and

00:09:23   forth like is anything weird gonna

00:09:24   happen like you can use it in that way

00:09:26   and then yeah like you said it's that

00:09:28   kind of testing where and the honestly

00:09:30   what I usually do is for the like trying

00:09:32   out a new user testing it's the you know

00:09:35   I - I currently use an iPhone 7 that's

00:09:37   the phone that I use like during

00:09:40   development like 98% of my development

00:09:42   happens on that phone and so before I

00:09:44   launch I need to you know very least run

00:09:46   it unlike a 5s and on a 7 plus I'm just

00:09:51   in terms of to make sure that there's

00:09:52   not any weird UI bugs or issues or

00:09:54   sometimes even just weird there's

00:09:57   performance things obviously going if

00:09:58   you go back to like a 5 or a 5s they're

00:10:01   much slower and so that's a I find a

00:10:02   great place to do the kind of on board a

00:10:04   2 on board testing because you know I'm

00:10:08   starting fresh on those devices half the

00:10:09   time anyway that will also say one thing

00:10:12   that has bitten me many many times in

00:10:14   the past and so now I always try and

00:10:16   make sure I do too is I do the fresh

00:10:20   user onboarding experience process but I

00:10:23   do that running the old version of the

00:10:26   app like the existing I could go to the

00:10:28   App Store and download whatever this is

00:10:30   whatever the current version of the app

00:10:32   is from the App Store on board a fresh

00:10:34   user and then install the version that

00:10:36   I'm trying to verify and test with nice

00:10:40   because often what I found is there's

00:10:41   issues that you can run into that like

00:10:43   the you know so usually it's a data

00:10:45   issue or a settings issue or something

00:10:47   that isn't being translated over

00:10:49   correctly and obviously the majority of

00:10:51   your users are going to be are like are

00:10:54   coming from that previous version and

00:10:56   then running your new version and so if

00:10:58   there's any issues there you want to

00:10:59   catch them but that kind of a this kind

00:11:01   of approach of just taking a lot of

00:11:03   devices like on my desk

00:11:04   I have a bunch of different you know

00:11:06   there's all the different iPhones like I

00:11:08   never sell an iPhone whenever I get a

00:11:10   new one and so I have all that all the

00:11:12   different testing devices available to

00:11:14   me that way

00:11:15   you know similarly like right now I'm

00:11:17   wearing two Apple watches I have a 38

00:11:20   millimeter Apple watch on my right hand

00:11:21   and a 42 millimeter repla watch on my

00:11:23   left hand and they're paired two

00:11:25   separate phones many seven sizes and I'm

00:11:27   just running the app that I'm testing

00:11:29   constantly now like I just he keep this

00:11:33   process of kind of banging up against it

00:11:34   and if in general I think that it's it's

00:11:37   you'll it's amazing how good much how

00:11:40   good the coverage you can get from it

00:11:42   that kind of a informal but like

00:11:46   constant high-volume testing like maybe

00:11:48   it's not as rigorous and so it loses

00:11:51   something there but it definitely

00:11:52   benefits from just doing it all the time

00:11:54   and if you build apps that you use you

00:11:57   certainly have that benefit at the very

00:11:59   least of exposure to it yeah I mean one

00:12:02   other note on the devices before I

00:12:04   forget

00:12:04   app review does not seem to test much if

00:12:08   at all on a variety of devices as far as

00:12:11   I can tell they test on one device only

00:12:13   and usually it's a recent one so as far

00:12:16   as I can tell so like a lot of times

00:12:18   people people will have issues where

00:12:19   like they'll get an update into the

00:12:21   store approved that like crashes on

00:12:24   launched on the iPhone 5s or something

00:12:26   like that right because it's like an

00:12:27   older device that is still supported and

00:12:29   it still runs on and that your customers

00:12:30   might still use but it doesn't work for

00:12:32   you know it and I purview didn't catch

00:12:35   it you know and so there's you can't

00:12:37   depend on app review to catch like the

00:12:40   big obvious Crashers because if it

00:12:42   happens on anything any conditions that

00:12:44   they're not trying including any other

00:12:46   device besides whatever one they happen

00:12:47   to be using you're not going to have a

00:12:49   good time so that's that's a very very

00:12:51   important to note and it's also probably

00:12:53   worth saying that app review is not QA

00:12:55   like app review is this very like course

00:13:00   sieve that every now and then will catch

00:13:03   problems and I love it when they do this

00:13:04   but like that is not at all what they're

00:13:07   doing they're not really looking to test

00:13:09   your app to find problems they're too

00:13:11   kind of running it through at a very

00:13:13   different level and so I would not as

00:13:16   certainly isn't one of those like well

00:13:17   I'll just

00:13:18   if a perv you can find any bugs and if

00:13:19   they don't I'm sure it's fine like that

00:13:21   is a tear there'll be a terrible

00:13:22   approach to take to you with your app

00:13:25   yeah it's kind it's like the seat belt

00:13:26   on a plane it's like that's that is not

00:13:29   the safety mechanism you should be

00:13:30   relying on this you should definitely

00:13:32   like it basically if a preview rejects

00:13:35   your app for a crash you have a serious

00:13:38   problem in your QA process it's like

00:13:39   whatever you submit to ever view should

00:13:41   not be in that kind of state where it

00:13:42   would crash in like the first four

00:13:44   minutes of somebody using it once you

00:13:46   know but you know overall like and I

00:13:49   should also you know suggest to that if

00:13:51   you are going to rely so heavily as we

00:13:54   do on self testing basically and again I

00:13:57   I do I apologize greatly to people who

00:14:01   enjoy formal testing and automated

00:14:03   testing and unit testing and then 100%

00:14:05   test coverage I apologize because this

00:14:07   this entire discussion must be driving

00:14:10   you totally crazy hi Casey

00:14:11   but ultimately it's hard for single

00:14:17   person developers to really justify the

00:14:20   additional time and/or expense that more

00:14:24   rigorous testing methods bring on for

00:14:27   things like this so now that's a lot of

00:14:29   qualifiers in that sentence obviously if

00:14:32   you have a larger staff if you have a

00:14:34   lot of money if your app is more

00:14:37   important and then like you know then

00:14:40   then you know like a sleep tracker for

00:14:42   your wrist or for the podcast player

00:14:45   like if you're really bad things happen

00:14:48   if you like lose people's important data

00:14:50   or you know if somebody will be

00:14:52   seriously put out or hurt if your app

00:14:55   breaks obviously that's a place where

00:14:57   you need a more formal testing procedure

00:14:59   and and framework in place but most

00:15:02   people who make iOS apps aren't working

00:15:04   in that kind of capacity and and not

00:15:07   only don't have those resources but it

00:15:09   would actually be be kind of a bad way

00:15:11   to spend resources basically like

00:15:13   tripling all of your work and this is a

00:15:15   huge debate in programmer circles of

00:15:17   whether test-driven development

00:15:18   is really more work or whether it ends

00:15:19   up saving you more work in the long run

00:15:21   I honestly don't care which one of those

00:15:24   it is I think it's probably unknowable

00:15:26   because it's probably like the answer

00:15:28   like everything else is probably it

00:15:29   depends but but ultimately

00:15:32   this isn't the way we do things and and

00:15:34   I think there's a lot of developers out

00:15:35   there who who do things more like what

00:15:37   we do anyway that being said what more

00:15:39   thing before I go into the sponsor for

00:15:41   the week and that is if you're relying

00:15:42   so much on self testing as we do if your

00:15:45   app has like you know different modes or

00:15:47   settings or ways you can do things

00:15:50   change it up like in your in your day to

00:15:53   day usage change it up every so often

00:15:55   and spend a lot of time in each kind of

00:15:57   condition so for instance overcast has

00:15:59   dark mode or light mode and during

00:16:01   development I usually switch about once

00:16:03   a week between which one of those I'm

00:16:05   using because I want to kind of get used

00:16:07   to it and live in it and make sure that

00:16:09   I'm that I'm designing and making the

00:16:11   best app I can for both of those things

00:16:12   same thing with streaming versus

00:16:14   downloading like you know I wrote this

00:16:15   whole streaming engine I hardly ever use

00:16:17   it and that's kind of a problem so

00:16:19   basically I force myself to change my

00:16:23   own preference and and spend a good

00:16:26   amount of time like a week or a month in

00:16:28   streaming mode in the middle of

00:16:30   development cycles where I would

00:16:31   otherwise be in downloading mode you

00:16:32   know just just to make sure I'm getting

00:16:34   a really good idea of like these vastly

00:16:38   different modes that my app can be in so

00:16:40   that my own testing is more effective

00:16:42   anyway we were sponsored this week by

00:16:45   Pingdom Pingdom can help remove the need

00:16:48   for you to have somebody who

00:16:49   automatically logs in your website every

00:16:51   minute and refreshes it to make sure

00:16:52   it's up because Pingdom does that

00:16:54   automatically with their testing

00:16:55   framework see for yourself at Pingdom

00:16:58   comm slash radar Pingdom is an awesome

00:17:01   monitoring service for websites and web

00:17:03   services and when you use offer code

00:17:05   radar at checkout you get a holiday

00:17:08   special of 50% off your first invoice

00:17:11   this is an awesome deal usually 20% now

00:17:14   it's 50 percent for the holidays so

00:17:16   check it out at Pingdom comm slash radar

00:17:18   Pingdom offers powerful easy to use

00:17:21   monitoring tools and services for your

00:17:24   website and web servers web sites are

00:17:26   very sophisticated these days and you

00:17:28   need you know there's all sorts of

00:17:30   things that can go wrong and they know

00:17:31   this they see it because they monitor it

00:17:33   for lots of people

00:17:34   you need Pingdom to be monitoring your

00:17:36   site they can monitor as often as every

00:17:38   minute by requesting pages by running

00:17:40   through checkouts by logging in by doing

00:17:42   you know transactions that they can do

00:17:44   all sorts of things all

00:17:46   simulated from there more than 70 global

00:17:48   test servers and they can do it as often

00:17:49   as once a minute and if anything goes

00:17:51   down they can alert you via so many

00:17:53   different means whatever you want you

00:17:55   can have email text push notification or

00:17:58   all an income any combination of these

00:17:59   things and you can set the granularity

00:18:01   it's incredibly customizable I've been

00:18:03   using Pingdom now for oh geez I think

00:18:06   eight years nine years something crazy

00:18:08   it's a very long time because I've used

00:18:10   it way before they were sponsor way

00:18:11   before I was even a podcaster I was

00:18:14   using Pingdom to monitor tumbler back in

00:18:15   the day and and I just love them I used

00:18:18   them constantly highly recommended that

00:18:20   you use Pingdom for your all your

00:18:21   monitoring needs it is really the best

00:18:24   and if you run any kind of website or

00:18:26   web service the last thing you want is

00:18:28   for strangers on Twitter to be telling

00:18:31   you when your site is down when you

00:18:32   didn't already know that like you do not

00:18:34   want to be learning that from Twitter

00:18:35   you want to know when your site is down

00:18:37   you want to be the first to know and

00:18:39   then you want to be able to go in and

00:18:41   fix it before all the people on Twitter

00:18:43   see it and start bugging you about it

00:18:44   and with Pingdom you can because Pingdom

00:18:47   will alert you the moment things go down

00:18:49   with that up to one minute granularity

00:18:51   that's really incredible and they are so

00:18:52   fast and so reliable

00:18:54   I love Pingdom check them out yourself

00:18:56   go to Pingdom comm slash of radar you

00:18:59   will get a 14-day free trial and when

00:19:01   you enter offer code radar at checkout

00:19:03   you will get a holiday special of 50%

00:19:05   off your first invoice check it out

00:19:08   today no be the first to know when your

00:19:09   site is down thank you very much to

00:19:10   Pingdom for sponsoring our show so I

00:19:13   think for the last part of the show it

00:19:15   seemed like some that may actually make

00:19:16   sense for us to focusing on is a bit of

00:19:19   the actual like the details of what this

00:19:22   kind of user testing approach that we've

00:19:25   just been talking about it looks like

00:19:27   and for me what that usually means is so

00:19:30   I get into this phase now where I'm

00:19:32   doing this final level of testing and

00:19:33   I'm just constantly trying to think of

00:19:35   things to try I'm trying them out I'm

00:19:38   using the app in different circumstances

00:19:39   in different places like I'll be using

00:19:41   it on different networks like hey I'm

00:19:42   out on a cell network I'm at home now

00:19:45   and so whenever I find an issue you're

00:19:46   inevitably you'll find a bug you'll see

00:19:48   something that you're like huh that's a

00:19:49   little interesting what I found is a

00:19:52   very important thing is to have some

00:19:53   kind of reliable way of collecting

00:19:57   all these little issues and bugs and

00:19:59   problems and for me usually that is I

00:20:01   have I'll have a list in OmniFocus that

00:20:05   I am just adding items to and anytime I

00:20:09   see something usually I'll grab a

00:20:11   screenshot of it and if it's obvious

00:20:14   I'll just attach the screenshot to the

00:20:16   task in OmniFocus or if not I'll take it

00:20:19   and run it through what used to be an

00:20:22   app that you made Marco which used to be

00:20:23   called buckshot what I think is now

00:20:24   called pin point mmm which is a great

00:20:26   little tool we have a link in the show

00:20:27   notes to it but it's just a little thing

00:20:29   for just drawing arrows basically or

00:20:30   circling things um it's great for screen

00:20:33   shot sort of when you're like huh

00:20:34   you know there's this weird graphics

00:20:36   glitch or something so you can mark it

00:20:38   up add it to it to get and then you know

00:20:41   I go through all of like and when I'm in

00:20:44   the face now like this morning I sat

00:20:45   down I had the head a list about five or

00:20:47   six little bugs I've noticed yesterday I

00:20:50   went through and I'm just knocking them

00:20:52   out and it's kind of like this I work

00:20:55   all the way down till I have no bugs and

00:20:57   then I make a build and then I go

00:20:58   through the process again and you kind

00:21:00   of just keep cycling through collecting

00:21:02   as you go and for me that kind of just

00:21:05   that kind of process of just keeping

00:21:08   this running log tea seems to work well

00:21:10   it I like doing it that way rather than

00:21:13   kind of you know having like sitting

00:21:17   down and just testing for hours never

00:21:19   really works for me I find that I get

00:21:20   kind of bored or I start doing the same

00:21:23   patterns over and over again and so it's

00:21:25   better I think for me I'd try and do it

00:21:26   spread it out throughout the day and

00:21:28   obviously if it's an application that

00:21:29   you would use normally and naturally

00:21:32   throughout the day then this makes it

00:21:33   easier but otherwise you just kind of

00:21:36   have to sort of remind yourself to do it

00:21:39   and this is also a little bit tricky if

00:21:41   you have an app that requires a certain

00:21:44   amount of time for testing so like for

00:21:47   example I have a sleep tracker like in

00:21:48   order for it to really like like I

00:21:51   cannot compressed beyond just running it

00:21:53   on multiple devices the testing of that

00:21:56   because it takes about eight hours to

00:21:58   have eight hours of data collection and

00:22:00   like I have simulated stuff in the app

00:22:03   for you know when I'm doing like debug

00:22:05   testing that I can kind of like create a

00:22:06   night's data and run my algorithms on it

00:22:09   but for actual testing there's nothing

00:22:11   you can really do other than wearing

00:22:13   local watches which is what I do like

00:22:15   when I'm doing extensive sleep you know

00:22:17   testing like I'll have all my watches on

00:22:19   my wrist during the night and I'm

00:22:21   running it you know three or four times

00:22:22   but you know sort of in the same way I

00:22:24   imagine for you with overcast like

00:22:26   there's a certain amount if you just

00:22:26   have to play a lots of audio because if

00:22:29   you don't you're never going to find all

00:22:31   the weird audio issues or glitches or

00:22:33   problems there's nothing that you can

00:22:35   really do to speed that up or compress

00:22:37   that testing yeah basically I mean I

00:22:40   like I just find ways to shove podcasts

00:22:44   into more of my life so so like you know

00:22:47   like like ivories and and also to test

00:22:48   out things like syncing you know like

00:22:50   this my syncing engine took forever to

00:22:52   really figure it mean took me almost a

00:22:53   year to really nail syncing and one of

00:22:57   the ways I started finally getting

00:22:58   better at that was I started needing it

00:23:00   more because I started using an iPad to

00:23:02   play podcast in the kitchen and I used

00:23:04   my iphone to play them everywhere else

00:23:06   so I will I was very frequently

00:23:08   switching between multiple devices that

00:23:10   would use the sink back-end and just by

00:23:12   doing that myself but by needing that

00:23:14   myself for a few months I really fixed a

00:23:17   ton of sink problems and got it to a

00:23:18   pretty good place now and and so yeah I

00:23:21   mean it really is just a matter of just

00:23:23   you know again it's like if you only

00:23:25   make apps that you will use a lot this

00:23:27   becomes a lot easier if if you don't

00:23:30   make apps that use a lot this you know

00:23:32   you're gonna have to find other ways to

00:23:33   do this effectively but it really works

00:23:36   that great for me and and I I will again

00:23:38   once again go in briefly about beta

00:23:40   testing also and and we pretty sure we

00:23:43   did a whole episode on it if not we will

00:23:45   and we should but I think we already did

00:23:46   we have so many episodes now I've kind

00:23:48   of lost track

00:23:49   but I do rely a lot on beta testers to

00:23:53   to really do like a final a final check

00:23:56   on the app they aren't my number right

00:23:58   there are my first testers I am my first

00:24:00   tester and I I will usually use changes

00:24:03   to the app for weeks or even months

00:24:05   before I give it to anybody else to test

00:24:07   just because I really want to make sure

00:24:09   that I know I have feel for it make sure

00:24:10   it's good make sure it's the right

00:24:12   choice and then I will finally share

00:24:13   with testers like right before the

00:24:14   public gets it but basically beta

00:24:17   testers can can help a little bit not a

00:24:19   ton but but a little bit I will also add

00:24:23   that the what you mentioned earlier

00:24:26   about your workflow about taking taking

00:24:28   screenshots and then adding arrows and

00:24:29   then adding them to OmniFocus and things

00:24:31   like that there if you want to go off

00:24:33   the deep end there are lots of crazy

00:24:35   tools and frameworks that automate that

00:24:37   process to varying degrees and have all

00:24:39   sorts of cool integrations so there are

00:24:40   some like you know like I made bug shot

00:24:42   kit which is basically shoved bug shot

00:24:45   into an app as you could do things like

00:24:47   it will detect screenshots being taken

00:24:50   and then automatically bring up a modal

00:24:53   controller with the screenshot setting

00:24:54   here mark this up and send it in to

00:24:56   report a bug you know and you can you

00:24:57   can give that to all your testers and

00:24:59   you can use that yourself during your

00:25:00   test process and so you can basically

00:25:01   have like easy or automated bug

00:25:03   reporting right right embedded into your

00:25:05   app and there's again there's lots of

00:25:08   ways to do this I'm not gonna recommend

00:25:09   any particular one because I haven't

00:25:10   looked but there are many diseases exist

00:25:12   now and you can you can really go off

00:25:14   the deep end with like the levels of

00:25:16   integrations you can have it like post

00:25:18   to slack and then it make an entry in

00:25:20   your bug tracker like all sorts of crazy

00:25:22   things you can do now if you really want

00:25:23   to get into all that stuff so the you

00:25:25   know the options are pretty good for now

00:25:27   for like tools and frameworks and things

00:25:29   like that to automate like the the

00:25:31   tedious stuff but for the for the real

00:25:34   just using the app really just again

00:25:37   just using it a lot yourself is a very

00:25:40   very effectively to do it obviously as I

00:25:42   know automated things exist sorry Casey

00:25:43   again but just using your app yourself

00:25:45   will show you so much and will tell you

00:25:47   so much and will tell you will show you

00:25:48   things like you know this this workflow

00:25:50   is kind of clunky or this animation

00:25:53   doesn't really look right or this I I

00:25:55   really need this text to be lighter when

00:25:57   it's really sunny outside or something

00:25:58   like that like there's a large category

00:26:01   of either problems or or good and bad

00:26:04   design decisions for which just using

00:26:07   yourself is really the only way to know

00:26:08   and the really the only way to test and

00:26:10   that isn't true for everything obviously

00:26:12   sorry again Casey but that's true for

00:26:14   quite a lot that we do in our apps and I

00:26:16   think with other thing I would say with

00:26:18   beta testing that's an important thing

00:26:19   too it's another use for beta testing is

00:26:23   the importance of not overly preparing

00:26:26   your beta testers for what you know for

00:26:30   the app so in terms of trying to give

00:26:32   him giving them an experience that is

00:26:33   similar to the experience that your app

00:26:36   store users are going to have

00:26:37   and so when they download the app and

00:26:40   they're like what do I do or this is

00:26:42   really confusing like some of the best

00:26:44   feedback I get from beta testers is sort

00:26:46   of accidental where they're not actually

00:26:49   they don't know it but they're telling

00:26:51   me something is fundamentally flawed or

00:26:53   is really confusing where they're asking

00:26:55   a question that like it implies missing

00:26:58   a fundamental misunderstanding about

00:27:00   something and that kind of feedback is

00:27:02   always really good and so I try whenever

00:27:03   I'm beta testing to just almost like

00:27:06   blindly send out the betas to people

00:27:08   where I'm not giving them too much

00:27:09   preparation or tell are like giving them

00:27:11   step-by-step walkthroughs or details

00:27:13   it's very much like as though you just

00:27:16   downloaded this from the App Store you

00:27:17   know maybe you've seen a screenshot or

00:27:18   two maybe you've had a high-level

00:27:20   discussion but you're working it out

00:27:22   just as you would as a regular user and

00:27:24   that's a great test because have you as

00:27:28   the developer of the app of ously I know

00:27:30   how everything works everything is

00:27:32   intuitive to me because that's the way I

00:27:34   made it but it is often not that not the

00:27:37   case for your users and so one of the

00:27:39   best uses for beta testing isn't it's

00:27:41   great when they find you know actual

00:27:43   bugs and issues and problems but it's

00:27:45   also it's really useful for capturing

00:27:47   those kind of like it's like if you're

00:27:49   asking this question you clearly have

00:27:51   not communicated the purpose of this

00:27:53   feature correctly or I've hidden it in a

00:27:55   weird way or you think it should do

00:27:57   something and it does the opposite and

00:27:58   those are the only thing that is almost

00:28:01   exclusively possible to find in beta

00:28:05   testing because no matter how hard you

00:28:07   try to kind of like reset your brain to

00:28:10   what if I was a fresh user opening the

00:28:11   application it's kind of impossible

00:28:13   where you know I the current app I'm

00:28:15   working on I have been working on it

00:28:17   essentially every day for months at this

00:28:20   point so I'm so in it that I just don't

00:28:24   see it's like I'm so far into the woods

00:28:26   that I have no idea that there is a

00:28:27   forest like all I have is trees like it

00:28:29   is deep down into it I mean and maybe

00:28:33   when I come back to it and a couple of

00:28:35   you know like I'll put the app to the

00:28:37   side for a little bit and I'll come back

00:28:39   to it and maybe a month maybe I'll I

00:28:40   will have that kind of fresh experience

00:28:42   but until that time the only way I can

00:28:45   get that is with beta testing and so it

00:28:46   is definitely a valuable and important

00:28:48   aspect of that

00:28:49   all right and I guess we're gonna see

00:28:52   your new app pretty soon right yeah it

00:28:54   should be hopefully launching next week

00:28:57   so should be kind of exciting and I

00:29:00   expect to go home sure we'll talk about

00:29:01   on the show how the lunch went and kind

00:29:03   of some of the details of it as the app

00:29:05   goes but it is very exciting to get to

00:29:07   this point and I will say as you know

00:29:09   it's like stalking all the other

00:29:10   independent developers out there it's

00:29:12   like like I just put in the work because

00:29:15   there's definitely been times in this

00:29:16   project where I was like oh gosh what

00:29:18   have I gotten myself into maybe I should

00:29:20   just like can this project move on but

00:29:22   there is hope eventually we'll get there

00:29:23   eventually we'll ship and it is

00:29:26   delightful

00:29:26   when you get there can't tell you a

00:29:28   secret to the audience you can I know

00:29:30   what this is and I've used it and it's

00:29:31   really good so look forward to it anyway

00:29:34   thanks a lot for listening everybody

00:29:35   we're out of time this week and we'll

00:29:37   talk to you next week bye