Under the Radar

00:00:00 ◼   ► welcome to under the radar a show about

00:00:02 ◼   ► independent iOS app development I'm mark

00:00:04 ◼   ► Ormond and I'm David Smith clear the

00:00:06 ◼   ► radar is never longer than 30 minutes so

00:00:08 ◼   ► let's get started so today we wanted to

00:00:12 ◼   ► talk a little bit about testing about

00:00:15 ◼   ► that process we go through towards the

00:00:17 ◼   ► end of developing something where we try

00:00:19 ◼   ► and take it from mostly working to

00:00:22 ◼   ► actually working or fully working and

00:00:25 ◼   ► this is particularly relevant and

00:00:26 ◼   ► interesting for me at this point because

00:00:28 ◼   ► I am in that very final of phases for my

00:00:33 ◼   ► next step when we thought about whole

00:00:34 ◼   ► bit about last week that I'm in the

00:00:37 ◼   ► phase now where functionally and you

00:00:39 ◼   ► know from a feature perspective I've

00:00:40 ◼   ► kind of drawn the line and said this is

00:00:43 ◼   ► the app this is as far as I'm gonna go

00:00:44 ◼   ► with it any other features or any other

00:00:46 ◼   ► things that come up are gonna be in the

00:00:48 ◼   ► next version you know version 1.1 or

00:00:50 ◼   ► I'll push it down the road all um now

00:00:52 ◼   ► now going to be doing is fixing bugs and

00:00:55 ◼   ► testing and finding all the little

00:00:56 ◼   ► issues and problems and it seemed like

00:00:59 ◼   ► something that would be an interesting

00:01:00 ◼   ► topic to kind of walk through because

00:01:03 ◼   ► ultimately the goal is to make sure that

00:01:06 ◼   ► we ship things that are you know quality

00:01:09 ◼   ► are not you know riddled with bugs or

00:01:12 ◼   ► having all kinds of problems and it's a

00:01:16 ◼   ► tricky thing I think especially at from

00:01:18 ◼   ► an indie perspective and I think we're

00:01:19 ◼   ► definitely fair to say but I'm

00:01:20 ◼   ► definitely talking about this from the

00:01:22 ◼   ► the more of the indie perspective rather

00:01:24 ◼   ► than you know if you are in a large

00:01:27 ◼   ► corporate environment where your app is

00:01:29 ◼   ► being worked on by lots of different

00:01:31 ◼   ► people you may actually have an entire

00:01:33 ◼   ► like QA department whose job it is to

00:01:35 ◼   ► test your application like I used to

00:01:38 ◼   ► when a job I used to work at you know

00:01:40 ◼   ► they had there's an a room full of par

00:01:42 ◼   ► like 20 people who all day just sat

00:01:44 ◼   ► there using the product testing is

00:01:47 ◼   ► trying all the crazy head you know edges

00:01:48 ◼   ► and they had these big test plans that

00:01:50 ◼   ► they had to go through before we did a

00:01:51 ◼   ► build and like that is a very different

00:01:53 ◼   ► world then I think either of us work in

00:01:55 ◼   ► and so we're not talking about that kind

00:01:57 ◼   ► of testing like that is a a discipline

00:02:00 ◼   ► unto itself it is useful and important

00:02:02 ◼   ► but for an independent I think that's

00:02:04 ◼   ► just not the reality both from like a

00:02:07 ◼   ► resources perspective like I don't have

00:02:09 ◼   ► the time

00:02:11 ◼   ► - realistically do that you know to do

00:02:13 ◼   ► 20 people times 40 hours or a week of

00:02:17 ◼   ► testing like that's just impossible no

00:02:20 ◼   ► so I'm not sure if it's necessary and so

00:02:23 ◼   ► this is coming into like a lot of

00:02:24 ◼   ► there's a certain category of bug that

00:02:27 ◼   ► I'm less worried about this is sort of

00:02:29 ◼   ► the in more like the integration bug

00:02:31 ◼   ► rather than the like functional bug

00:02:34 ◼   ► where sometimes you just have bugs that

00:02:36 ◼   ► come up because you had five people

00:02:37 ◼   ► working on a product and they're you

00:02:39 ◼   ► know one person changed something here

00:02:41 ◼   ► someone else changed it over there and

00:02:42 ◼   ► obviously myself in different time

00:02:44 ◼   ► domains is different like I can be you

00:02:47 ◼   ► know like two weeks ago me and my

00:02:48 ◼   ► current me can be conflicting but that

00:02:50 ◼   ► happens less often but either way like

00:02:54 ◼   ► being independent it's still something

00:02:55 ◼   ► that I have to do and it's a process

00:02:56 ◼   ► that I go through now and it seems like

00:02:59 ◼   ► kind of a good thing to walk through but

00:03:01 ◼   ► I don't do it in a formal way

00:03:03 ◼   ► marco do you do it any kind of formal

00:03:05 ◼   ► testing or processes that you have

00:03:07 ◼   ► around putting something out or do you

00:03:08 ◼   ► just kind of try it out a bunch and see

00:03:11 ◼   ► if it works I mostly just try out a

00:03:14 ◼   ► bunch I mean and so so we should clarify

00:03:16 ◼   ► also that test-driven development is not

00:03:19 ◼   ► what we're really talking about neither

00:03:21 ◼   ► of us really do that we've talked about

00:03:23 ◼   ► it briefly before and I don't think

00:03:25 ◼   ► either of us really has a strong opinion

00:03:27 ◼   ► on it except just to say that we don't

00:03:29 ◼   ► do it mostly as a choice because of our

00:03:32 ◼   ► resources and our style of coding

00:03:35 ◼   ► you know just being single person shops

00:03:37 ◼   ► it makes it harder to justify the the

00:03:40 ◼   ► additional amount of time it takes to

00:03:42 ◼   ► write all the tests and for test-driven

00:03:44 ◼   ► development so that's not what we're

00:03:46 ◼   ► talking about here today at least so

00:03:49 ◼   ► yeah so when it comes to whatever

00:03:52 ◼   ► testing people would call what we used

00:03:54 ◼   ► to call testing just like you or other

00:03:58 ◼   ► people like actually just using your app

00:04:01 ◼   ► and trying out all the different things

00:04:02 ◼   ► you know trying to turn you know testing

00:04:05 ◼   ► out edge cases you know the things that

00:04:07 ◼   ► a good tester a QA person would do that

00:04:09 ◼   ► is that's what we're talking about today

00:04:11 ◼   ► and so my answer to that is generally I

00:04:15 ◼   ► I mostly rely on my own usage and their

00:04:18 ◼   ► and reports from beta testers in order

00:04:21 ◼   ► to to do this and

00:04:23 ◼   ► this has pluses and minuses I mean

00:04:25 ◼   ► obviously it this is a less disciplined

00:04:28 ◼   ► approach than having like a formal unit

00:04:31 ◼   ► testing process a formal UI testing

00:04:33 ◼   ► process having dedicated testing people

00:04:37 ◼   ► or services I should look more into

00:04:41 ◼   ► those things I haven't to date and and

00:04:44 ◼   ► it mostly has not caused problems

00:04:46 ◼   ► there's and and part of this is because

00:04:48 ◼   ► of the way I developed the app in the

00:04:50 ◼   ► first place I mean step number one is I

00:04:52 ◼   ► only make apps that I use if you are

00:04:55 ◼   ► making an app that you don't use

00:04:57 ◼   ► yourself on a regular basis and there's

00:05:00 ◼   ► lots of reasons why you might want to do

00:05:01 ◼   ► this you know if you're a consultant

00:05:02 ◼   ► obviously that's a big one

00:05:04 ◼   ► but you know a lot of people have these

00:05:05 ◼   ► needs if that's the kind of need that

00:05:07 ◼   ► that you're working with or you don't

00:05:09 ◼   ► use the app I really don't know what to

00:05:10 ◼   ► tell you I think things like unit

00:05:12 ◼   ► testing make more sense there or having

00:05:14 ◼   ► having like dedicated QA people or

00:05:16 ◼   ► services that you can hire to do that

00:05:17 ◼   ► but because I use the app myself and and

00:05:22 ◼   ► I also tend to only implement features

00:05:25 ◼   ► that I will use and and this is not I I

00:05:29 ◼   ► don't do that 100% of the time like

00:05:30 ◼   ► there are certain features like in my

00:05:32 ◼   ► nitpicky details feature about the

00:05:34 ◼   ► showing the number of unplayed unplayed

00:05:36 ◼   ► episode as a red badge on the icon of

00:05:39 ◼   ► the app I think you can tell by the

00:05:41 ◼   ► description text in that interface that

00:05:43 ◼   ► I don't want to use this feature but

00:05:45 ◼   ► enough people requested it and it was a

00:05:47 ◼   ► small enough feature I mean that

00:05:49 ◼   ► features something like four lines of

00:05:50 ◼   ► code it's it's it's a very very very

00:05:52 ◼   ► tiny little feature and it satisfied

00:05:55 ◼   ► large user demand a lot of people

00:05:58 ◼   ► requested that feature always before I

00:05:59 ◼   ► added it and so it satisfies a large

00:06:02 ◼   ► user demand it was not that much work to

00:06:03 ◼   ► do and it's very easy for me to test

00:06:06 ◼   ► every so often even though I don't use

00:06:08 ◼   ► it myself so that's something I added

00:06:09 ◼   ► anyway despite not using it myself but

00:06:12 ◼   ► otherwise I try not to add features that

00:06:15 ◼   ► I won't use myself very often if at all

00:06:17 ◼   ► in part because I don't care and in part

00:06:21 ◼   ► because it makes it harder for me to

00:06:23 ◼   ► maintain the quality of the app because

00:06:24 ◼   ► I won't see problems in day-to-day

00:06:26 ◼   ► testing like one of the one of the

00:06:28 ◼   ► issues that I faced recently is I have a

00:06:30 ◼   ► pretty severe carplay bug in the most

00:06:32 ◼   ► recent build and I'm working on fixing

00:06:34 ◼   ► that now I should have it for the next

00:06:36 ◼   ► one but

00:06:37 ◼   ► I don't have a car play vehicle we don't

00:06:39 ◼   ► own a car play vehicle in our household

00:06:40 ◼   ► and we actually are about to get one in

00:06:42 ◼   ► about a month and that'll probably help

00:06:43 ◼   ► although it won't be my car which is

00:06:45 ◼   ► unfortunate but it's at least my wife

00:06:47 ◼   ► will have one so I will have hardware to

00:06:49 ◼   ► test on sometimes but like car plays

00:06:51 ◼   ► this whole area of the app that I never

00:06:53 ◼   ► see because I don't drive a car that's

00:06:56 ◼   ► car play enabled and so there are going

00:06:59 ◼   ► to be all these different areas where my

00:07:01 ◼   ► own day-to-day usage is not going to

00:07:03 ◼   ► cover it or is not going to reveal

00:07:06 ◼   ► subtle problems at least and so for

00:07:08 ◼   ► those kind of uses I really just rely on

00:07:10 ◼   ► beta testers and reports from users

00:07:13 ◼   ► hopefully you know hopefully before I

00:07:15 ◼   ► release these versions that have these

00:07:16 ◼   ► that have problems but you know

00:07:18 ◼   ► sometimes sometimes they get out there

00:07:20 ◼   ► and so in order to in order to minimize

00:07:22 ◼   ► the damage I do try to at least like at

00:07:26 ◼   ► least cover the the biggest most common

00:07:29 ◼   ► uses of the app in my own formal or

00:07:32 ◼   ► informal testing so for example what I'm

00:07:34 ◼   ► releasing a big version I will almost

00:07:36 ◼   ► always if I remember to which isn't

00:07:39 ◼   ► always the case which is a problem but

00:07:41 ◼   ► actually probably this is again this is

00:07:42 ◼   ► like one of the reasons why it might be

00:07:43 ◼   ► useful to have like a checklist or

00:07:45 ◼   ► something but or you know a more formal

00:07:47 ◼   ► structure of doing these things or an

00:07:48 ◼   ► automated way of doing these things but

00:07:50 ◼   ► one of the things that I do in my app is

00:07:53 ◼   ► to to basically start fresh start from a

00:07:56 ◼   ► brand new user every time I do a release

00:07:59 ◼   ► so like take a device that doesn't have

00:08:01 ◼   ► the app on an install a fresh version of

00:08:03 ◼   ► the app and then try creating a new

00:08:05 ◼   ► account and starting all over again and

00:08:07 ◼   ► this helps for a number of reasons

00:08:09 ◼   ► obviously that's a very important thing

00:08:12 ◼   ► for your app to make sure it works at

00:08:14 ◼   ► all and then secondly it shows me the

00:08:17 ◼   ► onboarding experience it shows me the

00:08:18 ◼   ► first run experience a lot like I see

00:08:21 ◼   ► that whenever I whenever I do we release

00:08:23 ◼   ► I I will see the first run experience

00:08:24 ◼   ► and at least once during that process so

00:08:27 ◼   ► that kind of helps remind me of any

00:08:30 ◼   ► shortcomings in it or it helps you know

00:08:32 ◼   ► helps show me if anything's broken about

00:08:33 ◼   ► it or if if things aren't quite right

00:08:35 ◼   ► you know it's so that's nice you know in

00:08:37 ◼   ► that way as well but this is all a very

00:08:40 ◼   ► long way of saying basically the way I

00:08:42 ◼   ► do testing is I just use the app a lot

00:08:45 ◼   ► myself before I let anybody else use it

00:08:46 ◼   ► and then when it gets close to release I

00:08:49 ◼   ► do beta tests

00:08:51 ◼   ► and I can talk a little bit more about

00:08:53 ◼   ► that maybe a little bit later but the

00:08:55 ◼   ► beta test I use tend to find almost any

00:08:58 ◼   ► other reigning problems yeah I think

00:09:00 ◼   ► it's to is it's important to say like I

00:09:02 ◼   ► mean I think my testing strategy is very

00:09:04 ◼   ► similar to yours it's the I use the app

00:09:06 ◼   ► a lot and I try and sort of use it like

00:09:13 ◼   ► as I get closer I tend to kind of use it

00:09:14 ◼   ► slightly and anger like I use it in a

00:09:16 ◼   ► way that it's kind of trying to break it

00:09:18 ◼   ► like I'm bashing on all the buttons and

00:09:20 ◼   ► like what if I change this setting

00:09:21 ◼   ► really quickly back and forth back and

00:09:23 ◼   ► forth like is anything weird gonna

00:09:24 ◼   ► happen like you can use it in that way

00:09:26 ◼   ► and then yeah like you said it's that

00:09:28 ◼   ► kind of testing where and the honestly

00:09:30 ◼   ► what I usually do is for the like trying

00:09:32 ◼   ► out a new user testing it's the you know

00:09:35 ◼   ► I - I currently use an iPhone 7 that's

00:09:37 ◼   ► the phone that I use like during

00:09:40 ◼   ► development like 98% of my development

00:09:42 ◼   ► happens on that phone and so before I

00:09:44 ◼   ► launch I need to you know very least run

00:09:46 ◼   ► it unlike a 5s and on a 7 plus I'm just

00:09:51 ◼   ► in terms of to make sure that there's

00:09:52 ◼   ► not any weird UI bugs or issues or

00:09:54 ◼   ► sometimes even just weird there's

00:09:57 ◼   ► performance things obviously going if

00:09:58 ◼   ► you go back to like a 5 or a 5s they're

00:10:01 ◼   ► much slower and so that's a I find a

00:10:02 ◼   ► great place to do the kind of on board a

00:10:04 ◼   ► 2 on board testing because you know I'm

00:10:08 ◼   ► starting fresh on those devices half the

00:10:09 ◼   ► time anyway that will also say one thing

00:10:12 ◼   ► that has bitten me many many times in

00:10:14 ◼   ► the past and so now I always try and

00:10:16 ◼   ► make sure I do too is I do the fresh

00:10:20 ◼   ► user onboarding experience process but I

00:10:23 ◼   ► do that running the old version of the

00:10:26 ◼   ► app like the existing I could go to the

00:10:28 ◼   ► App Store and download whatever this is

00:10:30 ◼   ► whatever the current version of the app

00:10:32 ◼   ► is from the App Store on board a fresh

00:10:34 ◼   ► user and then install the version that

00:10:36 ◼   ► I'm trying to verify and test with nice

00:10:40 ◼   ► because often what I found is there's

00:10:41 ◼   ► issues that you can run into that like

00:10:43 ◼   ► the you know so usually it's a data

00:10:45 ◼   ► issue or a settings issue or something

00:10:47 ◼   ► that isn't being translated over

00:10:49 ◼   ► correctly and obviously the majority of

00:10:51 ◼   ► your users are going to be are like are

00:10:54 ◼   ► coming from that previous version and

00:10:56 ◼   ► then running your new version and so if

00:10:58 ◼   ► there's any issues there you want to

00:10:59 ◼   ► catch them but that kind of a this kind

00:11:01 ◼   ► of approach of just taking a lot of

00:11:03 ◼   ► devices like on my desk

00:11:04 ◼   ► I have a bunch of different you know

00:11:06 ◼   ► there's all the different iPhones like I

00:11:08 ◼   ► never sell an iPhone whenever I get a

00:11:10 ◼   ► new one and so I have all that all the

00:11:12 ◼   ► different testing devices available to

00:11:14 ◼   ► me that way

00:11:15 ◼   ► you know similarly like right now I'm

00:11:17 ◼   ► wearing two Apple watches I have a 38

00:11:20 ◼   ► millimeter Apple watch on my right hand

00:11:21 ◼   ► and a 42 millimeter repla watch on my

00:11:23 ◼   ► left hand and they're paired two

00:11:25 ◼   ► separate phones many seven sizes and I'm

00:11:27 ◼   ► just running the app that I'm testing

00:11:29 ◼   ► constantly now like I just he keep this

00:11:33 ◼   ► process of kind of banging up against it

00:11:34 ◼   ► and if in general I think that it's it's

00:11:37 ◼   ► you'll it's amazing how good much how

00:11:40 ◼   ► good the coverage you can get from it

00:11:42 ◼   ► that kind of a informal but like

00:11:46 ◼   ► constant high-volume testing like maybe

00:11:48 ◼   ► it's not as rigorous and so it loses

00:11:51 ◼   ► something there but it definitely

00:11:52 ◼   ► benefits from just doing it all the time

00:11:54 ◼   ► and if you build apps that you use you

00:11:57 ◼   ► certainly have that benefit at the very

00:11:59 ◼   ► least of exposure to it yeah I mean one

00:12:02 ◼   ► other note on the devices before I

00:12:04 ◼   ► forget

00:12:04 ◼   ► app review does not seem to test much if

00:12:08 ◼   ► at all on a variety of devices as far as

00:12:11 ◼   ► I can tell they test on one device only

00:12:13 ◼   ► and usually it's a recent one so as far

00:12:16 ◼   ► as I can tell so like a lot of times

00:12:18 ◼   ► people people will have issues where

00:12:19 ◼   ► like they'll get an update into the

00:12:21 ◼   ► store approved that like crashes on

00:12:24 ◼   ► launched on the iPhone 5s or something

00:12:26 ◼   ► like that right because it's like an

00:12:27 ◼   ► older device that is still supported and

00:12:29 ◼   ► it still runs on and that your customers

00:12:30 ◼   ► might still use but it doesn't work for

00:12:32 ◼   ► you know it and I purview didn't catch

00:12:35 ◼   ► it you know and so there's you can't

00:12:37 ◼   ► depend on app review to catch like the

00:12:40 ◼   ► big obvious Crashers because if it

00:12:42 ◼   ► happens on anything any conditions that

00:12:44 ◼   ► they're not trying including any other

00:12:46 ◼   ► device besides whatever one they happen

00:12:47 ◼   ► to be using you're not going to have a

00:12:49 ◼   ► good time so that's that's a very very

00:12:51 ◼   ► important to note and it's also probably

00:12:53 ◼   ► worth saying that app review is not QA

00:12:55 ◼   ► like app review is this very like course

00:13:00 ◼   ► sieve that every now and then will catch

00:13:03 ◼   ► problems and I love it when they do this

00:13:04 ◼   ► but like that is not at all what they're

00:13:07 ◼   ► doing they're not really looking to test

00:13:09 ◼   ► your app to find problems they're too

00:13:11 ◼   ► kind of running it through at a very

00:13:13 ◼   ► different level and so I would not as

00:13:16 ◼   ► certainly isn't one of those like well

00:13:17 ◼   ► I'll just

00:13:18 ◼   ► if a perv you can find any bugs and if

00:13:19 ◼   ► they don't I'm sure it's fine like that

00:13:21 ◼   ► is a tear there'll be a terrible

00:13:22 ◼   ► approach to take to you with your app

00:13:25 ◼   ► yeah it's kind it's like the seat belt

00:13:26 ◼   ► on a plane it's like that's that is not

00:13:29 ◼   ► the safety mechanism you should be

00:13:30 ◼   ► relying on this you should definitely

00:13:32 ◼   ► like it basically if a preview rejects

00:13:35 ◼   ► your app for a crash you have a serious

00:13:38 ◼   ► problem in your QA process it's like

00:13:39 ◼   ► whatever you submit to ever view should

00:13:41 ◼   ► not be in that kind of state where it

00:13:42 ◼   ► would crash in like the first four

00:13:44 ◼   ► minutes of somebody using it once you

00:13:46 ◼   ► know but you know overall like and I

00:13:49 ◼   ► should also you know suggest to that if

00:13:51 ◼   ► you are going to rely so heavily as we

00:13:54 ◼   ► do on self testing basically and again I

00:13:57 ◼   ► I do I apologize greatly to people who

00:14:01 ◼   ► enjoy formal testing and automated

00:14:03 ◼   ► testing and unit testing and then 100%

00:14:05 ◼   ► test coverage I apologize because this

00:14:07 ◼   ► this entire discussion must be driving

00:14:10 ◼   ► you totally crazy hi Casey

00:14:11 ◼   ► but ultimately it's hard for single

00:14:17 ◼   ► person developers to really justify the

00:14:20 ◼   ► additional time and/or expense that more

00:14:24 ◼   ► rigorous testing methods bring on for

00:14:27 ◼   ► things like this so now that's a lot of

00:14:29 ◼   ► qualifiers in that sentence obviously if

00:14:32 ◼   ► you have a larger staff if you have a

00:14:34 ◼   ► lot of money if your app is more

00:14:37 ◼   ► important and then like you know then

00:14:40 ◼   ► then you know like a sleep tracker for

00:14:42 ◼   ► your wrist or for the podcast player

00:14:45 ◼   ► like if you're really bad things happen

00:14:48 ◼   ► if you like lose people's important data

00:14:50 ◼   ► or you know if somebody will be

00:14:52 ◼   ► seriously put out or hurt if your app

00:14:55 ◼   ► breaks obviously that's a place where

00:14:57 ◼   ► you need a more formal testing procedure

00:14:59 ◼   ► and and framework in place but most

00:15:02 ◼   ► people who make iOS apps aren't working

00:15:04 ◼   ► in that kind of capacity and and not

00:15:07 ◼   ► only don't have those resources but it

00:15:09 ◼   ► would actually be be kind of a bad way

00:15:11 ◼   ► to spend resources basically like

00:15:13 ◼   ► tripling all of your work and this is a

00:15:15 ◼   ► huge debate in programmer circles of

00:15:17 ◼   ► whether test-driven development

00:15:18 ◼   ► is really more work or whether it ends

00:15:19 ◼   ► up saving you more work in the long run

00:15:21 ◼   ► I honestly don't care which one of those

00:15:24 ◼   ► it is I think it's probably unknowable

00:15:26 ◼   ► because it's probably like the answer

00:15:28 ◼   ► like everything else is probably it

00:15:29 ◼   ► depends but but ultimately

00:15:32 ◼   ► this isn't the way we do things and and

00:15:34 ◼   ► I think there's a lot of developers out

00:15:35 ◼   ► there who who do things more like what

00:15:37 ◼   ► we do anyway that being said what more

00:15:39 ◼   ► thing before I go into the sponsor for

00:15:41 ◼   ► the week and that is if you're relying

00:15:42 ◼   ► so much on self testing as we do if your

00:15:45 ◼   ► app has like you know different modes or

00:15:47 ◼   ► settings or ways you can do things

00:15:50 ◼   ► change it up like in your in your day to

00:15:53 ◼   ► day usage change it up every so often

00:15:55 ◼   ► and spend a lot of time in each kind of

00:15:57 ◼   ► condition so for instance overcast has

00:15:59 ◼   ► dark mode or light mode and during

00:16:01 ◼   ► development I usually switch about once

00:16:03 ◼   ► a week between which one of those I'm

00:16:05 ◼   ► using because I want to kind of get used

00:16:07 ◼   ► to it and live in it and make sure that

00:16:09 ◼   ► I'm that I'm designing and making the

00:16:11 ◼   ► best app I can for both of those things

00:16:12 ◼   ► same thing with streaming versus

00:16:14 ◼   ► downloading like you know I wrote this

00:16:15 ◼   ► whole streaming engine I hardly ever use

00:16:17 ◼   ► it and that's kind of a problem so

00:16:19 ◼   ► basically I force myself to change my

00:16:23 ◼   ► own preference and and spend a good

00:16:26 ◼   ► amount of time like a week or a month in

00:16:28 ◼   ► streaming mode in the middle of

00:16:30 ◼   ► development cycles where I would

00:16:31 ◼   ► otherwise be in downloading mode you

00:16:32 ◼   ► know just just to make sure I'm getting

00:16:34 ◼   ► a really good idea of like these vastly

00:16:38 ◼   ► different modes that my app can be in so

00:16:40 ◼   ► that my own testing is more effective

00:16:42 ◼   ► anyway we were sponsored this week by

00:16:45 ◼   ► Pingdom Pingdom can help remove the need

00:16:48 ◼   ► for you to have somebody who

00:16:49 ◼   ► automatically logs in your website every

00:16:51 ◼   ► minute and refreshes it to make sure

00:16:52 ◼   ► it's up because Pingdom does that

00:16:54 ◼   ► automatically with their testing

00:16:55 ◼   ► framework see for yourself at Pingdom

00:16:58 ◼   ► comm slash radar Pingdom is an awesome

00:17:01 ◼   ► monitoring service for websites and web

00:17:03 ◼   ► services and when you use offer code

00:17:05 ◼   ► radar at checkout you get a holiday

00:17:08 ◼   ► special of 50% off your first invoice

00:17:11 ◼   ► this is an awesome deal usually 20% now

00:17:14 ◼   ► it's 50 percent for the holidays so

00:17:16 ◼   ► check it out at Pingdom comm slash radar

00:17:18 ◼   ► Pingdom offers powerful easy to use

00:17:21 ◼   ► monitoring tools and services for your

00:17:24 ◼   ► website and web servers web sites are

00:17:26 ◼   ► very sophisticated these days and you

00:17:28 ◼   ► need you know there's all sorts of

00:17:30 ◼   ► things that can go wrong and they know

00:17:31 ◼   ► this they see it because they monitor it

00:17:33 ◼   ► for lots of people

00:17:34 ◼   ► you need Pingdom to be monitoring your

00:17:36 ◼   ► site they can monitor as often as every

00:17:38 ◼   ► minute by requesting pages by running

00:17:40 ◼   ► through checkouts by logging in by doing

00:17:42 ◼   ► you know transactions that they can do

00:17:44 ◼   ► all sorts of things all

00:17:46 ◼   ► simulated from there more than 70 global

00:17:48 ◼   ► test servers and they can do it as often

00:17:49 ◼   ► as once a minute and if anything goes

00:17:51 ◼   ► down they can alert you via so many

00:17:53 ◼   ► different means whatever you want you

00:17:55 ◼   ► can have email text push notification or

00:17:58 ◼   ► all an income any combination of these

00:17:59 ◼   ► things and you can set the granularity

00:18:01 ◼   ► it's incredibly customizable I've been

00:18:03 ◼   ► using Pingdom now for oh geez I think

00:18:06 ◼   ► eight years nine years something crazy

00:18:08 ◼   ► it's a very long time because I've used

00:18:10 ◼   ► it way before they were sponsor way

00:18:11 ◼   ► before I was even a podcaster I was

00:18:14 ◼   ► using Pingdom to monitor tumbler back in

00:18:15 ◼   ► the day and and I just love them I used

00:18:18 ◼   ► them constantly highly recommended that

00:18:20 ◼   ► you use Pingdom for your all your

00:18:21 ◼   ► monitoring needs it is really the best

00:18:24 ◼   ► and if you run any kind of website or

00:18:26 ◼   ► web service the last thing you want is

00:18:28 ◼   ► for strangers on Twitter to be telling

00:18:31 ◼   ► you when your site is down when you

00:18:32 ◼   ► didn't already know that like you do not

00:18:34 ◼   ► want to be learning that from Twitter

00:18:35 ◼   ► you want to know when your site is down

00:18:37 ◼   ► you want to be the first to know and

00:18:39 ◼   ► then you want to be able to go in and

00:18:41 ◼   ► fix it before all the people on Twitter

00:18:43 ◼   ► see it and start bugging you about it

00:18:44 ◼   ► and with Pingdom you can because Pingdom

00:18:47 ◼   ► will alert you the moment things go down

00:18:49 ◼   ► with that up to one minute granularity

00:18:51 ◼   ► that's really incredible and they are so

00:18:52 ◼   ► fast and so reliable

00:18:54 ◼   ► I love Pingdom check them out yourself

00:18:56 ◼   ► go to Pingdom comm slash of radar you

00:18:59 ◼   ► will get a 14-day free trial and when

00:19:01 ◼   ► you enter offer code radar at checkout

00:19:03 ◼   ► you will get a holiday special of 50%

00:19:05 ◼   ► off your first invoice check it out

00:19:08 ◼   ► today no be the first to know when your

00:19:09 ◼   ► site is down thank you very much to

00:19:10 ◼   ► Pingdom for sponsoring our show so I

00:19:13 ◼   ► think for the last part of the show it

00:19:15 ◼   ► seemed like some that may actually make

00:19:16 ◼   ► sense for us to focusing on is a bit of

00:19:19 ◼   ► the actual like the details of what this

00:19:22 ◼   ► kind of user testing approach that we've

00:19:25 ◼   ► just been talking about it looks like

00:19:27 ◼   ► and for me what that usually means is so

00:19:30 ◼   ► I get into this phase now where I'm

00:19:32 ◼   ► doing this final level of testing and

00:19:33 ◼   ► I'm just constantly trying to think of

00:19:35 ◼   ► things to try I'm trying them out I'm

00:19:38 ◼   ► using the app in different circumstances

00:19:39 ◼   ► in different places like I'll be using

00:19:41 ◼   ► it on different networks like hey I'm

00:19:42 ◼   ► out on a cell network I'm at home now

00:19:45 ◼   ► and so whenever I find an issue you're

00:19:46 ◼   ► inevitably you'll find a bug you'll see

00:19:48 ◼   ► something that you're like huh that's a

00:19:49 ◼   ► little interesting what I found is a

00:19:52 ◼   ► very important thing is to have some

00:19:53 ◼   ► kind of reliable way of collecting

00:19:57 ◼   ► all these little issues and bugs and

00:19:59 ◼   ► problems and for me usually that is I

00:20:01 ◼   ► have I'll have a list in OmniFocus that

00:20:05 ◼   ► I am just adding items to and anytime I

00:20:09 ◼   ► see something usually I'll grab a

00:20:11 ◼   ► screenshot of it and if it's obvious

00:20:14 ◼   ► I'll just attach the screenshot to the

00:20:16 ◼   ► task in OmniFocus or if not I'll take it

00:20:19 ◼   ► and run it through what used to be an

00:20:22 ◼   ► app that you made Marco which used to be

00:20:23 ◼   ► called buckshot what I think is now

00:20:24 ◼   ► called pin point mmm which is a great

00:20:26 ◼   ► little tool we have a link in the show

00:20:27 ◼   ► notes to it but it's just a little thing

00:20:29 ◼   ► for just drawing arrows basically or

00:20:30 ◼   ► circling things um it's great for screen

00:20:33 ◼   ► shot sort of when you're like huh

00:20:34 ◼   ► you know there's this weird graphics

00:20:36 ◼   ► glitch or something so you can mark it

00:20:38 ◼   ► up add it to it to get and then you know

00:20:41 ◼   ► I go through all of like and when I'm in

00:20:44 ◼   ► the face now like this morning I sat

00:20:45 ◼   ► down I had the head a list about five or

00:20:47 ◼   ► six little bugs I've noticed yesterday I

00:20:50 ◼   ► went through and I'm just knocking them

00:20:52 ◼   ► out and it's kind of like this I work

00:20:55 ◼   ► all the way down till I have no bugs and

00:20:57 ◼   ► then I make a build and then I go

00:20:58 ◼   ► through the process again and you kind

00:21:00 ◼   ► of just keep cycling through collecting

00:21:02 ◼   ► as you go and for me that kind of just

00:21:05 ◼   ► that kind of process of just keeping

00:21:08 ◼   ► this running log tea seems to work well

00:21:10 ◼   ► it I like doing it that way rather than

00:21:13 ◼   ► kind of you know having like sitting

00:21:17 ◼   ► down and just testing for hours never

00:21:19 ◼   ► really works for me I find that I get

00:21:20 ◼   ► kind of bored or I start doing the same

00:21:23 ◼   ► patterns over and over again and so it's

00:21:25 ◼   ► better I think for me I'd try and do it

00:21:26 ◼   ► spread it out throughout the day and

00:21:28 ◼   ► obviously if it's an application that

00:21:29 ◼   ► you would use normally and naturally

00:21:32 ◼   ► throughout the day then this makes it

00:21:33 ◼   ► easier but otherwise you just kind of

00:21:36 ◼   ► have to sort of remind yourself to do it

00:21:39 ◼   ► and this is also a little bit tricky if

00:21:41 ◼   ► you have an app that requires a certain

00:21:44 ◼   ► amount of time for testing so like for

00:21:47 ◼   ► example I have a sleep tracker like in

00:21:48 ◼   ► order for it to really like like I

00:21:51 ◼   ► cannot compressed beyond just running it

00:21:53 ◼   ► on multiple devices the testing of that

00:21:56 ◼   ► because it takes about eight hours to

00:21:58 ◼   ► have eight hours of data collection and

00:22:00 ◼   ► like I have simulated stuff in the app

00:22:03 ◼   ► for you know when I'm doing like debug

00:22:05 ◼   ► testing that I can kind of like create a

00:22:06 ◼   ► night's data and run my algorithms on it

00:22:09 ◼   ► but for actual testing there's nothing

00:22:11 ◼   ► you can really do other than wearing

00:22:13 ◼   ► local watches which is what I do like

00:22:15 ◼   ► when I'm doing extensive sleep you know

00:22:17 ◼   ► testing like I'll have all my watches on

00:22:19 ◼   ► my wrist during the night and I'm

00:22:21 ◼   ► running it you know three or four times

00:22:22 ◼   ► but you know sort of in the same way I

00:22:24 ◼   ► imagine for you with overcast like

00:22:26 ◼   ► there's a certain amount if you just

00:22:26 ◼   ► have to play a lots of audio because if

00:22:29 ◼   ► you don't you're never going to find all

00:22:31 ◼   ► the weird audio issues or glitches or

00:22:33 ◼   ► problems there's nothing that you can

00:22:35 ◼   ► really do to speed that up or compress

00:22:37 ◼   ► that testing yeah basically I mean I

00:22:40 ◼   ► like I just find ways to shove podcasts

00:22:44 ◼   ► into more of my life so so like you know

00:22:47 ◼   ► like like ivories and and also to test

00:22:48 ◼   ► out things like syncing you know like

00:22:50 ◼   ► this my syncing engine took forever to

00:22:52 ◼   ► really figure it mean took me almost a

00:22:53 ◼   ► year to really nail syncing and one of

00:22:57 ◼   ► the ways I started finally getting

00:22:58 ◼   ► better at that was I started needing it

00:23:00 ◼   ► more because I started using an iPad to

00:23:02 ◼   ► play podcast in the kitchen and I used

00:23:04 ◼   ► my iphone to play them everywhere else

00:23:06 ◼   ► so I will I was very frequently

00:23:08 ◼   ► switching between multiple devices that

00:23:10 ◼   ► would use the sink back-end and just by

00:23:12 ◼   ► doing that myself but by needing that

00:23:14 ◼   ► myself for a few months I really fixed a

00:23:17 ◼   ► ton of sink problems and got it to a

00:23:18 ◼   ► pretty good place now and and so yeah I

00:23:21 ◼   ► mean it really is just a matter of just

00:23:23 ◼   ► you know again it's like if you only

00:23:25 ◼   ► make apps that you will use a lot this

00:23:27 ◼   ► becomes a lot easier if if you don't

00:23:30 ◼   ► make apps that use a lot this you know

00:23:32 ◼   ► you're gonna have to find other ways to

00:23:33 ◼   ► do this effectively but it really works

00:23:36 ◼   ► that great for me and and I I will again

00:23:38 ◼   ► once again go in briefly about beta

00:23:40 ◼   ► testing also and and we pretty sure we

00:23:43 ◼   ► did a whole episode on it if not we will

00:23:45 ◼   ► and we should but I think we already did

00:23:46 ◼   ► we have so many episodes now I've kind

00:23:48 ◼   ► of lost track

00:23:49 ◼   ► but I do rely a lot on beta testers to

00:23:53 ◼   ► to really do like a final a final check

00:23:56 ◼   ► on the app they aren't my number right

00:23:58 ◼   ► there are my first testers I am my first

00:24:00 ◼   ► tester and I I will usually use changes

00:24:03 ◼   ► to the app for weeks or even months

00:24:05 ◼   ► before I give it to anybody else to test

00:24:07 ◼   ► just because I really want to make sure

00:24:09 ◼   ► that I know I have feel for it make sure

00:24:10 ◼   ► it's good make sure it's the right

00:24:12 ◼   ► choice and then I will finally share

00:24:13 ◼   ► with testers like right before the

00:24:14 ◼   ► public gets it but basically beta

00:24:17 ◼   ► testers can can help a little bit not a

00:24:19 ◼   ► ton but but a little bit I will also add

00:24:23 ◼   ► that the what you mentioned earlier

00:24:26 ◼   ► about your workflow about taking taking

00:24:28 ◼   ► screenshots and then adding arrows and

00:24:29 ◼   ► then adding them to OmniFocus and things

00:24:31 ◼   ► like that there if you want to go off

00:24:33 ◼   ► the deep end there are lots of crazy

00:24:35 ◼   ► tools and frameworks that automate that

00:24:37 ◼   ► process to varying degrees and have all

00:24:39 ◼   ► sorts of cool integrations so there are

00:24:40 ◼   ► some like you know like I made bug shot

00:24:42 ◼   ► kit which is basically shoved bug shot

00:24:45 ◼   ► into an app as you could do things like

00:24:47 ◼   ► it will detect screenshots being taken

00:24:50 ◼   ► and then automatically bring up a modal

00:24:53 ◼   ► controller with the screenshot setting

00:24:54 ◼   ► here mark this up and send it in to

00:24:56 ◼   ► report a bug you know and you can you

00:24:57 ◼   ► can give that to all your testers and

00:24:59 ◼   ► you can use that yourself during your

00:25:00 ◼   ► test process and so you can basically

00:25:01 ◼   ► have like easy or automated bug

00:25:03 ◼   ► reporting right right embedded into your

00:25:05 ◼   ► app and there's again there's lots of

00:25:08 ◼   ► ways to do this I'm not gonna recommend

00:25:09 ◼   ► any particular one because I haven't

00:25:10 ◼   ► looked but there are many diseases exist

00:25:12 ◼   ► now and you can you can really go off

00:25:14 ◼   ► the deep end with like the levels of

00:25:16 ◼   ► integrations you can have it like post

00:25:18 ◼   ► to slack and then it make an entry in

00:25:20 ◼   ► your bug tracker like all sorts of crazy

00:25:22 ◼   ► things you can do now if you really want

00:25:23 ◼   ► to get into all that stuff so the you

00:25:25 ◼   ► know the options are pretty good for now

00:25:27 ◼   ► for like tools and frameworks and things

00:25:29 ◼   ► like that to automate like the the

00:25:31 ◼   ► tedious stuff but for the for the real

00:25:34 ◼   ► just using the app really just again

00:25:37 ◼   ► just using it a lot yourself is a very

00:25:40 ◼   ► very effectively to do it obviously as I

00:25:42 ◼   ► know automated things exist sorry Casey

00:25:43 ◼   ► again but just using your app yourself

00:25:45 ◼   ► will show you so much and will tell you

00:25:47 ◼   ► so much and will tell you will show you

00:25:48 ◼   ► things like you know this this workflow

00:25:50 ◼   ► is kind of clunky or this animation

00:25:53 ◼   ► doesn't really look right or this I I

00:25:55 ◼   ► really need this text to be lighter when

00:25:57 ◼   ► it's really sunny outside or something

00:25:58 ◼   ► like that like there's a large category

00:26:01 ◼   ► of either problems or or good and bad

00:26:04 ◼   ► design decisions for which just using

00:26:07 ◼   ► yourself is really the only way to know

00:26:08 ◼   ► and the really the only way to test and

00:26:10 ◼   ► that isn't true for everything obviously

00:26:12 ◼   ► sorry again Casey but that's true for

00:26:14 ◼   ► quite a lot that we do in our apps and I

00:26:16 ◼   ► think with other thing I would say with

00:26:18 ◼   ► beta testing that's an important thing

00:26:19 ◼   ► too it's another use for beta testing is

00:26:23 ◼   ► the importance of not overly preparing

00:26:26 ◼   ► your beta testers for what you know for

00:26:30 ◼   ► the app so in terms of trying to give

00:26:32 ◼   ► him giving them an experience that is

00:26:33 ◼   ► similar to the experience that your app

00:26:36 ◼   ► store users are going to have

00:26:37 ◼   ► and so when they download the app and

00:26:40 ◼   ► they're like what do I do or this is

00:26:42 ◼   ► really confusing like some of the best

00:26:44 ◼   ► feedback I get from beta testers is sort

00:26:46 ◼   ► of accidental where they're not actually

00:26:49 ◼   ► they don't know it but they're telling

00:26:51 ◼   ► me something is fundamentally flawed or

00:26:53 ◼   ► is really confusing where they're asking

00:26:55 ◼   ► a question that like it implies missing

00:26:58 ◼   ► a fundamental misunderstanding about

00:27:00 ◼   ► something and that kind of feedback is

00:27:02 ◼   ► always really good and so I try whenever

00:27:03 ◼   ► I'm beta testing to just almost like

00:27:06 ◼   ► blindly send out the betas to people

00:27:08 ◼   ► where I'm not giving them too much

00:27:09 ◼   ► preparation or tell are like giving them

00:27:11 ◼   ► step-by-step walkthroughs or details

00:27:13 ◼   ► it's very much like as though you just

00:27:16 ◼   ► downloaded this from the App Store you

00:27:17 ◼   ► know maybe you've seen a screenshot or

00:27:18 ◼   ► two maybe you've had a high-level

00:27:20 ◼   ► discussion but you're working it out

00:27:22 ◼   ► just as you would as a regular user and

00:27:24 ◼   ► that's a great test because have you as

00:27:28 ◼   ► the developer of the app of ously I know

00:27:30 ◼   ► how everything works everything is

00:27:32 ◼   ► intuitive to me because that's the way I

00:27:34 ◼   ► made it but it is often not that not the

00:27:37 ◼   ► case for your users and so one of the

00:27:39 ◼   ► best uses for beta testing isn't it's

00:27:41 ◼   ► great when they find you know actual

00:27:43 ◼   ► bugs and issues and problems but it's

00:27:45 ◼   ► also it's really useful for capturing

00:27:47 ◼   ► those kind of like it's like if you're

00:27:49 ◼   ► asking this question you clearly have

00:27:51 ◼   ► not communicated the purpose of this

00:27:53 ◼   ► feature correctly or I've hidden it in a

00:27:55 ◼   ► weird way or you think it should do

00:27:57 ◼   ► something and it does the opposite and

00:27:58 ◼   ► those are the only thing that is almost

00:28:01 ◼   ► exclusively possible to find in beta

00:28:05 ◼   ► testing because no matter how hard you

00:28:07 ◼   ► try to kind of like reset your brain to

00:28:10 ◼   ► what if I was a fresh user opening the

00:28:11 ◼   ► application it's kind of impossible

00:28:13 ◼   ► where you know I the current app I'm

00:28:15 ◼   ► working on I have been working on it

00:28:17 ◼   ► essentially every day for months at this

00:28:20 ◼   ► point so I'm so in it that I just don't

00:28:24 ◼   ► see it's like I'm so far into the woods

00:28:26 ◼   ► that I have no idea that there is a

00:28:27 ◼   ► forest like all I have is trees like it

00:28:29 ◼   ► is deep down into it I mean and maybe

00:28:33 ◼   ► when I come back to it and a couple of

00:28:35 ◼   ► you know like I'll put the app to the

00:28:37 ◼   ► side for a little bit and I'll come back

00:28:39 ◼   ► to it and maybe a month maybe I'll I

00:28:40 ◼   ► will have that kind of fresh experience

00:28:42 ◼   ► but until that time the only way I can

00:28:45 ◼   ► get that is with beta testing and so it

00:28:46 ◼   ► is definitely a valuable and important

00:28:48 ◼   ► aspect of that

00:28:49 ◼   ► all right and I guess we're gonna see

00:28:52 ◼   ► your new app pretty soon right yeah it

00:28:54 ◼   ► should be hopefully launching next week

00:28:57 ◼   ► so should be kind of exciting and I

00:29:00 ◼   ► expect to go home sure we'll talk about

00:29:01 ◼   ► on the show how the lunch went and kind

00:29:03 ◼   ► of some of the details of it as the app

00:29:05 ◼   ► goes but it is very exciting to get to

00:29:07 ◼   ► this point and I will say as you know

00:29:09 ◼   ► it's like stalking all the other

00:29:10 ◼   ► independent developers out there it's

00:29:12 ◼   ► like like I just put in the work because

00:29:15 ◼   ► there's definitely been times in this

00:29:16 ◼   ► project where I was like oh gosh what

00:29:18 ◼   ► have I gotten myself into maybe I should

00:29:20 ◼   ► just like can this project move on but

00:29:22 ◼   ► there is hope eventually we'll get there

00:29:23 ◼   ► eventually we'll ship and it is

00:29:26 ◼   ► delightful

00:29:26 ◼   ► when you get there can't tell you a

00:29:28 ◼   ► secret to the audience you can I know

00:29:30 ◼   ► what this is and I've used it and it's

00:29:31 ◼   ► really good so look forward to it anyway

00:29:34 ◼   ► thanks a lot for listening everybody

00:29:35 ◼   ► we're out of time this week and we'll

00:29:37 ◼   ► talk to you next week bye