PodSearch

Under the Radar

58: Prerelease Testing

 

00:00:00   Welcome to Under the Radar, a show about independent iOS app development.

00:00:04   I'm Mark Orment.

00:00:05   And I'm David Smith.

00:00:06   Under the Radar is never longer than 30 minutes, so let's get started.

00:00:10   So today we wanted to talk a little bit about testing, about that process we go through

00:00:16   towards the end of developing something where we try and take it from mostly working to

00:00:22   actually working or fully working.

00:00:25   And this is particularly relevant and interesting for me at this point because I am in that

00:00:29   very final of phases for my next app, the one we talked a little bit about last week,

00:00:36   that I'm in this phase now where functionally and from a feature perspective, I've kind

00:00:41   of drawn the line and said, this is the app.

00:00:43   This is as far as I'm going to go with it.

00:00:45   Any other features or any other things that come up are going to be in the next version,

00:00:49   version 1.1, or I'll push it down the road.

00:00:52   All I'm now going to be doing is fixing bugs and testing and finding all the little issues

00:00:57   and problems.

00:00:59   And it seemed like something that would be an interesting topic to kind of walk through

00:01:03   because ultimately the goal is to make sure that we ship things that are quality, are

00:01:10   not riddled with bugs, they're having all kinds of problems.

00:01:14   And it's a tricky thing, I think, especially from an indie perspective.

00:01:18   And I think it's really fair to say, I'm definitely talking about this from the more

00:01:23   of the indie perspective rather than if you are in a large corporate environment where

00:01:28   your app is being worked on by lots of different people.

00:01:31   You may actually have an entire QA department whose job it is to test your application.

00:01:36   I used to work at a job I used to work at.

00:01:40   There was a room full of probably 20 people who all day just sat there using the product,

00:01:46   testing it, trying all the crazy edges, and they had these big test plans that they had

00:01:50   to go through before we did a build.

00:01:52   That is a very different world than I think either of us work in.

00:01:55   And so we're not talking about that kind of testing.

00:01:58   That is a discipline unto itself.

00:02:01   It is useful and important.

00:02:02   But for an independent, I think that's just not the reality, both from a resources perspective.

00:02:09   I don't have the time to realistically do that, to do 20 people times 40 hours a week

00:02:17   of testing.

00:02:18   That's just impossible.

00:02:19   And also, I'm not sure if it's necessary.

00:02:22   And so this is coming into a certain category of bug that I'm less worried about.

00:02:29   This is sort of the integration bug rather than the functional bug, where sometimes you

00:02:35   just have bugs that come up because you had five people working on a product, and one

00:02:40   person changed something here, someone else changed it over there.

00:02:42   And obviously, myself, in different time domains, is different.

00:02:46   I can be, like two weeks ago, me and my current me can be conflicting, but that happens less

00:02:51   often.

00:02:52   But either way, being independent, it's still something that I have to do.

00:02:56   And it's a process that I go through now.

00:02:59   And it seems like kind of a good thing to walk through.

00:03:01   But I don't do it in a formal way.

00:03:03   Marco, do you do any kind of formal testing or processes that you have around putting

00:03:08   something out, or do you just kind of try it out a bunch and see if it works?

00:03:12   I mostly just try out a bunch.

00:03:15   And so we should clarify also that test-driven development is not what we're really talking

00:03:20   about.

00:03:21   Neither of us really do that.

00:03:23   We've talked about it briefly before.

00:03:25   And I don't think either of us really has a strong opinion on it, except just to say

00:03:28   that we don't do it mostly as a choice because of our resources and our style of coding,

00:03:35   of just being single person shops.

00:03:38   It makes it harder to justify the additional amount of time it takes to write all the tests

00:03:43   for test-driven development.

00:03:45   So that's not what we're talking about here today, at least.

00:03:48   So when it comes to whatever testing people would call what we used to call testing.

00:03:55   So yeah, just like you or other people actually just using your app and trying out all the

00:04:02   different things, testing out edge cases, the things that a good tester or QA person

00:04:08   would do.

00:04:09   That's what we're talking about today.

00:04:12   And so my answer to that is generally I mostly rely on my own usage and reports from beta

00:04:19   testers in order to do this.

00:04:23   And this has pluses and minuses.

00:04:25   I mean obviously this is a less disciplined approach than having like a formal unit testing

00:04:32   process, a formal UI testing process, having dedicated testing people or services.

00:04:39   I should look more into those things.

00:04:42   I haven't to date and it mostly has not caused problems.

00:04:47   And part of this is because of the way I developed the app in the first place.

00:04:50   I mean step number one is I only make apps that I use.

00:04:55   If you are making an app that you don't use yourself on a regular basis, and there's lots

00:05:00   of reasons why you might want to do this.

00:05:02   If you're a consultant, obviously that's a big one.

00:05:04   But lots of people have these needs.

00:05:06   If that's the kind of need that you're working with where you don't use the app, I really

00:05:10   don't know what to tell you.

00:05:11   I think things like unit testing make more sense there or having like dedicated QA people

00:05:16   or services that you can hire to do that.

00:05:18   But because I use the app myself, and I also tend to only implement features that I will

00:05:26   use.

00:05:27   And this is not, I don't do that 100% of the time.

00:05:30   There are certain features like my nitpicky details feature about showing the number of

00:05:35   unplayed episodes as a red badge on the icon of the app.

00:05:40   I think you can tell by the description text in that interface that I don't want to use

00:05:44   this feature.

00:05:45   But enough people requested it and it was a small enough feature.

00:05:48   I mean that feature is something like four lines of code.

00:05:50   It's a very, very, very tiny little feature.

00:05:54   And it satisfied large user demand.

00:05:57   A lot of people requested that feature always before I added it.

00:06:01   And so it satisfied a large user demand.

00:06:03   It was not that much work to do.

00:06:05   And it's very easy for me to test every so often, even though I don't use it myself.

00:06:08   So that's something I added anyway, despite not using it myself.

00:06:12   But otherwise, I try not to add features that I won't use myself very often, if at all.

00:06:18   In part because I don't care, and in part because it makes it harder for me to maintain

00:06:23   the quality of the app because I won't see problems in day to day testing.

00:06:28   One of the issues that I faced recently is I have a pretty severe carplay bug in the

00:06:32   most recent build.

00:06:34   And I'm working on fixing that.

00:06:35   I should have it for the next one, but I don't have a carplay vehicle.

00:06:38   We don't own a carplay vehicle in our household.

00:06:40   And we actually are about to get one in about a month.

00:06:43   And that'll probably help.

00:06:44   Although it won't be my car, which is unfortunate.

00:06:46   But at least my wife will have one, so I will have hardware to test on sometimes.

00:06:51   But carplay is this whole area of the app that I never see because I don't drive a

00:06:56   car that's carplay enabled.

00:06:57   And so there are gonna be all these different areas where my own day to day usage is not

00:07:02   going to cover it or is not gonna reveal subtle problems at least.

00:07:08   And so for those kind of uses I really just rely on beta testers and reports from users.

00:07:13   Hopefully it's before I release these versions that have problems, but sometimes they get

00:07:19   out there.

00:07:21   So in order to minimize the damage I do try to at least cover the biggest most common

00:07:29   uses of the app in my own formal or informal testing.

00:07:33   So for example, when I'm releasing a big version I will almost always, if I remember

00:07:38   to which isn't always the case, which is a problem, but I should probably, again this

00:07:42   is one of the reasons why it might be useful to have a checklist or something, or a more

00:07:47   formal structure of doing these things or an automated way of doing these things.

00:07:50   But one of the things that I do in my app is to basically start fresh, start from a

00:07:56   brand new user every time I do a release.

00:08:00   Take a device that doesn't have the app on it, install a fresh version of the app, and

00:08:04   then try creating a new account and starting all over again.

00:08:08   And this helps for a number of reasons.

00:08:10   Obviously that's a very important thing for your app to make sure it works at all.

00:08:15   And then secondly it shows me the onboarding experience, it shows me the first run experience

00:08:20   a lot.

00:08:21   Like I see that whenever I do a release.

00:08:23   I will see the first run experience at least once during that process.

00:08:27   So that kind of helps remind me of any shortcomings in it, or it helps show me if anything's

00:08:33   broken about it, or if things aren't quite right.

00:08:36   So that's nice in that way as well.

00:08:38   But this is all a very long way of saying basically the way I do testing is I just use

00:08:44   the app a lot myself before I let anybody else use it.

00:08:47   And then when it gets close to release I do beta tests.

00:08:52   And I can talk a little bit more about that maybe a little bit later, but the beta tests

00:08:56   I use tend to find almost any other ruining problems.

00:09:00   I think it's important to say like, I mean I think my testing strategy is very similar

00:09:04   to yours.

00:09:05   It's the, I use the app a lot and I try and sort of use it, like as I get closer I

00:09:13   tend to kind of use it slightly in anger.

00:09:15   Like I use it in a way that is kind of trying to break it.

00:09:18   Like I'm bashing on all the buttons and like what if I change this setting really

00:09:21   quickly back and forth, back and forth.

00:09:23   Like is anything weird going to happen?

00:09:24   Like you can use it in that way.

00:09:27   And then yeah like you said it's that kind of testing where, and honestly what I usually

00:09:30   do is for the like trying out a new user testing it's the, you know I currently use an iPhone

00:09:37   7.

00:09:38   That's the phone that I use like during development like 98% of my development happens

00:09:42   on that phone.

00:09:43   And so before I launch I need to at the very least run it on like a 5s and on a 7 plus.

00:09:51   Just in terms of to make sure that there's not any weird UI bugs or issues or sometimes

00:09:55   even just weird, there's performance things obviously going if you go back to like a 5

00:09:59   or a 5s they're much slower.

00:10:01   And so that's I find a great place to do the kind of onboarded, onboard testing because

00:10:07   you know I'm starting fresh on those devices half the time anyway.

00:10:10   I will also say one thing that has bitten me many, many times in the past and so now

00:10:15   I always try and make sure I do too is I do the fresh user onboarding experience process

00:10:22   but I do that running the old version of the app like the existing, like I go to the app

00:10:28   store and download whatever this is, whatever the current version of the app is from the

00:10:32   app store, onboard a fresh user and then install the version that I'm trying to verify and

00:10:39   test with.

00:10:40   Nice.

00:10:41   Because often what I've found is there's issues that you can run into that like the,

00:10:43   you know, usually it's a data issue or a settings issue or something that isn't being

00:10:48   translated over correctly and obviously the majority of your users are going to be, are

00:10:54   coming from that previous version and then running your new version and so if there's

00:10:58   any issues there you want to catch them.

00:11:00   But that kind of approach of just taking a lot of devices like on my desk I have a bunch

00:11:05   of different, all the different iPhones, like I never sell an iPhone whenever I get a new

00:11:10   one and so I have all the different testing devices available to me that way.

00:11:15   You know, similarly like right now I'm wearing two Apple watches.

00:11:18   I have a 38 millimeter Apple watch on my right hand and a 42 millimeter Apple watch on my

00:11:23   left hand and they're paired to separate phones, running seven sizes and I'm just running the

00:11:28   app that I'm testing constantly now.

00:11:30   Like I just keep this process of kind of banging up against it.

00:11:35   And if in general I think like you said, it's amazing how good the coverage you can get

00:11:41   from that kind of informal but like constant high volume testing.

00:11:48   Like maybe it's not as rigorous and so it loses something there but it definitely benefits

00:11:53   from just doing it all the time and if you build apps that you use, you know, you certainly

00:11:57   have that benefit at the very least of exposure to it.

00:12:00   - Yeah, I mean one other note on the devices before I forget, app review does not seem

00:12:07   to test much if at all on a variety of devices.

00:12:11   As far as I can tell they test on one device only and usually it's a recent one.

00:12:15   So as far as I can tell, so like a lot of times people will have issues where like they'll

00:12:20   get an update into the store approved that like crashes on launch on the iPhone 5S or

00:12:25   something like that, right?

00:12:26   Because it's like an older device that is still supported and then it still runs on

00:12:29   that your customers might still use but it doesn't work for, you know, and app review

00:12:35   didn't catch it, you know.

00:12:36   And so there's, you can't depend on app review to catch like the big obvious crashers because

00:12:42   if it happens on anything, any conditions that they're not trying including any other

00:12:46   device besides whatever one they happen to be using, you're not gonna have a good time.

00:12:50   So that's very, very important to note.

00:12:52   - And it's also probably worth saying that app review is not QA.

00:12:56   Like app review is this very like course sieve that every now and then will catch problems

00:13:03   and I love it when they do this but like that is not at all what they're doing.

00:13:07   They're not really looking to test your app to find problems.

00:13:11   They're kind of running it through at a very different level.

00:13:14   And so I would not, it certainly isn't one of those like, well, I'll just see if app

00:13:18   review can find any bugs and if they don't, then I'm sure it's fine.

00:13:21   Like that is a terrible, that would be a terrible approach to take to with your app.

00:13:25   - Yeah, it's like the seatbelt on a plane.

00:13:27   It's like that is not the safety mechanism you should be relying on.

00:13:31   You should definitely like, basically if app review rejects your app for a crash, you have

00:13:37   a serious problem in your QA process.

00:13:39   Like whatever you submit to app review should not be in that kind of state where it would

00:13:42   crash in like the first four minutes of somebody using it once, you know.

00:13:47   But you know, overall like, and I should also suggest too that if you are going to rely

00:13:52   so heavily as we do on self-testing basically.

00:13:57   And again, I do, I apologize greatly to people who enjoy formal testing and automated testing

00:14:04   and unit testing and 100% test coverage.

00:14:06   I apologize because this entire discussion must be driving you totally crazy.

00:14:11   Hi Casey.

00:14:12   But ultimately it's hard for single person developers to really justify the additional

00:14:20   time and or expense that more rigorous testing methods bring on for things like this.

00:14:28   So now, that's a lot of qualifiers in that sentence.

00:14:31   Obviously if you have a larger staff, if you have a lot of money, if your app is more important

00:14:37   than like, you know, than like a sleep tracker for your wrist or for a podcast player.

00:14:45   Like if really bad things happen, if you like lose people's important data or you know,

00:14:51   somebody will be seriously put out or hurt if your app breaks, obviously that's a place

00:14:57   where you need a more formal testing procedure and framework in place.

00:15:01   But most people who make iOS apps aren't working in that kind of capacity.

00:15:06   And not only don't have those resources, but it would actually be kind of a bad way

00:15:10   to spend resources basically like tripling all of your work.

00:15:14   And this is a huge debate in programmer circles of whether test driven development is really

00:15:18   more work or whether it ends up saving you more work in the long run.

00:15:21   I honestly don't care which one of those it is.

00:15:24   I think it's probably unknowable because it's probably like the answer like everything

00:15:28   else is probably it depends.

00:15:31   But ultimately this isn't the way we do things and I think there's a lot of developers

00:15:35   out there who do things more like what we do.

00:15:37   Anyway, that being said, one more thing before I go into the sponsor for the week and that

00:15:41   is if you're relying so much on self-testing as we do, if your app has different modes

00:15:47   or settings or ways you can do things, change it up.

00:15:52   In your day to day usage, change it up every so often and spend a lot of time in each kind

00:15:57   of condition.

00:15:58   So for instance, Overcast has dark mode or light mode and during development I usually

00:16:03   switch about once a week between which one of those I'm using because I want to kind

00:16:07   of get used to it and live in it and make sure that I'm designing and making the best

00:16:11   app I can for both of those things.

00:16:12   Same thing with streaming versus downloading.

00:16:14   Like I wrote this whole streaming engine and I hardly ever use it and that's kind of

00:16:18   a problem.

00:16:19   So basically I force myself to change my own preference and spend a good amount of time

00:16:26   like a week or a month in streaming mode in the middle of development cycles where I would

00:16:31   otherwise be in downloading mode.

00:16:33   Just to make sure I'm getting a really good idea of these vastly different modes that

00:16:39   my app can be in so that my own testing is more effective.

00:16:42   Anyway, we're sponsored this week by Pingdom.

00:16:46   Pingdom can help remove the need for you to have somebody who automatically logs into

00:16:50   your website every minute and refreshes it to make sure it's up because Pingdom does

00:16:53   that automatically with their testing framework.

00:16:56   See for yourself at Pingdom.com/radar.

00:17:00   Pingdom is an awesome monitoring service for websites and web services and when you use

00:17:05   offer code radar at checkout you get a holiday special of 50% off your first invoice.

00:17:12   This is an awesome deal, usually 20% now it's 50% for the holidays.

00:17:15   So check it out at Pingdom.com/radar.

00:17:19   Pingdom offers powerful easy to use monitoring tools and services for your website and web

00:17:25   servers.

00:17:26   Websites are very sophisticated these days and you need to, you know, there's all sorts

00:17:30   of things that can go wrong and they know this, they see it because they monitor it

00:17:33   for lots of people.

00:17:34   You need Pingdom to be monitoring your site.

00:17:37   They can monitor as often as every minute by requesting pages, by running through checkouts,

00:17:41   by logging in, by doing, you know, transactions.

00:17:44   They can do all sorts of things, all simulated from their more than 70 global test servers

00:17:49   and they can do it as often as once a minute.

00:17:51   And if anything goes down they can alert you via so many different means, whatever you

00:17:55   want.

00:17:56   You can have email, text, push notification or any combination of these things and you

00:18:00   can set the granularity.

00:18:01   It's incredibly customizable.

00:18:03   I've been using Pingdom now for, oh geez, I think eight years, nine years, something

00:18:07   crazy, it's a very long time because I've used it way before they were a sponsor, way

00:18:11   before I was even a podcaster.

00:18:13   I was using Pingdom to monitor Tumblr back in the day and I just love them.

00:18:17   I use them constantly.

00:18:19   Highly recommended that you use Pingdom for all your monitoring needs.

00:18:23   It is really the best and if you run any kind of website or web service, the last thing

00:18:28   you want is for strangers on Twitter to be telling you when your site is down when you

00:18:32   didn't already know that.

00:18:33   You do not want to be learning that from Twitter.

00:18:36   You want to know when your site is down.

00:18:38   You want to be the first to know and then you want to be able to go in and fix it before

00:18:42   all the people on Twitter see it and start bugging you about it.

00:18:45   And with Pingdom you can because Pingdom will alert you the moment things go down with that

00:18:49   up to one minute granularity.

00:18:51   That's really incredible and they are so fast and so reliable.

00:18:54   I love Pingdom.

00:18:55   Check them out yourself.

00:18:57   Go to Pingdom.com/Radar.

00:18:58   You will get a 14-day free trial and when you enter offer code radar at checkout, you

00:19:03   will get a holiday special of 50% off your first invoice.

00:19:07   Check it out today and you'll be the first to know when your site is down.

00:19:10   Thank you very much to Pingdom for sponsoring our show.

00:19:13   So I think for the last part of the show it seemed like something that may actually make

00:19:16   sense for us to focus in on is a bit of the actual like the details of what this kind

00:19:23   of user testing approach that we've just been talking about looks like.

00:19:28   And for me what that usually means is so I get into this phase now where I'm doing this

00:19:32   final level of testing and I'm just constantly trying to think of things to try.

00:19:36   I'm trying them out.

00:19:37   I'm using the app in different circumstances in different places like I'll be using it

00:19:41   on different networks like hey I'm out on a cell network.

00:19:43   I'm at home.

00:19:45   And so whenever I find an issue, you know inevitably you'll find a bug.

00:19:47   You'll see something that you're like huh that's a little interesting.

00:19:51   What I found is a very important thing is to have some kind of reliable way of collecting

00:19:58   all these little issues and bugs and problems.

00:20:00   For me usually that is I have a list in OmniFocus that I am just adding items to.

00:20:08   And so anytime I see something usually I'll grab a screenshot of it and if it's obvious

00:20:13   I'll just attach the screenshot to the task in OmniFocus or if not I'll take it and run

00:20:19   it through what used to be an app that you made Marco which used to be called Bugshot

00:20:24   what I think is now called Pinpoint which is a great little tool.

00:20:27   We have a link in the show notes to it but it's just a little thing for just drawing

00:20:29   arrows basically or circling things.

00:20:32   It's great for screenshot sort of when you're like huh you know there's this weird graphics

00:20:36   glitch or something and so you can mark it up add it to a ticket.

00:20:39   And then you know I go through all of like when I'm in the phase now like this morning

00:20:45   I sat down I had the head of list about five or six little bugs I've noticed yesterday.

00:20:50   I went through and I'm just knocking them out and it's kind of like this I work all the

00:20:55   way down till I have no bugs and then I make a build and then I go through the process

00:20:59   again and you kind of just keep cycling through collecting as you go.

00:21:04   And for me that kind of this that kind of process of just keeping this running log seems

00:21:10   to work well.

00:21:11   It I like doing it that way rather than kind of having like sitting down and just testing

00:21:17   for hours never really works for me.

00:21:19   I find that I get kind of bored or I start doing the same patterns over and over again

00:21:24   and so it's better I think for me I try and do it spread it out throughout the day and

00:21:28   obviously if it's an application that you would use normally and naturally throughout

00:21:32   the day then this makes it easier.

00:21:35   But otherwise you just kind of have to remind yourself to do it and this is also a little

00:21:41   bit tricky if you have an app that requires a certain amount of time for testing.

00:21:46   So like for example I have a sleep tracker like in order for it to really like I cannot

00:21:51   compressed beyond just running it on multiple devices the testing of that because it takes

00:21:57   about eight hours to have eight hours of data collection and like I have simulated stuff

00:22:02   in the app for you know when I'm doing like debug testing that I can kind of like create

00:22:06   a night's data and run my algorithms on it.

00:22:10   But for actual testing there's nothing you can really do other than wearing multiple

00:22:13   watches which is what I do like when I'm doing extensive sleep testing like I'll have all

00:22:18   my watches on my wrist during the night and I'm running it three or four times.

00:22:22   But you know sort of in the same way I imagine for you with overcast like there's a certain

00:22:26   amount if you just have to play lots of audio because if you don't you're never going to

00:22:30   find all the weird audio issues or glitches or problems.

00:22:33   There's nothing that you can really do to speed that up or compress that testing.

00:22:39   Yeah basically I mean I like I just find ways to shove podcasts into more of my life so

00:22:46   like you know like I've reached and also to test out things like syncing you know like

00:22:50   my syncing engine took forever to really figure out I mean it took me almost a year to really

00:22:54   nail syncing and one of the ways I started finally getting better at that was I started

00:22:59   needing it more because I started using an iPad to play podcasts in the kitchen and I

00:23:04   use my iPhone to play them everywhere else.

00:23:06   So I will I was very frequently switching between multiple devices that would use the

00:23:10   sync backend and just by doing that myself by needing that myself for a few months I

00:23:16   really fixed a ton of sync problems and got it to a pretty good place now.

00:23:20   And so yeah I mean it really is just a matter of just you know again it's like if you

00:23:24   only make apps that you will use a lot this becomes a lot easier.

00:23:28   If you don't make apps that use a lot this you know you're gonna have to find other

00:23:33   ways to do this effectively but it really works out great for me and I will again once

00:23:38   again go in briefly about beta testing also and we pretty sure we did a whole episode

00:23:43   on it if not we will and we should but I think we already did.

00:23:47   We have so many episodes now I've kind of lost track but I do rely a lot on beta testers

00:23:53   to really do like a final check on the app.

00:23:57   They aren't my first testers.

00:23:59   I am my first tester and I will usually use changes to the app for weeks or even months

00:24:06   before I give it to anybody else to test just because I really want to make sure that I

00:24:09   know I have a feel for it, make sure it's good, make sure it's the right choice and

00:24:12   then I will finally share with testers like right before the public gets it.

00:24:16   But basically beta testers can help a little bit not a ton but a little bit.

00:24:22   I will also add that the what you mentioned earlier about your workflow about taking taking

00:24:28   screenshots and then adding arrows and then adding them to OmniFocus and things like that.

00:24:33   If you want to go off the deep end there are lots of crazy tools and frameworks that automate

00:24:37   that process to varying degrees and have all sorts of cool integration.

00:24:40   So there are some like I made bug shot kit which is basically shoved bug shot into an

00:24:45   app as you could do things like it will detect screenshots being taken and then automatically

00:24:51   bring up a modal controller with the screenshot saying here mark this up and send it in to

00:24:56   report a bug.

00:24:57   You can give that to all your testers and you can use that yourself during your test

00:25:00   process and so you can basically have like easy or automated bug reporting right embedded

00:25:05   into your app.

00:25:06   And there's again there's lots of ways to do this.

00:25:08   I'm not going to recommend any particular one because I haven't really looked but

00:25:10   there are many of these that exist now and you can you can really go off the deep end

00:25:15   with like the levels of integrations.

00:25:17   You can have it like post to slack and then make an entry in your bug tracker.

00:25:21   All sorts of crazy things you can do now if you really want to get into all that stuff.

00:25:25   So the options are pretty good for now for like tools and frameworks and things like

00:25:30   that to automate like the tedious stuff.

00:25:33   But for the real just using the app really just again just using it a lot yourself is

00:25:39   a very very effective way to do it.

00:25:41   Obviously as I know automated things exist sorry Casey again but just using your app

00:25:45   yourself will show you so much and will tell you so much and will show you things like

00:25:49   you know this this workflow is kind of clunky or this animation doesn't really look right

00:25:54   or this I really need this text to be lighter when it's really sunny outside or something

00:25:58   like that like there's a large category of either problems or or good and bad design

00:26:04   decisions for which just using it yourself is really the only way to know and really

00:26:09   the only way to test and that isn't true for everything obviously sorry again Casey but

00:26:13   that's true for quite a lot that we do in our apps.

00:26:15   Yeah and I think with other building I would say with beta testing that's an important

00:26:19   thing to it or another use for beta testing is the importance of not overly preparing

00:26:26   your beta testers for what you know for the app so in terms of trying to give in giving

00:26:32   them an experience that is similar to the experience that your app store users are going

00:26:36   to have and so when they download the app and they're like what do I do or this is really

00:26:42   confusing like some of the best feedback I get from beta testers is sort of accidental

00:26:47   where they're not actually they don't know it but they're telling me something is fundamentally

00:26:53   flawed or is really confusing where they're asking a question that like implies a fundamental

00:26:59   misunderstanding about something and that kind of feedback is always really good and

00:27:03   so I try whenever I'm beta testing to just almost like blindly send out the betas to

00:27:08   people where I'm not giving them too much preparation or like giving them step by step

00:27:12   walkthroughs or details it's very much like as though you just downloaded this from the

00:27:17   app store you know maybe you've seen a screenshot or two you've had a high level discussion

00:27:21   but you're working it out just as you would as a regular user and that's a great test

00:27:26   because as the developer of the app obviously I know how everything works everything is

00:27:32   intuitive to me because that's the way I made it but it is often not the case for your users

00:27:37   and so one of the best uses for beta testing isn't you know it's great when they find you

00:27:42   know actual bugs and issues and problems but it's also it's really useful for capturing

00:27:47   those kind of like it's like if you're asking this question you clearly I clearly have not

00:27:51   communicated the purpose of this feature correctly or I've hidden it in a weird way or you think

00:27:56   it should do something and it does the opposite and those are the only thing that is almost

00:28:01   exclusively possible to find in beta testing because no matter how hard you try to kind

00:28:08   of like reset your brain to what if I was a fresh user opening the application it's

00:28:13   kind of impossible where you know like the current app I'm working on I have been working

00:28:16   on it essentially every day for months at this point so I'm so in it that I just don't

00:28:24   see it's like I'm so far into the woods that I have no idea that there is a forest

00:28:27   like all I have is trees like it is deep down into it I mean and maybe when I come back

00:28:34   to it in a couple of you know like I'll put the app to the side for a little bit and

00:28:38   I'll come back to it in maybe a month maybe I will have that kind of fresh experience

00:28:43   but until that time the only way I can get that is with beta testing and so it is definitely

00:28:47   a valuable and important aspect of that.

00:28:49   All right and I guess we're going to see your new app pretty soon right?

00:28:53   Yeah it should be hopefully launching next week so it should be kind of exciting and

00:29:00   I expect to I'm sure we'll talk about it on the show how the launch went and kind of

00:29:03   some of the details of it as the app goes but it is very exciting to get to this point

00:29:08   and I will say as you know it's like starting all the other independent developers out there

00:29:12   it's like I just put in the work because there's definitely been times in this project

00:29:17   where I was like oh gosh what have I gotten myself into maybe I should just like can this

00:29:21   project and move on but there is hope eventually we'll get there eventually we'll ship and

00:29:25   it is delightful when you get there.

00:29:27   Can't tell you a secret to the audience?

00:29:29   You can.

00:29:30   I know what this is and I've used it and it's really good so look forward to it anyway

00:29:34   thanks a lot for listening everybody we're out of time this week and we'll talk to

00:29:37   you next week.

00:29:38   Bye.

00:29:38   Bye!

00:29:39   [ Silence ]