00:00:00 ◼ ► Welcome to Under the Radar, a show about independent iOS app development. I'm Marco Arment.
00:00:04 ◼ ► And I'm David Smith. Under the Radar is never longer than 30 minutes, so let's get started.
00:00:09 ◼ ► I'm pretty tired because last night I finally finished and shipped my most recent beta for
00:00:18 ◼ ► I hope so. I hope it's yay. You never really know until like maybe a day after it's been in the
00:00:25 ◼ ► store and then you realize, okay, nothing seems to have gone horribly wrong. It seems okay. Now I
00:00:34 ◼ ► Well, but at the very least, you've crossed the point where now you've said this is 4.0,
00:00:40 ◼ ► like this is what this version is. And so it's now it's just fixes from here rather than like,
00:00:47 ◼ ► you've crossed that point where you're like considering everything in the known universe
00:00:51 ◼ ► as a possibility. It's like now you've crossed over to like this is what it's going to be and
00:01:00 ◼ ► Yeah. And I mean, this is a bit of a weird release for me because it's not a big feature release.
00:01:06 ◼ ► Like the list of new features is really short. It's basically a giant bug fix and UI tweaking
00:01:14 ◼ ► release. I did want to talk this week about the process of beta testing that I use this time. This
00:01:21 ◼ ► is this is the first time. So in the past, I've done small scale betas where I invite like, you
00:01:27 ◼ ► know, 30 or 40 friends and trusted people and people like bloggers and stuff like that. And
00:01:32 ◼ ► I've had relatively minimal success with those over time. You know, when when you're inviting
00:01:39 ◼ ► people to a beta, the problem that I always that I've always had and I do is when people invite me
00:01:44 ◼ ► to their betas, I have the exact same thing, which is you get to the you get the app, you poke around
00:01:49 ◼ ► a whole lot with the very first build you get, and then you basically don't ever do it again. Or
00:01:54 ◼ ► like, you know, then it becomes like regular usage of the app for you. You're not really like
00:01:59 ◼ ► thoroughly testing and thoroughly giving feedback on anything past like the very first one that you
00:02:04 ◼ ► get in a lot of in a lot of cases. And so it's very hard for small betas to really provide
00:02:10 ◼ ► meaningful feedback in terms of bug reporting. They are very useful, usually in terms of like
00:02:16 ◼ ► design, critique UI critique, you know, figuring out like what parts of your app are confusing
00:02:21 ◼ ► people and which parts aren't pretty much any number of people will reveal the same set of
00:02:27 ◼ ► problems with that. It's like if you have a beta test of 40 people and a beta test or beta test of
00:02:31 ◼ ► 4000 people, like they're going to have roughly the same output of like this parts of this part
00:02:37 ◼ ► of the app is confusing to us or this part doesn't really work for us. You'll just have more reports
00:02:41 ◼ ► of the same things if you have a bigger, a bigger group. So for that it's good, but for actually
00:02:46 ◼ ► finding bugs, it's not that useful to have a small group because they won't run into as many as a
00:02:53 ◼ ► bigger group will. So and because of that kind of drop off effect where people install the very
00:02:59 ◼ ► first version and then not much after that, it becomes hard to even keep the group going. In
00:03:04 ◼ ► TestFlight I have this group called like friends that you know of that group, TestFlight tells you
00:03:10 ◼ ► who installs each version and I can, you can just see like every time you send a new build, like it
00:03:16 ◼ ► gets fewer installations than the one before it. So if you're sending build to the same group of
00:03:20 ◼ ► people over and over again, you know, eventually you're not getting a meaningful number of people
00:03:25 ◼ ► installing it to really be worth the hassle at all. So one of the approaches I did in the past
00:03:31 ◼ ► was I basically set up a basic little form on the Overcast site and I announced on Twitter like,
00:03:40 ◼ ► this all became possible with TestFlight being part of Apple and having its limits raised. The
00:03:45 ◼ ► initial, you know, so back when it was like UDID provisioning based manual stuff, which was
00:03:51 ◼ ► horrible, that was limited to a hundred, right? And it was under devices, not even people.
00:03:55 ◼ ► Yeah. And the devices were tied to your developer account and you couldn't reset them. And so it was
00:04:00 ◼ ► a big pain. It was, you really couldn't have an open call because you would be tying, like locking
00:04:06 ◼ ► into someone, you know, for an entire calendar year, I think it was. Yeah. And then like, and
00:04:12 ◼ ► even, even if you finally got like your 30 friends and they were tested, then as soon as everyone got
00:04:17 ◼ ► new phones or new iPads or whatever, like it ruined it and you had to go reset it. It was,
00:04:21 ◼ ► you had like six devices from that, like one person who reviews phones and it's like, which
00:04:26 ◼ ► one of these are you currently using and taking up all these slots and yeah. So that was a pain.
00:04:30 ◼ ► So then when Apple bought it, then they, they, the initial like raise of the limit, I believe,
00:04:35 ◼ ► raised to a thousand, right? Yes, I believe so. It got a couple more raises. And then most recently,
00:04:41 ◼ ► I think this past summer they raised it to 20,000. So that's now a lot of people now, you know, if
00:04:47 ◼ ► you can, and now they've, there's much better management of groups in test flight. So you can,
00:04:52 ◼ ► you can have like a, like, you know, close friends group and then you can have like a public group
00:04:57 ◼ ► where you can have like send and you can send different builds to different groups. So you can
00:05:01 ◼ ► have, you can kind of stage it out where like you can try things first with a smaller group and then
00:05:06 ◼ ► stage it out to a bigger group. You can have a separate group for like people in the press,
00:05:09 ◼ ► stuff like that. So there's great flexibility now in test flight. And so this time I wanted to take
00:05:15 ◼ ► even more advantage of it than I, than I have before. I wanted to get way more people in there,
00:05:20 ◼ ► first of all. So before the group would be like 500 people. Now, if you send out 500 invitations,
00:05:26 ◼ ► you're going to get about 300 people who actually install and use the build. And then, you know,
00:05:30 ◼ ► and then of those 300 people, you're going to get maybe 20 people who actually ever email you
00:05:34 ◼ ► anything. So like you have to start like running the numbers here. It's like, okay, this is to get
00:05:39 ◼ ► a meaningful amount of feedback. You need a lot of people and you also need to make feedback easy
00:05:43 ◼ ► and there needs to be like a good place for it. The other issue I've had in, in the past,
00:05:48 ◼ ► when I've done those larger tests is I always get people asking me like, what's the official forum
00:05:53 ◼ ► for discussing this? Like, should I guess to be emailing you through the test flight response
00:05:57 ◼ ► emails, which are okay. Is that everyone always asks for like, is there like a forum or a bug
00:06:04 ◼ ► tracker or a Slack group or a chat room or like anything like that, any place to go to discuss
00:06:10 ◼ ► this beta. There's always a problem. And I didn't really have a way to communicate with the testers
00:06:16 ◼ ► except just emailing everybody, but nobody wants that and I don't want to do it. No one wants to
00:06:21 ◼ ► receive that. So, so I didn't have that either. So this, for this beta, I decided to go all out
00:06:26 ◼ ► and try something even crazier. Instead of, instead of, you know, announcing, or instead of
00:06:31 ◼ ► inviting 500 people, this time I invited 1500 people and I set up a Slack group for the very
00:06:37 ◼ ► first time for people to join the Overcast Slack and discuss the beta and report bugs and everything.
00:06:44 ◼ ► And this is basically what I want to talk about today is the experience of doing this, lessons
00:06:50 ◼ ► I've learned, things I'm definitely going to keep doing, things I'm definitely not going to do in
00:07:00 ◼ ► >> No, only informally, like in terms of I have had conversations on Slack about my apps with
00:07:06 ◼ ► individuals, but never on a, on the way that you're doing it where it's like, you're creating
00:07:11 ◼ ► essentially a, this, you know, a, a semi-public forum where, you know, hundreds of people can talk
00:07:19 ◼ ► about your app and the purpose of the room is to talk about the app. Like I've never done
00:07:24 ◼ ► anything like that. Like it's, it's always been way too intimidating of a thing to manage and to
00:07:29 ◼ ► feel like I would be able to reasonably engage with, I think is the thing that always made me
00:07:38 ◼ ► I want to, would want to be able to feel present in there so that it's not like they're just
00:07:43 ◼ ► sort of shouting to each other in the void. But yeah, so I've never gone, I've never gone down
00:07:52 ◼ ► All right. So first we are sponsored this week by Blue Apron, the number one recipe delivery
00:07:58 ◼ ► service with the freshest ingredients. Get $30 off your delivery and free shipping by going to
00:08:03 ◼ ► blueapron.com/radar today. Blue Apron makes incredible home cooking accessible to everyone.
00:08:09 ◼ ► And they support a more sustainable food system. They set the highest standards for their
00:08:13 ◼ ► ingredients and they're building a community of home chefs for less than $10 per meal. Blue
00:08:18 ◼ ► Apron delivers seasonal recipes with fresh high quality ingredients to make delicious home cooked
00:08:23 ◼ ► meals in 40 minutes or less. Every meal comes with a step-by-step easy to follow recipe card
00:08:29 ◼ ► and ingredients that are portioned out to exactly the amount you need. So they're reducing food
00:08:33 ◼ ► waste and the Blue Apron freshness guarantee promises that every ingredient makes it safe,
00:08:37 ◼ ► sound and ready to cook, or they will make it right. I've been using Blue Apron myself,
00:08:42 ◼ ► my wife for maybe two years now. It's been a long time since before they were a sponsor of any of
00:08:47 ◼ ► our shows. Absolutely love it. We stick with it. We pay for it too, just like everybody else. And
00:08:52 ◼ ► we stick with it. It's wonderful. We really are better chefs now. We've tried so many new foods,
00:08:59 ◼ ► not even by having to go to restaurants, but by cooking them ourselves. We've eaten things we've
00:09:03 ◼ ► never eaten before. And it's just wonderful that their hit rate is very high. We've even tried
00:09:08 ◼ ► other services that are similar and we have found them to be by far our favorite. They are the ones
00:09:13 ◼ ► we always go back to and it's just the best. So you can choose from a variety of new recipes each
00:09:17 ◼ ► week, or you can let Blue Apron's culinary team surprise you. There is no weekly commitments. You
00:09:22 ◼ ► only get deliveries when you want them. So if you're going to be super busy or out of town for
00:09:26 ◼ ► a week, you can just go on their app or their site and just say, all right, skip this delivery
00:09:29 ◼ ► and this delivery. And that's it. You don't get charged for them. They don't give you a hassle.
00:09:32 ◼ ► You don't have to call anybody. It's great. So try Blue Apron today. Go to blueapron.com/radar.
00:09:38 ◼ ► You get $30 off your first delivery and free shipping. You will love how great it feels and
00:09:52 ◼ ► a better way to cook. So I decided to do my big beta group and my Slack group as feedback
00:10:00 ◼ ► for the beta for the first time this time. So getting the big beta group was easy. I posted
00:10:06 ◼ ► on Twitter. I said, here's a web form. And it's just a little thing, as I said. It makes it easy
00:10:12 ◼ ► for me to dump out a CSV. And then I can import it right into TestFlight. So there's no real manual
00:10:18 ◼ ► work here involved in the testers. And it doesn't really matter how many you have. It's the same
00:10:22 ◼ ► amount of work. There's also, as more people get added to the list, you don't have to keep track of
00:10:28 ◼ ► who's been added or who hasn't. You can just export the entire list. And TestFlight is smart enough
00:10:32 ◼ ► not to import a bunch of duplicates. So you can just make a new version of the CSV that has 51
00:10:38 ◼ ► people instead of 50 people, import that, and it'll say, we imported one new tester. And if you want,
00:10:44 ◼ ► you can download a list of the 50 that we already had. It's quite clever, quite full featured.
00:10:49 ◼ ► So TestFlight is really great so far. It's been wonderful. They really have done quite a lot with
00:10:54 ◼ ► it. And I'm very happy with it. Slack, the idea of creating a Slack that had kind of a public
00:11:00 ◼ ► invitation used to be hard. It used to require that people make little web apps that would use
00:11:06 ◼ ► their API to generate one-off invitation links to every person who went there and tried to sign up.
00:11:12 ◼ ► This is no longer the case. Slack, you can now, and I should clarify, I only have a free
00:11:18 ◼ ► Slack account. I'm not paying for this. Their paid tier has lots of cool things for more pro use.
00:11:24 ◼ ► But for this kind of thing, you don't really need any of them. And it would actually be,
00:11:28 ◼ ► I think, quite cost prohibitive if you wanted to have 1,500 people in a Slack room with you.
00:11:33 ◼ ► - Yeah, exactly. That would be probably not on an indie budget. But anyway, so now Slack just gives
00:11:40 ◼ ► you these invite links that just last 30 days. I think maybe in 60 days. They last a long time,
00:11:45 ◼ ► so you can just have one and hardly ever change it. And they just invite anybody who knows the
00:11:49 ◼ ► link. And if you need to reset it, if it gets in the wrong hands, you can, but you probably don't
00:11:52 ◼ ► care. So it's very, very easy to have a bulk invite. So I just put that bulk invite in the
00:11:58 ◼ ► TestFlight emails and said, "Here, this is the official discussion channel for this release,
00:12:04 ◼ ► or you can just email through TestFlight." I gave people both options, and they used both options.
00:12:08 ◼ ► But by far, the most people used Slack. It was by far the most common feedback channel. Only a
00:12:15 ◼ ► handful of people emailed, as usual. But Slack got, I invited about 1,500 people. The Slack group
00:12:29 ◼ ► people who were invited join a Slack channel, I was not expecting that. Because a lot of people,
00:12:34 ◼ ► if you use Slack at all, you probably have too many Slacks. That's kind of like the problem.
00:12:40 ◼ ► It's like everyone has a Slack for something now. So you have people who don't use it and don't want
00:12:44 ◼ ► to use it, and that's fine. But people who do use it, they have too many. So I thought it would be
00:12:48 ◼ ► hard to get people to sign up just for this. And I expected the ratio of people who did sign up to
00:12:54 ◼ ► be lower than two-thirds. So that's pretty awesome. And this, again, this probably has a lot to do
00:13:00 ◼ ► with your audience. I mean, my audience is pretty nerdy, so that probably helped me a lot here.
00:13:10 ◼ ► when sending out that first batch of invitations is that, A, I should not send 1,500 invitations
00:13:17 ◼ ► at the same time. Because what that meant was I had most of these 900 people joining like that
00:13:26 ◼ ► first night, and the Slack channel went crazy. Like, it went from zero to 100 miles an hour
00:13:33 ◼ ► in two seconds, and even faster than your Tesla. And it was just like massive floods of information.
00:13:41 ◼ ► It very quickly became apparent that I needed some kind of slightly more structured way of reporting
00:13:48 ◼ ► bugs. I created a bugs room or channel in the Slack, and I had a little header thing in general
00:13:56 ◼ ► saying, "Please report bugs there," and general is for everything else. And that worked sort of
00:14:02 ◼ ► like, you know, about two-thirds of the bugs went there, about a third still came to general.
00:14:05 ◼ ► So, you know, you had this massive flood of chat going on. And it became very, very time consuming
00:14:13 ◼ ► for me to even just try to keep up with the slide, even just to read what was being said,
00:14:18 ◼ ► let alone to actually respond to anything or to have time to actually record these bugs down in
00:14:26 ◼ ► any kind of system or tracker or to-do list or anything. It became quite overwhelming at first.
00:14:32 ◼ ► The good thing is that that massive wave of activity was mostly just because everybody had
00:14:39 ◼ ► just joined, and were using the very first beta build at the same time. And so that has now calmed
00:14:44 ◼ ► down, and now it is, even though there's almost a thousand people in there, there's almost no chatter,
00:14:50 ◼ ► you know, now, because it's now just like a more of a regular discussion group. So it's not going
00:14:57 ◼ ► to be, and it's not like general discussion of like, you know, news and politics, it's just like
00:15:01 ◼ ► about this app. So I'm going to keep this group going, by the way. Now, you know, the beta is now
00:15:05 ◼ ► kind of over, but there's going to be more betas, and I'm just going to keep this group open and just,
00:15:09 ◼ ► you know, still monitor it, because now it's just like the overcast Slack. And if anybody wants to
00:15:14 ◼ ► join, I'll put the link in the show notes. So anybody, you don't have to be on the beta to join
00:15:16 ◼ ► the room. I don't care who joins. Please just please be nice, and then that's about it. So
00:15:21 ◼ ► anyway, the biggest thing that became apparent to me during this, so besides the fact that you
00:15:28 ◼ ► shouldn't send all your invitations all at once to avoid that massive flood, is that Slack is not
00:15:33 ◼ ► a bug tracker. And I'm sure every bug tracker has some kind of integration or app that I can use
00:15:41 ◼ ► with Slack, and I haven't tried any of them. I apologize. I honestly have not had time.
00:15:50 ◼ ► companies, I apologize. I'm sure you have good solutions. I would love to hear about them after
00:15:54 ◼ ► the iPhone 10 ships, but for now I don't have time. But Slack is really a terrible bug tracker
00:16:03 ◼ ► if you try to basically just use it itself as one, because, you know, A, there's no like
00:16:09 ◼ ► sorting or voting or easy way for people to check if there's been duplicates of the same thing.
00:16:16 ◼ ► So number one problem is tons of duplicate bug reports of bugs that were already reported like
00:16:22 ◼ ► 10 chat lines up from the person reporting it. But I don't expect every person to come in there
00:16:27 ◼ ► and read the entire history of Slack first before they report their bug. Like I don't want them to
00:16:31 ◼ ► have to do that, because then they won't report their bugs, and that's bad for me. So you can't
00:16:35 ◼ ► really expect people to do that. So there has to be some better way that I have to devise to
00:16:41 ◼ ► let people submit bug reports for people to be able to see what is currently being worked on
00:16:46 ◼ ► and what is currently already known. And I know, I mean, everybody who has a bug report system knows
00:16:52 ◼ ► that people report duplicates anyway, but I think I'd get a lot fewer of them, and that would reduce
00:16:56 ◼ ► the workload at least. Slack also doesn't have basic things like, suppose somebody reports a
00:17:03 ◼ ► really good bug and I want to get to it, but there's now 100 items below it that are newer,
00:17:08 ◼ ► and the next time I go to that page, it's gone. It's like, you know, it's like, it's, you know,
00:17:13 ◼ ► buried under newness. There's a few mechanisms like they had, like Slackbot has a remind feature.
00:17:21 ◼ ► You can like, you can say like, "Remind me of this message tomorrow or in an hour." It's a pretty
00:17:26 ◼ ► coarse system. It's not very well featured, and it just shows up as like DMs from Slackbot,
00:17:37 ◼ ► okay, I want to hit this, but I'm like seeing this on my iPad right now, right before I go to bed or
00:17:42 ◼ ► something. It's like, I don't, I can't deal with this right now. Slack also does not have a mark
00:17:47 ◼ ► as unread for DMs or anything. So you can't like, if somebody DMs me and says, "Hey, I'm not on the
00:17:54 ◼ ► beta. Can you please add this email address?" If I see that DM, that mark should have read.
00:18:09 ◼ ► I'll never remember. I'll never find it. So it's really not a good bug tracking system with the,
00:18:18 ◼ ► but the lack of an actual bug tracker and the lack of any kind of mark as unread to kind of hack it
00:18:26 ◼ ► tracking in Slack, but it's not very good at it. Where it is really good at it is the ability to
00:18:32 ◼ ► freely upload media and for other people to like thumbs up and emoji response things and stuff like
00:18:37 ◼ ► that. And so for me, a total game changer in this beta test that had not happened before is people
00:18:46 ◼ ► were able to record screen captures with the iOS 11 built-in screen recording ability. And so, so
00:18:53 ◼ ► many of the bug reports, I'd say most of the bug reports, people were also posting videos of them
00:18:59 ◼ ► recorded with iOS screen capture right from the phone. That is awesome. And that helped so much.
00:19:04 ◼ ► You know, a video is worth 10,000 words, right? As the saying goes. So that was a huge game changer.
00:19:11 ◼ ► And the, the built-in test flight emailing feedback thing, I don't think makes that easy,
00:19:24 ◼ ► Otherwise it was, it was really nice besides the bug reporting duplicate, Mark has read,
00:19:30 ◼ ► remind me, mediocrity. It was really nice to have a discussion group about the build because then
00:19:37 ◼ ► like I was able to have to like ask the, ask the room like, Hey, what if I do this? And I had like
00:19:43 ◼ ► a little like yes and no button as reactions and people would like vote on it. There's probably a
00:19:47 ◼ ► poll functionality now. I'm not, honestly, I'm not a Slack expert, so I probably could have done even
00:19:52 ◼ ► more with this. Stuff like that. And, and, and also because it was a chat, when people would,
00:19:58 ◼ ► sometimes people would report a bug and I wasn't around, then other people would tell them, Oh,
00:20:04 ◼ ► that's a bug. It's being fixed in the next build. Or, Oh, that's not a bug. Here's why. Or here's
00:20:08 ◼ ► like that's intended. Like other people were helping each other. And so that also, you know,
00:20:13 ◼ ► you get some of that with like a form and stuff, but I think it's easier with Slack. So ultimately
00:20:19 ◼ ► I do need a real bug tracker, but otherwise, Oh, I need a real bug tracker. I don't need to
00:20:25 ◼ ► send all the invites all at once, but otherwise it was a huge success. I got way more bugs reported
00:20:31 ◼ ► than I ever had before. I got tons of bugs reported that weren't even bugs in this beta,
00:20:35 ◼ ► but were bugs from earlier versions that like, you know, that just happened to still not be fixed.
00:20:48 ◼ ► but I will definitely do future betas in this exact same way. I think the only way it would
00:20:54 ◼ ► really fall down with doing like a large scale, mostly public beta like this is for secrecy.
00:21:00 ◼ ► If you wanted to keep things private, if you didn't want people posting screenshots or revealing
00:21:04 ◼ ► what your features were, then you have a problem and you have to make different arrangements then.
00:21:09 ◼ ► But I didn't, I decided with this release to kind of develop it in the open. And that alone is
00:21:14 ◼ ► possibly worth discussing. But the short version is like, I was tired of keeping it secret and this
00:21:19 ◼ ► version had a lot of like UI fan service almost like doing long standing requests in the UI,
00:21:26 ◼ ► basic stuff like one tap play and stuff like that. So I kind of felt secrecy wasn't important for
00:21:31 ◼ ► this. And so I just did it all in the public and it was fine. So yeah, ultimately success.
00:21:36 ◼ ► >> The biggest things that come to mind for me with this is, at first, do you have a concern about the,
00:21:44 ◼ ► like, I guess, the skew or the bias of the group that you have to, as whether it's representative
00:21:57 ◼ ► of your user base at large? Because one thing that comes to mind with this is like, it's incredibly
00:22:03 ◼ ► specific to the, you know, someone saw a link on Twitter who follows you or overcast, went to it,
00:22:11 ◼ ► signed up for it, and then knew what slack was, went through the process of creating a slack and
00:22:16 ◼ ► our slack account and like, gone into it like there's a there's a fairly high barrier to get
00:22:21 ◼ ► in there, which is good in some ways in the sense that these users are clearly invested and
00:22:27 ◼ ► interested and passionate and excited about your app. But the only thing that I kind of wonder
00:22:32 ◼ ► about is it's is it, you know, your user base is much broader from a experience or technical
00:22:38 ◼ ► savviness or whatever perspective. And so it's like, I would be nervous for myself of, you know,
00:22:44 ◼ ► having a having a place where I'm getting lots of reinforcement for choices or decisions that
00:22:50 ◼ ► may be biased in a particular direction. So I was curious if that's something that you're worried
00:22:55 ◼ ► about? >> It absolutely is. But I think that's unfortunately inherent in pretty much any beta
00:23:01 ◼ ► test that almost anybody would do, you know, because the number of people who are going to
00:23:05 ◼ ► be willing to join a slack probably has a lot of overlap with the people who are willing to join
00:23:09 ◼ ► a beta. And so, you know, that is one of the reasons why my my join rate was so high as a
00:23:15 ◼ ► percentage of people. But, you know, ultimately, I still have now, you know, like 800 to 1500
00:23:23 ◼ ► installations of this build running in the wild, getting crash reports that collect with the built
00:23:28 ◼ ► in crash reporter, having people experience this in 800 different ways like that's wonderful.
00:23:35 ◼ ► And I've, you know, so even though it's not 800 of the exact same kind of per it isn't like
00:23:41 ◼ ► a random representative sample of the user base, it's such a big sample of the user base that I
00:23:48 ◼ ► think it still has a lot of value. And ultimately, you know, I don't think I have a better alternative
00:23:55 ◼ ► for beta testing than doing something like this. Like I'm not going to capture a real random sample
00:24:00 ◼ ► through beta through beta testing, because not everyone wants to install a beta or even knows
00:24:03 ◼ ► how or has any interest in doing it, and then they won't even give feedback maybe. So like,
00:24:07 ◼ ► it's not there's no way to get a true random sample. So I think what I have now is pretty good.
00:24:11 ◼ ► Yeah. And I think it's the thing that would worry me less is the like, the beta testing is just
00:24:16 ◼ ► useful in so far as the having a good volume of things just to catch issues that you wouldn't see
00:24:21 ◼ ► otherwise. But yeah, it's the I would be worried about it changing my perspective of the app and
00:24:27 ◼ ► pushing the app in a direction that like I have to consciously I know for myself choose to try and
00:24:33 ◼ ► put myself in the mindset of like, whatever my I imagine my typical user is. And so like having
00:24:38 ◼ ► it'd be easy to get to to have something like this where you get this sense of like, well,
00:24:42 ◼ ► everyone loves it. Everyone loves it in the slack. I could see that being a tricky thing. Something
00:24:47 ◼ ► else I was curious is, how do you see this interacting with so like the way you do like
00:24:53 ◼ ► your general support. So I imagine you you know, you do there's an email address that people can
00:24:57 ◼ ► email. And within the app with a lot of caveats, you have a way for people to contact you and
00:25:02 ◼ ► report stuff there. Is that something that you expect to continue doing? Or is this sort of your
00:25:10 ◼ ► the direction that you're going to try and push that into that if you're like the feedback that
00:25:14 ◼ ► you receive for the app will come through this group, rather than you know, sort of, because I
00:25:20 ◼ ► would, it would be awkward, I imagine a certain point to be maintaining two totally separate
00:25:24 ◼ ► sort of inboxes that you have your email feedback and support, and then you have your slack email
00:25:31 ◼ ► support. Now you have these two places that you now have to do stuff between and manage and read.
00:25:38 ◼ ► And like you're in you could sort of add or the risky thing would be at a certain point,
00:25:48 ◼ ► you know, when you when you create one of these groups, you know, you are adding a new thing that
00:25:52 ◼ ► you have to check that you have to respond to. And so that, you know, normally, I'd be very
00:25:58 ◼ ► averse to that. The difference here is that I already use slack a lot. I'm already checking
00:26:02 ◼ ► it throughout the day for other in other slacks. And so like, it's not a big deal for me to add one
00:26:05 ◼ ► more thing here. And, and because it is, you know, semi public, it does have that effect of other
00:26:12 ◼ ► people will help each other and they have been so that that helps tremendously compared to email.
00:26:25 ◼ ► Sure. Yeah. And then we guess the last thing too, is I always wonder about is, it's like, I,
00:26:31 ◼ ► I always worry about from a sense of create, whenever you create a community, and you're sort
00:26:36 ◼ ► of the moderator or the steward of it, it's like, there's that feeling of responsibility for making
00:26:42 ◼ ► sure that everyone's being nice, and everyone's treating each other appropriately. And that side
00:26:47 ◼ ► of things would, you know, like, I'm just thinking of the way that I get, will get email, you know,
00:26:52 ◼ ► customer support email that can be very negative and can be very harsh or problematic. And it would
00:26:59 ◼ ► be something that would be in some ways, I'm glad that that's not a public thing. And then also,
00:27:04 ◼ ► it's, you know, it's something that I've had to put people, you know, someone else reads my first
00:27:08 ◼ ► line of customer support to avoid that having an impact on me personally. And so this feels even
00:27:15 ◼ ► more direct and even more connected in that way. So that's another thing that sort of comes to mind
00:27:24 ◼ ► I did think about that. And I'm, you know, long term, that might become a problem eventually,
00:27:28 ◼ ► but probably not. I mean, the because like, this is a very topic focused Slack. And it's,
00:27:34 ◼ ► you know, it's specifically about discussing this app and feedback about this app and bug reports
00:27:39 ◼ ► about this app. I think as long as I keep it relatively tightly focused to that, I'm not that
00:27:45 ◼ ► concerned about, you know, basically community problems, because running a community, you know,
00:27:49 ◼ ► your concern is very, very good and very correct. Like running a more general purpose community is
00:27:55 ◼ ► really hard. And there's so many problems with just humans, the way we talk to each other. It's
00:28:01 ◼ ► a big problem. And it's a huge challenge that I am not at all set up to do and have no interest in
00:28:06 ◼ ► doing. But this is such a small and focused group. I don't think it will be a problem. And if it does
00:28:14 ◼ ► Sure. Yeah, because I mean, the reality is, like, at the end of the day, like the proof is in the
00:28:18 ◼ ► pudding. Like I've been using the beta since, you know, the beginning of this process and have
00:28:23 ◼ ► watched it get better and better and better. And I think, you know, it's like the reality is it
00:28:28 ◼ ► work like it is working. And so even if there are sort of potential pitfalls or problems that may
00:28:34 ◼ ► come down the road, like this is certainly proved to be a very effective way to get a lot of useful
00:28:39 ◼ ► feedback that has made your app better over time. And so like, and from that perspective, it's just
00:28:44 ◼ ► like an absolutely clear win of like, yeah, this is certainly has worked well and is an interesting
00:28:49 ◼ ► case study, I think for other developers to consider. I mean, I think for myself, I am still
00:28:55 ◼ ► too intimidated by it. I'll probably just stick to email and, you know, have someone do my tier
00:28:59 ◼ ► one support and kind of filter it through like that works for me. But I can totally see where
00:29:03 ◼ ► this could be. It's like it is such a bigger potential upside, just at the expense of a few
00:29:10 ◼ ► potential downsides. But overall, like it's working. And I, you know, I'm loving the bill
00:29:17 ◼ ► Yeah. I mean, first of all, never underestimate how much I hate email. And, you know, you're
00:29:22 ◼ ► right though. I mean, this, this, the process worked in the sense that I got better and more
00:29:25 ◼ ► feedback than ever. And as a result, this is, in my opinion, to the best of my knowledge,
00:29:32 ◼ ► asterisk, asterisk, the least buggy and best version that I've ever shipped. And, you know,