Under the Radar

104: Public Beta Testing with Slack


00:00:00   Welcome to Under the Radar, a show about independent iOS app development. I'm Marco Arment.

00:00:04   And I'm David Smith. Under the Radar is never longer than 30 minutes, so let's get started.

00:00:09   I'm pretty tired because last night I finally finished and shipped my most recent beta for

00:00:15   Overcast 4.0 and it's going to the App Store now.

00:00:18   Yay!

00:00:18   I hope so. I hope it's yay. You never really know until like maybe a day after it's been in the

00:00:25   store and then you realize, okay, nothing seems to have gone horribly wrong. It seems okay. Now I

00:00:30   can finally like breathe, you know, breathe and like relax a little bit.

00:00:34   Well, but at the very least, you've crossed the point where now you've said this is 4.0,

00:00:40   like this is what this version is. And so it's now it's just fixes from here rather than like,

00:00:47   you've crossed that point where you're like considering everything in the known universe

00:00:51   as a possibility. It's like now you've crossed over to like this is what it's going to be and

00:00:55   now you can just fix and tweak from here on out. So it's an accomplishment either way.

00:01:00   Yeah. And I mean, this is a bit of a weird release for me because it's not a big feature release.

00:01:06   Like the list of new features is really short. It's basically a giant bug fix and UI tweaking

00:01:14   release. I did want to talk this week about the process of beta testing that I use this time. This

00:01:21   is this is the first time. So in the past, I've done small scale betas where I invite like, you

00:01:27   know, 30 or 40 friends and trusted people and people like bloggers and stuff like that. And

00:01:32   I've had relatively minimal success with those over time. You know, when when you're inviting

00:01:39   people to a beta, the problem that I always that I've always had and I do is when people invite me

00:01:44   to their betas, I have the exact same thing, which is you get to the you get the app, you poke around

00:01:49   a whole lot with the very first build you get, and then you basically don't ever do it again. Or

00:01:54   like, you know, then it becomes like regular usage of the app for you. You're not really like

00:01:59   thoroughly testing and thoroughly giving feedback on anything past like the very first one that you

00:02:04   get in a lot of in a lot of cases. And so it's very hard for small betas to really provide

00:02:10   meaningful feedback in terms of bug reporting. They are very useful, usually in terms of like

00:02:16   design, critique UI critique, you know, figuring out like what parts of your app are confusing

00:02:21   people and which parts aren't pretty much any number of people will reveal the same set of

00:02:27   problems with that. It's like if you have a beta test of 40 people and a beta test or beta test of

00:02:31   4000 people, like they're going to have roughly the same output of like this parts of this part

00:02:37   of the app is confusing to us or this part doesn't really work for us. You'll just have more reports

00:02:41   of the same things if you have a bigger, a bigger group. So for that it's good, but for actually

00:02:46   finding bugs, it's not that useful to have a small group because they won't run into as many as a

00:02:53   bigger group will. So and because of that kind of drop off effect where people install the very

00:02:59   first version and then not much after that, it becomes hard to even keep the group going. In

00:03:04   TestFlight I have this group called like friends that you know of that group, TestFlight tells you

00:03:10   who installs each version and I can, you can just see like every time you send a new build, like it

00:03:16   gets fewer installations than the one before it. So if you're sending build to the same group of

00:03:20   people over and over again, you know, eventually you're not getting a meaningful number of people

00:03:25   installing it to really be worth the hassle at all. So one of the approaches I did in the past

00:03:31   was I basically set up a basic little form on the Overcast site and I announced on Twitter like,

00:03:36   hey, sign up here and the first, you know, 500 people allowed to TestFlight and this,

00:03:40   this all became possible with TestFlight being part of Apple and having its limits raised. The

00:03:45   initial, you know, so back when it was like UDID provisioning based manual stuff, which was

00:03:51   horrible, that was limited to a hundred, right? And it was under devices, not even people.

00:03:55   Yeah. And the devices were tied to your developer account and you couldn't reset them. And so it was

00:04:00   a big pain. It was, you really couldn't have an open call because you would be tying, like locking

00:04:06   into someone, you know, for an entire calendar year, I think it was. Yeah. And then like, and

00:04:12   even, even if you finally got like your 30 friends and they were tested, then as soon as everyone got

00:04:17   new phones or new iPads or whatever, like it ruined it and you had to go reset it. It was,

00:04:21   you had like six devices from that, like one person who reviews phones and it's like, which

00:04:26   one of these are you currently using and taking up all these slots and yeah. So that was a pain.

00:04:30   So then when Apple bought it, then they, they, the initial like raise of the limit, I believe,

00:04:35   raised to a thousand, right? Yes, I believe so. It got a couple more raises. And then most recently,

00:04:41   I think this past summer they raised it to 20,000. So that's now a lot of people now, you know, if

00:04:47   you can, and now they've, there's much better management of groups in test flight. So you can,

00:04:52   you can have like a, like, you know, close friends group and then you can have like a public group

00:04:57   where you can have like send and you can send different builds to different groups. So you can

00:05:01   have, you can kind of stage it out where like you can try things first with a smaller group and then

00:05:06   stage it out to a bigger group. You can have a separate group for like people in the press,

00:05:09   stuff like that. So there's great flexibility now in test flight. And so this time I wanted to take

00:05:15   even more advantage of it than I, than I have before. I wanted to get way more people in there,

00:05:20   first of all. So before the group would be like 500 people. Now, if you send out 500 invitations,

00:05:26   you're going to get about 300 people who actually install and use the build. And then, you know,

00:05:30   and then of those 300 people, you're going to get maybe 20 people who actually ever email you

00:05:34   anything. So like you have to start like running the numbers here. It's like, okay, this is to get

00:05:39   a meaningful amount of feedback. You need a lot of people and you also need to make feedback easy

00:05:43   and there needs to be like a good place for it. The other issue I've had in, in the past,

00:05:48   when I've done those larger tests is I always get people asking me like, what's the official forum

00:05:53   for discussing this? Like, should I guess to be emailing you through the test flight response

00:05:57   emails, which are okay. Is that everyone always asks for like, is there like a forum or a bug

00:06:04   tracker or a Slack group or a chat room or like anything like that, any place to go to discuss

00:06:10   this beta. There's always a problem. And I didn't really have a way to communicate with the testers

00:06:16   except just emailing everybody, but nobody wants that and I don't want to do it. No one wants to

00:06:21   receive that. So, so I didn't have that either. So this, for this beta, I decided to go all out

00:06:26   and try something even crazier. Instead of, instead of, you know, announcing, or instead of

00:06:31   inviting 500 people, this time I invited 1500 people and I set up a Slack group for the very

00:06:37   first time for people to join the Overcast Slack and discuss the beta and report bugs and everything.

00:06:44   And this is basically what I want to talk about today is the experience of doing this, lessons

00:06:50   I've learned, things I'm definitely going to keep doing, things I'm definitely not going to do in

00:06:54   the future, et cetera. Have you ever used Slack for this kind of purpose before?

00:07:00   >> No, only informally, like in terms of I have had conversations on Slack about my apps with

00:07:06   individuals, but never on a, on the way that you're doing it where it's like, you're creating

00:07:11   essentially a, this, you know, a, a semi-public forum where, you know, hundreds of people can talk

00:07:19   about your app and the purpose of the room is to talk about the app. Like I've never done

00:07:24   anything like that. Like it's, it's always been way too intimidating of a thing to manage and to

00:07:29   feel like I would be able to reasonably engage with, I think is the thing that always made me

00:07:34   a little intimidated by something like that, where if someone is providing feedback,

00:07:38   I want to, would want to be able to feel present in there so that it's not like they're just

00:07:43   sort of shouting to each other in the void. But yeah, so I've never gone, I've never gone down

00:07:48   this road. So I'm very curious to hear the lessons you learned from going down it.

00:07:52   All right. So first we are sponsored this week by Blue Apron, the number one recipe delivery

00:07:58   service with the freshest ingredients. Get $30 off your delivery and free shipping by going to

00:08:03   blueapron.com/radar today. Blue Apron makes incredible home cooking accessible to everyone.

00:08:09   And they support a more sustainable food system. They set the highest standards for their

00:08:13   ingredients and they're building a community of home chefs for less than $10 per meal. Blue

00:08:18   Apron delivers seasonal recipes with fresh high quality ingredients to make delicious home cooked

00:08:23   meals in 40 minutes or less. Every meal comes with a step-by-step easy to follow recipe card

00:08:29   and ingredients that are portioned out to exactly the amount you need. So they're reducing food

00:08:33   waste and the Blue Apron freshness guarantee promises that every ingredient makes it safe,

00:08:37   sound and ready to cook, or they will make it right. I've been using Blue Apron myself,

00:08:42   my wife for maybe two years now. It's been a long time since before they were a sponsor of any of

00:08:47   our shows. Absolutely love it. We stick with it. We pay for it too, just like everybody else. And

00:08:52   we stick with it. It's wonderful. We really are better chefs now. We've tried so many new foods,

00:08:59   not even by having to go to restaurants, but by cooking them ourselves. We've eaten things we've

00:09:03   never eaten before. And it's just wonderful that their hit rate is very high. We've even tried

00:09:08   other services that are similar and we have found them to be by far our favorite. They are the ones

00:09:13   we always go back to and it's just the best. So you can choose from a variety of new recipes each

00:09:17   week, or you can let Blue Apron's culinary team surprise you. There is no weekly commitments. You

00:09:22   only get deliveries when you want them. So if you're going to be super busy or out of town for

00:09:26   a week, you can just go on their app or their site and just say, all right, skip this delivery

00:09:29   and this delivery. And that's it. You don't get charged for them. They don't give you a hassle.

00:09:32   You don't have to call anybody. It's great. So try Blue Apron today. Go to blueapron.com/radar.

00:09:38   You get $30 off your first delivery and free shipping. You will love how great it feels and

00:09:44   tastes to create an incredible home cooked meals with Blue Apron. So get started today

00:09:47   at blueapron.com/radar. Thank you to Blue Apron for supporting this show. Blue Apron,

00:09:52   a better way to cook. So I decided to do my big beta group and my Slack group as feedback

00:10:00   for the beta for the first time this time. So getting the big beta group was easy. I posted

00:10:06   on Twitter. I said, here's a web form. And it's just a little thing, as I said. It makes it easy

00:10:12   for me to dump out a CSV. And then I can import it right into TestFlight. So there's no real manual

00:10:18   work here involved in the testers. And it doesn't really matter how many you have. It's the same

00:10:22   amount of work. There's also, as more people get added to the list, you don't have to keep track of

00:10:28   who's been added or who hasn't. You can just export the entire list. And TestFlight is smart enough

00:10:32   not to import a bunch of duplicates. So you can just make a new version of the CSV that has 51

00:10:38   people instead of 50 people, import that, and it'll say, we imported one new tester. And if you want,

00:10:44   you can download a list of the 50 that we already had. It's quite clever, quite full featured.

00:10:49   So TestFlight is really great so far. It's been wonderful. They really have done quite a lot with

00:10:54   it. And I'm very happy with it. Slack, the idea of creating a Slack that had kind of a public

00:11:00   invitation used to be hard. It used to require that people make little web apps that would use

00:11:06   their API to generate one-off invitation links to every person who went there and tried to sign up.

00:11:12   This is no longer the case. Slack, you can now, and I should clarify, I only have a free

00:11:18   Slack account. I'm not paying for this. Their paid tier has lots of cool things for more pro use.

00:11:24   But for this kind of thing, you don't really need any of them. And it would actually be,

00:11:28   I think, quite cost prohibitive if you wanted to have 1,500 people in a Slack room with you.

00:11:32   - It's a pretty big company.

00:11:33   - Yeah, exactly. That would be probably not on an indie budget. But anyway, so now Slack just gives

00:11:40   you these invite links that just last 30 days. I think maybe in 60 days. They last a long time,

00:11:45   so you can just have one and hardly ever change it. And they just invite anybody who knows the

00:11:49   link. And if you need to reset it, if it gets in the wrong hands, you can, but you probably don't

00:11:52   care. So it's very, very easy to have a bulk invite. So I just put that bulk invite in the

00:11:58   TestFlight emails and said, "Here, this is the official discussion channel for this release,

00:12:04   or you can just email through TestFlight." I gave people both options, and they used both options.

00:12:08   But by far, the most people used Slack. It was by far the most common feedback channel. Only a

00:12:15   handful of people emailed, as usual. But Slack got, I invited about 1,500 people. The Slack group

00:12:23   has 945 people in it right now. That's an incredible ratio. To have two-thirds of the

00:12:29   people who were invited join a Slack channel, I was not expecting that. Because a lot of people,

00:12:34   if you use Slack at all, you probably have too many Slacks. That's kind of like the problem.

00:12:40   It's like everyone has a Slack for something now. So you have people who don't use it and don't want

00:12:44   to use it, and that's fine. But people who do use it, they have too many. So I thought it would be

00:12:48   hard to get people to sign up just for this. And I expected the ratio of people who did sign up to

00:12:54   be lower than two-thirds. So that's pretty awesome. And this, again, this probably has a lot to do

00:13:00   with your audience. I mean, my audience is pretty nerdy, so that probably helped me a lot here.

00:13:04   Anyway, so that was great. The one thing that became immediately apparent

00:13:10   when sending out that first batch of invitations is that, A, I should not send 1,500 invitations

00:13:17   at the same time. Because what that meant was I had most of these 900 people joining like that

00:13:26   first night, and the Slack channel went crazy. Like, it went from zero to 100 miles an hour

00:13:33   in two seconds, and even faster than your Tesla. And it was just like massive floods of information.

00:13:41   It very quickly became apparent that I needed some kind of slightly more structured way of reporting

00:13:48   bugs. I created a bugs room or channel in the Slack, and I had a little header thing in general

00:13:56   saying, "Please report bugs there," and general is for everything else. And that worked sort of

00:14:02   like, you know, about two-thirds of the bugs went there, about a third still came to general.

00:14:05   So, you know, you had this massive flood of chat going on. And it became very, very time consuming

00:14:13   for me to even just try to keep up with the slide, even just to read what was being said,

00:14:18   let alone to actually respond to anything or to have time to actually record these bugs down in

00:14:26   any kind of system or tracker or to-do list or anything. It became quite overwhelming at first.

00:14:32   The good thing is that that massive wave of activity was mostly just because everybody had

00:14:39   just joined, and were using the very first beta build at the same time. And so that has now calmed

00:14:44   down, and now it is, even though there's almost a thousand people in there, there's almost no chatter,

00:14:50   you know, now, because it's now just like a more of a regular discussion group. So it's not going

00:14:57   to be, and it's not like general discussion of like, you know, news and politics, it's just like

00:15:01   about this app. So I'm going to keep this group going, by the way. Now, you know, the beta is now

00:15:05   kind of over, but there's going to be more betas, and I'm just going to keep this group open and just,

00:15:09   you know, still monitor it, because now it's just like the overcast Slack. And if anybody wants to

00:15:14   join, I'll put the link in the show notes. So anybody, you don't have to be on the beta to join

00:15:16   the room. I don't care who joins. Please just please be nice, and then that's about it. So

00:15:21   anyway, the biggest thing that became apparent to me during this, so besides the fact that you

00:15:28   shouldn't send all your invitations all at once to avoid that massive flood, is that Slack is not

00:15:33   a bug tracker. And I'm sure every bug tracker has some kind of integration or app that I can use

00:15:41   with Slack, and I haven't tried any of them. I apologize. I honestly have not had time.

00:15:46   So for all the people who are using them or writing them or, you know, work for these

00:15:50   companies, I apologize. I'm sure you have good solutions. I would love to hear about them after

00:15:54   the iPhone 10 ships, but for now I don't have time. But Slack is really a terrible bug tracker

00:16:03   if you try to basically just use it itself as one, because, you know, A, there's no like

00:16:09   sorting or voting or easy way for people to check if there's been duplicates of the same thing.

00:16:16   So number one problem is tons of duplicate bug reports of bugs that were already reported like

00:16:22   10 chat lines up from the person reporting it. But I don't expect every person to come in there

00:16:27   and read the entire history of Slack first before they report their bug. Like I don't want them to

00:16:31   have to do that, because then they won't report their bugs, and that's bad for me. So you can't

00:16:35   really expect people to do that. So there has to be some better way that I have to devise to

00:16:41   let people submit bug reports for people to be able to see what is currently being worked on

00:16:46   and what is currently already known. And I know, I mean, everybody who has a bug report system knows

00:16:52   that people report duplicates anyway, but I think I'd get a lot fewer of them, and that would reduce

00:16:56   the workload at least. Slack also doesn't have basic things like, suppose somebody reports a

00:17:03   really good bug and I want to get to it, but there's now 100 items below it that are newer,

00:17:08   and the next time I go to that page, it's gone. It's like, you know, it's like, it's, you know,

00:17:13   buried under newness. There's a few mechanisms like they had, like Slackbot has a remind feature.

00:17:21   You can like, you can say like, "Remind me of this message tomorrow or in an hour." It's a pretty

00:17:26   coarse system. It's not very well featured, and it just shows up as like DMs from Slackbot,

00:17:32   which is a hack. I mean, it's fine. And I use that system on a few of them to like,

00:17:37   okay, I want to hit this, but I'm like seeing this on my iPad right now, right before I go to bed or

00:17:42   something. It's like, I don't, I can't deal with this right now. Slack also does not have a mark

00:17:47   as unread for DMs or anything. So you can't like, if somebody DMs me and says, "Hey, I'm not on the

00:17:54   beta. Can you please add this email address?" If I see that DM, that mark should have read.

00:17:59   If I don't do it right there, like right then, or some way to remind me of this later,

00:18:05   it's gone. Because then, you know, tomorrow they're going to be lost off my DM list.

00:18:09   I'll never remember. I'll never find it. So it's really not a good bug tracking system with the,

00:18:18   but the lack of an actual bug tracker and the lack of any kind of mark as unread to kind of hack it

00:18:22   and the coarseness of the reminder system. You can do things that are kind of like bug

00:18:26   tracking in Slack, but it's not very good at it. Where it is really good at it is the ability to

00:18:32   freely upload media and for other people to like thumbs up and emoji response things and stuff like

00:18:37   that. And so for me, a total game changer in this beta test that had not happened before is people

00:18:46   were able to record screen captures with the iOS 11 built-in screen recording ability. And so, so

00:18:53   many of the bug reports, I'd say most of the bug reports, people were also posting videos of them

00:18:59   recorded with iOS screen capture right from the phone. That is awesome. And that helped so much.

00:19:04   You know, a video is worth 10,000 words, right? As the saying goes. So that was a huge game changer.

00:19:11   And the, the built-in test flight emailing feedback thing, I don't think makes that easy,

00:19:16   but people were very much doing it in Slack and it was wonderful. So that was great.

00:19:24   Otherwise it was, it was really nice besides the bug reporting duplicate, Mark has read,

00:19:30   remind me, mediocrity. It was really nice to have a discussion group about the build because then

00:19:37   like I was able to have to like ask the, ask the room like, Hey, what if I do this? And I had like

00:19:43   a little like yes and no button as reactions and people would like vote on it. There's probably a

00:19:47   poll functionality now. I'm not, honestly, I'm not a Slack expert, so I probably could have done even

00:19:52   more with this. Stuff like that. And, and, and also because it was a chat, when people would,

00:19:58   sometimes people would report a bug and I wasn't around, then other people would tell them, Oh,

00:20:04   that's a bug. It's being fixed in the next build. Or, Oh, that's not a bug. Here's why. Or here's

00:20:08   like that's intended. Like other people were helping each other. And so that also, you know,

00:20:13   you get some of that with like a form and stuff, but I think it's easier with Slack. So ultimately

00:20:19   I do need a real bug tracker, but otherwise, Oh, I need a real bug tracker. I don't need to

00:20:25   send all the invites all at once, but otherwise it was a huge success. I got way more bugs reported

00:20:31   than I ever had before. I got tons of bugs reported that weren't even bugs in this beta,

00:20:35   but were bugs from earlier versions that like, you know, that just happened to still not be fixed.

00:20:40   It was wonderful. So I will not only keep this group going as just the overcast Slack,

00:20:48   but I will definitely do future betas in this exact same way. I think the only way it would

00:20:54   really fall down with doing like a large scale, mostly public beta like this is for secrecy.

00:21:00   If you wanted to keep things private, if you didn't want people posting screenshots or revealing

00:21:04   what your features were, then you have a problem and you have to make different arrangements then.

00:21:09   But I didn't, I decided with this release to kind of develop it in the open. And that alone is

00:21:14   possibly worth discussing. But the short version is like, I was tired of keeping it secret and this

00:21:19   version had a lot of like UI fan service almost like doing long standing requests in the UI,

00:21:26   basic stuff like one tap play and stuff like that. So I kind of felt secrecy wasn't important for

00:21:31   this. And so I just did it all in the public and it was fine. So yeah, ultimately success.

00:21:36   >> The biggest things that come to mind for me with this is, at first, do you have a concern about the,

00:21:44   like, I guess, the skew or the bias of the group that you have to, as whether it's representative

00:21:57   of your user base at large? Because one thing that comes to mind with this is like, it's incredibly

00:22:03   specific to the, you know, someone saw a link on Twitter who follows you or overcast, went to it,

00:22:11   signed up for it, and then knew what slack was, went through the process of creating a slack and

00:22:16   our slack account and like, gone into it like there's a there's a fairly high barrier to get

00:22:21   in there, which is good in some ways in the sense that these users are clearly invested and

00:22:27   interested and passionate and excited about your app. But the only thing that I kind of wonder

00:22:32   about is it's is it, you know, your user base is much broader from a experience or technical

00:22:38   savviness or whatever perspective. And so it's like, I would be nervous for myself of, you know,

00:22:44   having a having a place where I'm getting lots of reinforcement for choices or decisions that

00:22:50   may be biased in a particular direction. So I was curious if that's something that you're worried

00:22:55   about? >> It absolutely is. But I think that's unfortunately inherent in pretty much any beta

00:23:01   test that almost anybody would do, you know, because the number of people who are going to

00:23:05   be willing to join a slack probably has a lot of overlap with the people who are willing to join

00:23:09   a beta. And so, you know, that is one of the reasons why my my join rate was so high as a

00:23:15   percentage of people. But, you know, ultimately, I still have now, you know, like 800 to 1500

00:23:23   installations of this build running in the wild, getting crash reports that collect with the built

00:23:28   in crash reporter, having people experience this in 800 different ways like that's wonderful.

00:23:35   And I've, you know, so even though it's not 800 of the exact same kind of per it isn't like

00:23:41   a random representative sample of the user base, it's such a big sample of the user base that I

00:23:48   think it still has a lot of value. And ultimately, you know, I don't think I have a better alternative

00:23:55   for beta testing than doing something like this. Like I'm not going to capture a real random sample

00:24:00   through beta through beta testing, because not everyone wants to install a beta or even knows

00:24:03   how or has any interest in doing it, and then they won't even give feedback maybe. So like,

00:24:07   it's not there's no way to get a true random sample. So I think what I have now is pretty good.

00:24:11   Yeah. And I think it's the thing that would worry me less is the like, the beta testing is just

00:24:16   useful in so far as the having a good volume of things just to catch issues that you wouldn't see

00:24:21   otherwise. But yeah, it's the I would be worried about it changing my perspective of the app and

00:24:27   pushing the app in a direction that like I have to consciously I know for myself choose to try and

00:24:33   put myself in the mindset of like, whatever my I imagine my typical user is. And so like having

00:24:38   it'd be easy to get to to have something like this where you get this sense of like, well,

00:24:42   everyone loves it. Everyone loves it in the slack. I could see that being a tricky thing. Something

00:24:47   else I was curious is, how do you see this interacting with so like the way you do like

00:24:53   your general support. So I imagine you you know, you do there's an email address that people can

00:24:57   email. And within the app with a lot of caveats, you have a way for people to contact you and

00:25:02   report stuff there. Is that something that you expect to continue doing? Or is this sort of your

00:25:10   the direction that you're going to try and push that into that if you're like the feedback that

00:25:14   you receive for the app will come through this group, rather than you know, sort of, because I

00:25:20   would, it would be awkward, I imagine a certain point to be maintaining two totally separate

00:25:24   sort of inboxes that you have your email feedback and support, and then you have your slack email

00:25:31   support. Now you have these two places that you now have to do stuff between and manage and read.

00:25:38   And like you're in you could sort of add or the risky thing would be at a certain point,

00:25:43   you're just doubling the problem. So I'm curious what you thought about that.

00:25:46   Jared Ranerelle That's definitely a risk. I mean,

00:25:48   you know, when you when you create one of these groups, you know, you are adding a new thing that

00:25:52   you have to check that you have to respond to. And so that, you know, normally, I'd be very

00:25:58   averse to that. The difference here is that I already use slack a lot. I'm already checking

00:26:02   it throughout the day for other in other slacks. And so like, it's not a big deal for me to add one

00:26:05   more thing here. And, and because it is, you know, semi public, it does have that effect of other

00:26:12   people will help each other and they have been so that that helps tremendously compared to email.

00:26:16   But, you know, I also set such an incredibly bad standard for how I respond to email,

00:26:23   that anything's an improvement. Jon Sorrentino

00:26:25   Sure. Yeah. And then we guess the last thing too, is I always wonder about is, it's like, I,

00:26:31   I always worry about from a sense of create, whenever you create a community, and you're sort

00:26:36   of the moderator or the steward of it, it's like, there's that feeling of responsibility for making

00:26:42   sure that everyone's being nice, and everyone's treating each other appropriately. And that side

00:26:47   of things would, you know, like, I'm just thinking of the way that I get, will get email, you know,

00:26:52   customer support email that can be very negative and can be very harsh or problematic. And it would

00:26:59   be something that would be in some ways, I'm glad that that's not a public thing. And then also,

00:27:04   it's, you know, it's something that I've had to put people, you know, someone else reads my first

00:27:08   line of customer support to avoid that having an impact on me personally. And so this feels even

00:27:15   more direct and even more connected in that way. So that's another thing that sort of comes to mind

00:27:21   as potentially problematic. Jon Sorrentino

00:27:24   I did think about that. And I'm, you know, long term, that might become a problem eventually,

00:27:28   but probably not. I mean, the because like, this is a very topic focused Slack. And it's,

00:27:34   you know, it's specifically about discussing this app and feedback about this app and bug reports

00:27:39   about this app. I think as long as I keep it relatively tightly focused to that, I'm not that

00:27:45   concerned about, you know, basically community problems, because running a community, you know,

00:27:49   your concern is very, very good and very correct. Like running a more general purpose community is

00:27:55   really hard. And there's so many problems with just humans, the way we talk to each other. It's

00:28:01   a big problem. And it's a huge challenge that I am not at all set up to do and have no interest in

00:28:06   doing. But this is such a small and focused group. I don't think it will be a problem. And if it does

00:28:12   become a problem, I'll just shut it down. Jon Sorrentino

00:28:14   Sure. Yeah, because I mean, the reality is, like, at the end of the day, like the proof is in the

00:28:18   pudding. Like I've been using the beta since, you know, the beginning of this process and have

00:28:23   watched it get better and better and better. And I think, you know, it's like the reality is it

00:28:28   work like it is working. And so even if there are sort of potential pitfalls or problems that may

00:28:34   come down the road, like this is certainly proved to be a very effective way to get a lot of useful

00:28:39   feedback that has made your app better over time. And so like, and from that perspective, it's just

00:28:44   like an absolutely clear win of like, yeah, this is certainly has worked well and is an interesting

00:28:49   case study, I think for other developers to consider. I mean, I think for myself, I am still

00:28:55   too intimidated by it. I'll probably just stick to email and, you know, have someone do my tier

00:28:59   one support and kind of filter it through like that works for me. But I can totally see where

00:29:03   this could be. It's like it is such a bigger potential upside, just at the expense of a few

00:29:10   potential downsides. But overall, like it's working. And I, you know, I'm loving the bill

00:29:15   that I have now. So it's worked. Jon Sorrentino

00:29:17   Yeah. I mean, first of all, never underestimate how much I hate email. And, you know, you're

00:29:22   right though. I mean, this, this, the process worked in the sense that I got better and more

00:29:25   feedback than ever. And as a result, this is, in my opinion, to the best of my knowledge,

00:29:32   asterisk, asterisk, the least buggy and best version that I've ever shipped. And, you know,

00:29:37   the results speak for themselves. Anyway, thanks for listening, everybody. And we'll talk to you

00:29:41   next week. Bye.