0:00:00Welcome to Under the Radar, a show about independent iOS app development. I'm Marco Arment.
0:00:04And I'm David Smith. Under the Radar is never longer than 30 minutes, so let's get started.
0:00:09I'm pretty tired because last night I finally finished and shipped my most recent beta for
0:00:15Overcast 4.0 and it's going to the App Store now.
0:00:18Yay!
0:00:18I hope so. I hope it's yay. You never really know until like maybe a day after it's been in the
0:00:25store and then you realize, okay, nothing seems to have gone horribly wrong. It seems okay. Now I
0:00:30can finally like breathe, you know, breathe and like relax a little bit.
0:00:34Well, but at the very least, you've crossed the point where now you've said this is 4.0,
0:00:40like this is what this version is. And so it's now it's just fixes from here rather than like,
0:00:47you've crossed that point where you're like considering everything in the known universe
0:00:51as a possibility. It's like now you've crossed over to like this is what it's going to be and
0:00:55now you can just fix and tweak from here on out. So it's an accomplishment either way.
0:01:00Yeah. And I mean, this is a bit of a weird release for me because it's not a big feature release.
0:01:06Like the list of new features is really short. It's basically a giant bug fix and UI tweaking
0:01:14release. I did want to talk this week about the process of beta testing that I use this time. This
0:01:21is this is the first time. So in the past, I've done small scale betas where I invite like, you
0:01:27know, 30 or 40 friends and trusted people and people like bloggers and stuff like that. And
0:01:32I've had relatively minimal success with those over time. You know, when when you're inviting
0:01:39people to a beta, the problem that I always that I've always had and I do is when people invite me
0:01:44to their betas, I have the exact same thing, which is you get to the you get the app, you poke around
0:01:49a whole lot with the very first build you get, and then you basically don't ever do it again. Or
0:01:54like, you know, then it becomes like regular usage of the app for you. You're not really like
0:01:59thoroughly testing and thoroughly giving feedback on anything past like the very first one that you
0:02:04get in a lot of in a lot of cases. And so it's very hard for small betas to really provide
0:02:10meaningful feedback in terms of bug reporting. They are very useful, usually in terms of like
0:02:16design, critique UI critique, you know, figuring out like what parts of your app are confusing
0:02:21people and which parts aren't pretty much any number of people will reveal the same set of
0:02:27problems with that. It's like if you have a beta test of 40 people and a beta test or beta test of
0:02:314000 people, like they're going to have roughly the same output of like this parts of this part
0:02:37of the app is confusing to us or this part doesn't really work for us. You'll just have more reports
0:02:41of the same things if you have a bigger, a bigger group. So for that it's good, but for actually
0:02:46finding bugs, it's not that useful to have a small group because they won't run into as many as a
0:02:53bigger group will. So and because of that kind of drop off effect where people install the very
0:02:59first version and then not much after that, it becomes hard to even keep the group going. In
0:03:04TestFlight I have this group called like friends that you know of that group, TestFlight tells you
0:03:10who installs each version and I can, you can just see like every time you send a new build, like it
0:03:16gets fewer installations than the one before it. So if you're sending build to the same group of
0:03:20people over and over again, you know, eventually you're not getting a meaningful number of people
0:03:25installing it to really be worth the hassle at all. So one of the approaches I did in the past
0:03:31was I basically set up a basic little form on the Overcast site and I announced on Twitter like,
0:03:36hey, sign up here and the first, you know, 500 people allowed to TestFlight and this,
0:03:40this all became possible with TestFlight being part of Apple and having its limits raised. The
0:03:45initial, you know, so back when it was like UDID provisioning based manual stuff, which was
0:03:51horrible, that was limited to a hundred, right? And it was under devices, not even people.
0:03:55Yeah. And the devices were tied to your developer account and you couldn't reset them. And so it was
0:04:00a big pain. It was, you really couldn't have an open call because you would be tying, like locking
0:04:06into someone, you know, for an entire calendar year, I think it was. Yeah. And then like, and
0:04:12even, even if you finally got like your 30 friends and they were tested, then as soon as everyone got
0:04:17new phones or new iPads or whatever, like it ruined it and you had to go reset it. It was,
0:04:21you had like six devices from that, like one person who reviews phones and it's like, which
0:04:26one of these are you currently using and taking up all these slots and yeah. So that was a pain.
0:04:30So then when Apple bought it, then they, they, the initial like raise of the limit, I believe,
0:04:35raised to a thousand, right? Yes, I believe so. It got a couple more raises. And then most recently,
0:04:41I think this past summer they raised it to 20,000. So that's now a lot of people now, you know, if
0:04:47you can, and now they've, there's much better management of groups in test flight. So you can,
0:04:52you can have like a, like, you know, close friends group and then you can have like a public group
0:04:57where you can have like send and you can send different builds to different groups. So you can
0:05:01have, you can kind of stage it out where like you can try things first with a smaller group and then
0:05:06stage it out to a bigger group. You can have a separate group for like people in the press,
0:05:09stuff like that. So there's great flexibility now in test flight. And so this time I wanted to take
0:05:15even more advantage of it than I, than I have before. I wanted to get way more people in there,
0:05:20first of all. So before the group would be like 500 people. Now, if you send out 500 invitations,
0:05:26you're going to get about 300 people who actually install and use the build. And then, you know,
0:05:30and then of those 300 people, you're going to get maybe 20 people who actually ever email you
0:05:34anything. So like you have to start like running the numbers here. It's like, okay, this is to get
0:05:39a meaningful amount of feedback. You need a lot of people and you also need to make feedback easy
0:05:43and there needs to be like a good place for it. The other issue I've had in, in the past,
0:05:48when I've done those larger tests is I always get people asking me like, what's the official forum
0:05:53for discussing this? Like, should I guess to be emailing you through the test flight response
0:05:57emails, which are okay. Is that everyone always asks for like, is there like a forum or a bug
0:06:04tracker or a Slack group or a chat room or like anything like that, any place to go to discuss
0:06:10this beta. There's always a problem. And I didn't really have a way to communicate with the testers
0:06:16except just emailing everybody, but nobody wants that and I don't want to do it. No one wants to
0:06:21receive that. So, so I didn't have that either. So this, for this beta, I decided to go all out
0:06:26and try something even crazier. Instead of, instead of, you know, announcing, or instead of
0:06:31inviting 500 people, this time I invited 1500 people and I set up a Slack group for the very
0:06:37first time for people to join the Overcast Slack and discuss the beta and report bugs and everything.
0:06:44And this is basically what I want to talk about today is the experience of doing this, lessons
0:06:50I've learned, things I'm definitely going to keep doing, things I'm definitely not going to do in
0:06:54the future, et cetera. Have you ever used Slack for this kind of purpose before?
0:07:00>> No, only informally, like in terms of I have had conversations on Slack about my apps with
0:07:06individuals, but never on a, on the way that you're doing it where it's like, you're creating
0:07:11essentially a, this, you know, a, a semi-public forum where, you know, hundreds of people can talk
0:07:19about your app and the purpose of the room is to talk about the app. Like I've never done
0:07:24anything like that. Like it's, it's always been way too intimidating of a thing to manage and to
0:07:29feel like I would be able to reasonably engage with, I think is the thing that always made me
0:07:34a little intimidated by something like that, where if someone is providing feedback,
0:07:38I want to, would want to be able to feel present in there so that it's not like they're just
0:07:43sort of shouting to each other in the void. But yeah, so I've never gone, I've never gone down
0:07:48this road. So I'm very curious to hear the lessons you learned from going down it.
0:07:52All right. So first we are sponsored this week by Blue Apron, the number one recipe delivery
0:07:58service with the freshest ingredients. Get $30 off your delivery and free shipping by going to
0:08:03blueapron.com/radar today. Blue Apron makes incredible home cooking accessible to everyone.
0:08:09And they support a more sustainable food system. They set the highest standards for their
0:08:13ingredients and they're building a community of home chefs for less than $10 per meal. Blue
0:08:18Apron delivers seasonal recipes with fresh high quality ingredients to make delicious home cooked
0:08:23meals in 40 minutes or less. Every meal comes with a step-by-step easy to follow recipe card
0:08:29and ingredients that are portioned out to exactly the amount you need. So they're reducing food
0:08:33waste and the Blue Apron freshness guarantee promises that every ingredient makes it safe,
0:08:37sound and ready to cook, or they will make it right. I've been using Blue Apron myself,
0:08:42my wife for maybe two years now. It's been a long time since before they were a sponsor of any of
0:08:47our shows. Absolutely love it. We stick with it. We pay for it too, just like everybody else. And
0:08:52we stick with it. It's wonderful. We really are better chefs now. We've tried so many new foods,
0:08:59not even by having to go to restaurants, but by cooking them ourselves. We've eaten things we've
0:09:03never eaten before. And it's just wonderful that their hit rate is very high. We've even tried
0:09:08other services that are similar and we have found them to be by far our favorite. They are the ones
0:09:13we always go back to and it's just the best. So you can choose from a variety of new recipes each
0:09:17week, or you can let Blue Apron's culinary team surprise you. There is no weekly commitments. You
0:09:22only get deliveries when you want them. So if you're going to be super busy or out of town for
0:09:26a week, you can just go on their app or their site and just say, all right, skip this delivery
0:09:29and this delivery. And that's it. You don't get charged for them. They don't give you a hassle.
0:09:32You don't have to call anybody. It's great. So try Blue Apron today. Go to blueapron.com/radar.
0:09:38You get $30 off your first delivery and free shipping. You will love how great it feels and
0:09:44tastes to create an incredible home cooked meals with Blue Apron. So get started today
0:09:47at blueapron.com/radar. Thank you to Blue Apron for supporting this show. Blue Apron,
0:09:52a better way to cook. So I decided to do my big beta group and my Slack group as feedback
0:10:00for the beta for the first time this time. So getting the big beta group was easy. I posted
0:10:06on Twitter. I said, here's a web form. And it's just a little thing, as I said. It makes it easy
0:10:12for me to dump out a CSV. And then I can import it right into TestFlight. So there's no real manual
0:10:18work here involved in the testers. And it doesn't really matter how many you have. It's the same
0:10:22amount of work. There's also, as more people get added to the list, you don't have to keep track of
0:10:28who's been added or who hasn't. You can just export the entire list. And TestFlight is smart enough
0:10:32not to import a bunch of duplicates. So you can just make a new version of the CSV that has 51
0:10:38people instead of 50 people, import that, and it'll say, we imported one new tester. And if you want,
0:10:44you can download a list of the 50 that we already had. It's quite clever, quite full featured.
0:10:49So TestFlight is really great so far. It's been wonderful. They really have done quite a lot with
0:10:54it. And I'm very happy with it. Slack, the idea of creating a Slack that had kind of a public
0:11:00invitation used to be hard. It used to require that people make little web apps that would use
0:11:06their API to generate one-off invitation links to every person who went there and tried to sign up.
0:11:12This is no longer the case. Slack, you can now, and I should clarify, I only have a free
0:11:18Slack account. I'm not paying for this. Their paid tier has lots of cool things for more pro use.
0:11:24But for this kind of thing, you don't really need any of them. And it would actually be,
0:11:28I think, quite cost prohibitive if you wanted to have 1,500 people in a Slack room with you.
0:11:32- It's a pretty big company.
0:11:33- Yeah, exactly. That would be probably not on an indie budget. But anyway, so now Slack just gives
0:11:40you these invite links that just last 30 days. I think maybe in 60 days. They last a long time,
0:11:45so you can just have one and hardly ever change it. And they just invite anybody who knows the
0:11:49link. And if you need to reset it, if it gets in the wrong hands, you can, but you probably don't
0:11:52care. So it's very, very easy to have a bulk invite. So I just put that bulk invite in the
0:11:58TestFlight emails and said, "Here, this is the official discussion channel for this release,
0:12:04or you can just email through TestFlight." I gave people both options, and they used both options.
0:12:08But by far, the most people used Slack. It was by far the most common feedback channel. Only a
0:12:15handful of people emailed, as usual. But Slack got, I invited about 1,500 people. The Slack group
0:12:23has 945 people in it right now. That's an incredible ratio. To have two-thirds of the
0:12:29people who were invited join a Slack channel, I was not expecting that. Because a lot of people,
0:12:34if you use Slack at all, you probably have too many Slacks. That's kind of like the problem.
0:12:40It's like everyone has a Slack for something now. So you have people who don't use it and don't want
0:12:44to use it, and that's fine. But people who do use it, they have too many. So I thought it would be
0:12:48hard to get people to sign up just for this. And I expected the ratio of people who did sign up to
0:12:54be lower than two-thirds. So that's pretty awesome. And this, again, this probably has a lot to do
0:13:00with your audience. I mean, my audience is pretty nerdy, so that probably helped me a lot here.
0:13:04Anyway, so that was great. The one thing that became immediately apparent
0:13:10when sending out that first batch of invitations is that, A, I should not send 1,500 invitations
0:13:17at the same time. Because what that meant was I had most of these 900 people joining like that
0:13:26first night, and the Slack channel went crazy. Like, it went from zero to 100 miles an hour
0:13:33in two seconds, and even faster than your Tesla. And it was just like massive floods of information.
0:13:41It very quickly became apparent that I needed some kind of slightly more structured way of reporting
0:13:48bugs. I created a bugs room or channel in the Slack, and I had a little header thing in general
0:13:56saying, "Please report bugs there," and general is for everything else. And that worked sort of
0:14:02like, you know, about two-thirds of the bugs went there, about a third still came to general.
0:14:05So, you know, you had this massive flood of chat going on. And it became very, very time consuming
0:14:13for me to even just try to keep up with the slide, even just to read what was being said,
0:14:18let alone to actually respond to anything or to have time to actually record these bugs down in
0:14:26any kind of system or tracker or to-do list or anything. It became quite overwhelming at first.
0:14:32The good thing is that that massive wave of activity was mostly just because everybody had
0:14:39just joined, and were using the very first beta build at the same time. And so that has now calmed
0:14:44down, and now it is, even though there's almost a thousand people in there, there's almost no chatter,
0:14:50you know, now, because it's now just like a more of a regular discussion group. So it's not going
0:14:57to be, and it's not like general discussion of like, you know, news and politics, it's just like
0:15:01about this app. So I'm going to keep this group going, by the way. Now, you know, the beta is now
0:15:05kind of over, but there's going to be more betas, and I'm just going to keep this group open and just,
0:15:09you know, still monitor it, because now it's just like the overcast Slack. And if anybody wants to
0:15:14join, I'll put the link in the show notes. So anybody, you don't have to be on the beta to join
0:15:16the room. I don't care who joins. Please just please be nice, and then that's about it. So
0:15:21anyway, the biggest thing that became apparent to me during this, so besides the fact that you
0:15:28shouldn't send all your invitations all at once to avoid that massive flood, is that Slack is not
0:15:33a bug tracker. And I'm sure every bug tracker has some kind of integration or app that I can use
0:15:41with Slack, and I haven't tried any of them. I apologize. I honestly have not had time.
0:15:46So for all the people who are using them or writing them or, you know, work for these
0:15:50companies, I apologize. I'm sure you have good solutions. I would love to hear about them after
0:15:54the iPhone 10 ships, but for now I don't have time. But Slack is really a terrible bug tracker
0:16:03if you try to basically just use it itself as one, because, you know, A, there's no like
0:16:09sorting or voting or easy way for people to check if there's been duplicates of the same thing.
0:16:16So number one problem is tons of duplicate bug reports of bugs that were already reported like
0:16:2210 chat lines up from the person reporting it. But I don't expect every person to come in there
0:16:27and read the entire history of Slack first before they report their bug. Like I don't want them to
0:16:31have to do that, because then they won't report their bugs, and that's bad for me. So you can't
0:16:35really expect people to do that. So there has to be some better way that I have to devise to
0:16:41let people submit bug reports for people to be able to see what is currently being worked on
0:16:46and what is currently already known. And I know, I mean, everybody who has a bug report system knows
0:16:52that people report duplicates anyway, but I think I'd get a lot fewer of them, and that would reduce
0:16:56the workload at least. Slack also doesn't have basic things like, suppose somebody reports a
0:17:03really good bug and I want to get to it, but there's now 100 items below it that are newer,
0:17:08and the next time I go to that page, it's gone. It's like, you know, it's like, it's, you know,
0:17:13buried under newness. There's a few mechanisms like they had, like Slackbot has a remind feature.
0:17:21You can like, you can say like, "Remind me of this message tomorrow or in an hour." It's a pretty
0:17:26coarse system. It's not very well featured, and it just shows up as like DMs from Slackbot,
0:17:32which is a hack. I mean, it's fine. And I use that system on a few of them to like,
0:17:37okay, I want to hit this, but I'm like seeing this on my iPad right now, right before I go to bed or
0:17:42something. It's like, I don't, I can't deal with this right now. Slack also does not have a mark
0:17:47as unread for DMs or anything. So you can't like, if somebody DMs me and says, "Hey, I'm not on the
0:17:54beta. Can you please add this email address?" If I see that DM, that mark should have read.
0:17:59If I don't do it right there, like right then, or some way to remind me of this later,
0:18:05it's gone. Because then, you know, tomorrow they're going to be lost off my DM list.
0:18:09I'll never remember. I'll never find it. So it's really not a good bug tracking system with the,
0:18:18but the lack of an actual bug tracker and the lack of any kind of mark as unread to kind of hack it
0:18:22and the coarseness of the reminder system. You can do things that are kind of like bug
0:18:26tracking in Slack, but it's not very good at it. Where it is really good at it is the ability to
0:18:32freely upload media and for other people to like thumbs up and emoji response things and stuff like
0:18:37that. And so for me, a total game changer in this beta test that had not happened before is people
0:18:46were able to record screen captures with the iOS 11 built-in screen recording ability. And so, so
0:18:53many of the bug reports, I'd say most of the bug reports, people were also posting videos of them
0:18:59recorded with iOS screen capture right from the phone. That is awesome. And that helped so much.
0:19:04You know, a video is worth 10,000 words, right? As the saying goes. So that was a huge game changer.
0:19:11And the, the built-in test flight emailing feedback thing, I don't think makes that easy,
0:19:16but people were very much doing it in Slack and it was wonderful. So that was great.
0:19:24Otherwise it was, it was really nice besides the bug reporting duplicate, Mark has read,
0:19:30remind me, mediocrity. It was really nice to have a discussion group about the build because then
0:19:37like I was able to have to like ask the, ask the room like, Hey, what if I do this? And I had like
0:19:43a little like yes and no button as reactions and people would like vote on it. There's probably a
0:19:47poll functionality now. I'm not, honestly, I'm not a Slack expert, so I probably could have done even
0:19:52more with this. Stuff like that. And, and, and also because it was a chat, when people would,
0:19:58sometimes people would report a bug and I wasn't around, then other people would tell them, Oh,
0:20:04that's a bug. It's being fixed in the next build. Or, Oh, that's not a bug. Here's why. Or here's
0:20:08like that's intended. Like other people were helping each other. And so that also, you know,
0:20:13you get some of that with like a form and stuff, but I think it's easier with Slack. So ultimately
0:20:19I do need a real bug tracker, but otherwise, Oh, I need a real bug tracker. I don't need to
0:20:25send all the invites all at once, but otherwise it was a huge success. I got way more bugs reported
0:20:31than I ever had before. I got tons of bugs reported that weren't even bugs in this beta,
0:20:35but were bugs from earlier versions that like, you know, that just happened to still not be fixed.
0:20:40It was wonderful. So I will not only keep this group going as just the overcast Slack,
0:20:48but I will definitely do future betas in this exact same way. I think the only way it would
0:20:54really fall down with doing like a large scale, mostly public beta like this is for secrecy.
0:21:00If you wanted to keep things private, if you didn't want people posting screenshots or revealing
0:21:04what your features were, then you have a problem and you have to make different arrangements then.
0:21:09But I didn't, I decided with this release to kind of develop it in the open. And that alone is
0:21:14possibly worth discussing. But the short version is like, I was tired of keeping it secret and this
0:21:19version had a lot of like UI fan service almost like doing long standing requests in the UI,
0:21:26basic stuff like one tap play and stuff like that. So I kind of felt secrecy wasn't important for
0:21:31this. And so I just did it all in the public and it was fine. So yeah, ultimately success.
0:21:36>> The biggest things that come to mind for me with this is, at first, do you have a concern about the,
0:21:44like, I guess, the skew or the bias of the group that you have to, as whether it's representative
0:21:57of your user base at large? Because one thing that comes to mind with this is like, it's incredibly
0:22:03specific to the, you know, someone saw a link on Twitter who follows you or overcast, went to it,
0:22:11signed up for it, and then knew what slack was, went through the process of creating a slack and
0:22:16our slack account and like, gone into it like there's a there's a fairly high barrier to get
0:22:21in there, which is good in some ways in the sense that these users are clearly invested and
0:22:27interested and passionate and excited about your app. But the only thing that I kind of wonder
0:22:32about is it's is it, you know, your user base is much broader from a experience or technical
0:22:38savviness or whatever perspective. And so it's like, I would be nervous for myself of, you know,
0:22:44having a having a place where I'm getting lots of reinforcement for choices or decisions that
0:22:50may be biased in a particular direction. So I was curious if that's something that you're worried
0:22:55about? >> It absolutely is. But I think that's unfortunately inherent in pretty much any beta
0:23:01test that almost anybody would do, you know, because the number of people who are going to
0:23:05be willing to join a slack probably has a lot of overlap with the people who are willing to join
0:23:09a beta. And so, you know, that is one of the reasons why my my join rate was so high as a
0:23:15percentage of people. But, you know, ultimately, I still have now, you know, like 800 to 1500
0:23:23installations of this build running in the wild, getting crash reports that collect with the built
0:23:28in crash reporter, having people experience this in 800 different ways like that's wonderful.
0:23:35And I've, you know, so even though it's not 800 of the exact same kind of per it isn't like
0:23:41a random representative sample of the user base, it's such a big sample of the user base that I
0:23:48think it still has a lot of value. And ultimately, you know, I don't think I have a better alternative
0:23:55for beta testing than doing something like this. Like I'm not going to capture a real random sample
0:24:00through beta through beta testing, because not everyone wants to install a beta or even knows
0:24:03how or has any interest in doing it, and then they won't even give feedback maybe. So like,
0:24:07it's not there's no way to get a true random sample. So I think what I have now is pretty good.
0:24:11Yeah. And I think it's the thing that would worry me less is the like, the beta testing is just
0:24:16useful in so far as the having a good volume of things just to catch issues that you wouldn't see
0:24:21otherwise. But yeah, it's the I would be worried about it changing my perspective of the app and
0:24:27pushing the app in a direction that like I have to consciously I know for myself choose to try and
0:24:33put myself in the mindset of like, whatever my I imagine my typical user is. And so like having
0:24:38it'd be easy to get to to have something like this where you get this sense of like, well,
0:24:42everyone loves it. Everyone loves it in the slack. I could see that being a tricky thing. Something
0:24:47else I was curious is, how do you see this interacting with so like the way you do like
0:24:53your general support. So I imagine you you know, you do there's an email address that people can
0:24:57email. And within the app with a lot of caveats, you have a way for people to contact you and
0:25:02report stuff there. Is that something that you expect to continue doing? Or is this sort of your
0:25:10the direction that you're going to try and push that into that if you're like the feedback that
0:25:14you receive for the app will come through this group, rather than you know, sort of, because I
0:25:20would, it would be awkward, I imagine a certain point to be maintaining two totally separate
0:25:24sort of inboxes that you have your email feedback and support, and then you have your slack email
0:25:31support. Now you have these two places that you now have to do stuff between and manage and read.
0:25:38And like you're in you could sort of add or the risky thing would be at a certain point,
0:25:43you're just doubling the problem. So I'm curious what you thought about that.
0:25:46Jared Ranerelle That's definitely a risk. I mean,
0:25:48you know, when you when you create one of these groups, you know, you are adding a new thing that
0:25:52you have to check that you have to respond to. And so that, you know, normally, I'd be very
0:25:58averse to that. The difference here is that I already use slack a lot. I'm already checking
0:26:02it throughout the day for other in other slacks. And so like, it's not a big deal for me to add one
0:26:05more thing here. And, and because it is, you know, semi public, it does have that effect of other
0:26:12people will help each other and they have been so that that helps tremendously compared to email.
0:26:16But, you know, I also set such an incredibly bad standard for how I respond to email,
0:26:23that anything's an improvement. Jon Sorrentino
0:26:25Sure. Yeah. And then we guess the last thing too, is I always wonder about is, it's like, I,
0:26:31I always worry about from a sense of create, whenever you create a community, and you're sort
0:26:36of the moderator or the steward of it, it's like, there's that feeling of responsibility for making
0:26:42sure that everyone's being nice, and everyone's treating each other appropriately. And that side
0:26:47of things would, you know, like, I'm just thinking of the way that I get, will get email, you know,
0:26:52customer support email that can be very negative and can be very harsh or problematic. And it would
0:26:59be something that would be in some ways, I'm glad that that's not a public thing. And then also,
0:27:04it's, you know, it's something that I've had to put people, you know, someone else reads my first
0:27:08line of customer support to avoid that having an impact on me personally. And so this feels even
0:27:15more direct and even more connected in that way. So that's another thing that sort of comes to mind
0:27:21as potentially problematic. Jon Sorrentino
0:27:24I did think about that. And I'm, you know, long term, that might become a problem eventually,
0:27:28but probably not. I mean, the because like, this is a very topic focused Slack. And it's,
0:27:34you know, it's specifically about discussing this app and feedback about this app and bug reports
0:27:39about this app. I think as long as I keep it relatively tightly focused to that, I'm not that
0:27:45concerned about, you know, basically community problems, because running a community, you know,
0:27:49your concern is very, very good and very correct. Like running a more general purpose community is
0:27:55really hard. And there's so many problems with just humans, the way we talk to each other. It's
0:28:01a big problem. And it's a huge challenge that I am not at all set up to do and have no interest in
0:28:06doing. But this is such a small and focused group. I don't think it will be a problem. And if it does
0:28:12become a problem, I'll just shut it down. Jon Sorrentino
0:28:14Sure. Yeah, because I mean, the reality is, like, at the end of the day, like the proof is in the
0:28:18pudding. Like I've been using the beta since, you know, the beginning of this process and have
0:28:23watched it get better and better and better. And I think, you know, it's like the reality is it
0:28:28work like it is working. And so even if there are sort of potential pitfalls or problems that may
0:28:34come down the road, like this is certainly proved to be a very effective way to get a lot of useful
0:28:39feedback that has made your app better over time. And so like, and from that perspective, it's just
0:28:44like an absolutely clear win of like, yeah, this is certainly has worked well and is an interesting
0:28:49case study, I think for other developers to consider. I mean, I think for myself, I am still
0:28:55too intimidated by it. I'll probably just stick to email and, you know, have someone do my tier
0:28:59one support and kind of filter it through like that works for me. But I can totally see where
0:29:03this could be. It's like it is such a bigger potential upside, just at the expense of a few
0:29:10potential downsides. But overall, like it's working. And I, you know, I'm loving the bill
0:29:15that I have now. So it's worked. Jon Sorrentino
0:29:17Yeah. I mean, first of all, never underestimate how much I hate email. And, you know, you're
0:29:22right though. I mean, this, this, the process worked in the sense that I got better and more
0:29:25feedback than ever. And as a result, this is, in my opinion, to the best of my knowledge,
0:29:32asterisk, asterisk, the least buggy and best version that I've ever shipped. And, you know,
0:29:37the results speak for themselves. Anyway, thanks for listening, everybody. And we'll talk to you
0:29:41next week. Bye.