PodSearch

Developing Perspective

#140: Managing Beta Testing.

 

00:00:00 ◼   ► Hello and welcome to Developing Perspective,

00:00:02 ◼   ► Developing Perspective is a podcast discussing

00:00:04 ◼   ► News of Note and iOS development, Apple and the like.

00:00:07 ◼   ► I'm your host, David Smith.

00:00:08 ◼   ► I'm an independent iOS and Mac developer

00:00:10 ◼   ► based in Herndon, Virginia.

00:00:11 ◼   ► This is show number 140 and today is Friday, August 30th.

00:00:15 ◼   ► Developing Perspective is never longer than 15 minutes,

00:00:17 ◼   ► so let's get started.

00:00:19 ◼   ► All right, I'm gonna have a couple of quick topics

00:00:21 ◼   ► to talk through today.

00:00:22 ◼   ► The first is I'm gonna announce it here

00:00:24 ◼   ► probably for the first time.

00:00:25 ◼   ► I'm gonna be doing a, I say, short run

00:00:28 ◼   ► of developing perspective t-shirts.

00:00:31 ◼   ► T-shirts are always kind of one of these funny things.

00:00:33 ◼   ► I think it's kind of like everybody does them,

00:00:35 ◼   ► and it becomes a bit of a cliche.

00:00:36 ◼   ► But I kind of wanted to talk into why I'm doing it

00:00:38 ◼   ► and kind of what it's like.

00:00:39 ◼   ► And just sort of if that's the kind of thing

00:00:42 ◼   ► that you'd be interested in doing, by all means.

00:00:45 ◼   ► So first off, the t-shirt itself,

00:00:47 ◼   ► I'm doing it through Teespring because that's

00:00:49 ◼   ► ridiculously easy, and easy is good.

00:00:51 ◼   ► And so they're already just a company

00:00:53 ◼   ► that's kind of one of these things.

00:00:54 ◼   ► You go and buy it, and if you reach a certain threshold,

00:00:57 ◼   ► And it's sort of crowdfunded in that way, though I think I'll be setting my minimum

00:01:02 ◼   ► to the smallest number that's possible, because I just want to make sure that they exist far

00:01:06 ◼   ► more than anything else.

00:01:09 ◼   ► And the reason I'm doing it, which I thought was more of an interesting topic, is I'm a

00:01:13 ◼   ► big fan of these geeky t-shirts.

00:01:15 ◼   ► I have probably more of them than I should.

00:01:19 ◼   ► My closet is sort of half nice clothes and half geeky t-shirts.

00:01:23 ◼   ► And sometimes I think the geeky districts are gradually expanding and overwhelming.

00:01:29 ◼   ► The reason I have them, the reason I like them, is-- it's something that I kind of had

00:01:33 ◼   ► to step back and think about, is how the nature of being independent-- and I've talked about

00:01:37 ◼   ► this before-- is it's sort of intrinsically a little bit lonely in that I don't work with

00:01:43 ◼   ► people on a day-to-day basis, which is something I miss from kind of a more nine-to-five job.

00:01:47 ◼   ► I'm pretty much just on my own for a lot of the time and a lot of the work I do.

00:01:51 ◼   ► And so it's kind of difficult to sometimes have a sense of belonging, to have a sense

00:01:55 ◼   ► of camaraderie or a team or whatever you want to call it.

00:01:59 ◼   ► And there's things you can do around that.

00:02:00 ◼   ► But one of the things I kind of like is when I wear, wearing a t shirt, to me anyway, can

00:02:06 ◼   ► kind of feel like I'm part of something bigger, that it draws a sense of connection.

00:02:10 ◼   ► And you know, when I'm out, it's kind of like, I'm in on an in joke when I'm wearing like,

00:02:16 ◼   ► when I'm, you know, today I'm wearing my hypercritical t shirt, for example.

00:02:19 ◼   ► Like when I'm wearing that, like, most people have no idea what that is.

00:02:22 ◼   ► But if someone does, then I know they're my kind of people.

00:02:25 ◼   ► And even if they don't, like, it's kind of cool that I have, I sort of, and I know the

00:02:29 ◼   ► secret handshake, I know what's going on there.

00:02:31 ◼   ► And that's why I kind of I like that sense of that sense of community that kind of having

00:02:35 ◼   ► a, I guess you could say, uniform for encourages and kind of allows.

00:02:39 ◼   ► And so that's kind of a, it's kind of a fun thing.

00:02:42 ◼   ► And so that's kind of I was like, well, you know, I have all these other ones for all

00:02:46 ◼   ► the various things that I like, the things that I enjoy, different products I've been

00:02:50 ◼   ► associated with, etc. And so I thought, "Well, why don't I make one for developing perspective?"

00:02:54 ◼   ► It's a show that's been around for over two years now. This is episode 140, I think. It's

00:03:02 ◼   ► been around enough that I think it's probably worth the kind of thing to have a shirt for,

00:03:06 ◼   ► have something that I can wear, and that I can, you know, it's just sort of something

00:03:10 ◼   ► that I own, the community that I'm, I guess, directly part of, if not at the center of.

00:03:15 ◼   ► The design was actually done by friend of mine, Jory Raphael, who you may know from

00:03:18 ◼   ► things like symbolicons.

00:03:20 ◼   ► He is just a graphic designer, sort of vector artwork genius.

00:03:25 ◼   ► And when I decided on kind of generally what I wanted to do, I knew he would be the right

00:03:28 ◼   ► person to sort of take that and turn it into an actual design.

00:03:32 ◼   ► And so the design is pretty straightforward.

00:03:34 ◼   ► It's an underscore followed by a coffee mug.

00:03:37 ◼   ► It's actually my favorite coffee mug, which is, I'll have a picture of it in the show

00:03:40 ◼   ► notes.

00:03:41 ◼   ► But this particular coffee mug I got when I was visiting the Cupertino Apple

00:03:45 ◼   ► Campus a couple years ago, and it's a perfect shape

00:03:48 ◼   ► for drinking warm beverages.

00:03:50 ◼   ► If you're not into coffee, imagine it as a teacup, hot chocolate,

00:03:53 ◼   ► whatever floats your boat.

00:03:55 ◼   ► But that's kind of the concept.

00:03:56 ◼   ► And it was just something that I thought was a little bit geeky,

00:03:58 ◼   ► a little bit understated, and a little bit silly,

00:04:01 ◼   ► which is kind of one of the things that I enjoy in a t-shirt,

00:04:05 ◼   ► and something that I think is kind of fun.

00:04:07 ◼   ► The underscore at the beginning is, of course,

00:04:08 ◼   ► a reference to the fact that I now have a nickname, which

00:04:10 ◼   ► is kind of a funny thing.

00:04:11 ◼   ► Because for my entire life, I don't

00:04:13 ◼   ► think I've ever had a nickname.

00:04:14 ◼   ► The closest thing was being called Dave rather than David,

00:04:17 ◼   ► which I don't mind, which I guess is only vaguely

00:04:19 ◼   ► a nickname.

00:04:20 ◼   ► But for better or worse, I've ended up

00:04:21 ◼   ► being known as underscore in this community.

00:04:23 ◼   ► I think that sort of definitely peaked when I was recently--

00:04:26 ◼   ► I think it was Dr. Drang in one of an article

00:04:29 ◼   ► he referenced to me.

00:04:30 ◼   ► He referred to me, and then he abbreviated my name

00:04:33 ◼   ► as UDS, underscore David Smith.

00:04:35 ◼   ► So I have new initials, too.

00:04:36 ◼   ► So anyway, so that's kind of the concept.

00:04:39 ◼   ► It's fairly straightforward, fairly simple.

00:04:41 ◼   ► It's just a basic design.

00:04:42 ◼   ► There'll be a link in the show notes,

00:04:44 ◼   ► or you can go to teespring.com/underscore

00:04:46 ◼   ► to look at it and to buy it.

00:04:49 ◼   ► The price is $1,459, which is kind of a play on words

00:04:52 ◼   ► on the length of a-- my target length for a Developing

00:04:55 ◼   ► Perspective episode, which at that point,

00:04:58 ◼   ► it's really not for the money.

00:05:00 ◼   ► I'm not doing this to make a lot of money.

00:05:01 ◼   ► It's the kind of thing that's basically-- it's

00:05:03 ◼   ► a little bit above cost.

00:05:04 ◼   ► Essentially, my goal is to break even on the design

00:05:06 ◼   ► and the actual shipping of it.

00:05:08 ◼   ► My goal is to get more people to have it.

00:05:13 ◼   ► And so that's kind of it.

00:05:16 ◼   ► So anyway, I'll have a link in the show notes if that's something that you've been interested in, or if that's something that you want.

00:05:17 ◼   ► They should ship in time, I think for most of the fall conferences.

00:05:21 ◼   ► It should probably be arriving somewhere in mid-September when the campaign finishes in about, I think it finishes next week.

00:05:24 ◼   ► It'll just do a short one week kind of campaign for it.

00:05:30 ◼   ► So anyway, if that's your thing, great.

00:05:34 ◼   ► If not, I'm going to get into the rest of the topic for the show.

00:05:35 ◼   ► And that topic today is going to be, I'm going to talk about beta testing.

00:05:37 ◼   ► which is I always try-- this is a really tricky time for me

00:05:39 ◼   ► in terms of the time of year for coming up with new topics,

00:05:42 ◼   ► coming up with things to say, because a lot of what

00:05:44 ◼   ► I'm working on I can't talk about.

00:05:46 ◼   ► A lot of the new exciting things that are coming out

00:05:48 ◼   ► are either under NDA or unannounced.

00:05:50 ◼   ► And so we're in kind of this middle ground.

00:05:51 ◼   ► And so it's always kind of tricky.

00:05:52 ◼   ► But I think whenever that happens, the best thing to do

00:05:55 ◼   ► is just to fall back on what it is that I'm actually

00:05:56 ◼   ► doing it right now.

00:05:57 ◼   ► And right now, that's beta testing.

00:05:59 ◼   ► Right now, that is working on the final phases

00:06:02 ◼   ► of getting my apps all queued up to go out to Apple.

00:06:06 ◼   ► And so I'm going to talk a little bit about beta testing

00:06:08 ◼   ► generally, and then a few things that I've learned,

00:06:10 ◼   ► or situations, or struggles, or problems that you run into,

00:06:12 ◼   ► and what I do with that.

00:06:14 ◼   ► So first, I'm going to talk about what beta testing is

00:06:17 ◼   ► in iOS and how that works.

00:06:19 ◼   ► So generally, beta testing is the process

00:06:21 ◼   ► of taking your app right before it's ready for launch

00:06:25 ◼   ► and seeding it out to a small group of people

00:06:27 ◼   ► to get feedback on to identify bugs, problems, issues,

00:06:30 ◼   ► whatever it may be, and to do it in a way that

00:06:34 ◼   ► is constructive, hopefully.

00:06:35 ◼   ► Because ultimately, the goal, at least in my mind, for beta

00:06:37 ◼   ► testing, is to improve the product every time

00:06:41 ◼   ► you get around out.

00:06:43 ◼   ► There's some amount of it that is perhaps marketing oriented,

00:06:46 ◼   ► that it's certainly some amount of beta testing's value

00:06:49 ◼   ► is that it puts it in the hands of people who will hopefully

00:06:52 ◼   ► talk about it.

00:06:52 ◼   ► Mostly what you're really trying to do

00:06:54 ◼   ► is try to collect feedback from a small group

00:06:56 ◼   ► of trusted, interesting people who can help craft and sort

00:07:02 ◼   ► of direct the app in that final stage.

00:07:06 ◼   ► So typically, you've got the idea, the concept,

00:07:10 ◼   ► and then usually in sort of like 99% of the implementation

00:07:12 ◼   ► built. And now you're going to show it to some people

00:07:14 ◼   ► and get their feedback on it.

00:07:17 ◼   ► And that's always a dangerous thing.

00:07:19 ◼   ► And I'll say that, and mostly just in that,

00:07:22 ◼   ► it's always tricky when you open up an app to somebody else

00:07:25 ◼   ► and you're getting their feedback on it.

00:07:27 ◼   ► It can change your own view on that app.

00:07:30 ◼   ► It can change your direction for that app, for better or worse.

00:07:36 ◼   ► And so I think something I always want to emphasize

00:07:39 ◼   ► is it's important to be thoughtful about who

00:07:41 ◼   ► you show it to.

00:07:42 ◼   ► And this is something that I personally try--

00:07:45 ◼   ► my typical opinion on this is I show it

00:07:48 ◼   ► to a relatively small number of people.

00:07:51 ◼   ► And they're people whose opinions I trust,

00:07:54 ◼   ► who I really want to hear what they have to say,

00:08:00 ◼   ► and whose opinions, I think, are things that I tend to agree with,

00:08:03 ◼   ► are things that I would respect and respond well to.

00:08:07 ◼   ► I'm not the kind of person who tends to do these kind of betas

00:08:10 ◼   ► where you're trying to open it up to like 50 or 100 people,

00:08:13 ◼   ► many of whom you don't actually know.

00:08:15 ◼   ► They're just general customers.

00:08:17 ◼   ► And sometimes that can be useful if you're

00:08:19 ◼   ► solving a specific bug that customers have

00:08:22 ◼   ► and you want to have them reproduce it.

00:08:23 ◼   ► That makes sense to me.

00:08:24 ◼   ► But for a lot of times, what I find that's difficult

00:08:26 ◼   ► is the more and more beta testers you have,

00:08:28 ◼   ► the more time and energy it takes to actually manage

00:08:30 ◼   ► that process.

00:08:32 ◼   ► And so I'm not the kind of person who tends to just like,

00:08:34 ◼   ► every now and then I'll get requests on Twitter,

00:08:36 ◼   ► things like that.

00:08:36 ◼   ► People are like, hey, do you have any space in your beta

00:08:38 ◼   ► group to beta test something?

00:08:40 ◼   ► And I've almost done everything I've ever responded to someone

00:08:44 ◼   ► like that and said, yeah, that's great.

00:08:45 ◼   ► Let's put it in.

00:08:46 ◼   ► Because what I want to do is I want to--

00:08:49 ◼   ► in that critical phase of right before an episode is ready

00:08:52 ◼   ► to launch, what I want to do is to narrow in rather than

00:08:55 ◼   ► broaden out, necessarily.

00:08:56 ◼   ► And this is just the way that I work.

00:08:57 ◼   ► Some people do it differently, so the more the merrier.

00:09:00 ◼   ► Kind of open betas and things where you can get

00:09:02 ◼   ► lots of feedback.

00:09:03 ◼   ► What I'm trying to do is I'm trying to narrow down

00:09:06 ◼   ► my feedback and to be able to manage and really invest

00:09:08 ◼   ► and to have a lot of deep discussion back and forth,

00:09:11 ◼   ► which I just can't do with 50 people.

00:09:13 ◼   ► You can kind of collect crash reports

00:09:14 ◼   ► and those kinds of things, maybe things

00:09:15 ◼   ► that can be aggregated.

00:09:17 ◼   ► But for me, what I want to do is I want to have

00:09:19 ◼   ► a couple of conversations with people who I know

00:09:22 ◼   ► will have interesting feedback and can really help me

00:09:24 ◼   ► improve my product.

00:09:26 ◼   ► So what I do is you collect a group of people,

00:09:28 ◼   ► and you decide who it's going to be, and you send it out.

00:09:31 ◼   ► In iOS, the way you actually manage that is you

00:09:33 ◼   ► have to do what's called an ad hoc build, which

00:09:35 ◼   ► is if you've never done it, you just

00:09:37 ◼   ► go into the provisioning profile area in the developer portal.

00:09:42 ◼   ► And you set up-- it's a different signing certificate

00:09:44 ◼   ► and key.

00:09:45 ◼   ► It's similar to what you do when you're doing an app store build,

00:09:48 ◼   ► but you have to specify that the actual device

00:09:50 ◼   ► IDs for every device that you want that to run in--

00:09:53 ◼   ► and this gets into the whole kind of the way

00:09:55 ◼   ► that an iOS device is very locked down.

00:09:57 ◼   ► And so the only way to run an app is if it's signed.

00:09:59 ◼   ► And the only way to get it signed

00:10:00 ◼   ► is to have the-- you have to create an ad hoc

00:10:03 ◼   ► profile for the specific device that you want it to run on.

00:10:06 ◼   ► So you have to go and collect those IDs from everybody.

00:10:08 ◼   ► And most people in this community,

00:10:10 ◼   ► whether in the press or the developers,

00:10:11 ◼   ► should have no problems collecting that device ID.

00:10:14 ◼   ► If you need to do it from someone else who's

00:10:15 ◼   ► not quite as technical, there's all kinds of different tools

00:10:18 ◼   ► and things you can do to collect that.

00:10:20 ◼   ► Often the best is to use one of the tools, programs,

00:10:24 ◼   ► or services like TestFlight or Hockey,

00:10:28 ◼   ► which are two different services that

00:10:30 ◼   ► do very similar stuff around this kind of beta testing

00:10:32 ◼   ► and sending out builds.

00:10:34 ◼   ► But one of the things they often can do for you is they can collect

00:10:37 ◼   ► devices directly.

00:10:38 ◼   ► So you'll invite someone with an email.

00:10:39 ◼   ► They'll hit a button on the device,

00:10:41 ◼   ► and it'll pull down all the relevant information

00:10:44 ◼   ► that you're going to need.

00:10:45 ◼   ► So you prepare the build.

00:10:47 ◼   ► I definitely recommend that you send the builds

00:10:48 ◼   ► to people using one of those two services, TestFlight or Hockey.

00:10:51 ◼   ► I've used them both.

00:10:52 ◼   ► They're both pretty good.

00:10:53 ◼   ► You kind of go back and forth on which one's better.

00:10:55 ◼   ► I don't really know.

00:10:56 ◼   ► I think hockey is more of the kind of-- it's

00:10:59 ◼   ► a service you pay for.

00:11:00 ◼   ► It's a bit more robust in that way.

00:11:03 ◼   ► TestFlight is kind of more kind of-- it's a free service.

00:11:07 ◼   ► You can do lots of things with it.

00:11:08 ◼   ► So it's kind of sometimes nice for experimentation

00:11:11 ◼   ► or the first time you adopt this or if it's just

00:11:13 ◼   ► something you're doing on a hobby, maybe.

00:11:14 ◼   ► But you get what you pay for in a lot of ways.

00:11:17 ◼   ► And so it's just kind of a decision you have to make.

00:11:20 ◼   ► But whichever service you're going to use,

00:11:21 ◼   ► You put that-- you make your build, do it in Xcode,

00:11:25 ◼   ► upload it up to one of these services,

00:11:26 ◼   ► put in some useful distribution notes,

00:11:28 ◼   ► and then you can just send people links.

00:11:30 ◼   ► They take that link, and they download it onto their device

00:11:32 ◼   ► and they can get you feedback.

00:11:34 ◼   ► The actual way in which you collect feedback is-- a lot

00:11:37 ◼   ► of people do it very differently.

00:11:38 ◼   ► I've been recently on some great betas.

00:11:40 ◼   ► People have glassboard rooms on it

00:11:43 ◼   ► where you are collecting feedback there.

00:11:45 ◼   ► You have a discussion forum for beta testers.

00:11:47 ◼   ► You can have people email you.

00:11:49 ◼   ► They can IM you.

00:11:50 ◼   ► they can do kind of whatever makes sense for you.

00:11:53 ◼   ► And it's kind of hard to be prescriptive about that

00:11:55 ◼   ► because I think some people just react differently

00:11:58 ◼   ► to their feedback.

00:12:00 ◼   ► For me, I just want to, whatever is most convenient

00:12:02 ◼   ► for the person giving the feedback,

00:12:03 ◼   ► a lot of times it'll take emails,

00:12:05 ◼   ► it'll be taking items from people,

00:12:07 ◼   ► high messages, whatever it is.

00:12:08 ◼   ► And my goal is to try and understand the problems

00:12:11 ◼   ► that they're having, the things that are confusing,

00:12:13 ◼   ► number one, the things that are crashing, number two,

00:12:17 ◼   ► and the things that they wish it did.

00:12:20 ◼   ► And I do it in that order because first, things

00:12:23 ◼   ► that are confusing.

00:12:25 ◼   ► No matter what, if it's confusing,

00:12:26 ◼   ► that's very, very problematic for users,

00:12:29 ◼   ► and especially for support.

00:12:30 ◼   ► Where often, the worst thing you can ever have on a support

00:12:33 ◼   ► thing is where someone asks for a feature to-- oh,

00:12:36 ◼   ► I love the app, but I wish it did x.

00:12:38 ◼   ► And it actually already does x, but that

00:12:40 ◼   ► isn't clear in the interface, or it isn't well communicated.

00:12:43 ◼   ► And so making sure that things are clear, very important.

00:12:46 ◼   ► Making sure things are stable and solid.

00:12:48 ◼   ► Definitely in the beta phase, I highly

00:12:50 ◼   ► recommend putting in some kind of crash reporting.

00:12:52 ◼   ► I think both TestFlightHockey does it.

00:12:54 ◼   ► You can use Crashlytics, whatever, PLCrashReporter.

00:12:56 ◼   ► Whatever makes sense for you can incorporate crash reporting.

00:13:01 ◼   ► But it's really important in this phase

00:13:03 ◼   ► to be collecting that kind of stuff.

00:13:04 ◼   ► Because you'll find all the little bugs, all the little edge

00:13:05 ◼   ► cases, all the little things that you didn't expect

00:13:08 ◼   ► would happen, hopefully happen in your beta phase

00:13:10 ◼   ► so it doesn't happen broadly in production.

00:13:13 ◼   ► And then the last thing, which is really interesting,

00:13:15 ◼   ► is when you put an app out into actual people's hand

00:13:18 ◼   ► and you get them actually using it,

00:13:20 ◼   ► you'll get feedback about the things

00:13:21 ◼   ► that they wish it did additionally.

00:13:23 ◼   ► And this is, I've found, especially in Pod Wrangler,

00:13:26 ◼   ► what I wish I'd been doing most heavy beta testing for recently.

00:13:29 ◼   ► You find there are things that you

00:13:31 ◼   ► can add to your application.

00:13:33 ◼   ► And the best are the ones that are

00:13:35 ◼   ► these tiny little small features that have big impacts, things

00:13:39 ◼   ► that you can relatively easily add on,

00:13:41 ◼   ► that you find that your app is doing 99% of what they want.

00:13:44 ◼   ► But they just wish to, oh, I wish I just

00:13:46 ◼   ► did this one extra thing.

00:13:48 ◼   ► And adding that will make the app so much more useful,

00:13:50 ◼   ► so much more functional for that person.

00:13:52 ◼   ► And any time you can find this, it's a huge win and a big

00:13:56 ◼   ► benefit to find during the beta phase, so that you can

00:13:59 ◼   ► add that in.

00:14:00 ◼   ► And when you release it to the community at large, you can

00:14:05 ◼   ► take advantage of that.

00:14:06 ◼   ► And you can have a nice big splashy launch that has a lot

00:14:10 ◼   ► of the features, a lot of the things that you could squeeze

00:14:12 ◼   ► into in that time.

00:14:15 ◼   ► Anyway, so that's sort of beta testing.

00:14:16 ◼   ► That's kind of how the process works, kind of what I do.

00:14:19 ◼   ► I think it's important.

00:14:20 ◼   ► I think you can overemphasize it.

00:14:22 ◼   ► I think it's just something that you

00:14:23 ◼   ► want to do to collect that feedback,

00:14:24 ◼   ► to understand how your app's good, how it's bad,

00:14:28 ◼   ► and what you can do to make it better.

00:14:31 ◼   ► Anyway, that's it for today's show.

00:14:32 ◼   ► As always, if you have questions, comments, concerns,

00:14:34 ◼   ► complaints, I'm on Twitter @_davidsmith,

00:14:37 ◼   ► david@developingperspective.com.

00:14:38 ◼   ► Otherwise, have a great weekend.

00:14:40 ◼   ► If you're in the US, have a great Labor Day.

00:14:42 ◼   ► Long weekend.

00:14:43 ◼   ► And otherwise, have a great week.

00:14:44 ◼   ► and I'll see you next week. Bye.

00:14:48 ◼   ► [ Silence ]

00:15:06 ◼   ► [ Silence ]

00:15:24 ◼   ► [ Silence ]