#140: Managing Beta Testing.
00:00:00
◼
►
Hello and welcome to Developing Perspective,
00:00:02
◼
►
Developing Perspective is a podcast discussing
00:00:04
◼
►
News of Note and iOS development, Apple and the like.
00:00:07
◼
►
I'm your host, David Smith.
00:00:08
◼
►
I'm an independent iOS and Mac developer
00:00:10
◼
►
based in Herndon, Virginia.
00:00:11
◼
►
This is show number 140 and today is Friday, August 30th.
00:00:15
◼
►
Developing Perspective is never longer than 15 minutes,
00:00:17
◼
►
so let's get started.
00:00:19
◼
►
All right, I'm gonna have a couple of quick topics
00:00:21
◼
►
to talk through today.
00:00:22
◼
►
The first is I'm gonna announce it here
00:00:24
◼
►
probably for the first time.
00:00:25
◼
►
I'm gonna be doing a, I say, short run
00:00:28
◼
►
of developing perspective t-shirts.
00:00:31
◼
►
T-shirts are always kind of one of these funny things.
00:00:33
◼
►
I think it's kind of like everybody does them,
00:00:35
◼
►
and it becomes a bit of a cliche.
00:00:36
◼
►
But I kind of wanted to talk into why I'm doing it
00:00:38
◼
►
and kind of what it's like.
00:00:39
◼
►
And just sort of if that's the kind of thing
00:00:42
◼
►
that you'd be interested in doing, by all means.
00:00:45
◼
►
So first off, the t-shirt itself,
00:00:47
◼
►
I'm doing it through Teespring because that's
00:00:49
◼
►
ridiculously easy, and easy is good.
00:00:51
◼
►
And so they're already just a company
00:00:53
◼
►
that's kind of one of these things.
00:00:54
◼
►
You go and buy it, and if you reach a certain threshold,
00:00:57
◼
►
And it's sort of crowdfunded in that way, though I think I'll be setting my minimum
00:01:02
◼
►
to the smallest number that's possible, because I just want to make sure that they exist far
00:01:06
◼
►
more than anything else.
00:01:09
◼
►
And the reason I'm doing it, which I thought was more of an interesting topic, is I'm a
00:01:13
◼
►
big fan of these geeky t-shirts.
00:01:15
◼
►
I have probably more of them than I should.
00:01:19
◼
►
My closet is sort of half nice clothes and half geeky t-shirts.
00:01:23
◼
►
And sometimes I think the geeky districts are gradually expanding and overwhelming.
00:01:29
◼
►
The reason I have them, the reason I like them, is-- it's something that I kind of had
00:01:33
◼
►
to step back and think about, is how the nature of being independent-- and I've talked about
00:01:37
◼
►
this before-- is it's sort of intrinsically a little bit lonely in that I don't work with
00:01:43
◼
►
people on a day-to-day basis, which is something I miss from kind of a more nine-to-five job.
00:01:47
◼
►
I'm pretty much just on my own for a lot of the time and a lot of the work I do.
00:01:51
◼
►
And so it's kind of difficult to sometimes have a sense of belonging, to have a sense
00:01:55
◼
►
of camaraderie or a team or whatever you want to call it.
00:01:59
◼
►
And there's things you can do around that.
00:02:00
◼
►
But one of the things I kind of like is when I wear, wearing a t shirt, to me anyway, can
00:02:06
◼
►
kind of feel like I'm part of something bigger, that it draws a sense of connection.
00:02:10
◼
►
And you know, when I'm out, it's kind of like, I'm in on an in joke when I'm wearing like,
00:02:16
◼
►
when I'm, you know, today I'm wearing my hypercritical t shirt, for example.
00:02:19
◼
►
Like when I'm wearing that, like, most people have no idea what that is.
00:02:22
◼
►
But if someone does, then I know they're my kind of people.
00:02:25
◼
►
And even if they don't, like, it's kind of cool that I have, I sort of, and I know the
00:02:29
◼
►
secret handshake, I know what's going on there.
00:02:31
◼
►
And that's why I kind of I like that sense of that sense of community that kind of having
00:02:35
◼
►
a, I guess you could say, uniform for encourages and kind of allows.
00:02:39
◼
►
And so that's kind of a, it's kind of a fun thing.
00:02:42
◼
►
And so that's kind of I was like, well, you know, I have all these other ones for all
00:02:46
◼
►
the various things that I like, the things that I enjoy, different products I've been
00:02:50
◼
►
associated with, etc. And so I thought, "Well, why don't I make one for developing perspective?"
00:02:54
◼
►
It's a show that's been around for over two years now. This is episode 140, I think. It's
00:03:02
◼
►
been around enough that I think it's probably worth the kind of thing to have a shirt for,
00:03:06
◼
►
have something that I can wear, and that I can, you know, it's just sort of something
00:03:10
◼
►
that I own, the community that I'm, I guess, directly part of, if not at the center of.
00:03:15
◼
►
The design was actually done by friend of mine, Jory Raphael, who you may know from
00:03:18
◼
►
things like symbolicons.
00:03:20
◼
►
He is just a graphic designer, sort of vector artwork genius.
00:03:25
◼
►
And when I decided on kind of generally what I wanted to do, I knew he would be the right
00:03:28
◼
►
person to sort of take that and turn it into an actual design.
00:03:32
◼
►
And so the design is pretty straightforward.
00:03:34
◼
►
It's an underscore followed by a coffee mug.
00:03:37
◼
►
It's actually my favorite coffee mug, which is, I'll have a picture of it in the show
00:03:41
◼
►
But this particular coffee mug I got when I was visiting the Cupertino Apple
00:03:45
◼
►
Campus a couple years ago, and it's a perfect shape
00:03:48
◼
►
for drinking warm beverages.
00:03:50
◼
►
If you're not into coffee, imagine it as a teacup, hot chocolate,
00:03:53
◼
►
whatever floats your boat.
00:03:55
◼
►
But that's kind of the concept.
00:03:56
◼
►
And it was just something that I thought was a little bit geeky,
00:03:58
◼
►
a little bit understated, and a little bit silly,
00:04:01
◼
►
which is kind of one of the things that I enjoy in a t-shirt,
00:04:05
◼
►
and something that I think is kind of fun.
00:04:07
◼
►
The underscore at the beginning is, of course,
00:04:08
◼
►
a reference to the fact that I now have a nickname, which
00:04:10
◼
►
is kind of a funny thing.
00:04:11
◼
►
Because for my entire life, I don't
00:04:13
◼
►
think I've ever had a nickname.
00:04:14
◼
►
The closest thing was being called Dave rather than David,
00:04:17
◼
►
which I don't mind, which I guess is only vaguely
00:04:20
◼
►
But for better or worse, I've ended up
00:04:21
◼
►
being known as underscore in this community.
00:04:23
◼
►
I think that sort of definitely peaked when I was recently--
00:04:26
◼
►
I think it was Dr. Drang in one of an article
00:04:29
◼
►
he referenced to me.
00:04:30
◼
►
He referred to me, and then he abbreviated my name
00:04:33
◼
►
as UDS, underscore David Smith.
00:04:35
◼
►
So I have new initials, too.
00:04:36
◼
►
So anyway, so that's kind of the concept.
00:04:39
◼
►
It's fairly straightforward, fairly simple.
00:04:41
◼
►
It's just a basic design.
00:04:42
◼
►
There'll be a link in the show notes,
00:04:44
◼
►
or you can go to teespring.com/underscore
00:04:46
◼
►
to look at it and to buy it.
00:04:49
◼
►
The price is $1,459, which is kind of a play on words
00:04:52
◼
►
on the length of a-- my target length for a Developing
00:04:55
◼
►
Perspective episode, which at that point,
00:04:58
◼
►
it's really not for the money.
00:05:00
◼
►
I'm not doing this to make a lot of money.
00:05:01
◼
►
It's the kind of thing that's basically-- it's
00:05:03
◼
►
a little bit above cost.
00:05:04
◼
►
Essentially, my goal is to break even on the design
00:05:06
◼
►
and the actual shipping of it.
00:05:08
◼
►
My goal is to get more people to have it.
00:05:13
◼
►
And so that's kind of it.
00:05:16
◼
►
So anyway, I'll have a link in the show notes if that's something that you've been interested in, or if that's something that you want.
00:05:17
◼
►
They should ship in time, I think for most of the fall conferences.
00:05:21
◼
►
It should probably be arriving somewhere in mid-September when the campaign finishes in about, I think it finishes next week.
00:05:24
◼
►
It'll just do a short one week kind of campaign for it.
00:05:30
◼
►
So anyway, if that's your thing, great.
00:05:34
◼
►
If not, I'm going to get into the rest of the topic for the show.
00:05:35
◼
►
And that topic today is going to be, I'm going to talk about beta testing.
00:05:37
◼
►
which is I always try-- this is a really tricky time for me
00:05:39
◼
►
in terms of the time of year for coming up with new topics,
00:05:42
◼
►
coming up with things to say, because a lot of what
00:05:44
◼
►
I'm working on I can't talk about.
00:05:46
◼
►
A lot of the new exciting things that are coming out
00:05:48
◼
►
are either under NDA or unannounced.
00:05:50
◼
►
And so we're in kind of this middle ground.
00:05:51
◼
►
And so it's always kind of tricky.
00:05:52
◼
►
But I think whenever that happens, the best thing to do
00:05:55
◼
►
is just to fall back on what it is that I'm actually
00:05:56
◼
►
doing it right now.
00:05:57
◼
►
And right now, that's beta testing.
00:05:59
◼
►
Right now, that is working on the final phases
00:06:02
◼
►
of getting my apps all queued up to go out to Apple.
00:06:06
◼
►
And so I'm going to talk a little bit about beta testing
00:06:08
◼
►
generally, and then a few things that I've learned,
00:06:10
◼
►
or situations, or struggles, or problems that you run into,
00:06:12
◼
►
and what I do with that.
00:06:14
◼
►
So first, I'm going to talk about what beta testing is
00:06:17
◼
►
in iOS and how that works.
00:06:19
◼
►
So generally, beta testing is the process
00:06:21
◼
►
of taking your app right before it's ready for launch
00:06:25
◼
►
and seeding it out to a small group of people
00:06:27
◼
►
to get feedback on to identify bugs, problems, issues,
00:06:30
◼
►
whatever it may be, and to do it in a way that
00:06:34
◼
►
is constructive, hopefully.
00:06:35
◼
►
Because ultimately, the goal, at least in my mind, for beta
00:06:37
◼
►
testing, is to improve the product every time
00:06:41
◼
►
you get around out.
00:06:43
◼
►
There's some amount of it that is perhaps marketing oriented,
00:06:46
◼
►
that it's certainly some amount of beta testing's value
00:06:49
◼
►
is that it puts it in the hands of people who will hopefully
00:06:52
◼
►
talk about it.
00:06:52
◼
►
Mostly what you're really trying to do
00:06:54
◼
►
is try to collect feedback from a small group
00:06:56
◼
►
of trusted, interesting people who can help craft and sort
00:07:02
◼
►
of direct the app in that final stage.
00:07:06
◼
►
So typically, you've got the idea, the concept,
00:07:10
◼
►
and then usually in sort of like 99% of the implementation
00:07:12
◼
►
built. And now you're going to show it to some people
00:07:14
◼
►
and get their feedback on it.
00:07:17
◼
►
And that's always a dangerous thing.
00:07:19
◼
►
And I'll say that, and mostly just in that,
00:07:22
◼
►
it's always tricky when you open up an app to somebody else
00:07:25
◼
►
and you're getting their feedback on it.
00:07:27
◼
►
It can change your own view on that app.
00:07:30
◼
►
It can change your direction for that app, for better or worse.
00:07:36
◼
►
And so I think something I always want to emphasize
00:07:39
◼
►
is it's important to be thoughtful about who
00:07:41
◼
►
you show it to.
00:07:42
◼
►
And this is something that I personally try--
00:07:45
◼
►
my typical opinion on this is I show it
00:07:48
◼
►
to a relatively small number of people.
00:07:51
◼
►
And they're people whose opinions I trust,
00:07:54
◼
►
who I really want to hear what they have to say,
00:08:00
◼
►
and whose opinions, I think, are things that I tend to agree with,
00:08:03
◼
►
are things that I would respect and respond well to.
00:08:07
◼
►
I'm not the kind of person who tends to do these kind of betas
00:08:10
◼
►
where you're trying to open it up to like 50 or 100 people,
00:08:13
◼
►
many of whom you don't actually know.
00:08:15
◼
►
They're just general customers.
00:08:17
◼
►
And sometimes that can be useful if you're
00:08:19
◼
►
solving a specific bug that customers have
00:08:22
◼
►
and you want to have them reproduce it.
00:08:23
◼
►
That makes sense to me.
00:08:24
◼
►
But for a lot of times, what I find that's difficult
00:08:26
◼
►
is the more and more beta testers you have,
00:08:28
◼
►
the more time and energy it takes to actually manage
00:08:30
◼
►
that process.
00:08:32
◼
►
And so I'm not the kind of person who tends to just like,
00:08:34
◼
►
every now and then I'll get requests on Twitter,
00:08:36
◼
►
things like that.
00:08:36
◼
►
People are like, hey, do you have any space in your beta
00:08:38
◼
►
group to beta test something?
00:08:40
◼
►
And I've almost done everything I've ever responded to someone
00:08:44
◼
►
like that and said, yeah, that's great.
00:08:45
◼
►
Let's put it in.
00:08:46
◼
►
Because what I want to do is I want to--
00:08:49
◼
►
in that critical phase of right before an episode is ready
00:08:52
◼
►
to launch, what I want to do is to narrow in rather than
00:08:55
◼
►
broaden out, necessarily.
00:08:56
◼
►
And this is just the way that I work.
00:08:57
◼
►
Some people do it differently, so the more the merrier.
00:09:00
◼
►
Kind of open betas and things where you can get
00:09:02
◼
►
lots of feedback.
00:09:03
◼
►
What I'm trying to do is I'm trying to narrow down
00:09:06
◼
►
my feedback and to be able to manage and really invest
00:09:08
◼
►
and to have a lot of deep discussion back and forth,
00:09:11
◼
►
which I just can't do with 50 people.
00:09:13
◼
►
You can kind of collect crash reports
00:09:14
◼
►
and those kinds of things, maybe things
00:09:15
◼
►
that can be aggregated.
00:09:17
◼
►
But for me, what I want to do is I want to have
00:09:19
◼
►
a couple of conversations with people who I know
00:09:22
◼
►
will have interesting feedback and can really help me
00:09:24
◼
►
improve my product.
00:09:26
◼
►
So what I do is you collect a group of people,
00:09:28
◼
►
and you decide who it's going to be, and you send it out.
00:09:31
◼
►
In iOS, the way you actually manage that is you
00:09:33
◼
►
have to do what's called an ad hoc build, which
00:09:35
◼
►
is if you've never done it, you just
00:09:37
◼
►
go into the provisioning profile area in the developer portal.
00:09:42
◼
►
And you set up-- it's a different signing certificate
00:09:45
◼
►
It's similar to what you do when you're doing an app store build,
00:09:48
◼
►
but you have to specify that the actual device
00:09:50
◼
►
IDs for every device that you want that to run in--
00:09:53
◼
►
and this gets into the whole kind of the way
00:09:55
◼
►
that an iOS device is very locked down.
00:09:57
◼
►
And so the only way to run an app is if it's signed.
00:09:59
◼
►
And the only way to get it signed
00:10:00
◼
►
is to have the-- you have to create an ad hoc
00:10:03
◼
►
profile for the specific device that you want it to run on.
00:10:06
◼
►
So you have to go and collect those IDs from everybody.
00:10:08
◼
►
And most people in this community,
00:10:10
◼
►
whether in the press or the developers,
00:10:11
◼
►
should have no problems collecting that device ID.
00:10:14
◼
►
If you need to do it from someone else who's
00:10:15
◼
►
not quite as technical, there's all kinds of different tools
00:10:18
◼
►
and things you can do to collect that.
00:10:20
◼
►
Often the best is to use one of the tools, programs,
00:10:24
◼
►
or services like TestFlight or Hockey,
00:10:28
◼
►
which are two different services that
00:10:30
◼
►
do very similar stuff around this kind of beta testing
00:10:32
◼
►
and sending out builds.
00:10:34
◼
►
But one of the things they often can do for you is they can collect
00:10:37
◼
►
devices directly.
00:10:38
◼
►
So you'll invite someone with an email.
00:10:39
◼
►
They'll hit a button on the device,
00:10:41
◼
►
and it'll pull down all the relevant information
00:10:44
◼
►
that you're going to need.
00:10:45
◼
►
So you prepare the build.
00:10:47
◼
►
I definitely recommend that you send the builds
00:10:48
◼
►
to people using one of those two services, TestFlight or Hockey.
00:10:51
◼
►
I've used them both.
00:10:52
◼
►
They're both pretty good.
00:10:53
◼
►
You kind of go back and forth on which one's better.
00:10:55
◼
►
I don't really know.
00:10:56
◼
►
I think hockey is more of the kind of-- it's
00:10:59
◼
►
a service you pay for.
00:11:00
◼
►
It's a bit more robust in that way.
00:11:03
◼
►
TestFlight is kind of more kind of-- it's a free service.
00:11:07
◼
►
You can do lots of things with it.
00:11:08
◼
►
So it's kind of sometimes nice for experimentation
00:11:11
◼
►
or the first time you adopt this or if it's just
00:11:13
◼
►
something you're doing on a hobby, maybe.
00:11:14
◼
►
But you get what you pay for in a lot of ways.
00:11:17
◼
►
And so it's just kind of a decision you have to make.
00:11:20
◼
►
But whichever service you're going to use,
00:11:21
◼
►
You put that-- you make your build, do it in Xcode,
00:11:25
◼
►
upload it up to one of these services,
00:11:26
◼
►
put in some useful distribution notes,
00:11:28
◼
►
and then you can just send people links.
00:11:30
◼
►
They take that link, and they download it onto their device
00:11:32
◼
►
and they can get you feedback.
00:11:34
◼
►
The actual way in which you collect feedback is-- a lot
00:11:37
◼
►
of people do it very differently.
00:11:38
◼
►
I've been recently on some great betas.
00:11:40
◼
►
People have glassboard rooms on it
00:11:43
◼
►
where you are collecting feedback there.
00:11:45
◼
►
You have a discussion forum for beta testers.
00:11:47
◼
►
You can have people email you.
00:11:49
◼
►
They can IM you.
00:11:50
◼
►
they can do kind of whatever makes sense for you.
00:11:53
◼
►
And it's kind of hard to be prescriptive about that
00:11:55
◼
►
because I think some people just react differently
00:11:58
◼
►
to their feedback.
00:12:00
◼
►
For me, I just want to, whatever is most convenient
00:12:02
◼
►
for the person giving the feedback,
00:12:03
◼
►
a lot of times it'll take emails,
00:12:05
◼
►
it'll be taking items from people,
00:12:07
◼
►
high messages, whatever it is.
00:12:08
◼
►
And my goal is to try and understand the problems
00:12:11
◼
►
that they're having, the things that are confusing,
00:12:13
◼
►
number one, the things that are crashing, number two,
00:12:17
◼
►
and the things that they wish it did.
00:12:20
◼
►
And I do it in that order because first, things
00:12:23
◼
►
that are confusing.
00:12:25
◼
►
No matter what, if it's confusing,
00:12:26
◼
►
that's very, very problematic for users,
00:12:29
◼
►
and especially for support.
00:12:30
◼
►
Where often, the worst thing you can ever have on a support
00:12:33
◼
►
thing is where someone asks for a feature to-- oh,
00:12:36
◼
►
I love the app, but I wish it did x.
00:12:38
◼
►
And it actually already does x, but that
00:12:40
◼
►
isn't clear in the interface, or it isn't well communicated.
00:12:43
◼
►
And so making sure that things are clear, very important.
00:12:46
◼
►
Making sure things are stable and solid.
00:12:48
◼
►
Definitely in the beta phase, I highly
00:12:50
◼
►
recommend putting in some kind of crash reporting.
00:12:52
◼
►
I think both TestFlightHockey does it.
00:12:54
◼
►
You can use Crashlytics, whatever, PLCrashReporter.
00:12:56
◼
►
Whatever makes sense for you can incorporate crash reporting.
00:13:01
◼
►
But it's really important in this phase
00:13:03
◼
►
to be collecting that kind of stuff.
00:13:04
◼
►
Because you'll find all the little bugs, all the little edge
00:13:05
◼
►
cases, all the little things that you didn't expect
00:13:08
◼
►
would happen, hopefully happen in your beta phase
00:13:10
◼
►
so it doesn't happen broadly in production.
00:13:13
◼
►
And then the last thing, which is really interesting,
00:13:15
◼
►
is when you put an app out into actual people's hand
00:13:18
◼
►
and you get them actually using it,
00:13:20
◼
►
you'll get feedback about the things
00:13:21
◼
►
that they wish it did additionally.
00:13:23
◼
►
And this is, I've found, especially in Pod Wrangler,
00:13:26
◼
►
what I wish I'd been doing most heavy beta testing for recently.
00:13:29
◼
►
You find there are things that you
00:13:31
◼
►
can add to your application.
00:13:33
◼
►
And the best are the ones that are
00:13:35
◼
►
these tiny little small features that have big impacts, things
00:13:39
◼
►
that you can relatively easily add on,
00:13:41
◼
►
that you find that your app is doing 99% of what they want.
00:13:44
◼
►
But they just wish to, oh, I wish I just
00:13:46
◼
►
did this one extra thing.
00:13:48
◼
►
And adding that will make the app so much more useful,
00:13:50
◼
►
so much more functional for that person.
00:13:52
◼
►
And any time you can find this, it's a huge win and a big
00:13:56
◼
►
benefit to find during the beta phase, so that you can
00:13:59
◼
►
add that in.
00:14:00
◼
►
And when you release it to the community at large, you can
00:14:05
◼
►
take advantage of that.
00:14:06
◼
►
And you can have a nice big splashy launch that has a lot
00:14:10
◼
►
of the features, a lot of the things that you could squeeze
00:14:12
◼
►
into in that time.
00:14:15
◼
►
Anyway, so that's sort of beta testing.
00:14:16
◼
►
That's kind of how the process works, kind of what I do.
00:14:19
◼
►
I think it's important.
00:14:20
◼
►
I think you can overemphasize it.
00:14:22
◼
►
I think it's just something that you
00:14:23
◼
►
want to do to collect that feedback,
00:14:24
◼
►
to understand how your app's good, how it's bad,
00:14:28
◼
►
and what you can do to make it better.
00:14:31
◼
►
Anyway, that's it for today's show.
00:14:32
◼
►
As always, if you have questions, comments, concerns,
00:14:34
◼
►
complaints, I'm on Twitter @_davidsmith,
00:14:37
◼
►
david@developingperspective.com.
00:14:38
◼
►
Otherwise, have a great weekend.
00:14:40
◼
►
If you're in the US, have a great Labor Day.
00:14:42
◼
►
Long weekend.
00:14:43
◼
►
And otherwise, have a great week.
00:14:44
◼
►
and I'll see you next week. Bye.