Under the Radar

00:00:00 ◼   ► Welcome to Under the Radar, a show about independent iOS and app development.

00:00:03 ◼   ► I'm Marco Arment.

00:00:04 ◼   ► And I'm David Smith.

00:00:06 ◼   ► Under the Radar is never longer than 30 minutes, so let's get started.

00:00:09 ◼   ► So last week we mentioned at the end how we were going to tell this week tips and stories

00:00:15 ◼   ► and setups of how to run your own Linux servers.

00:00:18 ◼   ► Because this is now, you know, in the wake of the parse announcement last week, this

00:00:21 ◼   ► is now kind of in the news, and we think it's kind of a core competency that many app developers

00:00:25 ◼   ► should know just how to run your own servers.

00:00:27 ◼   ► And once you know this, it enables your apps to be better,

00:00:31 ◼   ► to be more capable.

00:00:33 ◼   ► And if you do kind of a standard server route

00:00:36 ◼   ► without doing these kind of more abstract proprietary services

00:00:39 ◼   ► like Parse, it gives you a lot of flexibility

00:00:41 ◼   ► with what you can do in the future.

00:00:43 ◼   ► And it kind of decouples you from depending

00:00:45 ◼   ► on a particular third party to keep their platform going.

00:00:49 ◼   ► Yeah.

00:00:49 ◼   ► And this is probably a fair disclaimer to say.

00:00:51 ◼   ► This is going to be an episode that

00:00:52 ◼   ► that goes a bit into the weeds,

00:00:54 ◼   ► like down at some pretty low level stuff,

00:00:57 ◼   ► and if that's for you, awesome.

00:00:59 ◼   ► If it isn't for you, maybe stick around,

00:01:01 ◼   ► you might learn something, and know that some episodes

00:01:04 ◼   ► are gonna be high level,

00:01:05 ◼   ► some episodes are gonna be low level,

00:01:07 ◼   ► and hopefully we can have a nice balance as a result.

00:01:10 ◼   ► So at a high level, it's probably worth starting

00:01:12 ◼   ► of just kind of talking about the spectrum of hosting

00:01:16 ◼   ► before we kind of end up with why we do it

00:01:18 ◼   ► the way we do it.

00:01:19 ◼   ► So ultimately, hosting, in many ways, comes down to levels of abstraction, like how abstracted

00:01:26 ◼   ► away the actual computer that you're running your servers and your backends and systems

00:01:32 ◼   ► on, how much that is abstracted away from you.

00:01:35 ◼   ► So at the lowest level, you have what I think would be called co-location or something like

00:01:39 ◼   ► that, where you have a computer that's your computer that you went to Best Buy or whatever

00:01:45 ◼   ► to buy, and you're taking it and you're putting it somewhere, typically in some kind of data

00:01:49 ◼   ► center that you put in a data center so that it has reliable power and reliable internet

00:01:53 ◼   ► and reliable air conditioning and all those types of things. But like it's your machine.

00:01:57 ◼   ► Most people don't do that. I would really not recommend doing something like that.

00:02:02 ◼   ► Most people with the very least we do, but I think most people call like bare metal

00:02:06 ◼   ► managed servers where it's not really your machine. It's a service that they buy machines,

00:02:12 ◼   ► they manage machines. If a hard drive dies in the machine, they'll go and fix it for you.

00:02:16 ◼   ► but it's you have actual like a dedicated piece of hardware

00:02:21 ◼   ► you know a set of hard drives, a motherboard, a CPU that's exactly yours

00:02:26 ◼   ► and you probably don't want that either I've gone down that road

00:02:29 ◼   ► it's fine but it has a lot of

00:02:32 ◼   ► difficulties that you aren't really offset by its costs both

00:02:37 ◼   ► in monetary and like ease of use so most people

00:02:41 ◼   ► who do any kind of hosting themselves will use something that's more

00:02:45 ◼   ► like a VPS, which stands for Virtual Private Server, which is basically like, it's a machine

00:02:52 ◼   ► from your app's perspective.

00:02:53 ◼   ► You are, your app sees a hard drive, it sees a CPU, it sees memory, but all of those aren't

00:02:59 ◼   ► actually physical things.

00:03:02 ◼   ► It's running on a shared environment with a bunch of other systems that all see the

00:03:06 ◼   ► same things and share resources.

00:03:09 ◼   ► And you can use something like Linode for this, or DigitalOcean is another one that's

00:03:14 ◼   ► That's pretty common in many ways.

00:03:16 ◼   ► EC2 kind of runs like this, but it's a bit different.

00:03:19 ◼   ► But you have these kinds of systems where you have a virtual Linux machine that you

00:03:23 ◼   ► can do, and you have some nice advantages with that because it's virtual.

00:03:27 ◼   ► You can do things like pick it up and move it around.

00:03:30 ◼   ► You take an image of your hard drive and you can copy it onto another one, and things like

00:03:35 ◼   ► that that are really easy to manage.

00:03:37 ◼   ► And from a cost perspective, it's way cheaper because you're sharing server resources.

00:03:43 ◼   ► And then above that you have things like virtual server environments, things like Azure or

00:03:46 ◼   ► Heroku, or total abstractions like CloudKit or Parse or those types of things that are

00:03:52 ◼   ► completely abstracting away from you.

00:03:55 ◼   ► And like I said, for most people you probably want to start at a VPS.

00:03:57 ◼   ► Like you want to find out where you're comfortable if the thought of administering a Linux server

00:04:02 ◼   ► even after this episode is completely overwhelming.

00:04:05 ◼   ► Like maybe go to something like Heroku or Azure where you have a little bit of control

00:04:10 ◼   ► over what you're doing, but mostly there's something there, but it's still at least general

00:04:15 ◼   ► purpose. But I think we both run VPS servers because once you know what you're doing, it's

00:04:22 ◼   ► really cheap, it's really efficient, and you have a lot of flexibility that you could move

00:04:27 ◼   ► providers, you can spin up new servers and take them down really quickly and easily,

00:04:32 ◼   ► and especially the cost is amazing. It's incredible the amount of performance you can get from

00:04:38 ◼   ► from a really basic setup.

00:04:39 ◼   ► I mean, at this point, I think I administer

00:04:41 ◼   ► 19 virtual private servers or something like that

00:04:44 ◼   ► to run my apps, and it's a pretty big number,

00:04:47 ◼   ► but the actual load of that,

00:04:49 ◼   ► of actually doing that is pretty straightforward.

00:04:51 ◼   ► - Oh yeah, and I mean, things have gotten so cheap

00:04:53 ◼   ► over the years, like I now pay for all of Overcast,

00:04:56 ◼   ► which is something like 11 VPS instances on Linode.

00:05:01 ◼   ► I now pay for all of them what I used to pay

00:05:03 ◼   ► for one database server for Instapaper back in the day,

00:05:07 ◼   ► and that wasn't that long ago.

00:05:08 ◼   ► That was like five years ago.

00:05:10 ◼   ► So what you can get now at the VPS level

00:05:12 ◼   ► is just incredibly good.

00:05:14 ◼   ► I was the biggest fan of bare metal dedicated servers before

00:05:17 ◼   ► but now even I have converted because you can just,

00:05:20 ◼   ► you know, the competition especially between Linode

00:05:22 ◼   ► and DigitalOcean has really made incredibly fast

00:05:26 ◼   ► hardware available for almost no money.

00:05:28 ◼   ► And it also makes it much easier to learn

00:05:30 ◼   ► because it lets you, and we'll get into this

00:05:32 ◼   ► in a little bit, but it lets you,

00:05:34 ◼   ► you know, if you can just like,

00:05:35 ◼   ► in a simple command on a web interface,

00:05:38 ◼   ► so not involving buying anything,

00:05:40 ◼   ► not involving emailing anybody

00:05:42 ◼   ► or placing any orders with anybody,

00:05:44 ◼   ► just in a web interface you can quickly create

00:05:47 ◼   ► an instance of a server, then you can do things like,

00:05:49 ◼   ► oh, let me see how easy is it to set up PHP or whatever.

00:05:53 ◼   ► Like you can do things like that,

00:05:54 ◼   ► or you can say, I wonder how,

00:05:56 ◼   ► whether this new distro of Linux makes more sense to me,

00:05:59 ◼   ► or I wonder if my software will run on this other distro

00:06:02 ◼   ► or this other web server or whatever.

00:06:04 ◼   ► you can just spin up a new instance in minutes

00:06:07 ◼   ► and run your install script and poke around,

00:06:11 ◼   ► try things out, see what works, see what breaks.

00:06:15 ◼   ► And if at any point you decide this wasn't for you

00:06:18 ◼   ► or this didn't go well, you can just delete the instance

00:06:20 ◼   ► and you've paid up to that point maybe 50 cents.

00:06:24 ◼   ► Like it's crazy how easy this is

00:06:27 ◼   ► to both administer servers and to learn

00:06:29 ◼   ► how to administer servers now in the world of VPSs.

00:06:33 ◼   ► - Yeah, 'cause it's lovely to think of,

00:06:34 ◼   ► most of these are charged by hour of use

00:06:37 ◼   ► or something like that, and so you can,

00:06:39 ◼   ► they're just kind of like utility in that way,

00:06:41 ◼   ► that you can just spin them up when you want 'em,

00:06:43 ◼   ► spin, take 'em down when you don't want 'em anymore,

00:06:45 ◼   ► and like, it's very straightforward and easy to do

00:06:48 ◼   ► in a way that is kind of scary the first time you do it,

00:06:51 ◼   ► that you just like hit a button and it's like,

00:06:52 ◼   ► oh, here's a machine.

00:06:54 ◼   ► Go ahead and SSH into it and get started.

00:06:57 ◼   ► - Yeah, I mean, it's so easy that like,

00:06:58 ◼   ► you know, one way you could've done it before

00:07:00 ◼   ► is by running something like VMware Fusion or Parallels

00:07:02 ◼   ► on your computer and virtualizing little Linux setups

00:07:06 ◼   ► on your computer and doing that.

00:07:08 ◼   ► But now it's so cheap and easy to go do it

00:07:10 ◼   ► on Linode or DigitalOcean that it's actually easier

00:07:14 ◼   ► to do that than it is to do it on your local computer.

00:07:18 ◼   ► Now I do all my experimentation on VPSs.

00:07:20 ◼   ► I don't even have virtualization software installed

00:07:21 ◼   ► anymore on my Mac.

00:07:23 ◼   ► - Yeah, and I'm the same way.

00:07:24 ◼   ► The only thing I use my Mac for when I'm doing

00:07:26 ◼   ► this kind of thing is the first level of,

00:07:30 ◼   ► the nice thing about using something like Linux

00:07:32 ◼   ► is that the basic underpinnings underneath our Macs

00:07:36 ◼   ► are like it's some kind of Unix-based thing.

00:07:40 ◼   ► I couldn't tell you all of the history of it,

00:07:41 ◼   ► but functionally, your Mac is running

00:07:43 ◼   ► the same basic stack kernel types of things

00:07:46 ◼   ► that your ultimate server is gonna run.

00:07:49 ◼   ► And so I use my Mac because I can learn

00:07:52 ◼   ► and kind of prototype and try some of the things

00:07:54 ◼   ► that ultimately I'm gonna run on my main server locally.

00:07:57 ◼   ► I have a development database that runs on my machine.

00:08:01 ◼   ► I have a web server that runs on my local machine,

00:08:03 ◼   ► which makes prototyping really quick.

00:08:05 ◼   ► But as soon as I wanna do anything

00:08:06 ◼   ► that's going to be simulating my end environment,

00:08:10 ◼   ► I just spin up an instance and run it.

00:08:12 ◼   ► And I have a couple of just instances

00:08:14 ◼   ► that you just bring up, like try it out.

00:08:16 ◼   ► If it works, great.

00:08:17 ◼   ► If it doesn't, you can just delete it and start again.

00:08:20 ◼   ► So once you've decided you're gonna build

00:08:23 ◼   ► a Linux instance and get started,

00:08:26 ◼   ► and that actual process is pretty straightforward.

00:08:29 ◼   ► you'll go to a virtual private server provider, and just sort of you'll set up your instance,

00:08:35 ◼   ► it'll give you some credentials and an IP address and a few things that you'll have

00:08:39 ◼   ► for like how to log into it, and then you just log in and get started. And for most

00:08:43 ◼   ► applications, you're going to install at least two things on that. You're going to install

00:08:47 ◼   ► a web server and a database of some kind. And starting out, you could probably put them

00:08:52 ◼   ► both on the same machine. Eventually, you'll probably do a thing where you'll split the

00:08:57 ◼   ► the web database onto its own machine

00:08:59 ◼   ► and put one or more web servers in front of it.

00:09:02 ◼   ► But at the very least, you're gonna install

00:09:04 ◼   ► some kind of web server environment,

00:09:05 ◼   ► some kind of database.

00:09:07 ◼   ► The web server that you choose is entirely up to you.

00:09:09 ◼   ► Like, I use Ruby on Rails for all my backends.

00:09:12 ◼   ► I think all of yours are in PHP.

00:09:14 ◼   ► - Yeah, I mean, and I have many opinions

00:09:17 ◼   ► on how you choose these tools.

00:09:20 ◼   ► I will summarize them, for the sake of this

00:09:21 ◼   ► being a short show, I will summarize them

00:09:23 ◼   ► as what you want most of all when choosing server software,

00:09:27 ◼   ► if you don't wanna be administering and tweaking

00:09:30 ◼   ► with your server constantly,

00:09:31 ◼   ► what you want is old, boring, and popular.

00:09:35 ◼   ► Those three things, old, boring, and popular.

00:09:38 ◼   ► New and trendy does not necessarily always mean better.

00:09:41 ◼   ► In this case, you know, especially like,

00:09:44 ◼   ► what makes some people think that the new stuff is better

00:09:47 ◼   ► is vague claims of it being quote, scalable,

00:09:51 ◼   ► And beware of any claims of things being scalable, because in reality, scaling web services up,

00:09:58 ◼   ► not only is it the kind of thing that you will probably never need for almost any project

00:10:02 ◼   ► that you do, but when you do need it, it is way more complicated. And there is no silver

00:10:08 ◼   ► bullet for scaling. There is nothing that automatically scales, unless it also automatically

00:10:13 ◼   ► scales all of your money away. But there is nothing that automatically and easily scales.

00:10:18 ◼   ► Scaling is always complicated and always tricky. It's just, you know, the new things that

00:10:22 ◼   ► claim scalability just scale differently. And the challenges are just different challenges.

00:10:29 ◼   ► The newer something is, the trendier or newer something is, that means, you know, the fewer

00:10:35 ◼   ► people have experience with it and the fewer large installations it has. And so that means

00:10:43 ◼   ► there's less knowledge in the world about how it fails or what the anti-patterns of

00:10:47 ◼   ► it might be. So as you go lower level in the stack, being this kind of conservative, preferring

00:10:53 ◼   ► things that are old, boring and popular becomes more important. So like if you want to use

00:10:58 ◼   ► like a brand new cutting edge CSS library, go for it. It doesn't matter really. You

00:11:02 ◼   ► know, if it works for you, go for it. Your risk is pretty low there. But you probably

00:11:07 ◼   ► would not want to use a beta version of the Linux kernel. You know, so like as you go

00:11:12 ◼   ► down the stack, you want to be more conservative because it's just kind of a bad idea to

00:11:17 ◼   ► to take risks lower down.

00:11:18 ◼   ► And I include things like the database in that.

00:11:21 ◼   ► MySQL and Postgres are great.

00:11:24 ◼   ► They're battle tested, they are old, boring, and popular.

00:11:27 ◼   ► That is what you want in your database.

00:11:30 ◼   ► Just like you wouldn't want a trendy new kind of brakes

00:11:32 ◼   ► on your car.

00:11:33 ◼   ► Like you want time tested brakes that everyone knows works

00:11:37 ◼   ► and everyone knows that you aren't gonna be pushing

00:11:38 ◼   ► the boundaries of them.

00:11:40 ◼   ► You want a really boring database.

00:11:43 ◼   ► And I would say you want a really boring language too

00:11:46 ◼   ► because really new cutting edge languages are still in flux.

00:11:51 ◼   ► And so if you want to have this kind of like set it

00:11:54 ◼   ► and forget it kind of server approach,

00:11:56 ◼   ► you don't want to have to deal with

00:11:58 ◼   ► a young languages frequent changes

00:12:01 ◼   ► and frequent needs for being updated.

00:12:04 ◼   ► You know, you want to minimize anything

00:12:06 ◼   ► that will require your attention

00:12:08 ◼   ► or that might break your app.

00:12:09 ◼   ► So you wanna minimize the need for, you know,

00:12:12 ◼   ► backwards incompatible updates and things like that

00:12:15 ◼   ► or crashes and having to install little bug fixes constantly for the language or database

00:12:20 ◼   ► you're using, you want to avoid all of that. Because new really, new means untested or

00:12:26 ◼   ► less tested. Another thing that you hit with new stuff is in the language layer, new means

00:12:33 ◼   ► that you have fewer third party libraries and generally less third party code. And this

00:12:38 ◼   ► is a problem like, I've been using PHP forever and PHP third party code is mostly garbage,

00:12:43 ◼   ► but there is so much of it out there that any need that I have, I can find something

00:12:49 ◼   ► to do it. And it isn't always good, but I can find something. And as I've started

00:12:53 ◼   ► writing some newer things in Go, and Go I like as a language conceptually, but because

00:12:58 ◼   ► it's such a younger language and it isn't that popular yet, and might never become popular,

00:13:04 ◼   ► who knows, but because it's younger and less popular, there is very little third-party

00:13:08 ◼   ► already go code for some very common things that I keep needing to do. And I keep, every

00:13:13 ◼   ► time I need to write something new in Go, I hit this wall of, "Oh, I'll just load up

00:13:17 ◼   ► the markdown module." And I look and there's like three of them that are all kind of half

00:13:22 ◼   ► baked and, you know, and I haven't listened in a while for markdown, so maybe that's a

00:13:25 ◼   ► bad example. But I keep hitting that wall whenever I do anything in a new language of,

00:13:29 ◼   ► "Oh, I just want to see how, oh, how do I resize an image? Oh, well, you kind of can't,

00:13:33 ◼   ► "Oh, unless you do this."

00:13:35 ◼   ► So it's always easier to use old, boring, and popular stuff.

00:13:39 ◼   ► It may be less interesting,

00:13:40 ◼   ► and if you are the kind of person

00:13:42 ◼   ► who wants cutting edge languages

00:13:45 ◼   ► and who finds that satisfying,

00:13:46 ◼   ► then you will trade time and maintenance for that.

00:13:50 ◼   ► And so for the goal of low maintenance

00:13:52 ◼   ► and low server maintenance,

00:13:55 ◼   ► I highly suggest that you use other parts of your app

00:13:58 ◼   ► to get the kind of newness out of your system.

00:14:01 ◼   ► Like yeah, use the higher level parts

00:14:03 ◼   ► use maybe the iOS side of it if you have an iOS and a web app, use that side of it to

00:14:07 ◼   ► do new cool stuff and make the server as boring, old and popular as possible. Because old,

00:14:12 ◼   ► boring, popular stuff works. It really works. It hardly ever breaks and you will never be

00:14:19 ◼   ► the biggest person using it because long before, you know, even suppose that you build something

00:14:24 ◼   ► with old, boring stuff. Suppose it gets like the size of Etsy, right? Etsy is huge. It's

00:14:29 ◼   ► It's not the biggest website on the internet,

00:14:31 ◼   ► but it's huge, right?

00:14:32 ◼   ► It gets the size of Etsy.

00:14:34 ◼   ► If you get that big, will you be the biggest user

00:14:38 ◼   ► of anything in your stack?

00:14:40 ◼   ► Because once you're the biggest user

00:14:41 ◼   ► of your database or something,

00:14:43 ◼   ► then you basically become a developer of that database.

00:14:46 ◼   ► You will hit all the bugs, you will have to go in

00:14:48 ◼   ► and make patches yourself to customize things

00:14:50 ◼   ► that aren't working well or aren't working right for you.

00:14:54 ◼   ► So Facebook rolls out tons of patches and things.

00:14:57 ◼   ► Twitter has rolled out tons of patches

00:14:58 ◼   ► things like Memcache, I mean like there's all these big companies, all these big web

00:15:02 ◼   ► companies end up having to basically partially take over parts of the stack they depend on

00:15:07 ◼   ► open source wise and make their own contributions and fixes because they hit tons of bugs. You

00:15:12 ◼   ► never want to be the biggest user of anything in your stack. And you know, when you start

00:15:16 ◼   ► out of course you won't be the biggest user of basically anything, but you could become

00:15:19 ◼   ► the biggest user. Like you'd be surprised, you know, you hear about new trendy things,

00:15:24 ◼   ► but you'd be surprised how few big companies

00:15:27 ◼   ► actually use them, so you never want to be the biggest

00:15:29 ◼   ► user of anything, you want Facebook to have fixed

00:15:32 ◼   ► every bug in that stack years before you ever hit it.

00:15:35 ◼   ► - Yeah, and I think we're both coming at it

00:15:37 ◼   ► from the perspective of being app developers,

00:15:39 ◼   ► like what makes our apps cool is not the backend stack.

00:15:42 ◼   ► That's got nothing to do with the thing that I'm gonna

00:15:45 ◼   ► be marketing my app as.

00:15:47 ◼   ► Like my customer shouldn't care, it's only,

00:15:51 ◼   ► it exists for the sole purpose of just getting

00:15:53 ◼   ► job done. Most of my backends are the most ridiculously boring, simple applications that

00:15:59 ◼   ► I can possibly make it. I put as much logic and smarts as I can in the apps itself. The

00:16:06 ◼   ► database is mostly just a nice wrapper for a database, and something that's simple and

00:16:10 ◼   ► straightforward that I can set up and then I never think about again. The only time I

00:16:16 ◼   ► ever really think about my servers is when there's some big security thing that I need

00:16:22 ◼   ► to make sure it was auto-updated,

00:16:24 ◼   ► or I'm adding major new functionality to the front end,

00:16:27 ◼   ► to my applications, and then I have to go back

00:16:30 ◼   ► and add something to it.

00:16:33 ◼   ► But otherwise, I just want it as simple and basic as can be.

00:16:35 ◼   ► It's just like a big database that I can put things in

00:16:39 ◼   ► and then later get them out.

00:16:41 ◼   ► And as long as I can do that, I'm good.

00:16:43 ◼   ► - And if anything, when you're using old, boring,

00:16:45 ◼   ► and popular, if anything ever goes wrong,

00:16:48 ◼   ► or if you have any questions whatsoever,

00:16:51 ◼   ► or anything comes up, you can just search the web

00:16:53 ◼   ► and you have hundreds of answers from the thousands

00:16:56 ◼   ► of other people who have used that already

00:16:58 ◼   ► and have documented it like three years earlier.

00:17:00 ◼   ► (laughs)

00:17:01 ◼   ► It's so much easier to make your server side stack

00:17:06 ◼   ► something that you can easily forget about most of the time

00:17:10 ◼   ► and you don't even have to deal with it.

00:17:12 ◼   ► - Exactly.

00:17:13 ◼   ► And I guess once you decide on which one you wanna do,

00:17:16 ◼   ► it's like the process from there is honestly

00:17:18 ◼   ► pretty straightforward.

00:17:19 ◼   ► you pick something, you install it. Usually these kind of things are a pretty straightforward

00:17:24 ◼   ► install process. Like, if you're, there should be some kind of conversion of like,

00:17:28 ◼   ► "Go get me MySQL, go get me Postgres, go get me Nginx, go get me Apache," and the server will go

00:17:36 ◼   ► get it, install it, configure it, you'll change a few configuration files, and you'll be up and

00:17:40 ◼   ► running because it's boring. And the process of getting it set up has been done by thousands of

00:17:47 ◼   ► people so everything's automated and scripted and the only thing that I'd mentioned there

00:17:51 ◼   ► is one thing that I learned early on is once I set up a server the first time for I always

00:17:58 ◼   ► document how I did it like the steps I did and then I will usually destroy what I did

00:18:02 ◼   ► the first time and do it again to make sure I actually know what I'm doing to avoid the

00:18:06 ◼   ► situation where down the road you are trying to like recreate a web server and you can

00:18:11 ◼   ► never get it to work quite right and it turns out it's because the first time you did it

00:18:15 ◼   ► you did this step that you forgot to document.

00:18:17 ◼   ► And so that was my first little pro tip,

00:18:21 ◼   ► is the first time when you think you got it right, that's great.

00:18:24 ◼   ► Now do it again.

00:18:25 ◼   ► Because if it works the second time,

00:18:27 ◼   ► then you know you actually understand your setup

00:18:30 ◼   ► and will be able to do it multiple times

00:18:32 ◼   ► if down the road you need to.

00:18:33 ◼   ► Because when you need to do it down the road,

00:18:35 ◼   ► it's usually under some kind of duress, because you're like,

00:18:39 ◼   ► wow, things are going really well.

00:18:40 ◼   ► I need another web server to keep up with the load.

00:18:43 ◼   ► Like that's a good thing, but it's also a bit of an urgent thing.

00:18:45 ◼   ► Yeah, and VPSs make this so much easier because not only can you like, you know, you can just

00:18:49 ◼   ► clone a server you already have if you need to, but also like you can do things like,

00:18:53 ◼   ► you know, automate this with a big shell script as you figure out, like this is what I do,

00:18:56 ◼   ► whenever I have to install something new or change something, I format everything I'm

00:19:01 ◼   ► doing as a shell script as I'm going, and then I can rerun it on a fresh server from

00:19:07 ◼   ► a VPS, like you know, so it's just like what you said, that you can kind of verify your

00:19:10 ◼   ► work and it's so much easier.

00:19:12 ◼   ► Oh man, it's like doing this, and I should also say,

00:19:15 ◼   ► like, anything that is not really easy to install

00:19:19 ◼   ► from a package manager, like, if,

00:19:21 ◼   ► suppose you're looking at, you know,

00:19:23 ◼   ► I'm looking at some PHP library to manipulate an image

00:19:26 ◼   ► or something, you know, this is just a weird example.

00:19:28 ◼   ► And if there's like three of them,

00:19:30 ◼   ► and one of them is in the repository

00:19:33 ◼   ► that my server comes with, like the yum or apt-get repo,

00:19:36 ◼   ► like, one of them is in the repo,

00:19:38 ◼   ► and I can install it with one word on the command line,

00:19:42 ◼   ► and one of them needs to be built from source,

00:19:44 ◼   ► I'm going with the one in the repo.

00:19:46 ◼   ► Because that's like, anything, as a general rule of thumb,

00:19:50 ◼   ► anything that you need to build from source

00:19:52 ◼   ► or manually download from somewhere

00:19:53 ◼   ► that is not in your package manager,

00:19:55 ◼   ► try to avoid needing that.

00:19:58 ◼   ► And everything becomes a lot easier.

00:20:00 ◼   ► And package managers are not flawless.

00:20:01 ◼   ► Sometimes you'll hit some weird conflict,

00:20:04 ◼   ► but that's very rare, especially if you stick

00:20:07 ◼   ► with the default repositories that come configured with it,

00:20:11 ◼   ► because those are so heavily tested by other people,

00:20:14 ◼   ► as again, old, boring, and popular,

00:20:16 ◼   ► that you might not be able to use the newest version

00:20:18 ◼   ► of something or a cutting edge feature of something

00:20:20 ◼   ► or an obscure module of something.

00:20:22 ◼   ► But if you can get away with not using it,

00:20:25 ◼   ► you are saving yourself time in the future.

00:20:27 ◼   ► - Exactly.

00:20:28 ◼   ► - We're sponsored this week by our friends at Igloo,

00:20:31 ◼   ► the intranet you will actually like.

00:20:33 ◼   ► With Igloo, you don't have to be stuck at your desk

00:20:35 ◼   ► to do your work.

00:20:36 ◼   ► You can manage your task list from your laptop

00:20:37 ◼   ► during a meeting, you can share status updates

00:20:39 ◼   ► from your phone as you're leaving a client site. You can even access the latest version

00:20:43 ◼   ► of a file from your home. Whether you're wearing pants or pajamas or anything, nobody will

00:20:47 ◼   ► even know. These days, everything is mobile. Your work should be too. Now, if you've ever

00:20:52 ◼   ► looked at a corporate intranet, you've possibly thought, "Whoever designed this must truly

00:20:56 ◼   ► hate me and everyone I know." With Igloo, those days are over. Igloo internets are actually

00:21:02 ◼   ► places you want to be. They're surprisingly configurable. You can even customize it and

00:21:06 ◼   ► and rebranded to give it the look and feel of your team.

00:21:08 ◼   ► They have group spaces, role-based access permissions,

00:21:10 ◼   ► easy to use drag and drop widget editors,

00:21:13 ◼   ► so you can reorganize the whole platform

00:21:14 ◼   ► to fit exactly how your teams work.

00:21:16 ◼   ► Now of course today, mobile is everywhere.

00:21:18 ◼   ► Igloo is great on mobile.

00:21:20 ◼   ► They are fully responsive, they work on every device,

00:21:22 ◼   ► at any screen size.

00:21:24 ◼   ► New devices come out, they work on day one,

00:21:26 ◼   ► and they have all these great integrations

00:21:28 ◼   ► with modern services that you actually use.

00:21:30 ◼   ► Things like Box, Google Drive, Dropbox,

00:21:33 ◼   ► All of this is integrated into one big,

00:21:35 ◼   ► easy to secure platform.

00:21:37 ◼   ► They do things like 256 bit encryption,

00:21:39 ◼   ► single sign on, active directory integrations.

00:21:41 ◼   ► You, if you know what any of those things are,

00:21:43 ◼   ► you know how safe and secure igloo is.

00:21:46 ◼   ► So share your files with your coworkers

00:21:47 ◼   ► for everyone to collaborate on,

00:21:48 ◼   ► do everything you want inside your intranet with igloo.

00:21:51 ◼   ► Break away from the intranet that you hate.

00:21:53 ◼   ► Go sign up for igloo right now

00:21:54 ◼   ► and you can try it for free for any team

00:21:56 ◼   ► with up to 10 people for as long as you want.

00:21:58 ◼   ► Sign up at igloosoftware.com/radar.

00:22:02 ◼   ► Once again, that's igloosoftware.com/radar.

00:22:04 ◼   ► Start your free trial today.

00:22:05 ◼   ► Thank you so much to igloo for supporting

00:22:07 ◼   ► Under the Radar and all of Relay FM.

00:22:10 ◼   ► - Okay, so once you now have the server set up,

00:22:14 ◼   ► it's running, your system's up and going,

00:22:17 ◼   ► the thing that of course seems worth pointing out

00:22:19 ◼   ► is some of the lessons we've learned

00:22:21 ◼   ► about how things typically go wrong

00:22:24 ◼   ► when they actually, you know,

00:22:26 ◼   ► so if you got this deployed, you got it running,

00:22:28 ◼   ► hopefully you have something running to monitor it.

00:22:31 ◼   ► I think I use Pingdom, I don't know what you use, but you have some kind of call where

00:22:35 ◼   ► I also use Pingdom.

00:22:36 ◼   ► There you go, they're a great service.

00:22:38 ◼   ► You give them a URL and they will hit it.

00:22:41 ◼   ► And one thing I always like with those kinds of URLs is don't point it to the root URL

00:22:47 ◼   ► of your site to make sure that it's up.

00:22:50 ◼   ► Have a special URL that it hits that will test your whole stack.

00:22:54 ◼   ► So it will go down to the database, make sure the database is running.

00:22:57 ◼   ► If you have a search query system, it'll go and hit that, and then report back results

00:23:03 ◼   ► so that you're actually testing your whole thing.

00:23:05 ◼   ► But you have something that's testing it.

00:23:08 ◼   ► Every now and then you'll get a little message that says, "Oh, it's down.

00:23:12 ◼   ► Why is that?"

00:23:14 ◼   ► And usually, on the nice thing, it doesn't actually happen very often.

00:23:18 ◼   ► For running as many services, I think we both run.

00:23:20 ◼   ► Outages are pretty rare, things are pretty reliable and stable these days in general.

00:23:25 ◼   ► and then you'll hit some kind of network outage in the data center, which you can't really

00:23:29 ◼   ► do anything about.

00:23:30 ◼   ► Which is awesome, because you can't do anything about it.

00:23:32 ◼   ► Yeah.

00:23:33 ◼   ► You know, you go on Twitter and you apologize to your users, and you just wait for the host

00:23:37 ◼   ► to fix it.

00:23:38 ◼   ► Yeah.

00:23:39 ◼   ► And that's great.

00:23:40 ◼   ► It's, it's, there's nothing to be done.

00:23:41 ◼   ► And that's going to happen no matter where you are, too.

00:23:43 ◼   ► Like, originally I'd think, oh, well, you know, maybe I, if I was using something else,

00:23:47 ◼   ► I wouldn't have this problem.

00:23:48 ◼   ► But it's like, everything has this problem.

00:23:51 ◼   ► Like,

00:23:52 ◼   ► I've been on a lot of hosts before.

00:23:53 ◼   ► they've all had outages.

00:23:55 ◼   ► >> Yeah, those are just built into it.

00:23:58 ◼   ► But beyond that, there's not that many of things that happen.

00:24:01 ◼   ► Usually the issues I run into, the most common thing that I'll hit is something like a machine

00:24:08 ◼   ► will run out of disk space, which seems kind of basic, but it's happened to me several

00:24:11 ◼   ► times where it turns out I have some log file that isn't being rotated correctly, and it's

00:24:18 ◼   ► just like my machine is completely out of space.

00:24:20 ◼   ► And so something—it'll just start falling over in a really confusing way.

00:24:24 ◼   ► So I just go and check on my servers every now and then, and just make sure that they

00:24:29 ◼   ► have space and the space isn't getting smaller and smaller, because most servers should just

00:24:32 ◼   ► be very stable and unchanging.

00:24:35 ◼   ► And once you do that, things are pretty good.

00:24:37 ◼   ► And then you run into, unfortunately, the next most likely case of problems is probably

00:24:44 ◼   ► what we would call human error.

00:24:46 ◼   ► And unfortunately, you are the human that is causing the error.

00:24:51 ◼   ► So when you're administering your own servers, you have all these little opportunities for

00:24:56 ◼   ► you to do something that ends up breaking your backend.

00:25:00 ◼   ► I've run a deployment to my RSS syncing system, Feed Wrangler, where I rolled a deployment

00:25:06 ◼   ► out that dropped the main feed item table accidentally.

00:25:11 ◼   ► That was an interesting test of how good my backup strategy was.

00:25:16 ◼   ► and took the service down for four hours,

00:25:18 ◼   ► but that's why you have backups,

00:25:20 ◼   ► and that's why you eventually learn

00:25:23 ◼   ► to be very, very, very careful

00:25:25 ◼   ► whenever you're touching your production databases.

00:25:28 ◼   ► - Yes, and yeah, database backup is very, very important.

00:25:32 ◼   ► Every VPS provider I know of

00:25:34 ◼   ► supports snapshotting the machine's backup.

00:25:36 ◼   ► Usually you have to do something special

00:25:38 ◼   ► to back up a database,

00:25:39 ◼   ► because you can't just copy the file

00:25:41 ◼   ► and have it be consistent,

00:25:42 ◼   ► but there are tools for every popular database

00:25:44 ◼   ► take snapshot backups of them and dump them into files or whatever. So look into that

00:25:48 ◼   ► for your database of choice. For MySQL, it's definitely Extra Backup is the one I recommend

00:25:53 ◼   ► from Percona. Postgres probably—you Postgres people, you're probably like, "Oh, well,

00:25:57 ◼   ► it just automatically does this for you. You don't have to even think about it because

00:25:59 ◼   ► Postgres is amazing, and MySQL is bad." Is that roughly correct?

00:26:04 ◼   ► Not exactly. I use a thing called Wall-E as my backup system, which is—yeah, the nice

00:26:11 ◼   ► thing about the most good backup systems too with databases is they can often allow you

00:26:16 ◼   ► to do point-in-time recovery, which is really helpful for that kind of a human error problem,

00:26:23 ◼   ► where you issued a particular command at a particular time and that destroyed all your

00:26:28 ◼   ► data. So even if you had your backup, if you had a snapshot right afterwards, it's not

00:26:34 ◼   ► like you need to recreate the database. You need to go back in time and undo queries.

00:26:38 ◼   ► And so if you have a system where that uses point-in-time recovery, which I know Wally

00:26:43 ◼   ► does and I'm sure there's MySQL things that do the same thing, you can kind of replay

00:26:47 ◼   ► what your database was doing and recreate it just before you were an idiot.

00:26:51 ◼   ► >> WESLEY CHUNG: Binary logging in MySQL.

00:26:52 ◼   ► >> STEVEN SCOTT: Yeah, which is awesome.

00:26:53 ◼   ► Being able to do that is a tremendously helpful thing.

00:26:57 ◼   ► Hopefully you never use it, but I think given that we've both done it or done things like

00:27:02 ◼   ► that, it will probably happen at some point and you just want to be prepared.

00:27:05 ◼   ► >> WESLEY CHUNG; Oh, yeah.

00:27:06 ◼   ► One thing also, earlier we talked about automating as much as possible with the setup. As much

00:27:11 ◼   ► as you can, automate the health of your servers. Pingdom monitoring for the overall uptime

00:27:18 ◼   ► of your app is great. One thing I do to prevent your disk space issue is I have two Pingdom

00:27:23 ◼   ► alerts for any service I run. One of them is, is it up? And that one's on high alert.

00:27:27 ◼   ► If anything goes down, like the main site is down, then everything of mine beeps and

00:27:34 ◼   ► vibrates, like everything alerts me. But I also have a lower class one, usually called

00:27:39 ◼   ► stats or vitals, where that hits a special URL which checks all the servers for things

00:27:45 ◼   ► like low disk space and very high CPU load and things like that. And it gives Pingdom

00:27:51 ◼   ► a down result if anything like that is out of whack and then it's logged in a place

00:27:57 ◼   ► that I can see like what caused this. So I can see, oh, at noon every day this server

00:28:02 ◼   ► has a really high load, or I can see,

00:28:05 ◼   ► oh, this server right here has low disk space right now.

00:28:07 ◼   ► So I'm never surprised by things like low disk space

00:28:11 ◼   ► or very high load, because I have that automated

00:28:13 ◼   ► and being automatically checked by another pingdom check.

00:28:16 ◼   ► - Yeah, that's a great idea.

00:28:17 ◼   ► Like, it's those types of things that you're just trying

00:28:20 ◼   ► to think of ways to anticipate the problem

00:28:23 ◼   ► before it happens.

00:28:24 ◼   ► - Yeah, I mean, even like a low-tech version of that,

00:28:26 ◼   ► which I've done on some servers,

00:28:27 ◼   ► is you just put a cron job that runs every five minutes

00:28:30 ◼   ► that emails you if disk space is too low.

00:28:32 ◼   ► Worst case scenario, you wake up to a pile of 40 emails,

00:28:35 ◼   ► okay, you go and fix it.

00:28:37 ◼   ► So it's, yeah, it's great.

00:28:38 ◼   ► There's so many easy ways to do this kind of automated

00:28:41 ◼   ► maintenance of servers now.

00:28:43 ◼   ► Running a server is really not as hard as you might think

00:28:47 ◼   ► if you've never done it before.

00:28:48 ◼   ► - Yeah, and I really think that if you can't,

00:28:51 ◼   ► if the things that we're talking about this week

00:28:53 ◼   ► sound scary, that's an important thing to get over

00:28:55 ◼   ► and get used to and comfortable with.

00:28:58 ◼   ► because as a developer, it feels like it's in some ways a very basic skill that if you

00:29:04 ◼   ► can't do, you're kind of holding both your work back in terms of the level of complexity

00:29:12 ◼   ► of the projects you can take on, as well as your career in terms of the things that you're

00:29:17 ◼   ► able to do and the jobs that you could potentially get and those kinds of things.

00:29:22 ◼   ► These are useful skills that don't go out of date.

00:29:24 ◼   ► If you know at a basic level how to administer a server, that's always going to be useful

00:29:29 ◼   ► and interesting.

00:29:30 ◼   ► I don't think that's going away.

00:29:31 ◼   ► As much as new high-level abstractions and things appear, that's cool and all, but

00:29:36 ◼   ► Unix has been around for a very long time for very good reasons.

00:29:39 ◼   ► All right, we're out of time this week.

00:29:41 ◼   ► We'll probably cover more of these topics in the future or expand on them, so keep listening

00:29:44 ◼   ► if you want more of this or anything else for that matter, because we are great and

00:29:48 ◼   ► you should keep listening to our show.

00:29:49 ◼   ► Thanks a lot for listening, everybody, and we'll talk to you next week.

00:29:52 ◼   ► Bye.