Under the Radar

14: Linux Servers for App Developers


00:00:00   Welcome to Under the Radar, a show about independent iOS and app development.

00:00:03   I'm Marco Arment.

00:00:04   And I'm David Smith.

00:00:06   Under the Radar is never longer than 30 minutes, so let's get started.

00:00:09   So last week we mentioned at the end how we were going to tell this week tips and stories

00:00:15   and setups of how to run your own Linux servers.

00:00:18   Because this is now, you know, in the wake of the parse announcement last week, this

00:00:21   is now kind of in the news, and we think it's kind of a core competency that many app developers

00:00:25   should know just how to run your own servers.

00:00:27   And once you know this, it enables your apps to be better,

00:00:31   to be more capable.

00:00:33   And if you do kind of a standard server route

00:00:36   without doing these kind of more abstract proprietary services

00:00:39   like Parse, it gives you a lot of flexibility

00:00:41   with what you can do in the future.

00:00:43   And it kind of decouples you from depending

00:00:45   on a particular third party to keep their platform going.

00:00:49   Yeah.

00:00:49   And this is probably a fair disclaimer to say.

00:00:51   This is going to be an episode that

00:00:52   that goes a bit into the weeds,

00:00:54   like down at some pretty low level stuff,

00:00:57   and if that's for you, awesome.

00:00:59   If it isn't for you, maybe stick around,

00:01:01   you might learn something, and know that some episodes

00:01:04   are gonna be high level,

00:01:05   some episodes are gonna be low level,

00:01:07   and hopefully we can have a nice balance as a result.

00:01:10   So at a high level, it's probably worth starting

00:01:12   of just kind of talking about the spectrum of hosting

00:01:16   before we kind of end up with why we do it

00:01:18   the way we do it.

00:01:19   So ultimately, hosting, in many ways, comes down to levels of abstraction, like how abstracted

00:01:26   away the actual computer that you're running your servers and your backends and systems

00:01:32   on, how much that is abstracted away from you.

00:01:35   So at the lowest level, you have what I think would be called co-location or something like

00:01:39   that, where you have a computer that's your computer that you went to Best Buy or whatever

00:01:45   to buy, and you're taking it and you're putting it somewhere, typically in some kind of data

00:01:49   center that you put in a data center so that it has reliable power and reliable internet

00:01:53   and reliable air conditioning and all those types of things. But like it's your machine.

00:01:57   Most people don't do that. I would really not recommend doing something like that.

00:02:02   Most people with the very least we do, but I think most people call like bare metal

00:02:06   managed servers where it's not really your machine. It's a service that they buy machines,

00:02:12   they manage machines. If a hard drive dies in the machine, they'll go and fix it for you.

00:02:16   but it's you have actual like a dedicated piece of hardware

00:02:21   you know a set of hard drives, a motherboard, a CPU that's exactly yours

00:02:26   and you probably don't want that either I've gone down that road

00:02:29   it's fine but it has a lot of

00:02:32   difficulties that you aren't really offset by its costs both

00:02:37   in monetary and like ease of use so most people

00:02:41   who do any kind of hosting themselves will use something that's more

00:02:45   like a VPS, which stands for Virtual Private Server, which is basically like, it's a machine

00:02:52   from your app's perspective.

00:02:53   You are, your app sees a hard drive, it sees a CPU, it sees memory, but all of those aren't

00:02:59   actually physical things.

00:03:02   It's running on a shared environment with a bunch of other systems that all see the

00:03:06   same things and share resources.

00:03:09   And you can use something like Linode for this, or DigitalOcean is another one that's

00:03:14   That's pretty common in many ways.

00:03:16   EC2 kind of runs like this, but it's a bit different.

00:03:19   But you have these kinds of systems where you have a virtual Linux machine that you

00:03:23   can do, and you have some nice advantages with that because it's virtual.

00:03:27   You can do things like pick it up and move it around.

00:03:30   You take an image of your hard drive and you can copy it onto another one, and things like

00:03:35   that that are really easy to manage.

00:03:37   And from a cost perspective, it's way cheaper because you're sharing server resources.

00:03:43   And then above that you have things like virtual server environments, things like Azure or

00:03:46   Heroku, or total abstractions like CloudKit or Parse or those types of things that are

00:03:52   completely abstracting away from you.

00:03:55   And like I said, for most people you probably want to start at a VPS.

00:03:57   Like you want to find out where you're comfortable if the thought of administering a Linux server

00:04:02   even after this episode is completely overwhelming.

00:04:05   Like maybe go to something like Heroku or Azure where you have a little bit of control

00:04:10   over what you're doing, but mostly there's something there, but it's still at least general

00:04:15   purpose. But I think we both run VPS servers because once you know what you're doing, it's

00:04:22   really cheap, it's really efficient, and you have a lot of flexibility that you could move

00:04:27   providers, you can spin up new servers and take them down really quickly and easily,

00:04:32   and especially the cost is amazing. It's incredible the amount of performance you can get from

00:04:38   from a really basic setup.

00:04:39   I mean, at this point, I think I administer

00:04:41   19 virtual private servers or something like that

00:04:44   to run my apps, and it's a pretty big number,

00:04:47   but the actual load of that,

00:04:49   of actually doing that is pretty straightforward.

00:04:51   - Oh yeah, and I mean, things have gotten so cheap

00:04:53   over the years, like I now pay for all of Overcast,

00:04:56   which is something like 11 VPS instances on Linode.

00:05:01   I now pay for all of them what I used to pay

00:05:03   for one database server for Instapaper back in the day,

00:05:07   and that wasn't that long ago.

00:05:08   That was like five years ago.

00:05:10   So what you can get now at the VPS level

00:05:12   is just incredibly good.

00:05:14   I was the biggest fan of bare metal dedicated servers before

00:05:17   but now even I have converted because you can just,

00:05:20   you know, the competition especially between Linode

00:05:22   and DigitalOcean has really made incredibly fast

00:05:26   hardware available for almost no money.

00:05:28   And it also makes it much easier to learn

00:05:30   because it lets you, and we'll get into this

00:05:32   in a little bit, but it lets you,

00:05:34   you know, if you can just like,

00:05:35   in a simple command on a web interface,

00:05:38   so not involving buying anything,

00:05:40   not involving emailing anybody

00:05:42   or placing any orders with anybody,

00:05:44   just in a web interface you can quickly create

00:05:47   an instance of a server, then you can do things like,

00:05:49   oh, let me see how easy is it to set up PHP or whatever.

00:05:53   Like you can do things like that,

00:05:54   or you can say, I wonder how,

00:05:56   whether this new distro of Linux makes more sense to me,

00:05:59   or I wonder if my software will run on this other distro

00:06:02   or this other web server or whatever.

00:06:04   you can just spin up a new instance in minutes

00:06:07   and run your install script and poke around,

00:06:11   try things out, see what works, see what breaks.

00:06:15   And if at any point you decide this wasn't for you

00:06:18   or this didn't go well, you can just delete the instance

00:06:20   and you've paid up to that point maybe 50 cents.

00:06:24   Like it's crazy how easy this is

00:06:27   to both administer servers and to learn

00:06:29   how to administer servers now in the world of VPSs.

00:06:33   - Yeah, 'cause it's lovely to think of,

00:06:34   most of these are charged by hour of use

00:06:37   or something like that, and so you can,

00:06:39   they're just kind of like utility in that way,

00:06:41   that you can just spin them up when you want 'em,

00:06:43   spin, take 'em down when you don't want 'em anymore,

00:06:45   and like, it's very straightforward and easy to do

00:06:48   in a way that is kind of scary the first time you do it,

00:06:51   that you just like hit a button and it's like,

00:06:52   oh, here's a machine.

00:06:54   Go ahead and SSH into it and get started.

00:06:57   - Yeah, I mean, it's so easy that like,

00:06:58   you know, one way you could've done it before

00:07:00   is by running something like VMware Fusion or Parallels

00:07:02   on your computer and virtualizing little Linux setups

00:07:06   on your computer and doing that.

00:07:08   But now it's so cheap and easy to go do it

00:07:10   on Linode or DigitalOcean that it's actually easier

00:07:14   to do that than it is to do it on your local computer.

00:07:18   Now I do all my experimentation on VPSs.

00:07:20   I don't even have virtualization software installed

00:07:21   anymore on my Mac.

00:07:23   - Yeah, and I'm the same way.

00:07:24   The only thing I use my Mac for when I'm doing

00:07:26   this kind of thing is the first level of,

00:07:30   the nice thing about using something like Linux

00:07:32   is that the basic underpinnings underneath our Macs

00:07:36   are like it's some kind of Unix-based thing.

00:07:40   I couldn't tell you all of the history of it,

00:07:41   but functionally, your Mac is running

00:07:43   the same basic stack kernel types of things

00:07:46   that your ultimate server is gonna run.

00:07:49   And so I use my Mac because I can learn

00:07:52   and kind of prototype and try some of the things

00:07:54   that ultimately I'm gonna run on my main server locally.

00:07:57   I have a development database that runs on my machine.

00:08:01   I have a web server that runs on my local machine,

00:08:03   which makes prototyping really quick.

00:08:05   But as soon as I wanna do anything

00:08:06   that's going to be simulating my end environment,

00:08:10   I just spin up an instance and run it.

00:08:12   And I have a couple of just instances

00:08:14   that you just bring up, like try it out.

00:08:16   If it works, great.

00:08:17   If it doesn't, you can just delete it and start again.

00:08:20   So once you've decided you're gonna build

00:08:23   a Linux instance and get started,

00:08:26   and that actual process is pretty straightforward.

00:08:29   you'll go to a virtual private server provider, and just sort of you'll set up your instance,

00:08:35   it'll give you some credentials and an IP address and a few things that you'll have

00:08:39   for like how to log into it, and then you just log in and get started. And for most

00:08:43   applications, you're going to install at least two things on that. You're going to install

00:08:47   a web server and a database of some kind. And starting out, you could probably put them

00:08:52   both on the same machine. Eventually, you'll probably do a thing where you'll split the

00:08:57   the web database onto its own machine

00:08:59   and put one or more web servers in front of it.

00:09:02   But at the very least, you're gonna install

00:09:04   some kind of web server environment,

00:09:05   some kind of database.

00:09:07   The web server that you choose is entirely up to you.

00:09:09   Like, I use Ruby on Rails for all my backends.

00:09:12   I think all of yours are in PHP.

00:09:14   - Yeah, I mean, and I have many opinions

00:09:17   on how you choose these tools.

00:09:20   I will summarize them, for the sake of this

00:09:21   being a short show, I will summarize them

00:09:23   as what you want most of all when choosing server software,

00:09:27   if you don't wanna be administering and tweaking

00:09:30   with your server constantly,

00:09:31   what you want is old, boring, and popular.

00:09:35   Those three things, old, boring, and popular.

00:09:38   New and trendy does not necessarily always mean better.

00:09:41   In this case, you know, especially like,

00:09:44   what makes some people think that the new stuff is better

00:09:47   is vague claims of it being quote, scalable,

00:09:51   And beware of any claims of things being scalable, because in reality, scaling web services up,

00:09:58   not only is it the kind of thing that you will probably never need for almost any project

00:10:02   that you do, but when you do need it, it is way more complicated. And there is no silver

00:10:08   bullet for scaling. There is nothing that automatically scales, unless it also automatically

00:10:13   scales all of your money away. But there is nothing that automatically and easily scales.

00:10:18   Scaling is always complicated and always tricky. It's just, you know, the new things that

00:10:22   claim scalability just scale differently. And the challenges are just different challenges.

00:10:29   The newer something is, the trendier or newer something is, that means, you know, the fewer

00:10:35   people have experience with it and the fewer large installations it has. And so that means

00:10:43   there's less knowledge in the world about how it fails or what the anti-patterns of

00:10:47   it might be. So as you go lower level in the stack, being this kind of conservative, preferring

00:10:53   things that are old, boring and popular becomes more important. So like if you want to use

00:10:58   like a brand new cutting edge CSS library, go for it. It doesn't matter really. You

00:11:02   know, if it works for you, go for it. Your risk is pretty low there. But you probably

00:11:07   would not want to use a beta version of the Linux kernel. You know, so like as you go

00:11:12   down the stack, you want to be more conservative because it's just kind of a bad idea to

00:11:17   to take risks lower down.

00:11:18   And I include things like the database in that.

00:11:21   MySQL and Postgres are great.

00:11:24   They're battle tested, they are old, boring, and popular.

00:11:27   That is what you want in your database.

00:11:30   Just like you wouldn't want a trendy new kind of brakes

00:11:32   on your car.

00:11:33   Like you want time tested brakes that everyone knows works

00:11:37   and everyone knows that you aren't gonna be pushing

00:11:38   the boundaries of them.

00:11:40   You want a really boring database.

00:11:43   And I would say you want a really boring language too

00:11:46   because really new cutting edge languages are still in flux.

00:11:51   And so if you want to have this kind of like set it

00:11:54   and forget it kind of server approach,

00:11:56   you don't want to have to deal with

00:11:58   a young languages frequent changes

00:12:01   and frequent needs for being updated.

00:12:04   You know, you want to minimize anything

00:12:06   that will require your attention

00:12:08   or that might break your app.

00:12:09   So you wanna minimize the need for, you know,

00:12:12   backwards incompatible updates and things like that

00:12:15   or crashes and having to install little bug fixes constantly for the language or database

00:12:20   you're using, you want to avoid all of that. Because new really, new means untested or

00:12:26   less tested. Another thing that you hit with new stuff is in the language layer, new means

00:12:33   that you have fewer third party libraries and generally less third party code. And this

00:12:38   is a problem like, I've been using PHP forever and PHP third party code is mostly garbage,

00:12:43   but there is so much of it out there that any need that I have, I can find something

00:12:49   to do it. And it isn't always good, but I can find something. And as I've started

00:12:53   writing some newer things in Go, and Go I like as a language conceptually, but because

00:12:58   it's such a younger language and it isn't that popular yet, and might never become popular,

00:13:04   who knows, but because it's younger and less popular, there is very little third-party

00:13:08   already go code for some very common things that I keep needing to do. And I keep, every

00:13:13   time I need to write something new in Go, I hit this wall of, "Oh, I'll just load up

00:13:17   the markdown module." And I look and there's like three of them that are all kind of half

00:13:22   baked and, you know, and I haven't listened in a while for markdown, so maybe that's a

00:13:25   bad example. But I keep hitting that wall whenever I do anything in a new language of,

00:13:29   "Oh, I just want to see how, oh, how do I resize an image? Oh, well, you kind of can't,

00:13:33   "Oh, unless you do this."

00:13:35   So it's always easier to use old, boring, and popular stuff.

00:13:39   It may be less interesting,

00:13:40   and if you are the kind of person

00:13:42   who wants cutting edge languages

00:13:45   and who finds that satisfying,

00:13:46   then you will trade time and maintenance for that.

00:13:50   And so for the goal of low maintenance

00:13:52   and low server maintenance,

00:13:55   I highly suggest that you use other parts of your app

00:13:58   to get the kind of newness out of your system.

00:14:01   Like yeah, use the higher level parts

00:14:03   use maybe the iOS side of it if you have an iOS and a web app, use that side of it to

00:14:07   do new cool stuff and make the server as boring, old and popular as possible. Because old,

00:14:12   boring, popular stuff works. It really works. It hardly ever breaks and you will never be

00:14:19   the biggest person using it because long before, you know, even suppose that you build something

00:14:24   with old, boring stuff. Suppose it gets like the size of Etsy, right? Etsy is huge. It's

00:14:29   It's not the biggest website on the internet,

00:14:31   but it's huge, right?

00:14:32   It gets the size of Etsy.

00:14:34   If you get that big, will you be the biggest user

00:14:38   of anything in your stack?

00:14:40   Because once you're the biggest user

00:14:41   of your database or something,

00:14:43   then you basically become a developer of that database.

00:14:46   You will hit all the bugs, you will have to go in

00:14:48   and make patches yourself to customize things

00:14:50   that aren't working well or aren't working right for you.

00:14:54   So Facebook rolls out tons of patches and things.

00:14:57   Twitter has rolled out tons of patches

00:14:58   things like Memcache, I mean like there's all these big companies, all these big web

00:15:02   companies end up having to basically partially take over parts of the stack they depend on

00:15:07   open source wise and make their own contributions and fixes because they hit tons of bugs. You

00:15:12   never want to be the biggest user of anything in your stack. And you know, when you start

00:15:16   out of course you won't be the biggest user of basically anything, but you could become

00:15:19   the biggest user. Like you'd be surprised, you know, you hear about new trendy things,

00:15:24   but you'd be surprised how few big companies

00:15:27   actually use them, so you never want to be the biggest

00:15:29   user of anything, you want Facebook to have fixed

00:15:32   every bug in that stack years before you ever hit it.

00:15:35   - Yeah, and I think we're both coming at it

00:15:37   from the perspective of being app developers,

00:15:39   like what makes our apps cool is not the backend stack.

00:15:42   That's got nothing to do with the thing that I'm gonna

00:15:45   be marketing my app as.

00:15:47   Like my customer shouldn't care, it's only,

00:15:51   it exists for the sole purpose of just getting

00:15:53   job done. Most of my backends are the most ridiculously boring, simple applications that

00:15:59   I can possibly make it. I put as much logic and smarts as I can in the apps itself. The

00:16:06   database is mostly just a nice wrapper for a database, and something that's simple and

00:16:10   straightforward that I can set up and then I never think about again. The only time I

00:16:16   ever really think about my servers is when there's some big security thing that I need

00:16:22   to make sure it was auto-updated,

00:16:24   or I'm adding major new functionality to the front end,

00:16:27   to my applications, and then I have to go back

00:16:30   and add something to it.

00:16:33   But otherwise, I just want it as simple and basic as can be.

00:16:35   It's just like a big database that I can put things in

00:16:39   and then later get them out.

00:16:41   And as long as I can do that, I'm good.

00:16:43   - And if anything, when you're using old, boring,

00:16:45   and popular, if anything ever goes wrong,

00:16:48   or if you have any questions whatsoever,

00:16:51   or anything comes up, you can just search the web

00:16:53   and you have hundreds of answers from the thousands

00:16:56   of other people who have used that already

00:16:58   and have documented it like three years earlier.

00:17:00   (laughs)

00:17:01   It's so much easier to make your server side stack

00:17:06   something that you can easily forget about most of the time

00:17:10   and you don't even have to deal with it.

00:17:12   - Exactly.

00:17:13   And I guess once you decide on which one you wanna do,

00:17:16   it's like the process from there is honestly

00:17:18   pretty straightforward.

00:17:19   you pick something, you install it. Usually these kind of things are a pretty straightforward

00:17:24   install process. Like, if you're, there should be some kind of conversion of like,

00:17:28   "Go get me MySQL, go get me Postgres, go get me Nginx, go get me Apache," and the server will go

00:17:36   get it, install it, configure it, you'll change a few configuration files, and you'll be up and

00:17:40   running because it's boring. And the process of getting it set up has been done by thousands of

00:17:47   people so everything's automated and scripted and the only thing that I'd mentioned there

00:17:51   is one thing that I learned early on is once I set up a server the first time for I always

00:17:58   document how I did it like the steps I did and then I will usually destroy what I did

00:18:02   the first time and do it again to make sure I actually know what I'm doing to avoid the

00:18:06   situation where down the road you are trying to like recreate a web server and you can

00:18:11   never get it to work quite right and it turns out it's because the first time you did it

00:18:15   you did this step that you forgot to document.

00:18:17   And so that was my first little pro tip,

00:18:21   is the first time when you think you got it right, that's great.

00:18:24   Now do it again.

00:18:25   Because if it works the second time,

00:18:27   then you know you actually understand your setup

00:18:30   and will be able to do it multiple times

00:18:32   if down the road you need to.

00:18:33   Because when you need to do it down the road,

00:18:35   it's usually under some kind of duress, because you're like,

00:18:39   wow, things are going really well.

00:18:40   I need another web server to keep up with the load.

00:18:43   Like that's a good thing, but it's also a bit of an urgent thing.

00:18:45   Yeah, and VPSs make this so much easier because not only can you like, you know, you can just

00:18:49   clone a server you already have if you need to, but also like you can do things like,

00:18:53   you know, automate this with a big shell script as you figure out, like this is what I do,

00:18:56   whenever I have to install something new or change something, I format everything I'm

00:19:01   doing as a shell script as I'm going, and then I can rerun it on a fresh server from

00:19:07   a VPS, like you know, so it's just like what you said, that you can kind of verify your

00:19:10   work and it's so much easier.

00:19:12   Oh man, it's like doing this, and I should also say,

00:19:15   like, anything that is not really easy to install

00:19:19   from a package manager, like, if,

00:19:21   suppose you're looking at, you know,

00:19:23   I'm looking at some PHP library to manipulate an image

00:19:26   or something, you know, this is just a weird example.

00:19:28   And if there's like three of them,

00:19:30   and one of them is in the repository

00:19:33   that my server comes with, like the yum or apt-get repo,

00:19:36   like, one of them is in the repo,

00:19:38   and I can install it with one word on the command line,

00:19:42   and one of them needs to be built from source,

00:19:44   I'm going with the one in the repo.

00:19:46   Because that's like, anything, as a general rule of thumb,

00:19:50   anything that you need to build from source

00:19:52   or manually download from somewhere

00:19:53   that is not in your package manager,

00:19:55   try to avoid needing that.

00:19:58   And everything becomes a lot easier.

00:20:00   And package managers are not flawless.

00:20:01   Sometimes you'll hit some weird conflict,

00:20:04   but that's very rare, especially if you stick

00:20:07   with the default repositories that come configured with it,

00:20:11   because those are so heavily tested by other people,

00:20:14   as again, old, boring, and popular,

00:20:16   that you might not be able to use the newest version

00:20:18   of something or a cutting edge feature of something

00:20:20   or an obscure module of something.

00:20:22   But if you can get away with not using it,

00:20:25   you are saving yourself time in the future.

00:20:27   - Exactly.

00:20:28   - We're sponsored this week by our friends at Igloo,

00:20:31   the intranet you will actually like.

00:20:33   With Igloo, you don't have to be stuck at your desk

00:20:35   to do your work.

00:20:36   You can manage your task list from your laptop

00:20:37   during a meeting, you can share status updates

00:20:39   from your phone as you're leaving a client site. You can even access the latest version

00:20:43   of a file from your home. Whether you're wearing pants or pajamas or anything, nobody will

00:20:47   even know. These days, everything is mobile. Your work should be too. Now, if you've ever

00:20:52   looked at a corporate intranet, you've possibly thought, "Whoever designed this must truly

00:20:56   hate me and everyone I know." With Igloo, those days are over. Igloo internets are actually

00:21:02   places you want to be. They're surprisingly configurable. You can even customize it and

00:21:06   and rebranded to give it the look and feel of your team.

00:21:08   They have group spaces, role-based access permissions,

00:21:10   easy to use drag and drop widget editors,

00:21:13   so you can reorganize the whole platform

00:21:14   to fit exactly how your teams work.

00:21:16   Now of course today, mobile is everywhere.

00:21:18   Igloo is great on mobile.

00:21:20   They are fully responsive, they work on every device,

00:21:22   at any screen size.

00:21:24   New devices come out, they work on day one,

00:21:26   and they have all these great integrations

00:21:28   with modern services that you actually use.

00:21:30   Things like Box, Google Drive, Dropbox,

00:21:33   All of this is integrated into one big,

00:21:35   easy to secure platform.

00:21:37   They do things like 256 bit encryption,

00:21:39   single sign on, active directory integrations.

00:21:41   You, if you know what any of those things are,

00:21:43   you know how safe and secure igloo is.

00:21:46   So share your files with your coworkers

00:21:47   for everyone to collaborate on,

00:21:48   do everything you want inside your intranet with igloo.

00:21:51   Break away from the intranet that you hate.

00:21:53   Go sign up for igloo right now

00:21:54   and you can try it for free for any team

00:21:56   with up to 10 people for as long as you want.

00:21:58   Sign up at igloosoftware.com/radar.

00:22:02   Once again, that's igloosoftware.com/radar.

00:22:04   Start your free trial today.

00:22:05   Thank you so much to igloo for supporting

00:22:07   Under the Radar and all of Relay FM.

00:22:10   - Okay, so once you now have the server set up,

00:22:14   it's running, your system's up and going,

00:22:17   the thing that of course seems worth pointing out

00:22:19   is some of the lessons we've learned

00:22:21   about how things typically go wrong

00:22:24   when they actually, you know,

00:22:26   so if you got this deployed, you got it running,

00:22:28   hopefully you have something running to monitor it.

00:22:31   I think I use Pingdom, I don't know what you use, but you have some kind of call where

00:22:35   I also use Pingdom.

00:22:36   There you go, they're a great service.

00:22:38   You give them a URL and they will hit it.

00:22:41   And one thing I always like with those kinds of URLs is don't point it to the root URL

00:22:47   of your site to make sure that it's up.

00:22:50   Have a special URL that it hits that will test your whole stack.

00:22:54   So it will go down to the database, make sure the database is running.

00:22:57   If you have a search query system, it'll go and hit that, and then report back results

00:23:03   so that you're actually testing your whole thing.

00:23:05   But you have something that's testing it.

00:23:08   Every now and then you'll get a little message that says, "Oh, it's down.

00:23:12   Why is that?"

00:23:14   And usually, on the nice thing, it doesn't actually happen very often.

00:23:18   For running as many services, I think we both run.

00:23:20   Outages are pretty rare, things are pretty reliable and stable these days in general.

00:23:25   and then you'll hit some kind of network outage in the data center, which you can't really

00:23:29   do anything about.

00:23:30   Which is awesome, because you can't do anything about it.

00:23:32   Yeah.

00:23:33   You know, you go on Twitter and you apologize to your users, and you just wait for the host

00:23:37   to fix it.

00:23:38   Yeah.

00:23:39   And that's great.

00:23:40   It's, it's, there's nothing to be done.

00:23:41   And that's going to happen no matter where you are, too.

00:23:43   Like, originally I'd think, oh, well, you know, maybe I, if I was using something else,

00:23:47   I wouldn't have this problem.

00:23:48   But it's like, everything has this problem.

00:23:51   Like,

00:23:52   I've been on a lot of hosts before.

00:23:53   they've all had outages.

00:23:55   >> Yeah, those are just built into it.

00:23:58   But beyond that, there's not that many of things that happen.

00:24:01   Usually the issues I run into, the most common thing that I'll hit is something like a machine

00:24:08   will run out of disk space, which seems kind of basic, but it's happened to me several

00:24:11   times where it turns out I have some log file that isn't being rotated correctly, and it's

00:24:18   just like my machine is completely out of space.

00:24:20   And so something—it'll just start falling over in a really confusing way.

00:24:24   So I just go and check on my servers every now and then, and just make sure that they

00:24:29   have space and the space isn't getting smaller and smaller, because most servers should just

00:24:32   be very stable and unchanging.

00:24:35   And once you do that, things are pretty good.

00:24:37   And then you run into, unfortunately, the next most likely case of problems is probably

00:24:44   what we would call human error.

00:24:46   And unfortunately, you are the human that is causing the error.

00:24:51   So when you're administering your own servers, you have all these little opportunities for

00:24:56   you to do something that ends up breaking your backend.

00:25:00   I've run a deployment to my RSS syncing system, Feed Wrangler, where I rolled a deployment

00:25:06   out that dropped the main feed item table accidentally.

00:25:11   That was an interesting test of how good my backup strategy was.

00:25:16   and took the service down for four hours,

00:25:18   but that's why you have backups,

00:25:20   and that's why you eventually learn

00:25:23   to be very, very, very careful

00:25:25   whenever you're touching your production databases.

00:25:28   - Yes, and yeah, database backup is very, very important.

00:25:32   Every VPS provider I know of

00:25:34   supports snapshotting the machine's backup.

00:25:36   Usually you have to do something special

00:25:38   to back up a database,

00:25:39   because you can't just copy the file

00:25:41   and have it be consistent,

00:25:42   but there are tools for every popular database

00:25:44   take snapshot backups of them and dump them into files or whatever. So look into that

00:25:48   for your database of choice. For MySQL, it's definitely Extra Backup is the one I recommend

00:25:53   from Percona. Postgres probably—you Postgres people, you're probably like, "Oh, well,

00:25:57   it just automatically does this for you. You don't have to even think about it because

00:25:59   Postgres is amazing, and MySQL is bad." Is that roughly correct?

00:26:04   Not exactly. I use a thing called Wall-E as my backup system, which is—yeah, the nice

00:26:11   thing about the most good backup systems too with databases is they can often allow you

00:26:16   to do point-in-time recovery, which is really helpful for that kind of a human error problem,

00:26:23   where you issued a particular command at a particular time and that destroyed all your

00:26:28   data. So even if you had your backup, if you had a snapshot right afterwards, it's not

00:26:34   like you need to recreate the database. You need to go back in time and undo queries.

00:26:38   And so if you have a system where that uses point-in-time recovery, which I know Wally

00:26:43   does and I'm sure there's MySQL things that do the same thing, you can kind of replay

00:26:47   what your database was doing and recreate it just before you were an idiot.

00:26:51   >> WESLEY CHUNG: Binary logging in MySQL.

00:26:52   >> STEVEN SCOTT: Yeah, which is awesome.

00:26:53   Being able to do that is a tremendously helpful thing.

00:26:57   Hopefully you never use it, but I think given that we've both done it or done things like

00:27:02   that, it will probably happen at some point and you just want to be prepared.

00:27:05   >> WESLEY CHUNG; Oh, yeah.

00:27:06   One thing also, earlier we talked about automating as much as possible with the setup. As much

00:27:11   as you can, automate the health of your servers. Pingdom monitoring for the overall uptime

00:27:18   of your app is great. One thing I do to prevent your disk space issue is I have two Pingdom

00:27:23   alerts for any service I run. One of them is, is it up? And that one's on high alert.

00:27:27   If anything goes down, like the main site is down, then everything of mine beeps and

00:27:34   vibrates, like everything alerts me. But I also have a lower class one, usually called

00:27:39   stats or vitals, where that hits a special URL which checks all the servers for things

00:27:45   like low disk space and very high CPU load and things like that. And it gives Pingdom

00:27:51   a down result if anything like that is out of whack and then it's logged in a place

00:27:57   that I can see like what caused this. So I can see, oh, at noon every day this server

00:28:02   has a really high load, or I can see,

00:28:05   oh, this server right here has low disk space right now.

00:28:07   So I'm never surprised by things like low disk space

00:28:11   or very high load, because I have that automated

00:28:13   and being automatically checked by another pingdom check.

00:28:16   - Yeah, that's a great idea.

00:28:17   Like, it's those types of things that you're just trying

00:28:20   to think of ways to anticipate the problem

00:28:23   before it happens.

00:28:24   - Yeah, I mean, even like a low-tech version of that,

00:28:26   which I've done on some servers,

00:28:27   is you just put a cron job that runs every five minutes

00:28:30   that emails you if disk space is too low.

00:28:32   Worst case scenario, you wake up to a pile of 40 emails,

00:28:35   okay, you go and fix it.

00:28:37   So it's, yeah, it's great.

00:28:38   There's so many easy ways to do this kind of automated

00:28:41   maintenance of servers now.

00:28:43   Running a server is really not as hard as you might think

00:28:47   if you've never done it before.

00:28:48   - Yeah, and I really think that if you can't,

00:28:51   if the things that we're talking about this week

00:28:53   sound scary, that's an important thing to get over

00:28:55   and get used to and comfortable with.

00:28:58   because as a developer, it feels like it's in some ways a very basic skill that if you

00:29:04   can't do, you're kind of holding both your work back in terms of the level of complexity

00:29:12   of the projects you can take on, as well as your career in terms of the things that you're

00:29:17   able to do and the jobs that you could potentially get and those kinds of things.

00:29:22   These are useful skills that don't go out of date.

00:29:24   If you know at a basic level how to administer a server, that's always going to be useful

00:29:29   and interesting.

00:29:30   I don't think that's going away.

00:29:31   As much as new high-level abstractions and things appear, that's cool and all, but

00:29:36   Unix has been around for a very long time for very good reasons.

00:29:39   All right, we're out of time this week.

00:29:41   We'll probably cover more of these topics in the future or expand on them, so keep listening

00:29:44   if you want more of this or anything else for that matter, because we are great and

00:29:48   you should keep listening to our show.

00:29:49   Thanks a lot for listening, everybody, and we'll talk to you next week.

00:29:52   Bye.