Under the Radar

13: External Dependencies


00:00:00   Welcome to Under the Radar, a show about independent iOS app development.

00:00:04   I'm Marco Arment.

00:00:05   And I'm David Smith.

00:00:06   Under the Radar is never longer than 30 minutes, so let's get started.

00:00:10   So today we wanted to unpack the situation that occurred,

00:00:17   I guess it was the beginning of this week, the end of last week, something like that,

00:00:21   where Parse, a fairly widely used platform for app backends,

00:00:27   announce that they will be shutting down. And while the actual situation and the nuances

00:00:33   of that aren't particularly generally applicable or interesting, the actual sort of -- the

00:00:38   fundamentals of that, of having this big general purpose platform that was used by lots of

00:00:43   apps finally shutting -- or deciding it was going to shut down has a lot of knock-on effects

00:00:48   that are probably worth unpacking.

00:00:49   Before we dive into that, it probably makes sense to just sort of talk about what Parse

00:00:53   And so Parse was this platform that made it relatively easy to make a backend for your

00:00:59   application that would do object persistence, user management, things like that, like a

00:01:04   very basic high-level, like, non-specific to a particular industry backend sort of system

00:01:10   that a lot of apps could use.

00:01:11   And it had fairly attractive pricing, and including a free tier, which is always a bit

00:01:16   of a trap for these kinds of things, where you can say, "Oh, we can use it for free,

00:01:19   and if our app gets really successful, then we -- only then will we have to pay."

00:01:23   And eventually, a couple years ago, it was bought by Facebook, which made everyone who

00:01:26   was using it kind of happy because, oh, now it's not this sort of fly-by-night thing.

00:01:30   It's backed by, you know, this major corporation.

00:01:33   But now it's being shut down.

00:01:36   Facebook has decided that that's not something that they want to invest in and continue to

00:01:39   maintain.

00:01:40   And so a year from now, they are going to be turning it off.

00:01:43   They're doing it pretty well.

00:01:45   Like they're giving a year's notice and a bunch of migration tools.

00:01:48   But at the end of the day, this thing that I've heard varying reports, but at least,

00:01:55   it's fair to say quite a lot of apps make use of, at least in part, is just going to

00:01:59   be turned off.

00:02:00   And as a result, the apps that use it, if they haven't been updated or migrated, are

00:02:04   just going to stop working.

00:02:06   And that's kind of tricky.

00:02:07   >>

00:02:07   - Yeah, I think one of the weird things about this

00:02:09   is like, you know, it kind of ties back to App Economics,

00:02:11   where in order for these apps to continue working,

00:02:15   it has to be worth their developers' time,

00:02:18   and their developers have to have the budget

00:02:20   to now do a noticeable update.

00:02:23   And you know, they've made it relatively easy.

00:02:26   They've released big parts of their service

00:02:28   as open source that you could just install on any server.

00:02:31   Microsoft Azure has started trying to attract people

00:02:33   to migrate to them, and they're making it a little bit easier

00:02:36   So there are migration options here that aren't going to be incredibly work heavy, but it

00:02:41   is still work.

00:02:42   You have to still do an update.

00:02:43   There are going to be things you have to change and rewrite.

00:02:46   And so it has to be worth that happening by the apps developers.

00:02:51   So if you're relying on an app that uses this that hasn't been updated in a long time, that

00:02:55   might never be updated for this.

00:02:57   It might never be worth somebody's time to update it.

00:03:00   And that's unfortunate.

00:03:01   And that is going to cause a lot of problems in the app store as these apps just kind of

00:03:03   slowly collect one-star reviews and stop working and they just kind of live as zombies forever.

00:03:10   >> And so as developers, when I see something like this, like I said, the specifics of the

00:03:16   par situation are sort of vaguely or intellectually interesting but aren't actually practically

00:03:21   that interesting. But what it makes me think about is it makes me evaluate the dependencies

00:03:27   that I have in my own apps and as I build apps, and increasingly, fewer and fewer of

00:03:33   the things that I build have no web component whatsoever.

00:03:37   Because it seems like these days, you're going to need some kind of backend for your application.

00:03:44   Either something like parse or that's sort of like off the shelf or something you build

00:03:49   yourself.

00:03:50   And you're probably going to need these for at least one of three reasons.

00:03:53   You're going to need some kind of, your app's probably going to do something backup-related,

00:03:57   where people are, if people are putting any amount of data into your application, they're

00:04:02   probably going to want to be able to have it backed up.

00:04:05   And this is something that for a long time I used to say, "Oh, I rely on the iTunes

00:04:09   iCloud backup system."

00:04:11   But that is all kinds of problems and issues that you'll run into, where, like, I have

00:04:16   a recipe manager, and I ran into issues where their recipes were fully backed up in the

00:04:22   the latest backup that they did,

00:04:23   but they accidentally deleted the app.

00:04:25   And so now the only way they can get their recipes back

00:04:27   is to do a full restore of an old backup onto their device,

00:04:32   potentially destroying newer data

00:04:36   on other apps and things.

00:04:38   It's a mess.

00:04:39   So you wanna be able to backup your data,

00:04:40   or you wanna be able to sync your data

00:04:42   between different devices,

00:04:44   so you'll need some kind of backend to do that.

00:04:46   Or you just have an app that has a core service.

00:04:49   Obviously, I imagine in Overcast,

00:04:51   you need a backend, or a lot of what you do wouldn't work

00:04:56   if you didn't have some kind of backend to run for it.

00:04:59   - Oh, sure, I mean, there are podcast apps

00:05:02   that don't use server-side backends as intermediaries

00:05:05   and just crawl feeds directly and everything,

00:05:07   but that's not how I built mine,

00:05:09   and it affords me a bunch of advantages

00:05:11   to have done it the way I do it,

00:05:12   but now I have this big dependency on my service.

00:05:15   - Yeah, and I think ultimately, that's the right word.

00:05:18   At the end of it, building these backends

00:05:21   that are maybe, aren't always required,

00:05:23   but are going to be required in a lot of cases,

00:05:26   the biggest thing that I think this situation

00:05:29   is instructive for is making us aware

00:05:32   of the things that we're dependent on

00:05:34   and that our apps are dependent on,

00:05:36   because we're always gonna be dependent

00:05:38   on something, it seems.

00:05:39   There's no way to really say,

00:05:41   okay, I'm gonna be completely independent,

00:05:42   because ultimately, you're gonna be,

00:05:44   I'm very reliant on Apple, for example, and iOS.

00:05:49   If Apple announced tomorrow that, hey, we've decided this iOS thing isn't really working

00:05:55   out, we're just going to turn it off.

00:05:58   We're just going to stop making iPhones.

00:05:59   I'm not saying that's likely, but if they did, my apps would stop working.

00:06:03   In the same way that if Parse going away meant that apps that relied on it go away.

00:06:08   Or maybe a more practical example on the Apple side of things is if they decide, you know

00:06:13   what, CloudKit isn't working out, we're going to turn that off.

00:06:16   Or those types of things.

00:06:18   or I rely on my hosting provider.

00:06:20   I host all my Linux servers on Linode,

00:06:23   and if they decide they're gonna go out of business

00:06:28   or they decide they're not gonna do the kind of hosting

00:06:30   that I need anymore, suddenly I'm in a big bit of a bind.

00:06:34   And so there's no way to avoid being dependent.

00:06:37   You're always dependent on something,

00:06:38   but there's a lot of dependencies

00:06:41   that you kind of have some choice in.

00:06:43   Do I wanna be so wed to a particular platform

00:06:47   back end or system? Or do I want to be a bit more flexible and do, you know, usually it

00:06:52   ends up like the more custom you make it and how much of it you control, you're going

00:06:56   to have more like portability and be able to be like, you know, if this particular host

00:07:01   goes away, I can just get another one. And you're going to be able to look at the trade-offs

00:07:05   and make more choices than if you're just all in on one thing. That's when you start

00:07:10   to get a little bit awkward.

00:07:12   awkward.

00:07:13   Yeah, and that's why the selection of what you depend on, this is why I'm usually very

00:07:18   conservative with these. Obviously I try to minimize how many external services and companies

00:07:23   and things I depend on, but you're always going to depend on something, as you said.

00:07:27   This is why I always try to choose as conservatively as possible. So yeah, Apple could shut down

00:07:33   the entire app store and that would really be disruptive for us, but that's very unlikely.

00:07:39   The App Store's continued success is pretty important

00:07:42   to Apple as well, so I've aligned my incentives

00:07:45   with this now pretty old and pretty important thing

00:07:48   to its parent company that it is very unlikely

00:07:51   that Apple will do that and that will be a problem for me.

00:07:54   Facebook shutting down Parse, this thing they bought,

00:07:58   did not have that kind of luxury.

00:08:00   If you were a Parse customer six months ago

00:08:01   and you're evaluating this dependency,

00:08:04   it isn't that important to Facebook to keep this running.

00:08:08   So that this was foreseeable,

00:08:09   that this was a high risk of happening,

00:08:12   that this company made this service,

00:08:13   it got big, it got bought,

00:08:15   the parent company didn't really depend

00:08:16   on its continued operation for their core strategy.

00:08:19   So this was always gonna be a risk, right?

00:08:22   So if Linode, Linode, however we're supposed

00:08:26   to be pronouncing it, I say Linode, you say Linode,

00:08:27   they say Linode, if that particular Linux VPS host

00:08:32   gets shut down, well, that's unlikely

00:08:35   'cause they're really big

00:08:36   and they've been around a while,

00:08:38   but even if that happens,

00:08:40   migrating away from that is not that big of a problem

00:08:43   because there are other Linux VPS hosts just like it.

00:08:48   And if every Linux VPS host went away,

00:08:52   you could get a Linux server somewhere

00:08:54   that behaved very similarly.

00:08:55   If every Linux server provider went away,

00:08:59   you could, as a last ditch, run one in your house.

00:09:02   You shouldn't, but you could.

00:09:04   So the transition options away from something

00:09:08   are also very important.

00:09:09   Parse shut down, they did a decent thing here

00:09:12   where they open sourced a big part of their server

00:09:15   and made it installable on your own stuff.

00:09:17   But what if they didn't do that?

00:09:18   Lots of things shut down and never do that

00:09:20   because they just either can't or won't

00:09:22   or don't feel like it.

00:09:24   So suppose you depend on an Amazon web service

00:09:29   for your business and Amazon shuts that down.

00:09:32   Most of the time, that is very hard to replace

00:09:35   'cause they are so custom and proprietary,

00:09:37   you can't just kinda do your own thing.

00:09:39   If you depend on any kind of high-level service like this,

00:09:43   then it is always a risk that the more custom

00:09:47   and proprietary and high-level something is,

00:09:49   the risk of it being hard to replace

00:09:52   if it ever does go away increases.

00:09:54   - Yeah, and I think that's ultimately probably

00:09:57   like the enticement and why it's this weird tension

00:10:00   that you find yourself in as you're developing a service

00:10:02   as you're thinking about a feature, you're saying, "If I do it with this high-level

00:10:08   construct that this company's providing, I can save myself a lot of time up front,

00:10:15   because I'm not having to build that again."

00:10:19   If there's this solution that they've come up with that means that user authentication

00:10:24   is just a thing that I can just plug into my app, and it handles all the secure password

00:10:31   storing and email resets and all that kind of stuff.

00:10:34   Say there's a service that does that off the shelf.

00:10:36   That's days, weeks, months of time that you weren't spending building that thing, but

00:10:42   instead you're just kind of more integrating directly into your application.

00:10:46   And so it's enticing.

00:10:48   And you're getting this enticement at the benefit of that upfront time, but it's sort

00:10:53   of at the detriment of this risk that you're increasing in your application.

00:10:58   And maybe that makes sense, like if you're just kind of prototyping something and throwing

00:11:02   it out there, or you aren't in a situation that's very time limited, that you have to

00:11:07   get, you know, if you don't ship your app in a month, it's you're going to miss some

00:11:12   kind of market window or opportunity that or like, that's the only amount of like, you

00:11:16   just have that much money to make a run at it, and you just kind of have to, then great,

00:11:21   like it, there's nothing bad about those types of things.

00:11:25   But it's this weird tension that you're finding of because you're so locked in at that point,

00:11:30   you're setting yourself up for difficulty down the road.

00:11:33   Because it's not necessarily your short-term benefit and long-term pain, because developing

00:11:40   it yourself, there's also long-term pain.

00:11:42   It's a different kind of pain, but you have to then be the one who's maintaining it, or

00:11:46   when security issues happen, you're the one going in and patching your web server or the

00:11:54   the Linux distribution you're installing on your servers

00:11:56   or whatever, at some point there's always

00:11:58   long-term challenges with these things.

00:12:01   But the difference is more one of you're totally locked in

00:12:05   and at the whim of whatever that company is.

00:12:08   And unless you're their biggest customer,

00:12:11   which for the kind of people who I imagine

00:12:13   listen to a show like this, you're unlikely to be

00:12:15   a service provider's biggest customer,

00:12:19   you're just gonna be kind of like,

00:12:21   sort of washed back and forth based on

00:12:23   make sense for them.

00:12:24   And that may or may not be a position

00:12:27   that you find yourself in that you'd be comfortable with.

00:12:31   This episode of Under the Radar is brought to you by Hover.

00:12:34   Quite simply, Hover is the best way

00:12:35   to buy a managed domain names.

00:12:37   And when it comes to buying a domain name,

00:12:38   Hover is the first place I check.

00:12:40   Now, when you have an idea for a project,

00:12:42   naming it can be difficult.

00:12:43   When you finally get that name, you

00:12:44   want to be able to quickly and easily get the domains

00:12:46   that you need.

00:12:47   Hover provides a simple, fast, and hassle-free method

00:12:50   of buying domains.

00:12:51   I don't want to be faced with 1,000 screens

00:12:53   and all these add-ons, high prices,

00:12:55   all these custom weird services that seem like scams.

00:12:58   I just want to get in, select what I need,

00:13:00   buy it and get on with my life and building my new idea.

00:13:02   Hover makes this very, very easy.

00:13:04   Their search is very nice.

00:13:06   It suggests things for you if nothing's available.

00:13:08   They can search all the TLDs, all the crazy new ones,

00:13:11   in addition to all the nice old ones.

00:13:13   And they have .com domains starting at just $12.99 a year.

00:13:17   Great prices on all the other ones as well.

00:13:19   All of these include Whois privacy for free

00:13:22   with every hover domain, because they believe

00:13:24   that you shouldn't have to pay extra for something like that.

00:13:27   Obviously, you want to keep your private information private.

00:13:29   They also have fantastic customer support.

00:13:31   If you want to call them, they have a no hold, no wait,

00:13:34   no transfer telephone support policy.

00:13:36   When you call them, you talk to an actual human being,

00:13:38   not a robot, not a menu.

00:13:40   You don't have to say like, operator,

00:13:42   like those stupid speech menus.

00:13:43   It's a real human being you can just talk to directly.

00:13:46   They pick up the phone.

00:13:47   And if you do, of course, prefer the robots,

00:13:49   they also have great support documents

00:13:51   and support guides on their website

00:13:52   for getting everything you need,

00:13:53   and you can email them as well if you'd like.

00:13:56   And they also have a valid transfer service

00:13:58   where they can take all the hassle out of switching

00:13:59   from your current provider,

00:14:00   simply because they do it all for you.

00:14:02   You can just give them your login to your old provider

00:14:04   and they will transfer names for you if you'd like.

00:14:06   All that for free, of course.

00:14:08   They have so much more great stuff.

00:14:09   They have volume discounts,

00:14:11   they have custom email addresses, storage and forwarding,

00:14:13   and so much more stuff.

00:14:15   Check it out today at hover.com.

00:14:18   Use code perspective at checkout.

00:14:20   That is once again, Code Perspective at checkout,

00:14:23   and you will get 10% off your first purchase at hover.com,

00:14:26   and you will show your support for Under the Radar

00:14:28   and all of Relay FM.

00:14:29   Thank you very much to Hover for sponsoring this episode.

00:14:32   - So it seems like we should probably also dive now,

00:14:34   dive into kind of like what we do, how we approach this,

00:14:38   because I think we both have found ourselves

00:14:41   at the end of the thought process

00:14:44   on how we should bake back ends for our service

00:14:46   with the like, well, we're gonna build them ourselves,

00:14:48   and we're gonna build custom applications

00:14:51   running on Linux VPSs that we use.

00:14:56   And probably worth saying why we kind of do that.

00:15:00   - I mean, to me, first of all,

00:15:02   it's all about control for me.

00:15:03   I'm a control freak, and I wanna do everything myself,

00:15:05   and I want everything to be under my control,

00:15:08   because I don't want to have major parts of my roadmap

00:15:12   dictated by a dumb change in my host that,

00:15:16   oh, all of a sudden this entire thing I depend on

00:15:18   is shutting down and I gotta change that.

00:15:21   Apple gives us enough of those things,

00:15:23   with new device releases and everything,

00:15:25   but those are kind of an unavoidable part

00:15:27   of working with Apple.

00:15:28   But when it comes to running your services,

00:15:30   you control a lot more of that

00:15:31   and you can avoid those things.

00:15:33   And so I love that part of it.

00:15:35   And for me, it's also a lot about capability and cost,

00:15:39   low cost and just being able to do a lot.

00:15:43   CloudKit is very appealing in a lot of ways.

00:15:45   And if I was making a new app today,

00:15:48   I would think very hard about whether I could just do it

00:15:50   all on CloudKit and whether that would be

00:15:51   the right move for me.

00:15:53   But it is still limited in what it can do, what it can't do.

00:15:58   And so for me, a website, like a regular Linux backend,

00:16:02   is the default for me.

00:16:03   I know how to do it.

00:16:04   It really isn't that hard,

00:16:06   which we'll get into in a little bit.

00:16:07   It really isn't that hard.

00:16:08   And it is surprisingly capable for surprisingly little cost.

00:16:13   - Yeah, exactly.

00:16:14   I think the reasons are fairly similar for me. I think the thing that I like most is

00:16:20   being able to tailor the backend of my application to not necessarily the application, but it's

00:16:27   tailored to the way that I think and the way that I solve problems and the way that I'm

00:16:31   thinking about the problems that are being solved in my app. So when I'm dealing with

00:16:37   something like sync, like the generic term for one of the hardest problems in computer

00:16:42   science. I like that I can—I'm solving that problem in a way that makes sense to

00:16:50   me, that I'm not having to kind of shoehorn my application and the way I think about it

00:16:55   into the model that a service provider provides. And they say, "Well, we handle conflict

00:17:01   resolution using last updated wins," or something like that. And maybe that works,

00:17:07   maybe it doesn't. And so when you build it yourself, I actually understand it. I have

00:17:12   to go—I've gone through and I've made the decisions at the various levels of like,

00:17:18   "Well, I want this to work this way, I want this to work that way." And so then down

00:17:22   the road, when I'm debugging something, I have a better understanding of how I expect

00:17:28   it to work, and when things go wrong, I have a sense of where they might be going wrong.

00:17:32   Like, is this an app problem, or is this a web service problem? And ultimately, it probably

00:17:38   also just makes my apps better and makes me a better programmer. Like, having this breadth

00:17:42   of experience that at this point, I can build something all the way from the UI in the application,

00:17:50   the business logic inside of the application, and then all the way through to the web service

00:17:57   that's managing that information and a database at the back that's storing that information.

00:18:03   Having been able to do all those things is just good for me from a career and personal

00:18:09   development perspective. Like, I've learned to solve more problems doing it this way that

00:18:15   ultimately I think makes me a better developer. Like, I write less. My apps are probably better

00:18:20   because they're the kind of calls they're making. Like, I know what the server is trying to do with

00:18:25   those calls. And so you don't end up just like, "Well, this is like the naive, obvious solution.

00:18:30   I'll just kind of throw all this data at the server." Or I'll, "Hey, let me just ask for

00:18:35   for all of it, all the time, and because if the servers are overwhelmed, that's not my

00:18:41   problem. Those are things that ultimately probably make my apps better. And like you

00:18:45   were saying, it is kind of crazy how inexpensive it is to do a lot of these things now. A lot

00:18:52   of my things are just backed by two or three VPSs that cost—at a basic one, it's like

00:19:00   $20 a month or something like that, $20, $40 a month. For a lot of my applications, I end

00:19:04   up spending maybe $100 a month in servers, and that's really not too bad for the kind

00:19:13   of capability and the throughput and the number of users that you can support even just at

00:19:18   that level.

00:19:19   Oh yeah, I mean, even the $20 a month server level on a modern host like LineOut or Digital

00:19:25   you can get so much for this money now.

00:19:28   And when you're using boring old fast tools

00:19:31   like MySQL or Postgres and you have like

00:19:35   a modern web language in front of it,

00:19:37   you know you have, even in the old ones,

00:19:38   PHP, Ruby, you know like Python,

00:19:41   or more recently you might have like Go,

00:19:44   these are so fast, you can do so much,

00:19:47   you can support so much usage,

00:19:48   it's way more than you think.

00:19:50   'Cause now you have these modern processors

00:19:52   doing the virtualization,

00:19:53   You have SSDs on almost all these hosts now.

00:19:56   It is incredibly fast to do.

00:19:58   And so you really can support a lot on very little hardware.

00:20:02   - Yeah, and I think ultimately that makes it a lot easier.

00:20:04   It's, they're the hardest problems I've ever had to solve.

00:20:08   The only time I kind of regretted doing backends myself

00:20:11   is they were the early days of Feed Wrangler,

00:20:14   my RSS syncing system, which I was doing stuff

00:20:18   that in retrospect was really foolish

00:20:21   and was just crushing my database.

00:20:24   My Postgres database was just constantly dying

00:20:28   and falling over.

00:20:29   And in retrospect, it was because I was being,

00:20:34   I'd made a few really bad assumptions upfront,

00:20:36   but even there, that's the only time I've ever really

00:20:41   had to do any low-level performance tuning

00:20:44   of any of my applications.

00:20:46   Otherwise, just out of the box,

00:20:48   things are just fast and work and it's fine in a way that,

00:20:53   it would be problematic if I really needed to be

00:20:56   a database administrator, a serious DBA,

00:21:00   whatever they call them these days,

00:21:02   doing that kind of work, but most of the times

00:21:04   I just install Postgres with the defaults,

00:21:07   tweak a few things the way I like it, and then it's fine.

00:21:10   And it just runs quickly enough for,

00:21:13   that my users don't even really notice

00:21:15   any kind of performance issues or problems.

00:21:17   - Yeah, I mean, you might think,

00:21:19   if you haven't done this before,

00:21:21   or the last time you did this was like 10 years ago,

00:21:23   you might think that running servers requires

00:21:25   lots of low-level tweaking and performance tuning

00:21:28   and getting these right config variables

00:21:29   to exactly the right buffer size and everything,

00:21:31   and you don't really need to do that anymore.

00:21:33   It's very rare for most people to need to get

00:21:37   that down into the nitty-gritty stuff.

00:21:38   It really is, as you said,

00:21:40   you can just install these things with the defaults,

00:21:43   and usually you're fine.

00:21:45   That's usually all you need to do,

00:21:46   because everything is just so good now.

00:21:48   There's so much headroom.

00:21:49   The software is very mature in a lot of these things,

00:21:53   and the hardware is very mature too.

00:21:54   So you really get a lot with just the defaults now.

00:21:58   - And I think one thing that I was kind of looking forward to

00:22:01   when we got into this topic is you said

00:22:03   you had a few little pro tips

00:22:05   for getting into this kind of administration,

00:22:07   because I think it can be a little bit intimidating

00:22:11   to go and install Linux.

00:22:15   Then you start to like, well, what version of Linux?

00:22:17   What should I do?

00:22:18   How to get started?

00:22:19   And I remember it being a little intimidating,

00:22:21   but at least the fun thing is once you get going,

00:22:24   there's tremendous resources,

00:22:26   and you can just kind of get going.

00:22:27   And once you know it, you know it,

00:22:28   because this stuff doesn't really change.

00:22:31   - Yeah, basically, Google is your friend,

00:22:34   not the corporate structure, but the search engine,

00:22:37   Stack Overflow, all these things.

00:22:39   These are your friend, because lots of people

00:22:42   have been running Linux servers for years.

00:22:44   And as you said, the tools and the commands

00:22:47   and what you need to do doesn't change very often.

00:22:50   Typically, you learn this stuff like once

00:22:52   and you have to learn something new maybe every two years.

00:22:55   It's pretty stable, it doesn't change much.

00:22:58   So number one tip I can give is to pick a very popular

00:23:03   but somewhat conservative Linux distribution

00:23:06   to do this with.

00:23:07   For years, I recommended CentOS,

00:23:09   which was based on Red Hat Enterprise Linux.

00:23:11   I think today, I think Ubuntu might have more momentum behind it. So I actually just managed

00:23:17   my first Ubuntu server recently and things are a little bit different but I was able

00:23:23   to figure it out. So between CentOS and Ubuntu, you can't really go wrong. Turn on auto updates

00:23:31   for as much of the system software as it makes sense to do that for. Usually every major

00:23:35   distro has a way to do this. It's very straightforward. That will take care of most security problems

00:23:40   for you. If you basically are not an idiot, which you're not, trust me, if you're not

00:23:46   an idiot and if you leave things mostly at their defaults with the distro and what it

00:23:50   comes with, modern Linux distros are very secure by default because they know that that

00:23:54   matters, like the defaults matter, so they've all adopted pretty conservative and pretty

00:23:59   secure defaults for the most part. Keeping things updated automatically is very easy

00:24:03   and things like that. On other high level stuff, only run the software that you need

00:24:08   to be running. And they're all very good at letting you manage this. So like, if you have

00:24:12   a server that you have your website on, don't also install like, "Oh, let me install FTP

00:24:16   so I can like trade files with my friends." No, just leave that off. That's just a liability.

00:24:20   Just don't do that, you know. Install what you need to install. And if you want to play

00:24:24   around with different things, you can create a second VPS for like five or 10 bucks a month

00:24:27   and play around on that. Don't play around on your main servers. Run only what you need

00:24:31   to be running on them. Take advantage of the built-in isolation in Linux machines, especially

00:24:38   with regard to networking. Almost every service that you'll be running will have some kind

00:24:44   of like listening port where you can say, "All right, this database should listen on

00:24:47   this interface, on this port." If you only have one server, make the internal stuff listen

00:24:52   on local host so that you can't log in to MySQL from outside. Like, you shouldn't need

00:24:58   to do that anyway. You should be doing things on the server if you need, you know, management

00:25:01   stuff. Lock that down. If you have multiple servers, use private networking. Every host

00:25:05   that's worth their salt supports private networking between your own machines. So if you have

00:25:10   multiple servers and you need to talk to each other, have them talk to each other only over

00:25:14   private interfaces. Have things like MySQL or Memcache listen only on private or local

00:25:19   interfaces. That helps a lot. Just make it, don't rely on like keeping your passwords

00:25:25   secure. Make it so that passwords don't even work from the outside. So that also applies

00:25:29   with things like SSH when you're doing login, remote login.

00:25:32   So disable root logins once you have a user set up,

00:25:35   have that user have pseudo access with the password,

00:25:38   and then that user account that you're logging in as,

00:25:40   say you're logging in as David,

00:25:41   make that the only user that can log in via SSH

00:25:43   and make that key authentication only,

00:25:46   disable password authentication in SSH.

00:25:49   This is very simple stuff to do.

00:25:50   You can Google how to do it.

00:25:52   So that, right there, you have no way

00:25:54   to log in with a password.

00:25:56   You have to have the encryption key to log in.

00:25:59   That knocks out massive brute force possibilities

00:26:03   and everything.

00:26:04   That helps so, so much.

00:26:06   Between that and private networking for private services,

00:26:09   you really eliminate a lot of problems.

00:26:11   Now, moving on slightly to user data.

00:26:14   Collect as little user data as possible to get your job done.

00:26:19   Because worst case scenario, somebody hacks into your server.

00:26:22   Worst case scenario, they take your database.

00:26:25   What do they have?

00:26:26   Think about it.

00:26:27   When you're designing your database,

00:26:29   you're designing your service,

00:26:30   what information do you really need from people

00:26:32   and what can you get away with not having?

00:26:34   If you don't need to get people's email addresses,

00:26:36   don't get their email addresses.

00:26:38   If you're taking passwords from people, hash those

00:26:41   so that people aren't getting just the MD5,

00:26:44   for God's sake don't do that.

00:26:45   Use secure password hashing like Bcrypt on strong settings.

00:26:49   There's lots of good practices for this,

00:26:50   lots of things to tell you how to do this.

00:26:52   I've considered even for overcast,

00:26:54   I do have the email addresses for people

00:26:56   because I figure I have email addresses

00:26:58   and I have hashed passwords with strong bcrypt,

00:27:01   but I'm like, I've been thinking recently,

00:27:02   do I even need the email address?

00:27:04   Could I hash that too?

00:27:06   'Cause then you have, if you steal my database,

00:27:08   you just have no email addresses.

00:27:10   That would be amazing.

00:27:11   And I was thinking, if you hash the email address

00:27:14   so it works just like the password basically,

00:27:16   then you could still have logins,

00:27:18   you could still have password resets.

00:27:20   The only thing you really can't do is,

00:27:22   I can't email people randomly out of my database,

00:27:26   but I've never done that.

00:27:27   I don't send a newsletter, I don't do any of that stuff.

00:27:31   So, you know, stuff like that.

00:27:32   Think about just like what data you have,

00:27:34   what you're collecting, and what you can afford

00:27:36   not to collect.

00:27:37   Simple security measures beyond that, you know.

00:27:41   You should have database backups.

00:27:42   You should also be encrypting those backups.

00:27:44   There's built-in stuff, there's a crypt command

00:27:45   you can pipe tar through and everything like this.

00:27:48   Really simple stuff on Unix to do all this very securely.

00:27:51   Make sure, though, that you are testing these backups.

00:27:54   Make sure you can decrypt them.

00:27:55   So that's very important.

00:27:57   Don't store the encryption key only on the server

00:28:00   because then if that server gets wiped

00:28:02   or gets lost or whatever, you've lost your data

00:28:05   and your backup decryption key, that's no good.

00:28:07   One strategy I employ there is I write my database backups,

00:28:13   I copy them onto a write only S3 account.

00:28:17   So the credentials that are on the machines

00:28:20   can only write to the bucket,

00:28:21   they can't read or delete from it.

00:28:23   So that way if somebody hacks into the machine,

00:28:25   they can't also go and delete all my backups.

00:28:27   So I have separate credentials that I can pull the backups

00:28:32   off of there and restore that never live on my servers.

00:28:34   Those stay with me and my personal documents.

00:28:36   Those never live on the servers.

00:28:38   So keep things as secure and separate as you can

00:28:41   just by design like this.

00:28:43   And that's really about it for basic security stuff.

00:28:46   It really is not as hard as you think.

00:28:49   And you don't have to do very much.

00:28:51   You don't have to constantly keep on top of your servers

00:28:54   and be constantly babysitting, for the most part,

00:28:56   you set it up and it basically runs itself.

00:28:58   And if you set it up with sensible default

00:29:00   using conservative software and some basic security settings

00:29:03   like what I've said here, you can be pretty much fine.

00:29:06   - Yeah, exactly, and I think that it's the kind of thing

00:29:08   that if you can't do this kind of thing,

00:29:12   and if everything that Marco just ran through

00:29:15   is complete gibberish to you,

00:29:17   you should probably do something about that.

00:29:19   It's an important skill to be a developer,

00:29:21   to understand some of these basics,

00:29:23   sort of like the fundamentals that run the internet.

00:29:26   Like, you should understand what this is

00:29:28   and to sort of take control of that.

00:29:30   And like you said, just get a $5 like,

00:29:35   VPS somewhere and start messing around

00:29:36   and start seeing, you know, learning.

00:29:38   'Cause that's how most people learn this stuff.

00:29:40   You just start doing it and you get better at it.

00:29:43   - All right, we're out of time this week.

00:29:45   Thanks for listening everybody.

00:29:46   And next week we're gonna go into a little more detail

00:29:48   about our server setups, lessons we've learned,

00:29:51   and how to minimize the workload.

00:29:52   We'll see you next week.

00:29:54   - Bye.