0:00:00[Music]
0:00:02this is hypercritical weekly talkshow
0:00:05ruminating on exactly what is wrong in
0:00:07the world of Apple and related
0:00:08technologies and businesses
0:00:09nothing is so perfect that it can't be
0:00:12complained about by my co-host John
0:00:14siracusa i'm dan benjamin this is
0:00:16episode number 57 today's Friday March
0:00:192nd want to make sure that we thank our
0:00:21two sponsors right at the top of the
0:00:23show tiny letter comm send email
0:00:26newsletters the easiest most simple way
0:00:29possible and Shopify create your own
0:00:32online store today we also want to
0:00:35mention the bandwidth for this episode
0:00:36is provided by Mac Mini Kolo I host
0:00:39Ansel the servers there to help run 5x5
0:00:41and you can - Mac Mini Colo net / 5x5 as
0:00:45a special discount for this is why we
0:00:48need faster file systems I'm learning
0:00:50how to use computers today John how are
0:00:52you I can relaunch my apps a little bit
0:00:55faster you have all SSD you've got
0:00:57mirrored and striped SSD drives in all
0:00:59of your chips Dave nanion the guy who
0:01:02makes SuperDuper I doesn't trust the SSD
0:01:05drives typically he doesn't trust Lee
0:01:08you know supposed to trust any drive Oh
0:01:10point is I'm gonna have but he trusts
0:01:12those less than he distraught he he
0:01:14distrusts them even more than he
0:01:16distrust his other what about the people
0:01:17of emailed asking you what about SSD is
0:01:20John yeah I probably distrust me too
0:01:22well today's show will be a follow up
0:01:24show on file systems so we will talk
0:01:28about all these issues we don't have any
0:01:29other thing to talk about today no no
0:01:32iPad 3 comments or predictions maybe
0:01:35we'll throw that in at the end ok
0:01:38alright really going here now that my
0:01:40audio is actually working I would like
0:01:41to blame this audio stuff online but
0:01:43this happened in Snow Leopard too like
0:01:45you know everything's all set up and the
0:01:46last thing I do is you know your testing
0:01:481 2 or 3 into my mic and noting the
0:01:51meters don't move at all I'm plugged mic
0:01:54plugged back in nothing so the old
0:01:57reboot this is the only time I ever
0:01:58reboot my Mac you know is when the audio
0:02:00stops working and yeah before you
0:02:01started doing the show you had an uptime
0:02:03of like three years and soon as you have
0:02:05this show you're lucky to get the full
0:02:06seven days well now as soon as I had my
0:02:09uptime was the space between security
0:02:10updates to Mac OS 10 the ones that made
0:02:12you restart
0:02:14alright let's go do it follow up I was
0:02:19listening to myself in hypercritical
0:02:21episode 56 and in the car this week as I
0:02:25always do and was cringing as I heard
0:02:27myself repeatedly used the word defer
0:02:29about deferring the costs and I wasn't I
0:02:33didn't mean deferring the costs as in
0:02:35pushing them off in time this was in
0:02:38connection to Apple's data centers and
0:02:41the cost of running them the cost of
0:02:43running iCloud and the developers using
0:02:45iCloud Hippias and no sooner had I
0:02:49gotten out of the car and logged onto
0:02:51the computer that I got a tweet from
0:02:54someone Nathan heretic I hope I got that
0:02:58right telling me that I should not be
0:02:59saying the word defer and I said I know
0:03:01I just heard myself saying in the car
0:03:02was horrible I should have said either
0:03:04offset or defray defray is probably the
0:03:07word that was getting mapped wrong in my
0:03:08brain and the point I made to him on
0:03:10Twitter is this interesting that
0:03:11listening to myself as soon as I said it
0:03:14you can hear all that's wrong with what
0:03:15is this guy talking about right but
0:03:16saying it I said it like 9 times without
0:03:19my brain triggering the warning you were
0:03:22using the wrong word
0:03:23right so I was kind of like writing and
0:03:25proofreading when I'm speaking things
0:03:28come out of my mouth that don't match
0:03:30what my brain was thinking so I
0:03:32apologize for that and thanks for the
0:03:33correction Nathan a correction from Mark
0:03:38Hofman about notifications in
0:03:40Notification Center in Mountain Lion
0:03:42we're just asking how that was one of
0:03:46the api's that was getting swept up into
0:03:48the group of things that are only
0:03:51available to Mac developers who put
0:03:53their application on the Mac App Store
0:03:54and I was saying how that was weird
0:03:56because yes notifications can be
0:03:59synchronized through the cloud from one
0:04:01device to the other and show up on your
0:04:02desktop for an event triggered on your
0:04:05phone and vice versa but notifications
0:04:08also work locally if you just have a
0:04:09local Mac application that wants to put
0:04:11up a little notification dialog says hey
0:04:14something happened over here take a look
0:04:15and they want to do something more than
0:04:17just have a dock icon badge or bounce or
0:04:19something and Notification Center is for
0:04:23that as well I said boy it's a shame
0:04:24that those guys that you can't
0:04:27use notification center for that yeah if
0:04:29you're not in the Mac App Store even
0:04:31though it has nothing to do with that
0:04:32cloud well it turns out that if you do
0:04:35only local notifications you can use
0:04:37Notification Center in Mountain Lion
0:04:39even if you're not on the Mac App Store
0:04:40so the only thing that you're barred
0:04:42from is so-called push notifications
0:04:45notifications that are synchronized
0:04:45through Apple's servers so that's good
0:04:48to know
0:04:52Andrew Lawrence was the first person to
0:04:54chime in to tell me that the NetApp
0:04:57Oracle lawsuit that I mentioned as a
0:04:59possible reason that Apple bailed on ZFS
0:05:01was settled shortly after son was
0:05:04acquired by Oracle I put a link to the
0:05:07press release in the show notes it was
0:05:09actually September 2010 that was still
0:05:11after long after Apple had decided had
0:05:15made his decision on ZFS oh I'm not sure
0:05:17settling helped to hurt one with the
0:05:19other but by then it was too late but
0:05:20it's a good point that it actually has
0:05:22been settled and it's not dragging on
0:05:23the settlement was one of those ones
0:05:24where it's you know dismissed without
0:05:27prejudice and the terms of the agreement
0:05:29are confidential so we don't know what
0:05:30actually happened they just probably
0:05:31exchanged a sufficient amount of money
0:05:33between parties and everything's all
0:05:35better JP I don't know I just have JP
0:05:40maybe that's all I could get out of his
0:05:41Twitter handle or email or something was
0:05:44the first to point out that btrfs is
0:05:46pronounced butter FS I had heard that
0:05:49enunciation but wasn't aware it was the
0:05:50official apparently it is the official
0:05:52pronunciation I don't I think I don't
0:05:54really like the idea of food mixing with
0:05:56my file systems so butter if that sounds
0:05:58a little weird to me but if it's butter
0:06:00FS then it's butter FS so I will
0:06:02endeavour to say butter FS from now on
0:06:08JP has some other information he
0:06:12apparently this is why he just says JP
0:06:14hey he worked in the Genius Bar for a
0:06:16while in an Apple Store and here's a
0:06:19quoting from his message we got almost
0:06:21as many okay hard drives according to
0:06:23smart status which were unrepairable in
0:06:25Disk Utility as we did actual fail to
0:06:27harden so what he's what he's saying is
0:06:29that you know people would come in to
0:06:30the Genius Bar and they would have a
0:06:31problem with their hard drive and half
0:06:33the time there was a hardware problem
0:06:35with the hard drive but the other half
0:06:37of the time it was just hfs+ corruption
0:06:39so
0:06:40fully 50% of their hard drive related
0:06:43problems were not due to hardware but
0:06:45were due to HFS corruption and he says
0:06:47all in all hard drive issues accounted
0:06:49for about 30% of all issues so about 15%
0:06:52for a directory failure so he's saying
0:06:5415% of all problems that come to the
0:06:56Genius Bar and his experience were due
0:06:58to hfs+ corruption and across all of
0:07:01apples customers this is a pretty big
0:07:03number I'm actually surprised that I'm
0:07:06surprised that the 30% of all Genius Bar
0:07:08t-shirts were hard drives a hard drive
0:07:11related but it's really said that half
0:07:13of the HOS 15% of all possible Guinness
0:07:16bar things are due to HFS+ corruption so
0:07:18it shows that this is really is a
0:07:19problem an apple I mean apples gotta
0:07:20track these stats and has to know geez
0:07:22you know 50 percent of the people come
0:07:24to the Genius Bar is due to bugs in our
0:07:26own software we really need to address
0:07:28that filesystem thing a couple people I
0:07:32don't remember who sent this might have
0:07:34been the chat room but I don't have a
0:07:35attribution for it send me a link to an
0:07:37lwn net article called a short history
0:07:40of butter FS and that I hadn't known too
0:07:45much of a butter of has except that it
0:07:46was the kind of like a you know the GPL
0:07:49Linux directed ZFS like filesystem and I
0:07:55also knew that it wasn't quite done yet
0:07:56so both of those are true but this this
0:07:59article that it linked in the show notes
0:08:01is written by someone who worked
0:08:03extensively on ZFS and it goes through
0:08:05the history of how butter FS came to be
0:08:07and what's different about it than ZFS I
0:08:12think it's worth I guess all worth
0:08:14hitting the highlights of instead of
0:08:16just saying go read this article because
0:08:17it is a long article and maybe if you're
0:08:18not really into this you won't read the
0:08:20whole darn thing
0:08:22so the be actually the be TR part in
0:08:26butter FS stands for be trees it's
0:08:29shorter be trees have been so it's
0:08:30another victim of the great valve sort
0:08:32shortage of 1972 which is stricken the
0:08:35UNIX landscape and Keaton used to be a
0:08:36problem they didn't even have an e
0:08:39leftover for the create system call
0:08:41that's how bad it was
0:08:42but apparently it's still going on so a
0:08:47file systems use b-trees because they're
0:08:51an efficient data structure for looking
0:08:53stuff up and
0:08:54starting stuff and deleting stuff if you
0:08:55take any computer science course you'll
0:08:57eventually do a chapter on trees and
0:08:58they're all different kinds of trees and
0:09:00you can see in the diagram of them it's
0:09:01like a circle with a bunch of lines
0:09:02connecting and there's you know out of
0:09:04each little circle there are you know
0:09:07one or more lines and then there are
0:09:09circles connected those and it just
0:09:10makes a big tree structure bee trees are
0:09:13a particular kind of tree that you know
0:09:16the characteristics of trees are how
0:09:17many how many different branches come
0:09:19out of each one how do you insert
0:09:21something into the tree how do you
0:09:22remove something from the tree I'm not
0:09:24going to go into the sto details of bee
0:09:26trees in particular but suffice it to
0:09:28say that B trees are specific variant of
0:09:31B trees are among the most efficient
0:09:33general-purpose tree structure for the
0:09:36you know they do everything pretty well
0:09:37you can insert things pretty quickly you
0:09:39can delete things pretty quickly they
0:09:40don't get unbalanced they don't require
0:09:43a lot of maintenance to keep balanced
0:09:44ever it's kind of like a good middle
0:09:46ground for the types of things that you
0:09:47might want to keep track of in a file
0:09:50system you do because you don't want to
0:09:51be something to be really quick to read
0:09:53from but really slow to write because
0:09:54file systems tend to do all that stuff
0:09:56read write and delete in and generally
0:09:58you know equal measures but B trees are
0:10:04not particularly good when it comes to
0:10:06the strategy the ZFS implements where
0:10:08they never overwrite data in place you
0:10:11know that's the this is the
0:10:12copy-on-write strategy where if you have
0:10:14new data to write write in to a new
0:10:16location
0:10:17rather than writing over on top of the
0:10:18open and if you want a mod that's to put
0:10:20the copy-on-write means and saying if
0:10:21it's time for you to write some data
0:10:23instead of overwriting the data find
0:10:25where that data is and then make a new
0:10:27copy of it somewhere else it's different
0:10:29than the old one but B trees are not
0:10:32compatible with that at all because if
0:10:34you were to naively implement copy and
0:10:35write with the B tree then anytime you
0:10:37changed anything the entire tree from
0:10:39top to bottom would have to be changed
0:10:40because the changes would propagate back
0:10:43up the tree because you're making a new
0:10:44instance so almost all B trees and file
0:10:46systems and by the way HFS+ and H of s
0:10:49use B trees extensively B trees and file
0:10:52systems tend to be updated in place
0:10:53because that preserves all the desirable
0:10:55performance characteristics of a B tree
0:10:57and once you start saying oh we can't
0:10:58update anything in place we always had
0:10:59to write a new dated a new location and
0:11:01B trees seemed useless so many years ago
0:11:05someone named Oh had Road
0:11:07a I don't know if I got this name right
0:11:08OHA D ro deh wrote an academic paper as
0:11:12people in academia tend to do about a
0:11:15new kind of bee tree that was friendly
0:11:18to copy-on-write he found a way that if
0:11:20you all you change bee trees in this way
0:11:21then we actually can get reasonably good
0:11:23performance characteristics while still
0:11:25not ever updating data in place and this
0:11:29academic paper probably sat around for a
0:11:30while until someone named Chris Mason
0:11:32found this and said hey if I have a bee
0:11:36tree structure that's also friendly -
0:11:37copy-on-write maybe I can use that in a
0:11:40new file system and his big insight for
0:11:44creating a new file system was to decide
0:11:47that everything in the file system will
0:11:50be stored in one of these copy-on-write
0:11:51friendly bee trees normally you have bee
0:11:54trees first restoring like oh I'll keep
0:11:56track of directory entries with the bee
0:11:59tree but when I'll have a separate data
0:12:00structure for keeping track of free
0:12:02space like a bitmap or something and
0:12:03I'll have a separate structure for
0:12:04keeping track of extensors you know it
0:12:08there are usually different kinds of
0:12:09data structures tailored for each
0:12:10purpose he decided well this data
0:12:12structure looks so great what if
0:12:13everything is stored in one of these
0:12:15copy IRB trees I'll star the directory
0:12:17entries the the free space lists the
0:12:19extents that the file data itself
0:12:22everything in one of these bee trees and
0:12:23that's a nice unifying concept because
0:12:27it means that you only really need to
0:12:28have one code path once you get your
0:12:31code for reading writing and modifying
0:12:32these bee trees like down pat and
0:12:34debugged you're all set you don't have
0:12:36to say okay well what about the code
0:12:37that manages the free space and what
0:12:38about the code that manages the direct
0:12:40Rangers well you not not 17 different
0:12:42algorithms and 17 different data
0:12:43structure just one thing and you know so
0:12:45the code is simpler smaller and a nicely
0:12:48uniform and programmers love this type
0:12:49of thing so that's the basis of btrfs
0:12:53building an entire file system on this
0:12:55this new data structure and that's very
0:12:59interesting to me I didn't I didn't know
0:13:01this before I had read it definitely
0:13:02hits all of my buttons of her computer
0:13:06science and programmers like oh it's
0:13:08beautiful and elegant and has great
0:13:09performance characteristics and has all
0:13:10these features of ZFS the article goes
0:13:12into much more detail about how ZFS and
0:13:15btrfs
0:13:16are the same and how they differ they
0:13:19both kind of arrived similar
0:13:20from different directions and they do
0:13:23have different advantages and
0:13:24disadvantages or asset each other so I
0:13:25encourage everyone to actually follow
0:13:26that link and read it is not that long
0:13:28and it's not too jargon inducing it's a
0:13:30short history of BTR oh I almost added a
0:13:32butter fs and we will be revisiting this
0:13:37topic in a bit once we get through the
0:13:39official follow up Lenny P Robert writes
0:13:44to tell me that there is a program that
0:13:48will look through your HFS+ volume and
0:13:54try to detect if there are any errors
0:13:57and how does it do this it check sums
0:14:01everything and writes a bunch of little
0:14:03checksum files to each directory and
0:14:05then the next time you ask the check it
0:14:07says ok I have this file here this has
0:14:09the checksum supposed to be blah
0:14:10I will recheck some all these files and
0:14:12see if it matches a lot of people were
0:14:15asking me hey I'm afraid that my data is
0:14:17going bad on my HFS+ volumes how do I
0:14:19tell and I did I had wasn't aware of
0:14:24this program before but other people
0:14:26wrote in with it so I replied so this is
0:14:28one of the tools available to you now if
0:14:29you think about it for too long you
0:14:31realize that this program which is
0:14:32called integrity checker by the way it's
0:14:36kind of in a catch-22 because it's
0:14:41storing the check sums on the same disk
0:14:44that you're not trusting to make sure
0:14:48your data is ok you know what I mean so
0:14:50if you have a bit error and it happens
0:14:53to be an error in the checksum file then
0:14:57it would look like all your data is
0:14:57corrupted and really it's the checksum
0:14:59that's corrupted so the very least this
0:15:01might tell you something is wrong what
0:15:04exactly is wrong what you can do but I
0:15:06don't know but you could use this for
0:15:08example to keep proving to yourself that
0:15:10your backups are the same as they were
0:15:12the last time you know like I haven't
0:15:15modified anything let me look at my time
0:15:16machine back up as of three months ago
0:15:18is that data still exactly what it was
0:15:20when I wrote it three months ago
0:15:22or is there something wrong with it when
0:15:24there is something wrong you may not be
0:15:25able to tell exactly what's wrong but at
0:15:26least you know your data is bad so I
0:15:29have a link to this product in the show
0:15:30notes I have not tried it myself I don't
0:15:32even know how much it costs
0:15:34but for people who are paranoid about
0:15:36data integrity it's it's seems that
0:15:39third parties are taking up the mantle
0:15:40of attempting to do something about this
0:15:44problem as you might imagine anything
0:15:47that's going to grind over your whole
0:15:48disk and checksum every single thing and
0:15:50check their the check sums are the same
0:15:51it's going to be tremendously i/o
0:15:53intensive and it probably takes a really
0:15:55long time that's what you get when you
0:15:58have to do it on top of an existing file
0:16:00system instead of having it built in and
0:16:05Gregory Williams pointed me to another
0:16:08third-party product this one allows you
0:16:13to tap into HFS+ native compression that
0:16:17was introduced in Snow Leopard I've
0:16:18talked about in last show how there was
0:16:20no like UI for that in the in the OS and
0:16:24Apple decided to compress a whole bunch
0:16:26of files sort of out of the box but you
0:16:27didn't really have any control over that
0:16:29well this is a preference pane called
0:16:31clusters from late night software and
0:16:35with it you can tell it hey any files
0:16:39under a particular folder flip that
0:16:41little HFS+ compression bit and compress
0:16:42them for you and it's complete
0:16:43transparent encryption you don't know
0:16:44they're encrypted there's no like
0:16:47decryption phase that you can see it all
0:16:49happens under the covers in the file
0:16:50system so you can save some space with
0:16:53this now don't turn this on for a folder
0:16:55full of QuickTime movies or something
0:16:56because they're already compressed and
0:16:58you're not going to get much more out of
0:16:59them if you have some easily
0:17:00compressible data and you would like to
0:17:01have it transparently compressed for you
0:17:03this may be the ticket on the other hand
0:17:07it's possible that there's a good reason
0:17:10that Apple has not exposed the HFS+
0:17:13compression in the user interface so I
0:17:17think this type of software is the type
0:17:19of thing that exists on the borderline
0:17:20between what Apple what Apple's
0:17:22operating system is capable of and what
0:17:24they recommend third-party developers
0:17:25use and we will touch on that again
0:17:27actually later in the show and I also
0:17:32put a link in the show notes to the
0:17:34section about hfs+ compression in my
0:17:37snow leopard review from 2009 alright so
0:17:42that's the end of the official follow-up
0:17:43and now we begin the unofficial fall off
0:17:46which is the main topic of the show
0:17:47which is more
0:17:48files isms because believe it or not the
0:17:50end of the last show I thought of 20
0:17:51things that I didn't talk about related
0:17:54to file systems and a lot of people
0:17:55asked about them hmm well let's do our
0:17:57first sponsor and then we'll kick that
0:17:58part off what do you think sounds good
0:18:00our first sponsor is a longtime sponsor
0:18:02shopify calm love these guys
0:18:05very simply I've told you about them
0:18:07before create your online store today
0:18:09with Shopify they make it super simple
0:18:12to make your own store and you can sell
0:18:14whatever you want one of the questions
0:18:16that I had people were asking me about
0:18:17this so well that's perfectly fine if
0:18:19you're selling you know physical goods
0:18:20if you're selling a t-shirt or you're
0:18:22selling electronics or you have a
0:18:24Kickstarter project that you know you
0:18:26need the storefront for it what if I
0:18:28want to sell something electronic what
0:18:29if I wrote a book you know and I want to
0:18:32sell the PDF of it or what if I have
0:18:34something that I want to control down
0:18:35yes of course you can do that with
0:18:38Shopify dot-com there are tons of
0:18:39add-ons and tons of little things that
0:18:41that people have written it for exactly
0:18:43this kind of thing in fact my friend
0:18:46Jeffrey's elderman they have a company
0:18:48called a book apart and all of the books
0:18:51that they do for designers and
0:18:52developers
0:18:52that's a Shopify store and they
0:18:54customized the heck out of it you
0:18:56wouldn't even recognize it
0:18:57it's totally so here's the deal normally
0:19:00you go in there and you sign up on
0:19:02Shopify com you get 30 days free use the
0:19:04coupon code 5x5 when you're signing up
0:19:07you'll get three months free and of
0:19:09course you can use the site and sell
0:19:11everything you want within that that
0:19:13time frame of course you can so go there
0:19:16today Shopify com coupon code 5x5 and
0:19:19join more than 20,000 stores selling
0:19:22their stuff on Shopify John himself
0:19:25you're even opening up a store I am NOT
0:19:29gonna be selling USB headsets
0:19:35all right already main topic yes the
0:19:39main topic is actually inspired by
0:19:40follow-up and you mentioned at the very
0:19:41top of the show because you've been
0:19:43seeing the same emails as I have
0:19:44everybody was asking in the last show
0:19:46all you know so I listened your show
0:19:47about file systems how does all this
0:19:50change in light of solid state disks
0:19:52that is the question of the week ah and
0:19:57I didn't talk about it in last show so
0:19:59we'll talk about it now I'll use it as a
0:20:00jumping-off point to talk more about
0:20:01these file systems in general one of the
0:20:05reasons I didn't talk about it is
0:20:06because there's no clear answer how does
0:20:11it change and it's not even we can't
0:20:13even say it changes completely or not at
0:20:15all it's somewhere in the middle so I'm
0:20:17trying to think of things that file
0:20:23systems do that clearly have spinning
0:20:28disks in mind as a way to get into the
0:20:32topic of how that might change with SSDs
0:20:34so the biggest one I could think of them
0:20:36are you going to go some more is putting
0:20:40related information close together on
0:20:42the disk
0:20:43because the big thing you want to avoid
0:20:44on a spinning disk is moving the disk
0:20:47head that is by far the most time
0:20:49consuming thing you have to do because
0:20:51especially with you know diskette disks
0:20:54are getting faster but the speed with
0:20:55which you can move a disk head from one
0:20:57place to another on a disk has not been
0:20:59getting faster nearly as much as the
0:21:01size of disks have been increasing you
0:21:03know because you have to move the little
0:21:05head and then wait for it to stop it
0:21:07exactly over some microscopic little
0:21:08track on the disk and then wait for it
0:21:10to stop wiggling back and forth and sort
0:21:12of settle down so we can read again and
0:21:14they can do that amazingly fast it's
0:21:16like a seek times like nine milliseconds
0:21:18or something like that but it's still
0:21:21super slow compared to how fast they can
0:21:23stream data off a single location on
0:21:25disk so putting stuff together on a
0:21:28spinning disk is really important you
0:21:31can see this yourself I think regular
0:21:32users have some idea about this when you
0:21:35have a like a 100 megabyte folder and
0:21:39but it's filled with thousands and
0:21:41thousands of files there are a few bytes
0:21:42each how long does it take to duplicate
0:21:45that versus how long does it take
0:21:47duplicate a single hundred megabyte
0:21:48phile if your disc is not full and you
0:21:50have lots of free space duplicating a
0:21:51single hundred megabyte file can be
0:21:53really fast amazingly fast basically if
0:21:55the hundred megabyte file is kind of all
0:21:57all the data of it is kind of in this
0:22:00similar place on the disk you don't need
0:22:01to do many seeks you read long sequences
0:22:03of bytes off of the disk before you have
0:22:05to move the disk head to get the rest of
0:22:07the bytes that's really fast but moving
0:22:10thousands and thousands of little files
0:22:12even the total amount of data is the
0:22:14same to get it thousands of thousands
0:22:15little files well you have to go read
0:22:17the directory entry and that's imagine
0:22:18this is hfs+ to go to go to the catalog
0:22:20file which is in a central location so
0:22:21you read the directory entry and then
0:22:22the catalog file points you'd okay
0:22:23here's the first file and here's what
0:22:25it's called and here's what the bytes
0:22:26are so move this can get the bytes for
0:22:28that file and that's really quick
0:22:29because there's only a few bytes and
0:22:30then write the file to the new location
0:22:32so make a new directory entry that's
0:22:33back to the catalog file and write the
0:22:35name of the file there and then write
0:22:36the bytes for the file okay let's write
0:22:38those flights with a file someplace else
0:22:39and dusk you've done like six seeks and
0:22:40all you've copied is like two bytes of
0:22:42data like if this was a tiny little file
0:22:44in fact most your time might have been
0:22:47spent just writing the metadata about
0:22:49the files name the date you know all
0:22:50these other other information but you're
0:22:52seeking all over the disk get a little
0:22:54bit of data to go back to the catalog
0:22:56file write the new location write the
0:22:57data it takes forever because seeking
0:22:59time starts to dominate how long the
0:23:02entire operation takes whereas the
0:23:03actual writing data to disk when you
0:23:05look at the stats on a disk like into a
0:23:0630 megabytes per second or 40 megabytes
0:23:08per second yeah that's how much you can
0:23:10write if the disk head is pretty much
0:23:12not moving and the disk is spinning
0:23:13underneath it once you start moving the
0:23:15diskette around you're spending all your
0:23:17time moving and very little time
0:23:18actually writing data so the big thing
0:23:21that file systems try to do in in the at
0:23:24all times but especially in the modern
0:23:26age is no disk seeks put related stuff
0:23:28close together pack it all next to each
0:23:29other don't make the disk ad move around
0:23:32so how does that relate to SSDs the big
0:23:37thing with us is these is you know
0:23:38there's no moving parts there's no disk
0:23:40kid moving back and forth trying to grab
0:23:44little pieces just it's just a bunch of
0:23:45memory chips and the naive view of
0:23:47memory checks is like oh I can get at
0:23:48any part of those memory chips instantly
0:23:50I don't have to move anything anywhere I
0:23:52just say give me byte number 56 and I
0:23:54get it give me byte number 3000 and I
0:23:56get it and asking for byte 56 directly
0:23:58after I asked for but 3000 it doesn't
0:24:00make any difference right
0:24:01there's no there's nothing to move well
0:24:05flash memory unfortunately is not as
0:24:07simple as the naive view of memory might
0:24:10be there are some limitations and the
0:24:14limitations are different for flash
0:24:16memory of different kinds a different
0:24:17for DRAM differing for all sorts of
0:24:19memory chips the stuff that goes on
0:24:20inside the memory chips we like to think
0:24:21of it as a black box that just stores
0:24:23our bits and gives us back but it's
0:24:24really fiendishly complex and you don't
0:24:26know how for example DRAM works I
0:24:28encourage you to go to Wikipedia and
0:24:30read about how DRAM works you will
0:24:32probably be horrified and amazed at any
0:24:34Forks at all pieces it's terrifyingly
0:24:35scary flash is similarly terrifying it
0:24:39has these crazy limitations that don't
0:24:40make any sense until you figure out how
0:24:42like the chips are actually laid out and
0:24:43how the physics work and the economics
0:24:45of fabbing and why they're made this
0:24:47play I'll try to give a reasonable
0:24:50high-level summary of it but the short
0:24:53answer is that it does help to have
0:24:57things near each other when when using
0:25:01flash memory because although it doesn't
0:25:04take any time - there's nothing to seek
0:25:06from one location to the other flash
0:25:09memory cannot be arbitrarily updated in
0:25:12exactly the size pieces that you want
0:25:14there are larger chunks that you know
0:25:17you can't just say okay I want I'm going
0:25:19to write two bytes here I'm going to
0:25:20write one byte there and about 1 byte
0:25:21over there and everything's hunky-dory
0:25:23if the flash chips themselves tend to
0:25:26have to do operations in larger evenly
0:25:28sized chunks than you would want so that
0:25:30might mean erasing an entire block we
0:25:33are reading the entire contents of
0:25:34entire block erasing the whole thing and
0:25:37writing it back plus the change instead
0:25:39of just writing your change I tried to
0:25:42highlight the one part that's relevant
0:25:43most relevant from the Wikipedia page to
0:25:46a NAND flash memory that's moved in most
0:25:49flash drives and the upshot according to
0:25:52Wikipedia and I don't know enough about
0:25:54it to know whether this is accurate but
0:25:55I assume it probably is because it's
0:25:57just a technical matter is that it says
0:26:01you can erase memory a block at a time
0:26:05but so you can read it in a little
0:26:08pieces and you can write it in little
0:26:10pieces but you can only erase it a block
0:26:11at the time and the next wrinkle to that
0:26:13is that
0:26:15when you erase you set all the bits but
0:26:18when you set them you can only clear
0:26:19bits so if you're setting something in a
0:26:22region that has all ones in it all you
0:26:25got to do is blank out the ones that are
0:26:26zeros to make your bit patterns so if
0:26:28you're writing one zero one zero and
0:26:29you've got one one one one
0:26:31you just blank out those two ones and
0:26:33voila one zero one zero you just wrote
0:26:34only the amount of data you needed ah
0:26:38but when you erase it you have to erase
0:26:40the entire thing the entire block and a
0:26:42block is much bigger than the region
0:26:44you're just interested in updating and
0:26:45since you can only clear disks if a
0:26:48region has more zeros than the data
0:26:49you're about to write there like I can't
0:26:51I can't write this because I all I can
0:26:53do is change things it all I can do is
0:26:57change things to zeros but it's already
0:26:58got too many zeros I can't change that
0:27:00zero to a one
0:27:00so if I can't change a zero to one my
0:27:03only option is to erase this entire
0:27:05block set it all to ones we're going to
0:27:07read the block first erase the entire
0:27:08block and write the entire block back
0:27:10plus my changes there are probably
0:27:14better articles about this and someone
0:27:16hasn't been the show at Stan and tech I
0:27:17was trying to Google for better links to
0:27:19this before I started so I will add this
0:27:21one they came up in the show notes we
0:27:23link to the print only version of it but
0:27:25I guess I'll throw them in there so it
0:27:29means the ETL details don't matter but
0:27:31the point is that operations to flash
0:27:34memory don't necessarily only modify the
0:27:36bits that you're interested in sometimes
0:27:38they have to modify entire large blocks
0:27:40of bits even if you're only interested
0:27:43in changing a small portion of so that
0:27:46argues for trying to make your data
0:27:50that's related close to each other
0:27:52otherwise you're going to spend your
0:27:53time erasing an entire block of data
0:27:55just to put one byte there and then
0:27:57erase it all totally unrelated block
0:27:59about to put to put another byte there
0:28:02now I'm not sure if this means that
0:28:05putting stuff close together is like a
0:28:07big win on flash certainly it's not as
0:28:09big a win as it is on spinning disks but
0:28:12at the very least it probably does no
0:28:14harm and there are reasons to think that
0:28:17it might be beneficial like these same
0:28:19file system layouts that put things
0:28:23close together would also benefit flash
0:28:25memory because then
0:28:27that's it do operations that modify more
0:28:31of the block the whole block rhythm I
0:28:33need to erase but the flipside of that
0:28:35is if there is a file system that goes
0:28:37through great lengths and bends over
0:28:38backwards and has a really complicated
0:28:40structure completely aimed at minimizing
0:28:42seeks it's like all we just got to pack
0:28:44everything together and we have to do
0:28:45all this complicated stuff and the code
0:28:47is complicated and the i/o scheduling is
0:28:49complicated also it can avoid seeks it
0:28:52could be that that complexity is not
0:28:55worth the benefit that you get in SSDs
0:28:57and a file system that doesn't go to
0:28:58such great lengths to make sure data is
0:29:00as close together but is much simpler
0:29:03much more reliable and as fewer bugs or
0:29:05whatever might be fine on an SSD because
0:29:08the things that hurt it on spinning disk
0:29:10don't hurt it as much there so that's
0:29:13not a very good answer the answer is
0:29:14basically it depends and and I don't
0:29:17think anyone is really sure and all the
0:29:19file systems that I've been talking
0:29:21about with the possible section of
0:29:23session of butter FS were conceived in a
0:29:26time when people didn't realize that
0:29:28SSDs would be the phenomenon that they
0:29:30are as ZFS is but I think it's close to
0:29:34ten years old now it's pretty old and
0:29:36back then yes SSDs existed and they
0:29:39certainly didn't want to do anything
0:29:40that would be silly on an SSD and I
0:29:43don't think they did but they wasn't
0:29:45designed for an SSD many of the
0:29:47considerations in the file system design
0:29:49were based on spinning disks and butter
0:29:52FS looks a lot like ZFS but it's newer
0:29:58so they may have more SSD stuff in mind
0:30:00certainly there is no file system that I
0:30:02know of that's in widespread use it was
0:30:05designed from the ground up to say
0:30:06forget spinning disks in fact if you run
0:30:09this on a spinning disk it might be
0:30:10hideous and terrible this is entirely a
0:30:12ended SSD Boyd Watters wrote in to say
0:30:16that to quote him plopping NAND flash
0:30:21behind a SATA interface is stupid so
0:30:24what he's put is showing is that thing
0:30:25of the storage hierarchy that exists in
0:30:27current things you got the CPU register
0:30:30file you got the the caching on the CPU
0:30:32then you've got the RAM then you've got
0:30:33the the i/o controller chip then you've
0:30:35got the the SATA interface and then
0:30:38you've got this flash disk and you put
0:30:39discs and
0:30:40scare quotes because it's not actually a
0:30:42disk why does that big structure exist
0:30:47why do these flash memory chips have to
0:30:51pretend they're a spinning disk does
0:30:53that seem like a silly kind of archaic
0:30:56it's kind of like a computational
0:30:58skeuomorphism where it's pretending to
0:31:02be a disk because everything expects
0:31:03there to be a disk there but it's
0:31:04totally not this is an interesting issue
0:31:09that has come up many times in the in
0:31:11the past decades like why do we had disk
0:31:13and memory wouldn't it be nice if
0:31:15everything was just memory that's the
0:31:18you know the computer science dream
0:31:19everything is just a big open field of
0:31:21bits and it can be addressed randomly
0:31:24written land rent they write or read
0:31:26randomly and there's no difference
0:31:29between something's in disk and it's in
0:31:31memory just one big continuum one way to
0:31:33do that would be to have notice can just
0:31:34have everything in memory because then
0:31:36everything is memory right but that's
0:31:40generally not feasible because memory is
0:31:42measured in you know gigabytes and disk
0:31:45space is memory measured in terabytes so
0:31:48they're not close and the other thing is
0:31:49just less I have a unified interface
0:31:50that yes it's flat flash is slower than
0:31:55then DRAM but the Ihram is also slower
0:31:58than l3 cache which is slower than l2
0:31:59cache which is slower than l1 cache
0:32:01which is slower than registers it's a
0:32:02hierarchy of memory how close quote
0:32:05unquote close is this memory to the CPU
0:32:08how long does it take the CPU to get to
0:32:10it that hierarchy exists and
0:32:12traditionally if you were to graph how
0:32:14you know how long does it take to get a
0:32:16piece of memory out of l1 cache l2 cache
0:32:18l3 cache it's like a hockey stick graph
0:32:20and especially when you hit disk it's
0:32:21like well now you might as well just go
0:32:23on vacation because that that is not
0:32:24going to be coming back for another 8
0:32:25billion cycles of the CPU so forget it
0:32:30flash memory is faster than just what
0:32:32that wall is still there that the gap
0:32:33between the speed of memory is still
0:32:35there but still the idea is that why not
0:32:38just have this is a uniform storage
0:32:39hierarchy with uniform interface and not
0:32:42have disk i/o to be this totally
0:32:44separate thing that's something that
0:32:48someone like Apple is not going to
0:32:49probably tackle anytime soon but it's
0:32:52been in the air
0:32:54probably since the 60s or 70s the idea
0:32:56that a unified interface to memory would
0:32:58be better than and probably a lot of old
0:33:00machines did have a unified interface
0:33:01but in the end the person computer age
0:33:05the split weakened disk in memory is
0:33:06pretty baked into the way that we do
0:33:09things so let's consider what might
0:33:12change if everything was unified well
0:33:17one thing that would have to change is
0:33:19that the operating system would have to
0:33:22be different because in the operating
0:33:23system the file IO calls are very
0:33:25different from the calls to manipulate
0:33:26memory file alcohols you've got you know
0:33:28opening files reading writing files
0:33:30they're just structured differently take
0:33:31different arguments have different
0:33:32semantics than the memory calls which
0:33:34are you know setting memory getting
0:33:37memory copying memory and just bit
0:33:39operations that happen in memory in two
0:33:43entirely different api's so you couldn't
0:33:46just say ok memory and disk or all one
0:33:47because everyone's programs wouldn't
0:33:49work anymore because that's you know
0:33:51you'd have to you'd have to make some
0:33:52sort of shim layer there and ideally
0:33:54you'd want one unified interface to a
0:33:56new set of api's that looks the same for
0:34:00both of those things and what does that
0:34:01new wear interface look like and who is
0:34:03going to adopt that new integration
0:34:04who's going to design it that's a big
0:34:05problem one thing that would go away are
0:34:08these really complicated api's that we
0:34:10have now for memory mapping files where
0:34:12you say look I want to manipulate this
0:34:16really big file but I want to read every
0:34:18single byte of it off disk and I want to
0:34:19manipulate it as if it's all in memory
0:34:21but I can't fit it all in memory so can
0:34:22i memory map the file and memory mapping
0:34:24will automatically make it look like
0:34:25your entire files in memory but just
0:34:26pulling the parts that you need on
0:34:28demand and that's a very complicated
0:34:30system and the API is very complicated
0:34:33that's kind of stuff would go away if
0:34:36you had a single unified interface to
0:34:38the entire storage hierarchy or maybe
0:34:40you would just shift and that becomes
0:34:41part of this unified interface but it's
0:34:43a complexity that exists now because we
0:34:45know that files are much bigger than
0:34:47memory we know we can't fit an entire
0:34:48file in memory so we have to make this
0:34:50weird illusion that it's kind of in
0:34:52memory when it's not ah a big one that I
0:34:55thought of is that you'd have to
0:34:56reconsider limits limits that we take
0:34:59for granted now so for example when you
0:35:03run out of disk space that's a dead end
0:35:05right now
0:35:07no space left on device right that's
0:35:10that's the end you know this there's no
0:35:12program in existence that says when I
0:35:15run out it's like what can it what kind
0:35:16of program do well jeez on that it can
0:35:18tell you I try to do that for you but
0:35:20your disk is full so and and by the way
0:35:23UNIX does not deal well with the disk
0:35:24being full for reasons we'll probably
0:35:26get into in a bit later uh and you don't
0:35:30want you just to be complete full lots
0:35:32of bad things happen but running out of
0:35:34RAM waiting it what do you say UNIX
0:35:36doesn't handle it well what does handle
0:35:38it well or is that is that part of that
0:35:41next topic yeah that what does handle it
0:35:44well in it in the days before UNIX for
0:35:46example in classic Mac OS in like really
0:35:48classic Mac OS like you know system 4.2
0:35:52you could have your disk filled up to
0:35:54the brim and everything would still
0:35:57pretty much work okay you know you
0:35:58couldn't save stuff but you wouldn't be
0:36:01using the thing and it would just blow
0:36:03up whereas the UNIX uses the disk space
0:36:06for all sorts of things and if your disk
0:36:10even gets close to full lots of things
0:36:12they're going to start falling down not
0:36:13just the program you're using so you
0:36:14can't just happily use your Mac with an
0:36:16entirely full disk in fact I don't even
0:36:18know if you can get your your disk
0:36:19entirely full without sort of cheating
0:36:20by putting it to single user mode and
0:36:22using DD to fill up the disk I don't
0:36:24even think you can boot with it with an
0:36:25entirely full disk once you get anywhere
0:36:27near close to disk full Mac os10 starts
0:36:29throwing dialogues in your faces said
0:36:30this is almost full seriously delete
0:36:32stuff you've got a problem and there's a
0:36:34good reason for that uh but in the
0:36:37modern era running out of RAM is not a
0:36:40dead end and I say the modern era being
0:36:43like Mac OS 10 certainly but even in the
0:36:45latter days of classic Mac OS what
0:36:47happens when we run out of RAM is we're
0:36:49not around all the time ah
0:36:52you run out of RAM and you swap some
0:36:56stuff to disk so you take some stuff
0:36:58that's in memory though I don't have
0:36:59room to put everything in memory that
0:37:00need to put in a memory let me take
0:37:02something that's in memory or write it
0:37:03out to this temporarily now I've read up
0:37:05some new room in memory and it tries to
0:37:07put stuff on to disk
0:37:08tries to swap stuff out that hasn't been
0:37:11used in a while and this is the basis of
0:37:13all modern virtual memory systems with
0:37:16swap where they will
0:37:19try to let it appear to your programs as
0:37:21if you have way more RAM than you do so
0:37:23programs back the old days and 32-bit
0:37:26thing they would think that they had you
0:37:27know two to four gigabytes of RAM when
0:37:30maybe your computer only at 128
0:37:32megabytes of RAM and the program was
0:37:35blissfully unaware of that it would just
0:37:36try to address memory in the operating
0:37:38system every time you filled up the
0:37:39actual RAM chips
0:37:41it would just swap things in and out to
0:37:43disk to make enough room for the stuff
0:37:44you currently using in ram now swapping
0:37:47is bad and it's slow because if your
0:37:48program thinks something's in memory and
0:37:50it asks the operating system for on the
0:37:51operating system says actually that's
0:37:53not in memory I had to put that in disk
0:37:54for you but I'll wait I'll go get it on
0:37:55the disk you just wait there it takes
0:37:57forever for the operating system to come
0:37:59back so here's this thing and the
0:38:00program expected that to come back
0:38:02really quickly so when your computer is
0:38:05thrashing or swapping or one of these
0:38:07other terms it means that it's trying to
0:38:09access a bunch of stuff that it thinks
0:38:10is in memory but a lot of it turns out
0:38:12to be on disk and your disk ad they're
0:38:13seeking all around to pull this stuff
0:38:15off of the disk and putting other stuff
0:38:16onto this because if it's on if it's on
0:38:19disk has to pull it out maybe that's to
0:38:20evict something else from RAM and put it
0:38:21down there so that's situation we're in
0:38:26now where a disk space fills up total
0:38:28dead end RAM fills up it's not a dead
0:38:30end but how is RAM filling up not a dead
0:38:32it uses disk space and that's why if
0:38:34your disk is full everything falls down
0:38:37because all these programs think that
0:38:38they have you know much more memory than
0:38:41they really have and if they did blondie
0:38:43continued to put stuff in memory in the
0:38:44operating system says well that actually
0:38:46can't fit in memory so what I'm supposed
0:38:48to do on behalf of this program is evict
0:38:50something from memory put that on disk
0:38:53and then put his thing in that spot and
0:38:54when it goes to evict something from
0:38:55memory and put it on disk the disk is
0:38:57full and that's why bad things start
0:38:59happening classic Mac OS way back and
0:39:01they did not have virtual memory which
0:39:03is different than swap but it certainly
0:39:04didn't have swap space so it could
0:39:07happily hum along with its you know 512
0:39:09K of RAM or whatever and if you hit the
0:39:12512 K Ram limit ah that was the end of
0:39:16that it wouldn't swap stuff out to disk
0:39:17so if your disk was full no big deal
0:39:20it's the only time it has to write
0:39:22something on to disk is if you ask it to
0:39:23not in your normal operation it didn't
0:39:26write stuff and UNIX of course has log
0:39:27files that are constantly being appended
0:39:29to and other things that are going on in
0:39:30the background slowly filling in your
0:39:31disk that
0:39:32know about and if your disk is full all
0:39:33those things freak out and say you know
0:39:36we've got a problem here they would
0:39:37probably they would probably try to log
0:39:38the fact that they got a no space left
0:39:40on device message and that would cause
0:39:41more Disgaea which would also fail and
0:39:43it's just a vaccine so there's an
0:39:47exception to this in the modern error
0:39:48which is iOS which some people also
0:39:51asked about iOS if you run out of disk
0:39:55space that's a dead end still if you run
0:39:58out of RAM on iOS it's also a didn't
0:40:02because iOS though it implements virtual
0:40:04memory virtual memory is the mechanism
0:40:06by which the programs think they have
0:40:08much more memory and they do then they
0:40:10really do and they think they all have
0:40:11their own memory space to themselves
0:40:14it has virtual memory but it does not
0:40:16use swap so when you fill up all the RAM
0:40:19on iOS it doesn't say oh let me put that
0:40:21on quote-unquote disk for you it's disk
0:40:23is obviously flash what it does is kills
0:40:25your program in fact to kill your
0:40:26program long before that iOS is a very
0:40:28hostile environment for programs and the
0:40:30the application environment for running
0:40:33GUI applications in iOS will send your
0:40:37application a low memory warning if it
0:40:38starts using too much RAM and if the
0:40:42program does not respond by freeing
0:40:43memory eventually the operating system
0:40:45will just kill that program say look I
0:40:46sent you a message I told you you're
0:40:47getting close to the memory limit I'm
0:40:48gonna have to kill you because if you
0:40:49ever hit the memory limit on iOS that's
0:40:53it it's not you know I need more space
0:40:55well RAM is full and there's no more RAM
0:40:57we can't make new Ram out of thin air
0:40:59and we're not going to swap the flash
0:41:00now one of the reasons that iOS doesn't
0:41:01swap to flash is because even though
0:41:04flash is faster than disk it's still way
0:41:06way WAY slower than Ram so that's one of
0:41:10the reasons that iOS has always been so
0:41:11responsive because to get that kind of
0:41:13responsiveness that's how it's running a
0:41:15variant of Mac OS 10 we're doing virtual
0:41:17memory but we cannot afford to like oh
0:41:20when I swipe something on the screen oh
0:41:21hang on a second I gotta pull a bunch of
0:41:23that stuff that you thought was in
0:41:24memory actually it's on my flash disk
0:41:26and I got to pull it off the disk and
0:41:28it's it can take a thousand times longer
0:41:29than you thought it would and then you
0:41:30get the stuttery animation or anything
0:41:32maybe someday that will change but for
0:41:35now as far as I'm aware maybe someone
0:41:37can correct me
0:41:38Iowa still does not use swap space it
0:41:41puts everything in RAM so if you're
0:41:44going to unify disk
0:41:45and ram into one type of thing
0:41:49everything would have to become a dead
0:41:50end
0:41:51because you have no other place to put
0:41:53stuff right like when you run out of
0:41:56whatever you're going to call this thing
0:41:58when you run out of memory that means
0:41:59that your caches are full you know they
0:42:02know you can put in a register can't put
0:42:04it in a cache you can't put it in RAM
0:42:05and uh your disk is also full and I
0:42:10think that would be a bad place to be
0:42:13because you could imagine getting
0:42:14machine into a deadlock situation
0:42:15hopefully not hopefully in a consistent
0:42:17state of some kind but where do you go
0:42:19from there you don't have an out you
0:42:20don't have a place to put stuff maybe
0:42:22we're just carve off it you would not do
0:42:23the same thing where it would say if you
0:42:25get close to the limit of my giant pool
0:42:27of memory I'm going to warn you and then
0:42:30I'm going to kill stuff off because we
0:42:31cannot hit that limit sort of like you
0:42:35know it does with disk space today where
0:42:36you just can't get close to that limit
0:42:38on disk pacer will start yelling about
0:42:39you but Ram the operating system will
0:42:41let you continue to use it and use it
0:42:42when you have virtual memory with swap
0:42:44until your computer becomes so slow that
0:42:46you're discouraged from using it any
0:42:47longer and you walk away and wait for
0:42:49everything to swap in and out so I think
0:42:52this is an interesting topic I don't
0:42:54know what will happen in this area I
0:42:57don't think any time in my lifetime
0:42:58we're going to see this grand unified a
0:43:02single pool of memory thing but if it is
0:43:07going to happen I think it will happen
0:43:08somewhere in the mobile space where the
0:43:10sizes are smaller and the entrenched
0:43:14system of dividing RAM and disk are not
0:43:19as prevalent as they are on the PC
0:43:26that's the end of my fake follow up
0:43:29follow up I was what's fake about that I
0:43:32thought it was very real well it was you
0:43:34know it's talking about SSD is not
0:43:37following up something we said last time
0:43:38but it's a new top oh I see yeah so your
0:43:41your confidence level of SSD over the
0:43:46traditional Drive as far as a strategy
0:43:50for Apple what are your thoughts on that
0:43:51long-term strategy Apple's hardware
0:43:54everything SSD yeah has to be mean it's
0:43:58going to have
0:43:59it's just a question of how long it
0:44:00takes with Apple sells consumer hardware
0:44:02so they don't have to deal with like
0:44:03well I need to store petabytes of data
0:44:05like they're not selling servers for the
0:44:07data center and SSDs are way more
0:44:10expensive than spinning disks we're just
0:44:14kind of that inflection point where they
0:44:15can say like when I got my MacBook Air
0:44:19like the biggest s is the you get was
0:44:20256 and that's what I got and that was
0:44:23still bigger than 160 gig spinning disk
0:44:25that was in my old MacBook Pro so it was
0:44:27still a step up but you can get like
0:44:30terabyte now I think terabyte laptop
0:44:32drives are available it's four times
0:44:34smaller than than the the biggest lap
0:44:37spinning laptop drive you can get and
0:44:39way way way more expensive so they're
0:44:43just on the borderline of like you can
0:44:45make people choke down a much higher
0:44:48price for less space just because the
0:44:50performance is so much better but they
0:44:52haven't quite crawled up to like can I
0:44:54convince someone to buy a Mac Pro with
0:44:56just SSDs in it someone who wants a Mac
0:44:59Pro probably wants a lot more internal
0:45:00storage space than 256 gigs and those
0:45:02drives are so expensive if you buy for
0:45:04them now the entire cost of your
0:45:06computer is basically just an SSD so the
0:45:09price keeps going down the storage space
0:45:10keeps going up eventually there'll be a
0:45:13point where spinning disks are just seen
0:45:17as archaic and everyone has to have SSDs
0:45:19and only the people who really need
0:45:21gigantic amount of space need spinning
0:45:22disks and eventually those will probably
0:45:24go away - so that's inevitable you just
0:45:27get used to the fact that as as these
0:45:29are coming how long before every single
0:45:33computer Apple makes has SSD by default
0:45:35the kinds of if they discontinue the Mac
0:45:38Pros thoughts on discontinuing the Mac
0:45:40Pro don't do it that's my thought please
0:45:43as you do it you'd like you like the Mac
0:45:45Pro and you're you're a big-time Mac Pro
0:45:47user yeah you don't want them to get rid
0:45:49of it I was I saw an article from I
0:45:52don't know if you've spoken with him but
0:45:54he's a friend of mine he used to do a
0:45:56show here as well James Duncan Davidson
0:45:59he's a photographer and right now or
0:46:02very recently he was out shooting Ted
0:46:04and he was saying that every year that
0:46:06he shot Ted which is a number of years
0:46:08these guys they spare no expense when it
0:46:11comes to having great
0:46:12hardware for the people who are doing
0:46:14the audio video editing and then the
0:46:17photography editing and things like that
0:46:18and they set up a whole lab I'll try and
0:46:21put I'll try and put one of these
0:46:22pictures that he's taken of the lab into
0:46:25the show notes and by the way thanks to
0:46:28help spot calm for the show notes and
0:46:31this year instead of having two dozen
0:46:35Mac Pros they now have two dozen IMAX
0:46:39with Thunderbolt drives connected to
0:46:42them raid Thunderbolt drives connected
0:46:44to them and they have officially moved
0:46:46away completely from Mac Pros and now
0:46:49they just have IMAX and it makes you
0:46:51think that if if a company like tad if a
0:46:54group like Ted is as as serious and
0:46:58dedicated as they are to doing real and
0:47:02if you've ever watched the Ted videos
0:47:04you know these are these are phenomenal
0:47:06videos they have the best audio-video
0:47:08photographer people in the business
0:47:09making them and editing them and turning
0:47:12them around in almost real-time it's
0:47:14it's crazy how fast they put these
0:47:16things out there they're content with
0:47:19IMAX why why would Apple keep making the
0:47:23Mac Pro who is that for isn't the
0:47:25Thunderbolt the obvious indication that
0:47:27they're serious about moving away from
0:47:29the Mac Pro well that particular
0:47:33application I think doesn't need the
0:47:36massive amounts of storage because kind
0:47:38of a way station for a workflow that
0:47:40takes assets from the day and grinds
0:47:42them up and spits out some new assets
0:47:44where you need the big spinning disks
0:47:47that can store so much more than SSDs
0:47:48for so much less money is where does the
0:47:51company that is responsible for
0:47:54recording all of Ted and producing all
0:47:56the content where does it distorts
0:47:58information where does it start all the
0:48:00raw videos all the raw images all the
0:48:02processed files everything forever
0:48:04archived maybe puts them all on tape and
0:48:09that's the answer but I think companies
0:48:11that have large storage requirements
0:48:13like people doing special effects for
0:48:14movies or something like that they'll
0:48:17use SSDs for the parts where speed
0:48:19matters but at some point that like they
0:48:21don't want to ever throw anything out
0:48:22you want to have all your assets and all
0:48:23everything redundantly stored in a
0:48:24million place and that takes up a
0:48:25tremendous and
0:48:26room even just for one movie let alone
0:48:28movie after movie like how many assets
0:48:29do you think Pixar has stored
0:48:31permanently yeah it accessible not on
0:48:34tape or some robot has to go pull a tape
0:48:36out and shove it in but accessible in
0:48:38semi real-time because you might want to
0:48:39use that asset for a new production you
0:48:41need some list to put all that stuff in
0:48:42SSDs just cost too much money to to do
0:48:45that now spinning disks are so much more
0:48:47economical especially since you have to
0:48:49have 20 of them because you have to have
0:48:50everything redundantly stored and you
0:48:52know I there is still a big market for
0:48:55it for spinning discs now is that is
0:48:58Apple does Apple have any part of that
0:48:59market maybe they're going to say look
0:49:02if you want to do that you're gonna have
0:49:03to buy disk storage from EMC or
0:49:05something just like we at Apple do for
0:49:07our data centers because we're not
0:49:08interested in giving you a machine that
0:49:11can store all that stuff or as you said
0:49:12our machines can't store it buy an iMac
0:49:15and connect one of these big raid things
0:49:17that we also don't make to a Thunderbolt
0:49:20cable you have really fast access to to
0:49:22lots of storage the Mac Pro as a product
0:49:25exists for reasons other than just
0:49:27internal drives though it's got the
0:49:28slots where you can put cards that you
0:49:29might need cards that don't fit inside
0:49:31the iMac because there's no place for
0:49:32cards at all and yet maybe you can use a
0:49:34Thunderbolt external chassis but the
0:49:35number of PCI Express Lanes over a
0:49:37thunderbolt wire is not that that bus is
0:49:41not as wide as an internal slot so there
0:49:44are many reasons why the Mac Pro still
0:49:45needs to exist the fact that they
0:49:46switched from Mac Pro Stimac for that
0:49:48Ted thing they probably could have
0:49:50gotten away with imax even sooner
0:49:53because it's just kind of a way station
0:49:54and not something that needs to have
0:49:56lots of storage and those big you know
0:49:58big special-purpose cards and yeah and
0:50:01also the other thing to keep in mind is
0:50:02that Ted probably has a lot of money you
0:50:05would think so I put I I did find the
0:50:07article and the pictures that go along
0:50:09with it I put it into the show notes and
0:50:10he says for Ted 2012 there's a pretty do
0:50:13you say SATA or sad at these days I say
0:50:16both okay for Ted 2012 there's a pretty
0:50:19big change in the media room instead of
0:50:20dozens of Mac Pros and piles of SATA
0:50:22drives like there have been in the last
0:50:24eight Ted events I've been part of the
0:50:26room is full of IMAX and Thunderbolt
0:50:28drives lots of Thunderbolt drives and he
0:50:30has a photograph here he says there are
0:50:3112 12 12 terabyte Pegasus are six arrays
0:50:36that's a hundred and forty four
0:50:37terabytes total
0:50:39he says it which is about a tenth of a
0:50:41petabyte of usual usable space for good
0:50:44measure there are a few more pegasus
0:50:45arrays including one that's dedicated to
0:50:47photography and he goes on to detail
0:50:49this and he says this is the biggest
0:50:52appointment of thunderbolt based storage
0:50:53that he's seen yeah I think Thunderbolt
0:50:56has made it feasible to have external
0:50:59storage that's really fast without
0:51:02having some weird interface card like
0:51:04fiber channel or something because
0:51:06internal disk was the way the way how
0:51:08you get the speed right because you know
0:51:10it's fast as possible as you can get
0:51:11it's connected right to the thing
0:51:12there's no long wire there's no weird
0:51:14interface it's you know sad is pretty
0:51:15fast and they would keep cranking up the
0:51:16speed and then once you went out over
0:51:19firewire like firewire 400 in particular
0:51:21was not enough to handle the output of a
0:51:24arrayed with many spindles or any large
0:51:27array like that firewire 800 was an
0:51:29improvement but still I think as the the
0:51:32Thunderbolt devices they're available
0:51:34today show they'll take like a four disc
0:51:36firewire 800 Drive and then I'll take a
0:51:39four disc Thunderbolt array maybe it's
0:51:42the same exact four mechanisms in each
0:51:44set light but that by eight mechanisms
0:51:45but form one form the other and the
0:51:46Thunderbolt ones just transit because
0:51:48Thunderbolt is like an external
0:51:50extension of an internal bus in this
0:51:52case it's the PCI Express bus so that
0:51:54that probably makes it feasible for them
0:51:56to have storage that's fast enough to do
0:51:59all this HD video streaming or whatever
0:52:00they're doing whilst just hooking it up
0:52:03to an iMac with that little Thunderbolt
0:52:04cable but the Mac pros are still an open
0:52:07question I think let's do our second
0:52:10sponsor its tiny letter comm they're
0:52:12just they're putting the finishing
0:52:13touches on a brand-new version and this
0:52:16is very exciting they have been working
0:52:18hard on this I talked to them about it
0:52:20and a tiny letter in general the best
0:52:23way for me to describe it to you is it
0:52:25is the most simple straightforward
0:52:27newsletter tool that has ever existed in
0:52:30the history of the earth it's elegant
0:52:33it's simple its straightforward it gets
0:52:36rid of all of the extra features and
0:52:38cruft and user interface layers and
0:52:41throws all that out the window now some
0:52:43people might want that go to MailChimp
0:52:45com if you want that you don't want that
0:52:47tiny letter super simple elegant
0:52:51straightforward tiny letter com
0:52:53they have a sneak peak video that they
0:52:55just put up to give you an idea of what
0:52:57it's going to be like within the next
0:53:01few days when this thing comes out so go
0:53:03check it out thanks very much two tiny
0:53:05letter calm for making the show possible
0:53:06I do actually have some more files of
0:53:10some stuff lurking at the bottom here
0:53:12we're going to talk about BOS no okay
0:53:15would be good ID now you give me more
0:53:16stuff I tried the Kappas to shows I had
0:53:18to assume well I actually I wanted to
0:53:21talk more about ZFS because all the
0:53:23stuff we talked about was EFS was
0:53:25focused on the particular failings of
0:53:27HFS+ and how they're addressed by ZFS
0:53:29with data integrity and all that but
0:53:32there are so many more things as e of us
0:53:33that I realize they didn't even get to
0:53:34that are interesting and other reasons
0:53:36why you might be interest why you might
0:53:39want this process like things that H of
0:53:41us aren't even a glimmer in HFS+ is I
0:53:44but that our most modern bus system cell
0:53:46and I imagine all sorry F SS but we
0:53:49didn't quite get to it okay so the first
0:53:51one which I think I mentioned this is
0:53:53these are advantages of ZFS or neat
0:53:55features of ZFS S and there a lot of
0:53:56them are not unique to ZFS but I'm using
0:53:58it as the poster boy for a modern ish
0:54:01file system first one is constant time
0:54:04snapshots and clones that is a mouthful
0:54:07there so I'll explain the pros also in
0:54:11the middle part snapshots so a snapshot
0:54:13is like the word says kind of at
0:54:17capturing the state of a disk at a
0:54:20moment in time and obviously we talked
0:54:24about this before it's a consistent
0:54:25state so the file system might be
0:54:28applications might be in the middle of
0:54:30doing stuff might be in the middle of
0:54:31writing files out or deleting stuff or
0:54:33whatever but you're going to take a
0:54:34snapshot and say look at that
0:54:35application might be in the null writing
0:54:36that file and it might have written half
0:54:38the file but as far as the file system
0:54:40concerned at this moment in time it
0:54:42knows where everything is it knows
0:54:43exactly the size of everything
0:54:44everything is all the bookkeeping agrees
0:54:46with each other that half written file
0:54:49it knows that that file is is one
0:54:51megabyte ah maybe the file will
0:54:53eventually be two megabytes when the
0:54:54applications done but right now one
0:54:56megabyte is on disk the metadata says
0:54:57one megabyte it is a moment in time
0:54:59snapshot ah
0:55:00and what ZFS can do is freeze that and
0:55:05say okay now at any
0:55:07in the future if you ever want to see
0:55:08what this file system was like at this
0:55:10particular moment in time it's available
0:55:12to you you can look at what it looked
0:55:14like at this moment in time the constant
0:55:18time part means that a time required to
0:55:22take a snapshot doesn't scale with the
0:55:25size of the disk so it's not like it
0:55:26makes a big giant copy like you so you
0:55:29have a terabyte drive when you say ok
0:55:31take a snapshot doesn't say ok I'm going
0:55:32to take the entire terabyte you know say
0:55:34this 500 you know gigabytes on there I
0:55:36take this entire 500 gigabytes and copy
0:55:38it someplace else it doesn't do that
0:55:41because of the way the file system is
0:55:43structured it can just make a note that
0:55:46this particular moment in time is
0:55:47important and it should be preserved
0:55:48remember that ZFS like many other file
0:55:50systems avoids overriding data in place
0:55:53it writes all new changes to a new
0:55:54location when it does that the old
0:55:57location eventually falls out of use
0:55:59when no more processes have that file
0:56:03open when nobody else is using it said
0:56:05that was the old version of this file
0:56:06the new one is way over there so now
0:56:09we'll just say ok this the space
0:56:10previously occupied by the old file is
0:56:12free to be thrown back into the pool all
0:56:13a snapshot does is tell the filesystem
0:56:17hey all of these things that make up
0:56:20this moment in time of the filesystem
0:56:21don't throw them out even when
0:56:23everybody's done with them keep them
0:56:24around so the literature on ZFS will say
0:56:27that this is instantaneous instantaneous
0:56:29snapshot Gration it's not instantaneous
0:56:31but the point is that it's constant time
0:56:33that it doesn't take twice as long to
0:56:35take a snapshot of a one terabyte discus
0:56:37as that is a 500 gig disk because the
0:56:39operation is basically a simple
0:56:40bookkeeping operation of saying all the
0:56:43stuff that you need with the snapshot is
0:56:44already on disk all I'm telling you is
0:56:47don't throw that stuff out later so it's
0:56:50very quick to note that like this moment
0:56:52in time it's a snapshot and you just
0:56:54store a little bit of metadata about
0:56:55here's a snapshot and here's you know
0:56:56that here's where it is in the time
0:56:58sequence of events and don't delete that
0:56:59stuff because it's preserving as the
0:57:01snapshot so it's really just a promise
0:57:05in the future to not ditch those old
0:57:08blocks that have new versions written to
0:57:10them and that's something you can't even
0:57:12imagine doing hfs+ because in nature I
0:57:14suppose everything is written in place
0:57:15and if you wanted to know what your disk
0:57:16looks like in a moment in time there's
0:57:17nothing available to you to do that
0:57:19the only thing you could possibly do
0:57:20is stopped all processes but you can't
0:57:23really do butts is always something
0:57:24vlogging in the background and UNIX or
0:57:25whatever but try to freeze it you know
0:57:27quit every application kill all
0:57:28processes is that the possible minimal
0:57:30maybe rebooting single user mode or
0:57:32something and then make an exact clone
0:57:34of this disk to a second location either
0:57:36a second disk or a second partition as
0:57:39you can imagine making an exact clone of
0:57:41a disk takes a long time and if it takes
0:57:43twice as long to make clone of a
0:57:45terabyte disk as a 500 gig disk and that
0:57:47amount of time is like measured in hours
0:57:49probably this is versus ZFS where you
0:57:52can just snap your fingers and set up
0:57:53make a snapshot of that so I can get
0:57:54back to it anytime now clones are
0:57:57writable snapshots clones let you say
0:58:01okay take a snapshot of it but actually
0:58:04let people make future changes to that
0:58:06snapshot so if you took a snapshot if
0:58:08you took if you made a clone right now
0:58:10what you essentially have our two file
0:58:13systems that look exactly identical and
0:58:15you can go into one of them and start
0:58:16messing with stuff of course the other
0:58:18one you know your main one also
0:58:19continues to go along and they start to
0:58:21diverge from each other this is all
0:58:23efficient because they share all the
0:58:25blocks in this because they have on
0:58:27count in common likes when you're making
0:58:28a snapshot all the disks are in common
0:58:30and then as time goes on something
0:58:32changes and what it does is okay well
0:58:35you know the same thing that always does
0:58:36someone wants to make a change this
0:58:37right the new version of it over there
0:58:39and to leave the old version over there
0:58:41because that's part of a snapshot so the
0:58:43the two resulting logical file system
0:58:47slowly diverged as as that more blocks
0:58:49changed as they differ snapshots don't
0:58:51let you modify that moment in time image
0:58:54of the thing but clones say just make it
0:58:57you know make a note here that this was
0:58:59a moment in time and now future changes
0:59:01to this clone should be separate from
0:59:02the future changes to the operating
0:59:03system that it was clone as if until the
0:59:05operation to the file system that it was
0:59:06cloned from these are two very powerful
0:59:11and very interesting technologies that
0:59:14I'm sure people who have never heard of
0:59:15them can think of many possible uses for
0:59:16them already but the fact that they are
0:59:19not giant copy operations and don't
0:59:21scale with the size of the file system
0:59:23is key the number of snapshots you can
0:59:26have on ZFS is supposedly unlimited but
0:59:30as you can imagine if you keeps taking
0:59:32snapshots what you're saying is the
0:59:34blocks used
0:59:35by the data currently on the disk can
0:59:36never be free eventually you'll fill
0:59:38your disk because you're you're telling
0:59:41it you can't delete this old stuff even
0:59:43if like so you take a snapshot then you
0:59:44delete a 1 gigabyte file off your disk
0:59:46well it can't actually give you one 2
0:59:48gigabyte of free space left because that
0:59:50gigabyte file has still exist in the
0:59:52snapshot so that will disappear from
0:59:53your current disk but if you were to
0:59:55look back in the snapshot it would still
0:59:56be there so eventually you will fill
0:59:58your entire disk by taking snapshots you
0:59:59can course
0:59:59can course
1:00:00trim old snapshots off the end and say
1:00:02okay this snapshot I'm done with it you
1:00:03can free up all the blocks there or
1:00:04whatever but the whole point of all this
1:00:05is is that it happens very very quickly
1:00:08basically instantaneously which is a far
1:00:11cry from how long it takes to do a
1:00:12super-duper backup or a Time Machine
1:00:14backup or anything like that the second
1:00:17related feature is sending and receiving
1:00:20block level deltas between snapshots
1:00:24this is where you can say okay I've got
1:00:28a snapshot over here and I've got a
1:00:30snapshot over there what is it is
1:00:33different between those two things find
1:00:35not the files that are different but
1:00:38find the individual blocks on disk that
1:00:40are different between snapshot a and
1:00:42snapshot B and send them from snapshot a
1:00:44snapshot B to make B match up with a now
1:00:50this is should bring to mind a backup
1:00:53type of strategy incremental backup
1:00:54strategy and it's a big contrast with
1:00:57the way time machine works this is a
1:00:59review of some of the stuff they wrote
1:01:01about in the leopard review and time
1:01:03machine was introduced the way time
1:01:05machine works is it also has to send the
1:01:09differences between the last time you
1:01:10backed up and the current state to your
1:01:12time machine drive so if you run time
1:01:13between the first time it copies
1:01:15everything to a second Drive right the
1:01:17second time you run it you don't want it
1:01:19to copy everything again you just wanted
1:01:20to say what has changed since the last
1:01:21time you did a Time Machine backup and
1:01:25then write those changes to the Time
1:01:26Machine backup so the first backup takes
1:01:29like hours and the second backup takes
1:01:30maybe five minutes ah the tricky bit is
1:01:34how do you tell what has changed since
1:01:36the last time you made a backup in the
1:01:39in Mac OS 10 and time machine what Mac
1:01:41OS 10 does is anytime anything makes a
1:01:44change to your disk it writes a little
1:01:47log of it there's a mechanism called FS
1:01:48events that maintains this highly
1:01:51compressed log that tries to coalesce
1:01:53multiple updates and all sorts of other
1:01:55things to try to make it so it's not
1:01:56filling your disk but right now as
1:01:58you're if you're sitting here on your
1:01:59computer or your Mac is doing stuff at
1:02:01home anytime something happens to the
1:02:04file system the operating system is
1:02:05writing to a little log file of what
1:02:08happened to the file system at a file
1:02:11granularity so it's going to say this
1:02:12file change
1:02:13file was deleted this file was created
1:02:18when Time Machine runs it reads it keeps
1:02:22track of where it wasn't that log last
1:02:23time so it says I did a time machine
1:02:25back up and I did all the updates up to
1:02:27file system event number 1 2 3 so I'm
1:02:29going to start from file since event
1:02:31number 1 2 4 and it says this file was
1:02:33deleted ok so I know that's the change
1:02:34you have to make in the back and it said
1:02:36this file was modified ok so I've got to
1:02:37grab the copy that file put it over
1:02:38there and says this file was deleted so
1:02:40you know it can go back go through the
1:02:43file system event log and find the
1:02:46things that have changed and make the
1:02:47new backup with them when it makes the
1:02:49backup it doesn't make it does use as
1:02:52hard links on the target destination to
1:02:54say all right so everything is exactly
1:02:56the same as the last backup except this
1:02:59one document change so it makes hard
1:03:01links to all of the rest of the disk
1:03:03except for the directory that has that
1:03:06one changed file in it and then it makes
1:03:07hard links to all the files in that
1:03:09directory except for the one file to
1:03:10change and then finally it copies the
1:03:12one file to changed over to the
1:03:13time-machine disk so it doesn't have to
1:03:19have to complete copies every date it's
1:03:20got one copy of all your data plus two
1:03:22copies of that one file that changed you
1:03:25can see this is kind of a very different
1:03:26way to do what copy and write does it a
1:03:28block level in something like ZFS the
1:03:32disadvantage is that many people use
1:03:35time as you know if you have a big file
1:03:37like a 2 gigabyte mail database and you
1:03:40get a single new piece of email and it
1:03:41modifies your 2 gigabyte mail database
1:03:43file the next time time machine runs is
1:03:47going to see a file system event that
1:03:48says aw since the last time I ran the
1:03:50mail database has changed and then it's
1:03:53going to copy the entire 2 gigabyte mail
1:03:55database all 2 gigabytes of it over to
1:03:58the time machine disk because that's the
1:04:00only thing it can do it's got the old
1:04:01version of the 2 gigabyte mail database
1:04:03on the time machine disk and now there's
1:04:05a new version it's got to copy the
1:04:07entire version over there if you have
1:04:09any large files that change frequently
1:04:10your time machine backups are going to
1:04:13be humongous you're going to be doing
1:04:14gigs and gigs of i/o when may be only a
1:04:16few kilobytes changed because it works
1:04:18at the file level the FS immense
1:04:21mechanism works at the file level and
1:04:23time machine works at the file level
1:04:24this is why mail programs that used to
1:04:26have 2k by database
1:04:27have since been changed to use thousands
1:04:29and thousands of little tiny files
1:04:30because when a little tiny file changes
1:04:32just a little file has to be copied not
1:04:34the entire file but there are still
1:04:35large files that are modified fairly
1:04:37frequently and just so wasteful to have
1:04:39to you know I add I add two three words
1:04:41to the end of this big Word document so
1:04:42I got to copy the two megabyte word
1:04:44document again if that happens over and
1:04:46over again you're filling your backup
1:04:47disk way fat way faster than you think
1:04:49you should uh and you're doing a lot
1:04:52more i/o you know you're copying tons
1:04:55and tons of stuff uh you know from one
1:04:57place to the other so everything takes
1:04:58longer now this this mechanism of
1:05:01figuring out what has changed through
1:05:05the file system the vent thing this was
1:05:07introduced uh I believe was introduced
1:05:09when spotlight was introduced because
1:05:10spotlight uses the same thing spotlight
1:05:12is the thing that indexes all the files
1:05:14on your just so you can search them and
1:05:16doesn't just index the file names it
1:05:17also indexes the file contents so
1:05:20anytime something happens on your disk
1:05:21in a file is written the spotlight
1:05:24daemon else on the spotlight demo one of
1:05:26the daemons related to spotlight is
1:05:28listening in on that file system event
1:05:30stream listening for every single thing
1:05:32that happens on your disk and when
1:05:34something happens it goes AHA a file has
1:05:36been modified or created or deleted I
1:05:38have to do something so say a file was
1:05:39created says up new file is created and
1:05:41it's notified in real time when that
1:05:42happens and it says well I gotta go
1:05:44index that file so it's six it's little
1:05:45index around that file and the index R
1:05:47reads some portion of the file and
1:05:48updates his index of stuff and says okay
1:05:50I read this file and I index and it's in
1:05:52there same thing when a file is deleted
1:05:53so this file was deleted I got to remove
1:05:55that file from my index so when I search
1:05:56for it later they don't find it because
1:05:57it's gone when this mechanism is reduced
1:06:01this firehose of file system events
1:06:04real-time streams of what's happening in
1:06:07the file system was only available to
1:06:10selected apple processes so the
1:06:13spotlight could do it and a few other
1:06:15things we're listening on the the
1:06:16firehose of file system events and the
1:06:19reason it was limited to only a few
1:06:22listeners was because this is all
1:06:27happening in the kernel of the operating
1:06:28system it's the only way to catch all I
1:06:30owe that happens to a disk is to talk
1:06:31into the kernel the file systems in that
1:06:33magazine was hooking to the crown so
1:06:34anytime any i/o happened it could go
1:06:36through this list of people who are
1:06:37interested hey is anybody listening for
1:06:39files as events I got tell them this
1:06:40happen Hey
1:06:41this file was created oh hey this file
1:06:43was deleted hey this file was modified
1:06:44things that happen inside the kernel
1:06:47first of all things that happen inside
1:06:49the kernel tend not to maybe able to be
1:06:52swapped out to disk for what I hope are
1:06:55obvious reasons because the kernels
1:06:56where you implement the swap stuff so if
1:06:58for example the code that swaps virtual
1:07:01memory to disk is swapped to disk and
1:07:03you need it to get the stuff off disk
1:07:05you're in a bad situation
1:07:06so most kernel memories wire down wire
1:07:08down means that it can't be swapped disk
1:07:10and the kernel has buffers for storing
1:07:14like messages that it's going to send
1:07:15and most of those buffers are fixed in
1:07:16size and small because that you know
1:07:18growing memory and in the kernel is
1:07:20another thing that tends to be frowned
1:07:22upon you want a bunch of small fixed
1:07:24size buffers so when you're sending
1:07:27messages to these things that are
1:07:30listening for filesystem events you've
1:07:32got a window of time when things have to
1:07:34happen
1:07:35the kernels got to put the notification
1:07:36in in some buffer queue for say okay by
1:07:39the way spotlight thing you want to know
1:07:41when something happens something
1:07:41happened well say tons of things are
1:07:44happening really really fast like 100
1:07:46files are created as fast as you
1:07:48possibly can the thing listening for
1:07:50filesystem events has a responsibility
1:07:52to consume those events to pull them off
1:07:55the queue and do something with them at
1:07:56a speed that's close to the speed that
1:08:00they're being filled in otherwise the
1:08:02buffer is going to get full and what
1:08:03happens when the buffer is full like oh
1:08:05it's it's like the Lucy with the
1:08:06assembly line chocolates thing right
1:08:09what episode of ice from I Love Lucy I
1:08:11don't know there's the one where she
1:08:12works in a chocolate factory in the
1:08:14chocolates are coming out faster than
1:08:15she can box him up yeah she was also
1:08:17funny she's she's the consumer in this
1:08:19she's supposed to be putting them in
1:08:20boxes and putting away but it's the
1:08:22producer at the chocolates cannot coming
1:08:24down the assembly line too fast you know
1:08:26what happens is the chocolates get
1:08:28dropped on the floor so that can happen
1:08:31with file system event consumers if the
1:08:33consumers are not pulling the events off
1:08:36the queue as fast as they can are
1:08:37they're coming in they will end up
1:08:40missing events and the whole point of
1:08:41this mechanism is not to miss any I have
1:08:45to know everything that happened so
1:08:46imagine for for the thing that writes
1:08:48the log of things that happen that's the
1:08:50whole point is it has to make a note of
1:08:52everything that happened and it really
1:08:54can't
1:08:55an event because then say files have
1:08:58been created really really fast and
1:08:59things like I'm alright I'm keeping up
1:09:01on pulling these events off the queue
1:09:02I'm keeping track of stuff I'm it would
1:09:04consolidate events to say like this if
1:09:05this file was changed 17 times in
1:09:07sequence the event log just needs to
1:09:09know the file was changed doesn't need
1:09:10to know how many times it was changed so
1:09:11we consolidate all those 17 changes into
1:09:14a single change event but eventually
1:09:15it's got to write out to its little
1:09:16compressed log file and disk this file
1:09:18change and it's got to do that fast
1:09:20enough that it doesn't get overwhelmed
1:09:21by the events that are coming in because
1:09:22if the kernel can't put another event on
1:09:24the queue for this listener
1:09:25it's going to send out a message and say
1:09:27look I had another event for you but
1:09:28there's no place to put it your queue is
1:09:30full I can't put it in your queue if I
1:09:32did that have to delete one of the
1:09:33things that's there so I'm just going to
1:09:34send you a message it says dude you
1:09:36didn't keep up you missed an event sorry
1:09:38about that
1:09:39if that happens the next time Time
1:09:41Machine runs it may not know about some
1:09:43file has been modified and then when you
1:09:44disk goes bad and you restore from your
1:09:46time machine back up you're like wait
1:09:47where's that file and if it's not there
1:09:49that's bad so there are certain
1:09:52constraints especially we're backup
1:09:54software but even for something like
1:09:55spotlight where like say you make a new
1:09:57file and something couldn't keep up and
1:09:59you go searching for it and you can't
1:10:00find it it's never going to get into the
1:10:01index because nobody knows that you know
1:10:03all you know all I can do is send to the
1:10:05the process you missed an event what is
1:10:08the recourse that time machine and
1:10:10spotlight have if they miss an event
1:10:13they the alternative strategy is really
1:10:17really bad because like our so I missed
1:10:19an event something happened and I don't
1:10:20know about the kernel is never going to
1:10:21tell me again because it's long since
1:10:23forgotten about that and moved on to
1:10:24other things and I don't know what
1:10:26happened maybe a really important file
1:10:28was modified or deleted or created or
1:10:30something and if I can't add to remove
1:10:32it to my index and I can't put it in my
1:10:35logs at a time machine knows to copy it
1:10:36to another disk we've got a bad
1:10:38situation here so what most of those
1:10:39programs do in response to you missed an
1:10:41event is to say oh I guess I got to scan
1:10:44the whole disk again from top to bottom
1:10:46maybe you've seen this if you do
1:10:49something that angers either Time
1:10:50Machine your spotlight where you log in
1:10:52and spotlight says it's indexing your
1:10:54disk again it's got to read every single
1:10:57file in every single directory starting
1:11:00from the beginning right now so it's
1:11:02gotta say forget it I'm totally screwed
1:11:04up I can do everything over again
1:11:05re-indexing with spotlight a one
1:11:07terabyte disk gets filled or
1:11:09are doing a brand-new time machine back
1:11:10up from the beginning and figuring out
1:11:11on a file-by-file basis is this file
1:11:13change no is this file change now it
1:11:15takes forever and grinds your disk to
1:11:17death
1:11:17this is all very very bad so that's why
1:11:23in when this mechanism introduced the
1:11:26public interface for applications was
1:11:29are you can't drink from the fire hose
1:11:30because you can't be held responsible
1:11:32for being able to keep up with the
1:11:33stream of stuff and furthermore if we
1:11:35have too many listeners listening for
1:11:36every single filesystem event that's
1:11:38happening we're going to have to slow
1:11:39down and we might not have enough kernel
1:11:41buffers for them and it's just you know
1:11:43this is not for you don't touch us but
1:11:45we'll have a public interface that will
1:11:48give you a much lower granularity of
1:11:51updates instead of telling you any every
1:11:53time something changes all we're going
1:11:55to tell you is something changed in this
1:11:58directory and we're not going to tell
1:11:59you what it was and those events can
1:12:01come kind of in a leisurely manner so if
1:12:03there's some huge amount of filesystem
1:12:04activity going on your application using
1:12:08this API is only going to find out about
1:12:10it perhaps after a whole bunch of stuff
1:12:11has changed and it's just and all you're
1:12:13gonna get is something changed in this
1:12:15directory and what happened I don't know
1:12:18that's not up to us so that is a much
1:12:20lower granularity of event then this
1:12:23file was deleted this file was created
1:12:24this file was modified this file was
1:12:26deleted you know that type of thing and
1:12:27that was the public interface to file
1:12:31system events now and time machine can
1:12:35consume that as well because but what
1:12:37that means is that what you get an event
1:12:38that says something changed in this
1:12:39directory you have to have a way to
1:12:41figure out all right well what changed
1:12:43one of the ways you could figure it out
1:12:45for example if your time machine is they
1:12:46could say something change in this
1:12:48directory all right well I've got a
1:12:50backup copy of that directory why don't
1:12:52I read that the entire content of that
1:12:54directory and compare it to the current
1:12:56contents and find the differences this
1:12:59is something similar to what and user
1:13:01programs do when they get a notification
1:13:03it's up to the program to decide like if
1:13:06you're interested in changes in this
1:13:07directory start by reading the contents
1:13:09of the directory and then listen and
1:13:10will tell you if anything changed and
1:13:12when they tell you something change
1:13:13reread the contents of the directory and
1:13:15compare it to the copy you had in memory
1:13:16and see what changed then you can tell
1:13:17oh this file is new this file is
1:13:19modified you know that's kind of
1:13:20annoying and cumbersome or whatever
1:13:22but you can do that sort of at your
1:13:23leisure without angering the colonel
1:13:25who's sending a little chocolates down
1:13:26the assembly line and without being
1:13:29responsible or pulling chocolates off
1:13:31the assembly line fast enough to keep up
1:13:32with what's going on with file i/o now
1:13:36all this is a very roundabout way to
1:13:38talk about how ZFS tackles the same
1:13:42problem ZFS doesn't have to have
1:13:44something in the kernel that listens to
1:13:47file system changes and writes them out
1:13:49to a compress log file
1:13:50there is no continually growing compress
1:13:52log of all the things that have happened
1:13:53in a ZFS system and the thing that has
1:13:57to send the difference between one
1:13:58snapshot and another doesn't read a big
1:14:01log of the things that happen the file
1:14:02system itself is essentially a log of
1:14:04all the things that it happened because
1:14:06it doesn't overwrite data it only writes
1:14:08data to new location so it's basically
1:14:10got a long stream of new data just being
1:14:14added and added and added and then being
1:14:16reaped off the back when it's no longer
1:14:18in use so the thing that has to find the
1:14:21the differences between one snapshot and
1:14:23another first of all it doesn't have to
1:14:24do with the file level because that's
1:14:25not how the file system works all the
1:14:27i/o is at a block level so if you change
1:14:29three bytes inside a two gigabyte file
1:14:32you're going to have it in ZFS those
1:14:34three bytes written to a new location
1:14:36the old three bytes in an old location
1:14:38and you'll know since last time I look
1:14:43at this or since from one snapshot the
1:14:44other one snapshot includes this this
1:14:46those three bytes in this block and the
1:14:48other snapshot includes those two bytes
1:14:49in this block and it can just send those
1:14:51bytes from one place to the other it
1:14:53doesn't have to send the entire
1:14:54two-qubit file it doesn't have to keep
1:14:56track of the fact that file change
1:14:57because when it sees it that this
1:14:59snapshot has one block for this file in
1:15:00snapshot has the other block for the
1:15:01file everything's check sounds it knows
1:15:03the contents of the of the blocks are
1:15:04different it knows which one came later
1:15:05it doesn't it's sort of like inherent in
1:15:08the structure of the file in itself what
1:15:11has happened who and what the
1:15:13differences are between these things
1:15:14it's not instantaneous to figure out
1:15:16what the block level dips are but it
1:15:19doesn't require scanning all of the
1:15:22source and all definition destination
1:15:24and comparing them that's one of the
1:15:27reasons why this type of strategy long
1:15:29ago is called log structured file format
1:15:30or does everything is appended to the
1:15:32end as a big linear stream of new things
1:15:34that have happened
1:15:35and snapshotting just really tells it
1:15:37which sets of these are relevant to each
1:15:39other so this is one of the reasons
1:15:44people were excited about ZFS you can
1:15:46imagine a time machine you rebuilt on
1:15:48top of something like ZFS your time
1:15:53machine backups would look like this
1:15:54first of all you could do Time Machine
1:15:56backups locally now people know about
1:16:00Lian know that it has local time machine
1:16:02backups too because this is important
1:16:03for laptop users which is increasingly
1:16:04pretty much a close approximation of Mac
1:16:08users because vast majority people by
1:16:10Macs are buying laptops and they tend to
1:16:12just have one disk so what do you do for
1:16:14backups well Apple since their straddle
1:16:17with H OS plus just add in another
1:16:18mechanism whereby anytime a file is
1:16:20saved it writes out it tries to write
1:16:22out the byte level dips between that
1:16:24file in this whole separate directory
1:16:25full of file there everything that
1:16:26happens in HFS+ is built on top of the
1:16:28filesystem rather being built into it so
1:16:30it's another bunch of files and another
1:16:32bunch of database files it's like a
1:16:34sequel Lite database and little chunks
1:16:36of files you know Selway someone saved
1:16:38the new version of this file well since
1:16:40there are laptop user and they're not
1:16:42connected to their time machine disk
1:16:43right now make a note of this change and
1:16:46figure out which bytes changed since
1:16:48last version of this file and write them
1:16:50out to the separate directory in a
1:16:51separate file it's all in an
1:16:52application-level type of thing and it's
1:16:54not particularly elegant or nice but
1:16:56this is something that they're trying to
1:16:58do to protect Mac users who are
1:17:01frequently away from a time machine
1:17:02drive well if you have ZFS you could
1:17:05take snapshots of the entire disk every
1:17:06five minutes or every time you save or
1:17:09any you know instantaneously at any time
1:17:11you wanted they wouldn't require any
1:17:13demons listening for things or any
1:17:15separate database files or any sequel
1:17:17Lite database of different changed
1:17:18ranges or any secret directory full of
1:17:20chunks of old files with directories
1:17:22named vector dates all this stuff all
1:17:24that stuff wouldn't have to exist it
1:17:25would be inherent in the file system
1:17:26itself so it should do a time machine
1:17:28back up you would take a snapshot and
1:17:30the reason you want to take a snapshot
1:17:32is because you want to do a backup as an
1:17:34ax point in time that's nothing that
1:17:37Time Machine can't do when you start
1:17:38doing your time machine backup you might
1:17:40realize as you're using a computer that
1:17:41by the time you finish doing your time
1:17:43machine back up the disk is changed so
1:17:45what what is the time machine backup
1:17:47right is it a backup as of the time
1:17:49it started or as at the time it ended
1:17:50it's kind of a melange of all of them
1:17:52combined like if if you're editing a
1:17:55file and you see the little time machine
1:17:56spinning thing you know it's backing up
1:17:58right now and you hit save you have no
1:18:00idea what the version that of that thing
1:18:02got saved in fact what time machine will
1:18:03do is after it's done running it will
1:18:04quickly see all right while I was
1:18:06running it anything happen and it will
1:18:07run again and it only does that leg once
1:18:09or twice and tries to sort of catch up
1:18:11but it's not a point in time of anything
1:18:13it's just kind of a mixture of different
1:18:14of different things so if you had ZFS
1:18:18you would take a snapshot and say look
1:18:19I've got a completely consistent point
1:18:21in time snapshot instantaneously this is
1:18:23about the context of the disk was then
1:18:25you can make a clone a writable snapshot
1:18:28of that on the backup disk or you know
1:18:31in place wherever you want to do it and
1:18:32then you could send the block level
1:18:33diffs between the snapshot in the clone
1:18:35any time something changes so you would
1:18:37have on the backup disk a series a
1:18:39series of yeah they have to be right up
1:18:42because you have to send the block level
1:18:43disk you would have a series of clones
1:18:45that you send the block level dips to
1:18:46and to figure out what has changed this
1:18:48last time you don't have to read some
1:18:50big log of stuff or anything like that
1:18:51and if a two gigabyte file has changed
1:18:53you don't have to send the entire two
1:18:55gig file you can just change send the
1:18:57blocks that have been modified it would
1:18:59be vastly more efficient vastly more
1:19:01reliable and just the number of neat
1:19:04things you could do with a constant time
1:19:06snapshots and clones and block little
1:19:08easy block level dips is just tremendous
1:19:10you might still need a mechanism but
1:19:12through which applications can listen
1:19:15for file system events and things that
1:19:17are changed but then that that suddenly
1:19:19takes it off of the path of things like
1:19:21backups ah which can't possibly miss
1:19:25anything because they don't have to
1:19:26worry about anything it's inherent in
1:19:27the file system itself there's no way
1:19:28they could possibly miss any event
1:19:30because there are no events being
1:19:32written it's just that's how the file
1:19:33system is structured I'm one one more
1:19:37final thing I want to talk about it
1:19:38there's more things receive us usually
1:19:39the ZFS wikipedia pages get a lot of
1:19:41stuff deduplication this is sort of an
1:19:44Enterprise II feature that ZFS has
1:19:46because it's an enterprise a file system
1:19:48what this means is that so
1:19:50ZFS is basically managing these big
1:19:52blobs of data a single large file could
1:19:56be made up of lots of variable size
1:19:57blobs that all make up the entire piece
1:19:59of data and each one of these blobs is
1:20:02check sums of
1:20:02that's what ZFS does well once it has
1:20:04these check sums which are small
1:20:05compared to the data you know you could
1:20:07have I don't know how big the blocks are
1:20:09but like a megabyte 512 kilobytes or
1:20:12whatever the checksum itself is like you
1:20:16know 256 bits it's very small so given
1:20:21that you have all these check sums what
1:20:24you can do is when you say you write out
1:20:26a 2 megabyte file and then you write out
1:20:30another juvett file a week later that
1:20:33happens to have the exact same contents
1:20:34as that first 2 megabyte file why are
1:20:37you storing the exact same 2 megabytes
1:20:39in two different places on your disk
1:20:41since ZFS checksum both of those 2 mega
1:20:44byte blocks say they're in stored in big
1:20:46contiguous 2 megabyte block it knows
1:20:47that they're the same and what it can do
1:20:50is say well we don't need two copies of
1:20:52this 2 megabyte file somewhere on disk
1:20:54I know these the contents are the same
1:20:56because the checksum is matched why
1:20:58don't I just get rid of one of them and
1:20:59now both of those files are talking to
1:21:01the same 2 megabyte block on disk that's
1:21:04that's deduplication and it happens at a
1:21:06block level not at a file level so if
1:21:07you have a 3 gigabyte file and there
1:21:09like 1 megabyte regions within them that
1:21:11are identical any region that's
1:21:13identically when we pulled out so one
1:21:15example is say some file format has a
1:21:17long header on it it's always identical
1:21:20for all files it's just like our
1:21:22preamble or PO stamp or something and
1:21:24you have hundreds and hundreds of those
1:21:25types of files you only have to store
1:21:28that preamble or post amble in the file
1:21:29format once for all those hundreds of
1:21:32files so you can get disk space back by
1:21:35finding out which portions of the file
1:21:37are the same as portions of other files
1:21:39and only storing one of those portions
1:21:41this is something that happens sort of
1:21:43offline where it doesn't happen
1:21:45necessarily in real-time or whatever but
1:21:47it's a way that you can say look my disk
1:21:48is almost full there any way you can
1:21:49give me more space it will find any
1:21:51little piece of any file it's the same
1:21:52as any other piece of any other little
1:21:53file and check out one of them or check
1:21:55out all the mix-up for one ah and why
1:21:58does this works like what about when
1:21:59something needs to modify that again
1:22:00well it's copy and write anything you
1:22:02want about anything you're always
1:22:03writing to a new location this doesn't
1:22:04hurt you at all in terms of making your
1:22:07data frozen in place or unable to modify
1:22:10this is one of the consequences of
1:22:12having a file system or you just write
1:22:13things out and never overwrite data in
1:22:16place
1:22:16uh and it's part of the file system so
1:22:20if Apple had deduplication as part of
1:22:22their file system I I would really not
1:22:24like to see them try to add this HFS
1:22:25boss but if they had a file so isn't it
1:22:26supported this if you just started to
1:22:28get full one of the things the operating
1:22:30system could do is say ok let me just go
1:22:31look for let me just start deduplicating
1:22:34blocks and let me find some common
1:22:35regions of files and it doesn't have to
1:22:37scan every single file to find common
1:22:39regions it just has to look at the check
1:22:40sums which are very very small compared
1:22:42to the file so we're going to say you
1:22:43know if it stores all its check sums in
1:22:45kind of like a sorted table a checksum
1:22:46somewhere they can very quickly find
1:22:48check sums are the same and just start
1:22:50reaping data and marking it as free
1:22:52space because it only needs one copy so
1:22:56that's it for my neat features of ZFS
1:22:59that HFS+ doesn't have I hope I've
1:23:01wetted people's appetites for ZFS
1:23:04another cool new file system there's not
1:23:06much they can do about it except hope
1:23:08wait and hope they can be as
1:23:11dissatisfied as I am I put a link in the
1:23:14show notes two more links to Jeff bond
1:23:17wigs blog about these carpet topics a an
1:23:20old ACM queue interview from 2007 with
1:23:23Jeff bond wick and a link to the
1:23:27original like PowerPoint presentation
1:23:30for ZFS it's called ZFS the last word in
1:23:34file systems these are all classic
1:23:36pieces of literature from ZFS oh and by
1:23:38the way on that on the butter FS thing
1:23:40that I link to as well there are many
1:23:44links to the academic PDF so the
1:23:46academic paper is on which is based even
1:23:48if you're someone who doesn't ever click
1:23:49on links to PDFs of academic papers did
1:23:51you expect it to be just horrible and
1:23:53dry and impossible to read these are
1:23:55surprisingly readable to anybody with
1:23:57even a tiny bit of a CS background so I
1:23:59encourage you to actually click on the
1:24:01PDFs and read the the academic papers
1:24:04they're short they're not too
1:24:05complicated they're not filled with
1:24:06equations that you're not going to
1:24:07understand it's pretty basic stuff that
1:24:12each oh yeah I was running my iPad 3
1:24:16predictions yes of course I've been
1:24:20going back and forth in this when I was
1:24:22much more sure like two weeks ago but
1:24:25now as we get closer I don't know it
1:24:28seems like there's there are a few
1:24:30points
1:24:30of that the rumors are addressing a few
1:24:34points the first point is the screen
1:24:36everybody seems to agree there will be a
1:24:38Retina display there's a second question
1:24:40which is will there continue to be a
1:24:42previous generation of iPad like the
1:24:44iPad 2 along with the new one there is
1:24:48the question of price points and and
1:24:52then finally I believe there's a
1:24:54question of naming conventions will this
1:24:56be the iPad 2s the iPad 3 the iPad HD
1:24:59there's even more question about yeah so
1:25:01those get paid once I'd love to hear all
1:25:03of them
1:25:04the only thing everyone agrees on is
1:25:05Retina display yeah I also agree on that
1:25:07yes
1:25:08I mean we've we've all known this is
1:25:10kind of like we all know that the the
1:25:11MacBook Pro without an optical drive is
1:25:13coming like we know it's coming yeah
1:25:15matter of time and now we know the time
1:25:16it's that's definitely come okay ah
1:25:20I am almost as certain that Apple will
1:25:23continue to sell an iPad with a non
1:25:25Retina display almost hundred percent
1:25:28certain because it just seems crazy to
1:25:30me that they wouldn't will it be the new
1:25:32iPad or will it be the existing iPad
1:25:34with without the Retina display like a
1:25:36two versions of the new one come out or
1:25:38the one that we currently have today as
1:25:40the iPad 2 continues to be sold
1:25:41I think they keep selling the old one
1:25:43because that I think that the economies
1:25:46of scale network well they already have
1:25:47the tooling the assembly lines the
1:25:49people you know everything ready to make
1:25:51those exact things and maybe you stamp
1:25:53something different on the back of it
1:25:54for branding if they wanted to change it
1:25:55but it's the same reason they sell still
1:25:56sell the iPhone 4 and the 3GS like once
1:25:58you do the investment in making those
1:26:00products it's much cheaper just to keep
1:26:03making them and if they you know if they
1:26:05want to do they could change something
1:26:06about the name but I don't even think
1:26:07they will I think it will be just be
1:26:09exactly identical it'll be the iPad 2 I
1:26:13don't think we'll even bother renaming
1:26:15it to be like the iPad 3 slow version
1:26:17you know because that would just be a
1:26:19matter of stamping something different
1:26:20on the back of the thing but I think
1:26:22they'll just continue to sell the old
1:26:23version the other one that I was really
1:26:26really sure about and now I'm kind of
1:26:28waffling but I'm still mostly sure is no
1:26:31LTE I was so sure there would be no LTE
1:26:34but ibex I'm reading too many rumor
1:26:36sites and it getting into my head I'm
1:26:39pretty certain olt that's I feel
1:26:41strongly about that I but now
1:26:43I have doubts doubts have been so about
1:26:45this rumor site so I should sure not
1:26:48read them should just stuck to my guns
1:26:49and say no you're still saying no LT
1:26:51yeah what does the price point at price
1:26:55is another thing that I used to be more
1:26:57certain of and now I'm doubting I I
1:26:59think I used to think and still kind of
1:27:02do that Apple at least shouldn't be
1:27:05afraid to sell a higher priced model to
1:27:09have one model available that is more
1:27:11expensive than any of the current models
1:27:13available i but now I think that Apple
1:27:18is afraid of that I don't think they
1:27:19should be I think they should extend in
1:27:21both directions I think they should be
1:27:23able to have high-end model for people
1:27:27who want it and if the high amount is
1:27:28like oh it's got 128 gigs of flash and
1:27:32the retina display and you know the LTE
1:27:35radio because I'm wrong on that or you
1:27:37know something like push out in the high
1:27:38end but as we get closer I think man
1:27:42Apple just really does not want to go in
1:27:44that direction the direction I think
1:27:46more people can agree on is they're
1:27:47going to try to push lower if that means
1:27:50a price drop on the iPad - if that means
1:27:52you know like whatever that means they
1:27:53want to keep pushing down that always
1:27:55want to push down push down with the
1:27:56iPods until they're so incredibly cheap
1:27:59they cost about as much as a case you
1:28:00get an iPod shuffle the same prices you
1:28:02can get like an iPhone good iPhone case
1:28:03all right pushing down down down I think
1:28:06the way they push down they're going to
1:28:08push down with the iPads the same way
1:28:09they would have with the iPhones which
1:28:10is selling previous models the real
1:28:14question is can Apple afford to sell an
1:28:16iPad with a Retina display at the same
1:28:20price points as the iPad 2 for the same
1:28:23other features so if you take an iPad 2
1:28:24that has a certain amount of flash
1:28:25memory and it has Wi-Fi and it has 3G
1:28:28can you just swap out that display and
1:28:29sell it at the same price certainly albo
1:28:31can they've got room in their margins to
1:28:33sell it with that but if they were going
1:28:35to I thought I would have seen more of a
1:28:40warning about future margins going as a
1:28:42thing for Horus to talk about nothing
1:28:44but in the earnings call I would have
1:28:47thought that there would have been more
1:28:48warnings about our margins are going to
1:28:50be lower next quarter so just keep that
1:28:51in mind and if you ask them why they
1:28:53would say we don't talk about our
1:28:54announced products and component
1:28:56prices in blah blah blah like they would
1:28:57be evasive as they always are but if
1:29:01they're going to have the exact same
1:29:01price points as the iPad 2 and maintain
1:29:03their margins I don't think it's
1:29:05possible because the screen just plain
1:29:07cost more no matter what kind of
1:29:08sweetheart deal they got on these
1:29:09screens and I'm sure they got a good one
1:29:10it doesn't cost the same as the old
1:29:13screen their margins have to go down so
1:29:16given that I think they may do some
1:29:21fancy mumbo-jumbo by shuffling the other
1:29:24features of the things so that the price
1:29:26points remain the same but what you're
1:29:27getting for the current cost of a 32 gig
1:29:30Wi-Fi only iPad isn't is not exactly
1:29:34comparable to what you get with the same
1:29:37price point a Retina Display iPad this
1:29:40is all lots of waffling which is to say
1:29:42that it's clear that Apple wants to hit
1:29:45exactly the same price points and extend
1:29:47down that's the ideal they want exactly
1:29:49the same price as the iPod - iPad - but
1:29:51now the price range goes lower so if you
1:29:53can't afford even the cheapest previous
1:29:54iPad now maybe you can because the
1:29:56cheapest model is even cheaper and yeah
1:29:57maybe it's last year's model I think
1:29:59they should extend up but I think
1:30:01they're afraid to maybe if they
1:30:04introduced one with more with more flash
1:30:06like a 128 gig one they might extend up
1:30:09but so if I had if you had to pin me
1:30:11down I would say same price points and
1:30:15the price range extends downward but not
1:30:17upward what else do we have the name
1:30:20three versus 2's I'm not really
1:30:23interested in that because it's not
1:30:24going to change the product that's just
1:30:26as well you refuse to comment to s3 a
1:30:33check those are the ones that are
1:30:35floating around today it probably
1:30:39depends on if it has a new case ah the
1:30:42rumors are that the case is going to be
1:30:45slightly thicker if that's the case
1:30:47they're going to call three if the case
1:30:49is exactly the same that you get away
1:30:51with 2s and I think they might do that
1:30:53but I'm leaning towards three and a
1:30:56different back slightly different back
1:30:58camera I'm assuming the camera will be
1:31:00improved but I also would not be shocked
1:31:02if it wasn't about other people would be
1:31:03shocked if it was I know but other
1:31:04people would flip out this could be one
1:31:05of those ways they maintain the price
1:31:07point because that camera that
1:31:08piece-of-crap camera in the
1:31:09back of the iPad it's got to cost them
1:31:11two cents by now and so they said well
1:31:13how can we hit the price point that we
1:31:15want to hit and how can we not raise
1:31:16prices given that we're putting the
1:31:17super expensive screen in there well
1:31:18keep the same crappy camera ah do you
1:31:21think a lot of people use the iPad 2
1:31:23cameras for the FaceTime video
1:31:26I bet the one in the front gets much
1:31:27more use than the one in the back yeah
1:31:28but you know we've all seen the people
1:31:30holding up the iPads to take pictures of
1:31:32stuff it happens everyone is like all
1:31:35that now they can put in the iPhone 4
1:31:37camera in there if not the 4s camera
1:31:38you're right they could but will they
1:31:40Adam I'm I doubt that a lot the CPU a 6
1:31:45verses a improved variant of the a5 I'm
1:31:48leaning pretty heavily towards the
1:31:50variant of the a5 which means a
1:31:53dual-core chip instead of quad-core I
1:31:54don't think the a6 is ready yet and I
1:31:57think the a5 with a better GPU and
1:31:58whatever in their hands as they need for
1:32:00the big honkin screen will be okay and
1:32:02some people have said that even the a5
1:32:03could handle a Retina display as is now
1:32:06maybe just a few other things need to be
1:32:08tweaks so what else is left a 5x same
1:32:14price points HD no LTE possibly the same
1:32:18crappy camera will there be anything
1:32:22announced at the event such as a new or
1:32:25an actual new version of the Apple TV
1:32:30something different that supports 1080p
1:32:32better newer CPU inside I would really
1:32:38like it if I I'm leaning towards know
1:32:42nothing about Apple TV but if there is
1:32:44something about Apple TV I'm I'm
1:32:48thinking it's a new little black Apple
1:32:51TV box that just has like bumped specs
1:32:55basically it's not like an Apple
1:32:57television set it works the same as the
1:32:59current little black square but you know
1:33:01supports 1080p or whatever you know but
1:33:03nothing big revolutionary which would
1:33:06save the Apple television set Apple
1:33:09saves TV whatever pipe dream that we all
1:33:11have for an end-of-the-year event and
1:33:13just give us 2012 to mostly say if
1:33:17you're going to buy an Apple TV now you
1:33:18can get an apple TV 3 which looks
1:33:20exactly like the Apple TV 2 it's little
1:33:21black square
1:33:22maybe the biggest change they make is
1:33:24that the the remote is Bluetooth but I
1:33:26don't think they've changed that it
1:33:27would be the same so there's going to be
1:33:29no announcement or it's going to be a
1:33:30little dinky black Apple TV 3 type thing
1:33:33su degree what you agree with John
1:33:34Gruber who said laser focus on the iPad
1:33:37and and maybe something supportive in
1:33:40the Apple TV to go along with 1080p in
1:33:45the iPad or something like yeah why
1:33:46would you take away from the iPad event
1:33:48by by showing the new version of the
1:33:51Apple TV and the flip side is if there
1:33:54was like a really interesting TV product
1:33:56like whatever Apple plans to do in TV
1:33:57that's different than this little box
1:33:59they've been making that would have been
1:34:00evident on the invitation because that's
1:34:02not something you do is like oh you know
1:34:04oh by the way one more thing we have an
1:34:06entirely new crazy product that is just
1:34:08insane that's going to revolutionize
1:34:10tell that's what the event is about if
1:34:11they're going to ever come out for you
1:34:13know we're gonna revolutionize
1:34:14television that's not going to be an
1:34:16afterthought in the event I just don't
1:34:18see that so I'm thinking nothing about
1:34:20television and then people would just be
1:34:21sad like I say what the heck am i except
1:34:23for maybe itunes store selling HD stuff
1:34:26that seems reasonable to do because
1:34:27there's like I get this read an iPad
1:34:29yeah my games look great but I gotta
1:34:30watch the 720p movies on it and I get
1:34:32zoomed up that's kind of icky so 1080p
1:34:35content could have come at any time
1:34:36because max can play 1080p now so I
1:34:38don't you know whatever Apple is waiting
1:34:39on to unleash the 1080p video content
1:34:42that could be waiting on deals or
1:34:45re-encoding video or they gotta
1:34:46renegotiate something at all those deals
1:34:48is kinda like the Beatles it's like what
1:34:50was stopping the Beatles thing from
1:34:51happening
1:34:51just stupid negotiation so I'll add the
1:34:54last question I've heard a lot of people
1:34:56out there speculating what apps or what
1:34:58kind of apps will be demoed and featured
1:35:00to show off the new Retina display I've
1:35:03heard you know if you remember earlier
1:35:05this week Adobe released photo photo
1:35:08shop touch and perhaps I hadn't thought
1:35:12of this but a couple people on Twitter
1:35:14and said to me oh I think that app was
1:35:16leaked it shouldn't have come out that
1:35:18soon it was supposed to be held back it
1:35:20came out too soon something like that
1:35:21and that was supposed to be the big app
1:35:23that Apple demoed on stage perhaps
1:35:27that's true perhaps not question for you
1:35:29what what kind of app do you think that
1:35:31they'll use to highlight the new Retina
1:35:34display games
1:35:36so given they're gonna do games because
1:35:38that it combines everything they want to
1:35:39show they want to show the more power
1:35:40slightly more probable GPU they want to
1:35:42show that the graphics aren't Jaggi or
1:35:44anything drawing app is probably a good
1:35:47choice the real problem as Gruber
1:35:49pointed out is the Retina display is
1:35:53very difficult to demonstrate because
1:35:54you're probably showing it in some
1:35:56little movie and like YouTube or even
1:35:59Apple's own HD movies you can't see the
1:36:03difference of like someone filming a
1:36:05Retina display and then compressing that
1:36:06video you know I think he said you need
1:36:08a Retina display to appreciate a Retina
1:36:11display
1:36:11yeah and they can do it in slides
1:36:13because on slides they'll zoom in the
1:36:15slide they'll show the zoomed in picture
1:36:16of a sprite from a game looking all
1:36:17blocky and I'll show the nice smooth
1:36:18line it's very easy to demonstrate in a
1:36:20slide but when they have the actual iPad
1:36:22thing in their hand what are they going
1:36:24to demonstrate that I think this is
1:36:27actually a problem for for this iPad
1:36:30announcement because we all know that is
1:36:32going to be Retina display we all know
1:36:34it's impossible to demonstrate and it's
1:36:36like what applications is this Retina
1:36:38display make possible that previously
1:36:40we're impossible
1:36:41pretty much none it just makes a whole
1:36:43bunch of them better and we all
1:36:43understand that but then like what do
1:36:45you talk about for the rest of the time
1:36:46if it's just an a5 X you can say oh and
1:36:48the graphics are you know X amount
1:36:50faster if it's not quad-core you can't
1:36:53say on this a double number of cores but
1:36:55even that like I demonstrate speed ah if
1:36:58even if is LTE that you know I don't
1:37:00know that I guess I guess I could look
1:37:03back at what they demonstrated the iPad
1:37:042 announcement there's not something
1:37:07kintyre lee knew that you couldn't do
1:37:09before that you can do now it just makes
1:37:10things better and faster it's kind of
1:37:12like a it's not a speed bump but it's
1:37:13kind of like when they do would do all
1:37:15new all new Powerbooks and what would be
1:37:19different about them well they'd have
1:37:20faster CPUs that hold more memory that
1:37:22this could be able to get bigger and
1:37:23maybe the screen is nicer in some way
1:37:27but that's it's not a speed bump that
1:37:30I'll then have a new case so this could
1:37:31have a new case too so I think their
1:37:34ability to do cool demos with the iPad
1:37:36is decreasing and they are they're going
1:37:38to be forced to demonstrate their market
1:37:42strength by saying look at all the cool
1:37:43applications we have so they're probably
1:37:45going to demo applications that if you
1:37:46think about it like well you could add
1:37:48that application on the iPad
1:37:49- and yeah it looks nice easy to write
1:37:51in a display but there's nothing about
1:37:53this new iPad that now makes this
1:37:55application possible it wasn't possibly
1:37:56for a good example is Microsoft Office
1:37:58Microsoft Office could exist in for the
1:37:59iPad 1 or the iPad 2 it happens to be if
1:38:02it happens to be available now that
1:38:04would be a good thing to demonstrate to
1:38:05say look we've got Microsoft Office on
1:38:07the iPad isn't that awesome and the same
1:38:09thing for games or any medical imaging
1:38:12application you want to show I don't
1:38:13really know what their pitch is going to
1:38:15be I just know that their job is getting
1:38:16harder it's much easier to show the iPad
1:38:19but the first time and the second one
1:38:20it's like thinner and his cameras and
1:38:22stuff but now it's not like going to
1:38:24keep adding geegaws and making new
1:38:27things possible on the iPad it's just
1:38:28going to be a demonstration of look at
1:38:30the neat new applications that are out
1:38:32for the iPad now a dock connector I
1:38:38think the doctors say is the same do you
1:38:42think that we will see a new dock
1:38:43connector sometime in the near future
1:38:45I really hope so maybe not this revision
1:38:49it almost sounds like we could do a
1:38:51whole show just talking about connectors
1:38:52like we did that time I wish I did
1:38:55oh i phone 5 is my bed for connector
1:38:57changes will it happen just for the
1:38:59iPhone or will it happen across the
1:39:02board album for the iPhone first that's
1:39:05my prediction okay no so it be with the
1:39:075 and it will not some people have
1:39:10speculated will it be a u.s. mini USB or
1:39:12micro USB and you're saying no it will
1:39:14just as we have the previous NOC
1:39:16connector it will be its own new thing
1:39:17that won't be compatible with anything
1:39:20else they'll probably sell an adapter
1:39:21for it but it's not like they're going
1:39:23to take a Thunderbolt port there or some
1:39:27kind of other that it will be Apple's
1:39:29own new port that we've never seen
1:39:31before I think I would love to use
1:39:33Thunderbolt I don't think there's room
1:39:34inside an iPhone right the controller
1:39:36chips to support this just it don't fit
1:39:38it's not going to not going to make it
1:39:40and never mind that the connector itself
1:39:42would has to have all this chips in and
1:39:43everything that's also big and bulky so
1:39:45they would love that Thunderbolt and
1:39:46thunderbolt is probably actually
1:39:48feasible on the iPad but not on the
1:39:49phone I think the reason they're going
1:39:50to go to a new connector on the phone if
1:39:52it doesn't happen in the iPhone 5 then
1:39:53maybe the 6 or something is that they're
1:39:57going to keep making that damn phone
1:39:58thinner and at a certain point it be
1:40:00like that the phone at the thin end is
1:40:03like
1:40:03as thin is the 30-pin connector is and
1:40:05it starts to become your limiting factor
1:40:07and you know also I've seen you know the
1:40:10people taking screens the inside how
1:40:11much that how much room on the inside of
1:40:13the phone that connector takes up it
1:40:15starts to become an issue it's like this
1:40:17this stupid connector is dictating the
1:40:19design of our phone to a large extent
1:40:22because it's taking up room on the
1:40:24inside and we can't make the edges any
1:40:25thinner than this because it gets a
1:40:26little bit ridiculous the ipod touches
1:40:27close to the as thin as you can make it
1:40:30and still support that 30 pin connector
1:40:31in a reasonable manner so I think it's
1:40:34it's days are numbered and and one of
1:40:37the easy predictions is no replacement
1:40:39the connector is gone and there is no
1:40:41other connector right they'll use they
1:40:43use Wi-Fi for syncing and capacitive
1:40:45charging further further charging and
1:40:48that's it there you don't plug your
1:40:49phone into anything well you could for
1:40:51charging you can always do a little
1:40:52pinhole like charging you can get away
1:40:54with a really tiny hole you know we
1:40:56think they would be able to repurpose
1:40:58the way they've done with the shuffle
1:41:00where you can charge with the headphone
1:41:02jack yeah it's possible you don't charge
1:41:06for the headphone jack on the shuffle or
1:41:07you charge with the well I guess it's
1:41:09the USB connector through the headphone
1:41:10jack but yeah right but it goes it in
1:41:12other words there's no separate there's
1:41:14only one hole in the shuffle and they
1:41:16could do the same thing with the iPhone
1:41:18yeah that's that's future if you chose
1:41:20like iPhone 6 or 7 like if we still have
1:41:22this 30 min connector by the time we're
1:41:24talking about iPhone 6 this time so I
1:41:25was thinking that that entire connector
1:41:26goes away which is it's weird because
1:41:28that connector doesn't just sync and
1:41:31charge it does all sorts of stuff with
1:41:33peripherals but at a certain point when
1:41:37this thing becomes a certain in
1:41:39particular becomes a certain size how
1:41:41can you attach something to this skinny
1:41:44little card it's just basically like a
1:41:46little screen in your aunt how do you
1:41:48attach something to that through to a
1:41:50port that connects with it it becomes
1:41:52cumbersome and unwieldy and just weird
1:41:56you know it's what weird already to
1:41:58attach things to the bottom of your
1:41:59phone even it's like a battery pack or
1:42:01something so I want the I want the
1:42:0330-pin connector go away and if
1:42:05something has to replace that I want to
1:42:07be much smaller so I'm predicting iPhone
1:42:085 but this this side new iPad I'm almost
1:42:11certain will have 30 pin connector on
1:42:13the bottom just like usual okay
1:42:16and I have no inside info just this
1:42:19disclaimer in case people think I
1:42:20actually know stuff I know nothing
1:42:21you'll see how wrong I am
1:42:23and when March 7th comes if you're
1:42:24writing down things that I predicted
1:42:26just check them off at it then that's it
1:42:33hundred minutes pretty good okay have a
1:42:39good week Joe you do now
1:42:49you