The cobbler’s son walks barefoot is such a perfect English idiom to describe how professionals apply themselves in their jobs, and how this differs from their personal lives. I invoke it regularly here for this raison, a typo I’m keeping because it makes me smile.
Lest the title of this post suggest I don’t do personal backups, I do. But while I’ll suggest a comprehensive discussion with a client about what types of data they have, how regularly they change, legislative requirements, their Recovery Time and Point Objectives (RTO and RPO), and a suite of tooling to backup, monitor, report, test, and restore backups… my backup regime at home is just a tad bit more pedestrian.
For years my backups were on Dropbox. Dropbox used to be awesome; the desktop client was lightweight, didn’t cost much, integrated with my favourite note apps on my iTelephone, and I could sync encrypted Mac OS X sparse bundles across Macs without issue. For those not versed in the intricacies of Mac volume management, sparse bundles appear as a logical disk image when mounted, but are in fact a bundle of small files beneath. In backup terminology we may say they’re “chunked”. This is crucial for a tool like Dropbox, as it only uploads files upon detecting changes. An entire contiguous disk image would need to be reuploaded every time a file within it was modified, but a chunked image lets Dropbox only upload tiny portions of the image at a time.
(I’m simplifying here, and Dropbox and their ilk may be using more sophisticated methods to diff changes and upload now. But that’s how it worked in the past).
When I decided to start self-hosting more of my material, I went down the rabbit hole of a few different tools. As I would with a client, I sketched out my requirements:
- The ability to back up my personal home folders on my Mac, BSD, Linux, and illumos machines to our FreeBSD file server. This in turn runs its own ZFS-based backup regime which is beyond the scope of this post.
Actually, huh, that was about it. That really didn’t warrant a list, did they? I’m not sure. This paragraph isn’t turning out to be that useful either.
So I began hunting for a solution, and ended up trialling a few different self-hosted file backup options:
-
Nextcloud (well, Owncloud at the time). It worked, but it came with a bunch of tools I didn’t need, so I couldn’t justify maintaining a whole stack for it. I should revisit at some point, as my needs have definitely grown over time.
-
Seafile. This was a simpler remote file store and sync tool, but for reasons likely down to user error, I couldn’t get it installing and working reliably. It also didn’t have integrations with notes apps on the phone, which I wanted at the time.
-
Syncthing. This is a distributed file share tool that worked pretty well. I installed it on the FreeBSD host and had it always available, then had the clients on each laptop.
I don’t know how or why, but after a while this cobbler decided that the approach he was taking didn’t make a whole lot of sense. He also started talking in the third person, for some reason.
I went back to the drawing board, and realised there was several types of state my laptops carry around, all of which could be handled with more basic tooling. I perused my home directory, and noted the key folders I care about:
-
~/documents. This is for local copies of immediate projects I’m working on that can’t be easily version controlled, or contain large assets. For example, I live in the real world, and therefore have to put together slideshows for client meetings.
-
~/music. This is currently being rebuilt for FLAC! Music is an important part of my life, so I always like having a local copy of my library for listening whenever or wherever I am.
-
~/repos. I’ve had this in my home directory since my Subversion days. This has all the main projects I work on daily, including this blog. It also has my repos for scripts and configuration dotfiles, which over time I’ve written to be cross-platform. This means I can clone, set symbolic links for my environment, and have everything I want configured, whether it be NetBSD, Tribblix, or CP/M. Okay, I’m not up to that last one… yet.
-
~/.ssh. This is how I auth with servers I maintain, log into my own kit remotely, and do cryptographic stuff that’s likely ill-advised but works (hey, remember I’m cobbler?).
These are very different types of files, and I realise don’t all need the same backup approach. Documents and music can be triggered with an rsync after changes to my FreeBSD box, which sits comfortably behind the family VPN. Repositories obviously exist on Git and Subversion, which I maintain copies of on that aforementioned server. About the only one I don’t have a good answer for is the ssh configuration. Don’t tell anyone, but I keep a tarball of that around which I copy across.
I still don’t think I have a great solution here, and I’ll likely stick with Syncthing as an automated fallback. Though maybe I should also bite the bullet and reinstall Nextcloud at some point as well. Some of my colleagues are enamoured with it, and what they show me looks cool.