RAID is not a backup!

In part 3: We continue to extend our backup regime by adding a remote NAS. (part 2 DAS Edition, part 1 Backups)

We’re lucky enough to have some family a few timezones away that were willing to stick a small NAS box on a shelf for us. At the time we went with a very simple, entry level Synology 2-bay NAS solution. We didn’t need RAID redundancy or any media server encoding. This was our off-site, cold-storage, 3rd copy backup. Hopefully, we won’t need it.

This is our ‘3’ and ‘1’ in the 3-2-1 backup rule. (3 copies, 2 mediums, 1 offsite).

To make this as easy as possible, we took advantage of our Unifi routers. Along with the NAS we also bought a Unifi Express. We created a private VPN between our office network and the remote network. Doing so makes all the devices on that network seem as if they are on our local network. We can SMB:// into a “local” IP address to our remote NAS.

Now that we have an 8TB hard drive in a NAS a few timezones away that seems like it is local, we can create a cron job which rsyncs from our local DAS to the remote NAS.

The current hardware setup is a 4-bay DAS connected to an older Mac mini. The first two bays contain 4TB hard drives for work and personal data. The 3rd bay is an 8TB hard drive which contains a copy of the first two 4TB drives. Each night the Mac mini runs a cron job to rsync the data to the 3rd drive. We do this rather than RAID for two reasons. We can have multiple drive sizes and if it were in RAID and we accidentally delete or corrupt a file, it is instantly also deleted or corrupted on the mirrored drive too. From time-to-time, having a daily delay has proven useful. With this setup we have ‘2’ copies of the data. The 4th hard drive is another 4TB drive we use as a Time Machine backup for our laptops. That makes ‘2’ copies there too (the laptop and the Time Machine backup)

We also have an external 4TB hard drive we use as an “air gapped” Time Machine. We plug that in about once a month to our laptops to also create a backup (just in case!).

Nightly, we also rsync from the 8TB drive on the DAS to the 8TB drive on the remote NAS. At the moment, it is not using the full disk space, so we also weekly rsync the 4TB Time Machine drive to the remote NAS. With time, we’ll fill-up the hard drive, but the remote NAS has 2-bays and one is still empty.

The way the Time Machine backups are structured means that a small incremental update causes a lot of files to change. If a large file changes just a tiny bit, we still need to rsync the whole file over to the remote NAS. We decided that a weekly remote Time Machine backup would be enough. We have two local copies (plus the laptop itself) if needed.

Webhosting

Most all our websites and web apps reside in one of two places: Heroku or Opalstack. Heroku is our goto service for web apps for customer projects. Everything you deploy there is via git version control, so we have a copy on heroku, locally and in a git repository (usually bitbucket or GitHub). 3-2-1 is covered. Any databases are backed-up nightly in the cloud, but technically that’s just ‘2’ locations and ‘1’ offsite. (A 3rd backup would be best)

Opalstack is our VPS (Virtual Private Server). We host a lot of static website, WordPress, personal projects and file storage there. We’ve had bad experiences with previous VPSes like Joyent, Textdrive, and WebFaction, from having no backups to being acquired. We needed to make sure that anything created there we have a local copy of as well.

To solve this, we have a cron task on the VPS that dumps all the databases to flat files. Then we added to our local rsync script to grab all the files: database dumps and html directories from the VPS and put them onto the DAS. Once they are there, they get absorbed into the 3-2-1 system.

Backups still not covered

We have done a great job getting to the 3-2-1 for all our local files. Our laptops and file server are all automatically being backed up. So what’s left?

Our devices (iPhones and iPads) are only backed up to iCloud. That’s mostly fine, except that photos are pretty important and (we feel) too fragile on iCloud. The way iCloud works, is to save space on your device sometimes you only get image thumbnails, and then if you delete a picture on your laptop to save space, it might purge it everywhere, including your phone. To mitigate this, we use the “Image Capture” app (or Photos.app) on the laptop to copy all recent images to the hard drive. Then it is part of our local backup regime.

The other major sources of data that are not currently in any sort of regular backup routine are our cloud-base services. Files in Google Drive, iCloud Drive, Trello, Asana, email, etc. are assumed to be “backed up” by the service provider, but we’ve had instances where that’s not true. Our email address could get flagged (for no reason) and our account suspended locking us out of our data. Our next big task is to list all the places that have data only in other systems (mastodon for instance) and get an automated cron job to save that date to our DAS and into our 3-2-1 backup regime.

Every iteration where improve the backup plan gets both more difficult and the data gets smaller. Looking back, we’ve gone from nothing (or Dropbox), to a pile of external hard drives and Time Machine, to a DAS with a schedule and system in place, to a remove NAS and syncing our website data. Each step has improved our redundancy and safe guarded the data we feel is important. We’ll never be “DONE”, but with every small step, we get closer.

Addendum: We got lucky!

Recently, we had one small backup disaster that was outside out 3-2-1 system. We recovered most, if not all, of the data by shear chance.

There were several old email accounts setup from previous projects on the iPhone that were just causing problems. The accounts closed several years ago and the login didn’t work. So we deleted them. It’s good practice to clean-up and not leave any un-used accounts lingering for longer than needed.

Then a day later we realized that half the contacts were gone. Somehow, that old account was the default for all new contacts created. Normally this wouldn’t be a big issue since the contacts would be synced with the server, but since that connection was broken awhile ago those contacts ONLY existing on the phone. Now they’re gone. We looked in Time Machine backups for the computer’s contacts database, but they were never synced there. We looked on iCloud but they were not there either, since they were never synced there. The laptop didn’t have a back-up of the iPhone since that’s now managed by iCloud. iCloud DOES have a backup of the phone, but only the most recent, and that’s already been overwritten and lost all the contacts.

It was only by luck that a few months ago, we upgraded the old iPhone 12 to a less old iPhone 13 and the iPhone 12’s backup was still in iCloud. We did the dance of backing up the iPhone 13, restoring the 12’s backup, getting all the lost contacts moved to iCloud, and restoring the 13’s backup again. It took a few hours, but we restored the contacts. This experience shed a light on a few areas of improvement in our backup regime. The first thing we did was check to make sure any new contact was being put into iCloud by default. We also took a manual vCard backup of the contacts and added that into the 3-2-1 system.

Remember L-O-C-K-S-S, Lots Of Copies, Keep, Stuff, Safe.