A while back I had my laptop stolen and while I had backups of all the important stuff my latest full-system backup was several months old since I have to dig out my backup harddrive from its alternate location every time I want to update the backup (I don't store my laptop and backup harddrive in the same building since theft/fire would mean loosing all copies).
So, being sick of imperfect backup solutions I decided to find a solution for online full-system backups that met the following criteria:
- 100% open source client
- Zero-knowledge encryption (server can't access data)
- Works on GNU/Linux
- Affordable (max $20 per month for ~1TB)
After some research I ended up using duplicity + backblaze. So far I have only run into one issue with this solution: duplicity doesn't support hardlinks. Luckily I don't use hardlinks very often.
duplicity is like rsync but with encryption so you can safely store your backups in an untrusted location.
backblaze provides low cost online storage:
Storage: $5 per TB per month
Download: $20 per TB
Upload: No charge
As long as you don't need to restore your backups backblaze is extremely cheap :), and if you do end up needing your backups then backblaze is still affordable. duplicity recently added support for backblaze as a storage backend. duplicity allows you to restore single files so you won't be charged $20 every time you need something from you 512 GB laptop HD backup.
First, sign up for Backblaze B2 Cloud Storage on backblaze.com.
Create a bucket with some name that makes sense to you (I keep a bucket for each computer I want to back up).
You should add a credit card if you'll be using more data than the free amount.
Install rsync:
sudo apt-get install rsync
Now install duplicity 0.7.06 or later.
If you previously installed duplicity manually you can remove it with:
sudo rm -rf /usr/local/bin/rdiffdir
sudo rm -rf /usr/local/bin/duplicity
sudo rm -rf /usr/local/lib/python2.7/dist-packages/duplicity-0.7.06.egg-info # change this to match the installed version
sudo rm -rf /usr/local/lib/python2.7/dist-packages/duplicity
If using a debian/ubunut based distro you can install using this PPA:
sudo add-apt-repository ppa:duplicity-team/ppa
sudo apt update
sudo apt install duplicity
To install manually if there isn't a package for your distro:
sudo apt-get build-dep duplicity
sudo apt-get install python-setuptools
wget https://code.launchpad.net/duplicity/0.7-series/0.7.16/+download/duplicity-0.7.16.tar.gz
tar xvzf duplicity-0.7.16.tar.gz
cd duplicity-0.7.16/
sudo python setup.py install
Install backblaze python package:
sudo pip install b2
If you still get errors about b2 not being install, update pip, wheel and setuptools:
sudo pip install -U pip wheel setuptools
Save the following script to /usr/local/bin/backup
#!/bin/bash
ACCOUNT_ID="your_account_id"
APP_KEY="your_application_id"
BUCKET_NAME="bucket_to_use_for_storage"
BACKUP="/" # what to back up
EXCLUDE=("/dev" "/media" "/mnt" "/proc" "/sys" "/tmp" "/run") # what to not back up
HARDDRIVES=("/dev/sda") # back up partition info for these harddrives
MBR_BACKUP="/opt/mbr_backup" # where to store partition info backup
if [[ $EUID -ne 0 ]]; then
echo "You must be root to read all system files for the backup"
exit 1
fi
# save mbr and partition information
mkdir -p $MBR_BACKUP
if [[ -d /etc/lvm/archive ]]; then
cp -a /etc/lvm/archive ./etc_lvm_archive # back up lvm partition info
fi
for HD in "${HARDDRIVES[@]}"; do
HD_NAME=`basename $HD`
dd if=${HD} of=${MBR_BACKUP}/${HD_NAME}.mbr bs=512 count=1 # save mbr
sfdisk -d $HD > ${HD_NAME}.sfdisk # save partition table (no lvm/crypto)
lsblk -f -b $HD > ${HD_NAME}.lsblk # save human readable partition tree for each hd
for PART in $(sfdisk -l ${HD} 2> /dev/null | grep "^/dev/" | awk '{ print $1 }'); do
# save blkid for each partition
PART_NAME=`basename $PART`
blkid -o value $PART | grep '^[0-9a-z]\{8\}-[0-9a-z]\{4\}-[0-9a-z]\{4\}-[0-9a-z]\{4\}-[0-9a-z]\{12\}' > "${MBR_BACKUP}/${PART_NAME}.blkid"
done
done
DUP_EXClUDE=""
for VAL in "${EXCLUDE[@]}"; do
DUP_EXCLUDE="$DUP_EXCLUDE --exclude $VAL"
done
echo "Running duplicity"
duplicity --progress --verbosity info $DUP_EXCLUDE $BACKUP "b2://${ACCOUNT_ID}:${APP_KEY}@${BUCKET_NAME}/"
Set permissions:
sudo chown root.root /usr/local/bin/backup
sudo chmod 700 /usr/local/bin/backup
Edit the file setting at least the first three variables. To get the account ID and application key you need to go to the Buckets page on backblaze and click the "Show Account ID and Application Key" link.
Note that this script creates the directory /opt/mbr_backup where it stores a copy of your master boot record, and partition information from each harddrive you specify in the the HARDDRIVES variable.
Unfortunately duplicity's --progress feature unfortunately doesn't work with backblaze yet so you don't get any progress information until it completes.
The first time you run this script it will take a long time since it's backing up your entire system. Every subsequent run will just create an incremental backup.
If you have any running databases then you should modify the script to save dumps of all databases to your hd before running the backup since you might otherwise end up with a backup of your database in an inconsistant state. You could also just shut down you databases before running the the script.
Use a bootable usb with e.g. lubuntu.
ToDo how to restore single files
To restore your MBR and partition table from the backup files, for each harddrive do something like
sudo dd if=sda.mbr of=/dev/sda
sudo sfdisk /dev/sda < sda.sfdisk
Now you'll have to format each partition using your preferred filesystem. If you have encrypted partitions and/or use lvm then you'll have to manually re-establish that setup manually. If you don't remember how it was set up then the *.lsblk files should be helpful.
ToDo how to restore the rest of the filesystem.
Since your new partitions will have different block IDs you'll have to restore the old block IDs for each partition. Assuming the partitions are ext2, ext3 or ext4 partitions, for each partition do something like:
sudo tune2fs /dev/sda1 -U `cat sda1.blkid`
Now your system should be restored and ready to boot.
It would be great to have a script to automatically run this backup script, but it should probably have the following features:
- some algorithm to decide when enough files have changed to trigger a new backup
- bandwidth and cpu limiting so it runs without impacting work
- whitelist/blacklist of (wifi) networks where backups are allowed/disallowed
The last point is important since you might not want to have automatic backups when I'm tethering.
this is awesome! thanks for writing it :D
one correction: script should be saved to /usr/local/bin/backup (edit one line above script and also in the lines for setting permissions)
<3!