title | slug | tags | author | created | |
---|---|---|---|---|---|
Home data storage with encrypted cloud backup |
data-storage-with-encrypted-cloud-backup |
|
Daniel |
2021-01-13 06:28:14 -0800 |
I thought it would be a good idea to consolidate all my data to a central storage. Before I set this up, I had data all over the place: photos and documents on iCloud and OneDrive, old programming projects and very old documents scattered on random hard drives, games and music on my PC.
I had a few requirements:
- Central location for all my devices: I want to mount a share and copy files
- Backups: the central location should have versioned backups
- No proprietary solutions: I want to be in control of my data
- No vendor lock-in: There should be an option to switch cloud storage providers
- Encryption: The cloud backups need to be encrypted
I decided to use an Odroid HC1 with an HDD. The Odroid will advertise an NFS share and periodically back up its contents to a cloud storage. I chose to go with Backblaze, which offers an S3-compatible object store at $0.005/GB. First 10 GB are free.
I downloaded Armbian for the HC1 here: https://www.armbian.com/odroid-hc1/ and wrote the image to an SD card using Etcher. I'm using OSX on my machine. If you're on Windows, Rufus is a good choice.
Once the Odroid boots up, find its MAC address in your router config and configure a static IP lease. Mine has 192.168.0.11
. Then SSH in:
$ ssh root@192.168.0.11
It will ask you to change the root password and create a new user. Once done, add your user to the sudo
group and update:
$ usermod -aG sudo <username>
$ apt update
$ apt upgrade
You should probably su <username>
, but then you will have to sudo
everything. ¯\_(ツ)_/¯
Now is a good time to mount the HDD in /mnt/hdd
and set it to mount on boot.
# find out where your disk is
$ fdisk -l
# create a fresh partition
$ parted /dev/sda
> mklabel gpt # beware: this will erase all data on the drive
> mkpart primary 0TB 1TB # or however large your disk is
> quit
# create an Ext4 filesystem
$ mkfs.ext4 /dev/sda1
# create the mount point
$ mkdir /mnt/hdd
# mount it to verify it works
$ mount /dev/sda1 /mnt/hdd
# find your disk's UUID
$ blkid -o list
# add this line to /etc/fstab to mount on boot
$ vi /etc/fstab
# ...snip...
UUID=<uuid> /mnt/hdd ext4 defaults 0 2
# after reboot, disk should be mounted in /mnt/hdd
$ reboot
Install the NFS server:
$ apt install autofs nfs-kernel-server nfs-common --install-recommends -f -y
$ reboot
Configure the share:
# the share will live in a subdirectory. That way you can add data to the disk that will not be shared
$ mkdir /mnt/hdd/data
# back up default NFS config
$ cp -a /etc/exports /etc/exports.backup
# configure the share
$ vi /etc/exports
# data_directory host(attributes)
/mnt/hdd/data *(rw,async,insecure,all_squash,no_subtree_check,nohide,anonuid=1000,anongid=1000)
Attributes:
rw
- read/writeasync
- allows the server to buffer writesinsecure
- allows clients (eg. Mac OS X) to use non-reserved ports to connect to the shareno_subtree_check
- improves speed and reliability by eliminating permission checks on parent directoriesnohide
- not a hidden shareall_squash
- treats all users as anonymous and assigns them uid=anonuid & gid=anongidanonuid=1000
- assigns anonymous users uid 1000anongid=1000
- assigns anonymous users gid 1000
Change the uid
and gid
to match the owner of the /mnt/hdd/data
directory. That way every client will have full access to the share. Use this only on your private network!
Restart to apply changes:
# if you make additional changes, run this first
$ exportfs -ra
# restart the server
$ service nfs-kernel-server restart
Mount the NFS share on your computer to verify it works.
# find out if the share is advertised
$ showmount -e 192.168.0.11
# mount it to /Users/<username>/odroid
$ mount -t nfs -o rsize=65536,wsize=65536,intr,hard,tcp,locallocks,rdirplus,readahead=128 192.168.0.11:/mnt/hdd/data /Users/<username>/odroid
Check if you see it in Finder and copy some files. If it works, unmount and set up an auto mount:
# unmount
$ umount /Users/<username>/odroid
# add auto_nfs to auto_master
$ sudo vi /etc/auto_master
/- auto_nfs -nobrowse,nosuid
# add the share to auto_nfs
$ sudo vi /etc/auto_nfs
/System/Volumes/Data/Users/<username>/odroid -fstype=nfs,noowners,nolockd,locallocks,rdirplus,hard,intr,rw,tcp,nfc,rsize=65536,wsize=65536 nfs://192.168.0.11:/mnt/hdd/data
# set proper permissions: a+rwx,u-x,g-wx,o-wx
$ sudo chmod 644 /etc/auto_nfs
# mount everything in auto_master
$ sudo automount -cv
You should find the volume in /Users/<username>/odroid
. Verify it still works.
Register for a Backblaze account at https://www.backblaze.com/b2/cloud-storage.html. (You can use any other cloud storage provider supported by rclone
, which is most of them.) Create a private bucket with versioning on (it's the default for B2).
Side note: if you create a public bucket, you can proxy traffic to it through Cloudflare to create your own private CDN.
Next, create an app key with permissions to your bucket and save the app key ID and the key to configure rclone
.
Rclone is an amazing CLI utility that supports a lot of cloud storage providers. It's basically rsync for the cloud. Let's install rclone and add B2 as a remote:
# install rclone
$ apt install rclone
# add a remote
$ rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> b2
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
...snip...
4 / Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio)
\ "s3"
5 / Backblaze B2
\ "b2"
...snip...
Storage> b2
** See help for b2 backend at: https://rclone.org/b2/ **
Account ID or Application Key ID
Enter a string value. Press Enter for the default ("").
account> 0000000000000000000
Application Key
Enter a string value. Press Enter for the default ("").
key> *876FD87SFGadsfSD08F6fD087Fadf07SF608D6fdsfzgjhsdfgd76*
Permanently delete files on remote removal, otherwise hide files.
Enter a boolean value (true or false). Press Enter for the default ("false").
hard_delete> false
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
--------------------
[b2]
account = 0000000000000000000
key = 876FD87SFGadsfSD08F6fD087Fadf07SF608D6fdsfzgjhsdfgd76
hard_delete = false
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
To test that it worked, create a file in /mnt/hdd/data
and run rclone:
# create a file
$ touch /mnt/hdd/data/test-file
# sync to B2
$ rclone sync /mnt/hdd/data b2:<bucket-name>
See if the file was uploaded in B2 management console.
To enable encryption, rclone has a crypt
remote type that encrypts files uploaded to it. Most of the configuration is straightforward and uses the defaults, except for filename and directory name encryption, which does not work with B2 versioning system, so we will disable it. If you insist on encrypting file and directory names, you could disable versioning and use the --backup-dir
option instead.
Let's add another remote:
$ rclone config
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> b2-crypt-backup
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
...snip...
9 / Encrypt/Decrypt a remote
\ "crypt"
...snip...
Storage> crypt
** See help for crypt backend at: https://rclone.org/crypt/ **
Remote to encrypt/decrypt.
Normally should contain a ':' and a path, eg "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
Enter a string value. Press Enter for the default ("").
remote> b2:<bucket-name>
How to encrypt the filenames.
Enter a string value. Press Enter for the default ("standard").
Choose a number from below, or type in your own value
1 / Don't encrypt the file names. Adds a ".bin" extension only.
\ "off"
2 / Encrypt the filenames see the docs for the details.
\ "standard"
3 / Very simple filename obfuscation.
\ "obfuscate"
filename_encryption> off
Option to either encrypt directory names or leave them intact.
Enter a boolean value (true or false). Press Enter for the default ("true").
Choose a number from below, or type in your own value
1 / Encrypt directory names.
\ "true"
2 / Don't encrypt directory names, leave them intact.
\ "false"
directory_name_encryption> false
Password or pass phrase for encryption.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
y/g/n> g
Password strength in bits.
64 is just about memorable
128 is secure
1024 is the maximum
Bits> 1024
Your password is: zFpNRENxx75Eks3GftRzhDdRYFO8U8oAWTIwezf5Qnj5HPwAxypXDZb4LGy1wJSUGPo4c9aj4GLhQ87Gcw3ar6ve7TET77NORkAyhKwkR6BwPk4jVGRi-YMkjIf6oqDPdRRB3RQUhtFKp5VtBQl-txe-luOQlHR2-zQ_YiamuAg
Use this password? Please note that an obscured version of this
password (and not the password itself) will be stored under your
configuration file, so keep this generated password in a safe place.
y) Yes
n) No
y/n> y
Password or pass phrase for salt. Optional but recommended.
Should be different to the previous password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
y/g/n> g
Password strength in bits.
64 is just about memorable
128 is secure
1024 is the maximum
Bits> 1024
Your password is: CadSQrSUqAAGOfNzr6-__Rsdv8Bauj2lG8Ee2Q7oFZh7MByhnRFyUGLoX5yXX1dKRKKxMCpoG5-OQlKgascKd-aU9p9sGAeVpzs301zzwrRI8ngE8XtPE8Qzx_HZvZAMqVSy1gl-zGhqqnsdCLuty5VbBtk2PJwQ9Hlq1XygxCA
Use this password? Please note that an obscured version of this
password (and not the password itself) will be stored under your
configuration file, so keep this generated password in a safe place.
y) Yes
n) No
y/n> y
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
--------------------
[b2-crypt-backup]
remote = b2:<bucket-name>
filename_encryption = off
directory_name_encryption = false
password = *** ENCRYPTED ***
password2 = *** ENCRYPTED ***
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Save the keys to your password manager. You will need them to restore your files.
Check that the encrypted remote works the same way we checked the B2 remote:
# create a file
$ echo "unencrypted text" > /mnt/hdd/data/test-plain-file
# sync to B2
$ rclone sync /mnt/hdd/data b2-crypt-backup:/
The file should show up in the management console with a .bin
extension and encrypted contents.
Test that you're able to restore the data. This step is crucial, without it you're uploading random bytes to the cloud. On the Odroid:
# copy your config
$ cat $(rclone config file | tail -n1)
On another machine:
# paste your config
$ rclone config edit
# restore the files, decrypting them in the process
$ rclone sync b2-crypt-backup:/ /Users/<username>/test
Confirm that the file is restored and its contents have been decrypted.
I used cron
to automate running rclone
every 30 minutes. Create a bash script in /home/<username>/rclone-cron.sh
:
#!/bin/bash
if pidof -o %PPID -x “rclone-cron.sh”; then
exit 1
fi
rclone sync --fast-list --stats-log-level NOTICE --stats-one-line --log-file=/var/log/rclone.log /mnt/hdd/data b2-crypt-backup:/
exit
If your storage provider bills you for every transaction (B2 does), I recommend to use --fast-list
, which will minimize the amount of transactions at the expense of higher memory usage and result in a lower monthly payment.
I enabled logging to file for easier monitoring with --stats-log-level NOTICE --stats-one-line --log-file=/var/log/rclone.log
.
The script runs rclone sync
, but only if another instance of the script is not already running.
# set the executable bit
$ chmod a+x rclone-cron.sh
# add this line to crontab
$ crontab -e
0,30 * * * * /home/<username>/rclone-cron.sh >/dev/null 2>&1
If you'd like to change the schedule, crontab.guru is handy when editing crontab.
That's it! Anything you store on the Odroid will be backed up to Backblaze every 30 minutes. Previous versions of edited or deleted files will remain in Backblaze. You can recover them using rclone --b2-versions
. You'll find more about that in the docs.