Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Btrfs guide to set up an LUKS-encrypted btrfs raid volume with included maintenance & recovery guide

Encrypted Btrfs storage setup and maintenance guide

Initial setup with LUKS/dm-crypt

This exemplary initial setup uses two devices /dev/sdb and /dev/sdc but can be applied to any amount of devices by following the steps with additional devices.

Create keyfile:

dd bs=64 count=1 if=/dev/urandom of=/etc/cryptkey iflag=fullblock
chmod 600 /etc/cryptkey

Encrypt devices:

cryptsetup -v -c aes-xts-plain64 -h sha512 -s 512 luksFormat /dev/sdb /etc/cryptkey
cryptsetup -v -c aes-xts-plain64 -h sha512 -s 512 luksFormat /dev/sdc /etc/cryptkey

Backup LUKS header:

cryptsetup luksHeaderBackup --header-backup-file ~/sdb.header.bak /dev/sdb
cryptsetup luksHeaderBackup --header-backup-file ~/sdc.header.bak /dev/sdc

Automatically unlock LUKS devices on boot by editing /etc/crypttab:

data1 UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /etc/cryptkey luks,noearly #,discard (for SSDs)
data2 UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /etc/cryptkey luks,noearly #,discard (for SSDs)
# Use 'blkid /dev/sdb' to get the UUID

Unlock encrypted devices now to create the filesystem in next step:

cryptsetup open --key-file=/etc/cryptkey --type luks /dev/sdb data1
cryptsetup open --key-file=/etc/cryptkey --type luks /dev/sdc data2

Create filesystem:

mkfs.btrfs -m raid1 -d raid1 /dev/mapper/data1 /dev/mapper/data2

Mount filesystem:

mount -t btrfs -o defaults,noatime,compress=zstd /dev/mapper/data1 /mnt/data

Automatically mount btrfs filesystem on boot by editing /etc/fstab:

/dev/mapper/data1 /mnt/data btrfs defaults,noatime,compress=zstd 0 2
# Add option 'autodefrag' to allow automatic defragmentation: useful for files with lot of random writes like databases or virtual machine images

Maintenance

In a btrfs raid setup it is necessary to frequently run a btrfs scrub to check for corrupted blocks/flipped bits and repair them using a healthy copy from one of the mirror disks.

In the example below a systemd-timer is used to run an automatic btrfs scrub job each month.

/etc/systemd/system/btrfs-scrub.timer:

[Unit]
Description=Monthly scrub btrfs filesystem, verify block checksums
Documentation=man:btrfs-scrub

[Timer]
# first saturday each month
OnCalendar=Sat *-*-1..7 3:00:00
RandomizedDelaySec=10min

[Install]
WantedBy=timers.target

/etc/systemd/system/btrfs-scrub.service:

[Unit]
Description=Scrub btrfs filesystem, verify block checksums
Documentation=man:btrfs-scrub

[Service]
Type=simple
ExecStart=/bin/btrfs scrub start -Bd /mnt/data
KillSignal=SIGINT
IOSchedulingClass=idle
CPUSchedulingPolicy=idle

Recovery from device failure

Example with one failed device:

  • /dev/mapper/data1 working device
  • /dev/mapper/data2 failed device
  • /dev/mapper/data3 new device
  • /mnt/data mountpoint

In case of failing/failed device, mount in degraded mode with the working device:

mount -t btrfs -o defaults,noatime,compress=zstd,degraded /dev/mapper/data1 /mnt/data

NOTE: Encrypt the new device before using it in the btrfs raid by following the steps above. Add new device to btrfs raid:

btrfs device add /dev/mapper/data3 /mnt/data

Remove the missing device (NOTE: missing is a special device name and not a placeholder):

btrfs device delete missing /mnt/data

Re-balance data of btrfs raid:

btrfs balance start /mnt/data

The fstab entry is left unmodified with one of the working devices:

/dev/mapper/data1 /mnt/data btrfs defaults,noatime,compress=zstd 0 2
@JakeTrock
Copy link

JakeTrock commented Dec 19, 2021

this is excellent! Kudos!

@learnitall
Copy link

learnitall commented Jan 5, 2022

Super helpful, thank you!

@george-andrei
Copy link

george-andrei commented Jan 14, 2022

Amazing, thanks a lot!

@iio7
Copy link

iio7 commented Oct 23, 2022

Where does GRUB fit into this? I see no grub-install command.

@MaxXor
Copy link
Author

MaxXor commented Oct 24, 2022

Where does GRUB fit into this? I see no grub-install command.

Hello @iio7, this is not guide which uses Btrfs on the system partition, so there's no need for a bootloader like grub. Btrfs here is used as data partition on separate disks, e.g. a NAS. Furthermore I wouldn't recommend using Btrfs configured as RAID1 as system partition. When one disk fails, your system won't boot anymore, unlike ZFS which automatically boots in degraded mode, but Btrfs unfortunately does not.

@iio7
Copy link

iio7 commented Oct 30, 2022

Hello @iio7, this is not guide which uses Btrfs on the system partition, so there's no need for a bootloader like grub. Btrfs here is used as data partition on separate disks, e.g. a NAS. Furthermore I wouldn't recommend using Btrfs configured as RAID1 as system partition. When one disk fails, your system won't boot anymore, unlike ZFS which automatically boots in degraded mode, but Btrfs unfortunately does not.

Thank you very much for your reply, I didn't know about Btrfs not booting in a degraded mode!

@MaxXor
Copy link
Author

MaxXor commented Oct 30, 2022

Thank you very much for your reply, I didn't know about Btrfs not booting in a degraded mode!

Well, it's not automatic or default for any distro I know of. So if you go into the recovery process and replace one of the Raid boot disks, the system won't boot anymore. Here is the description of the bug and a possible workaround: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1229456

@iio7
Copy link

iio7 commented Oct 30, 2022

Well, it's not automatic or default for any distro I know of. So if you go into the recovery process and replace one of the Raid boot disks, the system won't boot anymore. Here is the description of the bug and a possible workaround: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1229456

I found that kernel argument after looking into the issue. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment