Btrfs guide to set up an LUKS-encrypted btrfs raid volume with included maintenance & recovery guide

btrfs-guide.md

Encrypted Btrfs storage setup and maintenance guide

Initial setup with LUKS/dm-crypt

This exemplary initial setup uses two devices /dev/sdb and /dev/sdc but can be applied to any amount of devices by following the steps with additional devices.

Create keyfile:

dd bs=64 count=1 if=/dev/urandom of=/etc/cryptkey iflag=fullblock
chmod 600 /etc/cryptkey

Encrypt devices:

cryptsetup -v -c aes-xts-plain64 -h sha512 -s 512 luksFormat /dev/sdb /etc/cryptkey
cryptsetup -v -c aes-xts-plain64 -h sha512 -s 512 luksFormat /dev/sdc /etc/cryptkey

Backup LUKS header:

cryptsetup luksHeaderBackup --header-backup-file ~/sdb.header.bak /dev/sdb
cryptsetup luksHeaderBackup --header-backup-file ~/sdc.header.bak /dev/sdc

Automatically unlock LUKS devices on boot by editing /etc/crypttab:

data1 UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /etc/cryptkey luks,noearly #,discard (for SSDs)
data2 UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /etc/cryptkey luks,noearly #,discard (for SSDs)
# Use 'blkid /dev/sdb' to get the UUID

Unlock encrypted devices now to create the filesystem in next step:

cryptsetup open --key-file=/etc/cryptkey --type luks /dev/sdb data1
cryptsetup open --key-file=/etc/cryptkey --type luks /dev/sdc data2

Create filesystem:

mkfs.btrfs -m raid1 -d raid1 /dev/mapper/data1 /dev/mapper/data2

Mount filesystem:

mount -t btrfs -o defaults,noatime,compress=zstd /dev/mapper/data1 /mnt/data

Automatically mount btrfs filesystem on boot by editing /etc/fstab:

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /mnt/data btrfs defaults,noatime,compress=zstd 0 2
# Add option 'autodefrag' to allow automatic defragmentation: useful for files with lot of random writes like databases or virtual machine images
# Use 'blkid /dev/mapper/data1' to get the UUID of the Btrfs partition, it is common for all raid devices

Maintenance

In a btrfs raid setup it is necessary to frequently run a btrfs scrub to check for corrupted blocks/flipped bits and repair them using a healthy copy from one of the mirror disks.

In the example below a systemd-timer is used to run an automatic btrfs scrub job each month.

/etc/systemd/system/btrfs-scrub.timer:

[Unit]
Description=Monthly scrub btrfs filesystem, verify block checksums
Documentation=man:btrfs-scrub

[Timer]
# first saturday each month
OnCalendar=Sat *-*-1..7 3:00:00
RandomizedDelaySec=10min

[Install]
WantedBy=timers.target

/etc/systemd/system/btrfs-scrub.service:

[Unit]
Description=Scrub btrfs filesystem, verify block checksums
Documentation=man:btrfs-scrub

[Service]
Type=simple
ExecStart=/bin/btrfs scrub start -Bd /mnt/data
KillSignal=SIGINT
IOSchedulingClass=idle
CPUSchedulingPolicy=idle

Recovery from device failure

Example with one failed device:

• /dev/mapper/data1 working device
• /dev/mapper/data2 failed device
• /dev/mapper/data3 new device
• /mnt/data mountpoint

In case of failing/failed device, mount in degraded mode with one of the working devices:

mount -t btrfs -o defaults,noatime,compress=zstd,degraded /dev/mapper/data1 /mnt/data

Find the the device ID of the missing disk by executing btrfs device usage /mnt/data:

# btrfs device usage /mnt/data

/dev/mapper/data1, ID: 1
   Device size:             7.28TiB
   Device slack:              0.00B
   Data,RAID1:              5.46TiB
   Metadata,RAID1:          7.00GiB
   System,RAID1:           32.00MiB
   Unallocated:             1.81TiB

missing, ID: 2
   Device size:             7.28TiB
   Device slack:              0.00B
   Data,RAID1:              5.46TiB
   Metadata,RAID1:          7.00GiB
   System,RAID1:           32.00MiB
   Unallocated:             1.81TiB

NOTE: Encrypt the new device before using it in the btrfs raid by following the steps above.

Start the replace operation in background by adding the new device to the btrfs raid using the device ID of the missing disk:

btrfs replace start 2 /dev/mapper/data3 /mnt/data

(Optional) Check the replace progress:

btrfs replace status /mnt/data

Once the replace operation has finished, the fstab entry is left unmodified:

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /mnt/data btrfs defaults,noatime,compress=zstd 0 2
# Use 'blkid /dev/mapper/data1' to get the UUID of the Btrfs partition, it is common for all raid devices

If you want to retain the data - why not just plug the drives in and do the "automatically unlock drives" and "automatically mount drives" steps? Since you will be retaining your existing encryption and btrfs filesystem - there's no need to alter it.

Just do the cryptsetup open and mount to test that everything works on the new system, then do only the "automatically on boot" steps

I was having issues because I tried mounting only one at a time rather than both as intended. Problem solved.

Great cheat sheet! Thanks!

Does anyone have information on how to deal with defragmentation? The author of btrfs has a maintenance script collection which also includes the btrfs defrag script.

Now, because cryptsetup will normally use AES-XTS (which is a block-cipher mode), I would assume that the positions of reads/writes inside the encrypted volume match reads/writes on the physical disk. Therefore, defragmenting the data blocks inside the encrypted volume will also defragment the encrypted blocks on the physical disk, thus improving performance.

Can anyone please confirm or deny this with some evidence or expert knowledge...?

@MaxXor Thank you for this documentation!

I am also considering a Btrfs raid10 setup (consisting of four HDDs) together with LUKS encryption.

Has anyone experience regarding performance? Read/write throughput, latency and IOPS?

Because with raid10 (mirroring and striping) on top of seperate LUKS encrypted block devices, the CPU has to encrypt/decrypt multiple times.

---

Interesting would be benchmarking a Btrfs raid10 setup with and without LUKS encryption: https://openbenchmarking.org/suite/pts/disk

I had some WARNING: Multiple block group profiles detected, see 'man btrfs(5)' warnings after device replacement -- rebalance is required, see https://forums.gentoo.org/viewtopic-t-1140141-start-0-postdays-0-postorder-asc-highlight-btrfs%2Breplace.html

A faulty disk may be recognized by LUKS

# dmsetup table
data3: 0 23437737984 error

But it cannot be removed from device mapper, because it is still in use by BTRFS. So btrfs filesystem show will never show, that a device is missing:

 # btrfs filesystem show /data                                                                                                                                                             
Label: 'data'  uuid: b912b3b0-3436-4960-adf0-cb82e8ff2448                                                                                                                                                         
        Total devices 4 FS bytes used 4.87TiB                                                                                                                                                                     
        devid    1 size 10.91TiB used 2.90TiB path /dev/mapper/data1                                                                                                                                              
        devid    2 size 10.91TiB used 2.91TiB path /dev/mapper/data2                                                                                                                                              
        devid    3 size 10.91TiB used 2.90TiB path /dev/mapper/data3                                                                                                                                              
        devid    4 size 10.91TiB used 2.91TiB path /dev/mapper/data4

So it is a deadlock.

LUKS cannot close the faulty DM:

# cryptsetup close data3 
device-mapper: remove ioctl on data3  failed: Device or resource busy
# dmsetup remove -f   data3
device-mapper: remove ioctl on data3  failed: Device or resource busy
Command failed.

BTRFS is not notified about error in underlying device from DM, which is part of a RAID and still shows device as active, but generating Kernel Logs and btrfs io errors.

How do you deal with a situation where a hard disk can go offline unexpectedly?