dm-crypt + dm-integrity + dm-raid = awesome!

cryptsetup-with-luks2-and-integrity-demo.sh

#!/usr/bin/env bash
# 
#    Author: Markus (MawKKe) ekkwam@gmail.com
#    Date: 2018-03-19
#
#
# What?
#
#    Linux dm-crypt + dm-integrity + dm-raid (RAID1) 
#
#    = Secure, redundant array with data integrity protection
#
# Why? 
#
#   You see, RAID1 is dead simple tool for disk redundancy, 
#   but it does NOT protect you from bit rot. There is no way
#   for RAID1 to distinguish which drive has the correct data if rot occurs.
#   This is a silent killer.
#
#   But with dm-integrity, you can now have error detection
#   at the block level. But it alone does not provide error correction
#   and is pretty useless with just one disk (disks fail, shit happens).
#
#   But if you use dm-integrity *below* RAID1, now you have disk redundancy,
#   AND error checking AND error correction. Invalid data received from
#   a drive will cause a checksum error which the RAID array notices and 
#   replaces with correct data.
#
#   If you throw encryption into the mix, you'll have secure,
#   redundant array. Oh, and the data integrity can be protected with 
#   authenticated encryption, so no-one can tamper your data maliciously. 
#   
#   How cool is that?
#
#   Also: If you use RAID1 arrays as LVM physical volumes, the overall 
#   architecture is quite similar to ZFS! All with native Linux tools, 
#   and no hacky solaris compatibility layers or licencing issues!
#
#   (I guess you can use whatever RAID level you want, but RAID1 is the 
#    simplest and fastest to set up)
# 
#
#   Let's try it out!
#
#   ---
#   NOTE: The dm-integrity target is available since Linux kernel version 4.12.
#   NOTE: This example requires LUKS2 which is only recently released (2018-03)
#   NOTE: The authenticated encryption is still experimental (2018-03)
#   ---

set -eux

# 1) Make dummy disks

cd /tmp

truncate -s 500M disk1.img
truncate -s 500M disk2.img

# Format the disk with luksFormat:

dd if=/dev/urandom of=key.bin bs=512 count=1

cryptsetup luksFormat -q --type luks2 --integrity hmac-sha256 disk1.img key.bin
cryptsetup luksFormat -q --type luks2 --integrity hmac-sha256 disk2.img key.bin

# The luksFormat's might take a while since the --integrity causes the disks to be wiped.
# dm-integrity is usually configured with 'integritysetup' (see below), but as 
# it happens, cryptsetup can do all the integrity configuration automatically if
# the --integrity flag is specified.

# Open/attach the encrypted disks

cryptsetup luksOpen disk1.img disk1luks --key-file key.bin
cryptsetup luksOpen disk2.img disk2luks --key-file key.bin


# Create raid1:

mdadm \
  --create \
  --verbose --level 1 \
  --metadata=1.2 \
  --raid-devices=2 \
  /dev/md/mdtest \
  /dev/mapper/disk1luks \
  /dev/mapper/disk2luks


# Create a filesystem, add to LVM volume group, etc...

mkfs.ext4 /dev/md/mdtest

# Cool! Now you can 'scrub' the raid setup, which verifies
# the contents of each drive. Ordinarily detecting an error would
# be problematic, but since we are now using dm-integrity, the raid1
# *knows* which one has the correct data, and is able to fix it automatically. 
#
# To scrub the array:
#
#    $ echo check > /sys/block/md127/md/sync_action
#
# ... wait a while
#
#    $ dmesg | tail -n 30
#
# You should see 
#
#    [957578.661711] md: data-check of RAID array md127
#    [957586.932826] md: md127: data-check done.
#
#
# Let's simulate disk corruption:
# 
#    $ dd if=/dev/urandom of=disk2.img seek=30000 count=30 bs=1k conv=notrunc
#
# (this writes 30kB of random data into disk2.img)
#
#
# Run scrub again:
#
#    $ echo check > /sys/block/md127/md/sync_action
#
# ... wait a while
#
#    $ dmesg | tail -n 30
#
# Now you should see 
#	...
#    [959146.618086] md: data-check of RAID array md127
#    [959146.962543] device-mapper: crypt: INTEGRITY AEAD ERROR, sector 39784
#    [959146.963086] device-mapper: crypt: INTEGRITY AEAD ERROR, sector 39840
#    [959154.932650] md: md127: data-check done.
#
# But now if you run scrub yet again:
#    ...
#    [959212.329473] md: data-check of RAID array md127
#    [959220.566150] md: md127: data-check done.
#
# And since we didn't get any errors a second time, we can deduce that the invalid 
# data was repaired automatically.
#
# Great! We are done.
#
# --------
#
# If you don't need encryption, then you can use 'integritysetup' instead of cryptsetup. 
# It works in similar fashion: 
#
#    $ integritysetup format --integrity sha256 disk1.img
#    $ integritysetup format --integrity sha256 disk2.img
#    $ integritysetup open --integrity sha256 disk1.img disk1int
#    $ integritysetup open --integrity sha256 disk2.img disk2int
#    $ mdadm --create ...
#
# ...and so on. Though now you can detect and repair disk errors but have no protection
# against malicious cold-storage attacks. Data is also readable by anybody.
#
#	2018-03 NOTE: 
#
#       if you override the default --integrity value (whatever it is) during formatting, 
#       then you must specify it again when opening, like in the example above. For some 
#       reason the algorithm is not autodetected. I guess there is no header written onto 
#       disk like is with LUKS ?
#
# ----------
#
#  Read more:
#   https://fosdem.org/2018/schedule/event/cryptsetup/
#   https://gitlab.com/cryptsetup/cryptsetup/wikis/DMCrypt
#   https://gitlab.com/cryptsetup/cryptsetup/wikis/DMIntegrity
#   https://mirrors.edge.kernel.org/pub/linux/utils/cryptsetup/v2.0/v2.0.0-rc0-ReleaseNotes

@tomato42 -- it looks like my case just required a few scrubs to repair all of the corrupt data.

@Salamandar -- there are some resilience considerations with whether you run dm-crypt above or below the raid layer. if you run above the raid layer, you get automatic replication of your crypt headers, whereas if the headers are corrupted below the raid layer, you need to manually restore them (or re-image the entire disk).

@ggeorgovassilis -- i've been experimenting with a similar stack (dm-integrity as a separate layer on the bottom). what made you choose dm-raid over md for your use case?

@khimaros I wasn't aware that those are different things - I learned something. What I meant, I realise now, was md.

note: if you are using dm-crypt + integrity in Debian for your root filesystem, you may need to ensure dm_integrity module is in your initrd. one way to do this is echo dm_integrity >> /etc/initramfs-tools/modules && update-initramfs -u -k all

In regards to dm-integrity, mdadm and LVM (keeping LUKS/crypto out), any recommendations in regards to enable and configure this using the LVM tool or doing it Linux-native, as you all do above? I guess using LVM tool you're sure it's done "correctly". Going Linux-native I have to read this thread 4 times thoroughly :-P
And does the LVM-tools add further performance/stability enhancement compared going Linux-native, is it only for comfortability.

a discovery while working on https://github.com/khimaros/raid-explorations; separate dm-integrity layer below the md array (dm-integrity > md > dm-crypt > lvm > ext4) reduces sysbench (rndwr) performance by about 20% in a virtual machine compared to a combined dm-crypt/integrity (dm-crypt + integrity > md > lvm > ext4).

@MrM40 for dm-integrity to have any affect in protecting against silent data corruption it must be used below the raid level.
With md-raid that means the individual devices making up the md device must be dm-integrity backed; for lvm raid that means individual PVs must be dm-integrity backed.
While technically you can put LVM on LVM, I don't think that it will be any easier or more straightforward than doing dm-integrity > md > lvm.
It's a complex setup, no matter how you slice or dice it.

So if you have 5 disks in RAID5/6 you need to have 5 independent dm-integrity setups with 5 independent checksum "files" and 5 independent concurrent checksum writes?, I see the last could be problematic performance wise.
Since RHEL ditched brtfs and not too happy about ZFS and therefore seems to bet 100% on LVM (Stratis) for their storage solution one could hope it was optimized (if it is a bottleneck at all).
E.g. lvcreate --type <raid-level> --raidintegrity y -L <usable-size> -n <logical-volume> <volume-group>
So what I mean is, you can either do the setup yourself manually or do it all by lvcreate. Will LVM/lvcreate add any "optimization" or is it technical 100% the same as doing it yourself? Going lvcreate will probably give your less control, but also less possibility to mess something up/make configuration mistake

So if you have 5 disks in RAID5/6 you need to have 5 independent dm-integrity setups with 5 independent checksum "files" and 5 independent concurrent checksum writes?, I see the last could be problematic performance wise.

correct, and yes it has significant impact on performance, especially if the dm-integrity volume uses journaling

Since RHEL ditched brtfs and not too happy about ZFS and therefore seems to bet 100% on LVM (Stratis) for their storage solution one could hope it was optimized (if it is a bottleneck at all). E.g. lvcreate --type <raid-level> --raidintegrity y -L <usable-size> -n <logical-volume> <volume-group> So what I mean is, you can either do the setup yourself manually or do it all by lvcreate. Will LVM/lvcreate add any "optimization" or is it technical 100% the same as doing it yourself? Going lvcreate will probably give your less control, but also less possibility to mess something up/make configuration mistake

I don't think Stratis supports integrity just yet. I haven't inspected what raid + integrity on lvmcreate level do, or if that combination works at all.

The lvcreate cmd line was taking from RHEL's documentation so it should work ;-)
It's my impression Stratis is roughly just a stacking of current storage technologies wrapped in a new management-package, nothing new under the hood. And LVM is of course a key component and since it support mdadm and md-integrity Stratis do too. And so says the spec for Stratis 2.0.
It's been a good question for a long time if you should use LVM or native mdadm to create your md arrays (if your run LVM anyway of course). Not been able to find a good answer if LVM actually add any goodies other then a management layer.

@MawKKe about the note on the default integrity algorithm for cryptsetup:
https://gitlab.com/cryptsetup/cryptsetup/-/issues/754

Are there any known disadvantages with encryption for ZFS shipped with FreeBSD?
Obviously encryption happens in a separate layer there.
Main thread: https://forums.raspberrypi.com/viewtopic.php?p=2089261#p2089261

Currently (as of July 2023) there are some unresolved bugs with ZFS's native encryption and edge cases involving send/recv, that can potentially cause data corruption on both the home pool and a snapshot receiving pool.

See the following for references:

• https://discourse.practicalzfs.com/t/is-native-encryption-ready-for-production-use/532/3
• https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6swwBZXgXwdCPKgp4SbPZwTexCg/htmlview
• https://github.com/openzfs/zfs/issues?q=is%3Aopen+is%3Aissue+label%3A%22Component%3A+Encryption%22

While these seem to be reported in regards to linux systems, that's likely simply because that's where the major users are, I personally wouldn't expect FreeBSD to be free from these issues.

Basically if you don't send/recv an encrypted dataset and instead use something like rsync, then you should be fine.

I have read through the thread. Great discussion, and a lot of useful info!

@khimaros , @tomato42

Does anyone know whether disabling journaling for dm-integrity (--integrity-no-journal) is safe with a dm-integrity > md-raid > filesystem kind of setup?

I’m trying to research this, but there is hardly any info. My logic is this: If journaling is off, and there is a corruption in the dm-integrity layer, e.g. due to a power-cut, then this would be presented to the md-raid layer as a normal unrecoverable read error. Then, md-raid would correct the error the same way as in any other case. The fact that the error was a result of disabled journaling is irrelevant.

The only problem I can see with this setup is that if the integrity error is not discovered in time, and only found out when the md-raid layer doesn’t have a copy (Raid-1) or a checksum (Raid-5 or 6) of the lost data. E.g. there is a power cut -> faulty block is written in the dm-integrity layer -> sometime later a disk dies -> we try to repair the array, but the faulty block is causing a problem.

But thinking it further, if we do a Raid scrub every time after a power-cut, then is it safe to disable journaling? Am I right with this logic?

Apart from a power-cut or yanking a disk out mid-operation, are there any situations where dm-integrity would write faulty blocks silently, and where this wouldn't happen with journaling enabled?

Given it's likely that both copies will be written at once, there's high chance that both copies will be mismatched with the checksums. So, no, it's definitely not safe in general.

This is a nice example to get ZFS-like error detection with regular filesystems like EXT4 or XFS, but what about alignment issues and write amplification? Do we have to correct mdadm or the filesystem chunk/stride calculation for alignment, or is it now taken care of by default?

The linked article https://securitypitfalls.wordpress.com/2020/09/27/making-raid-work-dm-integrity-with-md-raid/ suggest:

• avoiding read-modify-write by passing --chunk 2048 to mdadm
• having a /sys/block/md0/md/stripe_cache_size of 4096
• using stripe unit and stripe width when doing mkfs to reflect the hardware realities as the mdadm device wrongly looks like 4kn

Alignment issues are no worse than with any RAID: for a file system it looks like regular MD RAID.
There is journaling in integrity level so write amplification is on the horrible side. May be better if you have a device with configurable sector size (like 520 instead of 512 bytes).

If this is for the root drive, one potential advantage of doing LUKS over mdadm RAID1 instead of the reverse is that LUKS over a degraded RAID will boot normally since this patch, whereas a degraded RAID over LUKS array may fail the boot process and drop into an initramfs shell due to a missing encrypted root disk. That may require typing exit to continue, although this may not be a big deal to some.