Skip to content

Instantly share code, notes, and snippets.

@ixmatus ixmatus/ forked from techhazard/
Created Jul 30, 2017

What would you like to do?
Nixos with ZFS on encrypted LUKS as root filesystem

NixOS with ZFS on LUKS

After some effort (and asking for help on the nix-dev mailing list) I installed ZFS on an encrypted partition. The relevant configuration is below.


I do not have a custom iso yet, so you'll need two USBs. One for the NixOS iso, and one for these files. You'll have to mount the second stick manually.

  1. Boot into the nixos environment and find the uuid or id of the disk you want to install to. Do not use /dev/sda but /dev/disk/by-..., use lsblk and blkid.
  2. export it to the environment as rootdisk:
# whole disk please, no partition
export rootdisk="/dev/disk/by-id/ata-Some-Storage-Device"
  1. use keyfile and/or configure passphrase usage (see sections below)
  2. run it:
bash /path/to/

Use keyfile

It is possible to use a keyfile (e.g. on a usb stick). If you want a keyfile and not have a passphrase for backup, see Configure without passphrase below.

# part of step 3
export keyfile="/dev/disk/by-id/usb-Some-Usb-Stick"
# optional, default is 4096
export keysize="8192"

Configure passphrase

It is possible to pass the passphrase in an environment variable to make the install fully automated. This is generally unwise, but since we are in a temporary live enviorment I consider it safe enough. You can also put it as passphrase="your passphrase here" in on line 16 instead. If you add a keyfile as well, both are added.

# part of step 3
export passphrase="your passphrase here"

Configure without passphrase

If you only want to add a keyfile and not set a passphrase, set use_passphrase to no. This is not recommended.

# part of step 3
export use_passphrase="no"
# see Use keyfile above
export keyfile="/path/to/keyfile"

Misc commands

I always run these command right after booting the install usb.

# I use programmer dvorak instead of qwerty
loadkeys dvorak-programmer

To Do

  • use nixos-rebuild to make an iso containing the files
  • customise the iso with ZFS support and these files
  • find the location of in the built iso.

Resources I used

I used the following resources:

Installing with old script

use this version of the files: old version. All text below is about those versions, not the ones you see here.

The commands in I run manually, (so no sed :-P)

The is used to set up a single-disk ZFS root filesystem inside of an encrypted LUKS container.

The two *.nix files have the minimum config needed for this (compare them with the generated ones in /mnt/etc/nixos/); The UUIDs should be filled-in by nixos-generate-config; the "usb_storage" addition is not needed for everyone, just like the keyfile options; the other important changes are the hostId, which is required by ZFS; and the boot.supportedFilesystems which I'm not even sure of if that's necessary

# you only need to set this to the disk to want to install to
# use keyfile (optional)
# set to "no" to not set a passphrase
# I highly recommed setting a passphrase and storing it in a safe location
# You can set the passphrase here (so you can view it in plaintext,
# but do not forget to remove it.
# TODO: check if this file is on a tempfs just like /etc/nixos/configuration.nix
# Probably no need to change anything below or
# in the other scripts, there be dragons
# exit on error
set -e
# abort if no root disk is set
if [[ "${rootdisk}" != "NONE" ]]; echo "please set rootdisk with: \`rootdisk=/dev/disk/by-id/disk_id_for_root_device $0\`"; exit 1; fi
if [[ "${use_passphrase}" == "no" ]] && [[ "${keyfile}" == "NONE" ]]; echo "please use at least one of the following: keyfile, password."; exit 2; fi
if [[ "${use_passphrase}" == "no" ]]; echo "using a passprase is highly recommended, since keyfiles can get corrupt or lost."; fi
export rootdisk keyfile keysize use_passphrase;
# absolute location for this script (directory the files are in)
export scriptlocation=$(dirname $(readlink -f $0))
nixos-generate-config --root /mnt
if [[ "$keyfile" != "NONE" ]];
# add lukskeyfile.nix
cp "${scriptlocation}/lukskeyfile.nix" /mnt/etc/nixos/
# replace `/path/to/device` with actual keyfile
echo "path to device"
blkdevice="$(basename "$(ls -l /dev/disk/by-partlabel/cryptroot | awk '{print $11}')")";
device="$(ls -l /dev/disk/by-partuuid/ | grep "$blkdevice" | awk '{print $9}')"
sed -i'' -e "s!/path/to/device!${device}!" /mnt/etc/nixos/lukskeyfile.nix;
# replace `/path/to/keyfile` with actual keyfile
sed -i '' -e "s!/path/to/keyfile!${keyfile}!" /mnt/etc/nixos/lukskeyfile.nix;
# replace placeholder keysize with actual keysize
sed -i '' -e "s!keyfile_size_here!${keysize}!" /mnt/etc/nixos/lukskeyfile.nix;
# add ./lukskeyfile.nix to the imports of configuration.nix
sed -i '' -e "s!\(./hardware-configuration.nix\)!\1\n ./lukskeyfile.nix!" /mnt/etc/nixos/configuration.nix
# add the zfs.nix
cp "${scriptlocation}/zfs.nix" /mnt/etc/nixos/
# generate and insert a unique hostid
hostid="$(head -c4 /dev/urandom | od -A none -t x4)"
sed -i '' -e "s!cafebabe!${hostid}!" /mnt/etc/nixos/zfs.nix
# add ./zfs.nix to the imports of configuration.nix
sed -i '' -e "s!\(./hardware-configuration.nix\)!\1\n ./zfs.nix!" /mnt/etc/nixos/configuration.nix
echo "Done!"
echo "Please check if if everything looks allright in all the files in /mnt/etc/nixos/"
exit 0
set -e
# temporary keyfile, will be removed (8k, ridiculously large)
dd if=/dev/urandom of=/tmp/keyfile bs=1k count=8
## now we encrypt the second partition
# pick a strong passphrase
echo "Creating the encrypted partition, follow the instructions and use a strong password!"
# formats the partition with luks and adds the temporary keyfile.
echo "YES" | cryptsetup luksFormat /dev/disk/by-partlabel/cryptroot --key-size 512 --hash sha512 --key-file /tmp/keyfile
if [[ $use_passphrase != "no" ]];
# sets the given passphrase or asks for one
if [[ "${passphrase}" != "NONE" ]];
echo "$passphrase" | cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot --key-file /tmp/keyfile
cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot --key-file /tmp/keyfile
echo "added passphrase"
if [[ "${keyfile}" != "NONE" ]];
cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot -d /tmp/keyfile --new-keyfile-size="${keysize}" "${keyfile}"
echo "added keyfile"
# mount the cryptdisk at /dev/mapper/nixroot
cryptsetup luksOpen /dev/disk/by-partlabel/cryptroot nixroot -d /tmp/keyfile
# remove the temporary keyfile
cryptsetup luksRemoveKey /dev/disk/by-partlabel/cryptroot /tmp/keyfile
rm -f /tmp/keyfile
exit 0
set -e
# Install zfs in the live environment
# TODO: do this in the custom bootable iso
sed -i '' -e 's/^}/ boot.supportedFilesystems = ["zfs"];\n}/' /etc/nixos/configuration.nix;
nixos-rebuild switch
## the actual zpool create is below
# zpool create \
# -O atime=on \ #
# -O relatime=on \ # only write access time (requires atime, see man zfs)
# -O compression=lz4 \ # compress all the things! (man zfs)
# -O snapdir=visible \ # ever so sligthly easier snap management (man zfs)
# -O xattr=sa \ # selinux file permissions (man zfs)
# -o ashift=12 \ # 4k blocks (man zpool)
# -o altroot=/mnt \ # temp mount during install (man zpool)
# rpool \ # new name of the pool
# /dev/mapper/nixroot # devices used in the pool (in my case one, so no mirror or raid)
zpool create \
-O atime=on \
-O relatime=on \
-O compression=lz4 \
-O snapdir=visible \
-O xattr=sa \
-o ashift=12 \
-o altroot=/mnt \
rpool \
/dev/mapper/nixroot \
# dataset for / (root)
zfs create -o mountpoint=none rpool/root
echo created root dataset
zfs create -o mountpoint=legacy rpool/root/nixos
# dataset for home, make copies of all files against corruption
zfs create -o copies=2 -o mountpoint=legacy rpool/home
# dataset for swap
zfs create -o compression=off -V 8G rpool/swap
mkswap -L SWAP /dev/zvol/rpool/swap
swapon /dev/zvol/rpool/swap
# mount the root dataset at /mnt
mount -t zfs rpool/root/nixos /mnt
# mount the home datset at future /home
mkdir -p /mnt/home
mount -t zfs rpool/home /mnt/home
# mount EFI partition at future /boot
mkdir -p /mnt/boot
mount /dev/disk/by-partlabel/efiboot /mnt/boot
# set boot filesystem
zpool set bootfs=rpool/root/nixos rpool
# enable auto snapshots for home dataset
# defaults to keeping:
# - 4 frequent snapshots (1 per 15m)
# - 24 hourly snapshots
# - 7 daily snapshots
# - 4 weekly snapshots
# - 12 monthly snapshots
zfs set com.sun:auto-snapshot=true rpool/home
exit 0
# /etc/nixos/lukskeyfile.nix (Don't forget to add it to configuration.nix)
# The minimal config required for a luksencrypted root with a keyfile
# You can skip this file if you use a passphrase
{ config, lib, pkgs, ... }:
# remove if you do not use an usb-stick with a keyfile
boot.kernelModules = [ "usb_storage" ];
boot.extraModulePackages = [ ];
boot.initrd.luks.devices."nixroot" = {
keyFile = "/path/to/keyfile";
keyFileSize = keyfile_size_here;
# This partitions the given rootdisk into two partitions: one for EFI (300MB) and the rest for LUKS (which will contain ZFS)
# This then formats the resulting EFI partition with FAT32
set -e
## The actual command is below the comment block.
# we will create a new GPT table
# o: create new GPT table
# y: confirm creation
# with the new partition table,
# we now create the EFI partition
# n: create new partion
# 1: partition number
# 2048: start position
# +300M: make it 300MB big
# ef00: set an EFI partition type
# With the EFI partition, we
# use the rest of the disk for LUKS
# n: create new partition
# 2: partition number
# <empty>: start partition right after first
# <empty>: use all remaining space
# 8300: set generic linux partition type
# We only need to set the partition labels
# c: change partition label
# 1: partition to label
# efiboot: name of the partition
# c: change partition label
# 2: partition to label
# cryptroot: name of the partition
# w: write changes and quit
# y: confirm write
gdisk ${rootdisk} >/dev/null <<end_of_commands
# check for the newly created partitions
# this sometimes gives unrelated errors
# so we change it to `partprobe || true`
partprobe "${rootdisk}" >/dev/null || true
# wait for label to show up
while [[ ! -e /dev/disk/by-partlabel/efiboot ]];
sleep 2;
# wait for label to show up
while [[ ! -e /dev/disk/by-partlabel/cryptroot ]];
sleep 2;
# check if both labels exist
ls /dev/disk/by-partlabel/efiboot >/dev/null
ls /dev/disk/by-partlabel/cryptroot >/dev/null
## format the EFI partition
mkfs.vfat /dev/disk/by-partlabel/efiboot
exit 0
# /etc/nixos/zfs.nix (Don't forget to add it to configuration.nix)
# These are the options ZFS requires, but a normal system has, of course,
# more options (like a bootloader, or installed software).
{ config, pkgs, ... }:
# remove this after 1st boot
# see
boot.kernelParams = ["zfs_force=1"];
boot.zfs.forceImportRoot = false;
boot.zfs.forceImportAll = false;
boot.supportedFilesystems = [ "zfs" ];
# required by ZFS
# see
networking.hostId = "cafebabe";
# this enables the zfs-auto-snapshot
services.zfs.autoSnapshot = {
enable = true;
flags = "-k -p --utc";
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.