ixmatus/Readme.md

## Readme.md

      
    Raw
  

              Readme.md
            
          
    NixOS with ZFS on LUKS

After some effort (and asking for help on the nix-dev mailing list) I
installed ZFS on an encrypted partition. The relevant configuration is
below.

Installing

Use keyfile
Configure passphrase
Configure without passphrase


Misc commands
To Do
Resources I used
Installing with old script

Installing

I do not have a custom iso yet, so you'll need two USBs. One for the NixOS iso, and one for these files. You'll have to mount the second stick manually.

Boot into the nixos environment and find the uuid or id of the disk you want to install to. Do not use /dev/sda but /dev/disk/by-..., use lsblk and blkid.
export it to the environment as rootdisk:

# whole disk please, no partition
export rootdisk="/dev/disk/by-id/ata-Some-Storage-Device"

use keyfile and/or configure passphrase usage (see sections below)
run it:

bash /path/to/automated_install.sh
Use keyfile

It is possible to use a keyfile (e.g. on a usb stick). If you want a keyfile and not have a passphrase for backup, see Configure without passphrase below.
# part of step 3
export keyfile="/dev/disk/by-id/usb-Some-Usb-Stick"
# optional, default is 4096
export keysize="8192"
Configure passphrase

It is possible to pass the passphrase in an environment variable to make the install fully automated. This is generally unwise, but since we are in a temporary live enviorment I consider it safe enough. You can also put it as passphrase="your passphrase here" in automated_install.sh on line 16 instead.
If you add a keyfile as well, both are added.
# part of step 3
export passphrase="your passphrase here"
Configure without passphrase

If you only want to add a keyfile and not set a passphrase, set use_passphrase to no. This is not recommended.
# part of step 3
export use_passphrase="no"
# see Use keyfile above
export keyfile="/path/to/keyfile"
Misc commands

I always run these command right after booting the install usb.
# I use programmer dvorak instead of qwerty
loadkeys dvorak-programmer
To Do


 use nixos-rebuild to make an iso containing the files
 customise the iso with ZFS support and these files
 find the location of automated_install.sh in the built iso.

Resources I used

I used the following resources:

https://nixos.org/wiki/ZFS_on_NixOS#How_to_install_NixOS_on_a_ZFS_root_filesystem
https://nixos.org/wiki/Encrypted_Root_on_NixOS
http://www.pavelkogan.com/2014/05/23/luks-full-disk-encryption/
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.10-Root-on-ZFS

Installing with old script

use this version of the files: old version. All text below is about those versions, not the ones you see here.
The commands in init.sh I run manually, (so no sed :-P)
The zfscreate.sh is used to set up a single-disk ZFS root filesystem inside of an encrypted LUKS container.
The two *.nix files have the minimum config needed for this (compare them with the generated ones in /mnt/etc/nixos/); The UUIDs should be filled-in by nixos-generate-config;
the "usb_storage" addition is not needed for everyone, just like the keyfile options; the other important changes
are the hostId, which is required by ZFS; and the boot.supportedFilesystems which I'm not even sure of if that's necessary

  
## automated_install.sh
# you only need to set this to the disk to want to install to
# IT WILL BE WIPED
rootdisk="${rootdisk:-NONE}";

# use keyfile (optional)
keyfile="${keyfile:-NONE}";
keysize="${keysize:-4096}";

# set to "no" to not set a passphrase
# I highly recommed setting a passphrase and storing it in a safe location
use_passphrase="${use_passphrase:-yes}";
# You can set the passphrase here (so you can view it in plaintext,
# but do not forget to remove it.
# TODO: check if this file is on a tempfs just like /etc/nixos/configuration.nix
passphrase="${passphrase:-NONE}"

# Probably no need to change anything below or
# in the other scripts, there be dragons

# exit on error
set -e
# abort if no root disk is set
if [[ "${rootdisk}" != "NONE" ]]; echo "please set rootdisk with: \`rootdisk=/dev/disk/by-id/disk_id_for_root_device $0\`"; exit 1; fi

if [[ "${use_passphrase}" == "no" ]] && [[ "${keyfile}" == "NONE" ]]; echo "please use at least one of the following: keyfile, password."; exit 2; fi

if [[ "${use_passphrase}" == "no" ]]; echo "using a passprase is highly recommended, since keyfiles can get corrupt or lost."; fi

export rootdisk keyfile keysize use_passphrase;
# absolute location for this script (directory the files are in)
export scriptlocation=$(dirname $(readlink -f $0))

bash partition.sh
bash formatluks.sh
bash formatzfs.sh

nixos-generate-config --root /mnt

if [[ "$keyfile" != "NONE" ]];
then
  # add lukskeyfile.nix
  cp "${scriptlocation}/lukskeyfile.nix" /mnt/etc/nixos/

  # replace `/path/to/device` with actual keyfile
  echo "path to device"
  blkdevice="$(basename "$(ls -l /dev/disk/by-partlabel/cryptroot | awk '{print $11}')")";
  device="$(ls -l /dev/disk/by-partuuid/ | grep "$blkdevice" | awk '{print $9}')"
  sed -i'' -e "s!/path/to/device!${device}!" /mnt/etc/nixos/lukskeyfile.nix;

  # replace `/path/to/keyfile` with actual keyfile
  sed -i '' -e "s!/path/to/keyfile!${keyfile}!" /mnt/etc/nixos/lukskeyfile.nix;

  # replace placeholder keysize with actual keysize
  sed -i '' -e "s!keyfile_size_here!${keysize}!"             /mnt/etc/nixos/lukskeyfile.nix;

  # add ./lukskeyfile.nix to the imports of configuration.nix
  sed -i '' -e "s!\(./hardware-configuration.nix\)!\1\n      ./lukskeyfile.nix!" /mnt/etc/nixos/configuration.nix
fi

# add the zfs.nix
cp "${scriptlocation}/zfs.nix" /mnt/etc/nixos/

# generate and insert a unique hostid
hostid="$(head -c4 /dev/urandom | od -A none -t x4)"
sed -i '' -e "s!cafebabe!${hostid}!"                                   /mnt/etc/nixos/zfs.nix

# add ./zfs.nix to the imports of configuration.nix
sed -i '' -e "s!\(./hardware-configuration.nix\)!\1\n      ./zfs.nix!" /mnt/etc/nixos/configuration.nix

echo "Done!"
echo "Please check if if everything looks allright in all the files in /mnt/etc/nixos/"
exit 0

## formatluks.sh
#!/bin/bash
set -e

# temporary keyfile, will be removed (8k, ridiculously large)
dd if=/dev/urandom of=/tmp/keyfile bs=1k count=8

## now we encrypt the second partition
# pick a strong passphrase

echo "Creating the encrypted partition, follow the instructions and use a strong password!"
# formats the partition with luks and adds the temporary keyfile.
echo "YES" | cryptsetup luksFormat /dev/disk/by-partlabel/cryptroot --key-size 512 --hash sha512 --key-file /tmp/keyfile

if [[ $use_passphrase != "no" ]];
then
	# sets the given passphrase or asks for one
	if [[ "${passphrase}" != "NONE" ]];
	then
		echo "$passphrase" | cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot --key-file /tmp/keyfile
	else
		cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot --key-file /tmp/keyfile
	fi
	echo "added passphrase"
fi


if [[ "${keyfile}" != "NONE" ]];
then
	cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot  -d /tmp/keyfile --new-keyfile-size="${keysize}" "${keyfile}"
	echo "added keyfile"
fi


# mount the cryptdisk at /dev/mapper/nixroot
cryptsetup luksOpen /dev/disk/by-partlabel/cryptroot nixroot -d /tmp/keyfile

# remove the temporary keyfile
cryptsetup luksRemoveKey /dev/disk/by-partlabel/cryptroot /tmp/keyfile
rm -f /tmp/keyfile

exit 0

## formatzfs.sh
#!/bin/bash
set -e

# Install zfs in the live environment
# TODO: do this in the custom bootable iso
sed -i '' -e 's/^}/  boot.supportedFilesystems = ["zfs"];\n}/' /etc/nixos/configuration.nix;
nixos-rebuild switch


## the actual zpool create is below
#
# zpool create        \
# -O atime=on         \ #
# -O relatime=on      \ # only write access time (requires atime, see man zfs)
# -O compression=lz4  \ # compress all the things! (man zfs)
# -O snapdir=visible  \ # ever so sligthly easier snap management (man zfs)
# -O xattr=sa         \ # selinux file permissions (man zfs)
# -o ashift=12        \ # 4k blocks (man zpool)
# -o altroot=/mnt     \ # temp mount during install (man zpool)
# rpool               \ # new name of the pool
# /dev/mapper/nixroot   # devices used in the pool (in my case one, so no mirror or raid)

zpool create        \
-O atime=on         \
-O relatime=on      \
-O compression=lz4  \
-O snapdir=visible  \
-O xattr=sa         \
-o ashift=12        \
-o altroot=/mnt     \
rpool               \
/dev/mapper/nixroot \


# dataset for / (root)
zfs create -o mountpoint=none rpool/root
echo created root dataset
zfs create -o mountpoint=legacy rpool/root/nixos

# dataset for home, make copies of all files against corruption
zfs create -o copies=2 -o mountpoint=legacy rpool/home

# dataset for swap
zfs create -o compression=off -V 8G rpool/swap
mkswap -L SWAP /dev/zvol/rpool/swap
swapon /dev/zvol/rpool/swap

# mount the root dataset at /mnt
mount -t zfs rpool/root/nixos /mnt

# mount the home datset at future /home
mkdir -p /mnt/home
mount -t zfs rpool/home /mnt/home

# mount EFI partition at future /boot
mkdir -p /mnt/boot
mount  /dev/disk/by-partlabel/efiboot /mnt/boot

# set boot filesystem
zpool set bootfs=rpool/root/nixos rpool

# enable auto snapshots for home dataset
# defaults to keeping:
# - 4 frequent snapshots (1 per 15m)
# - 24 hourly snapshots
# - 7 daily snapshots
# - 4 weekly snapshots
# - 12 monthly snapshots
zfs set com.sun:auto-snapshot=true rpool/home

exit 0

## lukskeyfile.nix
# /etc/nixos/lukskeyfile.nix (Don't forget to add it to configuration.nix)
# The minimal config required for a luksencrypted root with a keyfile
# You can skip this file if you use a passphrase
{ config, lib, pkgs, ... }:
{
  # remove if you do not use an usb-stick with a keyfile
  boot.kernelModules = [ "usb_storage" ];
  boot.extraModulePackages = [ ];

  boot.initrd.luks.devices."nixroot" = {
    keyFile = "/path/to/keyfile";
    keyFileSize = keyfile_size_here;
  };
}

## partition.sh
#!/bin/bash
# This partitions the given rootdisk into two partitions: one for EFI (300MB) and the rest for LUKS (which will contain ZFS)
# This then formats the resulting EFI partition with FAT32

set -e
## The actual command is below the comment block.
# we will create a new GPT table
#
# o:         create new GPT table
#         y: confirm creation
#
# with the new partition table,
# we now create the EFI partition
#
# n:         create new partion
#         1: partition number
#      2048: start position
#     +300M: make it 300MB big
#      ef00: set an EFI partition type
#
# With the EFI partition, we
# use the rest of the disk for LUKS
#
# n:         create new partition
#         2: partition number
#   <empty>: start partition right after first
#   <empty>: use all remaining space
#      8300: set generic linux partition type
#
# We only need to set the partition labels
#
# c:         change partition label
#         1: partition to label
#   efiboot: name of the partition
# c:         change partition label
#         2: partition to label
# cryptroot: name of the partition
#
# w:	     write changes and quit
#         y: confirm write

gdisk ${rootdisk} >/dev/null <<end_of_commands
o
y
n
1
2048
+300M
ef00
n
2


8300
c
1
efiboot
c
2
cryptroot
w
y
end_of_commands


# check for the newly created partitions
# this sometimes gives unrelated errors
# so we change it to  `partprobe || true`
partprobe "${rootdisk}" >/dev/null || true

# wait for label to show up
while [[ ! -e /dev/disk/by-partlabel/efiboot ]];
do
	sleep 2;
done
# wait for label to show up
while [[ ! -e /dev/disk/by-partlabel/cryptroot ]];
do
	sleep 2;
done
# check if both labels exist
ls /dev/disk/by-partlabel/efiboot   >/dev/null
ls /dev/disk/by-partlabel/cryptroot >/dev/null

## format the EFI partition
mkfs.vfat /dev/disk/by-partlabel/efiboot

exit 0

## zfs.nix
# /etc/nixos/zfs.nix (Don't forget to add it to configuration.nix)
# These are the options ZFS requires, but a normal system has, of course,
#   more options (like a bootloader, or installed software).
{ config, pkgs, ... }:
{
  # remove this after 1st boot
  # see https://nixos.org/nixos/options.html#boot.zfs.forceimportroot
  boot.kernelParams = ["zfs_force=1"];

  boot.zfs.forceImportRoot = false;
  boot.zfs.forceImportAll = false;

  boot.supportedFilesystems = [ "zfs" ];

  # required by ZFS
  # see https://nixos.org/nixos/options.html#networking.hostid
  networking.hostId = "cafebabe";

  # this enables the zfs-auto-snapshot
  services.zfs.autoSnapshot = {
    enable = true;
    flags = "-k -p --utc";
  };
}
	# you only need to set this to the disk to want to install to
	# IT WILL BE WIPED
	rootdisk="${rootdisk:-NONE}";

	# use keyfile (optional)
	keyfile="${keyfile:-NONE}";
	keysize="${keysize:-4096}";

	# set to "no" to not set a passphrase
	# I highly recommed setting a passphrase and storing it in a safe location
	use_passphrase="${use_passphrase:-yes}";
	# You can set the passphrase here (so you can view it in plaintext,
	# but do not forget to remove it.
	# TODO: check if this file is on a tempfs just like /etc/nixos/configuration.nix
	passphrase="${passphrase:-NONE}"

	# Probably no need to change anything below or
	# in the other scripts, there be dragons

	# exit on error
	set -e
	# abort if no root disk is set
	if [[ "${rootdisk}" != "NONE" ]]; echo "please set rootdisk with: \`rootdisk=/dev/disk/by-id/disk_id_for_root_device $0\`"; exit 1; fi

	if [[ "${use_passphrase}" == "no" ]] && [[ "${keyfile}" == "NONE" ]]; echo "please use at least one of the following: keyfile, password."; exit 2; fi

	if [[ "${use_passphrase}" == "no" ]]; echo "using a passprase is highly recommended, since keyfiles can get corrupt or lost."; fi

	export rootdisk keyfile keysize use_passphrase;
	# absolute location for this script (directory the files are in)
	export scriptlocation=$(dirname $(readlink -f $0))

	bash partition.sh
	bash formatluks.sh
	bash formatzfs.sh

	nixos-generate-config --root /mnt

	if [[ "$keyfile" != "NONE" ]];
	then
	# add lukskeyfile.nix
	cp "${scriptlocation}/lukskeyfile.nix" /mnt/etc/nixos/

	# replace `/path/to/device` with actual keyfile
	echo "path to device"
	blkdevice="$(basename "$(ls -l /dev/disk/by-partlabel/cryptroot \| awk '{print $11}')")";
	device="$(ls -l /dev/disk/by-partuuid/ \| grep "$blkdevice" \| awk '{print $9}')"
	sed -i'' -e "s!/path/to/device!${device}!" /mnt/etc/nixos/lukskeyfile.nix;

	# replace `/path/to/keyfile` with actual keyfile
	sed -i '' -e "s!/path/to/keyfile!${keyfile}!" /mnt/etc/nixos/lukskeyfile.nix;

	# replace placeholder keysize with actual keysize
	sed -i '' -e "s!keyfile_size_here!${keysize}!" /mnt/etc/nixos/lukskeyfile.nix;

	# add ./lukskeyfile.nix to the imports of configuration.nix
	sed -i '' -e "s!\(./hardware-configuration.nix\)!\1\n ./lukskeyfile.nix!" /mnt/etc/nixos/configuration.nix
	fi

	# add the zfs.nix
	cp "${scriptlocation}/zfs.nix" /mnt/etc/nixos/

	# generate and insert a unique hostid
	hostid="$(head -c4 /dev/urandom \| od -A none -t x4)"
	sed -i '' -e "s!cafebabe!${hostid}!" /mnt/etc/nixos/zfs.nix

	# add ./zfs.nix to the imports of configuration.nix
	sed -i '' -e "s!\(./hardware-configuration.nix\)!\1\n ./zfs.nix!" /mnt/etc/nixos/configuration.nix

	echo "Done!"
	echo "Please check if if everything looks allright in all the files in /mnt/etc/nixos/"
	exit 0
	#!/bin/bash
	set -e

	# temporary keyfile, will be removed (8k, ridiculously large)
	dd if=/dev/urandom of=/tmp/keyfile bs=1k count=8

	## now we encrypt the second partition
	# pick a strong passphrase

	echo "Creating the encrypted partition, follow the instructions and use a strong password!"
	# formats the partition with luks and adds the temporary keyfile.
	echo "YES" \| cryptsetup luksFormat /dev/disk/by-partlabel/cryptroot --key-size 512 --hash sha512 --key-file /tmp/keyfile

	if [[ $use_passphrase != "no" ]];
	then
	# sets the given passphrase or asks for one
	if [[ "${passphrase}" != "NONE" ]];
	then
	echo "$passphrase" \| cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot --key-file /tmp/keyfile
	else
	cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot --key-file /tmp/keyfile
	fi
	echo "added passphrase"
	fi


	if [[ "${keyfile}" != "NONE" ]];
	then
	cryptsetup luksAddKey /dev/disk/by-partlabel/cryptroot -d /tmp/keyfile --new-keyfile-size="${keysize}" "${keyfile}"
	echo "added keyfile"
	fi


	# mount the cryptdisk at /dev/mapper/nixroot
	cryptsetup luksOpen /dev/disk/by-partlabel/cryptroot nixroot -d /tmp/keyfile

	# remove the temporary keyfile
	cryptsetup luksRemoveKey /dev/disk/by-partlabel/cryptroot /tmp/keyfile
	rm -f /tmp/keyfile

	exit 0
	#!/bin/bash
	set -e

	# Install zfs in the live environment
	# TODO: do this in the custom bootable iso
	sed -i '' -e 's/^}/ boot.supportedFilesystems = ["zfs"];\n}/' /etc/nixos/configuration.nix;
	nixos-rebuild switch


	## the actual zpool create is below
	#
	# zpool create \
	# -O atime=on \ #
	# -O relatime=on \ # only write access time (requires atime, see man zfs)
	# -O compression=lz4 \ # compress all the things! (man zfs)
	# -O snapdir=visible \ # ever so sligthly easier snap management (man zfs)
	# -O xattr=sa \ # selinux file permissions (man zfs)
	# -o ashift=12 \ # 4k blocks (man zpool)
	# -o altroot=/mnt \ # temp mount during install (man zpool)
	# rpool \ # new name of the pool
	# /dev/mapper/nixroot # devices used in the pool (in my case one, so no mirror or raid)

	zpool create \
	-O atime=on \
	-O relatime=on \
	-O compression=lz4 \
	-O snapdir=visible \
	-O xattr=sa \
	-o ashift=12 \
	-o altroot=/mnt \
	rpool \
	/dev/mapper/nixroot \


	# dataset for / (root)
	zfs create -o mountpoint=none rpool/root
	echo created root dataset
	zfs create -o mountpoint=legacy rpool/root/nixos

	# dataset for home, make copies of all files against corruption
	zfs create -o copies=2 -o mountpoint=legacy rpool/home

	# dataset for swap
	zfs create -o compression=off -V 8G rpool/swap
	mkswap -L SWAP /dev/zvol/rpool/swap
	swapon /dev/zvol/rpool/swap

	# mount the root dataset at /mnt
	mount -t zfs rpool/root/nixos /mnt

	# mount the home datset at future /home
	mkdir -p /mnt/home
	mount -t zfs rpool/home /mnt/home

	# mount EFI partition at future /boot
	mkdir -p /mnt/boot
	mount /dev/disk/by-partlabel/efiboot /mnt/boot

	# set boot filesystem
	zpool set bootfs=rpool/root/nixos rpool

	# enable auto snapshots for home dataset
	# defaults to keeping:
	# - 4 frequent snapshots (1 per 15m)
	# - 24 hourly snapshots
	# - 7 daily snapshots
	# - 4 weekly snapshots
	# - 12 monthly snapshots
	zfs set com.sun:auto-snapshot=true rpool/home

	exit 0
	# /etc/nixos/lukskeyfile.nix (Don't forget to add it to configuration.nix)
	# The minimal config required for a luksencrypted root with a keyfile
	# You can skip this file if you use a passphrase
	{ config, lib, pkgs, ... }:
	{
	# remove if you do not use an usb-stick with a keyfile
	boot.kernelModules = [ "usb_storage" ];
	boot.extraModulePackages = [ ];

	boot.initrd.luks.devices."nixroot" = {
	keyFile = "/path/to/keyfile";
	keyFileSize = keyfile_size_here;
	};
	}
	#!/bin/bash
	# This partitions the given rootdisk into two partitions: one for EFI (300MB) and the rest for LUKS (which will contain ZFS)
	# This then formats the resulting EFI partition with FAT32

	set -e
	## The actual command is below the comment block.
	# we will create a new GPT table
	#
	# o: create new GPT table
	# y: confirm creation
	#
	# with the new partition table,
	# we now create the EFI partition
	#
	# n: create new partion
	# 1: partition number
	# 2048: start position
	# +300M: make it 300MB big
	# ef00: set an EFI partition type
	#
	# With the EFI partition, we
	# use the rest of the disk for LUKS
	#
	# n: create new partition
	# 2: partition number
	# <empty>: start partition right after first
	# <empty>: use all remaining space
	# 8300: set generic linux partition type
	#
	# We only need to set the partition labels
	#
	# c: change partition label
	# 1: partition to label
	# efiboot: name of the partition
	# c: change partition label
	# 2: partition to label
	# cryptroot: name of the partition
	#
	# w: write changes and quit
	# y: confirm write

	gdisk ${rootdisk} >/dev/null <<end_of_commands
	o
	y
	n
	1
	2048
	+300M
	ef00
	n
	2


	8300
	c
	1
	efiboot
	c
	2
	cryptroot
	w
	y
	end_of_commands


	# check for the newly created partitions
	# this sometimes gives unrelated errors
	# so we change it to `partprobe \|\| true`
	partprobe "${rootdisk}" >/dev/null \|\| true

	# wait for label to show up
	while [[ ! -e /dev/disk/by-partlabel/efiboot ]];
	do
	sleep 2;
	done
	# wait for label to show up
	while [[ ! -e /dev/disk/by-partlabel/cryptroot ]];
	do
	sleep 2;
	done
	# check if both labels exist
	ls /dev/disk/by-partlabel/efiboot >/dev/null
	ls /dev/disk/by-partlabel/cryptroot >/dev/null

	## format the EFI partition
	mkfs.vfat /dev/disk/by-partlabel/efiboot

	exit 0
	# /etc/nixos/zfs.nix (Don't forget to add it to configuration.nix)
	# These are the options ZFS requires, but a normal system has, of course,
	# more options (like a bootloader, or installed software).
	{ config, pkgs, ... }:
	{
	# remove this after 1st boot
	# see https://nixos.org/nixos/options.html#boot.zfs.forceimportroot
	boot.kernelParams = ["zfs_force=1"];

	boot.zfs.forceImportRoot = false;
	boot.zfs.forceImportAll = false;

	boot.supportedFilesystems = [ "zfs" ];

	# required by ZFS
	# see https://nixos.org/nixos/options.html#networking.hostid
	networking.hostId = "cafebabe";

	# this enables the zfs-auto-snapshot
	services.zfs.autoSnapshot = {
	enable = true;
	flags = "-k -p --utc";
	};
	}