Install NixOS on ZFS With Opt-In State

README.md

Install NixOS on ZFS With Opt-In State

Commands

# nvme0n1
# ├─nvme0n1p1    BOOT
# └─nvme0n1p2    LUKS CONTAINER
#   └─cryptroot  LUKS MAPPER
#     └─vg-swap  SWAP
#     └─vg-root  ZFS

# ON TARGET MACHINE

# Enable sshd (done by default in 22.11)
sudo systemctl start sshd

# Set password
passwd

# Get IP address
ifconfig

# ON OTHER MACHINE

# SSH in
ssh nixos@192.168.1.X

# Become root
sudo su

# Create convenient alias for drive
DISK=/dev/disk/by-id/<id-goes-here>

# Clear disk
wipefs -af $DISK
sgdisk -Zo $DISK

# Create GPT partition table
parted $DISK -- mklabel gpt

# Create boot partition
parted $DISK -- mkpart ESP fat32 1MiB 512MiB
parted $DISK -- set 1 boot on

# Create root partition
parted $DISK -- mkpart primary 512MiB 100%

# Create LUKS container
cryptsetup luksFormat --type luks2 --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random $DISK-part2

# Open LUKS container
cryptsetup open $DISK-part2 cryptroot

# Set up LVM
pvcreate /dev/mapper/cryptroot
vgcreate vg /dev/mapper/cryptroot
lvcreate --name swap --size 16G vg
lvcreate --name root --extents '100%FREE' vg

# Enable swap
mkswap /dev/vg/swap
swapon /dev/vg/swap

# Format root volume as ZFS
zpool create -f -R /mnt -O mountpoint=none -O compression=zstd -O atime=off -O xattr=sa -O acltype=posixacl tank /dev/vg/root

# Create root ZFS dataset
zfs create -p -o mountpoint=legacy tank/local/root

# Create blank snapshot of root ZFS dataset
zfs snapshot tank/local/root@blank

# Mount root ZFS dataset
mount -t zfs tank/local/root /mnt

# Format boot partition as FAT32
mkfs.vfat -F32 $DISK-part1

# Mount boot partition
mkdir /mnt/boot
mount $DISK-part1 /mnt/boot

# Create and mount dataset for `/nix`
zfs create -p -o mountpoint=legacy tank/local/nix
mkdir /mnt/nix
mount -t zfs tank/local/nix /mnt/nix

# Create and mount dataset for `/home`
zfs create -p -o mountpoint=legacy tank/safe/home
mkdir /mnt/home
mount -t zfs tank/safe/home /mnt/home

# Create and mount dataset for persisted state
zfs create -p -o mountpoint=legacy tank/safe/persist
mkdir /mnt/persist
mount -t zfs tank/safe/persist /mnt/persist

# Generate initial NixOS configuration
nixos-generate-config --root /mnt

# Edit the NixOS configuration to include the following:
# {
#   # Configure networking
#   networking.networkmanager.enable = true;
#
#   # Configure LUKS
#   # blkid --match-tag UUID --output value "$DISK-part2"
#   boot.initrd.luks.devices."cryptroot".device = "/dev/disk/by-uuid/<disk-uuid>";
#
#   # Configure ZFS
#   boot.supportedFilesystems = [ "zfs" ];
#   networking.hostId = "<random 8-digit hex string>"; # head -c8 /etc/machine-id
#   boot.zfs.devNodes = "/dev/vg/root";
#
#   # Roll back to blank snapshot on boot
#   boot.initrd.postDeviceCommands = lib.mkAfter ''
#     zfs rollback -r tank/local/root@blank
#   '';
#
#   # Add users
#   users.users = {
#     root.initialPassword = "alpine";
#     evanrelf = {
#       isNormalUser = true;
#       extraGroups = [ "wheel" "networkmanager" ];
#       initialPassword = "banana";
#     };
#   };
#
#  # Persist state
#  environment.etc = {
#    "nixos".source = "/persist/etc/nixos";
#    "NetworkManager/system-connections".source = "/persist/etc/NetworkManager/system-connections";
#    "adjtime".source = "/persist/etc/adjtime";
#    "NIXOS".source = "/persist/etc/NIXOS";
#  };
#
#  systemd.tmpfiles.rules = [
#    "L /var/lib/NetworkManager/secret_key - - - - /persist/var/lib/NetworkManager/secret_key"
#    "L /var/lib/NetworkManager/seen-bssids - - - - /persist/var/lib/NetworkManager/seen-bssids"
#    "L /var/lib/NetworkManager/timestamps - - - - /persist/var/lib/NetworkManager/timestamps"
#  ];
#
#  security.sudo.extraConfig = ''
#    Defaults lecture = never
#  '';
# }
# Also make sure to uncomment any useful options already in the file.
vim /mnt/etc/nixos/configuration.nix

# Install NixOS
nixos-install

# Reboot
reboot

# Configuring what state to keep using instructions from "Erase Your Darlings":
# https://grahamc.com/blog/erase-your-darlings#opting-in

Resources

• https://grahamc.com/blog/erase-your-darlings
• https://nixos.wiki/wiki/NixOS_on_ZFS#How_to_install_NixOS_on_a_ZFS_root_filesystem
• https://nixos.org/nixos/manual/index.html#sec-installation
• https://mt-caret.github.io/blog/posts/2020-06-29-optin-state.html
• https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/
• https://elis.nu/blog/2020/06/nixos-tmpfs-as-home/

Todo:

• local vs safe naming?
• zstd compression
• Look into this https://github.com/jimsalterjrs/sanoid/
• Look into using regular rsync.net plan (not attic) for ZFS features (backing up snapshots and such?). Maybe not any better than borg or restic...

You have to add the persist stuff after the first boot. And you have to manually copy over things to /persist.