Skip to content

Instantly share code, notes, and snippets.

@evanrelf
Last active October 9, 2023 20:45
Show Gist options
  • Save evanrelf/562102d6e8bc5b0f386fe8e91c40e863 to your computer and use it in GitHub Desktop.
Save evanrelf/562102d6e8bc5b0f386fe8e91c40e863 to your computer and use it in GitHub Desktop.
Install NixOS on ZFS With Opt-In State

Install NixOS on ZFS With Opt-In State

Commands

# nvme0n1
# ├─nvme0n1p1    BOOT
# └─nvme0n1p2    LUKS CONTAINER
#   └─cryptroot  LUKS MAPPER
#     └─vg-swap  SWAP
#     └─vg-root  ZFS

# ON TARGET MACHINE

# Enable sshd (done by default in 22.11)
sudo systemctl start sshd

# Set password
passwd

# Get IP address
ifconfig

# ON OTHER MACHINE

# SSH in
ssh nixos@192.168.1.X

# Become root
sudo su

# Create convenient alias for drive
DISK=/dev/disk/by-id/<id-goes-here>

# Clear disk
wipefs -af $DISK
sgdisk -Zo $DISK

# Create GPT partition table
parted $DISK -- mklabel gpt

# Create boot partition
parted $DISK -- mkpart ESP fat32 1MiB 512MiB
parted $DISK -- set 1 boot on

# Create root partition
parted $DISK -- mkpart primary 512MiB 100%

# Create LUKS container
cryptsetup luksFormat --type luks2 --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random $DISK-part2

# Open LUKS container
cryptsetup open $DISK-part2 cryptroot

# Set up LVM
pvcreate /dev/mapper/cryptroot
vgcreate vg /dev/mapper/cryptroot
lvcreate --name swap --size 16G vg
lvcreate --name root --extents '100%FREE' vg

# Enable swap
mkswap /dev/vg/swap
swapon /dev/vg/swap

# Format root volume as ZFS
zpool create -f -R /mnt -O mountpoint=none -O compression=zstd -O atime=off -O xattr=sa -O acltype=posixacl tank /dev/vg/root

# Create root ZFS dataset
zfs create -p -o mountpoint=legacy tank/local/root

# Create blank snapshot of root ZFS dataset
zfs snapshot tank/local/root@blank

# Mount root ZFS dataset
mount -t zfs tank/local/root /mnt

# Format boot partition as FAT32
mkfs.vfat -F32 $DISK-part1

# Mount boot partition
mkdir /mnt/boot
mount $DISK-part1 /mnt/boot

# Create and mount dataset for `/nix`
zfs create -p -o mountpoint=legacy tank/local/nix
mkdir /mnt/nix
mount -t zfs tank/local/nix /mnt/nix

# Create and mount dataset for `/home`
zfs create -p -o mountpoint=legacy tank/safe/home
mkdir /mnt/home
mount -t zfs tank/safe/home /mnt/home

# Create and mount dataset for persisted state
zfs create -p -o mountpoint=legacy tank/safe/persist
mkdir /mnt/persist
mount -t zfs tank/safe/persist /mnt/persist

# Generate initial NixOS configuration
nixos-generate-config --root /mnt

# Edit the NixOS configuration to include the following:
# {
#   # Configure networking
#   networking.networkmanager.enable = true;
#
#   # Configure LUKS
#   # blkid --match-tag UUID --output value "$DISK-part2"
#   boot.initrd.luks.devices."cryptroot".device = "/dev/disk/by-uuid/<disk-uuid>";
#
#   # Configure ZFS
#   boot.supportedFilesystems = [ "zfs" ];
#   networking.hostId = "<random 8-digit hex string>"; # head -c8 /etc/machine-id
#   boot.zfs.devNodes = "/dev/vg/root";
#
#   # Roll back to blank snapshot on boot
#   boot.initrd.postDeviceCommands = lib.mkAfter ''
#     zfs rollback -r tank/local/root@blank
#   '';
#
#   # Add users
#   users.users = {
#     root.initialPassword = "alpine";
#     evanrelf = {
#       isNormalUser = true;
#       extraGroups = [ "wheel" "networkmanager" ];
#       initialPassword = "banana";
#     };
#   };
#
#  # Persist state
#  environment.etc = {
#    "nixos".source = "/persist/etc/nixos";
#    "NetworkManager/system-connections".source = "/persist/etc/NetworkManager/system-connections";
#    "adjtime".source = "/persist/etc/adjtime";
#    "NIXOS".source = "/persist/etc/NIXOS";
#  };
#
#  systemd.tmpfiles.rules = [
#    "L /var/lib/NetworkManager/secret_key - - - - /persist/var/lib/NetworkManager/secret_key"
#    "L /var/lib/NetworkManager/seen-bssids - - - - /persist/var/lib/NetworkManager/seen-bssids"
#    "L /var/lib/NetworkManager/timestamps - - - - /persist/var/lib/NetworkManager/timestamps"
#  ];
#
#  security.sudo.extraConfig = ''
#    Defaults lecture = never
#  '';
# }
# Also make sure to uncomment any useful options already in the file.
vim /mnt/etc/nixos/configuration.nix

# Install NixOS
nixos-install

# Reboot
reboot

# Configuring what state to keep using instructions from "Erase Your Darlings":
# https://grahamc.com/blog/erase-your-darlings#opting-in

Resources

@evanrelf
Copy link
Author

Todo:

  • local vs safe naming?
  • zstd compression
  • Look into this https://github.com/jimsalterjrs/sanoid/
  • Look into using regular rsync.net plan (not attic) for ZFS features (backing up snapshots and such?). Maybe not any better than borg or restic...

@evanrelf
Copy link
Author

You have to add the persist stuff after the first boot. And you have to manually copy over things to /persist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment