NixOS installation with opt-in state (darling erasure)

nixos-install-with-opt-in-state.txt

==================================
INSTALLING NIXOS WITH OPT-IN STATE

Based mostly on:

  grahamc.com/blog/erase-your-darlings
  gist.github.com/mx00s/ea2462a3fe6fdaa65692fe7ee824de3e
  nixos.org/manual/nixos/stable/

Also relevant:

  www.reddit.com/r/NixOS/comments/su5bwl/whos_here_runs_nixos_with_opt_in_state/
  github.com/nix-community/impermanence
  www.reddit.com/r/NixOS/comments/lyuxv5/erasing_root_on_every_boot/
  mt-caret.github.io/blog/posts/2020-06-29-optin-state.html


=====
STEPS

Acquire a machine on which to perform the installation.
It must support booting from ISO
For VPSs (virtual private servers), I use vultr.com

Download the NixOS ISO image from nixos.org/download.html
Boot your machine into downloaded ISO

Add SSH key (optional)
> SSH_KEY=<your ssh key>
> mkdir -p /home/nixos/.ssh
> echo "$SSH_KEY" > /home/nixos/.ssh/authorized_keys
> chmod 700 /home/nixos/.ssh
> chmod 600 /home/nixos/.ssh/authorized_keys
> chown -R nixos /home/nixos/.ssh

Connect to machine, via SSH or otherwise

Become root, for convenience
> sudo su

Are you on UEFI or BIOS?
> [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
Remember the result! This will be important later.

List partitions
> lsblk

Set DISK to the partition on which to perform the installation
Set PART_SIZE to the desired NixOS partition size; the rest will be used for swap
> DISK=<disk name, eg /dev/vda>
> PART_SIZE=<partition size, eg 150G>

Note: while partitioning you may see a warning about updating /etc/fstab; this can be ignored.

Create boot table and root & swap partitions

If on BIOS:
  > parted $DISK mklabel msdos
  > parted $DISK mkpart primary 2M $PART_SIZE
  > parted $DISK mkpart primary linux-swap $PART_SIZE 100%

If on UEFI:
  > parted $DISK mklabel gpt
  > parted $DISK mkpart primary 512MB $PART_SIZE
  > parted $DISK mkpart primary linux-swap $PART_SIZE 100%
  > parted $DISK mkpart ESP fat32 1MB 512MB
  > parted $DISK set 3 esp on
  > mkfs.fat -F32 -n boot ${DISK}3  # might be `${DISK}p3`; check `lsblk`

Initialize swap
> mkswap -L swap ${DISK}2  # might be `${DISK}p2`; check `lsblk`

Create ZFS root pool (≈ storage)
> zpool create -f rpool ${DISK}1  # might be `${DISK}p1`; check `lsblk`
> zfs set compression=on rpool

Create ZFS datasets (≈ filesystems) and mount them
We create the following filesystem structure:
  / will get wiped on reboot, except that
  /{boot,nix,per} will all not get wiped ("per" stands for "persistent")
Note: "eyd" stands for "erase your darlings", the name of the first hyperlink in this document
> zfs create -p -o mountpoint=legacy rpool/eyd/root
> zfs set xattr=sa rpool/eyd/root
> zfs set acltype=posixacl rpool/eyd/root
> zfs snapshot rpool/eyd/root@blank
> zfs create -p -o mountpoint=legacy rpool/eyd/nix
> zfs create -p -o mountpoint=legacy rpool/eyd/per
> mkdir -p /mnt
> mount -t zfs rpool/eyd/root /mnt
> mkdir -p /mnt/{nix,boot,per}
> mount -t zfs rpool/eyd/nix /mnt/nix
> mount -t zfs rpool/eyd/per /mnt/per

If on BIOS:
  > zfs create -p -o mountpoint=legacy rpool/eyd/boot
  > mount -t zfs rpool/eyd/boot /mnt/boot

If on UEFI:
  > mount ${DISK}3 /mnt/boot  # might be `${DISK}p3`; check `lsblk`

Generate nix configuration file
> nixos-generate-config --root /mnt

And edit it
> vim /mnt/etc/nixos/configuration.nix
Add the following. Note that this should not be copy-pasted as is; it requires modification

```
## Boot stuff ##
boot.supportedFilesystems = [ "zfs" ];
boot.loader.grub.device = "${DISK}";  # replace `${DISK}` with the actual disk name, eg /dev/nvme0n1

# Might be required if running on a virtual machine
boot.zfs.devNodes = "/dev/disk/by-path";

# Required by zfs. generate with 'head -c4 /dev/urandom | od -t x4 | cut -c9-16'
networking.hostId = "<your host id>";

boot.kernelPackages = pkgs.zfs.latestCompatibleLinuxPackages;


## User and SSH stuff ##
services.openssh.enable = true;  # If using VPS

security.sudo.wheelNeedsPassword = false;
users.users."<your username>" = {
  isNormalUser = true;
  password = "password";  # Change this once your computer is set up!
  home = "/home/<your username>";
  extraGroups = [ "wheel" "networkmanager" ];
  openssh.authorizedKeys.keys = [ "<your ssh key>" ];  # If using VPS
};


## Packages ##
environment.systemPackages = with pkgs; [ vim wirelesstools git ];

# Enable nmcli
networking.wireless.enable = false;
networking.networkmanager.enable = true;
# These two might also be necessary:
networking.useDHCP = false;
networking.networkmanager.wifi.scanRandMacAddress = false;
```

It might also be good to import configuration from `nixos-hardware`, ala
```
imports = [
  "${builtins.fetchGit { url = "https://github.com/NixOS/nixos-hardware.git"; }}/path/to/machine"
];
```
(This was once necessary for my `ncmli` to work)

Perform NixOS installation
> nixos-install --verbose --no-root-password

Remove the ISO from the machine and reboot

Connect back into machine and run
> sudo su

Move NixOS configuration into persistent storage and enable wipe-on-boot
> mkdir -p /per/etc
> mv /etc/nixos /per/etc
> vim /per/etc/nixos/configuration.nix
Add the following:

```
# Make NixOS read config from /per
environment.etc = { nixos.source = /per/etc/nixos; };

# Enables wipe-on-boot
# Might need mkAfter or might need mkBefore
# Remember to add 'lib' as a param to the enclosing function
boot.initrd.postDeviceCommands = lib.mkBefore ''
  zfs rollback -r rpool/eyd/root@blank
'';
```

Activate the new config
> nixos-rebuild -I nixos-config=/per/etc/nixos/configuration.nix switch

Note: when you run nixos-rebuild you will always need to point it to the new place for configuration.nix

Now let's test it! Create two files, one in the persistent /per and one in the ephemeral /etc
> touch /per/hello
> touch /etc/hello
Restart the machine. /per/hello should persist but /etc/hello should get deleted
I sometimes find it a little fiddly at first. Try restarting multiple times and playing with
using lib.mkBefore vs lib.mkAfter.
But after it works the first time it seems to work every time!

Done!




nb.

- If you need to tinker on the system, boot via USB and use `mount-t zfs rpool/eyd/root /mnt`. This may be necessary to e.g. modify `/etc/shadow`.

Can you make a tutorial on how to setup agenix/sops-nix with erase your darling?

I don't know anything about sops-nix, sorry.