configuration.nix

{
  systemd.targets.maintenance = {                                                                                    
    description = "Maintenance target with only sshd";                                                               
    after = [ "network-online.target" "network-setup.service" "sshd.service" ];                                      
    requires = [ "network-online.target" "network-setup.service" "sshd.service" ];                                   
    unitConfig = {                                                                                                   
      AllowIsolate = "yes";                                                                                          
    };                                                                                                               
  };                                                                                                                 

  # ---- Stage 1 -----
  # boot.supportedFilesystems = [ "zfs" ];
  # networking.hostId = "9e16a79b"; # generate a random Id
  # ------------------
  
  # ---- Stage 2 -----
  # fileSystems."/"     = { fsType = "zfs"; device = "zpool/root"; };                                                                                                                                                                      
  # fileSystems."/boot" = { fsType = "zfs"; device = "zpool/root/boot"; };                                                                                                                                                                 
  # fileSystems."/etc"  = { fsType = "zfs"; device = "zpool/root/etc"; };                                                                                                                                                                  
  # fileSystems."/nix"  = { fsType = "zfs"; device = "zpool/root/nix"; };                                                                                                                                                                  
  # fileSystems."/tmp"  = { fsType = "zfs"; device = "zpool/root/tmp"; };                                                                                                                                                                  
  # fileSystems."/var"  = { fsType = "zfs"; device = "zpool/root/var"; };                                                                                                                                                                  
  # ------------------
}

migrate_raid_to_zfs.md

/!\ This setup was tested successfully in a VM, I didn’t do it in production yet! Use at your own risk

Context

We want to migrate a RAID1 array to ZFS on a distant server (without physical access).

We have /dev/sda3 and /dev/sdb3 in RAID1 containing the whole root system. A (temporary) spare disk /dev/sdc is available with at least the size of /dev/sda

Deployment is usually done via nixops, but configuration should not be too different with local configuration.nix

Instance: Add the new disk to RAID array and remove /dev/sda3

# Copy partitionning
sfdisk -d /dev/sda | grep -v ^sector-size: | sfdisk /dev/sdc
# Add the new disk to RAID1 array
mdadm --grow /dev/md0 --level=1 --raid-devices=3 --add /dev/sdc3
# Wait until it is synchronized
cat /proc/mdstat
# (...)
# Remove /dev/sda3 from array
mdadm /dev/md0 --fail /dev/sda3 --remove /dev/sda3
mdadm --grow /dev/md0 --raid-devices=2

Host: Add zfs configuration

Uncoment Stage 1 in configuration.nix, and run nixops deploy (nixos-rebuild switch)

Instance: Create zpool

zpool create -f zpool sda3
zfs create zpool/root
zfs create zpool/root/boot
zfs create zpool/root/nix
zfs create -o xattr=sa -o acltype=posixacl zpool/root/var
zfs create -o xattr=sa -o acltype=posixacl zpool/root/tmp
zfs create -o xattr=sa -o acltype=posixacl zpool/root/etc

# This will only synchronize rootfs
rsync -a --one-file-system / /zpool/root/

Host: Add filesystems and apply partially

Uncomment Stage 2 in configuration.nix and run nixops deploy --dry-activate (nixos-rebuild dry-activate)

Instance: Go in degraded mode and finalize migration

Above stages can be still run concurrently with a running system. From there, it must be all be ran in one go

# Stop every services
systemctl isolate maintenance.target
systemctl stop systemd-journald

# Resynchronize whatever changed since the first time
rsync -a --one-file-system / /zpool/root/

# Switch the system to new array for next boot (!).
# All commands below must be run safely, otherwise the system may not boot properly! (they are all quick to run)
nixos-enter --root /zpool/root/ -- /nix/var/nix/profiles/system/bin/switch-to-configuration boot
zfs unmount zpool/root
zfs set mountpoint=legacy zpool/root
zfs set mountpoint=legacy zpool/root/boot
zfs set mountpoint=legacy zpool/root/nix
zfs set mountpoint=legacy zpool/root/var
zfs set mountpoint=legacy zpool/root/tmp
zfs set mountpoint=legacy zpool/root/etc

# phew. From there the risk is no more boot,
# but possible data loss if (1) the server reboots before synchronization is complete AND (2) writes data to disk AND (2) sda3 dies

# Remove sdb3 from raid array
mdadm /dev/md0 --fail /dev/sdb3 --remove /dev/sdb3

# Attach sdb3 as mirror in zpool
zpool attach -f zpool sda3 sdb3

# Wait until resynchronisation is finished (might be long)
zpool status

# Reboot!
shutdown -r now