zfs.md

Setting up root on ZFS with EFI boot. Can't use MBR boot because GRUB doesn't like ZFS with Debian Jessie.

1. Grab Debian live CD ("iso-hybrid") 64-bit; username user password live

2. (optional) set sshd_config to allow password login, ssh into live environment - easier than a console window when you can copy-paste

3. Install ZoL in live environment

   sudo -i
   wget http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_6_all.deb
   dpkg -i zfsonlinux_6_all.deb
   apt-get update
   apt-get install linux-image-amd64 debian-zfs
   modprobe zfs
   

4. Partition boot device: gpt, with EFI system partition of at least 200 MB (and no more than 400 MB on SuperMicro motherboards).

   1. fdisk

5. Format boot partition as FAT32: mkdosfs -F 32 /dev/xxx

6. Create zpool:

   1. RAID-Z: zpool create -o ashift=12 tank raidz1 /dev/disk/by-id/... replacing ... with a glob representing your disk(s)
      • OR RAID-1: zpool create -o ashift=12 tank mirror /dev/disk/by-id/... replacing ... with a glob representing your disk(s)
   2. Optional read cache (L2ARC) and write cache (ZIL):
      • zpool add tank log mirror /dev/disk/by-id/... for ZIL - must be less than 50% of RAM size
      • zpool add tank cache /dev/disk/by-id/... for L2ARC - can be any size, pure read performance if this is SSD-backed
   3. If you want LZ4 compression (recommended for hard drives - higher CPU usage during I/O but higher throughput): zfs set compression=lz4 tank

7. Create root fs

   1. zfs create tank/root

8. zfs umount -a

9. zfs set mountpoint=/ tank/root

10. zpool set bootfs=tank/root tank

11. zpool export tank

12. zpool import -d /dev/disk/by-id -R /mnt tank

13. mkdir -p /mnt/boot/efi

14. mount /dev/xxx /mnt/boot/efi replacing xxx with a partition (not raw disk) where your EFI System Partition will be

15. apt-get install debootstrap

16. debootstrap jessie /mnt

17. cp /etc/hostname /etc/hosts /mnt/etc/

18. Add to /mnt/etc/fstab:

    1. /dev/disk/by-id/xxxbootpartition /boot/efi auto defaults 0 1 replacing xxxbootpartition with your EFI System Partition

19. Add interfaces to /mnt/etc/network/interfaces, e.g.:

    auto lo
    

iface lo inet loopback auto eth0 iface eth0 inet dhcp 20. Chroot into new system: `systemd-nspawn -D /mnt /bin/bash --login` 21. Install ZFS on new system - it must compile again apt-get install locales locale-gen en_US.UTF-8 wget http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_6_all.deb apt-get install lsb-release dpkg -i zfsonlinux_6_all.deb apt-get update apt-get install linux-image-amd64 debian-zfs apt-get install zfs-initramfs apt-get dist-upgrade ```

22. Add boot support files

    1. Make /etc/kernel/postinst.d/zz-update-efistub:

       #!/bin/sh
       cp /vmlinuz /initrd.img /boot/efi/EFI/debian/
       

    2. chmod +x /etc/kernel/postinst.d/zz-update-efistub

    3. mkdir -p /boot/efi/EFI/debian

    4. /etc/kernel/postinst.d/zz-update-efistub

    5. Also add to /etc/initramfs/post-update.d/

23. Set root password (passwd root)
24. (optional) snapshot new root zfs snapshot tank/root@cleaninstall
25. exit from chroot env
26. zfs umount -a
27. umount -l /mnt
28. zpool export tank

ZFS now installed. Next: set up boot

29. Boot into live CD with EFI support (Debian Jessie Live does not). Suggestion: Lubuntu. Must be 64-bit. Username lubuntu no password
30. Install efibootmgr
31. Add boot entry: efibootmgr -d /dev/xxx -c -g -L "Debian (EFI stub)" -l '\EFI\debian\vmlinuz' -u "add_efi_memmap boot=zfs rpool=tank bootfs=tank/root root=ZFS=tank/root initrd=\\EFI\\debian\\initrd.img ro"
    • Add vmwgfx.enable_fbdev=1 required to work around plymouth blank screen bug in jessie on VMware
    • Add quiet to suppress startup messages. Unnecessary and not recommended especially for servers.
    • Add security=apparmor to enable AppArmor, which is a requirement for LXD.
32. Boot! (and pray)