ZFSBootMenu is a bootloader for Linux kernels residing in a ZFS pool. It basically searches your ZFS pools for datasets that look like a Linux root filesystem and it presents them to you as options to boot from (source). Once an OS has been selected to boot, it lets go of the pool and boots the OS, at which point it's the OS's responsibility to mount the ZFS pool(s) (source).
All this tool does it provide a convenient means to load up your kernel from a zpool and boot it, that's it (and some nice pre-boot tools).
For Linux to boot from a ZFS root, it needs to have the ZFS module loaded very early in the boot process, before the filesystem is required. Each distro will have its own way of getting ZFS module to load at boot and it will also have its own implementation as to and what datasets are mounted.
Dracut is a popular initramfs generator for this and OpenZFS even has support for by means of a hook.
This hook will first import the pool (without mounting any datasets) then mount the boot/root dataset, which must have a mountpoint of "/".
It will then mount some subset of root dataset's children which it deems essential.
It expects some other tool to mount any remaining datasets after the initial boot is complete.
The ArchZFS repos provides 3rd party tools for using ZFS on Arch Linux as there is no official support due to Licensing issues. It provides a mkinitcpio hook to mount the root dataset at boot. Like dracut, it also imports the pool without mounting any datasets, then mounts the root dataset, then mounts all the children that ZFS would've automatically mounted (source). It also expects some other tool to mount any remaining datasets (outside the root dataset) after the initial boot is complete.
Once you've created your ZFS pool (which I will henceforth refer to as zroot),
you'll want to carve out some datasets for your setup.
The simplest (but conventional) dataset layout for boot environments (source) is:
zrootROOT(canmount=off, mountpoint=none) This is just a container dataset.debian(canmount=noauto, mountpoint=/) This is the root (/) dataset for the Debian boot environment.arch(canmount=noauto, mountpoint=/) This is the root (/) dataset for the Arch Linux boot environment.- ...
In this layout you have some root datasets representing different isolated boot environments. Each boot environment could be a different distro, different versions of a distro, same version of a distro but with different software suites (e.g. X vs Wayland), really anything you want.
This and any further dataset layouts described here are just conventions/recommendations, feel free to break or go against them according to your needs. The only vital thing to consider when choosing a layout is to make sure your bootloader (e.g. ZFSBootMenu, Grub, rEFInd, etc), ZFS boot module (e.g. Dracut, ArchZFS, etc) and init system (e.g. systemd, runit, etc) (across all boot envs) support your chosen layout. Every time you add a boot env, support for your layout needs to considered again.
ZFS allows you to take snapshots of datasets and rollback your filesystem to these snapshots. For boot environments, this can be useful for undoing problematic system updates. If after you perform an update, you discover a new bug, your GPU driver starts misbehaving, a feature you were relying on got removed, your distro "broke" ZFS support in the kernel, etc. You can just rollback your filesystem to before this update happened (assuming you took a snapshot before performing the update).
Whilst this is possible with the simple layout above, when a rollback is performed, any user file changes you made after the snapshot will be rolled back as well. If you made any code changes in your text editor, saved your progress in a video game, had a database updated by a webserver, etc. All of these file updates will be lost in the rollback, which is probably not what you want.
To prevent this, you can move any directories containing "user data" to a different dataset which will remain untouched when you roll back the root dataset.
Some directories to consider moving to a separate dataset:
/home- This is almost entirely user data. (Keep in mind the awkward problem of backward incompatible software upgrades)/tmp/var/log- No reason to undo logs in the rollback. They can be handy to see the difference in software behaviour./var/spool/var/cache/var/tmp/var/mail- Some software stores emails here./var/snap- If you use Snap packages/var/lib/AccountsService- If you use Gnome./var/lib/docker- If you use docker./usr/local
This list is far from exhaustive and this consideration should be made for any software you install in your boot env. Also see Filesystem Hierarchy Standard and File Hierarchy.
TODO
By default only root can perform admistrative tasks on the ZFS pool and dataset,
but in practice you want some non-root users to be able to run some commands on their datasets, like take snapshots and sending their datasets.
ZFS Allow
Permissions to delegate to a user for their home directory. zfs allow -u <user> perm1,perm2,... zroot/home/<user>
bookmark- Useful for snapshots andsending.hold,release- Useful for snapshots.snapshot,rollback,diff- Useful for snapshots.send,receive- Useful for backupscreate,mount- Useful for creating child datasets under their home directory. (Can be quota'd)
Additional permissions to delegate to user's child datasets. zfs allow -c <user> perm1,perm2,... zroot/home/<user>
destroy,rename,clone,promote- Useful for removing child datasets.canmount- Useful for organisation.compression,copies,recordsize- Useful for tweaking.load-key,change-key- Useful for encryption