Create an Ubuntu 18.04 droplet, and use the smallest SSD/HDD size. This way you can make a snapshot of the droplet and have the ability to later deploy that snapshot into a droplet of any size.
In this example, I use the "Optimized Droplet" that is 20GB, since that's the smallest one currently available.
After creating the droplet, power it down and open a ticket with DigitalOcean support, and ask to enable the Ubuntu 18.04 recovery ISO. It can take an hour or two before they do this. By the time you read this guide, hopefully they will have an automated way of doing it.
Once they get back to you, power your droplet back up and choose the recovery ISO's terminal/console option to get a shell. Now shrink the ext4 root to make room for the zfs root:
e2fsck -f /dev/vda1
resize2fs /dev/vda1 1600M
The ext4 shrinking process is done, but you still need to resize the partition table entry using parted. To do this properly, you'll need the block size/count of the disk, sector size of the disk, and the start sector of the ext4 partition.
Get the block size and block count using this command:
tune2fs -l /dev/vda1 | grep "Block"
Also run this command:
parted -m /dev/vda unit s print
to get the sector size and start sector of the ext4 partition. I've **double starred** the sector size and start sector in the output below:
/dev/vda:52428800s:virtblk:**512**:512:gpt:Virtio Block Device:;
14:2048s:10239s:8192s:::bios_grub;
15:10240s:227327s:217088s:fat32::msftdata;
1:**227328s**:3504127s:3276800s:ext4::;
Now plug the correct values from the tune2fs and parted commands above into the first 4 lines of the python script below and run the python script:
block_count = 409600
block_size = 4096
sector_size = 512
start_sector = 227328
fs_size = block_count * block_size
fs_sectors = fs_size/sector_size
part_sectors = ((fs_sectors-1)/2048+1)*2048
end_sector = start_sector + part_sectors - 1
print "Partition start: %d end: %d size: %d"%(start_sector,end_sector,part_sectors)
print "Resize in parted with: \nresizepart <number> %ds"%(end_sector)
This will tell you the exact command to run to change the partition table entry.
When I run this, this script outputted resizepart <number> 3504127s, so I run it like below. For <number>, use the partition number of the ext4 partition, which should be 1:
resizepart 1 3504127s
Your ext4 partition should now be shrunk down, leaving you plenty (18GB or so) of empty space for your zfs root.
Now make the zfs partition. I've noticed that DO may just revert the ext4 partition to its full size again if we don't do this now.
sgdisk -n2:0:0 -t1:BF01 /dev/vda
Now let's stop using the recovery ISO and boot back into the ext4 root to make sure it still works. We can also finish the rest of the zfs-on-root process there.
To stop using the Recovery ISO and boot back into the ext4 root, you have a choice: a) Open a DO support ticket and ask to get off of the recovery ISO. Power down and boot back up and you'll be back into your ext4 root, albeit a shrunken one. With this method, you'll need to wait the 1-2 hours for DO to respond to your ticket. b) Power down, create a snapshot, and spin up a new droplet based on that snapshot. The new droplet won't have the recovery ISO mounted anymore, but will still contain your work so far. At that point you can just destroy the old droplet. This saves some time, so it's what I do, but if you're not familiar with DigitalOcean snapshot creation and deployment, it might be trickier.
You should now be booted up to your droplet and have a shrunken (1.6GB) ext4 root partition. 'parted -l' should show something like:
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
14 1049kB 5243kB 4194kB bios_grub
15 5243kB 116MB 111MB fat32 msftdata
1 116MB 1794MB 1678MB ext4
2 1794MB 21.5GB 19.7GB
Now install the zfs packages, create the pool/dataset, copy the existing Ubuntu 18.04 installation into the new zfs root, then finally chroot into it.
apt update
apt install --yes debootstrap gdisk zfs-initramfs
zpool create -o ashift=12 -O atime=off -O canmount=off -O compression=lz4 -O normalization=formD -O mountpoint=/ -R /mnt zp /dev/vda2
zfs create -o canmount=off -o mountpoint=none zp/ROOT
zfs create -o canmount=noauto -o mountpoint=/ zp/ROOT/ubuntu1804
zfs mount zp/ROOT/ubuntu1804
rsync -aHAXxvP / /mnt/
rm -rf /mnt/boot/*
mount --rbind /dev /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys /mnt/sys
mount --rbind /run /mnt/run
chroot /mnt /bin/bash --login
Note: I've created a very simple dataset scheme here, but feel free to do something more advanced.
Now that we're chrooted into our new zfs root, we'll create a mount point for the old ext4 partition, because we will be reusing its /boot directory.
mkdir /ext4root
Note: Though you may be tempted to try to get rid of the old ext4 root, I've found it's very important to keep it around, because: a) Booting a zfs root without an ext4-based /boot gets tricky on DigitalOcean (I haven't successfully done so). b) The DigitalOcean snapshot deployment process will error out if the ext4 partition doesn't exist. Meaning, without the ext4 in place, you will not be able to deploy a new droplet based on a DO snapshot that you create using this guide. This limitation is significant, since you'd have to go through this entire guide every time you want to spin up a new DO instance with a ZFS root.
Edit the /etc/fstab file and revise it so that the old ext4 root is mounted to /ext4root and /boot is bind mounted to /ext4root/boot Your fstab should look something like:
LABEL=cloudimg-rootfs /ext4root ext4 defaults 0 0
/ext4root/boot /boot none defaults,bind 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
Note: Line ordering is important!
Now run:
mkdir /ext4root
mount -a
Run mount and confirm that the output is sane. ls /boot and confirm that the bind mount worked by ensuring the kernel and initramfs files are there.
I suggest going into /etc/default/grub and setting GRUB_HIDDEN_TIMEOUT line to:
GRUB_HIDDEN_TIMEOUT=1.0
This gives you a chance to get into grub if you need to, but doesn't slow down the reboot process by much.
Now run the various grub commands:
apt install --yes grub-pc # Some form of grub is already installed on the 1804 image that DO provides, so this is maybe not necessary? I didn't want to re-do this guide in order to find out, so feel free to find out for yourself and propose an edit to the guide based on that.
grub-install /dev/vda2
update-grub
update-initramfs -u -k all # Also not sure if this is strictly necessary.
After this, you can confirm that your /boot/grub/grub.cfg lines reference the zfs root, for example I have many lines such as:
/boot/vmlinuz-4.15.0-20-generic root=ZFS=zp/ROOT/ubuntu1804 ro recovery nomodeset
Exit the chroot and make a snapshot if you wish:
zfs snapshot zp/ROOT/ubuntu1804@install
Unmount and export:
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export zp
Now reboot your droplet and you should be in your new zfs root! At this point you can (and I would recommend) powering down and creating a DigitalOcean snapshot from your droplet. This will allow you to easily deploy a zfs root DigitalOcean instance at any time instead of needing to redo this entire guide every time.
If you deploy your new zfsroot DO snapshot to a droplet that's larger than the one you used to create the original snapshot, you'll have to grow your pool.
An example of what this might look like follows, where I expanded the pool to fill a 50GB disk:
$ parted /dev/vda
(parted) resizepart
Partition number? 2
End? [21.5GB]? 53.7GB
Now enable autoexpand on the pool and re-online it to cause the expansion:
zpool set autoexpand=on zp
zpool online -e zp /dev/vda2
That's it! Your pool should now be expanded:
# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zp 48.2G 648M 47.6G - 0% 1% 1.00x ONLINE -
These days you can easily switch between recovery and regular boot which makes it much easier. But there's another shortcut you can take. Instead of resizing the volume after creation you can do the following:
sgdiskdisk step.