ubnt1604zor.md

Ubuntu 16.04 with root on ZFS

This is a guide to install ubuntu 16.04 minimal on a ZFS root partition. It supports EFI or MBR (bios) booting, and has been tested (too many) times on VMWare workstation, but it should work anywhere. Note, you will need to install using a Live-CD: I'd recommend using using Lubuntu because it's lightwieght. You could also install a stock copy of ubuntu on to a USB stick and use that.

To start, we are going to install and set up SSH access to the liveCD so that we can copy and paste things easier. I made a little script to install needed stuff and enable SSH. To run it easier, run the following command:

wget -O bootstrap https://img.sharo.se/YdmjT.txt; sudo bash bootstrap

Ok, now connect to the liveCD over ssh (The script will tell you the IP and randomly generated password unless you provided one yourself). We'll now start to actually get everything set up, starting with some environment variables we'll set.

The disk we'll be partitioning and installing to. /dev/disk/by-id only please! If you just use /dev/sdX linux has a chance to change those paths on boot, causing your pool to not mount.

export SELECTEDDISK=/dev/disk/by-id/mydisk

The hostname for the new install. testzor stands for TEST Zfs On Root, if curious.

export HOSTNAME=testzor

This can be the same as the hostname, but I usually prefer something like os-hostname.

export POOLNAME=os-testzor

Size of the partition the pool will use. I keep mine small so I can use the rest as swap or write cache. If you want to use as much as possible, just set this to 0.

export PARTSIZE=10GB

The timezone of the installed server. See: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

export TIMEZONE=Canada/Newfoundland

You can set up a preferred mirror here. http://mirrors.ubuntu.com/mirrors.txt shows a list of mirrors in your country.

export APTSERVER=http://archive.ubuntu.com/ubuntu

Ok, now that all that config is all done, let's start actually running commands. Remember how we work for EFI or BIOS booting? Well, pick now. As a side note, it actually uses GPT partitioning either way. Please note for EFI mode to work, you need to boot the liveCD into EFI mode. Otherwise it can't install grub. Check with the following snippet:

[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS

BIOS Edition:

sgdisk -Z $SELECTEDDISK
sgdisk -a1 -n1:34:2047 -t1:EF02 $SELECTEDDISK
sgdisk -n2:2048:+$PARTSIZE -t2:A504 $SELECTEDDISK

EFI Edition:

sgdisk -Z $SELECTEDDISK
sgdisk -a1 -n1:1M:64MB -t1:EF00 $SELECTEDDISK
sgdisk -n2:0:+$PARTSIZE -t2:A504 $SELECTEDDISK

Ok, now we'll create the pool that your OS will install on. This is gonna be seperate just for the fact that if there are any errors, you want to see them.

zpool create -o ashift=12 -O atime=off -O canmount=off -O compression=lz4 -O mountpoint=/ -R /mnt $POOLNAME $SELECTEDDISK-part2

Now, I'm going to make seperate datasets for things like /home, /root, etc so that it's easier to roll back the OS without rolling back data or vice versa. If you don't want to, and want it all on one dataset, just skip the optional parts.

These parts are required.

zfs create -o canmount=off -o mountpoint=none $POOLNAME/ROOT
zfs create -o canmount=noauto -o mountpoint=/ $POOLNAME/ROOT/ubuntu
zfs mount $POOLNAME/ROOT/ubuntu

And these are the optional datasets.

zfs create -o setuid=off $POOLNAME/home
zfs create -o mountpoint=/root $POOLNAME/home/root
zfs create -o canmount=off -o setuid=off -o exec=off $POOLNAME/var
zfs create -o com.sun:auto-snapshot=false $POOLNAME/var/cache
zfs create $POOLNAME/var/log
zfs create $POOLNAME/var/spool
zfs create -o com.sun:auto-snapshot=false -o exec=on $POOLNAME/var/tmp

Ok, now we are going to make sure we chmod /var/tmp so that nothing breaks, and actually install ubuntu on to the pool. This will take a while, since it's basically re-downloading the entire OS.

mkdir -p /mnt/var/tmp # Just in case you didn't make the datasets, this one folder we DO need.
chmod 1777 /mnt/var/tmp
debootstrap xenial /mnt

Ok, now let's quickly configure a bunch of stuff.

zfs set devices=off $POOLNAME
echo $HOSTNAME > /mnt/etc/hostname
echo 127.0.1.1 $HOSTNAME >> /mnt/etc/hosts
echo $TIMEZONE > /mnt/etc/timezone
ln -fs /mnt/usr/share/zoneinfo/${TIMEZONE} /mnt/etc/localtime
export INTERFACE=$(ip -o addr show scope global | awk '{gsub(/\/.*/,"",$2); print $2}')
echo -e "auto $INTERFACE\niface $INTERFACE inet dhcp" > /mnt/etc/network/interfaces.d/$INTERFACE
echo deb $APTSERVER xenial main universe > /mnt/etc/apt/sources.list
echo deb-src $APTSERVER xenial main universe >> /mnt/etc/apt/sources.list
echo deb $APTSERVER xenial-security main universe >> /mnt/etc/apt/sources.list
echo deb-src $APTSERVER xenial-security main universe >> /mnt/etc/apt/sources.list
echo deb $APTSERVER xenial-updates main universe >> /mnt/etc/apt/sources.list
echo deb-src $APTSERVER xenial-updates main universe >> /mnt/etc/apt/sources.list
mount --rbind /dev  /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys  /mnt/sys
chroot /mnt /bin/bash --login

Now we are all chrooted into our new install. Before we continue, let's set a password. You can also set up a new user and add them to the sudoers list, but I'm not gonna bother.

passwd

Now let's set up a locale, and update apt. I'm just going to set it to en_US, you can change it if you want.

locale-gen en_US.UTF-8
echo 'LANG="en_US.UTF-8"' > /etc/default/locale
ln -s /proc/mounts /etc/mtab
dpkg-reconfigure --frontend noninteractive tzdata
apt-get update

Now I'm going to install the linux image. It uses the image version that the LiveCD uses so that nothing breaks: I had issues with it downloading a newer one before rebooting. Include the extra image because the default image is missing some helpful drivers, like less common networking, or usb keyboards.

apt-get install --yes --no-install-recommends linux-signed-image-$(uname -r) linux-image-extra-$(uname -r)

Now let's install ZFS and some other helper utilities. I'm including nano here because I hate vi with a passion.

apt-get install --yes ubuntu-minimal nano zfs-initramfs

You should, at this point, be able to run zpool status and see your root pool. If not? You've got problems. Now, let's actually install grub so we can boot this sucker. As usual, there's a BIOS version and EFI. Pick the same one as before.

BIOS Edition:

Please note to MAKE SURE you select the drive root (ex sda), not the second partition (ex sda2) when installing!

apt-get install --yes grub-pc

EFI Edition:

apt-get --yes install dosfstools
# Make sure you hold on here so commands don't get lost!
mkdosfs -F 32 -n EFI $SELECTEDDISK-part1
mkdir /boot/efi
echo PARTUUID=$(blkid -s PARTUUID -o value $SELECTEDDISK-part1) /boot/efi vfat defaults 0 1 >> /etc/fstab
mount /boot/efi
modprobe efivars
apt-get install --yes grub-efi-amd64-signed efibootmgr fwupdate-signed shim-signed

Let's make sure grub can see your pool.

grub-probe / # Should return 'zfs'
# Now update initramFS.
update-initramfs -c -k all

Now, just in case grub gives us any problems, we are going to make some quick edits to it's config for debugging. We can change this later, or even skip the step: it's not needed. Here's some sed commands for speed.

sed -i 's,^\(GRUB_HIDDEN_TIMEOUT=0\),#\1,' /etc/default/grub
sed -i 's,^\(GRUB_CMDLINE_LINUX_DEFAULT=\).*,\1"",' /etc/default/grub
sed -i 's,^\(#GRUB_TERMINAL=console\),GRUB_TERMINAL=console,' /etc/default/grub

Now let's update grub's configuration.

update-grub

And now install the bootloader. BIOS and EFI versions below.

BIOS Edition:

grub-install $SELECTEDDISK

EFI Edition:

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck --no-floppy

Awesome, all done. Now, just install whatever else you might need, just openssh-server, which your new install of ubuntu doesn't actually have at the moment, and reboot.