Skip to content

Instantly share code, notes, and snippets.

@naftulikay
Created October 25, 2021 19:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save naftulikay/3398cce7e25762eb675e50b0ed5de7a6 to your computer and use it in GitHub Desktop.
Save naftulikay/3398cce7e25762eb675e50b0ed5de7a6 to your computer and use it in GitHub Desktop.

I have observed this issue as well.

Forgive, me, I'll be documenting a lot of what I've done in order to benefit future users and also so that I have a canonical source on how I do this as I'll be installing Odin on two other systems shortly.

⚠️ NOTE: I'm terribly sorry for how long this is, there is a summary at the bottom, but I needed to provide context for everything and discuss how things used to work in order to discuss how things "work" now.

If it would be preferable for me to host all of this in a Gist, I can do that, but sorry for such a long answer. You might also be wondering how I know all of this, and all I can say is

History 📖

I have a complicated disk setup, not using LVM but using BTRFS for multiple subvolumes (with compression) so that I reduce the amount of read/write on the device and increase the amount of storage space. I won't go into that right now, but it's something I've been doing for a while and ubiquity (the previous, Ubuntu installer), would actually allow you to do what you're discussing above, elementary OS Odin does not. I'd also usually pass a flag to tell it to not install a bootloader, as I like to use rEFInd, which is much prettier and nicer to work with than grub.

Disk Setup

Note that when I use $LUKS_UUID, I mean the output of blkid -s UUID -o value /dev/nvme0np3); when I open the device, I give it a name luks-$LUKS_UUID, yielding a device path of /dev/mapper/luks-$LUKS_UUID; when I use $ROOT_UUID, I mean the internal BTRFS filesystem UUID acquired via blkid -o UUID -s value /dev/mapper/luks-$LUKS_UUID; I also assume EFI, so $EFI_UUID is the FAT32 partition's UUID for /dev/nvme0np1, and I assume that a second partition exists for hosting the initramfs and other boot stuff at /dev/nvme0n1p2 (filesystem choice largely irrelevant but I use BTRFS) and I'll use $BOOT_UUID to reference its UUID.

The way to get it working before was:

  1. Setup the dm-crypt/LUKS/cryptsetup volume at /dev/nvme0n1p3

    cryptsetup luksFormat --cipher aes-cbc-essiv:sha3-256 --hash sha3-256 \
        --iter-time 5000 --pbkdf argon2id ... /dev/nvme0n1p3
    
  2. Unlock the device using:

    cryptsetup open /dev/nvme0n1p3 luks-$LUKS_UUID
    
  3. Create the inner BTRFS filesystem:

    mkfs.btrfs -L my-computer-name /dev/mapper/luks-$LUKS_UUID
    
  4. Mount the BTRFS root:

    mount /dev/mapper/luks-$(blkid-s UUID -o value) /mnt
    
  5. Create subvolumes:

    btrfs subvolume create /mnt/@           # mount point: /
    btrfs subvolume create /mnt/@home       # mount point: /home
    btrfs subvolume create /mnt/@var        # mount point: /var
    btrfs subvolume create /mnt/@virtualbox # mount point: /virtualbox
    

    Ignore the VirtualBox mount point, VirtualBox has always had problems with BTRFS, so I symlink the disk image directory to this mount point and mount with nodatacow.

  6. Unmount:

    umount /mnt
    
  7. Run the installer without installing a bootloader.

The old installer would allow you to do this, it would allow you to use the /dev/mapper/luks-$LUKS_UUID as / and Ubuntu used the same subvolume names as listed above so it played fairly nicely.

chroot and Configure the System

After running the installer, it then became necessary to chroot into the machine's filesystem, so:

# btrfs subvolumes
mount -o defaults,subvol=@,compress=lzo /dev/mapper/luks-$LUKS_UUID /mnt
mount -o defaults,subvol=@var,compress=lzo /dev/mapper/luks-$LUKS_UUID /mnt/var
mount -o defaults,subvol=@home,compress=lzo /dev/mapper/luks-$LUKS_UUID /mnt/home
mount -o defaults,subvol=@virtualbox,compress=lzo /dev/mapper/luks-$LUKS_UUID /mnt/virtualbox
# unencrypted boot partition containing initramfs etc.
mount /dev/nvme0n1p2 /mnt/boot
# the EFI volume
mount /dev/nvme0n1p1 /mnt/boot/efi
# system mount points to ensure chroot works
mount --bind /dev /mnt/dev
mount --bind /run /mnt/run
mount -t proc proc /mnt/proc
mount -t sysfs sysfs /mnt/sys

Bonus points: since the installer most likely did not mount with compression enabled, now would be a good time to compress the filesystem recursively, as doing it in a live system is probably not a great idea, so:

btrfs filesystem defrag -r -v -clzo

This compresses the filesystem using LZO, which is a good middle ground compression algorithm, not too CPU expensive, with decent compression overall.

Next: chroot /mnt and if you did everything correctly, you'll now be living virtually inside of your install, so you'll be able to do customization here.

chroot

chroot /mnt

⚠️ From this point on, you are now inside your elementary OS install. I'll let you know when to exit it.

Setup /etc/fstab

The installer never quite got /etc/fstab right, so install an editor and edit /etc/fstab:

UUID=$BOOT_UUID /boot btrfs defaults 0 2
UUID=$EFI_UUID /boot/efi vfat umask=0077 0 1

/dev/mapper/luks-$LUKS_UUID / btrfs defaults,compress=lzo,subvol=@ 0 1
/dev/mapper/luks-$LUKS_UUID /home btrfs defaults,compress=lzo,subvol=@home 0 2
/dev/mapper/luks-$LUKS_UUID /var btrfs defaults,compress=lzo,subvol=@var 0 2
/dev/mapper/luks-$LUKS_UUID /virtualbox btrfs defaults,nodatacow,subvol=@virtualbox 0 2

Be sure to replace those variables with your actual UUIDs.

/etc/crypttab

I wasn't exactly sure how /etc/crypttab worked, it usually didn't seem to do anything at all for me, but I'd always set it up anyway perhaps out of paranoia:

luks-$LUKS_UUID /dev/nvme0n1p3 none luks

Install rEFInd (optional)

Install the rEFInd PPA if you want a nice graphical boot manager that's easy to customize:

apt install -y software-properties-common
add-apt-repository -y ppa:rodsmith/refind
apt update
apt install -y refind

Ensure Crypto Modules in initramfs

After doing this, the initramfs would usually not contain the dm-crypt and cryptsetup modules, which was super frustrating to debug, but I'd usually go and edit /usr/share/initramfs-tools/hooks/cryptroot and force the hook to install the cryptsetup dependencies. The hooks have changed significantly, so this did not seem necessary in my Odin install, but on all of the previous elementary versions, I had to do this; I'd usually add an echo statement to the hook to emit a simple message indicating that it was, in fact, adding the modules to the initramfs.

Rebuild the initramfs via update-initramfs -u -k all.

At this point your initramfs and your actual machine config should be able to boot. However, we're not done, we need to update the kernel command-line to use arguments that were necessary to ensure the kernel knew that it needed to decrypt the root filesystem.

Updating the Kernel Command Line

Note: I'm using rEFInd, you're probably using grub, so you'll need to look up the way to edit the grub kernel command line. I've had correspondence with Rod Smith and he's been extremely helpful in the past, can't recommend the bootloader enough, I wish it could be the default bootloader but everybody still uses grub.

To set the kernel command-line in rEFInd, you'll need to edit /boot/refind_linux.conf. Each line here represents a different boot option. Specifically, we'll need to set the following:

  • root, which indicates to the kernel where the root filesystem should be found; our root filesystem is inside of a LUKS container, so if you only pass this, the kernel won't be able to find your root.
  • rootflags=subvol=@, which tells the kernel to pass subvol=@ when trying to mount the root filesystem; this makes sure that we're using the correct subvolume at boot time.
  • cryptopts=target=luks-$LUKS_UUID,source=UUID=$LUKS_UUID, which tells the kernel to look for a partition with the $LUKS_UUID as its identifier, and to open it as luks-$LUKS_UUID.

Remember to replace the variables as you edit /boot/refind_linux.conf:

"Normal"   "ro root=UUID=$ROOT_UUID rootwait rootflags=subvol=@ cryptopts=target=luks-$LUKS_UUID,source=UUID=$LUKS_UUID,options=luks quiet splash vt.handoff=7"
"Recovery" "ro root=UUID=$ROOT_UUID rootwait rootflags=subvol=@ cryptopts=target=luks-$LUKS_UUID,source=UUID=$LUKS_UUID,options=luks quiet splash"

💪 Great! Now we should be able to boot, provided we got everything right. I often screw things up so I have to enter the installer, setup the chroot, try to fix things, update-initramfs -u -k all and iterate. Try booting the recovery option to see more of what the kernel is doing if you don't get boot right away.


New Sauce ✨

Okay, so, as mentioned at the start, elementary OS Odin does not use ubiquity anymore, and from what I've seen, you basically cannot tell elementary OS Odin to do full-disk encryption or even user home directory encryption, which was unacceptable to me, at least. I lost about 8-10 hours trying to do the above with the Odin installer, and no dice, but I do have a booting system with full-disk encryption using LUKS.

In the end, and I really hate that I had to do this, but I essentially created a stage-3 ala Gentoo by destroying my LUKS container and having it create an unencrypted filesystem on /dev/nvme0n1p3. Still within the installer:

  1. rsync everything in /dev/nvme0n1p3 to an external disk.

  2. Create the LUKS container.

  3. Unlock it.

  4. Create the BTRFS partition.

  5. Mount it, create the BTRFS subvolumes, unmount it.

  6. Mount /, /boot, /boot/efi, /var, /home.

  7. rsync from the external disk to where you mounted your root, which so far has been /mnt, so

    rsync -av /media/elementary/my-external-disk/elementary-odin-stage-3/ /mnt/
    
  8. Setup the chroot with the system mount points, /dev, /proc, /sys, /run.

  9. Enter the chroot.

  10. Setup /etc/fstab.

  11. Setup /etc/crypttab.

  12. Install rEFInd (optional).

  13. update-initramfs -u -k all

  14. Setup kernel command line options.

  15. Pray Hope Reboot and just see what happens.

If you were able to do all of this, you should be able to get to boot 🎉, though I had to boot recovery a few times to fix various issues.

The Broken ❌

Well, Ubuntu has changed the way it does kernel command line arguments for LUKS, and uses a new shiny systemd configuration in the early initramfs stages to setup your LUKS container. The cryptopts argument seems outdated, and seems to protest using what I previously used, and so that led me down this rabbit trail:

There are new options starting with luks* that seem to supersede the old cryptopts parameter. I haven't gotten these working yet, but the man page does direct us to another interesting place:

It seems that what is implied here is that the actual initramfs is configured when it is created/updated that /etc/crypttab is finally actually used to embed configuration into the initramfs, so cryptopts may no longer be necessary, and using the option generates warnings for me at boot time. Maybe it just works, but I am unsure and I haven't tested it out yet, but I likely will do so in VirtualBox to see what is actually necessary.


Summary

In short, I documented above what used to work as a basis for getting things working again in Odin. What I have found so far is that there is no way in the installer to enable full-disk-encryption and I didn't see an option for user home directory encryption either.

As a result, I essentially ran the installer, copied all of the installed filesystem files to an external hard drive via rsync, created my LUKS container and filesystems etc., rsynced into the encrypted container, modified some essential files in a chroot, updated the initramfs, updated the kernel command-line, and I can at least boot now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment