Remote unlocking of native-encrypted ZFS-on-root on Debian/Ubuntu using Dropbear SSH

remote-unlock-zfs-with-dropbear.md

Introduction

The following describes a way to remote unlock a server with natively-encrypted ZFS-on-root. It uses Dropbear SSH to start a SSH server in the initramfs listing on a specified port.

All server commands are executed as root on Debian 10. For Ubuntu, sudo su or adding sudo to all command should be sufficient.

Requirements

The following requires the recent ZoL 2.x, which adds support to Unlock encrypted root filesystem via SSH, making it merely a no-brainer to unlock the data set (besides setting up Dropbear).

ZoL 2.x is available from the debian-backports repository for Debian 10 and should be also available in Ubuntu 21.04 (and will hopefully backported to Ubuntu 20.04).

Update 2022-10-16: Debian Bullseye natively contains ZoL 2.x

Related Articles

• Remotely unlock a LUKS-encrypted Linux server using Dropbear
• LUKS Verschlüsseltes Ubuntu 20.04 remote über SSH entschlüsseln
• How to install LUKS encrypted Ubuntu 18.04.x Server and enable remote unlocking
• https://github.com/dynerose/Remote-unlock-native-ZFS

Installation (Debian/Ubuntu)

1. Install dropbear

   apt install dropbear

2. Please note RSA host key fingerprint during installation (to check it later when SSH-ing into Dropbear for the first time...)

Dropbear Server Configuration

Change port

As Dropbear is using a different private key than OpenSSH (which is pretty wise due to security reasons...), you will likely get scary (and annoying) warnings when SSH-ing into your server due to a possible man-in-the-middle attack. This is simply for the fact, that the server keys are different before and after unlocking the root partition.

Thus, to avoid such warnings, it is best to run Dropbear SSH on another port than OpenSSH.

The config file of the dropbear-initramfs is placed under /etc/dropbear-initramfs/config.

Listen on port 2311

To make Dropbear listing on port 2311, add the following to DROPBEAR_OPTIONS

DROPBEAR_OPTIONS="-p 2311"

Disable "normal" SSH operations

As we just want to unlock our root dataset, the following options should also be added to DROPBEAR_OPTIONS:

DROPBEAR_OPTIONS="-s -j -k -I 60"

with

• -s: Disables password logins -> only keyfile allowed.
• -j: Disables local port forwarding.
• -k: Disables remote port forwarding.
• -I 60: Idle timeout after 60 seconds.

For more information and options, see man dropbear.

Create authorized_keys and allow access for client

Atm dropbear doesn't support Ed25519. Thus, if you're normally using Ed25519, an explicit RSA-based SSH key is necessary for dropbear.

Update 2022-10-16: Debian Bullseye brought support for Ed25519 🥳.

1. Create a Ed25519-Key using 100 derivate function rounds for Dropbear on the client you're using to remote-unlock the server!

   # Generate an Ed25519 key with email as a comment:
   ssh-keygen -t ed25519 -a 100 -C "<YOUREMAIL@DOMAIN>"

2. [LEGACY] RSA-Key Create a 4096-bit RSA-Key for Dropbear on the client you're using to remote-unlock the server

   # Generate an RSA 4096 bit key with email as a comment:
   ssh-keygen -t rsa -b 4096 -C "<YOUREMAIL@DOMAIN>"

3. Add a RSA-Key to dropbears authorized_keys file under /etc/dropbear-initramfs/. Therefore, copy the public key of your client and import it! Create the file if not existing:

   sudo vim /etc/dropbear-initramfs/authorized_keys

(OPTIONAL) Directly call zfsunlock (effectively restrict access)

Add the following to the host entry in the authorized_keys file before ssh-rsa begins, so it looks like this:

no-port-forwarding,no-agent-forwarding,no-x11 forwarding,command="/bin/zfsunlock" ssh-rsa ...

Doing so, zfsunlock will be directly called when SSH-ing into Dropbear SSH. Thus, after ssh server-unlock it will directly ask for the password to unlock the root dataset.

Manage host keys of Dropbear

Delete DSA/ECDSA keys (Update 2022-10-16: and RSA as well!)

DSA and ECDSA are insecure and should thus not be used anymore. To ensure that they could not be used by mistake, delete the DSA and ECDSA keys:

rm /etc/dropbear-initramfs/dropbear_dss_host_key /etc/dropbear-initramfs/dropbear_ecdsa_host_key

Update 2022-10-16: In case you prefer/used ECC (Ed25519), you should delete Dropbears RSA host key as well:

rm /etc/dropbear-initramfs/dropbear_rsa_host_key

Generate Ed25519 host key

Dropbear needs a valid server key. If a host key for the desired algorithm is missing, you need to generate it by yourself:

./dropbearkey -t ed25519 -f dropbear_ed25519_host_key

For more information, see Dropbears README

Update initramfs

Finally, initramfs must be updated to include the recent changes in the Dropbear configuration:

update-initramfs -u -k all

Host-Configuration

As explained above, Dropbear is using a different RSA-Key than OpenSSH (at least in Ubuntu/Debian), which could make accessing the server a bit inconvenient as every host in the known_hosts file could exactly have one host and one port. Thus, one would constantly get errors after rebooting the server, making it necessary to constantly modify the known_hosts file of the client.

However, this could be easily avoided by using a specific hosts file for such initramfs/Dropbear servers, e.g. known_hosts.initramfs.

Therefore, it is best to add an "-unlock" SSH-host entry to your ~/.ssh/config:

host server-unlock
    Hostname <ip>
    Port <port>
    User root
    HostKeyAlgorithms <THE HOST KEY ALGORITHM>
    UserKnownHostsFile ~/.ssh/known_hosts.initramfs

• Replace <ip> and <port with the respective values you choose.
• HostKeyAlgorithms enforces the connection to use Ed25519 or RSA (to ensure using the correct SSH id in case you forgot deleting the other keys :))
• UserKnownHostsFile ~/.ssh/known_hosts.initramfs the name of extra hoists file for initramfs servers.

Reboot and unlock ZFS pool via SSH

1. Reboot your server
2. Your server should be waiting at Begin: Starting dropbear...
3. Login into your server: ssh server-unlock
4. Check the SSH fingerprint against the one noted above
5. Confirm with yes (or no if the fingerprint differs...)
6. You should be dropping into the BusyBox now :)
7. zfsunlock and enter your password. The server should be resuming its boot sequence afterwards :)