(Raspberry Pi OS Debian Bullseye)
This is useful for testing but the 3B sure is slow! Follow instructions at your own risk (and time) and no support is provided.
These instructions also include setup for SSH certificates and host certificates using a self-hosted Certificate Authority (step-ca). I also have a self-hosted zone (me.home.arpa) on my network to provide a local DNS. This process should also work with SSH keys.
- Raspberry Pi 3B
- A 3B+ or newer will likely be faster
- Earlier than 3B may be even slower or not work
- Step CA Service
- Provides ACME for short-term certificates
- Also use SSH certificates for even easier login
- Unbound + NSD (DNS services)
- Unbound provides recursive DNS out to internet
- Unbound directs
me.home.arpato NSD - NSD provides authoritative DNS for
me.home.arpaon local network - Pi-Hole can be set to upstream to Unbound
- Proxmox Virtual Environment experience
- It normally is easier to install on appropriate hardware; a Pi is not a great first introduction
Instructions are for using MacOS (BSD) to load the microSD card, and should be modified for GNU or Windows. You will need local console access on the Pi to finish configuration if/when the network is unavailable.
Download the image and verify:
$ wget https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-09-26/2022-09-22-raspios-bullseye-arm64-lite.img.xz
$ wget https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-09-26/2022-09-22-raspios-bullseye-arm64-lite.img.xz.sha256
$ wget https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-09-26/2022-09-22-raspios-bullseye-arm64-lite.img.xz.sig
$ gpg --verify 2022-09-22-raspios-bullseye-arm64-lite.img.xz.sig 2022-09-22-raspios-bullseye-arm64-lite.img.xz
gpg: Signature made Mon 26 Sep 01:29:13 2022 MDT
gpg: using RSA key 54C3DD610D9D1B4AF82A37758738CD6B956F460C
gpg: Can't check signature: No public key
$ shasum -c 2022-09-22-raspios-bullseye-arm64-lite.img.xz.sha256
2022-09-22-raspios-bullseye-arm64-lite.img.xz: OK
Decompress:
$ xz -T 0 -d 2022-09-22-raspios-bullseye-arm64-lite.img.xz
Restore onto SD card:
$ sudo diskutil list
/dev/disk4 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: *31.3 GB disk4
$ sudo dd if=2022-09-22-raspios-bullseye-arm64-lite.img of=/dev/rdisk4 bs=1m'
Eject the SD card with all its partitions.
$ sudo diskutil ejectdisk /dev/disk4
Put the card in the Pi, and plug in the cables. HDMI for monitor, USB keyboard, ethernet cable, and finally the micro-USB power cable.
The username is not too important as PVE will be set up using the root account, but the account will remain after install. You can use the default "pi" user, and later set up certificate mappings that let a remote user login as different users.
Password-less sudo is pre-enabled, so you may use "sudo" without having to enter your password at least once per session. This is already configured in /etc/sudoers.d/010_pi-nopasswd.
Use raspi-config to change the Pi settings before PVE is installed.
$ sudo raspi-config
1 -> Hostname -> `pinode1`
1 -> Wait for network at boot -> No
3 -> SSH -> Enable
5 -> Locale -> disable `en_GB`, enable `en_US.UTF-8`, set default locale to `en_US.UTF-8`
5 -> Timezone -> `America/Edmonton`
5 -> Keyboard Layout -> `Generic 104-key US QWERTY layout`
5 -> WLAN Country -> `CA (Canada)`
6 -> Network Interface Names -> No
Edit /etc/dhcpcd.conf to set the static IP configuration for eth0. Dhcpcd will be removed later by the PVE install, but is necessary until then.
interface eth0
static ip_address 192.168.1.7/24
static routers=192.168.1.1
static domain_name_servers=192.168.1.2
Reboot, and check that the IP is working and DNS is being reached at 192.168.1.2. It should be possible to ping websites outside of the local network and receive responses.
If you use a local apt cache, then update:
$ sudo -i
# echo 'Acquire::http { Proxy "http://aptcache.me.home.arpa:3142"; };' > /etc/apt/apt.conf.d/02proxy
# apt update
# apt upgrade -y
Reboot to finish the updates. Set up step client for SSH certificate support. Note that armv7 must be downloaded.
$ wget https://github.com/smallstep/cli/releases/download/v0.22.0/step_linux_0.22.0_armv7.tar.gz
$ tar xzf step_linux_0.22.0_armv7.tar.gz
$ sudo cp step_0.22.0/bin/step /usr/local/bin/step
$ step ca bootstrap --ca-url https://ca.me.home.arpa --fingerprint <step-ca-fingerprint>
$ step certificate install $(step path)/certs/root_ca.crt
$ step ssh config --roots | sudo tee /etc/ssh/ssh_user_key.pub
$ step ssh certificate --insecure --no-password --host pinode1.me.home.arpa ssh_host_ecdsa_key
Copy the keys to /etc/ssh:
$ cp ssh_host_ecdsa_key ssh_host_ecdsa_key-cert.pub ssh_host_ecdsa_key.pub /etc/ssh/.
Add an authorized principals file, with a mapping for your SSH certificate principal(s) (certificate usernames to system usernames):
# mkdir /etc/ssh/auth_principals/
# echo "access-root" > /etc/ssh/auth_principals/root
And an addition to /etc/ssh/sshd_config:
TrustedUserCAKeys /etc/ssh/ssh_user_key.pub
HostKey /etc/ssh/ssh_host_ecdsa_key
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
Restart ssh, it should not have any errors:
# systemctl restart ssh
# journalctl -u ssh
If your SSH client has been configured with your SSH certificate (from the Step CA), you can issue yourself a short-term SSH certificate and connect to the Pi now without a password or SSH Key. Here is a sample that will require a provisioner to provide a key (password) to generate the certificate. When you set up Step CA it will have you create a default JWK provisioner that can be used here too.
mac$ cd ~/.ssh
mac$ step ssh certificate me@example.com id_ecdsa --principal access-root
Restart the Pi and begin the install process for Proxmox VE. Enable the no-subscriptions repository now; I had some issues with the incorrect version of ifupdown2 being installed.
$ wget https://enterprise.proxmox.com/debian/proxmox-release-bullseye.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bullseye.gpg
$ echo "deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list
Edit the hosts file, replacing 127.0.1.1 line with:
192.168.1.7 pinode1.me.home.arpa pinode1
Download the custom installer script from PiMox:
$ sudo -s
$ curl https://raw.githubusercontent.com/pimox/pimox7/master/RPiOS64-IA-Install.sh > RPiOS64-IA-Install.sh
$ chmod +x RPiOS64-IA-Install.sh
( edit the script to allow installation on RPi 3 - remove an exit statement on line ~39 )
$ ./RPiOS64-IA-Install.sh
The installer warns about a 30 minute install on a 4, but my 3B took about an hour. It looks like it "locked up" at some point and a hard reset had a partial install unfinished and networking is not working. This can be recovered.
Edit the network interface manually in /etc/network/interfaces:
auto lo
iface lo inet loopback
iface eth0 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.1.7/24
gateway 192.168.1.1
bridge-ports eth0
bridge-stp off
bridge-fd 0
This is the typical vmbr0 in PVE that uses the eth0 physical interface. The virtual bridge provides some more flexibility for networking the hosts and containers. Save the interfaces, then reboot. If the network does not come online automatically, then login and try using sudo ifup vmbr0 to find the error. (Maybe try using systemctl to unmask networking, I cannot confirm if that was an actual fix.)
Once the networking is working, use apt to finish the installation.
$ sudo apt install -f
After the install is completed, a final restart should confirm that PVE is booting with networking and SSH working.
Visiting https://pinode1.me.home.arpa:8006/ works as I set up the DNS A records for my Mac's browser. I have DNS and a Certificate Authority configured, so we do not have to settle for accessing the PVE web interface with its default self-signed certificate and the browser warnings.
Add ACME Provisioner
If your Step CA does not have an ACME provisioner already then you will need to add one. Login to the CA and set the STEPPATH. Then create a new ACME provisioner.
# export STEPPATH=/etc/step-ca
# step ca provisioner add acme --type ACME
# systemctl restart step-ca.service
The directory is based on the name given (acme) and its type (acme): https://ca.me.home.arpa/acme/acme/directory
Back on pinode1, PVE can be set up to use the ACME service for automatically renewed certificates.
pinode1 # pvenode acme account register pinode1 pve@pinode1.me.home.arpa --directory https://ca.me.home.arpa/acme/acme/directory
# pvenode config set --acme domains=pinode1.me.home.arpa
# pvenode acme cert order
That last command may need --force if you manually added any certificates. For setup on additional PVE nodes, I re-ran this command with a different id (pinode1) and domains (pinode1.me.home.arpa).
Now the web admin site works with HTTPS and the certificate is trusted as the root CA has been installed in our browser's trust store.
Completed: PVE 7 running on Pi 3B
It is possible to run this in a cluster, but I had occasional issues with the Pi being unavailable from the Web GUI despite having SSH access. I only use the Pi to run a secondary DNS and (slow) fallover services.
LXC containers will need an ARM-based image, and the PVE default list does not include them. It is fortunately only a few steps to get started.
The images are located at https://us.lxd.images.canonical.com/images/ and there seem to be local mirrors around the world. In my case I want to run an LXC on Ubuntu 20.04, so I will choose ubuntu/focal/arm64/default/<date>/rootfs.tar.xz.
If you are storing templates on the Pi, download the rootfs file to the container storage there (usually /var/lib/vz/template/cache/). I recommend renaming to something that will be self-explanatory in the Proxmox web UI, e.g. ubuntu-20.04-standard_arm64.tar.xz.
$ sudo -s
# cd /var/lib/vz/template/cache/
# curl https://us.lxd.images.canonical.com/images/ubuntu/focal/arm64/default/20221023_19:59/rootfs.tar.xz -o ubuntu-20.04-standard_arm64.tar.xz
This template can now be used to start up an LXC on the Pi. I suggest creating a new LXC named something generic like focal-arm64 and setting up your preferred base set of packages, shutting down, and converting that LXC to a template that you can re-use. In my case I added the Step CLI binary and pre-set the bootstrap for the local Certificate Authority, and then only a few steps are required to customize the LXC for a custom domain.
Good luck!