These playbooks are for compiling the Pi OS kernel to make it compatible with Envoy which is used under the hood of Cilium, Consul Connect and maybe Istio. Enyoy will refuse to start on Pi OS due to a memory allocation error. Long discussion here. The two options are
- Building Envoy from source (not fun)
- Compile the Pi OS kernel with some extra flags (not fun, but doable)
- Use Ubuntu or maybe Debian built for the Pi (not fun either)
Pi OS needs to be compiled with the following options:
CONFIG_ARM64_VA_BITS_39=n
CONFIG_ARM64_VA_BITS_48=y
CONFIG_ARM64_VA_BITS=48
CONFIG_PGTABLE_LEVELS=4
CONFIG_PGTABLE_LEVELS
I'm not 100% sure about, but its been mentioned in a few resources do I added it anyway. This Blog got me started but I found it slightly light on details. I also wanted to do a cross compile from an x86 machine, so I combined it with this gist.
Compiling the kernel on a raspberry pi 4b took just over two hours and the resulting .img
didn't seem to work, hence the desire to build on an x86 platform. On a Hetzner 8 core Debian 11 host the build time was under 20 minutes, including the time to provision dependencies.
This ansible playbook is not pretty, I am not an expert, but it'll work. Update the hosts.yml
with a valid IP then run
ansible-playbook -i hosts.yml kernel_compiler.yml --tags setup
Let it finish, shell into the build machine, do cd /root/Pi4-Kernel/linux
then run the following
CONFIG_ARM64_VA_BITS_39=n \
CONFIG_ARM64_VA_BITS_48=y \
CONFIG_ARM64_VA_BITS=48 \
ARCH=arm64 \
CROSS_COMPILE=aarch64-linux-gnu- \
make -j< NUM OF CORES> \
CXXFLAGS="-march=armv8-a+crc -mtune=cortex-a72" \
CFLAGS="-march=armv8-a+crc -mtune=cortex-a72" \
deb-pkg
Depending on the amount of cores assigned it'll eventually finish and spit out 3 .deb
files. Go back to you ansible host and run
ansible-playbook -i hosts.yml kernel_compiler.yml --tags package
This gathers everything into a zip and pulls it back down onto your local machine. Next you'll need to install the new kernel and its associated junk on your pi. I'm using DietPi so there was a few extra steps to keep various bits that it needs. If your on regular Pi OS just delete out anything that references DietPi :-)
ansible-playbook -i hosts.yml pi_upgrade.yml
Your Pi should reboot, if this hasn't bricked it. To check if its worked run uname -v
and it should spit out a version date with the compile time, or check dmesg
. If you want to check that the compile options are correct run
modprobe configs
zcat /proc/config.gz | grep CONFIG_ARM64_VA_BITS
To actually know if it works download envoy and run envoy --version
, and it should give a version number, instead of a depressing error message.
Congratulations! You can now run Consul Connect, or Cilium with --set l7Proxy=true
.
looks legit, but I can't tell if it actaully make a difference or not. Might be a good topic for a video!