Raspberry Pi OS kernel compile for envoy support

README.MD

These playbooks are for compiling the Pi OS kernel to make it compatible with Envoy which is used under the hood of Cilium, Consul Connect and maybe Istio. Enyoy will refuse to start on Pi OS due to a memory allocation error. Long discussion here. The two options are

• Building Envoy from source (not fun)
• Compile the Pi OS kernel with some extra flags (not fun, but doable)
• Use Ubuntu or maybe Debian built for the Pi (not fun either)

Pi OS needs to be compiled with the following options:

CONFIG_ARM64_VA_BITS_39=n
CONFIG_ARM64_VA_BITS_48=y
CONFIG_ARM64_VA_BITS=48
CONFIG_PGTABLE_LEVELS=4

CONFIG_PGTABLE_LEVELS I'm not 100% sure about, but its been mentioned in a few resources do I added it anyway. This Blog got me started but I found it slightly light on details. I also wanted to do a cross compile from an x86 machine, so I combined it with this gist.

Compiling the kernel on a raspberry pi 4b took just over two hours and the resulting .img didn't seem to work, hence the desire to build on an x86 platform. On a Hetzner 8 core Debian 11 host the build time was under 20 minutes, including the time to provision dependencies.

This ansible playbook is not pretty, I am not an expert, but it'll work. Update the hosts.yml with a valid IP then run

ansible-playbook -i hosts.yml kernel_compiler.yml --tags setup

Let it finish, shell into the build machine, do cd /root/Pi4-Kernel/linux then run the following

CONFIG_ARM64_VA_BITS_39=n \
  CONFIG_ARM64_VA_BITS_48=y \
  CONFIG_ARM64_VA_BITS=48 \
  ARCH=arm64 \
  CROSS_COMPILE=aarch64-linux-gnu- \
  make -j< NUM OF CORES> \
  CXXFLAGS="-march=armv8-a+crc -mtune=cortex-a72" \
  CFLAGS="-march=armv8-a+crc -mtune=cortex-a72" \
  deb-pkg

Depending on the amount of cores assigned it'll eventually finish and spit out 3 .deb files. Go back to you ansible host and run

ansible-playbook -i hosts.yml kernel_compiler.yml --tags package

This gathers everything into a zip and pulls it back down onto your local machine. Next you'll need to install the new kernel and its associated junk on your pi. I'm using DietPi so there was a few extra steps to keep various bits that it needs. If your on regular Pi OS just delete out anything that references DietPi :-)

ansible-playbook -i hosts.yml pi_upgrade.yml

Your Pi should reboot, if this hasn't bricked it. To check if its worked run uname -v and it should spit out a version date with the compile time, or check dmesg. If you want to check that the compile options are correct run

modprobe configs
zcat /proc/config.gz | grep CONFIG_ARM64_VA_BITS

To actually know if it works download envoy and run envoy --version, and it should give a version number, instead of a depressing error message.

Congratulations! You can now run Consul Connect, or Cilium with --set l7Proxy=true.

hosts.yml

hetz:
  hosts:
    hetz01:
      ansible_host: <your host ip>
  vars:
    ansible_ssh_private_key_file: ~/.ssh/hetz_host_rsa
    ansible_ssh_user: root
pis:
  hosts:
    pi01
  vars:
    ansible_ssh_private_key_file: ~/.ssh/pi_host_rsa
    ansible_ssh_user: root

kernel_complier.yml

# https://gist.github.com/G-UK/ee7edc4844f14fec12450b2211fc886e
# build on hetz debian 11
# take 20 minutes on a an 8 core CPX41

- hosts: hetz
  vars:
    core_count: 8
  tasks:

    - name: setup
      tags:
        - setup
      block:
        - name: do update and upgrade
          apt:
            update_cache: yes
            upgrade: 'yes'

        - name: install dependencies
          apt:
            name:
              - htop
              - git
              - bc
              - bison
              - flex
              - libssl-dev
              - make
              - ncurses-dev
              - libncurses-dev
              - gcc
              - subversion
              - screen
              - dpkg-dev
              - zip
              - gcc-aarch64-linux-gnu
              - build-essential

        - name: ensure build dir
          file:
            path: /root/Pi4-Kernel
            state: directory

        - ansible.builtin.git:
            repo: https://github.com/raspberrypi/linux.git
            depth: 1
            dest: /root/Pi4-Kernel/linux
            version: rpi-5.15.y

        - stat:
            path: /root/Pi4-Kernel/boot
          register: boot_path

        - debug:
            var: boot_path

        - subversion:
            repo: https://github.com/raspberrypi/firmware/trunk/boot
            dest: /root/Pi4-Kernel/boot
            export: yes
          when: boot_path.stat.exists == false

        - shell: |
            rm -f boot/kernel*
            rm -f boot/*.dtb
            rm -f boot/overlays/*.dtbo
          args:
            chdir: /root/Pi4-Kernel/

        - shell: |
              make \
                CONFIG_ARM64_VA_BITS_39=n \
                CONFIG_ARM64_VA_BITS_48=y \
                CONFIG_ARM64_VA_BITS=48 \
                CONFIG_PGTABLE_LEVELS=4 \
                ARCH=arm64 \
                CROSS_COMPILE=aarch64-linux-gnu- \
                bcm2711_defconfig
          args:
            chdir: /root/Pi4-Kernel/linux

        - ansible.builtin.lineinfile:
            path: /root/Pi4-Kernel/linux/.config
            search_string : "{{ item.old }}"
            line: "{{ item.new }}"
          loop:
            - old: 'CONFIG_ARM64_VA_BITS_39=y'
              new: '# CONFIG_ARM64_VA_BITS_39=n'
            - old: '# CONFIG_ARM64_VA_BITS_48 is not set'
              new: 'CONFIG_ARM64_VA_BITS_48=y'
            - old: 'CONFIG_ARM64_VA_BITS=39'
              new: 'CONFIG_ARM64_VA_BITS=48'
              # https://github.com/envoyproxy/envoy/issues/15235
            - old: 'CONFIG_PGTABLE_LEVELS=3'
              new: 'CONFIG_PGTABLE_LEVELS=4'

    - name: compile
      tags:
        - compile
        - never
      block:

        - debug:
            msg:
              - this requires interaction on the cli
              - so don't run it via ansible
              - its here for reference
              
        - shell: |
              CONFIG_ARM64_VA_BITS_39=n \
                CONFIG_ARM64_VA_BITS_48=y \
                CONFIG_ARM64_VA_BITS=48 \
                ARCH=arm64 \
                CROSS_COMPILE=aarch64-linux-gnu- \
                make -j{{ core_count }} \
                CXXFLAGS="-march=armv8-a+crc -mtune=cortex-a72" \
                CFLAGS="-march=armv8-a+crc -mtune=cortex-a72" \
                deb-pkg
          args:
            chdir: /root/Pi4-Kernel/linux

    - name: package and fetch
      tags:
        - package
      block:
        - shell: |
            cp arch/arm64/boot/Image ../boot/kernel8.img
            cp arch/arm64/boot/dts/overlays/*.dtbo ../boot/overlays/
            cp arch/arm64/boot/dts/broadcom/*.dtb ../boot/
          args:
            chdir: /root/Pi4-Kernel/linux

        - shell: |
            mkdir -p /root/kernel
            cp *.deb /root/kernel
            cp -r boot /root/kernel
            zip -r kernel.zip /root/kernel
          args:
            chdir: /root/Pi4-Kernel/

        - name: fetch kernel.zip
          ansible.builtin.fetch:
            src: /root/Pi4-Kernel/kernel.zip
            dest: ./
            flat: yes

        - debug:
            msg:
              - https://gist.github.com/G-UK/ee7edc4844f14fec12450b2211fc886e#kernel-installation

pi_upgrade.yml

- hosts: pis
  tasks:

    - name: copy kernel zip
      copy:
        src: kernel.zip
        dest: /usr/src

    - name: Unarchive kernel
      ansible.builtin.unarchive:
        src: /usr/src/kernel.zip
        dest: /usr/src/
        remote_src: yes

    - name: check boot dir 
      stat:
        path: /boot/config.txt
      register: boot_config_txt

    - name: perform pre install
      shell: |
        apt purge -y linux-headers-* linux-image-* || true
        cp -f /boot/*.txt .
        cp -rf /boot/dietpi/ .
        rm -rf /boot/*
      args:
        chdir: /usr/src/root/kernel/boot
      when: boot_config_txt.stat.exists == true

    - name: Check /boot contents
      find:
        paths: /boot
      register: boot_contnents

    - name: install debs
      shell: dpkg -i linux-headers-*.deb linux-image-*.deb linux-libc-*.deb
      args:
        chdir: /usr/src/root/kernel/
      when: boot_contnents.matched == 0 

    - name: copy boot files
      shell: cp -a ./ /boot/
      args:
        chdir: /usr/src/root/kernel/boot

    - name: reboot pis
      reboot:
        reboot_command: /usr/sbin/poweroff --reboot

I beseech @geerlingguy, our Pi Lord and Kernel Compile Saviour for any advice!

Hehe, thanks for the query—though do you have any specific questions? This is a lot of code to try to understand at a glance.

Hehe, thanks for the query—though do you have any specific questions? This is a lot of code to try to understand at a glance.
Sorry Jeff. This is like crying out for jesus and have him actually respond.
I would take any tips on doing cross compiles of the RP kernel. The blog posts and gists I found are of old. This for instance:

  CXXFLAGS="-march=armv8-a+crc -mtune=cortex-a72" 
  CFLAGS="-march=armv8-a+crc -mtune=cortex-a72" 

looks legit, but I can't tell if it actaully make a difference or not. Might be a good topic for a video!