Skip to content

Instantly share code, notes, and snippets.

@Abyss-W4tcher

Abyss-W4tcher/AArch64_RaspberryPi emulation_and_kernel_cross-compilation.md

Last active January 30, 2024 09:13
Show Gist options
  • Save Abyss-W4tcher/8442b6b6b85f725158fe7e9b99e507be to your computer and use it in GitHub Desktop.
Save Abyss-W4tcher/8442b6b6b85f725158fe7e9b99e507be to your computer and use it in GitHub Desktop.
Download ZIP
AArch64 RaspberryPi emulation and kernel cross-compilation
Raw
AArch64_RaspberryPi emulation_and_kernel_cross-compilation.md

AArch64 RaspberryPi emulation and kernel cross-compilation

General ressources

Disclaimer

The following setup assumes you already ran an existing RaspberryPi VM, and know which kernel you want to cross compile. If not, please follow this tutorial : https://interrupt.memfault.com/blog/emulating-raspberry-pi-in-qemu. This is not mandatory, but if you want to compile a custom kernel closely matching a VM one, you will need to run it first and inspect it a bit.

Compile custom RaspberryPi kernel

We will try to get as close as possible to the kernel used by the raspios image.

Get target kernel informations

As we want to mimic the kernel from the raspios image, we will need to extract some informations :

$ cat /proc/version
Linux version 6.1.21-v8+ (dom@buildbot) (aarch64-linux-gnu-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023
$ gcc --version
gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Docker

We will be using a docker container to keep our host machine clean from apt packages.

Place yourself in a directory dedicated to this project (e.g. emu_raspberry/).

Let's define a custom package name to isolate each build :

echo 'my_kernel' > /bind/kpack

Launch and access the container :

docker run -d -it --name rasp_compil -w /bind/ -v $(pwd):/bind gcc:10.2
docker exec -it rasp_compil bash

Install common dependencies :

apt-get update
apt-get install -y git bc rsync subversion libncurses-dev gawk flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf llvm cpio build-essential
apt-get install -y gcc-aarch64-linux-gnu

Kernel cross-compilation

https://gist.github.com/azbesthu/3891645#file-get-rpi-kernel-source-sh-L4

Following assumes you are in the container terminal, at following path :

cd /bind/

Clone the kernel and navigate to its root tree :

# Use the technique at https://web.archive.org/web/20240124142754/https://gist.github.com/azbesthu/3891645#file-get-rpi-kernel-source-sh-L4 to determine the kernel source corresponding to the /proc/version informations
git clone --depth 1 -b '1.20230405' https://github.com/raspberrypi/linux.git
cd linux/

Export needed cross compilation variables :

export ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- KERNEL=kernel8

Set configuration :

make clean
file .config || make bcm2711_defconfig # You can use an existing .config or make a default one
# Command to extract config from a live Raspberry VM : sudo modprobe configs; zcat /proc/config.gz > /boot/.config

Now, you need to run make menuconfig and activate the following options, in order. This is not mandatory, but the setup will assume the VM has virtio support (it is much faster) :

  • PCI support
    • PCI controller drivers
      • Generic PCI host controller
  • Device Drivers
    • Virtio drivers
      • PCI driver for virtio devices
      • Virtio balloon driver
      • Virtio input driver
      • Platform bus driver for memory mapped virtio devices
    • Network device support
      • Virtio network driver
    • Block devices
      • Virtio block driver

Based on https://www.qemu.org/docs/master/system/arm/virt.html#linux-guest-kernel-configuration, https://www.kernelconfig.io/config_pci_host_generic and https://stackoverflow.com/a/26890069.
If you want vmlinux :

  • Kernel hacking
    • Compile-time checks and compiler options
      • Debug information
        • Rely on the toolchain's implicit default DWARF version

Cross compile the kernel :

CUSTOM_KPKG_NAME=$(cat /bind/kpack)
make -j$(nproc) LOCALVERSION="-$CUSTOM_KPKG_NAME" KDEB_PKGVERSION="$(make kernelversion)-1" Image modules dtbs

Export compiled objects :

KSAVE_DIR="/bind/kernel-$CUSTOM_KPKG_NAME"
mkdir -p $KSAVE_DIR/boot/overlays
cp arch/arm64/boot/Image $KSAVE_DIR/boot/kernel8.img
cp arch/arm64/boot/dts/overlays/*.dtbo $KSAVE_DIR/boot/overlays/
cp arch/arm64/boot/dts/broadcom/*.dtb $KSAVE_DIR/boot/
cp vmlinux $KSAVE_DIR/

Ressources

Cross compilation bug

When cross compiling the kernel, many utilities needed to build external modules aren't cross compiled. Doing so, if we compile the kernel with :

make bindeb-pkg -j$(nproc)

Then transfer the .deb (linux-image, linux-headers etc.) to the VM, and install them, we will have x86_64 binaries instead of AArch64 ones. This is a known bug, and forces us to compile our modules outside of the VM (see following arm-cpusysregs example).

Potential patch :

Emulation

Following assumes you are not in the Docker container, and at the root of the directory dedicated to this project (e.g. emu_raspberry/).

Setup convenient aliases :

export RASPIOS_IMG_PATH='2023-05-03-raspios-bullseye-arm64.img'
export RASPIOS_IMG_MOUNT_POINT='/mnt/raspi_img/'
alias mount_img="sudo losetup -P /dev/loop0 $RASPIOS_IMG_PATH \
;sudo mkdir -p $RASPIOS_IMG_MOUNT_POINT \
;sudo mount /dev/loop0p2 $RASPIOS_IMG_MOUNT_POINT \
;sudo mount /dev/loop0p1 $RASPIOS_IMG_MOUNT_POINT/boot"
alias umount_img='sudo losetup -d /dev/loop0 ; sudo umount /dev/loop0*'
alias qemu_rasp="qemu-system-aarch64 -machine virt -cpu cortex-a72 -smp 8 -m 1G \
    -kernel kernel8.img -append 'root=/dev/vda2 rootfstype=ext4 earlyprintk rw panic=0 console=ttyAMA0 fsck.repair=yes rootwait rootdelay=1 dwc_otg.lpm_enable=0' \
    -drive format=raw,file=$RASPIOS_IMG_PATH,if=none,id=hd0,cache=writeback \
    -device virtio-blk,drive=hd0,bootindex=0 \
    -netdev user,id=mynet,hostfwd=tcp::2222-:22 \
    -device virtio-net-pci,netdev=mynet \
    -nographic"
# For 52 bits VA support (https://github.com/hermit-os/loader/issues/108) : -cpu max

(re-)Download the raspios image, and mount it :

wget https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2023-05-03/2023-05-03-raspios-bullseye-arm64.img.xz
xz -d 2023-05-03-raspios-bullseye-arm64.img.xz
mount_img

Override current kernel, and setup username + password :

CUSTOM_KPKG_NAME=$(cat kpack)
KSAVE_DIR="kernel-$CUSTOM_KPKG_NAME"

sudo cp -r $KSAVE_DIR/boot/* $RASPIOS_IMG_MOUNT_POINT/boot/
cp $KSAVE_DIR/boot/kernel8.img .
# CREDENTIALS : "pi:raspberry"
echo 'pi:$6$rBoByrWRKMY1EHFy$ho.LISnfm83CLBWBE/yqJ6Lq1TinRlxw/ImMTPcvvMuUfhQYcMmFnpFXUPowjy2br1NA0IACwF9JKugSNuHoe0' | sudo tee $RASPIOS_IMG_MOUNT_POINT/boot/userconf

Resize the image, and unmount it :

qemu-img resize ./2023-05-03-raspios-bullseye-arm64.img 16G
umount_img

Launch the emulation :

qemu_rasp
# Connect with "pi:raspberry"

# Recommended
sudo apt-get update
sudo apt-get install -y ntp

(optional) Expand the disk :

umount_img
qemu_rasp
# Connect with "pi:raspberry"
# https://muizidn.medium.com/increase-raspberry-pi-disk-size-in-qemu-d6d33666a930
sudo raspi-config 
# advanced options -> expand filesystem
# CTRL + A then X to quit qemu
qemu_rasp
# launch the VM again and quit on login screen, disk has been expanded

Ressources

arm-cpusysreg

This module allows to read AArch64 CPU registers conveniently. See https://github.com/lelegard/arm-cpusysregs.

Module cross compilation

Following assumes you are in the container terminal, at following path :

cd /bind/linux/ # The previously compiled kernel source

Cross compile the module :

INSTALL_MOD_PATH=../modules make -j$(nproc) modules_install
cd ../
git clone https://github.com/lelegard/arm-cpusysregs
cd arm-cpusysregs/kernel/linux/
make -C /bind/modules/lib/modules/*/build M=$PWD
cd /bind/linux/

Save the compiled module :

CUSTOM_KPKG_NAME=$(cat /bind/kpack)
KSAVE_DIR="/bind/kernel-$CUSTOM_KPKG_NAME"
cp /bind/arm-cpusysregs/kernel/linux/cpusysregs.ko $KSAVE_DIR/

Module insertion

Following assumes you are not in the Docker container, and at the root of the directory dedicated to this project (e.g. emu_raspberry/).

mount_img
sudo cp $KSAVE_DIR/cpusysregs.ko $RASPIOS_IMG_MOUNT_POINT/root/
umount_img
qemu_rasp
# Following is in the VM
apt-get install -y python3 python-is-python3 build-essential
git clone https://github.com/lelegard/arm-cpusysregs
cd arm-cpusysregs/
make -j$(nproc)
mv apps/sysregs /usr/bin/sysregs
cd /root/
insmod cpusysregs.ko # Previously transferred
sysregs -a

Sample values :

TCR_EL1           00400034-F5567516
TPIDRRO_EL0       00000000-00000000
TPIDR_EL0         000003FF-B8826B80
TPIDR_EL1         FFFFFE00-36C00000
TTBR0_EL1         00000000-52990000
TTBR1_EL1         1C280000-411D0000

Memory capture

# VM is launchded
# "CTRL+A" then "C" to open qemu console
dump-guest-memory rasp_dump

Additional ressources

Get exact kernel commit from live VM

$ sudo apt-get install -y raspberrypi-kernel-headers
$ gunzip -c /usr/share/doc/raspberrypi-bootloader/changelog.Debian.gz > changelog
$ head changelog 
raspberrypi-firmware (1:1.20230405-1) bullseye; urgency=medium

  * firmware as of 055e044d5359ded1aacc5a17a8e35365373d0b8b
  * Linux version 6.1.21

Replicate exact GCC version

Goal was to replicate an existing kernel identically. It is not worth the hassle :)

root@raspberrypi:~# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/aarch64-linux-gnu/10/lto-wrapper
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 10.2.1-6' --with-bugurl=file:///usr/share/doc/gcc-10/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-10 --program-prefix=aarch64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --disable-libquadmath-support --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-mutex
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.2.1 20210110 (Debian 10.2.1-6)

Compiling

CONFIG_CC_VERSION_TEXT="aarch64-linux-gnu-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0" # from .config

# https://packages.debian.org/bullseye/amd64/gcc-10-aarch64-linux-gnu
mkdir libs_aarch64 && cd libs_aarch64

wget http://ftp.de.debian.org/debian/pool/main/g/gcc-10-cross/gcc-10-cross-base_10.2.1-6cross1_all.deb
wget http://ftp.de.debian.org/debian/pool/main/g/gcc-10-cross/libgcc-10-dev-arm64-cross_10.2.1-6cross1_all.deb
wget http://ftp.de.debian.org/debian/pool/main/g/gcc-10-cross/gcc-10-aarch64-linux-gnu-base_10.2.1-6cross1_amd64.deb
wget http://ftp.de.debian.org/debian/pool/main/g/gcc-10-cross/cpp-10-aarch64-linux-gnu_10.2.1-6cross1_amd64.deb
wget http://ftp.de.debian.org/debian/pool/main/g/gcc-10-cross/gcc-10-aarch64-linux-gnu_10.2.1-6cross1_amd64.deb

apt-get install -y binutils-aarch64-linux-gnu
apt-get install -y libgcc-12-dev-arm64-cross libasan6-arm64-cross libtsan0-arm64-cross
# dpkg -i --force-depends *.deb
ln -s /usr/bin/aarch64-linux-gnu-gcc-10 /usr/bin/aarch64-linux-gnu-gcc
# https://launchpad.net/ubuntu/+source/gcc-8-cross/33ubuntu2
apt-get update
apt-get install -y binutils-aarch64-linux-gnu
apt-get install -y libgcc-12-dev-arm64-cross libasan6-arm64-cross libtsan0-arm64-cross

mkdir libs_aarch64_8 && cd libs_aarch64_8
wget http://ftp.de.debian.org/debian/pool/main/g/gcc-8-cross/libgcc1-amd64-cross_8.3.0-2cross1_all.deb
wget http://ftp.de.debian.org/debian/pool/main/g/gcc-8-cross/libasan5-arm64-cross_8.3.0-2cross1_all.deb
wget http://launchpadlibrarian.net/465590960/libisl22_0.22.1-1_amd64.deb
wget http://launchpadlibrarian.net/470696662/gcc-8-cross-base_8.4.0-3ubuntu1cross1_all.deb
wget http://launchpadlibrarian.net/470696794/libgcc-8-dev-arm64-cross_8.4.0-3ubuntu1cross1_all.deb
wget http://launchpadlibrarian.net/470696654/gcc-8-aarch64-linux-gnu-base_8.4.0-3ubuntu1cross1_amd64.deb
wget http://launchpadlibrarian.net/470696610/cpp-8-aarch64-linux-gnu_8.4.0-3ubuntu1cross1_amd64.deb
wget http://launchpadlibrarian.net/470696656/gcc-8-aarch64-linux-gnu_8.4.0-3ubuntu1cross1_amd64.deb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment