ARM64 Linux on Win10 x64

arm64-on-Win10-x64.md

Below are the steps to get an ARM64 version of Ubuntu running in the QEMU emulator on Windows 10.

Install QEMU

Install for Windows from choco package manager

choco install qemu --version=2021.5.5 -confirm
PS D:\arm64-ubuntu2004\qemu> qemu-system-aarch64 --help

Check appending Window System PATH C:\Program Files\qemu applied.

Then typing the terminal with some commands.

qemu-img --help

qemu-system-aarch64 -M virt -cpu help

Copy the Firmware and OS images

Into your working directory...

• Copy the ubuntu-20.04-server-cloudimg-arm64.img OS image from https://cloud-images.ubuntu.com/releases/20.04/release/
• Copy the QEMU_EFI.fd firmware image from https://releases.linaro.org/components/kernel/uefi-linaro/latest/release/qemu64/
• Download user-data.zip from below zip raw file on this Gist. then, extract to user-data.img.

#cloud-config
password: asdfqwer
chpasswd: { expire: False }
ssh_pwauth: True

Login first password will be asdfqwer

(Optional) Create the configuration data image

The Ubuntu server images require configuration data be provided as an image, such as setting auth credentials.

We need input password below example <WE_HAVE_PASSWORD_MUST_BE_CREDENTIAL>.

sudo apt -y install cloud-image-utils

cat >user-data <<EOF
#cloud-config
password: <WE_HAVE_PASSWORD_MUST_BE_CREDENTIAL>
chpasswd: { expire: False }
ssh_pwauth: True
EOF

cloud-localds user-data.img user-data

The tool used to convert the config text file into an image file only runs on Linux, so I've attached a user-data.img file (and the text file used to create it) in a zip file to this Gist. Extract the user-data.img file to the working directory.

In user-data text file contents.

#cloud-config
password: asdfqwer
chpasswd: { expire: False }
ssh_pwauth: True

The user-data.img file was created for password authentication as outlined in https://stackoverflow.com/a/53373376

Resize storage

If booting is confirmed, we will power off for a while and change the storage capacity.

Resize your image in the host what you want. :)

qemu-img resize ubuntu-20.04-server-cloudimg-arm64.img +32G

Start the vm, ssh into it and enter the following command

sudo growpart /dev/vda 1

Launch the emulator from the working directory

Run the below to boot the image, you will some some benign errors at startup. Wait until the output settles down (even after you see the login prompt, as the post-boot config may not have completed yet).

Run example

qemu-system-aarch64 -m 4096 -cpu cortex-a72 -smp 4 -M virt -M gic_version=3 -nographic -bios QEMU_EFI.fd -drive if=none,file=ubuntu-20.04-server-cloudimg-arm64.img,id=hd0 -device virtio-blk-device,drive=hd0 -drive file=user-data.img,format=raw -device virtio-net-device,netdev=net0 -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0

Run NUMA Example

qemu-system-aarch64 -m 16384M,slots=4 -cpu cortex-a72 -smp cpus=8 -object memory-backend-ram,id=mem0,size=4096M -numa node,cpus=0-1,nodeid=0,memdev=mem0 -object memory-backend-ram,id=mem1,size=4096M -numa node,cpus=2-3,nodeid=1,memdev=mem1 -object memory-backend-ram,id=mem2,size=4096M -numa node,cpus=4-5,nodeid=2,memdev=mem2 -object memory-backend-ram,id=mem3,size=4096M -numa node,cpus=6-7,nodeid=3,memdev=mem3 -M virt -M gic_version=3 -nographic -bios QEMU_EFI.fd -drive if=none,file=ubuntu-18.04-server-cloudimg-arm64.img,id=hd0 -device virtio-blk-device,drive=hd0 -drive file=user-data.img,format=raw -device virtio-net-device,netdev=net0 -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0

Long command need indent, enter.

We can check below.

qemu-system-aarch64 -m 4096 -cpu cortex-a72 -smp 4 -M virt -M gic_version=3 -nographic \
    -bios QEMU_EFI.fd \
    -drive if=none,file=ubuntu-20.04-server-cloudimg-arm64.img,id=hd0 \
    -device virtio-blk-device,drive=hd0 \
    -drive file=user-data.img,format=raw \
    -device virtio-net-device,netdev=net0 \
    -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0

To break down these lines:

• qemu-system-aarch64 -m 4096 -cpu cortex-a72 -smp 4 -M virt -M gic_version=3 -nographic
  • run the ARM64 virtual platform emulator with 4GB RAM and 4 Cortex-A72 cores with no GUI support.
• -M virt -M gic_version=3
  • use the QEMU's Virt DTB mem map and GIC v3.
• -bios QEMU_EFI.fd
  • use the firmware downloaded above.
• -drive if=none,file=ubuntu-20.04-server-cloudimg-arm64.img,id=hd0
  • use the Ubuntu image file
• -device virtio-blk-device,drive=hd0
  • mount drive from above as a block device
• -drive file=user-data.img,format=raw
  • use the configuration data image file
• -device virtio-net-device,netdev=net0
  • create a virtual network device
• -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0
  • set up the networking stack and forward the SSH port

Play with SSH

Then from a good Terminal emulator (I recommend the new Windows Terminal app with one of the Powerline fonts) you can connect over SSH with the below, and the configured password (sample file have asdfqwer):

Do terminal with ssh.

ssh ubuntu@localhost -p 2222

Upload my-file.sh from host to vm.

 scp -P 2222 my-file.sh ubuntu@localhost:/home/ubuntu

Download my-file.sh from vm to host.

 scp -P 2222 ubuntu@localhost:/home/ubuntu/my-file.sh .

dbgsym-ubuntu2004.sh

#!/bin/bash

codename=$(lsb_release -cs)

sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename}      main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

sudo apt update -y
sudo apt install -y ubuntu-dbgsym-keyring
sudo apt update -y
sudo apt install -y linux-image-$(uname -r)-dbgsym

is_kdump_ok.sh

#!/bin/bash

echo -e "##################################################"
echo -e ">> cat /proc/cmdline"
echo -e "##################################################"
cat /proc/cmdline

echo -e "##################################################"
echo -e ">> dmesg | grep -i crash"
echo -e "##################################################"
dmesg | grep -i crash

echo -e "##################################################"
echo -e ">> cat /proc/iomem | grep 'Crash kernel'"
echo -e "##################################################"
cat /proc/iomem | grep 'Crash kernel'

echo -e "##################################################"
echo -e ">> cat /sys/kernel/kexec_crash_loaded"
echo -e "##################################################"
cat /sys/kernel/kexec_crash_loaded

echo -e "##################################################"
echo -e ">> kdump-config show"
echo -e "##################################################"
kdump-config show

qemu-img-resize-32g.bat

qemu-img resize ubuntu-20.04-server-cloudimg-arm64.img +32G

qemu.bat

qemu-system-aarch64 -m 4096 -cpu cortex-a72 -smp 4 -M virt -nographic -bios QEMU_EFI.fd -drive if=none,file=ubuntu-20.04-server-cloudimg-arm64.img,id=hd0 -device virtio-blk-device,drive=hd0 -drive file=user-data.img,format=raw -device virtio-net-device,netdev=net0 -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0

ubuntu-16.04-server-cloudimg-arm64-uefi1.png

user-data.zip

Building the Kernel

You must set the following flags when you build the kernel:

CONFIG_KEXEC=y
CONFIG_DEBUG_INFO=y
CONFIG_CRASH_DUMP=y
CONFIG_SYSFS=y
CONFIG_PROC_VMCORE=y

• 리눅스커널 - vmcore를 받기 위한 kdump 설정 방법 II (Ubuntu)
• Raspberry Pi Kdump

Install kdump and crash utility

sudo apt -y install linux-crashdump crash

Kdump setup

crashkernel=512M-:192M 이 들어가 있어야 함.

$ cat /etc/default/grub.d/kdump-tools.cfg 
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT crashkernel=512M-:192M"

$ sudo update-grub

reboot 후 확인하기

Login root

sudo -s

is_kdump_ok.sh 스크립트 참고

#!/bin/bash

echo -e "##################################################"
echo -e ">> cat /proc/cmdline"
echo -e "##################################################"
cat /proc/cmdline

echo -e "##################################################"
echo -e ">> dmesg | grep -i crash"
echo -e "##################################################"
dmesg | grep -i crash

echo -e "##################################################"
echo -e ">> cat /proc/iomem | grep 'Crash kernel'"
echo -e "##################################################"
cat /proc/iomem | grep 'Crash kernel'

echo -e "##################################################"
echo -e ">> cat /sys/kernel/kexec_crash_loaded"
echo -e "##################################################"
cat /sys/kernel/kexec_crash_loaded

echo -e "##################################################"
echo -e ">> kdump-config show"
echo -e "##################################################"
kdump-config show

dbgsym-ubuntu2004.sh 스크립트 참고

#!/bin/bash

codename=$(lsb_release -cs)

sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename}      main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

sudo apt update -y
sudo apt install -y ubuntu-dbgsym-keyring
sudo apt update -y
sudo apt install -y linux-image-$(uname -r)-dbgsym

Let kernel panic

echo c > /proc/sysrq-trigger

ARM64 NUMA

• https://futurewei-cloud.github.io/ARM-Datacenter/qemu/how-to-configure-qemu-numa-nodes/
• https://github.com/rf972/lisa-qemu

qemu-system-aarch64 -m 16384 \
    -cpu cortex-a72 -smp cpus=8 \
    -numa node,cpus=0-1,nodeid=0 -numa node,cpus=2-3,nodeid=1 -numa node,cpus=4-5,nodeid=2 -numa node,cpus=6-7,nodeid=3 \
    -M virt -M gic_version=3 \
    -nographic -bios QEMU_EFI.fd \
    -drive if=none,file=ubuntu-18.04-server-cloudimg-arm64.img,id=hd0 -device virtio-blk-device,drive=hd0 
    -drive file=user-data.img,format=raw \
    -device virtio-net-device,netdev=net0 \
    -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0

# -numa node,cpus=0,nodeid=0 -numa node,cpus=1,nodeid=1 \
# -enable-kvm

~/qemu/build/qemu-system-aarch64 -S \
        -cpu cortex-a72 \
        -smp cpus=2,maxcpus=2 \
        -machine type=virt -machine gic_version=3 \
        -m 2048 \
        -nographic \
        -no-reboot \
        -drive file=busybox/root.bin,format=raw,id=hd0 \
        -device virtio-blk-device,drive=hd0 \
        -kernel linux/arch/arm64/boot/Image \
        -append "root=/dev/vda console=ttyAMA0 nokaslr" -s

Debug Kernel Config

• https://git.rockylinux.org/staging/rpms/kernel/-/blob/r9/SOURCES/kernel-aarch64-debug-rhel.config

Add more debug config

• https://gist.github.com/paranlee/f3cb2fa8568b93cd5110cfad4fe5bd84

QEMU

• https://hlyani.github.io/notes/openstack/qemu_make.html
• https://www.bhral.com/post/practice-risc-v-linux-kernel-qemu를-활용하여-동적-분석하기

The Linux kernel .deb custom kernel package compiled with extra debugging enabled.

커널의 디버그 설정을 활성화한 커널-디버그 패키지를 Ubuntu에서 만드는 방법을 소개합니다.

Ubuntu

우분투 배포판 커널 GIT 레포지토리에서 소스를 클론합니다.

• https://code.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy

설치에 필요한 패키지 설치

 apt install -y build-essential debhelper kernel-wedge gawk

cd jammy
chmod a+x debian/rules
chmod a+x debian/scripts/*
chmod a+x debian/scripts/misc/*

menuconfig
LANG=C fakeroot debian/rules editconfigs

디버그 커널은 generic 커널 기준으로 빌드할 예정이므로,
중간에 다른 아키텍쳐 또는 generic 이외의 lowlatency 커널에 대한 설정이 나오면, ctrl + c 를 눌러서 종료합니다.

# Clean
LANG=C fakeroot debian/rules clean
# you need to go through each (Y, Exit, Y, Exit..) or get a complaint about config later
LANG=C fakeroot debian/rules editconfigs

기본 컨피그에
debian.master/config/amd64/config.common.amd64

flavour 컨피그를 오버라이드하는 한 다음 빌드에 사용하는 형식입니다.
debian.master/config/amd64/config.flavour.generic

우분투 커널 디버그 파일 예시

# 클린
LANG=C fakeroot debian/rules clean

# 빌드
LANG=C fakeroot debian/rules binary-headers binary-generic binary-perarch skipdbg=false

# 클린 & 빌드
LANG=C fakeroot debian/rules clean
LANG=C fakeroot debian/rules binary-headers binary-generic binary-perarch skipdbg=false

4 NUMA Nodes

qemu-system-aarch64 --version
QEMU emulator version 6.0.0 (v6.0.0-11869-g800a25ea45-dirty)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

qemu-system-aarch64 -m 16384M,slots=4,maxmem=24G -cpu cortex-a72 -smp cpus=8 -object memory-backend-ram,id=mem0,size=4096M -numa node,cpus=0-1,nodeid=0,memdev=mem0 -object memory-backend-ram,id=mem1,size=4096M -numa node,cpus=2-3,nodeid=1,memdev=mem1 -object memory-backend-ram,id=mem2,size=4096M -numa node,cpus=4-5,nodeid=2,memdev=mem2 -object memory-backend-ram,id=mem3,size=4096M -numa node,cpus=6-7,nodeid=3,memdev=mem3 -M virt -nographic -bios QEMU_EFI.fd -drive if=none,file=ubuntu-18.04-server-cloudimg-arm64.img,id=hd0 -device virtio-blk-device,drive=hd0 -drive file=user-data.img,format=raw -device virtio-net-device,netdev=net0 -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0

ubuntu@ubuntu:~$ sudo numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7
cpubind: 0 1 2 3
nodebind: 0 1 2 3
membind: 0 1 2 3

ubuntu@ubuntu:~$ sudo numastat
                           node0           node1           node2           node3
numa_hit                 1593718         1702695         1603111         1812539
numa_miss                      0               0               0               0
numa_foreign                   0               0               0               0
interleave_hit              9776            9812            9782            9787
local_node               1593088         1692547         1592684         1802108
other_node                   630           10148           10427           10431

https://gist.github.com/brion/3f68154b4c19e966d0ec616ac8e1988d

https://cdn.kernel.org/pub/linux/kernel/people/will/docs/qemu/qemu-arm64-howto.html