Skip to content

Instantly share code, notes, and snippets.

@Abyss-W4tcher

Abyss-W4tcher/AArch64_Android_emulation_and_kernel_cross-compilation.md

Last active February 12, 2024 00:47
Show Gist options
  • Save Abyss-W4tcher/f1833623c975193446315d48c106750e to your computer and use it in GitHub Desktop.
Save Abyss-W4tcher/f1833623c975193446315d48c106750e to your computer and use it in GitHub Desktop.
Download ZIP
AArch64 Android emulation and kernel cross-compilation
Raw
AArch64_Android_emulation_and_kernel_cross-compilation.md

AArch64 Android emulation and kernel cross-compilation

The following assumes you are using a AArch64 host.

Android SDK installation

Setup SDK and emulator :

# https://developer.android.com/studio/index.html#command-line-tools-only
sudo apt-get install unzip openjdk-17-jdk gradle -y
mkdir -p Android/cmdline-tools
wget https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip -O commandlinetools-linux-11076708_latest.zip
unzip commandlinetools-linux-11076708_latest.zip -d Android/cmdline-tools
mv Android/cmdline-tools/cmdline-tools Android/cmdline-tools/latest
export ANDROID_HOME=$(pwd)/Android
export PATH="$ANDROID_HOME/emulator:$ANDROID_HOME/cmdline-tools/latest:$ANDROID_HOME/cmdline-tools/latest/bin:$ANDROID_HOME/tools:$ANDROID_HOME/tools/bin:$ANDROID_HOME/platform-tools:$PATH"

# Get emulator download link from https://ci.android.com/builds/branches/aosp-emu-master-dev/grid (needs a manual download, or get the link from your navigator downloads and paste it in the following wget parameter)
# e.g. https://ci.android.com/builds/submitted/8632828/emulator-linux_aarch64/latest/sdk-repo-linux_aarch64-emulator-8632828.zip
wget '' -O emulator.zip # 31.3.8
unzip -qq emulator.zip -d Android/

# Edit the XML <revision> tag at the end with your emulator version
curl 'https://chromium.googlesource.com/android_tools/+/refs/heads/main/sdk/emulator/package.xml?format=TEXT' | base64 -d > Android/emulator/package.xml

Setup dependencies :

yes | sdkmanager --licenses
sdkmanager "build-tools;34.0.0" "platform-tools" "platforms;android-34" "tools"

Ressources

Emulation

Fetch avd system image :

# sdkmanager --list # List available packages
TARGET_IMAGE='system-images;android-27;default;arm64-v8a'
sdkmanager $TARGET_IMAGE

Create avd :

# If you encounter any java error when launching avdmanager, please follow https://stackoverflow.com/a/62610046
yes '' | avdmanager create avd -n my_avd -k $TARGET_IMAGE

Start emulation :

# Headless
emulator -avd my_avd -no-snapshot -no-window

# Graphical
emulator -avd my_avd -no-snapshot -gpu swiftshader_indirect

Delete avd :

avdmanager delete avd -n my_avd

Compile custom android kernel

We will try to get as close as possible to kernels used in Android sdkmanager available devices.

Docker

We will be using a docker container to keep our host machine clean from apt packages.

Place yourself in the top directory where you installed the Android SDK (e.g. emu_android/).

Launch and access the container :

docker run -d -it --name android_compil -w /bind/ -v $(pwd):/bind gcc:9.5
docker exec -it android_compil bash

Install common dependencies :

apt-get update
# https://source.android.com/docs/setup/build/initializing?hl=fr
apt-get install -y git-core gnupg flex bison build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 libncurses5 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z1-dev libgl1-mesa-dev libxml2-utils xsltproc unzip fontconfig
apt-get install -y python3 python-is-python3 wget bc cpio

GCC (older kernels)

Get target kernel informations

As we want to mimic the kernel from our avd (system-images;android-27;default;arm64-v8a), we will need to extract some informations :

$ adb shell cat /proc/version
Linux version 3.18.94+ (emulator@emulator-ThinkPad-P50) (gcc version 4.9.x 20150123 (prerelease) (GCC) ) #17 SMP PREEMPT Mon May 1 07:31:27 PDT 2023
$ adb shell getprop ro.build.version.release
8.1.0
$ adb pull /proc/config.gz
$ gzip -dc config.gz > .config

Kernel cross-compilation

Following assumes you are in the container terminal, at following path :

cd /bind/

Setup the toolchain :

# GCC toolchain
git clone https://android.googlesource.com/platform/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9
cd aarch64-linux-android-4.9
# Notice how we are using a tag close to the identified "8.1.0" android version
git checkout android-8.1.0_r1
cd ../

Clone the kernel and navigate to its root tree :

# Find kernel source commit from Makefile log history : https://android.googlesource.com/kernel/goldfish.git/+log/refs/heads/android-goldfish-3.18/Makefile
TARGET_KERNEL_COMMIT='7aa67ee8fa0849d41ee8081f74bf77fdb9eb43d7'
mkdir -p kernel
cd kernel
wget https://android.googlesource.com/kernel/goldfish.git/+archive/$TARGET_KERNEL_COMMIT.tar.gz
mkdir $TARGET_KERNEL_COMMIT/
tar -xf $TARGET_KERNEL_COMMIT.tar.gz -C $TARGET_KERNEL_COMMIT/
cd $TARGET_KERNEL_COMMIT/

Export needed cross compilation variables :

export CC_PATH=/bind/aarch64-linux-android-4.9/bin
export ARCH=arm64 SUBARCH=arm64 CROSS_COMPILE=$CC_PATH/aarch64-linux-android-

Cross compile the kernel :

# Use our existing AVD config (RECOMMENDED)
cp /bind/.config . ||  make defconfig # "make help" to list all possible targets

# You need to toggle on/off the following with "make menuconfig" :
# (https://cateee.net/lkddb/web-lkddb/DEVMEM.html) && (NOT https://cateee.net/lkddb/web-lkddb/HARDENED_USERCOPY.html) -> needed for AVML memory capture
make menuconfig 

make -j$(nproc) LOCALVERSION="-custom-android" KDEB_PKGVERSION="$(make kernelversion)-1" Image

LLVM toolchain (more recent kernels)

Get target kernel informations

As we want to mimic the kernel from our avd (system-images;android-31;default;arm64-v8a), we will need to extract some informations :

$ adb shell cat /proc/version
Linux version 5.10.110-android12-9-00004-gb92ac325368e-ab8731800 (build-user@build-host) (Android (7284624, based on r416183b) clang version 12.0.5 (https://android.googlesource.com/toolchain/llvm-project c935d99d7cf2016289302412d708641d52d2f7ee), LLD 12.0.5 (/buildbot/src/android/llvm-toolchain/out/llvm-project/lld c935d99d7cf2016289302412d708641d52d2f7ee)) #1 SMP PREEMPT Tue Jun 14 13:40:53 UTC 2022
$ adb shell getprop ro.build.version.release
12
$ adb pull /proc/config.gz
$ gzip -dc config.gz > .config

Kernel cross-compilation

Following assumes you are in the container terminal, at following path :

cd /bind/

Setup the toolchain :

# Fetch cross compilation dependencies
# Determine the NDK version to fetch based on clang version
# https://github.com/search?q=repo%3Aandroid%2Fndk+r416183b&type=wikis
# https://github.com/android/ndk/wiki/Unsupported-Downloads
NDK_VERSION='23'
wget https://dl.google.com/android/repository/android-ndk-r$NDK_VERSION-linux.zip
unzip android-ndk-r$NDK_VERSION-linux.zip

Clone the kernel and navigate to its root tree :

# Find kernel source commit from Makefile log history : https://android.googlesource.com/kernel/common/+log/refs/heads/android12-5.10/Makefile
TARGET_KERNEL_COMMIT='95f4203fc961cd30eddc01f4c418c6bcf2f0378f'
mkdir -p kernel
cd kernel
wget https://android.googlesource.com/kernel/common/+archive/$TARGET_KERNEL_COMMIT.tar.gz
mkdir $TARGET_KERNEL_COMMIT/
tar -xf $TARGET_KERNEL_COMMIT.tar.gz -C $TARGET_KERNEL_COMMIT/
cd $TARGET_KERNEL_COMMIT/

Export cross compiler binaries path :

export CC_PATH=/bind/android-ndk-r$NDK_VERSION/toolchains/llvm/prebuilt/linux-x86_64/bin/

Cross compile the kernel :

# Use our existing AVD config (RECOMMENDED)
cp /bind/.config . || PATH="${CC_PATH}:${PATH}" make gki_defconfig \
                      ARCH=arm64 \
                      CC=clang \
                      CROSS_COMPILE=aarch64-linux-android- \
                      LLVM=1 # "make help" to list all possible targets

# You need to toggle on/off the following with "make menuconfig" :
# (https://cateee.net/lkddb/web-lkddb/DEVMEM.html) && (NOT https://cateee.net/lkddb/web-lkddb/HARDENED_USERCOPY.html) -> needed for AVML memory capture
PATH="${CC_PATH}:${PATH}" make menuconfig \
                      ARCH=arm64 \
                      CC=clang \
                      CROSS_COMPILE=aarch64-linux-android- \
                      LLVM=1

# You might encounter errors when compiling kernels, and will need to fix it by hand
# This occurs mostly when using non stable kernels
# here : ./scripts/config --disable CONFIG_UNUSED_KSYMS_WHITELIST
PATH="${CC_PATH}:${PATH}" make LOCALVERSION="-custom-android" KDEB_PKGVERSION="$(make kernelversion)-1" -j$(nproc --all) Image \
                      ARCH=arm64 \
                      CC=clang \
                      CROSS_COMPILE=aarch64-linux-android- \
                      LLVM=1

Emulation with custom kernel

Create avd :

# If you encounter any java error when launching avdmanager, please follow https://stackoverflow.com/a/62610046
yes '' | avdmanager create avd -n my_avd -k $TARGET_IMAGE

Start emulation :

# Headless
emulator -avd my_avd -no-snapshot -no-window -kernel [PATH TO : arch/arm64/boot/Image]

Ressources

Memory capture

AVML

Cross build AVML on your host :

git clone https://github.com/microsoft/avml.git
cd avml 

# Install MUSL
sudo apt-get install musl-dev musl-tools musl

# Install Rust via rustup
curl https://sh.rustup.rs -sSf | sh -s -- -y
exec $SHELL 

# Install cross
cargo install cross --git https://github.com/cross-rs/cross

# Build
cross build --release --target aarch64-linux-android

# Verify architecture
file target/aarch64-linux-android/release/avml

Capture memory of your running AVD device :

# Memory capture
adb root 
adb push avml /data/local/tmp/
adb shell 'chmod +x /data/local/tmp/avml'
adb shell '/data/local/tmp/avml /data/local/tmp/avml_dump.lime'
adb pull /data/local/tmp/avml_dump.lime .

If you encounter errors like "write_error" with AVML, this means the memory dump exceeds the VM storage size. An easy fix is to run the avd with -qemu -m 512 (512MB of RAM).

LiME

Cross compile LiME module

When available, prefer using AVML.

Go in the container, where you compiled the kernel :

cd /bind/
git clone https://github.com/504ensicsLabs/LiME.git
cd LiME/src
cp Makefile Makefile.backup
echo '' > Makefile

Replace the makefile content with the following (be sure to set the correct CCPATH and KERNEL_SOURCE) :

obj-m := lime.o
lime-objs := tcp.o disk.o main.o hash.o deflate.o

KERNEL_SOURCE := /bind/kernel/
PWD := $(shell pwd)
#CCPATH := /bind/aarch64-linux-android-4.9/bin/
#CCPATH := /bind/android-ndk-rXX/toolchains/llvm/prebuilt/linux-x86_64/bin/

default:
	# cross-compile for Android emulator
	$(MAKE) ARCH=arm64 CROSS_COMPILE=$(CCPATH)/aarch64-linux-android- -C $(KERNEL_SOURCE EXTRA_CFLAGS=-fno-pic M="$(PWD)" modules

	$(MAKE) tidy

tidy:
	rm -f *.o *.mod.c Module.symvers Module.markers modules.order \.*.o.cmd \.*.ko.cmd \.*.o.d
	rm -rf \.tmp_versions

clean:
	$(MAKE) tidy
	rm -f *.ko

Then, follow https://github.com/504ensicsLabs/LiME?tab=readme-ov-file#examples-

Fix for an error occuring when inserting LiME (logged in dmesg) on a 3.18.94 kernel :

adb snapshot

Using this method to capture memory isn't advised, as many were proven unusable with Volatility3 :

adb emu avd snapshot save my_snapshot
# will be locate here : ~/.android/avd/my_avd.avd/snapshots/my_snapshot/ram.bin

Additional ressources

Memory layout debugging

cat /proc/iomem
cat /sys/kernel/debug/kernel_page_tables

Using an SDCARD with AVD

Status : sdcard is detected but size isn't correct. I was needing an SDCARD to store the AVML dump when the AVD was configured with a significant RAM size.

mksdcard -l mysdcard 2048M mysdcard.img

avdmanager delete avd -n my_avd
yes '' | avdmanager create avd -n my_avd -k $TARGET_IMAGE -c $(realpath mysdcard.img) # -c 2G
emulator -avd my_avd -no-snapshot -no-window -kernel Image -verbose -show-kernel -qemu -m 512 # -sdcard $(realpath mysdcard.img)

Interesting AVML volatile extraction method (bypass space limits of the AVD):

AArch64 Android emulation on x86_64

Status : kernel never boots up completely

# https://thecodeartist.blogspot.com/2011/12/booting-android-completely-over.html
emulator -avd vol3 -cores $(nproc) -noaudio -no-snapshot-load -show-kernel -qemu -machine virt

# SELINUX errors
emulator -avd vol3 -noaudio -no-snapshot-load -show-kernel -qemu -machine virt -append 'androidboot.selinux=permissive security=selinux selinux=0 rootdelay=1' -kernel arch/arm64/boot/Image

Useful links :

Android studio

# https://ubunlog.com/fr/android-studio-instalacion-ubuntu/
# https://developer.android.com/studio/install
wget https://redirector.gvt1.com/edgedl/android/studio/ide-zips/2023.1.1.27/android-studio-2023.1.1.27-linux.tar.gz
sudo tar -xvf android-studio-2023.1.1.27-linux.tar.gz -C /opt/
/opt/android-studio/bin/studio.sh
sudo ln -sf /opt/android-studio/bin/studio.sh /bin/android-studio

Compile your own LLVM toolchain

# https://clang.llvm.org/get_started.html
TARGET_TOOLCHAIN_COMMIT='c935d99d7cf2016289302412d708641d52d2f7ee'
wget https://android.googlesource.com/toolchain/llvm-project/+archive/$TARGET_TOOLCHAIN_COMMIT.tar.gz
mkdir $TARGET_TOOLCHAIN_COMMIT/
tar -xf $TARGET_TOOLCHAIN_COMMIT.tar.gz -C $TARGET_TOOLCHAIN_COMMIT/ 
cd $TARGET_TOOLCHAIN_COMMIT
mkdir build && cd build
cmake -DLLVM_ENABLE_PROJECTS="clang" -DCMAKE_BUILD_TYPE=Release -G "Ninja" ../llvm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment