Running dockerd on linux/arm64 with QEMU

dockerd-on-qemu-arm64.md

Running dockerd on linux/arm64 with QEMU

Why

• You need to pull, build, or push linux/arm64 Docker images with tooling which does not support Docker's experimental --platform argument.
• You are otherwise blocked by moby/moby#36552.
• You are unable to configure your local dockerd and/or cannot use user-space emulation with qemu-user-static (see below).

Why not

• Emulating a full aarch64 system is sloooow 🐌
• Consider using multiarch/qemu-user-static for building linux/arm64 images with binfmt. It is much easier and a lot faster!

Prerequisites

You will need curl, QEMU with AArch64 support, and genisoimage.

We are using Atomic Linux, as it is one of the few Linux distributions with pre-built aarch64 images which have dockerd already part of the base image.

OS_URL="https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-29-updates-20190121.0/AtomicHost/aarch64/images/Fedora-AtomicHost-29-20190121.0.aarch64.qcow2"
OS_IMG=$(basename "${OS_URL}")
test -f ${OS_IMG} || curl -L -o ${OS_IMG} "${OS_URL}"

AArch64 uses UEFI, for which we can fetch the necessary firmware image as follows:

UEFI_URL="https://releases.linaro.org/components/kernel/uefi-linaro/16.02/release/qemu64/QEMU_EFI.fd"
UEFI_IMG=$(basename "${UEFI_URL}")
test -f ${UEFI_IMG} || curl -L -o ${UEFI_IMG} "${UEFI_URL}"

Configuring the Guest VM

We are using Cloud-Init to configure the guest and expose an unauthenticated dockerd remote API on port 2375. Create the required meta-data and user-data file as follows:

cat > meta-data <<EOF
instance-id: iid-local01
local-hostname: hydrogen
EOF

cat > user-data <<EOF
#cloud-config
bootcmd:
  - ["cloud-init-per", "once", "docker_opts", "sed", "-i", "s|^OPTIONS='[^']*|& -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375|", "/etc/sysconfig/docker"]
EOF

Enable SSH access (optional)

By default, you will not be able to login into your Atomic Linux guest at all. If you would like to access it via SSH, you will need to create an authorized user:

cat >> user-data <<EOF
users:
  - name: ${USER}
    groups: wheel, docker
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    ssh-authorized-keys:
      - '$(cat ~/.ssh/id_rsa.pub)'
EOF

In addition, you will need to forward the SSH port on your QEMU host. Append the following forwarding rule to the -netdev argument in the qemu-system-aarch64 invocation below: hostfwd=tcp:127.0.0.1:2222-:22

You will then be able to login in to your guest (once it is booted) with the following command: ssh -o NoHostAuthenticationForLocalhost=yes -p 2222 localhost

Prepare the disk images

The guest VM expects the above cloud-init files as a ISO9660 image:

CLOUD_ISO="cloud.iso"
genisoimage -output "${CLOUD_ISO}" -volid cidata -joliet -rock user-data meta-data

To avoid unwanted changes to the Atomic Linux boot disk, create a copy-on-write disk image as follows:

DISK_IMG="atomic.img"
qemu-img create -f qcow2 -b "${OS_IMG}" "${DISK_IMG}"

Booting up the Guest

The following command will boot up a AArch64 guest with two cores and 2GB of RAM. It will print all the serial console output to your terminal. To terminate it, press and release CTRL+A then press x.

qemu-system-aarch64 \
    -smp 2 \
    -m 2048 \
    -M virt \
    -cpu cortex-a57 \
    -bios ${UEFI_IMG} \
    -nographic \
    -device virtio-rng-pci \
    -device virtio-blk-device,drive=image \
    -drive if=none,id=image,file=${DISK_IMG} \
    -device virtio-blk-device,drive=cloud \
    -drive if=none,id=cloud,file=${CLOUD_ISO},format=raw \
    -device virtio-net-device,netdev=net0 \
    -netdev user,id=net0,hostfwd=tcp:127.0.0.1:2375-:2375

Access dockerd from your guest

Once the guest VM is fully booted (after 5-10 minutes), you will be able to access the Docker API locally on TCP port 2375:

export DOCKER_HOST=localhost:2375
docker version

You should see that your server engine is now running on linux/arm64:

Server:
 Engine:
  Version:          1.13.1
  API version:      1.26 (minimum version 1.12)
  Go version:       go1.11beta2
  Git commit:       baec8f1-unsupported
  Built:            Wed Jul 25 18:57:04 2018
  OS/Arch:          linux/arm64
  Experimental:     false