🖥 Lightweight diskless VM with QEMU

README.md

🖥 Lightweight diskless VM with QEMU

This document describes a process to quickly setup a Linux VM for testing or kernel development.

The VM requires no disk image, instead it boots off a directory on the host (using virtiofs). This is:

• more performant (no FS-in-FS overhead)
• more convenient for experiments (both host and guest can modify files concurrently)
• safer (no corruption* in case of kernel panic)
• easier to set up

The lightweight microvm platform is used, with minimal VIRTIO hardware (serial console, filesystem, network, ballooning).

It assumes the user has Arch Linux, but shouldn't be hard to adapt to other distros (except for the part of creating the rootfs directory, which is highly distro-dependent).

Steps

• Prerequisites:

  sudo pacman -S qemu-system-x86 qemu-virtiofsd arch-install-scripts

• Build a recent x86 Linux kernel (see kernel config below) and cd to its source tree.

• Prepare the rootfs directory for the VM:

  sudo mkdir qemu-rootfs   # <- sudo, so that directory is owned by root
  sudo mount -m --bind qemu-rootfs qemu-rootfs-bind
  sudo pacstrap -K qemu-rootfs-bind base
  sudo arch-chroot qemu-rootfs-bind passwd   # <- set a root password
  sudo make modules_install INSTALL_MOD_PATH=qemu-rootfs

• Place the provided qemu script in the directory, and run it to start the VM:

  curl -fLO https://gist.github.com/mildsunrise/08f7fbe5da4378ad855fa51150cf2403/raw/qemu
  chmod +x qemu
  ./qemu

Using the network

The qemu script adds a network interface with the user backend, which requires no setup or special host privileges. You can use it to access the Internet from the VM, just make sure the interface is configured with DHCP, i.e.:

cat > /etc/systemd/network/20-wired.network <<EOF
[Match]
Name=eth0

[Network]
DHCP=yes
EOF

systemctl enable systemd-networkd
systemctl enable systemd-resolved
reboot

The script also sets up a port redirection that makes the VM's SSH port accessible at port 22001 on the host. To use it, start an SSH server:

pacman -S openssh
# enable root-login in /etc/ssh/sshd-config
systemctl enable sshd
reboot

And use ssh -p 22001 root@localhost. For convenience you can set up an entry in your ~/.ssh/config:

Host linux-vm
Hostname localhost
Port 22001
User root

Then use ssh linux-vm.

Kernel config

For the kernel to work with this setup, you'll need to enable at least:

• CONFIG_SERIAL_8250_PNP=y
• CONFIG_VIRTIO_MMIO=y
• CONFIG_VIRTIO_FS=y [more info]
• CONFIG_VIRTIO_BALLOON=y or m
• CONFIG_VIRTIO_NET=y or m

qemu

#!/bin/sh
cd "$(dirname "$0")"

sudo /usr/lib/qemu/virtiofsd \
  --socket-path=qemu-virtiofs.sock --socket-group="$(id -grn)" \
  -o source="$PWD"/qemu-rootfs -o cache=always &

sleep 0.1s

# FIXME: enable DAX when upstreamed

exec qemu-system-x86_64 \
  -enable-kvm -cpu host \
  -smp 1 -m 512 \
  \
  -nodefaults -no-user-config -nographic \
  -machine microvm,memory-backend=mem,x-option-roms=off,pit=off,pic=off,isa-serial=on,rtc=off \
  -bios /usr/share/qemu/qboot.rom \
  -global virtio-mmio.force-legacy=false \
  -serial stdio \
  \
  -device virtio-balloon-device \
  \
  -chardev socket,id=char0,path=qemu-virtiofs.sock \
  -device vhost-user-fs-device,queue-size=1024,chardev=char0,tag=rootfs \
  -object memory-backend-memfd,id=mem,size=512M,share=on \
  \
  -netdev user,id=net0,hostfwd=tcp:127.0.0.1:22001-:22 \
  -device virtio-net-device,netdev=net0 \
  \
  -kernel arch/x86/boot/bzImage \
  -append "earlyprintk=ttyS0 console=ttyS0 rootfstype=virtiofs root=rootfs rw"