Skip to content

Instantly share code, notes, and snippets.

@daemonhorn

daemonhorn/FreeBSD Arm64 Qemu.md

Last active October 3, 2023 12:20
Show Gist options
  • Save daemonhorn/a3a3d271a7e3f4528975cdcbcfc06c71 to your computer and use it in GitHub Desktop.
Save daemonhorn/a3a3d271a7e3f4528975cdcbcfc06c71 to your computer and use it in GitHub Desktop.
Download ZIP
FreeBSD-arm64-aarch64 via Qemu from ports
Raw
FreeBSD Arm64 Qemu.md

Intro

This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or newer), and provides an example configuration for running arm64 (aarch64) FreeBSD guest on an amd64 FreeBSD Host. Concepts can be applied to other architectures as desired, but syntax and capabilities will vary.

Dependancies

  • Install qemu pkg install qemu or pkg instal qemu-nox11. Latest pre-built package release as of this writeup is 8.1.0
  • Sufficient disk space (50+GB) on a mounted FreeBSD Host disk (e.g.: /qemu-data in this example)
  • Network environment that allows for multiple mac addresses on one switch port (or vswitch port configuration) for bridging mode

Getting Started

  1. Download desired qcow2 image from FreeBSD site, decompress, and resize (to increase by desired size. Example shows adding 40GB):
mkdir /qemu-data
cd /qemu-data
fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA4/aarch64/Latest/FreeBSD-14.0-BETA4-arm64-aarch64.qcow2.xz
unxz FreeBSD-14.0-BETA4-arm64-aarch64.qcow2.xz
qemu-img resize FreeBSD-14.0-BETA4-arm64-aarch64.qcow2 +40G
  1. Configure networking bridge for use by Qemu guests (replace vmx0 with host network interface name). Note: 802.11 Wireless (Wi-Fi) host NICs usually will have issues with bridging, so stick with wired ethernet
ifconfig bridge0 create
ifconfig bridge0 addm vmx0 edge vmx0 up

Make permanant in /etc/rc.conf by adding:

ifconfig_bridge0="addm vmx0 edge vmx0 up"
cloned_interfaces="bridge0"
  1. Modify Qemu network ifup/ifdown scripts /usr/local/etc/qemu-if[up|down]
  • Example shown is designed to handle bridge and tap devices automatically as the guest is started/stopped.
    vi /usr/local/etc/qemu-ifup
#!/bin/sh
ifconfig bridge0 addm $1 up
ifconfig $1 up

vi /usr/local/etc/qemu-ifdown

#!/bin/sh
ifconfig $1 down
ifconfig bridge0 deletem $1
  1. Boot Qemu with the pre-installed qcow2 image (text mode). Alternatively, see shell script example below.
qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt  \
        -bios edk2-aarch64-code.fd -serial telnet:localhost:4444,mux=on,server,wait=off -vnc :0,websocket=on \
        -drive if=none,file=/qemu_data/FreeBSD-14.0-BETA4-arm64-aarch64.qcow2,id=hd0 \
        -device virtio-blk-device,drive=hd0 \
        -device virtio-net-device,netdev=net0 \
        -netdev tap,id=net0
  1. Login as root with no password, immediately set new password using passwd. (See below for accessing guest console via telnet)
  2. Create users using adduser, start sshd using service sshd enable && service sshd start
  3. Edit configuration file in /etc/rc.conf and give a hostname. Verify other settings.
  4. Use the new guest VM of Arm64(aarch64). Since this is now a higher tier variant, there will be pre-built packages for use with pkg utility as well as all of the other normal FreeBSD base things. Package builds for betas may lag ISO releases.

Example Qemu startup script

This example sh script provides an arm64/aarch64 qemu guest launch(/qemu-data/start_guest.sh):

  • Command line parameter(optional) to perform qemu host->guest USB passthrough based upon usb device string
    • Supports any USB device mapping from the Host to Guest based upon detected device seen by usbconfig list
    • Supports USB mapping via nested virtualization/emulation (VMWare VM of FreeBSD amd64 hosting Qemu VM of FreeBSD aarch64)
    • e.g.: sh /qemu-data/start_guest.sh fido will check the usb device list for a FIDO USB device, and passthrough that device to the guest on startup. Any device string that matches a substring of usbconfig output should work (e.g. ugen0.4 or fido or Mouse or Yubikey)
  • EFI firmware image from qemu-aarch64 -bios edk2-aarch64-code.fd (loads from /usr/local/share/qemu/ by default)
  • Simple Example telnet console for guest (localhost port 4444) and qemu monitor (localhost port 4445)
  • tap device cleanup (if not in use)
  • Error checking to ensure that qemu is not already running.
  • Machine Configuration: CPU = 4, RAM = 4GB (Customize as desired)
  • Adjust FreeBSD-14.0-BETA4-arm64-aarch64.qcow2 disk image name to match applicable qcow2 image desired.
#!/bin/sh
# Safety check.
ps -aux | grep -v grep | grep -cq qemu-system && \
        echo "Qemu is already running. Shutdown the guest(s), then retry." && exit 1
# if arg(1) is set, use it as a string to match on for usb passthrough to guest.
if [ ! -z ${1} ] ; then
        echo "Attempting to passthrough usb host device based on query string: ${1}"
        usb_map_count=$(usbconfig | grep -cie ${1})
        [ ${usb_map_count} -ne 1 ] && \
                 echo "Total devices matched: ${usb_map_count} is not equal to 1, please refine." && \
                 usbconfig && exit 1
        usb_map=$(usbconfig | grep -ie ${1} | grep -o -e [0-9]\.[0-9])
        usb_map_bus=$(echo ${usb_map} | grep -o -e ^[0-9])
        usb_map_addr=$(echo ${usb_map} | grep -o -e [0-9]$)
        usb_qemu_cli="-device usb-host,hostbus=${usb_map_bus},hostaddr=${usb_map_addr},id=${1}"
        echo "Mapping usb device $(usbconfig | grep -ie ${1}) into the guest."
        echo -n "In qemu monitor, you can inspect attached usb guest devices with \"info usb\" "
        echo "command, or delete the usb device mapping with \"device_del ${1}\""
fi
# Cleanup tap0 interfaces that are not in use anymore.
ifconfig tap0 2>/dev/null | grep -cq -e "Opened by PID" || ifconfig tap0 destroy 2>/dev/null
echo "Starting Qemu in background..."
qemu-system-aarch64 -m 4096M -cpu max -smp cpus=4 -M virt \
        -bios edk2-aarch64-code.fd \
        -serial telnet:localhost:4444,mux=on,server,wait=off \
        -monitor telnet:localhost:4445,mux=on,server,wait=off \
        -display none \
        -drive if=none,file=/qemu-data/FreeBSD-14.0-BETA4-arm64-aarch64.qcow2,id=hd0 \
        -device virtio-blk-device,drive=hd0 \
        -device virtio-net-device,netdev=net0 \
        -netdev tap,id=net0 \
        -usb \
        -device qemu-xhci,id=xhci \
        ${usb_qemu_cli} \
        -daemonize
echo "Connect to guest console (telnet localhost 4444), or qemu monitor (telnet localhost 4445)"

Access Qemu guest console

Connect over telnet protocol from localhost to the guest console. Note: Use Ctrl+] to get a telnet prompt, then quit to exit telnet

telnet localhost 4444

Access Qemu monitor console

Connect over telnet protocol from localhost to the qemu monitor. Note: Use Ctrl+] to get a telnet prompt, then quit to exit telnet. Typing quit at monitor (qemu) prompt non-gracefully kills qemu guest.

telnet localhost 4445

Example monitor commands

See https://qemu-project.gitlab.io/qemu/system/monitor.html for complete details.

  • system_reset - Reboot the vm
  • system_pwerdown - Send Powerdown event (may require qemu-guestd)
  • device_add usb-host,hostbus=xxx,hostaddr=yyy,id=friendlyname
  • device_del id (where id=friendlyname)
  • info status to see if the vm is running/paused
  • info cpu to check how many cpus are running/pids
  • info usb or info usbhost for information on usb guest/host devices respectively.
  • info network for network configuration and mac address
  • info chardev for serial/character device configuration.

Example Fresh Install from ISO

  1. Prepare to install:
  • Download iso file (example is 13.2-RELEASE-AARCH64-bootonly variant )
fetch https://download.freebsd.org/releases/ISO-IMAGES/13.2/FreeBSD-13.2-RELEASE-arm64-aarch64-bootonly.iso
  • Create new blank drive image using qemu-img create
qemu-img create -f qcow2 FreeBSD-13.2-Release.qcow2 50G
  • Make modifications to qemu command line (change/addition from example above) to attach new qcow2 disk and virtual cdrom for ISO
  -cdrom /qemu-data/FreeBSD-13.2-RELEASE-arm64-aarch64-bootonly.iso \
  -drive if=none,file=/qemu-data/FreeBSD-13.2-Release.qcow2,id=hd0 \
  • Start new amd64 aarch64 13.2 guest and start installation by following the prompts.
sh /qemu-data/start_guest.sh
telnet localhost 4444
  • After installation is complete and system reboots, you can remove the -cdrom... line from the shell script as desired.

Yubikey usb host->guest passthrough with hidraw(4) usb driver

  • Untested if only guest has hidraw(4) kernel module loaded. (should work)
  • Seems to require qemu-xhci virtual usb device, ehci variant was problematic for unknown reason.
  • Works if both host and guest have enabled hidraw kernel module and sysctl. Usually second device /dev/hidraw1 is correct, depends on kldstat -v including hkbd or not. Example /boot/loader.conf:
hidraw_load="YES"
hkbd_load="YES"
hw.usb.usbhid.enable="1"
  • Requires patches to libfido (included in 14.0-RELEASE, and stable after 9/23/23) - libfido2 1.13 for automatic detection
  • Manual workaround for older OS code is to pass device entry explicitly ssh-keygen -v -K -O device=/dev/hidraw1
  • Using string fido or yubikey for device selection and passthrough and above script works.

TODO

  • Write a detection script to check if there is a pid attached, and destroy if not: ifconfig tap0 | grep -cq -e "Opened by PID" || ifconfig tap0 destroy
  • Document USB Yubikey passthrough from ESXi->FreeBSD VM(amd64)->FreeBSD QEMU(aarch64)
  • Document ISO mapping syntax for install media
  • Document Fresh aarch64 install via 13.2 bootonly ISO
  • Document qemu-guest-agent install and configuration
  • Document u2f virtual device (emulated and passthrough) - ? - needs investigation
  • Determine why attempting to destroy tapX interface using ifconfig tap0 destroy format hangs instead of erroring out when there is still a pid associated with the interface.

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment