Skip to content

Instantly share code, notes, and snippets.

@atatsu

atatsu/void-arch-vga-passthrough.md

Last active May 27, 2020 15:00
Show Gist options
  • Save atatsu/433ace3b65a137f89837acbadaba4540 to your computer and use it in GitHub Desktop.
Save atatsu/433ace3b65a137f89837acbadaba4540 to your computer and use it in GitHub Desktop.
Download ZIP
Adventures in VGA passthrough land
Raw
void-arch-vga-passthrough.md

Install packages

  • virt-manager
  • libvirt
  • qemu
    • usermod -aG kvm <username>
    • modprobe kvm-amd
  • ovmf
    • isn't available for Void... copying the binaries from my old Arch install and we'll see what happens
  • ebtables (networking)
  • dnsmasq (networking)
  • bridge-utils (networking, obviously)

If ebtables, dnsmasq, or bridge-utils were installed after the libvirtd daemon was already running don't forget to restart it.

Update grub.cfg

After having enabled IOMMU in BIOS also update grub to support it.

  • nvim /etc/default/grub
    • GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on"
  • grub-mkconfig -o /boot/grub/grub.cfg

Also enable static huge pages.

  • nvim /etc/default/grub
    • GRUB_CMDLINE_LINUX="hugepages=8192"
    • default size 2048kB per huge page (16G with above setting)
  • grub-mkconfig -o /boot/grub/grub.cfg

Configure mount point for HugeTLB

I never did this in Arch which makes me wonder if the VM was even using the hugepages shit. That'd be glorious if I had 16G of memory stashed away and then the VM was taking my nix ram. Whatever.

  • mkdir /hugepages
  • mount -t hugetlbfs hugetlbfs /hugepages
  • add entry to fstab for hugepages
    • hugetlbfs /hugepages hugetlbfs defaults 0 0
  • update qemu.conf with hugepages mount mount
    • hugetlbfs_mount = "/hugepages"

Isolate the GPU and USB controllers

Typically a person would add their GPU (by device id) to /etc/modprobe.d/vfio.conf like so:

    options vfio-pci ids=10de:1c03,10de:10f1

However, since I had to hijack my USB controllers by other means I just lumped the GPU in with those. So instead my vfio.conf looks more like this:

# /etc/modprobe.d/vfio.conf
install vfio-pci /usr/local/bin/vfio-pci-override

And the referenced file in /usr/local/bin

# /usr/local/bin/vfio-pci-override

   #!/bin/sh

# IF ANYTHING IN THIS FILE CHANGES THE INITRAMFS MUST BE REGENERATED!

# 0000:04:00.0 0000:04:00.1 -- GPU/HDMI addresses
# 0000:00:12.0 0000:00:12.2 -- USB controller addresses 
DEVS="0000:04:00.0 0000:04:00.1 0000:00:12.0 0000:00:12.2"

echo "Snagging devices for vfio-pci..."

for DEV in $DEVS; do
if [ -e /sys/bus/pci/devices/$DEV ]; then
        # couple of checks that I didn't need with my arch setup
	# for whatever reason
	if [ -e /sys/bus/pci/drivers/ehci-pci/$DEV ]; then
	    echo -n ":: Unbinding [ehci-pci] $DEV..."
	    echo $DEV > /sys/bus/pci/drivers/ehci-pci/unbind
	    echo "done."
	fi
	if [ -e /sys/bus/pci/drivers/ohci-pci/$DEV ]; then
	    echo -n ":: Unbinding [ohci-pci] $DEV..."
	    echo $DEV > /sys/bus/pci/drivers/ohci-pci/unbind
	    echo "done."
	fi
	
	# and now the original bits
	if [ -e /sys/bus/pci/devices/$DEV/driver ]; then
		echo -n ":: Unbinding $DEV..."
		echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver/unbind
		echo "done."
	fi
	echo -n ":: Setting up $DEV..."
	echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
	echo "done."
else
	echo "$DEV does not exist."
fi
done

echo "Finished snagging devices for vfio-pci."

# -i ignores the `install` option so that this script isn't executed
# in a loop (from /etc/modprobe.d/vfio.conf `install` line)
modprobe -i vfio-pci

If it isn't obvious if listing shit in /etc/modprobe.d/vfio.conf devices are listed by id, whereas the script in /sbin references them by address.

Instruct necessary modules to load at boot

Artix / Arch

Edit /etc/mkinitcpio.conf and add the following, in the exact order, to the MODULES:

# /etc/mkinitcpio.conf
MODULES=(... vfio_pci vfio vfio_iommu_type1 vfio_virqfd ...)

Make sure modconf is included in the HOOKS:

# /etc/mkinitcpio.conf
HOOKS=(... modconf ...)

Add the vfio-pci-override script from above into the FILES array:

# /etc/mkinitcpio.conf
FILES=(... /usr/local/bin/vfio-pci-override ...)

Regenerate the initramfs

mkinitcpio -p linux

Void

If necessary create the directory /etc/modules-load.d/. In it create vfio.conf and place inside it:

# /etc/modules-load.d/vfio.conf
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Configure libvirt (ovmf shit)

So... as I said above I copied the binaries from my Arch install and hopefully those will work. We'll see.

Update /etc/libvirt/qemu.conf to reflect the following:

nvram = [
    "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd",
]

Restart libvirtd.

UEFI Firmware not listed/available in imported VM or new setup

It seems as though virt-manager/libvirt fucks up on a semi-regular basis and you can be presented with an error similar to the following:

libvirt did not detect any ovmf/uefi firmware blah blah blah

This can result in your imported VM's Firmware being shown as BIOS (when it should infact be UEFI) or when setting up a new VM being unable to actually select a UEFI firmware and being stuck with only BIOS as a selection.

So far I only have experience with getting an existing VM xml export to work again but I would think you could create a new one and add the below and it would also work. Anyway, edit the VM's config (virsh edit <vm-name>) and locate the section. Ensure that the following to lines are present:

<loader readonly="yes" type="pflash">/usr/share/ovmf/x64/OVMF_CODE.fd</loader>
<nvram>/usr/share/ovmf/x64/OVMF_VARS.fd</nvram>

Update polkit rules

Add a new rule to polkit so that users in the kvm group can manage libvirt with non-root access.

# /etc/polkit-1/rules.d/50-libvirt.rules
/* Allow users in kvm group to manage the libvirt
daemon without authentication */
polkit.addRule(function(action, subject) {
    if (action.id == "org.libvirt.unix.manage" &&
        subject.isInGroup("kvm")) {
            return polkit.Result.YES;
    }
});

Import existing image

virsh define /path/to/image/config.xml --validate

Existing Windows image fails to boot with VirtIO disk

Windows is a douche. It's apparently too stupid to load drivers that it has already been using when importing it in this way. Perform the following to help it fix its shit:

  • after having set the boot disk to IDE load up the vm and open an elevated command prompt
  • instruct Windows to reboot into safe mode (this apparently gets it to include the necessary drivers in the kernel)
    • bcdedit /set safeboot minimal
  • shutdown the vm and change the boot drive back to VirtIO
  • boot up the vm and remove the safe mode boot instruction
    • bcdedit /deletevalue safeboot
  • shutdown the vm again
  • start it up and it should actually boot now using a VirtIO boot drive

Network setup (external access)

Setup a bridge and configure the vm to use it.

Setup the bridge and add host interface to it

Artix/Arch

  • sudo ip link add name br0 type bridge
  • sudo ip link set br0 up
  • sudo ip link set enp6s0 master br0
  • sudo dhclient br0
  • sudo ip addr add dev br0 192.168.11.26/24

Void

  • sudo ip link add br0 type bridge
  • #sudo ip addr flush dev enp6s0
  • sudo dhcpcd --release enp6s0
  • sudo ip link set enp6s0 master br0
  • sudo ip tuntap add dev tap0 mode tap user $(whoami)
  • sudo ip link set tap0 master br0
  • sudo ip link set dev br0 up
  • sudo ip link set tap0 up

Tear down the bridge

  • sudo ip link set br0 down
  • sudo ip link set enp6s0 nomaster
  • sudo ip link set tap0 nomaster
  • sudo ip link delete br0 type bridge
  • sudo ip tuntap del tap0 mode tap
  • sudo dhclient -r enp6s0
  • sudo dhclient enp6s0

Instruct VM to utilize bridge

Virt Manager

  • Network source: Specify shared device name
  • Bridge name: br0

virsh edit domain

<interface type='bridge'>
  ...
  <source bridge='br0'/>
  ...
</interface>

Use bridge link to check the devices utilizing the above created bridge.

Useful tidbits

Listing USB controller groups

for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done

Check which driver is being used for a device

lspci -nnk -d <device_id>

Bridge teardown

  • ip link delete br0 type bridge
  • dhclient -r <interface>
  • dhclient <interface>

Old notes

Shit I did different than the wiki

The stuff for OVMF is different than the OVMF shit on the libvirt page... and doesn't seem to work. For shits I followed the libvirt OVMF instructions (https://wiki.archlinux.org/index.php/libvirt#OVMF_-_QEMU_workaround) and that actually worked. However, not until after a reboot (no fuckin' clue why restarting libvirtd isn't enough).

Nvidia is a douche

(turns out this is also on the Arch wiki... but I somehow fucking missed it and had all this typed up and done before I realized it) Apparently there is a "bug" in the nvidia driver which causes the driver to disable the device if it detects the hypervisor. Brilliant. They're also apparently aware of it and have no intention of fixing it. As such some tweaks need to be made so they hypervisor isn't broadcast to the vm. Using virsh modify the XML for the domain.

sudo virsh edit <my_vm_name_created_in_virt_manager_or_whatever>

Update the section to reflect the following:

<features>
  <acpi/>
  <apic/>
  <hyperv>
    <relaxed state='on'/>
    <vapic state='on'/>
    <spinlocks state='on' retries='8191'/>
    <vendor_id state='on' value='Nvidia43FIX'/>
  </hyperv>
  <kvm>                                                                                                                          
    <hidden state='on'/>
  </kvm>
  <vmport state='off'/>
</features>

Adding an entire physical disk to the vm

According to RedHat the unadultered disk shouldn't be added (as in /dev/sdd) as it poses a security risk. So I just created a physical partition that took up the entire space of the drive. Then add the following to your domain xml:

sudo virsh edit <my_vm_name_created_in_virt_manager_or_whatever>

<disk type='block' device='disk'> 
  <driver name='qemu' type='raw'/>
  <source dev='/dev/disk/by-id/wwn-0x50014ee0044d18d7-part1'/>
  <target dev='vdb' bus='virtio'/>
</disk>

It's added using the disk's id. That way the disk can change which sata controller it resides on and you don't have to worry about shit getting fucked up.

Can't run virsh commands as non-root

If after setting up permissions you still can't run virsh commands as your normal user (or you can but can't see any domains or networks) check that you've exported LIBVIRT_DEFAULT_URI (in shellrc file):

export LIBVIRT_DEFAULT_URI=qemu:///system

Requested operation is not valid: network 'default' is not active

virsh net-start default

Install guest drivers

Hardware notes

I bought a kvm switch (hdmi) with all the usual peripheral shits. We'll see how that ends up working out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment