Skip to content

Instantly share code, notes, and snippets.

@Gictorbit
Forked from MaxXor/gpu-passthrough.md
Created March 25, 2022 19:20
Show Gist options
  • Save Gictorbit/05035107a06dfa3fcca07264f90adb74 to your computer and use it in GitHub Desktop.
Save Gictorbit/05035107a06dfa3fcca07264f90adb74 to your computer and use it in GitHub Desktop.
Arch GPU Passthrough Summary

Arch Linux GPU-Passthrough

A quick guide on how to setup a GPU-Passthorugh. Below are some of my Resources

The Arch Wiki is the goto place for additional information and performance tweaks like CPU-Pinning.

Requirements

  • IGPU or second dedicated GPU for Host system (unless you want to go the hard way and use one GPU for HOST and GUEST)
  • Linux Kernel 4.1 or later (vfio-pci)
  • sudo pacman -S qemu bridge-utils

IOMMU

I/O Memory Management Unit, essential tech for KVM. Your Hardware(CPU, MB-Chipset) better supports it ;)

1 Enable IOMMU

  • edit /etc/default/grub
  • append intel_iommu=on or amd_iommu=on to GRUB_CMDLINE_LINUX_DEFAULT
  • should look similar: GRUB_CMDLINE_LINUX_DEFAULT="[...] intel_iommu=on"
  • make a new grub config: sudo grub-mkconfig -o /boot/grub/grub.cfg
  • reboot
  • execute sudo dmesg | grep -e DMAR -e IOMMU
  • look for DMAR: IOMMU enabled to ensure IOMMU is enabled
  • take a look at your IOMMU Groups with this script:
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;
  • identify the target GPU and remember its ID's e.g. 1002:67b1 and 1002:aac8

VFIO

Virtual Function I/O "driver" needed to pass GPUs to a KVM

2.1 Enable VFIO Driver for target GPU

  • ensure vfio-pci is available: modinfo vfio-pci
  • edit /etc/modprobe.d/vfio.conf
  • add this line options vfio-pci ids=1002:67b1,1002:aac8

2.2 Ensure that VFIO will be loaded before other drivers

  • edit /etc/mkinitcpio.conf
  • append vfio vfio_iommu_type1 vfio_pci vfio_virqfd to MODULES
  • VFIO modules must precede other drivers! (if you want to use the GPU on host and guest look at 2.2)
  • should looke simmilar: MODULES="[...] vfio vfio_iommu_type1 vfio_pci vfio_virqfd [nouveau radeon amdgpu ...]"
  • ensure modconf is included in HOOKS
  • should look similar: HOOKS="[...] modconf [...]"
  • regenerate initramfs: mkinitcpio -p linux
  • reboot (system might be seem bricked now, I needed to enter BIOS and tell it to use IGPU)
  • first reboot might take longer since the system might need to figure out stuff. You also might need to switch your monitors input source.
  • execute dmesg | grep -i vfio andOr lspci -nnkd 1002:67b1 && lspci -nnkd 1002:aac8
  • verify that vfio-pci has loaded properly and that it is now bound to the right devices

2.3 [OPTIONAL] Dynamic rebinding

This is how you could do a GPU-Passthorugh with only one GPU i guess, but I never tested this without the IGPU. In order to dynamicly use your GPU on the HOST or GUEST you need to load your GPU driver first in /etc/mkinitcpio.conf. Use the following script (Functions taken from Reddit user /u/glowtape) to ensure everything works. When I start a VM, the startscript exits my desktopenviroment binds vfio and then reloads my session and starts the vm. There are reports of People switching drivers with XServer running, but sadly I could not replicate this.

#!/bin/bash

# Which device and which related HDMI audio device. They're usually in pairs.
export VGA_DEVICE=0000:01:00.0
export AUDIO_DEVICE=0000:01:00.1
export VGA_DEVICE_ID=1002:67b1
export AUDIO_DEVICE_ID=1002:aac8

vfiobind() {
	DEV="$1"

	# Check if VFIO is already bound, if so, return.
	VFIODRV="$( ls -l /sys/bus/pci/devices/${DEV}/driver | grep vfio )"
	if [ -n "$VFIODRV" ];
	then
		echo VFIO was already bound to this device!
		return 0
	fi

	echo -n Binding VFIO to ${DEV}...

	echo ${DEV} > /sys/bus/pci/devices/${DEV}/driver/unbind
	sleep 0.5

	echo vfio-pci > /sys/bus/pci/devices/${DEV}/driver_override
	echo ${DEV} > /sys/bus/pci/drivers/vfio-pci/bind
	# echo > /sys/bus/pci/devices/${DEV}/driver_override

	sleep 0.5

	echo OK!
}

vfiounbind() {
	DEV="$1"

	echo -n Unbinding VFIO from ${DEV}...

	echo > /sys/bus/pci/devices/${DEV}/driver_override
	#echo ${DEV} > /sys/bus/pci/drivers/vfio-pci/unbind
	echo 1 > /sys/bus/pci/devices/${DEV}/remove
	sleep 0.2

	echo OK!
}

pcirescan() {

	echo -n Rescanning PCI bus...

	su -c "echo 1 > /sys/bus/pci/rescan"
	sleep 0.2

	echo OK!

}

# Xorg shouldn't run.
if [ -n "$( ps -C xinit | grep xinit )" ];
then
	echo Don\'t run this inside Xorg!
	exit 1
fi

lspci -nnkd $VGA_DEVICE_ID && lspci -nnkd $AUDIO_DEVICE_ID
# Bind specified graphics card and audio device to vfio.
echo Binding specified graphics card and audio device to vfio

vfiobind $VGA_DEVICE
vfiobind $AUDIO_DEVICE

lspci -nnkd $VGA_DEVICE_ID && lspci -nnkd $AUDIO_DEVICE_ID

echo Adios vfio, reloading the host drivers for the passedthrough devices...

sleep 0.5

# Don't unbind audio, because it fucks up for whatever reason.
# Leave vfio-pci on it.
vfiounbind $AUDIO_DEVICE
vfiounbind $VGA_DEVICE

pcirescan

lspci -nnkd $VGA_DEVICE_ID && lspci -nnkd $AUDIO_DEVICE_ID

QEMU

3.1 Setup a Network Bridge

  • sudo brctl addbr bridge_qemu_0
  • brctl addif bridge_qemu_0 yourNetworkDevice
  • ip link set up dev bridge_qemu_0
  • edit /etc/qemu/bridge.conf
  • paste allow bridge_qemu_0

3.2 Create a VM with dedicated GPU

  • create a QCOW2 Drive: qemu-img create -f qcow2 qemu_vm.qcow2 30G
  • start your VM:
#!/bin/bash
# CPU settings.
# Don't expose KVM as the hypervisor via the MSR hypervisor nodes. (usefull for nvidia drivers)
OPTS="-cpu host,kvm=off"
OPTS="$OPTS -smp 4,sockets=1,cores=4,threads=1"
# Enable KVM full virtualization support.
OPTS="$OPTS -enable-kvm"
# Assign memory to the vm.
OPTS="$OPTS -m 12000"
# VFIO GPU ~and GPU sound~ passthrough.
OPTS="$OPTS -device vfio-pci,host=01:00.0,multifunction=on"
#OPTS="$OPTS -device vfio-pci,host=00:01.1" # dont need audio
# Load our created VM image as a harddrive.
OPTS="$OPTS -hda /home/groot/vm/win10/qemu_vm.qcow2"
# Load our OS setup image e.g. ISO file.
#OPTS="$OPTS -cdrom /home/groot/vm/iso/Win10_1703_English_x64.iso"
# Use the following emulated video device ("none" for disabled).
OPTS="$OPTS -vga qxl"
# Redirect QEMU's console 10.
OPTS="$OPTS -monitor stdio"
# Setup networking, need to try virtio at some point
OPTS="$OPTS -net nic -net bridge,br=bridge_qemu_0"

sudo qemu-system-x86_64 $OPTS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment