Skip to content

Instantly share code, notes, and snippets.

@ph1048
Forked from 0chroma/VFIO Setup Guide.md
Created December 26, 2020 00:29
Show Gist options
  • Save ph1048/73b9038c5846e445fc738cfa7f4913a2 to your computer and use it in GitHub Desktop.
Save ph1048/73b9038c5846e445fc738cfa7f4913a2 to your computer and use it in GitHub Desktop.
Windows 10 VFIO QEMU Setup

VFIO Setup Guide

I play games regularly, and the sad reality is that it forces me to use Windows on my desktop. There's a Linux installation on there, but rebooting into it is such a massive interruption that I usually just move over to my laptop for programming. Working on a laptop leads to all sorts of ergonomic issues, and it felt like a massive waste to not develop on the desktop hardware I invested so much in. So after extensively researching what the VFIO community has been doing, I've deleted my Windows installation and moved all my gaming into a virtual machine on a Linux host.

Normally VMs are too slow for gaming, but thanks to a feature called VFIO you can run games at near-native performance by passing graphics cards and USB controllers directly to a virtual machine. The only requirement is that your board supports IOMMU, which most modern systems have. In this guide I'll walk you through a setting up a VFIO-enabled virtual machine.

The setup I use involves using integrated Intel graphics for the host operating system, and passing an NVidia graphics card to the VM. While the VM is turned off, I can use the discrete GPU from the host using Bumblebee. Bumblebee lets you render on the NVidia dGPU, and pass the resulting framebuffer to the internal Intel graphics for display.

Preparing the host

You'll want to make sure that Bumblebee is installed, along with Intel and NVidia drivers for your cards. In your BIOS, you'll want to make sure that the main display card is set to "internal graphics", and that VT-d is enabled. Plug your monitors into the integrated graphics outputs, and your system should be prepared. Ensure your IOMMU groups are good, if not you may need to patch your kernel or get a different motherboard.

Many guides on the VFIO subreddit wiki will point you in the right direction if you need more information on specific hardware or any of these steps. Just make sure you ignore anything about graphics-related kernel modules or pci-stub, as it isn't really nessesary with this setup.

Using lots of monitors on integrated graphics

If you want to use 3 monitors but your integrated graphics card doesn't have enough outputs you can get a DisplayPort hub to add extra connectivity. I use this one from EVGA, but most should work. Just keep in mind that it may be harder for the GPU to keep up if you're pushing lots of pixels, especially if you have 4k monitors. Personally I have two 1440p monitors and a 1080p monitor, and it runs smoothly when using Wayland.

Running games on the host

To run games and CUDA applications on the dGPU, you'll use optirun <command> to run them. For non-steam applications I edit their respective .desktop file entry and add optirun before the command entry. For steam games, you'll add LD_PRELOAD="libpthread.so.0 libGL.so.1" __GL_THREADED_OPTIMIZATIONS=1 optirun %command% to each game's launch options, or just launch steam through optirun.

Setting up the VM

From here you can pretty much follow the virt-manager section of any VFIO guide. There's a good list of them on the VFIO subreddit wiki. Past here, I'll list a few optimizations that have increased performance for me, as well as specific troubleshooting steps that helped me diagnose issues.

Storage

  • Having a proper storage setup is vital for playing games that stream a lot of content from disk, and can reduce guest locking.
  • I passed in a virtio iso into my VM during installation so that I could install the virtio-scsi drivers. VirtIO is paravirtualized storage, so it's more performant than using the default virtual SATA controller. It's well worth it.
    • Add the virtio-scsi controller to your config, set the queues to the number of vCPUs you have, and then set all your disk images to use SCSI in virt-manager and it should just work.
    • Supposedly NVMe passthrough is the fastest option for passing a disk into the VM, but you need a dedicated NVMe drive for the VM from what I understand. vhost-scsi will the next best option once windows drivers are written for it.
  • My non-sparse raw disk images are stored on an SSD, so it's optimized for that setup. See this slideshow for optimal choices with other configurations.
  • I install my games on a separate disk image. I recommend this, since it allows you to recreate the VM without having to re-download all your games, and this way you can also have multiple images across HDDs/SSDs depending on the performance you want for each game you install. Steam and Blizzard's app specifically are both great at re-adding games you installed previously.

PCI Passthrough + Compatibility

  • If NVIDIA's drivers don't recognize your card check the card in Windows device manager for a Code 43 error. If it's there, follow this wiki section to get around it. Also, try changing your chipset to q35, as it emulates PCI-related things better (although is newer and thus less stable)
  • If sound output on the NVIDIA card sounds timewarped or garbled make sure the nvidia sound card is configured as a function of the graphics card, this is something the virt-manager UI doesn't do well. Search for multifunction in my config for what your setup should look like, make sure the two devices have the same domain/bus/slot and different function entries. Also worth looking into Message Signaled Interrupts if it's still a problem.
  • To get Oculus Rift working make sure your cpu model is set to host-passthrough or something new enough that will make Oculus happy (Skylake-Client also worked for me). In the virt-manager UI there might not be a drop down option for this, so just enter it in the textbox if it's missing.
  • If you're having issues with ASMedia USB 3.1 Controllers try messing around with the PCI bus it's connected to in the guest. I tried a config where it was attached to domain='0x0000' bus='0x05' slot='0x00' function='0x0' in the guest, and it caused all sorts of problems where the device would get into a weird state and I had to reset the host. But when I tried domain='0x0000' bus='0x00' slot='0x02' function='0x0' like in the config below, it worked fine (although sometimes isn't usable in the host after shutdown, which I'm okay with since it's all windows-only hardware anyway).

Performance tuning

  • CPU pinning and topology makes a big difference in reducing guest locking. It mostly just ensures that KVM isn't fighting itself for resources when something is making high utilization of a given CPU core. An added benefit is it can make better use of the CPU's cache.
  • I have 1 vcpu with 3 cores (6 threads), pinning them to 3 of my host machine's cores. I run the IO threads and the emulator thread on the spare core.
    • The order in my config is weird because I looked at the output of cat /proc/cpuinfo and paired the virtual threads in the VM with the real threads on my CPU.
  • Supposedly the q35 chipset is newer and emulates PCI stuff better, but is less tested.
  • Make sure you remove as many devices as possible from your config. The fewer devices you need to emulate, the less CPU overhead you'll have.
  • In windows, go to Power Settings -> Power Profiles -> Show More -> choose High Performance. I have no idea if this is a placebo effect but I did feel like there was less stuttering during games after changing this.

evdev passthrough

  • This will let you share a mouse and keyboard between the host and guest without having to have a physical usb switch
  • Switch by hitting both ctrl keys
  • I haven't noticed any input latency with this, and use it to play Overwatch (doesn't interfere with tracking at all when playing Tracer)
  • Only limitation is that side mouse buttons don't currently work due to it being hard-coded in qemu
  • Search qemu:commandline in my config and copy that section, replacing any of the /dev/input/by-id stuff with the mouse/keyboard you want to pass in
  • copy the <input type='...' bus='...'> sections over, making sure the bus/slot codes are unique to your config
  • make sure you have the virtio-input drivers installed on the guest or it will fallback to using the PS/2 bus
  • make sure you're running the VMs as some user that's been added to the input user group (grep for user in /etc/libvirt/qemu.conf for that setting). I got lazy and just set it to me.
  • Information came from these pages: 1 2 3 4
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
virsh edit Gaming
or other application using the libvirt API.
-->
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>Gaming</name>
<uuid>3503a85f-a44f-4578-bc76-cbdbce44e9f0</uuid>
<memory unit='KiB'>16384000</memory>
<currentMemory unit='KiB'>16384000</currentMemory>
<vcpu placement='static'>6</vcpu>
<iothreads>2</iothreads>
<cputune>
<vcpupin vcpu='0' cpuset='0'/>
<vcpupin vcpu='1' cpuset='1'/>
<vcpupin vcpu='2' cpuset='2'/>
<vcpupin vcpu='3' cpuset='3'/>
<vcpupin vcpu='4' cpuset='4'/>
<vcpupin vcpu='5' cpuset='5'/>
<emulatorpin cpuset='6-7'/>
<iothreadpin iothread='1' cpuset='6'/>
<iothreadpin iothread='2' cpuset='7'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-q35-2.9'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/ovmf/ovmf_code_x64.bin</loader>
<nvram>/var/lib/libvirt/qemu/nvram/Gaming_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='animeistrash'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
</features>
<cpu mode='host-passthrough' check='none'>
<topology sockets='1' cores='3' threads='2'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/sbin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/win10.img'/>
<target dev='sdh' bus='scsi'/>
<boot order='1'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/games-fast.img'/>
<target dev='sdi' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/run/media/hflw/4e8bb340-f6b1-48ff-bf93-ef1c82c1254c/VM Images/games-slow.img'/>
<target dev='sdj' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='2'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
</controller>
<controller type='scsi' index='0' model='virtio-scsi'>
<driver queues='6'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<interface type='direct'>
<mac address='52:54:00:60:ea:05'/>
<source dev='enp0s31f6' mode='bridge'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
</interface>
<input type='mouse' bus='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
</input>
<input type='keyboard' bus='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x1' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x046d'/>
<product id='0x0826'/>
</source>
<address type='usb' bus='0' port='1'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</memballoon>
</devices>
<qemu:commandline>
<qemu:arg value='-object'/>
<qemu:arg value='input-linux,id=kbd,evdev=/dev/input/by-id/usb-04d9_USB_Keyboard-event-kbd,grab_all=on,repeat=on'/>
<qemu:arg value='-object'/>
<qemu:arg value='input-linux,id=mouse,evdev=/dev/input/by-id/usb-Kingsis_Peripherals_ZOWIE_Gaming_mouse-event-mouse'/>
</qemu:commandline>
</domain>

VR Capable Mini-ITX VFIO build

  • Case: NCases M1
  • CPU: Intel 7700K
  • Cooler: Noctua - NH-U9S
    • I used a NH-L12 when I built this, but the VRM heat sink on my board interferes with the optimal orientation, so I'd recommend this cooler instead. I cross-referenced it's measurements with the dimensions inside the case and it should squeeze in.
    • You might have more options if you remove the VRM heat sink and replace it with something less obstructive, but that's a bit scary so I'd just go with the tower cooler
    • Another option is to get a water cooler, the M1 can fit pretty large radiators
  • Memory: Corsair LPX 32GB (2x16GB) 3200MHz DDR4
  • Motherboard: EVGA Z270 Stinger
    • Two rear USB 3.1 ports, an M.2 slot, and a U.2 slot which can be hard to come by. Plenty of USB 3.0 ports as well.
    • IOMMU groups on this board are great and I don't need to use the ACS override patch
    • I pass the USB 3.1 controller into the VM and use an Anker USB-C hub for 5 type A ports, enough for an Oculus Rift, Xbox One controller and a USB audio interface.
      • If you opt to do this make sure the rift headset is plugged into the 3.1 type A port, as it will likely use the most bandwidth of your devices
    • I use an EVGA DisplayPort Hub for a triple monitor set up (2x 1440p and 1x 1080p monitors), I just had to make sure one of the 1440p monitors is connected to the HDMI port to avoid bandwidth issues.
    • Another option I considered was the ROG Strix Z270I, but it had worse USB connectivity, and I would rather have one M.2 and one U.2 instead of two M.2.
  • GPU: EVGA GTX GTX 970 Superclocked
    • Sub-optimal pick that you can swap out with a newer card (I'll be doing this soon)
    • Any full-length card should work, just check the M1 Compatibility Spreadsheet (edits are public so careful)
  • PSU: Corsair SF Series SF600
    • might be overkill depending on the parts you pick, they make a 450W version as well
    • avoid Silverstone's SFX power supplies, they're terrible. I had two fail on me.
  • Drives: Pick as you see fit. The M1 can fit pretty much any reasonable configuration.
    • I personally have two 3.5in HDDs, an old 2.5in SSD which I'll probably upgrade soon, and a slim Blu-ray writer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment