I play games regularly, and the sad reality is that it forces me to use Windows on my desktop. There's a Linux installation on there, but rebooting into it is such a massive interruption that I usually just move over to my laptop for programming. Working on a laptop leads to all sorts of ergonomic issues, and it felt like a massive waste to not develop on the desktop hardware I invested so much in. So after extensively researching what the VFIO community has been doing, I've deleted my Windows installation and moved all my gaming into a virtual machine on a Linux host.
Normally VMs are too slow for gaming, but thanks to a feature called VFIO you can run games at near-native performance by passing graphics cards and USB controllers directly to a virtual machine. The only requirement is that your board supports IOMMU, which most modern systems have. In this guide I'll walk you through a setting up a VFIO-enabled virtual machine.
The setup I use involves using integrated Intel graphics for the host operating system, and passing an NVidia graphics card to the VM. While the VM is turned off, I can use the discrete GPU from the host using Bumblebee. Bumblebee lets you render on the NVidia dGPU, and pass the resulting framebuffer to the internal Intel graphics for display.
You'll want to make sure that Bumblebee is installed, along with Intel and NVidia drivers for your cards. In your BIOS, you'll want to make sure that the main display card is set to "internal graphics", and that VT-d is enabled. Plug your monitors into the integrated graphics outputs, and your system should be prepared. Ensure your IOMMU groups are good, if not you may need to patch your kernel or get a different motherboard.
Many guides on the VFIO subreddit wiki will point you in the right direction if you need more information on specific hardware or any of these steps. Just make sure you ignore anything about graphics-related kernel modules or pci-stub, as it isn't really nessesary with this setup.
If you want to use 3 monitors but your integrated graphics card doesn't have enough outputs you can get a DisplayPort hub to add extra connectivity. I use this one from EVGA, but most should work. Just keep in mind that it may be harder for the GPU to keep up if you're pushing lots of pixels, especially if you have 4k monitors. Personally I have two 1440p monitors and a 1080p monitor, and it runs smoothly when using Wayland.
To run games and CUDA applications on the dGPU, you'll use optirun <command> to run them. For non-steam applications I edit their respective .desktop file entry and add optirun before the command entry. For steam games, you'll add LD_PRELOAD="libpthread.so.0 libGL.so.1" __GL_THREADED_OPTIMIZATIONS=1 optirun %command% to each game's launch options, or just launch steam through optirun.
From here you can pretty much follow the virt-manager section of any VFIO guide. There's a good list of them on the VFIO subreddit wiki. Past here, I'll list a few optimizations that have increased performance for me, as well as specific troubleshooting steps that helped me diagnose issues.
- Having a proper storage setup is vital for playing games that stream a lot of content from disk, and can reduce guest locking.
- I passed in a virtio iso into my VM during installation so that I could install the virtio-scsi drivers. VirtIO is paravirtualized storage, so it's more performant than using the default virtual SATA controller. It's well worth it.
- Add the virtio-scsi controller to your config, set the queues to the number of vCPUs you have, and then set all your disk images to use SCSI in virt-manager and it should just work.
- Supposedly NVMe passthrough is the fastest option for passing a disk into the VM, but you need a dedicated NVMe drive for the VM from what I understand.
vhost-scsiwill the next best option once windows drivers are written for it.
- My non-sparse raw disk images are stored on an SSD, so it's optimized for that setup. See this slideshow for optimal choices with other configurations.
- I install my games on a separate disk image. I recommend this, since it allows you to recreate the VM without having to re-download all your games, and this way you can also have multiple images across HDDs/SSDs depending on the performance you want for each game you install. Steam and Blizzard's app specifically are both great at re-adding games you installed previously.
- If NVIDIA's drivers don't recognize your card check the card in Windows device manager for a
Code 43error. If it's there, follow this wiki section to get around it. Also, try changing your chipset toq35, as it emulates PCI-related things better (although is newer and thus less stable) - If sound output on the NVIDIA card sounds timewarped or garbled make sure the nvidia sound card is configured as a function of the graphics card, this is something the virt-manager UI doesn't do well. Search for
multifunctionin my config for what your setup should look like, make sure the two devices have the samedomain/bus/slotand differentfunctionentries. Also worth looking into Message Signaled Interrupts if it's still a problem. - To get Oculus Rift working make sure your cpu model is set to
host-passthroughor something new enough that will make Oculus happy (Skylake-Clientalso worked for me). In the virt-manager UI there might not be a drop down option for this, so just enter it in the textbox if it's missing. - If you're having issues with ASMedia USB 3.1 Controllers try messing around with the PCI bus it's connected to in the guest. I tried a config where it was attached to
domain='0x0000' bus='0x05' slot='0x00' function='0x0'in the guest, and it caused all sorts of problems where the device would get into a weird state and I had to reset the host. But when I trieddomain='0x0000' bus='0x00' slot='0x02' function='0x0'like in the config below, it worked fine (although sometimes isn't usable in the host after shutdown, which I'm okay with since it's all windows-only hardware anyway).
- CPU pinning and topology makes a big difference in reducing guest locking. It mostly just ensures that KVM isn't fighting itself for resources when something is making high utilization of a given CPU core. An added benefit is it can make better use of the CPU's cache.
- I have 1 vcpu with 3 cores (6 threads), pinning them to 3 of my host machine's cores. I run the IO threads and the emulator thread on the spare core.
- The order in my config is weird because I looked at the output of
cat /proc/cpuinfoand paired the virtual threads in the VM with the real threads on my CPU.
- The order in my config is weird because I looked at the output of
- Supposedly the
q35chipset is newer and emulates PCI stuff better, but is less tested. - Make sure you remove as many devices as possible from your config. The fewer devices you need to emulate, the less CPU overhead you'll have.
- In windows, go to Power Settings -> Power Profiles -> Show More -> choose High Performance. I have no idea if this is a placebo effect but I did feel like there was less stuttering during games after changing this.
- This will let you share a mouse and keyboard between the host and guest without having to have a physical usb switch
- Switch by hitting both ctrl keys
- I haven't noticed any input latency with this, and use it to play Overwatch (doesn't interfere with tracking at all when playing Tracer)
- Only limitation is that side mouse buttons don't currently work due to it being hard-coded in qemu
- it looks like it might be an easy fix though: virtio-input-hid.c
- Search
qemu:commandlinein my config and copy that section, replacing any of the/dev/input/by-idstuff with the mouse/keyboard you want to pass in - copy the
<input type='...' bus='...'>sections over, making sure the bus/slot codes are unique to your config - make sure you have the virtio-input drivers installed on the guest or it will fallback to using the PS/2 bus
- make sure you're running the VMs as some user that's been added to the
inputuser group (grep foruserin/etc/libvirt/qemu.conffor that setting). I got lazy and just set it to me. - Information came from these pages: 1 2 3 4
I have a similar setup currently working perfectly in Arch with GNOME/Xorg. I was experimenting with GNOME/Wayland the other day, and found that it hangs when starting the VM. I think Xwayland is binding to the nvidia driver on startup, and won't let go when vfio tries to grab it. As far as I can tell there's no config file to tell Xwayland to ignore the nvidia card. Did you have any problems like this in Wayland?