GPU passthrough + CPU pinning

gamingVM.md

Windows VM on Linux (Debian) with GPU passthrough and CPU pinning

Short tutorial on how to create optimized gaming Windows VM with GPU passthrough and CPU pinning on Linux Debian-based systems using QEMU/KVM

PART 1 - HOST SYSTEM SETUP - qemu and virt-manager installation + gpu isolation

Part 1 helpers

lspci -nn - list pci devices - get graphics card video/audio ids (example id: [8284:7bfd])

lspci -k | grep "AMD" - other option + narrow down results

Steps

1. Install qemu and virt-manager + dependencies

sudo apt-get install qemu-kvm libvirt-bin bridge-utils virt-manager qemu virt-viewer spice-vdagent

2. Turn on virtualization in BIOS settings (VT-x/AMD-V/IOMMU - often one of them, depending on bios/cpu)

3. Edit grub to allow virtualization + pass graphics card video/audio ids for later isolation

sudo nano /etc/default/grub

Modify line with GRUB_CMDLINE_LINUX_DEFAULT:

GRUB_CMDLINE_LINUX_DEFAULT="{intel/amd}_iommu=on vfio-pci.ids={video:id},{audio:id}"

By default, there is "quiet" inside - remove it so eventual errors will be shown at boot. Also: look at Part 1 helpers to get proper ids

4. Apply changes to grub

sudo grub-mkconfig -o /boot/grub/grub.cfg

5. Reboot

6. Isolate graphics card drivers at boot

sudo nano /etc/modprobe.d/vfio.conf

Modify/add following lines:

select ids to isolate:

options vfio-pci ids={video:id},{audio:id}

delay load of proprietary drivers, so vfio is able to take control over graphics card:

softdep {amd/nvidia}gpu pre: vfio-pci

7. Update initramfs (while doing it, focus on if there are any vfio related errors/warning - there shouldn't be any tho :P)

sudo update-initramfs -u

8. Reboot

9. Check if kernel drivers are isolated correctly with lspci -k ("Kernel driver in use: vfio-pci")

PART 2 - VM - setting up windows VM

Steps

1. Creating windows VM, settings/tips:

   • chipset Q35
   • firmware UEFI
   • add video and audio of graphics card isolated before
   • use virtio network - by doing so, network won't be accessible during setup, so you will be able to create offline account
   • add virtio-win.iso as CDROM device (used to later install network drivers)
   • virtio disks were causing bluescreens in my case - had to use default SATA ones

2. Unplug monitor from isolated graphics card, it makes things smoother/easier while setting up system

3. Start VM. Windows installation tips:

   • DON'T use "N" versions - they are lacking some directX modules (I was personally unable to install them later)
   • after couple of initial steps, to use offline account in windows 11:
     • shift + F10 - to open terminal
     • oobe\bypassnro - to reboot system installation process with offline option

4. Enter device manager and install network drivers from virtio-win.iso

   • may be important (it was in my case): don't install other drivers (you probably don't need to), cuz they can mess up some things (or just confuse you in various ways) later

5. (optional, I guess) clean up useless programs/processes, recommended tool: CTT's Windows Utility

   • run PowerShell as administrator
   • iwr -useb https://christitus.com/win | iex
   • explanation video
   • recommended settings for gaming in specific tabs:
     • INSTALL - all .NET and C++ things
     • TWEAKS - desktop recommended tweaks + remove all ms store apps
     • CONFIG - .NET (note: hyperV bricked my system)
     • UPDATES - Security

6. Plug monitor back to the isolated graphics card, enter display settings, set display only on proper monitors, remove monitor from VM (reboot potentially needed)

7. Shut down system, passthrough devices like keyboard/mouse/headphones

PART 3 - Further optimization - CPU pinning

We need to make libvirt hooks to run scripts that pin CPU cores at VM startup and unpin at shutdown

Part 3 helpers

lscpu -e - list cpu cores (while pinning, CORE numbers corresponding to isolated CPU numbers should not be separated between host and VM - they have to be adjusted to your CPU topology - for example, if CPU 0 has CORE 0, and CPU 8 has CORE 0, both CPU0 and CPU8 should either be isolated or not)

Steps

1. Create VM startup hook in /etc/libvirt/hooks/qemu.d/{VM_name}/prepare/begin/{fileName}.sh:

pkill brave
pkill Discord #optional processes to turn off before starting VM
systemctl set-property --runtime -- user.slice AllowedCPUs={CPU numbers that should stay on host}
systemctl set-property --runtime -- system.slice AllowedCPUs={CPU numbers that should stay on host}
systemctl set-property --runtime -- init.scope AllowedCPUs={CPU numbers that should stay on host}

{CPU numbers that should stay on host} - for example: 14-19 or 0,1,2,3

2. Create VM shutdown hook in /etc/libvirt/hooks/qemu.d/{VM_name}/release/end/{fileName2}.sh:

systemctl set-property --runtime -- user.slice AllowedCPUs={CPU numbers released for host to use}
systemctl set-property --runtime -- system.slice AllowedCPUs={CPU numbers released for host to use}
systemctl set-property --runtime -- init.scope AllowedCPUs={CPU numbers released for host to use}

{CPU numbers released for host to use} - typically all CPU numbers, for example: 0-19

3. File structure should look like this (tree /etc/libvirt/hooks):

/etc/libvirt/hooks
├── qemu
└── qemu.d
    └── {VM_name}
        ├── prepare
        │   └── begin
        │       └── {fileName}.sh
        └── release
            └── end
                └── {fileName2}.sh

4. Make created .sh files executable sudo chmod +x {/etc/libvirt/(...).sh}

5. Edit XML file of your VM in QEMU/KVM GUI, add following lines:

<vcpu placement="static" cpuset="0-13">14</vcpu>
<cputune>
    <vcpupin vcpu="0" cpuset="0-1"/>
    <vcpupin vcpu="1" cpuset="2-3"/>
    <vcpupin vcpu="2" cpuset="4-5"/>
    <vcpupin vcpu="3" cpuset="6-7"/>
    <vcpupin vcpu="4" cpuset="8-9"/>
    <vcpupin vcpu="5" cpuset="10-11"/>
    <vcpupin vcpu="6" cpuset="12"/>
    <vcpupin vcpu="7" cpuset="13"/>
</cputune>

• <vcpu> is made automatically, you have to add cpuset inside <> to separate selected CPU numbers (look at helpers to find CPU numbers to put here)
• <vcpupin />: cpuset should match requirements of your CPU topology (Part 3 helpers), above example is based on my CPU (libvirt XML CPU documentation: https://libvirt.org/formatdomain.html#cpu-tuning)