Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Setting up KVM with GPU passthrough in Debian Buster

Here's how to set up a Windows 10 virtual machine in KVM with PCI passthrough. The VM will have access to an NVIDIA graphics card while the host machine (running Debian Buster) uses Intel integrated graphics. This is mostly for my own reference so I don't forget how I did it.

Hardware

  • Intel i5 (an old one) with integrated graphics: this will be used as the graphics card for the host machine running Debian Buster
  • NVIDIA Geforce 1070: this will be used as the graphics card for the Windows 10 VM

Step 1: Enable IOMMU

In order to do hardware passthrough with KVM at all, you need to enable the Intel Vt-d virtualization extensions. Edit /etc/default/grub and edit the GRUB_CMDLINE_LINUX_DEFAULT line so that it reads like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

The intel_iommu=on enables the Intel virtualization extensions for KVM. iommu=pt turns on passthrough support. Then run

sudo grub-mkconfig -o /boot/grub/grub.cfg

to rebuild your Grub config.

Step 2: Tell VFIO we want to pass through the NVIDIA card

Most types of devices can be used by the host and then passed through to the VM on demand once you actually start it. Graphics cards can't do that. If the host loads the driver for your card and starts talking to it then you can't pass it through to the VM. Linux will load the driver for any card that's plugged in even if it's not your default graphics card, so to get around that we need to tell the host's Linux kernel that we intend to use the NVIDIA card for a virtual machine, and we need to do that before the kernel gets the chance to load the driver for it (which in this case is the Nouveau open source driver).

There are a few ways to do that. You could tell the kernel to outright block the Nouveau module completely. I ended up instead telling it to wait to load the Nouveau module until after the card had already been initialized for use by VFIO passthrough. This will stop Nouveau from trying to do anything with it. Since the NVIDIA card also uses the Intel HDA module for audio output over HDMI we'll do the same thing with Intel HDA.

To do that, find the PCI ids for your GPU using:

lspci -vnn

In my case the id of the GPU is 10de:1b81 and the HDMI sound output is 10de:10f0. Note the part beneath the GPU where it says Kernel driver in use: nouveau. If everything works correctly that should change by the time we're done. To flag the card for use by VFIO, create the file /etc/modprobe.d/vfio.conf with the contents:

softdep nouveau pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci
options vfio-pci ids=10de:1b81,10de:10f0

Then run

sudo update-initramfs -u

to update the boot filesystem image with that config. You'll need to reboot at this point. To check if everything worked correctly, run lspci -vnn again and find the GPU. Beneath both of the NVIDIA devices we passed through you should see Kernel driver in use: vfio-pci.

Step 3: Create the Windows VM without GPU passthrough

I recommend using virt-manager and setting up a regular Windows 10 VM using the default QXL video card before trying to do any passthrough stuff. When creating the VM, make sure to select "Customize before install" and set the Firmware option to "UEFI". Create the VM and go through the Windows installer until you have a working Windows 10 installation with no GPU passthrough, then shut down the VM.

Step 4: Pass through the NVIDIA card

In virt-manager, go to your VM settings and click "Add Hardware", then "PCI Host Device". This will give you a list of all your PCI devices where you can select the NVIDIA GPU and click "Finish" to add it. Repeat the process for the NVIDIA Audio Controller.

You can attempt to launch the VM at this point, but if you do, Windows will install the NVIDIA driver but the card still won't work. If you go into Device Manager in Windows, you'll see the NVIDIA card with a little yellow caution icon and opening the device properties will reveal an enigmatic "Code 43" error.

Step 5: Fix the Windows Code 43 error

This error seems to happen because the NVIDIA driver realizes that it's running inside a VM and will disable itself. Since we don't want that we need to "hide" the fact that there's a VM from the driver. KVM has a mechanism for doing that but it's not exposed in virt-manager, so we'll need to edit the XML config for the virtual machine manually. To do that, run:

sudo virsh edit win10

where win10 is the name of the VM that you gave when you created it inside virt-manager. You'll need to edit the contents of the <features> tag in the following way:

Inside the <hyperv> tag: add the line:

<vendor_id state='on' value='1234567890ab'/>

(the actual value of the vendor_id is arbitrary, but it should be a 12 digit hex number).

Inside the <kvm> tag: add the line:

<hidden state='on'/>

Inside the <features> tag: add the line:

<ioapic driver='kvm'/>

The end result should look something like:

  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
    <ioapic driver='kvm'/>
  </features>

If you boot the machine up again, the NVIDIA driver should actually work! Windows will probably default to using the GPU as the primary card, which means that the Windows login prompt will likely appear on the display connected to the video card rather than the QXL display that you can see in virt-manager.

References

PCI passthrough on the Arch Linux wiki

VGA passthrough on Debian wiki

Heiko Siegler: running Windows 10 on Linux using KVM with VGA passthrough

Matthias Hueber: error 43

@mhdzumair
Copy link

Can I use with nvidia driver? I'm using nvidia-driver 418.181.07-1 on Debian buster?

When I enable nvidia pci then virt-manager hang on. If I close and reopen it then there is no qemu/kvm virtual machine available

@ozboss
Copy link

ozboss commented Mar 28, 2021

There is a typo in /etc/modprobe.d/vfio.conf:

...
- softdep snd_hda_intel_pre: vfio-pci
+ softdep snd_hda_intel pre: vfio-pci
...

@hp600
Copy link

hp600 commented Apr 8, 2021

I'm trying to do vfio with two identical gpus on Debian Buster with no success.
I've tried numerous guides with workarounds for identical gpus, and nothing seems to work.

Ok, seems my hdmi switch was defective and I was not getting a signal from the video card.
I used the guide at level1techs and installed the "Vendor Reset" kernel module for use with my amd card.
Now everything is working flawlessly.

@ogmkp
Copy link

ogmkp commented Apr 8, 2021

I'm trying to do vfio with two identical gpus on Debian Buster with no success.
I've tried numerous guides with workarounds for identical gpus, and nothing seems to work.

On my side, I just had to activate intel_iommu=on in grub and it worked automatically without adding anything.
For nvidia cards and because of their drivers, you have to add the specific flags in the features.
For all other hardware no problem.
Maybe depends on the cpu and the motherboard of the host.

@DaaNMaGeDDoN
Copy link

hey peeps, been googling on this subject: "qemu manager" (qemu i remembered from a long time ago, realized i wasnt fully gnu being a virtualbox user) and learned about nvidia passthrough via https://www.youtube.com/watch?v=12JtV1ou6EQ ended up here. Just wanted to say thanks for sharing! And i will surely be following this and try it out. The only real thing that could need gpu time on my backend is probably factorio for i dont know ill make it up streaming-device but thats ok, learning a lot.

@mhdzumair
Copy link

my system is bullseye, i tried as you stated here with nouveau driver. i got error code 28 instead of 43
any help is appreciable
Screenshot_20210516_210457

@nephest
Copy link

nephest commented Jun 21, 2021

In case someone needs the raw qemu options for step4-5:

-cpu host,kvm=off,hv_relaxed,hv_vapic,hv_time,hv_spinlocks=0x1fff,hv_vendor_id=${VENDOR} \
-device vfio-pci,host=${ADDR},multifunction=on,x-vga=on \
-device vfio-pci,host=${ADDR2} \

Where ${ADDR} and ${ADDR2} is your device address, like 01:00.0 and 01:00.1(lspci), and ${VENDOR} is a vendor id string, like AuthenticAMD(can be anything)

@davealvarado
Copy link

There is a typo in /etc/modprobe.d/vfio.conf:

...
- softdep snd_hda_intel_pre: vfio-pci
+ softdep snd_hda_intel pre: vfio-pci
...

Thanks for catching this. For anybody else following this guide, make sure you remove that underscore before "pre".

@Debianer
Copy link

Debianer commented Oct 5, 2021

Thank you, this helped me, but I also needed an option I found on here:

https://www.reddit.com/r/unRAID/comments/cox27o/usb_pci_card_vm_passthrough_tip/

I added the mentioned pci=noaer to the grub line above to get this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pci=noaer"

Now it works, when it crashed the machine before.

@DownloadableFox
Copy link

DownloadableFox commented Apr 16, 2022

Sorry for my ignorance. How do I revert changes? (just in case I need to).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment