Skip to content

Instantly share code, notes, and snippets.

@midi1996

midi1996/README.md

Last active May 3, 2024 22:26
Show Gist options
  • Save midi1996/91e1a3b4f3510767496e8edcb63eeb38 to your computer and use it in GitHub Desktop.
Save midi1996/91e1a3b4f3510767496e8edcb63eeb38 to your computer and use it in GitHub Desktop.
Download ZIP
M2000M passthrough experiences.
Raw
README.md

I've seen people passing their laptop dGPUs yet I haven't been successful because of a lot of reasons, mainly extracting the vbios (that somehow some laptop users can do but I cannot, and I wasn't the only one) and having it UEFI (mine wasn't which is weird as it's a SKL laptop) and they said that some Muxed laptops are easy to deal with. And after a while with the help of some people in the hackintosh discord server, we made an even larger layout list of display outputs we stumbled upon while helping some new users with their laptops (thanks /u/dhinakg):

dGPU Laptop setups

My laptop (Thinkpad P50) happened to be something like the Macbooks where there is a mux between the iGPU and dGPU to the eDP (this has been confirmed with my laptop's schematics).

Great! So you can just pass the dGPU and see stuff on either the display or output!

WRONG! Lenovo and Novideo know best, and you better like it! /s

If I want to pass my dGPU I have to fulfill a lot of requirements which I've compiled in a list for your convenience:

  • Integrate my vBIOS into the OVMF image with an SSDT hack (so that the nvidia driver loads the vBIOS from ACPI or something of the sort)
  • vBIOS (with UEFI GOP) injected through the PCIe section of virsh and qemu otherwise your dGPU will crap itself once the VM is shutdown and you'll be required to reboot the system to get it working again (this is required for my setup, maybe others wont be needing it, needs testing)
  • Proper PCIe addressing to match the one in your setup (otherwise nvidia driver no likey)
  • Masquerading your device IDs and subsystem IDs (the latter is more important) to get the drivers to even install

If any of these isn't fulfilled, you'll be met with the good ol' Error 43.

Note that this is mostly me sharing my experiences, there may be some unnecessary parts or steps that shouldn't be there, you're welcome to point any issues and correct/add more info! You may take it as guide, but again, you're responsible of anything you do with your hardware.

This whole thing started 2 years ago when I tried passing my nvidia dGPU to get it working on a Windows guest for Autodesk programs (aka, that joyful company that hates everything non-Windows and free), but lo and behold, it didn't work! Shocker! So I followed through 2 sources:

Other main sources I used (there are many more but I would be making a list longer than the post itself):

My Thinkpad P50 with Quadro M2000M, it has a gpu layout that goes something like this (from the schematics of the laptop):

Untitled Diagram

My host is as follows:

  • Lenovo Thinkpad P50
  • CPU: Xeon E3-1505M V5
  • GPUs: Intel HD P530 (used for GVT) - Nvidia Quadro M2000M (passed to the VM)
  • OS: Manjaro (I tried this on regular Arch, Ubuntu and Pop!OS too)
  • Qemu 5.2.0 patched with this
  • libvirtd 7.1.0 (official from repo)
  • modules I'm loading: vfio_pci vfio vfio_iommu_type1 vfio_virqfd vfio_mdev kvmgt (for gvt too)
  • options for modules:
blacklist nvidia
blacklist nvidia-drm
options i915 enable_guc=0
options i915 enable_fbc=0
options i915 enable_gvt=1
  • kernel boot parameters: iommu=pt intel_iommu=on kvm.ignore_msrs=1

The keys for my setup were the proper PCIe location in the VM and passing a GOP updated vBIOS, otherwise I'll be met with Error 43 or no display. To get the M2000M working properly: (I assume you've read Archlinux's wiki about OVMF GPU Passthrough and how to add stuff like qemu commands in a virsh xml)

  • Dump the vBIOS (there are many ways, through the windows registry (manually or with a script) or by using VBiosExtractor following this guide. Note that you will need to put the full path to the BIOS updater executable, the VBiosExtractor script doesn't seem to do well with relative paths.
  • Add UEFI GOP to it (as it's PC-AT only) (I used Maxwell MXM)
  • Patch the OVMF image with the regular (or GOP updated) vBIOS by building it automatically (thanks to @Ashymad's original gist) if you're on an Arch-based OS (you'll have to download the whole gist) or check the PKGBUILD and make the patching manually (this guide has the same steps too, don't forget the dos2unix commands, otherwise the patches wont be applied). You might want to add the path of the OVMF in /etc/libvirtd/qemu.conf accordingly or edit the PKGBUILD to copy them to /usr/share/edk2-ovmf/x64/ instead (where the edk2-ovmf official package store them).
  • Make a VM with virt-manager as you would normally, add the dGPU and the other hardware that comes with it (for me only Audio, for others it may be also USB or others peripherals that come with a GPU nowadays), but in the XML section (make sure you enable editing), remove all PCI addressing lines (to avoid conflicting addresses) and add the proper address for the dGPU to the hostdev section (note that it usually just means that the source address and the hostdev address match). When applying, virsh will properly populate the other addresses.
  <source>
    <address domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
  </source>
  <!-- The address above of the source has its domain, bus, slot and function, matching that of the hostdev address! multifunction is added by virsh if you have more devices under the same bus -->  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0" multifunction="on" />
  • Add the UEFI GOP vBIOS as ROM in the devhost, otherwise you won't be getting any image (in the firmware boot or in windows), you can add the vBIOS to all the passed nvidia parts (not sure if that matters, but I did it anyways):
  <source>
    <address domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
  </source>
  <rom file="/ext/qemu/vbios_10de_13b0_1_updGOP.rom"/>
  <!-- match it to your own path -->
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0" multifunction="on" />
  • Masquerade the device IDs and subsystem IDs otherwise the drivers won't even install (the first 2 are subsystem in decimal values and the last 2 are in hex, either of the formats work, I did this just to show that there are many ways of putting these values).check below to know how to get these values
    <qemu:arg value="-set"/>
    <qemu:arg value="device.hostdev1.x-pci-sub-vendor-id=6058"/>
    <qemu:arg value="-set"/>
    <qemu:arg value="device.hostdev1.x-pci-sub-device-id=8750"/>
    <qemu:arg value="-set"/>
    <qemu:arg value="device.hostdev1.x-pci-vendor-id=0x10de"/>
    <qemu:arg value="-set"/>
    <qemu:arg value="device.hostdev1.x-pci-device-id=0x13b0"/>
    <qemu:arg value="-acpitable"/>
    <qemu:arg value="file=/home/midi/Downloads/SSDT_BAT.dat"/>

These were the requirement to get my setup working and I got output through HDMI (and the TB3 port through a hub, no need to pass the hub through it as it supported DP-Alt). If any of them isn't present then I'll be met with either no display or Error 43.

How to get your subsystem IDs, 2 ways:

  • lspci -nnk and you'll get something like this:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107GLM [Quadro M2000M] [10de:13b0] (rev a2)
    Subsystem: Lenovo GM107GLM [Quadro M2000M] [17aa:222e]
    Kernel driver in use: vfio-pci
    Kernel modules: nouveau, nvidia_drm, nvidia
  • Windows > Right Click on Start > Device Manager > Double click on your dGPU > Details > Hardware Ids > read it as VEN_XXXX for Vendor ID, DEV_XXXX for Device ID, SUBSYS_XXXXYYYY for Sub Device (XXXX) and Sub Vendor (YYYY).

Windows Device Hardware Ids

Issues I encountered:

  • Running the VM without passing the ROM (or rom bar="off") will result in no display in the VM and also when the VM crashes and powers off, the dGPU becomes in a broken state and I can only fix that by rebooting the system (resetting the device by removing it and then rescanning pci devices did not work)
  • GPU is running hot, probably because it cannot sense CPU temps or something like that and just cannot properly manager its temps, probably a solution would be to undervolt it, but I haven't done that as the temps for me don't exceed the 70C (which I guess is fine for a mobile chip, the fans are running properly to cool it down when not in use)
  • My laptop can no longer enter sleep mode (and sometime even rebooting or shutting down) when enabling vfio/kvm modules (no idea why, but I'm keeping here in case someone knows what's up), even when no VM is running.
  • My laptop has an option to use the dGPU-only, and disabling the iGPU. I followed the single GPU passthrough guide which worked (ofc with the requirements above) but only for the external displays, not the internal one (obligatory u/Lellow_Yedbetter) And funny enough, I could extract 2 vBIOS of my nvidia GPU (different versions) from the registry for both dGPU+iGPU configuration and dGPU-only. For my vBIOS test, I used the one I extracted from the BIOS update, I might try the ones I pulled from registry.

Other finding:

  • I added GVT to the same VM and to my surprise I could still use the nvidia GPU through it, although with a really bad output as the spice viewer is slow and introduces a lot of latency (like 30fps max), I tried to mitigate this as much as I could by patching qemu with this, it's smoother (~60fps) but still has quite some latency. I did not try looking-glass yet.
  • Windows' feature to select which GPU to use through the Graphics Settings (in Display Settings) is working pretty good as I tested furmark and GB5 Vulkan and OpenGL benchmarks (with and without a display attached to the dGPU) and the results were the same. I'm going to do more testing on regular windows and see if the performances matches.
  • Without repeating the same information: your vm needs to have all the lines that hide the fact that it's a VM (it's everywhere on the internet and on r/VFIO)
  • Having the laptop in battery mode while trying this is a big no-no as the performances tank so hard!

What I'm trying to do next:

  • Get the internal display work with the dGPU-only setup
  • Try to pass some sensor data to the VM (including battery charge)
  • Try to pass my SMBIOS data and some of my SSDTs to activate Windows (as I want to pass a whole nvme drive to the VM, the magic of PCIe and vfio)

My XML for my setup here. DO NOT use it as it is, it makes no sense if you do without knowing what you're doing, just follow any guide that uses virt manager and do your modifications following what I posted (and hope it works for you).

I'll post some spicy benchmark numbers when I'm done doing them (as that's what some people really want to see)

@plantroon
Copy link

Was this stable? Like did you use it for extended periods of time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment