Skip to content

Instantly share code, notes, and snippets.

@jb-alvarado
Last active October 27, 2023 02:16
Show Gist options
  • Save jb-alvarado/d6aef18ddb965939442838d7310c5b31 to your computer and use it in GitHub Desktop.
Save jb-alvarado/d6aef18ddb965939442838d7310c5b31 to your computer and use it in GitHub Desktop.
PCI Passthrough on Fedora 31

PCI Passthrough

Hardware:

  • Mainboard ASUS PRIME Z390-A Z390
  • ASUS Dual-GTX1650-O4G
  • Intel i9 9900
  • 32GB Ram
  • NEC Corporation uPD720200 USB 3.0 Host Controller
  • Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (1 TB)
  • Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 (256 GB)

Operating Systems:

  • Host: Fedora 31
  • Guest: Windows 10

Bios Settings:

  • enable VMX
  • enable VT-d

Host preparation:

  • install fedora on the 1TB M.2

  • use custom partitioning and let the half free for data transfer between host and guest

  • after installation, update system and install virtualization and uefi bios for the VM:

    • dnf install @virtualization
    • dnf install edk2-ovmf
  • prepare grub with nano /etc/default/grub and add to GRUB_CMDLINE_LINUX= at the end:

    • intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
  • activate modules:

    • echo "vfio" > /etc/modules-load.d/vfio.conf
    • echo "vfio-pci" > /etc/modules-load.d/vfio-pci.conf
    • echo "vfio_iommu_type1" > /etc/modules-load.d/vfio_iommu_type1.conf
    • echo "vfio_virqfd" > /etc/modules-load.d/vfio_virqfd.conf
  • reconfigure grub:

    • dracut -f --kver $(uname -r)
    • grub2-mkconfig > /etc/grub2-efi.cfg
    • reboot
  • check iommu groups and get device IDs:

    • for g in /sys/kernel/iommu_groups/*; do echo "IOMMU Group ${g##*/}:"; for d in $g/devices/*; do echo -e "\t$(lspci -nns ${d##*/})"; done; done
    • get all IDs within the group where you devices is member, example output:
      • IOMMU Group 1:
             00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 0d)
             01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
             01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
        
      IDs are: 8086:1901, 10de:1f82, 10de:10fa
      
      
  • for passthrough I need the IDs from graphic card, USB Controller, and the small M.2:

    • 8086:1901,10de:1f82,10de:10fa,144d:a804,1033:0194
  • add this IDs to /etc/modprobe.d/vfio.conf:

    • options vfio-pci ids=8086:1901,10de:1f82,10de:10fa,144d:a804,1033:0194
    • reconfigure kernel and grub again:
    • dracut -f --kver $(uname -r)
    • grub2-mkconfig > /etc/grub2-efi.cfg
    • reboot
  • check if VFIO driver are loaded:

    • dmesg | grep -i vfio
    • and also every device with: lspci -nnk -d 10de:1f82

install VM

  • use virt-manager
  • add PCI devices to the VM:
    • add hardware > pci host device > ...:
      • add graphic card, with audio device
      • add the small M.2
      • add the USB controller
  • don't create an virtual hard drive, we use the M.2
  • don't create many CPUs, create 1 socket, with cores and threads
  • install windows 10
  • after installation shutdown VM and edit xml virsh edit vmname:
    • after editing <features></features> should look like:
    <features>
        <acpi/>
        <apic/>
        <hyperv>
            <relaxed state='on'/>
            <vapic state='on'/>
            <spinlocks state='on' retries='8191'/>
            <vendor_id state='on' value='10DE'/>
        </hyperv>
        <kvm>
            <hidden state='on'/>
        </kvm>
        <vmport state='off'/>
    </features>
    
  • boot guest again and install all update

host tuning

  • change fedora window manager from wayland to x11 in /etc/gdm/custom.conf:
    ...
    [daemon]
    WaylandEnable=false
    DefaultSession=gnome-xorg.desktop
    ...
    
  • restart host
  • install and configure barrier
  • format the free space from the bigger M.2 and share it with samba

References:

@fleekix
Copy link

fleekix commented Mar 22, 2020

Hey there jb, thanks for the write up. I followed your guide since it is the most relevant for my system (being fedora 31). I am running into very degraded performance issues though and I'm wondering if there is something stupid I missed perhaps?

  • MOTHERBOARD: Gigabyte Aorus Master X299
  • CPU: i9-9900X (6host, 4guest)
  • GPU1: RTX2070 Super (guest)
  • GPU2: RTX2060 Super (host)
  • 32GB Ram (8GB guest)

I'm running awesomewm and have set up my vms via virt-manager. I edited the xml on virt-manager as well.
I have installed windows about 4 times now using different settings including initially with bios, then q35/uefi and also i440fx uefi.

Here is the current i440fx xml:

<domain type="kvm">
  <name>win10-i440</name>
  <uuid>e1948277-a2ea-4a5e-8f71-4d907ae7c65e</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">8388608</memory>
  <currentMemory unit="KiB">8388608</currentMemory>
  <vcpu placement="static">4</vcpu>
  <os>
    <type arch="x86_64" machine="pc-i440fx-4.1">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10-i440_VARS.fd</nvram>
    <boot dev="cdrom"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vendor_id state="on" value="10DE"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
  </features>
  <cpu mode="host-model" check="partial">
    <model fallback="allow"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pci-root"/>
    <interface type="network">
      <mac address="52:54:00:56:ca:3c"/>
      <source network="default"/>
      <model type="e1000e"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </interface>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <sound model="ac97">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </sound>
    <video>
      <model type="virtio" heads="1" primary="yes">
        <acceleration accel3d="no"/>
      </model>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0xb3" slot="0x00" function="0x0"/>
      </source>
      <rom bar="off"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x65" slot="0x00" function="0x0"/>
      </source>
      <rom bar="off"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x65" slot="0x00" function="0x1"/>
      </source>
      <rom bar="off"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x65" slot="0x00" function="0x2"/>
      </source>
      <rom bar="off"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x09" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x65" slot="0x00" function="0x3"/>
      </source>
      <rom bar="off"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0a" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1532"/>
        <product id="0x0207"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc539"/>
      </source>
      <address type="usb" bus="0" port="2"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0b" function="0x0"/>
    </memballoon>
  </devices>
</domain>

Any help is appreciated greatly, thanks in advance.

@jb-alvarado
Copy link
Author

Hi fleekix, sorry for my late response. What I see is, that you have a different machine type. I use pc-q35-4.1, and in your hostdev I don't see any vfio drivers.

Here is my xml, but not everything has to match:

<domain type='kvm' id='1'>
  <name>win10</name>
  <uuid>cb331742-4231-466a-948e-13b26fe64323</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>18432000</memory>
  <currentMemory unit='KiB'>18432000</currentMemory>
  <vcpu placement='static' current='8'>32</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-4.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='10DE'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='4' threads='8'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='ibpb'/>
    <feature policy='require' name='amd-ssbd'/>
    <feature policy='require' name='rdctl-no'/>
    <feature policy='require' name='ibrs-all'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='mds-no'/>
    <feature policy='disable' name='hle'/>
    <feature policy='disable' name='rtm'/>
    <feature policy='disable' name='mpx'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='fdc' index='0'>
      <alias name='fdc0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:dd:2c:5e'/>
      <source network='br0' portid='691e3257-55f2-4863-9f2a-b077fc452d3e' bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich9'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c586,c777</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c586,c777</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

For the network you should use virtio driver, also for the hard drive when is only a virtual one. Have you also install the virtio drivers in Windows? They are important.

@barii
Copy link

barii commented Apr 14, 2020

have you used the integrated GPU on the host?
I have the same Motherboard and CPU, I have a Radeon card on the PCI slot closest to the CPU, and an Nvidia in the middle slot, and they appear to be in the same IOMMU slot :(

@jb-alvarado
Copy link
Author

Yes I use the GPU from CPU on host. It look like, there is a kernel Patch for that:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Bypassing_the_IOMMU_groups_(ACS_override_patch)

https://github.com/Somersall-Natalie/fedora-acs-override

But I have no idea how good and stable this will work.

@barii
Copy link

barii commented Apr 14, 2020

yes, your second link is what I tried, but it faild compiling kernel. I haven't looked it in more detail.
also even when the CPU's GPU is enablet, it is not on the list when I run
for d in /sys/kernel/iommu_groups//devices/; do n=${d#/iommu_groups/}; n=${n%%/}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##/}"; done;
maybe it doesn't need to be I guess.

@jb-alvarado
Copy link
Author

By me the CPU GPU was the default, it can also be configured in the BIOS. But yes, it don't need to be listed.

@thatnerdjosh
Copy link

thatnerdjosh commented May 22, 2020

This is giving me a black screen where I can't even see anything past grub on the host, also tried ctrl+alt+F* to switch tty

@jb-alvarado
Copy link
Author

Can it be that you have blacklisted your primary GPU?

@adam-hanna
Copy link

Thanks for the great writeup!

FYI - I wasn't able to use the included command to check iommu groups and get device IDs. Rather, I had to use $ lspci -nn | grep "VGA\|Audio\|USB"

@LaakJoonas
Copy link

I followed the guide to the letter, but the system is not booting after changing the Grub settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment