PCI Passthrough on Fedora 31

pci-passthrough.md

PCI Passthrough

Hardware:

• Mainboard ASUS PRIME Z390-A Z390
• ASUS Dual-GTX1650-O4G
• Intel i9 9900
• 32GB Ram
• NEC Corporation uPD720200 USB 3.0 Host Controller
• Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (1 TB)
• Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 (256 GB)

Operating Systems:

• Host: Fedora 31
• Guest: Windows 10

Bios Settings:

• enable VMX
• enable VT-d

Host preparation:

• install fedora on the 1TB M.2

• use custom partitioning and let the half free for data transfer between host and guest

• after installation, update system and install virtualization and uefi bios for the VM:

  • dnf install @virtualization
  • dnf install edk2-ovmf

• prepare grub with nano /etc/default/grub and add to GRUB_CMDLINE_LINUX= at the end:

  • intel_iommu=on iommu=pt rd.driver.pre=vfio-pci

• activate modules:

  • echo "vfio" > /etc/modules-load.d/vfio.conf
  • echo "vfio-pci" > /etc/modules-load.d/vfio-pci.conf
  • echo "vfio_iommu_type1" > /etc/modules-load.d/vfio_iommu_type1.conf
  • echo "vfio_virqfd" > /etc/modules-load.d/vfio_virqfd.conf

• reconfigure grub:

  • dracut -f --kver $(uname -r)
  • grub2-mkconfig > /etc/grub2-efi.cfg
  • reboot

• check iommu groups and get device IDs:

  • for g in /sys/kernel/iommu_groups/*; do echo "IOMMU Group ${g##*/}:"; for d in $g/devices/*; do echo -e "\t$(lspci -nns ${d##*/})"; done; done
  • get all IDs within the group where you devices is member, example output:

    • IOMMU Group 1:
           00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 0d)
           01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
           01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
      

    IDs are: 8086:1901, 10de:1f82, 10de:10fa
    
    

• for passthrough I need the IDs from graphic card, USB Controller, and the small M.2:

  • 8086:1901,10de:1f82,10de:10fa,144d:a804,1033:0194

• add this IDs to /etc/modprobe.d/vfio.conf:

  • options vfio-pci ids=8086:1901,10de:1f82,10de:10fa,144d:a804,1033:0194
  • reconfigure kernel and grub again:
  • dracut -f --kver $(uname -r)
  • grub2-mkconfig > /etc/grub2-efi.cfg
  • reboot

• check if VFIO driver are loaded:

  • dmesg | grep -i vfio
  • and also every device with: lspci -nnk -d 10de:1f82

install VM

• use virt-manager
• add PCI devices to the VM:
  • add hardware > pci host device > ...:
    • add graphic card, with audio device
    • add the small M.2
    • add the USB controller
• don't create an virtual hard drive, we use the M.2
• don't create many CPUs, create 1 socket, with cores and threads
• install windows 10
• after installation shutdown VM and edit xml virsh edit vmname:
  • after editing <features></features> should look like:

  <features>
      <acpi/>
      <apic/>
      <hyperv>
          <relaxed state='on'/>
          <vapic state='on'/>
          <spinlocks state='on' retries='8191'/>
          <vendor_id state='on' value='10DE'/>
      </hyperv>
      <kvm>
          <hidden state='on'/>
      </kvm>
      <vmport state='off'/>
  </features>
  

• boot guest again and install all update

host tuning

• change fedora window manager from wayland to x11 in /etc/gdm/custom.conf:

  ...
  [daemon]
  WaylandEnable=false
  DefaultSession=gnome-xorg.desktop
  ...
  

• restart host
• install and configure barrier
• format the free space from the bigger M.2 and share it with samba

References:

• PCI_passthrough_via_OVMF
• VFIO-Fedora-Notes
• fedora_29_mkinitcpio_doesnt_exist_or

Thanks for the great writeup!

FYI - I wasn't able to use the included command to check iommu groups and get device IDs. Rather, I had to use $ lspci -nn | grep "VGA\|Audio\|USB"

I followed the guide to the letter, but the system is not booting after changing the Grub settings.