QEMU with PCI Passthrough

win.sh

#!/bin/bash

# About

## Hardware
#### Motherboard: Asus Z170 Pro Gaming
#### 16GB RAM
#### Intel onboard GFX
#### GPU: 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1b81] (rev a1)

## Kernel
### Linux 4.5.1-1-ARCH #1 SMP PREEMPT Thu Apr 14 19:19:32 CEST 2016 x86_64 GNU/Linux


# Setup

## Make sure Virtualisation is turned on in BIOS

## Turn on IOMMU
### Need to add relevant iommu option to your kernel command line
### e.g. in /etc/defaults/grub
### For intel processors, add `intel_iommu=on`

## Permissions
### add udev rule to tag vfio device with 'kvm' group
### add self to kvm group
### Related: http://www.evonide.com/non-root-gpu-passthrough-setup/

## I got error 'VFIO_MAP_DMA cannot allocate memory'
### need to increase locked memory ulimit
### https://bugzilla.redhat.com/show_bug.cgi?id=912277
### edited /etc/security/limits.conf, allowed 10GB (I guess it needs to be more than GFX card RAM?):
### daurnimator     hard    memlock     10485760
ulimit -l 10485760

## Trying to use ed2k EFI instead of seabios
### Grab ovmf image from https://www.kraxel.org/repos/jenkins/edk2
ed2k_location="/usr/local/share/edk2.git/ovmf-x64"

## Windows driver cd for virtio. download from https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.118-2/
driver_iso=virtio-win-0.1.118.iso
install_iso=Win10_1607_EnglishInternational_x64.iso

# Disk to mount
## Make sure to have permissions
disk=/dev/disk/by-id/ata-ST2000DL001-9VT156_5YD4EAVL
sudo chown "$USER" "$disk"

# Assemble qemu args
opts="-name WindowsVM"

## Turn off unused features
opts="$opts -parallel null"
opts="$opts -serial null"

## Q35 is a good base chipset, run `qemu-system-x86_64 -machine help` to get a list
opts="$opts -machine type=q35,accel=kvm"
## KVM allows for faster guests
opts="$opts -enable-kvm"
## Guest gets same CPU model as host (only works with KVM enabled).
### Don't let guest use KVM itself. Also apparently solves an issue:
#### From https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#QEMU_commands:
#### 'kvm=off is used for NVIDIA cards to stop it detecting a hypervisor and therefore exiting with an error.'
### Also want to add hypervisor goodies
### Fix from http://vfio.blogspot.com/2014/08/vfiovga-faq.html?showComment=1452870650776#c3703276360523701353
opts="$opts -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX"
## I have a quad core (non hyperthreading) processor. let guest have 3 cores
opts="$opts -smp sockets=1,cores=3,threads=1"
## I have 16GB of RAM, let guest have 8GB.
guest_mem=8G
opts="$opts -m $guest_mem"

## Give guest an emulated intel sound card
#opts="$opts -soundhw hda"
#export QEMU_AUDIO_DRV=pa

## Network
### Have to manually create 'br0' bridge
### /etc/qemu/bridge.conf contains: allow br0
opts="$opts -netdev bridge,id=hn0 -device virtio-net-pci,netdev=hn0,id=nic1"

## Input
### https://www.kraxel.org/blog/2016/04/linux-evdev-input-support-in-qemu-2-6/
#opts="$opts -object input-linux,id=mouse,evdev=/dev/input/by-id/usb-Logitech_Gaming_Mouse_G502_0A6D355D3130-event-mouse"
#opts="$opts -object input-linux,id=mouse,evdev=/dev/input/by-id/usb-Logitech_Gaming_Mouse_G502_0A6D355D3130-if01-event-kbd"
if [[ "$input" = "true" ]]; then
	## Razer Blackwidow keyboard
	opts="$opts -usb -usbdevice host:1532:010e"
	## Logitech mouse
	opts="$opts -usb -usbdevice host:046d:c332"
else
	## Give USB mouse instead of PS/2 (otherwise e.g. wheel doesn't work)
	opts="$opts -usbdevice mouse"
fi


## Give guest video card.
### Need to add the vendor + product to /etc/modprobe.d/vfio.conf
#### options vfio-pci ids=10de:1b81,10de:10f0
### Then the GFX card itself
#### If you pass x-vga=on it complains: 'vfio: Device does not support requested feature x-vga'
#### Info on x-vga: http://www.firewing1.com/howtos/fedora-20/create-gaming-virtual-machine-using-vfio-pci-passthrough-kvm
opts="$opts -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on"
opts="$opts -device vfio-pci,host=01:00.1"

### Use ed2k EFI as BIOS
opts="$opts -drive if=pflash,format=raw,readonly,file=$ed2k_location/OVMF_CODE-pure-efi.fd"
OVMF_VARS=win.efivars
if [[ ! -e "$OVMF_VARS" ]]; then
	#### Copy vars
	cp "$ed2k_location"/OVMF_VARS-pure-efi.fd "$OVMF_VARS"
fi
opts="$opts -drive if=pflash,format=raw,file=$OVMF_VARS"

## Hard Drive
### Created with 'qemu-img create -f raw win.img 32G'
opts="$opts -drive file=win.img,format=raw"
### Also an actual hard drive to give to VM
opts="$opts -device virtio-scsi-pci,id=scsi"
opts="$opts -drive file=${disk},id=disk,format=raw,if=none"
opts="$opts -device scsi-hd,drive=disk"

if [[ "$install" = "true" ]]; then
	opts="$opts -cdrom $install_iso -boot once=d"
	drivers=true
fi
if [[ "$drivers" = "true" ]]; then
	opts="$opts -drive file=$driver_iso,id=virtiocd,if=none"
	opts="$opts -device ide-cd,bus=ide.1,drive=virtiocd"
fi

# Start Qemu
exec qemu-system-x86_64 $opts $*

Sir, were you successful in passing through and utilizing your nVidia GPU inside the Windows guest?

Sir, were you successful in passing through and utilizing your nVidia GPU inside the Windows guest?

Yes. Got it all working :)