Current QEMU dual-VM configuration. Thanks to everyone who helped me!

gpu-hotplug.md

Check it's working

DRI_PRIME=1 glxinfo | grep OpenGL

Unbind AMD GPU from Radeon driver

echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/new_id echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/remove_id echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/new_id echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/remove_id

Use it in QEMU

qemu-system-x86_64
-name "win81_amd"

-bios /usr/share/seabios/bios.bin
-enable-kvm

-usb
-device usb-kbd
-device usb-mouse

-vga none
-device qxl

-m 12G

-no-hpet
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic
-smp 4,sockets=1,cores=2,threads=2

-soundhw ac97

-M q35
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio

-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

-net none

-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0

-monitor telnet:127.0.0.1:1234,server,nowait

Now you need to remove PCI device

echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove echo 1 > /sys/bus/pci/devices/0000:01:00.1/remove

Rescan will attach it back to Radeon

echo 1 > /sys/bus/pci/rescan

Check it via PRIME once again

DRI_PRIME=1 glxinfo | grep OpenGL

qemu_shared

Introduction:
 This is my configuration for VMs and I'm change it from time to time.
 
 At moment I'm decide to use following configuration that work best for hardware I have:
  - By default I have "i915.enable_hd_vgaarb=0" because most of time I don't need Nvidia GPU.
  - When I only need VM with AMD HD6950 I boot with option unchanged to keep Intel DRI working on host
  - Once I need to run both VMs simultaneously I just change it to "i915.enable_hd_vgaarb=1" at boot time.
  - Have two sets of options for AMD VM because w/o Nvidia one I have more RAM and don't to use taskset



Hardware:
 - ASRock Z87 Extreme6
 - i7 4771 and 32GB RAM
 - 60GB SSD (for /), 2x 250GB HDDs in RAID (for /home), 1TB HDD with NTFS partition for Windows content
 - AMD HD 6950 1GB and Nvidia GTX 260
 
 
 
Software:
 - Ubuntu 14.04
 - Linux Kernel 3.19 with ACS and i915 VGAarbiter patches
   - Preemption Model set to Low Latency Desktop
   - Timer frequency set to 1000HZ 
 - QEMU 2.0.0~rc1+dfsg-0ubuntu3 from repositories
   - In past I used QEMU 2.2.1 from ppa:syseleven-platform/virtualization, but then I decide to use older Nvidia driver instead.
     Also with 2.2.1 I had weird disconnect issue in "Path of Exile" game while changing areas.
 
 
 
System configuration:
 - /etc/default/grub:
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash i915.enable_hd_vgaarb=1 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"
 - /etc/modprobe.d/blacklist-gpu.conf:
    blacklist radeon
    blacklist nouveau
 - /etc/qemu/bridge.conf:
    allow br0
 - /etc/sysctl.d/40-hugepage.conf:
    vm.nr_hugepages = 8200
 - /etc/fstab:
    hugetlbfs /dev/hugepages hugetlbfs mode=1770,gid=78 0 0



Both VMs using same "backing files" that not modified in runtime:
- /win81-base.qcow2 [13GB only!], compressed
    This QCOW2 image contain installed and configured Windows 8.1
- /dev/disk/by-uuid/0AD949C94647B394
    It's NTFS partition on 1TB HDD with games and some software installed.
Those files are base for QCOW2 images that used by virtual machines:
qemu-img create -b /win81-base.qcow2 -f qcow2 /win81-writable-amd.qcow2
qemu-img create -b /win81-base.qcow2 -f qcow2 /win81-writable-nvidia.qcow2
qemu-img create -b /dev/disk/by-uuid/0AD949C94647B394 -f qcow2 /home/gameshdd-writable-amd.qcow2
qemu-img create -b /dev/disk/by-uuid/0AD949C94647B394 -f qcow2 /home/gameshdd-writable-nvidia.qcow2
So I basically store VM-specific changes to my system drive on SSD and changes for content stored on HDD RAID.



Kernel build:
apt-get install kernel-package
apt-get build-dep linux-image-$(uname -r)
apt-get build-dep linux-image-3.19.0-23-generic
apt-get source linux-image-$(uname -r)
apt-get source linux-image-3.19.0-23-generic
cd linux-*
# Now just moment to hate Ubuntu packaging, I do not use VBOX!
rm -rf ubuntu/vbox
# Comment out virtualbox crap
nano ubuntu/Makefile
# Apply patches (my patch files are above kernel directory)
# Personally I just used patch version published by someone on google drive 
https://gist.github.com/ArseniyShestakov/768a426e79e2b901f9eb
patch -p1 < ../override_for_missing_acs_capabilities.patch 
patch -p1 < ../i915_317.patch
# Copy your current kernel config
cp -vi /boot/config-`uname -r` .config
# Change some options for better performance
# Processor Type and Features > Preemption Model > Preemptible Kernel (Low Latency Desktop)
#  (CONFIG_PREEMPT=y)
# Processor Type and Features > Timer Frequency > 1000 Hz
# (CONFIG_HZ_1000=y CONFIG_HZ=1000)
make menuconfig
# Now build kernel
fakeroot make-kpkg -j 8 --initrd kernel-image kernel-headers --append-to-version=-acs-igdarb-lowpremption-timer1000hz
# Install built packages
dpkg -i ../*acs-igdarb-lowpremption-timer1000*.deb


All useful information may be found there:
https://bbs.archlinux.org/viewtopic.php?id=162768
https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit#gid=0
http://vfio.blogspot.com



My personal expirience of why I use some of options:
https://www.reddit.com/r/linux_gaming/comments/3dr62f/set_up_gpu_passthrough_without_using_the_cli/ct8gr34
https://www.reddit.com/r/linux_gaming/comments/2z2y7h/gpu_passthrough_or_how_to_play_any_game_at_near/cpfh76x

qemu_vm_amd

# This system use real network card (00:19.0) as I have two in my mobo.


#################################################
# Exclusive configuration when no other VM running (more RAM, all cores)
#################################################
qemu-system-x86_64 \
-name "win81_amd_exclusive" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 10G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 8,sockets=1,cores=4,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-amd.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net none \
\
-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0


#################################################
# Shared configuration when Nvidia VM is used too:
#################################################
taskset -c 0,2,4,6 qemu-system-x86_64 \
-name "win81_amd" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 8G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 4,sockets=1,cores=2,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-amd.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net none \
\
-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0

qemu_vm_nvidia

# Driver on guest is 335.23, it's last driver that no break KVM and Hyper-V
# Anyway this doesn't make much sense because I only run D3D9 games on that card and those not sensetive to old drivers

# Optional removal of existing data in Nvidia VM
# This VM used for testing so I drop all changes on ever reboot
rm /win81-writable-nvidia.qcow2
rm /home/gameshdd-writable-nvidia.qcow2
cp /win81-writable-nvidia.qcow2.backup /win81-writable-nvidia.qcow2
cp /home/gameshdd-writable-nvidia.qcow2.backup /home/gameshdd-writable-nvidia.qcow2

taskset -c 1,3,5,7 qemu-system-x86_64 \
-name "win81_nvidia" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 7G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 4,sockets=1,cores=2,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-nvidia.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-nvidia.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net nic,model=virtio \
-net bridge,br=br0 \
\
-serial stdio \
-nographic

# Nvidia reset. Boot to reset GPU between usages to avoid host freeze.
# Also make it set fans to minimum so it's not so loud when VM not used.
timeout 10 qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-m 512 \
-cpu host \
-rtc base=localtime \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin \
-boot menu=off,strict=on -vga none -nographic \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on


# Everything else only needed if i915 VGA arbitration not used
# I don't really use it because
##########################################
# kvm=off required to bypass Nvidia driver virtualization limitations
# Hyper-V hv_ options can't be used for same reason

# The way I got consistent VM startup:
 - First boot up "win81_nvidia_reset" configuration with livecd that use Nouveau.
 - Once monitor activated shutdown livecd or just kill QEMU.
   Display will remain with frozen image on it (or shutdown black screen, but active).
 - Now boot "win81_nvidia" configuration.
 - In "Device Manager" disable and enable device.
   Now monitor must lost connection and go idle / switch to different active port.
 - Disable and enable device once again. It's should be working now.

qemu-system-x86_64 \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
\
-m 1G \
\
-cpu host \
-smp 1,sockets=1,cores=1,threads=1 \
\
-name "win81_nvidia_reset" \
-M q35 \
\
-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,x-vga=on \
\
-drive file=/opt/kubuntu-15.04-desktop-amd64.iso,id=isocd \
-device ide-cd,bus=ide.1,drive=isocd

vfio-setup.sh

#!/bin/sh
service lightdm stop

modprobe vfio-pci
echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/remove_id

echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind
echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/remove_id

echo "8086 153b" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:00:19.0" > /sys/bus/pci/devices/0000:00:19.0/driver/unbind
echo "0000:00:19.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "8086 153b" > /sys/bus/pci/drivers/vfio-pci/remove_id

echo "10de 05e2" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:02:00.0" > /sys/bus/pci/devices/0000:02:00.0/driver/unbind
echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "10de 05e2" > /sys/bus/pci/drivers/vfio-pci/remove_id

service lightdm star

xorg.conf.d

/etc/X11/xorg.conf.d/10-intel.conf:
Section "Device"
    Identifier "Intel Graphics"
    Driver "intel"
    Option "DRI" "3"
EndSection

/etc/X11/xorg.conf.d/20-radeon.conf:
Section "Device"
    Identifier "Radeon"
    Driver "radeon"
    Option "DRI3" "1"
EndSection