Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Current QEMU dual-VM configuration. Thanks to everyone who helped me!

Check it's working

DRI_PRIME=1 glxinfo | grep OpenGL

Unbind AMD GPU from Radeon driver

echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/new_id echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/remove_id echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/new_id echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/remove_id

Use it in QEMU

qemu-system-x86_64
-name "win81_amd"

-bios /usr/share/seabios/bios.bin
-enable-kvm

-usb
-device usb-kbd
-device usb-mouse

-vga none
-device qxl

-m 12G

-no-hpet
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic
-smp 4,sockets=1,cores=2,threads=2

-soundhw ac97

-M q35
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio

-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

-net none

-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0

-monitor telnet:127.0.0.1:1234,server,nowait

Now you need to remove PCI device

echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove echo 1 > /sys/bus/pci/devices/0000:01:00.1/remove

Rescan will attach it back to Radeon

echo 1 > /sys/bus/pci/rescan

Check it via PRIME once again

DRI_PRIME=1 glxinfo | grep OpenGL

Introduction:
This is my configuration for VMs and I'm change it from time to time.
At moment I'm decide to use following configuration that work best for hardware I have:
- By default I have "i915.enable_hd_vgaarb=0" because most of time I don't need Nvidia GPU.
- When I only need VM with AMD HD6950 I boot with option unchanged to keep Intel DRI working on host
- Once I need to run both VMs simultaneously I just change it to "i915.enable_hd_vgaarb=1" at boot time.
- Have two sets of options for AMD VM because w/o Nvidia one I have more RAM and don't to use taskset
Hardware:
- ASRock Z87 Extreme6
- i7 4771 and 32GB RAM
- 60GB SSD (for /), 2x 250GB HDDs in RAID (for /home), 1TB HDD with NTFS partition for Windows content
- AMD HD 6950 1GB and Nvidia GTX 260
Software:
- Ubuntu 14.04
- Linux Kernel 3.19 with ACS and i915 VGAarbiter patches
- Preemption Model set to Low Latency Desktop
- Timer frequency set to 1000HZ
- QEMU 2.0.0~rc1+dfsg-0ubuntu3 from repositories
- In past I used QEMU 2.2.1 from ppa:syseleven-platform/virtualization, but then I decide to use older Nvidia driver instead.
Also with 2.2.1 I had weird disconnect issue in "Path of Exile" game while changing areas.
System configuration:
- /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash i915.enable_hd_vgaarb=1 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"
- /etc/modprobe.d/blacklist-gpu.conf:
blacklist radeon
blacklist nouveau
- /etc/qemu/bridge.conf:
allow br0
- /etc/sysctl.d/40-hugepage.conf:
vm.nr_hugepages = 8200
- /etc/fstab:
hugetlbfs /dev/hugepages hugetlbfs mode=1770,gid=78 0 0
Both VMs using same "backing files" that not modified in runtime:
- /win81-base.qcow2 [13GB only!], compressed
This QCOW2 image contain installed and configured Windows 8.1
- /dev/disk/by-uuid/0AD949C94647B394
It's NTFS partition on 1TB HDD with games and some software installed.
Those files are base for QCOW2 images that used by virtual machines:
qemu-img create -b /win81-base.qcow2 -f qcow2 /win81-writable-amd.qcow2
qemu-img create -b /win81-base.qcow2 -f qcow2 /win81-writable-nvidia.qcow2
qemu-img create -b /dev/disk/by-uuid/0AD949C94647B394 -f qcow2 /home/gameshdd-writable-amd.qcow2
qemu-img create -b /dev/disk/by-uuid/0AD949C94647B394 -f qcow2 /home/gameshdd-writable-nvidia.qcow2
So I basically store VM-specific changes to my system drive on SSD and changes for content stored on HDD RAID.
Kernel build:
apt-get install kernel-package
apt-get build-dep linux-image-$(uname -r)
apt-get build-dep linux-image-3.19.0-23-generic
apt-get source linux-image-$(uname -r)
apt-get source linux-image-3.19.0-23-generic
cd linux-*
# Now just moment to hate Ubuntu packaging, I do not use VBOX!
rm -rf ubuntu/vbox
# Comment out virtualbox crap
nano ubuntu/Makefile
# Apply patches (my patch files are above kernel directory)
# Personally I just used patch version published by someone on google drive
https://gist.github.com/ArseniyShestakov/768a426e79e2b901f9eb
patch -p1 < ../override_for_missing_acs_capabilities.patch
patch -p1 < ../i915_317.patch
# Copy your current kernel config
cp -vi /boot/config-`uname -r` .config
# Change some options for better performance
# Processor Type and Features > Preemption Model > Preemptible Kernel (Low Latency Desktop)
# (CONFIG_PREEMPT=y)
# Processor Type and Features > Timer Frequency > 1000 Hz
# (CONFIG_HZ_1000=y CONFIG_HZ=1000)
make menuconfig
# Now build kernel
fakeroot make-kpkg -j 8 --initrd kernel-image kernel-headers --append-to-version=-acs-igdarb-lowpremption-timer1000hz
# Install built packages
dpkg -i ../*acs-igdarb-lowpremption-timer1000*.deb
All useful information may be found there:
https://bbs.archlinux.org/viewtopic.php?id=162768
https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit#gid=0
http://vfio.blogspot.com
My personal expirience of why I use some of options:
https://www.reddit.com/r/linux_gaming/comments/3dr62f/set_up_gpu_passthrough_without_using_the_cli/ct8gr34
https://www.reddit.com/r/linux_gaming/comments/2z2y7h/gpu_passthrough_or_how_to_play_any_game_at_near/cpfh76x
# This system use real network card (00:19.0) as I have two in my mobo.
#################################################
# Exclusive configuration when no other VM running (more RAM, all cores)
#################################################
qemu-system-x86_64 \
-name "win81_amd_exclusive" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 10G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 8,sockets=1,cores=4,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-amd.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net none \
\
-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0
#################################################
# Shared configuration when Nvidia VM is used too:
#################################################
taskset -c 0,2,4,6 qemu-system-x86_64 \
-name "win81_amd" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 8G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 4,sockets=1,cores=2,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-amd.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net none \
\
-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0
# Driver on guest is 335.23, it's last driver that no break KVM and Hyper-V
# Anyway this doesn't make much sense because I only run D3D9 games on that card and those not sensetive to old drivers
# Optional removal of existing data in Nvidia VM
# This VM used for testing so I drop all changes on ever reboot
rm /win81-writable-nvidia.qcow2
rm /home/gameshdd-writable-nvidia.qcow2
cp /win81-writable-nvidia.qcow2.backup /win81-writable-nvidia.qcow2
cp /home/gameshdd-writable-nvidia.qcow2.backup /home/gameshdd-writable-nvidia.qcow2
taskset -c 1,3,5,7 qemu-system-x86_64 \
-name "win81_nvidia" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 7G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 4,sockets=1,cores=2,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-nvidia.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-nvidia.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net nic,model=virtio \
-net bridge,br=br0 \
\
-serial stdio \
-nographic
# Nvidia reset. Boot to reset GPU between usages to avoid host freeze.
# Also make it set fans to minimum so it's not so loud when VM not used.
timeout 10 qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-m 512 \
-cpu host \
-rtc base=localtime \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin \
-boot menu=off,strict=on -vga none -nographic \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on
# Everything else only needed if i915 VGA arbitration not used
# I don't really use it because
##########################################
# kvm=off required to bypass Nvidia driver virtualization limitations
# Hyper-V hv_ options can't be used for same reason
# The way I got consistent VM startup:
- First boot up "win81_nvidia_reset" configuration with livecd that use Nouveau.
- Once monitor activated shutdown livecd or just kill QEMU.
Display will remain with frozen image on it (or shutdown black screen, but active).
- Now boot "win81_nvidia" configuration.
- In "Device Manager" disable and enable device.
Now monitor must lost connection and go idle / switch to different active port.
- Disable and enable device once again. It's should be working now.
qemu-system-x86_64 \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
\
-m 1G \
\
-cpu host \
-smp 1,sockets=1,cores=1,threads=1 \
\
-name "win81_nvidia_reset" \
-M q35 \
\
-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,x-vga=on \
\
-drive file=/opt/kubuntu-15.04-desktop-amd64.iso,id=isocd \
-device ide-cd,bus=ide.1,drive=isocd
#!/bin/sh
service lightdm stop
modprobe vfio-pci
echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/remove_id
echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind
echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/remove_id
echo "8086 153b" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:00:19.0" > /sys/bus/pci/devices/0000:00:19.0/driver/unbind
echo "0000:00:19.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "8086 153b" > /sys/bus/pci/drivers/vfio-pci/remove_id
echo "10de 05e2" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:02:00.0" > /sys/bus/pci/devices/0000:02:00.0/driver/unbind
echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "10de 05e2" > /sys/bus/pci/drivers/vfio-pci/remove_id
service lightdm star
/etc/X11/xorg.conf.d/10-intel.conf:
Section "Device"
Identifier "Intel Graphics"
Driver "intel"
Option "DRI" "3"
EndSection
/etc/X11/xorg.conf.d/20-radeon.conf:
Section "Device"
Identifier "Radeon"
Driver "radeon"
Option "DRI3" "1"
EndSection
@fld

This comment has been minimized.

Copy link

commented Dec 12, 2016

Hi,

What versions of Xorg/Mesa/Intel DDX are you using?

I'm on Debian jessie, running kernel v4.7.8-1~bpo8+1, using xorg-server v1.16.4-1 with backported xserver-xorg-video-intel v2.99.917+git20161105-1 and backported Mesa v12.0.4-2. I'm using Haswell graphics for desktop (with DRI3 enabled) and PRIME offloading games to Radeon HD7770. Works great, until I try unbinding the HD7770 from the radeon driver while Xorg is running... That's when the kernel just freezes dead :) Apparently you've managed to unbind your radeon while Xorg is running?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.