VGA Passthrough on virtual machines in CentOS 7

README.md

Mainline kernel

After your fresh CentOS 7 setup (make sure you install the QEMU/KVM virtualization tools and virt-manager), install the mainline kernel:

sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
sudo rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
sudo yum --enablerepo=elrepo-kernel install kernel-ml
sudo yum -y update

Reboot and be sure to select the mainline (4.0) kernel from the boot menu. Add the menu entry you selected to the GRUB_DEFAULT in /etc/default/grub in order to save your selection.

You may also want to enable updates for the mainline kernel with sudo yum-config-manager --enable elrepo-kernel and disable kernel updates from the CentOS Base repository by adding exclude=kernel* to the bottom of the [updates] section in /etc/yum.repos.d/CentOS-Base.repo.

UEFI firmware

To install the OVMF firmware, create the file kraxel.repo at /etc/yum.repos.d/ with the following contents:

include=https://www.kraxel.org/repos/firmware.repo

Then install the firmware with yum install edk2.git-ovmf-x64.

To advertise UEFI in libvirt, add the following to /etc/libvirt/qemu.conf:

nvram = [
    "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd",
    "/usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw:/usr/share/edk2.git/aarch64/vars-template-pflash.raw",
]

Restart libvirt with sudo systemctl restart libvirtd for these changes to take effect.

The latest libvirt and qemu

Download the following packages from https://repos.fedorapeople.org/repos/openstack/.virt-upstream-el7/ to a folder called qemu-2.2:

• libvirt-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-client-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-config-network-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-config-nwfilter-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-driver-interface-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-driver-lxc-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-driver-network-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-driver-nodedev-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-driver-nwfilter-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-driver-qemu-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-driver-secret-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-driver-storage-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-daemon-kvm-1.2.11-1.el7.centos.x86_64.rpm
• libvirt-python-1.2.11-1.el7.centos.x86_64.rpm
• qemu-guest-agent-2.2.0-1.el7.centos.test.x86_64.rpm
• qemu-img-2.2.0-1.el7.centos.test.x86_64.rpm
• qemu-kvm-2.2.0-1.el7.centos.test.x86_64.rpm
• qemu-kvm-common-2.2.0-1.el7.centos.test.x86_64.rpm
• qemu-kvm-tools-2.2.0-1.el7.centos.test.x86_64.rpm

Change into your qemu-2.2 directory and run the following command to install these packages: sudo yum --nogpgcheck localinstall *.rpm.

You can verify that this worked by running yum list installed | egrep '(libvirt|qemu)' and making sure that the libvirt packages are listed as 1.2.11 and the qemu packages are listed as 2.2.0.

Setting up Networking

Run the following to enable IP forwarding in your kernel:

echo "net.ipv4.ip_forward = 1"|sudo tee /etc/sysctl.d/99-ipforward.conf
sudo sysctl -p /etc/sysctl.d/99-ipforward.conf

Find the name of your network device with ls -la /etc/sysconfig/network-scripts/ (the one that starts with ifcfg), then comment out all the IP-related lines, like so:

TYPE=Ethernet
#BOOTPROTO=dhcp
#DEFROUTE=yes
#IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
## Your name, UUID, and device will probably be different from mine!
NAME=enp8s0
UUID=a258d1da-cc97-4255-bd8e-dcbde55952be
DEVICE=enp8s0
ONBOOT=yes
#PEERDNS=yes
#PEERROUTES=yes
#IPV6_PEERDNS=yes
#IPV6_PEERROUTES=yes
#IPV6_PRIVACY=no
BRIDGE=br0

Notice that we added an extra line for a bridge, so now we create the bridge with sudo vim /etc/sysconfig/network-scripts/ifcfg-br0. It should contain something like this:

DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes
## Your IP address settings will probably be different from mine!
IPADDR=192.168.0.2
NETMASK=255.255.255.0
GATEWAY=192.168.0.1
DNS1=75.75.75.75

The Network Manager GUI is going to want to undo what you've done, so disable it:

sudo systemctl stop NetworkManager 
sudo systemctl disable NetworkManager

Boot options for VGA passthrough

I have an Nvidia card, so I did a lspci -nn | grep -i nvidia to figure out the host address and device ID for my graphics card and HD audio. The host address will look like this: 01:00.0, and your device ID will look like this: 10de:0fc1.

Using your own device IDs, add iommu=on pci-stub.ids=10de:0fc1,10de:0e1b to the end of the GRUB_CMDLINE_LINUX in /etc/default/grub, regenerate the grub configuration with sudo grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg, then reboot. Note that if you don't use UEFI firmware on your host machine, the location of your grub configuration file may be different!

We also need to disable the Nouveau drivers with echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf, then execute the following as root:

mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
dracut -v /boot/initramfs-$(uname -r).img $(uname -r)

SELinux

You might get security errors, so run this as root:

grep qemu-kvm /var/log/audit/audit.log | audit2allow -M mypol
semodule -i mypol.pp
grep ebtables /var/log/audit/audit.log | audit2allow -M mypol
semodule -i mypol.pp

Permission errors on home directory

libvirt has a pretty strict security model, even when running as root. If you'd like to make disks in your home directory accessible, run chmod go+x $HOME.

Virtual Machine configuration

Before trying to install your guest OS, it's very important that you find a Windows 7 installation CD that supports UEFI. I spent 2 weeks trying to get VGA passthrough to work and probably a week of that was due to a bad Windows 7 ISO.

Another thing to note is that the Tiano UEFI that we've configured probably doesn't have any useful drivers for recognizing disks. If you find yourself dumped on a strange UEFI command line that doesn't recognize any disks, download the rEFInd CD image and load it into a virtual CDROM. On boot you can navigate into it, install some drivers, then remove the rEFInd CD and try again with your OS DVD.

The last thing you should be aware of is that you should not attempt the VGA passthrough on your first install of the guest OS. Install the OS, make sure all your devices work (you will likely need Windows Virtio Drivers for many of the virtual devices), and then you can reboot and add PCI Host devices to your virtual machine.

To conclude, I've posted the contents of my virtual machine (accessible via sudo virsh dumpxml <vm_name>) below:

<domain type='kvm' id='8' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win7</name>
  <uuid>527f4b3d-deed-4c36-9858-6339b2703464</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win7_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/VirtualMachines/win7.img'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb0'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb0'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb0'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:ca:67:fc'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x0461'/>
        <product id='0x4d75'/>
        <address bus='6' device='6'/>
      </source>
      <alias name='hostdev2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x056a'/>
        <product id='0x003f'/>
        <address bus='5' device='7'/>
      </source>
      <alias name='hostdev3'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x08bb'/>
        <product id='0x2902'/>
        <address bus='5' device='6'/>
      </source>
      <alias name='hostdev4'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,kvm=off'/>
    <qemu:arg value='-machine'/>
    <qemu:arg value='pc-i440fx-2.2,accel=kvm,usb=off'/>
    <qemu:arg value='-vga'/>
    <qemu:arg value='none'/>
  </qemu:commandline>
</domain>

In order to add the <qemu:commandline> section at the bottom (and add the xmlns:qemu attribute to the <domain> tag at the top), you must edit the XML using sudo virsh edit <vm_name>. We need to add them here because these options are not yet available in the Virtual Machine Manager GUI.

Other Considerations

Sound

You'll notice that the sound coming from your Windows 7 machine sucks. There's nothing you can do about that from within the Virtual Machine Manager (and this thread seems to suggest it's actually a playback buffer limitation from within Windows 7 audio drivers). My advice is to remove the sound from your guest machine and get yourself a cheap USB sound card. I personally went with a Behringer U-Control UCA202. You can find them for about $30 (probably cheaper on eBay).

If you only have a single set of speakers and you don't want to be changing your cables from your USB device to your PC sound card manually, I've got a fix for that:

1. Plug your PC sound card's line-out to your speakers
2. Plug your USB sound card's line-out to your PC sound card's line-in
3. Run the following command on your terminal so that your line-in sound loops back through to your line-out: pactl load-module module-loopback

To make that last command permanent, add the following to the end of your /etc/pulse/default.pa file: load-module module-loopback.

Hello, thanks for the documented part on the network bridge.
Did you got troubles after rebooting with passtrhough enable to assign usb host devices like mouse and keyboard to the guest?

i cant find the packages you refer in https://repos.fedorapeople.org/repos/openstack/.virt-upstream-el7/ do you by any chance have a mirror or a copy of these you could provide ?