- Need OVMF UEFI firmware for 64 bit machines installed. Be sure guest is using UEFI firmware! You can't change this after you create a machine in virt-manager!
- Be sure using i440FX chipset.
- Enable a qemu-ga channel for guest. Make a VirtIO serial controller.
- Make sure your hard disk is using VirtIO, not IDE.
- Be sure you have the VirtIO floppy image for Windows 7 install, you'll need to load drivers so installer can see the VirtIO disk.
- Bridging to main Ethernet interface works fine, depsite the warnings. Use the VirtIO ethernet type.
- The VirtIO 0.1.141 drivers work well for me in Win 7 Pro 64 bit.
- To avoid the NVIDIA "Code 43" error, see the Arch Linux wiki about modifying the hypervisor name. Or somehow find older NVIDIA drivers prior to version 337 that work with your card.
- Remove the "sound" device, sound pass through sucks. Just pass through a USB sound card/device but this isn't needed at install time.
- Windows 7 was not fun, Windows 10 shows more promise. Win7 apparently has issues with pure UEFI machines.
- Ran into issue during Win10 install where USB devices on the passed-thru PCIe card would not enumerate to the installer's questions after the 2nd reboot during the install process. In the first part of the installer they worked fine, however. I had to not pass thru the entire PCIe USB card but just a keyboard and mouse USB device to get past this, probably best to pass thru only the USB devices for the installer, until full Windows is loading correctly.
- For NVIDIA "code 43" errors, I had to do this within my
virsh editconfig editor (see full config file in other part of gist), the "hidden state" part isn't explained well in many of the good docs online, most focus only on the vendor_id part:
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='lovenpassion'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
- https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
- https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/index
- https://wiki.libvirt.org/page/I_created_an_external_snapshot,_but_libvirt_will_not_let_me_delete_or_revert_to_it
- https://wiki.archlinux.org/index.php/libvirt
- You can pass thru both the PCIe graphics and a USB keyboard and mouse.
- Be sure you have the VirtIO drivers floppy attached to the VM.
- Don't use the VirtIO driver ISO image to install all the needed VirtIO drivers, just use the floppy during install for VirtIO disks. Then go get the SPICE guest tools installer and use that, it'll do everything for you automatically: https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-latest.exe
- Remove the spice devices from the VM configuration once you're done with the install of Windows and want to do PCIe passthrough.
- Make sure that all devices in the same IOMMU group as your PCIe graphics card only get bound by the vfio driver (watch out for HDMI audio devices, remove them from PCI bus with a hackjob udev rule if present).
- Pass through your PCIe graphics card and a USB keyboard and mouse. Better is to pass through an entire USB controller if you can, then you can avoid passing individual devices.
- You can't do normal snapshots when using UEFI. You get to do them manually. See above resource for making external snapshots. I make snapshots like this (domain is the name of the VM, diskspec is where you want the snapshot output file to live):
sudo virsh snapshot-create-as --domain win7 --name=20180720-01 --description="Fresh Windows 7 install" --diskspec vda,file=/var/lib/libvirt/images/win7-20180720-01.qcow2 --disk-only --atomic --no-metadataand then the snapshot file points back to your normal disk and will grow as you make changes to the VM's disk. You don't need to adjust anything in virt-manager or virsh. - If you use the
--no-metadataoption, you can't directly restore to a given snapshot state, you'd have to do a new snapshot with metadata first. So using it might not always be the best. But it's not yet clear to me how the metadata is handled when doing avirsh blockcommitto squash snapshots onto their base and if metadata would need to be cleaned up in virsh or not... - I haven't yet tried to boot from a snapshot or do any of the snapshot compression things...
- When creating snapshots, generally using the
--no-metadataoption is recommended in the docs. This makes deleting or commiting or pulling snapshots around less horrible (as you don't have to go back and deal with the now incorrect metadata) but it seems that some of the tools for looking at what snapshots apply to a given host withvirshdon't really work then. - When trying to commit snapshots, use of the
virsh blockcommittool requires setting the/etc/libvirt/qemu.conffile to includesecurity_driver = "none"or else SELinux/AppArmor will prevent QEMU from "reopening" the lower level read-only qcow2 files as read-write so that the commit can happen. This seems to be a bug fixed in Ubuntu for an older version but is still present in Debian testing today: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1554031 - After committing a set of snapshots together (or assume pulling, too),
virsh blockcommitdoesn't actually delete the old upper layer qcow2 files even though they are completely useless now (if you change lower level snapshots, you CANNOT use the upper level ones any longer, the upper level ones are just deltas, like overlayfs but at a 4k disk level, not at a file level). You have to delete these now useless upper level snapshot files yourself. - I'm leaving virt-manager behind now, I think it's actively hurting my ability to do what I want. Even though the
virshcommand line tool isn't super easy, the man page is pretty decent. THIS MAY NOT BE TRUE, I MAY HAVE MISDIAGNOSED ISSUES I WAS HAVING - Rather than do snapshots of the VM, I'm thinking I will just have one snapshot after a clean install into a qcow2 disk then I'll just snapshot the entire host filesystem including the qcow2 files. Since I need backups anyways of everything on my PC and since it should be pretty easy to schedule hourly/daily/weekly snapshots when using btrfs, I can just go grab and older qcow2 file-set from an old snapshot if needed. I will likely need to pause the Windows vm with
virsh suspendthen do snapshotting then unpause the vmvirsh resume.
Ideally, we want to put our Windows 10 guest onto its own CPUs so as to avoid high load on the host machine causing
the guest to stutter or otherwise bog down. The best way I've found to do this so far is to give my Windows 10 guest 2 dedicated "CPUs"
(in quotes because it's really just a hyperthread pair on a single core). To do this, find out the hyperthread pairs
using sudo virsh capabilities | grep siblings and pick CPUs which are siblings but are not CPU number 0. I'm using
CPU 11 and 23 as they are siblings and should be the last hyperthread pair on my E5-2687Wv4. Use these CPU numbers
to pass to the host Linux kernel command line isolcpus=11,23 (likely in your bootloader configuration, I use grub2).
This will prevent the Linux kernel scheduler from using these CPUs when scheduling tasks. We can then tell the VM
to use these two CPUs as its virtual CPUs in sudo virsh edit <VM> like:
<vcpu placement='static'>2</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='11'/>
<vcpupin vcpu='1' cpuset='23'/>
</cputune>
Now the guest will get full access to CPUs 11 and 23. This solves my high load (bitbaking) on the host causing the Windows 10 guest to stutter and be very slow. Even with the host fully loaded (all 22 other "CPUs" at full tilt) the guest doesn't stutter, although it is slightly slower to perform some operations, it's not unbearable. Likely the slowness is due to QEMU/KVM emulator processes still being able to roam between other CPUs for the overhead operations. This can be constrained by other options but I'm not sure it's worth it for now.
See https://libvirt.org/formatdomain.html#hypervisor-features :