Skip to content

Instantly share code, notes, and snippets.

@Redecorating
Last active February 8, 2022 04:16
Show Gist options
  • Star 6 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Redecorating/956a672e6922e285de83fdd7d9982e5e to your computer and use it in GitHub Desktop.
Save Redecorating/956a672e6922e285de83fdd7d9982e5e to your computer and use it in GitHub Desktop.
Hybrid Graphics on t2 macbooks

Where'd the bit on enabling the iGPU go????

It's here now.

Quirks for VFIO gpu passthrough on a MacBookPro16,1:

  1. The Arch wiki has a very good article on this, follow it, add what is mentioned here, and that ought to get this working for you. You can ask on the t2 Linux discord if you have issues.
  2. Isolate AMDGPU from boot with vfio-pci.ids=1002:7340 in kernel parameters (add to /etc/default/grub). If you have a different gpu, check lspci -nn|grep AMD and use your gpu's id. This does make the dGPU use more power than if it wasn't isolated. You don't need to bother with the audio component here.
  3. Install the vendor-reset kernel module (aur: vendor-reset-dkms-git), load it as early as possible (add to /etc/modules-load.d/).
  4. USB and Thunderbolt controllers are in same iommu group as the GPU (at least for MBP 16,1). Pass through the two USB controllers, But don't pass through the Thunderbolt controllers (GPU driver fails in Windows guest if you do).
  5. Thunderbolt controllers must be disabled before the VM boots, i.e. sudo virsh nodedev-detach pci_0000_08_00_0 && sudo virsh nodedev-detach pci_0000_7e_00_0. This means everything in the iommu group is passed through or inactive. You can reattach them with virsh nodedev-reattach after the VM shuts down if you'd like.
  6. When USB controllers are passed through, some USB devices are still accessible by the host. It seems to be everything connected to a USB-A port on a hub that is still accesable to the host.
  7. GPU hostdev element in libvirt xml must have managed='no', i.e. <hostdev mode='subsystem' type='pci' managed='no'>. This stops it from handing the GPU back to Linux and putting in a state where it can't be given back to Windows. This lets you reboot the VM.
  8. If you can't install normal AMDGPU drivers in your Windows guest, use the Bootcamp support software drivers. Get them from Bootcamp Assistant in macOS, transfer the folder to Windows, and run WindowsSupport/BootCamp/Drivers/AMD/AMDGraphics/Setup.exe.
  9. Intel GVT-G is working.

No warranty, idk what I'm doing.

IOMMU Group 0:
00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3ec4] (rev 07)
IOMMU Group 1:
00:01.0 PCI bridge [0604]: Intel Corporation 6th-9th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 07)
00:01.2 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x4) [8086:1909] (rev 07)
01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev 43)
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 14 [Radeon RX 5500/5500M / Pro 5500M] [1002:7340] (rev 43) <-- Isolate with `vfio_pci.ids=1002:xxxx` in kernel arguments, and set managed to no in xml
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38] <-- Don't need to isolate or set managed to no in xml
06:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
07:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
07:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
07:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
07:04.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
08:00.0 System peripheral [0880]: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [8086:15eb] (rev 06) <-- Don't pass through
09:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [8086:15ec] (rev 06) <-- Pass through
7c:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
7d:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
7d:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
7d:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
7d:04.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
7e:00.0 System peripheral [0880]: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [8086:15eb] (rev 06) <-- Don't pass through
7f:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [8086:15ec] (rev 06) <-- Pass through
IOMMU Group 10:
04:00.0 Mass storage controller [0180]: Apple Inc. ANS2 NVMe Controller [106b:2005] (rev 01)
04:00.1 Non-VGA unclassified device [0000]: Apple Inc. T2 Bridge Controller [106b:1801] (rev 01)
04:00.2 Non-VGA unclassified device [0000]: Apple Inc. T2 Secure Enclave Processor [106b:1802] (rev 01)
04:00.3 Multimedia audio controller [0401]: Apple Inc. Apple Audio Device [106b:1803] (rev 01)
IOMMU Group 11:
05:00.0 Network controller [0280]: Broadcom Inc. and subsidiaries BCM4364 802.11ac Wireless Network Adapter [14e4:4464] (rev 04)
IOMMU Group 2:
00:02.0 VGA compatible controller [0300]: Intel Corporation UHD Graphics 630 (Mobile) [8086:3e9b] <-- Only appears if apple-set-os has loaded correctly
IOMMU Group 3:
00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU Group 4:
00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 5:
00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU Group 6:
00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 [8086:a340] (rev f0)
IOMMU Group 7:
00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0)
IOMMU Group 8:
00:1e.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller [8086:a328] (rev 10)
IOMMU Group 9:
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a313] (rev 10)
00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>win10-gpu</name>
<uuid>798b0f23-b57c-4741-8b41-2f78960ae0b7</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static'>8</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='2'/>
<vcpupin vcpu='1' cpuset='3'/>
<vcpupin vcpu='2' cpuset='4'/>
<vcpupin vcpu='3' cpuset='5'/>
<vcpupin vcpu='4' cpuset='8'/>
<vcpupin vcpu='5' cpuset='9'/>
<vcpupin vcpu='6' cpuset='10'/>
<vcpupin vcpu='7' cpuset='11'/>
<emulatorpin cpuset='1,7'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-q35-5.2'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win10-gpu_VARS.fd</nvram>
<bootmenu enable='no'/>
<smbios mode='host'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vpindex state='on'/>
<runtime state='on'/>
<synic state='on'/>
<stimer state='on'>
<direct state='on'/>
</stimer>
<reset state='on'/>
<vendor_id state='on' value='Microsoft Hv'/>
<frequencies state='on'/>
<reenlightenment state='on'/>
<tlbflush state='on'/>
<ipi state='on'/>
<evmcs state='off'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
<ioapic driver='kvm'/>
</features>
<cpu mode='host-passthrough' check='none' migratable='on'>
<topology sockets='1' dies='1' cores='4' threads='2'/>
<feature policy='disable' name='hypervisor'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
<timer name='tsc' present='yes' mode='native'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='yes'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<controller type='usb' index='0' model='nec-xhci' ports='15'>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='1' port='0x8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='2' port='0x9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='3' port='0xa'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='4' port='0xb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='5' port='0xc'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='6' port='0xd'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='7' port='0xe'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='8' port='0xf'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
</controller>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice'>
<listen type='none'/>
<gl enable='yes' rendernode='/dev/dri/by-path/pci-0000:00:02.0-render'/>
</graphics>
<video>
<model type='none'/>
</video>
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
<source>
<address uuid='eb32c120-a312-43a3-92d5-b01c9d20879c'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='no'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='no'>
<source>
<address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
</source>
<boot order='1'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x7f' slot='0x00' function='0x0'/>
</source>
<boot order='2'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</hostdev>
<memballoon model='none'/>
</devices>
<qemu:commandline>
<qemu:arg value='-set'/>
<qemu:arg value='device.hostdev0.x-igd-opregion=on'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.hostdev0.romfile=/vbios_gvt_uefi.rom'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.hostdev0.ramfb=on'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.hostdev0.driver=vfio-pci-nohotplug'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.hostdev0.display=on'/>
<qemu:env name='MESA_LOADER_DRIVER_OVERRIDE' value='i965'/>
</qemu:commandline>
</domain>
@Redecorating
Copy link
Author

Closed the game and then turned off display.

panic(cpu 1 caller 0xfffffff0147bc190): x86 CPU CATERR detected
Debugger message: panic
Memory ID: 0xff
OS release type: User
OS version: 18P4663
macOS version: Not set
Kernel version: Darwin Kernel Version 20.5.0: Fri May  7 22:04:14 PDT 2021; root:xnu-7195.121.3~8/RELEASE_ARM64_T8010
Kernel UUID: 873F4AC5-B66C-3016-8EAC-39D538EEB5A3
iBoot version: iBoot-6723.120.36
secure boot?: YES
x86 EFI Boot State: 0xd
x86 System State: 0x0
x86 Power State: 0x0
x86 Shutdown Cause: 0xf3
x86 Previous Power Transitions: 0x70707060400
PCIeUp link state: 0x1614
Paniclog version: 13
Kernel slide:      0x000000000c860000
Kernel text base:  0xfffffff013864000
mach_absolute_time: 0x32e9452d7c
Epoch Time:        sec       usec
  Boot    : 0x60e121c1 0x0004b9f9
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x60e14554 0x000b15b0

Panicked task 0xffffffe19a080630: 3229 pages, 226 threads: pid 0: kernel_task
Panicked thread: 0xffffffe19a159400, backtrace: 0xffffffe80433b700, tid: 373
		  lr: 0xfffffff013faf49c  fp: 0xffffffe80433b750
		  lr: 0xfffffff013faf2f4  fp: 0xffffffe80433b7c0
		  lr: 0xfffffff0140dadf0  fp: 0xffffffe80433b890
		  lr: 0xfffffff0145d55fc  fp: 0xffffffe80433b8a0
		  lr: 0xfffffff013faf028  fp: 0xffffffe80433bc20
		  lr: 0xfffffff013faf028  fp: 0xffffffe80433bc80
		  lr: 0xfffffff014fdf200  fp: 0xffffffe80433bca0
		  lr: 0xfffffff0147bc190  fp: 0xffffffe80433bcd0
		  lr: 0xfffffff0147aa520  fp: 0xffffffe80433bd30
		  lr: 0xfffffff0147ac41c  fp: 0xffffffe80433bdc0
		  lr: 0xfffffff0147a9bc4  fp: 0xffffffe80433be50
		  lr: 0xfffffff0146ae814  fp: 0xffffffe80433be80
		  lr: 0xfffffff01452a010  fp: 0xffffffe80433bec0
		  lr: 0xfffffff014529890  fp: 0xffffffe80433bf00
		  lr: 0xfffffff0145e05a0  fp: 0x0000000000000000

@Redecorating
Copy link
Author

amdgpu.bapm=0 amdgpu.msi=0 amdgpu.dpm=0 amdgpu.aspm=0 amdgpu.runpm=0 seems to fix it, need to figure out which one it was but i'll do that later

@Redecorating
Copy link
Author

amdgpu.dpm=0 fixes it without too much impact on battery life. Games feel a bit slower with dri_prime, i need to see if that's just dri_prime or if disabling dpm has an impact.

@fakhrytatanaki
Copy link

fakhrytatanaki commented Jul 12, 2021

I had similar issues with my MacBook Pro 16,1. For instance, when I try to change the resolution, the device freezes with a black screen then automatically reboots with CATERR message, applying amdgpu.dpm=0 fixed it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment