ASUS ROG Zephyrus G14 2022 GA402RJ pci passthrough

Readme.md

ASUS ROG Zephyrus G14 2022 GA402RJ pci passthrough

This is a record for my own passthrough setup, I can finally use a single laptop for windows gaming and linux programming at the same time.

The device

My G14 is GA402RJ(6800HS + 6700s) with MT7922 WiFi/BT card.

At this time, BIOS version is 309.

Prepare

TLDR: see next section

You need sufficient knowledge about UEFI, QEMU, PCIe, linux. You need a working linux system for this setup (linux will be the host).

Resizable BAR/SAM is not supported in passed-through windows guest yet (confirmed that in my RX460D and 5700xt machine). The first thing is to disable it in bios. Obviously, G14's bios is typical "laptop" bios that no advanced options shown in the menu.

So let's try a little hack to do this. First, get a bios from asus site, unzip it and find it's a UEFI capsule (good job asus).

Use UEFITool to extract it and search keyword 'Resize', then find it is located in PciDynamicSetup

Then extract that efi file and use Universal-IFR-Extractor or IFRExtractor-RS to scan it, then we found

0x8A64 Form Set: PCI Subsystem Settings [ACA9F304-21E2-4852-9875-7FF4881D67A5], ClassGuid0 [93039971-8545-4B04-B45E-32EB8326040E] {0E A7 04 F3 A9 AC E2 21 52 48 98 75 7F F4 88 1D 67 A5 06 00 02 00 01 71 99 03 93 45 85 04 4B B4 5E 32 EB 83 26 04 0E}
0x8A8B  Guid: [0F0B1735-87A0-4193-B266-538C38AF48CE] {5F 15 35 17 0B 0F A0 87 93 41 B2 66 53 8C 38 AF 48 CE 03 88 00}
0x8AA0  Guid: [0F0B1735-87A0-4193-B266-538C38AF48CE] {5F 15 35 17 0B 0F A0 87 93 41 B2 66 53 8C 38 AF 48 CE 04 00 00}
0x8AB5  Default Store: , DefaultId: 0x0 {5C 06 00 00 00 00}
0x8ABB  Default Store: , DefaultId: 0x1 {5C 06 00 00 01 00}
0x8AC1  VarStore: VarStoreId: 0xCCCC [ACA9F304-21E2-4852-9875-7FF4881D67A5], Size: 0x8, Name: PCI_COMMON {24 21 04 F3 A9 AC E2 21 52 48 98 75 7F F4 88 1D 67 A5 CC CC 08 00 50 43 49 5F 43 4F 4D 4D 4F 4E 00}
0x8AE2  Form: PCI Subsystem Settings, FormId: 0x1 {01 86 01 00 06 00}
0x8AE8          Subtitle: Statement.Prompt: 00.00, Flags: 0x0 {02 87 40 00 00 00 00}
0x8AEF          End {29 02}
0x8AF1          Subtitle: Statement.Prompt: , Flags: 0x0 {02 87 02 00 00 00 00}
0x8AF8          End {29 02}
0x8AFA          Subtitle: Statement.Prompt: PCI Settings Common for all Devices:, Flags: 0x0 {02 87 07 00 00 00 00}
0x8B01          End {29 02}
0x8B03          Suppress If {0A 82}
0x8B05                  64 Bit Unsigned Int: 0x1 {45 8A 01 00 00 00 00 00 00 00}
0x8B0F                          64 Bit Unsigned Int: 0x0 {45 0A 00 00 00 00 00 00 00 00}
0x8B19                          Equal {2F 02}
0x8B1B                  End {29 02}
0x8B1D                  Checkbox: Above 4G Decoding, VarStoreInfo (VarOffset/VarName): 0x3, VarStore: 0xCCCC, QuestionId: 0x7006 {06 8E 0C 00 0D 00 06 70 CC CC 03 00 10 00}
0x8B2B                          Default: DefaultId: 0x0, Value (8 bit): 0x1 {5B 06 00 00 00 01}
0x8B31                  End {29 02}
0x8B33          End If {29 02}
0x8B35          Checkbox: Re-Size BAR Support, VarStoreInfo (VarOffset/VarName): 0x4, VarStore: 0xCCCC, QuestionId: 0x702B {06 8E 10 00 11 00 2B 70 CC CC 04 00 10 00}
0x8B43                  Default: DefaultId: 0x0, Value (8 bit): 0x1 {5B 06 00 00 00 01}
0x8B49          End {29 02}
0x8B4B          Suppress If {0A 82}
0x8B4D                  64 Bit Unsigned Int: 0x1 {45 8A 01 00 00 00 00 00 00 00}
0x8B57                          64 Bit Unsigned Int: 0x0 {45 0A 00 00 00 00 00 00 00 00}
0x8B61                          Equal {2F 02}
0x8B63                  End {29 02}
0x8B65                  Checkbox: SR-IOV Support, VarStoreInfo (VarOffset/VarName): 0x5, VarStore: 0xCCCC, QuestionId: 0x7007 {06 8E 12 00 13 00 07 70 CC CC 05 00 10 00}
0x8B73                          Default: DefaultId: 0x0, Value (8 bit): 0x0 {5B 06 00 00 00 00}
0x8B79                  End {29 02}
0x8B7B          End If {29 02}
...

This option is located at PCI_COMMON store offset 0x4.

Disable Resizable BAR

This will disable Resizeable BAR, and harm a little performance at real machine Windows (I can't tell the difference).

If not doing this, guest windows amd driver will give you a code 43, linux is not affected.

This may BRICK your device, at your own risk!

Use grub-mod-setup_var to modify it:

# disable Resizable BAR
setup_var_cv PCI_COMMON 0x4 0x1 0x0
# optional you can enable sriov also
setup_var_cv PCI_COMMON 0x5 0x1 0x1

Configure kernel

Not sure which version of kernel is minimal for this setup, but for my case, it's 5.17.1-arch1-1

My kernel args is root=UUID=someuuid rw vsyscall=emulate iommu=pt pci=nommconf, only iommu=pt matters.

modprobe.d configs is decribed in files: etc_modprobe.d_kvm.conf at /etc/modprobe.d/kvm.conf, etc_modprobe.d_vfio.conf at /etc/modprobe.d/vfio.conf (filename does not matter, you can name it any.conf)

Then regenerate your initcpio/initramdisk:

# for arch with mkinitcpio:
sudo mkinitcpio -P

Start QEMU

Reboot system with newly generated initcpio, then

sudo ./hugepages.sh
sudo ./net.sh
./start.sh

Your VM should work fine.

etc_modprobe.d_kvm.conf

# ignore unknown msrs for windows and disable warning
options kvm ignore_msrs=1 report_ignored_msrs=0
# see https://forum.level1techs.com/t/svm-avic-iommu-avic-improves-interrupt-performance-on-zen-2-etc-based-processors/155226
options kvm_amd nested=1 avic=1 npt=1

etc_modprobe.d_vfio.conf

# use vfio_pci stub for 6700s
# I donot have a GA402RK, you can tell me what 6800s is
options vfio_pci ids=1002:73ef,1002:ab28

hugepages.sh

#!/bin/sh

# this number should be a little more than (memory in MB)/2
# keep execute this script until it show
#  HugePages_Total: 16400

echo 16400 > /proc/sys/vm/nr_hugepages
grep HugePages_Total /proc/meminfo

net.sh

#!/bin/sh

# this script creates bridge br0 and set ${link} master br0

link=enp8s0f4u1u3

ip link add name br0 type bridge
ip link set "${link}" master br0
ip link set br0 up
ip link set "${link}" up

systemctl start dhcpcd@br0
systemctl stop dhcpcd@"${link}"

iptables -P FORWARD ACCEPT

start.sh

#!/bin/sh

# this shell invokes a qemu command for passing through 6700s to guest
# needs prepare hugepages and network
# usb-host,vendorid=0x13d3,productid=0x3568 is MT7922 BT card, use your own (you may have an Intel one) or remove it
# at GA402RJ bios ver 309, 6700s is at 03:00.0 and 03:00.1

sudo qemu-system-x86_64 \
    -drive file=OVMF_CODE.fd,format=raw,if=pflash \
    -drive file=OVMF_VARS.fd,format=raw,if=pflash \
    -machine type=q35,accel=kvm,kernel-irqchip=split \
    -m 32768M \
    -mem-path /dev/hugepages \
    -mem-prealloc \
    -rtc clock=host,base=localtime \
    -smp cpus=16,cores=8,threads=2 \
    -global ICH9-LPC.disable_s3=1 \
    -global ICH9-LPC.disable_s4=1 \
    -nodefaults \
    -enable-kvm \
    -cpu host,kvm=off,+topoext,-hypervisor,hv_vapic,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vendor_id=12alphanum \
    -parallel none \
    -serial none \
    -net none \
    -vga none \
    -monitor stdio \
    -nographic \
    -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=pcie.1 \
    -device vfio-pci,host=03:00.0,bus=pcie.1,addr=00.0,multifunction=on \
    -device vfio-pci,host=03:00.1,bus=pcie.1,addr=00.1 \
    -device virtio-rng-pci,bus=pcie.0 \
    -netdev bridge,id=net0 \
    -device virtio-net-pci,bus=pcie.0,netdev=net0 \
    -device qemu-xhci,id=xhci,bus=pcie.0 \
    -device usb-host,vendorid=0x13d3,productid=0x3568,bus=xhci.0 \
    -device virtio-scsi-pci-transitional \
    -device scsi-hd,drive=system \
    -drive file=/dev/disk/by-id/someid,format=raw,if=none,id=system,cache=none,aio=io_uring,id=system \

exit $?

I've seen a strange problem.

Guest got stuck when opening aida64 "scanning PCI devices" with 100% utilization in all 16 cores, system_reset qemu monitor command will reset guest normally.

(even with this simple setup)

sudo qemu-system-x86_64 \
    -drive file=OVMF_CODE.fd,format=raw,if=pflash \
    -drive file=OVMF_VARS.fd,format=raw,if=pflash \
    -machine type=q35,accel=kvm \
    -enable-kvm \
    -m 4096M \
    -cpu max,kvm=off \
    -smp cpus=4 \
    -nodefaults \
    -monitor stdio \
    -device qemu-xhci \
    -device usb-storage,drive=disk \
    -drive file=/dev/sda,format=raw,if=none,id=disk \
    -device VGA

This problem gone away after I set kvm_amd options nested=0 avic=1 npt=1 (this will disable nested kvm).

Not sure what causes this, maybe should I create an issue to qemu or kernel?

Found a discussion seems to be the same: https://www.reddit.com/r/VFIO/comments/tyhhtl/windows_11_becomes_unresponsive_at_random_when/

Update: using -hypervisor in qemu -cpu options may fix this, this may be a windows 11 hyperv type 1 hypervisor bug

Hello and thank you very much for the detailed how to!
This helped some folks in our community to setup the VM.
Could you confirm or advice how to reverse the setting you mentioned that's done with the UEFI variables?
Would setup_var_cv PCI_COMMON 0x4 0x1 0x1 (last value 1 instead of 0) set BAR to enabled again? I'm not sure about the pattern and the parameters of the command and what value stands for what data we see in the dump above.

@Hyphone

setup_var_cv PCI_COMMON 0x4 0x1 0x1 will enable Resizable BAR.

In extracted ifr list above, Resizable BAR is decribed as

0x8AC1  VarStore: VarStoreId: 0xCCCC [ACA9F304-21E2-4852-9875-7FF4881D67A5], Size: 0x8, Name: PCI_COMMON {24 21 04 F3 A9 AC E2 21 52 48 98 75 7F F4 88 1D 67 A5 CC CC 08 00 50 43 49 5F 43 4F 4D 4D 4F 4E 00}
...                           ^0xcccc matches below 0xcccc                              ^this is store name
0x8B35          Checkbox: Re-Size BAR Support, VarStoreInfo (VarOffset/VarName): 0x4, VarStore: 0xCCCC, QuestionId: 0x702B {06 8E 10 00 11 00 2B 70 CC CC 04 00 10 00}
                                                                                   ^offset here ^0xcccc matches above 0xcccc 
0x8B43                  Default: DefaultId: 0x0, Value (8 bit): 0x1 {5B 06 00 00 00 01}
                                                        ^size   ^default here
0x8B49          End {29 02}

which meaning that config is located at PCI_COMMON store, offset 0x4 with size 0x1 (8bit->1byte) default 0x1

while setup_var_cv usage is

setup_var_cv nameOfVarStore offsetInVarStore [optional variable size] [optional value to write]

you can also use setup_var_cv PCI_COMMON 0x4 0x1 to get this config state

see https://github.com/datasone/grub-mod-setup_var#setup_var_cv for detail.

BTW, I just cannot feel any boost with Resizable BAR enabled, maybe my games are not huge enough to benefit from it.

thank you very much again for this very detailed answer!

Would you be okay with it I would implement some of it to the VFIO guide of asus-linux.org when I tag you and this gist? You are also of course invited to make a contribution to the website if you want to - it's on gitlab.

@Hyphone Just use it as you like. But in fact, I applied refunding for my ga402rj, I will sell it if they reject refunding and will never buy asus laptops. My laptop have returned two times, the first time the main board is broken, second time screen or gpu is broken, the last time is keyboard, I just got tired of returning. So I may not be able to contribute other things in the asus-linux project.