Skip to content

Instantly share code, notes, and snippets.

@geerlingguy
Last active October 14, 2024 11:25
Show Gist options
  • Save geerlingguy/9d78ea34cab8e18d71ee5954417429df to your computer and use it in GitHub Desktop.
Save geerlingguy/9d78ea34cab8e18d71ee5954417429df to your computer and use it in GitHub Desktop.
Increase the BAR memory address space for PCIe devices on the Raspberry Pi Compute Module 4
#!/bin/bash
# The default BAR address space available on the CM4 may be too small to allow
# some devices to initialize correctly. To avoid 'failed to assign memory'
# errors on boot, you can increase the range of the PCIe bus in the Raspberry
# Pi's Device Tree (a .dtb file specific to each Pi model).
#
# You should probably read up on Device Trees if you don't know what they are:
# https://www.raspberrypi.org/documentation/configuration/device-tree.md
#
# NOTE: The default BAR allocation was increased to 1 GB in this commit:
# https://github.com/raspberrypi/linux/commit/54db4b2fa4d17251c2f6e639f849b27c3b553939
# Download the trial firmware from this post:
# https://www.raspberrypi.org/forums/viewtopic.php?p=1761834#p1761834
# and replace the corresponding files in the boot volume.
# Back up current Device Tree for the CM4.
sudo cp /boot/bcm2711-rpi-cm4.dtb /boot/bcm2711-rpi-cm4.dtb.bak
# Decompile the current Device Tree to a dts (source) file.
dtc -I dtb -O dts /boot/bcm2711-rpi-cm4.dtb -o ~/test.dts
# Edit the file and change the PCIe bus range as mentioned in:
# https://www.raspberrypi.org/forums/viewtopic.php?p=1746665#p1746665
nano ~/test.dts
# Replace the line that allocates 1 GB of RAM in the pci section:
# ranges = <0x02000000 0x0 0xc0000000 0x6 0x00000000 0x0 0x40000000>;
#
# with the following line, which provides up to 8 GB of RAM:
# ranges = <0x02000000 0x0 0x00000000 0x6 0x00000000 0x2 0x00000000>;
# Also replace the scb ranges to:
# ranges = <0x0 0x7c000000 0x0 0xfc000000 0x0 0x03800000>,
# <0x0 0x40000000 0x0 0xff800000 0x0 0x00800000>,
# <0x6 0x00000000 0x6 0x00000000 0x2 0x00000000>,
# <0x0 0x00000000 0x0 0x00000000 0x0 0xfc000000>;
# Recompile the Device Tree from the dts (source) file.
dtc -I dts -O dtb ~/test.dts -o ~/test.dtb
# Copy the new Device Tree into place.
sudo mv ~/test.dtb /boot/bcm2711-rpi-cm4.dtb
# Reboot.
sudo reboot
@geerlingguy
Copy link
Author

geerlingguy commented Oct 24, 2020

This change is necessary to make the Raspberry Pi Compute Module 4 handle many 3rd party PCIe adapters (e.g. GPUs, SATA cards, etc.), as they require more BAR (Base Address Register) PCI configuration space than is provided by default (~64 MB). The default amount was adequate for simpler devices, like the VL805 USB 3.0 controller used in the Raspberry Pi 4, but many PCIe devices require at least hundreds of MB of BAR space, so this script increases the limit to 1 GB.

The default limit may be increased in future versions of Raspberry Pi OS, since this change is fairly innocuous.

For some examples of devices that may need larger BAR space, see my Raspberry Pi PCIe device compatibility database

@geerlingguy
Copy link
Author

@pelwell
Copy link

pelwell commented Oct 28, 2020

It's worth noting that the outbound window isn't allocating RAM, it's reserving a chunk of the PCI address space for devices and mapping it into the ARM's physical address space. No RAM is lost in the process.

The default outbound window on rpi-5.9.y (available from our "next" firmware branch) and rpi-5.10.y has now been expanded to 1GB, so in the future the dtb surgery described above won't be necessary.

@geerlingguy
Copy link
Author

geerlingguy commented Oct 28, 2020

@pelwell - ah, thanks for clearing up my misunderstanding, and also for helping get this change into the updated firmware! That'll make it easier and I can avoid patching the DTB.

And the commit: raspberrypi/linux@54db4b2

@geerlingguy
Copy link
Author

geerlingguy commented Nov 4, 2020

Also can do 0xffffffff for 4 GB, or 0x80000000 for 2 GB.

Note: Error: /home/pi/test.dts:1970.59-70 Value out of range for 32-bit array element if I tried 0x100000000.

@geerlingguy
Copy link
Author

geerlingguy commented Nov 27, 2020

Another update: this post in the Pi Forums contains some trial firmware that allows the BAR space to take up the maximum possible space on the Pi (8GB of PCIe space for any Pi with >= 4 GB of RAM). To use it:

  1. Downloaded the trial firmware and replace the corresponding files in the boot volume.

  2. Set scb ranges to:

       ranges = <0x0 0x7c000000  0x0 0xfc000000  0x0 0x03800000>,
                <0x0 0x40000000  0x0 0xff800000  0x0 0x00800000>,
                <0x6 0x00000000  0x6 0x00000000  0x2 0x00000000>,
                <0x0 0x00000000  0x0 0x00000000  0x0 0xfc000000>;
    
  3. Set pcie ranges to:

       ranges = <0x02000000 0x0 0x00000000 0x6 0x00000000
                 0x2 0x00000000>;
    

More details on the entire process used to try to get a GPU working are in this comment: geerlingguy/raspberry-pi-pcie-devices#6 (comment)

@TheMindVirus
Copy link

Is the "ranges = /bits/ 64 <0x0 0x600000000 0x00000000c0000000>;" syntax applicable to this version of the Device Tree Compiler (dtc)?
https://github.com/vagrantc/device-tree-compiler/blob/063d02187b584e40998a700d64a3530f287c68e2/Documentation/dts-format.txt#L52
Alternatively it would be ideal not to have to specify the number of bits at all and for the compiler not to complain of the 32-bit limit.
At least the meaning is somewhat semantically human readable this way. I'm already reading it thinking parameter 2 is 0x7c000000 -_(\

@RoboTakoh
Copy link

RoboTakoh commented Apr 5, 2021

I've been trying to get this to work for my firewire card for a few days, but I can't get it to work. Tried 64bit and 32bit raspberry pi os. Tried different amounts of ram, up to 4gb. I've got a cm4 with 4gb or ram and emmc.
I'm not sure what i'm doing wrong. Do I need to recompile the kernel? I'm not sure what to change if I do :'(
Any suggestions? Thank you

(Also, I have another firewire card that has a TI chip. Wonder if i should try using that one as well.)

uname -a
Linux raspberrypi 5.10.25-v7l+ #1408 SMP Mon Mar 22 12:49:24 GMT 2021 armv7l GNU/Linux

>  [    1.202310] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:

> [    1.202341] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
> [    1.202421] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x06fffffffe -> 0x00c0000000
> [    1.202521] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
> [    1.256089] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
> [    1.256493] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
> [    1.256514] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    1.256536] pci_bus 0000:00: root bus resource [mem 0x600000000-0x6fffffffe] (bus address [0xc0000000-0x1bffffffe])
> [    1.256625] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
> [    1.256874] pci 0000:00:00.0: PME# supported from D0 D3hot
> [    1.260339] PCI: bus0: Fast back to back transfers disabled
> [    1.260364] pci 0000:00:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
> [    1.260615] pci 0000:01:00.0: [1b21:1080] type 01 class 0x060400
> [    1.260973] pci 0000:01:00.0: supports D1 D2
> [    1.260991] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    1.264348] PCI: bus1: Fast back to back transfers disabled
> [    1.264374] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [    1.264558] pci_bus 0000:02: extended config space not accessible
> [    1.264696] pci 0000:02:00.0: [1106:3044] type 00 class 0x0c0010
> [    1.264763] pci 0000:02:00.0: reg 0x10: [mem 0x00000000-0x000007ff]
> [    1.264803] pci 0000:02:00.0: reg 0x14: [io  0x0000-0x007f]
> [    1.265035] pci 0000:02:00.0: supports D2
> [    1.265052] pci 0000:02:00.0: PME# supported from D2 D3hot D3cold
> [    1.268451] PCI: bus2: Fast back to back transfers disabled
> [    1.268472] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
> [    1.268501] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 02
> [    1.268545] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
> [    1.268568] pci 0000:01:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
> [    1.268586] pci 0000:01:00.0: BAR 7: no space for [io  size 0x1000]
> [    1.268603] pci 0000:01:00.0: BAR 7: failed to assign [io  size 0x1000]
> [    1.268626] pci 0000:02:00.0: BAR 0: assigned [mem 0x600000000-0x6000007ff]
> [    1.268652] pci 0000:02:00.0: BAR 1: no space for [io  size 0x0080]
> [    1.268669] pci 0000:02:00.0: BAR 1: failed to assign [io  size 0x0080]
> [    1.268688] pci 0000:01:00.0: PCI bridge to [bus 02]
> [    1.268718] pci 0000:01:00.0:   bridge window [mem 0x600000000-0x6000fffff]
> [    1.268758] pci 0000:00:00.0: PCI bridge to [bus 01-02]
> [    1.268783] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
> [    1.269171] pcieport 0000:00:00.0: enabling device (0140 -> 0142)
> [    1.269396] pcieport 0000:00:00.0: PME: Signaling with IRQ 64
> [    1.269677] pci 0000:01:00.0: Disabling ASPM L0s/L1
> [    1.275651] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
> [    1.278668] iproc-rng200 fe104000.rng: hwrng registered
> [    1.278982] vc-mem: phys_addr:0x00000000 mem_base=0x3ec00000 mem_size:0x40000000(1024 MiB)
> [    1.279840] gpiomem-bcm2835 fe200000.gpiomem: Initialised: Registers at 0xfe200000

> 00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
> 01:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)
> 02:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev 80)

 
> lspci -vmmnn
> Slot:	00:00.0
> Class:	PCI bridge [0604]
> Vendor:	Broadcom Limited [14e4]
> Device:	Device [2711]
> Rev:	20
> 
> Slot:	01:00.0
> Class:	PCI bridge [0604]
> Vendor:	ASMedia Technology Inc. [1b21]
> Device:	ASM1083/1085 PCIe to PCI Bridge [1080]
> Rev:	01
> 
> Slot:	02:00.0
> Class:	FireWire (IEEE 1394) [0c00]
> Vendor:	VIA Technologies, Inc. [1106]
> Device:	VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [3044]
> SVendor:	VIA Technologies, Inc. [1106]
> SDevice:	VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [3044]
> Rev:	80
> ProgIf:	10
 

> lspci -nnk
> 00:00.0 PCI bridge [0604]: Broadcom Limited Device [14e4:2711] (rev 20)
> 	Kernel driver in use: pcieport
> 01:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01)
> 02:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] (rev 80)
> 	Subsystem: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044]



> lspci -vvnnk
> 00:00.0 PCI bridge [0604]: Broadcom Limited Device [14e4:2711] (rev 20) (prog-if 00 [Normal decode])
> 	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Latency: 0, Cache Line Size: 64 bytes
> 	Interrupt: pin A routed to IRQ 64
> 	Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
> 	I/O behind bridge: 00000000-00000fff
> 	Memory behind bridge: c0000000-c00fffff
> 	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> 	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
> 		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> 	Capabilities: access denied
> 	Kernel driver in use: pcieport
> 
> 01:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01) (prog-if 00 [Normal decode])
> 	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Interrupt: pin A routed to IRQ 64
> 	Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
> 	Memory behind bridge: c0000000-c00fffff
> 	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> 	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B-
> 		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> 	Capabilities: access denied
> 
> 02:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] (rev 80) (prog-if 10 [OHCI])
> 	Subsystem: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044]
> 	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping+ SERR+ FastB2B- DisINTx-
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Interrupt: pin A routed to IRQ 0
> 	Region 0: Memory at 600000000 (32-bit, non-prefetchable) [disabled] [size=2K]
> 	Region 1: I/O ports at <unassigned> [disabled]
> 	Capabilities: access denied


@geerlingguy
Copy link
Author

It looks like the memory is being assigned. Note that no IO BAR space is available on the Pi, so you'll never see IO space allocated.

I so far haven't seen anyone get a Firewire card working :(

@RoboTakoh
Copy link

RoboTakoh commented Apr 8, 2021

Thank you very much for the reply and your helpful posts.
Ah that sucks. That's the whole reason why I bought the cm4 & io board.

@sbates130272
Copy link

@geerlingguy I just tried a 512MB BAR device using the latest release of Pi OS (32 bit) and it worked fine! I am still seeing issues with BAR size allocated on Ubuntu Server 20.04.2 LTS though.

@geerlingguy
Copy link
Author

Just updated this Gist to reflect the new default (1 GB BAR space), as well as how to increase it to 8 GB following the guide linked earlier in the comments.

@l00nix
Copy link

l00nix commented Oct 17, 2021

Hi there -

Is this bar space increase still a required procedure with latest firmware and kernels? Asking as I am getting kernel panics when booting the CM4 on the CM4 i/o board with the ASHATA PCE4PCE-A01 1 to 4 Port PCIe Switch AND the ELUTENG PCIE USB 3.0 Card 4 Ports PCI Expree to USB Expansion Card. With both plugged in individually the CM4 boots up.

@geerlingguy
Copy link
Author

@l00nix - No, this shouldn't be necessary unless you need a full 8 GB of BAR space (and vanishingly few cards I've tested need that).

For the kernel panic, feel free to open an issue on my Pi PCIe site/repo for that card and see if anyone else can help debug.

Copy link

ghost commented Dec 5, 2021

Hello,
I'm trying to make a Xonar DX audio card work but there's some issues.
The CMI8788 chipset seems to be ok, has snd_virtuoso as driver. The issue seems to be with the PLX bridge, a PEX8112.
From what I've seen for linux it should use the shpchp driver, which I compiled into the kernel, but doesn't seem to load it.

dmesg :

[    1.340364] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    1.340833] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.340883] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.340985] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[    1.341094] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x007fffffff -> 0x0400000000
[    1.366959] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
[    1.367344] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    1.367381] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.367417] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[    1.367529] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    1.367791] pci 0000:00:00.0: PME# supported from D0 D3hot
[    1.371322] PCI: bus0: Fast back to back transfers disabled
[    1.371363] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.371704] pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
[    1.375429] PCI: bus1: Fast back to back transfers disabled
[    1.375471] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.375681] pci_bus 0000:02: extended config space not accessible
[    1.376296] pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
[    1.376379] pci 0000:02:04.0: reg 0x10: [io  0x42000000-0x420000ff]
[    1.376651] pci 0000:02:04.0: supports D1 D2
[    1.379665] PCI: bus2: Fast back to back transfers disabled
[    1.379701] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[    1.379746] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 02
[    1.379802] pci 0000:01:00.0: BAR 7: no space for [io  size 0x1000]
[    1.379835] pci 0000:01:00.0: BAR 7: failed to assign [io  size 0x1000]
[    1.379870] pci 0000:02:04.0: BAR 0: no space for [io  size 0x0100]
[    1.379902] pci 0000:02:04.0: BAR 0: failed to assign [io  size 0x0100]
[    1.379936] pci 0000:01:00.0: PCI bridge to [bus 02]
[    1.380005] pci 0000:00:00.0: PCI bridge to [bus 01-02]
[    1.380641] pcieport 0000:00:00.0: PME: Signaling with IRQ 62
[    4.941635] snd_virtuoso 0000:02:04.0: invalid PCI I/O range

journalctl:

Dec 05 15:17:01 rbnet kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x007fffffff -> 0x0400000000
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
Dec 05 15:17:01 rbnet kernel: pci_bus 0000:00: root bus resource [bus 00-ff]
Dec 05 15:17:01 rbnet kernel: pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
Dec 05 15:17:01 rbnet kernel: pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
Dec 05 15:17:01 rbnet kernel: pci 0000:00:00.0: PME# supported from D0 D3hot
Dec 05 15:17:01 rbnet kernel: PCI: bus0: Fast back to back transfers disabled
Dec 05 15:17:01 rbnet kernel: pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: supports D1
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: PME# supported from D0 D1 D3hot
Dec 05 15:17:01 rbnet kernel: PCI: bus1: Fast back to back transfers disabled
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 05 15:17:01 rbnet kernel: pci_bus 0000:02: extended config space not accessible
Dec 05 15:17:01 rbnet kernel: pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
Dec 05 15:17:01 rbnet kernel: pci 0000:02:04.0: reg 0x10: [io  0x46000000-0x460000ff]
Dec 05 15:17:01 rbnet kernel: pci 0000:02:04.0: supports D1 D2
Dec 05 15:17:01 rbnet kernel: PCI: bus2: Fast back to back transfers disabled
Dec 05 15:17:01 rbnet kernel: pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
Dec 05 15:17:01 rbnet kernel: pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 02
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: BAR 7: no space for [io  size 0x1000]
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: BAR 7: failed to assign [io  size 0x1000]
Dec 05 15:17:01 rbnet kernel: pci 0000:02:04.0: BAR 0: no space for [io  size 0x0100]
Dec 05 15:17:01 rbnet kernel: pci 0000:02:04.0: BAR 0: failed to assign [io  size 0x0100]
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: PCI bridge to [bus 02]
Dec 05 15:17:01 rbnet kernel: pci 0000:00:00.0: PCI bridge to [bus 01-02]
Dec 05 15:17:01 rbnet kernel: pcieport 0000:00:00.0: PME: Signaling with IRQ 62
Dec 05 15:17:01 rbnet kernel: snd_virtuoso 0000:02:04.0: invalid PCI I/O range

and lspci:

00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 10) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 62
	Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
	I/O behind bridge: 00000000-00000fff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
		RootCap: CRSVisible+
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
	Capabilities: [240 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=1us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Kernel driver in use: pcieport

01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 62
	Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Power Budgeting <?>

02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
	Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 62
	Region 0: I/O ports at <ignored> [disabled]
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel modules: snd_virtuoso

Any ideas on how to go about it?
Also do you have any issues with power order between Pi and PCIe boards? I had a r8168 board (that I sadly bricked) that wouldn't be recognized if I didn't pulse it's PCIERST pin low a bit after powering the Pi. Connecting that pin from Pi to the ethernet board didn't help.
Is this common with Pi4 and PCIe devices, or do they usually work without powerup sequence being important?
I want to make a setup that I can just power on at once without worries about which gets powered first.

edit: I'm on Pi4 with the latest beta firmware. was the same with the latest stable one.
also I don't have the ethernet PHY chip and neither the VL805 one, they got toasted in a mishap. Also the wifi/bt is missing, so could this help me free some resources? I tried to software disable/remove the ethernet PHY but didn't manage, and I also have a crappy timeout at boot.

2nd edit: I think the issue for the Pi4 is the fact that only PCIe_nRST (pin 109 on CM4) is present at the VL805 pins. PCIe_CLK_nREQ (pin 102 on CM4) is missing from the VL805 connections. I suppose they rather used it as a GPIO maybe? I think this pin is what causes issues with finding different PCIe boards on the bus, for the RPi4. I think on PCIe_CLK_nREQ devices on the bus keep the BCM2711 bus active.
CM4 breaks out both of these pins. would be interesting to know if and what pin it's broken out to on Pi4.

@TheMindVirus
Copy link

From what I can see the drivers you mentioned are loading, perhaps apart from shpchp.


A Quick Look-Up in the source yielded this:
https://github.com/raspberrypi/linux/blob/a1d0f808d4a0b7f7053095cd4ab97a4276bed9ff/drivers/pci/hotplug/Kconfig#L106
https://cateee.net/lkddb/web-lkddb/HOTPLUG_PCI_SHPC.html
https://github.com/raspberrypi/linux/blob/e9e302fd9aede393bcf7f264a53b8d98e1cd8df2/arch/arm64/configs/bcm2711_defconfig

You may have looked for a CONFIG_HOTPLUG_PCI_SHPC but I'm not sure the BCM2711-related board support has...support... for the PCIe Hot-plug Motherboard Feature in Firmware that the shpchp driver is meant to be in control of.


What has hot-plug been like on any of the other devices you've tested, does it still work after the card is disconnected and reconnected? (Perhaps with a power riser board in between to avoid touching a device which is still powered on)
I haven't had the hardware to test with but previously PCIe hot-plug has been a feature delegated to enterprise motherboards.


If the Pi is missing pins itself then maybe a downstream device (such as your PCIe bridge) might be able to handle the re-clocking part of the hot-plug event independently of the Pi, but the OS will not know that this has happened and will manually have to check for new devices. Again, speaking from past experience of not having such hardware but it might be theoretically possible.


I would be interested to see if the Smart-NIC / DPU-Market could use an extra PCIe Female Connector to sit between a test device (such as your Xonar DX Audio Card) and the Pi to not only diagnose things like this but also do re-clocking, link training and essential low-level communication for legacy devices that the OS itself is unable to support electronically from where it is.


Copy link

ghost commented Dec 9, 2021

I made a mistake where I left the pullup resistor when I connected PCIe_nRST from Pi to the PCIe breakout pcb. Removing the resistor seems to make the Pi correctly control the PCIe connected device.
So until I have a board that doesn't work without the PCIe_CLK_nREQ connection then I can't say for sure it's absolutely necessary.

I got curious and looked at the lspci details on my x86_64 machine (for Xonar DX), and there isn't much difference. There's no driver shown for the PLX chip, just like on the Pi4. So I'm not sure if that shpchp driver is really required. On the Pi there's no difference in reported info between no PCI hotplug driver, just PCI hotplug driver or both PCI/PCIe hotplug drivers. I also enabled more verbosity for PCI and here's how it looks at this moment:

	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 62
	Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: fff00000-000fffff
	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Power Budgeting <?>

02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
	Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 62
	Region 0: I/O ports at <ignored> [disabled]
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel modules: snd_virtuoso
dmesg:
[    0.000000] PCIe ASPM is forcibly enabled
[    1.353502] PCI: CLS 0 bytes, default 64
[    1.377120] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    1.377682] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.377740] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.377825] brcm-pcie fd500000.pcie: Parsing ranges property...
[    1.377899] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[    1.377966] brcm-pcie fd500000.pcie: Parsing dma-ranges property...
[    1.378048] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x007fffffff -> 0x0400000000
[    1.407936] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
[    1.408176] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    1.408219] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.408260] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[    1.408308] pci_bus 0000:00: scanning bus
[    1.408406] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    1.408732] pci 0000:00:00.0: PME# supported from D0 D3hot
[    1.408776] pci 0000:00:00.0: PME# disabled
[    1.412362] pci_bus 0000:00: fixups for bus
[    1.412401] PCI: bus0: Fast back to back transfers disabled
[    1.412448] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
[    1.412467] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.412541] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
[    1.412836] pci_bus 0000:01: scanning bus
[    1.412931] pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
[    1.416701] pci_bus 0000:01: fixups for bus
[    1.416748] PCI: bus1: Fast back to back transfers disabled
[    1.416796] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
[    1.416834] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.416907] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[    1.417048] pci_bus 0000:02: extended config space not accessible
[    1.417182] pci_bus 0000:02: scanning bus
[    1.417637] pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
[    1.417731] pci 0000:02:04.0: reg 0x10: [io  0x0a00-0x0aff]
[    1.418055] pci 0000:02:04.0: supports D1 D2
[    1.421110] pci_bus 0000:02: fixups for bus
[    1.421148] PCI: bus2: Fast back to back transfers disabled
[    1.421187] pci_bus 0000:02: bus scan returning with max=02
[    1.421209] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[    1.421259] pci_bus 0000:01: bus scan returning with max=02
[    1.421279] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 02
[    1.421326] pci_bus 0000:00: bus scan returning with max=02
[    1.421362] pci 0000:01:00.0: BAR 7: no space for [io  size 0x1000]
[    1.421402] pci 0000:01:00.0: BAR 7: failed to assign [io  size 0x1000]
[    1.421444] pci 0000:02:04.0: BAR 0: no space for [io  size 0x0100]
[    1.421482] pci 0000:02:04.0: BAR 0: failed to assign [io  size 0x0100]
[    1.421521] pci 0000:01:00.0: PCI bridge to [bus 02]
[    1.421600] pci 0000:00:00.0: PCI bridge to [bus 01-02]
[    1.421690] pci 0000:00:00.0: Max Payload Size set to  512/ 512 (was  128), Max Read Rq  512
[    1.421788] pci 0000:01:00.0: Max Payload Size set to  128/ 128 (was  128), Max Read Rq  128
[    1.422268] pcieport 0000:00:00.0: assign IRQ: got 62
[    1.422339] pcieport 0000:00:00.0: enabling bus mastering
[    1.422554] pcieport 0000:00:00.0: PME: Signaling with IRQ 62
[    1.422743] pcieport 0000:00:00.0: saving config space at offset 0x0 (reading 0x271114e4)
[    1.422766] pcieport 0000:00:00.0: saving config space at offset 0x4 (reading 0x100144)
[    1.422788] pcieport 0000:00:00.0: saving config space at offset 0x8 (reading 0x6040010)
[    1.422894] pcieport 0000:00:00.0: saving config space at offset 0xc (reading 0x10010)
[    1.422917] pcieport 0000:00:00.0: saving config space at offset 0x10 (reading 0x0)
[    1.422938] pcieport 0000:00:00.0: saving config space at offset 0x14 (reading 0x0)
[    1.422960] pcieport 0000:00:00.0: saving config space at offset 0x18 (reading 0x20100)
[    1.422981] pcieport 0000:00:00.0: saving config space at offset 0x1c (reading 0x20000000)
[    1.423003] pcieport 0000:00:00.0: saving config space at offset 0x20 (reading 0xfff0)
[    1.423024] pcieport 0000:00:00.0: saving config space at offset 0x24 (reading 0x1fff1)
[    1.423044] pcieport 0000:00:00.0: saving config space at offset 0x28 (reading 0x0)
[    1.423065] pcieport 0000:00:00.0: saving config space at offset 0x2c (reading 0x0)
[    1.423086] pcieport 0000:00:00.0: saving config space at offset 0x30 (reading 0x0)
[    1.423108] pcieport 0000:00:00.0: saving config space at offset 0x34 (reading 0x48)
[    1.423128] pcieport 0000:00:00.0: saving config space at offset 0x38 (reading 0x0)
[    1.423149] pcieport 0000:00:00.0: saving config space at offset 0x3c (reading 0x3013e)
[    1.423645] pcieport 0000:01:00.0: assign IRQ: got 62
[    4.354949] pci 0000:01:00.0: enabling bus mastering
[    4.354971] snd_virtuoso 0000:02:04.0: invalid PCI I/O range

Tried different cmdline.txt parameters for pci and didn't make any difference.

And here's lspci from my PC for the DX board. As you can see there isn't a listed kernel driver.

        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 37
        IOMMU group: 14
        Bus: primary=23, secondary=24, subordinate=24, sec-latency=32
        I/O behind bridge: 0000e000-0000efff [size=4K]
        Memory behind bridge: fff00000-000fffff [disabled]
        Prefetchable memory behind bridge: fff00000-000fffff [disabled]
        Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 26.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr+ FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100 v1] Power Budgeting <?>

24:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
        Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 32 (500ns min, 6000ns max)
        Interrupt: pin A routed to IRQ 37
        IOMMU group: 14
        Region 0: I/O ports at e000 [size=256]
        Capabilities: [c0] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: snd_virtuoso
        Kernel modules: snd_virtuoso

So, this is not a problem that can be fixed by increasing the BAR? I have 2G of RAM on my Pi4.

@TheMindVirus
Copy link

TheMindVirus commented Dec 9, 2021

For your PC it says: I/O behind bridge: 0000e000-0000efff [size=4K]
and for the Pi it says: I/O behind bridge: 0000f000-00000fff

That seems fine but then it says:
For the PC: Region 0: I/O ports at e000 [size=256]
For the Pi: Region 0: I/O ports at <ignored> [disabled] (shouldn't that read Region 0: I/O ports at f000 [size=256]?)

Combining these two pieces of random documentation together:
https://support.xilinx.com/s/question/0D52E00006iHlNoSAK/lspci-reports-bar-0-disabled?language=en_US
https://wiki.osdev.org/PCI

The command seems to be: setpci -s 02:04.0 COMMAND=0x05 which sets I/O+ and BusMaster+ like you have on your PC.
This also sets SERR- where it is SERR+ on the Pi for some reason. That would make sense why it isn't allowed to send any data.

I'd hold off from entering the command immediately until you have the right device ID entered and you're sure the command is correct.
Never had to do this before though, usually this sort of thing is left correctly configured by default rather than having a million off-switches.

Copy link

ghost commented Dec 9, 2021

setpci -s 02:04.0 COMMAND=0x05

02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
	Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (500ns min, 6000ns max)
	Interrupt: pin A routed to IRQ 61
	Region 0: I/O ports at 0000
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel modules: snd_virtuoso

Sets it at 0000 instead of f000. And the PLX chip still shows
I/O behind bridge: 0000f000-00000fff [disabled]

Copy link

ghost commented Dec 10, 2021

Also tried it on aarch64 and behaves the same.
dmesg:

[    1.669941] PCI: CLS 0 bytes, default 64
[    2.345316] pciehp: pcie_port_service_register = 0
[    2.345449] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    3.468976] ehci-pci: EHCI PCI platform driver
[    3.489589] ohci-pci: OHCI PCI platform driver
[    8.468012] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    8.475207] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    8.484761] brcm-pcie fd500000.pcie: Parsing ranges property...
[    8.484787] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[    8.493208] brcm-pcie fd500000.pcie: Parsing dma-ranges property...
[    8.493236] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00bfffffff -> 0x0000000000
[    8.532089] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
[    8.538764] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    8.549306] pci_bus 0000:00: root bus resource [bus 00-ff]
[    8.566028] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[    8.577111] pci_bus 0000:00: scanning bus
[    8.577157] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    8.583358] pci 0000:00:00.0: PME# supported from D0 D3hot
[    8.588944] pci 0000:00:00.0: PME# disabled
[    8.594378] pci_bus 0000:00: fixups for bus
[    8.594394] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
[    8.594495] pci_bus 0000:01: scanning bus
[    8.594540] pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
[    8.601244] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    8.612296] pci_bus 0000:01: fixups for bus
[    8.612309] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
[    8.612315] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    8.620484] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[    8.620567] pci_bus 0000:02: extended config space not accessible
[    8.626863] pci_bus 0000:02: busn_res: can not insert [bus 02-01] under [bus 01] (conflicts with (null) [bus 01])
[    8.637317] pci_bus 0000:02: scanning bus
[    8.637517] pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
[    8.643693] pci 0000:02:04.0: reg 0x10: [io  0xffffff00-0xffffffff]
[    8.650197] pci 0000:02:04.0: supports D1 D2
[    8.656606] pci_bus 0000:02: fixups for bus
[    8.656649] pci_bus 0000:02: bus scan returning with max=02
[    8.656661] pci_bus 0000:02: busn_res: [bus 02-01] end is updated to 02
[    8.663454] pci_bus 0000:02: busn_res: can not insert [bus 02] under [bus 01] (conflicts with (null) [bus 01])
[    8.680096] pci 0000:01:00.0: devices behind bridge are unusable because [bus 02] cannot be assigned for them
[    8.689506] pci_bus 0000:01: bus scan returning with max=02
[    8.703127] pci 0000:00:00.0: bridge has subordinate 01 but max busn 02
[    8.703139] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
[    8.703147] pci_bus 0000:00: bus scan returning with max=01
[    8.703167] pci 0000:01:00.0: BAR 13: no space for [io  size 0x1000]
[    8.719759] pci 0000:01:00.0: BAR 13: failed to assign [io  size 0x1000]
[    8.719767] pci 0000:02:04.0: BAR 0: no space for [io  size 0x0100]
[    8.719771] pci 0000:02:04.0: BAR 0: failed to assign [io  size 0x0100]
[    8.719774] pci 0000:01:00.0: PCI bridge to [bus 02]
[    8.730394] pci 0000:00:00.0: PCI bridge to [bus 01]
[    8.780561] pcieport 0000:00:00.0: assign IRQ: got 61
[    8.780596] pcieport 0000:00:00.0: enabling bus mastering
[    8.780738] pcieport 0000:00:00.0: PME: Signaling with IRQ 61
[    8.793529] pcieport 0000:00:00.0: AER: enabled with IRQ 61
[    8.800190] pcieport 0000:00:00.0: saving config space at offset 0x0 (reading 0x271114e4)
[    8.805636] pcieport 0000:00:00.0: saving config space at offset 0x4 (reading 0x100004)
[    8.805640] pcieport 0000:00:00.0: saving config space at offset 0x8 (reading 0x6040010)
[    8.805644] pcieport 0000:00:00.0: saving config space at offset 0xc (reading 0x10000)
[    8.805648] pcieport 0000:00:00.0: saving config space at offset 0x10 (reading 0x0)
[    8.805652] pcieport 0000:00:00.0: saving config space at offset 0x14 (reading 0x0)
[    8.805655] pcieport 0000:00:00.0: saving config space at offset 0x18 (reading 0x10100)
[    8.805659] pcieport 0000:00:00.0: saving config space at offset 0x1c (reading 0x20000000)
[    8.805663] pcieport 0000:00:00.0: saving config space at offset 0x20 (reading 0xfff0)
[    8.805669] pcieport 0000:00:00.0: saving config space at offset 0x24 (reading 0x1fff1)
[    8.805673] pcieport 0000:00:00.0: saving config space at offset 0x28 (reading 0x0)
[    8.805677] pcieport 0000:00:00.0: saving config space at offset 0x2c (reading 0x0)
[    8.805681] pcieport 0000:00:00.0: saving config space at offset 0x30 (reading 0x0)
[    8.805684] pcieport 0000:00:00.0: saving config space at offset 0x34 (reading 0x48)
[    8.805688] pcieport 0000:00:00.0: saving config space at offset 0x38 (reading 0x0)
[    8.805692] pcieport 0000:00:00.0: saving config space at offset 0x3c (reading 0x2013d)
[    8.806243] pcieport 0000:01:00.0: assign IRQ: got 61
[    9.402928] pci 0000:01:00.0: enabling bus mastering
[    9.402941] snd_virtuoso 0000:02:04.0: invalid PCI I/O range

After I issued the setpci command I rescaned the bus and this is what dmesg shows after:


[  144.734267] pci_bus 0000:00: scanning bus
[  144.735147] pcieport 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
[  144.735160] pci_bus 0000:01: scanning bus
[  144.735928] pci 0000:01:00.0: scanning [bus 02-02] behind bridge, pass 0
[  144.735937] pci_bus 0000:02: scanning bus
[  144.736776] pci_bus 0000:02: bus scan returning with max=02
[  144.736784] pci 0000:01:00.0: devices behind bridge are unusable because [bus 02] cannot be assigned for them
[  144.746870] pci 0000:01:00.0: scanning [bus 02-02] behind bridge, pass 1
[  144.746883] pci_bus 0000:01: bus scan returning with max=02
[  144.746887] pcieport 0000:00:00.0: bridge has subordinate 01 but max busn 02
[  144.754049] pcieport 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
[  144.754056] pci_bus 0000:00: bus scan returning with max=01

lspci info doesn't change between 32bit and 64bit versions

01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 61
	Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: fff00000-000fffff [disabled]
	Prefetchable memory behind bridge: fff00000-000fffff [disabled]
	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr+ FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Power Budgeting <?>

02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
	Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (500ns min, 6000ns max)
	Interrupt: pin A routed to IRQ 61
	Region 0: I/O ports at 0000
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel modules: snd_virtuoso

later edit:
Sadly I don't think this will be possible. this Raspberry Pi Engineer/Moderator says (https://forums.raspberrypi.com/viewtopic.php?p=1746705#p1746705) the bcm2711 datasheet mentions it doesn't support I/O space for PCIe. Only memory space. And that's correctly mapped. I see some I/O bar fails for a r8169 ethernet card, but that is working fine even so. But I can't see the Xonar in /proc/asound/cards

@TheRemote
Copy link

TheRemote commented Mar 2, 2022

I'd never heard of this before until it came up at https://jamesachambers.com/pcie-1x-nvme-on-raspberry-pi-compute-module-4-guide/#comment-12952. John tried a Xawai adapter that I had previously tested with and had no trouble with. He is encountering the same error as RoboTakoh. Since I've used this adapter successfully before it can't be impossible like may be the case with the firewire card.

I can't explain why I never encountered it with the same adapter yet. Hopefully John will test some suggestions I had and some suggestions here. I may try doing it again with the latest Pi OS (I used 64 bit previously, one test I suggested was he try the 64 bit flavor to see if it makes any difference) to see if I can reproduce it on my end. I also linked him here to try some of the tweaks in the gist.

Most likely it just needs a firmware update as it sounds like this should be resolved and would explain why I never encountered it (I was late to the CM4 party).

Very strange issue, great work on this Jeff!

@5erv3
Copy link

5erv3 commented Oct 14, 2024

Thanks for the script @geerlingguy !
Do you know how to modify this on a PI5?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment