Skip to content

Instantly share code, notes, and snippets.

@geerlingguy
Last active November 10, 2024 20:47
Show Gist options
  • Save geerlingguy/9d78ea34cab8e18d71ee5954417429df to your computer and use it in GitHub Desktop.
Save geerlingguy/9d78ea34cab8e18d71ee5954417429df to your computer and use it in GitHub Desktop.
Increase the BAR memory address space for PCIe devices on the Raspberry Pi Compute Module 4
#!/bin/bash
# The default BAR address space available on the CM4 may be too small to allow
# some devices to initialize correctly. To avoid 'failed to assign memory'
# errors on boot, you can increase the range of the PCIe bus in the Raspberry
# Pi's Device Tree (a .dtb file specific to each Pi model).
#
# You should probably read up on Device Trees if you don't know what they are:
# https://www.raspberrypi.org/documentation/configuration/device-tree.md
#
# NOTE: The default BAR allocation was increased to 1 GB in this commit:
# https://github.com/raspberrypi/linux/commit/54db4b2fa4d17251c2f6e639f849b27c3b553939
# Download the trial firmware from this post:
# https://www.raspberrypi.org/forums/viewtopic.php?p=1761834#p1761834
# and replace the corresponding files in the boot volume.
# Back up current Device Tree for the CM4.
sudo cp /boot/bcm2711-rpi-cm4.dtb /boot/bcm2711-rpi-cm4.dtb.bak
# Decompile the current Device Tree to a dts (source) file.
dtc -I dtb -O dts /boot/bcm2711-rpi-cm4.dtb -o ~/test.dts
# Edit the file and change the PCIe bus range as mentioned in:
# https://www.raspberrypi.org/forums/viewtopic.php?p=1746665#p1746665
nano ~/test.dts
# Replace the line that allocates 1 GB of RAM in the pci section:
# ranges = <0x02000000 0x0 0xc0000000 0x6 0x00000000 0x0 0x40000000>;
#
# with the following line, which provides up to 8 GB of RAM:
# ranges = <0x02000000 0x0 0x00000000 0x6 0x00000000 0x2 0x00000000>;
# Also replace the scb ranges to:
# ranges = <0x0 0x7c000000 0x0 0xfc000000 0x0 0x03800000>,
# <0x0 0x40000000 0x0 0xff800000 0x0 0x00800000>,
# <0x6 0x00000000 0x6 0x00000000 0x2 0x00000000>,
# <0x0 0x00000000 0x0 0x00000000 0x0 0xfc000000>;
# Recompile the Device Tree from the dts (source) file.
dtc -I dts -O dtb ~/test.dts -o ~/test.dtb
# Copy the new Device Tree into place.
sudo mv ~/test.dtb /boot/bcm2711-rpi-cm4.dtb
# Reboot.
sudo reboot
@TheMindVirus
Copy link

From what I can see the drivers you mentioned are loading, perhaps apart from shpchp.


A Quick Look-Up in the source yielded this:
https://github.com/raspberrypi/linux/blob/a1d0f808d4a0b7f7053095cd4ab97a4276bed9ff/drivers/pci/hotplug/Kconfig#L106
https://cateee.net/lkddb/web-lkddb/HOTPLUG_PCI_SHPC.html
https://github.com/raspberrypi/linux/blob/e9e302fd9aede393bcf7f264a53b8d98e1cd8df2/arch/arm64/configs/bcm2711_defconfig

You may have looked for a CONFIG_HOTPLUG_PCI_SHPC but I'm not sure the BCM2711-related board support has...support... for the PCIe Hot-plug Motherboard Feature in Firmware that the shpchp driver is meant to be in control of.


What has hot-plug been like on any of the other devices you've tested, does it still work after the card is disconnected and reconnected? (Perhaps with a power riser board in between to avoid touching a device which is still powered on)
I haven't had the hardware to test with but previously PCIe hot-plug has been a feature delegated to enterprise motherboards.


If the Pi is missing pins itself then maybe a downstream device (such as your PCIe bridge) might be able to handle the re-clocking part of the hot-plug event independently of the Pi, but the OS will not know that this has happened and will manually have to check for new devices. Again, speaking from past experience of not having such hardware but it might be theoretically possible.


I would be interested to see if the Smart-NIC / DPU-Market could use an extra PCIe Female Connector to sit between a test device (such as your Xonar DX Audio Card) and the Pi to not only diagnose things like this but also do re-clocking, link training and essential low-level communication for legacy devices that the OS itself is unable to support electronically from where it is.


Copy link

ghost commented Dec 9, 2021

I made a mistake where I left the pullup resistor when I connected PCIe_nRST from Pi to the PCIe breakout pcb. Removing the resistor seems to make the Pi correctly control the PCIe connected device.
So until I have a board that doesn't work without the PCIe_CLK_nREQ connection then I can't say for sure it's absolutely necessary.

I got curious and looked at the lspci details on my x86_64 machine (for Xonar DX), and there isn't much difference. There's no driver shown for the PLX chip, just like on the Pi4. So I'm not sure if that shpchp driver is really required. On the Pi there's no difference in reported info between no PCI hotplug driver, just PCI hotplug driver or both PCI/PCIe hotplug drivers. I also enabled more verbosity for PCI and here's how it looks at this moment:

	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 62
	Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: fff00000-000fffff
	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Power Budgeting <?>

02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
	Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 62
	Region 0: I/O ports at <ignored> [disabled]
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel modules: snd_virtuoso
dmesg:
[    0.000000] PCIe ASPM is forcibly enabled
[    1.353502] PCI: CLS 0 bytes, default 64
[    1.377120] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    1.377682] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.377740] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.377825] brcm-pcie fd500000.pcie: Parsing ranges property...
[    1.377899] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[    1.377966] brcm-pcie fd500000.pcie: Parsing dma-ranges property...
[    1.378048] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x007fffffff -> 0x0400000000
[    1.407936] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
[    1.408176] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    1.408219] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.408260] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[    1.408308] pci_bus 0000:00: scanning bus
[    1.408406] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    1.408732] pci 0000:00:00.0: PME# supported from D0 D3hot
[    1.408776] pci 0000:00:00.0: PME# disabled
[    1.412362] pci_bus 0000:00: fixups for bus
[    1.412401] PCI: bus0: Fast back to back transfers disabled
[    1.412448] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
[    1.412467] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.412541] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
[    1.412836] pci_bus 0000:01: scanning bus
[    1.412931] pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
[    1.416701] pci_bus 0000:01: fixups for bus
[    1.416748] PCI: bus1: Fast back to back transfers disabled
[    1.416796] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
[    1.416834] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.416907] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[    1.417048] pci_bus 0000:02: extended config space not accessible
[    1.417182] pci_bus 0000:02: scanning bus
[    1.417637] pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
[    1.417731] pci 0000:02:04.0: reg 0x10: [io  0x0a00-0x0aff]
[    1.418055] pci 0000:02:04.0: supports D1 D2
[    1.421110] pci_bus 0000:02: fixups for bus
[    1.421148] PCI: bus2: Fast back to back transfers disabled
[    1.421187] pci_bus 0000:02: bus scan returning with max=02
[    1.421209] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[    1.421259] pci_bus 0000:01: bus scan returning with max=02
[    1.421279] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 02
[    1.421326] pci_bus 0000:00: bus scan returning with max=02
[    1.421362] pci 0000:01:00.0: BAR 7: no space for [io  size 0x1000]
[    1.421402] pci 0000:01:00.0: BAR 7: failed to assign [io  size 0x1000]
[    1.421444] pci 0000:02:04.0: BAR 0: no space for [io  size 0x0100]
[    1.421482] pci 0000:02:04.0: BAR 0: failed to assign [io  size 0x0100]
[    1.421521] pci 0000:01:00.0: PCI bridge to [bus 02]
[    1.421600] pci 0000:00:00.0: PCI bridge to [bus 01-02]
[    1.421690] pci 0000:00:00.0: Max Payload Size set to  512/ 512 (was  128), Max Read Rq  512
[    1.421788] pci 0000:01:00.0: Max Payload Size set to  128/ 128 (was  128), Max Read Rq  128
[    1.422268] pcieport 0000:00:00.0: assign IRQ: got 62
[    1.422339] pcieport 0000:00:00.0: enabling bus mastering
[    1.422554] pcieport 0000:00:00.0: PME: Signaling with IRQ 62
[    1.422743] pcieport 0000:00:00.0: saving config space at offset 0x0 (reading 0x271114e4)
[    1.422766] pcieport 0000:00:00.0: saving config space at offset 0x4 (reading 0x100144)
[    1.422788] pcieport 0000:00:00.0: saving config space at offset 0x8 (reading 0x6040010)
[    1.422894] pcieport 0000:00:00.0: saving config space at offset 0xc (reading 0x10010)
[    1.422917] pcieport 0000:00:00.0: saving config space at offset 0x10 (reading 0x0)
[    1.422938] pcieport 0000:00:00.0: saving config space at offset 0x14 (reading 0x0)
[    1.422960] pcieport 0000:00:00.0: saving config space at offset 0x18 (reading 0x20100)
[    1.422981] pcieport 0000:00:00.0: saving config space at offset 0x1c (reading 0x20000000)
[    1.423003] pcieport 0000:00:00.0: saving config space at offset 0x20 (reading 0xfff0)
[    1.423024] pcieport 0000:00:00.0: saving config space at offset 0x24 (reading 0x1fff1)
[    1.423044] pcieport 0000:00:00.0: saving config space at offset 0x28 (reading 0x0)
[    1.423065] pcieport 0000:00:00.0: saving config space at offset 0x2c (reading 0x0)
[    1.423086] pcieport 0000:00:00.0: saving config space at offset 0x30 (reading 0x0)
[    1.423108] pcieport 0000:00:00.0: saving config space at offset 0x34 (reading 0x48)
[    1.423128] pcieport 0000:00:00.0: saving config space at offset 0x38 (reading 0x0)
[    1.423149] pcieport 0000:00:00.0: saving config space at offset 0x3c (reading 0x3013e)
[    1.423645] pcieport 0000:01:00.0: assign IRQ: got 62
[    4.354949] pci 0000:01:00.0: enabling bus mastering
[    4.354971] snd_virtuoso 0000:02:04.0: invalid PCI I/O range

Tried different cmdline.txt parameters for pci and didn't make any difference.

And here's lspci from my PC for the DX board. As you can see there isn't a listed kernel driver.

        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 37
        IOMMU group: 14
        Bus: primary=23, secondary=24, subordinate=24, sec-latency=32
        I/O behind bridge: 0000e000-0000efff [size=4K]
        Memory behind bridge: fff00000-000fffff [disabled]
        Prefetchable memory behind bridge: fff00000-000fffff [disabled]
        Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 26.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr+ FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100 v1] Power Budgeting <?>

24:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
        Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 32 (500ns min, 6000ns max)
        Interrupt: pin A routed to IRQ 37
        IOMMU group: 14
        Region 0: I/O ports at e000 [size=256]
        Capabilities: [c0] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: snd_virtuoso
        Kernel modules: snd_virtuoso

So, this is not a problem that can be fixed by increasing the BAR? I have 2G of RAM on my Pi4.

@TheMindVirus
Copy link

TheMindVirus commented Dec 9, 2021

For your PC it says: I/O behind bridge: 0000e000-0000efff [size=4K]
and for the Pi it says: I/O behind bridge: 0000f000-00000fff

That seems fine but then it says:
For the PC: Region 0: I/O ports at e000 [size=256]
For the Pi: Region 0: I/O ports at <ignored> [disabled] (shouldn't that read Region 0: I/O ports at f000 [size=256]?)

Combining these two pieces of random documentation together:
https://support.xilinx.com/s/question/0D52E00006iHlNoSAK/lspci-reports-bar-0-disabled?language=en_US
https://wiki.osdev.org/PCI

The command seems to be: setpci -s 02:04.0 COMMAND=0x05 which sets I/O+ and BusMaster+ like you have on your PC.
This also sets SERR- where it is SERR+ on the Pi for some reason. That would make sense why it isn't allowed to send any data.

I'd hold off from entering the command immediately until you have the right device ID entered and you're sure the command is correct.
Never had to do this before though, usually this sort of thing is left correctly configured by default rather than having a million off-switches.

Copy link

ghost commented Dec 9, 2021

setpci -s 02:04.0 COMMAND=0x05

02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
	Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (500ns min, 6000ns max)
	Interrupt: pin A routed to IRQ 61
	Region 0: I/O ports at 0000
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel modules: snd_virtuoso

Sets it at 0000 instead of f000. And the PLX chip still shows
I/O behind bridge: 0000f000-00000fff [disabled]

Copy link

ghost commented Dec 10, 2021

Also tried it on aarch64 and behaves the same.
dmesg:

[    1.669941] PCI: CLS 0 bytes, default 64
[    2.345316] pciehp: pcie_port_service_register = 0
[    2.345449] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    3.468976] ehci-pci: EHCI PCI platform driver
[    3.489589] ohci-pci: OHCI PCI platform driver
[    8.468012] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    8.475207] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    8.484761] brcm-pcie fd500000.pcie: Parsing ranges property...
[    8.484787] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[    8.493208] brcm-pcie fd500000.pcie: Parsing dma-ranges property...
[    8.493236] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00bfffffff -> 0x0000000000
[    8.532089] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
[    8.538764] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    8.549306] pci_bus 0000:00: root bus resource [bus 00-ff]
[    8.566028] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[    8.577111] pci_bus 0000:00: scanning bus
[    8.577157] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    8.583358] pci 0000:00:00.0: PME# supported from D0 D3hot
[    8.588944] pci 0000:00:00.0: PME# disabled
[    8.594378] pci_bus 0000:00: fixups for bus
[    8.594394] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
[    8.594495] pci_bus 0000:01: scanning bus
[    8.594540] pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
[    8.601244] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    8.612296] pci_bus 0000:01: fixups for bus
[    8.612309] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
[    8.612315] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    8.620484] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[    8.620567] pci_bus 0000:02: extended config space not accessible
[    8.626863] pci_bus 0000:02: busn_res: can not insert [bus 02-01] under [bus 01] (conflicts with (null) [bus 01])
[    8.637317] pci_bus 0000:02: scanning bus
[    8.637517] pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
[    8.643693] pci 0000:02:04.0: reg 0x10: [io  0xffffff00-0xffffffff]
[    8.650197] pci 0000:02:04.0: supports D1 D2
[    8.656606] pci_bus 0000:02: fixups for bus
[    8.656649] pci_bus 0000:02: bus scan returning with max=02
[    8.656661] pci_bus 0000:02: busn_res: [bus 02-01] end is updated to 02
[    8.663454] pci_bus 0000:02: busn_res: can not insert [bus 02] under [bus 01] (conflicts with (null) [bus 01])
[    8.680096] pci 0000:01:00.0: devices behind bridge are unusable because [bus 02] cannot be assigned for them
[    8.689506] pci_bus 0000:01: bus scan returning with max=02
[    8.703127] pci 0000:00:00.0: bridge has subordinate 01 but max busn 02
[    8.703139] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
[    8.703147] pci_bus 0000:00: bus scan returning with max=01
[    8.703167] pci 0000:01:00.0: BAR 13: no space for [io  size 0x1000]
[    8.719759] pci 0000:01:00.0: BAR 13: failed to assign [io  size 0x1000]
[    8.719767] pci 0000:02:04.0: BAR 0: no space for [io  size 0x0100]
[    8.719771] pci 0000:02:04.0: BAR 0: failed to assign [io  size 0x0100]
[    8.719774] pci 0000:01:00.0: PCI bridge to [bus 02]
[    8.730394] pci 0000:00:00.0: PCI bridge to [bus 01]
[    8.780561] pcieport 0000:00:00.0: assign IRQ: got 61
[    8.780596] pcieport 0000:00:00.0: enabling bus mastering
[    8.780738] pcieport 0000:00:00.0: PME: Signaling with IRQ 61
[    8.793529] pcieport 0000:00:00.0: AER: enabled with IRQ 61
[    8.800190] pcieport 0000:00:00.0: saving config space at offset 0x0 (reading 0x271114e4)
[    8.805636] pcieport 0000:00:00.0: saving config space at offset 0x4 (reading 0x100004)
[    8.805640] pcieport 0000:00:00.0: saving config space at offset 0x8 (reading 0x6040010)
[    8.805644] pcieport 0000:00:00.0: saving config space at offset 0xc (reading 0x10000)
[    8.805648] pcieport 0000:00:00.0: saving config space at offset 0x10 (reading 0x0)
[    8.805652] pcieport 0000:00:00.0: saving config space at offset 0x14 (reading 0x0)
[    8.805655] pcieport 0000:00:00.0: saving config space at offset 0x18 (reading 0x10100)
[    8.805659] pcieport 0000:00:00.0: saving config space at offset 0x1c (reading 0x20000000)
[    8.805663] pcieport 0000:00:00.0: saving config space at offset 0x20 (reading 0xfff0)
[    8.805669] pcieport 0000:00:00.0: saving config space at offset 0x24 (reading 0x1fff1)
[    8.805673] pcieport 0000:00:00.0: saving config space at offset 0x28 (reading 0x0)
[    8.805677] pcieport 0000:00:00.0: saving config space at offset 0x2c (reading 0x0)
[    8.805681] pcieport 0000:00:00.0: saving config space at offset 0x30 (reading 0x0)
[    8.805684] pcieport 0000:00:00.0: saving config space at offset 0x34 (reading 0x48)
[    8.805688] pcieport 0000:00:00.0: saving config space at offset 0x38 (reading 0x0)
[    8.805692] pcieport 0000:00:00.0: saving config space at offset 0x3c (reading 0x2013d)
[    8.806243] pcieport 0000:01:00.0: assign IRQ: got 61
[    9.402928] pci 0000:01:00.0: enabling bus mastering
[    9.402941] snd_virtuoso 0000:02:04.0: invalid PCI I/O range

After I issued the setpci command I rescaned the bus and this is what dmesg shows after:


[  144.734267] pci_bus 0000:00: scanning bus
[  144.735147] pcieport 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
[  144.735160] pci_bus 0000:01: scanning bus
[  144.735928] pci 0000:01:00.0: scanning [bus 02-02] behind bridge, pass 0
[  144.735937] pci_bus 0000:02: scanning bus
[  144.736776] pci_bus 0000:02: bus scan returning with max=02
[  144.736784] pci 0000:01:00.0: devices behind bridge are unusable because [bus 02] cannot be assigned for them
[  144.746870] pci 0000:01:00.0: scanning [bus 02-02] behind bridge, pass 1
[  144.746883] pci_bus 0000:01: bus scan returning with max=02
[  144.746887] pcieport 0000:00:00.0: bridge has subordinate 01 but max busn 02
[  144.754049] pcieport 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
[  144.754056] pci_bus 0000:00: bus scan returning with max=01

lspci info doesn't change between 32bit and 64bit versions

01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 61
	Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: fff00000-000fffff [disabled]
	Prefetchable memory behind bridge: fff00000-000fffff [disabled]
	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr+ FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Power Budgeting <?>

02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
	Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (500ns min, 6000ns max)
	Interrupt: pin A routed to IRQ 61
	Region 0: I/O ports at 0000
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel modules: snd_virtuoso

later edit:
Sadly I don't think this will be possible. this Raspberry Pi Engineer/Moderator says (https://forums.raspberrypi.com/viewtopic.php?p=1746705#p1746705) the bcm2711 datasheet mentions it doesn't support I/O space for PCIe. Only memory space. And that's correctly mapped. I see some I/O bar fails for a r8169 ethernet card, but that is working fine even so. But I can't see the Xonar in /proc/asound/cards

@TheRemote
Copy link

TheRemote commented Mar 2, 2022

I'd never heard of this before until it came up at https://jamesachambers.com/pcie-1x-nvme-on-raspberry-pi-compute-module-4-guide/#comment-12952. John tried a Xawai adapter that I had previously tested with and had no trouble with. He is encountering the same error as RoboTakoh. Since I've used this adapter successfully before it can't be impossible like may be the case with the firewire card.

I can't explain why I never encountered it with the same adapter yet. Hopefully John will test some suggestions I had and some suggestions here. I may try doing it again with the latest Pi OS (I used 64 bit previously, one test I suggested was he try the 64 bit flavor to see if it makes any difference) to see if I can reproduce it on my end. I also linked him here to try some of the tweaks in the gist.

Most likely it just needs a firmware update as it sounds like this should be resolved and would explain why I never encountered it (I was late to the CM4 party).

Very strange issue, great work on this Jeff!

@5erv3
Copy link

5erv3 commented Oct 14, 2024

Thanks for the script @geerlingguy !
Do you know how to modify this on a PI5?

@geerlingguy
Copy link
Author

This may be useful on the Pi 5 as well (to your question @5erv3), since some larger graphics cards are setting down to 256 MB of BAR space as they have 12+ GB of VRAM: geerlingguy/raspberry-pi-pcie-devices#680

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment