Skip to content

Instantly share code, notes, and snippets.

@geerlingguy
Last active May 7, 2024 15:31
Show Gist options
  • Save geerlingguy/9d78ea34cab8e18d71ee5954417429df to your computer and use it in GitHub Desktop.
Save geerlingguy/9d78ea34cab8e18d71ee5954417429df to your computer and use it in GitHub Desktop.
Increase the BAR memory address space for PCIe devices on the Raspberry Pi Compute Module 4
#!/bin/bash
# The default BAR address space available on the CM4 may be too small to allow
# some devices to initialize correctly. To avoid 'failed to assign memory'
# errors on boot, you can increase the range of the PCIe bus in the Raspberry
# Pi's Device Tree (a .dtb file specific to each Pi model).
#
# You should probably read up on Device Trees if you don't know what they are:
# https://www.raspberrypi.org/documentation/configuration/device-tree.md
#
# NOTE: The default BAR allocation was increased to 1 GB in this commit:
# https://github.com/raspberrypi/linux/commit/54db4b2fa4d17251c2f6e639f849b27c3b553939
# Download the trial firmware from this post:
# https://www.raspberrypi.org/forums/viewtopic.php?p=1761834#p1761834
# and replace the corresponding files in the boot volume.
# Back up current Device Tree for the CM4.
sudo cp /boot/bcm2711-rpi-cm4.dtb /boot/bcm2711-rpi-cm4.dtb.bak
# Decompile the current Device Tree to a dts (source) file.
dtc -I dtb -O dts /boot/bcm2711-rpi-cm4.dtb -o ~/test.dts
# Edit the file and change the PCIe bus range as mentioned in:
# https://www.raspberrypi.org/forums/viewtopic.php?p=1746665#p1746665
nano ~/test.dts
# Replace the line that allocates 1 GB of RAM in the pci section:
# ranges = <0x02000000 0x0 0xc0000000 0x6 0x00000000 0x0 0x40000000>;
#
# with the following line, which provides up to 8 GB of RAM:
# ranges = <0x02000000 0x0 0x00000000 0x6 0x00000000 0x2 0x00000000>;
# Also replace the scb ranges to:
# ranges = <0x0 0x7c000000 0x0 0xfc000000 0x0 0x03800000>,
# <0x0 0x40000000 0x0 0xff800000 0x0 0x00800000>,
# <0x6 0x00000000 0x6 0x00000000 0x2 0x00000000>,
# <0x0 0x00000000 0x0 0x00000000 0x0 0xfc000000>;
# Recompile the Device Tree from the dts (source) file.
dtc -I dts -O dtb ~/test.dts -o ~/test.dtb
# Copy the new Device Tree into place.
sudo mv ~/test.dtb /boot/bcm2711-rpi-cm4.dtb
# Reboot.
sudo reboot
Copy link

ghost commented Dec 10, 2021

Also tried it on aarch64 and behaves the same.
dmesg:

[    1.669941] PCI: CLS 0 bytes, default 64
[    2.345316] pciehp: pcie_port_service_register = 0
[    2.345449] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    3.468976] ehci-pci: EHCI PCI platform driver
[    3.489589] ohci-pci: OHCI PCI platform driver
[    8.468012] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    8.475207] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    8.484761] brcm-pcie fd500000.pcie: Parsing ranges property...
[    8.484787] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[    8.493208] brcm-pcie fd500000.pcie: Parsing dma-ranges property...
[    8.493236] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00bfffffff -> 0x0000000000
[    8.532089] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
[    8.538764] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    8.549306] pci_bus 0000:00: root bus resource [bus 00-ff]
[    8.566028] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[    8.577111] pci_bus 0000:00: scanning bus
[    8.577157] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    8.583358] pci 0000:00:00.0: PME# supported from D0 D3hot
[    8.588944] pci 0000:00:00.0: PME# disabled
[    8.594378] pci_bus 0000:00: fixups for bus
[    8.594394] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
[    8.594495] pci_bus 0000:01: scanning bus
[    8.594540] pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
[    8.601244] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    8.612296] pci_bus 0000:01: fixups for bus
[    8.612309] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
[    8.612315] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    8.620484] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[    8.620567] pci_bus 0000:02: extended config space not accessible
[    8.626863] pci_bus 0000:02: busn_res: can not insert [bus 02-01] under [bus 01] (conflicts with (null) [bus 01])
[    8.637317] pci_bus 0000:02: scanning bus
[    8.637517] pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
[    8.643693] pci 0000:02:04.0: reg 0x10: [io  0xffffff00-0xffffffff]
[    8.650197] pci 0000:02:04.0: supports D1 D2
[    8.656606] pci_bus 0000:02: fixups for bus
[    8.656649] pci_bus 0000:02: bus scan returning with max=02
[    8.656661] pci_bus 0000:02: busn_res: [bus 02-01] end is updated to 02
[    8.663454] pci_bus 0000:02: busn_res: can not insert [bus 02] under [bus 01] (conflicts with (null) [bus 01])
[    8.680096] pci 0000:01:00.0: devices behind bridge are unusable because [bus 02] cannot be assigned for them
[    8.689506] pci_bus 0000:01: bus scan returning with max=02
[    8.703127] pci 0000:00:00.0: bridge has subordinate 01 but max busn 02
[    8.703139] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
[    8.703147] pci_bus 0000:00: bus scan returning with max=01
[    8.703167] pci 0000:01:00.0: BAR 13: no space for [io  size 0x1000]
[    8.719759] pci 0000:01:00.0: BAR 13: failed to assign [io  size 0x1000]
[    8.719767] pci 0000:02:04.0: BAR 0: no space for [io  size 0x0100]
[    8.719771] pci 0000:02:04.0: BAR 0: failed to assign [io  size 0x0100]
[    8.719774] pci 0000:01:00.0: PCI bridge to [bus 02]
[    8.730394] pci 0000:00:00.0: PCI bridge to [bus 01]
[    8.780561] pcieport 0000:00:00.0: assign IRQ: got 61
[    8.780596] pcieport 0000:00:00.0: enabling bus mastering
[    8.780738] pcieport 0000:00:00.0: PME: Signaling with IRQ 61
[    8.793529] pcieport 0000:00:00.0: AER: enabled with IRQ 61
[    8.800190] pcieport 0000:00:00.0: saving config space at offset 0x0 (reading 0x271114e4)
[    8.805636] pcieport 0000:00:00.0: saving config space at offset 0x4 (reading 0x100004)
[    8.805640] pcieport 0000:00:00.0: saving config space at offset 0x8 (reading 0x6040010)
[    8.805644] pcieport 0000:00:00.0: saving config space at offset 0xc (reading 0x10000)
[    8.805648] pcieport 0000:00:00.0: saving config space at offset 0x10 (reading 0x0)
[    8.805652] pcieport 0000:00:00.0: saving config space at offset 0x14 (reading 0x0)
[    8.805655] pcieport 0000:00:00.0: saving config space at offset 0x18 (reading 0x10100)
[    8.805659] pcieport 0000:00:00.0: saving config space at offset 0x1c (reading 0x20000000)
[    8.805663] pcieport 0000:00:00.0: saving config space at offset 0x20 (reading 0xfff0)
[    8.805669] pcieport 0000:00:00.0: saving config space at offset 0x24 (reading 0x1fff1)
[    8.805673] pcieport 0000:00:00.0: saving config space at offset 0x28 (reading 0x0)
[    8.805677] pcieport 0000:00:00.0: saving config space at offset 0x2c (reading 0x0)
[    8.805681] pcieport 0000:00:00.0: saving config space at offset 0x30 (reading 0x0)
[    8.805684] pcieport 0000:00:00.0: saving config space at offset 0x34 (reading 0x48)
[    8.805688] pcieport 0000:00:00.0: saving config space at offset 0x38 (reading 0x0)
[    8.805692] pcieport 0000:00:00.0: saving config space at offset 0x3c (reading 0x2013d)
[    8.806243] pcieport 0000:01:00.0: assign IRQ: got 61
[    9.402928] pci 0000:01:00.0: enabling bus mastering
[    9.402941] snd_virtuoso 0000:02:04.0: invalid PCI I/O range

After I issued the setpci command I rescaned the bus and this is what dmesg shows after:


[  144.734267] pci_bus 0000:00: scanning bus
[  144.735147] pcieport 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
[  144.735160] pci_bus 0000:01: scanning bus
[  144.735928] pci 0000:01:00.0: scanning [bus 02-02] behind bridge, pass 0
[  144.735937] pci_bus 0000:02: scanning bus
[  144.736776] pci_bus 0000:02: bus scan returning with max=02
[  144.736784] pci 0000:01:00.0: devices behind bridge are unusable because [bus 02] cannot be assigned for them
[  144.746870] pci 0000:01:00.0: scanning [bus 02-02] behind bridge, pass 1
[  144.746883] pci_bus 0000:01: bus scan returning with max=02
[  144.746887] pcieport 0000:00:00.0: bridge has subordinate 01 but max busn 02
[  144.754049] pcieport 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
[  144.754056] pci_bus 0000:00: bus scan returning with max=01

lspci info doesn't change between 32bit and 64bit versions

01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 61
	Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: fff00000-000fffff [disabled]
	Prefetchable memory behind bridge: fff00000-000fffff [disabled]
	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr+ FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Power Budgeting <?>

02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
	Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (500ns min, 6000ns max)
	Interrupt: pin A routed to IRQ 61
	Region 0: I/O ports at 0000
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel modules: snd_virtuoso

later edit:
Sadly I don't think this will be possible. this Raspberry Pi Engineer/Moderator says (https://forums.raspberrypi.com/viewtopic.php?p=1746705#p1746705) the bcm2711 datasheet mentions it doesn't support I/O space for PCIe. Only memory space. And that's correctly mapped. I see some I/O bar fails for a r8169 ethernet card, but that is working fine even so. But I can't see the Xonar in /proc/asound/cards

@TheRemote
Copy link

TheRemote commented Mar 2, 2022

I'd never heard of this before until it came up at https://jamesachambers.com/pcie-1x-nvme-on-raspberry-pi-compute-module-4-guide/#comment-12952. John tried a Xawai adapter that I had previously tested with and had no trouble with. He is encountering the same error as RoboTakoh. Since I've used this adapter successfully before it can't be impossible like may be the case with the firewire card.

I can't explain why I never encountered it with the same adapter yet. Hopefully John will test some suggestions I had and some suggestions here. I may try doing it again with the latest Pi OS (I used 64 bit previously, one test I suggested was he try the 64 bit flavor to see if it makes any difference) to see if I can reproduce it on my end. I also linked him here to try some of the tweaks in the gist.

Most likely it just needs a firmware update as it sounds like this should be resolved and would explain why I never encountered it (I was late to the CM4 party).

Very strange issue, great work on this Jeff!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment