Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Increase the BAR memory address space for PCIe devices on the Raspberry Pi Compute Module 4
#!/bin/bash
# The default BAR address space available on the CM4 is too small to allow
# devices like GPUs to initialize correctly. To avoid 'failed to assign memory'
# errors on boot, we need to increase the range of the PCIe bus in the Raspberry
# Pi's Device Tree (a .dtb file specific to each Pi model).
#
# You should probably read up on Device Trees if you don't know what they are:
# https://www.raspberrypi.org/documentation/configuration/device-tree.md
#
# NOTE: The default BAR allocation was increased to 1 GB in this commit:
# https://github.com/raspberrypi/linux/commit/54db4b2fa4d17251c2f6e639f849b27c3b553939
#
# Once that hits the stable Raspberry Pi OS image (or you can install via the
# beta firmware channel), the BAR change instructions below will not be needed.
# Back up current Device Tree for the CM4.
sudo cp /boot/bcm2711-rpi-cm4.dtb /boot/bcm2711-rpi-cm4.dtb.bak
# Decompile the current Device Tree to a dts (source) file.
dtc -I dtb -O dts /boot/bcm2711-rpi-cm4.dtb -o ~/test.dts
# Edit the file and change the PCIe bus range as mentioned in:
# https://www.raspberrypi.org/forums/viewtopic.php?p=1746665#p1746665
nano ~/test.dts
# Replace the line that allocates ~64 MB of RAM in the pci section:
# ranges = < 0x2000000 0x00 0xf8000000 0x06 0x00 0x00 0x4000000 >;
#
# with the following line, which provides up to 1 GB of RAM:
# ranges = <0x02000000 0x0 0xc0000000 0x6 0x00000000 0x0 0x40000000>;
#
# alternatively, use the following line, for ~256 MB of RAM:
# ranges = <0x02000000 0x0 0xe0000000 0x6 0x00000000 0x0 0x10000000>;
# Recompile the Device Tree from the dts (source) file.
dtc -I dts -O dtb ~/test.dts -o ~/test.dtb
# Copy the new Device Tree into place.
sudo mv ~/test.dtb /boot/bcm2711-rpi-cm4.dtb
# Reboot.
sudo reboot
@geerlingguy

This comment has been minimized.

Copy link
Owner Author

@geerlingguy geerlingguy commented Oct 24, 2020

This change is necessary to make the Raspberry Pi Compute Module 4 handle many 3rd party PCIe adapters (e.g. GPUs, SATA cards, etc.), as they require more BAR (Base Address Register) PCI configuration space than is provided by default (~64 MB). The default amount was adequate for simpler devices, like the VL805 USB 3.0 controller used in the Raspberry Pi 4, but many PCIe devices require at least hundreds of MB of BAR space, so this script increases the limit to 1 GB.

The default limit may be increased in future versions of Raspberry Pi OS, since this change is fairly innocuous.

For some examples of devices that may need larger BAR space, see my Raspberry Pi PCIe device compatibility database

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

@geerlingguy geerlingguy commented Oct 26, 2020

@pelwell

This comment has been minimized.

Copy link

@pelwell pelwell commented Oct 28, 2020

It's worth noting that the outbound window isn't allocating RAM, it's reserving a chunk of the PCI address space for devices and mapping it into the ARM's physical address space. No RAM is lost in the process.

The default outbound window on rpi-5.9.y (available from our "next" firmware branch) and rpi-5.10.y has now been expanded to 1GB, so in the future the dtb surgery described above won't be necessary.

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

@geerlingguy geerlingguy commented Oct 28, 2020

@pelwell - ah, thanks for clearing up my misunderstanding, and also for helping get this change into the updated firmware! That'll make it easier and I can avoid patching the DTB.

And the commit: raspberrypi/linux@54db4b2

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

@geerlingguy geerlingguy commented Nov 4, 2020

Also can do 0xffffffff for 4 GB, or 0x80000000 for 2 GB.

Note: Error: /home/pi/test.dts:1970.59-70 Value out of range for 32-bit array element if I tried 0x100000000.

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

@geerlingguy geerlingguy commented Nov 27, 2020

Another update: this post in the Pi Forums contains some trial firmware that allows the BAR space to take up the maximum possible space on the Pi (8GB of PCIe space for any Pi with >= 4 GB of RAM). To use it:

  1. Downloaded the trial firmware and replace the corresponding files in the boot volume.

  2. Set scb ranges to:

       ranges = <0x0 0x7c000000  0x0 0xfc000000  0x0 0x03800000>,
                <0x0 0x40000000  0x0 0xff800000  0x0 0x00800000>,
                <0x6 0x00000000  0x6 0x00000000  0x2 0x00000000>,
                <0x0 0x00000000  0x0 0x00000000  0x0 0xfc000000>;
    
  3. Set pcie ranges to:

       ranges = <0x02000000 0x0 0x00000000 0x6 0x00000000
                 0x2 0x00000000>;
    

More details on the entire process used to try to get a GPU working are in this comment: geerlingguy/raspberry-pi-pcie-devices#6 (comment)

@TheMindVirus

This comment has been minimized.

Copy link

@TheMindVirus TheMindVirus commented Dec 27, 2020

Is the "ranges = /bits/ 64 <0x0 0x600000000 0x00000000c0000000>;" syntax applicable to this version of the Device Tree Compiler (dtc)?
https://github.com/vagrantc/device-tree-compiler/blob/063d02187b584e40998a700d64a3530f287c68e2/Documentation/dts-format.txt#L52
Alternatively it would be ideal not to have to specify the number of bits at all and for the compiler not to complain of the 32-bit limit.
At least the meaning is somewhat semantically human readable this way. I'm already reading it thinking parameter 2 is 0x7c000000 -_(\

@RoboTakoh

This comment has been minimized.

Copy link

@RoboTakoh RoboTakoh commented Apr 5, 2021

I've been trying to get this to work for my firewire card for a few days, but I can't get it to work. Tried 64bit and 32bit raspberry pi os. Tried different amounts of ram, up to 4gb. I've got a cm4 with 4gb or ram and emmc.
I'm not sure what i'm doing wrong. Do I need to recompile the kernel? I'm not sure what to change if I do :'(
Any suggestions? Thank you

(Also, I have another firewire card that has a TI chip. Wonder if i should try using that one as well.)

uname -a
Linux raspberrypi 5.10.25-v7l+ #1408 SMP Mon Mar 22 12:49:24 GMT 2021 armv7l GNU/Linux

>  [    1.202310] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:

> [    1.202341] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
> [    1.202421] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x06fffffffe -> 0x00c0000000
> [    1.202521] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
> [    1.256089] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
> [    1.256493] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
> [    1.256514] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    1.256536] pci_bus 0000:00: root bus resource [mem 0x600000000-0x6fffffffe] (bus address [0xc0000000-0x1bffffffe])
> [    1.256625] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
> [    1.256874] pci 0000:00:00.0: PME# supported from D0 D3hot
> [    1.260339] PCI: bus0: Fast back to back transfers disabled
> [    1.260364] pci 0000:00:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
> [    1.260615] pci 0000:01:00.0: [1b21:1080] type 01 class 0x060400
> [    1.260973] pci 0000:01:00.0: supports D1 D2
> [    1.260991] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    1.264348] PCI: bus1: Fast back to back transfers disabled
> [    1.264374] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [    1.264558] pci_bus 0000:02: extended config space not accessible
> [    1.264696] pci 0000:02:00.0: [1106:3044] type 00 class 0x0c0010
> [    1.264763] pci 0000:02:00.0: reg 0x10: [mem 0x00000000-0x000007ff]
> [    1.264803] pci 0000:02:00.0: reg 0x14: [io  0x0000-0x007f]
> [    1.265035] pci 0000:02:00.0: supports D2
> [    1.265052] pci 0000:02:00.0: PME# supported from D2 D3hot D3cold
> [    1.268451] PCI: bus2: Fast back to back transfers disabled
> [    1.268472] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
> [    1.268501] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 02
> [    1.268545] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
> [    1.268568] pci 0000:01:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
> [    1.268586] pci 0000:01:00.0: BAR 7: no space for [io  size 0x1000]
> [    1.268603] pci 0000:01:00.0: BAR 7: failed to assign [io  size 0x1000]
> [    1.268626] pci 0000:02:00.0: BAR 0: assigned [mem 0x600000000-0x6000007ff]
> [    1.268652] pci 0000:02:00.0: BAR 1: no space for [io  size 0x0080]
> [    1.268669] pci 0000:02:00.0: BAR 1: failed to assign [io  size 0x0080]
> [    1.268688] pci 0000:01:00.0: PCI bridge to [bus 02]
> [    1.268718] pci 0000:01:00.0:   bridge window [mem 0x600000000-0x6000fffff]
> [    1.268758] pci 0000:00:00.0: PCI bridge to [bus 01-02]
> [    1.268783] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
> [    1.269171] pcieport 0000:00:00.0: enabling device (0140 -> 0142)
> [    1.269396] pcieport 0000:00:00.0: PME: Signaling with IRQ 64
> [    1.269677] pci 0000:01:00.0: Disabling ASPM L0s/L1
> [    1.275651] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
> [    1.278668] iproc-rng200 fe104000.rng: hwrng registered
> [    1.278982] vc-mem: phys_addr:0x00000000 mem_base=0x3ec00000 mem_size:0x40000000(1024 MiB)
> [    1.279840] gpiomem-bcm2835 fe200000.gpiomem: Initialised: Registers at 0xfe200000

> 00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
> 01:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)
> 02:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev 80)

 
> lspci -vmmnn
> Slot:	00:00.0
> Class:	PCI bridge [0604]
> Vendor:	Broadcom Limited [14e4]
> Device:	Device [2711]
> Rev:	20
> 
> Slot:	01:00.0
> Class:	PCI bridge [0604]
> Vendor:	ASMedia Technology Inc. [1b21]
> Device:	ASM1083/1085 PCIe to PCI Bridge [1080]
> Rev:	01
> 
> Slot:	02:00.0
> Class:	FireWire (IEEE 1394) [0c00]
> Vendor:	VIA Technologies, Inc. [1106]
> Device:	VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [3044]
> SVendor:	VIA Technologies, Inc. [1106]
> SDevice:	VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [3044]
> Rev:	80
> ProgIf:	10
 

> lspci -nnk
> 00:00.0 PCI bridge [0604]: Broadcom Limited Device [14e4:2711] (rev 20)
> 	Kernel driver in use: pcieport
> 01:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01)
> 02:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] (rev 80)
> 	Subsystem: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044]



> lspci -vvnnk
> 00:00.0 PCI bridge [0604]: Broadcom Limited Device [14e4:2711] (rev 20) (prog-if 00 [Normal decode])
> 	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Latency: 0, Cache Line Size: 64 bytes
> 	Interrupt: pin A routed to IRQ 64
> 	Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
> 	I/O behind bridge: 00000000-00000fff
> 	Memory behind bridge: c0000000-c00fffff
> 	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> 	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
> 		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> 	Capabilities: access denied
> 	Kernel driver in use: pcieport
> 
> 01:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01) (prog-if 00 [Normal decode])
> 	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Interrupt: pin A routed to IRQ 64
> 	Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
> 	Memory behind bridge: c0000000-c00fffff
> 	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> 	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B-
> 		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> 	Capabilities: access denied
> 
> 02:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] (rev 80) (prog-if 10 [OHCI])
> 	Subsystem: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044]
> 	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping+ SERR+ FastB2B- DisINTx-
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Interrupt: pin A routed to IRQ 0
> 	Region 0: Memory at 600000000 (32-bit, non-prefetchable) [disabled] [size=2K]
> 	Region 1: I/O ports at <unassigned> [disabled]
> 	Capabilities: access denied


@geerlingguy

This comment has been minimized.

Copy link
Owner Author

@geerlingguy geerlingguy commented Apr 7, 2021

It looks like the memory is being assigned. Note that no IO BAR space is available on the Pi, so you'll never see IO space allocated.

I so far haven't seen anyone get a Firewire card working :(

@RoboTakoh

This comment has been minimized.

Copy link

@RoboTakoh RoboTakoh commented Apr 8, 2021

Thank you very much for the reply and your helpful posts.
Ah that sucks. That's the whole reason why I bought the cm4 & io board.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment