Increase the BAR memory address space for PCIe devices on the Raspberry Pi Compute Module 4

increase-pci-bar-space.sh

#!/bin/bash

# The default BAR address space available on the CM4 is too small to allow
# devices like GPUs to initialize correctly. To avoid 'failed to assign memory'
# errors on boot, we need to increase the range of the PCIe bus in the Raspberry
# Pi's Device Tree (a .dtb file specific to each Pi model).
#
# You should probably read up on Device Trees if you don't know what they are:
# https://www.raspberrypi.org/documentation/configuration/device-tree.md
#
# NOTE: The default BAR allocation was increased to 1 GB in this commit:
# https://github.com/raspberrypi/linux/commit/54db4b2fa4d17251c2f6e639f849b27c3b553939
#
# Once that hits the stable Raspberry Pi OS image (or you can install via the
# beta firmware channel), the BAR change instructions below will not be needed.

# Back up current Device Tree for the CM4.
sudo cp /boot/bcm2711-rpi-cm4.dtb /boot/bcm2711-rpi-cm4.dtb.bak

# Decompile the current Device Tree to a dts (source) file.
dtc -I dtb -O dts /boot/bcm2711-rpi-cm4.dtb -o ~/test.dts

# Edit the file and change the PCIe bus range as mentioned in:
# https://www.raspberrypi.org/forums/viewtopic.php?p=1746665#p1746665
nano ~/test.dts

# Replace the line that allocates ~64 MB of RAM in the pci section:
#   ranges = < 0x2000000 0x00 0xf8000000 0x06 0x00 0x00 0x4000000 >;
#
# with the following line, which provides up to 1 GB of RAM:
#   ranges = <0x02000000 0x0 0xc0000000 0x6 0x00000000 0x0 0x40000000>;
#
# alternatively, use the following line, for ~256 MB of RAM:
#   ranges = <0x02000000 0x0 0xe0000000 0x6 0x00000000 0x0 0x10000000>;

# Recompile the Device Tree from the dts (source) file.
dtc -I dts -O dtb ~/test.dts -o ~/test.dtb

# Copy the new Device Tree into place.
sudo mv ~/test.dtb /boot/bcm2711-rpi-cm4.dtb

# Reboot.
sudo reboot

This change is necessary to make the Raspberry Pi Compute Module 4 handle many 3rd party PCIe adapters (e.g. GPUs, SATA cards, etc.), as they require more BAR (Base Address Register) PCI configuration space than is provided by default (~64 MB). The default amount was adequate for simpler devices, like the VL805 USB 3.0 controller used in the Raspberry Pi 4, but many PCIe devices require at least hundreds of MB of BAR space, so this script increases the limit to 1 GB.

The default limit may be increased in future versions of Raspberry Pi OS, since this change is fairly innocuous.

For some examples of devices that may need larger BAR space, see my Raspberry Pi PCIe device compatibility database

See related gist: Set up the Nvidia GeForce GT 710 on Raspberry Pi Compute Module 4

It's worth noting that the outbound window isn't allocating RAM, it's reserving a chunk of the PCI address space for devices and mapping it into the ARM's physical address space. No RAM is lost in the process.

The default outbound window on rpi-5.9.y (available from our "next" firmware branch) and rpi-5.10.y has now been expanded to 1GB, so in the future the dtb surgery described above won't be necessary.

@pelwell - ah, thanks for clearing up my misunderstanding, and also for helping get this change into the updated firmware! That'll make it easier and I can avoid patching the DTB.

And the commit: raspberrypi/linux@54db4b2

Also can do 0xffffffff for 4 GB, or 0x80000000 for 2 GB.

Note: Error: /home/pi/test.dts:1970.59-70 Value out of range for 32-bit array element if I tried 0x100000000.

Another update: this post in the Pi Forums contains some trial firmware that allows the BAR space to take up the maximum possible space on the Pi (8GB of PCIe space for any Pi with >= 4 GB of RAM). To use it:

1. Downloaded the trial firmware and replace the corresponding files in the boot volume.

2. Set scb ranges to:

      ranges = <0x0 0x7c000000  0x0 0xfc000000  0x0 0x03800000>,
               <0x0 0x40000000  0x0 0xff800000  0x0 0x00800000>,
               <0x6 0x00000000  0x6 0x00000000  0x2 0x00000000>,
               <0x0 0x00000000  0x0 0x00000000  0x0 0xfc000000>;
   

3. Set pcie ranges to:

      ranges = <0x02000000 0x0 0x00000000 0x6 0x00000000
                0x2 0x00000000>;
   

More details on the entire process used to try to get a GPU working are in this comment: geerlingguy/raspberry-pi-pcie-devices#6 (comment)

Is the "ranges = /bits/ 64 <0x0 0x600000000 0x00000000c0000000>;" syntax applicable to this version of the Device Tree Compiler (dtc)?
https://github.com/vagrantc/device-tree-compiler/blob/063d02187b584e40998a700d64a3530f287c68e2/Documentation/dts-format.txt#L52
Alternatively it would be ideal not to have to specify the number of bits at all and for the compiler not to complain of the 32-bit limit.
At least the meaning is somewhat semantically human readable this way. I'm already reading it thinking parameter 2 is 0x7c000000 -_(\