-
-
Save geerlingguy/9d78ea34cab8e18d71ee5954417429df to your computer and use it in GitHub Desktop.
#!/bin/bash | |
# The default BAR address space available on the CM4 may be too small to allow | |
# some devices to initialize correctly. To avoid 'failed to assign memory' | |
# errors on boot, you can increase the range of the PCIe bus in the Raspberry | |
# Pi's Device Tree (a .dtb file specific to each Pi model). | |
# | |
# You should probably read up on Device Trees if you don't know what they are: | |
# https://www.raspberrypi.org/documentation/configuration/device-tree.md | |
# | |
# NOTE: The default BAR allocation was increased to 1 GB in this commit: | |
# https://github.com/raspberrypi/linux/commit/54db4b2fa4d17251c2f6e639f849b27c3b553939 | |
# Download the trial firmware from this post: | |
# https://www.raspberrypi.org/forums/viewtopic.php?p=1761834#p1761834 | |
# and replace the corresponding files in the boot volume. | |
# Back up current Device Tree for the CM4. | |
sudo cp /boot/bcm2711-rpi-cm4.dtb /boot/bcm2711-rpi-cm4.dtb.bak | |
# Decompile the current Device Tree to a dts (source) file. | |
dtc -I dtb -O dts /boot/bcm2711-rpi-cm4.dtb -o ~/test.dts | |
# Edit the file and change the PCIe bus range as mentioned in: | |
# https://www.raspberrypi.org/forums/viewtopic.php?p=1746665#p1746665 | |
nano ~/test.dts | |
# Replace the line that allocates 1 GB of RAM in the pci section: | |
# ranges = <0x02000000 0x0 0xc0000000 0x6 0x00000000 0x0 0x40000000>; | |
# | |
# with the following line, which provides up to 8 GB of RAM: | |
# ranges = <0x02000000 0x0 0x00000000 0x6 0x00000000 0x2 0x00000000>; | |
# Also replace the scb ranges to: | |
# ranges = <0x0 0x7c000000 0x0 0xfc000000 0x0 0x03800000>, | |
# <0x0 0x40000000 0x0 0xff800000 0x0 0x00800000>, | |
# <0x6 0x00000000 0x6 0x00000000 0x2 0x00000000>, | |
# <0x0 0x00000000 0x0 0x00000000 0x0 0xfc000000>; | |
# Recompile the Device Tree from the dts (source) file. | |
dtc -I dts -O dtb ~/test.dts -o ~/test.dtb | |
# Copy the new Device Tree into place. | |
sudo mv ~/test.dtb /boot/bcm2711-rpi-cm4.dtb | |
# Reboot. | |
sudo reboot |
See related gist: Set up the Nvidia GeForce GT 710 on Raspberry Pi Compute Module 4
It's worth noting that the outbound window isn't allocating RAM, it's reserving a chunk of the PCI address space for devices and mapping it into the ARM's physical address space. No RAM is lost in the process.
The default outbound window on rpi-5.9.y (available from our "next" firmware branch) and rpi-5.10.y has now been expanded to 1GB, so in the future the dtb surgery described above won't be necessary.
@pelwell - ah, thanks for clearing up my misunderstanding, and also for helping get this change into the updated firmware! That'll make it easier and I can avoid patching the DTB.
And the commit: raspberrypi/linux@54db4b2
Also can do 0xffffffff
for 4 GB, or 0x80000000
for 2 GB.
Note: Error: /home/pi/test.dts:1970.59-70 Value out of range for 32-bit array element
if I tried 0x100000000
.
Another update: this post in the Pi Forums contains some trial firmware that allows the BAR space to take up the maximum possible space on the Pi (8GB of PCIe space for any Pi with >= 4 GB of RAM). To use it:
-
Downloaded the trial firmware and replace the corresponding files in the boot volume.
-
Set scb ranges to:
ranges = <0x0 0x7c000000 0x0 0xfc000000 0x0 0x03800000>, <0x0 0x40000000 0x0 0xff800000 0x0 0x00800000>, <0x6 0x00000000 0x6 0x00000000 0x2 0x00000000>, <0x0 0x00000000 0x0 0x00000000 0x0 0xfc000000>;
-
Set pcie ranges to:
ranges = <0x02000000 0x0 0x00000000 0x6 0x00000000 0x2 0x00000000>;
More details on the entire process used to try to get a GPU working are in this comment: geerlingguy/raspberry-pi-pcie-devices#6 (comment)
Is the "ranges = /bits/ 64 <0x0 0x600000000 0x00000000c0000000>;" syntax applicable to this version of the Device Tree Compiler (dtc)?
https://github.com/vagrantc/device-tree-compiler/blob/063d02187b584e40998a700d64a3530f287c68e2/Documentation/dts-format.txt#L52
Alternatively it would be ideal not to have to specify the number of bits at all and for the compiler not to complain of the 32-bit limit.
At least the meaning is somewhat semantically human readable this way. I'm already reading it thinking parameter 2 is 0x7c000000 -_(\
I've been trying to get this to work for my firewire card for a few days, but I can't get it to work. Tried 64bit and 32bit raspberry pi os. Tried different amounts of ram, up to 4gb. I've got a cm4 with 4gb or ram and emmc.
I'm not sure what i'm doing wrong. Do I need to recompile the kernel? I'm not sure what to change if I do :'(
Any suggestions? Thank you
(Also, I have another firewire card that has a TI chip. Wonder if i should try using that one as well.)
uname -a
Linux raspberrypi 5.10.25-v7l+ #1408 SMP Mon Mar 22 12:49:24 GMT 2021 armv7l GNU/Linux
> [ 1.202310] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
> [ 1.202341] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
> [ 1.202421] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x06fffffffe -> 0x00c0000000
> [ 1.202521] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
> [ 1.256089] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
> [ 1.256493] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
> [ 1.256514] pci_bus 0000:00: root bus resource [bus 00-ff]
> [ 1.256536] pci_bus 0000:00: root bus resource [mem 0x600000000-0x6fffffffe] (bus address [0xc0000000-0x1bffffffe])
> [ 1.256625] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
> [ 1.256874] pci 0000:00:00.0: PME# supported from D0 D3hot
> [ 1.260339] PCI: bus0: Fast back to back transfers disabled
> [ 1.260364] pci 0000:00:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
> [ 1.260615] pci 0000:01:00.0: [1b21:1080] type 01 class 0x060400
> [ 1.260973] pci 0000:01:00.0: supports D1 D2
> [ 1.260991] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [ 1.264348] PCI: bus1: Fast back to back transfers disabled
> [ 1.264374] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [ 1.264558] pci_bus 0000:02: extended config space not accessible
> [ 1.264696] pci 0000:02:00.0: [1106:3044] type 00 class 0x0c0010
> [ 1.264763] pci 0000:02:00.0: reg 0x10: [mem 0x00000000-0x000007ff]
> [ 1.264803] pci 0000:02:00.0: reg 0x14: [io 0x0000-0x007f]
> [ 1.265035] pci 0000:02:00.0: supports D2
> [ 1.265052] pci 0000:02:00.0: PME# supported from D2 D3hot D3cold
> [ 1.268451] PCI: bus2: Fast back to back transfers disabled
> [ 1.268472] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
> [ 1.268501] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 02
> [ 1.268545] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
> [ 1.268568] pci 0000:01:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
> [ 1.268586] pci 0000:01:00.0: BAR 7: no space for [io size 0x1000]
> [ 1.268603] pci 0000:01:00.0: BAR 7: failed to assign [io size 0x1000]
> [ 1.268626] pci 0000:02:00.0: BAR 0: assigned [mem 0x600000000-0x6000007ff]
> [ 1.268652] pci 0000:02:00.0: BAR 1: no space for [io size 0x0080]
> [ 1.268669] pci 0000:02:00.0: BAR 1: failed to assign [io size 0x0080]
> [ 1.268688] pci 0000:01:00.0: PCI bridge to [bus 02]
> [ 1.268718] pci 0000:01:00.0: bridge window [mem 0x600000000-0x6000fffff]
> [ 1.268758] pci 0000:00:00.0: PCI bridge to [bus 01-02]
> [ 1.268783] pci 0000:00:00.0: bridge window [mem 0x600000000-0x6000fffff]
> [ 1.269171] pcieport 0000:00:00.0: enabling device (0140 -> 0142)
> [ 1.269396] pcieport 0000:00:00.0: PME: Signaling with IRQ 64
> [ 1.269677] pci 0000:01:00.0: Disabling ASPM L0s/L1
> [ 1.275651] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
> [ 1.278668] iproc-rng200 fe104000.rng: hwrng registered
> [ 1.278982] vc-mem: phys_addr:0x00000000 mem_base=0x3ec00000 mem_size:0x40000000(1024 MiB)
> [ 1.279840] gpiomem-bcm2835 fe200000.gpiomem: Initialised: Registers at 0xfe200000
> 00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
> 01:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)
> 02:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev 80)
> lspci -vmmnn
> Slot: 00:00.0
> Class: PCI bridge [0604]
> Vendor: Broadcom Limited [14e4]
> Device: Device [2711]
> Rev: 20
>
> Slot: 01:00.0
> Class: PCI bridge [0604]
> Vendor: ASMedia Technology Inc. [1b21]
> Device: ASM1083/1085 PCIe to PCI Bridge [1080]
> Rev: 01
>
> Slot: 02:00.0
> Class: FireWire (IEEE 1394) [0c00]
> Vendor: VIA Technologies, Inc. [1106]
> Device: VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [3044]
> SVendor: VIA Technologies, Inc. [1106]
> SDevice: VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [3044]
> Rev: 80
> ProgIf: 10
> lspci -nnk
> 00:00.0 PCI bridge [0604]: Broadcom Limited Device [14e4:2711] (rev 20)
> Kernel driver in use: pcieport
> 01:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01)
> 02:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] (rev 80)
> Subsystem: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044]
> lspci -vvnnk
> 00:00.0 PCI bridge [0604]: Broadcom Limited Device [14e4:2711] (rev 20) (prog-if 00 [Normal decode])
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin A routed to IRQ 64
> Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
> I/O behind bridge: 00000000-00000fff
> Memory behind bridge: c0000000-c00fffff
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: access denied
> Kernel driver in use: pcieport
>
> 01:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01) (prog-if 00 [Normal decode])
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 64
> Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
> Memory behind bridge: c0000000-c00fffff
> Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B-
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: access denied
>
> 02:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] (rev 80) (prog-if 10 [OHCI])
> Subsystem: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044]
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping+ SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 0
> Region 0: Memory at 600000000 (32-bit, non-prefetchable) [disabled] [size=2K]
> Region 1: I/O ports at <unassigned> [disabled]
> Capabilities: access denied
It looks like the memory is being assigned. Note that no IO BAR space is available on the Pi, so you'll never see IO space allocated.
I so far haven't seen anyone get a Firewire card working :(
Thank you very much for the reply and your helpful posts.
Ah that sucks. That's the whole reason why I bought the cm4 & io board.
@geerlingguy I just tried a 512MB BAR device using the latest release of Pi OS (32 bit) and it worked fine! I am still seeing issues with BAR size allocated on Ubuntu Server 20.04.2 LTS though.
Just updated this Gist to reflect the new default (1 GB BAR space), as well as how to increase it to 8 GB following the guide linked earlier in the comments.
Hi there -
Is this bar space increase still a required procedure with latest firmware and kernels? Asking as I am getting kernel panics when booting the CM4 on the CM4 i/o board with the ASHATA PCE4PCE-A01 1 to 4 Port PCIe Switch AND the ELUTENG PCIE USB 3.0 Card 4 Ports PCI Expree to USB Expansion Card. With both plugged in individually the CM4 boots up.
@l00nix - No, this shouldn't be necessary unless you need a full 8 GB of BAR space (and vanishingly few cards I've tested need that).
For the kernel panic, feel free to open an issue on my Pi PCIe site/repo for that card and see if anyone else can help debug.
Hello,
I'm trying to make a Xonar DX audio card work but there's some issues.
The CMI8788 chipset seems to be ok, has snd_virtuoso as driver. The issue seems to be with the PLX bridge, a PEX8112.
From what I've seen for linux it should use the shpchp driver, which I compiled into the kernel, but doesn't seem to load it.
dmesg :
[ 1.340364] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 1.340833] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 1.340883] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 1.340985] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[ 1.341094] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x007fffffff -> 0x0400000000
[ 1.366959] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
[ 1.367344] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[ 1.367381] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.367417] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[ 1.367529] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[ 1.367791] pci 0000:00:00.0: PME# supported from D0 D3hot
[ 1.371322] PCI: bus0: Fast back to back transfers disabled
[ 1.371363] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.371704] pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
[ 1.375429] PCI: bus1: Fast back to back transfers disabled
[ 1.375471] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.375681] pci_bus 0000:02: extended config space not accessible
[ 1.376296] pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
[ 1.376379] pci 0000:02:04.0: reg 0x10: [io 0x42000000-0x420000ff]
[ 1.376651] pci 0000:02:04.0: supports D1 D2
[ 1.379665] PCI: bus2: Fast back to back transfers disabled
[ 1.379701] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[ 1.379746] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 02
[ 1.379802] pci 0000:01:00.0: BAR 7: no space for [io size 0x1000]
[ 1.379835] pci 0000:01:00.0: BAR 7: failed to assign [io size 0x1000]
[ 1.379870] pci 0000:02:04.0: BAR 0: no space for [io size 0x0100]
[ 1.379902] pci 0000:02:04.0: BAR 0: failed to assign [io size 0x0100]
[ 1.379936] pci 0000:01:00.0: PCI bridge to [bus 02]
[ 1.380005] pci 0000:00:00.0: PCI bridge to [bus 01-02]
[ 1.380641] pcieport 0000:00:00.0: PME: Signaling with IRQ 62
[ 4.941635] snd_virtuoso 0000:02:04.0: invalid PCI I/O range
journalctl:
Dec 05 15:17:01 rbnet kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie: MEM 0x0600000000..0x063fffffff -> 0x00c0000000
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x007fffffff -> 0x0400000000
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
Dec 05 15:17:01 rbnet kernel: brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
Dec 05 15:17:01 rbnet kernel: pci_bus 0000:00: root bus resource [bus 00-ff]
Dec 05 15:17:01 rbnet kernel: pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
Dec 05 15:17:01 rbnet kernel: pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
Dec 05 15:17:01 rbnet kernel: pci 0000:00:00.0: PME# supported from D0 D3hot
Dec 05 15:17:01 rbnet kernel: PCI: bus0: Fast back to back transfers disabled
Dec 05 15:17:01 rbnet kernel: pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: supports D1
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: PME# supported from D0 D1 D3hot
Dec 05 15:17:01 rbnet kernel: PCI: bus1: Fast back to back transfers disabled
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 05 15:17:01 rbnet kernel: pci_bus 0000:02: extended config space not accessible
Dec 05 15:17:01 rbnet kernel: pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
Dec 05 15:17:01 rbnet kernel: pci 0000:02:04.0: reg 0x10: [io 0x46000000-0x460000ff]
Dec 05 15:17:01 rbnet kernel: pci 0000:02:04.0: supports D1 D2
Dec 05 15:17:01 rbnet kernel: PCI: bus2: Fast back to back transfers disabled
Dec 05 15:17:01 rbnet kernel: pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
Dec 05 15:17:01 rbnet kernel: pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 02
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: BAR 7: no space for [io size 0x1000]
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: BAR 7: failed to assign [io size 0x1000]
Dec 05 15:17:01 rbnet kernel: pci 0000:02:04.0: BAR 0: no space for [io size 0x0100]
Dec 05 15:17:01 rbnet kernel: pci 0000:02:04.0: BAR 0: failed to assign [io size 0x0100]
Dec 05 15:17:01 rbnet kernel: pci 0000:01:00.0: PCI bridge to [bus 02]
Dec 05 15:17:01 rbnet kernel: pci 0000:00:00.0: PCI bridge to [bus 01-02]
Dec 05 15:17:01 rbnet kernel: pcieport 0000:00:00.0: PME: Signaling with IRQ 62
Dec 05 15:17:01 rbnet kernel: snd_virtuoso 0000:02:04.0: invalid PCI I/O range
and lspci:
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 10) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 62
Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
I/O behind bridge: 00000000-00000fff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=1us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Kernel driver in use: pcieport
01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 62
Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Power Budgeting <?>
02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 62
Region 0: I/O ports at <ignored> [disabled]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel modules: snd_virtuoso
Any ideas on how to go about it?
Also do you have any issues with power order between Pi and PCIe boards? I had a r8168 board (that I sadly bricked) that wouldn't be recognized if I didn't pulse it's PCIERST pin low a bit after powering the Pi. Connecting that pin from Pi to the ethernet board didn't help.
Is this common with Pi4 and PCIe devices, or do they usually work without powerup sequence being important?
I want to make a setup that I can just power on at once without worries about which gets powered first.
edit: I'm on Pi4 with the latest beta firmware. was the same with the latest stable one.
also I don't have the ethernet PHY chip and neither the VL805 one, they got toasted in a mishap. Also the wifi/bt is missing, so could this help me free some resources? I tried to software disable/remove the ethernet PHY but didn't manage, and I also have a crappy timeout at boot.
2nd edit: I think the issue for the Pi4 is the fact that only PCIe_nRST (pin 109 on CM4) is present at the VL805 pins. PCIe_CLK_nREQ (pin 102 on CM4) is missing from the VL805 connections. I suppose they rather used it as a GPIO maybe? I think this pin is what causes issues with finding different PCIe boards on the bus, for the RPi4. I think on PCIe_CLK_nREQ devices on the bus keep the BCM2711 bus active.
CM4 breaks out both of these pins. would be interesting to know if and what pin it's broken out to on Pi4.
From what I can see the drivers you mentioned are loading, perhaps apart from shpchp
.
A Quick Look-Up in the source yielded this:
https://github.com/raspberrypi/linux/blob/a1d0f808d4a0b7f7053095cd4ab97a4276bed9ff/drivers/pci/hotplug/Kconfig#L106
https://cateee.net/lkddb/web-lkddb/HOTPLUG_PCI_SHPC.html
https://github.com/raspberrypi/linux/blob/e9e302fd9aede393bcf7f264a53b8d98e1cd8df2/arch/arm64/configs/bcm2711_defconfig
You may have looked for a CONFIG_HOTPLUG_PCI_SHPC
but I'm not sure the BCM2711-related board support has...support... for the PCIe Hot-plug Motherboard Feature in Firmware that the shpchp
driver is meant to be in control of.
What has hot-plug been like on any of the other devices you've tested, does it still work after the card is disconnected and reconnected? (Perhaps with a power riser board in between to avoid touching a device which is still powered on)
I haven't had the hardware to test with but previously PCIe hot-plug has been a feature delegated to enterprise motherboards.
If the Pi is missing pins itself then maybe a downstream device (such as your PCIe bridge) might be able to handle the re-clocking part of the hot-plug event independently of the Pi, but the OS will not know that this has happened and will manually have to check for new devices. Again, speaking from past experience of not having such hardware but it might be theoretically possible.
I would be interested to see if the Smart-NIC / DPU-Market could use an extra PCIe Female Connector to sit between a test device (such as your Xonar DX Audio Card) and the Pi to not only diagnose things like this but also do re-clocking, link training and essential low-level communication for legacy devices that the OS itself is unable to support electronically from where it is.
I made a mistake where I left the pullup resistor when I connected PCIe_nRST from Pi to the PCIe breakout pcb. Removing the resistor seems to make the Pi correctly control the PCIe connected device.
So until I have a board that doesn't work without the PCIe_CLK_nREQ connection then I can't say for sure it's absolutely necessary.
I got curious and looked at the lspci details on my x86_64 machine (for Xonar DX), and there isn't much difference. There's no driver shown for the PLX chip, just like on the Pi4. So I'm not sure if that shpchp driver is really required. On the Pi there's no difference in reported info between no PCI hotplug driver, just PCI hotplug driver or both PCI/PCIe hotplug drivers. I also enabled more verbosity for PCI and here's how it looks at this moment:
Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 62
Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Power Budgeting <?>
02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 62
Region 0: I/O ports at <ignored> [disabled]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel modules: snd_virtuoso
dmesg:
[ 0.000000] PCIe ASPM is forcibly enabled
[ 1.353502] PCI: CLS 0 bytes, default 64
[ 1.377120] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 1.377682] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 1.377740] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 1.377825] brcm-pcie fd500000.pcie: Parsing ranges property...
[ 1.377899] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[ 1.377966] brcm-pcie fd500000.pcie: Parsing dma-ranges property...
[ 1.378048] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x007fffffff -> 0x0400000000
[ 1.407936] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
[ 1.408176] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[ 1.408219] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.408260] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[ 1.408308] pci_bus 0000:00: scanning bus
[ 1.408406] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[ 1.408732] pci 0000:00:00.0: PME# supported from D0 D3hot
[ 1.408776] pci 0000:00:00.0: PME# disabled
[ 1.412362] pci_bus 0000:00: fixups for bus
[ 1.412401] PCI: bus0: Fast back to back transfers disabled
[ 1.412448] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
[ 1.412467] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.412541] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
[ 1.412836] pci_bus 0000:01: scanning bus
[ 1.412931] pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
[ 1.416701] pci_bus 0000:01: fixups for bus
[ 1.416748] PCI: bus1: Fast back to back transfers disabled
[ 1.416796] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
[ 1.416834] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.416907] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[ 1.417048] pci_bus 0000:02: extended config space not accessible
[ 1.417182] pci_bus 0000:02: scanning bus
[ 1.417637] pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
[ 1.417731] pci 0000:02:04.0: reg 0x10: [io 0x0a00-0x0aff]
[ 1.418055] pci 0000:02:04.0: supports D1 D2
[ 1.421110] pci_bus 0000:02: fixups for bus
[ 1.421148] PCI: bus2: Fast back to back transfers disabled
[ 1.421187] pci_bus 0000:02: bus scan returning with max=02
[ 1.421209] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[ 1.421259] pci_bus 0000:01: bus scan returning with max=02
[ 1.421279] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 02
[ 1.421326] pci_bus 0000:00: bus scan returning with max=02
[ 1.421362] pci 0000:01:00.0: BAR 7: no space for [io size 0x1000]
[ 1.421402] pci 0000:01:00.0: BAR 7: failed to assign [io size 0x1000]
[ 1.421444] pci 0000:02:04.0: BAR 0: no space for [io size 0x0100]
[ 1.421482] pci 0000:02:04.0: BAR 0: failed to assign [io size 0x0100]
[ 1.421521] pci 0000:01:00.0: PCI bridge to [bus 02]
[ 1.421600] pci 0000:00:00.0: PCI bridge to [bus 01-02]
[ 1.421690] pci 0000:00:00.0: Max Payload Size set to 512/ 512 (was 128), Max Read Rq 512
[ 1.421788] pci 0000:01:00.0: Max Payload Size set to 128/ 128 (was 128), Max Read Rq 128
[ 1.422268] pcieport 0000:00:00.0: assign IRQ: got 62
[ 1.422339] pcieport 0000:00:00.0: enabling bus mastering
[ 1.422554] pcieport 0000:00:00.0: PME: Signaling with IRQ 62
[ 1.422743] pcieport 0000:00:00.0: saving config space at offset 0x0 (reading 0x271114e4)
[ 1.422766] pcieport 0000:00:00.0: saving config space at offset 0x4 (reading 0x100144)
[ 1.422788] pcieport 0000:00:00.0: saving config space at offset 0x8 (reading 0x6040010)
[ 1.422894] pcieport 0000:00:00.0: saving config space at offset 0xc (reading 0x10010)
[ 1.422917] pcieport 0000:00:00.0: saving config space at offset 0x10 (reading 0x0)
[ 1.422938] pcieport 0000:00:00.0: saving config space at offset 0x14 (reading 0x0)
[ 1.422960] pcieport 0000:00:00.0: saving config space at offset 0x18 (reading 0x20100)
[ 1.422981] pcieport 0000:00:00.0: saving config space at offset 0x1c (reading 0x20000000)
[ 1.423003] pcieport 0000:00:00.0: saving config space at offset 0x20 (reading 0xfff0)
[ 1.423024] pcieport 0000:00:00.0: saving config space at offset 0x24 (reading 0x1fff1)
[ 1.423044] pcieport 0000:00:00.0: saving config space at offset 0x28 (reading 0x0)
[ 1.423065] pcieport 0000:00:00.0: saving config space at offset 0x2c (reading 0x0)
[ 1.423086] pcieport 0000:00:00.0: saving config space at offset 0x30 (reading 0x0)
[ 1.423108] pcieport 0000:00:00.0: saving config space at offset 0x34 (reading 0x48)
[ 1.423128] pcieport 0000:00:00.0: saving config space at offset 0x38 (reading 0x0)
[ 1.423149] pcieport 0000:00:00.0: saving config space at offset 0x3c (reading 0x3013e)
[ 1.423645] pcieport 0000:01:00.0: assign IRQ: got 62
[ 4.354949] pci 0000:01:00.0: enabling bus mastering
[ 4.354971] snd_virtuoso 0000:02:04.0: invalid PCI I/O range
Tried different cmdline.txt parameters for pci and didn't make any difference.
And here's lspci from my PC for the DX board. As you can see there isn't a listed kernel driver.
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 37
IOMMU group: 14
Bus: primary=23, secondary=24, subordinate=24, sec-latency=32
I/O behind bridge: 0000e000-0000efff [size=4K]
Memory behind bridge: fff00000-000fffff [disabled]
Prefetchable memory behind bridge: fff00000-000fffff [disabled]
Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 26.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr+ FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Power Budgeting <?>
24:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 32 (500ns min, 6000ns max)
Interrupt: pin A routed to IRQ 37
IOMMU group: 14
Region 0: I/O ports at e000 [size=256]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: snd_virtuoso
Kernel modules: snd_virtuoso
So, this is not a problem that can be fixed by increasing the BAR? I have 2G of RAM on my Pi4.
For your PC it says: I/O behind bridge: 0000e000-0000efff [size=4K]
and for the Pi it says: I/O behind bridge: 0000f000-00000fff
That seems fine but then it says:
For the PC: Region 0: I/O ports at e000 [size=256]
For the Pi: Region 0: I/O ports at <ignored> [disabled]
(shouldn't that read Region 0: I/O ports at f000 [size=256]
?)
Combining these two pieces of random documentation together:
https://support.xilinx.com/s/question/0D52E00006iHlNoSAK/lspci-reports-bar-0-disabled?language=en_US
https://wiki.osdev.org/PCI
The command seems to be: setpci -s 02:04.0 COMMAND=0x05
which sets I/O+
and BusMaster+
like you have on your PC.
This also sets SERR-
where it is SERR+
on the Pi for some reason. That would make sense why it isn't allowed to send any data.
I'd hold off from entering the command immediately until you have the right device ID entered and you're sure the command is correct.
Never had to do this before though, usually this sort of thing is left correctly configured by default rather than having a million off-switches.
setpci -s 02:04.0 COMMAND=0x05
02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0 (500ns min, 6000ns max)
Interrupt: pin A routed to IRQ 61
Region 0: I/O ports at 0000
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel modules: snd_virtuoso
Sets it at 0000 instead of f000. And the PLX chip still shows
I/O behind bridge: 0000f000-00000fff [disabled]
Also tried it on aarch64 and behaves the same.
dmesg:
[ 1.669941] PCI: CLS 0 bytes, default 64
[ 2.345316] pciehp: pcie_port_service_register = 0
[ 2.345449] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 3.468976] ehci-pci: EHCI PCI platform driver
[ 3.489589] ohci-pci: OHCI PCI platform driver
[ 8.468012] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 8.475207] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 8.484761] brcm-pcie fd500000.pcie: Parsing ranges property...
[ 8.484787] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[ 8.493208] brcm-pcie fd500000.pcie: Parsing dma-ranges property...
[ 8.493236] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x00bfffffff -> 0x0000000000
[ 8.532089] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
[ 8.538764] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[ 8.549306] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 8.566028] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[ 8.577111] pci_bus 0000:00: scanning bus
[ 8.577157] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[ 8.583358] pci 0000:00:00.0: PME# supported from D0 D3hot
[ 8.588944] pci 0000:00:00.0: PME# disabled
[ 8.594378] pci_bus 0000:00: fixups for bus
[ 8.594394] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
[ 8.594495] pci_bus 0000:01: scanning bus
[ 8.594540] pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
[ 8.601244] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force'
[ 8.612296] pci_bus 0000:01: fixups for bus
[ 8.612309] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
[ 8.612315] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 8.620484] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[ 8.620567] pci_bus 0000:02: extended config space not accessible
[ 8.626863] pci_bus 0000:02: busn_res: can not insert [bus 02-01] under [bus 01] (conflicts with (null) [bus 01])
[ 8.637317] pci_bus 0000:02: scanning bus
[ 8.637517] pci 0000:02:04.0: [13f6:8788] type 00 class 0x040100
[ 8.643693] pci 0000:02:04.0: reg 0x10: [io 0xffffff00-0xffffffff]
[ 8.650197] pci 0000:02:04.0: supports D1 D2
[ 8.656606] pci_bus 0000:02: fixups for bus
[ 8.656649] pci_bus 0000:02: bus scan returning with max=02
[ 8.656661] pci_bus 0000:02: busn_res: [bus 02-01] end is updated to 02
[ 8.663454] pci_bus 0000:02: busn_res: can not insert [bus 02] under [bus 01] (conflicts with (null) [bus 01])
[ 8.680096] pci 0000:01:00.0: devices behind bridge are unusable because [bus 02] cannot be assigned for them
[ 8.689506] pci_bus 0000:01: bus scan returning with max=02
[ 8.703127] pci 0000:00:00.0: bridge has subordinate 01 but max busn 02
[ 8.703139] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
[ 8.703147] pci_bus 0000:00: bus scan returning with max=01
[ 8.703167] pci 0000:01:00.0: BAR 13: no space for [io size 0x1000]
[ 8.719759] pci 0000:01:00.0: BAR 13: failed to assign [io size 0x1000]
[ 8.719767] pci 0000:02:04.0: BAR 0: no space for [io size 0x0100]
[ 8.719771] pci 0000:02:04.0: BAR 0: failed to assign [io size 0x0100]
[ 8.719774] pci 0000:01:00.0: PCI bridge to [bus 02]
[ 8.730394] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 8.780561] pcieport 0000:00:00.0: assign IRQ: got 61
[ 8.780596] pcieport 0000:00:00.0: enabling bus mastering
[ 8.780738] pcieport 0000:00:00.0: PME: Signaling with IRQ 61
[ 8.793529] pcieport 0000:00:00.0: AER: enabled with IRQ 61
[ 8.800190] pcieport 0000:00:00.0: saving config space at offset 0x0 (reading 0x271114e4)
[ 8.805636] pcieport 0000:00:00.0: saving config space at offset 0x4 (reading 0x100004)
[ 8.805640] pcieport 0000:00:00.0: saving config space at offset 0x8 (reading 0x6040010)
[ 8.805644] pcieport 0000:00:00.0: saving config space at offset 0xc (reading 0x10000)
[ 8.805648] pcieport 0000:00:00.0: saving config space at offset 0x10 (reading 0x0)
[ 8.805652] pcieport 0000:00:00.0: saving config space at offset 0x14 (reading 0x0)
[ 8.805655] pcieport 0000:00:00.0: saving config space at offset 0x18 (reading 0x10100)
[ 8.805659] pcieport 0000:00:00.0: saving config space at offset 0x1c (reading 0x20000000)
[ 8.805663] pcieport 0000:00:00.0: saving config space at offset 0x20 (reading 0xfff0)
[ 8.805669] pcieport 0000:00:00.0: saving config space at offset 0x24 (reading 0x1fff1)
[ 8.805673] pcieport 0000:00:00.0: saving config space at offset 0x28 (reading 0x0)
[ 8.805677] pcieport 0000:00:00.0: saving config space at offset 0x2c (reading 0x0)
[ 8.805681] pcieport 0000:00:00.0: saving config space at offset 0x30 (reading 0x0)
[ 8.805684] pcieport 0000:00:00.0: saving config space at offset 0x34 (reading 0x48)
[ 8.805688] pcieport 0000:00:00.0: saving config space at offset 0x38 (reading 0x0)
[ 8.805692] pcieport 0000:00:00.0: saving config space at offset 0x3c (reading 0x2013d)
[ 8.806243] pcieport 0000:01:00.0: assign IRQ: got 61
[ 9.402928] pci 0000:01:00.0: enabling bus mastering
[ 9.402941] snd_virtuoso 0000:02:04.0: invalid PCI I/O range
After I issued the setpci command I rescaned the bus and this is what dmesg shows after:
[ 144.734267] pci_bus 0000:00: scanning bus
[ 144.735147] pcieport 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
[ 144.735160] pci_bus 0000:01: scanning bus
[ 144.735928] pci 0000:01:00.0: scanning [bus 02-02] behind bridge, pass 0
[ 144.735937] pci_bus 0000:02: scanning bus
[ 144.736776] pci_bus 0000:02: bus scan returning with max=02
[ 144.736784] pci 0000:01:00.0: devices behind bridge are unusable because [bus 02] cannot be assigned for them
[ 144.746870] pci 0000:01:00.0: scanning [bus 02-02] behind bridge, pass 1
[ 144.746883] pci_bus 0000:01: bus scan returning with max=02
[ 144.746887] pcieport 0000:00:00.0: bridge has subordinate 01 but max busn 02
[ 144.754049] pcieport 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
[ 144.754056] pci_bus 0000:00: bus scan returning with max=01
lspci info doesn't change between 32bit and 64bit versions
01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 61
Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000f000-00000fff [disabled]
Memory behind bridge: fff00000-000fffff [disabled]
Prefetchable memory behind bridge: fff00000-000fffff [disabled]
Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr+ FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Power Budgeting <?>
02:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar DX)
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0 (500ns min, 6000ns max)
Interrupt: pin A routed to IRQ 61
Region 0: I/O ports at 0000
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel modules: snd_virtuoso
later edit:
Sadly I don't think this will be possible. this Raspberry Pi Engineer/Moderator says (https://forums.raspberrypi.com/viewtopic.php?p=1746705#p1746705) the bcm2711 datasheet mentions it doesn't support I/O space for PCIe. Only memory space. And that's correctly mapped. I see some I/O bar fails for a r8169 ethernet card, but that is working fine even so. But I can't see the Xonar in /proc/asound/cards
I'd never heard of this before until it came up at https://jamesachambers.com/pcie-1x-nvme-on-raspberry-pi-compute-module-4-guide/#comment-12952. John tried a Xawai adapter that I had previously tested with and had no trouble with. He is encountering the same error as RoboTakoh. Since I've used this adapter successfully before it can't be impossible like may be the case with the firewire card.
I can't explain why I never encountered it with the same adapter yet. Hopefully John will test some suggestions I had and some suggestions here. I may try doing it again with the latest Pi OS (I used 64 bit previously, one test I suggested was he try the 64 bit flavor to see if it makes any difference) to see if I can reproduce it on my end. I also linked him here to try some of the tweaks in the gist.
Most likely it just needs a firmware update as it sounds like this should be resolved and would explain why I never encountered it (I was late to the CM4 party).
Very strange issue, great work on this Jeff!
Thanks for the script @geerlingguy !
Do you know how to modify this on a PI5?
This change is necessary to make the Raspberry Pi Compute Module 4 handle many 3rd party PCIe adapters (e.g. GPUs, SATA cards, etc.), as they require more BAR (Base Address Register) PCI configuration space than is provided by default (~64 MB). The default amount was adequate for simpler devices, like the VL805 USB 3.0 controller used in the Raspberry Pi 4, but many PCIe devices require at least hundreds of MB of BAR space, so this script increases the limit to 1 GB.
The default limit may be increased in future versions of Raspberry Pi OS, since this change is fairly innocuous.
For some examples of devices that may need larger BAR space, see my Raspberry Pi PCIe device compatibility database