Skip to content

Instantly share code, notes, and snippets.

@smunaut

smunaut/gist:933c674bce9301fee8fa85f224670705 Secret

Last active February 24, 2022 09:28
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save smunaut/933c674bce9301fee8fa85f224670705 to your computer and use it in GitHub Desktop.
Save smunaut/933c674bce9301fee8fa85f224670705 to your computer and use it in GitHub Desktop.
Download ZIP
LitePCIe behavior
Raw
gistfile1.md

Initial testing Feb 22:

  • Card plugged in mobo PCIe1 (16x)

    • Only detected if bitstream is loaded after system is booted and trigger rescan
    • Not detected if booted with bitstream loaded
    • Doesn't actual work ( trying to show ID report \xff.... )
    • Link speed reported as 2.5GT/s (gen1)
    • Link width is correct

  • Card plugged in mobo PCIe3 (1x) via a 'usb cable extender'

    • Needs to limit link speed to gen2. In gen3, tons of errors are reported.
    • Card detected fine and ID report is correct, dma_test works -> It works !
    • Link speeds reported as 5GT/s (expected, limited on purpose in bios)
    • Link width is 1x (expected ... that slot it 1x only)

  • Card plugged in an external PCIe switch, the PCIe switch connected to mobo PCIe3

    • PCIe switch shows up correcty with expected link speed
    • Card behavior is the same as when it's plugged in PCIe1 directly (the mobo x16 slot): i.e. not working
    • (note: The PCIe presence jumper needs to be set to 1x or the pcie switch doesn't see the card)

  • Card plugged in an external PCIe switch, the PCIe switch connected to mobo PCIe1

    • Same behavior as if switch plugged in PCIe1

  • Card plugged in mobo PCIe1 (16x) via a 'usb cable extender' (limiting to 1x)

    • Same behavior as direct connection expect link width is 1x

Update Feb 23:

So I think I've got several issues in my PCIe setup and that's why the symptoms are weird and the different cases result make little sense.

  • If the card isn't detected at boot, the bios seems to not bother to configure the PCIe root port. So I have to manually write the LinkControl register in linux to set it up properly to get gen3 support and get it to train properly.

  • When doing a pcie rescan, even if it detects the device ... linux is dumb as a brick and will not configure the memory zone through the upstream switches/root ports, they remain [disabled]. Solution for that is to do a delete of the pcie root port where the card is plugged, and then do a rescan. When linux will re-add the bridge, it will then properly configure it for the downstream devices.

  • For some reason, once the PCIe core has trained once ... it cannot go through a reboot cycle, that will lock it up. So if I configure it before the machine is started, then boot, it's fine. Or if I boot with the card unconfigured and do it all once in linux, that works too. But if I get the card up and try a warm boot, it will never be seen ever again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment