Skip to content

Instantly share code, notes, and snippets.

@nuumio

nuumio/001-rp64-uboot-tests-2021-11-2021-12.md

Last active December 5, 2021 08:35
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nuumio/a89369122cc3fa815cc14e9efed065f9 to your computer and use it in GitHub Desktop.
Save nuumio/a89369122cc3fa815cc14e9efed065f9 to your computer and use it in GitHub Desktop.
Download ZIP
ROCKPro64 uboot testing 2021-11 - 2021-12
Raw
001-rp64-uboot-tests-2021-11-2021-12.md

Hosts under test, other equipment used in tests

Few "pre-notes"

  • more pics:

  • guineapig had trouble booting with "rescue sd" having uboot 2020.01-ayufan-2013-gdb96b24f never got to login with that. Getting a lot of this:

    Starting kernel ...
"Synchronous Abort" handler, esr 0x02000000
elr: ffffffff0f6eb000 lr : 0000000000202354 (reloc)
elr: 0000000003410000 lr : 00000000f3f27354
x0 : 00000000f09e4000 x1 : 0000000000000000
x2 : 0000000000000000 x3 : 0000000000000000
x4 : 0000000002080000 x5 : 0000000000000001
x6 : 0000000000000008 x7 : 0000000000000000
x8 : 00000000f1f49f60 x9 : 0000000000000002
x10: 000000000a200023 x11: 0000000000000002
x12: 0000000000000002 x13: 00000000f1f1cdde
x14: 000000000000c768 x15: 00000000f3f26a68
x16: 0000000000000000 x17: 0000000000000080
x18: 00000000f1f1cdc8 x19: 00000000f3fe4f20
x20: 0000000000000000 x21: 0000000000000000
x22: 0000000000000003 x23: 00000000f1f08cc8
x24: 0000000000000003 x25: 00000000f1f08cc8
x26: 00000000f3fcf190 x27: 00000000f3fc6ef6
x28: 00000000f3fbd341 x29: 00000000f1f08a80
Code: 00000000 00000000 00000000 00000000 (cccccccc) 
Resetting CPU ...
resetting ...
    • it did start kernel few times but didn't get to login
    • it booted just fine with "retrogator sd" though (uboot 2021.01, not sure how patched, buildroot thingy)

How to tell difference between board models

In addition to differences in components (see picture links above) boards have different RAM chips and sdram_print_ddr_info() in uboot's drivers/ram/rockchip/sdram_common.c shows a little bit different info for them:

  • guineapig / "old" model:

    U-Boot TPL 2021.10-3 (Nov 27 2021 - 06:29:26)
Channel 0: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
Channel 1: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
256B stride

  • crashtestdummy / "new" model:

    U-Boot TPL 2021.10-3 (Nov 27 2021 - 06:29:26)
Channel 0: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=16/15 CS=1 Die BW=16 Size=2048MB
Channel 1: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=16/15 CS=1 Die BW=16 Size=2048MB
256B stride

  • diff:

    --- guineapig-tpl-ram.txt	2021-12-04 15:45:27.249412185 +0200
+++ crashtestdummy-tpl-ram.txt	2021-12-04 15:45:28.156086960 +0200
@@ -1,6 +1,6 @@
U-Boot TPL 2021.10-3 (Nov 27 2021 - 06:29:26)
Channel 0: LPDDR4, 50MHz
-BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
+BW=32 Col=10 Bk=8 CS0 Row=16/15 CS=1 Die BW=16 Size=2048MB
Channel 1: LPDDR4, 50MHz
-BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
+BW=32 Col=10 Bk=8 CS0 Row=16/15 CS=1 Die BW=16 Size=2048MB
256B stride

Test command and getting results

# HUT is guineapig or crashtestdummy
# Just keep rebooting HUT until stopped
HUT=host_under_test; \
RB=1; \
while true; do \
  echo "Reboot $RB"; \
  let RB=RB+1; \
  ssh ${HUT} "date; cat /sys/class/thermal/thermal_zone?/temp; sudo reboot" 2>&1 | sed 's/^/  /'; \
  echo "  Wait ${HUT} down..."; \
  while timeout 0.5 ping -c1 -n ${HUT} &> /dev/null; do \
    sleep 0.5; \
  done; \
  echo "  Wait ${HUT} up...";
  while ! timeout 0.5 ping -c1 -n ${HUT} &> /dev/null; do \
    sleep 0.5; \
  done; \
  sleep 5; \
done

Manual reset is done on HUT if it's found getting stuck on boot.

  • getting results from serial console logs:

    • (using strings in case grep thinks it's binary)

    • number of all boots (rp64-uboot-tests-202111-xxx is added to CONFIG_IDENT_STRING):

      $ strings uart.log | grep rp64-uboot-tests-202111 | wc -l

    • ok boots (it hangs before Starting kernel in case of failure)

      $ strings uart.log | grep "Starting kernel" | wc -l

Test notes

Format changes in test notes:

  • 2021-12-04: Tests for each patch as 2nd level heading instead of list. As the doc is getting longer this makes it easier to use results in chat / forums / issue trackers because headings can be added to href.

All builds are based on Manjaro uboot-rockpro64 2021.10-1 unless otherwise noted.

test-001 & test-002: First reproducing the original issue

test-003: Manjaro uboot, no extra patches added

  • test notes:

test-004: with enable emmc phy patch

test-005: with emmc phy patch and PREBOOT enabled

  • test notes:
    • building:
    • test results (just unpacked packages and dd'ed uboot files):

      • guineapig: 3 OK, 35 FAIL

      • crashtestdummy: 50 OK, 0 FAIL

      • guineapig failed like with the first test package except it happened more frequently, got stuck at:

        75433 bytes read in 37 ms (1.9 MiB/s)
Moving Image from 0x2080000 to 0x2200000, end=3be0000
## Flattened Device Tree blob at 01f00000
  Booting using the fdt blob at 0x1f00000

test-006: with emmc phy patch, PREBOOT enabled and with "PBP config"

test-007: Icenowy's patch (BAD BUILD)

  • test notes:
    • NOTE: this run is bad, forgot to apply the patch in PKGBUILD. But forgotten USB device in crashtestdummy created interesting results. Testing Icenowy's patch suggested by sigmaris on #pine64 / #rock64 chat: that'll be Manjaro uboot-rockpro64 2021.10 with dts-rockchip-rk3399-enable-emmc-phy-for-spl.patch and Icenowy's patch, preboot enabled (will be build on top of test-005).
    • downloaded: https://github.com/sigmaris/u-boot/commit/a311daabe9983aa55322341fb3cf6a8851f73163.patch as 0004-phy-rockchip-inno-usb2-fix-hang-when-multiple-controllers-exit.patch (sha256: 0ddee85c2a26c9650a2177465db1e53d0b536cf76e29096bc9849505137471f5)
    • building:
    • test results (just unpacked packages and dd'ed uboot files):

      • guineapig: 2 OK, 22 FAIL

      • crashtestdummy: 33 OK, 19 FAIL

      • NOTE: I forgot to remove a Logitech unifying receiver from crashtestdummy's upper USB-2 port before testing. While it the receiver was connected crashtestdummy suffered from the same "getting stuck" problem as guineapig. After removing the receiver crashtestdummy started booting just fine. All 19 failed cases were before that.

      • This is the where they both got stuck (before the receiver was removed from crashtestdummy):

        75433 bytes read in 36 ms (2 MiB/s)
Moving Image from 0x2080000 to 0x2200000, end=3be0000
## Flattened Device Tree blob at 01f00000
  Booting using the fdt blob at 0x1f00000

      • NOTE: this run (test-007) is bad, forgot to apply the patch in PKGBUILD. But forgotten USB device in crashtestdummy created interesting results.

test-008: Icenowy's patch (FIXED BUILD)

  • test notes:
    • Testing Icenowy's patch suggested by sigmaris on #pine64 / #rock64 chat: that'll be Manjaro uboot-rockpro64 2021.10 with dts-rockchip-rk3399-enable-emmc-phy-for-spl.patch and Icenowy's patch, preboot enabled (will be build on top of test-007).
    • Icenowy's patch already downloaded on test-007
    • building:
    • test results (just unpacked packages and dd'ed uboot files):
      • test set 001: no usb gadgets
        • guineapig: 100 OK, 0 FAIL
        • crashtestdummy: 100 OK, 0 FAIL
      • test set 002: Logitech unifying receiver on upper USB 2 port, new receiver on guineapig, old on crashtestdummy, keyboard was not connected to receiver until the final round (one kb attached to another receiver, another powered off)
        • guineapig: 51 OK, 0 FAIL
        • crashtestdummy: 51 OK, 0 FAIL
        • Keyboard via the receiver works on uboot on both hosts. Tried only once on each host:
          • Stopped uboot's autoboot using keyboard
          • Booted with command: boot
      • test set 003: Logitech unifying receiver on USB 3 port, new receiver on guineapig, old on crashtestdummy

        • guineapig: 51 OK, 0 FAIL

        • crashtestdummy: 51 OK, 0 FAIL

        • Keyboard via the receiver works on uboot only on guineapig (older board). Tried only once on each host:

          • Stopped uboot's autoboot using keyboard
          • Booted with command: boot

        • crashtestdummy (newer board, older receiver) fails to get device descriptor for USB device in uboot.

        • uboot USB output for guineapig, works ok:

          starting USB...
Bus usb@fe380000: USB EHCI 1.00
Bus usb@fe3a0000: USB OHCI 1.0
Bus usb@fe3c0000: USB EHCI 1.00
Bus usb@fe3e0000: USB OHCI 1.0
Bus usb@fe800000: usb maximum-speed not found
Register 2000140 NbrPorts 2
Starting the controller
USB XHCI 1.10
Bus usb@fe900000: usb maximum-speed not found
Register 2000140 NbrPorts 2
Starting the controller
USB XHCI 1.10
scanning bus usb@fe380000 for devices... 1 USB Device(s) found
scanning bus usb@fe3a0000 for devices... 1 USB Device(s) found
scanning bus usb@fe3c0000 for devices... 1 USB Device(s) found
scanning bus usb@fe3e0000 for devices... 1 USB Device(s) found
scanning bus usb@fe800000 for devices... 1 USB Device(s) found
scanning bus usb@fe900000 for devices... 2 USB Device(s) found
      scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  2

        • uboot USB output for crashtestdummy, fails:

          starting USB...
Bus usb@fe380000: USB EHCI 1.00
Bus usb@fe3a0000: USB OHCI 1.0
Bus usb@fe3c0000: USB EHCI 1.00
Bus usb@fe3e0000: USB OHCI 1.0
Bus usb@fe800000: usb maximum-speed not found
Register 2000140 NbrPorts 2
Starting the controller
USB XHCI 1.10
Bus usb@fe900000: usb maximum-speed not found
Register 2000140 NbrPorts 2
Starting the controller
USB XHCI 1.10
scanning bus usb@fe380000 for devices... 1 USB Device(s) found
scanning bus usb@fe3a0000 for devices... 1 USB Device(s) found
scanning bus usb@fe3c0000 for devices... 1 USB Device(s) found
scanning bus usb@fe3e0000 for devices... 1 USB Device(s) found
scanning bus usb@fe800000 for devices... 1 USB Device(s) found
scanning bus usb@fe900000 for devices... unable to get device descriptor (error=-1)
1 USB Device(s) found
      scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  2

        • after a bit more testing it seems that old receiver doesn't work on USB3 port on either host in uboot, works ok in linux. New receiver works in all situations.

test-009: dsimic's 1st patch

  • test notes:
    • Testing dsimic's 1st patch which replaces Icenowy's patch
    • download https://termbin.com/s4nd as 0005-dsimic-0001.patch (sha256: 00994f5b43b486452cf25e9e6f9b0d8788f694f0a413d6432e351b2d57a6015b)
    • building:
    • test results (just unpacked packages and dd'ed uboot files):
      • test set 001: no usb gadgets
        • guineapig: 3 OK, 7 FAIL
        • crashtestdummy: 10 OK, 0 FAIL

test-010: dsimic's 2nd patch

  • test notes:
    • Testing dsimic's 2nd patch which replaces Icenowy's patch
    • download https://termbin.com/frnh as 0005-dsimic-0002.patch (sha256: 5fab15500d7234450d2062e9e87fb708e2599e602998ea7aa9c3ab0141b1927b)
    • building:
    • test results (just unpacked packages and dd'ed uboot files):

      • test set 001: no usb gadgets

        • guineapig boot to linux: 150 OK, 0 FAIL
        • crashtestdummy boot to linux: 150 OK, 0 FAIL

      • test set 002: older model logitech unifying receiver on upper USB2 port + logitech k360 keyboard

        • guineapig boot to linux: 5 OK, 0 FAIL
          • keyboard in uboot: 5 OK, 0 FAIL (manually tested, can stop autoboot)
        • crashtestdummy boot to linux: 5 OK, 0 FAIL
          • keyboard in uboot: 5 OK, 0 FAIL (manually tested, can stop autoboot)

      • test set 003: older model logitech unifying receiver on USB3 port + logitech k360 keyboard

        • guineapig boot to linux: 5 OK, 0 FAIL

          • keyboard in uboot: 0 OK, 5 FAIL (manually tested, can NOT stop autoboot)

        • crashtestdummy boot to linux: 5 OK, 0 FAIL

          • keyboard in uboot: 0 OK, 5 FAIL (manually tested, can NOT stop autoboot)

        • on both boards usb port scan failed like with Icenowy's patch (this was expected):

          USB XHCI 1.10
scanning bus usb@fe380000 for devices... 1 USB Device(s) found
scanning bus usb@fe3a0000 for devices... 1 USB Device(s) found
scanning bus usb@fe3c0000 for devices... 1 USB Device(s) found
scanning bus usb@fe3e0000 for devices... 1 USB Device(s) found
scanning bus usb@fe800000 for devices... 1 USB Device(s) found
scanning bus usb@fe900000 for devices... unable to get device descriptor (error=-1)

      • test set 004: newer model logitech unifying receiver on upper USB2 port + logitech craft keyboard

        • guineapig boot to linux: 5 OK, 0 FAIL
          • keyboard in uboot: 5 OK, 0 FAIL (manually tested, can stop autoboot)
        • crashtestdummy boot to linux: 5 OK, 0 FAIL
          • keyboard in uboot: 5 OK, 0 FAIL (manually tested, can stop autoboot)

      • test set 005: newer model logitech unifying receiver on USB3 port + logitech craft keyboard

        • guineapig boot to linux: 5 OK, 0 FAIL
          • keyboard in uboot: 5 OK, 0 FAIL (manually tested, can stop autoboot)
        • crashtestdummy boot to linux: 5 OK, 0 FAIL
          • keyboard in uboot: 5 OK, 0 FAIL (manually tested, can stop autoboot)

      • these test verify that behavior on RockPRO64 with dsimic's 2nd patch is equal to Icenowy's patch.

        • newer model logitech unifying receiver works fine on both USB2 and USB3 ports
        • older model fails in USB3 port

Conclusions / final notes, ROCKPro64 + U-Boot 2021.10 + eMMC boot

  • Doesn't work at all without patches (test round 003)
  • Newer boards (crashtestdummy) work with enable-emmc-phy-for-spl.patch (test round 005)
  • Older boards (guineapig) need USB hacks in addition to enable-emmc-phy-for-spl.patch
  • The root cause of these boot problems is currently unknown but USB is at least somehow related.
  • Boot problems started with 2020.10 (2020.07 was ok, iirc) and it feels like the problem is even worse on 2021.10 (only few succesful boots on old board).
  • I probably have third variant of ROCKPro64 boards too because the "rescue sd" didn't work on the old board / guineapig, but I'm quite confident that it works on two boards which are even older (from 2018).
  • Icenowy's patch fixes USB related boot issues
  • Keyboard via old logitech unifying receiver in USB 3 port doesn't work in uboot, new receiver works. Both work fine in linux in all ports.
  • dsimic's 2nd patch fixes USB related boot issues. Details in test-010: dsimic's 2nd patch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment