Skip to content

Instantly share code, notes, and snippets.

@elFarto
Last active May 27, 2023 10:52
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save elFarto/1f9ba845e5ba3539a2c914aae1f4a1e4 to your computer and use it in GitHub Desktop.
Save elFarto/1f9ba845e5ba3539a2c914aae1f4a1e4 to your computer and use it in GitHub Desktop.
Remotely debugging PI over a serial connection

The first thing you'll need is someway to connect to the serial port on the Pi. This can be done with a USB to UART adapter (preferably one that has a individual DuPont connnectors), or via a second Pi (in which case you'll need three DuPont jumper wires, with female ends).

You'll want to be running GDB from the machine/VM that's also building the kernel, as the debugger will need access to all the source code. If you're running the build from a VM, and using a USB-serial adapter you'll need to make sure you can add this device to the VM.

note In theory you should have a cross-compiled GDB, one that runs on your system that is targetted to arm64. I believe Ubuntu has the 'gdb-multiarch' package available that contains a compatible version (I did my testing on Fedora). However, I didn't seem to have any issues using my x86-64 GDB to debug it. It might only be an issue when it needs to disassemble instructions.

The first step is to recompiling the kernel for the target Pi you want to debug. Ensure these config options are set:

#compiles with debug symbols enabled
CONFIG_DEBUG_INFO=y
#enables auto-loading the gdb script
CONFIG_GDB_SCRIPTS=y

After you've finished compiling the kernel, you'll also need to do make scripts_gdb (with whatever options you need to pass for cross-compiling). This will create the files needed for the GDB extension, which is really something you need to debug easily. This should leave you with a vmlinux-gdb.py file in the kernel directory (this will be used later).

The second step is to enable the correct UART on the Pi, and enable kernel debugging on it. Refer to [1], and make sure you have a UART that has AMA in it's name (that is, ignore the mini-UARTs), and it isn't being used by the bluetooth adapter. For my Pi1, I didn't have to do anything. For my Pi3 I needed to disable bluetooth using an overlay. This gave me /dev/ttyAMA0 connected to GPIOs 14 and 15. You can use the additional UARTs on the Pi4, but you'll need to check which GPIOs/pins they use[2].

Once that's done, you can edit the /boot/cmdline.txt file and add the following, where ttyAMA0 is the UART you've configured above:

kgdboc=ttyAMA0,115200

Also ensure you have a matching console line for that UART (yours might refer to serial0, see [1] for the aliases that are created):

console=ttyAMA0,115200

Reboot the Pi, you should now be able to connect to it over serial from your host. Depending on when you connect the Pi, you might need to press enter to get a login prompt.

If you're using a second Pi as a host to connect the target, you'll also need to configure that Pi's UARTs aswell (you don't need to touch the cmdline). When connecting the Pis together, connect the Rx pin of each the the Tx pin of the other and be sure to connect their GND pins together (any GND pins will do), you will get weird errors if you don't do this.

If you want to just use the second Pi as a proxy (effectively just using it as a network connected serial port), you can download [3], which will expose the serial port as a socket. Once compiled, you can run ./agent-proxy 5550^5551 0 /dev/ttyAMA0,115200 (changing UARTs as necessary). You will need to telnet to the first port (5550), before you can debug with the second port (5551). (You can also use agent-proxy when connected directly to the target Pi, which negates the need to do the sys-rq magic to drop into debug mode.)

Now for the third step, connecting the debugger. On your host (where you're building the kernel), change to the directory where you built the kernel and run: gdb vmlinux

Once you have a '(gdb)' prompt you can connect to the target. If you're connected directly, you'll firstly need to drop the kernel into debug mode by running this on the target Pi (as root):

echo g > /proc/sysrq-trigger

Then on your host you can type this into GDB:

target remote /dev/ttyS0

Where ttyS0 is the serial port on the host that's connected to the Pi (it'll be one of the ttyAMA on a Pi, or maybe one of ttyUSB if you're using an adapter).

If you're connecting via a second Pi, type:

target remote <second pi's IP>:5551

Where 5551 is the second port your configured when starting agent-proxy. There's no need to run the sysrq-trigger command above, as the agent-proxy will deal with that for you.

Once you're connected to it, it should pause execution in the kernel. Now type this into GDB:

lx-symbols

If this gives you a command not found, you might need to add your kernel build directory to the gdb config. In ~/.gdbinit add, replacing the path with the correct one, and restarting gdb:

add-auto-load-safe-path /path/to/kernel/build

The lx-symbols command loads all the symbols from your local kernel build. You are now ready to debug. Set a breakpoint somewhere close to where it's crashing, preferably on the function name (although file.c:lineno should work aswell):

b amdgpu_ring_init

At this point it will say that it can't find that symbol, do you want to add it when modules are loaded, say y to this. Now type c to continue execution.

Now you can modprobe the module from a separate SSH connection to the target Pi, and GDB should break at entry into that method. You can now step through line by line using the s command (you don't need to type the command everytime, pressing enter will repeat the last command entered while stepping through, so you should literally be able to hold enter until it crashes although it might not be quick enough to keep up).

That should be enough to get you started. There's some more information about the kernel debugging here [4]. And some more information on Raspberry Pi serial/UART connections here [5].

[1] https://www.raspberrypi.org/documentation/configuration/uart.md
[2] https://raspberrypi.stackexchange.com/questions/104464/where-are-the-uarts-on-the-raspberry-pi-4
[3] https://git.kernel.org/pub/scm/utils/kernel/kgdb/agent-proxy.git
[4] https://www.kernel.org/doc/html/v5.9/dev-tools/kgdb.html
[5] https://elinux.org/RPi_Serial_Connection

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment