mproffitt/kmod-dbg.md

## kmod-dbg.md

      
    Raw
  

              kmod-dbg.md
            
          
    Kernel module debugging with gdb

This talks through the process of setting up and starting kernel level debugging for dynamic modules.
There is an order to be preserved in this that's easy to forget but once the session starts, is a pain to get back to.
Set up your base VM

In this example, I'm using ubuntu:22.10 base image from osboxes, running on virtualbox.
It is up to you the virtualisation layer you choose, your instructions may differ.

Download and configure the base image, don't forget to include the guest-additions.
These are requred for copy/pasting memory registers out of the VM.
Install any and all updates - you'll regret this later when your kernel version changes.

Install debug symbols and all relevant build tools

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
codename=$(lsb_release -c | awk  '{print $2}')
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename}      main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

Edit /etc/apt/sources.list and uncomment the relevant lines for:
deb-src http://us.archive.ubuntu.com/ubuntu/ kinetic main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ kinetic-updates universe

Now update apt and install packages
sudo apt update
sudo apt build-dep linux linux-image-$(uname -r)
sudo apt install libncurses-dev gawk flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf llvm

Once these are installed, we now need to install the debug-symbols for the running kernel
sudo apt install linux-image-$(uname -r)-dbgsym

Reboot to activate the kernel changes and then confirm that the kernel config contains the relevant sections for allowing
kernel level debugging to continue.
$ grep '\(CONFIG_FRAME_POINTER\|CONFIG_KGDB\|CONFIG_KGDB_SERIAL_CONSOLE\|CONFIG_KGDB_KDB\|CONFIG_KDB_KEYBOARD\)=' /boot/config-5.19.0-35-generic 
CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y

If it looks like this, we're mostly all set.
Edit /etc/default/grub and set GRUB_CMDLINE_LINUX as follows:
GRUB_CMDLINE_LINUX_DEFAULT=quiet splash kgdboc=ttyS0,115200 nokaslr


kgdboc=ttyS0,115200 Tell the debugger to be active on serial port ttyS0 and the baud rate at which it operates.
nokaslr Tells the kernel not to randomize the address space. this allows the debugger to continually
locate the function addresses correctly.

Now update the bootloader and shutdown the machine.
$ sudo update-grub
$ shutdown -h now

Edit th VM preferences and go to Serial Ports.

Go to tab Port 1.

Check Enable serial port.
Choose Port Number=COM1,
Port Mode=Host Pipe
Set Path/Address=/tmp/pipe
Ensure "Connect to existing pipe/socket" is not checked.


In this example, we're debugging a bluetooth USB device. It is preferred to use a separate bluetooth dongle
for this so as not to interfere with any additional bluetooth usage (such as music).

Map the local source to a path somewhere inside your VM
Ensure the VM can see any required USB devices (including bluetooth adaptors)

Restart the VM.
Debugging

Inside the VM open up a console and make sure the kernel module is loaded. You may need to compile this first
If so, you might want the following additional entries in your module Makefile
# Debugging symbols
MY_CFLAGS += -g -DDEBUG
ccflags-y += ${MY_CFLAGS}
CC += ${MY_CFLAGS}

modules_debug:
        EXTRA_CFLAGS="$(MY_CFLAGS)" $(MAKE) -C $(KDIR) M=$(PWD) modules

modules_debug_install:
        EXTRA_CFLAGS="$(MY_CFLAGS)" $(MAKE) -C $(KDIR) M=$(PWD) modules_install

In the example, I have Host path ~/src/digimend mapped to src/digimend inside the VM. This is useful as I can edit
the code natively in my own natural tools whilst compiling and running it inside the VM.
We need the vmlinux file from the VM available on the host. Copy this to the shared location.
cd ~/src/digimend
sudo cp /usr/lib/debug/boot/vmlinux-$(uname -r) vmlinux

Now build and install the kernel module from the same shared location and whilst still inside the VM.
sudo make modules_debug
sudo make modules_debug_install

It is important that this step is done inside the VM but that the kernel module is available on the host
alongside the vmlinux binary. These will both be used by gdb in the next step.
If this is an extension of an existing module, check that it's not loaded first, then load the recently compiled version
sudo rmmod hid_uclogic
sudo modprobe hid_uclogic

Next, we need to grab the memory location for this.
$ grep -e hid_uclogic /proc/modules | sed -n 1p | awk '{print $6}'
0xffffffffc0640000

Copy this to your clipboard as you will need it shortly.
Setup the debug session

Now we're ready to start debugging.

On the VM, turn on the kernel debugger. This will freeze the VM but don't worry, it will resume in a moment.
echo g > /proc/sysrq-trigger


On the host, run the following somewhere where it will not interfere with your session but is
accessible if you need to restart it
until $(socat -d /tmp/pipe TCP4-LISTEN:65335); do sleep 1; echo .; done

The until is useful if you restart and the socket is still in use. It will basically wait until it comes free
Go to your source location and start the debugger with the VMs vmlinux file attached.
cd ~/src/digimend
gdb ./vmlinux
...
Reading symbols from ./vmlinux...
(gdb)


Now we're inside the debugger and the kernel symbols are loaded, we need to load our kernel module.
Note. the memory address here is the one copied from the VM earlier.

gdb should be started from the location of the kernel module. In this instance it's built in ~/src/digimend.
Your location may differ.

(gdb) add-symbol-file hid-uclogic.ko 0xffffffffc0640000
add symbol table from file "hid-uclogic.ko" at
        .text_addr = 0xffffffffc0651000
(y or n) y

Now we have the module loaded, set up any breakpoints you desire. These need to go on the module symbol.
(gdb) b uclogic_probe
Breakboint 1 at 0xffffffffc0651790: file /home/osboxes/src/digimend/hid-uclogic-core.c, line 199.

Start the debug session

(gdb) target remote :65335
Remote debugging using :65335
warning: multi-threaded target stopped without sending a thread-id, using first non-exited thread
[Changing to Thread 4294967294]
kgdb_breakpoint() at /build/linux-tGckOR/linux-5.19.0/kernel/debug/debug_core.c:1231

This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.ubuntu.com
Enable debuginfod for this session? (and or [n]) n
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
1231 /build/linux-tGckOR/linux-5.19.0/kernel/debug/debug_core.c: Success.
Continue the debugger and carry out any action required to trigger your breakpoints (such as turning on the device)
(gdb) cont
Continuing


At this point, gdb will hang until the breakpoint is hit but your VM will have resumed from its stall.
The debugger may also hang indefinitely if you do not load the same symbols into the debugger as are running in the VM.
This is why it is important to load the files from the same location on both the host and the VM.
If your kernel changes, you'll need to download a new copy of vmlinux
If you rebuild your module, you need to reload the new symbols into the debugger.
This is a slow and painful, but highly rewarding process if done right. Have patience.