Debugging kernel code line by line with Qemu and GDB

debugging_kernel.txt

We are going to use buildroot to download, configure and compile the kernel.

First, download  and uncompress buildroot: https://buildroot.org/download.html

Go to the directory and:

make qemu_x86_defconfig

make menuconfig

In Build options:
  - Select 'build packages with debugging symbols'
  - Go to 'gcc debug level' and select level 3
  - Go to 'strip command for binaries on target' and select none
  - Goto to 'gcc optimization level' and select 'optimize for debugging'
  
In Kernel:
  - Go to 'Kernel version' and select 'Latest version'

Save and exit

make
make linux-menuconfig
Select 'CONFIG_DEBUG_INFO'
make

Start virtualization:
qemu-system-i386 -kernel output/images/bzImage -hda output/images/rootfs.ext2 -append "root=/dev/sda rw" -s -S &

Start GDB:
gdb

Then run the commands:
  file ./output/build/linux-4.7.2/vmlinuz
  target remote :1234
  hbreak start_kernel
  continue
  
Done. You are at the kernel's "main" function.

================================================================================

You can also do this by using VMware Workstation:

- Go to your virtual machine settings
- Look for the configuration file path
- add debugStub.listen.guest64 = "1" for x86_64 kernels or debugStub.listen.guest32 = "1" for 32 bits
- The log file will be showing the following line now:
      Debug stub: VMware Workstation is listening for debug connection on port 8864
      
Source: http://stackframe.blogspot.com.br/2007/04/debugging-linux-kernels-with.html

Debugging Linux kernels with Workstation 6.0
We just quietly added an exciting feature to Workstation 6.0. I believe it will make WS6 a great tool for Linux kernel development. You can now use gdb on your host to debug the Linux kernel running inside the VM. No kdb, no recompiling and no need for second machine. All you need is a single line in VM's configuration file.

To use the new feature, grab the latest build of Workstation here, or free 30-day evaluation here. Put this line into configuration file of your Linux VM:

debugStub.listen.guest32=1

Now whenever you run the virtual machine, you'll see the following in the vmware.log file (debug builds will also print this message to Host console):

VMware Workstation is listening for debug connection on port 8832.

Run gdb on the Host, reference it to the kernel with symbols and attach to the virtual machine:

% gdb
(gdb) file vmlinux-2.4.21-27.EL.debug
(gdb) target remote localhost:8832

That's it. The VM is blocked now, so you can "continue" it and "^C" back to gdb. Breakpoints, single step, memory inspection - all this works as usual. If you have SMP VM, then each VCPU is mapped on a thread, so use "info threads" and "thread NN" to switch between them.

Debugging the 64-bit kernel works in the same way, except you need to use a different option:

debugStub.listen.guest64=1

and connect to port 8864. Since gdb starts in 32-bit mode by default, you may also need to switch it to i386:x64-64 before connecting:

(gdb) set architecture i386:x86-64
(gdb) target remote localhost:8864

The kernels with symbols are sadly lacking on most distributions, but if you use RHEL then this website may help (look for kernel-debuginfo rpm):

http://people.redhat.com/duffy/debuginfo/index-js.html

The gdb support in WS6 is experimental, so there may be rough edges here and there. Please post on community forums if something doesn't work right or if you have a suggestion:

http://communities.vmware.com/community/vmtn/general/guestdebugmonitor

There are more debugging specific features in WS6 (for example, you can use gdb hand-in-hand with Record/Replay!). I will describe them shortly.

Updated 4/20/07: added explanation of 64-bit support.
Updated 5/14/07: release build prints "waiting for gdb" message into vmware.log only.
Updated 7/24/07: pointers to new build and discussion forum.

http://blog.dynofu.me/post/2015/10/08/linux-live-debugging-with-kgdb-vmware-workstion.html

Thank you so much for this wonderful tutorial.
:)

I cannot stop at start_kernel, and fixed it with following kernel configuration.

"You should disable KASLR in your kernel command line with nokaslr option, or disable kernel option "Randomize the kernel memory sections" inside "Processor type and features" when you build your kernel image."

Related setup: https://github.com/cirosantilli/linux-kernel-module-cheat

Suppose a little typo for following content: ./output/build/linux-4.7.2/vmlinuz -> ./output/build/linux-4.7.2/vmlinux
Start GDB:
gdb

Then run the commands:
file ./output/build/linux-4.7.2/vmlinuz
target remote :1234
hbreak start_kernel
continue

Can you cross debug an arm guest linux kernel from a X86 host with your solutions?

Can you cross debug an arm guest linux kernel from a X86 host with your solutions?

http://doppioandante.github.io/2015/07/10/Simple-ARM-programming-on-linux.html

@hngouveia01
Hi Sir,

I followed your these steps to enable debug:

make linux-menuconfig
Select 'CONFIG_DEBUG_INFO'
make

In the menu, where is CONFIG_DEBUG_INFO, I cannot find it, can you please provide the exact steps?
Instead, I add CONFIG_DEBUG_INFO=y to .config file, but the generated vmlinux did not have debug symbols.
Thanks in advance!

@zlkneron, which kernel version are you trying to debug?
Things changed a bit from the initial kernel version to, lets say, kernel 5.
https://unix.stackexchange.com/questions/712513/why-is-the-flag-config-debug-info-reset-when-building-the-linux-kernel

Hello, I followed this document, use vmware to debug linux kernel 5.15.0, set debugStub.listen.guest64=1，but I can’t stop at start_kernel, kaslr has already disable，I use vmware 12.

Hi, @wangjinnana. Are you using network (port) or serial debugging?
Double check the values for the config here.
I see that this conf is set to “TRUE”. Does VMWare Workstation prints a message "VMware Workstation is listening for debug connection on port 8832." into the vmware.log file?

https://communities.vmware.com/t5/VMware-Fusion-Discussions/Using-debugStub-to-debug-a-guest-linux-kernel/td-p/394906?darkschemeovr=1

https://wiki.osdev.org/VMware

Yeah, I use network, vmware log the port is listened, I can break on syscall, but I can’t stop at start_kernel

To debug the start_kernel function in VMware 12, follow these steps to ensure your kernel is configured to load symbols:

1. Enable Debugging Options:
   Open the kernel configuration menu using one of the following commands:

   make menuconfig   # Text-based configuration
   make xconfig      # Graphical configuration (requires additional packages)

2. In the configuration menu, enable the following options:

   CONFIG_DEBUG_INFO=y
   CONFIG_DEBUG_KERNEL=y
   CONFIG_DEBUG_INFO_DWARF4=y  # For DWARF debug info format

   Save your configuration and exit the menu.

3. Build the Kernel:
   Compile the kernel with the new configuration:

   make -j$(nproc)
   make modules
   make modules_install
   make install

4. Copy Kernel Image and System.map:
   Copy the compiled kernel image and System.map file to the appropriate location. Replace <kernel_version> with the actual version:

   cp arch/x86/boot/bzImage /boot/vmlinuz-<kernel_version>
   cp System.map /boot/System.map-<kernel_version>

5. Update GRUB:
   Update your bootloader configuration to include the new kernel and System.map:

   update-grub  # For GRUB
   grub-mkconfig -o /boot/grub/grub.cfg  # For GRUB2

6. Configure VMware for Kernel Debugging:
   In VMware 12, ensure that the following settings are configured in your virtual machine:

   • Set the kernel command line parameters to include debugging options. Edit the GRUB configuration file or use the kernel command line during boot to add:

     kgdbwait kgdboc=ttyS0,115200

     This waits for the debugger to attach and directs kernel messages to the specified serial port.

   • Make sure the virtual machine's serial port is configured for debugging. In VMware settings, add a serial port and set it to use named pipe or TCP, depending on your preference.

7. Reboot:
   Reboot your virtual machine to load the newly built kernel with debug symbols.

8. Attach Debugger:
   Use your preferred debugger (e.g., GDB) to attach to the virtual machine's serial port. You may use a tool like socat or nc to connect the debugger to the named pipe or TCP port you configured in VMware.

Yeah, I use network, vmware log the port is listened, I can break on syscall, but I can’t stop at start_kernel

Also, you could try to configure a second vm and try to debug from there.