Debugging kernel issues with perf. Specifically used some of those for debugging our KVM networking issue. Extracted from instructions here: http://comments.gmane.org/gmane.comp.emulators.kvm.devel/113211

gistfile1.txt

This is using perf 3.4 which seems to be a little old and missing some options.

Establish probes and start recording:

$ perf probe start_xmit
$ perf probe virtqueue_kick
$ perf record -a -e net:net_dev_xmit -e net:net_dev_queue -e probe:start_xmit -e probe:virtqueue_kick -R&

  [...attempt to transmit packets inside guest...]

$ kill %1
$ perf script -i perf.data

Built kernel with CONFIG_DYNAMIC_DEBUG=y

$ echo -n 'module virtio_net +p' > /sys/kernel/debug/dynamic_debug/control
$ echo -n 'module virtio +p' >/sys/kernel/debug/dynamic_debug/control
$ echo 8 > /proc/sys/kernel/printk

Generating a core file

$ gdb
(gdb) generate-core-file [file]