perf is very handy, it comes with linux and works without a lot of setup (contrast: oprofile, systemtap)
probes work for kernel and userspace and you can record function arguments and local frame and other variables in the trace
you can use perf script to view the output of such traces, and it will display the variables bundled with the trace, by name, and with the format given when the probe was added
recently someone was asked to record what happened in systemd at source_dispatch, in sd-event; to see what the value of s->description was at the time, they were advised to use gdb