Skip to content

Instantly share code, notes, and snippets.

@djanatyn

djanatyn/bpf-notes.md

Created October 12, 2022 22:08
Show Gist options
  • Save djanatyn/7a2784e477be403a15a8a2215a97b9bf to your computer and use it in GitHub Desktop.
Save djanatyn/7a2784e477be403a15a8a2215a97b9bf to your computer and use it in GitHub Desktop.
Download ZIP
reading PF Performance Tools
Raw
bpf-notes.md

I started diving into the BPF Performance Tools book, and there's so many utilities from the BCC that I wasn't aware of!

execsnoop does what i always wanted ps to do

❯ sudo execsnoop
PCOMM            PID    PPID   RET ARGS
uname            252856 1994     0 /run/current-system/sw/bin/uname
sh               252857 2544     0 /bin/sh -c tmux     send-keys -X cancel; /home/djanatyn/.tmux/plugins/tmux-copycat/scripts/copycat_mode_quit.sh; true
tmux             252858 252857   0 /nix/var/nix/profiles/default/bin/tmux send-keys -X cancel
copycat_mode_qu  252859 252857   0 /home/djanatyn/.tmux/plugins/tmux-copycat/scripts/copycat_mode_quit.sh
bash             252859 252857   0 /run/current-system/sw/bin/bash /home/djanatyn/.tmux/plugins/tmux-copycat/scripts/copycat_mode_quit.sh
dirname          252861 252860   0 /run/current-system/sw/bin/dirname /home/djanatyn/.tmux/plugins/tmux-copycat/scripts/copycat_mode_quit.sh
tmux             252865 252864   0 /nix/var/nix/profiles/default/bin/tmux display-message -p #{session_id}-#{window_index}-#{pane_index}
sed              252866 252864   0 /run/current-system/sw/bin/sed s/\$//
tmux             252867 252862   0 /nix/var/nix/profiles/default/bin/tmux show-option -gqv @copycat_mode_0-1-1
uname            252868 1994     0 /run/current-system/sw/bin/uname

The output reveals which processes were executed while tracing: processes that may be so short-lived that they are invisible to other tools.

I ran execsnoop when the system was supposed to be idle, and discovered that it wasn't! Every second these processes were launched, and they were perturbing our benchmarks. The cause turned out to be a misconfigured service that was attempting to launch every second, failing, and starting again.

The output from execsnoop aids a performance analysis methodology called workload characterization, which is supported by many other BPF tools in this book.

Try running execsnoop on your systems and leave it running for an hour. What do you find?

biolatency...summarizes block device I/O (disk I/O) as a latency histogram.

❯ sudo biolatency
Tracing block device I/O... Hit Ctrl-C to end.
^C
     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 14       |******                                  |
        16 -> 31         : 82       |****************************************|
        32 -> 63         : 30       |**************                          |
        64 -> 127        : 6        |**                                      |
       128 -> 255        : 0        |                                        |
       256 -> 511        : 0        |                                        |
       512 -> 1023       : 0        |                                        |
      1024 -> 2047       : 3        |*                                       |
      2048 -> 4095       : 1        |                                        |
      4096 -> 8191       : 0        |                                        |
      8192 -> 16383      : 38       |******************                      |

You can use dumpcap to view the instructions generated for a packet filter you'd use in tcpdump:

❯ sudo dumpcap -d -f 'tcp port 80'
Capturing on 'enp5s0'
(000) ldh      [12]
(001) jeq      #0x86dd          jt 2    jf 8
(002) ldb      [20]
(003) jeq      #0x6             jt 4    jf 19
(004) ldh      [54]
(005) jeq      #0x50            jt 18   jf 6
(006) ldh      [56]
(007) jeq      #0x50            jt 18   jf 19
(008) jeq      #0x800           jt 9    jf 19
(009) ldb      [23]
(010) jeq      #0x6             jt 11   jf 19
(011) ldh      [20]
(012) jset     #0x1fff          jt 19   jf 13
(013) ldxb     4*([14]&0xf)
(014) ldh      [x + 14]
(015) jeq      #0x50            jt 18   jf 16
(016) ldh      [x + 16]
(017) jeq      #0x50            jt 18   jf 19
(018) ret      #262144
(019) ret      #0 

❯ fend '80 in hex'
50
  • port 80 is referenced in (008) jeq #0x50 jt 18 jf 19,
  • (019) 0 is NOMATCH,
  • (018) #262144 is MATCH

It's really helpful to learn some of the history (so much of it is recent):

In July 2014...BPF was an obscure technology for improving packet filter performance, and Alexei Starovoitov had a vision of extending it far beyond packets. Alexei had been working with another network engineer, Daniel Borkmann, to turn BPF into a general-purpose virtual machine, capable of running advanced networking and other programs.

BPF can be difficult to explain precisely because it can do so much. It provides a way to run mini programs on a wide variety of kernel and application events. If you are familiar with JavaScript, you may see some similarities: JavaScript allows a website to run mini programs on browser events such as mouse clicks, enabling a wide variety of web-based applications. BPF allows the kernel to run mini programs on system and application events, such as disk I/O, thereby enabling new system technologies.

BCC (BPF Compiler Collection) was the first higher-level tracing framework developed for BPF. It provides a C programming environment for writing kernel BPF code and other languages for the user-level interface: Python, Lua, and C++. It is also the origin of the libbcc and the current libbpf libraries, which provides functions for instrumenting events with BPF programs. The BCC repository also contains more than 70 BPF tools for performance analysis and troubleshooting (like execsnoop and biolatency)

bpftrace is a newer front end that provides a special-purpose, high-level language for developing BPF tools.bpftrace is built upon the libbcc and libbpf libraries.

Especially with related tools and libraries (tcpdump, tshark / wireshark, perf, libpcap):

The first libbpf was developed by Wang Nan for use with perf. libbpf is now part of the kernel source.

(from tshark docs) Capture filters are based on BPF syntax, which tcpdump also uses.

The libpcap interface supports a filtering mechanism based on the architecture in the BSD packet filter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment