The BCC checklist
Ubuntu Server installs bpfcc-tools by default (with bpftrace). Debian and Ubuntu add -bpfcc to every tool name so it cannot clash with other packages: execsnoop is execsnoop-bpfcc, in /usr/sbin. All need sudo, all have -h. They are Python scripts that compile a small C program when they start (a second or two).
Gregg's follow-up to the 60-second list, in the bcc docs, is a list of tools to run next:
| Tool | Answers | Typical find |
|---|---|---|
execsnoop-bpfcc | which processes start (every exec) | a script forking hundreds of short commands a second |
opensnoop-bpfcc | which files are opened, and failures (-x) | a missing config file retried in a loop |
ext4slower-bpfcc | file system operations slower than N ms (default 10) | slow fsync on a database log |
biolatency-bpfcc | disk I/O latency as a histogram (-D per disk) | a slow tail the average hides |
biosnoop-bpfcc | each disk I/O with process, size and latency | who issues the slow requests |
cachestat-bpfcc | page cache hits and misses per second | a scan evicting the hot data |
tcpconnect-bpfcc | outgoing TCP connections | an unexpected dependency, a connect storm |
tcpaccept-bpfcc | incoming TCP connections | who calls this service |
tcpretrans-bpfcc | each TCP retransmit with addresses | the lossy peer |
runqlat-bpfcc | how long threads wait for a CPU | CPU saturation as latency |
profile-bpfcc | CPU stacks, like perf record | hot code, without perf installed |
offcputime-bpfcc | where threads block, by stack | locks, fsync, network waits |
Snoopers (*snoop, tcpconnect, tcpretrans) print a line per event until Ctrl+C:
$ sudo execsnoop-bpfcc -T
TIME COMM PID PPID RET ARGS
20:00:08 df 19292 17385 0 /usr/bin/df -P
20:00:08 awk 19293 17385 0 /usr/bin/awk NR>1 {print $5, $6}
20:00:08 tr 19294 17385 0 /usr/bin/tr -d %
Short-lived processes never show in top or pidstat (they are gone before the next sample); execsnoop catches every one. The PPID column leads to the parent that starts them.
Summarizers (*lat, *dist, cachestat) collect in the kernel and print a histogram on Ctrl+C, or every interval: runqlat-bpfcc 5 1 = one 5-second histogram. This one was taken while a backup read the vdb disk (the disk lab); on this box right now it shows only a few fast I/Os:
$ sudo biolatency-bpfcc -D
Tracing block device I/O... Hit Ctrl-C to end.
^C
disk = vdb
usecs : count distribution
0 -> 1 : 0 | |
...
4096 -> 8191 : 9 | |
8192 -> 16383 : 311 |************* |
16384 -> 32767 : 949 |****************************************|
32768 -> 65535 : 921 |************************************** |
65536 -> 131071 : 481 |******************** |
Each row is a power-of-two bucket: here most reads took 16-65 ms, and a fifth over 65 ms - on a virtual disk that should answer in ~1 ms. A histogram shows what an average hides: two modes (cache vs disk), or a slow tail.
Overhead. Snoopers on very frequent events (opens on a busy box, every block I/O at 50k IOPS) cost CPU and print too much: filter (-p PID, -n NAME, -x), and run them for seconds, not hours.
In an interview: "After the counters, I reach for the bcc tools: execsnoop for short-lived processes, opensnoop for file errors, biolatency and ext4slower for storage latency, tcpretrans for network loss, runqlat for CPU queueing, profile and offcputime for on- and off-CPU time."