OnCallReadyStart learning

Linux Lesson 8.21 · Linux Performance & eBPF · 6 min read

The bcc tools: answers in one command

In plain words

The bcc tools are a toolbox of ready-made eBPF programs, each built for one question. One prints every new process as it starts. One prints every file opened. One draws a chart of how long disk requests took. You do not write any eBPF yourself: you pick the tool whose name matches your question, run it for a few seconds, and read the answer.

The BCC checklist

Ubuntu Server installs bpfcc-tools by default (with bpftrace). Debian and Ubuntu add -bpfcc to every tool name so it cannot clash with other packages: execsnoop is execsnoop-bpfcc, in /usr/sbin. All need sudo, all have -h. They are Python scripts that compile a small C program when they start (a second or two).

Gregg's follow-up to the 60-second list, in the bcc docs, is a list of tools to run next:

ToolAnswersTypical find
execsnoop-bpfccwhich processes start (every exec)a script forking hundreds of short commands a second
opensnoop-bpfccwhich files are opened, and failures (-x)a missing config file retried in a loop
ext4slower-bpfccfile system operations slower than N ms (default 10)slow fsync on a database log
biolatency-bpfccdisk I/O latency as a histogram (-D per disk)a slow tail the average hides
biosnoop-bpfcceach disk I/O with process, size and latencywho issues the slow requests
cachestat-bpfccpage cache hits and misses per seconda scan evicting the hot data
tcpconnect-bpfccoutgoing TCP connectionsan unexpected dependency, a connect storm
tcpaccept-bpfccincoming TCP connectionswho calls this service
tcpretrans-bpfcceach TCP retransmit with addressesthe lossy peer
runqlat-bpfcchow long threads wait for a CPUCPU saturation as latency
profile-bpfccCPU stacks, like perf recordhot code, without perf installed
offcputime-bpfccwhere threads block, by stacklocks, fsync, network waits

Snoopers (*snoop, tcpconnect, tcpretrans) print a line per event until Ctrl+C:

$ sudo execsnoop-bpfcc -T
TIME     COMM             PID     PPID    RET ARGS
20:00:08 df               19292   17385     0 /usr/bin/df -P
20:00:08 awk              19293   17385     0 /usr/bin/awk NR>1 {print $5, $6}
20:00:08 tr               19294   17385     0 /usr/bin/tr -d %

Short-lived processes never show in top or pidstat (they are gone before the next sample); execsnoop catches every one. The PPID column leads to the parent that starts them.

Summarizers (*lat, *dist, cachestat) collect in the kernel and print a histogram on Ctrl+C, or every interval: runqlat-bpfcc 5 1 = one 5-second histogram. This one was taken while a backup read the vdb disk (the disk lab); on this box right now it shows only a few fast I/Os:

$ sudo biolatency-bpfcc -D
Tracing block device I/O... Hit Ctrl-C to end.
^C

disk = vdb
     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
...
      4096 -> 8191       : 9        |                                        |
      8192 -> 16383      : 311      |*************                           |
     16384 -> 32767      : 949      |****************************************|
     32768 -> 65535      : 921      |**************************************  |
     65536 -> 131071     : 481      |********************                    |

Each row is a power-of-two bucket: here most reads took 16-65 ms, and a fifth over 65 ms - on a virtual disk that should answer in ~1 ms. A histogram shows what an average hides: two modes (cache vs disk), or a slow tail.

Overhead. Snoopers on very frequent events (opens on a busy box, every block I/O at 50k IOPS) cost CPU and print too much: filter (-p PID, -n NAME, -x), and run them for seconds, not hours.

In an interview: "After the counters, I reach for the bcc tools: execsnoop for short-lived processes, opensnoop for file errors, biolatency and ext4slower for storage latency, tcpretrans for network loss, runqlat for CPU queueing, profile and offcputime for on- and off-CPU time."

Why it helps

Most performance questions that counters cannot answer already have a bcc tool: short-lived processes, failing opens, slow file system operations, disk latency distributions, run queue delays, retransmits. They are installed on every Ubuntu server, they are safe and bounded, and knowing a dozen of them by name turns a long investigation into a few commands. They are also the templates for your own bpftrace programs.

On the blog

Commands in this lesson

execsnoop-bpfcc biolatency-bpfcc

FAQ

Why are the names different on Ubuntu?

Ubuntu's bpfcc-tools package installs the tools in /usr/sbin with a -bpfcc suffix, for example execsnoop-bpfcc and biolatency-bpfcc, to avoid clashing with other packages. Upstream and other distributions use the bare names, so documentation shows execsnoop where Ubuntu needs execsnoop-bpfcc.

What is the difference between snoopers and summarizers?

Snoopers print one line per event: execsnoop per new process, opensnoop per open, biosnoop per disk request, tcpconnect per connection. Summarizers aggregate in the kernel and print a table or histogram at the end: biolatency, runqlat, cachestat. Summarizers are cheaper on busy events.

When do I use execsnoop?

When CPU or system time is high but top shows nothing obvious. Short-lived processes, for example a script forking in a loop, exit before top or pidstat sample them. execsnoop-bpfcc logs every exec with its PPID, which leads back to the parent that keeps starting them.

How do I find a missing file an app keeps looking for?

sudo opensnoop-bpfcc -x prints only failed opens with the error number; ERR 2 is ENOENT, no such file. Add -n NAME or -p PID to narrow it to one program. It names the path even when the application's own log does not.

How much do these tools cost?

It depends on the event rate. execsnoop on a quiet box costs nothing; a snooper on every read of a busy database can cost a noticeable share of a CPU. Prefer the summarizers for hot events, give the tool a duration, and filter by PID.

In an interview Mid

Which eBPF tools would you reach for when the counters are not enough?

The bcc tools, installed on Ubuntu as bpfcc-tools with a -bpfcc suffix: execsnoop for short-lived processes, opensnoop for failing opens, biolatency and ext4slower for storage latency, tcpretrans for retransmits, runqlat for CPU queueing, and profile and offcputime for on-CPU and off-CPU time. Snoopers print events, summarizers aggregate in the kernel, so I prefer summarizers on hot events.

Also asked: What is the difference between a snooper and a summarizer? · When would you use execsnoop? · How would you find the file an application fails to open?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.