OnCallReadyStart learning

LinuxPerformance · 6 min read

bpftrace one-liners that answer real production questions

Eight bpftrace one-liners for who is busy, which open fails, what keeps starting, how big the reads are and where the CPU goes, with their real output.

strace answers questions about one process and slows it down while it does. The bcc tools (opensnoop-bpfcc, execsnoop-bpfcc, ...) answer common questions about the whole box. bpftrace answers your question, about every process, usually in one line, and at a cost low enough for production. On Ubuntu Server it is already installed: since 24.04, bpftrace and bpfcc-tools are part of every server installation. The examples here are bpftrace 0.25.

Every program has the same shape:

output
probe /filter/ { action }

The probe is the event (a tracepoint, a kernel function, a timer). The optional filter decides whether the action runs. The action usually adds to a map (@name[key]), and when the program ends, with Ctrl+C or exit(), bpftrace prints every map. That is enough to read all the one-liners below.

Before you start: root, and finding probes

terminal
$ bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
ERROR: Missing CAP_DAC_READ_SEARCH capability, please run bpftrace as the root user.
$ sudo bpftrace -l '*openat*'
tracepoint:syscalls:sys_enter_openat
tracepoint:syscalls:sys_enter_openat2
tracepoint:syscalls:sys_exit_openat
tracepoint:syscalls:sys_exit_openat2
kprobe:do_sys_openat2

Loading BPF programs needs root (or CAP_BPF and CAP_PERFMON plus the file-access capabilities bpftrace checks for). -l lists probes matching a pattern. Prefer tracepoints: as bpftrace's documentation puts it, they "generally provide a more stable interface than kprobes do", because kprobes attach to kernel functions that can be renamed or inlined between kernel versions. Probe types have short names: t: tracepoint, k:/kr: kprobe/kretprobe, i: interval, p: profile.

1. Who is making all the system calls?

terminal
$ sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
Attached 1 probe
^C

@[bash]: 6
@[nginx]: 12
@[systemd-journal]: 58
@[systemd]: 60
@[logwriter]: 68
@[sshd-session]: 70
@[java]: 600
@[pricing]: 4284

raw_syscalls:sys_enter fires on every system call. @[comm] is a map keyed by process name, and count() adds one each time. Maps print sorted by value, so the busiest process comes last.

2. Which file opens are failing, and with which error?

A worker logs "error loading configuration, retrying in 50ms" and nothing else. No path. Ask the kernel:

terminal
$ sudo bpftrace -e 'tracepoint:syscalls:sys_exit_openat /args.ret < 0/ { @[comm, args.ret] = count(); }'
Attached 1 probe
^C

@[invoice-worker, -2]: 40

sys_exit_openat fires when openat() returns. args.ret is the return value, and a negative value is -errno: -2 is ENOENT, "No such file or directory" (-13 would be EACCES, a permission problem). Two keys in the map make two columns.

3. ...and which file?

The file name is an argument on entry, so save the pointer on entry and print it on a failed exit, keyed by thread:

terminal
$ sudo bpftrace -e 't:syscalls:sys_enter_openat { @f[tid] = args.filename; } t:syscalls:sys_exit_openat /@f[tid] && args.ret < 0/ { printf("%-16s %4d %s\n", comm, args.ret, str(@f[tid])); } t:syscalls:sys_exit_openat { delete(@f[tid]); }'
Attached 3 probes
invoice-worker     -2 /etc/invoice/worker.yaml
invoice-worker     -2 /etc/invoice/worker.yaml
invoice-worker     -2 /etc/invoice/worker.yaml
^C

Two things to notice. args.filename is a pointer into the process's memory, and str() reads the string it points to. comm is already a string. And the map is keyed by tid, so two threads opening files at the same time never mix up their names. delete() keeps the map from growing.

This is exactly what the bcc tool does, ready-made:

terminal
$ sudo opensnoop-bpfcc -x
PID    COMM               FD ERR PATH
17385  invoice-worker     -1   2 /etc/invoice/worker.yaml
17385  invoice-worker     -1   2 /etc/invoice/worker.yaml
^C

-x shows only failed opens. Write the one-liner when the bcc tool does not ask your question.

4. What keeps starting processes?

Short-lived processes (a health check every 2 seconds, a cron job, a script in a loop) never stay long enough to show up in top:

terminal
$ sudo bpftrace -e 't:syscalls:sys_enter_execve { printf("%-6d %s\n", pid, str(args.filename)); }'
Attached 1 probe
17408  /usr/bin/curl
17410  /usr/bin/curl
17412  /usr/bin/curl
17414  /usr/bin/curl
17416  /usr/bin/curl
^C

Replace the printf with @[str(args.filename)] = count(); to get a count per program instead of a stream. execsnoop-bpfcc -T adds the parent PID and the arguments, and the parent PID leads you to the script or unit that starts them. Thousands of these a second show up as high system CPU with nothing visible in top.

5. How big are the reads? (a histogram)

terminal
$ sudo bpftrace -e 'tracepoint:syscalls:sys_exit_read /comm == "pricing"/ { @bytes = hist(args.ret); }'
Attached 1 probe
^C

@bytes:
[256, 512)           996 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)              0 |                                                    |
[1K, 2K)             294 |@@@@@@@@@@@@@@@                                     |

hist() puts each value into a power-of-2 bucket: [1K, 2K) means 1024-2047. Two humps mean two kinds of reads. An average would have said "about 600 bytes", which describes neither of them. lhist(x, min, max, step) gives linear buckets when you know the range.

6. A rate per second that stops by itself

terminal
$ sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @ = count(); } interval:s:1 { print(@); clear(@); } interval:s:3 { exit(); }'
Attached 3 probes
@: 1732
@: 1692
@: 1726

interval:s:1 fires every second: print the map, then empty it. interval:s:3 calls exit(), so the program ends on its own. That makes it safe to paste into a script or a runbook, where nobody is around to press Ctrl+C.

7. Bytes read per process (a kernel function)

terminal
$ sudo bpftrace -e 'kretprobe:vfs_read /retval > 0/ { @bytes[comm] = sum(retval); }'
Attached 1 probe
^C

@bytes[sshd-session]: 432
@bytes[bash]: 768
@bytes[systemd]: 3072
@bytes[systemd-journal]: 7104
@bytes[java]: 82944
@bytes[pricing]: 727644

A kretprobe fires when a kernel function returns, with the return value in retval. For vfs_read, the kernel function behind read() on every filesystem, that is the number of bytes read. There is no tracepoint inside the VFS read path, so a kprobe is the way in. Check the function exists on your kernel first: sudo bpftrace -l 'kprobe:vfs_r*'.

8. Where does a process spend its CPU?

terminal
$ sudo bpftrace -e 'profile:hz:99 /comm == "pricing"/ { @[ustack] = count(); }'
Attached 1 probe
^C

...
@[
        check_matching+964
        re_search_internal+68
        regexec+700
        match_rule+484
        apply_promotions+112
        price_cart+200
        handle_request+700
        worker_loop+872
        start_thread+960
]: 60
@[
        parse_expression+16
        parse_branch+32
        parse_reg_exp+992
        re_compile_internal+356
        regcomp+992
        match_rule+484
        apply_promotions+112
        price_cart+200
        handle_request+700
        worker_loop+872
        start_thread+960
]: 108

profile:hz:99 samples every CPU 99 times a second (99, not 100, so the samples do not line up with timer-driven work). ustack is the user-space stack at that moment. The most common stacks come last. Here the hottest one compiles a regular expression (regcomp) inside match_rule, on every request, so the fix is to compile it once. For a whole-picture view, the same samples feed a flame graph (perf record or profile-bpfcc -f).

When the question is about the disk rather than one process, the bcc tool biolatency-bpfcc draws the latency histogram for you: see high iowait: which disk, which process. And when a service is slow but none of these probes shows it busy, check whether it is being throttled by a CPU limit or stuck in D state.

The building blocks, in one table

KindWhat you use
builtinspid, tid, uid, comm, nsecs, cpu, retval, args (tracepoint fields), arg0... (kprobe), kstack, ustack
aggregationscount(), sum(x), avg(x), min(x), max(x), stats(x), hist(x), lhist(x, min, max, step)
functionsprintf(), str(), print(@m), clear(@m), delete(@m[k]), exit()
filters/comm == "pricing"/, /pid == 1234/, /args.ret < 0/

Is it safe on a production box?

Mostly, yes. The kernel's verifier checks every BPF program before it loads (bounded loops, no invalid memory access), which is what makes eBPF safe to load into a running kernel. The cost depends on how often the probe fires. A timer at 99 Hz costs almost nothing. A probe on every system call of a box doing a million per second does cost something, so keep those short, filter early (/comm == "x"/ runs before the action), and end them with exit() or Ctrl+C rather than leaving them running.

Practise it

In the Linux Performance & eBPF chapter, the lesson bpftrace: one-liners for any question teaches the language, and the lab Five questions, five one-liners has you answer five questions about a busy pricing service with bpftrace alone. The bcc lesson and the labs Who keeps starting curl? and The worker that will not say what is wrong use the ready-made tools for questions 3 and 4.

OnCallReady is free, with no ads and no tracking. RSS · All posts