OnCallReadyStart learning

Linux Lesson 8.24 · Linux Performance & eBPF · 9 min read

bpftrace: one-liners for any question

In plain words

bpftrace is a pocket calculator for the kernel. You write one short line: when this happens, if this is true, count it by this name. bpftrace turns the line into an eBPF program, attaches it, and keeps counting until you press Ctrl+C. Then it prints the totals. It is how you ask a question no ready-made tool answers, in one line instead of a whole program.

Ask your own question

The bcc tools answer common questions. bpftrace (installed by default, version 0.25) answers yours, often in one line. A program is probe /filter/ { actions }:

$ sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
Attached 1 probe
^C

@[systemd-journal]: 60
@[java]: 594
@[pricing]: 9534
  • probe: tracepoint:raw_syscalls:sys_enter fires on every system call;
  • map: @[comm] is a map keyed by process name; count() adds one;
  • Ctrl+C stops it, and bpftrace prints every map, sorted by value.

Finding probes. sudo bpftrace -l 'tracepoint:syscalls:sys_enter_open*' lists matches; -lv adds the fields (args) a tracepoint has. Short names work: t: tracepoint, k: kprobe, kr: kretprobe, p: profile, i: interval.

The pieces:

KindExamples
builtinspid, tid, uid, comm, nsecs (ns since boot), elapsed, cpu, retval (kretprobe), args (tracepoint fields), arg0.. (kprobe), kstack, ustack
maps@name[key] = ...; @ alone is fine; $x is a per-event variable
aggregationscount(), sum(x), avg(x), min(x), max(x), stats(x), hist(x) (powers of 2), lhist(x, min, max, step)
functionsprintf("%s %d\n", comm, pid), str(args.filename), print(@m), clear(@m), time("%H:%M:%S\n"), exit()
filters/comm == "pricing"/, /pid == 1234/, /args.ret < 0/

One-liners worth knowing (Gregg's bpftrace tutorial):

# files opened, by process and path
sudo bpftrace -e 't:syscalls:sys_enter_openat { @[comm, str(args.filename)] = count(); }'
# new processes with their arguments
sudo bpftrace -e 't:syscalls:sys_enter_execve { printf("%-6d %s\n", pid, str(args.filename)); }'
# read() sizes for one process, as a histogram
sudo bpftrace -e 't:syscalls:sys_exit_read /comm == "pricing"/ { @bytes = hist(args.ret); }'
# syscalls per second, every second, for 5 seconds
sudo bpftrace -e 't:raw_syscalls:sys_enter { @ = count(); } i:s:1 { print(@); clear(@); } i:s:5 { exit(); }'
# on-CPU stacks of one process, sampled at 99 Hz
sudo bpftrace -e 'profile:hz:99 /comm == "pricing"/ { @[ustack] = count(); }'

A histogram prints like this - power-of-two buckets, [1K, 2K) meaning 1024-2047:

@bytes:
[256, 512)          4410 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)              0 |                                                    |
[1K, 2K)            1890 |@@@@@@@@@@@@@@@@@@@@@@                              |

Kernel functions. kprobes reach what has no tracepoint. kretprobe gives the return value in retval; arguments are arg0, arg1... For vfs_read the return value is the number of bytes read:

sudo bpftrace -e 'kretprobe:vfs_read /retval > 0/ { @bytes[comm] = sum(retval); }'

Function names are not a stable interface: they can be renamed or inlined between kernel versions. sudo bpftrace -l 'kprobe:vfs_*' checks what this kernel has before you write the program. Prefer a tracepoint when one exists; use fentry: (BTF, typed args) when the kernel has BTF, as Ubuntu's does.

Scripts. Longer programs go in a file (.bt), run with sudo bpftrace file.bt. The bpftrace project ships classic tools as such scripts (opensnoop.bt, biolatency.bt, tcpretrans.bt); they are the best examples to copy from.

Strings. args.filename is a pointer into user memory: wrap it in str() to read it (printing it bare gives the address, 0xffff...). comm is already a string.

Timing a pair of events uses a map keyed by thread: save the start, subtract at the end.

sudo bpftrace -e 't:syscalls:sys_enter_read { @s[tid] = nsecs; }
  t:syscalls:sys_exit_read /@s[tid]/ { @us = hist((nsecs - @s[tid]) / 1000); delete(@s[tid]); }'

In an interview: "With bpftrace I attach to a tracepoint or kprobe and aggregate in a map: @[comm] = count() for who, hist() for distributions; Ctrl+C prints the maps. For example, which processes open which files: sys_enter_openat with comm and str(args.filename) as keys."

Why it helps

The bcc tools cover common questions; bpftrace covers everything else in one line: which process writes the most, what size reads a service does, how long a particular kernel function takes. It is the fastest way to test a theory during an incident, and many classic tools ship as short bpftrace scripts you can read and adapt. Interviewers like asking for a one-liner.

On the blog

Commands in this lesson

bpftrace

FAQ

How is a bpftrace program built?

probe /filter/ { action }. The probe says where to attach, for example tracepoint:syscalls:sys_enter_openat; the optional filter decides which events count, for example /comm == "nginx"/; the action does the work, usually updating a map such as @[comm] = count().

How do I find the probes I can use?

sudo bpftrace -l lists them, with wildcards: bpftrace -l 'tracepoint:syscalls:open' or bpftrace -l 'kprobe:vfs_*'. Add -v to a tracepoint to see its argument names, which you then read as args.NAME in the program. Kernel functions differ between kernel versions, so list before you write.

What do count, hist and lhist do?

count() counts events per key. hist() builds a power-of-two histogram of a value, for example read sizes or latencies. lhist(value, min, max, step) builds a linear histogram with fixed buckets. All of them are maps, printed when the program exits or you press Ctrl+C.

Why do I need str() around a filename?

args.filename is a pointer to a string in the process's memory, not the string itself. str() copies the string into the program so it can be printed or used as a map key. Without it you get a number, the address.

How do I time something with bpftrace?

Store the start time in a map keyed by thread when the event begins, for example @start[tid] = nsecs on entry, and on the exit probe compute nsecs - @start[tid], put it into hist() and delete the start entry. That gives a latency histogram for any pair of events.

In an interview Mid

Write a bpftrace one-liner that counts files opened per process.

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm, str(args.filename)] = count(); }' attaches to the openat tracepoint and counts each process and path in a map; str() copies the filename from user memory. Ctrl+C prints the map sorted by count. I find probe names with bpftrace -l and use hist() instead of count() when I want a distribution.

Also asked: How do you list the probes bpftrace can attach to? · What is the difference between hist and lhist? · How would you measure a latency with bpftrace?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.