Ask your own question
The bcc tools answer common questions. bpftrace (installed by default, version 0.25) answers yours, often in one line. A program is probe /filter/ { actions }:
$ sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
Attached 1 probe
^C
@[systemd-journal]: 60
@[java]: 594
@[pricing]: 9534
- probe:
tracepoint:raw_syscalls:sys_enterfires on every system call; - map:
@[comm]is a map keyed by process name;count()adds one; - Ctrl+C stops it, and bpftrace prints every map, sorted by value.
Finding probes. sudo bpftrace -l 'tracepoint:syscalls:sys_enter_open*' lists matches; -lv adds the fields (args) a tracepoint has. Short names work: t: tracepoint, k: kprobe, kr: kretprobe, p: profile, i: interval.
The pieces:
| Kind | Examples |
|---|---|
| builtins | pid, tid, uid, comm, nsecs (ns since boot), elapsed, cpu, retval (kretprobe), args (tracepoint fields), arg0.. (kprobe), kstack, ustack |
| maps | @name[key] = ...; @ alone is fine; $x is a per-event variable |
| aggregations | count(), sum(x), avg(x), min(x), max(x), stats(x), hist(x) (powers of 2), lhist(x, min, max, step) |
| functions | printf("%s %d\n", comm, pid), str(args.filename), print(@m), clear(@m), time("%H:%M:%S\n"), exit() |
| filters | /comm == "pricing"/, /pid == 1234/, /args.ret < 0/ |
One-liners worth knowing (Gregg's bpftrace tutorial):
# files opened, by process and path
sudo bpftrace -e 't:syscalls:sys_enter_openat { @[comm, str(args.filename)] = count(); }'
# new processes with their arguments
sudo bpftrace -e 't:syscalls:sys_enter_execve { printf("%-6d %s\n", pid, str(args.filename)); }'
# read() sizes for one process, as a histogram
sudo bpftrace -e 't:syscalls:sys_exit_read /comm == "pricing"/ { @bytes = hist(args.ret); }'
# syscalls per second, every second, for 5 seconds
sudo bpftrace -e 't:raw_syscalls:sys_enter { @ = count(); } i:s:1 { print(@); clear(@); } i:s:5 { exit(); }'
# on-CPU stacks of one process, sampled at 99 Hz
sudo bpftrace -e 'profile:hz:99 /comm == "pricing"/ { @[ustack] = count(); }'
A histogram prints like this - power-of-two buckets, [1K, 2K) meaning 1024-2047:
@bytes:
[256, 512) 4410 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K) 0 | |
[1K, 2K) 1890 |@@@@@@@@@@@@@@@@@@@@@@ |
Kernel functions. kprobes reach what has no tracepoint. kretprobe gives the return value in retval; arguments are arg0, arg1... For vfs_read the return value is the number of bytes read:
sudo bpftrace -e 'kretprobe:vfs_read /retval > 0/ { @bytes[comm] = sum(retval); }'
Function names are not a stable interface: they can be renamed or inlined between kernel versions. sudo bpftrace -l 'kprobe:vfs_*' checks what this kernel has before you write the program. Prefer a tracepoint when one exists; use fentry: (BTF, typed args) when the kernel has BTF, as Ubuntu's does.
Scripts. Longer programs go in a file (.bt), run with sudo bpftrace file.bt. The bpftrace project ships classic tools as such scripts (opensnoop.bt, biolatency.bt, tcpretrans.bt); they are the best examples to copy from.
Strings. args.filename is a pointer into user memory: wrap it in str() to read it (printing it bare gives the address, 0xffff...). comm is already a string.
Timing a pair of events uses a map keyed by thread: save the start, subtract at the end.
sudo bpftrace -e 't:syscalls:sys_enter_read { @s[tid] = nsecs; }
t:syscalls:sys_exit_read /@s[tid]/ { @us = hist((nsecs - @s[tid]) / 1000); delete(@s[tid]); }'
In an interview: "With bpftrace I attach to a tracepoint or kprobe and aggregate in a map: @[comm] = count() for who, hist() for distributions; Ctrl+C prints the maps. For example, which processes open which files: sys_enter_openat with comm and str(args.filename) as keys."