offcputime-bpfcc: Summarize off-CPU time by stack trace
offcputime-bpfcc [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f] [-s] [--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M MAX_BLOCK_TIME] [--state STATE] [duration]
What it does
Summarize off-CPU time by stack trace. Part of the bcc tools (bpfcc-tools 0.35.0 on Ubuntu, installed as /usr/sbin/offcputime-bpfcc; upstream and other distributions call it offcputime). It loads an eBPF program, so it needs root.
Options you will use
duration- duration of trace, in seconds
-p, --pid PID- trace these PIDs only, comma separated list
-t, --tid TID- trace these TIDs only, comma separated list
-u, --user-threads-only- user threads only (no kernel threads)
-k, --kernel-threads-only- kernel threads only (no user threads)
-U, --user-stacks-only- show stacks from user space only (no kernel space stacks)
-K, --kernel-stacks-only- show stacks from kernel space only (no user space stacks)
-d, --delimited- insert delimiter between kernel/user stacks
-f, --folded- output folded format
-s, --offset- show address offsets
--stack-storage-size STACK_STORAGE_SIZE- the number of unique stack traces that can be stored and displayed (default 16384)
-m, --min-block-time MIN_BLOCK_TIME- the amount of time in microseconds over which we store traces (default 1)
-M, --max-block-time MAX_BLOCK_TIME- the amount of time in microseconds under which we store traces (default U64_MAX)
--state STATE- filter on this thread state bitmask (eg, 2 == TASK_UNINTERRUPTIBLE) see include/linux/sched.h
Examples
$ sudo offcputime-bpfcctrace off-CPU stack time until Ctrl-C
$ sudo offcputime-bpfcc 5trace for 5 seconds only
$ sudo offcputime-bpfcc -f 55 seconds, and output in folded format
$ sudo offcputime-bpfcc -s 55 seconds, and show symbol offsets
$ sudo offcputime-bpfcc -m 1000trace only events that last more than 1000 usec
$ sudo offcputime-bpfcc -M 10000trace only events that last less than 10000 usec
$ sudo offcputime-bpfcc -p 185,175,165only trace threads for PID 185,175,165
$ sudo offcputime-bpfcc -t 188,120,134only trace threads 188,120,134
Gotchas
- Time blocked per stack, in microseconds; -f folded output for off-CPU flame graphs.
- As a normal user it fails with "could not open bpf map: ..., error: Operation not permitted" (unprivileged BPF is off on Ubuntu).
Taught in
Try offcputime-bpfcc in a real terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.