OnCallReadyStart learning

offcputime-bpfcc: Summarize off-CPU time by stack trace

offcputime-bpfcc [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f] [-s] [--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M MAX_BLOCK_TIME] [--state STATE] [duration]

What it does

Summarize off-CPU time by stack trace. Part of the bcc tools (bpfcc-tools 0.35.0 on Ubuntu, installed as /usr/sbin/offcputime-bpfcc; upstream and other distributions call it offcputime). It loads an eBPF program, so it needs root.

Options you will use

duration
duration of trace, in seconds
-p, --pid PID
trace these PIDs only, comma separated list
-t, --tid TID
trace these TIDs only, comma separated list
-u, --user-threads-only
user threads only (no kernel threads)
-k, --kernel-threads-only
kernel threads only (no user threads)
-U, --user-stacks-only
show stacks from user space only (no kernel space stacks)
-K, --kernel-stacks-only
show stacks from kernel space only (no user space stacks)
-d, --delimited
insert delimiter between kernel/user stacks
-f, --folded
output folded format
-s, --offset
show address offsets
--stack-storage-size STACK_STORAGE_SIZE
the number of unique stack traces that can be stored and displayed (default 16384)
-m, --min-block-time MIN_BLOCK_TIME
the amount of time in microseconds over which we store traces (default 1)
-M, --max-block-time MAX_BLOCK_TIME
the amount of time in microseconds under which we store traces (default U64_MAX)
--state STATE
filter on this thread state bitmask (eg, 2 == TASK_UNINTERRUPTIBLE) see include/linux/sched.h

Examples

$ sudo offcputime-bpfcc

trace off-CPU stack time until Ctrl-C

$ sudo offcputime-bpfcc 5

trace for 5 seconds only

$ sudo offcputime-bpfcc -f 5

5 seconds, and output in folded format

$ sudo offcputime-bpfcc -s 5

5 seconds, and show symbol offsets

$ sudo offcputime-bpfcc -m 1000

trace only events that last more than 1000 usec

$ sudo offcputime-bpfcc -M 10000

trace only events that last less than 10000 usec

$ sudo offcputime-bpfcc -p 185,175,165

only trace threads for PID 185,175,165

$ sudo offcputime-bpfcc -t 188,120,134

only trace threads 188,120,134

Gotchas

Taught in

Try offcputime-bpfcc in a real terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.