runqlat-bpfcc: Summarize run queue (scheduler) latency as a histogram
runqlat-bpfcc [-h] [-T] [-m] [-P] [--pidnss] [-L] [-p PID] [interval] [count]
What it does
Summarize run queue (scheduler) latency as a histogram. Part of the bcc tools (bpfcc-tools 0.35.0 on Ubuntu, installed as /usr/sbin/runqlat-bpfcc; upstream and other distributions call it runqlat). It loads an eBPF program, so it needs root.
Options you will use
interval- output interval, in seconds
count- number of outputs
-T, --timestamp- include timestamp on output
-m, --milliseconds- millisecond histogram
-P, --pids- print a histogram per process ID
--pidnss- print a histogram per PID namespace
-L, --tids- print a histogram per thread ID
-p, --pid PID- trace this PID only
Examples
$ sudo runqlat-bpfccsummarize run queue latency as a histogram
$ sudo runqlat-bpfcc 1 10print 1 second summaries, 10 times
$ sudo runqlat-bpfcc -mT 11s summaries, milliseconds, and timestamps
$ sudo runqlat-bpfcc -Pshow each PID separately
$ sudo runqlat-bpfcc -p 185trace PID 185 only
Gotchas
- How long runnable tasks wait for a CPU: microseconds when healthy, milliseconds when CPUs are saturated.
- As a normal user it fails with "could not open bpf map: ..., error: Operation not permitted" (unprivileged BPF is off on Ubuntu).
Try runqlat-bpfcc in a real terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.