biolatency-bpfcc: Summarize block device I/O latency as a histogram
biolatency-bpfcc [-h] [-T] [-Q] [-m] [-D] [-F] [-e] [-j] [-d DISK] [interval] [count]
What it does
Summarize block device I/O latency as a histogram. Part of the bcc tools (bpfcc-tools 0.35.0 on Ubuntu, installed as /usr/sbin/biolatency-bpfcc; upstream and other distributions call it biolatency). It loads an eBPF program, so it needs root.
Options you will use
interval- output interval, in seconds
count- number of outputs
-T, --timestamp- include timestamp on output
-Q, --queued- include OS queued time in I/O time
-m, --milliseconds- millisecond histogram
-D, --disks- print a histogram per disk device
-F, --flags- print a histogram per set of I/O flags
-e, --extension- summarize average/total value
-j, --json- json output
-d, --disk DISK- Trace this disk only
Examples
$ sudo biolatency-bpfccsummarize block I/O latency as a histogram
$ sudo biolatency-bpfcc 1 10print 1 second summaries, 10 times
$ sudo biolatency-bpfcc -mT 11s summaries, milliseconds, and timestamps
$ sudo biolatency-bpfcc -Qinclude OS queued time in I/O time
$ sudo biolatency-bpfcc -Dshow each disk device separately
$ sudo biolatency-bpfcc -Fshow I/O flags separately
$ sudo biolatency-bpfcc -jprint a dictionary
$ sudo biolatency-bpfcc -eshow extension summary(total, average)
Gotchas
- A healthy virtual disk has one hump under a few ms; a tail in tens of ms means queueing.
- As a normal user it fails with "could not open bpf map: ..., error: Operation not permitted" (unprivileged BPF is off on Ubuntu).
Taught in
On the blog
Try biolatency-bpfcc in a real terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.