The disk is slow - or is it?
iostat -xz 1 is the view. One device under a backup job:
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz ... aqu-sz %util
vdb 956.04 117179.39 19.12 1.96 25.00 122.57 40.77 448.37 7.34 15.25 38.80 11.00 ... 25.48 100.00
| Column | Meaning | Here |
|---|---|---|
r/s, w/s | requests completed per second | 956 reads |
rkB/s, wkB/s | throughput | 117 MB/s read |
rareq-sz | average read size (kB) | 122 kB: big sequential reads |
r_await, w_await | average ms per request, queue + device | 25 ms reads, 39 ms writes |
aqu-sz | average number of requests in flight | 25 queued |
%util | share of time the device had at least one request | 100% |
Latency is the truth, %util is a hint. %util only says the device was never idle. A spinning disk at 100% is full; an SSD, NVMe or a virtio disk serves many requests in parallel and can be at 100% with room to spare. The question is: are r_await/w_await high for this device (single-digit ms on a VM disk is normal) and is aqu-sz growing? Here, 25-39 ms with 25 queued: saturated. The small writes of the orders API (11 kB each) wait 39 ms behind the backup's reads.
(svctm used to be a column; sysstat 12 removed it because it could not be computed correctly.)
What the CPU sees. vmstat shows wa (iowait) and b (tasks blocked in D); PSI has the disk view: /proc/pressure/io "some" = share of time some task waited for I/O.
Who is it? pidstat -d 1 lists read and write rates per process - as root (other users' /proc/PID/io is unreadable, so pidstat silently hides their processes):
$ sudo pidstat -d 1 1
20:00:08 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
20:00:09 0 352 0.00 19.86 0.00 0 systemd-journal
20:00:09 1001 1302 0.00 85.29 0.00 0 logwriter
20:00:09 0 17385 116525.82 0.00 0.00 0 tar
iodelay is 0 here even for tar: block I/O delay accounting is compiled into Ubuntu's kernel but off by default. sudo sysctl kernel.task_delayacct=1 turns it on; from then on iodelay shows the clock ticks each task spent waiting for I/O.
Reading a latency histogram. biolatency-bpfcc -D (the bcc lesson) groups every I/O by how long it took. A healthy virtual disk has one hump under a millisecond or two. Two humps mean two kinds of I/O (cache hits on the host vs real reads); a hump in the tens of milliseconds next to a long tail means queueing: the requests are fine, they wait their turn. biosnoop-bpfcc shows the individual requests with the process, the sector and the size - big sequential reads from one process behind small writes from another is the classic "batch job ruins the database" picture.
The page cache. Reads that hit the cache never reach the disk; free's buff/cache is that cache. A job reading cold data (a backup, a scan) misses, and can also evict what the application needed. cachestat-bpfcc shows hits and misses per second (the eBPF lesson).
Fixes, in order of preference. Move the job (schedule it off-peak), slow it down, or give the important I/O priority. Slowing it down works everywhere with cgroup io.max: sudo systemctl set-property nightly-backup.service IOReadBandwidthMax="/dev/vdb 20M" caps the job's reads at 20 MB/s (systemd counts these suffixes in powers of 1000) at once, while it runs, and writes the drop-in under /etc/systemd/system.control/ so the limit stays (the cgroup file shows 253:16 rbps=20000000 ...). (ionice priorities only work with a scheduler that honours them, like BFQ; virtio disks on Ubuntu use none, where ionice changes nothing.)
In an interview: "For disks I look at await and the queue size in iostat -x, not just %util, because SSDs and virtual disks do work in parallel. Then pidstat -d as root to see which process is doing the I/O."