The orders API got slow at 14:00. The CPUs are more than half idle, and vmstat says something else:
$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r b swpd free buff cache si so bi bo in cs us sy id wa st gu
1 1 12544 2141372 88020 2845980 0 0 9443 68 150 230 1 1 95 3 0 0
1 1 12544 2141372 88020 2845980 0 0 107018 441 1135 1248 4 5 53 38 0 0
1 1 12544 2141372 88020 2845980 0 0 101723 440 1048 1247 4 5 54 37 0 0
1 1 12544 2141372 88020 2845980 0 0 102070 435 1326 1223 4 5 54 37 0 0
1 1 12544 2141372 88020 2845980 0 0 117180 472 1146 1196 4 5 56 35 0 0wa is 35-38%, b (tasks blocked in uninterruptible sleep) is 1, and bi says about 100 MB a second is being read from block devices. Ignore the first line: it is the average since boot. Every later line is one second.
What iowait actually means
wa (%iowait in mpstat) is idle time: the share of time a CPU had nothing to run while at least one task was waiting for disk I/O. The Linux man page for /proc/stat (proc_stat(5)) calls the number "not reliable": "The CPU will not wait for I/O to complete; iowait is the time that a task is waiting for I/O to complete", and another task can run on that CPU in the meantime. Two things follow from that:
- High iowait says "look at the disks". It does not mean the CPUs are busy. Another process could use them right now.
- Low iowait does not prove the disks are fine. On a box whose CPUs are busy, the same I/O wait shows up as
us/syinstead.
So iowait only tells you to look at the disks. iostat tells you which one, and how bad it is.
Step 1: which device, and how slow (iostat -x)
$ iostat -dxy vdb 1 1
Linux 7.0.0-31-generic (oncall-lab) 09/22/26 _aarch64_ (2 CPU)
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
vdb 811.70 102069.14 16.23 1.96 24.78 125.75 34.89 407.36 6.28 15.25 37.99 11.68 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 21.44 100.00-x gives the extended columns, -d only the devices, -y skips the first report (since boot, like vmstat's first line). Run iostat -xz 1 without a device name first to see every disk that is doing something. Here only vdb, the /data disk, was busy. The columns that matter, as sysstat 12.7's man page defines them:
| Column | Meaning | Here |
|---|---|---|
r/s, w/s | requests completed per second | 812 reads, 35 writes |
rkB/s | read throughput | ~100 MB/s |
rareq-sz, wareq-sz | average request size (KiB) | 126 KiB reads (big, sequential), 12 KiB writes |
r_await, w_await | average ms per request, time in the queue plus service time | 25 ms reads, 38 ms writes |
aqu-sz | average number of requests queued or in service (was avgqu-sz) | 21 |
%util | share of time the device had at least one request | 100% |
Latency is what you check. %util only hints. The man page says it directly: %util near 100% means saturation "for devices serving requests serially", but "for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits." A virtio disk or an NVMe drive can sit at 100% with room to spare. The question is whether r_await/w_await are high for this device and whether aqu-sz keeps growing. On a VM disk that normally answers in a millisecond or two, 25-38 ms with 21 requests queued is saturated.
The small writes are the API: 12 KiB each, waiting 38 ms behind someone's 126 KiB reads.
(If you remember a svctm column: sysstat removed it in 12.1.2 because it could not be computed correctly.)
Step 2: which process (pidstat -d, as root)
$ pidstat -d 1 1
Linux 7.0.0-31-generic (oncall-lab) 09/22/26 _aarch64_ (2 CPU)
20:00:15 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
Average: UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
$ sudo pidstat -d 1 1
Linux 7.0.0-31-generic (oncall-lab) 09/22/26 _aarch64_ (2 CPU)
20:00:17 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
20:00:18 0 352 0.00 21.74 0.00 0 systemd-journal
20:00:18 1001 1302 0.00 106.43 0.00 0 logwriter
20:00:18 0 17385 100535.34 0.00 0.00 0 tarThe empty table is the trap. pidstat reads /proc/PID/io, which only the process owner (or root) may read, and it skips what it cannot read without a warning. Always run it with sudo.
tar reads 100 MB/s. A PID alone tells you nothing, so ask systemd what it belongs to:
$ systemctl status $(pgrep -x tar) --no-pager | head -3
● nightly-backup.service - Nightly backup of /data
Loaded: loaded (/etc/systemd/system/nightly-backup.service; disabled; preset: enabled)
Active: active (running) since Tue 2026-09-22 20:00:03 UTC; 16s agoThe nightly backup, started by hand at two in the afternoon. ps shows the same process in state D, waiting in the page cache code for its reads:
$ ps -eo pid,stat,wchan:20,comm | awk '$2 ~ /D/'
17385 D folio_wait_bit_commo tarThe iodelay column is 0 even for tar. It is not broken: block I/O delay accounting is compiled into Ubuntu's kernel but disabled at boot. sudo sysctl kernel.task_delayacct=1 turns it on, and from then on iodelay shows the clock ticks each task spent waiting for I/O. It only counts tasks from the moment you enable it.
Step 3: the distribution, not the average (optional)
An average of 25 ms can hide a tail of 100 ms. biolatency-bpfcc (installed by default on Ubuntu Server since 24.04, with the rest of the bcc tools and bpftrace) times every block I/O and draws a histogram per disk:
$ sudo biolatency-bpfcc -D 5 1
Tracing block device I/O... Hit Ctrl-C to end.
disk = vdb
usecs : count distribution
4096 -> 8191 : 19 | |
8192 -> 16383 : 522 |************* |
16384 -> 32767 : 1595 |****************************************|
32768 -> 65535 : 1524 |************************************** |
65536 -> 131071 : 791 |******************* |
131072 -> 262143 : 39 | |(The empty buckets and the idle vda are cut here.) Most I/O takes 16-64 ms, and some takes over 100 ms. A healthy virtual disk has one hump around a millisecond. A hump in the tens of milliseconds means queueing: each request is served quickly once it reaches the device, but it waits behind many others first.
Pressure stall information (PSI) gives the same verdict as one number. some is the share of time at least one task was stalled on I/O, full the share when all non-idle tasks were:
$ cat /proc/pressure/io
some avg10=41.43 avg60=37.73 avg300=21.82 total=978941700
full avg10=15.81 avg60=14.55 avg300=8.68 total=391576680Step 4: slow it down instead of killing it
The backup has to finish today, so killing it is not an option. A cgroup I/O limit caps it without stopping it. In systemd that is IOReadBandwidthMax= (the cgroup v2 io.max file, bytes per second, K/M/G suffixes in powers of 1000):
$ sudo mkdir -p /etc/systemd/system/nightly-backup.service.d
$ printf '[Service]\nIOReadBandwidthMax=/dev/vdb 20M\n' | sudo tee /etc/systemd/system/nightly-backup.service.d/io.conf
[Service]
IOReadBandwidthMax=/dev/vdb 20M
$ sudo systemctl daemon-reload && sudo systemctl restart nightly-backup
$ iostat -dxy vdb 1 3
Linux 7.0.0-31-generic (oncall-lab) 09/22/26 _aarch64_ (2 CPU)
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
vdb 155.39 18980.57 3.11 1.96 2.28 122.14 38.91 382.61 7.00 15.25 14.89 9.83 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.93 64.17
...
$ vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r b swpd free buff cache si so bi bo in cs us sy id wa st gu
1 0 12544 2141372 88020 2845980 0 0 1759 69 150 230 1 1 97 1 0 0
1 0 12544 2141372 88020 2845980 0 0 19391 448 398 1107 4 5 81 10 0 0
1 0 12544 2141372 88020 2845980 0 0 21677 437 396 1109 4 4 81 11 0 0Reads drop to the cap (19,000 KiB/s, just under 20 million bytes a second), r_await falls from 25 ms to about 2, the queue from 21 to under 1, and the API's writes stop waiting behind the backup (w_await 38 -> 15 ms). iowait is down to 10-11% and b is 0. The backup now takes about five times longer. That is the trade you chose.
Before you reach for ionice: it only does something when the disk's I/O scheduler honours priorities. On none, the usual choice for NVMe and many virtual disks, ionice changes nothing. io.max works whatever the scheduler.
The real fix is boring: run the job at 02:00 as intended, with the drop-in kept in place.
When iowait is high but no disk is busy
If wa is high and b is large but iostat shows every disk nearly idle, the waiting is not on a local disk. The usual cause is an NFS or other network mount that stopped answering, with processes stuck in D that even kill -9 cannot remove. That is a different problem: load average 15 on 4 cores, but the CPU is 90% idle. And if the CPUs are idle, there is no iowait, and a service is still slow, look at its cgroup: CPU throttling.
The short version
vmstat 1:waandbup means some task is waiting on I/O.iostat -xz 1: which device. Judge it byr_await/w_awaitandaqu-sz, not by%util.sudo pidstat -d 1: which process. Without sudo, pidstat hides other users' processes.systemctl status PID: which unit.- Cap it with
IOReadBandwidthMax=/IOWriteBandwidthMax=, reschedule it, or make it someone else's disk.
Practise it
In the Linux Performance & eBPF chapter, the lesson Disk I/O: latency, queues and who is reading covers these columns, and the lab The backup that ate the disk starts this exact backup on your box. You find it, measure it with biolatency, and cap it so that it keeps running and the API's writes stop queuing behind it.