OnCallReadyStart learning

Linux Lesson 8.9 · Linux Performance & eBPF · 6 min read

Disk I/O: latency, queues and who is reading

In plain words

A disk is like a post office counter. %util says whether the clerk was busy at all; it does not say how long the line in front of the counter is. The real question for a customer is how long it took from joining the line to walking out with the stamp. That is await, and the number of people in the line is the queue size.

The disk is slow - or is it?

iostat -xz 1 is the view. One device under a backup job:

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz ...  aqu-sz  %util
vdb            956.04 117179.39    19.12   1.96   25.00   122.57   40.77    448.37     7.34  15.25   38.80    11.00 ...   25.48 100.00
ColumnMeaningHere
r/s, w/srequests completed per second956 reads
rkB/s, wkB/sthroughput117 MB/s read
rareq-szaverage read size (kB)122 kB: big sequential reads
r_await, w_awaitaverage ms per request, queue + device25 ms reads, 39 ms writes
aqu-szaverage number of requests in flight25 queued
%utilshare of time the device had at least one request100%

Latency is the truth, %util is a hint. %util only says the device was never idle. A spinning disk at 100% is full; an SSD, NVMe or a virtio disk serves many requests in parallel and can be at 100% with room to spare. The question is: are r_await/w_await high for this device (single-digit ms on a VM disk is normal) and is aqu-sz growing? Here, 25-39 ms with 25 queued: saturated. The small writes of the orders API (11 kB each) wait 39 ms behind the backup's reads.

(svctm used to be a column; sysstat 12 removed it because it could not be computed correctly.)

What the CPU sees. vmstat shows wa (iowait) and b (tasks blocked in D); PSI has the disk view: /proc/pressure/io "some" = share of time some task waited for I/O.

Who is it? pidstat -d 1 lists read and write rates per process - as root (other users' /proc/PID/io is unreadable, so pidstat silently hides their processes):

$ sudo pidstat -d 1 1
20:00:08      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
20:00:09        0       352      0.00     19.86      0.00       0  systemd-journal
20:00:09     1001      1302      0.00     85.29      0.00       0  logwriter
20:00:09        0     17385 116525.82      0.00      0.00       0  tar

iodelay is 0 here even for tar: block I/O delay accounting is compiled into Ubuntu's kernel but off by default. sudo sysctl kernel.task_delayacct=1 turns it on; from then on iodelay shows the clock ticks each task spent waiting for I/O.

Reading a latency histogram. biolatency-bpfcc -D (the bcc lesson) groups every I/O by how long it took. A healthy virtual disk has one hump under a millisecond or two. Two humps mean two kinds of I/O (cache hits on the host vs real reads); a hump in the tens of milliseconds next to a long tail means queueing: the requests are fine, they wait their turn. biosnoop-bpfcc shows the individual requests with the process, the sector and the size - big sequential reads from one process behind small writes from another is the classic "batch job ruins the database" picture.

The page cache. Reads that hit the cache never reach the disk; free's buff/cache is that cache. A job reading cold data (a backup, a scan) misses, and can also evict what the application needed. cachestat-bpfcc shows hits and misses per second (the eBPF lesson).

Fixes, in order of preference. Move the job (schedule it off-peak), slow it down, or give the important I/O priority. Slowing it down works everywhere with cgroup io.max: sudo systemctl set-property nightly-backup.service IOReadBandwidthMax="/dev/vdb 20M" caps the job's reads at 20 MB/s (systemd counts these suffixes in powers of 1000) at once, while it runs, and writes the drop-in under /etc/systemd/system.control/ so the limit stays (the cgroup file shows 253:16 rbps=20000000 ...). (ionice priorities only work with a scheduler that honours them, like BFQ; virtio disks on Ubuntu use none, where ionice changes nothing.)

In an interview: "For disks I look at await and the queue size in iostat -x, not just %util, because SSDs and virtual disks do work in parallel. Then pidstat -d as root to see which process is doing the I/O."

Why it helps

Slow storage is behind a large share of latency incidents, and the most quoted number, %util, is the least reliable one on SSDs and virtual disks that serve many requests in parallel. Reading await and the queue, then finding the process that does the I/O, gives you a diagnosis and a culprit. The fixes - io.max caps, ionice, moving a batch job - follow directly from it.

On the blog

Commands in this lesson

pidstat

FAQ

What is await?

iostat -x shows r_await and w_await: the average time in milliseconds for a read or a write, from the moment it was queued until it completed. It includes the time in the queue, so it is what the application feels. A healthy virtual disk answers in a millisecond or two.

Why is %util 100% not always saturation?

%util is the share of time the device had at least one request in flight. A single spinning disk at 100% is saturated, but SSDs, NVMe and virtual disks serve many requests at once and can do more. Look at aqu-sz and await growing instead.

How do I find the process doing the I/O?

sudo pidstat -d 1 shows kB_rd/s and kB_wr/s per process. Without sudo it hides other users' processes, because it cannot read their /proc/PID/io. iotop and biosnoop-bpfcc show the same from other angles, biosnoop per request.

What is iodelay in pidstat?

The time the process spent waiting for block I/O, from the kernel's delay accounting. On Ubuntu that accounting is off by default to save overhead, so the column shows 0 until you set kernel.task_delayacct=1 with sysctl. It is then a direct measure of who is hurt by the slow disk.

How do I stop a batch job from hurting everyone else?

Move it to a quiet time or another disk first. If it must run, cap it with IOReadBandwidthMax= or IOWriteBandwidthMax= in its unit (io.max in the cgroup), or lower its priority with ionice -c3, which only works with schedulers that honour priorities.

In an interview Mid

How do you tell whether a disk is the bottleneck?

I run iostat -xz 1 and read r_await and w_await, the latency including queueing, and aqu-sz, the queue length, rather than just %util, because SSDs and virtual disks serve requests in parallel. If latency is high I run sudo pidstat -d 1 to see which process does the I/O, and biolatency-bpfcc for the latency distribution. The fix is moving or capping the heavy job, for example with io.max.

Also asked: Why can %util be misleading on an SSD? · Why does pidstat -d show nothing for some processes? · How would you limit a batch job's disk bandwidth?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.