OnCallReadyStart learning

Linux Lesson 8.4 · Linux Performance & eBPF · 7 min read

Load, run queues and CPU saturation

In plain words

Imagine a supermarket with two checkouts. The number of shoppers being served plus the ones waiting in line is the load. If there are eight people and two checkouts, six are waiting. Linux also counts shoppers who stepped away to fetch something from the back store, which is why a box can have a high load even when the checkouts are empty.

What "load 8" means

Linux's load average is an exponentially damped average of the number of tasks that are running or runnable, plus tasks in uninterruptible sleep (state D, usually waiting on disk or NFS). It is not a CPU percentage, and on Linux it is not even only CPU: a box with 20 tasks stuck on a dead NFS server shows load 20 with idle CPUs (the processes chapter showed it). The three numbers are 1, 5 and 15 minute averages: compare them to see the trend.

Saturation for CPUs is the run queue: more runnable threads than CPUs. Three places show it:

  • vmstat 1: the r column (running + waiting) above the CPU count (nproc);
  • pidstat 1: %wait per task, the time it was ready but did not get a CPU;
  • PSI, the kernel's pressure stall information: /proc/pressure/cpu says what share of the time some task was waiting for a CPU.
$ cat /proc/pressure/cpu
some avg10=90.13 avg60=89.49 avg300=79.99 total=623624351
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

some avg10=90.13: in the last 10 seconds, 90% of the time at least one task was waiting for a CPU; avg60 and avg300 are the same share over 1 and 5 minutes (compare them for the trend), total the stall time in microseconds since boot. (full is always 0 for the CPU at system level.) PSI is on in Ubuntu's kernel; io and memory have the same format and the same meaning for disks and memory.

How long they wait is the run queue latency, measured with eBPF (runqlat-bpfcc, later in this chapter): microseconds on a healthy box, milliseconds when saturated - added to every request.

The CPU columns. us user code, sy kernel (system calls, page faults, interrupts), ni niced user code, id idle, wa iowait, st steal:

  • iowait is idle time: the CPU had nothing to run while some task it ran waited for a disk. High wa says "look at the disks", not "the CPU is busy". A CPU-bound job next to it makes wa disappear without the disk getting any faster.
  • steal is time the hypervisor gave this vCPU's turn to another guest. It matters on cloud VMs (burstable instances, noisy neighbours); this lab VM reports 0.

Rising or falling? If the 1-minute load is well above the 15-minute one, the problem is new and growing; the reverse means it is already over. A load of exactly the number of busy threads (here 8: four pricing workers, three indexer threads, one more) is a sign of CPU-bound work; a load that keeps climbing with idle CPUs points at tasks piling up in D.

Context switches (cs in vmstat, cswch/s and nvcswch/s in pidstat -w) come in two kinds. Voluntary switches happen when a task blocks (I/O, a lock, a sleep): normal. Involuntary (nvcswch/s) happen when the scheduler takes the CPU away because someone else must run: many of them on a task is another sign of CPU saturation.

One CPU at 100%. mpstat -P ALL 1 shows each CPU. A single-threaded program (a Python script, a Node.js process, one busy Redis) can never use more than one CPU: the box looks 50% idle on 2 CPUs while that program is completely CPU-bound. pidstat -t -p PID 1 shows its threads.

Who gets the CPU. The scheduler (EEVDF since kernel 6.6, the successor of CFS) shares CPUs by weight. nice 0 is weight 1024; each nice step is about 25% less (nice 19 is ~15). Change it with renice -n 19 -p PID (only root may lower it again), or for a unit with Nice= or CPUWeight= (cgroup cpu.weight, default 100): sudo systemctl set-property reindex.service CPUWeight=20. Weights only matter when CPUs are contended - an idle box still gives the batch job everything.

$ sudo renice -n 19 -p $(pgrep indexer)
17386 (process ID) old priority 0, new priority 19

The load does not drop after renicing: the same threads are still runnable. What drops is the service's %wait. That is the right fix for a batch job: it finishes later, nobody waits for it.

In an interview: "Load average counts runnable and uninterruptible tasks, so I compare it with the CPU count and check vmstat r and wa: high load with idle CPUs is usually tasks stuck in D on I/O, not CPU."

Why it helps

The load average is the number everyone looks at first and most people misread. On Linux it includes tasks stuck waiting for disks in state D, so high load with idle CPUs is an I/O problem, not a CPU problem. Knowing how to read the run queue, iowait, steal and per-CPU usage separates "add CPUs" from "fix the disk" or "make it parallel", which is a classic interview question.

Commands in this lesson

cat renice

FAQ

What does the load average count on Linux?

Tasks that are running or runnable (state R) plus tasks in uninterruptible sleep (state D), averaged over 1, 5 and 15 minutes. Other systems count only runnable tasks; Linux adds D, so slow disks or a hung NFS server raise the load without using any CPU.

How do I see CPU saturation?

vmstat's r column above the number of CPUs (nproc) means tasks wait to run. pidstat 1 shows %wait per process, /proc/pressure/cpu shows the share of time tasks were stalled, and runqlat-bpfcc shows how long they waited. Utilization alone cannot show saturation.

Is iowait CPU usage?

No. iowait is idle time: the CPU had nothing to run, and at least one task it ran was waiting for I/O. It points at the disks, but it is a weak signal - a CPU-bound job can make it vanish without the disk getting faster. iostat and PSI io show the disk side properly.

What is steal time?

On a virtual machine, the time the virtual CPU wanted to run but the hypervisor gave the physical CPU to another guest. High steal means a noisy neighbour or an oversubscribed host. You cannot fix it inside the VM; you move the VM or buy guaranteed CPUs.

Why is one CPU at 100% and the other idle?

A single-threaded program can only use one CPU at a time. mpstat -P ALL 1 shows the imbalance and pidstat -t confirms one hot thread. More CPUs will not help; parallelising the work or a faster CPU will. nice and CPUWeight only decide who goes first.

In an interview Mid

The load average is 12 on a 4-CPU box but the CPUs are mostly idle. What is going on?

Linux load counts running or runnable tasks plus tasks in uninterruptible sleep (D). With idle CPUs the extra load is almost always tasks in D waiting on I/O. I check vmstat for b and wa, ps for processes in state D and where they wait (wchan, /proc/PID/stack), and iostat -x for the disk's latency. Adding CPUs would not change anything.

Also asked: How do you show CPU saturation rather than utilization? · Why is iowait not CPU usage? · What does steal time mean on a virtual machine?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.