What "load 8" means
Linux's load average is an exponentially damped average of the number of tasks that are running or runnable, plus tasks in uninterruptible sleep (state D, usually waiting on disk or NFS). It is not a CPU percentage, and on Linux it is not even only CPU: a box with 20 tasks stuck on a dead NFS server shows load 20 with idle CPUs (the processes chapter showed it). The three numbers are 1, 5 and 15 minute averages: compare them to see the trend.
Saturation for CPUs is the run queue: more runnable threads than CPUs. Three places show it:
vmstat 1: thercolumn (running + waiting) above the CPU count (nproc);pidstat 1:%waitper task, the time it was ready but did not get a CPU;- PSI, the kernel's pressure stall information:
/proc/pressure/cpusays what share of the time some task was waiting for a CPU.
$ cat /proc/pressure/cpu
some avg10=90.13 avg60=89.49 avg300=79.99 total=623624351
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some avg10=90.13: in the last 10 seconds, 90% of the time at least one task was waiting for a CPU; avg60 and avg300 are the same share over 1 and 5 minutes (compare them for the trend), total the stall time in microseconds since boot. (full is always 0 for the CPU at system level.) PSI is on in Ubuntu's kernel; io and memory have the same format and the same meaning for disks and memory.
How long they wait is the run queue latency, measured with eBPF (runqlat-bpfcc, later in this chapter): microseconds on a healthy box, milliseconds when saturated - added to every request.
The CPU columns. us user code, sy kernel (system calls, page faults, interrupts), ni niced user code, id idle, wa iowait, st steal:
- iowait is idle time: the CPU had nothing to run while some task it ran waited for a disk. High
wasays "look at the disks", not "the CPU is busy". A CPU-bound job next to it makeswadisappear without the disk getting any faster. - steal is time the hypervisor gave this vCPU's turn to another guest. It matters on cloud VMs (burstable instances, noisy neighbours); this lab VM reports 0.
Rising or falling? If the 1-minute load is well above the 15-minute one, the problem is new and growing; the reverse means it is already over. A load of exactly the number of busy threads (here 8: four pricing workers, three indexer threads, one more) is a sign of CPU-bound work; a load that keeps climbing with idle CPUs points at tasks piling up in D.
Context switches (cs in vmstat, cswch/s and nvcswch/s in pidstat -w) come in two kinds. Voluntary switches happen when a task blocks (I/O, a lock, a sleep): normal. Involuntary (nvcswch/s) happen when the scheduler takes the CPU away because someone else must run: many of them on a task is another sign of CPU saturation.
One CPU at 100%. mpstat -P ALL 1 shows each CPU. A single-threaded program (a Python script, a Node.js process, one busy Redis) can never use more than one CPU: the box looks 50% idle on 2 CPUs while that program is completely CPU-bound. pidstat -t -p PID 1 shows its threads.
Who gets the CPU. The scheduler (EEVDF since kernel 6.6, the successor of CFS) shares CPUs by weight. nice 0 is weight 1024; each nice step is about 25% less (nice 19 is ~15). Change it with renice -n 19 -p PID (only root may lower it again), or for a unit with Nice= or CPUWeight= (cgroup cpu.weight, default 100): sudo systemctl set-property reindex.service CPUWeight=20. Weights only matter when CPUs are contended - an idle box still gives the batch job everything.
$ sudo renice -n 19 -p $(pgrep indexer)
17386 (process ID) old priority 0, new priority 19
The load does not drop after renicing: the same threads are still runnable. What drops is the service's %wait. That is the right fix for a batch job: it finishes later, nobody waits for it.
In an interview: "Load average counts runnable and uninterruptible tasks, so I compare it with the CPU count and check vmstat r and wa: high load with idle CPUs is usually tasks stuck in D on I/O, not CPU."