Load average 15 on 4 cores, but the CPU is 90% idle

The alert says load is 15. The box has 4 cores. You open top expecting a fire, and the CPU is nearly idle. Both numbers are telling the truth.

Load average is not CPU usage

On Linux, the load average counts tasks that are:

• R - running, or waiting for a CPU, and
• D - in uninterruptible sleep, usually waiting for disk or network storage.

The three numbers are 1-, 5- and 15-minute averages. A pile of processes stuck in D adds to the load while using no CPU at all. So "high load + idle CPU" points straight at I/O.

Find the D-state processes

$ ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /D/'

The STAT column shows D. wchan is the kernel function each one is sleeping in, and it usually points at the storage layer involved (NFS, a block device, a filesystem). As root, /proc/<pid>/stack shows the whole kernel stack. In top, a high wa (I/O wait) tells the same story.

Why kill -9 does nothing

A process in D is inside a kernel call that can't be interrupted, not even by SIGKILL. The signal is queued and only delivered once the kernel call returns, and if the storage never answers, that can be a very long time. (Some newer kernel code paths, NFS among them, use "killable" waits that still show as D but do let SIGKILL through. When kill -9 does nothing, you're in one that doesn't.)

$ sudo kill -9 4211
$ ps -o pid,stat,comm -p 4211
  PID STAT COMMAND
 4211 D    backup-sync

That's the real answer to the interview question "a process won't die even with kill -9, what state is it in and what does it tell you?" It's in D. The problem is the storage it's waiting on, not the process. (Z is the other state that shrugs off kill -9, for the opposite reason: zombies are already dead.)

What to do

• Find what they're waiting on: an NFS server (can the client still resolve its name?), a failing disk, a full or hung storage backend.
• Fix that, and the processes finish their call, receive the signal and go away on their own.
• For network mounts, choose mount options on purpose (timeouts, soft vs hard): they decide what happens the next time the server disappears.

Don't reboot first. If the mount comes back the same way after the reboot, so does the problem. On Kubernetes, a pod whose process is stuck in D stays Terminating past its grace period: the final SIGKILL can't land either.