Start from a method
The problem. "The site is slow" arrives with no detail. The tempting moves are the anti-methods: look at the tool you know (the streetlight anti-method), change things at random until it gets better (random change), or blame whatever you looked at last (blame someone else). They waste the first hour of an incident and sometimes make it worse.
The USE method (Brendan Gregg) turns it into a checklist. For every resource - CPUs, memory, disks, network interfaces, and software resources like a connection pool or a cgroup limit - ask three questions:
| Letter | Question | CPU example | Disk example |
|---|---|---|---|
| Utilization | How busy is it (% of time, or % of capacity)? | %usr + %sys in mpstat | %util in iostat |
| Saturation | Is work queuing because it is full? | run queue longer than the CPU count, PSI cpu | aqu-sz, await growing |
| Errors | Is it failing? | (rare: machine checks in dmesg) | I/O errors in dmesg, SMART |
Saturation is the one people skip, and the one that hurts: a CPU at 100% with nothing waiting is fine; a CPU at 100% with six threads waiting for it adds milliseconds to every request.
Three words, three meanings. Latency is how long one operation takes (the user feels it). Throughput is how many per second. Utilization is how busy the resource is. A box can have low utilization and terrible latency (a lock, a slow disk, CPU throttling); the reverse is also common (a batch job using 100% CPU that nobody is waiting for).
Where the numbers come from. Every classic tool reads counters the kernel keeps:
| Tool | Reads |
|---|---|
uptime, top | /proc/loadavg, /proc/stat, /proc/PID/stat |
vmstat, mpstat | /proc/stat, /proc/meminfo, /proc/vmstat |
iostat | /proc/diskstats |
sar -n | /proc/net/dev, /proc/net/snmp |
pidstat | /proc/PID/stat, /proc/PID/io, /proc/PID/schedstat |
They are cheap (counters, no tracing) and always there, which is why you start with them. The eBPF tools later in this chapter answer the questions counters cannot: which process, which file, which code path, how long each operation took.
$ cat /proc/loadavg
8.00 6.94 3.96 3/47 17391
The four layers you check, top to bottom: the application (its own latency metrics), the runtime (JVM GC, Python GIL), the kernel (scheduler, page cache, block layer, TCP), the hardware or hypervisor (disks, vCPUs, steal). Most incidents are found in the first two minutes by the counters; the rest need tracing.
The USE table for this box. Fill it in once, and the first minute of every incident is a lookup instead of a search:
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| CPUs (2) | mpstat -P ALL 1: 100 - %idle per CPU | vmstat 1 r > 2, /proc/pressure/cpu, pidstat %wait | dmesg (rare) |
| Memory (6 GB) | free -m: used vs available | vmstat si/so, /proc/pressure/memory, OOM kills | dmesg OOM lines |
| Disks (vda, vdb, vdc) | iostat -xz 1 %util | aqu-sz, await, /proc/pressure/io | dmesg I/O errors |
| Network (enp0s1) | sar -n DEV 1 kB/s vs link speed | sar -n EDEV drops, ss -tl accept queues | sar -n EDEV errors, retransmits |
| cgroup CPU limits | cpu.stat usage | nr_throttled, cpu.pressure | - |
| File descriptors | ls /proc/PID/fd | wc -l vs ulimit -n | - | EMFILE in the app's log |
Software resources (a thread pool, a connection pool, a lock) belong in the table too; the application's own metrics usually answer them.
Known unknowns. The method also tells you what you did not check. "CPU and disks are fine, I have not looked at the network" is a useful sentence in an incident channel; "it's not the server" is not.
Write it down. Start a note with the time, what you ran and what you saw, before you change anything. A fix without the evidence cannot be reviewed, and the postmortem needs it.
In an interview: "I use the USE method: for each resource I check utilization, saturation and errors, starting with the cheap counters (uptime, vmstat, mpstat, iostat) and only then tracing. Saturation, things queuing, is usually where latency comes from."