OnCallReadyStart learning

Linux Lesson 8.1 · Linux Performance & eBPF · 6 min read

Performance work is a method, not a hunch

In plain words

A doctor with a sick patient does not guess. They check the same things in the same order every time: temperature, pulse, blood pressure, breathing. The USE method is that routine for a computer. For every part of the box - CPUs, memory, disks, network - you ask three questions: how busy is it, is work waiting in line for it, and is it reporting errors. The answers point at the part that is sick.

Start from a method

The problem. "The site is slow" arrives with no detail. The tempting moves are the anti-methods: look at the tool you know (the streetlight anti-method), change things at random until it gets better (random change), or blame whatever you looked at last (blame someone else). They waste the first hour of an incident and sometimes make it worse.

The USE method (Brendan Gregg) turns it into a checklist. For every resource - CPUs, memory, disks, network interfaces, and software resources like a connection pool or a cgroup limit - ask three questions:

LetterQuestionCPU exampleDisk example
UtilizationHow busy is it (% of time, or % of capacity)?%usr + %sys in mpstat%util in iostat
SaturationIs work queuing because it is full?run queue longer than the CPU count, PSI cpuaqu-sz, await growing
ErrorsIs it failing?(rare: machine checks in dmesg)I/O errors in dmesg, SMART

Saturation is the one people skip, and the one that hurts: a CPU at 100% with nothing waiting is fine; a CPU at 100% with six threads waiting for it adds milliseconds to every request.

Three words, three meanings. Latency is how long one operation takes (the user feels it). Throughput is how many per second. Utilization is how busy the resource is. A box can have low utilization and terrible latency (a lock, a slow disk, CPU throttling); the reverse is also common (a batch job using 100% CPU that nobody is waiting for).

Where the numbers come from. Every classic tool reads counters the kernel keeps:

ToolReads
uptime, top/proc/loadavg, /proc/stat, /proc/PID/stat
vmstat, mpstat/proc/stat, /proc/meminfo, /proc/vmstat
iostat/proc/diskstats
sar -n/proc/net/dev, /proc/net/snmp
pidstat/proc/PID/stat, /proc/PID/io, /proc/PID/schedstat

They are cheap (counters, no tracing) and always there, which is why you start with them. The eBPF tools later in this chapter answer the questions counters cannot: which process, which file, which code path, how long each operation took.

$ cat /proc/loadavg
8.00 6.94 3.96 3/47 17391

The four layers you check, top to bottom: the application (its own latency metrics), the runtime (JVM GC, Python GIL), the kernel (scheduler, page cache, block layer, TCP), the hardware or hypervisor (disks, vCPUs, steal). Most incidents are found in the first two minutes by the counters; the rest need tracing.

The USE table for this box. Fill it in once, and the first minute of every incident is a lookup instead of a search:

ResourceUtilizationSaturationErrors
CPUs (2)mpstat -P ALL 1: 100 - %idle per CPUvmstat 1 r > 2, /proc/pressure/cpu, pidstat %waitdmesg (rare)
Memory (6 GB)free -m: used vs availablevmstat si/so, /proc/pressure/memory, OOM killsdmesg OOM lines
Disks (vda, vdb, vdc)iostat -xz 1 %utilaqu-sz, await, /proc/pressure/iodmesg I/O errors
Network (enp0s1)sar -n DEV 1 kB/s vs link speedsar -n EDEV drops, ss -tl accept queuessar -n EDEV errors, retransmits
cgroup CPU limitscpu.stat usagenr_throttled, cpu.pressure-
File descriptorsls /proc/PID/fd | wc -l vs ulimit -n-EMFILE in the app's log

Software resources (a thread pool, a connection pool, a lock) belong in the table too; the application's own metrics usually answer them.

Known unknowns. The method also tells you what you did not check. "CPU and disks are fine, I have not looked at the network" is a useful sentence in an incident channel; "it's not the server" is not.

Write it down. Start a note with the time, what you ran and what you saw, before you change anything. A fix without the evidence cannot be reviewed, and the postmortem needs it.

In an interview: "I use the USE method: for each resource I check utilization, saturation and errors, starting with the cheap counters (uptime, vmstat, mpstat, iostat) and only then tracing. Saturation, things queuing, is usually where latency comes from."

Why it helps

Without a method, performance work turns into guessing: restarting the service, adding CPUs, blaming the network. Those anti-methods sometimes work and teach you nothing. A short, repeatable checklist finds the saturated resource quickly, shows which resources you have not checked yet, and gives the incident channel a clear sentence instead of "it's not the server". Interviewers ask for it by name.

Commands in this lesson

cat

FAQ

What is the difference between utilization and saturation?

Utilization is how busy a resource is, for example a CPU at 80%. Saturation is how much work is waiting because it is busy: tasks in the run queue, requests queued at a disk. Latency usually comes from saturation, so a resource at 70% with a long queue matters more than one at 95% without one.

Which resources belong in the USE table?

The hardware ones - CPUs, memory, disks, network interfaces - and the software ones: cgroup CPU and memory limits, file descriptors, thread and connection pools, locks. The application's own metrics usually answer the software resources; the Linux counters answer the hardware ones.

What is an anti-method?

A way of working that feels productive but is not driven by evidence: the streetlight anti-method (look only where your favourite tool shines), random changes until the problem goes away, or blaming another team. They waste time and leave you without an explanation for the write-up.

Why start with counters instead of tracing?

Counters such as vmstat, mpstat and iostat are always there, cost nothing and cover the whole box in a minute. Tracing is precise but narrow and costs something per event. The counters tell you which resource to trace, so you only pay for the tracing that answers a real question.

Where do the numbers come from?

Almost all of them come from the kernel: /proc and /sys files that tools like vmstat and iostat read and turn into rates. Some come from the application itself (GC pauses, pool sizes) and some from the hypervisor (steal time). Knowing the source tells you what a number can and cannot see.

In an interview Mid

What is the USE method?

For every resource - CPUs, memory, disks, network and software resources such as pools and cgroup limits - you check utilization (how busy it is), saturation (is work queuing for it) and errors. It turns "it is slow" into a checklist, starts with cheap counters like vmstat, mpstat and iostat, and catches saturation, which is where latency usually comes from.

Also asked: What is the difference between utilization and saturation? · Which software resources would you add to a USE checklist? · Why start with counters rather than tracing?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.