OnCallReadyStart learning

Linux Lesson 8.2 · Linux Performance & eBPF · 7 min read

The 60-second checklist

In plain words

When you walk into a room where something is wrong, you look around before you touch anything: is there smoke, is water on the floor, is a window broken. The 60-second checklist is that first look for a Linux box. Ten short commands, each one glancing at a different part of the machine, and after a minute you know which part to look at closely.

Ten commands, one minute, most of the answer

Netflix's performance team published this list in 2015 ("Linux Performance Analysis in 60,000 Milliseconds"). It is still the best first minute on any Linux box: all ten tools ship with Ubuntu Server (sysstat is installed by default and collects every 10 minutes).

#CommandLook for
1uptimeload averages: rising (1 min > 15 min) or falling? compare with nproc
2sudo dmesg -T | tailOOM kills, I/O errors, TCP drops, hung tasks
3vmstat 1r (runnable) > CPU count = saturation, b blocked, si/so swapping, us sy id wa st
4mpstat -P ALL 1one CPU hot while others idle = a single-threaded bottleneck
5pidstat 1which processes use the CPU, per second (top's data, but scrolling)
6iostat -xz 1r_await/w_await latency, aqu-sz queue, %util per disk
7free -mavailable, not free; swap used
8sar -n DEV 1interface throughput vs what the link can do
9sar -n TCP,ETCP 1new connections per second, retransmits (retrseg/s)
10topeverything at once, a final check

dmesg is root-only on Ubuntu (kernel.dmesg_restrict = 1), hence the sudo. The tools taking an interval (1) run until Ctrl+C: give them three to five lines and stop them.

The first line lies. vmstat 1 and iostat 1 print an average since boot first; only the lines after it are the last second. iostat -y skips it. mpstat and sar do not have this line: their first report is already an interval.

On a busy box, the first three already tell the story:

$ uptime
 20:00:05 up  2:17,  1 user,  load average: 8.00, 6.94, 3.96
$ nproc
2
$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 8  0  12544 1766796  88020 2845980    0    0   145   103  150  230  7  1 92  0  0  0
 8  0  12544 1766796  88020 2845980    0    0  1465   996  422 4222 91  6  3  0  0  0
 8  0  12544 1766796  88020 2845980    0    0  1460   929  408 4414 93  6  1  0  0  0

Load 8 on 2 CPUs, rising (8.00 now, 3.96 over 15 minutes), r = 8 runnable threads for 2 CPUs, idle near 0, no I/O wait, no swapping: CPU saturation. The next question is who:

$ pidstat 1 1
Linux 7.0.0-31-generic (oncall-lab) 	10/22/26 	_aarch64_	(2 CPU)

20:00:10      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
20:00:11     1001      1210    1.14    0.16    0.00    0.00    1.29     1  java
20:00:11     1001     17385   96.55    7.15    0.00   19.32  103.69     0  pricing
20:00:11     1001     17386   82.51    4.55    0.00   59.07   87.07     0  indexer

Average:      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
Average:     1001      1210    1.14    0.16    0.00    0.00    1.29     -  java
Average:     1001     17385   96.55    7.15    0.00   19.32  103.69     -  pricing
Average:     1001     17386   82.51    4.55    0.00   59.07   87.07     -  indexer

%CPU above 100 means several threads (together they keep both CPUs busy). %wait is the part that hurts latency: the time the task was ready to run but waited for a CPU (from /proc/PID/schedstat).

The disk line to read in iostat -xz 1: -x extended columns, -z hides idle devices. r_await/w_await are milliseconds per request including the queue; aqu-sz the average queue length; %util the share of time the device had work. On virtual and SSD devices that serve many requests in parallel, 100% util is not "full" - latency is the truth.

The TCP line to read in sar -n TCP,ETCP 1: active/s connections this box opened, passive/s connections it accepted, and retrseg/s retransmitted segments - a network or remote problem when it is more than a tiny fraction of oseg/s. (Older sysstat named the column retrans/s; the article you will find online shows that name.)

sar keeps history. sar without an interval reads today's file in /var/log/sysstat/: sar -u, sar -q, sar -n DEV for "what happened at 03:10?". sar -f /var/log/sysstat/sa21 reads another day.

In an interview: "First minute: uptime, dmesg, vmstat, mpstat, pidstat, iostat, free, sar for network and TCP, top. That tells me which resource is saturated and roughly who, before I touch anything."

Why it helps

The first minute of an incident decides whether the next hour is spent on the right thing. Ten standard commands cover load, kernel errors, CPU, memory, disks and the network, and they work on every Linux box without installing anything extra on Ubuntu. Doing them in the same order every time means you do not forget the one resource that turns out to be the problem.

On the blog

Commands in this lesson

uptime nproc vmstat pidstat

FAQ

Why does the first line of vmstat lie?

vmstat's first line shows averages since boot, not the current state. Only the lines after it, one per interval, show what is happening now. The same is true for iostat without -y and for mpstat's first report. Always read the second line onwards.

What does dmesg -T | tail tell me?

Whether the kernel has complained recently: OOM kills, I/O errors from a disk, TCP drops, a hung task. Those are the errors in USE, and they are easy to miss because no performance counter shows them. -T prints human-readable times so you can match them to the incident.

Which iostat columns matter first?

r_await and w_await (average milliseconds per read and write, queueing included), aqu-sz (the average queue length) and %util. Latency is the truth: a disk with 30 ms writes is a problem even if %util is not at 100. The -z option hides idle devices.

What does sar -n TCP,ETCP 1 show?

TCP connection rates (active/s, passive/s) and the error counters, most importantly retrseg/s (retrans/s in older sysstat), segments retransmitted per second. Retransmits add hundreds of milliseconds to the requests they hit, so a few percent of segments explains tail latency without any bandwidth problem.

Can I see what happened before I logged in?

Yes, if sysstat collection is on, which it is by default on Ubuntu: sar reads its history files under /var/log/sysstat every ten minutes. sar -u, sar -d or sar -n DEV with -s and -e show the CPU, disks or network at the time of the alert.

In an interview Mid

What do you run in the first minute on a slow Linux box?

uptime for the load trend, dmesg -T | tail for kernel errors, vmstat 1 for the run queue, swapping and CPU modes, mpstat -P ALL 1 for one hot CPU, pidstat 1 for the busy processes, iostat -xz 1 for disk latency and queues, free -m for available memory, sar -n DEV 1 and sar -n TCP,ETCP 1 for traffic and retransmits, then top. Together they name the saturated resource and roughly who.

Also asked: Why do you ignore the first line of vmstat? · What does sar give you that the other commands do not? · What are you looking for in dmesg?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.