Ten commands, one minute, most of the answer
Netflix's performance team published this list in 2015 ("Linux Performance Analysis in 60,000 Milliseconds"). It is still the best first minute on any Linux box: all ten tools ship with Ubuntu Server (sysstat is installed by default and collects every 10 minutes).
| # | Command | Look for |
|---|---|---|
| 1 | uptime | load averages: rising (1 min > 15 min) or falling? compare with nproc |
| 2 | sudo dmesg -T | tail | OOM kills, I/O errors, TCP drops, hung tasks |
| 3 | vmstat 1 | r (runnable) > CPU count = saturation, b blocked, si/so swapping, us sy id wa st |
| 4 | mpstat -P ALL 1 | one CPU hot while others idle = a single-threaded bottleneck |
| 5 | pidstat 1 | which processes use the CPU, per second (top's data, but scrolling) |
| 6 | iostat -xz 1 | r_await/w_await latency, aqu-sz queue, %util per disk |
| 7 | free -m | available, not free; swap used |
| 8 | sar -n DEV 1 | interface throughput vs what the link can do |
| 9 | sar -n TCP,ETCP 1 | new connections per second, retransmits (retrseg/s) |
| 10 | top | everything at once, a final check |
dmesg is root-only on Ubuntu (kernel.dmesg_restrict = 1), hence the sudo. The tools taking an interval (1) run until Ctrl+C: give them three to five lines and stop them.
The first line lies. vmstat 1 and iostat 1 print an average since boot first; only the lines after it are the last second. iostat -y skips it. mpstat and sar do not have this line: their first report is already an interval.
On a busy box, the first three already tell the story:
$ uptime
20:00:05 up 2:17, 1 user, load average: 8.00, 6.94, 3.96
$ nproc
2
$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r b swpd free buff cache si so bi bo in cs us sy id wa st gu
8 0 12544 1766796 88020 2845980 0 0 145 103 150 230 7 1 92 0 0 0
8 0 12544 1766796 88020 2845980 0 0 1465 996 422 4222 91 6 3 0 0 0
8 0 12544 1766796 88020 2845980 0 0 1460 929 408 4414 93 6 1 0 0 0
Load 8 on 2 CPUs, rising (8.00 now, 3.96 over 15 minutes), r = 8 runnable threads for 2 CPUs, idle near 0, no I/O wait, no swapping: CPU saturation. The next question is who:
$ pidstat 1 1
Linux 7.0.0-31-generic (oncall-lab) 10/22/26 _aarch64_ (2 CPU)
20:00:10 UID PID %usr %system %guest %wait %CPU CPU Command
20:00:11 1001 1210 1.14 0.16 0.00 0.00 1.29 1 java
20:00:11 1001 17385 96.55 7.15 0.00 19.32 103.69 0 pricing
20:00:11 1001 17386 82.51 4.55 0.00 59.07 87.07 0 indexer
Average: UID PID %usr %system %guest %wait %CPU CPU Command
Average: 1001 1210 1.14 0.16 0.00 0.00 1.29 - java
Average: 1001 17385 96.55 7.15 0.00 19.32 103.69 - pricing
Average: 1001 17386 82.51 4.55 0.00 59.07 87.07 - indexer
%CPU above 100 means several threads (together they keep both CPUs busy). %wait is the part that hurts latency: the time the task was ready to run but waited for a CPU (from /proc/PID/schedstat).
The disk line to read in iostat -xz 1: -x extended columns, -z hides idle devices. r_await/w_await are milliseconds per request including the queue; aqu-sz the average queue length; %util the share of time the device had work. On virtual and SSD devices that serve many requests in parallel, 100% util is not "full" - latency is the truth.
The TCP line to read in sar -n TCP,ETCP 1: active/s connections this box opened, passive/s connections it accepted, and retrseg/s retransmitted segments - a network or remote problem when it is more than a tiny fraction of oseg/s. (Older sysstat named the column retrans/s; the article you will find online shows that name.)
sar keeps history. sar without an interval reads today's file in /var/log/sysstat/: sar -u, sar -q, sar -n DEV for "what happened at 03:10?". sar -f /var/log/sysstat/sa21 reads another day.
In an interview: "First minute: uptime, dmesg, vmstat, mpstat, pidstat, iostat, free, sar for network and TCP, top. That tells me which resource is saturated and roughly who, before I touch anything."