Linux Performance & eBPF
Find out why a Linux box is slow in minutes: the USE method and the 60-second checklist, CPU saturation and cgroup throttling, disk and network latency, perf profiles and flame graphs, and eBPF tracing with the bcc tools and bpftrace.
In plain words
When a car feels slow, a good mechanic does not start by taking the engine apart. First the dashboard: speed, revs, temperature, fuel. Then a closer look at the part the dashboard points at. Only at the end special tools that watch one part while the engine runs. This chapter does the same for a Linux box: cheap counters first, then profiles, then eBPF tracing that watches the kernel while it works.
Why it matters on call
"The service is slow" is the most common page an on-call engineer gets, and the most common wrong reaction is to restart things or add CPUs at random. A method that finds the saturated resource in a minute, and tools that name the process, the stack or the file behind it, turn that page into a short, evidence-based write-up. The same questions come up in every SRE interview: load average, iowait, throttling, flame graphs and eBPF.
Lessons
- 8.1Performance work is a method, not a hunch
- 8.2The 60-second checklist
- 8.4Load, run queues and CPU saturation
- 8.7CPU limits in cgroups: quota and throttling
- 8.9Disk I/O: latency, queues and who is reading
- 8.11Network: throughput, connections and retransmits
- 8.13perf: counting events
- 8.15Profiling: perf record, report and script
- 8.17Flame graphs
- 8.19eBPF: safe programs inside the kernel
- 8.21The bcc tools: answers in one command
- 8.24bpftrace: one-liners for any question
- 8.26Off-CPU time and tail latency
- 8.31System calls: what a process asks the kernel
- 8.34Tracing in production, and writing it up
Open this chapter in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.
Questions people ask
Which tools does this chapter use?
sysstat (mpstat, pidstat, iostat, sar), vmstat and top for the counters, perf and the FlameGraph scripts for profiles, the bcc tools (installed on Ubuntu with a -bpfcc suffix), bpftrace and bpftool for eBPF, plus renice, taskset and ionice to change who gets the CPU and the disk.
Are these tools installed on a fresh Ubuntu server?
sysstat, bpfcc-tools and bpftrace are installed on every Ubuntu Server since 24.04 LTS. perf is not: on Ubuntu 26.04 it comes from the linux-perf package. bpftool is its own package too. The labs make you install the two that are missing, as you would on a real box.
What is simulated here?
The load itself: the lab services are small programs whose CPU, disk, network and system call behaviour is modelled, and the tools read that model. Output formats, options, error messages and permissions follow the real tools' source code; the numbers come from the scenario, not from real hardware.
Do I need root for all of this?
For most of it, yes. On Ubuntu perf_event_paranoid is 4, so perf needs sudo; bpftrace, the bcc tools and bpftool need root because unprivileged BPF is off. vmstat, mpstat, iostat and sar work as a normal user, and pidstat -d shows only your own processes without sudo.
How does this connect to the Linux chapters?
It uses everything they taught: process states (R, S, D), systemd units and their cgroups, the page cache, file descriptors and the journal. The Linux chapters explained what those things are; this chapter measures them when something is slow and turns the numbers into a diagnosis.