Tracing a live server is like a doctor running tests on a patient who is still at work. The tests must be short, gentle and stop on their own, so they do not make the patient sicker. And at the end the doctor writes a short note: when, what the tests showed, what was done, and whether it helped. That note is what the next doctor reads.
Use it on a live box without making things worse
Cost goes with the event rate. Counters (vmstat, iostat, sar) cost nothing. Sampling (perf record -F 99, profile-bpfcc) costs a fixed, small amount. Tracing costs per event: a bpftrace probe on raw_syscalls:sys_enter on a box doing a million syscalls a second can take whole percent of CPU; on sched_process_exec it costs nothing. Before attaching to something hot, count it first (funccount-bpfcc, or a bpftrace count() for one second).
Bound everything. Use a duration (runqlat-bpfcc 5 1, profile-bpfcc 10, perf record -- sleep 10, bpftrace interval:s:10 { exit(); }), a PID filter where you can, and print summaries rather than events. A snooper left running in a tmux session for a week is how tracing gets a bad name.
Root and audit. All of it runs as root: say in the incident channel what you are attaching and for how long. sudo bpftool prog list shows anything still attached afterwards.
Kernel versions. Tracepoints and the bcc/bpftrace tools work on any Ubuntu LTS kernel since 20.04; kprobe-based one-liners may need a different function name on another kernel version (bpftrace -l 'kprobe:*name*' finds it).
Profile on-CPU (perf record / profile-bpfcc) or off-CPU (offcputime-bpfcc).
A specific tracer for the question that is left (execsnoop, opensnoop, biolatency, tcpretrans, ext4slower, a bpftrace one-liner).
The write-up is the deliverable. Five lines are enough:
14:20 orders-api p99 2.1 s. CPUs 20% busy, iowait 30% (vmstat).
14:22 vdb: w_await 31 ms, aqu-sz 45 (iostat -x); rsync 90 MB/s (pidstat -d) =
restore-reports.service, one-off ticket OPS-4411.
14:24 stopped restore-reports; vdb w_await 1.2 ms, p99 back to 180 ms at 14:26.
Follow-up: run restores off-peak with IOWriteBandwidthMax=; alert on vdb w_await > 20 ms.
Time, evidence (with the tool that showed it), action, the result, the follow-up.
From one box to the fleet. Everything here has a fleet version, and this chapter is what you do when you are on the box.
Later (Ch 27, 43, 44, 45): node_exporter exports the same /proc counters (CPU modes, PSI, disk times, TCP retransmits) and cAdvisor the cgroup throttling to Prometheus; the Observability chapters build the dashboards and alerts on them. Continuous profilers (Parca, Pyroscope, Grafana's eBPF profiler) do the same for profiles: profile-style sampling all the time, the flame graphs kept for when you need them.
In an interview: "I keep tracing bounded: a duration, a PID filter, summaries in the kernel instead of per-event output, and I count an event's rate before attaching to it. Then I write up time, evidence, action and result."
Why it helps
Every tool in this chapter runs on boxes that are already in trouble, often as root. Bounding the tools, counting before tracing and saying in the channel what you attached keeps you from making the incident worse. The write-up - time, evidence, action, result, follow-up - is what turns your investigation into something the team can learn from, and what reviewers and interviewers actually see.
FAQ
How do I keep tracing cheap?
Count the event's rate first with funccount-bpfcc or a one-second bpftrace count(), prefer summarizers that aggregate in the kernel over snoopers that print every event, filter by PID, and give every tool a duration, such as runqlat-bpfcc 5 1 or perf record -- sleep 10.
What is the difference between sampling and tracing?
Sampling looks at the system at a fixed rate, for example perf record -F 99, so its cost does not depend on how busy the box is. Tracing runs on every event, so its cost grows with the event rate. Sampling shows where time goes; tracing shows every occurrence of something specific.
How do I make sure nothing is left attached?
Run tools with a duration or stop them with Ctrl+C, never leave a snooper in a background session, and check afterwards with sudo bpftool prog list: tracing programs from bcc or bpftrace should be gone once the tool exits.
What goes into the write-up?
Five lines are enough: the time and symptom, the evidence with the tool that showed it, the action you took, the result with numbers after the change, and the follow-up that prevents a repeat. Anyone reading it later should be able to check every claim.
How does this scale to many servers?
The same counters are exported continuously by node_exporter (CPU modes, PSI, disk await, retransmits) to a monitoring system, cgroup throttling by cAdvisor, and continuous profilers keep flame graphs for every service; the Observability chapters build that. On-box tools remain what you use when the dashboards point at one box.
In an interview Mid
How do you use tracing tools safely on a production box?
I keep tracing bounded: a duration on every tool, a PID filter where I can, summaries aggregated in the kernel instead of per-event output, and I count an event's rate with funccount-bpfcc or a bpftrace count() before attaching to it. I say in the incident channel what I attach, check with bpftool prog list afterwards, and write up time, evidence, action and result.
Also asked: What is the difference between sampling and tracing overhead? · What belongs in a performance incident write-up? · How do on-box tools relate to fleet monitoring?