The kernel's own profiler
perf is the tool that comes with the Linux kernel source (tools/perf). It counts events (perf stat), samples stacks (perf record), and shows a live profile (perf top). On Ubuntu 26.04 it is the package linux-perf - not installed by default (linux-tools-* no longer contains it):
$ perf
Command 'perf' not found, but can be installed with:
sudo apt install linux-perf
$ sudo apt install -y linux-perf
...
Setting up linux-perf:arm64 (7.0.0-38.38) ...
Events. Hardware events come from the CPU's performance monitoring unit (PMU): cycles, instructions, cache misses. Software events are kernel counters: task-clock, cpu-clock, context-switches, page-faults. Tracepoints are the kernel's static trace points (sched:sched_switch, block:block_rq_issue...). perf list shows what this box has.
This box has no PMU. It is a VM on Apple Silicon, and the hypervisor does not give guests the performance counters. Hardware events say <not supported>; profiling falls back to a software timer (perf does it for you): task-clock when you profile a process or a command, cpu-clock when you profile every CPU (-a). That is why the report says task-clock:pppH instead of cycles:P. Many cloud VM sizes are the same. Everything in this chapter works on the timer.
Who may use it. Ubuntu's kernel sets kernel.perf_event_paranoid to 4, stricter than upstream's 2: no performance events at all for unprivileged users.
$ cat /proc/sys/kernel/perf_event_paranoid
4
$ perf stat -- sleep 1
Error:
No supported events found.
Access to performance monitoring and observability operations is limited.
...
Use sudo. (Lowering it with sysctl kernel.perf_event_paranoid=2 lets users profile their own processes; on a shared production box, keep 4.)
perf stat runs a command, or watches a PID or the whole box, and counts:
$ cd ~/oncall-lab/labs/perf
$ sudo perf stat gzip -9 -c access-2026-10.log > /dev/null
Performance counter stats for 'gzip -9 -c access-2026-10.log':
<not supported> cpu-cycles
65 context-switches # 3.0 cs/sec cs_per_second
1 cpu-migrations # 0.0 migrations/sec migrations_per_second
183 page-faults # 8.5 faults/sec page_faults_per_second
21,461.33 msec task-clock # 1.0 CPUs CPUs_utilized
21.506768133 seconds time elapsed
18.885973333 seconds user
2.575360000 seconds sys
task-clock is CPU time used; next to it, CPUs utilized = CPU time / wall time (1.0: one CPU busy the whole time - CPU-bound and single-threaded). seconds user / sys split it into program code and kernel. perf stat writes to stderr, so > /dev/null hides only gzip's output. Other targets: -p PID -- sleep 10 (a running process for 10 s), -a -- sleep 5 (all CPUs), -e context-switches,page-faults (only these), -e 'syscalls:sys_enter_openat' (count a tracepoint), -r 5 (five runs, mean and spread).
perf top is top for functions: a live, system-wide list of the hottest symbols across all processes and the kernel. sudo perf top -p PID for one process. It needs symbols (the next lesson) to be readable.
In an interview: "perf stat counts events for a command or a process: task-clock and CPUs utilized tell me if it is CPU-bound; on bare metal I add cycles and instructions for IPC. perf top and perf record tell me where the CPU time goes."