OnCallReadyStart learning

Linux Lesson 8.13 · Linux Performance & eBPF · 6 min read

perf: counting events

In plain words

perf is a tally counter you can attach to a program. You tell it what to count - how much CPU time the program used, how often it was switched out, how many page faults it took - and run the program. At the end it shows the totals. On real hardware it can also count what happens inside the processor itself, like cycles and cache misses.

The kernel's own profiler

perf is the tool that comes with the Linux kernel source (tools/perf). It counts events (perf stat), samples stacks (perf record), and shows a live profile (perf top). On Ubuntu 26.04 it is the package linux-perf - not installed by default (linux-tools-* no longer contains it):

$ perf
Command 'perf' not found, but can be installed with:
sudo apt install linux-perf
$ sudo apt install -y linux-perf
...
Setting up linux-perf:arm64 (7.0.0-38.38) ...

Events. Hardware events come from the CPU's performance monitoring unit (PMU): cycles, instructions, cache misses. Software events are kernel counters: task-clock, cpu-clock, context-switches, page-faults. Tracepoints are the kernel's static trace points (sched:sched_switch, block:block_rq_issue...). perf list shows what this box has.

This box has no PMU. It is a VM on Apple Silicon, and the hypervisor does not give guests the performance counters. Hardware events say <not supported>; profiling falls back to a software timer (perf does it for you): task-clock when you profile a process or a command, cpu-clock when you profile every CPU (-a). That is why the report says task-clock:pppH instead of cycles:P. Many cloud VM sizes are the same. Everything in this chapter works on the timer.

Who may use it. Ubuntu's kernel sets kernel.perf_event_paranoid to 4, stricter than upstream's 2: no performance events at all for unprivileged users.

$ cat /proc/sys/kernel/perf_event_paranoid
4
$ perf stat -- sleep 1
Error:
No supported events found.
Access to performance monitoring and observability operations is limited.
...

Use sudo. (Lowering it with sysctl kernel.perf_event_paranoid=2 lets users profile their own processes; on a shared production box, keep 4.)

perf stat runs a command, or watches a PID or the whole box, and counts:

$ cd ~/oncall-lab/labs/perf
$ sudo perf stat gzip -9 -c access-2026-10.log > /dev/null

 Performance counter stats for 'gzip -9 -c access-2026-10.log':

   <not supported>      cpu-cycles
                65      context-switches                 #      3.0 cs/sec  cs_per_second
                 1      cpu-migrations                   #      0.0 migrations/sec  migrations_per_second
               183      page-faults                      #      8.5 faults/sec  page_faults_per_second
         21,461.33 msec task-clock                       #      1.0 CPUs  CPUs_utilized

      21.506768133 seconds time elapsed

      18.885973333 seconds user
       2.575360000 seconds sys

task-clock is CPU time used; next to it, CPUs utilized = CPU time / wall time (1.0: one CPU busy the whole time - CPU-bound and single-threaded). seconds user / sys split it into program code and kernel. perf stat writes to stderr, so > /dev/null hides only gzip's output. Other targets: -p PID -- sleep 10 (a running process for 10 s), -a -- sleep 5 (all CPUs), -e context-switches,page-faults (only these), -e 'syscalls:sys_enter_openat' (count a tracepoint), -r 5 (five runs, mean and spread).

perf top is top for functions: a live, system-wide list of the hottest symbols across all processes and the kernel. sudo perf top -p PID for one process. It needs symbols (the next lesson) to be readable.

In an interview: "perf stat counts events for a command or a process: task-clock and CPUs utilized tell me if it is CPU-bound; on bare metal I add cycles and instructions for IPC. perf top and perf record tell me where the CPU time goes."

Why it helps

perf is the standard Linux profiler and part of the kernel source. perf stat answers "is this CPU-bound?" in one command, and the rest of perf (record, report, top) answers "where does the CPU time go?". Knowing why it fails - not installed, perf_event_paranoid, no PMU in a VM - saves the first ten minutes of every profiling session, and the questions come up in interviews.

Commands in this lesson

perf apt cat cd

FAQ

How do I install perf on Ubuntu 26.04?

sudo apt install linux-perf. Older releases shipped it in linux-tools packages matching the kernel; on 26.04 the linux-perf package provides /usr/bin/perf. perf version prints the version, which follows the kernel series, so a box on kernel 7.0 gets perf 7.0.

Why does perf refuse to run as a normal user?

kernel.perf_event_paranoid is 4 on Ubuntu, a distribution setting stricter than upstream's 2: no perf events at all for unprivileged users. Run perf with sudo, or lower the sysctl on a development machine. The error message prints the current value and the levels.

Why does perf stat say <not supported> for cycles?

The box has no PMU events: the hypervisor does not expose the processor's hardware counters to the VM, as on this lab's Apple Silicon host. Software events such as task-clock, context-switches and page-faults still work, and perf record falls back to sampling on a software timer (task-clock for a process, cpu-clock with -a).

What does CPUs utilized mean in perf stat?

task-clock (the CPU time the command used) divided by the elapsed time. 1.0 means one CPU busy for the whole run, 2.0 two CPUs. A value well below 1 for a single-threaded command means it spent much of its time waiting, not computing.

What is perf top for?

A live view of the hottest functions on the whole box, refreshed every couple of seconds, like top but per function instead of per process. It needs root and symbols. It is good for a quick look; perf record keeps the samples so you can study them afterwards.

In an interview Mid

What does perf stat tell you about a program?

perf stat counts events while a command or a process (-p PID) runs: task-clock and CPUs utilized say whether it is CPU-bound, context-switches and page-faults show how often it blocked or touched new memory, and seconds user / sys split the time. On bare metal I add cycles and instructions for IPC; in a VM without a PMU those show <not supported>. It needs sudo on Ubuntu because perf_event_paranoid is 4.

Also asked: Why does perf fail for normal users on Ubuntu? · What is a PMU and why might a VM not have one? · What does a low instructions-per-cycle value suggest?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.