OnCallReadyStart learning

Linux Lesson 8.15 · Linux Performance & eBPF · 7 min read

Profiling: perf record, report and script

In plain words

Imagine taking a photo of a busy kitchen 99 times a second and noting what each cook is doing in every photo. After ten seconds you have about a thousand photos. If the chef appears chopping onions in 400 of them, chopping onions is 40% of the work. A profiler does that with a program: it samples the stack many times a second and counts where it was.

Where does the CPU time go?

A profiler interrupts each CPU many times a second and writes down what was running: the process and its stack (which function called which). Functions that use the CPU show up in many samples. It needs no change to the program, and at 99 samples a second the overhead is tiny.

$ sudo perf record -F 99 -g -p $(pgrep pricing) -- sleep 10
[ perf record: Woken up 5 times to write data ]
[ perf record: Captured and wrote 0.546 MB perf.data (1742 samples) ]
  • -F 99: 99 samples per second per CPU. Not 100, so sampling does not run in lockstep with something that happens every 10 ms.
  • -g: record the call stack of each sample.
  • -p PID one process (or -a all CPUs); -- sleep 10 only sets how long to record.
  • It writes ./perf.data (mode 0600, owned by root): read it with sudo too.

perf report --stdio --no-children lists functions by self time - samples where that function was the one running:

# Samples: 1K of event 'task-clock:pppH'
# Event count (approx.): 17595959596
#
# Overhead  Command  Shared Object      Symbol
# ........  .......  .................  ............................
#
    21.01%  pricing  libc.so.6          [.] parse_expression
            |
            ---parse_expression
               parse_branch
               parse_reg_exp
               re_compile_internal
               regcomp
               match_rule
               apply_promotions
               ...
    12.00%  pricing  libc.so.6          [.] check_matching

[.] is user code, [k] kernel. The tree under each line shows who called it, leaf first. Without --no-children (the default when stacks were recorded) perf adds a Children column: time in the function and everything it called. main has ~100% children, ~0% self. Sort by self to find the hot leaf, by children to find the expensive path. On a terminal, perf report without --stdio opens an interactive browser (arrows, Enter to expand).

Read the profile above: half the CPU is in regcomp and its helpers - compiling a regular expression - called from match_rule on every request. Compile once, reuse: that is the fix.

Symbols and stacks. A profile is only as good as two things:

  1. Symbols to turn addresses into names. Packaged libraries are stripped down to the functions they export; internal ones show as hex addresses or [unknown]. Install the debug symbols (libc6-dbg for glibc, *-dbgsym packages from Ubuntu's ddebs archive), or let perf download them while it records: perf record --debuginfod=https://debuginfod.ubuntu.com ... (perf ignores DEBUGINFOD_URLS unless you ask, and sudo drops the variable anyway). The lab box has glibc's debug symbols installed, which is why regex internals like parse_expression have names in the report above.
  2. Complete stacks. -g walks frame pointers. Ubuntu builds its packages with frame pointers since 24.04, so system libraries give complete stacks; your own code needs -fno-omit-frame-pointer, or use --call-graph dwarf (bigger, slower).

JIT runtimes (Java, Node.js) generate code at run time; perf sees addresses without symbols unless the runtime writes /tmp/perf-PID.map (Java: jcmd PID Compiler.perfmap; Node.js: --perf-basic-prof; Python 3.12+: python3 -X perf).

perf script prints every sample with its stack - the raw input for flame graphs:

$ sudo perf script | head -4
         pricing   17385 [001]  8226.614100:   10101010 task-clock:pppH: 
	    ffffa1913150 parse_expression+0x10 (/usr/lib/aarch64-linux-gnu/libc.so.6)
	    ffffa1695f30 parse_branch+0x20 (/usr/lib/aarch64-linux-gnu/libc.so.6)
	    ffffa188cb38 parse_reg_exp+0x3e0 (/usr/lib/aarch64-linux-gnu/libc.so.6)

In an interview: "I profile with perf record -F 99 -g on the process for a few seconds, then perf report sorted by self time for the hot functions and by children for the expensive call paths; or I turn it into a flame graph."

Why it helps

Counters say the CPU is busy; a profile says which function keeps it busy, which is what you can actually change. perf record and perf report are the standard way to get that on Linux, for any language that has symbols and stacks. Knowing self versus children time, and why stacks come out broken, decides whether the profile leads you to the bug or to a wild goose chase.

Commands in this lesson

perf

FAQ

Why sample at 99 Hz and not 100?

So the sampling does not run in lockstep with something that happens exactly 100 times a second (timers, scheduler ticks), which would bias the profile towards whatever runs at that moment. 99 samples a second per CPU is also cheap enough for production.

What do -g and -p do in perf record?

-g records the call stack with every sample, not just the current function, so you can see who called it. -p PID profiles one running process instead of a new command; add -- sleep 10 to stop after ten seconds. -a profiles the whole box.

What is the difference between self and children?

Self is the time spent in the function's own code. Children adds the time of everything it calls. perf report --no-children sorts by self and shows the hot leaf functions; the default children view shows the expensive call paths from the top down.

Why do I see hex addresses or [unknown]?

Missing symbols (a stripped binary, or JIT code perf cannot see) or broken stacks (code built without frame pointers). Install debug symbols, use a runtime option that writes a perf map (Java, Node.js, Python 3.12+), or try --call-graph dwarf.

What does perf script do?

It prints every sample in perf.data as text: the command, PID, time, event and the stack, one frame per line. That text is the input for the FlameGraph scripts and other tools, so it is the bridge from a recording to a flame graph.

In an interview Mid

How do you find which function burns the CPU in a running process?

sudo perf record -F 99 -g -p PID -- sleep 10 samples its stacks for ten seconds, then perf report --stdio --no-children lists the functions by self time, the hot leaves, and the default children view shows the expensive call paths. If stacks are broken I fix symbols or frame pointers first. For a picture I turn it into a flame graph with perf script.

Also asked: What is the difference between self and children time? · Why do profiles sometimes show [unknown] frames? · Why sample at 99 Hz?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.