Where does the CPU time go?
A profiler interrupts each CPU many times a second and writes down what was running: the process and its stack (which function called which). Functions that use the CPU show up in many samples. It needs no change to the program, and at 99 samples a second the overhead is tiny.
$ sudo perf record -F 99 -g -p $(pgrep pricing) -- sleep 10
[ perf record: Woken up 5 times to write data ]
[ perf record: Captured and wrote 0.546 MB perf.data (1742 samples) ]
-F 99: 99 samples per second per CPU. Not 100, so sampling does not run in lockstep with something that happens every 10 ms.-g: record the call stack of each sample.-p PIDone process (or-aall CPUs);-- sleep 10only sets how long to record.- It writes
./perf.data(mode 0600, owned by root): read it withsudotoo.
perf report --stdio --no-children lists functions by self time - samples where that function was the one running:
# Samples: 1K of event 'task-clock:pppH'
# Event count (approx.): 17595959596
#
# Overhead Command Shared Object Symbol
# ........ ....... ................. ............................
#
21.01% pricing libc.so.6 [.] parse_expression
|
---parse_expression
parse_branch
parse_reg_exp
re_compile_internal
regcomp
match_rule
apply_promotions
...
12.00% pricing libc.so.6 [.] check_matching
[.] is user code, [k] kernel. The tree under each line shows who called it, leaf first. Without --no-children (the default when stacks were recorded) perf adds a Children column: time in the function and everything it called. main has ~100% children, ~0% self. Sort by self to find the hot leaf, by children to find the expensive path. On a terminal, perf report without --stdio opens an interactive browser (arrows, Enter to expand).
Read the profile above: half the CPU is in regcomp and its helpers - compiling a regular expression - called from match_rule on every request. Compile once, reuse: that is the fix.
Symbols and stacks. A profile is only as good as two things:
- Symbols to turn addresses into names. Packaged libraries are stripped down to the functions they export; internal ones show as hex addresses or
[unknown]. Install the debug symbols (libc6-dbgfor glibc,*-dbgsympackages from Ubuntu's ddebs archive), or let perf download them while it records:perf record --debuginfod=https://debuginfod.ubuntu.com ...(perf ignoresDEBUGINFOD_URLSunless you ask, andsudodrops the variable anyway). The lab box has glibc's debug symbols installed, which is why regex internals likeparse_expressionhave names in the report above. - Complete stacks.
-gwalks frame pointers. Ubuntu builds its packages with frame pointers since 24.04, so system libraries give complete stacks; your own code needs-fno-omit-frame-pointer, or use--call-graph dwarf(bigger, slower).
JIT runtimes (Java, Node.js) generate code at run time; perf sees addresses without symbols unless the runtime writes /tmp/perf-PID.map (Java: jcmd PID Compiler.perfmap; Node.js: --perf-basic-prof; Python 3.12+: python3 -X perf).
perf script prints every sample with its stack - the raw input for flame graphs:
$ sudo perf script | head -4
pricing 17385 [001] 8226.614100: 10101010 task-clock:pppH:
ffffa1913150 parse_expression+0x10 (/usr/lib/aarch64-linux-gnu/libc.so.6)
ffffa1695f30 parse_branch+0x20 (/usr/lib/aarch64-linux-gnu/libc.so.6)
ffffa188cb38 parse_reg_exp+0x3e0 (/usr/lib/aarch64-linux-gnu/libc.so.6)
In an interview: "I profile with perf record -F 99 -g on the process for a few seconds, then perf report sorted by self time for the hot functions and by children for the expensive call paths; or I turn it into a flame graph."