A flame graph is a picture of thousands of stack samples stacked on top of each other. Each box is a function, and the functions it called sit on top of it. The wider a box, the more samples it appeared in, so the more time it took. You look for the widest boxes near the top of the flames: that is where the CPU spends its time.
A whole profile in one picture
A long perf report is hard to read. A flame graph (Brendan Gregg) draws all the stacks at once:
each box is a function; the box above it is a function it called;
the width is the number of samples that contained it - its share of the CPU time;
the x-axis is not time: boxes are sorted alphabetically to merge identical stacks;
colours are random warm tones, only to tell neighbours apart.
Look for wide plateaus at the top: a function that is on-CPU itself (its own width, nothing above it). Then read down to see who calls it.
Making one. The scripts are not packaged; clone them:
out.folded is the interesting middle step: one line per unique stack, frames joined by ; from the root, then the number of samples. It is plain text - you can grep it, sort it, diff it:
Open flame.svg in a browser (copy it to your laptop with scp): hover for names and percentages, click to zoom into a tower.
Patterns to recognise:
one wide plateau at the top: a single hot function (the regex compile in the labs);
a wide tower ending in many thin flames: an expensive path whose cost is spread out - look at the function at the base of the spread;
a large [unknown] block: missing symbols or frame pointers (JIT code, stripped binaries);
kernel towers (el0_svc, __arm64_sys_* on arm64, entry_SYSCALL_64 on x86) under user code: time in system calls - check which ones;
swapper in a system-wide (-a) graph: idle CPU time, not work.
Compare before and after. Keep the folded file of the slow version; after the fix, fold a new profile and compare the widths of the same frames. difffolded.pl old.folded new.folded | ./FlameGraph/flamegraph.pl > diff.svg colours what grew and what shrank.
Other inputs, same picture.profile-bpfcc -f and offcputime-bpfcc -f print folded stacks directly. An off-CPU flame graph (from offcputime) shows where threads wait - locks, disk, network - instead of where they compute. Two folded files can be compared with difffolded.pl.
In an interview: "Width is the share of samples, height is stack depth, x is sorted, not time. I look for the widest frames at the top of the towers and then read down the stack to find the caller I can change."
Why it helps
A perf report with thousands of lines is hard to read; a flame graph shows the same profile in one picture where the expensive path is obvious at a glance. It is the most common way profiles are shared in incident reviews and pull requests, continuous profilers draw them all day, and "how do you read a flame graph?" is a standard interview question.
No. Frames are sorted alphabetically at each level so that identical stacks merge into one wide box. Only the width matters: the share of samples. A flame chart, where x is time, is a different picture that some tools also draw.
How do I make one from perf?
perf script turns perf.data into text, stackcollapse-perf.pl folds it into one line per unique stack with a count, and flamegraph.pl draws an SVG from the folded file. The scripts come from Brendan Gregg's FlameGraph repository, cloned with git.
What are folded stacks?
One line per unique stack: the frames joined by semicolons from the root to the leaf, a space, and the number of samples. It is a small text format that grep and diff work on, and every flame graph tool reads it, so it is worth keeping next to the SVG.
What does a large [unknown] block mean?
Frames whose symbols perf could not resolve: JIT code without a perf map, a stripped binary, or code without frame pointers. The width is real time, but you cannot see where it goes until you fix the symbols, so fix them before you draw conclusions.
What is an off-CPU flame graph?
The same picture made from time spent blocked instead of time on the CPU, for example from offcputime-bpfcc -f. Width is then time waiting: for a disk, a lock, the network. It answers the question an on-CPU flame graph cannot: where a slow but idle program spends its time.
In an interview Mid
How do you read a flame graph?
Each box is a function and its callees sit above it. Width is the share of samples, height is stack depth, and the x-axis is sorted alphabetically, not time. I look for wide plateaus at the top of the towers - the hot functions - then read down the stack to the caller in our code that I can change. I make it with perf script, stackcollapse-perf.pl and flamegraph.pl.
Also asked: What does a flame graph's x-axis mean? · What is the folded stack format? · What is an off-CPU flame graph for?