Linux Performance & eBPF: glossary
96 terms, in plain words, with where each one is taught.
- %util (util)
- iostat's share of time the device had at least one request in flight. 100% means saturated only for devices that serve one request at a time; SSDs and virtual disks can do more. Lesson 8.9
- %wait (pidstat) (%wait)
- pidstat's share of time a task was runnable but waiting for a CPU: CPU saturation seen per process. Lesson 8.4
- 60-second checklist (Linux performance analysis in 60 seconds, first minute)
- Ten commands run in order on a slow box: uptime, dmesg -T | tail, vmstat 1, mpstat -P ALL 1, pidstat 1, iostat -xz 1, free -m, sar -n DEV 1, sar -n TCP,ETCP 1, top. Together they name the saturated resource. Lesson 8.2
- ABI (application binary interface, stable ABI)
- An interface at the binary level that programs rely on without being rebuilt. Kernel tracepoints are treated as one, so tools that use them keep working across kernel versions; kprobe targets are not. Lesson 8.19
- anti-method (streetlight anti-method, random change anti-method, blame-someone-else)
- A way of working on a slow system that is not driven by evidence: looking only where your favourite tool shines, changing things at random until the problem goes away, or blaming another team. Lesson 8.1
- aqu-sz (avgqu-sz, queue size)
- iostat's average number of requests queued or in flight at the device. Growing aqu-sz with growing await means the disk is saturated. Lesson 8.9
- await (r_await, w_await)
- iostat's average time in milliseconds per read (r_await) or write (w_await), from queuing to completion: what the application feels. Lesson 8.9
- bcc tools (bpfcc-tools, bcc, -bpfcc)
- A collection of ready-made eBPF tools in Python (execsnoop, biolatency, tcpretrans...). On Ubuntu each is installed as NAME-bpfcc in /usr/sbin. Lesson 8.21
- BFQ (mq-deadline, I/O scheduler)
- BFQ (Budget Fair Queueing) is a Linux I/O scheduler that shares disk time between processes and honours ionice priorities. virtio disks on Ubuntu use none, which ignores them. Lesson 8.9
- biolatency (biolatency-bpfcc, biosnoop, biosnoop-bpfcc)
- biolatency draws a histogram of block I/O latency (-D per disk); biosnoop prints each disk request with process, sector, size and latency. Lesson 8.21
- BPF map (eBPF map)
- A key-value store shared between an eBPF program and user space. Tools count and build histograms in maps inside the kernel and read the result once. Lesson 8.19
- bpftool (bpftool prog list, bpftool map, bpftool feature probe)
- The kernel's own tool to list and inspect loaded eBPF programs and maps and to probe what the kernel supports. Needs root; a separate package on Ubuntu. Lesson 8.19
- bpftrace (bpftrace -e, bpftrace -l)
- A high-level language for eBPF one-liners and short scripts: probe /filter/ { action }. Ctrl+C prints its maps; -l lists probes. Lesson 8.24
- bpftrace map (@, @[comm], count())
- A variable starting with @ that lives in the kernel: @[comm] = count() counts per process, hist() and lhist() build histograms. Printed at exit or Ctrl+C. Lesson 8.24
- BTF (BPF Type Format, /sys/kernel/btf/vmlinux)
- Compact type information about the running kernel, in /sys/kernel/btf/vmlinux on Ubuntu. eBPF tools use it to read kernel structures without kernel headers. Lesson 8.19
- cachestat (cachestat-bpfcc)
- Page cache hits, misses and hit ratio per interval: shows whether reads are served from memory or go to disk. Lesson 8.21
- children time (Children, inclusive time)
- Samples in a function plus everything it calls. perf report's default view: the expensive call paths. Lesson 8.15
- CO-RE (compile once, run everywhere)
- Compile once, run everywhere: a compiled eBPF program adjusts its structure offsets with BTF when it loads, so one binary works across kernel versions. Lesson 8.19
- continuous profiling (Parca, Pyroscope)
- Sampling profiles taken all the time on every host and stored, so flame graphs exist for any past moment instead of only after someone logs in. Lesson 8.34
- CPU affinity (taskset, pinning)
- The set of CPUs a task may run on. taskset -p shows or changes it; pinning a busy process to one CPU can make that CPU the bottleneck. Lesson 8.4
- CPU throttling (throttled, cpu throttling)
- A cgroup used its CPU quota for the current period and is stopped until the next one, even if CPUs are idle. It shows up as tail latency, not as high CPU. Lesson 8.7
- cpu.max (CFS quota, CFS bandwidth, quota and period)
- A cgroup v2 file with a CPU quota and a period in microseconds, for example 50000 100000 = half a CPU. systemd writes it from CPUQuota=; max means no limit. Lesson 8.7
- cpu.stat (nr_throttled, throttled_usec, nr_periods)
- A cgroup's CPU counters: usage_usec, user_usec, system_usec and, with a quota, nr_periods, nr_throttled and throttled_usec. Read it twice; growing throttle counters prove throttling. Lesson 8.7
- CPUWeight (CPUWeight=)
- A systemd setting for cpu.weight (1-10000, default 100): a unit's share of CPU when CPUs are contended. Unlike a quota it never holds back a unit on an idle box. Lesson 8.7
- differential flame graph (difffolded.pl)
- A flame graph that compares two profiles of the same program and colours the frames that grew or shrank. Lesson 8.17
- eBPF (BPF, extended BPF)
- Small programs loaded into the Linux kernel that run on events (system calls, kernel functions, timers, packets). Checked by the verifier, compiled to native code, results kept in maps. Lesson 8.19
- EEVDF (EEVDF scheduler)
- Earliest Eligible Virtual Deadline First: the Linux CPU scheduler since kernel 6.6, the successor of CFS. It shares CPUs by weight (nice values, cpu.weight); quotas still come from cgroup bandwidth control. Lesson 8.4
- ERR (opensnoop) (ERR)
- The opensnoop column with the error number of a failed open: 2 = ENOENT (no such file), 13 = EACCES (permission denied). Successful opens show the file descriptor in FD instead. Lesson 8.21
- ETCP (EDEV)
- sar -n keywords: EDEV = network interface errors and drops, ETCP = TCP errors (retrseg/s, segments retransmitted per second; isegerr/s, orsts/s). DEV and TCP are the normal traffic counters. Lesson 8.2
- execsnoop (execsnoop-bpfcc)
- Prints every new process (exec) with its PID, PPID, return code and arguments. Finds short-lived processes that top never shows. Lesson 8.21
- ext4slower (ext4slower-bpfcc, ext4dist)
- Prints ext4 reads, writes, opens and fsyncs slower than a threshold (default 10 ms) with process, size and file name. ext4dist draws the latency histograms. Lesson 8.26
- fentry (fexit)
- A faster kind of kernel function hook that uses BTF, so arguments come with their types. Lesson 8.19
- flame graph (flamegraph, flamegraph.pl, FlameGraph)
- A picture of sampled stacks: each box is a function with its callees above it, width = share of samples, x sorted alphabetically (not time). Wide plateaus at the top are the hot functions. Lesson 8.17
- folded stacks (stackcollapse-perf.pl, collapsed stacks, .folded)
- One line per unique stack: frames joined by semicolons, a space, the sample count. stackcollapse-perf.pl makes it from perf script output; flamegraph.pl draws from it. Lesson 8.17
- frame pointer (frame pointers, --call-graph dwarf, -fno-omit-frame-pointer)
- A register each function uses to point at its caller's frame, which lets a profiler walk the stack cheaply. Code built without it gives broken stacks; Ubuntu builds its packages with it. Lesson 8.15
- fsync (fdatasync)
- A system call that waits until a file's data is on the disk. Databases and ledgers call it per transaction, so a slow disk turns directly into slow requests. Lesson 8.26
- funccount (funccount-bpfcc)
- Counts calls to kernel functions or tracepoints matching a pattern ('vfs_*'): the event rate to check before attaching a heavier tracer. Lesson 8.31
- GIL (global interpreter lock)
- The lock in CPython that lets only one thread run Python code at a time, so a multi-threaded Python program uses at most about one CPU for Python code (3.13+ has an optional build without it). Lesson 8.1
- interface drops (rxdrop/s, txdrop/s, sar -n EDEV)
- Packets an interface or the kernel threw away, usually because a buffer was full. sar -n EDEV shows them with the error counters per interface. Lesson 8.11
- involuntary context switch (nvcswch/s)
- The scheduler takes the CPU away from a task because another must run. Many of them on a task is a sign of CPU saturation. Lesson 8.4
- io.max (IOReadBandwidthMax=, IOWriteBandwidthMax=, IOReadIOPSMax=)
- A cgroup v2 file that caps a cgroup's bytes or operations per second on a device. systemd writes it from IOReadBandwidthMax= / IOWriteBandwidthMax= in a unit. Lesson 8.9
- iodelay (delay accounting, task_delayacct, kernel.task_delayacct)
- pidstat -d's time a process waited for block I/O, from the kernel's delay accounting. Off by default on Ubuntu: the column stays 0 until sysctl kernel.task_delayacct=1. Lesson 8.9
- ionice (ionice -c3, idle I/O class)
- Sets a process's I/O scheduling class and priority (realtime, best-effort 0-7, idle). Only I/O schedulers that honour priorities, such as BFQ, act on it. Lesson 8.9
- iostat (iostat -x, iostat -xz)
- Per-device disk statistics per interval. -x adds the extended columns (r_await, w_await, aqu-sz, %util), -z hides idle devices. The first report is since boot. Lesson 8.2
- iowait (%iowait, wa)
- CPU idle time during which at least one task the CPU ran waits for I/O. It is idle, not work: a busy CPU makes it disappear without the disk getting faster. Lesson 8.4
- IPC (instructions per cycle, insn per cycle)
- Instructions completed per CPU cycle, from the PMU. Low IPC (under 1) usually means the CPU waits on memory; high IPC means it computes. Lesson 8.13
- known unknowns
- The resources you have not checked yet. Saying them out loud ("CPU and disks are fine, network not checked") keeps an investigation honest. Lesson 8.1
- kprobe (kprobes, kretprobe)
- A dynamic hook on almost any kernel function (kretprobe: its return). Powerful but tied to the kernel version, since functions get renamed or inlined. Lesson 8.19
- load average (loadavg)
- On Linux: the average number of running, runnable and uninterruptible (D) tasks over 1, 5 and 15 minutes. High load with idle CPUs points at tasks waiting on I/O. Lesson 8.4
- log2 histogram (power-of-two histogram, hist())
- A histogram whose buckets double in size (0-1, 2-3, 4-7, 8-15...): the form bcc and bpftrace use for latencies and sizes, so a wide range fits in a few lines. Lesson 8.21
- LTS (long-term support)
- A long-term support release: Ubuntu LTS versions (24.04, 26.04) get five years of standard security updates, against nine months for the releases in between. Lesson 8.34
- mpstat (mpstat -P ALL)
- Prints CPU time split into modes (%usr, %sys, %iowait, %steal, %idle...) per CPU and interval. -P ALL shows each CPU, which reveals one CPU at 100% while the others idle. Lesson 8.2
- nice value (renice, nice -n, NI)
- A process's CPU priority from -20 (most favoured) to 19 (least). It changes its share only when CPUs are contended; renice changes it for a running process (lowering needs root). Lesson 8.4
- nsecs (@start[tid], retval, tid)
- bpftrace built-ins: nsecs is a timestamp in nanoseconds, tid the thread id, retval a function's return value. Storing nsecs at entry and subtracting at exit times any pair of events. Lesson 8.24
- nstat (TcpRetransSegs, SNMP counters)
- Prints the kernel's network counters (TcpRetransSegs, TcpExtListenDrops...) as deltas since its last run. Lesson 8.11
- off-CPU flame graph (offcpu flame graph)
- A flame graph where width is time spent blocked rather than running, made from offcputime -f output. Lesson 8.17
- off-CPU time (off-CPU, offcpu)
- Time a thread is not running: blocked on I/O, a lock, the network, a sleep, or waiting in the run queue. On-CPU profiles cannot see it. Lesson 8.26
- offcputime (offcputime-bpfcc)
- Sums the time threads spend blocked per stack, in microseconds; -p PID for one process, -f folded output for an off-CPU flame graph. Lesson 8.26
- opensnoop (opensnoop-bpfcc, opensnoop -x)
- Prints every file open with the process, file descriptor or error and the path; -x only failed opens (ERR 2 = no such file). Lesson 8.21
- p99 (99th percentile, tail)
- The latency 99% of requests stay under; the slowest 1% are the tail. Off-CPU waits like fsync and retransmits usually live there. Lesson 8.26
- perf (linux-perf, perf_events, perf stat)
- The Linux profiler that lives in the kernel source. perf stat counts events, perf record samples stacks, perf report and perf script read them. On Ubuntu 26.04 it comes from the linux-perf package. Lesson 8.13
- perf script (perf script output)
- Prints every sample in perf.data as text with its stack, one frame per line: the input for flame graph scripts. Lesson 8.15
- perf_event_paranoid (kernel.perf_event_paranoid)
- The sysctl that decides what unprivileged users may measure with perf. Ubuntu sets 4: nothing without root. Lesson 8.13
- perf.data (perf record, perf.data.old)
- The file perf record writes: the samples with their stacks. perf report and perf script read it; a new recording renames the old file to perf.data.old. Lesson 8.15
- pidstat (pidstat -d, pidstat -w, pidstat -t)
- Per-process statistics per interval: CPU (%usr, %system, %wait), -d disk kB read and written, -w context switches, -t per thread. Without root it hides other users' processes for -d. Lesson 8.2
- PMU (performance monitoring unit, hardware counters, <not supported>)
- The processor's hardware event counters (cycles, instructions, cache misses). Many VMs have none, so perf shows those events as <not supported> and uses software events. Lesson 8.13
- pressure stall information (PSI, /proc/pressure, some avg10)
- Per-resource stall time in /proc/pressure/{cpu,io,memory} and in every cgroup: some = at least one task stalled, full = all non-idle tasks stalled at once; averages over 10, 60 and 300 s. Lesson 8.31
- probe (bpftrace) (probe /filter/ { action }, interval:s:1)
- Where a bpftrace program attaches: tracepoint:..., kprobe:..., uprobe:..., profile:hz:99, interval:s:1, begin/end. An optional /filter/ decides which events run the action. Lesson 8.24
- ptrace overhead (strace -c, ptrace)
- strace stops the traced process at every system call (ptrace), which can slow a busy program down ten times or more. eBPF counting avoids it. Lesson 8.31
- RADDR (RPORT, LADDR, LPORT)
- Columns in the bcc TCP tools: LADDR/LPORT = the local address and port, RADDR/RPORT = the remote one, the peer. Lesson 8.11
- resource (USE) (software resource)
- Anything work can queue for: a CPU, memory, a disk, a network link, or a software limit such as a thread pool, a connection pool, a lock, a cgroup quota or the file descriptor limit. Lesson 8.1
- retransmit (retrans/s, retrseg/s, TCP retransmission)
- A TCP segment sent again because it was not acknowledged in time. Each one delays that request by a timeout (200 ms or more), so a few percent raises tail latency. Lesson 8.11
- run queue (runqueue, vmstat r)
- The tasks ready to run but waiting for a CPU. vmstat's r column above the number of CPUs means CPU saturation. Lesson 8.4
- run queue latency (scheduler latency, runqlat)
- How long a runnable task waits before it gets a CPU. A few microseconds on a healthy box; milliseconds when the CPUs are saturated. runqlat-bpfcc shows the distribution. Lesson 8.4
- sampling profiler (profiler, -F 99, sampling frequency)
- A tool that records where a program is at a fixed rate, for example 99 times a second, and counts the stacks. Its cost does not grow with what the program does. Lesson 8.15
- sar (sar -n DEV, sar -n TCP,ETCP, sar -n EDEV)
- The sysstat reporter for almost every counter: CPU (-u), disks (-d), interfaces (-n DEV, -n EDEV), TCP (-n TCP,ETCP). Live with an interval, or from the history files for the past (-s, -e). Lesson 8.2
- self time (--no-children, Self)
- Samples in a function's own code, not in what it calls. perf report --no-children sorts by it: the hot leaf functions. Lesson 8.15
- since boot (first line)
- The first report of vmstat, iostat and similar tools averages everything since the box started, not the current state. Read the lines after it. Lesson 8.2
- snooper (snoop tool)
- A tool that prints one line per event: execsnoop per new process, opensnoop per open, biosnoop per disk request. Cheap on rare events, expensive on hot ones. Lesson 8.21
- software event (task-clock, cpu-clock, page-faults)
- An event the kernel counts itself, without the PMU: task-clock (CPU time used), context-switches, page-faults, cpu-migrations, and the task-clock / cpu-clock timers perf samples with when there is no PMU. Lesson 8.13
- steal (%steal, st)
- Time a virtual CPU wanted to run but the hypervisor ran another guest instead. High steal is a noisy neighbour or an oversubscribed host; nothing inside the VM fixes it. Lesson 8.4
- str() (str(args.filename))
- A bpftrace function that copies a string from the traced process's memory; without it a filename argument is just an address. Lesson 8.24
- summarizer (summary tool)
- A tool that aggregates events in the kernel and prints a table or histogram at the end (biolatency, runqlat, cachestat): cheap even on busy events. Lesson 8.21
- symbol (symbols, debug symbols, [unknown])
- The name of a function at an address. Without symbols a profile shows hex addresses or [unknown]; JIT runtimes need a perf map file to name their code. Lesson 8.15
- syscount (syscount-bpfcc, syscount -x, syscount -L)
- Counts system calls box-wide, per process (-P) or for one PID (-p); -x only failed calls, -L the time spent in each. Lesson 8.31
- sysstat (sysstat-collect.timer, /var/log/sysstat, sadc)
- The package with mpstat, pidstat, iostat and sar. Installed on Ubuntu Server by default; its timer records the box's counters every ten minutes so sar can show the past. Lesson 8.2
- tcpretrans (tcpretrans-bpfcc)
- Prints each TCP retransmit with the local and remote address and port and the connection state, which names the lossy peer. Lesson 8.21
- tracepoint (tracepoints, sys_enter_openat, static tracepoint)
- A stable hook built into the kernel at a known event (a system call, a scheduler switch, a block I/O). Preferred over kprobes because it survives kernel upgrades. Lesson 8.19
- tracing overhead (bounded tracing, sampling vs tracing)
- Sampling costs a fixed amount; tracing costs per event, so it grows with the event rate. Count first, filter by PID, give every tool a duration. Lesson 8.34
- uninterruptible sleep (D state, state D)
- A task waiting for something it cannot be interrupted from, usually disk or NFS I/O. Signals wait until it wakes; Linux counts these tasks in the load average. Lesson 8.4
- unprivileged_bpf_disabled (kernel.unprivileged_bpf_disabled)
- The sysctl that decides whether normal users may load eBPF programs. Ubuntu sets 2: only root, and it cannot be lowered without a reboot. Lesson 8.19
- uprobe (uprobes, uretprobe)
- A dynamic hook on a function in a user program or library, such as a function in libc or a database binary. Lesson 8.19
- USDT (USDT probe, user statically defined tracing)
- A static probe point compiled into an application (some databases and language runtimes ship them), the user-space version of a tracepoint. Lesson 8.19
- USE method (utilization, saturation, errors)
- For every resource (CPUs, memory, disks, network, and software resources such as pools and cgroup limits) check how busy it is, whether work queues for it, and whether it reports errors. Saturation is where latency usually comes from. Lesson 8.1
- verifier (BPF verifier)
- The kernel component that checks an eBPF program before it loads: it must terminate, only access allowed memory through helpers and stay within limits. It refuses anything it cannot prove safe. Lesson 8.19
- voluntary context switch (cswch/s)
- A task gives up the CPU because it blocks (I/O, a lock, a sleep). Normal for any program that waits. Lesson 8.4