OnCallReadyStart learning

Linux Lesson 8.19 · Linux Performance & eBPF · 5 min read

eBPF: safe programs inside the kernel

In plain words

The kernel is the engine room of the computer, and normally you cannot put anything in there without rebuilding the engine. eBPF lets you hand the engine room a tiny, checked program, like a note with instructions: "every time this happens, add one to this counter". An inspector, the verifier, reads the note first and refuses it unless it is certain the note cannot break anything.

Run your code where the events happen

eBPF lets you load a small program into the running kernel and attach it to an event: a kernel function, a tracepoint, a user-space function, a network packet, a timer. The program runs each time the event fires, records what it needs in a map, and user space reads the map. Counting and summarizing happen in the kernel, so only results cross into user space.

Why it is safe in production - the reason it replaced loading kernel modules for tracing:

  1. The verifier checks every program before it runs: every path terminates (bounded loops), every memory access is checked, it only calls allowed helper functions, it cannot crash or hang the kernel. A program that fails verification is rejected, not run.
  2. The JIT compiles it to native code (Ubuntu forces the JIT on), so it is fast.
  3. Overhead is per event: a probe on something that fires a million times a second costs real CPU; on something that fires 100 times a second, nothing measurable.

Where programs attach:

HookWhatStable?
tracepointstatic points the kernel developers placed (sched:sched_switch, syscalls:sys_enter_openat)yes: a kernel ABI
kprobe / kretprobeentry/return of almost any kernel functionno: functions change between versions
fentry / fexitthe same, faster, with typed arguments from BTFno
uprobe / uretprobeany function in a user binary or librarydepends on the binary
USDTstatic probes compiled into applications (JVM, Python, PostgreSQL)yes
perf eventstimed sampling (profiling), hardware countersyes

Prefer tracepoints when one exists; kprobes reach everything else.

BTF and CO-RE. Kernel structures differ between versions. BTF is compact type information the kernel publishes about itself (/sys/kernel/btf/vmlinux, present on Ubuntu). With it, tools compiled once can relocate their field accesses at load time - CO-RE, "compile once, run everywhere". bpftrace and the libbpf-based tools use it; the older bcc Python tools compile C at start-up instead (slower to start, need kernel headers).

Privileges. Ubuntu sets kernel.unprivileged_bpf_disabled = 2: only root (or CAP_BPF + CAP_PERFMON) loads programs. Every tool in the next lessons runs with sudo.

bpftool (package bpftool) lists what is loaded - systemd already uses eBPF for device access control and IP filtering of its units:

$ sudo bpftool prog list
2: cgroup_device  name sd_devices  tag 05287f4466946aab  gpl
	loaded_at 2026-10-22T17:43:01+0000  uid 0
	xlated 560B  jited 400B  memlock 4096B
3: cgroup_skb  name sd_fw_egress  tag 6deef7357e7b4530  gpl
	loaded_at 2026-10-22T17:43:01+0000  uid 0
	xlated 64B  jited 80B  memlock 4096B

xlated is the verified bytecode size, jited the native code. bpftool map list shows the maps; bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h writes the header CO-RE tools compile against.

What the verifier refuses. Unbounded loops, reading memory without a checked helper, dereferencing a pointer that might be NULL, a stack over 512 bytes, too many instructions. The tools you use have passed it already; when you write your own (bpftrace or C), its message says which instruction it could not prove safe. That refusal is the safety: it happens before the program runs, not as a crash at 3 a.m.

Overhead in numbers. An eBPF program on a tracepoint costs on the order of a hundred nanoseconds per event. 100 events a second: nothing. A million a second (every system call on a busy box): a noticeable share of a CPU. Count first (funccount-bpfcc, or bpftrace count() for a second), then decide.

Three front ends, one engine. bcc (*-bpfcc tools): ready-made tools in Python. bpftrace: a one-liner language. libbpf + CO-RE: compiled tools and products (Cilium, Falco, Pixie, the Observability chapters' agents).

In an interview: "eBPF runs verified, JIT-compiled programs on kernel or user events; the verifier guarantees they terminate and only touch memory they may, so it is safe in production. Results are aggregated in maps in the kernel, which keeps the overhead low."

Why it helps

eBPF is how modern Linux observability, networking and security tools work: the bcc tools, bpftrace, Cilium, Falco and continuous profilers are all built on it. Understanding what makes it safe (the verifier), cheap (maps, aggregation in the kernel) and portable (BTF and CO-RE) lets you use it on production boxes with confidence and explain why it is not the same risk as loading a kernel module.

On the blog

Commands in this lesson

bpftool

FAQ

What does the verifier check?

Before a program is loaded it proves that the program always terminates, never reads memory it should not (only through checked helpers, pointers checked for NULL), keeps its stack under 512 bytes and stays under the instruction limit. A program it cannot prove safe is refused before it ever runs.

What is a map?

A key-value store shared between the eBPF program in the kernel and the tool in user space. Programs count, sum or build histograms in maps, and the tool reads the result once. That in-kernel aggregation is why eBPF tracing is so much cheaper than copying every event to user space.

Where can programs attach?

Tracepoints (stable kernel hooks), kprobes and kretprobes (almost any kernel function), fentry and fexit (the same with BTF types), uprobes (functions in user programs and libraries), USDT probes built into applications, and perf events such as a timer for sampling. Prefer tracepoints when one exists.

What are BTF and CO-RE?

BTF is compact type information about the running kernel, in /sys/kernel/btf/vmlinux on Ubuntu. CO-RE (compile once, run everywhere) uses it to adjust a compiled program's structure accesses when it is loaded, so one binary tool works across kernel versions without kernel headers.

Can a normal user load eBPF programs?

Not on Ubuntu: kernel.unprivileged_bpf_disabled is 2, so loading programs needs root (CAP_BPF plus CAP_PERFMON or CAP_SYS_ADMIN). bpftool, bpftrace and the bcc tools all run with sudo. sudo bpftool prog list shows every program currently loaded.

In an interview Mid

What makes eBPF safe to use in production?

Every program passes the verifier before it loads: it must terminate, only touch memory through checked helpers and stay within size limits, so it cannot crash or hang the kernel. Programs are JIT-compiled to native code and aggregate results in maps inside the kernel, so the cost is a small amount per event. The remaining risk is overhead on very hot events, which you bound by counting first.

Also asked: What is the difference between a tracepoint and a kprobe? · What are BTF and CO-RE for? · How do you list the eBPF programs loaded on a box?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.