Run your code where the events happen
eBPF lets you load a small program into the running kernel and attach it to an event: a kernel function, a tracepoint, a user-space function, a network packet, a timer. The program runs each time the event fires, records what it needs in a map, and user space reads the map. Counting and summarizing happen in the kernel, so only results cross into user space.
Why it is safe in production - the reason it replaced loading kernel modules for tracing:
- The verifier checks every program before it runs: every path terminates (bounded loops), every memory access is checked, it only calls allowed helper functions, it cannot crash or hang the kernel. A program that fails verification is rejected, not run.
- The JIT compiles it to native code (Ubuntu forces the JIT on), so it is fast.
- Overhead is per event: a probe on something that fires a million times a second costs real CPU; on something that fires 100 times a second, nothing measurable.
Where programs attach:
| Hook | What | Stable? |
|---|---|---|
| tracepoint | static points the kernel developers placed (sched:sched_switch, syscalls:sys_enter_openat) | yes: a kernel ABI |
| kprobe / kretprobe | entry/return of almost any kernel function | no: functions change between versions |
| fentry / fexit | the same, faster, with typed arguments from BTF | no |
| uprobe / uretprobe | any function in a user binary or library | depends on the binary |
| USDT | static probes compiled into applications (JVM, Python, PostgreSQL) | yes |
| perf events | timed sampling (profiling), hardware counters | yes |
Prefer tracepoints when one exists; kprobes reach everything else.
BTF and CO-RE. Kernel structures differ between versions. BTF is compact type information the kernel publishes about itself (/sys/kernel/btf/vmlinux, present on Ubuntu). With it, tools compiled once can relocate their field accesses at load time - CO-RE, "compile once, run everywhere". bpftrace and the libbpf-based tools use it; the older bcc Python tools compile C at start-up instead (slower to start, need kernel headers).
Privileges. Ubuntu sets kernel.unprivileged_bpf_disabled = 2: only root (or CAP_BPF + CAP_PERFMON) loads programs. Every tool in the next lessons runs with sudo.
bpftool (package bpftool) lists what is loaded - systemd already uses eBPF for device access control and IP filtering of its units:
$ sudo bpftool prog list
2: cgroup_device name sd_devices tag 05287f4466946aab gpl
loaded_at 2026-10-22T17:43:01+0000 uid 0
xlated 560B jited 400B memlock 4096B
3: cgroup_skb name sd_fw_egress tag 6deef7357e7b4530 gpl
loaded_at 2026-10-22T17:43:01+0000 uid 0
xlated 64B jited 80B memlock 4096B
xlated is the verified bytecode size, jited the native code. bpftool map list shows the maps; bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h writes the header CO-RE tools compile against.
What the verifier refuses. Unbounded loops, reading memory without a checked helper, dereferencing a pointer that might be NULL, a stack over 512 bytes, too many instructions. The tools you use have passed it already; when you write your own (bpftrace or C), its message says which instruction it could not prove safe. That refusal is the safety: it happens before the program runs, not as a crash at 3 a.m.
Overhead in numbers. An eBPF program on a tracepoint costs on the order of a hundred nanoseconds per event. 100 events a second: nothing. A million a second (every system call on a busy box): a noticeable share of a CPU. Count first (funccount-bpfcc, or bpftrace count() for a second), then decide.
Three front ends, one engine. bcc (*-bpfcc tools): ready-made tools in Python. bpftrace: a one-liner language. libbpf + CO-RE: compiled tools and products (Cilium, Falco, Pixie, the Observability chapters' agents).
In an interview: "eBPF runs verified, JIT-compiled programs on kernel or user events; the verifier guarantees they terminate and only touch memory they may, so it is safe in production. Results are aggregated in maps in the kernel, which keeps the overhead low."