Linux Performance & eBPF: interview questions
The question you are most likely to get for each topic, a model answer, and what else comes up.
A service on a Linux box is slow. How do you find out why? Mid
With the USE method: for each resource I check utilization, saturation and errors with the cheap counters first - uptime, dmesg, vmstat 1, mpstat -P ALL 1, pidstat 1, iostat -xz 1, free -m and sar for the network. That names the saturated resource and roughly who. Then I profile (perf record, a flame graph) for CPU, or trace with the bcc tools or bpftrace for the off-CPU questions, keeping every tracer bounded, and write up the evidence.
Also asked: What does the load average measure on Linux? · How do you find CPU throttling from a cgroup limit? · What makes eBPF safe to run in production?
What is the USE method? Mid
For every resource - CPUs, memory, disks, network and software resources such as pools and cgroup limits - you check utilization (how busy it is), saturation (is work queuing for it) and errors. It turns "it is slow" into a checklist, starts with cheap counters like vmstat, mpstat and iostat, and catches saturation, which is where latency usually comes from.
Also asked: What is the difference between utilization and saturation? · Which software resources would you add to a USE checklist? · Why start with counters rather than tracing?
What do you run in the first minute on a slow Linux box? Mid
uptime for the load trend, dmesg -T | tail for kernel errors, vmstat 1 for the run queue, swapping and CPU modes, mpstat -P ALL 1 for one hot CPU, pidstat 1 for the busy processes, iostat -xz 1 for disk latency and queues, free -m for available memory, sar -n DEV 1 and sar -n TCP,ETCP 1 for traffic and retransmits, then top. Together they name the saturated resource and roughly who.
Also asked: Why do you ignore the first line of vmstat? · What does sar give you that the other commands do not? · What are you looking for in dmesg?
Learn it: 8.2 The 60-second checklist
The load average is 12 on a 4-CPU box but the CPUs are mostly idle. What is going on? Mid
Linux load counts running or runnable tasks plus tasks in uninterruptible sleep (D). With idle CPUs the extra load is almost always tasks in D waiting on I/O. I check vmstat for b and wa, ps for processes in state D and where they wait (wchan, /proc/PID/stack), and iostat -x for the disk's latency. Adding CPUs would not change anything.
Also asked: How do you show CPU saturation rather than utilization? · Why is iowait not CPU usage? · What does steal time mean on a virtual machine?
Learn it: 8.4 Load, run queues and CPU saturation
A service uses 50% CPU, the box is idle, but its p99 latency is bad. What do you check? Mid
CFS throttling from a CPU quota: CPUQuota= on a systemd unit sets cpu.max, a quota per 100 ms period. A multi-threaded service burns it early in the period and is frozen for the rest, so requests stall while average CPU looks fine. I read nr_throttled and throttled_usec in the cgroup's cpu.stat twice; if they grow, I raise the limit or switch to weights.
Also asked: What is the difference between a CPU weight and a CPU quota? · Where do you find a service's cgroup CPU counters? · Should a latency-sensitive service have a CPU quota at all?
How do you tell whether a disk is the bottleneck? Mid
I run iostat -xz 1 and read r_await and w_await, the latency including queueing, and aqu-sz, the queue length, rather than just %util, because SSDs and virtual disks serve requests in parallel. If latency is high I run sudo pidstat -d 1 to see which process does the I/O, and biolatency-bpfcc for the latency distribution. The fix is moving or capping the heavy job, for example with io.max.
Also asked: Why can %util be misleading on an SSD? · Why does pidstat -d show nothing for some processes? · How would you limit a batch job's disk bandwidth?
Latency is bad but the network link is nowhere near full. Could it still be the network? Mid
Yes: retransmits. I check retrseg/s with sar -n TCP,ETCP 1 or nstat and compare it with the segments sent; a few percent retransmitted adds a timeout to every affected request, so p99 rises while bandwidth is idle. sar -n EDEV shows drops and errors on the interface, and tcpretrans-bpfcc or ss -ti name the connection and the peer.
Also asked: How do you measure the retransmit rate on a box? · What does a receive drop on an interface usually mean? · How do you find which peer is losing packets?
Learn it: 8.11 Network: throughput, connections and retransmits
What does perf stat tell you about a program? Mid
perf stat counts events while a command or a process (-p PID) runs: task-clock and CPUs utilized say whether it is CPU-bound, context-switches and page-faults show how often it blocked or touched new memory, and seconds user / sys split the time. On bare metal I add cycles and instructions for IPC; in a VM without a PMU those show <not supported>. It needs sudo on Ubuntu because perf_event_paranoid is 4.
Also asked: Why does perf fail for normal users on Ubuntu? · What is a PMU and why might a VM not have one? · What does a low instructions-per-cycle value suggest?
Learn it: 8.13 perf: counting events
How do you find which function burns the CPU in a running process? Mid
sudo perf record -F 99 -g -p PID -- sleep 10 samples its stacks for ten seconds, then perf report --stdio --no-children lists the functions by self time, the hot leaves, and the default children view shows the expensive call paths. If stacks are broken I fix symbols or frame pointers first. For a picture I turn it into a flame graph with perf script.
Also asked: What is the difference between self and children time? · Why do profiles sometimes show [unknown] frames? · Why sample at 99 Hz?
How do you read a flame graph? Mid
Each box is a function and its callees sit above it. Width is the share of samples, height is stack depth, and the x-axis is sorted alphabetically, not time. I look for wide plateaus at the top of the towers - the hot functions - then read down the stack to the caller in our code that I can change. I make it with perf script, stackcollapse-perf.pl and flamegraph.pl.
Also asked: What does a flame graph's x-axis mean? · What is the folded stack format? · What is an off-CPU flame graph for?
Learn it: 8.17 Flame graphs
What makes eBPF safe to use in production? Mid
Every program passes the verifier before it loads: it must terminate, only touch memory through checked helpers and stay within size limits, so it cannot crash or hang the kernel. Programs are JIT-compiled to native code and aggregate results in maps inside the kernel, so the cost is a small amount per event. The remaining risk is overhead on very hot events, which you bound by counting first.
Also asked: What is the difference between a tracepoint and a kprobe? · What are BTF and CO-RE for? · How do you list the eBPF programs loaded on a box?
Which eBPF tools would you reach for when the counters are not enough? Mid
The bcc tools, installed on Ubuntu as bpfcc-tools with a -bpfcc suffix: execsnoop for short-lived processes, opensnoop for failing opens, biolatency and ext4slower for storage latency, tcpretrans for retransmits, runqlat for CPU queueing, and profile and offcputime for on-CPU and off-CPU time. Snoopers print events, summarizers aggregate in the kernel, so I prefer summarizers on hot events.
Also asked: What is the difference between a snooper and a summarizer? · When would you use execsnoop? · How would you find the file an application fails to open?
Write a bpftrace one-liner that counts files opened per process. Mid
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm, str(args.filename)] = count(); }' attaches to the openat tracepoint and counts each process and path in a map; str() copies the filename from user memory. Ctrl+C prints the map sorted by count. I find probe names with bpftrace -l and use hist() instead of count() when I want a distribution.
Also asked: How do you list the probes bpftrace can attach to? · What is the difference between hist and lhist? · How would you measure a latency with bpftrace?
Latency is high but the process uses almost no CPU. Where is the time? Mid
Off-CPU: the threads are blocked on I/O, locks, the network or the run queue. I run offcputime-bpfcc -p PID for a few seconds to see which stacks block and for how many microseconds, then confirm the specific cause with the matching tool - ext4slower or biolatency for storage, tcpretrans for the network, runqlat for CPU queueing. An on-CPU profile would show nothing useful here.
Also asked: What is the difference between on-CPU and off-CPU analysis? · How would you confirm a slow fsync? · Why look at p99 rather than the average latency?
Learn it: 8.26 Off-CPU time and tail latency
How do you see what a running process is doing without its source code or slowing it down? Mid
I count its system calls with eBPF: sudo syscount-bpfcc -p PID for the top calls, -x for the ones that fail and -L for where the time goes. A loop of failing openat calls with clock_nanosleep in between is a retry loop; futex or epoll_pwait at the top is a mostly waiting program. strace -c gives a similar table but stops the process on every call. PSI in /proc/pressure then tells me which resource tasks stall on.
Also asked: Why is strace risky on a production service? · What does the full line in a PSI file mean? · How would you find which file a failing openat is looking for?
How do you use tracing tools safely on a production box? Mid
I keep tracing bounded: a duration on every tool, a PID filter where I can, summaries aggregated in the kernel instead of per-event output, and I count an event's rate with funccount-bpfcc or a bpftrace count() before attaching to it. I say in the incident channel what I attach, check with bpftool prog list afterwards, and write up time, evidence, action and result.
Also asked: What is the difference between sampling and tracing overhead? · What belongs in a performance incident write-up? · How do on-box tools relate to fleet monitoring?
Practise these answers with flashcards and labs Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.