A worker can be slow for two reasons: they are busy working, or they are standing around waiting - for a delivery, for a meeting room, for an answer. A normal profiler only watches the worker while they work. Off-CPU analysis watches the waiting too: how long they stood still, and what they were waiting for. For many slow requests the waiting is where all the time goes.
Slow, but not busy
A request that takes 200 ms but uses 5 ms of CPU spent 195 ms off-CPU: waiting for a disk, a lock, the network, a timer, or a CPU (run queue). CPU profiles cannot see it - they only sample what is running. Off-CPU analysis records where threads stop running and for how long.
offcputime-bpfcc traces the scheduler: every time a thread leaves the CPU it notes the stack, and when it comes back, the time it was away. The output is stacks with microseconds blocked:
$sudo offcputime-bpfcc -p $(pgrep ledger) 5
Tracing off-CPU time (us) of PIDs 17385 by user + kernel stack for 5 secs.
finish_task_switch.isra.0
__schedule
schedule
jbd2_log_wait_commit
jbd2_complete_transaction
ext4_sync_file
vfs_fsync_range
do_fsync
__arm64_sys_fdatasync
...
fdatasync
wal_append
commit_entry
request_loop
start_thread
- ledger (17385)
3050000
Read it bottom-up like a profile: commit_entry calls fdatasync, which waits for the ext4 journal (jbd2_log_wait_commit). 3.05 s of 5 s blocked there. -f prints folded stacks for an off-CPU flame graph.
The two kinds of sleeping.S (interruptible: waiting for an event, a socket, a timer) is normal for idle servers - most off-CPU time on any box is threads with nothing to do. D (uninterruptible: waiting on I/O) is the one that counts in load and usually in latency. --state 2 limits offcputime to D. Filter to the process you care about (-p), or the output is dominated by idle event loops.
Confirm with the file system view.ext4slower-bpfcc 10 prints every ext4 read, write, open and fsync slower than 10 ms:
Tracing ext4 operations slower than 10 ms
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
20:14:02 ledger 17385 S 0 0 35.18 journal.wal
20:14:02 ledger 17385 S 0 0 41.02 journal.wal
T is the operation: R read, W write, O open, S sync. Every commit waits ~38 ms for fsync.
Tail latency. The p99 (the slowest 1%) is where users feel it, and off-CPU events cause most of it: a retransmit (200 ms), a throttled period (up to 100 ms), a slow fsync, a GC pause, a run queue wait. Averages hide them. Histograms (biolatency, runqlat, bpftrace hist()) and the application's own latency percentiles show them.
In an interview: "If latency is high but CPU usage is low, the time is off-CPU. I use offcputime on the process to see which stacks block and for how long, then the matching tool: ext4slower or biolatency for storage, tcpretrans for the network, runqlat for CPU queueing."
Why it helps
Many latency problems use almost no CPU: requests wait on a disk, a lock, the network or the run queue. On-CPU profiles and top show nothing for those, which is why they are hard to find. Off-CPU analysis with offcputime, confirmed with ext4slower, biolatency or tcpretrans, finds where the time goes when the CPU is idle, and tail latency (p99) is usually made of exactly these waits.
Time a thread spends not running: blocked on I/O, waiting for a lock, sleeping, or runnable but waiting in the run queue. A request that takes 2 seconds while using 20 ms of CPU spent the rest off-CPU. An on-CPU profile cannot see any of it.
What does offcputime-bpfcc show?
For each stack, the total time threads spent blocked there, in microseconds, measured from the moment the scheduler switched them out until they ran again. -p PID limits it to one process, -f prints folded stacks for an off-CPU flame graph, and a duration bounds it.
What is the difference between S and D sleep?
S is interruptible sleep: waiting for something that may never come, such as a network message or a timer, and a signal can wake it. D is uninterruptible: usually waiting for disk I/O that must finish first. Both are off-CPU; D also counts in the load average.
How do I confirm that the file system is slow?
sudo ext4slower-bpfcc 10 prints every ext4 read, write, open or fsync slower than 10 ms with the process, the size and the file name. If the slow stacks from offcputime end in fsync and ext4slower shows slow fsyncs on the same file, the storage is the cause.
Why does the p99 matter more than the average?
Averages hide the slow requests: if 1% of requests wait 2 seconds on a disk, the average barely moves but one user in a hundred sees a timeout, and a page that makes 50 calls almost always hits one. Off-CPU waits like fsync and retransmits usually live in that tail.
In an interview Mid
Latency is high but the process uses almost no CPU. Where is the time?
Off-CPU: the threads are blocked on I/O, locks, the network or the run queue. I run offcputime-bpfcc -p PID for a few seconds to see which stacks block and for how many microseconds, then confirm the specific cause with the matching tool - ext4slower or biolatency for storage, tcpretrans for the network, runqlat for CPU queueing. An on-CPU profile would show nothing useful here.
Also asked: What is the difference between on-CPU and off-CPU analysis? · How would you confirm a slow fsync? · Why look at p99 rather than the average latency?