OnCallReadyStart learning

Linux Lesson 8.31 · Linux Performance & eBPF · 5 min read

System calls: what a process asks the kernel

In plain words

A program cannot do anything outside its own memory on its own. To read a file, send a message or wait, it has to ask the kernel, like a guest asking hotel reception. Each request is a system call. If you count the requests a guest makes - and which ones reception refuses - you learn a lot about what they are up to, without ever entering their room.

The conversation between a program and the kernel

Everything a process does outside its own memory - read a file, send a packet, wait, start a process - is a system call. Counting them answers "what is it doing?" without its source:

$ sudo syscount-bpfcc -P
Tracing syscalls, printing top 10... Ctrl+C to quit.
^C[20:00:11]
PID    COMM               COUNT
1210   java                 583
17385  invoice-worker       181
1302   logwriter             75
OptionShows
(none)top 10 system calls, box-wide
-Ptop 10 processes
-p PIDone process's system calls
-xonly failed calls (return < 0)
-Lthe time spent in each (µs; -m for ms)
-i 5print every 5 seconds

Failures are the interesting part. syscount-bpfcc -x on a healthy box shows a handful; a program retrying a missing file, a refused connection or a permission error shows thousands. Then opensnoop -x or a bpftrace one-liner on sys_exit_* with /args.ret < 0/ names them.

Time, not just counts. With -L a few calls with large times are usually waiting: futex (locks, condition variables), epoll_pwait (an event loop with nothing to do), clock_nanosleep (sleeping), fdatasync (the disk). Waiting syscalls are normal; the question is whether a request is waiting on them.

Why not strace? strace -c -p PID gives a similar table, but strace stops the process at every system call (ptrace) - it can slow a busy program down 10x or more. On a production service, use the eBPF tools; keep strace for a single command you are debugging (strace -f -e trace=openat,connect ./tool).

Kernel functions. funccount-bpfcc 'vfs_*' counts calls to every kernel function matching the pattern (kprobes): vfs_read, vfs_write, vfs_open... It is how you check the rate of an event before you attach a heavier tracer to it. Tracepoints work too: funccount-bpfcc t:sched:sched_process_exec.

Pressure stall information (PSI) sums up saturation per resource in three files (here while a restore job writes to the disk, as in the next lab):

$ cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory
some avg10=0.11 avg60=0.12 avg300=0.10 total=8920695
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some avg10=35.49 avg60=35.65 avg300=30.45 total=215369200
full avg10=14.47 avg60=13.67 avg300=12.15 total=86065450
some avg10=0.00 avg60=0.00 avg300=0.00 total=164460
full avg10=0.00 avg60=0.00 avg300=0.00 total=82230

some = share of time at least one task was stalled on that resource; full = all non-idle tasks were stalled at once (real lost work). Here: no CPU or memory pressure, but tasks wait for I/O 35% of the time and the whole box is stalled on I/O 14% of the time. Every cgroup has the same files (cpu.pressure, io.pressure, memory.pressure), and systemd-oomd uses them.

In an interview: "To see what a process does without slowing it down I count its system calls with eBPF (syscount, with -x for failures and -L for time); strace works but stops the process on every call. PSI in /proc/pressure tells me which resource tasks are stalled on."

Why it helps

Counting system calls tells you what a program does without its source code: a busy loop of failing opens, a process that only sleeps, a service hammering fsync. eBPF tools count them cheaply on a live service, where strace would slow it down badly. Pressure stall information adds one number per resource that says how much time tasks spend stalled, which is the saturation in USE measured directly.

Commands in this lesson

syscount-bpfcc cat

FAQ

What does syscount-bpfcc show?

By default the top 10 system calls on the whole box with their counts. -P groups by process instead, -p PID limits it to one process, -x counts only failed calls, -L adds the time spent in each call, and -i 5 prints every five seconds.

Why are failed system calls interesting?

A healthy program rarely fails a system call. Thousands of failures a second usually mean a retry loop: a missing file, a refused connection, a permission error. syscount-bpfcc -x shows which call fails, and opensnoop-bpfcc -x or a bpftrace one-liner on the exit probe names what it was trying to reach.

Why not just use strace?

strace uses ptrace, which stops the process at every system call and switches to strace and back. On a busy program that can slow it down ten times or more. It is fine for one command you are debugging; on a production service the eBPF tools give similar counts at a tiny fraction of the cost.

What does funccount-bpfcc count?

Calls to kernel functions matching a pattern, such as 'vfs_*', using kprobes, or tracepoints like t:sched:sched_process_exec. It is how you check how often an event happens before you attach a heavier per-event tracer to it, so the tracer does not surprise you with its cost.

How do I read /proc/pressure/io?

The some line is the share of time at least one task was stalled waiting for I/O; the full line is the share of time all non-idle tasks were stalled at once, which is lost work. avg10, avg60 and avg300 are averages over 10, 60 and 300 seconds; total is in microseconds.

In an interview Mid

How do you see what a running process is doing without its source code or slowing it down?

I count its system calls with eBPF: sudo syscount-bpfcc -p PID for the top calls, -x for the ones that fail and -L for where the time goes. A loop of failing openat calls with clock_nanosleep in between is a retry loop; futex or epoll_pwait at the top is a mostly waiting program. strace -c gives a similar table but stops the process on every call. PSI in /proc/pressure then tells me which resource tasks stall on.

Also asked: Why is strace risky on a production service? · What does the full line in a PSI file mean? · How would you find which file a failing openat is looking for?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.