OnCallReady

Lesson 3.12 · Processes & Signals · 12 min read

lsof and strace

In plain words

Imagine a library where every book, every phone line and every letterbox is signed out in one big ledger: who has it and since when. lsof reads that ledger. Ask "who has phone line 8080?" (lsof -i :8080) or "what has this person checked out?" (lsof -p 1210). If you are not the librarian (root), you only see your own entries, so always ask with sudo.

strace is different: it is like standing behind someone and writing down every request they make at the desk, "open this", "read that", "wait for a letter". If the last line says "waiting for a letter" and nothing ever arrives, you know exactly why they are stuck. That is read() on a socket with no timeout.

"Something is on port 8080" and "it is up but does nothing"

Two incident questions come up every week: which process holds this port or this file? and this process looks healthy, so what is it actually waiting for? lsof answers the first, strace the second.

What you need to know already: 1.7 (file descriptors 0, 1, 2), 2.1 (systemctl status), 3.1 (PIDs), 3.3 (system calls).

Words first

lsof: list open files

lsof lists every file descriptor of every process - files, sockets, pipes:

sudo lsof -i :8080          -i :PORT = network sockets on that port (who listens, who is connected)
sudo lsof -p 1210           -p PID  = everything this process has open
sudo lsof /var/log/app.log  a path  = who has this file open
sudo lsof -u appuser        -u USER = everything this user's processes have open
sudo lsof +L1               files with a link count below 1: DELETED but still open (Chapter 4)
sudo lsof -t -i :8080       -t      = just the PIDs, for feeding into kill

Its main columns: COMMAND (program name), PID, USER, FD (the fd number plus r/w/u for read/write/both), TYPE (REG file, DIR, IPv4 socket...), NAME (the path, or the address and port, with (LISTEN) for a listener).

Run it with sudo. Without root, lsof can only look at your processes, so lsof -i :8080 comes back empty and you conclude nothing is listening. It is the single most common way to be misled by this tool.

ss -tlnp is the faster answer for "what is listening": -t TCP, -l only listening sockets, -n show numbers not names, -p show the process. (It also needs sudo to show other users' processes.) lsof wins when you want which files, not just which sockets.

From a PID back to a unit

systemctl status 1210

Give systemctl status a PID instead of a unit name and it tells you which unit that process belongs to. Very useful when lsof or top hands you a number and you need to know what owns it - and if the answer is "no unit", you have found something started by hand that nobody supervises.

strace: what is it actually doing?

strace prints every system call a process makes - "open this file", "read from that socket", "wait for data" - one per line, as it happens:

sudo strace -p 1210              -p PID: attach to a running process (only its main thread)
sudo strace -f -p 1210           -f: also its other threads, and anything it starts
sudo strace -e trace=openat ls   -e trace=: only these system calls (here: run ls and show its file opens)
sudo strace -c -p 1210           -c: count and time system calls instead of printing each one
sudo timeout 5 strace -p 1210    timeout 5: stop strace after five seconds

A thread (3.1) is one line of work inside a process. Busy servers keep a thread pool: a fixed set of threads, each handling one request at a time. On a multi-threaded program like the orders Java service, plain -p shows only the main thread, which just waits for the others: one futex(... FUTEX_WAIT ...) line (futex = "wait until another thread wakes me") and silence. Use -f; each line then starts with [pid NNNN], the thread it came from.

What you are looking for is the last, unfinished line:

read(214, <unfinished ...>       waiting for data on fd 214
epoll_pwait(88,                  waiting for any of many sockets (normal for servers)
futex(0xffff8c0008b8, FUTEX_WAIT_BITSET_PRIVATE, ...   waiting for another thread

read() that never returns means it is waiting for the other end of a socket to send something. If that socket is a call to another service (an upstream) with no timeout configured - no "give up after N seconds" - it will wait forever. So will the request behind it, and the thread. Eventually every thread in the pool is stuck, and the service is down while every process looks perfectly healthy.

A note on cost: strace pauses the process at every system call. On a busy production service it can slow it down tenfold. Attach briefly, prefer -c, and use timeout.

What you can now do

Why it helps

"Address already in use" on a deploy, "who is holding this file", "why is disk space not freed", "what is this service actually connected to" are daily questions, and lsof answers all of them. Forgetting sudo and concluding "nothing listens on 8080" is a genuine source of wrong incident conclusions.

strace is how you debug a process that looks healthy but does nothing: a thread pool exhausted by upstream calls without timeouts, a service blocked on a lock file, a config file opened from an unexpected path. The orders scenario in this chapter, read() hanging because read.timeout.ms=0, is a very common production pattern and a great story for interviews about debugging hung services.

Commands in this lesson

lsof systemctl

FAQ

Should I use lsof or ss to see listening ports?

ss -tlnp is faster and purpose-built for sockets: TCP, listening, numeric, with the owning process (needs sudo for other users' processes). lsof -i :8080 also works and shows established connections to that port, but is slower on busy boxes because it scans every process. Use ss for "what listens where" and lsof for "which files and sockets does this process hold".

Why does lsof show nothing without sudo?

lsof reads /proc/PID/fd for every process, and those directories are only readable by the process owner and root. As a normal user you see only your own processes, so a port held by a service user looks free. There is no error message, just missing output. Always run it with sudo when looking at services.

Is strace safe to run on production?

It is safe in the sense that it will not change what the program does, but it is not free: it uses ptrace, the kernel's debugging interface, which stops the traced process at every system call, and can slow a busy service by an order of magnitude. Attach briefly with timeout 5, filter with -e trace=, or use -c for a summary. Lower-overhead tracing tools exist for when you need to watch for longer.

How do I go from a PID to the service that owns it?

systemctl status PID prints the unit (or session scope) containing that process, with its status. cat /proc/PID/cgroup gives the cgroup path, which names the service or the login session. If the answer is a user session scope rather than a service, something was started by hand and nothing supervises it.

What do the lines at the end of strace output mean?

The last line is what the process is doing right now. read(214, <unfinished ...> means it is waiting for data on fd 214; check what that fd is in /proc/PID/fd. epoll_wait or poll means an event loop idling, normal for servers. futex(... FUTEX_WAIT ...) means a thread waiting on a lock or condition. connect stuck means a remote host not answering.

In an interview Junior

A deploy fails with "Address already in use" on port 8080. How do you find what holds the port?

sudo ss -tlnp (TCP, listening, numeric, with the process) or sudo lsof -i :8080. The sudo matters: without root both only see your own processes, so the port looks free and you conclude nothing is there.

Then turn the PID into an owner: systemctl status PID names the unit whose cgroup holds it - often the old instance that did not stop. If it has no unit, it was started by hand and nobody supervises it; ps -o pid,ppid,etime,args -p PID shows how long it has been there.

Stop the right owner properly (systemctl stop), not kill -9 on whatever lsof -t returns. And if a process is up but does nothing, sudo strace -f -p PID shows what it is blocked on: a read() that never returns is waiting on a socket with no timeout.

Also asked: A service is running but requests hang. How would you see what it is waiting for? · How do you see which files a process has open? · Why is strace risky on a busy production service?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.