OnCallReady

Lesson 3.6 · Processes & Signals · 12 min read

Signals

In plain words

Signals are like notes a teacher can pass to a pupil. "Please pack up now" is a note the pupil can read and act on sensibly. "Hang up the phone" once meant the line was cut, and nowadays it is the agreed code for "re-read your instructions". "Freeze" pauses the pupil like a statue; "carry on" unfreezes them. And one note is not a note at all: the headteacher simply carries the pupil out of the room.

That last one is SIGKILL (9): the kernel removes the process with no chance to clean up. SIGTERM (15) is "please pack up", the default of kill. SIGHUP (1) makes nginx reload. SIGINT (2) is Ctrl+C. SIGSTOP and SIGCONT freeze and unfreeze. pgrep finds the pupil before pkill passes the note.

Stopping, pausing and poking processes

You will need to stop a runaway process, make a web server re-read its config without dropping visitors, or pause a job that is hammering a database. All of that is done with signals, and picking the wrong one turns a clean stop into lost data.

What you need to know already: 2.10 (systemd treats SIGTERM as a clean stop, SIGKILL not), 3.1 (PIDs, STAT T).

What a signal is

A signal is a small numbered message the kernel delivers to a process. For most signals the process can install a handler - its own code that runs when the signal arrives ("flush my files, then exit"). Such a signal is catchable. If there is no handler, the kernel applies the signal's default action - for most signals, that is "terminate the process".

kill is the command that sends signals. Despite the name, it sends any of them.

The six you need

SIGTERM 15  "please stop". Catchable. The default of kill. The process gets
            to flush buffers, close connections, run its shutdown code.
SIGKILL  9  the kernel destroys the process. NOT catchable, NOT ignorable,
            NOT blockable. No cleanup of any kind.
SIGHUP   1  "hang up": originally "your terminal went away". By convention,
            daemons (background services) reload their configuration on it -
            nginx, sshd, rsyslog all do.
SIGINT   2  what Ctrl+C sends. Catchable.
SIGSTOP     pause. Not catchable either. The STAT becomes T.
SIGCONT     resume a paused process.

kill -l lists them all (numbers 1-64, with 32 and 33 missing: the C library keeps those for itself). You will use six.

Why -9 is not the default

kill -9 on a Java service means its shutdown code never runs: no finishing the requests in flight, no flushing buffers to disk. On a database it means an unclean shutdown and a slow recovery on the next start. On anything with a write-ahead log (a journal of changes the program writes before applying them, replayed after a crash) it means running the recovery path you have never tested.

The order is always: TERM, wait, then KILL if you must. systemctl stop does exactly that for you (2.24).

The one case where TERM cannot help is D state - and then KILL cannot either.

Targeting processes

kill 1234                  SIGTERM to PID 1234
kill -9 1234               SIGKILL (same as kill -KILL 1234)
kill -HUP 1234             SIGHUP: reload
pgrep -a nginx             list PIDs whose NAME matches, -a = with command lines
pgrep -f 'java.*orders'    -f = match the FULL command line, not just the name
pkill -f 'java.*orders'    same matching as pgrep, but sends SIGTERM to them
killall nginx              signal every process named exactly nginx

The pattern is a regular expression: . means "any character" and .* "anything", so java.*orders means "java, then later orders". (Chapter 7 goes deeper.)

Always pgrep before pkill. -f matches the whole command line, and the command line of the shell you are typing in may contain the pattern you just typed. People have killed their own session this way, and worse.

Jobs: your shell's own background processes

A job is a command your interactive shell started and keeps track of. & at the end runs it in the background (you get the prompt back at once); %1 means "job number 1" wherever a PID is expected:

sleep 600 &     run in the background -> [1] 12345   (job 1, PID 12345)
jobs            list this shell's jobs
kill -STOP %1   pause job 1          (or Ctrl+Z on the job in the foreground)
kill -CONT %1   resume it
fg %1 / bg %1   bring it to the foreground / resume it in the background

SIGHUP and reload

nginx is the web server on this box. It runs as one master process (root, it owns the config) plus worker processes that serve the requests.

sudo kill -HUP $(pgrep -o nginx)

(pgrep -o = the oldest match, which is the master.) nginx re-reads its config, starts new workers with it, and lets the old workers finish their in-flight requests before exiting. Zero dropped connections. Its error log narrates the whole thing.

That is what systemctl reload nginx does under the hood: its ExecReload= runs nginx -s reload, which just sends SIGHUP to the master. (sshd's unit is even more literal: ExecReload=/bin/kill -HUP $MAINPID.)

What you can now do

Why it helps

Signals are how every supervisor talks to processes: systemd stop and reload, log rotation telling daemons to reopen files, Ctrl+C in your terminal. Using the right one is the difference between a zero-downtime config change (kill -HUP or systemctl reload nginx) and dropped connections, and between a clean shutdown and a database replaying its write-ahead log for ten minutes after kill -9.

It also prevents self-inflicted incidents: pkill -f with a loose pattern has killed SSH sessions and unrelated services more than once. "TERM, wait, then KILL" and "pgrep before pkill" are habits interviewers listen for, and signal numbers explain every exit code 130, 137 and 143 you will see on stopped services.

Commands in this lesson

kill sleep

FAQ

Which signals cannot be caught?

SIGKILL and SIGSTOP. The kernel handles them directly; a process cannot install a handler, ignore them or block them. Everything else, including SIGTERM, SIGINT and SIGHUP, can be caught or ignored. That is why SIGKILL is the last resort for a stuck process and why a process can be paused with SIGSTOP even if it does not want to be.

Does every program reload on SIGHUP?

No, it is only a convention. Daemons like nginx, sshd and rsyslog re-read configuration on SIGHUP. For a program with no handler, SIGHUP's default action is to terminate, so sending it to the wrong process kills it. Check the program's documentation or its ExecReload=, and prefer systemctl reload unit, which uses whatever the unit defines.

What is the difference between pkill, killall and kill?

kill takes PIDs (or job specs like %1). pkill matches by process name, or with -f by full command line, using regular expressions, plus filters like -u user. killall on Linux matches exact process names. On some other Unix systems killall kills every process, so be careful with muscle memory on unfamiliar systems. Always preview with pgrep -a before pkill.

Why did pkill -f kill my own SSH session?

-f matches the pattern against the full command line of every process. Your shell or a wrapper command often contains the same text, for example ssh host 'pkill -f java' runs bash -c 'pkill -f java', whose command line contains "java". pkill excludes itself but not its parent shell. Preview with pgrep -af pattern and anchor the regex tightly.

What happens to a process stopped with Ctrl+Z?

Ctrl+Z sends SIGTSTP, which stops the foreground job: state T, no CPU, but it keeps its memory, open files and locks. bash prints [1]+ Stopped. fg resumes it in the foreground, bg in the background. A forgotten stopped job keeps holding locks and ports, a classic cause of "the port is in use" or "the database is locked".

In an interview Junior

What is the difference between SIGTERM, SIGKILL, SIGHUP and SIGINT?

The rule: TERM, wait, then KILL only if you must - kill -9 on a database means an unclean shutdown and a recovery on the next start. And neither reaches a process in D state. Find targets with pgrep -a before you pkill.

Also asked: How do you reload nginx's configuration without dropping connections? · Why should kill -9 not be your first resort? · How do you pause a process and resume it later?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.