Think of a cookie cutter. One cutter, many cookies, and you can write a name on each one with icing. You do not need a separate cutter for every friend.
[email protected] is the cutter: one file on disk. worker@alpha and worker@beta are cookies, each a real running service, and inside the file %i is the name written in icing ("alpha"). That is how getty@tty1 gives every console its own login prompt. The second half of the lesson is about what happens when a cookie breaks: OnFailure= starts another unit to raise the alarm, SuccessExitStatus= says which endings are fine, and WatchdogSec= means "if you stop telling me you are alive, I will restart you".
Why this matters
You need four copies of the same worker program, each handling a different queue (a list of jobs waiting to be processed). Four near-identical unit files would drift apart the first time someone edits one. A template is one file that systemd stamps out as many times as you ask. And when a job fails, you want someone told - automatically.
What you need to know already: 2.5 (writing a unit, Type=oneshot), 2.10 (Restart=, SuccessExitStatus=), 2.12 (failed state).
One file, many instances
A unit file whose name contains @ before the suffix is a template:
You never start [email protected] itself - you start an instance (one running copy made from the template), and the bit after the @ is available inside the file as %i:
[Unit]
Description=Lab worker for %i
[Service]
ExecStart=/usr/local/bin/worker --queue %i
'worker@*' is a pattern: * matches anything. The quotes stop your shell from trying to match it against file names first.
%i is a specifier: a placeholder systemd fills in when it loads the unit. The ones you will use:
%i the instance name, as given
%I the instance name, unescaped (- becomes /, \x2d becomes -)
%n the full unit name, [email protected]
%N the same without the suffix
%p the prefix, worker
%H the hostname
Ubuntu itself uses templates: [email protected] is the login prompt on the console's first screen (one instance per screen), and [email protected] is a per-user systemd for the user with UID 1000 - you.
Unit names cannot contain /, so when the instance is a path it gets escaped (rewritten with - for /): systemd-escape -p /var/lib/data turns it into var-lib-data, which is exactly how .mount units are named.
When it fails
[Unit]
OnFailure=notify-oncall@%n.service
Start that unit whenever this one enters a failed state. The handler can be anything: a script that sends a chat message or an email to whoever is on call (the person responsible for responding to problems right now, chapter 0). Passing %n means the handler knows which unit failed.
[Service]
SuccessExitStatus=143 SIGTERM
Exit codes and signals to treat as success (from 2.10). 143 is 128+15, what many programs return after handling SIGTERM - without this line every clean stop of such a program is recorded as a failure.
A watchdog is a "still alive?" check. The program must send systemd a "still alive" message (a call named sd_notify(WATCHDOG=1) - the program has to be written to do it) at least every 30 seconds. If the messages stop - the program is stuck, not crashed - systemd kills it (with SIGABRT, a signal that also saves a core dump: a snapshot of the program's memory for debugging) and Restart=on-failure brings it back. The danger: too tight a value restarts a healthy service that is merely busy.
What you can now do
write a template unit and start several instances of it
run another unit when one fails
explain what a watchdog catches that Restart= alone does not
Why it helps
Templates show up whenever you run several copies of the same thing on one server: one worker per queue, one per customer, one tunnel per remote site, one backup job per database. One file means one place to fix a bug, and systemctl list-units 'worker@*' shows the whole set at a glance.
Failure handling turns systemd into a small monitoring system: OnFailure=notify@%n.service sends an alert with the failed unit's name, with nothing else to install. WatchdogSec catches the process that is hung but not dead - still "running", doing nothing - which a restart policy alone misses. It comes with a real trade-off: set it too tight and you restart healthy services that are merely busy.
No. A template file is only a pattern; systemd needs an instance name to make a real unit. systemctl start worker@alpha creates the instance with %i=alpha. You can enable instances too (systemctl enable worker@alpha), which creates a link named after the instance. DefaultInstance= in [Install] sets which instance is used if you enable the template without a name.
What is the difference between %i and %I?
%i is the instance name exactly as written in the unit name. %I is the same name with systemd's escaping undone: - becomes /. That matters when the instance stands for a path: systemd-escape -p /var/lib/data turns the path into var-lib-data so it fits in a unit name, and %I turns it back into /var/lib/data inside the file.
When does OnFailure= fire?
When the unit enters the failed state: a non-zero exit not listed in SuccessExitStatus, an unclean signal, a timeout, a missed watchdog ping, or hitting the start limit. With Restart= on, each crash goes to activating (auto-restart) instead of failed, so in practice OnFailure fires when systemd gives up, for example at the start limit. Pass %n so the alert knows which unit failed.
How does the watchdog know the program is alive?
The program has to tell it. With WatchdogSec=30, the program must send systemd a small "still alive" message at least every 30 seconds, using a library call that talks to systemd. If the messages stop, systemd kills the program and applies Restart=. So it only works for programs written to support it; for anything else you need an outside check instead.
How do I list all instances of a template?
systemctl list-units 'worker@*' shows the loaded instances and their states; add --all to include stopped ones. systemctl status 'worker@*' prints the status of each. Keep the quotes, or the shell tries to expand the * itself as a file-name pattern. You can also act on all of them at once: sudo systemctl restart 'worker@*'.
In an interview Junior
What is a template unit like [email protected], and when would you use one?
A unit file with @ before the suffix is a template: one file systemd stamps out as many times as you ask. You never start [email protected] itself; you start instances - sudo systemctl start worker@alpha worker@beta - and inside the file the part after the @ is available as %i:
Use it when you need several copies of the same thing that differ only by a name: four workers for four queues, without four near-identical files drifting apart. Ubuntu itself does it: [email protected] is the console login prompt, [email protected] your per-user systemd. systemctl list-units 'worker@*' shows the running instances; %n is the full unit name, handy in OnFailure=notify-oncall@%n.service.
Also asked: How can a service tell someone when it fails, without extra monitoring software? · What does a watchdog (WatchdogSec=) catch that Restart= alone does not? · What is %i, and what are specifiers?