OnCallReady

Lesson 2.5 · systemd · 11 min read

Your first service

In plain words

Think of hiring a babysitter for a child who should play in the living room. The babysitter watches that one child. If the child sneaks out the back door and a friend takes its place, the babysitter thinks the child left, and panics.

systemd is the babysitter and your program is the child. With Type=simple, systemd watches the exact process it started. If your script sends the real work into the background with & and then exits, systemd thinks the service has ended. So under systemd your program stays in the foreground and prints to its normal output; systemd does the "run in the background" part. daemon-reload is the babysitter re-reading the note you left, enable --now means "every evening, and also right now", and journalctl -u demo -f is listening at the door.

Why this matters

You have a small program that must keep running: after you log out, after it crashes, after a reboot. Running it by hand in your terminal is not enough - it dies the moment your SSH session ends. Turning it into a service hands the job to systemd. This lesson walks you through doing that for a tiny script.

What you need to know already: 2.1 (unit files and sections), 2.3 (why your files go in /etc/systemd/system), redirection and stdout (1.7).

Two words: script and executable

Foreground, not background

A program is in the foreground when it keeps running and holds on to the terminal until it is done, printing its output as it goes. A traditional daemon instead daemonises: it starts a copy of itself in the background and the original exits straight away.

Under systemd, your program should stay in the foreground and print its messages to stdout (the normal output stream). systemd is the one that runs it in the background, and it catches everything the program prints. If your script backgrounds itself (& at the end of a line), systemd sees the process it started exit immediately and thinks the service ended.

Type= decides what "started" means

Type= in [Service] tells systemd when to consider the service started:

simple    (default) the program systemd runs IS the service. It counts as
          started as soon as systemd has launched it. Use for any
          foreground program: a web server, a Python app, your script.
exec      like simple, but systemd waits until the program was actually
          launched successfully. Slightly better error reporting.
forking   the old daemon style: the program starts a background copy of
          itself and the original exits. Needs PIDFile= (a file where the
          daemon writes its PID) so systemd knows which process to watch.
oneshot   runs, does one job, exits. systemd waits for it to finish.
          Add RemainAfterExit=yes if it should still show as active after.
notify    the program tells systemd "I am ready now" itself. Best of all -
          "started" then means "actually ready" - but the program must be
          written to do it.

The rules for ExecStart

ExecStart= is the command systemd runs. It is not typed into a shell, so:

A minimal unit

[Unit]
Description=systemd service to run looped script

[Service]
User=learner
ExecStart=/home/learner/oncall-lab/labs/1a-linux/systemd/demo/script.sh
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

User=learner runs it as your user instead of root (the all-powerful admin account) - a service should have only the power it needs.

The commands after writing it

sudo systemctl daemon-reload     # make systemd re-read unit files from disk
sudo systemctl enable --now demo # start at every boot (enable) AND start now
systemctl status demo            # did it work?
journalctl -u demo -f            # watch its output live; Ctrl+C to stop

daemon-reload is not optional. Edit a unit without it and systemd keeps using the old definition. There is no error - only a one-line Warning: The unit file ... changed on disk. Run 'systemctl daemon-reload' above the output, which is easy to skim past. You will lose twenty minutes to this exactly once. (A brand-new unit file is picked up on first use; it is changes that need the reload. Run it every time anyway.)

Anything your service writes to stdout or stderr goes straight into the journal, tagged with the unit. That is why services today do not write their own log files.

What you can now do

Why it helps

Turning a script or program into a proper service is one of the most common platform tasks: a small sync agent on a server, a metrics collector, a helper on a jump box. Done right, the process restarts when it fails, its output lands in the journal, it starts at boot and it can be stopped cleanly.

The mistakes in this lesson are the ones that produce services that keep flapping in production: a script that backgrounds itself, a relative ExecStart, shell syntax like | inside ExecStart, a forgotten daemon-reload. They are also a classic interview task ("turn this nohup command into a systemd service"), and the reason hand-made log files fill disks: under systemd, printing to the journal is the right answer, and the journal cleans up after itself.

Commands in this lesson

systemctl

FAQ

When should I use Type=simple, exec, oneshot or notify?

simple is the default and fine for most programs that stay in the foreground; systemd counts it as started the moment the process exists. exec waits until the program was actually launched, so a missing program makes systemctl start fail. oneshot is for a job that runs and exits, like a cleanup. notify waits until the program itself says "ready", which needs support inside the program.

Can I use pipes or && in ExecStart?

No. systemd splits ExecStart into words and runs the program directly, with no shell in between, so |, &&, > and * are handed over as plain text. If you really need shell features, use ExecStart=/bin/bash -c 'cmd | other', at the price of awkward quoting. Usually better: put the logic in a script that starts with a #! line, and point ExecStart at the script.

Where does my program's output go?

To the journal. By default everything a service prints, both normal output and error output, is caught by journald and tagged with the unit name, the PID and priority info. Read it with journalctl -u demo, and follow it live with -f. You do not need your own log files or a clean-up job for them; journald limits its own size and age.

What does enable --now do exactly?

Two separate things at once. enable creates the link described in [Install] - for WantedBy=multi-user.target, a link in /etc/systemd/system/multi-user.target.wants/ - which only matters at the next boot. --now also starts the unit immediately. disable --now removes the link and stops the unit. Without --now, enable changes nothing about what is running right now.

Should my service run as root?

Not unless it has to. Set User= to a dedicated account without special rights, like appuser on this box. A service running as root turns any bug in it into control of the whole machine; running as its own user, the damage stays inside that user's files. 2.26 adds DynamicUser=yes, which creates a throwaway user for each run.

In an interview Junior

How would you turn a program you run by hand into a systemd service?

  1. Put it at a fixed absolute path and make it executable (chmod +x; a script needs its shebang).
  2. Make it run in the foreground and print to stdout - no &, no backgrounding itself, or systemd thinks it exited.
  3. Write /etc/systemd/system/app.service: [Service] with User= (not root), ExecStart=/usr/local/bin/app (absolute path; no pipes, ~ or other shell syntax), Restart=on-failure, RestartSec=10; [Install] with WantedBy=multi-user.target.
  4. sudo systemctl daemon-reload, then sudo systemctl enable --now app: enable = at every boot, --now = also start it now.
  5. Check: systemctl status app and journalctl -u app -f.

What you get over running it by hand: it survives logout and reboot, comes back after a crash, and everything it prints lands in the journal with a timestamp.

Also asked: What does Type=simple mean, and when would you use Type=oneshot? · Why is daemon-reload needed after editing a unit? · Why do services print their logs instead of writing their own log files?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.