OnCallReady

Lesson 2.3 · systemd · 9 min read

Precedence and drop-ins

In plain words

Imagine a school rulebook printed by the head office. You are not allowed to scribble in it, because every year a new edition arrives and your notes vanish. So instead you stick a small note on the page: "in this school, lunch is at 12:30". Anyone reading sees the printed rule and your sticky note, and your note wins.

The printed book is the package's unit in /usr/lib/systemd/system. The sticky note is a drop-in, /etc/systemd/system/cron.service.d/override.conf, which systemctl edit cron creates for you. You write only the lines you change. systemctl revert cron peels every sticky note off. And some rules are lists: a note adds a line to the list, so to replace the list you first write an empty line to clear it.

Why this matters

You need cron to run with one extra setting. The obvious move is to open its unit file and type it in. It works - until the next sudo apt upgrade installs a new version of the package, replaces the file, and your change silently vanishes. Nobody notices until 3am. This lesson is about where changes go instead.

What you need to know already: 2.1 (units, sections, systemctl cat), paths and directories (1.3), sudo (1.11).

Three directories, one winner

systemd looks for foo.service in these directories, in this order, and the first file it finds wins completely:

  1. /etc/systemd/system/ - yours, the administrator's. Survives upgrades.
  2. /run/systemd/system/ - generated at runtime; /run is emptied at every reboot.
  3. /usr/lib/systemd/system/ - the package's (what apt installed). /lib is a symlink (a shortcut that points to another path) to /usr/lib on modern Ubuntu, so both paths name the same file.

systemd-analyze unit-paths prints the full search list, in order.

Never edit the file under /usr/lib. Not because it will not work - it works perfectly, until apt upgrade replaces the file and your change silently disappears, usually on the machine nobody remembers you touched.

Drop-ins are the right answer

A drop-in is a small extra file that changes only the directives you list, on top of the packaged unit, instead of replacing the whole file. It is also called an override. It lives in a directory named after the unit plus .d:

/etc/systemd/system/cron.service.d/override.conf

Any .conf file in that directory is merged on top of the packaged unit, in alphabetical order. You write only what you change - with its section header, so systemd knows where the directive belongs:

[Service]
Environment=LAB=1

(Environment= sets an environment variable for the program: a named value, here LAB = 1, that the program can read when it starts. 2.21 is all about them.)

sudo systemctl edit cron creates that file for you, in the right place, opens it in a text editor, and tells systemd to re-read it when you save. Use it rather than typing the path by hand.

The editor it opens is nano, a simple terminal text editor. The keys you need: type normally to edit; Ctrl+O then Enter saves ("write Out"); Ctrl+X exits. The shortcut list at the bottom of the screen uses ^ to mean Ctrl.

Things worth knowing:

Undo

sudo systemctl revert cron

Deletes every drop-in and any copy under /etc, putting the unit back to exactly what the package ships. It prints each file it deleted.

systemctl cat and systemctl status both show drop-ins, so you can always see what has been layered on: cat prints each drop-in after the main file with its path, and status grows a Drop-In: line.

What you can now do

Why it helps

Drop-ins are how you change a service that a package installed without losing the change at the next apt upgrade: raising a limit on the web server, adding a setting to cron, capping the memory of a noisy agent. The package keeps improving underneath, and your change stays small and visible.

When debugging, the Drop-In: line in systemctl status is often the whole answer: someone set a limit months ago and nobody remembers. And knowing the empty-assignment rule for lists (ExecStart= on its own line, then the new one) saves you from the classic "Service has more than one ExecStart= setting" failure after an override.

Commands in this lesson

ls

FAQ

Why not just edit the file in /usr/lib/systemd/system?

Because the package owns it. The next apt upgrade of that package overwrites the file without asking, and your change silently disappears. Files in /etc/systemd/system belong to you, the admin, and packages never touch them. A drop-in keeps your change small, easy to see and easy to undo, while the package's own fixes keep arriving underneath.

What is the difference between systemctl edit and systemctl edit --full?

systemctl edit unit creates or opens /etc/systemd/system/unit.d/override.conf, a drop-in merged on top of the package's file. --full copies the whole unit to /etc/systemd/system/unit, which then replaces the package's file completely. Use --full only when a drop-in cannot express the change; otherwise you freeze an old copy and miss future fixes. Both reload systemd when you save.

Why did my drop-in with a new ExecStart= break the service?

ExecStart= is a list. A second one in a drop-in adds a second command instead of replacing the first, and a normal service may only have one, so systemd refuses to load it. Clear the list first with an empty line, then set the new value: ExecStart= on its own, followed by ExecStart=/new/command. Other list settings, like Environment= and After=, add up the same way.

In what order are multiple drop-ins applied?

All *.conf files in the unit's .d directories are sorted by file name and applied in that order, so a later file wins for single-value settings. If the same file name exists in the /etc and /usr/lib drop-in directories, the /etc one wins. That is why people use number prefixes like 10-limits.conf and 50-override.conf: the numbers make the order obvious.

How do I see all overrides on a box?

systemd-delta lists every unit on the system that is overridden, extended by drop-ins or masked, with the file responsible. For one unit, systemctl cat unit prints the package's file and every drop-in with its path, and systemctl status unit shows a Drop-In: line. Running systemd-delta is a good first step on any server you take over.

In an interview Junior

How do you change a setting of a service installed by a package, for example add an environment variable?

With a drop-in: sudo systemctl edit cron, and write only the change, under its section:

[Service]
Environment=LAB=1

That creates /etc/systemd/system/cron.service.d/override.conf and reloads systemd when you save; then sudo systemctl restart cron, because a running process keeps the settings it started with.

Why not edit the file itself: the package's unit lives in /usr/lib/systemd/system/ and apt upgrade replaces it, silently dropping your change. /etc/systemd/system/ wins over /run and /usr/lib, and survives upgrades. Check with systemctl cat cron (shows the drop-in after the main file) and the Drop-In: line in systemctl status. systemctl revert cron undoes it. Gotcha: list settings like ExecStart= add up - clear with an empty ExecStart= first.

Also asked: Which directories does systemd search for unit files, and which one wins? · When would you use systemctl edit --full instead of a drop-in? · You changed a service's settings but nothing happened. What do you check?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.