OnCallReady

Lesson 2.21 · systemd · 10 min read

Exec directives, environment and secrets

In plain words

Imagine giving a courier a note taped to the outside of a parcel. Everyone who handles the parcel - the sorting office, the van driver, the neighbour who signs for it - can read that note. Now imagine instead a locked box that only the recipient has a key for.

Environment variables (the NAME=value settings a process starts with) are the taped note. Environment=DB_PASSWORD=... can be read by any user through systemctl show -p Environment, and by root in /proc/PID/environ. LoadCredential= is the locked box: systemd reads the secret file as root and gives the service a private copy in a folder only it can read. The Exec family (ExecStartPre, ExecStartPost, ExecReload, ExecStop) are the steps before, after and around the main job, and a leading - means "a failure here is fine".

Why this matters

Real services need a little work around the main program: check the config file exists before starting, clean up a leftover file, tell the program to re-read its settings. They also need settings - which database, which port, and sometimes a password. How you hand over that password decides who else on the box can read it.

What you need to know already: 2.3 (drop-ins, Environment=), 2.10 (signals, kill), 2.5 (User=).

The exec family

Besides ExecStart=, a service can run commands before, after and around it:

[Service]
ExecStartPre=/usr/bin/test -f /opt/app/config.yaml
ExecStartPre=-/usr/bin/rm -f /run/app.lock
ExecStart=/opt/app/bin/orders-server
ExecStartPost=/usr/local/bin/notify-deploy.sh
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/local/bin/drain.sh

(test -f FILE succeeds only if FILE exists; rm -f deletes without complaining if the file is not there.)

Other prefixes exist (+ runs that one command as root, ignoring User= and the sandbox of 2.26); you rarely need them.

Environment

An environment variable is a named value (KEY=VALUE) handed to a program when it starts; the program reads it like a setting. Your shell has them too: echo $HOME prints one.

Environment=LOG_LEVEL=info
Environment="APP_NAME=orders api"          # quote if it contains spaces
EnvironmentFile=/etc/default/orders        # a file, one KEY=VALUE per line
EnvironmentFile=-/etc/default/orders.local # the - means "fine if missing"

The - prefix on EnvironmentFile is the difference between "optional local override" and "service fails to start on a machine that does not have it".

Environment files are not shell scripts. No export, no $OTHER to refer to another variable, no $(command). systemd reads plain KEY=VALUE lines.

Environment is fixed when the process starts: after changing it you must restart the service; daemon-reload alone only updates systemd's copy.

Environment variables are not a secret store

A secret is a value only the service should know: a password, an API key. Anything in Environment= is visible to:

systemctl show -p Environment orders        # any user, no privileges
sudo cat /proc/<PID>/environ | tr '\0' '\n'  # root, or the service's own user

/proc is a folder the kernel fills with live information about every process; /proc/<PID>/environ holds that process's environment, separated by invisible zero bytes - tr '\0' '\n' swaps each one for a new line so you can read it. (Chapter 4 tours /proc and /sys.)

systemctl show needs no privileges at all. A password in Environment= is readable by every user on the box. EnvironmentFile= is better only because you can lock the file so only root can read it (chmod 600, chapter 4) - the value is still in /proc/PID/environ.

The systemd-native answer:

LoadCredential=db-password:/etc/creds/db-password

systemd (as root) reads the file and gives the service its own copy at $CREDENTIALS_DIRECTORY/db-password - a private folder kept in memory, readable only by that service's user, not in the environment (so not in systemctl show or /proc/PID/environ), gone when the unit stops. The program reads the password from that file.

Bigger setups use a dedicated secrets service, but the principle is the same: a file only the service can read, never an environment variable.

What you can now do

Why it helps

Secrets in environment variables are one of the most common findings in security reviews. Knowing that systemctl show reveals them to every user on the box, and that they are copied into every program the service starts, lets you make a concrete case in a code review instead of a vague "this is insecure".

The Exec directives matter day to day: an ExecStartPre= config check (like sshd -t, which tests sshd's config) stops you restarting into a broken config; ExecReload= makes systemctl reload do something; and the - prefix on an EnvironmentFile decides whether a missing optional file breaks startup - or, used carelessly, hides a missing required one (the incident in 2.23).

Commands in this lesson

cat

FAQ

Can I use shell syntax in an EnvironmentFile?

No. systemd reads it as simple KEY=VALUE lines with basic quoting. Recent versions tolerate an export in front, but $OTHER references, $(command) and if are not run the way a shell would run them. If a shell script also reads the same file, keep it to plain assignments. For values that must be computed, use a small script or an ExecStartPre= that writes the file.

What does the - prefix do on ExecStartPre and EnvironmentFile?

On an Exec*= line, - means a failure of that command is ignored: the start carries on even if it exits non-zero. On EnvironmentFile=, it means "do not fail if the file is missing". Without it, a missing environment file makes the unit fail with Result: resources. Use it for optional files only, never for settings the service genuinely needs.

Why can ExecReload use $MAINPID?

systemd fills in a few variables itself when it runs Exec lines, and $MAINPID holds the PID of the service's main process. ExecReload=/bin/kill -HUP $MAINPID therefore sends the "reload your config" signal (SIGHUP) to exactly the right process. No shell is involved; systemd does the substitution. Variables from Environment= can be used the same way.

Is an EnvironmentFile readable only by root safe enough for secrets?

Safer than Environment=, because other users cannot read the file and systemctl show does not display its contents. But once loaded, the values live in the process environment: readable by root and by the service's own user in /proc/PID/environ, copied into every program the service starts, and often printed by crash reports or debug pages. LoadCredential= keeps them out of the environment entirely.

How can I check what a running service actually received?

Two views. systemctl show svc -p Environment shows what the unit file set with Environment= (not the contents of an EnvironmentFile). The running process's real environment is in /proc/<PID>/environ, readable by root: sudo cat /proc/$(systemctl show -p MainPID --value svc)/environ | tr '\0' '\n'. The entries are separated by an invisible NUL character, and tr turns each into a newline.

In an interview Junior

How should a systemd service get a database password, and why not with Environment=?

Not with Environment=: environment variables are not a secret store. systemctl show -p Environment svc prints them to any user, no privileges needed, and root can read them in /proc/PID/environ.

EnvironmentFile= is a little better, because you can lock the file with chmod 600 - but the value still ends up in /proc/PID/environ.

The systemd-native way is LoadCredential=db-password:/etc/creds/db-password. systemd, as root, reads the file and gives the service a private copy at $CREDENTIALS_DIRECTORY/db-password: kept in memory, readable only by the service's user, not in the environment, and gone when the unit stops. The program reads its password from that file. The principle: a file only the service can read, never an environment variable.

Also asked: What are ExecStartPre=, ExecReload= and ExecStop= used for? · What does the - prefix do in EnvironmentFile=-/etc/default/app? · You changed Environment= and ran daemon-reload, but the program still sees the old value. Why?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.