OnCallReadyStart learning

systemd: glossary

143 terms, in plain words, with where each one is taught.

/proc/PID/limits (limits file)
A file the kernel keeps for every running process, listing the limits it really got.
/sys/fs/cgroup (system.slice)
A folder where the kernel shows every cgroup as a directory, with files for its limits and usage; system.slice is the group holding all system services. Lesson 2.26
/tmp
The shared scratch folder any user or program can write temporary files into. Lesson 2.26
/usr/local/bin
The folder for programs installed by hand rather than by apt, and the right home for a service's scripts. Lesson 2.26
137 (exit code 137, exit 137)
The exit code of a process killed with SIGKILL: 128 + 9. When you see it, think "killed by force", often for using too much memory. Lesson 2.9
143
The exit code of a program that ended because of SIGTERM: 128 + 15, the signal number. Lesson 2.10
203/EXEC (203, status=203/EXEC)
systemd's exit code for "I could not start that program at all": not executable, wrong path, bad shebang, or blocked by a sandbox setting.
ABRT (SIGABRT)
The signal a program sends itself when it gives up on purpose (a failed internal check), so it dies and can leave a core dump.
anacron
An old cron add-on that ran jobs missed while the machine was off. Timers do this with Persistent=true. Lesson 2.16
AppArmor
An Ubuntu security feature that limits what each program may touch, loaded at boot as apparmor.service. Lesson 2.33
bad-setting
systemd's load state for a unit file with a setting it refuses, such as a relative path or ~ in ExecStart. The unit cannot be started at all. Lesson 2.5
batch job
A program that works through a pile of data and then exits, rather than running forever.
BindsTo (BindsTo=, PartOf=, PartOf)
Stricter relationships: BindsTo= stops me whenever the other goes away; PartOf= passes the other's stop/restart on to me; Conflicts= stops the other when I start. Lesson 2.14
boot ID (--list-boots)
A random ID the journal gives each boot. journalctl --list-boots lists them; -b -1 reads the previous boot. Lesson 2.30
calendar expression (OnCalendar, OnCalendar=)
A schedule written as DayOfWeek Year-Month-Day Hour:Minute:Second, where * means any: Mon *-*-* 09:00:00 is every Monday at 09:00. systemd-analyze calendar "..." explains one and shows when it next fires. Lesson 2.16
cgroup (cgroups, control group, control groups)
A box the kernel puts around a set of processes so they can be counted and limited together. systemd gives every service its own. The CGroup: line in systemctl status shows the processes in the box; systemd-cgls shows the whole tree. Lesson 2.9
changed on disk
systemd's warning that a unit file was edited after it was loaded; nothing in the edit applies until systemctl daemon-reload. Lesson 2.5
CHDIR (200/CHDIR)
systemd's exit code 200: it could not change into the WorkingDirectory= of the unit (missing, or not allowed), so the program never started.
chmod
The command that changes a file's permissions ("change mode"); chmod +x makes a file executable. Chapter 4 covers permissions fully. Lesson 2.5
clean exit
A program that ended by itself with exit code 0, or (for systemd) by SIGTERM, SIGINT, SIGHUP or SIGPIPE. Restart=on-failure does not restart it. Lesson 2.10
command substitution ($(command), $(...))
$(command) inside another command is replaced by that command's output: kill $(systemctl show -p MainPID --value demo).
commented out (comment out)
A line (or rest of a line) starting with #, ignored by the program reading the file. Commenting a line out disables it.
console
The machine's own screen and keyboard (for a VM, its window), as opposed to logging in over SSH. Lesson 2.18
core dump (core-dump, dumped core)
A copy of a crashed program's memory, saved to a file so someone can debug the crash later. Lesson 2.9
CPU core
One processor inside the CPU that can run one thing at a time. A machine with 2 cores can run two things truly at once. Lesson 2.26
CPUQuota (CPUQuota=)
How much CPU time a service may use, as a percentage of one core: 20% = a fifth of one core, 200% = two whole cores. Lesson 2.26
critical chain (critical-chain)
The chain of units the boot really waited on (systemd-analyze critical-chain). blame just lists units by start time, which can mislead. Lesson 2.33
CRLF (CRLF line endings, \r)
The Windows way of ending lines with an extra invisible character (\r). A script saved that way breaks its shebang on Linux. Lesson 2.7
cron (crontab)
The older Unix job scheduler: a daemon that runs commands listed in a crontab file at set times. Lesson 2.1
daemon-reexec
Restart systemd's own program (PID 1) in place, keeping its state; used after upgrading systemd. Lesson 2.33
daemon-reload (systemctl daemon-reload)
Tell systemd to read all unit files from disk again. Without it, edits to a unit file are ignored. Lesson 2.1
daemonise (daemonize, daemonises, daemonised)
The old way for a program to become a daemon: start a background copy of itself and let the original exit. Lesson 2.5
default target (get-default, set-default, graphical.target)
The target the machine boots into. systemctl get-default shows it, set-default changes it. Lesson 2.18
directive (directives)
One Name=value setting line in a unit file, like Restart=on-failure. Lesson 2.1
dmesg
A command that prints the kernel's own messages; journalctl -k shows the same. Lesson 2.30
draining
Letting a service finish the work it already has before it stops, instead of cutting it off. Lesson 2.21
DynamicUser (DynamicUser=)
A setting that makes systemd invent a temporary user account for a service while it runs, with strict sandboxing switched on. Lesson 2.26
enable (systemctl enable)
Set a unit to start at every boot. It does not start it now; start does that, and enable --now does both. On disk, enabling is just a symlink in a .wants directory. Lesson 2.1
EnvironmentFile (EnvironmentFile=)
A directive naming a file of NAME=value lines to load as environment variables. A leading - means "fine if the file is missing". Lesson 2.9
exec (execs)
To ask the kernel to load a program file and run it. systemd execs ExecStart after preparing the process. Lesson 2.5
ExecStart (ExecStart=)
The directive holding the exact command systemd runs for a service. It needs an absolute path and is not run by a shell. Lesson 2.1
ExecStartPre (ExecStartPre=, ExecStartPost=, ExecStop=)
Commands a service runs around its main one: before start (Pre), after start (Post), on systemctl reload (Reload), before stopping (Stop). A leading - on the command means "ignore it if it fails". Lesson 2.3
exposure score (systemd-analyze security, exposure level)
The 0-10 score from systemd-analyze security: 0 = locked down, 10 = wide open. Under about 7 is respectable. Lesson 2.26
for loop (for ... do ... done)
Shell syntax that repeats commands: for i in 1 2 3; do COMMANDS; done runs COMMANDS three times. Chapter 6 covers loops.
foreground
A program in the foreground keeps running and printing where it was started until it is done. Services should run this way under systemd. Lesson 2.5
getty (getty@tty1)
The program that shows the login prompt on the console; one instance per console screen. Lesson 2.28
grace period
The time a program gets to finish cleanly after being asked to stop, before it is killed by force. Lesson 2.24
hardening (sandboxing, sandbox)
Taking away everything a service does not need (files, powers, devices), so a broken or attacked service can do less damage. Lesson 2.26
identifier (SYSLOG_IDENTIFIER, syslog identifier)
The name a journal line is written under, shown before the PID: in sshd[700]: it is "sshd". journalctl -t name filters by it. Lesson 2.32
interpreter
The program that runs a script, named in its shebang: bash for a bash script, python3 for a Python one. Lesson 2.7
invocation ID (INVOCATION_ID)
A random ID systemd creates each time a unit starts, stamped on every journal line of that run. Lesson 2.9
IO (I/O, IOWeight, IOWeight=)
Input/output: reading from and writing to disks. IOWeight= sets a service's disk priority relative to others. Lesson 2.26
is-active (systemctl is-active)
Prints one word (active, inactive, failed...) and sets the exit code: 0 means active. The right check for scripts. Lesson 2.9
is-enabled (systemctl is-enabled)
Prints one word for a unit's boot setting: enabled, disabled, static, masked, alias... Lesson 2.18
isolate (systemctl isolate)
Switch to a target right now, stopping every unit it does not want. Over SSH that includes sshd, so only do it on the console. Lesson 2.18
kill
The command that sends a signal to a process by PID: kill PID sends SIGTERM, kill -9 PID sends SIGKILL. Despite the name, it just sends signals. Chapter 3 covers them all. Lesson 2.9
KillMode (KillMode=, KillSignal=)
Which processes of a service get the stop signal: all of them in its cgroup (control-group, the default) or only the main one (process). Lesson 2.24
kubelet
An agent program that runs on each node, checks in with the central system and runs its work. It refuses to run while swap is on. Chapter 17 covers what it is part of. Lesson 2.33
LimitNOFILE (NOFILE, RLIMIT_NOFILE, LimitNOFILE=)
The systemd setting for how many file descriptors each process of a service may have open ("NOFILE" = number of open files). Check the running process with grep "open files" /proc/PID/limits.
list-dependencies (systemctl list-dependencies)
Prints a unit's dependencies as a tree; --reverse shows who depends on it instead. Lesson 2.14
list-timers (systemctl list-timers)
Shows every timer on the box: NEXT (next run), LEFT (time until), LAST (last run), PASSED (time since), UNIT, ACTIVATES (the service it starts). Lesson 2.16
listening
A program is listening when it has opened a port and is ready to accept connections on it. Lesson 2.9
LoadCredential (LoadCredential=, CREDENTIALS_DIRECTORY)
systemd's safe way to hand a service a secret: it reads a file as root and gives only that service a private copy, outside the environment. Lesson 2.9
log priority (log level, -p err)
How serious a log line is, 0 (emerg) to 7 (debug). -p err shows 0 to 3: that level and worse. Lesson 2.30
LSB (Linux Standard Base)
An old Linux standard that gave names to exit codes 1-7 (1 FAILURE, 3 NOTIMPLEMENTED...). systemd prints those names next to the numbers. Most programs ignore the convention, so read the number, not the name. Lesson 2.9
main PID (MainPID)
The PID of the one process systemd is supervising for a service. systemctl show -p MainPID --value unit prints just the number.
mask (masked, unmask, systemctl mask)
Make a unit impossible to start by any route (by hand, as a dependency, at boot) by pointing it at /dev/null. unmask undoes it. Lesson 2.18
MemoryHigh (MemoryHigh=)
A soft memory limit: above it the kernel slows the service and takes memory back, but does not kill it. Lesson 2.26
MemoryMax (MemoryMax=, memory limit, memory.max)
A service's hard memory limit. Going over it gets the process killed by the kernel (SIGKILL, exit 137). Lesson 2.26
mount namespace
A private view of the filesystem that the kernel can give one process, so it sees different (or hidden) folders than everyone else.
multi-user.target
The target that means "the system is fully up for logins, no graphical desktop". Normal servers boot to it. Lesson 2.1
network-online.target (network.target)
The target meaning "the network is actually usable". network.target only means the network software is set up. Use After= and Wants= on network-online.target. Lesson 2.1
node (nodes)
One machine in a group of machines that a central system hands work to.
NoNewPrivileges (NoNewPrivileges=, PrivateTmp, PrivateTmp=)
Sandbox settings: NoNewPrivileges= stops the service ever gaining more power than it started with; PrivateTmp= gives it its own private /tmp. Lesson 2.26
NotReady
The central system's mark on a node that stopped checking in: "do not send it work".
NUL (NUL character, null byte)
An invisible character with code 0. Some kernel files, like /proc/PID/environ, use it to separate entries instead of newlines.
OnFailure (OnFailure=)
A directive naming a unit to start whenever this one fails, e.g. one that sends an alert. Lesson 2.28
OnUnitActiveSec (OnActiveSec, OnBootSec, OnUnitActiveSec=)
Relative timer settings: fire so long after boot (OnBootSec), after the timer started (OnActiveSec), or after the service last ran (OnUnitActiveSec, the repeater). Lesson 2.16
OOM killer (OOM, out of memory, OOM-killed)
The part of the kernel that kills a process when memory runs out (OOM = out of memory), or when a service goes over its memory limit. Chapter 5 is all about it. Lesson 2.26
ordering (After=, Before=)
A relationship "if we are both starting, I go after that one" (After=). It does not start the other unit by itself. Lesson 2.14
path unit (.path)
A unit type that starts a service when a file or directory changes.
persistent (persistent storage)
Kept on disk, so it survives a reboot. The journal is persistent once /var/log/journal exists. Lesson 2.16
Persistent= (Persistent=true)
A timer setting: if the machine was off when the job should have run, run it once as soon as it is back. Lesson 2.16
port (ports)
A numbered door on a machine that network connections arrive at; a program listens on a port, e.g. SSH on 22. Lesson 2.1
PostgreSQL
A popular database server program. Apps that store data connect to it over the network. Lesson 2.14
property (properties, systemctl show)
One named setting or live value systemd keeps for a unit, printed as Name=value by systemctl show. systemctl show unit -p Name --value is the stable way for scripts to read unit state.
ProtectHome (ProtectHome=)
A sandbox setting that hides /home, /root and /run/user from the service. A script stored in /home then fails with 203/EXEC. Lesson 2.7
ProtectSystem (ProtectSystem=, ReadWritePaths=)
A sandbox setting that makes system folders read-only for the service; strict makes everything read-only except the paths in ReadWritePaths=. Lesson 2.26
queue (queues)
A list of jobs waiting to be processed, one after another. Lesson 2.28
RandomizedDelaySec (RandomizedDelaySec=)
A timer setting that adds a random delay up to this long, so many machines do not all fire at the same second. Lesson 2.16
regex (regular expression, regular expressions, regexes)
A small pattern language for matching text: a|b means "a or b", . means "any character". Plain words match themselves. Lesson 2.32
rescue.target (emergency.target, rescue mode)
Repair modes with almost nothing running and a root shell on the console; emergency.target is even more minimal. Lesson 2.14
reset-failed (systemctl reset-failed)
Clears a unit's failed state and its start counter so it can be started again. It fixes nothing by itself. Lesson 2.12
Restart= (restart policy)
The directive that says when systemd should start a service again after it stops: no, on-failure, always, ... Lesson 2.1
RestartSec (RestartSec=)
How long systemd waits before restarting a service that stopped. Default 100ms. Lesson 2.1
Result (Result=)
systemd's one-word reason a unit failed: exit-code, signal, timeout, oom-kill, start-limit-hit, resources... Lesson 2.9
revert (systemctl revert)
Undo all local changes to a unit: delete its drop-ins and any copy under /etc, back to what the package ships. Lesson 2.3
runlevel (runlevels)
The older numbered system modes (1 repair, 3 server, 5 desktop). systemd uses targets instead. Lesson 2.18
search index (reindex)
The lookup table a search feature reads so it can answer quickly; rebuilding it is heavy work.
secret (secrets, credential, credentials)
A value only a service should know, like a password or API key. Never put one in an environment variable. Lesson 2.21
section (sections, [Unit], [Service])
A part of a unit file that starts with a name in square brackets, such as [Service]; the settings under it belong to that part. [Unit] = what it is and its relationships; [Service] = how to run the program; [Install] = what enable does. Lesson 2.1
sed (sed -i)
A command that edits text line by line; sed -i 's|old|new|' file replaces text inside the file in place. Chapter 7 covers it fully.
SEGV (SIGSEGV)
The signal the kernel sends a program that touched memory it does not own (a crash, a "segmentation fault"). The program dies unless it handles it.
set-property (systemctl set-property, system.control)
Changes a unit's resource settings at runtime; systemd saves them as drop-ins under /etc/systemd/system.control/. Lesson 2.26
shadow
When a unit file in a higher-priority directory (/etc) hides a same-named file in a lower one (/usr/lib), completely.
shebang (#!)
The first line of a script, starting with #!, naming the program that runs it: #!/usr/bin/env bash. Lesson 2.5
SIGHUP (HUP)
The "hang up" signal (number 1). Many daemons treat it as "re-read your config file without restarting". Lesson 2.10
signal handler
Code in a program that runs when a signal arrives, e.g. to finish current work and exit cleanly on SIGTERM. Lesson 2.24
socket activation (socket unit, .socket)
systemd holds a network port open itself and starts the matching service only when the first connection arrives. That is why ssh.service can say disabled yet you can still log in: ssh.socket is enabled. Lesson 2.1
soft limit (hard limit)
The soft limit is the one actually enforced; a process may raise it itself, up to the hard limit, which only root can raise. Lesson 2.26
specifier (specifiers, %i, %n)
A placeholder like %i (the instance name) or %n (the full unit name) that systemd fills in when it loads a unit. Lesson 2.28
start limit (StartLimitIntervalSec, StartLimitBurst, start-limit-hit)
systemd's brake on crash loops: more than StartLimitBurst starts within StartLimitIntervalSec and it gives up, marking the unit failed (start-limit-hit). These two go in [Unit], not [Service]. Lesson 2.12
static unit
The is-enabled answer for a unit with no [Install] section: nothing to enable or disable, other units pull it in. Lesson 2.18
Storage= (journald.conf)
The setting in /etc/systemd/journald.conf that says where the journal is kept: auto, persistent, volatile or none. Lesson 2.30
sub-state (substate, SubState)
The finer detail after a unit's state, in brackets: in active (running), "running" is the sub-state. Lesson 2.9
SuccessExitStatus (SuccessExitStatus=)
Extra exit codes or signals systemd should count as success, e.g. 143 for a program that exits that way after SIGTERM. Lesson 2.10
swapoff (swapon, swapon --show, swapoff -a)
swapon --show lists active swap; sudo swapoff -a turns all swap off until the next boot (fstab turns it back on).
systemd-analyze
systemd's inspection tool: checks unit files (verify), scores sandboxing (security), measures the boot (time, blame), explains schedules (calendar). Lesson 2.3
systemd-analyze verify
Checks a unit file for mistakes (unknown keys, wrong sections, missing programs) without loading it. Silence means it found nothing. Lesson 2.7
systemd-cgls (systemd-cgtop)
systemd-cgls prints the tree of cgroups and their processes; systemd-cgtop is a live view of CPU, memory and IO per cgroup. Lesson 2.26
systemd-escape
Rewriting characters a name cannot contain: systemd-escape turns the path /var/lib/data into var-lib-data for a unit name. Lesson 2.28
systemd-journald (journald)
The daemon that collects all log lines and stores the journal. Lesson 2.18
systemd-networkd-wait-online
A service that just waits until the network is up; it is what makes network-online.target true, and a common cause of slow boots. Lesson 2.33
systemd-run (--scope)
Runs a one-off command inside its own temporary (transient) unit, so you can give it limits: systemd-run --scope -p MemoryMax=200M cmd. Lesson 2.18
target (targets, .target)
A unit that is just a named group of other units, used as a milestone during boot, like multi-user.target. Lesson 2.1
TasksMax (TasksMax=)
The maximum number of processes and threads a service may have at once. Lesson 2.9
template (template unit, templates, [email protected])
A unit file with @ before the suffix ([email protected]) that systemd can stamp out as many instances as you ask. Lesson 2.28
TimeoutStopSec (TimeoutStopSec=, TimeoutStopUSec)
How long systemd waits after SIGTERM before sending SIGKILL when stopping a service. Default 90 seconds. Lesson 2.24
timers.target
The boot milestone that starts every enabled timer; timers use WantedBy=timers.target. Lesson 2.16
tr
A command that translates characters: tr '\0' '\n' turns every NUL into a newline. Lesson 2.21
TriggeredBy
The line in systemctl status naming the timer or socket unit that starts this service on demand.
Type= (Type=simple, Type=oneshot, Type=forking)
The directive that tells systemd when a service counts as started: simple (as soon as launched), oneshot (a job that runs and exits), notify (the program says when it is ready)... Lesson 2.1
Type=exec
A service type like the default one, except systemctl start waits until the program was actually launched, so a missing or unrunnable program makes the start command fail. Lesson 2.9
uniq
A command that collapses identical neighbouring lines into one; -c puts a count in front of each. Usually after sort, so identical lines are next to each other. Lesson 2.32
unit (units)
One thing systemd manages and can start, stop and order: a service, a timer, a target, a mount... The suffix of the file name says which kind: .service, .timer, .target, .socket, .mount. Lesson 2.1
User=
The directive naming the Linux user account a service runs as. Services should not run as root unless they must. Lesson 2.1
userspace (user space)
Everything that runs outside the kernel: systemd and all the programs it starts. Lesson 2.33
vacuum (--vacuum-time, --vacuum-size, SystemMaxUse=)
Deleting old journal files to free disk: --vacuum-time=7d keeps a week, --vacuum-size=200M keeps 200M. Lesson 2.30
vendor file (vendor unit)
The unit file a package installs, under /usr/lib/systemd/system. Package upgrades overwrite it, so you never edit it.
vendor preset
The package author's suggestion of whether a unit should be enabled at boot. Shown on the Loaded: line; only a suggestion. Lesson 2.9
volatile (volatile storage)
Kept only in memory and lost at reboot. A volatile journal forgets everything each boot. Lesson 2.30
WantedBy (WantedBy=)
The [Install] directive that says which target should pull this unit in when it is enabled; almost always multi-user.target. Lesson 2.1
watchdog (WatchdogSec, WatchdogSec=)
A "still alive?" check: the program must report in regularly, and if it goes quiet systemd kills and restarts it. Lesson 2.9