OnCallReady

Lesson 2.30 · systemd · 14 min read

journald: priorities, boots and persistence

In plain words

Imagine a ship's logbook where every entry is stamped with the time, who wrote it, how serious it is, and which voyage it belongs to. You can ask the logbook "show me only the captain's urgent notes from the voyage before this one", instead of reading every page.

journald is that logbook, and journalctl is how you ask it. -u nginx filters by who wrote it, -p err by how serious (errors and worse), -b -1 by voyage (the previous boot). The catch: if the ship keeps its logbook in pencil on a whiteboard, it is wiped every time the ship docks. That is a volatile journal, kept in memory under /run/log/journal. Creating /var/log/journal switches to ink on disk, so the reason for the last reboot survives.

Why this matters

The machine rebooted at 04:12 and nobody knows why. The answer is in the log of the boot before this one - if the log survived. This lesson is how to find things in the journal fast, and how to make sure it remembers across reboots.

What you need to know already: 2.5 (journalctl -u -f), 2.9 (reading status output).

The journal is structured, not a text file

The journal is kept by the systemd-journald daemon. Every entry is a record with named fields: the message, a priority (how serious), a timestamp, the unit, the PID, the UID (user ID number of the process's user), the boot ID (a random ID for each boot), the program. journalctl is a search tool over those fields, not just a way to print a text file.

journalctl -u nginx                 one unit
journalctl -u nginx -f              follow it
journalctl -u nginx -n 100          last 100 lines
journalctl -u nginx --since "15 min ago" --until "5 min ago"
journalctl -p err -b                errors and worse, this boot
journalctl -k                       kernel messages only (same as dmesg)
journalctl -g 'timed out'           search message text for 'timed out'
journalctl _PID=1234                any process, by field
journalctl _SYSTEMD_UNIT=ssh.service _UID=0
journalctl -o json-pretty           every field, for scripting
journalctl -o cat                   just the messages, no metadata

Flags in that list: -n number of lines, --since/--until a time window (plain English like "15 min ago" works), -p priority, -b this boot, -k kernel (the kernel's own messages, which the dmesg command also shows), -g search ("grep") the message text, FIELD=value match a field exactly, -o output format (json-pretty shows every field of each record; cat only the message).

A normal output line reads: Sep 22 17:43:01 oncall-lab sshd[700]: Server listening on 0.0.0.0 port 22. - timestamp, hostname, program name with its PID in brackets, message.

The eight priorities

Each record has a priority (also called log level), from most to least serious:

0 emerg   1 alert   2 crit   3 err
4 warning 5 notice  6 info   7 debug

-p err means 0 through 3, not "only err" - priority filters are "this level and worse". -p warning on a noisy box is usually the right first look.

-x adds catalog text: for well-known messages, a paragraph explaining what the message means and what to do. -e jumps to the end. Try journalctl -xe after a failed unit.

Boots

journalctl --list-boots
journalctl -b        this boot
journalctl -b -1     the previous boot

--list-boots prints one row per boot the journal remembers: an index (0 = this boot, -1 = the one before, ...), the boot ID, and the first and last timestamp. -b -1 is how you find out why a machine rebooted: whatever happened is in the previous boot's log; the current boot's log starts after the fact.

...unless the journal is volatile

$ ls /var/log/journal
ls: cannot access '/var/log/journal': No such file or directory

On your VM: Ubuntu Server ships /var/log/journal, so your real journal is persistent from day one. The lab box starts volatile on purpose (simulator), so you see the failure mode and fix it once - on other distros and minimal images it is real.

If that directory does not exist, journald keeps everything in /run/log/journal - which lives on tmpfs, a filesystem held only in RAM. Every reboot wipes it. This is called volatile storage (lost at power off), as opposed to persistent storage (kept on disk). journalctl -b -1 finds nothing, and the reason your machine rebooted is gone forever.

Storage= in /etc/systemd/journald.conf (journald's own settings file):

auto        (default) persistent IF /var/log/journal exists, else volatile
persistent  create the directory and always persist
volatile    memory only
none        discard everything

Ubuntu ships auto, and a minimal image may not have the directory. Making it persistent is one command:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald     # or: sudo journalctl --flush

Keeping it from eating the disk

journalctl --disk-usage
sudo journalctl --vacuum-time=7d
sudo journalctl --vacuum-size=200M

--vacuum-time=7d deletes archived journal files older than 7 days; --vacuum-size=200M deletes the oldest until the total is under 200M. Or set SystemMaxUse= in journald.conf. By default the journal takes up to 10% of the filesystem, capped at 4G - which on a small disk is exactly the kind of thing you discover during an incident.

What you can now do

Why it helps

The most important question after an unexpected reboot is "what happened just before it?", and the answer is journalctl -b -1 - which only works if the journal is kept on disk. Finding that out during an incident, when the evidence is already gone, is a classic postmortem finding (Chapter 0), and making the journal persistent is a standard setup step on new servers.

Priorities and field filters turn a wall of logs into an answer: -p warning -b for what went wrong this boot, -k for the kernel's own messages (disk errors, programs killed for memory), _PID= for one process. And knowing the disk-usage and clean-up options stops the journal from quietly filling a small disk.

Commands in this lesson

ls

FAQ

Does -p err show only errors?

No: a priority filter includes the level you name and everything more serious. -p err shows priorities 0 to 3: emerg, alert, crit and err. -p warning adds warnings (4). To get exactly one level or a band, use a range: -p warning..warning or -p notice..warning. And remember that whatever a service prints is logged at info (6) unless the program marks it otherwise.

How do I know if my journal is persistent?

Check whether /var/log/journal exists, and whether journalctl --list-boots shows more than the current boot. journalctl --disk-usage and journald's own start-up message ("System Journal (/var/log/journal/...)" versus "Runtime Journal (/run/log/journal/...)") tell you too. The Storage= setting in /etc/systemd/journald.conf sets the policy.

Does Ubuntu still write /var/log/syslog?

Yes. On Ubuntu Server, rsyslog is still installed and receives a copy of everything from journald, writing classic text files like /var/log/syslog and /var/log/auth.log (logins and sudo). So many logs exist twice. The journal has more detail and better filtering; the text files suit tools that expect plain files. Very small installations may have neither.

What is the difference between journalctl -k and dmesg?

Both show kernel messages. dmesg reads the kernel's own small memory buffer directly: old messages fall out when it fills, it is wiped at reboot, and on Ubuntu it needs sudo. journalctl -k shows the kernel messages journald collected, with proper timestamps, per boot, and kept across reboots if the journal is persistent. journalctl -k -b -1 gives the previous boot's kernel log, which dmesg cannot.

How much disk can the journal use?

By default up to 10% of the disk it lives on, at most 4G, and it always leaves at least 15% of that disk free. On a small disk that can still be a lot. journalctl --disk-usage shows current use; --vacuum-time=7d or --vacuum-size=200M deletes old archived entries now; SystemMaxUse= in journald.conf sets a permanent cap.

In an interview Junior

A server rebooted unexpectedly overnight. How do you find out why?

The answer is in the log of the boot before this one; the current boot's log starts after the fact.

  1. journalctl --list-boots - is the previous boot there at all? Index 0 is this boot, -1 the one before.
  2. journalctl -b -1 -e - jump to the end of the previous boot: the last lines before it went down.
  3. journalctl -b -1 -p err - errors and worse (0-3) from that boot, and journalctl -b -1 -k for the kernel's messages.

The trap: if /var/log/journal does not exist, the journal is volatile - kept in /run/log/journal in RAM - and every reboot wipes it, so -b -1 finds nothing and the reason is gone. That is why I make it persistent with sudo mkdir -p /var/log/journal and sudo journalctl --flush (or Storage=persistent), and keep its size in check with --vacuum-time=7d or SystemMaxUse=.

Also asked: Which journalctl options do you use most, and for what? · What does journalctl -p err include, and what does it miss? · How do you stop the journal from filling the disk?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.