03:12, the primary is down and every service that writes to it fails. Nobody deployed. The cluster list says it plainly:
$ pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
18 main 5432 down postgres /var/lib/postgresql/18/main /var/log/postgresql/postgresql-18-main.logRead the server log first
The service manager only knows the process stopped. PostgreSQL's own log knows why:
$ sudo tail -n 13 /var/log/postgresql/postgresql-18-main.log
2026-09-22 20:00:14.600 UTC [17391] PANIC: could not write to file "pg_wal/xlogtemp.17391": No space left on device
2026-09-22 20:00:14.600 UTC [17385] LOG: WAL writer process (PID 17391) was terminated by signal 6: Aborted
2026-09-22 20:00:14.600 UTC [17385] LOG: terminating any other active server processes
2026-09-22 20:00:14.600 UTC [17385] LOG: all server processes terminated; reinitializing
2026-09-22 20:00:14.600 UTC [17417] LOG: database system was interrupted; last known up at 2026-09-22 20:00:03 UTC
2026-09-22 20:00:14.600 UTC [17417] LOG: database system was not properly shut down; automatic recovery in progress
2026-09-22 20:00:14.600 UTC [17417] LOG: redo starts at 0/7AE33B0
2026-09-22 20:00:14.600 UTC [17417] LOG: redo done at 0/7AE53B0 system usage: CPU: user: 0.02 s, system: 0.01 s, elapsed: 0.04 s
2026-09-22 20:00:14.600 UTC [17417] LOG: checkpoint starting: end-of-recovery fast wait
2026-09-22 20:00:14.600 UTC [17417] PANIC: could not write to file "pg_wal/xlogtemp.17417": No space left on device
2026-09-22 20:00:14.600 UTC [17385] LOG: startup process (PID 17417) was terminated by signal 6: Aborted
2026-09-22 20:00:14.600 UTC [17385] LOG: aborting startup due to startup process failure
2026-09-22 20:00:14.600 UTC [17385] LOG: database system is shut downTwo PANICs. The first is the crash: PostgreSQL cannot continue if it cannot write WAL, the log every change goes to before it touches a table. The second is why it stays down: crash recovery ends with a checkpoint, which needs to write WAL too. With a full disk, restarting it in a loop changes nothing.
$ df -h /var/lib/postgresql
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 19G 18G 8.1M 100% /Free space, but never in pg_wal
The WAL directory looks like a pile of old log files. It is not. Those segments are what crash recovery replays and what replicas and backups still need. Deleting them by hand, or running pg_resetwal, corrupts the database or throws away committed transactions. Find space outside the data directory:
$ sudo du -sh /var/lib/postgresql/backups /var/lib/postgresql/18/main/pg_wal
9.4G /var/lib/postgresql/backups
97M /var/lib/postgresql/18/main/pg_wal
$ sudo ls -lh /var/lib/postgresql/backups
total 9.4G
-rw-r----- 1 postgres postgres 9.4G Sep 22 20:00 base-2026-08-01.tar.gzA base backup from August that newer backups replaced, on the same disk as the database it protects. Remove it (or move it), then start the server:
$ sudo rm /var/lib/postgresql/backups/base-2026-08-01.tar.gz
$ sudo systemctl start postgresql@18-main
$ pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
18 main 5432 online postgres /var/lib/postgresql/18/main /var/log/postgresql/postgresql-18-main.logNo space anywhere else? Grow the volume (lvextend -r, or a bigger cloud disk), or move pg_wal to another disk with a symlink while the server is stopped. Still never delete inside it. The same "find the space, then find what holds it" order works for any No space left on device, and a deleted file that a process keeps open is the other classic.
Why the WAL grew: the slot
The server is up, but the cause is still there. PostgreSQL normally recycles WAL after each checkpoint. Something told it to keep more. The usual suspects are a failing archive_command, a huge wal_keep_size, and replication slots:
$ sudo -u postgres psql -c "select slot_name, slot_type, active, wal_status, restart_lsn, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as retained from pg_replication_slots"
slot_name | slot_type | active | wal_status | restart_lsn | retained
---------------+-----------+--------+------------+-------------+----------
analytics_cdc | physical | f | reserved | 0/296AFF0 | 81 MB
(1 row)A slot is a promise: "keep every WAL segment from restart_lsn on, until my consumer has read it". That is what lets a replica or a CDC tool (Debezium, a logical decoding client) disconnect and catch up later without losing anything. active = f means nobody is reading. Here the analytics change-data-capture job stopped weeks ago, and the primary has kept every byte of WAL since. In this small lab it is 81 MB; on a busy primary the same query shows hundreds of gigabytes, growing with every write until the disk is gone.
Drop it, then cap every slot
Check with the owner that the consumer is really gone (a dropped slot cannot be recreated at the same position; the consumer has to start over from a fresh snapshot). Then:
$ sudo -u postgres psql -c "select pg_drop_replication_slot('analytics_cdc')"
pg_drop_replication_slot
--------------------------
(1 row)
$ sudo -u postgres psql -c "alter system set max_slot_wal_keep_size = '10GB'" -c "select pg_reload_conf()"
ALTER SYSTEM
pg_reload_conf
----------------
t
(1 row)
$ sudo -u postgres psql -c "checkpoint"
CHECKPOINT
$ df -h /var/lib/postgresql
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 19G 7.8G 9.4G 46% /The checkpoint is what releases the old segments; dropping the slot only allows it. max_slot_wal_keep_size (PostgreSQL 13+, default -1 = unlimited) is the cap: a slot that falls further behind is invalidated (wal_status = lost) instead of taking the primary down. That trades a broken replica or CDC pipeline for a running database, which is the right trade for a primary.
The alerts that would have caught it
- Inactive slots:
select slot_name from pg_replication_slots where not activereturning anything for more than a few minutes. - Retained WAL per slot, from the query above, against a threshold well below your free space.
- Disk trend, not disk level: "90% full" fired at 3 a.m. here, minutes before the end. A
predict_linearrule on the filesystem metric fires hours earlier; the same idea as alerting on what users feel rather than on a number that is already too late.
When the outage is slots of a different kind, connection slots, see sorry, too many clients already.
Practise it
Incident: disk full on the primary is this outage: the PANIC, the old backup, the slot, and a runbook rule that nothing in the data directory is deleted by hand. Drop a stale replication slot is the calm version, with a healthy replica next to the dead slot so you have to drop the right one.