OnCallReadyStart learning

PostgreSQLFilesystemSRE · 5 min read

PostgreSQL disk full: an inactive replication slot kept all the WAL

pg_wal filled the disk and PostgreSQL PANICked. Free space safely, find the slot that pins the WAL, drop it, and cap it with max_slot_wal_keep_size.

03:12, the primary is down and every service that writes to it fails. Nobody deployed. The cluster list says it plainly:

terminal
$ pg_lsclusters
Ver Cluster Port Status Owner    Data directory              Log file
18  main    5432 down   postgres /var/lib/postgresql/18/main /var/log/postgresql/postgresql-18-main.log

Read the server log first

The service manager only knows the process stopped. PostgreSQL's own log knows why:

terminal
$ sudo tail -n 13 /var/log/postgresql/postgresql-18-main.log
2026-09-22 20:00:14.600 UTC [17391] PANIC:  could not write to file "pg_wal/xlogtemp.17391": No space left on device
2026-09-22 20:00:14.600 UTC [17385] LOG:  WAL writer process (PID 17391) was terminated by signal 6: Aborted
2026-09-22 20:00:14.600 UTC [17385] LOG:  terminating any other active server processes
2026-09-22 20:00:14.600 UTC [17385] LOG:  all server processes terminated; reinitializing
2026-09-22 20:00:14.600 UTC [17417] LOG:  database system was interrupted; last known up at 2026-09-22 20:00:03 UTC
2026-09-22 20:00:14.600 UTC [17417] LOG:  database system was not properly shut down; automatic recovery in progress
2026-09-22 20:00:14.600 UTC [17417] LOG:  redo starts at 0/7AE33B0
2026-09-22 20:00:14.600 UTC [17417] LOG:  redo done at 0/7AE53B0 system usage: CPU: user: 0.02 s, system: 0.01 s, elapsed: 0.04 s
2026-09-22 20:00:14.600 UTC [17417] LOG:  checkpoint starting: end-of-recovery fast wait
2026-09-22 20:00:14.600 UTC [17417] PANIC:  could not write to file "pg_wal/xlogtemp.17417": No space left on device
2026-09-22 20:00:14.600 UTC [17385] LOG:  startup process (PID 17417) was terminated by signal 6: Aborted
2026-09-22 20:00:14.600 UTC [17385] LOG:  aborting startup due to startup process failure
2026-09-22 20:00:14.600 UTC [17385] LOG:  database system is shut down

Two PANICs. The first is the crash: PostgreSQL cannot continue if it cannot write WAL, the log every change goes to before it touches a table. The second is why it stays down: crash recovery ends with a checkpoint, which needs to write WAL too. With a full disk, restarting it in a loop changes nothing.

terminal
$ df -h /var/lib/postgresql
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   19G   18G  8.1M 100% /

Free space, but never in pg_wal

The WAL directory looks like a pile of old log files. It is not. Those segments are what crash recovery replays and what replicas and backups still need. Deleting them by hand, or running pg_resetwal, corrupts the database or throws away committed transactions. Find space outside the data directory:

terminal
$ sudo du -sh /var/lib/postgresql/backups /var/lib/postgresql/18/main/pg_wal
9.4G	/var/lib/postgresql/backups
97M	/var/lib/postgresql/18/main/pg_wal
$ sudo ls -lh /var/lib/postgresql/backups
total 9.4G
-rw-r----- 1 postgres postgres 9.4G Sep 22 20:00 base-2026-08-01.tar.gz

A base backup from August that newer backups replaced, on the same disk as the database it protects. Remove it (or move it), then start the server:

terminal
$ sudo rm /var/lib/postgresql/backups/base-2026-08-01.tar.gz
$ sudo systemctl start postgresql@18-main
$ pg_lsclusters
Ver Cluster Port Status Owner    Data directory              Log file
18  main    5432 online postgres /var/lib/postgresql/18/main /var/log/postgresql/postgresql-18-main.log

No space anywhere else? Grow the volume (lvextend -r, or a bigger cloud disk), or move pg_wal to another disk with a symlink while the server is stopped. Still never delete inside it. The same "find the space, then find what holds it" order works for any No space left on device, and a deleted file that a process keeps open is the other classic.

Why the WAL grew: the slot

The server is up, but the cause is still there. PostgreSQL normally recycles WAL after each checkpoint. Something told it to keep more. The usual suspects are a failing archive_command, a huge wal_keep_size, and replication slots:

terminal
$ sudo -u postgres psql -c "select slot_name, slot_type, active, wal_status, restart_lsn, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as retained from pg_replication_slots"
   slot_name   | slot_type | active | wal_status | restart_lsn | retained
---------------+-----------+--------+------------+-------------+----------
 analytics_cdc | physical  | f      | reserved   | 0/296AFF0   | 81 MB
(1 row)

A slot is a promise: "keep every WAL segment from restart_lsn on, until my consumer has read it". That is what lets a replica or a CDC tool (Debezium, a logical decoding client) disconnect and catch up later without losing anything. active = f means nobody is reading. Here the analytics change-data-capture job stopped weeks ago, and the primary has kept every byte of WAL since. In this small lab it is 81 MB; on a busy primary the same query shows hundreds of gigabytes, growing with every write until the disk is gone.

Drop it, then cap every slot

Check with the owner that the consumer is really gone (a dropped slot cannot be recreated at the same position; the consumer has to start over from a fresh snapshot). Then:

terminal
$ sudo -u postgres psql -c "select pg_drop_replication_slot('analytics_cdc')"
 pg_drop_replication_slot
--------------------------

(1 row)
$ sudo -u postgres psql -c "alter system set max_slot_wal_keep_size = '10GB'" -c "select pg_reload_conf()"
ALTER SYSTEM
 pg_reload_conf
----------------
 t
(1 row)
$ sudo -u postgres psql -c "checkpoint"
CHECKPOINT
$ df -h /var/lib/postgresql
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   19G  7.8G  9.4G  46% /

The checkpoint is what releases the old segments; dropping the slot only allows it. max_slot_wal_keep_size (PostgreSQL 13+, default -1 = unlimited) is the cap: a slot that falls further behind is invalidated (wal_status = lost) instead of taking the primary down. That trades a broken replica or CDC pipeline for a running database, which is the right trade for a primary.

The alerts that would have caught it

  • Inactive slots: select slot_name from pg_replication_slots where not active returning anything for more than a few minutes.
  • Retained WAL per slot, from the query above, against a threshold well below your free space.
  • Disk trend, not disk level: "90% full" fired at 3 a.m. here, minutes before the end. A predict_linear rule on the filesystem metric fires hours earlier; the same idea as alerting on what users feel rather than on a number that is already too late.

When the outage is slots of a different kind, connection slots, see sorry, too many clients already.

Practise it

Incident: disk full on the primary is this outage: the PANIC, the old backup, the slot, and a runbook rule that nothing in the data directory is deleted by hand. Drop a stale replication slot is the calm version, with a healthy replica next to the dead slot so you have to drop the right one.

OnCallReady is free, with no ads and no tracking. RSS · All posts