The page says the root filesystem of a Docker host is full, and two containers are already failing writes (an app that cannot write often exits, and then restarts every few seconds):
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 19G 18G 0 100% /The first responder runs Docker's own disk report and finds nothing alarming:
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 3 2 206MB 0B (0%)
Containers 2 2 2B 0B (0%)
Local Volumes 0 0 0B 0B (0%)
Build Cache 1 0 0B 0B18 GB used, and Docker accounts for about 200 MB of it. Both outputs are correct.
What lives under /var/lib/docker
Docker keeps everything in its data root, /var/lib/docker by default, on the root filesystem unless you moved it:
overlay2/- image layers, plus each container's writable layer (files a container writes outside volumes).volumes/- named and anonymous volumes: your data.buildkit/- the build cache.containers/<id>/- per-container files, including<id>-json.log: everything the container printed to stdout and stderr.
docker system df reports images, containers' writable layers, volumes and build cache. It does not count those log files. With the default logging driver, json-file, they are never rotated: a chatty container writes until the disk is full.
The diagnosis path
1. Confirm which filesystem is full
df -h / (or df -h /var/lib/docker if Docker has its own mount). Also check df -i: running out of inodes gives the same "No space left on device" with free bytes - one of three ways df can show free space while writes fail.
2. Find the directory with du - as root, glob included
$ sudo du -sh /var/lib/docker/* | sort -h
du: cannot access '/var/lib/docker/*': No such file or directoryThe classic trap: /var/lib/docker is mode 0710, owned by root, so your shell cannot list it to expand the *, and du receives a literal *. Expand it as root:
$ sudo sh -c 'du -sh /var/lib/docker/* | sort -h'
$ sudo sh -c 'du -sh /var/lib/docker/containers/*' | sort -h | tail -3
12K /var/lib/docker/containers/2f19369e7c24...
10G /var/lib/docker/containers/aff0567c0d6f...The first command shows containers/ dwarfing overlay2/; the second narrows it to one container directory holding 10 GB. The directory name is the container's full ID.
3. Turn the ID into a container
$ docker inspect -f '{{.Name}} {{.LogPath}}' $(docker ps -q)
/fx-feed /var/lib/docker/containers/aff0567c0d6f.../aff0567c0d6f...-json.log
/audit /var/lib/docker/containers/2f19369e7c24.../2f19369e7c24...-json.log
$ docker inspect -f '{{json .HostConfig.LogConfig}}' fx-feed
{"Type":"json-file","Config":{}}Config: {} means no max-size: unlimited. docker logs --tail 2 fx-feed shows why it grew so fast - a debug-level logger printing every price quote:
$ docker logs --tail 2 fx-feed
{"level":"debug","msg":"quote","pair":"EURUSD","bid":1.0912}
listening on :92004. Check the rest only after that
docker system df -v lists every image, container and volume with sizes. That is where you look when the space really is images or build cache - on build agents it often is.
Freeing space safely
Pruning means deleting everything unused of one kind. From gentle to dangerous:
docker container prune stopped containers
docker image prune dangling images (<none>:<none> leftovers)
docker image prune -a every image no container uses
docker builder prune the build cache
docker system prune the four above, without -a: stopped containers, unused networks, dangling images, build cache
docker volume prune unused anonymous volumes (Docker 23+); -a adds named onesVolumes hold data. docker system prune leaves them alone unless you add --volumes, and that is the flag to think twice about. Every prune command asks for confirmation; read what it lists.
None of these shrink a running container's log. And rm on the log file does not free the space either: dockerd still has the file open, so the blocks stay allocated (sudo lsof +L1 shows it, the same trap as the 40 GB log that df still counts).
The fix: rotation, then recreate
Set rotation as the daemon default in /etc/docker/daemon.json:
$ echo '{"log-driver": "json-file", "log-opts": {"max-size": "10m", "max-file": "3"}}' | sudo tee /etc/docker/daemon.json
{"log-driver": "json-file", "log-opts": {"max-size": "10m", "max-file": "3"}}
$ jq empty /etc/docker/daemon.json && sudo systemctl restart dockermax-size and max-file are strings in this file. jq empty prints nothing for valid JSON and an error otherwise, which is why it gates the restart: a syntax error in daemon.json stops dockerd from starting, and with it every container on the host.
The default only applies to containers created after the restart. Existing containers keep the log config they were created with, so recreate the noisy one. Recreating also removes its old log file:
$ docker rm -f fx-feed && docker run -d --name fx-feed --restart unless-stopped fx-feed:2
fx-feed
7229f1a9e4f2dec46a3fdecf87cfddb9e3743934446337a36a7508345c0558be
$ docker inspect -f '{{json .HostConfig.LogConfig}}' fx-feed
{"Type":"json-file","Config":{"max-size":"10m","max-file":"3"}}
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 19G 7.9G 9.3G 46% /If you cannot recreate the container right now, sudo truncate -s 0 on its LogPath frees the space immediately. You lose that log history, and the file grows again until you fix rotation.
Keeping it from coming back
- Rotation as the daemon default on every host, in the image you build hosts from. The
locallogging driver is a good alternative: rotated and compressed by default, anddocker logsstill works. - Log level
infoin production. Debug logging to stdout is a disk-filling machine. - Docker on its own filesystem (
"data-root"in daemon.json pointing at a separate mount), so a runaway container fills Docker's disk and not the host's root. - Alert on disk usage at 80%, well before writes fail at 100%.
Practise it
Incident: the Docker host is out of disk (11.28) gives you this host at 97%, with the log to find and the rotation to set up without losing any volume data.