OnCallReady

Lesson 11.26 · Docker Runtime & Networking · 13 min read

Logging drivers, and the log file that fills the disk

In plain words

Imagine a parrot in a room that repeats everything it hears into a notebook, and nobody ever tears out old pages. It works fine for a while, then one day the notebook fills the whole room and nobody can get in.

That is Docker's default json-file logging driver. Everything PID 1 prints to stdout and stderr is appended to one file on the host, /var/lib/docker/containers/<id>/<id>-json.log, and by default it is never rotated. docker logs reads from that file. The fix is rotation: max-size and max-file per container, or as the default in /etc/docker/daemon.json, or the local driver which rotates by itself.

Why this matters

A classic 3am page: the root disk of a Docker host is at 100%. Someone runs Docker's own disk report and it "looks fine". The culprit is a container's log file, which by default grows forever and which that report does not count.

What you need to know already: df and du for finding space (4.19); deleted-but-open files (4.21); log rotation and journald limits (2.30); JSON and jq (7.11); restarting a systemd service (2.5).

Where docker logs come from

Docker hands a container's output to a logging driver - the plug-in that decides where log lines go. The default, json-file, writes every line PID 1 prints to a file on the host, one JSON object per line:

$ docker inspect -f '{{.LogPath}}' web
/var/lib/docker/containers/3f4e1a2b.../3f4e1a2b...-json.log
$ sudo tail -2 "$(docker inspect -f '{{.LogPath}}' web)"
{"log":"2026/09/23 10:12:01 [notice] 1#1: start worker processes\n","stream":"stderr","time":"2026-09-23T10:12:01.458123000Z"}
{"log":"172.17.0.1 - - [23/Sep/2026:10:12:09 +0000] \"GET / HTTP/1.1\" 200 615\n","stream":"stdout","time":"2026-09-23T10:12:09.001234000Z"}

Each object: log = the line, stream = stdout or stderr, time = when. docker logs reads that file. By default it is never rotated (rotation = start a new file at a size limit and delete the oldest, like journald's limits in 2.30). A chatty container writes until the disk is full.

# on a box where one container has logged for weeks (the glob has to expand as root, hence sh -c)
sudo sh -c 'du -sh /var/lib/docker/containers/*' | sort -h | tail -3
12K	/var/lib/docker/containers/8a7b...
640K	/var/lib/docker/containers/3f4e...
9.1G	/var/lib/docker/containers/c2d4...

(df says / is full; du -sh sizes each directory; sort -h sorts human sizes. The directory name is the container's full ID.)

Rotation: per container, or as the default

docker run --log-opt max-size=10m --log-opt max-file=3 ...

--log-opt passes an option to the driver: keep at most three files of 10MB (-json.log, -json.log.1, -json.log.2). For every container on the host, set it as the daemon's default in /etc/docker/daemon.json, dockerd's config file:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
sudo systemctl restart docker

dockerd reads the file only when it starts, hence the restart. Two traps:

The local driver ("log-driver": "local") is a good alternative default: compressed, rotated automatically, and docker logs still works.

Other drivers

journald      lines go to the systemd journal (journalctl CONTAINER_NAME=web)
syslog/gelf/fluentd/awslogs/splunk   ship to a central log system
none          nothing is kept

With none (and some shipping drivers), docker logs fails with configured logging driver does not support reading.

Disk usage in general

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          14        6         4.21GB    2.64GB (62%)
Containers      9         4         118MB     22.1MB (18%)
Local Volumes   5         2         1.04GB    312MB (30%)
Build Cache     61        0         1.83GB    1.83GB

docker system df is Docker's own disk report. Columns: TOTAL how many, ACTIVE in use by a container, SIZE on disk, RECLAIMABLE what cleanup could free. Note what is not there: container log files. That is why a log-filled disk surprises people who "checked Docker".

Cleanup, from gentle to brutal (prune = delete everything unused of that kind):

docker container prune           stopped containers
docker image prune               dangling images: untagged leftovers, <none>:<none>
docker image prune -a            every image not used by a container
docker builder prune             the build cache
docker volume prune              unused anonymous volumes (-a: named too - data!)
docker system prune              containers + networks + dangling images + build cache
docker system prune -a --volumes the lot

None of these shrink a running container's log. Rotation does. (Deleting the log file with rm does not free space either - dockerd keeps it open, the deleted-but-open trap from 4.21.)

What you can now do

Why it helps

A disk at 100% on a Docker host is a classic page, and a runaway container log is one of the top causes. docker system df does not count log files, so someone "checked Docker" and found nothing. You will know to run du on /var/lib/docker/containers and find the 9GB file named after a container ID.

The daemon.json traps are the ones that cause second incidents: rotation settings only apply to containers created after the daemon restart, and a trailing comma in daemon.json keeps dockerd from starting, which takes every container on the host down. Reviewing a host setup, checking for log rotation is a quick win. And it is the same lesson as journald's size limits in chapter 2: any log without a cap eventually becomes a disk-full page.

Commands in this lesson

docker tail

FAQ

Why is my disk full although docker system df looks fine?

docker system df counts images, containers' writable layers, volumes and build cache, but not the containers' log files. With the default json-file driver and no rotation, one chatty container can write gigabytes to /var/lib/docker/containers/<id>/<id>-json.log. Find it with sudo sh -c 'du -sh /var/lib/docker/containers/*' | sort -h | tail; the directory name is the container ID.

I set log rotation in daemon.json. Why is the big container still not rotating?

Logging options are fixed when a container is created. The daemon default in /etc/docker/daemon.json only applies to containers created after dockerd restarts with the new config. Existing containers keep their old settings, which you can see with docker inspect -f '{{json .HostConfig.LogConfig}}' <c>. The chatty container has to be recreated, with docker rm and docker run, or a Compose up that changes it.

Can I just delete the big log file?

Deleting it with rm while the container runs does not free space: dockerd still holds the file open, which is the deleted-but-open case from chapter 4. Truncating it, sudo truncate -s 0 <file>, frees the space immediately and is a common emergency move, but you lose that history and it will grow again. The real fix is rotation plus recreating the container.

Which logging driver should I use?

On a single host, json-file with max-size and max-file, or the local driver, which is compressed and rotated by default, are both good defaults, and docker logs works with both. journald fits hosts where everything already goes to the journal. Shipping drivers like fluentd, gelf, awslogs or splunk send logs to a central system, but some make docker logs fail unless dual logging is enabled. Many platforms keep a local driver and let an agent ship the files.

Why should an app log to stdout instead of a file inside the container?

Because only PID 1's stdout and stderr reach the logging driver. A file inside the container lands in the writable layer: docker logs is empty, the disk use is hidden from the usual places, rotation is your problem, and the file is deleted with the container, just when you need it for a post-mortem. Logging to stdout lets the runtime store, rotate and ship the lines. The nginx image links its log files to /dev/stdout and /dev/stderr for this reason.

In an interview Junior

A Docker host's root disk is full. How do you find and fix the cause?

The usual culprit is a container's log. The default json-file logging driver writes everything PID 1 prints to /var/lib/docker/containers/<id>/<id>-json.log, and by default never rotates it. docker system df does not count log files, which is why it "looks fine".

  1. df -h to confirm, then sudo sh -c 'du -sh /var/lib/docker/containers/*' | sort -h | tail -3 - the directory name is the container ID.
  2. Set rotation as the default in /etc/docker/daemon.json ("log-opts": {"max-size": "10m", "max-file": "3"}), validate it with jq . (bad JSON stops dockerd), sudo systemctl restart docker.
  3. Recreate the chatty container - existing containers keep the log settings they were created with.

Do not rm the log file: dockerd keeps it open, so the space is not freed. Then docker image prune / container prune for the rest.

Also asked: Where does docker logs get its data from? · What does docker system df show, and what does it miss? · What is a logging driver?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.