Lesson 11.5 · Docker Runtime & Networking · 16 min read

The debugging toolkit: logs, exec, inspect, events, stats

Q: Why does grep find nothing in docker logs when the line is there?

docker logs keeps stdout and stderr separate and replays them on your terminal the same way. A pipe only carries stdout, so anything the app wrote to stderr, like nginx's error log or most Java stack traces, skips grep and just prints to the screen. Use docker logs web 2&1 grep ... to merge the streams first.

Q: Why can't I exec bash into my container?

exec runs a binary that exists inside the image, and many images do not ship bash. Alpine and slim images usually have sh; distroless and scratch images have no shell at all. For those, attach a tool container to the target's namespaces (--network container:x, or --pid container:x), or use nsenter from the host with the PID from inspect. exec also needs the container to be running.

Q: How do you look at the logs of a container, including one that already stopped?

docker logs NAME prints what PID 1 wrote to stdout and stderr - and it works on exited containers too, so it is the first command after docker ps -a. - --tail 50 the last lines, -f follow, --since 10m a time window, -t timestamps. - Grep with docker logs web 2&1 grep ... - Docker keeps the two streams apart, and a pipe only carries stdout, so error lines (often on stderr) skip grep without 2&1. - An app that logs to a file inside the container has empty docker logs; images like nginx link their log files to /dev/stdout and /dev/stderr for that reason. The rest of the toolkit: docker inspect -f for settings and state, docker events for the timeline, docker stats for usage, docker exec into a running one, docker cp to copy a file out.

In plain words

Imagine a sealed fish tank. You cannot climb in, but you have tools: you can read the notes the fish slips under the lid (logs), put a small camera in through the feeding hole (exec), read the label on the tank that lists its size, water type and filter (inspect), check the diary of everything that happened to it (events), and watch the thermometer and water level (stats).

That is the Docker toolkit. docker logs shows what PID 1 wrote to stdout and stderr, docker exec runs a command inside a running container, docker inspect prints its full configuration and state as JSON, docker events gives a timeline, and docker stats shows live CPU, memory and PID use against the limits.

Why this matters

A container is a sealed box: no SSH into it, often no shell in it. When one misbehaves you still need answers - what did it print, what settings did it really get, what happened to it and when, how much memory is it using. Docker has one command for each question. This lesson is the toolbox.

What you need to know already: stdout, stderr and 2>&1 (6.17); pipes and grep -c (7.1); host PIDs and /proc (3.14); how 11.1 reads status and exit codes.

logs: stdout and stderr, kept apart

Every process has two output streams: stdout (normal output) and stderr (errors), as in 6.17. Docker captures what PID 1 writes to those two - nothing else. An app that logs to /var/log/app.log inside the container has empty docker logs (the official nginx image links its log files to /dev/stdout and /dev/stderr for exactly this reason).

docker logs web                     everything so far
docker logs --tail 50 web           only the last 50 lines
docker logs -f web                  follow: keep printing new lines (Ctrl+C stops following)
docker logs --since 10m web         only the last 10 minutes (or --since 2026-09-23T10:00:00)
docker logs -t web                  prefix each line with its timestamp

-f works like tail -f and journalctl -f (2.30).

The two streams stay separate on your side too, which catches people:

# web = the crash-looping nginx of the mission below
docker logs web | grep -c emerg
0
docker logs web 2>&1 | grep -c emerg
1

nginx writes its error log to stderr, and a pipe | only carries stdout - the stderr lines went straight to your screen and skipped grep. 2>&1 ("send stream 2 where stream 1 goes", 6.17) first, every time you grep container logs.

exec: run something in a running container

docker exec CONTAINER COMMAND starts an extra process inside a running container - same namespaces, same filesystem:

docker exec web nginx -t                      one command (here: nginx checks its config)
docker exec -it web sh                        an interactive shell (-i stdin, -t terminal)
docker exec -u 0 web id                       as another user (-u 0 = root)
docker exec -w /etc/nginx web ls              in another working directory (-w)
docker exec -e DEBUG=1 web env                with an extra environment variable (-e)

exec needs a running container and a program that exists in the image:

$ docker run -d --name tiny alpine:3.20 sleep 600
$ docker exec -it tiny bash
OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown

Alpine has no bash - try sh. Distroless images have neither; use the borrowed-tools and nsenter techniques from chapter 10 (10.38).

inspect --format: the scriptable view

docker inspect NAME prints everything Docker knows about a container as one big JSON document (JSON as in 7.11 with jq). -f / --format takes a Go template - a small text-template language: {{ ... }} marks a value to print, and .State.Status is a path into the JSON, like .State.Status in jq. A handful of paths cover almost every question:

docker inspect -f '{{.State.Status}} {{.State.ExitCode}} {{.State.OOMKilled}}' api
docker inspect -f '{{.RestartCount}}' api                  how often it was restarted
docker inspect -f '{{.State.Pid}}' api                     host PID of its PID 1
docker inspect -f '{{json .Config.Env}}' api               the environment it really got
docker inspect -f '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}}' api    its limits
docker inspect -f '{{json .Mounts}}' api                   volumes and bind mounts
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' api
docker inspect -f '{{json .State.Health}}' api             healthcheck results

Three template words: json prints any value as JSON; range loops over a list or map (like .[] in jq); index reaches into maps with awkward keys ({{index .Config.Labels "com.docker.compose.service"}}). You can always fall back to docker inspect api | jq ....

A trap: {{.NetworkSettings.IPAddress}} is only filled for the default bridge network. On a network you created (11.15) it is empty; use the range form.

events: what happened, in order

docker events is Docker's own timeline - every create, start, die, restart, oom:

# api = the OOM-looping container from the limits mission
docker events --since 30m --until now --filter container=api
2026-09-23T10:02:11.482113000+00:00 container start 4f1e2d3c... (image=api:2, name=api)
2026-09-23T10:02:14.905210000+00:00 container oom 4f1e2d3c... (image=api:2, name=api)
2026-09-23T10:02:14.911002000+00:00 container die 4f1e2d3c... (exitCode=137, image=api:2, name=api)
2026-09-23T10:02:15.012344000+00:00 container restart 4f1e2d3c... (name=api)

Each line: timestamp, object type, event, ID, and details in brackets. oom right before die with exitCode=137 is the whole story in two lines.

--since 30m start 30 minutes ago; --until now stop at now. Without --until it keeps following forever (Ctrl+C), like -f.
--filter container=api only this container; --filter event=die only deaths.

stats and top

$ docker stats --no-stream
CONTAINER ID   NAME   CPU %    MEM USAGE / LIMIT     MEM %    NET I/O          BLOCK I/O     PIDS
4f1e2d3c4b5a   api    98.50%   498.2MiB / 512MiB     97.30%   1.2kB / 894B     4.1MB / 0B    38

docker stats is a live top for containers; --no-stream prints one snapshot and exits. Columns:

CPU % - as a percentage of one core (200% = two cores busy)
MEM USAGE / LIMIT - memory used against the container's memory limit (the host's RAM if it has none)
NET I/O - bytes received / sent over the network
BLOCK I/O - bytes read / written to disk
PIDS - number of processes and threads inside

A container pinned near its memory limit is one allocation away from exit 137.

docker top api lists its processes with host PIDs and host user names - which is how you notice that uid 1000 inside is learner outside (4.3).

diff and cp

docker diff api                    files the container changed: A added, C changed, D deleted
docker cp api:/app/logs/app.log .  copy a file out (works on stopped containers too)
docker cp fix.conf api:/etc/app/   copy a file in (not a substitute for rebuilding)

diff compares the container's writable layer (10.3) with its image.

Getting into an image that will not stay up

exec needs a running container. If it exits at start, replace what it runs:

docker run --rm -it --entrypoint sh api:2                  a shell instead of the app
docker run --rm --entrypoint cat api:2 /app/config.yml     print one file
docker run --rm -it --entrypoint sh api:2 -c 'env; ls -la /app'

--entrypoint takes a single program; its arguments go after the image name.

What you can now do

read a container's logs without losing stderr
pull any fact out of docker inspect with a template
rebuild a timeline with docker events, and watch usage with docker stats

Why it helps

When the payments container misbehaves at 3am, you need facts fast, and each tool answers one question. Is it logging? docker logs --since 10m, with 2>&1 so grep sees stderr too. What environment did it really get? inspect -f '{{json .Config.Env}}', which settles "but I set that variable" arguments. Why did it restart? docker events showing oom then die exitCode=137 is the whole incident in two lines. Is it about to die? docker stats at 97% of its memory limit.

The inspect templates also make you useful in automation: scripts and CI checks that read one field instead of parsing JSON by eye. And the questions themselves - what did it print, what config did it get, what happened when - are the ones you ask of any service, the same as journalctl and systemctl show for a unit.

Commands in this lesson

docker

FAQ

Why is docker logs empty when the app is clearly logging?

Docker only captures what PID 1 writes to stdout and stderr. If the app writes to a file such as /var/log/app.log inside the container, that goes to the writable layer and never reaches docker logs. Configure the app to log to stdout, or symlink its log files to /dev/stdout and /dev/stderr like the official nginx image does. Also check that you are not missing stderr in a pipe.

Why does grep find nothing in docker logs when the line is there?

docker logs keeps stdout and stderr separate and replays them on your terminal the same way. A pipe only carries stdout, so anything the app wrote to stderr, like nginx's error log or most Java stack traces, skips grep and just prints to the screen. Use docker logs web 2>&1 | grep ... to merge the streams first.

Why can't I exec bash into my container?

exec runs a binary that exists inside the image, and many images do not ship bash. Alpine and slim images usually have sh; distroless and scratch images have no shell at all. For those, attach a tool container to the target's namespaces (--network container:x, or --pid container:x), or use nsenter from the host with the PID from inspect. exec also needs the container to be running.

Why is .NetworkSettings.IPAddress empty in docker inspect?

That top-level field is only filled for the default bridge network. A container on a user-defined network, which includes every Compose project, has its addresses under .NetworkSettings.Networks.<name>.IPAddress, one entry per network it is attached to. Use the range form: docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' api.

Is docker cp a good way to fix a config in a container?

Only as a temporary emergency measure. The copied file lives in that container's writable layer: the next recreate, whether a new docker run, a Compose change or a new deploy, starts from the image again and your fix is gone. It also leaves the running container different from what is in version control. The real fix is a rebuilt image or a mounted config. docker cp out, however, is very handy for grabbing logs or dumps, even from stopped containers.

In an interview Junior

How do you look at the logs of a container, including one that already stopped?

docker logs NAME prints what PID 1 wrote to stdout and stderr - and it works on exited containers too, so it is the first command after docker ps -a.

--tail 50 the last lines, -f follow, --since 10m a time window, -t timestamps.
Grep with docker logs web 2>&1 | grep ... - Docker keeps the two streams apart, and a pipe only carries stdout, so error lines (often on stderr) skip grep without 2>&1.
An app that logs to a file inside the container has empty docker logs; images like nginx link their log files to /dev/stdout and /dev/stderr for that reason.

The rest of the toolkit: docker inspect -f for settings and state, docker events for the timeline, docker stats for usage, docker exec into a running one, docker cp to copy a file out.

Also asked: What is docker inspect good for, and which fields do you check most? · How do you get a shell in a running container? · How would you reconstruct why a container restarted overnight?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.