LinuxFilesystem
I deleted a 40 GB log and df didn't change. Where did the space go?
rm removes a name, not the data. Why df and du disagree after deleting an open file, how lsof +L1 finds it, and how to get the space back without a reboot.
Blog · topic
Linux
Linux on real servers: the gotchas that page you at 3 a.m. and the commands that explain them.
11 postsRSS feed
NetworkingLinuxKubernetesSRE
DNS resolution failed for every service: resolv.conf, systemd-resolved, CoreDNS
When every app fails name lookups at the same minute, one shared resolver is down. Find it with resolv.conf, getent vs dig, and one query per hop.
DockerLinuxSRE
Docker container keeps restarting: exit codes, restart policies, docker logs
A container stuck in Restarting (1) is a crash loop. Read the exit code, the restart count and the logs of every attempt, then fix the startup, not the policy.
DockerLinuxFilesystemSRE
Docker host out of disk but docker system df looks fine
A full Docker host is usually container logs, which docker system df does not count. Find the space with df and du, prune safely, and set log rotation.
LinuxProcesses
Load average 15 on 4 cores, but the CPU is 90% idle
Linux load average counts processes waiting on disk or NFS, not just CPU. How to spot D-state processes, why kill -9 cannot touch them, and what to fix instead.
KubernetessystemdLinux
Node NotReady: kubelet stuck in activating (auto-restart) because swap is on
The kubelet refuses to run with swap enabled by default. How to read it from systemctl and journalctl, why swapoff alone comes back after a reboot, and the full fix.
NetworkingLinuxDockerSRE
nginx 502 Bad Gateway but the API is up: reading error.log
A 502 is nginx saying its own connection to the upstream failed. How to read error.log, tell refused from timed out, and test from where nginx stands.
LinuxFilesystem
"No space left on device" but df shows free space: the 3 causes, in the order to check
ENOSPC with a half-empty disk. Check inodes (df -i), then deleted-but-open files (lsof +L1), then ext4 reserved blocks (tune2fs) - what each looks like and how to fix it.
BashLinux
Why set -e doesn't catch a failure in foo | bar (and the other gaps)
set -euo pipefail is the right header, but it has holes - pipelines, conditions, && chains and local x=$(cmd). What each one misses and how to close it.
systemdLinux
systemd: your StartLimitIntervalSec is in the wrong section (and silently ignored)
StartLimitIntervalSec and StartLimitBurst belong in [Unit]. In [Service] systemd ignores them with a warning you only see in systemd-analyze verify. Plus the default that never trips.
ProcessesLinuxDocker
Zombie processes (<defunct>): why kill doesn't work and what does
A zombie is a child that has exited but was never reaped by its parent. Why kill -9 does nothing, how to find the parent, and why containers need a PID 1 that reaps.