bpftrace one-liners that answer real production questions
Eight bpftrace one-liners for who is busy, which open fails, what keeps starting, how big the reads are and where the CPU goes, with their real output.
Linux on real servers: the gotchas that page you at 3 a.m. and the commands that explain them.
20 postsRSS feed
Eight bpftrace one-liners for who is busy, which open fails, what keeps starting, how big the reads are and where the CPU goes, with their real output.
A CPU limit below the service's peak shows up as tail latency, not as high CPU. Prove it with cpu.max and nr_throttled, then raise or drop the quota.
vmstat says wa 37%. Read iostat -x the right way (await and aqu-sz, not just %util), name the process with pidstat -d, then cap it with io.max.
curl error 60: no chain to a trusted root. Either the server skips its intermediate or your box lacks the root. Tell which, then fix nginx, the OS or Java.
apt update hangs, Session Manager says TargetNotConnected. Read the private route table, spot the blackhole, rebuild the NAT gateway in a public subnet.
A playbook that changes nothing says changed=3 and restarts nginx every run. Why command and shell always report changed, how handlers fire, and the fix.
UNREACHABLE means Ansible never got an SSH session; Missing sudo password means it got in and become failed. Six real causes, their exact errors and fixes.
After a reboot every app that needs a secret fails with 503 Vault is sealed. Why Vault restarts sealed, how to unseal with key shares, and auto-unseal.
The app renewed its Vault token and still died at 3 a.m.: renewal stops at the max TTL and the leases die with it. How to read it, and the Vault Agent fix.
rm removes a name, not the data. Why df and du disagree after deleting an open file, how lsof +L1 finds it, and how to get the space back without a reboot.
When every app fails name lookups at the same minute, one shared resolver is down. Find it with resolv.conf, getent vs dig, and one query per hop.
A container stuck in Restarting (1) is a crash loop. Read the exit code, the restart count and each attempt's logs, then fix the startup, not the policy.
A full Docker host is usually container logs, which docker system df does not count. Find the space with df and du, prune safely, and set log rotation.
Linux load average counts processes waiting on disk or NFS, not just CPU. Spot D-state processes, why kill -9 cannot touch them, and what to fix instead.
The kubelet refuses to run with swap on. Read it from systemctl and journalctl, why swapoff alone comes back after a reboot, and the full fix.
"502 Bad Gateway" over nginx/1.26.3 (Ubuntu) means nginx could not reach your app. Read error.log, tell refused from timed out, and fix the right link.
ENOSPC with a half-empty disk: check inodes (df -i), then deleted-but-open files (lsof +L1), then ext4 reserved blocks (tune2fs) - and how to fix each.
set -euo pipefail is the right header, but it has holes - pipelines, conditions, && chains and local x=$(cmd). What each one misses and how to close it.
In [Service], systemd ignores StartLimitIntervalSec: "Unknown key name ... ignoring." It belongs in [Unit]. How to spot it, and the default limit.
A zombie is a child that exited but was never reaped. Why kill -9 does nothing, how to find the parent, and why containers need a PID 1 that reaps.