https://oncallready.dev/blog/Real incidents and Linux / Kubernetes / SRE gotchas, explained.enSat, 03 Oct 2026 08:00:00 GMT https://oncallready.dev/blog/argocd-outofsync-ignoredifferences/https://oncallready.dev/blog/argocd-outofsync-ignoredifferences/Sat, 03 Oct 2026 08:00:00 GMTSynced, then OutOfSync again seconds later - something rewrites a field git declares. Find it with argocd app diff, then fix git or ignore narrowly.GitOpsKubernetes https://oncallready.dev/blog/deleted-log-df-not-freed/https://oncallready.dev/blog/deleted-log-df-not-freed/Sat, 03 Oct 2026 08:00:00 GMTrm removes a name, not the data. Why df and du disagree after deleting an open file, how lsof +L1 finds it, and how to get the space back without a reboot.LinuxFilesystem https://oncallready.dev/blog/dns-resolution-failed-systemd-resolved/https://oncallready.dev/blog/dns-resolution-failed-systemd-resolved/Sat, 03 Oct 2026 08:00:00 GMTWhen every app fails name lookups at the same minute, one shared resolver is down. Find it with resolv.conf, getent vs dig, and one query per hop.NetworkingLinuxKubernetesSRE https://oncallready.dev/blog/docker-container-restarting-exit-code/https://oncallready.dev/blog/docker-container-restarting-exit-code/Sat, 03 Oct 2026 08:00:00 GMTA container stuck in Restarting (1) is a crash loop. Read the exit code, the restart count and the logs of every attempt, then fix the startup, not the policy.DockerLinuxSRE https://oncallready.dev/blog/docker-no-space-left-on-device/https://oncallready.dev/blog/docker-no-space-left-on-device/Sat, 03 Oct 2026 08:00:00 GMTA full Docker host is usually container logs, which docker system df does not count. Find the space with df and du, prune safely, and set log rotation.DockerLinuxFilesystemSRE https://oncallready.dev/blog/exit-code-137-oomkilled-jvm/https://oncallready.dev/blog/exit-code-137-oomkilled-jvm/Sat, 03 Oct 2026 08:00:00 GMT137 = 128 + SIGKILL. How to tell a cgroup OOM kill from a host OOM, why -Xmx equal to the memory limit gets your Java pod killed, and how to size it.MemoryKubernetesJVM https://oncallready.dev/blog/high-load-idle-cpu-d-state/https://oncallready.dev/blog/high-load-idle-cpu-d-state/Sat, 03 Oct 2026 08:00:00 GMTLinux load average counts processes waiting on disk or NFS, not just CPU. How to spot D-state processes, why kill -9 cannot touch them, and what to fix instead.LinuxProcesses https://oncallready.dev/blog/kubeadm-x509-certificate-has-expired/https://oncallready.dev/blog/kubeadm-x509-certificate-has-expired/Sat, 03 Oct 2026 08:00:00 GMTA kubeadm cluster a year old and never upgraded stops answering kubectl. Check expiry, renew, restart the static pods, refresh your kubeconfig.Kubernetes https://oncallready.dev/blog/kubelet-swap-node-notready/https://oncallready.dev/blog/kubelet-swap-node-notready/Sat, 03 Oct 2026 08:00:00 GMTThe kubelet refuses to run with swap enabled by default. How to read it from systemctl and journalctl, why swapoff alone comes back after a reboot, and the full fix.KubernetessystemdLinux https://oncallready.dev/blog/kubernetes-crashloopbackoff-debug/https://oncallready.dev/blog/kubernetes-crashloopbackoff-debug/Sat, 03 Oct 2026 08:00:00 GMTCrashLoopBackOff is a symptom, not a cause. Read Last State and the exit code, get the logs of the previous container, check probes and events - a short, ordered checklist.KubernetesSRE https://oncallready.dev/blog/kubernetes-createcontainerconfigerror-missing-configmap-key/https://oncallready.dev/blog/kubernetes-createcontainerconfigerror-missing-configmap-key/Sat, 03 Oct 2026 08:00:00 GMTNew pods stuck in CreateContainerConfigError or ContainerCreating after a config cleanup? Read the Events, find the missing ConfigMap or key, fix the config.Kubernetes https://oncallready.dev/blog/kubernetes-pod-deletion-sigterm-grace-sigkill/https://oncallready.dev/blog/kubernetes-pod-deletion-sigterm-grace-sigkill/Sat, 03 Oct 2026 08:00:00 GMTSIGTERM, the grace period, SIGKILL - and the endpoint removal running at the same time. Why pods drop requests on deploy, and how preStop and graceful shutdown fix it.KubernetesProcesses https://oncallready.dev/blog/kubernetes-pod-pending-failedscheduling/https://oncallready.dev/blog/kubernetes-pod-pending-failedscheduling/Sat, 03 Oct 2026 08:00:00 GMT"0/3 nodes are available" is a per-node tally. Decode each reason - Insufficient cpu, untolerated taint, node affinity, unbound PVC - and fix the right one.Kubernetes https://oncallready.dev/blog/kubernetes-pvc-stuck-terminating-finalizer/https://oncallready.dev/blog/kubernetes-pvc-stuck-terminating-finalizer/Sat, 03 Oct 2026 08:00:00 GMTA deleted PVC that stays Terminating is protected, not broken. Find the pod that still mounts it, and why removing the finalizer is the wrong fix.Kubernetes https://oncallready.dev/blog/kubernetes-rolling-update-downtime-readiness-prestop/https://oncallready.dev/blog/kubernetes-rolling-update-downtime-readiness-prestop/Sat, 03 Oct 2026 08:00:00 GMT20-40 seconds of 503s on every rollout of a two-second app. Check the strategy, then readiness, then shutdown - and prove zero downtime with one rollout.KubernetesSRE https://oncallready.dev/blog/kubernetes-service-no-endpoints-selector/https://oncallready.dev/blog/kubernetes-service-no-endpoints-selector/Sat, 03 Oct 2026 08:00:00 GMTHealthy pods, a Service, and Connection refused. Check the selector (empty endpoints), readiness (ready false) and targetPort - in that order.KubernetesNetworking https://oncallready.dev/blog/nginx-502-bad-gateway-upstream/https://oncallready.dev/blog/nginx-502-bad-gateway-upstream/Sat, 03 Oct 2026 08:00:00 GMTA 502 is nginx saying its own connection to the upstream failed. How to read error.log, tell refused from timed out, and test from where nginx stands.NetworkingLinuxDockerSRE https://oncallready.dev/blog/no-space-left-df-shows-free/https://oncallready.dev/blog/no-space-left-df-shows-free/Sat, 03 Oct 2026 08:00:00 GMTENOSPC with a half-empty disk. Check inodes (df -i), then deleted-but-open files (lsof +L1), then ext4 reserved blocks (tune2fs) - what each looks like and how to fix it.LinuxFilesystem https://oncallready.dev/blog/openshift-application-is-not-available-503/https://oncallready.dev/blog/openshift-application-is-not-available-503/Sat, 03 Oct 2026 08:00:00 GMTThe grey router page means the router could not hand the request to a pod. Wrong host or path, no ready endpoints, or a Route pointing at the wrong port - and the commands that tell them apart.OpenShiftKubernetesNetworking https://oncallready.dev/blog/prometheus-p99-latency-histogram-quantile/https://oncallready.dev/blog/prometheus-p99-latency-histogram-quantile/Sat, 03 Oct 2026 08:00:00 GMTSlow requests that end in 200 are successes, and abandoned ones are never counted. How to read p99 from Prometheus buckets and find the wait behind it.ObservabilityJVMSRE https://oncallready.dev/blog/set-e-does-not-catch-pipe/https://oncallready.dev/blog/set-e-does-not-catch-pipe/Sat, 03 Oct 2026 08:00:00 GMTset -euo pipefail is the right header, but it has holes - pipelines, conditions, && chains and local x=$(cmd). What each one misses and how to close it.BashLinux https://oncallready.dev/blog/systemd-startlimitintervalsec-ignored/https://oncallready.dev/blog/systemd-startlimitintervalsec-ignored/Sat, 03 Oct 2026 08:00:00 GMTStartLimitIntervalSec and StartLimitBurst belong in [Unit]. In [Service] systemd ignores them with a warning you only see in systemd-analyze verify. Plus the default that never trips.systemdLinux https://oncallready.dev/blog/terraform-count-index-shift-destroy/https://oncallready.dev/blog/terraform-count-index-shift-destroy/Sat, 03 Oct 2026 08:00:00 GMTRemoving one item from a count list re-indexes everything after it, so Terraform replaces them. Switch to for_each and move state with moved blocks.TerraformAzureSRE https://oncallready.dev/blog/terraform-plan-replace-production-wrong-state/https://oncallready.dev/blog/terraform-plan-replace-production-wrong-state/Sat, 03 Oct 2026 08:00:00 GMTA dev plan that renames prod resources is reading prod's state. How to read the plan, find the backend key, and re-init with -reconfigure, not -migrate-state.TerraformAzureCI/CDSRE https://oncallready.dev/blog/terraform-secret-leaked-in-ci-log/https://oncallready.dev/blog/terraform-secret-leaked-in-ci-log/Sat, 03 Oct 2026 08:00:00 GMTsensitive = true only hides values in plan output. How a prod password reaches a CI log, how to confirm it without printing it, and how to rotate it.TerraformCI/CDAzureSRE https://oncallready.dev/blog/terraform-state-lock-force-unlock/https://oncallready.dev/blog/terraform-state-lock-force-unlock/Sat, 03 Oct 2026 08:00:00 GMTA stale Terraform state lock blocks every pipeline run. Read Lock Info, prove the holder is dead, force-unlock that ID, then import what it left behind.TerraformAzureCI/CDSRE https://oncallready.dev/blog/zombie-processes-defunct/https://oncallready.dev/blog/zombie-processes-defunct/Sat, 03 Oct 2026 08:00:00 GMTA zombie is a child that has exited but was never reaped by its parent. Why kill -9 does nothing, how to find the parent, and why containers need a PID 1 that reaps.ProcessesLinuxDocker