ALB 502 vs 503 vs 504 on AWS: what each one says about your targets
From an AWS ALB: 502 = the target refused or broke the connection, 503 = no target, 504 = no answer in time. Real outputs for each, and the fixes.
Incident response, SLOs and on-call habits: the practice behind the commands.
22 postsRSS feed
From an AWS ALB: 502 = the target refused or broke the connection, 503 = no target, 504 = no answer in time. Real outputs for each, and the fixes.
The end of an AWS AccessDenied message names the policy type that said no. Five real S3 errors decoded, from a missing ListBucket to a KMS key and an SCP.
pg_wal filled the disk and PostgreSQL PANICked. Free space safely, find the slot that pins the WAL, drop it, and cap it with max_slot_wal_keep_size.
A metadata-only ALTER TABLE waits for its lock behind one long transaction, and every query queues behind it. See the queue, cancel it, use lock_timeout.
max_connections is a budget that every pod's pool spends. How to get in when it is gone, find who took it, stop the bleeding without a restart, and pool.
A playbook that changes nothing says changed=3 and restarts nginx every run. Why command and shell always report changed, how handlers fire, and the fix.
An incident commander does not debug. A SEV2 from four people changing production at once to a clean handoff, and what the IC says at each step.
Who is affected and how, what you are doing, and when the next update comes - with templates for customers, executives and support.
After a reboot every app that needs a secret fails with 503 Vault is sealed. Why Vault restarts sealed, how to unseal with key shares, and auto-unseal.
The policy says kv/reports/* and the token still gets a 403. KV v2 reads at kv/data/ and lists at kv/metadata/: read the URL, test the token, fix it.
The app renewed its Vault token and still died at 3 a.m.: renewal stops at the max TTL and the leases die with it. How to read it, and the Vault Agent fix.
When every app fails name lookups at the same minute, one shared resolver is down. Find it with resolv.conf, getent vs dig, and one query per hop.
A container stuck in Restarting (1) is a crash loop. Read the exit code, the restart count and each attempt's logs, then fix the startup, not the policy.
A full Docker host is usually container logs, which docker system df does not count. Find the space with df and du, prune safely, and set log rotation.
CrashLoopBackOff is a symptom, not a cause. Read Last State and the exit code, get the previous container's logs, check probes and events: a checklist.
20-40 seconds of 503s on every rollout of a two-second app. Check the strategy, then readiness, then shutdown - and prove zero downtime with one rollout.
"502 Bad Gateway" over nginx/1.26.3 (Ubuntu) means nginx could not reach your app. Read error.log, tell refused from timed out, and fix the right link.
Slow requests that end in 200 are successes, and abandoned ones are never counted. How to read p99 from Prometheus buckets and find the wait behind it.
Removing one item from a count list re-indexes everything after it, so Terraform replaces them. Switch to for_each and move state with moved blocks.
A dev plan that renames prod resources is reading prod's state. Read the plan, find the backend key, and re-init with -reconfigure, not -migrate-state.
sensitive = true only hides values in plan output. How a prod password reaches a CI log, how to confirm it without printing it, and how to rotate it.
"Error acquiring the state lock" or "state blob is already locked": read Lock Info, prove the holder is dead, run terraform force-unlock ID, then clean up.