Istio 503s explained: Envoy response flags UH, UF, NR, URX and friends
Every request through an Istio sidecar is logged with a flag saying who failed and why. How to read UH, UF, URX, NR, NC and UT, and the command for each.
Production gotchas in Linux, Kubernetes and the cloud: what the output means, why it happens and the fix, in a few minutes each. And every one is a lab you can run in a real terminal.
RSS feed58 posts · free, no ads, no tracking
Every request through an Istio sidecar is logged with a flag saying who failed and why. How to read UH, UF, URX, NR, NC and UT, and the command for each.
Policy check failed, must have required property 'owner', missing or empty kind, Unknown field: what each Backstage catalog error means and how to fix it.
Eight bpftrace one-liners for who is busy, which open fails, what keeps starting, how big the reads are and where the CPU goes, with their real output.
A CPU limit below the service's peak shows up as tail latency, not as high CPU. Prove it with cpu.max and nr_throttled, then raise or drop the quota.
An XR stuck at SYNCED False or READY False: trace the tree, read the first unhealthy object, then fix the Composition, the patch or the credentials.
Patch and transform in a Crossplane v2 pipeline: patch types, map and string transforms, readinessChecks, and testing it all with composition render.
vmstat says wa 37%. Read iostat -x the right way (await and aqu-sz, not just %util), name the process with pidstat -d, then cap it with io.max.
Lag that only grows means the group is not committing. Read kafka-consumer-groups --describe and --state, spot a rebalance storm, and fix max.poll.records.
Producers with acks=all get NotEnoughReplicasException when the ISR is below min.insync.replicas. Find the missing brokers and fix them, not the setting.
kafka-consumer-groups --reset-offsets --to-datetime: stop the group, mind the milliseconds and UTC, dry-run, export a backup, then execute, and undo it.
The Certified Kubernetes Security Specialist in 2026: format, the six domains and their weights, the exam environment, and a typical task for each domain.
Every Secret read fails with InternalError after a key rotation: what the error means, how to get the old key back, and the rotation order that avoids it.
Both scan Kubernetes YAML before it reaches the cluster, differently. One manifest through both, the score traps (a pass that exits 2), and the fixes.
Kyverno 1.19 deprecates ClusterPolicy and plans to remove it in 1.20. How the CEL policy types map, one rule converted, and a rollout that breaks nothing.
curl error 60: no chain to a trusted root. Either the server skips its intermediate or your box lacks the root. Tell which, then fix nginx, the OS or Java.
From an AWS ALB: 502 = the target refused or broke the connection, 503 = no target, 504 = no answer in time. Real outputs for each, and the fixes.
The end of an AWS AccessDenied message names the policy type that said no. Five real S3 errors decoded, from a missing ListBucket to a KMS key and an SCP.
apt update hangs, Session Manager says TargetNotConnected. Read the private route table, spot the blackhole, rebuild the NAT gateway in a public subnet.
pg_wal filled the disk and PostgreSQL PANICked. Free space safely, find the slot that pins the WAL, drop it, and cap it with max_slot_wal_keep_size.
A metadata-only ALTER TABLE waits for its lock behind one long transaction, and every query queues behind it. See the queue, cancel it, use lock_timeout.
max_connections is a budget that every pod's pool spends. How to get in when it is gone, find who took it, stop the bleeding without a restart, and pool.
A playbook that changes nothing says changed=3 and restarts nginx every run. Why command and shell always report changed, how handlers fire, and the fix.
ansible-vault encrypts the file, not the output. How debug, verbose results and diffs leak a secret into CI logs, what no_log hides, and the cleanup order.
UNREACHABLE means Ansible never got an SSH session; Missing sudo password means it got in and become failed. Six real causes, their exact errors and fixes.
The Certificate never becomes Ready and the issuer says it is fine. Follow CertificateRequest, Order and Challenge to the HTTP-01 self-check that fails.
You applied an HTTPRoute, the Gateway returns 404 and kubectl get looks fine. The answer is in the route's status conditions: read each reason, fix it.
An incident commander does not debug. A SEV2 from four people changing production at once to a clean handoff, and what the IC says at each step.
Who is affected and how, what you are doing, and when the next update comes - with templates for customers, executives and support.
After a reboot every app that needs a secret fails with 503 Vault is sealed. Why Vault restarts sealed, how to unseal with key shares, and auto-unseal.
The policy says kv/reports/* and the token still gets a 403. KV v2 reads at kv/data/ and lists at kv/metadata/: read the URL, test the token, fix it.
The app renewed its Vault token and still died at 3 a.m.: renewal stops at the max TTL and the leases die with it. How to read it, and the Vault Agent fix.
Synced, then OutOfSync again seconds later - something rewrites a field git declares. Find it with argocd app diff, then fix git or ignore narrowly.
rm removes a name, not the data. Why df and du disagree after deleting an open file, how lsof +L1 finds it, and how to get the space back without a reboot.
When every app fails name lookups at the same minute, one shared resolver is down. Find it with resolv.conf, getent vs dig, and one query per hop.
A container stuck in Restarting (1) is a crash loop. Read the exit code, the restart count and each attempt's logs, then fix the startup, not the policy.
A full Docker host is usually container logs, which docker system df does not count. Find the space with df and du, prune safely, and set log rotation.
137 = 128 + SIGKILL. How to tell a cgroup OOM kill from a host OOM, why -Xmx equal to the memory limit gets your Java pod killed, and how to size it.
Linux load average counts processes waiting on disk or NFS, not just CPU. Spot D-state processes, why kill -9 cannot touch them, and what to fix instead.
A kubeadm cluster a year old and never upgraded stops answering kubectl. Check expiry, renew, restart the static pods, refresh your kubeconfig.
The kubelet refuses to run with swap on. Read it from systemctl and journalctl, why swapoff alone comes back after a reboot, and the full fix.
CrashLoopBackOff is a symptom, not a cause. Read Last State and the exit code, get the previous container's logs, check probes and events: a checklist.
New pods stuck in CreateContainerConfigError or ContainerCreating after a config cleanup? Read the Events, find the missing ConfigMap or key, fix it.
SIGTERM, the grace period, SIGKILL - and endpoint removal at the same time. Why pods drop requests on deploy, and how preStop and graceful shutdown fix it.
"0/3 nodes are available" is a per-node tally. Decode each reason - Insufficient cpu, untolerated taint, node affinity, unbound PVC - and fix that one.
A deleted PVC that stays Terminating is protected, not broken. Find the pod that still mounts it, and why removing the finalizer is the wrong fix.
20-40 seconds of 503s on every rollout of a two-second app. Check the strategy, then readiness, then shutdown - and prove zero downtime with one rollout.
Healthy pods, a Service, and Connection refused. Check the selector (empty endpoints), readiness (ready false) and targetPort - in that order.
"502 Bad Gateway" over nginx/1.26.3 (Ubuntu) means nginx could not reach your app. Read error.log, tell refused from timed out, and fix the right link.
ENOSPC with a half-empty disk: check inodes (df -i), then deleted-but-open files (lsof +L1), then ext4 reserved blocks (tune2fs) - and how to fix each.
The grey router page means no pod got the request: wrong host or path, no ready endpoints, or a Route on the wrong port. The oc commands that tell which.
Slow requests that end in 200 are successes, and abandoned ones are never counted. How to read p99 from Prometheus buckets and find the wait behind it.
set -euo pipefail is the right header, but it has holes - pipelines, conditions, && chains and local x=$(cmd). What each one misses and how to close it.
In [Service], systemd ignores StartLimitIntervalSec: "Unknown key name ... ignoring." It belongs in [Unit]. How to spot it, and the default limit.
Removing one item from a count list re-indexes everything after it, so Terraform replaces them. Switch to for_each and move state with moved blocks.
A dev plan that renames prod resources is reading prod's state. Read the plan, find the backend key, and re-init with -reconfigure, not -migrate-state.
sensitive = true only hides values in plan output. How a prod password reaches a CI log, how to confirm it without printing it, and how to rotate it.
"Error acquiring the state lock" or "state blob is already locked": read Lock Info, prove the holder is dead, run terraform force-unlock ID, then clean up.
A zombie is a child that exited but was never reaped. Why kill -9 does nothing, how to find the parent, and why containers need a PID 1 that reaps.