OnCallReady

Lesson 34.3 · Kubernetes: Ingress, Gateway API & Service Mesh · 13 min read

Reading a 404, 502, 503 and 504 from ingress-nginx

In plain words

Think of a parcel service. If the address does not exist, the parcel comes back "unknown address". If the address exists but nobody lives there any more, it comes back "no one to deliver to". If the courier rings and the door slams in their face, that is a different note again, and if they wait on the doorstep for an hour and give up, that is a fourth one.

ingress-nginx sends four different notes too: 404 (no rule matched the host and path), 503 (a rule matched but the Service has no ready pods), 502 (it reached a pod and the connection broke or spoke the wrong protocol) and 504 (it reached the pod and waited too long). This lesson matches each code to its log line and its usual cause.

One request, five places it can fail

When a user reports "the site is down", the controller has already decided who failed. Each status code points at a different layer, and the controller's log has one line that says why.

What you need to know already: the access log fields and [upstream] (16.19), the 404/502/503 incidents of 16.24-16.25, EndpointSlices and readiness (16.1, 15.14), NetworkPolicy (16.29), curl -v, -w, --resolve (9.27).

client ─> node port / LB ─> controller ─(rule?)─> endpoints? ─> connect ─> answer in time?
            unreachable:       404            503           502         504
            curl (7)/(28)

404: no rule matched

The controller has no server/location for this host + path, so its default server answers with its built-in 404 page. The access log names the upstream [upstream-default-backend]:

10.244.1.1 - - [07/Oct/2026:10:00:01 +0000] "GET /blog HTTP/1.1" 404 146 "-" "curl/8.14.1" 82 0.001 [upstream-default-backend] [] 127.0.0.1:8181 146 0.001 404 4f1c...

Check in this order: the Host header (-H 'Host: ...' or --resolve), the Ingress class (an Ingress the controller ignores is the same as no Ingress: ADDRESS empty), pathType (Exact vs Prefix, 16.19), and a regex annotation on another Ingress of the same host that changed how all its paths match.

A 404 can also come from the app: the upstream field then names a real Service ([shop-web-80]) and the last status column is the pod's own 404.

Teams often replace the plain 404 page with a branded one: --default-backend-service=ns/name in the controller args sends every unmatched request to that Service. The controller still logs [upstream-default-backend], so the diagnosis is the same.

503: matched, but nowhere to send it

W1007 10:00:05.101221       7 controller.go:1128] Service "shop/api" does not have any active Endpoint.

No ready pods behind the Service (selector, readiness probe, a rollout that never became ready), the Service does not exist, or the Ingress names a port the Service does not have. kubectl get endpointslice -l kubernetes.io/service-name=api answers it in one command.

502: reached a pod, the connection broke

2026/10/07 10:00:07 [error] 38#38: *1004 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.1.1, server: shop.lab, request: "GET /api/orders HTTP/1.1", upstream: "http://10.244.2.15:8080/api/orders", host: "shop.lab"

The pod IP is right but nothing listens on that port: the Service's targetPort is wrong, or the app crashed between the readiness check and now. Two other 502 lines you will see:

recv() failed (104: Connection reset by peer) while reading response header from upstream
SSL_do_handshake() failed (SSL: error:0A00010B:SSL routines::wrong version number) while SSL handshaking to upstream

The first: the pod accepted and then dropped the connection (an app that crashes per request, or a mesh sidecar in STRICT mTLS refusing plaintext - later in this chapter). The second: backend-protocol: HTTPS on an Ingress whose pods speak plain HTTP.

504: reached it, gave up waiting

2026/10/07 10:00:20 [error] 38#38: *1009 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.244.1.1, server: shop.lab, request: "GET /reports/export HTTP/1.1", upstream: "http://10.244.1.22:8080/reports/export", host: "shop.lab"

"while reading response header" = connected fine, the pod was too slow (longer than proxy-read-timeout). "while connecting to upstream" with a timeout = the connection never opened: a NetworkPolicy dropping the controller's packets (it is a pod like any other, 16.29) or a dead node.

Two more classics

The redirect loop. A cloud load balancer terminates TLS and sends plain HTTP to the controller; the controller sees http, the host has tls, so it answers 308 to https; the browser follows, the LB sends http again... Fix: use-forwarded-headers: "true" (trust X-Forwarded-Proto from the LB) or ssl-redirect: "false" on that Ingress.

"It works with curl from my laptop but not from the pod." Inside the cluster, call the Service directly (http://web.shop) - going out to the node port and back through the controller adds every layer above for nothing.

first command          kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=20
then, by code          404 -> rules/class   503 -> endpoints   502 -> targetPort/backend   504 -> timeout/policy

What you can now do:

Why it helps

"The site returns 502" is one of the most common tickets a platform team gets, and each code points to a different object. A 404 is a routing problem in the Ingress, a 503 is about endpoints and readiness, a 502 about the pod or the protocol, a 504 about a timeout or a slow backend. Guessing wastes the first half hour; reading the code and the controller's error log takes a minute.

The lesson also covers two classics that fool many engineers: an HTTPS backend behind an HTTP proxy, and the redirect loop when TLS ends at a load balancer in front of the controller. Recognising them on sight is the kind of thing interviewers love to ask about.

Commands in this lesson

curl

FAQ

Where is the default backend?

When no rule matches, the controller answers itself with a 404 page (or sends the request to a custom default backend set with --default-backend-service). The access log shows the upstream as upstream-default-backend or the custom one, which is how you tell "no rule matched" from "the app said 404".

Why 503 and not 502 when my pods are down?

If the Service has no ready endpoints, nginx has nowhere to connect, so it answers 503 Service Temporarily Unavailable without trying. A 502 means it did pick a pod and the connection failed or broke: the pod is crashing, the port is wrong, or the protocol does not match.

How do I find the log line for one failing request?

kubectl logs on the controller pod: the access log line has the status, the upstream name (namespace-service-port), the pod address it used and the time it waited. Errors such as connect() failed (111: Connection refused) or upstream timed out (110 are in the same stream, with the client and host they belong to.

What causes the ssl-redirect loop?

TLS ends at a load balancer in front of the controller, so the controller receives plain HTTP and redirects to HTTPS; the load balancer sends the next request as plain HTTP again, and so on. The fix is to make the controller trust the forwarded protocol header from that load balancer, or not redirect there.

Can a NetworkPolicy cause a 504?

Yes. If a policy in the app's namespace does not allow traffic from the controller's namespace, packets are dropped silently and nginx waits for its connect timeout, then answers 504 (or 502 after retries). Allowing ingress from the controller's pods by namespace label fixes it.

In an interview Mid

Users get a 502 from ingress-nginx, but the pods look Running. What do you check?

A 502 means nginx chose a pod and the connection broke, so I look at the controller's error log for that request first: connect() failed (111: Connection refused) means nothing listens on the targetPort, so I compare the Service's targetPort with the container port; recv() failed (104: Connection reset by peer) means the app closed the connection; an SSL handshake error suggests the backend speaks HTTPS and needs the backend-protocol annotation. I also check readiness, because Running is not Ready, and kubectl get endpointslices to see which pod addresses nginx can pick.

Also asked: What is the difference between a 503 and a 504 from an ingress controller? · How do you tell a 404 from the app from a 404 from the controller? · What causes an endless redirect loop behind a cloud load balancer?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.