Lesson 34.2 · Kubernetes: Ingress, Gateway API & Service Mesh · 17 min read

ingress-nginx in depth: the generated config, the ConfigMap, the annotations

Q: Does a config change restart the controller pod?

No. The controller regenerates nginx.conf and tells nginx to reload, which starts new worker processes with the new config while old ones finish their requests. Endpoint changes do not even need a reload: the list of backend pods is updated inside nginx through Lua, which is why /dbg backends shows pod IPs that the config file does not.

In plain words

Imagine a hotel receptionist who works from one big rule book. Managers never write in the book directly; they hand in small request slips ("guests for room 12 go to the third floor", "close the bar at midnight"), and every so often the receptionist rewrites the book from all the slips and starts using the new one without closing the desk.

ingress-nginx works like that. Your Ingress objects, their annotations and the controller's ConfigMap are the slips; the controller turns them into one generated nginx.conf and reloads nginx. In this lesson you read that generated file, change timeouts globally and per Ingress, use rewrites, redirects, canaries and allowlists, see why snippet annotations are switched off, and learn what the project's retirement in March 2026 means for you.

What the controller actually runs

An Ingress is a request; the controller turns it into nginx configuration. When a route misbehaves, read what it generated instead of guessing from the YAML.

What you need to know already: the Ingress object, pathType, IngressClass and the controller's access log (16.19), TLS on an Ingress and ssl-redirect (16.22), nginx itself as a reverse proxy (9.23), kubectl exec (15.9).

$ kubectl exec -n ingress-nginx deploy/ingress-nginx-controller -- cat /etc/nginx/nginx.conf

The file is long (most of it Lua plumbing). The parts you read:

## start server shop.lab           one server block per host in any Ingress
server {
    server_name shop.lab ;
    listen 80  ;
    listen 443  ssl;
    location /api/ {                    one location per path (Prefix /api -> "/api/")
        set $namespace      "shop";
        set $ingress_name   "shop";
        set $service_name   "api";
        proxy_read_timeout  60s;        the effective timeouts for THIS location
        set $proxy_upstream_name "shop-api-80";
        proxy_pass http://upstream_balancer;
    }
}

Two surprises in there:

There is no upstream block per Service. ingress-nginx keeps the pod IPs in Lua memory and changes them without a reload when endpoints change. /dbg backends list and /dbg backends get NAME (a debug tool in the controller image) show that live list.
$proxy_upstream_name is the [shop-api-80] field of every access log line (16.19): namespace-service-port.

nginx -T prints the same file after nginx has parsed it; nginx -t only checks it.

The controller ConfigMap: global defaults

The controller reads one ConfigMap (--configmap=ingress-nginx/ingress-nginx-controller in its args). Its keys are global defaults for every Ingress:

proxy-read-timeout      "60"     seconds nginx waits for the pod's answer (default 60)
proxy-connect-timeout   "5"      seconds to open the connection (default 5)
proxy-body-size         "1m"     the largest request body (413 above it)
ssl-redirect            "true"   308 to https for hosts with tls
allow-snippet-annotations "false"   raw nginx config in annotations: off
use-forwarded-headers   "false"  trust X-Forwarded-For from a load balancer in front

Edit it with kubectl edit cm or kubectl patch. The controller notices, regenerates nginx.conf and reloads - its log says so:

I1007 10:12:01.120113       7 event.go:377] Event(...): type: 'Normal' reason: 'UPDATE' ConfigMap ingress-nginx/ingress-nginx-controller
I1007 10:12:01.122301       7 controller.go:193] "Configuration changes detected, backend reload required"
I1007 10:12:01.310442       7 controller.go:214] "Backend successfully reloaded"

A reload is cheap but not free: long-lived connections (websockets) on old worker processes are closed after worker-shutdown-timeout.

Annotations: per-Ingress overrides

An annotation on one Ingress beats the ConfigMap for that Ingress's locations. The ones you will meet on call:

nginx.ingress.kubernetes.io/proxy-read-timeout: "15"     this route may take 15 s
nginx.ingress.kubernetes.io/proxy-connect-timeout: "3"
nginx.ingress.kubernetes.io/rewrite-target: /$2          change the path (with a regex path)
nginx.ingress.kubernetes.io/use-regex: "true"            paths are regexes (whole host!)
nginx.ingress.kubernetes.io/app-root: /store             "/" answers 302 to /store
nginx.ingress.kubernetes.io/permanent-redirect: https://new.lab/   301 for everything
nginx.ingress.kubernetes.io/ssl-redirect: "false"        keep plain http for this one
nginx.ingress.kubernetes.io/allowlist-source-range: 10.64.0.0/24   others get 403
nginx.ingress.kubernetes.io/backend-protocol: HTTPS      talk TLS to the pods

Annotation values are strings: "15", not 15 - a bare number in YAML is an integer and the API refuses it (annotations are a map of strings).

Timeouts. A request slower than proxy-read-timeout gets a 504 and an error log line upstream timed out (110: Operation timed out) while reading response header from upstream. Raise the timeout on the one Ingress that needs it (a report export), not in the ConfigMap for everyone - a long global timeout lets a slow dependency tie up every nginx worker.

Rewrites. rewrite-target only makes sense with a regex path and capture groups: path /store(/|$)(.*) with rewrite-target /$2 sends /store/static/app.js to the pod as /static/app.js. Two traps:

rewrite-target: / with no capture group rewrites every path under the prefix to / - images, scripts, the API - all get the home page.
the app still writes links like /static/app.js (no prefix) into its HTML; the browser asks for shop.lab/static/app.js, which no rule matches: 404. The app must know its prefix (a base path setting), or you need a rule for /static too.

Allowlists. allowlist-source-range (the newer name of whitelist-source-range) is checked against the client address nginx sees. Behind a NodePort with externalTrafficPolicy: Cluster that address is a node's (16.6) - so either every client is allowed or none. The fix is externalTrafficPolicy: Local on the controller Service (or PROXY protocol from a load balancer in front).

Canaries in ingress-nginx

A second Ingress for the same host and path, with canary: "true", becomes an alternative backend of the first one's location instead of a duplicate (the webhook would refuse a plain duplicate, 16.21):

metadata:
  name: shop-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"          20% of requests
    nginx.ingress.kubernetes.io/canary-by-header: X-Canary   "always" / "never" overrides the weight

The header rule wins over the weight; the weight is per request, so 20 out of 100 is "about 20". In the generated config the canary has no location of its own: $proxy_alternative_upstream_name on the main location names it.

Snippets are switched off

configuration-snippet and server-snippet let an annotation inject raw nginx config. Anyone who can create an Ingress could then read the controller's secrets (CVE-2021-25742, and the "IngressNightmare" bugs of March 2025). Since v1.9 they are disabled by default, and v1.12 added a risk level that blocks them unless annotations-risk-level: Critical:

Error from server (BadRequest): error when creating "snip.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: nginx.ingress.kubernetes.io/configuration-snippet annotation cannot be used. Snippet directives are disabled by the Ingress administrator

Do not turn them back on to get one header in: there is almost always a dedicated annotation, or a better place (the app, a Gateway filter).

The retirement: what it means for you

On 11 November 2025 Kubernetes SIG Network and the Security Response Committee announced that ingress-nginx would be retired. Best-effort maintenance ran until March 2026: the last releases came out on 19 March 2026 (controller v1.15.1, with v1.14.5 and v1.13.9 for the older lines), and the repository was archived a few days later. Since then:

no new releases, no bugfixes, no security fixes - a CVE found today stays open in every cluster that runs it;
existing installs keep working, images and charts stay downloadable;
the recommended way forward is Gateway API with a maintained implementation (Envoy Gateway, NGINX Gateway Fabric, Istio, Cilium, kgateway, Traefik, a cloud provider's), helped by the ingress2gateway tool.

Do not confuse the project with the API: the Ingress resource (networking.k8s.io/v1) is GA and stays. Other controllers (Traefik, HAProxy, cloud ones, NGINX Inc.'s own "NGINX Ingress Controller", a different product) keep serving it.

This lab runs ingress-nginx v1.13.3 - realistic: most clusters you will inherit still run some 1.1x version. Your job there is to keep it alive and plan the move (later in this chapter).

In an interview: "ingress-nginx was retired in March 2026, so it gets no security fixes. I would inventory our Ingresses and annotations, run ingress2gateway to see what translates, and move host by host to a Gateway API implementation, running both side by side behind the same DNS name."

Quick check

$ kubectl get cm -n ingress-nginx ingress-nginx-controller -o jsonpath='{.data}'

What does the lab's controller ConfigMap set?

only allow-snippet-annotations "false"
proxy-read-timeout "60"
nothing, it is empty

Answer

only allow-snippet-annotations "false" - The defaults (60 s read timeout, 5 s connect, 1m body) are built in; the ConfigMap only lists overrides. This one only spells out that snippets stay disabled.

What you can now do:

read the generated nginx.conf and find the server, location and upstream for a request
change a global default in the ConfigMap, or one Ingress with an annotation, and say which wins
set up a canary Ingress, an allowlist and a rewrite without the classic traps
explain the ingress-nginx retirement and what it does (and does not) deprecate

Why it helps

Most clusters you join will still have ingress-nginx, and its behaviour is in the generated config, not in your YAML. When a 504 appears after 60 seconds, or a canary gets no traffic, cat /etc/nginx/nginx.conf inside the controller shows exactly what nginx was told.

Annotations are where most surprises live: they are free-form strings, typos are silently ignored, and some (snippets) were a real security hole. Knowing the common ones (timeouts, rewrite-target, canary, allowlist-source-range) lets you fix most edge tickets. And because the project is retired, knowing what you depend on is the inventory you need before any migration to Gateway API.

Commands in this lesson

kubectl

FAQ

Why is my annotation ignored without any error?

Annotations are plain strings that only the controller reads. A typo in the key, a missing nginx.ingress.kubernetes.io/ prefix or a value in the wrong format is accepted by the API server and then skipped. Check the generated nginx.conf for the directive, and the controller log for a warning about that Ingress.

ConfigMap or annotation: which wins?

The controller's ConfigMap sets defaults for every Ingress the controller serves; an annotation overrides that default for one Ingress. So proxy-read-timeout in the ConfigMap is the global value, and the same annotation on one Ingress changes only that Ingress's locations.

Why were snippet annotations disabled?

A snippet puts raw nginx configuration from an Ingress into the shared config. Anyone allowed to create an Ingress could then read files from the controller, including its service account token, which can read Secrets across the cluster. Recent releases disable them by default, and the admission webhook refuses Ingresses that use them.

Does a config change restart the controller pod?

No. The controller regenerates nginx.conf and tells nginx to reload, which starts new worker processes with the new config while old ones finish their requests. Endpoint changes do not even need a reload: the list of backend pods is updated inside nginx through Lua, which is why /dbg backends shows pod IPs that the config file does not.

What exactly happened with the retirement?

Kubernetes SIG Network announced on 11 November 2025 that the community ingress-nginx project would be retired, with best-effort maintenance until March 2026. After that there are no releases and no security fixes. The Ingress API itself is not removed, and other controllers still implement it, but this particular controller is finished.

In an interview Mid

ingress-nginx is retired. What would you do about it in a company that depends on it?

I would say what the retirement means first: since March 2026 the project ships no security fixes, so every new vulnerability stays open, but traffic keeps flowing. Then I would inventory: every Ingress, its annotations and the controller ConfigMap, because annotations are where the behaviour lives. I would run ingress2gateway to see what translates to Gateway API fields and what does not (snippets, some auth annotations), choose a Gateway API implementation, and move host by host, running both side by side behind the same DNS name so each cutover is a DNS change that can be rolled back.

Also asked: How do you see the nginx configuration ingress-nginx actually generated? · How does a canary Ingress choose which requests it gets? · Why are configuration snippets considered dangerous?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.