OnCallReady

Lesson 16.22 · Kubernetes: Networking & Storage · 14 min read

TLS termination at the Ingress

In plain words

Think of a sealed letter sent to a big company. The mailroom at the front door has the key, opens the envelope, reads which department it's for, and walks it inside. The seal protected it on the street; inside the building it travels open (unless the mailroom seals it again). The mailroom needs a proper company stamp to prove to senders that it really is that company; if the stamp is missing, it shows a fake one and careful senders refuse to hand over the letter.

That is TLS termination at the Ingress. The controller holds the certificate and key from a kubernetes.io/tls Secret, decrypts, routes on Host and path, and talks HTTP to the pods. SNI tells it which certificate to use. The "Kubernetes Ingress Controller Fake Certificate" is the fake stamp.

Why HTTPS at the Ingress

Users must reach the shop over HTTPS. You could give every app its own certificate and TLS code, and renew them all separately. Or you let the one place every request passes through - the Ingress controller - do TLS for all of them. That is the normal setup, and its failure modes (the wrong certificate, the "Fake Certificate", redirect loops) are daily work.

What you need to know already: the TLS handshake, certificate chains, SNI, trust stores (9.15), openssl for certificates (9.17), curl -v (9.21), Secrets (15.33), Ingress, the controller and --resolve (16.19).

Where TLS ends

TLS termination means: the TLS session ends here. At the Ingress, the browser's encrypted connection ends at the controller, which decrypts, routes on Host and path, and talks plain HTTP to the pods (or opens a new TLS connection to them, with the annotation backend-protocol: HTTPS).

The certificate and its private key live in a Secret of type kubernetes.io/tls in the Ingress's namespace. kubectl create secret tls makes one from two files: --cert the certificate (PEM, 9.17), --key the private key:

# shop.lab.crt/.key = a cert from the lab CA (the TLS mission issues one)
k create secret tls shop-tls -n shop --cert=shop.lab.crt --key=shop.lab.key
secret/shop-tls created
k get secret shop-tls -n shop
NAME       TYPE                DATA   AGE
shop-tls   kubernetes.io/tls   2      4s

TYPE kubernetes.io/tls tells the controller what the Secret holds; DATA 2 = two keys, tls.crt and tls.key.

tls.crt should hold the leaf certificate and the intermediates (the "full chain", 9.15): the controller serves whatever is in it, and a missing intermediate is the classic "works in Chrome, fails in curl" of 9.17.

The Ingress points at the Secret in a tls: section:

spec:
  ingressClassName: nginx
  tls:
  - hosts: [shop.lab]
    secretName: shop-tls
  rules:
  - host: shop.lab
    http: ...

hosts must match the rules' hosts and the names in the certificate. One controller serves many certificates on one port by SNI (9.15): the client says which name it wants in its first TLS message (the ClientHello), and the controller picks the matching Secret.

What you see from outside

curl -sv shows the handshake (-v writes to stderr, so 2>&1 merges it before grep); --resolve pins shop.lab to a node IP so the SNI name is right:

# the TLS mission's Ingress, reached through ingress-nginx
curl -sv --resolve shop.lab:30443:10.64.0.11 https://shop.lab:30443/ 2>&1 | grep -E 'subject:|issuer:|verify'
*  subject: CN=shop.lab
*  issuer: C=RO; O=LabCorp; CN=LabCorp Issuing CA G2
*  SSL certificate verify ok.

subject = whose certificate (CN = the name), issuer = who signed it, verify ok = curl trusted the chain. oncall-lab trusts the LabCorp root (installed with update-ca-certificates in 9.17), so the chain verifies.

openssl s_client shows the chain the controller sends. -connect HOST:PORT where to connect, -servername the SNI name to ask for, </dev/null so it closes right away:

openssl s_client -connect shop.lab:30443 -servername shop.lab </dev/null 2>/dev/null | head -5
CONNECTED(00000003)
---
Certificate chain
 0 s:CN=shop.lab
   i:C=RO, O=LabCorp, CN=LabCorp Issuing CA G2

0 is the first certificate in the chain, s: its subject, i: its issuer.

The HTTP to HTTPS redirect

Once a host has TLS, plain HTTP requests are redirected - with 308 Permanent Redirect, which (unlike 301) keeps the method: a POST stays a POST. curl -si shows the response headers (-i):

curl -si -H 'Host: shop.lab' http://10.64.0.11:30080/ | head -4
HTTP/1.1 308 Permanent Redirect
Server: nginx
Date: Tue, 22 Sep 2026 20:00:13 GMT
Location: https://shop.lab/

The annotation nginx.ingress.kubernetes.io/ssl-redirect: "false" turns it off per Ingress. Note the Location has no port: behind a NodePort, the redirect points at 443, which nobody serves here. Real deployments put a load balancer on 443.

The fake certificate

A name the controller has no certificate for - or a secretName that does not exist, or a Secret of the wrong type - gets the controller's built-in default certificate, signed by nobody:

curl -sv --resolve admin.lab:30443:10.64.0.11 https://admin.lab:30443/ 2>&1 | grep -E 'subject|problem'
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate problem: self-signed certificate
curl: (60) SSL certificate problem: self-signed certificate

"Kubernetes Ingress Controller Fake Certificate" in a browser warning means exactly one thing: the controller did not find a usable certificate for that SNI name. The controller log says why:

W0922 20:00:14.201114       7 backend_ssl.go:47] Error obtaining X.509 certificate: no object matching key "shop/shop-tls" in local store

(shop/shop-tls = namespace/name of the Secret it looked for.)

curl -k skips verification - fine to prove routing, never a fix.

Where certificates really come from

Nobody creates production certificates by hand. cert-manager is a controller you install that does it for you:

The Ingress only ever references the Secret name.

What still bites teams: Let's Encrypt's HTTP-01 check (it fetches a special URL from your site to prove you own the name) must be reachable through the same Ingress controller, and a renewal that silently fails leaves you with an expiry incident 30 days later - so alert on certificate expiry.

Other TLS modes you should recognise:

What you can now do:

Why it helps

Expired or wrong certificates are among the most common causes of customer-facing outages, and on Kubernetes they almost always sit at the ingress. When a browser shows "Kubernetes Ingress Controller Fake Certificate", you'll know it means the controller found no usable Secret for that name, and where to read why. When "it works in Chrome but not in curl or Java", you'll check the Secret holds the full chain.

On a banking platform you'll deal with internal CAs, cert-manager ClusterIssuers, renewals that silently fail 30 days before an outage, and mTLS for B2B APIs. This lesson gives you the checks: curl -v --resolve, openssl s_client -servername, and the controller log line.

FAQ

Why do I see "Kubernetes Ingress Controller Fake Certificate"?

The controller has no usable certificate for the SNI name the client asked for, so it serves its built-in self-signed default. Causes: no tls section for that host, a secretName that doesn't exist or is in another namespace, a Secret of the wrong type, or hosts that don't match the certificate. The controller log says which, for example "no object matching key shop/shop-tls in local store".

Does the traffic between the ingress and the pod stay encrypted?

Not by default. With termination at the Ingress, the controller decrypts and talks plain HTTP to the pod over the cluster network. If the backend must be encrypted too, use nginx.ingress.kubernetes.io/backend-protocol: HTTPS (re-encryption, the pod serves its own certificate), passthrough (the pod terminates TLS, no path routing), or a service mesh with mTLS between pods.

Why "works in Chrome, fails in curl and Java"?

A missing intermediate certificate. Browsers often fill in missing intermediates from cache or by fetching them; curl, OpenSSL and Java don't. The controller serves exactly what is in tls.crt, so it must contain the leaf and the intermediates (the full chain). openssl s_client -connect host:443 -servername host shows which certificates are actually sent.

Why test with --resolve instead of -H 'Host: ...'?

A faked Host header gets HTTP routing right, but the TLS handshake still uses the IP or the URL's hostname for SNI, so the controller picks the wrong certificate (or the fake one). curl --resolve shop.lab:30443:10.64.0.11 https://shop.lab:30443/ pins the name to the IP, so both SNI and the Host header are correct.

Why is the HTTP redirect a 308 and not a 301?

A 308 Permanent Redirect keeps the method and body, so a POST stays a POST after the redirect. With 301 many clients switch to GET, which silently breaks API calls. ingress-nginx redirects HTTP to HTTPS with 308 as soon as a host has TLS; ssl-redirect: "false" turns it off per Ingress.

In an interview Junior

How do you configure TLS for an application behind an Ingress?

Terminate TLS at the Ingress controller:

  1. Put the certificate - leaf plus intermediates - and its private key in a Secret of type kubernetes.io/tls in the Ingress's namespace: k create secret tls shop-tls --cert=shop.lab.crt --key=shop.lab.key.
  2. Reference it in the Ingress: tls: - hosts: [shop.lab] secretName: shop-tls. The hosts must match the rules and the certificate's names.
  3. The controller picks the certificate per request by SNI, decrypts, routes on host and path, and talks HTTP to the pods (or HTTPS with an annotation). Plain HTTP gets a 308 redirect.

Check it: curl -sv --resolve shop.lab:443:IP https://shop.lab/ or openssl s_client -connect IP:443 -servername shop.lab. "Kubernetes Ingress Controller Fake Certificate" means the controller found no usable Secret for that name - wrong name, wrong type or missing; its log says which.

In production, cert-manager issues and renews the Secret from an Issuer; still alert on expiry.

Also asked: How would you prevent certificate-expiry outages on a Kubernetes platform? · What does the Ingress controller's Fake Certificate tell you? · What is TLS passthrough, and what do you lose with it?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.