Lesson 34.29 · Kubernetes: Ingress, Gateway API & Service Mesh · 15 min read

Mutual TLS and authorization in the mesh

Q: What does a plain-text client see against STRICT?

A connection reset: curl prints (56) Recv failure: Connection reset by peer. The server's sidecar never finds a matching filter chain for plain text, and its access log shows a line with response flag NR and filterchainnotfound. No HTTP status is involved, because the HTTP layer was never reached.

Q: You turned on STRICT mTLS and some clients broke. How do you work out which and fix it?

PeerAuthentication in STRICT means the workload accepts only mTLS, so clients without a sidecar now get a connection reset, curl's (56) Recv failure, and the server's proxy logs NR filterchainnotfound. Those log lines give me the source addresses of the plain-text clients. The safe path is back to PERMISSIVE, then either put those clients in the mesh or, if one port must stay open, use portLevelMtls for that port only, then STRICT again. If a client gets 403 RBAC: access denied instead, mTLS works and an AuthorizationPolicy denies it.

In plain words

Imagine a company where every employee badge has a photo and a name, printed by the company itself. Doors check the badge, not the person's word. Some doors only check that you have a badge at all; others also have a list of exactly which names may come in, and maybe only on weekdays.

In the mesh every workload gets a certificate with a SPIFFE ID like spiffe://cluster.local/ns/shop/sa/web, based on its service account. With auto mTLS sidecars encrypt and present those badges to each other. PeerAuthentication says what a workload accepts: badges only (STRICT) or badges and plain visitors (PERMISSIVE). AuthorizationPolicy is the list on the door: which identities may call which methods and paths. A denied caller gets 403 with the body RBAC: access denied.

Identity first

Every workload in the mesh gets a certificate from istiod's CA whose identity is a SPIFFE ID built from its namespace and ServiceAccount:

spiffe://cluster.local/ns/shop/sa/web
         trust domain   ns   service account

The certificate lives about 24 hours and is rotated by pilot-agent long before that - nobody copies or renews anything. The identity is the ServiceAccount, so give each workload its own ServiceAccount: with everything on default, every pod in the namespace is the same "person".

What you need to know already: mutual TLS (9.19), ServiceAccounts (17.18), the previous two lessons, NetworkPolicy as the L3/L4 comparison (16.29).

Auto mTLS and PeerAuthentication

Out of the box Istio uses auto mTLS: a client sidecar talking to a pod that has a sidecar uses mTLS automatically; to a pod without one it sends plaintext. The server side decides what it accepts, with PeerAuthentication:

PERMISSIVE   accept mTLS and plaintext (the default - safe for migrations)
STRICT       accept mTLS only: plaintext connections are reset
DISABLE      plaintext only

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system      # the root namespace = the whole mesh
spec:
  mtls:
    mode: STRICT

Precedence: a policy with a workload selector beats the namespace-wide one (no selector, in that namespace), which beats the mesh-wide one (no selector, in istio-system). portLevelMtls (on a workload policy) overrides one port - keyed by the container port, not the Service port.

What a non-mesh client sees against STRICT is not an HTTP error:

$ curl -sS http://ledger.bank:8000/get
curl: (56) Recv failure: Connection reset by peer

and the server's sidecar logs a line with no request in it:

[2026-10-07T10:02:11.120Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.244.2.31:8080 10.244.1.17:44912 - -

filter_chain_not_found: the inbound listener only has a filter chain for TLS, the client spoke plaintext. That line is the signature of "STRICT broke a client that is not in the mesh".

Migrating to STRICT safely: leave PERMISSIVE, check that every caller already uses mTLS (the metric label connection_security_policy="mutual_tls" on istio_requests_total, or XFCC headers), inject or exempt the stragglers, then switch.

The client side: DestinationRule TLS

A DestinationRule's trafficPolicy.tls.mode overrides auto mTLS for traffic to a host: ISTIO_MUTUAL (force mesh mTLS), DISABLE (force plaintext), SIMPLE/MUTUAL (TLS to something outside the mesh). The classic mistake: DISABLE towards a STRICT server - every call fails with 503 and flag UC ("upstream connect error ... reset reason: connection termination").

Seeing it

The server sidecar adds an X-Forwarded-Client-Cert (XFCC) header to requests that arrived over mTLS - httpbin's /headers shows it:

"X-Forwarded-Client-Cert": ["By=spiffe://cluster.local/ns/demo/sa/httpbin;Hash=0dcf...;Subject=\"\";URI=spiffe://cluster.local/ns/demo/sa/curl"]

By = the server, URI = the client. istioctl x describe pod prints the effective mode, and istioctl proxy-config secret POD the proxy's certificate (ACTIVE, its serial, not before / not after).

AuthorizationPolicy

mTLS answers "who is calling". AuthorizationPolicy answers "may they":

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-allow
  namespace: shop
spec:
  selector:
    matchLabels:
      app: orders               # applies to the orders pods' sidecars
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/shop/sa/frontend"]    # no spiffe:// prefix
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/api/*"]

The rules every on-call engineer must know:

Evaluation order: CUSTOM, then DENY, then ALLOW. A matching DENY always wins.
If a workload has no ALLOW policy, everything not denied is allowed. As soon as it has one, only requests matching some ALLOW rule pass.
An ALLOW policy with an empty spec (spec: {}) matches nothing: deny all. The usual baseline, per namespace, before opening what is needed.
principals and namespaces come from the mTLS certificate: they only match mTLS traffic. Plaintext callers have no principal - so a rule with principals never matches them.
Denied requests get 403 with the body RBAC: access denied from the server sidecar; the access log shows rbac_access_denied_matched_policy[...]. A 403 the app never saw.

istioctl x authz check POD       the policies that apply to a pod and their rule counts

AuthorizationPolicy is L7 (methods, paths, headers) and identity-based; NetworkPolicy is L3/L4 and address-based. Keep both: NetworkPolicy still protects against pods without a sidecar.

In an interview: "PeerAuthentication says what a workload accepts - PERMISSIVE or STRICT mTLS; AuthorizationPolicy says who may call what, using the SPIFFE identity from the certificate. A plaintext client against STRICT gets a connection reset; an mTLS client a policy denies gets 403 RBAC: access denied."

Quick check

$ kubectl explain peerauthentication.spec.mtls

With no PeerAuthentication anywhere, which mode does a sidecar run in?

STRICT
PERMISSIVE
DISABLE

Answer

PERMISSIVE - PERMISSIVE accepts both, so adding the mesh never breaks existing plaintext callers. STRICT is a decision you take later, on purpose.

What you can now do:

read a workload's SPIFFE identity and prove a call used mTLS
set STRICT at mesh, namespace or workload level and predict who breaks
write an allow-list AuthorizationPolicy and read a 403 RBAC denial

Why it helps

Zero-trust networking inside the cluster is a common security requirement, and the mesh is how many companies meet it. Turning on STRICT is also one of the classic ways to break production: a client outside the mesh (a legacy pod, a monitoring job, a node-level health check) suddenly gets connection resets.

Knowing the two objects and how they combine, PeerAuthentication for the transport and AuthorizationPolicy for who may call what, lets you roll out mTLS safely: PERMISSIVE first, check who still sends plain text, then STRICT. And the two symptoms are different enough to diagnose at a glance: a reset means the transport policy, a 403 means an authorization policy.

Commands in this lesson

curl kubectl

FAQ

Is traffic encrypted before I set any PeerAuthentication?

Between two sidecars, yes: auto mTLS makes the client proxy use mTLS whenever the server has a sidecar, and the default mode PERMISSIVE accepts it. What PERMISSIVE also accepts is plain text from clients without a sidecar. STRICT removes that option.

Which PeerAuthentication wins?

The most specific: one with a workload selector beats the namespace-wide one, which beats the mesh-wide one in the root namespace (istio-system). Inside a policy, portLevelMtls for a specific container port overrides the policy's mode for that port.

What does a plain-text client see against STRICT?

A connection reset: curl prints (56) Recv failure: Connection reset by peer. The server's sidecar never finds a matching filter chain for plain text, and its access log shows a line with response flag NR and filter_chain_not_found. No HTTP status is involved, because the HTTP layer was never reached.

How are AuthorizationPolicies evaluated?

CUSTOM first, then DENY: if any DENY policy matches, the request is refused. Then, if any ALLOW policy selects the workload, the request must match at least one of them, otherwise it is refused. If no ALLOW policy applies, the request is allowed. An ALLOW policy with no rules denies everything.

Why must principals not start with spiffe://?

In AuthorizationPolicy the principal is written without the scheme: cluster.local/ns/shop/sa/web. The validation webhook rejects the spiffe:// prefix. The full SPIFFE ID appears in the X-Forwarded-Client-Cert header the server receives, which is a handy way to see who called.

In an interview Mid

You turned on STRICT mTLS and some clients broke. How do you work out which and fix it?

PeerAuthentication in STRICT means the workload accepts only mTLS, so clients without a sidecar now get a connection reset, curl's (56) Recv failure, and the server's proxy logs NR filter_chain_not_found. Those log lines give me the source addresses of the plain-text clients. The safe path is back to PERMISSIVE, then either put those clients in the mesh or, if one port must stay open, use portLevelMtls for that port only, then STRICT again. If a client gets 403 RBAC: access denied instead, mTLS works and an AuthorizationPolicy denies it.

Also asked: What is a SPIFFE ID, and where does it come from? · How does Istio evaluate ALLOW and DENY AuthorizationPolicies? · What is the difference between PERMISSIVE and STRICT mode?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.