Vault on Kubernetes: the auth method, Agent Injector, ESO and CSI

The problem: a pod needs a password

The orders API runs in Kubernetes and needs its database password. You have already seen the two bad answers: the password in the image (anyone who pulls the image has it) and a Kubernetes Secret written in git (base64 is an encoding, not encryption - anyone who can read the repo can decode it). The password belongs in Vault. The question this lesson answers is: how does the pod get it out, without a Vault token baked into the pod either?

What you need to know already: ServiceAccounts and their tokens (17.33), RBAC and ClusterRoleBindings (17.30), Secrets and why base64 is not security (17.41), init containers and sidecars (15.35), the apiserver's request path and admission (15.5), CRDs (16.26), and this chapter's lessons on the Kubernetes-free basics: tokens and auth methods, KV v2 and the data/ path, policies, and Vault Agent (auto-auth, sinks, templates).

The words you need first

• Workload identity - proof of "which program is this" that the platform hands out, so the program never needs a stored password to prove it. In Kubernetes it is the pod's ServiceAccount token: a signed JWT the kubelet mounts at /var/run/secrets/kubernetes.io/serviceaccount/token.
• TokenReview - the Kubernetes API call that answers "is this JWT valid, and whose is it?". Anyone who wants to trust a ServiceAccount token asks the apiserver with a TokenReview.
• Mutating webhook - an admission step (15.5) that may change an object on its way into the cluster. The Vault Agent Injector is one: it rewrites pods.
• Secret zero - the first credential a program needs to fetch all the others. The whole game is making secret zero something the platform gives you (a ServiceAccount token) instead of something you store.

Four ways to get a secret into a pod

All four start the same way: the pod (or the controller acting for it) logs in to Vault with a Kubernetes ServiceAccount token through Vault's kubernetes auth method. So that comes first.

The Kubernetes auth method

The flow, every time a pod logs in:

1. The pod reads its ServiceAccount JWT and sends it to Vault: PUT /v1/auth/kubernetes/login with role=orders jwt=....
2. Vault does not trust the JWT on its own: it sends a TokenReview to the cluster's apiserver ("is this token valid?").
3. The apiserver answers with the token's user: system:serviceaccount:shop:orders.
4. Vault checks the answer against the role: is the ServiceAccount name in bound_service_account_names, the namespace in bound_service_account_namespaces?
5. If yes, Vault issues a Vault token with the role's token_policies and TTL.

Setting it up is three writes. Vault here runs on the box (oncall-lab, 10.64.0.2), outside the cluster, and talks to the apiserver at https://10.64.0.10:6443:

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
$ vault write auth/kubernetes/config kubernetes_host=https://10.64.0.10:6443
Success! Data written to: auth/kubernetes/config
$ vault write auth/kubernetes/role/orders bound_service_account_names=orders bound_service_account_namespaces=shop token_policies=orders-db ttl=1h
Success! Data written to: auth/kubernetes/role/orders

vault read shows what the role really holds - note that ttl is the old name of token_ttl, and both are shown:

$ vault read auth/kubernetes/role/orders
Key                                         Value
---                                         -----
alias_name_source                           serviceaccount_uid
audience                                    n/a
bound_service_account_names                 [orders]
bound_service_account_namespace_selector    n/a
bound_service_account_namespaces            [shop]
max_ttl                                     0s
period                                      0s
policies                                    [orders-db]
token_bound_cidrs                           []
token_explicit_max_ttl                      0s
token_max_ttl                               0s
token_no_default_policy                     false
token_num_uses                              0
token_period                                0s
token_policies                              [orders-db]
token_ttl                                   1h
token_type                                  default
ttl                                         1h

You can play the pod by hand: kubectl create token asks the apiserver for a fresh ServiceAccount token (17.33), and vault write auth/kubernetes/login is exactly the call the agent makes.

Who is allowed to ask "is this token valid?"

A TokenReview is itself an API call, and RBAC (17.30) guards it: the caller needs create on tokenreviews.authentication.k8s.io. The built-in ClusterRole system:auth-delegator grants exactly that. Two ways to wire it:

• A reviewer ServiceAccount (the usual production setup): one ServiceAccount (say vault/vault-auth) gets system:auth-delegator, and its long-lived token goes into auth/kubernetes/config token_reviewer_jwt=.... Vault uses it for every review.
• The client's own JWT (what you get with no token_reviewer_jwt and Vault outside the cluster): every application ServiceAccount must be bound to system:auth-delegator itself. Simple for a lab, noisy for a platform.

When Vault runs inside the cluster (the official chart), it can use its own pod's token as the reviewer and you configure nothing.

$ kubectl create clusterrolebinding vault-tokenreview --clusterrole=system:auth-delegator --serviceaccount=shop:orders
clusterrolebinding.rbac.authorization.k8s.io/vault-tokenreview created
$ vault write auth/kubernetes/login role=orders jwt=$(kubectl create token orders -n shop)
Key                                       Value
---                                       -----
token                                     hvs.CAESIK7YFJ0lWMnHhAbkwrod9uxbryuY18vYsaposwGh4KHGh2cy4q8AdXLB0SKv8C1qmhudL59Ik0YDUR8ot
token_accessor                            Hc74jlxxLNQxeUUHnOErZkPQ
token_duration                            1h
token_renewable                           true
token_policies                            ["default" "orders-db"]
identity_policies                         []
policies                                  ["default" "orders-db"]
token_meta_role                           orders
token_meta_service_account_name           orders
token_meta_service_account_namespace      shop
token_meta_service_account_secret_name    n/a
token_meta_service_account_uid            4dc32e95-13bc-4346-96a0-0650bd54c0c5

The token's metadata records which ServiceAccount it was issued to: that is what the audit log shows when you later ask "who read this secret?".

The error ladder

Read the error top to bottom; each one means the previous step passed:

The Agent Injector

The injector is a small webhook server (hashicorp/vault-k8s, 1.7.6 at the time of writing), normally installed with the official Vault chart (injector.enabled=true, and injector.externalVaultAddr when Vault lives outside the cluster, as here). In the lab the platform team has already installed it:

$ kubectl get mutatingwebhookconfigurations
NAME                       AGE
vault-agent-injector-cfg   16s
$ kubectl -n vault get deploy,pods
NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-agent-injector   1/1     1            1           16s

NAME                                        READY   STATUS    RESTARTS   AGE
pod/vault-agent-injector-z8xd88smw9-8zfgw   1/1     Running   0          15s

You never talk to it. You annotate the pod template, and every new pod that carries vault.hashicorp.com/agent-inject: "true" is rewritten on its way in:

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "orders"
        vault.hashicorp.com/agent-inject-secret-db: "secret/data/shop/orders/db"
    spec:
      serviceAccountName: orders

• agent-inject: "true" - inject. A quoted string: YAML true without quotes is a boolean, and annotations must be strings.
• role - the Vault Kubernetes auth role to log in with.
• agent-inject-secret-<name>: <path> - render the secret at that API path (for KV v2 that is secret/data/..., not the vault kv get path) into the file /vault/secrets/<name>.

What the webhook adds, in short:

• an init container vault-agent-init: logs in, renders every file once, exits. Your app container only starts after it, so the file is there on the app's first read;
• a sidecar vault-agent: stays running, renews the token, re-renders when a secret changes or a lease nears its end;
• a shared in-memory emptyDir volume mounted at /vault/secrets in every container.

$ kubectl get pods -n shop
NAME                          READY   STATUS     RESTARTS   AGE
orders-api-ss6zfq5cj5-r9lj2   0/2     Init:0/1   0          3s
$ kubectl get pods -n shop
NAME                          READY   STATUS    RESTARTS   AGE
orders-api-ss6zfq5cj5-r9lj2   2/2     Running   0          33s

2/2: your container and the sidecar. A pod stuck at Init:0/1 means the init agent cannot log in or cannot read a secret - its log says which: kubectl logs <pod> -c vault-agent-init. A pod that comes up 1/1 with no vault-agent container means the injector never saw it: no webhook, the annotation on the Deployment's metadata instead of the pod template, or a typo.

$ kubectl logs -n shop deploy/orders-api -c vault-agent | tail -5
2026-09-22T20:00:50.000Z [INFO]  agent.template.server: template server received new token
2026-09-22T20:00:50.000Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2026-09-22T20:00:50.000Z [INFO]  agent: (runner) creating watcher
2026-09-22T20:00:50.000Z [INFO]  agent: (runner) starting
2026-09-22T20:00:50.000Z [INFO]  agent: (runner) rendered "(dynamic)" => "/vault/secrets/db"

The default template is a Go map dump

Without a template annotation, the injector renders the whole secret with Go's default formatting. For KV v2 that is rarely what an app can read:

$ kubectl exec -n shop deploy/orders-api -c app -- cat /vault/secrets/db
data: map[DB_PASSWORD:orders-static-2023 DB_USER:orders_app]
metadata: map[created_time:2026-09-22T20:00:38.104545576Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

Give the file a shape with agent-inject-template-<name> - the same template language as Vault Agent (consul-template):

        vault.hashicorp.com/agent-inject-template-db: |
          {{- with secret "secret/data/shop/orders/db" -}}
          DB_USER={{ .Data.data.DB_USER }}
          DB_PASSWORD={{ .Data.data.DB_PASSWORD }}
          {{- end }}

.Data is the API response's data; for KV v2 the values sit one level deeper, in .Data.data. For a dynamic database secret (no extra level) it is .Data.username. Other annotations you will meet: agent-pre-populate-only: "true" (only the init container, no sidecar: for Jobs, or when rotation needs a restart anyway), agent-inject-command-<name> (run a command after a render, e.g. send the app a SIGHUP), agent-inject-file-<name> (another file name), service (another Vault address).

External Secrets Operator

The injector puts files in pods. Many teams instead want a plain Kubernetes Secret: their charts and manifests expect one, deployment tools compare objects, and other operators read Secrets. External Secrets Operator (ESO) is a controller that keeps a Secret in sync with an external store (Vault, cloud secret managers and many more). The Vault side is split in two objects:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore          # platform-owned: HOW to reach Vault
metadata:
  name: vault
spec:
  provider:
    vault:
      server: https://10.64.0.2:8200
      path: secret                # the KV mount
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: eso
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret              # team-owned: WHAT to copy, into which Secret
metadata:
  name: payments-api-key
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault
  target:
    name: payments-api-key
  data:
  - secretKey: API_KEY
    remoteRef:
      key: shop/payments/api-key  # the key inside the mount: NO data/ here
      property: API_KEY

Two path conventions in one chapter: the injector wants the full API path (secret/data/shop/payments/api-key), the ESO vault provider wants the mount in path and the key without data/ (it adds it because version: v2).

The store logs in as the operator's ServiceAccount, so the same error ladder applies, shown in the store's status:

$ kubectl get clustersecretstore vault
NAME    AGE   STATUS                  CAPABILITIES   READY
vault   5s    InvalidProviderConfig   ReadWrite      False
$ kubectl get clustersecretstore vault -o jsonpath='{.status.conditions[0].message}{"\n"}'
unable to log in to auth method: unable to log in with Kubernetes auth: Error making API request.

URL: PUT https://10.64.0.2:8200/v1/auth/kubernetes/login
Code: 400. Errors:

* invalid role name "eso"

With the role and the auth-delegator binding in place:

$ kubectl get clustersecretstore vault
NAME    AGE   STATUS   CAPABILITIES   READY
vault   76s   Valid    ReadWrite      True
$ kubectl get externalsecret -n shop
NAME               STORETYPE            STORE   REFRESH INTERVAL   STATUS         READY
payments-api-key   ClusterSecretStore   vault   1m                 SecretSynced   True
$ kubectl get secret payments-api-key -n shop -o jsonpath='{.data.API_KEY}' | base64 -d; echo
pk_live_7f3a9c21e8

A changed value in Vault reaches the Secret within refreshInterval. To skip the wait, change the force-sync annotation - ESO syncs on any change of it:

$ kubectl annotate externalsecret payments-api-key -n shop force-sync=$(date +%s) --overwrite
externalsecret.external-secrets.io/payments-api-key annotated

A wrong key does not break the existing Secret: it keeps the last good value and the ExternalSecret turns SecretSyncedError, with the reason in its events:

$ kubectl describe externalsecret payments-api-key -n shop | tail -1
  Warning  UpdateFailed  8s   external-secrets  error retrieving secret at .data[0], key: shop/payments/apikey, err: Secret does not exist

The price of ESO: the value is now in etcd as a Kubernetes Secret (17.41), so anyone with get secrets in the namespace reads it, and you need encryption at rest for etcd and tight RBAC. The injector never writes the secret to the Kubernetes API.

The CSI driver and the Vault Secrets Operator

Two more you will see in job ads:

• Secrets Store CSI driver with the Vault provider: a SecretProviderClass object and a CSI volume in the pod. The kubelet mounts the secret as files at pod start; no sidecar. Optional sync to a Secret. Rotation by polling.
• Vault Secrets Operator (VSO): HashiCorp's own operator, VaultStaticSecret and VaultDynamicSecret objects that write Kubernetes Secrets - like ESO but Vault-only, and it can restart a Deployment when a secret changes.

(simulator) The lab cluster has the injector and ESO; the CSI driver and VSO are described here, not installed.

Choosing

• Charts and manifests that expect Secrets, several secret stores: ESO (or VSO).
• No secret in the Kubernetes API, dynamic secrets that must be renewed, files: the injector.
• Only files, no sidecar per pod: the CSI driver.
• The app needs a fresh credential per request, or transit encryption: call Vault from the app with its ServiceAccount token.

In an interview: "the pod never holds a Vault token in its image or manifest: it proves who it is with its ServiceAccount token, Vault checks that with a TokenReview, and the role maps ServiceAccount + namespace to policies."

What you can now do

• Configure the Kubernetes auth method and a role, and log in by hand with kubectl create token.
• Read the login error ladder: invalid role name, permission denied (TokenReview / system:auth-delegator), service account name not authorized, namespace not authorized.
• Annotate a pod template for the Agent Injector, template the file, and debug a pod stuck at Init:0/1.
• Sync a Vault value into a Secret with ESO, read the store's and the ExternalSecret's status, and force a refresh.
• Say when you would pick the injector, ESO, the CSI driver or the app itself.