OnCallReady

Lesson 22.27 · Azure I: CLI, Identity & Data Planes · 11 min read

From Key Vault into a pod: the Secrets Store CSI driver

In plain words

Imagine a school where each pupil has a locker, and the school nurse keeps the medicine in a locked cabinet. Instead of pupils carrying their medicine around, a helper checks each pupil's ID at the start of the day, fetches exactly their medicine from the cabinet, and puts it in their locker. If the helper can't get it, the pupil isn't let into class until it's sorted.

The Secrets Store CSI driver is that helper. When a pod starts, it authenticates as the pod's workload identity, reads the secrets listed in a SecretProviderClass from Key Vault, and writes them as files into a tmpfs mount like /mnt/secrets/orders-db-password. Nothing is committed to Git or baked into an image. If the fetch fails, the pod stays in ContainerCreating with a FailedMount event.

The password now lives in Key Vault. But the orders pod needs it at start-up, and a Kubernetes Secret committed to git (15.33) is exactly what you were trying to avoid. So: how do secrets get from Key Vault into a pod without being committed anywhere? (One of the Notion questions.)

What you need to know already: 22.9 (workload identity), 22.23 (Key Vault and its roles), 16.37 (volumes and volumeMounts), 15.33 (Kubernetes Secrets), 15.22 (DaemonSets).

The answer has three parts: an identity for the pod (workload identity, 22.9), a role on the vault (Secrets User, 22.23), and a mount that fetches the secret at pod start. The mount is the Secrets Store CSI driver.

CSI (Container Storage Interface) is the standard plug-in interface Kubernetes uses to attach storage to pods (Ch 16 used it for disks). A CSI driver is one such plug-in. The Secrets Store CSI driver is a driver whose "storage" is a secret store: it fetches secrets and presents them as files in the pod.

The add-on

On Azure's managed Kubernetes you get the driver as an add-on (a component Microsoft installs and updates for you). --query addonProfiles.azureKeyvaultSecretsProvider shows its settings:

$ az aks show -g rg-oncall-lab -n aks-sysop --query addonProfiles.azureKeyvaultSecretsProvider
{
  "config": {
    "enableSecretRotation": "true",
    "rotationPollInterval": "2m"
  },
  "enabled": true,
  "identity": { "clientId": "...", "objectId": "...", "resourceId": ".../azurekeyvaultsecretsprovider-aks-sysop" }
}

az aks enable-addons --addons azure-keyvault-secrets-provider (or key_vault_secrets_provider in Terraform) installs two DaemonSets in kube-system: the CSI driver and the Azure provider. On every node, when a pod that mounts the volume starts, kubelet asks the driver to mount it, the provider authenticates as the pod's workload identity, reads the secrets from Key Vault, and writes them as files into a tmpfs (a filesystem in RAM) in the pod. enableSecretRotation and rotationPollInterval are about re-reading secrets later - see Rotation below.

The add-on has its own identity too (azurekeyvaultsecretsprovider-<cluster>). You can grant that one Secrets User and point every SecretProviderClass at it - then every pod on the cluster can read the vault. Use workload identity per app.

SecretProviderClass

A SecretProviderClass is a custom Kubernetes resource the driver adds: it says which vault, which identity and which secrets. One per app, in the app's namespace.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: orders-kv
  namespace: orders
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "<clientId of id-orders>"        # workload identity: WHICH identity
    keyvaultName: kv-shop-prod
    tenantId: 11111111-2222-3333-4444-555555555555
    objects: |
      array:
        - |
          objectName: orders-db-password
          objectType: secret
          objectVersion: ""                    # empty = latest

Every value is a string: "false", the quoted clientID. objects is a YAML string containing more YAML - indentation errors inside it are the most common reason a SecretProviderClass silently mounts nothing.

The pod side

spec:
  serviceAccountName: orders-api            # annotated with the same client-id
  containers:
    - name: orders-api
      volumeMounts:
        - name: secrets
          mountPath: /mnt/secrets
          readOnly: true
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: orders-kv

The app reads /mnt/secrets/orders-db-password. Nothing in git, nothing in an image, nothing in a Kubernetes Secret unless you ask for one.

When it fails, the pod does not start

A failed mount keeps the pod in ContainerCreating, and the reason is an event (a Kubernetes record of something that happened to an object, shown by kubectl describe and kubectl get events), not a log line of your app (your app never ran):

Warning  FailedMount  kubelet  MountVolume.SetUp failed for volume "secrets" : rpc error:
code = Unknown desc = failed to mount secrets store objects for pod orders/orders-api-7b9c6d4f5-q2x8z,
err: rpc error: code = Unknown desc = failed to mount objects, error: failed to get objectType:secret,
objectName:orders-db-password, objectVersion:: GET https://kv-shop-prod.vault.azure.net/secrets/orders-db-password/
RESPONSE 403: 403 Forbidden
ERROR CODE: Forbidden
{"error":{"code":"Forbidden","message":"Caller is not authorized ... Action: 'Microsoft.KeyVault/vaults/secrets/getSecret/action' ...","innererror":{"code":"ForbiddenByRbac"}}}

kubectl describe pod shows it. The message embeds the Key Vault error, so the same reading skills apply: 403 + ForbiddenByRbac = role missing; AADSTS70021 = the federated credential does not match; SecretNotFound = typo in objectName.

Kubernetes Secrets as well (optional)

Many apps want environment variables, which come from a Kubernetes Secret. The driver can create one while a pod mounts the volume:

spec:
  secretObjects:
    - secretName: orders-db
      type: Opaque
      data:
        - objectName: orders-db-password
          key: password

Then env.valueFrom.secretKeyRef: { name: orders-db, key: password }. Two caveats: the Kubernetes Secret only exists while at least one pod mounts the volume, and it is only base64 in etcd - anyone who can get secrets in the namespace reads it. Files are the better default.

Rotation

With enableSecretRotation the provider re-reads every rotationPollInterval (2 minutes) and updates the mounted files and any synced Secret. What it does not do is restart your app: environment variables never change in a running process, and an app that read the file once at startup keeps the old value. So rotation needs the app to re-read the file (or a reloader, a small controller that restarts pods when the Secret they use changes). And every poll is a SecretGet per pod per object - visible in the vault's audit log and on its bill.

Later (Ch 24): the FailedMount event is also a row in Log Analytics (KubeEvents), and External Secrets Operator is the other way to get Key Vault secrets into a cluster - it syncs them into Kubernetes Secrets from a controller, without mounts.

What you can now do

Why it helps

"How do secrets get from Key Vault into a pod without being committed anywhere?" is a question you'll answer in interviews and implement at work. On Azure's managed Kubernetes this driver is one of the two standard answers, and you need to know how it fails: a pod stuck in ContainerCreating, with the real reason in an event that embeds the Key Vault error.

You'll also get tickets about rotation: "we rotated the password and the app still uses the old one". The driver updates the mounted file, but environment variables never change in a running process, and an app that read the file once keeps the old value. Knowing that, and the difference between file mounts and synced Kubernetes Secrets, lets you choose the right pattern for each team.

Commands in this lesson

az

FAQ

Why is my pod stuck in ContainerCreating?

With the CSI driver, a failed secret mount blocks the container from starting, and the reason is an event, not an application log, because the app never ran. kubectl describe pod shows FailedMount with the embedded Key Vault error. 403 with ForbiddenByRbac means the identity lacks a role, AADSTS70021 means the federated credential doesn't match the ServiceAccount, and SecretNotFound usually means a typo in objectName.

Do I get a Kubernetes Secret or a file?

By default a file in a tmpfs mount inside the pod, which is the safer option: the value is never stored in etcd. If the app needs environment variables, secretObjects in the SecretProviderClass makes the driver also create a Kubernetes Secret, but only while at least one pod mounts the volume, and that Secret is just base64 in etcd, readable by anyone with get secrets in the namespace. Prefer files when the app can read them.

Does rotation restart my pods?

No. With enableSecretRotation, the provider re-reads Key Vault every rotationPollInterval, 2 minutes by default, and updates the mounted files and any synced Secret. But a running process never sees new environment variables, and an app that read the file once at startup keeps the old value. So either the app re-reads the file, or a tool like Reloader restarts pods when the synced Secret changes. Each poll is also a SecretGet per pod per object, visible in logs and billing.

Why is the objects field a YAML string inside YAML?

Because the driver's parameters are a flat map of strings passed to the provider, so the list of objects is encoded as a string containing more YAML. That's also why values like usePodIdentity: "false" must be quoted strings. Indentation mistakes inside the objects string are the most common reason a SecretProviderClass mounts nothing or fails, so check it carefully, or generate it with a template or Terraform.

Should I use the CSI driver or External Secrets Operator?

Both are valid. The CSI driver mounts secrets as files at pod start, keeps them out of etcd by default, and needs each pod to mount a volume. External Secrets Operator syncs Key Vault into Kubernetes Secrets from a controller, which suits apps that use environment variables and works without volume mounts, but the values then live in etcd. Know which one your platform standardises on and why.

In an interview Mid

How do you get secrets from Azure Key Vault into a pod without storing them in git?

Three parts, nothing secret anywhere in git or the image:

  1. An identity for the pod - workload identity: a user-assigned identity with a federated credential for the pod's ServiceAccount (system:serviceaccount:<ns>:<sa>).
  2. A role on the vault - Key Vault Secrets User for that identity, scoped to the vault.
  3. A mount - the Secrets Store CSI driver (an add-on): a SecretProviderClass in the app's namespace names the vault, the identity's client ID and the secrets; the pod mounts a CSI volume using it. At pod start the provider authenticates as the pod's identity, reads the secrets and writes them as files in a tmpfs: /mnt/secrets/orders-db-password.

Optionally secretObjects syncs them into a Kubernetes Secret for env vars - but that is only base64 in etcd, so files are the better default.

When it fails the pod stays in ContainerCreating and the reason is a FailedMount event (kubectl describe pod), embedding the Key Vault error: 403 = role, AADSTS70021 = federated credential mismatch, SecretNotFound = typo. Rotation updates the files, not a running app's environment.

Also asked: A pod using the Key Vault CSI driver is stuck in ContainerCreating. How do you find out why? · What are the trade-offs between mounting secrets as files and exposing them as environment variables? · What happens to a running app when a secret in Key Vault is rotated?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.