OnCallReadyStart learning

KubernetesSecuritySecrets · 6 min read

Rotating the etcd encryption key without locking your Secrets: "no matching key was found"

Every Secret read fails with InternalError after a key rotation: what the error means, how to get the old key back, and the rotation order that avoids it.

After a laptop with a copy of the infra repo was stolen, someone rotated the key Kubernetes uses to encrypt Secrets in etcd. The next morning every deploy that touches a Secret fails:

terminal
$ kubectl get secret db-creds -n payments
Error from server (InternalError): Internal error occurred: [no matching key was found for the provided AES transformer, identity transformer tried to read encrypted data]
$ kubectl get secrets -n payments
Error from server (InternalError): Internal error occurred: unable to transform key "/registry/secrets/payments/api-token": [no matching key was found for the provided AES transformer, identity transformer tried to read encrypted data]

Nothing is wrong with RBAC, the network or etcd. The API server reads the bytes from etcd fine. It just has no key that can decrypt them. The kubelet cannot read them either, so new pods that mount those Secrets will not start.

What the error says

Encryption at rest is set up by an EncryptionConfiguration file, passed to kube-apiserver with --encryption-provider-config. It lists providers in order, and each provider can hold several keys. The Kubernetes docs put the two rules in one sentence: "the keys are tried in order for decryption, and if the provider is the first provider, the first key is used for encryption."

Every value the API server writes starts with a plain prefix that names the provider and the key, for example k8s:enc:aescbc:v1:key1:, followed by the ciphertext. On a read, each provider whose prefix matches tries to decrypt. If none can, the API server returns all their errors together, which is the bracketed list:

  • no matching key was found for the provided AES transformer: the aescbc provider recognised its own prefix, but none of its keys has the name in the data (key1). The text comes from staging/src/k8s.io/apiserver/pkg/server/options/encryptionconfig/config.go; secretbox has the same message with "Secretbox".
  • identity transformer tried to read encrypted data: identity (no encryption, usually last in the list so old plain-text values stay readable) refuses anything that starts with k8s:enc:.
  • unable to transform key "/registry/secrets/...": on a list, the etcd key of the first object that failed.

Together: the data was written with a key the running API server no longer has.

Diagnose: compare the config with the data

On the control plane node, see which keys the config holds and which key wrote each Secret. The API server decrypts for you, so ask etcd directly:

terminal
$ ssh cp-1
$ sudo ls -l /etc/kubernetes/enc/
total 8
-rw------- 1 root root 271 Sep 22 20:00 enc.yaml
-rw------- 1 root root 271 Sep 22 20:00 enc.yaml.bak
$ sudo grep -A1 'name:' /etc/kubernetes/enc/enc.yaml /etc/kubernetes/enc/enc.yaml.bak
/etc/kubernetes/enc/enc.yaml:            - name: key2
/etc/kubernetes/enc/enc.yaml-              secret: ZmVkY2JhOTg3NjU0MzIxMGZlZGNiYTk4NzY1NDMyMTA=
/etc/kubernetes/enc/enc.yaml.bak:            - name: key1
/etc/kubernetes/enc/enc.yaml.bak-              secret: MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY=
$ for s in db-creds api-token stripe-key; do sudo ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key get /registry/secrets/payments/$s --print-value-only | head -c 23; echo "  $s"; done
k8s:enc:aescbc:v1:key1:  db-creds
k8s:enc:aescbc:v1:key1:  api-token
k8s:enc:aescbc:v1:key2:  stripe-key

There is the whole incident. The running config has only key2. Every Secret written before last night is still on key1. Only stripe-key, updated after the change, is on key2. Whoever rotated replaced the key instead of adding one, and never rewrote the existing Secrets.

The fix: put the old key back as a read key

Do not delete and recreate the Secrets. Give the API server key1 again, second, so it can read the old data while key2 keeps encrypting new writes:

terminal
$ sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<'EOF'
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key2
              secret: ZmVkY2JhOTg3NjU0MzIxMGZlZGNiYTk4NzY1NDMyMTA=
            - name: key1
              secret: MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY=
      - identity: {}
EOF

Saving the file changes nothing yet. kube-apiserver reads the encryption config once, when it starts, unless it runs with --encryption-provider-config-automatic-reload=true. In a kubeadm cluster it is a static pod, the same as when its certificates expire. Stop its container and the kubelet starts a new one that reads the file:

terminal
$ sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)
2160cc8d33e788162add86f7c4d940afa716c2364917fef02f848678a056d701
$ exit
$ kubectl get secrets -n payments
NAME         TYPE     DATA   AGE
api-token    Opaque   1      25s
db-creds     Opaque   1      25s
stripe-key   Opaque   1      25s

Everything is readable again. With several control plane nodes, the same file goes to every one of them before you restart any. One API server without key1 is enough to fail some of the requests.

Finish the rotation

The leaked key still protects most of the data. Rewrite every Secret so it is encrypted with key2. A replace with unchanged content is enough, because every write goes through the first key:

terminal
$ kubectl get secrets -A -o json | kubectl replace -f -
secret/api-token replaced
secret/db-creds replaced
secret/stripe-key replaced
secret/cart-db replaced
$ ssh cp-1 'for s in db-creds api-token stripe-key; do sudo ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key get /registry/secrets/payments/$s --print-value-only | head -c 23; echo "  $s"; done'
k8s:enc:aescbc:v1:key2:  db-creds
k8s:enc:aescbc:v1:key2:  api-token
k8s:enc:aescbc:v1:key2:  stripe-key

Check in etcd, not with kubectl: kubectl get secret decrypts with whatever key works and cannot tell you which one wrote the data. On a big cluster, the docs suggest splitting the rewrite by namespace. Re-running it is safe, and a conflict on a Secret that changed meanwhile just needs a retry.

Only now can key1 go: out of enc.yaml, then restart the API server. The config was not the only copy: enc.yaml.bak on the node has it too, and so do backups, the repo and that laptop. A key that leaked has to go everywhere you control, and the Secrets it protected should be treated as exposed. Rotate the passwords and tokens themselves, not just the key that wraps them. Secrets that rotate on their own, like Vault's dynamic database credentials, make that part of the job much smaller.

The order that avoids all of this

The official procedure ("Rotate a decryption key" in the docs page above):

  1. Add the new key as the second key of the current provider on every control plane node, and restart every kube-apiserver. Now all of them can decrypt it.
  2. Make a safe backup of the new key.
  3. Move the new key first and restart every kube-apiserver. New writes use it.
  4. kubectl get secrets --all-namespaces -o json | kubectl replace -f - to rewrite everything with the new key.
  5. Only after that, remove the old key and restart.

Step 1 exists because of the rolling restart. If the first API server to restart already writes with the new key, the ones not restarted yet cannot read what it writes. With a single control plane node, steps 1 and 3 can be one edit (keys: [new, old]). Step 5 too early is exactly the incident above.

If every copy of a key is lost, the docs are blunt: the Secrets written with it cannot be read, and the only way out is to delete those entries from etcd directly and recreate them. Back up the key first.

Two more things worth knowing

  • aescbc is the exam's provider, not production's. The docs mark aescbc as weak ("not recommended due to CBC's vulnerability to padding oracle attacks") and keep its key on the control plane host, next to the data. The production answer is the kms v2 provider (stable since Kubernetes v1.29): the key stays in an external KMS and rotation happens there.
  • Automatic reload (--encryption-provider-config-automatic-reload=true) makes the API server poll the file every minute, so key changes need no restart. The rotation order is the same: add, promote, rewrite, remove.

Practise it

The lab Incident: the key rotation that locked the Secrets in the Kubernetes Security (CKS) chapter starts exactly here: a cluster where key1 was replaced before the rewrite, a backup of the old config on the node, and Secrets on both keys. You get them readable again, finish the rotation and retire the leaked key. Before it, Encrypt the payments Secrets in etcd turns encryption at rest on from scratch: the config, the flag, the hostPath mount and the rewrite.

OnCallReady is free, with no ads and no tracking. RSS · All posts