A Secret is only base64 - in etcd too
By default the apiserver stores Secrets in etcd as plain protobuf: anyone with an etcd backup or the etcd client certificates reads every password. Prove it on cp-1:
sudo ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key \
get /registry/secrets/payments/db-creds
The value is in the output in clear. Encryption at rest makes the apiserver encrypt before writing to etcd:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <32 random bytes, base64>
- identity: {}
The order matters. The first provider encrypts every new write; all of them are tried for reads. identity last means "can still read the old, unencrypted ones". identity first would mean no encryption at all. aescbc/aesgcm/secretbox keep the key in the file on the node; kms keeps it in an external key service (the production answer).
Point the apiserver at it: --encryption-provider-config=/etc/kubernetes/enc/enc.yaml plus a hostPath volume and a volumeMount for /etc/kubernetes/enc. The apiserver reads the file when it starts and refuses to start with a bad one: a key that is not 16, 24 or 32 bytes, a typo in a provider name, a file it cannot see. crictl logs on cp-1 shows the reason, e.g. err="error while parsing file: resources[0].providers[0].aescbc.keys[0].secret: Invalid value: \"REDACTED\": secret is not of the expected length, got 6, expected one of [16 24 32]". Now etcd shows k8s:enc:aescbc:v1:key1: and ciphertext.
The trap: only Secrets written after the change are encrypted. Rewrite the old ones:
kubectl get secrets -A -o json | kubectl replace -f -
Rotating the key
A key leaks (a laptop with the repo is stolen, an engineer leaves) and has to change. The apiserver reads the config only when it starts (unless --encryption-provider-config-automatic-reload=true), so every step is "edit the file, restart the apiserver". The official order, for a cluster with several control plane nodes:
- Add the new key as the second key of the provider, on every control plane node; restart every apiserver. Now all of them can read the new key.
- Make a safe backup of the new key.
- Move the new key first; restart every apiserver. New writes use it.
- Rewrite everything:
kubectl get secrets -A -o json | kubectl replace -f -. - Only then remove the old key; restart.
Why step 1 exists: with three apiservers restarted one by one, the first one up would write with key2 while the other two cannot read key2 yet. With one control plane node (this lab, most exam clusters) steps 1 and 3 can be one edit: keys: [key2, key1].
The trap is step 5 too early. Remove key1 before the rewrite and every Secret still on key1 is unreadable - for kubectl, for the kubelet starting pods, for the rewrite itself:
$ kubectl get secret db-creds -n payments
Error from server (InternalError): Internal error occurred: [no matching key was found for the provided AES transformer, identity transformer tried to read encrypted data]
$ kubectl get secrets -n payments
Error from server (InternalError): Internal error occurred: unable to transform key "/registry/secrets/payments/api-token": [no matching key was found for the provided AES transformer, identity transformer tried to read encrypted data]
Each provider is tried in order: aescbc finds the k8s:enc:aescbc:v1: prefix but no key named key1, identity matches anything and refuses encrypted data, and the apiserver reports both. The fix is to put key1 back as a second key (from wherever a copy survived), restart, rewrite, then remove it. Lose every copy of a key and the Secrets written with it are gone: delete and recreate them.
Turning it off is the same dance: identity first, the aescbc block after it (so the old data still reads), restart, rewrite.
In an interview: "base64 is encoding, not encryption; encryption at rest protects etcd and its backups, RBAC protects the API - you need both."
On the exam: domain "Cluster setup / Minimize microservice vulnerabilities": create the EncryptionConfiguration, wire it in, re-encrypt, verify with etcdctl.