OnCallReadyStart learning

Security and secrets Lesson 23.4 · Kubernetes Security (CKS) · 9 min read

Encryption at rest: Secrets in etcd

In plain words

Secrets in Kubernetes are like letters in a filing cabinet with no lock: base64 only changes the handwriting, anyone who opens the drawer can read them. Encryption at rest puts a lock on the cabinet. The API server encrypts each Secret before it goes into etcd and decrypts it again when an authorised user asks, so a stolen copy of the cabinet is useless without the key.

A Secret is only base64 - in etcd too

By default the apiserver stores Secrets in etcd as plain protobuf: anyone with an etcd backup or the etcd client certificates reads every password. Prove it on cp-1:

sudo ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/payments/db-creds

The value is in the output in clear. Encryption at rest makes the apiserver encrypt before writing to etcd:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <32 random bytes, base64>
      - identity: {}

The order matters. The first provider encrypts every new write; all of them are tried for reads. identity last means "can still read the old, unencrypted ones". identity first would mean no encryption at all. aescbc/aesgcm/secretbox keep the key in the file on the node; kms keeps it in an external key service (the production answer).

Point the apiserver at it: --encryption-provider-config=/etc/kubernetes/enc/enc.yaml plus a hostPath volume and a volumeMount for /etc/kubernetes/enc. The apiserver reads the file when it starts and refuses to start with a bad one: a key that is not 16, 24 or 32 bytes, a typo in a provider name, a file it cannot see. crictl logs on cp-1 shows the reason, e.g. err="error while parsing file: resources[0].providers[0].aescbc.keys[0].secret: Invalid value: \"REDACTED\": secret is not of the expected length, got 6, expected one of [16 24 32]". Now etcd shows k8s:enc:aescbc:v1:key1: and ciphertext.

The trap: only Secrets written after the change are encrypted. Rewrite the old ones:

kubectl get secrets -A -o json | kubectl replace -f -

Rotating the key

A key leaks (a laptop with the repo is stolen, an engineer leaves) and has to change. The apiserver reads the config only when it starts (unless --encryption-provider-config-automatic-reload=true), so every step is "edit the file, restart the apiserver". The official order, for a cluster with several control plane nodes:

  1. Add the new key as the second key of the provider, on every control plane node; restart every apiserver. Now all of them can read the new key.
  2. Make a safe backup of the new key.
  3. Move the new key first; restart every apiserver. New writes use it.
  4. Rewrite everything: kubectl get secrets -A -o json | kubectl replace -f -.
  5. Only then remove the old key; restart.

Why step 1 exists: with three apiservers restarted one by one, the first one up would write with key2 while the other two cannot read key2 yet. With one control plane node (this lab, most exam clusters) steps 1 and 3 can be one edit: keys: [key2, key1].

The trap is step 5 too early. Remove key1 before the rewrite and every Secret still on key1 is unreadable - for kubectl, for the kubelet starting pods, for the rewrite itself:

$ kubectl get secret db-creds -n payments
Error from server (InternalError): Internal error occurred: [no matching key was found for the provided AES transformer, identity transformer tried to read encrypted data]
$ kubectl get secrets -n payments
Error from server (InternalError): Internal error occurred: unable to transform key "/registry/secrets/payments/api-token": [no matching key was found for the provided AES transformer, identity transformer tried to read encrypted data]

Each provider is tried in order: aescbc finds the k8s:enc:aescbc:v1: prefix but no key named key1, identity matches anything and refuses encrypted data, and the apiserver reports both. The fix is to put key1 back as a second key (from wherever a copy survived), restart, rewrite, then remove it. Lose every copy of a key and the Secrets written with it are gone: delete and recreate them.

Turning it off is the same dance: identity first, the aescbc block after it (so the old data still reads), restart, rewrite.

In an interview: "base64 is encoding, not encryption; encryption at rest protects etcd and its backups, RBAC protects the API - you need both."

On the exam: domain "Cluster setup / Minimize microservice vulnerabilities": create the EncryptionConfiguration, wire it in, re-encrypt, verify with etcdctl.

Why it helps

Anyone with an etcd backup, a disk snapshot or the etcd client certificates can read every unencrypted Secret in the cluster. Encryption at rest closes that gap, and it is a CKS favourite because of one trap: Secrets that existed before you turned it on stay unencrypted until something rewrites them. Knowing how to verify it at the storage layer is the part most people miss.

On the blog

Commands in this lesson

kubectl

FAQ

Is base64 encryption?

No. It is an encoding that anyone can reverse in one command, so a Secret in etcd is readable unless encryption at rest is turned on. RBAC protects the API, but it does nothing for someone who reads etcd or its backups directly.

Which provider encrypts new Secrets?

The first one in the EncryptionConfiguration's providers list. The other providers are only tried when reading, which lets the apiserver still read data written with an older key or without encryption. Order the list deliberately: writer first, readers after it.

Why put identity last?

So the apiserver can still read Secrets written before encryption was turned on. Putting identity first would mean every new write is stored unencrypted, so the configuration would look right and protect nothing. Remove identity only after every Secret has been rewritten.

How do I encrypt the Secrets that already exist?

Rewrite them through the API: kubectl get secrets -A -o json | kubectl replace -f -. Each Secret is written again, and the write goes through the first provider, so afterwards the old ones are encrypted too.

How do I prove it worked?

Read the key in etcd with etcdctl get on the control plane node: the value must start with k8s:enc:aescbc:v1:key1: or your provider. Checking with kubectl proves nothing, because the apiserver decrypts for you. Check one old and one new Secret to be sure both are encrypted.

In an interview Mid

Are Kubernetes Secrets encrypted?

Not by default - they are base64-encoded and stored in etcd as they are. With an EncryptionConfiguration passed to the apiserver through --encryption-provider-config plus a mount, the first provider (aescbc, or kms in production) encrypts every new write; existing Secrets need rewriting with kubectl replace, and you verify with etcdctl get, where the value starts with k8s:enc:.

Also asked: Why does the provider order matter? · How do you rotate the encryption key? · Why prefer a kms provider in production?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.