OnCallReady

Lesson 17.41 · Kubernetes: Scheduling, Health & Security · 10 min read

Secrets, and why base64 is not a security measure

In plain words

Imagine writing your diary in a secret code where A becomes B, B becomes C, and so on. It looks unreadable at a glance, but anyone who knows the trick (and everyone does) reads it in seconds. It keeps the page tidy, not private. Real privacy is a locked drawer, and who holds the key.

Kubernetes Secrets are base64-encoded: an encoding for binary data, not encryption. base64 -d reveals the value. Protecting a Secret means controlling every place it can be read: RBAC (who can get or list it, and who can create pods that mount it), etcd (plain text unless you enable encryption at rest), the node (mounted files) and the process (env vars). In a bank, the real source of truth is an external secrets manager (a vault service).

base64 is an encoding

The problem. Teams put database passwords and API keys into Kubernetes Secrets and think they are protected. They are only encoded. This lesson shows who can really read a Secret and what to do about each path.

What you need to know already: Secrets and base64 (15.33), RBAC and list (17.30), env vars in /proc/PID/environ (2.21), etcd (15.5), tmpfs (11.22).

kubectl create secret generic NAME --from-literal=key=value creates a Secret from the command line (generic = an ordinary key/value Secret):

# an illustration: pay/db-creds and psp-key-v7 (the Secrets mission)
$ kubectl create secret generic db-creds -n pay --from-literal=username=ledger --from-literal=password='s3cr3t-p@ssw0rd'
secret/db-creds created
$ kubectl get secret db-creds -n pay -o yaml
apiVersion: v1
data:
  password: czNjcjN0LXBAc3N3MHJk
  username: bGVkZ2Vy
kind: Secret
metadata:
  name: db-creds
  namespace: pay
type: Opaque
$ kubectl get secret db-creds -n pay -o jsonpath='{.data.password}' | base64 -d; echo
s3cr3t-p@ssw0rd

base64 exists so that binary data (TLS keys, keystores) survives JSON and YAML. It has no key; anyone who can read the object reads the secret. kubectl describe secret hides the values (password: 15 bytes) as a courtesy against shoulder-surfing, not as protection - -o yaml is one flag away.

So "who can read a Secret" is the whole question, and the answer has four layers:

  1. RBAC - who may get/list/watch secrets through the API. Remember list returns full objects (17.30), and edit includes secrets. Also: anyone who can create pods in the namespace can mount any secret in that namespace into a pod and read it - pod creation is secret access.
  2. etcd - by default secrets are stored in etcd in plain text (well, base64 inside protobuf, the binary format etcd stores). An etcd backup file, a stolen disk, or anyone with the etcd client certificate reads them all.
  3. the node - a mounted secret is a tmpfs file on the node; root on the node reads it.
  4. the process - an env var is visible in /proc/PID/environ (as in 2.21), in crash dumps, in "print all env" debug endpoints, and to every child process.

Encryption at rest

The apiserver can encrypt secrets before writing them to etcd. A KMS (Key Management Service) is an external service that holds the encryption key so it never sits on the node's disk; an HSM (hardware security module) is a tamper-proof device that stores keys:

# /etc/kubernetes/enc/encryption-config.yaml, passed as --encryption-provider-config
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: ["secrets"]
  providers:
  - kms:                      # production: an external KMS (a cloud key vault, an HSM) holds the key
      apiVersion: v2
      name: cloud-kms
      endpoint: unix:///var/run/kmsplugin/socket.sock
  - aescbc:                   # or a local key in the file (better than nothing, key on disk)
      keys:
      - name: key1
        secret: <32 random bytes, base64>
  - identity: {}              # read-only fallback so existing plaintext data stays readable

The first provider encrypts new writes; the others can decrypt. After enabling it, existing secrets are still plaintext until rewritten: kubectl get secrets -A -o json | kubectl replace -f - (read every Secret and write it back unchanged). You verify by reading etcd directly with etcdctl get /registry/secrets/... (etcdctl = etcd's own command-line client): plaintext shows the value, encrypted data starts with k8s:enc:aescbc:v1:key1: or k8s:enc:kms:v2:. Managed cloud clusters offer the same as a setting.

What to do instead, in a bank

Later (Ch 22): Azure Key Vault is the vault you will use, with pods authenticating through workload identity.

Two useful Secret features

apiVersion: v1
kind: Secret
metadata:
  name: psp-key-v7
immutable: true            # cannot be changed, only deleted: no accidental edits,
type: Opaque               # and the kubelet stops watching it (less apiserver load)
stringData:                # write plain text, the apiserver base64-encodes into data
  key: sk_live_4f9a2c77
# an illustration: pay/db-creds and psp-key-v7 (the Secrets mission)
kubectl edit secret psp-key-v7        # change the key, save
error: secrets "psp-key-v7" is invalid
The Secret "psp-key-v7" is invalid: data: Forbidden: field is immutable when `immutable` is set

Rotation with immutable secrets = create psp-key-v8, point the Deployment at it (a rollout), delete v7. Every change is visible and reversible.

What you can now do

Why it helps

"Secrets are just base64" is one of the first things a security auditor asks about, and on a banking platform you'll be designing the answer: etcd encryption with KMS, no human list secrets in production, an external vault as the source of truth fetched by the Secrets Store CSI driver with the pod's own identity, and Sealed Secrets or SOPS if Git must carry them.

You'll also catch the everyday mistakes in PR reviews: Secret manifests committed to Git, credentials in env vars that leak through /proc and debug endpoints, rotation that requires a restart because env vars are fixed at start. Knowing that pod creation is secret access changes how you design RBAC. And "how do you manage secrets in Kubernetes?" is a standard interview question with a clear strong answer.

Commands in this lesson

kubectl

FAQ

If Secrets are only base64, why use them instead of ConfigMaps?

Because they're treated differently where it matters: separate RBAC (you can give view without secrets access), encryption at rest can be enabled for them specifically, the kubelet stores mounted secrets on tmpfs rather than disk, and tooling hides their values in describe. They're a better container, not a safe. The real protection is access control and encryption.

Are Secrets encrypted in etcd?

Not by default: they're stored as base64 in protobuf, readable from an etcd backup or by anyone with the etcd client certificate. You enable encryption at rest with an EncryptionConfiguration on the API server, ideally with a KMS provider (an external key service or an HSM holds the key). Existing secrets stay plaintext until rewritten, for example with kubectl get secrets -A -o json | kubectl replace -f -.

Env var or mounted file for a secret?

Prefer a mounted file. Env vars are visible in /proc/PID/environ, crash dumps, "print all env" debug endpoints and to every child process, and they're fixed at container start. A mounted secret is updated by the kubelet within about a minute after the Secret changes (unless mounted with subPath), so rotation can work without a restart if the app re-reads it.

Can I store Secrets in Git with the rest of my YAML?

Not as plain Secret manifests: base64 in Git is plaintext in Git, forever, in every clone. Use Sealed Secrets (encrypted for the cluster's controller key) or SOPS-encrypted files (with a KMS key), which are decrypted only in the cluster. Or better, keep secrets in an external vault and sync them with the External Secrets Operator or mount them with the Secrets Store CSI driver.

What does immutable: true do on a Secret?

The Secret's data can't be changed, only the object deleted and recreated. It prevents accidental edits and lets the kubelet stop watching it, which reduces load on the API server in large clusters. Rotation becomes explicit: create key-v8, point the Deployment at it (a rollout), delete key-v7. Every change is visible and reversible.

In an interview Mid

How would you manage application secrets on a Kubernetes platform in a regulated environment?

First, who can read a Secret today - four layers:

  1. RBAC - get, list and watch on secrets (list returns full objects), and anyone who can create pods in the namespace can mount any secret there.
  2. etcd - without encryption at rest, secrets are stored as plain base64, in every backup.
  3. The node - root reads mounted secrets from tmpfs.
  4. The process - env vars leak through /proc/PID/environ, crash dumps and debug endpoints.

Then the controls:

Also asked: Are Kubernetes Secrets secure? · How do you enable encryption at rest for Secrets, and verify it works? · Why is permission to create pods also access to secrets?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.