Lesson 15.33 · Kubernetes: Architecture & Workloads · 16 min read

Secrets, base64, and the Downward API

Q: Why is my password rejected even though it looks right?

Probably a trailing newline. echo 'S3cr3t!pw' base64 encodes the newline too, so the stored value is the password plus \n. Use echo -n, or better avoid manual encoding: kubectl create secret generic --from-literal=... or stringData in the manifest encode exactly what you give them.

Q: Should secrets be consumed as env vars or files?

Prefer files. Environment variables leak into crash dumps, /proc/<pid/environ, child processes and debug endpoints that print the environment. Secret volumes are mounted on tmpfs (RAM), decoded, and can have restrictive modes like defaultMode: 0400. The same update rules apply as for ConfigMaps: env vars never change, mounted files update.

Q: Who can read a Secret?

Anyone with get on it, and anyone with list on secrets in the namespace, because list returns full objects. Also anyone who can create a pod in the namespace, since they can mount the Secret or reference it as env and read it from inside. So RBAC for secrets must consider pod creation rights too, and cluster-admins can read everything.

Q: Are Kubernetes Secrets secure?

Not by themselves. A Secret is a ConfigMap with better manners: values are base64-encoded, which is an encoding, not encryption - k get secret db-creds -o jsonpath='{.data.password}' base64 -d prints the password. They are kept on tmpfs on the node, and describe hides values. Where the protection really comes from: - Who may read them: anyone who can get or list Secrets in a namespace - or create a pod there - can read them. That is a permissions question. - Encryption at rest: without an EncryptionConfiguration (ideally backed by a KMS), etcd and its backups hold them as plain base64. - Not in git: a manifest with stringData is a plaintext password. - Files over env vars: env vars leak into crash dumps, /proc/<pid/environ and child processes. The careful pattern keeps them out of Kubernetes objects: a driver mounts secrets from an external store (a Key Vault) into the pod.

In plain words

Writing a secret message in mirror writing looks mysterious, but anyone with a mirror reads it instantly. That isn't a lock, it's just a different way of writing it down. The real protection is who is allowed into the room where the message is kept, and whether the message is kept in a proper safe.

A Kubernetes Secret stores values base64-encoded, which is mirror writing: base64 -d reveals them. Protection comes from RBAC (who can get or list secrets, or create pods that mount them), encryption at rest in etcd with a KMS provider, and, better, keeping the value in an external secret store and mounting it from there. Secret volumes live on tmpfs; files are safer than env vars. The Downward API lets a pod learn its own name, namespace, node and limits.

Why this matters

What you need to know already: ConfigMaps and how pods consume them (15.29), base64 (9.17), tmpfs and /proc/<pid>/environ (3.14, 5.3), TLS certificates (9.15), registries (10.54), jsonpath (15.38 uses it more).

Database passwords, API tokens and TLS keys must reach the app too - but you do not want them readable by everyone who can list config. Kubernetes has a separate kind for them, the Secret. The trap is believing the name: this lesson shows how little a Secret protects by itself, and how to handle it carefully.

A Secret is a ConfigMap with better manners

Same shape, same consumption modes (secretKeyRef, secretRef in envFrom, secret: volumes), with three differences: values are base64-encoded in data, there are typed variants, and the kubelet keeps secret volumes on tmpfs (RAM), never on the node's disk.

k create secret generic NAME --from-literal=K=V makes one (generic = the plain key/value type, Opaque):

$ k create secret generic db-creds --from-literal=username=orders --from-literal=password='S3cr3t!pw'
secret/db-creds created
$ k get secret db-creds -o yaml
apiVersion: v1
data:
  password: UzNjcjN0IXB3
  username: b3JkZXJz
kind: Secret
metadata:
  name: db-creds
  namespace: default
  ...
type: Opaque

base64 is not encryption

$ k get secret db-creds -o jsonpath='{.data.password}' | base64 -d
S3cr3t!pw

(-o jsonpath='{.data.password}' prints just that one field of the object; base64 -d decodes it.) That is all it takes. base64 is an encoding so binary data survives JSON; it provides no confidentiality at all. Anyone who can get the Secret can read it - and list returns full objects, so granting list on secrets is granting read on every secret in the namespace. Also anyone who can create a pod in the namespace can mount it. Permissions on who may read Secrets are where their protection really lives.

Later (Ch 17): RBAC, Kubernetes' permission system, decides who may get, list or mount Secrets.

describe is polite - it shows sizes, not values:

$ k describe secret db-creds
Name:        db-creds
Namespace:   default
Labels:      <none>
Annotations: <none>

Type:  Opaque

Data
====
password:  9 bytes
username:  6 bytes

The classic encoding bug:

$ echo 'S3cr3t!pw' | base64          # WRONG: echo adds a newline
UzNjcjN0IXB3Cg==
$ echo -n 'S3cr3t!pw' | base64       # right
UzNjcjN0IXB3

The first one decodes to the password plus \n, and the database login fails with a correct-looking password. --from-literal and stringData avoid it:

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
stringData:                # plain text in, stored base64 in data:
  username: orders
  password: S3cr3t!pw

stringData is write-only - reading the object back shows only data. And a manifest with stringData in git is a plaintext password in git.

Types

Opaque                                 arbitrary key/values (the default)
kubernetes.io/tls                      tls.crt + tls.key - a certificate and its key (used in Ch 16)
kubernetes.io/dockerconfigjson         registry credentials for imagePullSecrets
kubernetes.io/service-account-token    legacy long-lived SA tokens (avoid)
kubernetes.io/basic-auth, ssh-auth     conventions, validated keys

k create secret tls web-tls --cert=tls.crt --key=tls.key
k create secret docker-registry regcred --docker-server=registry.lab \
    --docker-username=ci --docker-password="$TOKEN"

(--cert/--key take the certificate and key files; --docker-server, --docker-username, --docker-password are the registry login.) A pod pulls from a private registry with that secret referenced in spec.imagePullSecrets: [{name: regcred}].

Consuming

env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-creds
      key: password
volumeMounts:
- name: creds
  mountPath: /etc/creds
  readOnly: true
volumes:
- name: creds
  secret:
    secretName: db-creds          # note: secretName, not name
    defaultMode: 0400

# app = a pod with the mounts above (the next mission builds one)
k exec app -- ls -la /etc/creds
lrwxrwxrwx    1 root     root         15 Sep 23 10:01 password -> ..data/password
lrwxrwxrwx    1 root     root         15 Sep 23 10:01 username -> ..data/username
k exec app -- cat /etc/creds/password
S3cr3t!pw

Files are decoded - the app reads plain text. Prefer files to env vars for secrets: env vars leak into crash dumps, /proc/<pid>/environ (3.14), child processes, and "print the environment" debug endpoints. The same env-vs-volume update rule applies as for ConfigMaps.

What a careful company actually does

Secrets in etcd (the cluster's database, 15.5) are only base64 unless you turn on encryption at rest: an EncryptionConfiguration file given to the API server (--encryption-provider-config), ideally backed by a KMS (key management service - a cloud key store that holds the encryption key), so etcd and its backups hold ciphertext.

Better still, keep the secret out of Kubernetes objects entirely: a driver mounts secrets straight from an external secret store (like the Key Vault of 12.3) into the pod as files. Rotation happens in the store; nothing sensitive sits in git or etcd. The other common pattern is an operator that copies secrets from the store into Secret objects.

immutable: true works for Secrets exactly as for ConfigMaps: no edits, recreate instead, and no kubelet watch.

The Downward API

The Downward API lets a container learn facts about its own pod - its name, namespace, node, IP - without calling the API server. Each fieldRef names a field of the pod object:

env:
- name: POD_NAME
  valueFrom:
    fieldRef:
      fieldPath: metadata.name
- name: POD_NAMESPACE
  valueFrom:
    fieldRef:
      fieldPath: metadata.namespace
- name: NODE_NAME
  valueFrom:
    fieldRef:
      fieldPath: spec.nodeName
- name: POD_IP
  valueFrom:
    fieldRef:
      fieldPath: status.podIP
- name: MEM_LIMIT
  valueFrom:
    resourceFieldRef:
      resource: limits.memory

As a volume (downwardAPI: with items), labels and annotations are available as files that update when the labels change. The logging library that tags every log line with pod and node name is the Downward API at work. (limits.memory is the container's memory limit - Ch 17.)

What you can now do:

Create, read and decode a Secret, and avoid the echo newline bug.
Mount a Secret as read-only files (preferred) or as env vars.
Explain why base64 is not protection, and pass pod facts with the Downward API.

Why it helps

Security reviews in a bank will ask where secrets live and who can read them, and "they're in Kubernetes Secrets, so they're encrypted" is the wrong answer you'll need to correct. Knowing that list on secrets means reading every secret in the namespace changes how you write RBAC. The echo without -n bug, a trailing newline in a base64-encoded password, causes baffling "correct password rejected" incidents. stringData in a git manifest is a plaintext password in git. And mounting secrets from an external store, with each app's own identity, is the pattern you'll likely implement at work. Secrets and the Downward API appear in exam tasks too.

Commands in this lesson

echo

FAQ

Is base64 encryption?

No. It's an encoding so that binary data survives JSON; anyone who can read the Secret decodes it with base64 -d. Confidentiality comes from RBAC on secrets, from encryption at rest in etcd (an EncryptionConfiguration with a KMS provider, a cloud key store), and from not storing the value in Kubernetes at all.

Why is my password rejected even though it looks right?

Probably a trailing newline. echo 'S3cr3t!pw' | base64 encodes the newline too, so the stored value is the password plus \n. Use echo -n, or better avoid manual encoding: kubectl create secret generic --from-literal=... or stringData in the manifest encode exactly what you give them.

Should secrets be consumed as env vars or files?

Prefer files. Environment variables leak into crash dumps, /proc/<pid>/environ, child processes and debug endpoints that print the environment. Secret volumes are mounted on tmpfs (RAM), decoded, and can have restrictive modes like defaultMode: 0400. The same update rules apply as for ConfigMaps: env vars never change, mounted files update.

Who can read a Secret?

Anyone with get on it, and anyone with list on secrets in the namespace, because list returns full objects. Also anyone who can create a pod in the namespace, since they can mount the Secret or reference it as env and read it from inside. So RBAC for secrets must consider pod creation rights too, and cluster-admins can read everything.

What is the Downward API?

A way for a pod to learn about itself without calling the apiserver: fieldRef exposes metadata like the pod name, namespace, node name and pod IP as env vars, and resourceFieldRef exposes its requests and limits, like limits.memory. As a downwardAPI volume, labels and annotations become files that update when they change. Useful for tagging logs with pod and node, or sizing a JVM heap from the memory limit.

In an interview Junior

Are Kubernetes Secrets secure?

Not by themselves. A Secret is a ConfigMap with better manners: values are base64-encoded, which is an encoding, not encryption - k get secret db-creds -o jsonpath='{.data.password}' | base64 -d prints the password. They are kept on tmpfs on the node, and describe hides values.

Where the protection really comes from:

Who may read them: anyone who can get or list Secrets in a namespace - or create a pod there - can read them. That is a permissions question.
Encryption at rest: without an EncryptionConfiguration (ideally backed by a KMS), etcd and its backups hold them as plain base64.
Not in git: a manifest with stringData is a plaintext password.
Files over env vars: env vars leak into crash dumps, /proc/<pid>/environ and child processes.

The careful pattern keeps them out of Kubernetes objects: a driver mounts secrets from an external store (a Key Vault) into the pod.

Also asked: Why is base64 not protection? · What is the classic bug when you base64-encode a password with echo? · What is the Downward API?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.