OnCallReady

Lesson 26.22 · GitOps, Argo CD & Delivery · 13 min read

Secrets when git is the source of truth

In plain words

Imagine you want to leave a secret note for a friend in the class post box, which everyone can open. One way: you lock the note in a small box, and only your friend's key opens it; anyone can see the box, nobody else can read the note. Another way: you leave a note that just says "ask the teacher for the paper in drawer 7", and the teacher only hands it to your friend.

Sealed Secrets is the locked box: kubeseal encrypts with the controller's public key, the SealedSecret goes into git, and only the controller in the cluster can open it into a normal Secret. External Secrets is the note pointing to the drawer: an ExternalSecret in git names a key in Azure Key Vault, and the operator fetches the value into a Secret.

The problem

In GitOps everything the cluster runs comes from git - including the database password. But a Kubernetes Secret manifest (15.33) only base64-encodes its values, and encoding is not encryption:

$ echo 'U3VtbWVyMjAyNiE=' | base64 -d
Summer2026!

(base64 -d decodes; anyone can run it.) Commit that to a config repo and every clone, fork, CI cache and backup has the password forever - rewriting git history does not reach the copies. You still want the existence of the secret, its name and its keys in git; only the value must not be readable there. There are two families of answers.

What you need to know already: 15.33 and 17.41 (Secrets, base64), 26.2 (overlays, patches), 26.4 (Argo CD applies what is in git), 22.23 (Key Vault), 22.12 (workload identity: a pod's ServiceAccount signs in to Azure without a password), 25.24 (Helm's checksum annotation).

Sealed Secrets: encrypt it for the cluster

Sealed Secrets is a controller (sealed-secrets-controller in kube-system) plus a command, kubeseal. It uses asymmetric encryption: a key pair where what the public key encrypts only the matching private key can decrypt. The controller keeps the private key; anyone can have the public one. So anyone can seal, and only that controller can unseal:

kubectl create secret generic payments-db -n pay-dev \
  --from-literal=DB_PASSWORD="$(cat pw.txt)" --dry-run=client -o yaml \
  | kubeseal --format yaml > overlays/dev/payments-db-sealed.yaml

Piece by piece:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: payments-db
  namespace: pay-dev
spec:
  encryptedData:
    DB_PASSWORD: AgBy3i4OJSWK+PiTySYZZA1rO43cGDEq...     # ciphertext: useless without the private key
  template:
    metadata: {name: payments-db, namespace: pay-dev}   # what the resulting Secret looks like

The SealedSecret goes into git; Argo CD applies it; the controller decrypts it into a normal Secret it owns. kubeseal --fetch-cert > pub.pem saves the public key (as a certificate) so people and CI can seal offline with --cert pub.pem.

Sealing scope

Scope - where the ciphertext may be opened - is baked into the encryption:

strict (default)   bound to namespace AND name: copy it elsewhere and it does not open
namespace-wide     any name, same namespace   (kubeseal --scope namespace-wide)
cluster-wide       anywhere                   (kubeseal --scope cluster-wide)

A strict SealedSecret copied to another namespace fails with no key could decrypt secret (DB_PASSWORD) - on purpose: otherwise anyone who can create a SealedSecret in their own namespace could copy prod's there and read it.

Operational facts:

External Secrets Operator: keep the value outside

External Secrets Operator (ESO) leaves the value in a real secret manager (Azure Key Vault from 22.23, AWS Secrets Manager, HashiCorp Vault...) and copies it into a Kubernetes Secret. An operator is a controller that manages some outside system or app for you. Git only holds a pointer:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore              # platform-owned: HOW to reach the vault
metadata: {name: azure-kv}
spec:
  provider:
    azurekv:
      vaultUrl: https://kv-payments-lab.vault.azure.net/
      authType: WorkloadIdentity      # ESO signs in to Azure as its ServiceAccount (22.12)
      serviceAccountRef: {name: external-secrets, namespace: external-secrets}
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret                  # team-owned: WHAT to fetch, into which Secret
metadata: {name: payments-api-key, namespace: pay-dev}
spec:
  refreshInterval: 1m                 # re-read the vault every minute
  secretStoreRef: {kind: ClusterSecretStore, name: azure-kv}
  target: {name: payments-api-key}    # the Kubernetes Secret to create
  data:
    - secretKey: API_KEY              # key in the Kubernetes Secret
      remoteRef: {key: payments-api-key}   # name of the secret in Key Vault

Rotation happens in Key Vault (az keyvault secret set ..., 22.23) and reaches the cluster within refreshInterval. Access control is the vault's (RBAC on the vault, workload identity for the operator - chapter 22), the audit trail is the vault's logs, and a leaked git repo contains no secrets at all.

Neither restarts your pods

Both write a Secret. Environment variables from a Secret are read when the container starts, so a rotated value reaches running pods only on restart. (Files from a mounted Secret volume update in place after a minute or so - if the app re-reads them.) Options:

Others you will meet

Choosing

Sealed Secrets   simple, no external dependency, value lives in git (encrypted);
                 rotation = re-seal + commit; the sealing key is a crown jewel
ESO + Key Vault  the value never touches git, central rotation and audit, needs the
                 vault and identity plumbing; the standard in Azure-heavy banks

What you can now do

Why it helps

Every GitOps adoption hits the secrets question in week one, and the wrong answer, base64 Secrets in git, is a breach. You will have to choose between Sealed Secrets and External Secrets with Key Vault for your team, set up the ClusterSecretStore with workload identity, and explain rotation to developers.

The operational details are where incidents come from. A strict SealedSecret copied to another namespace fails with no key could decrypt secret. Losing the sealing keys with a cluster makes every SealedSecret in git garbage, so key backup is part of disaster recovery. And a rotated value in Key Vault does not reach running pods that read it as an env var until they restart, the classic "we rotated the password and the app broke an hour later". Interviewers ask about all three.

Commands in this lesson

echo

FAQ

Isn't a Kubernetes Secret already encrypted?

No. The values in a Secret manifest are base64-encoded, which anyone can decode with base64 -d. Inside the cluster, Secrets can be encrypted at rest in etcd if the cluster is configured for it, and access is controlled by RBAC, but a Secret manifest in git is effectively plain text. That is why GitOps needs Sealed Secrets, SOPS, or a pointer to an external secret manager.

What is the difference between Sealed Secrets and External Secrets Operator?

Sealed Secrets keeps the value in git, encrypted with the cluster controller's public key; only that controller can decrypt it. It is simple and self-contained, but rotation means re-sealing and committing, and the sealing key is critical. External Secrets keeps the value in a secret manager like Azure Key Vault and syncs it into a Secret; git holds only a reference. Rotation, access control and audit happen in the vault.

Why can't I copy a SealedSecret to another namespace?

The default scope is strict: the ciphertext is bound to the Secret's namespace and name. Decrypting it elsewhere fails with no key could decrypt secret. That is deliberate, otherwise anyone able to create a SealedSecret in their own namespace could replay a prod secret there. Use kubeseal --scope namespace-wide or cluster-wide if you really need portability, and prefer re-sealing per namespace.

Why didn't my pods pick up the rotated secret?

Environment variables from a Secret are read only when the container starts, so running pods keep the old value until they restart. Mounted Secret volumes update in place after a short delay, but only help if the application re-reads the file. Options: restart the Deployment after rotation, use a controller like Stakater Reloader that rolls workloads when a referenced Secret changes, or have the app read secrets from files and reload them.

What happens if the Sealed Secrets controller's keys are lost?

Every SealedSecret in git becomes unreadable, because only the matching private keys can decrypt them. A rebuilt cluster with a new controller generates new keys and cannot open the old ciphertexts. So back up the key Secrets in kube-system (label sealedsecrets.bitnami.com/sealed-secrets-key) securely, outside the cluster, and include restoring them in the disaster recovery runbook. Otherwise you must re-seal every secret from the original values.

In an interview Mid

How do you manage secrets in a GitOps workflow?

Not as a Secret manifest in git: its values are only base64 (base64 -d reads them), and every clone, fork and backup keeps the password forever. Git should hold the secret's existence, name and keys - never a readable value. Two families:

Either way, pods using env vars need a restart (or Reloader) to see a rotated value.

Also asked: Compare Sealed Secrets, SOPS and External Secrets Operator. · Why does a SealedSecret copied to another namespace fail to decrypt? · A secret was rotated in Key Vault but the pods still use the old value. Why?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.