OnCallReady

Lesson 15.29 · Kubernetes: Architecture & Workloads · 19 min read

ConfigMaps: env vs volumes, and what happens when you change one

In plain words

Imagine a toy robot that reads instructions two ways. Some instructions are written on a card you slot in when you switch it on; it reads that card once, and if you change the card later, it doesn't notice until you switch it off and on again. Other instructions are on a whiteboard it looks at every few minutes; change the whiteboard and it notices soon.

A ConfigMap holds configuration for pods. Consumed as environment variables (env with configMapKeyRef, or envFrom), it's the card: read once at container start. Mounted as a volume, it's the whiteboard: the kubelet updates the files within about a minute, through an atomic ..data symlink swap, except subPath mounts, which never update. Changing a ConfigMap never restarts pods; rollout restart, a checksum annotation, or versioned names make changes take effect.

Why this matters

What you need to know already: environment variables (6.1, 2.21), Docker volumes and bind mounts (11.22), symlinks (4.13), pods and Deployments (15.14, 15.16), rollouts (15.16).

The same image runs in dev and prod; only its settings differ - log level, greeting text, a config file. Baking settings into the image means rebuilding it for every change. Kubernetes keeps them in a separate object, a ConfigMap (a named set of key/value pairs, each value a string or a whole file), and hands it to the container. The surprise this lesson is about: how you hand it over decides whether a change ever reaches the running app.

Creating one

k create configmap NAME (short: cm) builds one from the command line: --from-literal=KEY=VALUE adds one key, --from-file=FILE adds a file's content under its file name (or KEY=FILE to rename it), --from-env-file=FILE reads KEY=VALUE lines.

k create configmap app-config --from-literal=LOG_LEVEL=info --from-literal=GREETING=Hello
k create configmap nginx-conf --from-file=default.conf          # key = file name
k create configmap nginx-conf --from-file=site.conf=default.conf # key = site.conf
k create configmap env-config --from-env-file=app.env           # KEY=VALUE lines
$ k get cm app-config -o yaml
apiVersion: v1
data:
  GREETING: Hello
  LOG_LEVEL: info
kind: ConfigMap
metadata:
  name: app-config
  namespace: default
  ...

The object has no spec: just data, a map of key to string. Values are always strings. In YAML that means quoting anything that looks like something else: PORT: "8080", DEBUG: "true" - an unquoted 8080 is a number and the apiserver rejects the ConfigMap (cannot unmarshal number into Go struct field ConfigMap.data of type string). Max 1 MiB per ConfigMap.

Consuming it, four ways

In the pod template: env + valueFrom.configMapKeyRef copies one key into an environment variable; envFrom copies all keys; a volume of type configMap turns keys into files, mounted with volumeMounts at a path in the container (like docker run -v, 11.22). subPath mounts a single file instead of a directory.

spec:
  containers:
  - name: app
    image: registry.lab/shop/greeter:1.0
    env:
    - name: GREETING                    # 1. one key as one variable
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: GREETING
    envFrom:
    - configMapRef:                     # 2. every key as a variable
        name: app-config
      prefix: APP_                      #    (optional: APP_GREETING, APP_LOG_LEVEL)
    volumeMounts:
    - name: cfg
      mountPath: /etc/greeter           # 3. every key as a file in a directory
    - name: cfg
      mountPath: /etc/nginx/conf.d/default.conf
      subPath: default.conf             # 4. one key as one file, via subPath
  volumes:
  - name: cfg
    configMap:
      name: app-config
      items:                            # optional: only these keys, renamed
      - key: GREETING
        path: greeting.txt

The critical difference

Environment variables are resolved once, when the container starts. They are copied into the process's environment by the kubelet - like Environment= in a systemd unit (2.21), read once at start. Change the ConfigMap and a running container never sees it - not in a minute, not in a day - until the container is restarted.

Mounted files are updated in place. The kubelet periodically re-syncs ConfigMap volumes (its sync loop plus the object cache - in practice up to about a minute) and swaps the files atomically. The app sees the new content the next time it reads the file.

You can watch both with the lab's greeter app, which reads GREETING from its environment once at startup and re-reads /etc/greeter/message.txt every 10 seconds:

(k logs deploy/greeter --tail=2 prints the last 2 log lines of one of the Deployment's pods; k patch cm NAME -p 'JSON' changes fields in place; k exec deploy/greeter -- CMD runs a command in one of its pods.)

# greeter is what the next mission builds
k logs deploy/greeter --tail=2
2026-09-23T10:00:16Z greeting(env)="Hello" message(file)="first message"
k patch cm greeter-config -p '{"data":{"GREETING":"Salut","message.txt":"second message"}}'
configmap/greeter-config patched
k logs deploy/greeter --tail=2          # about a minute later
2026-09-23T10:01:16Z greeting(env)="Hello" message(file)="second message"

The file changed; the env var did not. Inside the container:

k exec deploy/greeter -- printenv GREETING
Hello
k exec deploy/greeter -- cat /etc/greeter/message.txt
second message

How the atomic swap works - look at the directory:

# greeter is what the next mission builds
k exec deploy/greeter -- ls -la /etc/greeter
total 12
drwxr-xr-x    2 root     root       4096 Sep 23 10:01 .
drwxr-xr-x    2 root     root       4096 Sep 23 10:01 ..
drwxr-xr-x    2 root     root       4096 Sep 23 10:01 ..2026_09_23_10_01_12.201312345
lrwxrwxrwx    1 root     root         31 Sep 23 10:01 ..data -> ..2026_09_23_10_01_12.201312345
lrwxrwxrwx    1 root     root         18 Sep 23 10:01 message.txt -> ..data/message.txt

message.txt is a symlink to ..data/message.txt; ..data is a symlink to a timestamped directory. An update writes a new timestamped directory and flips ..data in one rename - the app never reads a half-written file. It also means inotify watches on the file itself break on update (inotify is the Linux "tell me when this file changes" facility); watch the directory.

Two consequences people learn in production:

The rollout problem

Changing a ConfigMap never restarts pods. Nothing in Kubernetes links "this ConfigMap changed" to "roll this Deployment". Three ways to make a change take effect:

1. Restart the rollout by hand

k rollout restart deploy/greeter

Adds kubectl.kubernetes.io/restartedAt to the pod template - a template change - so the Deployment rolls every pod, respecting maxUnavailable.

2. A checksum annotation in the template (the pattern deploy tools automate)

spec:
  template:
    metadata:
      annotations:
        checksum/config: 3f7a9c...    # sha256 of the ConfigMap's content
sha256sum greeter-config.yaml        # recompute on every change

The config change changes the checksum, which changes the template, which rolls the Deployment - automatically, through the normal rollout machinery.

Later (Ch 25): Helm, a templating tool for manifests, computes this checksum for you on every deploy.

3. Versioned names

greeter-config-v7 -> greeter-config-v8, and the Deployment references the new name. Also a template change, and the old ConfigMap stays around for rollback. Some manifest tools generate such names for you, with a hash of the content as the suffix.

immutable

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-v8
immutable: true
data: {...}

Immutable ConfigMaps cannot be edited (data: Forbidden: field is immutable when immutable is set) - only deleted and recreated - and the kubelet stops watching them (a watch is a long-lived request asking the API server to push every change). On a cluster with thousands of pods that removes a large share of the API server's load. It pairs naturally with versioned names.

Missing ConfigMaps

A pod that references a ConfigMap that does not exist does not start:

as a volume:     Warning  FailedMount  MountVolume.SetUp failed for volume "cfg" : configmap "app-config" not found
                 -> STATUS ContainerCreating, forever
as env:          Warning  Failed  Error: configmap "app-config" not found
                 -> STATUS CreateContainerConfigError
missing key:     Error: couldn't find key DB_URL in ConfigMap default/app-config

optional: true on the reference makes it start anyway with the variable unset or the directory empty.

What you can now do:

Why it helps

"I changed the ConfigMap and nothing happened" is one of the most common Kubernetes support questions, and you'll answer it for developers regularly: env vars are frozen at start, subPath mounts never update, and the app must re-read files. Knowing the checksum/config annotation pattern and hash-suffixed ConfigMap names means config changes roll out through the normal, safe rollout machinery instead of someone manually restarting pods. You'll also debug pods stuck in ContainerCreating (FailedMount, missing ConfigMap as a volume) or CreateContainerConfigError (missing as env). And immutable: true is a real apiserver load reduction on large clusters.

FAQ

Why doesn't my app see the new ConfigMap value?

If it's consumed as an environment variable, it was copied into the process at container start and never changes; restart the pods. If it's a mounted file, the kubelet updates it within about a minute, but only if it's not a subPath mount, and only if the app re-reads the file. Many apps read config once at startup, making the volume no better than env.

Does changing a ConfigMap restart my pods?

No. Nothing links a ConfigMap change to a Deployment rollout. Use kubectl rollout restart deploy/NAME, put a checksum of the config in a pod template annotation, or use versioned ConfigMap names that the Deployment references. The last two make config changes go through the normal rollout, with its safety and rollback.

Why do my inotify watches break when the ConfigMap updates?

Mounted files are symlinks: message.txt points to ..data/message.txt, and ..data points to a timestamped directory. An update writes a new directory and atomically flips ..data. The file you watched is never modified in place, so watch the directory instead. The swap guarantees the app never reads a half-written file.

Why was my ConfigMap rejected with "cannot unmarshal number"?

ConfigMap values must be strings. In YAML, an unquoted 8080 or true is a number or boolean, so the apiserver rejects it. Quote them: PORT: "8080", DEBUG: "true". The maximum size of a ConfigMap is 1 MiB.

What happens if the ConfigMap a pod references doesn't exist?

The pod doesn't start. As a volume, you get FailedMount ... configmap not found and the pod stays ContainerCreating. As env, you get CreateContainerConfigError. A missing key gives "couldn't find key". Adding optional: true to the reference lets the pod start with the variable unset or the directory empty.

In an interview Junior

What is a ConfigMap, and how can a pod use it?

A ConfigMap is a named set of key/value pairs - each value a string or a whole file - kept outside the image, so the same image runs in dev and prod: k create configmap app-config --from-literal=LOG_LEVEL=info --from-file=default.conf.

A pod uses it as:

The critical difference: env vars are read once, at container start - a change never reaches a running container. Mounted files update in place within about a minute (an atomic swap of the ..data symlink) - but subPath mounts never update, and the app must re-read the file.

Changing a ConfigMap never restarts pods. To roll it out: k rollout restart deploy/NAME, a checksum annotation in the pod template, or a versioned ConfigMap name. A missing ConfigMap leaves the pod in CreateContainerConfigError or stuck mounting.

Also asked: How do you roll out a configuration change safely? · Why does a ConfigMap mounted with subPath not update? · What does immutable: true do on a ConfigMap?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.