OnCallReady

Lesson 26.2 · GitOps, Argo CD & Delivery · 12 min read

kustomize: a base and overlays, without templates

In plain words

Imagine a school uniform. Every class wears the same base: shirt and trousers. Class 4A adds a red badge, class 4B wears shorts in summer. Nobody redraws the whole uniform per class; each class has a small note, "same as the base, plus a badge", and you can still wear the base uniform on its own.

kustomize works like that. base/ holds plain, valid Kubernetes YAML you could apply directly. Each overlays/<env>/kustomization.yaml says "take the base, then change these things": namespace, replicas, image digest, a patch that raises the memory limit. There is no template language. kubectl kustomize overlays/prod shows the finished uniform; the configMapGenerator adds a content hash to the ConfigMap name so a config change rolls the pods.

The problem

The config repo (26.1) must say "payments runs with 1 replica in dev and 3 in prod, prod has more memory, both run image 1.5.0". Copying the whole Deployment YAML into a dev folder and a prod folder works for a week, then the copies drift apart and nobody knows which differences are on purpose. You want one shared description plus a short list of what each environment changes.

What you need to know already: 26.1 (config repo, environments as directories), 15.16 (Deployments), 15.29 (ConfigMaps, envFrom), 15.40 (kubectl apply, declarative YAML), 25.19 and 25.24 (Helm charts and templates, for the comparison at the end), 10.5 (image tags and digests).

Patching instead of templating

Helm (25.19) solves this with templates: YAML with {{ }} holes, filled from values. kustomize does it differently. It takes plain, valid Kubernetes YAML and applies transformations to it - edits such as "set the namespace", "change the replicas", "merge in this partial object". There is no template language.

kustomize is built into kubectl: kubectl kustomize DIR prints the result, kubectl apply -k DIR (-k = "this is a kustomize directory") applies it. Argo CD recognises a directory with a kustomization.yaml by itself.

A base

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:                      # the files that make up the base
  - deployment.yaml
  - service.yaml
configMapGenerator:             # build a ConfigMap from these key=value pairs
  - name: payments-config
    literals:
      - SPRING_PROFILES_ACTIVE=default
      - PAYMENTS_GATEWAY_TIMEOUT=2s

An overlay

# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: pay-prod            # sets metadata.namespace on everything
resources:
  - ../../base                 # a directory with its own kustomization.yaml
replicas:
  - name: payments-api         # the Deployment's name
    count: 3
images:
  - name: registry.lab/pay/payments-api    # matches this image NAME in any container
    newTag: "1.5.0"                        # or digest: sha256:...  (or newName: to change the name)
configMapGenerator:
  - name: payments-config
    behavior: merge            # add to / override keys of the base's generator of the same name
    literals:
      - SPRING_PROFILES_ACTIVE=prod
patches:
  - path: memory.yaml          # a strategic-merge patch, below

A patch

A patch is a partial object: only the fields you want to change, plus enough to identify the object (kind + name). kustomize merges it into the base.

# overlays/prod/memory.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  template:
    spec:
      containers:
        - name: payments-api   # lists of containers merge by NAME
          resources:
            limits:
              memory: 768Mi

This kind is called a strategic merge patch: Kubernetes-aware merging, where a list like containers is matched item by item by its name field instead of being replaced whole.

kubectl kustomize overlays/prod prints the finished YAML; kubectl apply -k overlays/prod applies it. Always read the output before applying: it is the only place the whole, final object exists.

The transformers you will use

A transformer is one kind of edit kustomize can make:

namespace: x                 put every object into namespace x (and fix references like RoleBinding subjects)
namePrefix: / nameSuffix:    rename every object - and every reference to it
labels:                      add labels (includeSelectors: true also changes selectors - careful)
images:                      change an image's name/tag/digest without a patch
replicas:                    change replicas of a Deployment/StatefulSet by name
patches:                     strategic merge (a partial object, as above) or JSON 6902:
  - target: {kind: Deployment, name: payments-api}
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/imagePullPolicy
        value: IfNotPresent

A JSON 6902 patch (named after RFC 6902, the standard that defines it) is a list of operations - add, replace, remove - each on a path into the object. /containers/0/ means "the first container": items are addressed by index (position). That breaks silently when someone adds a container to the base and the positions move. Prefer strategic merge unless you must delete or replace a list item.

Generators and the hash

A generator creates an object for you. configMapGenerator and secretGenerator create a ConfigMap or Secret whose name gets a hash of its content appended: payments-config-7k2fdm9t4c (a hash is a short fingerprint computed from the data; different data, different hash).

kustomize rewrites every reference to it (envFrom, volumes, valueFrom) to the hashed name. So change one literal and:

  1. the ConfigMap's name changes,
  2. so the Deployment's pod template changes (it names the ConfigMap),
  3. so the Deployment rolls its pods (15.16) - they pick up the new config.

Plain ConfigMaps do not do that: pods keep the old env values until restarted (15.29). This is the same effect as Helm's checksum annotation (25.24), for free. The old ConfigMap stays in the cluster until something prunes it (deletes what is no longer in git) - Argo CD with prune: true does, 26.9. generatorOptions: {disableNameSuffixHash: true} turns the hash off - and with it the automatic roll.

secretGenerator exists, but a Secret generated from a literal in git is a password in git. Lesson 26.22 is about what to do instead.

Helm or kustomize?

helm        packaging and distribution (versioned charts, repos, OCI), parameters with
            defaults, logic (if/range), hooks, a release history. Best for things you
            SHIP to others: platform base charts, third-party software.
kustomize   plain YAML, environment differences as patches, no release state.
            Best for YOUR OWN environments of a service.

They combine. Argo CD can render a Helm chart with a values file per environment, and kustomize can render a chart itself (helmCharts: in kustomization.yaml, run with --enable-helm) and then patch the output - the usual answer when a third-party chart lacks one setting you need. In bank-style platforms you see both: a Helm base chart owned by the platform team, and per-service overlays or values files in a config repo.

What you can now do

Why it helps

Your own services' environment differences are usually small: a namespace, a replica count, a digest, a memory limit. kustomize expresses exactly that, and Argo CD detects it automatically. When you review a promotion PR, the change is one digest: line in overlays/prod. When a pod does not roll after a config change, you check whether someone set disableNameSuffixHash: true. When a JSON 6902 patch suddenly edits the wrong container after a sidecar was added to the base, you know index-based paths are the reason.

"Helm or kustomize?" is a very common interview question, and the good answer is "both, for different jobs": Helm for things you ship to others, kustomize for your own environments, and kustomize inflating a third-party chart when it lacks a knob.

FAQ

What is the difference between Helm and kustomize?

Helm renders Go templates from values; it has packaging (versioned charts, repositories, OCI), logic, hooks and release history. kustomize has no templates: it takes plain YAML and applies transformations and patches per environment, with no release state. Helm fits software you ship to others, like a platform base chart; kustomize fits your own service's environments. They combine: kustomize can inflate a Helm chart with helmCharts: and patch the result.

Why does the ConfigMap name have a random-looking suffix?

configMapGenerator and secretGenerator append a hash of the content, like payments-config-7k2fdm9t4c, and rewrite every reference to it. Change a value and the name changes, so the Deployment's pod template changes and the pods roll, like Helm's checksum annotation. Old ConfigMaps remain until pruned, for example by Argo CD with prune: true. Disabling the hash also disables the automatic rollout.

When should I use a strategic merge patch versus a JSON 6902 patch?

Prefer strategic merge: you write a partial object identified by kind and name, and lists like containers merge by name, so it keeps working when the base changes. JSON 6902 patches are operations on paths such as /spec/template/spec/containers/0/..., addressing list items by index; they break silently when a container is added. Use 6902 only when you must remove or replace a specific list element.

Is kustomize the same as kubectl apply -k?

kustomize is built into kubectl: kubectl kustomize dir renders, kubectl apply -k dir renders and applies. The standalone kustomize binary is usually newer and has extras like kustomize edit set image, which pipelines use to update a digest. The embedded version can lag the standalone one by a few releases, so a feature documented for the latest kustomize may not exist in your kubectl.

What does the images transformer do?

It changes the image of every container whose image name matches, without a patch: images: - name: registry.lab/pay/payments-api with newTag, newName or digest. It applies across Deployments, StatefulSets and other workloads. kustomize edit set image registry.lab/pay/payments-api@sha256:... rewrites that entry, which is why a promotion commit in a kustomize config repo is a single-line diff.

In an interview Mid

What is kustomize, and how do you choose between it and Helm?

kustomize (built into kubectl) takes plain, valid Kubernetes YAML and applies transformations - no template language:

kubectl kustomize overlays/prod prints the finished YAML - read it before kubectl apply -k. A configMapGenerator appends a content hash to the ConfigMap's name, so a config change renames it, changes the pod template, and rolls the pods.

Choosing: Helm for things you ship to others - versioned charts in repos, parameters with defaults, logic, hooks, a release history (platform base charts, third-party software). kustomize for your own environments of a service - plain YAML and per-environment patches. They combine: Argo CD renders a chart with per-environment values, or kustomize renders a chart and patches the one setting it lacks.

Also asked: What is a strategic merge patch, and when would you use a JSON 6902 patch instead? · How does a configMapGenerator make pods restart when the config changes? · A kustomize overlay patch stopped working after the base changed. How do you debug it?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.