OnCallReady

Lesson 25.11 · CI/CD Pipelines & Helm · 14 min read

Build once, deploy many: promotion by digest, environments and approvals

In plain words

Imagine a school science fair. The judges test one volcano model and put a gold sticker on it. Now the class wants to show it in the library and in the town hall. If they build a new volcano for each place "the same way", the judges' sticker is meaningless; the new ones were never tested. So they carry the very same model with the sticker to each place, and only the table cloth changes.

Promotion in CD works like that. The Build stage tests and pushes one image and records its digest, the sticker. Dev and Prod stages deploy that digest, with only values-dev or values-prod differing. The approval to enter prod is attached to the payments-prod environment, like a guard at the town hall door that no student can remove.

The problem

Lesson 25.1 said "build once, deploy many". This lesson builds a pipeline that does it: one image, recorded by digest, deployed to dev automatically and to prod after a human approves. First, the pipeline most teams start with, and why it is wrong.

What you need to know already: 25.1 (build once, digests), 25.4 (stages, jobs, tasks), 25.7 (output variables, service connections), 10.5 (tags and digests).

Words for this lesson

The wrong pipeline, first

- stage: Dev
  jobs:
  - job: deploy
    steps:
    - task: Docker@2                      # builds an image FOR dev
      inputs: { command: buildAndPush, repository: pay/payments-api, tags: dev }
- stage: Prod
  jobs:
  - job: deploy
    steps:
    - script: ./mvnw -B package -DskipTests      # rebuilds the jar
    - task: Docker@2                              # builds ANOTHER image for prod
      inputs: { command: buildAndPush, repository: pay/payments-api, tags: prod }

Everything tested in dev is thrown away at the Prod stage. The prod image is built from the same commit but not the same bytes: FROM eclipse-temurin:21-jre may resolve to a newer base image, Maven may resolve a newer transitive dependency (a library your library needs), and -DskipTests means the prod jar was never tested at all. When prod breaks and dev did not, nobody can tell whether it is the environment or the artifact.

The right shape

  Build stage:  test -> build image ONCE -> push -> record the DIGEST
  Dev stage:    deploy <digest> with values-dev      (automatic)
  Prod stage:   deploy <digest> with values-prod     (after an approval)
- stage: Build
  jobs:
  - job: build
    steps:
    - script: ./mvnw -B verify
    - task: Docker@2
      inputs:
        containerRegistry: registry-lab
        repository: pay/payments-api
        command: buildAndPush
        tags: |
          $(VERSION)
          $(Build.BuildId)
    - bash: |
        set -euo pipefail
        digest=$(docker inspect --format '{{index .RepoDigests 0}}' $(IMAGE):$(VERSION) | cut -d@ -f2)
        echo "##vso[task.setvariable variable=digest;isOutput=true]$digest"
      name: image

- stage: Prod
  jobs:
  - deployment: deploy
    environment: payments-prod           # the approval lives on the environment
    variables:
      digest: $[ stageDependencies.Build.build.outputs['image.digest'] ]
    strategy:
      runOnce:
        deploy:
          steps:
          - bash: helm upgrade --install payments-api ... --set image.digest=$(digest) --wait

Reading it: the Build stage pushes two tags, then the step named image asks Docker for the digest (docker inspect --format '{{index .RepoDigests 0}}' prints the first name@sha256:... the registry gave back; cut -d@ -f2 keeps the part after @) and publishes it as the output variable digest. The Prod stage reads it with stageDependencies (25.7) and passes it to helm upgrade --install - Helm, lesson 25.19, installs or updates the app's Kubernetes objects; --set image.digest= overrides one setting and --wait waits for ready pods.

docker inspect's RepoDigests gives registry.lab/pay/payments-api@sha256:... after a push. The digest travels as an output variable (or as a small artifact file like release.env, which also documents what was built). The Helm chart then renders image: registry.lab/pay/payments-api@sha256:... - a reference that cannot silently change.

Tags that help, tags that hurt

  1.5.0          semantic version, ONE build per version; registry refuses overwrites
  9c41e7a / 1041 commit or build id: unique, traceable back to the source
  latest, main,  MUTABLE (they move to a new image every build). Fine to find "the newest", never to deploy
  dev, prod, 1.5

registry.lab (like Azure Container Registry, ACR, with tag immutability enabled) refuses to overwrite a MAJOR.MINOR.PATCH tag:

unknown: The image tag '1.5.0' already exists in the 'pay/payments-api' repository and cannot be overwritten because the repository is immutable.

That error is a feature: it forces a version bump instead of a silent swap.

Deployment jobs and environments

- deployment: deploy            # instead of `job:`
  environment: payments-prod    # created on first use; history of what ran where
  strategy:
    runOnce:                    # deploy once; other strategies exist for VMs
      deploy:
        steps: [...]

A deployment job:

strategy: runOnce: deploy: steps: is the fixed nesting a deployment job needs: run the deploy steps once. In the output below, ci show shows the run parked at the gate, ci approve RUN approves it, and ci env NAME shows the environment's checks and deployment history.

# a run waiting at the Prod gate, as in the next mission
ci show
  ...
  ⏸ Stage Prod   Waiting for approval: environment 'payments-prod' (ci approve 1041)
      · deploy  -> environment payments-prod   pending
ci approve 1041
Approved: stage 'Prod' of run #1041 (environment payments-prod)
ci env payments-prod
Environment payments-prod
  Checks: Approvals (Learner)
  Deployments:
    run #1041 20260924.3 9c41e7a 2026-09-24T10:14:02.000Z

The approval is attached to the environment, not to the YAML: whoever edits the pipeline cannot remove it. That is the property auditors care about.

How to promote without rebuilding, in one sentence

Build and push once, capture the digest, and have every later stage reference that digest, with only the values file differing per environment.

What you can now do

Why it helps

This is the most common design question in CD interviews and the root cause of a whole class of incidents: "it worked in test" when prod runs a rebuild. When you review a pipeline that runs docker build in its Prod stage, or mvn package -DskipTests, you now have the argument to reject it.

In your daily work, the pieces here are what you will wire: capturing the digest after buildAndPush, passing it as an output variable, helm upgrade --set image.digest=$(digest) --wait, and an environment with approvals and an exclusive lock. Immutable tags mean the error the image tag '1.5.0' already exists becomes a useful signal to bump the version. And auditors care that the approval lives on the environment, where a YAML edit cannot remove it.

FAQ

What is the difference between a deployment job and a normal job?

A deployment: job targets an environment. It records each deployment in that environment's history (which run, which commit), runs the environment's checks before starting (approvals, branch control, business hours, exclusive lock), does not check out the repo unless you add checkout: self, and downloads the run's pipeline artifacts automatically. It uses a strategy such as runOnce, rolling or canary. A normal job has none of this.

How do I get the digest after pushing an image?

After docker push, the output line 1.5.0: digest: sha256:... size: ... contains it, and docker inspect --format '{{index .RepoDigests 0}}' image:tag returns registry/repo@sha256:.... With BuildKit, docker buildx build --push --metadata-file meta.json writes it to a file. Capture it once in the build stage and pass it forward as an output variable or a small artifact such as release.env.

Is it fine to deploy the latest tag to dev?

It is common but it costs traceability: you cannot tell from the manifest what is running, and Kubernetes will not roll pods when the tag content changes because the pod spec did not change. If you deploy by digest everywhere, dev behaves like prod and a rollback is exact. Mutable tags like latest, main or dev are useful to find the newest image, not to deploy it.

Why put the approval on the environment rather than in the YAML?

Anything in the YAML can be edited by anyone who can change the pipeline file, including removing the approval. Checks on an environment are configured in Azure DevOps by its administrators and apply to every pipeline that deploys to it. That separation of duties is what auditors look for: the person who writes the deploy cannot also waive its gate. GitLab has the same idea with protected environments.

Do environments need different images if they need different config?

No. Configuration that differs per environment belongs in values files, ConfigMaps, Secrets and environment variables, not baked into the image. The same payments-api@sha256:... runs with values-dev.yaml and values-prod.yaml. If an image contains an environment's URLs or credentials, it can only be promoted by rebuilding, which is exactly what build once, deploy many forbids. This is the "config in the environment" rule from the twelve-factor app.

In an interview Mid

How do you pass the built image from the build stage to the deployment stages, and gate production?

Build once, carry the digest:

  1. The Build stage pushes the image once and captures its digest: docker inspect --format '{{index .RepoDigests 0}}', publishing it with ##vso[task.setvariable variable=digest;isOutput=true] from a step with a name:.
  2. Later stages read it with stageDependencies.Build.build.outputs['image.digest'] (a $[ ] runtime expression) and deploy --set image.digest=$(digest), so the chart renders image: ...@sha256:... - a reference that cannot silently change. Only the values file differs per environment.

Gate prod with an environment: a deployment job (deployment: + environment: payments-prod) records what was deployed where and runs the environment's checks first - approvals, branch must be main, business hours, an exclusive lock so two runs cannot deploy at once. The approval lives on the environment, not in the YAML, so whoever edits the pipeline cannot remove it - which is what auditors care about.

Mutable tags (latest, main, prod) are fine for finding the newest image, never for deploying.

Also asked: What are mutable and immutable image tags, and why does it matter for deployments? · Why should a pipeline not rebuild the image for each environment? · Why do approvals belong on the environment rather than in the pipeline YAML?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.