OnCallReady

Lesson 26.18 · GitOps, Argo CD & Delivery · 7 min read

Promotion in a GitOps world

In plain words

Imagine a relay race. The baton is marked with a serial number. Runner one takes it around the dev track. Only when the coach has seen that runner finish cleanly does the same baton, same number, pass to the next runner on the prod track, and the handover is written in the race log with the number. Nobody gets a fresh baton "that looks the same".

Promotion in GitOps is that handover. The pipeline pushes the image once, then commits kustomize edit set image IMG@DIGEST to overlays/dev. For prod, a pull request copies the same digest to overlays/prod; the review is the approval, the merge is the deploy, and git log -- overlays/prod is the race log. Rollback is git revert.

The problem

In 25.11 you promoted an image by digest, but the pipeline did the deploying: it held cluster credentials and ran kubectl/helm itself. With Argo CD in the cluster, that is exactly what GitOps wants to take away. So what does the pipeline do now, and how does a build still get from dev to prod with a human approval in between?

What you need to know already: 26.1 (promotion = a commit that changes a digest), 26.2 (the images: entry), 25.4 and 25.7 (pipeline stages, secret variables), 25.11 (build once, promote by digest), 25.2 (pull requests, revert), 26.16 (sync windows).

The new pipeline

Build     test, build the image ONCE, push, record the digest          (unchanged)
Dev       clone the config repo, kustomize edit set image IMG@DIGEST,
          commit "dev: payments-api 1.5.1 @ sha256:...", push           (new)
          -> Argo CD syncs dev
Prod      a pull request that copies the same digest to overlays/prod,
          reviewed and merged by people                                 (new)
          -> Argo CD syncs prod

The Dev stage's one step:

- bash: |
    set -euo pipefail
    git clone -q "https://ci-bot:[email protected]/pay/payments-gitops.git" gitops
    cd gitops/overlays/dev
    kustomize edit set image $(IMAGE)@$(digest)
    git -c user.name=ci-bot -c [email protected] commit -qam "dev: payments-api $(VERSION) @ $(digest)"
    git push -q
  env:
    CONFIG_TOKEN: $(CONFIG_TOKEN)      # a secret variable: a token that can push to ONE repo

Line by line:

For Helm-based config repos the same step is yq -i '.image.digest = "..."' envs/dev/values.yaml (yq is jq for YAML, 7.11; -i edits the file in place).

What the pipeline lost: the Kubernetes@1 task, kubeconfigs, helm upgrade, and every permission on the cluster. What it needs instead: a token that can push to one repository (and in a strict setup, only to a bot branch that merges itself for dev).

Prod: the pull request is the gate

For prod the pipeline (or a bot) opens a pull request changing overlays/prod.

Tools that write these commits for you:

Rollback, freeze, audit

Order of environments

Dev is automatic on every main build. Staging and prod promote the digest that passed the previous environment - never "the latest build". Some teams make promotion a pipeline stage that opens the pull request only after dev is Healthy (argocd app wait payments-dev --health with a read-only Argo CD token), which keeps the chain honest without giving CI write access to the cluster.

What you can now do

Why it helps

This is what your pipelines will look like after the move to GitOps, and you will write them. The deploy step shrinks to cloning the config repo, editing one line and pushing, with a token that can push to one repo instead of a kubeconfig for prod. When an auditor asks who deployed 1.5.1 to prod and who approved it, the answer is the merged PR.

In an incident, "which build was running at 03:10?" is a grep for the digest in git log --format='%h %an %s' -- overlays/prod, which only works if commits carry the digest. When someone proposes Argo CD Image Updater for prod, you can explain why automatic promotion by tag is risky there and Renovate PRs fit better. Freezes via sync windows are a standard bank requirement.

FAQ

What does the pipeline still do in GitOps?

Everything up to the image: test, build once, scan, push, record the digest. Then, instead of helm upgrade, it clones the config repo, updates the digest with kustomize edit set image or yq, commits with the version and digest in the message, and pushes. For prod it opens a pull request. It no longer needs a kubeconfig or any cluster permission, only a token that can push to one repository.

Should promotion to prod be automatic?

Usually not in regulated environments. Dev is automatic on every main build. Prod is a pull request that copies a digest that already passed the previous environment, reviewed by CODEOWNERS, with a green check that renders the overlay and shows the diff. Some teams open the prod PR automatically only after argocd app wait payments-dev --health succeeds, but a human still merges it.

What is the difference between Argo CD Image Updater and Renovate?

Argo CD Image Updater watches a registry and, when a new tag matching a constraint appears, commits the new image to git or sets an Application parameter. It is fast and hands-off, good for dev, risky for prod. Renovate opens pull requests for new image tags, chart versions and other dependencies, with changelogs, so people review and merge them. It is the common choice for prod promotion and base-chart upgrades.

How do I freeze deployments in GitOps?

With sync windows on the AppProject: kind: deny, a cron schedule such as 0 16 * * 5 and a duration like 64 hours, matching *-prod applications. Argo CD then does not auto-sync those apps during the window, even if a PR is merged. manualSync controls whether manual emergency syncs are still possible. Branch protection can additionally block merges, but the sync window is what stops the cluster from changing.

Why keep the digest in the commit message and not just the version?

Because during an incident you work from what the cluster reports: pods show imageID with the digest, and registry and scanner findings reference digests. A version can in theory be re-pushed; a digest identifies exactly one build. git log -- overlays/prod | grep sha256:3f0d then tells you instantly when that build reached prod, who approved it and which commit removed it.

In an interview Mid

How do you promote a release from dev to prod with GitOps?

The pipeline stops deploying; it commits:

  1. Build stays the same: test, build the image once, push, record the digest.
  2. Dev: clone the config repo, kustomize edit set image IMG@DIGEST in overlays/dev (or yq -i on a values file), commit "dev: payments-api 1.5.1 @ sha256:...", push. Argo CD syncs dev.
  3. Prod: open a pull request changing the same digest line in overlays/prod - only after dev is healthy (argocd app wait payments-dev --health). The review is the approval, CODEOWNERS says who may approve, branch protection requires green checks (renders, argocd app diff --revision). Merging is deploying.

What the pipeline lost: kubeconfigs, helm upgrade, every cluster permission. What it needs: a token that can push to one repo.

Around it: rollback = git revert of the promotion commit; freezes = AppProject syncWindows; audit = git log -- overlays/prod, with the digest in every message. Tools like Renovate or Argo CD Image Updater can open these pull requests.

Also asked: What permissions does a pipeline need in a GitOps setup, and how do you minimise them? · A bad release reached production through GitOps. How do you roll back? · How do you enforce a change freeze with Argo CD?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.