Lesson 25.1 · CI/CD Pipelines & Helm · 19 min read

Delivery: from a commit to a running pod

Q: What is the difference between an image tag and an image digest?

A tag is a movable name like payments-api:1.5.0; anyone with push rights can point it at different content tomorrow unless the registry forbids overwriting. A digest is a SHA-256 hash of the image manifest, so it is derived from the content and can never point anywhere else. Humans use tags to find images; deployments should use digests, or at least verify that the deployed digest matches the one CI produced.

Q: Where do I see the digest a pod is actually running?

In the pod status: kubectl get pod -o jsonpath='{.items[].status.containerStatuses[0].imageID}'. That field is what the kubelet resolved and pulled, in the form registry/repo@sha256:.... The image: field in the spec only shows what was requested, which may be a tag. The push step of the CI log prints the digest it uploaded, so the two can be compared directly.

Q: What is "ci" in the lab, and will I find it on a real system?

No, ci is a simulator command that stands in for the Azure DevOps and GitLab web UI: queue a run, read logs, approve a stage. Everything the pipeline runs is real on the box, as the user azagent, with real git, docker, helm and kubectl. On a real system you would do the same things in the browser or with az pipelines and glab CLIs.

Q: How would you prove that production runs exactly the image that passed testing?

By building it once and promoting the same bytes: - Build once, deploy many: CI builds and tests one image; every later environment deploys that image, and only configuration (values: replicas, URLs, flags) differs. A prod stage that runs docker build again ships an untested artifact - a newer base layer, a different dependency resolved that afternoon. - Identify it by digest, not by tag. A tag like payments-api:1.5.0 is a name that can be moved; the digest (sha256:...) is computed from the content and cannot change. Tag with version and commit for humans, make release tags immutable in the registry, deploy by digest. - Check it: docker push in the CI log prints the digest; kubectl get pod -n pay-prod -l app.kubernetes.io/name=payments-api -o jsonpath='{..imageID}' shows what the node actually pulled. Same digest = same artifact. Add the audit trail: the deploy is a commit in the config repo ("prod: 1.4.1 - 1.5.0"), reviewed and revertible.

In plain words

Imagine a school photo. The photographer takes one picture, prints it, and every parent gets a copy of that same print. Some frame it in wood, some in plastic, some hang it in the kitchen. The frames differ; the photo does not. If one parent got a retaken photo "because the kid looked the same", it would not be the photo everyone approved.

In delivery the photo is the container image, and its fingerprint is the digest sha256:3f0d.... The frames are the values files for dev, test and prod. CI takes the photo once; CD hangs it in each environment. The tag 1.5.0 is only the name written on the back, and names can be rewritten; the digest cannot.

The problem

A developer merges a fix. Somewhere between that merge and customers getting the fix, a lot happens: code is compiled, tests run, an image is built, pushed, and started in three environments. When it is done by hand, someone forgets a step, builds prod from a different commit, or cannot say afterwards what is actually running. This chapter is about automating that path - and knowing what state everything is in when it breaks halfway.

What you need to know already: 1.17 (git: commits, push), 10.5 (images, tags and digests), 10.54 (registries), 15.1 (kubectl), 15.16 (Deployments and rolling updates).

Words for this lesson

Delivery - the whole path from "code is merged" to "it runs for users".
Environment - one place the app runs: dev (developers try things), test (checks before release), prod (production: real users). Each is, for us, usually its own Kubernetes namespace or cluster.
Artifact - the thing a build produces and you ship. Here: a container image (Ch 10).
Pipeline - an automated list of steps a server runs for you on every change: build, test, package, deploy. The next lessons build one.

You already ship code at work through pipelines someone else built. This chapter is about building and owning that path yourself, and about the questions an interviewer asks to find out whether you really understand it:

What exactly goes to production, and how do you know it is the thing that was tested?
What happens between git push and a new pod serving traffic?
When it breaks halfway, what state is everything in, and how do you get out?

The answer to all three is the same discipline: build one artifact that never changes, move ("promote") that same artifact from environment to environment, and let every environment differ only in configuration.

CI and CD are different jobs

CI = continuous integration: every change is built and tested automatically, as soon as it is pushed. CD = continuous delivery (or deployment): the artifact CI produced is put into the environments.

  CI  (continuous integration)          CD  (continuous delivery / deployment)
  ------------------------------------  -----------------------------------------
  every push, every merge request       after CI produced an artifact
  compile, unit test, lint, scan        put THAT artifact into an environment
  produce ONE artifact: an image        dev -> test -> prod, with gates between
  fast, cheap, no secrets for prod      slow, controlled, audited
  fails -> the commit is bad            fails -> the release is bad; roll back

Reading the table: a merge request (GitLab's name; GitHub says pull request, PR) is a proposal to merge a branch into main that others review. Lint = a tool that checks code style and obvious mistakes without running it (like shellcheck in Ch 6). Scan = a tool that looks for known security holes in the dependencies. A gate is a check that must pass before the next environment, often a human approval.

Continuous delivery means every good build could go to production at the press of a button (an approval). Continuous deployment means it does, with no human in between. Banks almost always do delivery: a human (or a change ticket) approves prod.

Build once, deploy many

The single most important rule of the whole chapter:

   commit 9c41e7a
        |
        v
   CI: ./mvnw verify  ->  docker build  ->  registry.lab/pay/payments-api:1.5.0
                                             digest sha256:3f0d...a91c   <- THE artifact
        |
        +--> dev   runs sha256:3f0d...a91c  with values-dev
        +--> test  runs sha256:3f0d...a91c  with values-test
        +--> prod  runs sha256:3f0d...a91c  with values-prod

Read it top to bottom: one commit is built once (./mvnw verify compiles and tests the Java code - Maven, Ch 20; docker build makes the image, Ch 10). The image is pushed to the registry. Then every environment runs that same image. The "values" files are the per-environment settings (the Helm lessons later in this chapter explain the name).

What differs between environments is configuration: replica counts, URLs, feature flags (on/off switches for features, read by the app at start), the names of the secrets to use. What never differs is the bytes of the image. If your prod stage runs docker build again, prod gets an image that was never tested: a newer base image layer, a different dependency downloaded that afternoon, a different compiler. "It passed in test" is then a statement about a different artifact.

Rebuilding per environment is the anti-pattern the Notion question asks about. The fix is promotion: move the same digest forward.

Three names for one build

  1.5.0                      the version a human reads. Semantic: MAJOR.MINOR.PATCH
  9c41e7a                    the commit it was built from (git rev-parse --short HEAD)
  sha256:3f0d8e...a91c       the image digest: a hash of the image manifest itself

Semantic version (semver) - a version number MAJOR.MINOR.PATCH; the next lesson explains when each part goes up.
git rev-parse --short HEAD prints the short id of the commit you are on.
The digest (Ch 10.5) is computed from the image's content, so the same bytes always give the same digest.

Tags are names, and names can be moved. payments-api:1.5.0 can be pushed again tomorrow with different content unless the registry refuses it. A digest cannot change: it is computed from the content. So:

tag images with the version and the commit (1.5.0, 9c41e7a) so humans and tools can find them;
make release tags immutable (the registry refuses to overwrite them) - registry.lab refuses to overwrite a MAJOR.MINOR.PATCH tag, you will see the error;
deploy by digest, or at least check that the digest you deploy is the digest CI produced.

Here is how you compare the two. docker push prints the digest the registry stored; kubectl get pod ... -o jsonpath=... prints one field of the pods (Ch 15.38): -n pay-prod the namespace, -l app.kubernetes.io/name=payments-api only pods with that label, and the jsonpath picks imageID, the digest the node actually pulled.

# what CI printed, and what prod runs (registry.lab and pay-prod are not on this box)
docker push registry.lab/pay/payments-api:1.5.0
...
1.5.0: digest: sha256:3f0d8e21c9b04aa1f7e6f0e7b3f4a2d5c8e1f9a07b6c3d2e1f0a9b8c7d6e5a91c size: 1560
kubectl -n pay-prod get pod -l app.kubernetes.io/name=payments-api \
    -o jsonpath='{.items[*].status.containerStatuses[0].imageID}'
registry.lab/pay/payments-api@sha256:3f0d8e21c9b04aa1f7e6f0e7b3f4a2d5c8e1f9a07b6c3d2e1f0a9b8c7d6e5a91c

imageID is what the kubelet actually pulled. If it does not match the digest in the CI log, prod is not running what CI built.

App repo and config repo

A repo (repository) is one git project. Delivery usually uses two:

  git.lab/pay/payments-api          git.lab/pay/payments-config
  ------------------------          ----------------------------
  source code, tests                the Helm chart (charts/payments-api)
  Dockerfile                        envs/dev/values.yaml
  azure-pipelines.yml (CI)          envs/prod/values.yaml
  owned by the developers           owned by whoever may change prod
  changes every hour                changes on every deploy - and that IS the deploy

The app repo holds the code and the recipe to build it (azure-pipelines.yml is the pipeline file - lesson 25.4).
The config repo holds what runs where: which version, how many replicas, which settings, per environment. Helm is a tool that installs a set of Kubernetes YAML files as one unit, filling in per-environment numbers; a chart is such a packaged set. Lessons 25.19 onwards teach it; for now, "the chart" is just "the Kubernetes YAML for this app".

Why two repos: different people may approve changes to each; a deploy becomes a reviewable commit ("prod: 1.4.1 -> 1.5.0") with its own history and git revert; and CI on the app repo does not need credentials to prod.

Later (Ch 26): GitOps takes the config repo one step further: the pipeline only commits to it, and a tool in the cluster makes the cluster match it.

The lab

  oncall-lab (you)                      the lab network
  ------------------                      ---------------
  git, helm, kubectl                      git.lab        GitLab-like server: repos, protected branches
  ci      (simulator)                     registry.lab   image + OCI chart registry (immutable semver tags)
  azagent (the pipeline agent's user)     charts.lab     Helm chart repository of the platform team
                                          10.64.0.10  the kubeadm cluster from chapter 15

git.lab - a git server like GitLab or GitHub: it stores the repos and enforces rules such as "nobody may force-push to main".
registry.lab - where images (and, later, Helm charts) are pushed. OCI = the Open Container Initiative, the standard format for images; an "OCI registry" can store anything packaged in that format.
charts.lab - a web server that hands out Helm charts (lesson 25.29).
azagent - the Linux user the pipeline runs as (lesson 25.4).

ci is the one thing that does not exist on a real box. It stands in for the Azure DevOps (and GitLab) web UI: start a run, read the logs, approve a stage. Everything the pipeline does is real: it runs your YAML on this machine as the user azagent, with real git, docker, helm and kubectl.

ci help lists its subcommands; ci pipelines lists the pipelines the server knows, which repo and file each one reads, and when it last ran:

$ ci help
ci - the lab's CI/CD server (simulator: stands in for the Azure DevOps and GitLab web UI)
...
$ ci pipelines
PIPELINE      KIND             REPO                      FILE                 LAST RUN
payments-api  Azure Pipelines  git.lab/pay/payments-api  azure-pipelines.yml  -

What this chapter covers, in order

  1. git as the ledger of every change (branches, rebase, revert, tags)
  2. Azure Pipelines: triggers, stages, jobs, steps, variables, secrets, templates
  3. promotion: images by digest, environments with approvals
  4. GitLab CI equivalents (and what Jenkins looks like)
  5. Helm: charts, values, templates, hooks, releases and their failure states

What you can now do

Say what CI and CD each do, and the difference between continuous delivery and deployment.
Explain "build once, deploy many" and why a digest, not a tag, identifies what runs.
Tell an app repo from a config repo.

Why it helps

The first question in any post-release incident is "is prod running what we tested?" This lesson gives you the three names of a build (version, commit, digest) and the one command that answers it: the pod's imageID compared with the digest in the CI log. If they differ, you stop debugging the code and start debugging the pipeline.

It also sets up the rest of the chapter. The split between the app repo and the config repo explains why a deploy is a commit, and why CI does not need prod credentials. And "what is the difference between CI and CD, and what goes to prod" is asked in nearly every DevOps interview; answering with digests and promotion instead of "Jenkins builds it" is what separates you.

FAQ

What is the difference between an image tag and an image digest?

A tag is a movable name like payments-api:1.5.0; anyone with push rights can point it at different content tomorrow unless the registry forbids overwriting. A digest is a SHA-256 hash of the image manifest, so it is derived from the content and can never point anywhere else. Humans use tags to find images; deployments should use digests, or at least verify that the deployed digest matches the one CI produced.

Where do I see the digest a pod is actually running?

In the pod status: kubectl get pod -o jsonpath='{.items[*].status.containerStatuses[0].imageID}'. That field is what the kubelet resolved and pulled, in the form registry/repo@sha256:.... The image: field in the spec only shows what was requested, which may be a tag. The push step of the CI log prints the digest it uploaded, so the two can be compared directly.

Why keep the Helm chart and values in a separate config repo?

Different people approve changes to each: developers own the code, whoever may change prod owns the config. A deploy becomes a small reviewable commit ("prod: 1.4.1 -> 1.5.0") with its own history and a clean git revert. And CI on the app repo never needs credentials for prod. Some teams keep the chart in the app repo and only the values in the config repo; the principle is the same.

What is "ci" in the lab, and will I find it on a real system?

No, ci is a simulator command that stands in for the Azure DevOps and GitLab web UI: queue a run, read logs, approve a stage. Everything the pipeline runs is real on the box, as the user azagent, with real git, docker, helm and kubectl. On a real system you would do the same things in the browser or with az pipelines and glab CLIs.

Is a semantic version still useful if I deploy by digest?

Yes. People and changelogs speak in versions: "1.5.0 added refunds" means something, a hash does not. The version also drives the chart's appVersion, release notes and the tag in git. The digest is for machines: it guarantees identity. Good practice is to tag images with the version and the commit, make release tags immutable, and record the digest that each environment runs.

In an interview Mid

How would you prove that production runs exactly the image that passed testing?

By building it once and promoting the same bytes:

Build once, deploy many: CI builds and tests one image; every later environment deploys that image, and only configuration (values: replicas, URLs, flags) differs. A prod stage that runs docker build again ships an untested artifact - a newer base layer, a different dependency resolved that afternoon.
Identify it by digest, not by tag. A tag like payments-api:1.5.0 is a name that can be moved; the digest (sha256:...) is computed from the content and cannot change. Tag with version and commit for humans, make release tags immutable in the registry, deploy by digest.
Check it: docker push in the CI log prints the digest; kubectl get pod -n pay-prod -l app.kubernetes.io/name=payments-api -o jsonpath='{..imageID}' shows what the node actually pulled. Same digest = same artifact.

Add the audit trail: the deploy is a commit in the config repo ("prod: 1.4.1 -> 1.5.0"), reviewed and revertible.

Also asked: What is the difference between CI and CD? · What is the difference between continuous delivery and continuous deployment? · Why would you separate the application repository from the deployment configuration repository?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.