OnCallReady

Lesson 25.9 · CI/CD Pipelines & Helm · 17 min read

Templates, parameters, matrix builds and caching

In plain words

Imagine every class in a school writing its own fire drill instructions. After a year they are all slightly different, and two forget the step "close the windows". Better: the head teacher writes one drill sheet with blanks ("your room number: ___") and every class fills in only the blanks. Change the sheet once, and every class gets the fix.

Pipeline templates are that sheet: a YAML file with parameters: that other pipelines include with template: or extends:, expanded before the run starts. A matrix is the same drill run for several rooms at once (JDK 17 and 21). A cache is keeping the Maven downloads in the cupboard between drills, keyed on the hash of pom.xml, so you only fetch what changed.

The problem

Ten services with the same "build, test, scan, push" stages means ten copies of the same YAML drifting apart: someone fixes a bug in one copy and nine keep it. And every run downloads the same hundreds of Maven dependencies again. This lesson is about writing the pipeline once (templates), running one job in several variants (a matrix), and not downloading what you already have (a cache).

What you need to know already: 25.4 (stages, jobs, steps, the agent's workspace), 25.7 (the three syntaxes: ${{ }} at compile time, $( ) and $[ ] later).

Templates

A template (pipeline sense) is a YAML file with parameters: and one of steps:, jobs:, stages: or variables:: a reusable piece of pipeline. A parameter is an input the caller fills in, like a function argument. The template is pasted in at compile time, before the run starts:

# templates/maven-build.yml
parameters:
- name: goals
  type: string
  default: verify
- name: jdk
  type: string
  default: '21'
  values: ['17', '21']          # compile-time validation

steps:
- script: |
    set -euo pipefail
    ./mvnw -B ${{ parameters.goals }}
  displayName: Maven ${{ parameters.goals }} (JDK ${{ parameters.jdk }})
# azure-pipelines.yml
steps:
- template: templates/maven-build.yml
  parameters:
    goals: package

Reading the two files: the template declares two parameters, each with a type and a default (used when the caller gives nothing); values: lists the only allowed values. Inside, ${{ parameters.goals }} is replaced by the caller's value. The pipeline uses it with - template: FILE plus the parameters: it wants to set - here goals: package, so the step becomes ./mvnw -B package. jdk keeps its default '21'.

Parameter types: string, number, boolean, object, step, stepList, job, jobList, stage, stageList. A missing parameter without a default, a wrong type or a value outside values: fails the run before it starts:

/templates/maven-build.yml: The 'jdk' parameter value '11' is not a valid value.

Compile-time logic

steps:
- ${{ if eq(parameters.publish, true) }}:
  - publish: target
    artifact: jar
- ${{ each env in parameters.environments }}:
  - script: echo "deploy to ${{ env }}"

${{ if COND }}: keeps the indented lines under it only when COND is true (here: publish the jar only if the publish parameter is true). ${{ each X in LIST }}: repeats the lines under it once per item (one deploy step per environment in the list).

${{ if }} and ${{ each }} are evaluated while compiling - they shape the pipeline. They only see parameters and variables written in the YAML, not runtime values like $(Build.BuildId) or a variable set by a script. For runtime decisions use condition: on the step or job.

Sharing templates across repos

A platform team keeps shared templates in their own repo. resources: repositories: declares that other repo under a short alias (templates); ref: says which branch or tag to read. extends: then says "this whole pipeline IS that template, with these parameters"; FILE@templates means FILE from the repo with alias templates.

resources:
  repositories:
  - repository: templates
    type: git
    name: platform/pipeline-templates
    ref: refs/tags/v3            # pin a version of the shared templates

extends:
  template: java-service.yml@templates     # the whole pipeline comes from the platform team
  parameters:
    serviceName: payments-api

extends is how platform teams enforce standards: the service's pipeline can only fill in parameters; the scan and approval stages are not optional. Pin the template repo to a tag - a change to main of the templates repo otherwise changes every pipeline at once.

Matrix builds

A matrix runs the same job several times with different variables - here once per Java version (JDK = Java Development Kit, Ch 20). maxParallel caps how many copies run at the same time.

- job: build
  strategy:
    matrix:
      jdk17:
        JDK: '17'
      jdk21:
        JDK: '21'
    maxParallel: 2
  steps:
  - script: echo "building on JDK $(JDK)"

One job definition becomes one job per matrix entry, named build_jdk17 and build_jdk21, each with its variables. They run in parallel if there are enough agents - with one self-hosted agent they queue one after the other. Matrix values are runtime variables: $(JDK) works, ${{ variables.JDK }} does not (it is empty at compile time), and passing $(JDK) into a template parameter that has values: fails the compile-time check because the parameter literally receives the text $(JDK).

Caching

A cache (pipeline sense) is a directory the pipeline saves at the end of a job and restores at the start of a later one, so the dependencies Maven downloaded (into .m2/repository, its local store) do not have to be downloaded again. An artifact is what you ship; a cache is only a speed-up, and the job must still work without it.

Cache@2 inputs: key names the cache, restoreKeys are fall-backs, path is the directory to save and restore. -Dmaven.repo.local=DIR tells Maven to use DIR as its local store, so it uses the cached one.

- task: Cache@2
  inputs:
    key: 'maven | "$(Agent.OS)" | pom.xml'
    restoreKeys: |
      maven | "$(Agent.OS)"
    path: $(Pipeline.Workspace)/.m2/repository
- script: ./mvnw -B verify -Dmaven.repo.local=$(Pipeline.Workspace)/.m2/repository

The key is a list of segments joined by |: literal strings, and file paths or globs whose content hash (a fingerprint of the file: change one byte and it changes) becomes part of the key. Change pom.xml and the key changes: a cache miss, a full download, and a new cache saved at the end of the job (a post-job step: one the task adds after your last step). restoreKeys are prefixes tried when the exact key misses: you get the closest older cache and Maven only downloads what changed.

Resolving key:
 - maven [string]
 - "Linux" [string]
 - pom.xml [file pattern; matches: 1]
   - s/pom.xml --> CFA97E63D1ED9B5FD5FF1B6613BED54500AB41DE
Resolved to: maven|"Linux"|NkE3RUIy...=
...
There is a cache miss.
...
##[section]Starting: Post-job: Cache the Maven repository
...
Cache saved successfully

The log shows the key being worked out from its three parts, then the miss, then the save at the end.

Rules: cache dependencies, never build outputs you then ship (that is what artifacts are for); a cache is saved only when the job succeeds; caches are scoped per branch with fallback to the default branch.

Self-hosted agents and "clean"

On a self-hosted agent $(Pipeline.Workspace) survives between runs, so a local Maven repo there is effectively a free cache - and also a source of "works on the agent, fails on a fresh one". workspace: clean: all on a job wipes it first (workspace: is a job setting, at the same level as steps:), which makes self-hosted behave like a fresh VM. Do that whenever you want to prove a pipeline is reproducible.

See what will actually run

Templates, parameters, ${{ if }} and each make the file you read differ from the pipeline that runs. The web UI has "Download full YAML"; here ci validate PIPELINE (simulator) compiles the pipeline from the repo on git.lab and prints the result, with every template pasted in:

$ ci validate payments-api
stages:
- jobs:
  - job: build
    steps:
    - displayName: Cache the Maven repository
      inputs: ...

Always look at the expanded form when a template "does nothing".

What you can now do

Why it helps

On a platform team you will own the templates that dozens of services extend. When a service's pipeline "skips the scan", it is because it does not extends the platform template. When one change to the templates repo breaks every pipeline at once, it is because nobody pinned ref: refs/tags/v3. When a template if does nothing, you know to download the full expanded YAML, because compile-time logic cannot see runtime values.

Caching is what turns a 12-minute Maven build into a 3-minute one, and misusing it (caching build outputs instead of dependencies, or a key that never changes) is a common review comment. Matrix builds are how you test a library on JDK 17 and 21 without copy-paste. All three are standard topics when an interviewer asks how you would scale CI for many teams.

FAQ

What is the difference between template: and extends: in Azure Pipelines?

template: includes a piece of YAML (steps, jobs, stages or variables) wherever you place it; the including pipeline stays in control and can add anything around it. extends: means the entire pipeline comes from the template, and the service can only fill in the parameters the template exposes. Platform teams use extends to enforce mandatory stages like scanning and approvals, often combined with a required-template check on environments.

Why doesn't my ${{ if }} see a variable set by a script or a matrix value?

Because ${{ }} is evaluated at compile time, before any job runs. Values set with ##vso[task.setvariable], at queue time or by a matrix entry do not exist yet, so the expression sees empty. For runtime decisions use condition: on the step or job. Likewise, passing $(JDK) into a parameter with values: fails validation, because the parameter literally receives the text $(JDK).

What is the difference between a cache and a pipeline artifact?

A cache speeds things up: dependencies like ~/.m2 or node_modules, restored if a key matches, best effort, shared across runs. Losing it only makes the build slower. A pipeline artifact is a guaranteed output of one run, like the jar or a release.env with the digest, used to move files between jobs and stages. Never ship something restored from a cache; ship artifacts.

How does the Cache@2 key work, and what are restoreKeys?

The key is a list of segments joined by |: literal strings and file paths or globs whose content hash becomes part of the key. maven | "$(Agent.OS)" | pom.xml changes whenever pom.xml changes, giving a cache miss and a new cache saved after the job succeeds. restoreKeys are prefixes tried when the exact key misses, so you get the most recent similar cache and only download the difference.

Should I pin shared templates to a tag, and why?

Yes. resources.repositories with ref: refs/tags/v3 means a change to the templates repo's main does not change every service's pipeline at once. Teams opt into new versions by bumping the ref, ideally after reading a changelog. Without pinning, one bad template commit breaks all pipelines simultaneously and nobody can reproduce yesterday's green build. Treat templates like any shared library: SemVer, changelog, pinned versions.

In an interview Mid

How would you avoid duplicating pipeline YAML across many services?

Templates: a YAML file with typed parameters: (type, default, allowed values:) and steps:/jobs:/stages:, pasted in at compile time with - template: templates/maven-build.yml plus parameters:. A wrong or missing parameter fails the run before it starts.

For many repos:

Related: a matrix runs one job per variant (JDK 17 and 21); Cache@2 keyed on pom.xml restores Maven dependencies. And look at the expanded YAML (ci validate, "Download full YAML") when a template "does nothing".

Also asked: Your Maven build spends most of its time downloading dependencies. What do you do? · How can a platform team make security scanning mandatory in every team's pipeline? · What is a matrix build and when would you use one?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.