Lesson 25.7 · CI/CD Pipelines & Helm · 22 min read

Variables, secrets and service connections

Q: What is the difference between ${{ }}, $( ) and $[ ] in Azure Pipelines?

${{ variables.x }} is a template expression, evaluated when the YAML is compiled, before anything runs; it can shape the pipeline but only sees parameters and YAML variables. $(x) is a macro, substituted into a task's inputs just before that task runs; an unknown name stays as literal text. $[ variables.x ] is a runtime expression, used in conditions and for variables that read other jobs' outputs.

In plain words

Imagine a class play with a script. Some words in the script are placeholders: "[the teacher's name]". There are three moments they can be filled in: when the script is printed, just before each scene starts, or live on stage when an actor looks at a note. And the secret password to the costume cupboard is never printed in the script at all; the teacher whispers it only to the actor who needs it, and if that actor shouts it out in a funny accent, nobody can stop them.

In Azure Pipelines, ${{ }} is filled when the YAML compiles, $(x) just before a task runs, and $[ ] at runtime. Secrets come from variable groups or Key Vault, are masked in logs, and reach a script only when mapped with env:.

The problem

A pipeline needs values: a version number, a registry name, and passwords or tokens to push images and commit to repos. Put a password in the YAML and it is in git forever, readable by everyone with a clone. This lesson is about where each kind of value belongs and how the pipeline reads it.

What you need to know already: 25.4 (pipeline YAML, $(Name) variables, logs), 6.12 (bash parameter expansion), 22.14 (a service principal for CI), 22.21 (a pipeline identity with no secret: OIDC), 22.23 (Key Vault).

Words for this lesson

variable (pipeline) - a named value the pipeline can use, like VERSION.
secret variable - a variable whose value the server stores encrypted and hides from logs.
masking - replacing a secret's value with *** wherever it shows up in a log.
service connection - a stored, named credential (registry, cluster, Azure subscription) that tasks use by name, without the pipeline ever seeing it.
compile (a pipeline) - the step where Azure DevOps reads your YAML, fills in templates and works out the list of stages, jobs and steps, before any agent starts.

Three syntaxes, three moments

Azure Pipelines has three ways to write "the value of x", and each is filled in at a different moment:

  ${{ variables.x }}   template expression   expanded when the YAML is compiled,
                                              before anything runs. Can shape the pipeline
                                              (insert steps, pick templates). Knows only
                                              parameters and variables defined in YAML.
  $(x)                 macro                  replaced in a task's inputs just before the
                                              task runs. Unknown name -> left as literal $(x)
  $[ variables.x ]     runtime expression     evaluated at runtime; used for conditions and
                                              for variables computed from other jobs' outputs

template expression ${{ }} - worked out while compiling. Use it when the value decides the shape of the pipeline (lesson 25.9).
macro $( ) - text replacement just before one step runs. The common one.
runtime expression $[ ] - worked out while the run is going, when a value only exists after an earlier job produced it.

The macro rule "unknown stays literal" has a nasty consequence in bash:

- script: echo "deploying $(ImageTag)"      # typo: the variable is imageTag ... no:

Variable names are case-insensitive in Azure Pipelines, so $(imagetag) works - but a real typo like $(ImageTga) stays in the script text, and bash treats $(ImageTga) as command substitution (Ch 6: "run the command ImageTga and paste its output"):

2026-09-24T10:00:41.9786725Z bash: line 1: ImageTga: command not found
2026-09-24T10:00:41.9786725Z deploying

The step does not even fail (echo is the last command, lesson 25.4's trap).

Where variables come from, and who wins

variables:                         # pipeline level
  VERSION: 1.4.0
  - group: payments-common         # a variable GROUP from the Library (list form only)

stages:
- stage: Deploy
  variables:                       # stage level overrides pipeline level
    ENV: dev
  jobs:
  - job: deploy
    variables:                     # job level overrides stage level
      ENV: dev-eu

The most specific scope (where it is defined) wins: job > stage > pipeline. Two more sources override YAML for the rest of the job: variables set when someone starts a run by hand ("queue time", if the variable allows it), and variables a script sets with ##vso[task.setvariable] (see "Logging commands" below).

(That first block is deliberately wrong: you cannot mix the map form (KEY: value) and the list form (- name: / - group:) at one level. As soon as you need a group, use the list form everywhere:)

variables:
- name: VERSION
  value: 1.4.0
- group: payments-common

Variable groups

A variable group (in the web UI under Pipelines > Library) holds values shared by many pipelines: registry names, the URL of a code-quality server, a team's chat webhook. A group can be linked to an Azure Key Vault (22.23): the group then shows the vault's secrets, fetched at run time with the permissions of a service connection - the secret never lives in Azure DevOps at all. That is the typical bank setup: secrets in Key Vault, pipelines reading them through a linked group or the AzureKeyVault@2 task.

ci groups (simulator) lists the groups: their variables, and whether they are linked to a vault:

$ ci groups
GROUP             VARIABLES                          LINKED
payments-common   REGISTRY, SONAR_URL, TEAMS_HOOK    -
payments-prod-kv  payments-db-password               Key Vault kv-payments-prod

Secret variables

Marking a variable secret does three things, and not a fourth:

The value is stored encrypted and never shown again in the UI.
The agent masks it: any log line containing the exact value shows ***.
It is not available to scripts as an environment variable. You must map it explicitly with env:.
It does not stop a script from leaking it.

env: on a step sets environment variables for that one step only. Here it hands the secret to the script as $CONFIG_TOKEN:

- bash: |
    set -euo pipefail
    git clone -q "https://ci-bot:[email protected]/pay/payments-config.git" cfg
  env:
    CONFIG_TOKEN: $(CONFIG_TOKEN)     # without this line, $CONFIG_TOKEN is EMPTY

(https://USER:TOKEN@host/... is how you give git a username and token in the URL itself; -q = quiet.) Without the mapping, the clone runs with an empty password and the server says so:

remote: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. ...
fatal: Authentication failed for 'https://git.lab/pay/payments-config.git/'

Masking only matches the exact string. These all leak it:

echo "$CONFIG_TOKEN" | base64          # a different string: printed in full
echo "${CONFIG_TOKEN:0:6}"              # a substring (first 6 chars): printed
echo "$CONFIG_TOKEN" | rev              # reversed: printed
set -x                                  # xtrace prints commands AFTER expansion ...
                                        # ... masked, but only if it is the exact value

(base64 re-encodes text as other letters, Ch 15.33.) And a secret written to a file that you then publish as an artifact is not masked at all. Treat masking as a safety net for accidents, not as access control.

What never goes in a variable

In the YAML file: no secret, ever. The file is in git: every clone, fork and CI log of the repo has it, and git revert does not remove it from history. A secret that was committed is burned: rotate it (revoke the old value and issue a new one).
Plain (non-secret) variables for anything sensitive: they are shown in the UI, in ci vars, and exported to every script's environment (env | sort prints them).
Long-lived cloud credentials when you can avoid them: use a service connection with workload identity federation (OIDC, 22.21). Azure DevOps presents a short-lived token that Entra ID trusts; there is no client secret to store, leak or rotate.

Output variables: passing values between jobs and stages

A job's variables disappear when it ends. To hand a value (say, the image digest) to a later job, a step publishes it as an output variable:

- job: build
  steps:
  - bash: echo "##vso[task.setvariable variable=digest;isOutput=true]sha256:3f0d..."
    name: meta                          # the step NAME is part of the address

- job: deploy_dev
  dependsOn: build
  variables:
    digest: $[ dependencies.build.outputs['meta.digest'] ]          # same stage

- stage: Prod
  jobs:
  - job: deploy
    variables:
      digest: $[ stageDependencies.Build.build.outputs['meta.digest'] ]   # other stage

The address reads: from the job build, the output digest of the step named meta. dependencies. works inside the same stage, stageDependencies.STAGE. across stages. Note the $[ ] runtime syntax: the value only exists once build ran.

Three details make or break it: the step must have a name:, isOutput=true must be set, and the consuming job must depend on the producer (directly or through its stage). A missing piece gives you an empty string, not an error.

Service connections

A service connection is a named, permissioned credential the pipeline can use without seeing it: a Docker registry, a Kubernetes cluster, an Azure subscription. Tasks take its name:

- task: Docker@2
  inputs:
    containerRegistry: registry-lab          # the service connection
    repository: pay/payments-api
    command: buildAndPush
    tags: $(VERSION)
- task: Kubernetes@1
  inputs:
    connectionType: Kubernetes Service Connection
    kubernetesServiceEndpoint: lab-cluster
    command: login                           # sets KUBECONFIG for the rest of the job

The connection can be restricted to specific pipelines and protected by the same approvals as an environment (lesson 25.11). The agent user (azagent) has none of your credentials; after Kubernetes@1 login the job has a temporary kubeconfig (Ch 15.1) in _work/_temp, and kubectl/helm in later steps use it.

Logging commands you will use

A logging command is a specially formatted line a script prints to its output; the agent reads it and acts on it instead of just logging it:

##vso[task.setvariable variable=NAME]value                   set for later steps of this job
##vso[task.setvariable variable=NAME;isOutput=true]value     ... and as an output
##vso[task.setvariable variable=NAME;issecret=true]value     ... and mask it from now on
##vso[task.logissue type=error]message                       an error annotation
##vso[task.complete result=SucceededWithIssues;]             mark the step
##vso[build.updatebuildnumber]1.5.0-rc.2                     rename the run

A script that fetches a token at runtime should set it with issecret=true immediately - before anything can print it.

What you can now do

Choose between ${{ }}, $( ) and $[ ] by when the value is known.
Store a token as a secret variable and hand it to one step with env:.
Pass a value from one job to another with an output variable.

Why it helps

Secrets in pipelines are where real breaches and real outages come from. You will review a PR that puts a token in YAML and need to say "burned, rotate it", not "move it to a variable". You will debug a git clone that fails with Authentication failed because the secret was never mapped via env:, so $CONFIG_TOKEN was empty. And you will explain in an interview why masking is not access control: echo "$TOKEN" | base64 prints it in full.

The three expression syntaxes explain half the confusing pipeline bugs: a template if that "does nothing" because it compiled before the value existed, or a typo that became ImageTga: command not found. Output variables are how the digest gets from Build to Prod, and service connections with workload identity federation are how a bank's pipeline reaches Azure without a stored client secret.

FAQ

What is the difference between ${{ }}, $( ) and $[ ] in Azure Pipelines?

${{ variables.x }} is a template expression, evaluated when the YAML is compiled, before anything runs; it can shape the pipeline but only sees parameters and YAML variables. $(x) is a macro, substituted into a task's inputs just before that task runs; an unknown name stays as literal text. $[ variables.x ] is a runtime expression, used in conditions and for variables that read other jobs' outputs.

Why is my secret variable empty in the script?

Secret variables are deliberately not exported to the environment of scripts. You must map them explicitly on the step: env: { CONFIG_TOKEN: $(CONFIG_TOKEN) }. Without that line $CONFIG_TOKEN is empty, and tools fail with an authentication error rather than a clear "missing secret". Non-secret variables are exported automatically, which is another reason never to put sensitive values in plain variables.

If secrets are masked in the log, can a script still leak them?

Yes. Masking replaces only the exact secret string in log output. Base64-encoding it, printing a substring, reversing it or writing it to a file that is published as an artifact all bypass the mask. A malicious or careless script with access to the secret can exfiltrate it any way it likes. Masking protects against accidents; access control is about which pipelines and branches can get the secret at all.

What is a service connection, and why use workload identity federation?

A service connection is a named credential managed by Azure DevOps (for Azure, a registry or a cluster) that tasks use without the pipeline seeing the secret, with its own permissions and approvals. Workload identity federation means the connection has no client secret at all: Azure DevOps issues a short-lived OIDC token that Entra ID trusts for that specific connection. Nothing to store, leak or rotate.

Why is my output variable empty in the next job or stage?

Usually one of three pieces is missing. The producing step needs a name:, because the address is stepName.variable. The logging command must include isOutput=true. And the consuming job must depend on the producer, directly or through its stage, and use the right syntax: dependencies.job.outputs['step.var'] in the same stage, stageDependencies.Stage.job.outputs['step.var'] across stages. Any gap gives an empty string, not an error.

In an interview Mid

How do you handle secrets in a pipeline?

Never in the YAML. It is in git: every clone and fork has it, and a later commit does not remove it from history. A committed secret is burned - rotate it.
Secret variables: stored encrypted, masked (***) in logs, and not exported to scripts - map them to one step with env: CONFIG_TOKEN: $(CONFIG_TOKEN). Masking matches only the exact string: base64, a substring or a file you publish leaks it. A safety net, not access control.
Better: no secret in Azure DevOps at all - a variable group linked to Key Vault, or the AzureKeyVault@2 task, fetching at run time.
Service connections for registries, clusters and subscriptions: tasks use them by name, the pipeline never sees the credential, and they can be restricted to pipelines and gated by approvals.
No long-lived cloud credentials: a service connection with workload identity federation (OIDC) - a short-lived token Entra ID trusts, nothing to leak or rotate.
A script that fetches a token at runtime sets it with ##vso[task.setvariable ...;issecret=true] immediately.

Also asked: What is the difference between ${{ }}, $( ) and $[ ] variables in Azure Pipelines? · A developer committed a production password and removed it in the next commit. What do you do? · How do you pass a value from one job to a later stage?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.