CI/CD Pipelines & Helm: interview questions

Walk me through what happens between a git push and a new pod serving traffic. Mid

1. Trigger - the push to main matches the pipeline's trigger; a run starts on an agent.
2. CI - checkout, compile and test (./mvnw -B verify), lint and scan. Any failure stops here.
3. Build once - docker build, push to the registry with the version and commit as tags (1.5.0, 9c41e7a), and record the digest the registry returned as an output variable.
4. Deploy to dev - a deployment job runs helm upgrade --install ... -f values-dev.yaml --set image.digest=<digest> --wait. Helm renders the chart, records a new revision, applies the objects.
5. Kubernetes - the Deployment's pod template changed, so a rolling update starts: new pods are scheduled, pull the image by digest, start, and receive traffic only once their readiness probe passes; old pods are drained.
6. Promote - the Prod stage waits for an approval on the payments-prod environment, then deploys the same digest with values-prod.yaml.

Every environment differs only in configuration; the bytes of the image never change. --wait makes a crash-looping rollout fail the pipeline instead of reporting green.

Also asked: What does "build once, deploy many" mean and how do you implement it? · What is Helm and what problems does it solve? · How do you handle secrets in a pipeline?

How would you prove that production runs exactly the image that passed testing? Mid

By building it once and promoting the same bytes:

• Build once, deploy many: CI builds and tests one image; every later environment deploys that image, and only configuration (values: replicas, URLs, flags) differs. A prod stage that runs docker build again ships an untested artifact - a newer base layer, a different dependency resolved that afternoon.
• Identify it by digest, not by tag. A tag like payments-api:1.5.0 is a name that can be moved; the digest (sha256:...) is computed from the content and cannot change. Tag with version and commit for humans, make release tags immutable in the registry, deploy by digest.
• Check it: docker push in the CI log prints the digest; kubectl get pod -n pay-prod -l app.kubernetes.io/name=payments-api -o jsonpath='{..imageID}' shows what the node actually pulled. Same digest = same artifact.

Add the audit trail: the deploy is a commit in the config repo ("prod: 1.4.1 -> 1.5.0"), reviewed and revertible.

Also asked: What is the difference between CI and CD? · What is the difference between continuous delivery and continuous deployment? · Why would you separate the application repository from the deployment configuration repository?

Learn it: 25.1 Delivery: from a commit to a running pod

Production is broken after a deploy that was a commit to the config repo. How do you roll back? Mid

With git, because the config repo is the ledger of what runs where:

1. git log --oneline - find the commit ("prod: payments-api 1.4.1"), git show --stat to confirm what it touched.
2. git revert <sha> - a new commit that undoes it. History keeps both: what happened and that it was undone.
3. git push - the pipeline deploys the reverted config like any other change, through the same checks.

Never git reset --hard plus git push --force on a shared branch: it rewrites what everyone else has, and on a protected main the server refuses it (pre-receive hook declined). That refusal is what keeps the ledger trustworthy.

If the push is rejected (fetch first / non-fast-forward) because someone pushed meanwhile: git pull --rebase, then push - never --force. Pipelines that commit to the repo do the same in a small retry loop. Directories per environment on one branch (envs/dev, envs/prod) keep this simple; branches per environment drift.

Also asked: What is the difference between git merge and git rebase? · Two pipelines commit to the same config repo and one push is rejected. How do you make this reliable? · What is semantic versioning, and when does each number go up?

Learn it: 25.2 Git as the delivery ledger: branches, rebase, revert, tags

Explain the structure of an Azure DevOps YAML pipeline. Mid

azure-pipelines.yml lives in the app repo and is read from the commit that triggered the run:

• trigger - which pushes start a run (branches, paths).
• pool - which agent pool; an agent (Microsoft-hosted or self-hosted) executes the work.
• stages > jobs > steps. A stage is the boundary for approvals, environments and dependsOn. A job runs on one agent with its own workspace. Steps run in order: script/bash, or a task like Docker@2 with inputs.
• Jobs never share files - move them as pipeline artifacts (publish / download).
• Variables: predefined ($(Build.SourceBranch)), your own, groups, secrets.
• Conditions: condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main')) - a custom condition without succeeded() runs even after a failure.

The trap to mention: a multi-line script: runs without set -e, so its result is the last command's - tests fail, echo succeeds, the step is green. Start scripts with set -euo pipefail.

Also asked: A command works in your terminal but fails in the pipeline. What do you check? · Why can a pipeline step be green when the tests in it failed? · How do you stop a pipeline from deploying to production from a feature branch?

Learn it: 25.4 Azure Pipelines YAML: triggers, stages, jobs, steps and the agent

How do you handle secrets in a pipeline? Mid

• Never in the YAML. It is in git: every clone and fork has it, and a later commit does not remove it from history. A committed secret is burned - rotate it.
• Secret variables: stored encrypted, masked (***) in logs, and not exported to scripts - map them to one step with env: CONFIG_TOKEN: $(CONFIG_TOKEN). Masking matches only the exact string: base64, a substring or a file you publish leaks it. A safety net, not access control.
• Better: no secret in Azure DevOps at all - a variable group linked to Key Vault, or the AzureKeyVault@2 task, fetching at run time.
• Service connections for registries, clusters and subscriptions: tasks use them by name, the pipeline never sees the credential, and they can be restricted to pipelines and gated by approvals.
• No long-lived cloud credentials: a service connection with workload identity federation (OIDC) - a short-lived token Entra ID trusts, nothing to leak or rotate.
• A script that fetches a token at runtime sets it with ##vso[task.setvariable ...;issecret=true] immediately.

Also asked: What is the difference between ${{ }}, $( ) and $[ ] variables in Azure Pipelines? · A developer committed a production password and removed it in the next commit. What do you do? · How do you pass a value from one job to a later stage?

Learn it: 25.7 Variables, secrets and service connections

How would you avoid duplicating pipeline YAML across many services? Mid

Templates: a YAML file with typed parameters: (type, default, allowed values:) and steps:/jobs:/stages:, pasted in at compile time with - template: templates/maven-build.yml plus parameters:. A wrong or missing parameter fails the run before it starts.

For many repos:

• Keep the templates in a platform repo, declared under resources: repositories: and pinned to a tag (ref:) - otherwise one change to its main changes every pipeline at once.
• Use extends: template: ...@templates so the service's pipeline only fills in parameters; the scan and approval stages are not optional - that is how a platform team enforces standards.
• ${{ if }} and ${{ each }} shape the pipeline at compile time; they only see parameters and YAML variables, not runtime values (use condition: for those).

Related: a matrix runs one job per variant (JDK 17 and 21); Cache@2 keyed on pom.xml restores Maven dependencies. And look at the expanded YAML (ci validate, "Download full YAML") when a template "does nothing".

Also asked: Your Maven build spends most of its time downloading dependencies. What do you do? · How can a platform team make security scanning mandatory in every team's pipeline? · What is a matrix build and when would you use one?

Learn it: 25.9 Templates, parameters, matrix builds and caching

How do you pass the built image from the build stage to the deployment stages, and gate production? Mid

Build once, carry the digest:

1. The Build stage pushes the image once and captures its digest: docker inspect --format '{{index .RepoDigests 0}}', publishing it with ##vso[task.setvariable variable=digest;isOutput=true] from a step with a name:.
2. Later stages read it with stageDependencies.Build.build.outputs['image.digest'] (a $[ ] runtime expression) and deploy --set image.digest=$(digest), so the chart renders image: ...@sha256:... - a reference that cannot silently change. Only the values file differs per environment.

Gate prod with an environment: a deployment job (deployment: + environment: payments-prod) records what was deployed where and runs the environment's checks first - approvals, branch must be main, business hours, an exclusive lock so two runs cannot deploy at once. The approval lives on the environment, not in the YAML, so whoever edits the pipeline cannot remove it - which is what auditors care about.

Mutable tags (latest, main, prod) are fine for finding the newest image, never for deploying.

Also asked: What are mutable and immutable image tags, and why does it matter for deployments? · Why should a pipeline not rebuild the image for each environment? · Why do approvals belong on the environment rather than in the pipeline YAML?

Learn it: 25.11 Build once, deploy many: promotion by digest, environments and approvals

Compare Azure Pipelines and GitLab CI. What changes when you move between them? Mid

The ideas are the same - stages, jobs, an agent/runner, variables, artifacts, environments with approvals - only the words change. In .gitlab-ci.yml every top-level key that is not a keyword is a job; stages: is just a list of names; needs: starts a job as soon as its inputs finish; rules: - if: decides whether a job exists; predefined variables are $CI_... environment variables.

The differences that bite:

• Script lines fail fast in GitLab (the first non-zero line fails the job); in Azure Pipelines a multi-line script only reports its last command. A pipe like ./mvnw verify | tee log still hides a failure without pipefail.
• rules are evaluated when the pipeline is created, not while it runs.
• Cache is best effort for dependencies; artifacts are guaranteed and move files between jobs.
• Variables can be protected (only on protected branches and tags, so a feature branch cannot read the prod token), masked, or of type file.
• Global keywords (image, cache, default) are not job names.

Jenkins: a Groovy Jenkinsfile with the same stages, agent label, withCredentials and an input gate.

Also asked: How do you prevent a feature-branch pipeline in GitLab from deploying to production? · Your GitLab job shows green but the Maven tests failed. What could cause it? · What is the difference between cache and artifacts in GitLab CI?

Learn it: 25.14 The same pipeline in GitLab CI (and what Jenkins looks like)

What is Helm and what problems does it solve? Mid

Helm installs an app's Kubernetes objects as one unit. It does two separate jobs:

1. Templating - a chart is a directory of templates plus default values.yaml; per-environment values fill the blanks, so one chart serves dev (1 replica) and prod (3) without copied YAML. helm template renders it without touching a cluster.
2. Release management - helm install records a release (name + namespace) with numbered revisions; every upgrade and rollback adds one. That record lets Helm delete objects a new version no longer renders (kubectl apply cannot), roll back to an exact earlier state (helm rollback creates a new revision with the old content), and show who deployed what (helm history).

Details interviewers like: version is the chart package's version, appVersion the app's; the release record is a Secret per revision in the namespace (so releases are per namespace, and anyone reading Secrets reads every value passed); --history-max 10 limits rollbacks. Before touching a release: helm history, helm get values, and helm get manifest | kubectl diff -f - to catch drift.

Also asked: What is the difference between a chart's version and its appVersion? · Someone says helm upgrade succeeded but the change is not in the cluster. How do you investigate? · Where does Helm store release state, and what follows from that?

Learn it: 25.19 Helm: charts, releases and revisions

How do you manage different configurations for dev, test and prod with Helm, and what are the traps? Mid

One chart, one values file per environment, every value passed by the pipeline every time: helm upgrade --install app ./chart -f values.yaml -f prod.yaml.

The merge rules: the chart's values.yaml < -f files left to right (the last wins) < --set. So the order of -f matters, and a --set in the pipeline beats any file.

The traps:

• Maps merge key by key; lists are replaced whole. null deletes a default; {} does not.
• A wrong key is not an error - --set replica=5 gives Upgrade complete and changes nothing. A values.schema.json with additionalProperties: false makes it fail.
• Types: tag: 1.10 is the float 1.1 - quote versions. Big numbers from files render as 1.5e+06 unless piped through int.
• Upgrade and old values: with any -f/--set Helm uses only this command's values plus defaults - a manual helm upgrade --set image.tag=... drops prod's settings. --reset-then-reuse-values for manual changes; better, no manual upgrades.

Check: helm get values (supplied) and --all (computed).

Also asked: Prod lost its settings after someone ran helm upgrade by hand. What happened and how do you prevent it? · What YAML type pitfalls do you watch for in Helm values files? · How do you see the values a release was actually deployed with?

Learn it: 25.21 Values: where every number comes from

How do you make pods roll automatically when their configuration changes? Mid

Changing a ConfigMap does not restart the pods that read it - the Deployment's pod template did not change, so nothing rolls and the pods keep the old config.

The Helm fix is the checksum annotation on the pod template:

spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

include renders the ConfigMap template to text and sha256sum hashes it. Any config change changes the hash, the pod template differs, and the Deployment does a normal rolling update with its readiness checks - no rollout restart step in the pipeline.

Limits: it only sees what the chart renders. A Secret created outside the chart needs a controller that watches it and restarts the pods (Reloader) or an explicit restart.

Related rules: use include (pipeable), not template; never put anything that changes between releases into selector labels - spec.selector is immutable and the upgrade fails with field is immutable.

Also asked: How do you debug a Helm chart that fails to render or produces wrong YAML? · What is the difference between include and template in a Helm chart? · Why must a Deployment's selector labels never contain the version?

Learn it: 25.24 Templates: Go templates, named templates, tpl and the checksum trick

A helm upgrade in the pipeline timed out. What state is the release in, and what do you do? Mid

It depends where helm upgrade stopped. The order is: render, write revision N+1 as pending-upgrade (the lock), pre-upgrade hooks, apply the manifests, wait, post-upgrade hooks, mark deployed.

• Timed out waiting (--wait, context deadline exceeded, resource Deployment/... not ready): N+1 is failed but its objects are applied - the Deployment is mid-rollout or stuck; Kubernetes' rolling update keeps the old pods serving. Read the pods (kubectl describe, logs) to see why they are not ready - or whether --timeout was simply shorter than replicas x startup / maxSurge.
• A pre-upgrade hook failed: N+1 failed, cluster untouched.
• The helm process was killed (agent OOM-killed, pipeline cancelled with SIGKILL): N+1 stays pending-upgrade forever and every next upgrade fails with another operation ... is in progress. Check helm history, then helm rollback to the last deployed revision to clear it.

helm history tells you which case you are in. For shared environments --rollback-on-failure returns to the last good revision automatically - but removes the evidence, so not in dev.

Also asked: How do you run database migrations as part of a Helm deployment? · What does --rollback-on-failure do, and when would you not use it? · Why is helm rollback not enough to undo a database migration?

Learn it: 25.26 Hooks, waiting and failure: what Helm does when things go wrong

How would you design and version a platform base chart used by many services? Mid

The base chart (java-service) holds the Deployment, probes, security context, labels and resource defaults; each service declares it as a dependency in Chart.yaml with a version range (~1.3.0 = patch updates only) and supplies only values, under the subchart's name. A platform fix then ships by bumping one dependency.

Reproducible builds: helm dependency update resolves the ranges and writes Chart.lock - committed to git; CI runs helm dependency build to fetch exactly those versions.

Distribution: push packaged charts (helm package, helm push) to an OCI registry next to the images, install with oci://registry.lab/charts/... --version. Chart versions immutable, promoted like image digests.

Versioning = the chart's interface (SemVer): patch for a template fix or new appVersion; minor for new optional values with unchanged defaults; major for renamed or removed values or changed selector labels (immutable - the upgrade fails). Teams read the changelog before a major bump.

Guardrails: values.schema.json, helm lint --strict and helm template in CI, helm diff before prod.

Also asked: What is Chart.lock and why should it be committed? · What is the difference between a classic Helm chart repository and an OCI registry? · When is a change to a chart a major version?

Learn it: 25.29 Dependencies, repositories, OCI and versioning