GitOps, Argo CD & Delivery: interview questions

What is GitOps, and how is it different from deploying from a pipeline? Mid

Push-based CD (a pipeline running helm upgrade): something outside the cluster pushes changes in. Every pipeline and agent holds credentials that can change prod, the cluster is whatever the last run made it, and a kubectl edit goes unnoticed.

GitOps turns it around - the four OpenGitOps principles:

1. Declarative - the desired state is data (YAML, a chart and values, a kustomize overlay).
2. Versioned and immutable - in git: author, review, timestamp, revert.
3. Pulled - an agent in the cluster (Argo CD or Flux) pulls it; nothing outside pushes.
4. Continuously reconciled - it compares live with desired all the time and fixes drift.

What changes in practice: the pipeline only builds, pushes and commits a digest to the config repo; only the in-cluster controller has cluster credentials; promotion is a pull request, rollback is git revert, and the deployment history is git log overlays/prod.

It does not solve secrets, database migrations or the build itself - and it adds a controller that can disagree with git for its own reasons.

Also asked: How do you manage secrets in a GitOps workflow? · How do you promote a release from dev to prod with GitOps? · Explain rolling update, blue-green and canary deployments.

What problems does GitOps solve compared with running kubectl or helm from a pipeline? Mid

Three problems of push-based CD:

• Credentials - every pipeline and agent can change prod. With GitOps only the controller inside the cluster can; "CI can kubectl apply anything to prod" is an audit finding, "a controller applies what was merged to a protected branch after review" is a control.
• Source of truth - with push, the cluster is the sum of whatever runs happened. With GitOps it is git: what is merged is what should run.
• Drift - a hand edit stays silent until the next deploy overwrites it (or does not). With GitOps the controller keeps reconciling, so drift is detected in minutes and optionally reverted.

Plus a free audit trail: promotion is a pull request changing one digest line, the review is the approval, the merge commit is the record, and rollback is git revert.

How it is laid out: an app repo (code, Dockerfile, pipeline) and a config repo (what runs where), with environments as directories (overlays/dev, overlays/prod) on main, not branches, and CODEOWNERS on prod.

Also asked: How would you structure a GitOps config repository for several services and environments? · Why should environments be directories rather than branches in a config repo? · Someone hotfixed production with kubectl edit and a minute later the change was gone. Explain.

Learn it: 26.1 GitOps: git as the only way in

What is kustomize, and how do you choose between it and Helm? Mid

kustomize (built into kubectl) takes plain, valid Kubernetes YAML and applies transformations - no template language:

• a base: ordinary manifests you could kubectl apply as they are;
• an overlay per environment: a kustomization.yaml that points at the base and lists edits - namespace:, images: (tag or digest), replicas:, and patches (partial objects merged in; a strategic merge patch matches list items like containers by name).

kubectl kustomize overlays/prod prints the finished YAML - read it before kubectl apply -k. A configMapGenerator appends a content hash to the ConfigMap's name, so a config change renames it, changes the pod template, and rolls the pods.

Choosing: Helm for things you ship to others - versioned charts in repos, parameters with defaults, logic, hooks, a release history (platform base charts, third-party software). kustomize for your own environments of a service - plain YAML and per-environment patches. They combine: Argo CD renders a chart with per-environment values, or kustomize renders a chart and patches the one setting it lacks.

Also asked: What is a strategic merge patch, and when would you use a JSON 6902 patch instead? · How does a configMapGenerator make pods restart when the config changes? · A kustomize overlay patch stopped working after the base changed. How do you debug it?

Learn it: 26.2 kustomize: a base and overlays, without templates

Explain how Argo CD works. Mid

Argo CD is a controller installed in the cluster (namespace argocd) whose desired state comes from git. The repo-server clones git and renders the manifests (helm template, kustomize build or plain YAML); the application controller compares them with the live objects and syncs (applies) the difference; argocd-server is the API, UI and what the argocd CLI talks to.

You give it Applications: "this path of this repo, kept applied to this namespace of this cluster". It notices commits by polling (every 3 minutes by default) or immediately via a webhook; argocd app get APP --refresh forces it. Objects it applies carry a tracking-id annotation, which is how it knows what it owns.

Every app has two independent statuses - read both:

• Sync: Synced / OutOfSync (does the cluster match git?).
• Health: Healthy, Progressing, Degraded, Missing, Suspended (does it work?).

Synced + Degraded = git was applied and the version in git is broken: fix it in git. OutOfSync + Healthy = drift or an unsynced commit. Rollback is git revert; argocd app rollback is refused while auto-sync is on.

Also asked: An Argo CD application shows Synced but Degraded. What does it mean and how do you proceed? · You pushed a commit and Argo CD did nothing for a few minutes. Why? · Why does argocd app rollback refuse to run when auto-sync is enabled?

Learn it: 26.4 Argo CD: how it works, and the Application

What do automated sync, prune and self-heal do in Argo CD? Mid

syncPolicy:
  automated:
    prune: true
    selfHeal: true

• automated (auto-sync): every new commit that renders differently is applied without a human. Without it Argo CD only reports OutOfSync and someone runs argocd app sync.
• prune: delete live objects that are no longer in git. Without it, removing a file leaves the object running and the app OutOfSync ("requires pruning"). Many teams auto-sync prod but prune in a reviewed manual sync, and protect data with Prune=false on a PVC.
• selfHeal: re-sync when the live state drifts, not only when git changes - so a kubectl edit is reverted within moments. That is what makes git the only way in.

Two details: auto-sync does not retry a commit that already failed (add retry: with backoff for transient failures), and allowEmpty: false refuses to sync a path that suddenly renders nothing.

Ordering is separate: PreSync hooks (a migration Job) run before the apply, and sync waves apply lower waves first and wait for them to be healthy.

Also asked: How do you order resources during an Argo CD sync, for example a migration before the app? · What sync policies would you choose for dev and for prod, and why? · What happens to the deployed objects when you delete an Argo CD Application?

Learn it: 26.9 Sync policies: automated, prune, self-heal, waves and hooks

An Argo CD application keeps flapping between Synced and OutOfSync. How do you fix it properly? Mid

Argo CD compares the fields you declare in git with the live object (normalised). Defaults the API server adds are not compared. So an app that is OutOfSync again right after a successful sync has something in the cluster rewriting a field git declares - and with self-heal on, that becomes a loop of syncs of the same commit.

Usual culprits: an HPA changing replicas, or a mutating admission webhook (a policy engine forcing imagePullPolicy, injecting labels or resources).

1. See it: argocd app diff APP - < lines are live, > lines are git. kubectl get mutatingwebhookconfigurations lists the webhooks.
2. Make git match reality first: if an HPA owns replicas, remove replicas from the Deployment in git; if policy forces Always, declare Always.
3. Ignore narrowly only where a controller legitimately owns a field you must declare: ignoreDifferences on that kind and path (or managedFieldsManagers), with RespectIgnoreDifferences=true so the sync stops overwriting it.
4. For webhooks, server-side diff puts the mutation on both sides.

Never ignore a whole object or /spec - that hides the drift GitOps exists to catch.

Also asked: What does OutOfSync mean in Argo CD, and what are common causes? · How do you read the output of argocd app diff? · Why should a Deployment scaled by an HPA not declare replicas in git?

Learn it: 26.12 How Argo CD diffs, and ignoreDifferences

How do you manage many Argo CD Applications without creating each one by hand? Mid

Applications are Kubernetes objects, so they belong in git too - an Application created with argocd app create is unreviewed cluster state.

• App of apps: a root Application points at a directory (apps/) that contains only Application and AppProject manifests. One kubectl apply -f root.yaml is the bootstrap; after that, adding a service or environment is a commit. Sync waves order them (the AppProject at -1, platform apps before services). With prune on the root, deleting a file deletes the child app - and, with the resources finalizer, what it deployed - so review apps/ like prod.
• ApplicationSet: when the apps follow a pattern, one object generates them from a template - a list, git (one per directory), clusters, matrix, or pull request generator (preview environments).

Rule of thumb: app of apps when each app is different, ApplicationSet for a pattern (40 services x 3 environments).

And fence each team with an AppProject: allowed source repos, destinations (namespaces), resource kinds, roles, sync windows. An app pointing outside it gets InvalidSpecError instead of syncing. The default project allows everything.

Also asked: How do you provide multi-tenancy for several teams on one Argo CD instance? · How would you bootstrap a new cluster with Argo CD so it reaches the full desired state from git? · When would you use an ApplicationSet instead of app of apps?

Learn it: 26.16 App of apps, ApplicationSet and projects

How do you promote a release from dev to prod with GitOps? Mid

The pipeline stops deploying; it commits:

1. Build stays the same: test, build the image once, push, record the digest.
2. Dev: clone the config repo, kustomize edit set image IMG@DIGEST in overlays/dev (or yq -i on a values file), commit "dev: payments-api 1.5.1 @ sha256:...", push. Argo CD syncs dev.
3. Prod: open a pull request changing the same digest line in overlays/prod - only after dev is healthy (argocd app wait payments-dev --health). The review is the approval, CODEOWNERS says who may approve, branch protection requires green checks (renders, argocd app diff --revision). Merging is deploying.

What the pipeline lost: kubeconfigs, helm upgrade, every cluster permission. What it needs: a token that can push to one repo.

Around it: rollback = git revert of the promotion commit; freezes = AppProject syncWindows; audit = git log -- overlays/prod, with the digest in every message. Tools like Renovate or Argo CD Image Updater can open these pull requests.

Also asked: What permissions does a pipeline need in a GitOps setup, and how do you minimise them? · A bad release reached production through GitOps. How do you roll back? · How do you enforce a change freeze with Argo CD?

Learn it: 26.18 Promotion in a GitOps world

How do you manage secrets in a GitOps workflow? Mid

Not as a Secret manifest in git: its values are only base64 (base64 -d reads them), and every clone, fork and backup keeps the password forever. Git should hold the secret's existence, name and keys - never a readable value. Two families:

• Encrypt it for the cluster - Sealed Secrets. kubectl create secret ... --dry-run=client -o yaml | kubeseal encrypts with the controller's public key; the SealedSecret goes into git and only the controller's private key opens it. Scope (strict by default) binds it to a namespace and name, so a copy elsewhere does not decrypt. Back up the sealing keys; rotation = re-seal and commit. SOPS is similar but encrypts only the values inside normal YAML.
• Keep the value outside - External Secrets Operator. The value lives in Key Vault; an ExternalSecret in git is only a pointer, and the operator (via workload identity) copies it into a Secret every refreshInterval. Rotation, access control and audit are the vault's; a leaked repo has no secrets. The usual choice in Azure-heavy banks. (Or the Secrets Store CSI driver, mounting files.)

Either way, pods using env vars need a restart (or Reloader) to see a rotated value.

Also asked: Compare Sealed Secrets, SOPS and External Secrets Operator. · Why does a SealedSecret copied to another namespace fail to decrypt? · A secret was rotated in Key Vault but the pods still use the old value. Why?

Learn it: 26.22 Secrets when git is the source of truth

Explain rolling update, blue-green and canary deployments. Mid

• Rolling update (the Deployment default): replace pods a few at a time. maxSurge = extra pods allowed, maxUnavailable = how many may be missing (maxSurge: 1, maxUnavailable: 0 never drops capacity). Needs readiness probes to mean anything. Cheap; rollback is another rollout; both versions run during it. Recreate kills all old pods first - downtime, never two versions at once.
• Blue-green: run the new version (green) at full size next to the old (blue), test it through a preview Service, then switch the production Service's selector. Instant switch and instant rollback (blue still runs); costs double capacity.
• Canary: send a small share of real traffic to the new version, watch error rate and latency against stable, increase step by step or abort. By replica ratio (1 canary pod of 4 = ~25%, coarse) or by traffic split (ingress canary annotations, Gateway API weights, a service mesh: exact percentages, routing by header). Argo Rollouts automates the steps and the analysis.

All of them run two versions against one database at some point, so schema changes must be expand / contract. Feature flags are separate: they release a feature, not a build.

Also asked: How would you configure a rolling update for a latency-sensitive service with a slow startup? · What decides whether a canary is promoted or rolled back? · Why does every deployment strategy need backwards-compatible database changes?

Learn it: 26.26 Delivery strategies: rolling, blue-green, canary