Argo CD app OutOfSync right after a sync: mutating webhooks, controllers and ignoreDifferences

$ argocd app get payments-prod | head -16
Name:               argocd/payments-prod
...
Sync Policy:        Automated (Prune)
Sync Status:        OutOfSync from HEAD (b5e2be1)
Health Status:      Healthy
...
$ argocd app history payments-prod | tail -5
0       2026-09-22 19:57:06 +0000 UTC  HEAD (b5e2be1)
1       2026-09-22 19:57:09 +0000 UTC  HEAD (b5e2be1)
2       2026-09-22 19:57:16 +0000 UTC  HEAD (b5e2be1)
3       2026-09-22 19:57:35 +0000 UTC  HEAD (b5e2be1)
4       2026-09-22 19:58:30 +0000 UTC  HEAD (b5e2be1)

payments-prod has shown OutOfSync for 45 minutes. Its history has dozens of syncs, all of the same commit, and nobody has pushed since yesterday. The pods look fine. Syncing again won't change anything: it has already synced 30 times.

What is actually happening

For every object an Application tracks, Argo CD compares two things: the manifest git renders, and the live object in the cluster. It normalises both first (so "1000m" CPU and "1" compare equal), and the rule that matters is this:

• A field you declare in git is compared. If the live value differs, the app is OutOfSync.
• A field you don't declare isn't compared. Defaults the API server fills in (imagePullPolicy, terminationGracePeriodSeconds, a Service's clusterIP) don't make an app OutOfSync.

So an app that's OutOfSync right after a successful sync has something in the cluster that rewrites a field git declares, every time it's applied. The usual suspects:

• a mutating admission webhook, a program the API server calls on every create and update that may change the object before storing it (policy engines like Kyverno or Gatekeeper that force a pull policy, add resources, inject labels or sidecars);
• a controller that owns a field: an HPA owns spec.replicas, cert-manager fills a caBundle, an operator writes back its own spec.

With selfHeal on, this becomes a loop. Sync, mutate, OutOfSync, self-heal syncs again, mutate. Argo CD backs off between attempts, but the history fills up and the status stops meaning anything. When the status is always red, nobody notices real drift, like a hand-edited Service selector that leaves pods Running with no traffic.

Diagnosis

1. Confirm it's a loop, not drift

app get says OutOfSync but Healthy, and the history shows the same revision synced again and again (the output above). Real drift, someone's kubectl edit, gets fixed by one sync and stays fixed.

2. Ask Argo CD what it sees as different

$ argocd app diff payments-prod

===== apps/Deployment pay-prod/payments-api ======
29c29
<         imagePullPolicy: Always
---
>         imagePullPolicy: IfNotPresent

The header names the object (group/kind, namespace/name). The rest is classic diff output with the live object first: < is the cluster, > is git. Git says IfNotPresent, the cluster says Always, right after a sync that applied IfNotPresent.

3. Find who rewrites it

$ kubectl get mutatingwebhookconfigurations
NAME                     AGE
platform-policy-mutate   3m1s
$ kubectl get mutatingwebhookconfiguration platform-policy-mutate -o yaml | grep -A3 namespaceSelector && kubectl get ns pay-prod --show-labels
  namespaceSelector:
    matchLabels:
      policy.lab/image-pull: always
  rules:
NAME       STATUS   AGE    LABELS
pay-prod   Active   3m1s   kubernetes.io/metadata.name=pay-prod,sysop/lab=ch25,policy.lab/image-pull=always

The platform team's webhook forces imagePullPolicy: Always in every namespace with that label, and pay-prod has it. When no webhook matches, look at who last wrote the field: kubectl get deploy payments-api -n pay-prod -o yaml --show-managed-fields lists each field's manager (the client or controller that set it), for example kube-controller-manager for replicas changed by an HPA.

The fix

Fix it in this order.

1. Make git agree with reality, or stop declaring the field. The webhook is the desired state here: it's platform policy. Remove the patch that declares IfNotPresent:

$ git rm -q pull-policy.yaml && sed -i '/save registry traffic/d; /path: pull-policy.yaml/d' kustomization.yaml && git commit -qam "prod: do not declare imagePullPolicy (platform policy owns it)" && git push -q
rm 'overlays/prod/pull-policy.yaml'
[main 72e773b] prod: do not declare imagePullPolicy (platform policy owns it)
 2 files changed, 12 deletions(-)
 delete mode 100644 overlays/prod/pull-policy.yaml
...
$ argocd app wait payments-prod --sync --health --timeout 180
...
Sync Status:        Synced to HEAD (72e773b)
Health Status:      Healthy

A minute later it's still Synced to HEAD (72e773b), and the history has stopped growing. The same rule covers the HPA case: remove replicas from the Deployment in git when an HPA owns it. That's the documented pattern, not an ignore rule.

2. Ignore narrowly, when a controller legitimately owns a field you also have to declare:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: payments-api
      jsonPointers:
        - /spec/template/spec/containers/0/imagePullPolicy
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

jqPathExpressions and managedFieldsManagers select fields in other ways. Without RespectIgnoreDifferences=true, the field is only ignored for the status: every sync still applies git's value, and the fight goes on. Never ignore a whole object or all of /spec, because that hides the drift GitOps exists to catch.

3. Server-side diff. Newer Argo CD versions can diff against a server-side dry-run apply (the annotation argocd.argoproj.io/compare-options: ServerSideDiff=true). The dry run passes through admission, so webhook mutations show up on both sides and stop causing diffs. That fixes webhook noise. It doesn't fix a controller that writes later.

How to prevent it

• When a platform team adds a mutating policy, tell app teams which fields it owns, so they stop declaring them.
• Treat an app that flips between Synced and OutOfSync as a bug, not as noise.
• Run argocd app diff in CI on pull requests (it exits 1 on differences) to catch fights before they merge.

Deploys through Argo CD still go through a normal Deployment rollout. If they cause errors, the problem is in the rollout, not in GitOps (downtime on every deploy). And the same "who owns this field" question comes up in Terraform when a plan wants to replace something in production.

Practise it

Chapter 26's incident "OutOfSync forever" sets up this webhook fight with self-heal on. You fix it through git, and the lesson before it covers how Argo CD diffs and every ignoreDifferences form.