OnCallReady

Lesson 26.12 · GitOps, Argo CD & Delivery · 9 min read

How Argo CD diffs, and ignoreDifferences

In plain words

Imagine a teacher comparing your homework with the answer sheet. The teacher only checks the answers you wrote; blank spaces the school fills in for everyone, like the date stamp, are not counted as mistakes. But if a classroom helper secretly changes one of your written answers every time you hand it in, the teacher marks it wrong every time, you fix it, and it changes again.

Argo CD's diff works like that. A field you declare in git is compared; defaults the API server adds are not. An app that stays OutOfSync right after syncing has something rewriting a declared field: a mutating webhook, an HPA owning spec.replicas, or normalisation like 1000m becoming 1. First make git match reality; only then ignore narrowly with ignoreDifferences and RespectIgnoreDifferences=true.

The problem

An app shows OutOfSync. Before you sync, you need to know what differs and why. Sometimes the answer is "someone kubectl-edited it" (drift, 26.11). But sometimes an app stays OutOfSync even right after a successful sync, forever - and with self-heal on, Argo CD then syncs again and again. This lesson is how Argo CD decides "different", and how to fix the case where something else legitimately changes your objects.

What you need to know already: 26.4 (sync status), 26.9 (self-heal, sync options), 15.40 (kubectl apply, the last-applied annotation, kubectl diff), 17.28 (the HorizontalPodAutoscaler), 17.39 (admission: the API server checking objects before storing them), 7.11 (jq paths).

What "OutOfSync" actually compares

For each object Argo CD tracks, it takes three things: the manifest git renders, the live object, and (for normal client-side apply) the last-applied-configuration annotation kubectl leaves on objects (15.40). It normalises both sides (puts them in one standard form, e.g. "1000m" CPU and "1" become the same) and compares. The practical rule:

So an app that is OutOfSync forever, even right after a successful sync, has something in the cluster that changes a field git declares, every time:

mutating admission webhooks     a program the API server calls on every create/update
                                that may REWRITE the object before it is stored - e.g. a
                                company policy engine (Kyverno, Gatekeeper) forcing
                                imagePullPolicy, adding resources, injecting labels or
                                sidecar containers
controllers                     an HPA owns spec.replicas (17.28); cert-manager (a
                                controller that issues TLS certificates) fills a
                                caBundle field; an operator writes back its own spec
the API server normalising      "1000m" becomes "1", "0.5Gi" becomes "512Mi"

A mutating admission webhook is registered with a MutatingWebhookConfiguration object (kubectl get mutatingwebhookconfigurations lists them).

With self-heal on, that turns into a loop: sync, mutate, OutOfSync, self-heal sync, mutate... Argo CD waits a little longer between tries, but the app flaps between statuses and the history fills with syncs of the same commit.

Fix it in this order

  1. Make git match reality. If the platform policy forces imagePullPolicy: Always, declare Always (or stop declaring the field at all). If an HPA owns replicas, remove replicas from the Deployment in git - that is the documented pattern, not an ignore rule.
  2. Ignore narrowly, when a controller legitimately owns a field you must also declare. ignoreDifferences in the Application lists fields Argo CD should not compare:
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: payments-api                  # omit to match every Deployment of the app
      jsonPointers:                       # a path like /spec/replicas (RFC 6901, 26.2's patch paths)
        - /spec/replicas
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jqPathExpressions:                  # the same, written as a jq path (7.11)
        - .webhooks[]?.clientConfig.caBundle
    - group: apps
      kind: Deployment
      managedFieldsManagers:              # ignore every field this manager owns
        - kube-controller-manager
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true     # and do not overwrite those fields on sync

managedFieldsManagers relies on managed fields: the API server records, in metadata.managedFields, which tool (a "manager") last set each field.

Without RespectIgnoreDifferences=true, the field is ignored for the sync status, but a sync still applies git's value - and the fight continues on every sync.

  1. System-wide rules go in argocd-cm (resource.customizations.ignoreDifferences.apps_Deployment) - for things every app hits, like a platform injecting the same annotation everywhere.

Never ignore a whole object or /spec: you would hide exactly the drift GitOps is supposed to catch.

Reading a diff

# after someone kubectl-edited imagePullPolicy (in sync, it prints nothing and exits 0)
argocd app diff payments-prod

===== apps/Deployment pay-prod/payments-api ======
61c61
<         imagePullPolicy: Always
---
>         imagePullPolicy: IfNotPresent

Read it line by line:

Argo CD's UI shows the same with colours. Exit code 1 means differences, which makes argocd app diff usable as a CI check on a pull request (render the pull request's commit with --revision).

Server-side diff

Newer Argo CD versions can ask the API server to dry-run the apply (go through all the checks and webhooks but store nothing) and diff the result (argocd.argoproj.io/compare-options: ServerSideDiff=true, or controller.diff.server.side: "true" for all apps). Because the dry run goes through admission, mutations by webhooks appear on both sides and stop causing diffs. It is the modern fix for webhook noise - but it costs an API call per object per reconcile, and it only helps with webhooks, not with controllers that write later.

What you can now do

Why it helps

An app that flaps between Synced and OutOfSync, with self-heal syncing the same commit dozens of times an hour, is one of the most common Argo CD tickets on a platform team. It fills the history, fires alerts and trains people to ignore OutOfSync, which defeats the point of GitOps.

This lesson gives you the diagnosis order: argocd app diff (reading < as live and > as git), identify who changes the field (a policy webhook, an HPA, cert-manager, an operator), then fix git first: stop declaring replicas when an HPA owns it, declare imagePullPolicy: Always if policy forces it. Only then a narrow ignore rule. Knowing that ignoring the whole /spec hides real drift is what a reviewer should catch.

FAQ

Why is my app OutOfSync right after a successful sync?

Something in the cluster changes a field you declare in git, every time. Typical culprits: a mutating admission webhook (Kyverno, Gatekeeper mutation, a platform webhook) rewriting imagePullPolicy or injecting resources; a controller owning a field, like an HPA with spec.replicas or cert-manager filling a caBundle; or the API server normalising values such as 1000m to 1. Run argocd app diff to see which field.

Why doesn't a default added by the API server make the app OutOfSync?

Argo CD compares the fields you declare. Fields you never wrote in git, such as terminationGracePeriodSeconds, strategy defaults or a Service's clusterIP, are not part of the desired state and are ignored. A field you declared before and have since removed is still compared through the last-applied annotation, since an apply would delete it, so it shows up in the diff until synced.

How do I read the output of argocd app diff?

It runs as diff LIVE TARGET: lines starting with < are what is live in the cluster, lines starting with > are what git wants. So < imagePullPolicy: Always and > imagePullPolicy: IfNotPresent means the cluster has Always and git says IfNotPresent. The exit code is 1 when there are differences, which makes it usable as a CI check on a pull request with --revision.

What does RespectIgnoreDifferences=true add?

ignoreDifferences alone only affects the sync status: the field no longer makes the app OutOfSync, but a sync still applies git's value to it, and the tug of war with the webhook or controller continues on every sync. With the sync option RespectIgnoreDifferences=true, Argo CD also leaves those fields alone when applying, so the controller that owns them keeps its value.

What is server-side diff and when does it help?

With server-side diff, Argo CD asks the API server to dry-run the apply and compares with that result. The dry run passes through admission, so webhook mutations appear on both sides and stop causing diffs. It is the modern fix for webhook noise. It costs an API call per resource per reconcile, and it does not help with controllers that change fields later, such as an HPA.

In an interview Mid

An Argo CD application keeps flapping between Synced and OutOfSync. How do you fix it properly?

Argo CD compares the fields you declare in git with the live object (normalised). Defaults the API server adds are not compared. So an app that is OutOfSync again right after a successful sync has something in the cluster rewriting a field git declares - and with self-heal on, that becomes a loop of syncs of the same commit.

Usual culprits: an HPA changing replicas, or a mutating admission webhook (a policy engine forcing imagePullPolicy, injecting labels or resources).

  1. See it: argocd app diff APP - < lines are live, > lines are git. kubectl get mutatingwebhookconfigurations lists the webhooks.
  2. Make git match reality first: if an HPA owns replicas, remove replicas from the Deployment in git; if policy forces Always, declare Always.
  3. Ignore narrowly only where a controller legitimately owns a field you must declare: ignoreDifferences on that kind and path (or managedFieldsManagers), with RespectIgnoreDifferences=true so the sync stops overwriting it.
  4. For webhooks, server-side diff puts the mutation on both sides.

Never ignore a whole object or /spec - that hides the drift GitOps exists to catch.

Also asked: What does OutOfSync mean in Argo CD, and what are common causes? · How do you read the output of argocd app diff? · Why should a Deployment scaled by an HPA not declare replicas in git?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.