Lesson 26.4 · GitOps, Argo CD & Delivery · 15 min read

Argo CD: how it works, and the Application

Q: Which Argo CD component does what?

argocd-server serves the API, UI and CLI endpoint with auth and RBAC. argocd-repo-server clones git and renders manifests (helm template, kustomize build, plain YAML); rendering errors show up in its logs. argocd-application-controller compares desired and live state, computes sync and health and performs syncs. Redis caches, dex handles SSO, and the applicationset-controller generates Applications from templates.

In plain words

Imagine a librarian with a list of which books should be on each shelf. Every few minutes the librarian walks the shelves, compares them with the list, and marks each shelf "matches" or "doesn't match". Separately, the librarian checks whether the books are readable or have torn pages. A shelf can match the list perfectly and still hold a torn book, or be messy but full of good books.

Argo CD is that librarian. An Application says "this git path goes to this cluster and namespace". The repo-server renders the manifests, the application-controller compares them with the live cluster. The list check is sync status (Synced or OutOfSync); the readable check is health (Healthy, Progressing, Degraded). It rechecks git every 3 minutes, or immediately on a webhook.

The problem

You now have overlays in git (26.2), and you applied them by hand with kubectl apply -k. GitOps (26.1) says a program in the cluster should do that for you, and keep doing it. Argo CD is that program. This lesson is how it is built, the one object you give it (the Application), how to read what it tells you, and the argocd command.

What you need to know already: 26.1 (GitOps, pull, reconcile, drift), 26.2 (kustomize overlays), 15.9 (the reconciliation loop: a controller compares desired and actual state and fixes the difference), 15.16 (Deployments and ReplicaSets), 15.26 (namespaces, labels and annotations).

What Argo CD is

Argo CD is a set of programs you install into the cluster, in their own namespace argocd. Together they are a controller (15.9): a loop that reads the desired state - here from git, not from the Kubernetes API - and makes the cluster match.

The pieces

You do not have to operate these today, but their names show up in pod lists and error messages:

argocd-server                  the API, the web UI, and what the argocd CLI talks to
                               (logins, permissions, single sign-on)
argocd-repo-server             clones git and RENDERS the manifests: runs helm template,
                               kustomize build, or reads plain YAML. Keeps no state.
argocd-application-controller  the reconciler: compares the rendered desired state with
                               the live cluster, works out sync + health, runs syncs
argocd-redis                   a cache (redis = a small in-memory database)
argocd-dex-server              single sign-on: lets people log in with Entra ID (22.8),
                               GitHub or a company directory
argocd-applicationset-controller   creates Applications from a template (26.16)

Render means turn a chart or an overlay into the plain Kubernetes YAML it stands for - what helm template (25.19) or kubectl kustomize (26.2) print.

Everything about Argo CD is itself configured with Kubernetes objects in the argocd namespace: its own object kinds Application and AppProject (added to the API as CRDs, custom resource definitions - new kinds of object a program adds to Kubernetes), the argocd-cm and argocd-rbac-cm ConfigMaps (settings and who may do what), and Secrets holding repo passwords and cluster credentials. So Argo CD's own setup can live in git too.

The Application

An Application is one entry in Argo CD's list of things to deploy: "take this path of this repo, and keep it applied to this namespace of this cluster".

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-dev
  namespace: argocd                 # Applications live in argocd's namespace
spec:
  project: default                  # an AppProject: which repos/clusters/namespaces are allowed (26.16)
  source:                           # WHERE the desired state is
    repoURL: https://git.lab/pay/payments-gitops.git
    targetRevision: HEAD            # which version: a branch, a tag, or a commit (HEAD = latest on the default branch)
    path: overlays/dev              # kustomization.yaml -> kustomize; Chart.yaml -> helm; else plain YAML
    # helm: { valueFiles: [values-dev.yaml], parameters: [...] }   for charts
  destination:                      # WHERE to apply it
    server: https://kubernetes.default.svc   # the cluster Argo CD itself runs in
    namespace: pay-dev              # namespace for objects that do not name one
  syncPolicy:                       # leave it out = you sync by hand
    automated: { prune: true, selfHeal: true }   # 26.9
    syncOptions: [CreateNamespace=true]          # create pay-dev if missing

A sync is Argo CD applying what git renders to the cluster - its kubectl apply. With no syncPolicy you trigger each sync yourself.

Two statuses, always read both

Sync status answers "does the cluster match git?":

Synced       every object Argo CD tracks matches what git renders
OutOfSync    something differs, is missing, or exists in the cluster but not in git
Unknown      Argo CD could not render or compare (bad YAML, repo unreachable)

Health status answers "does what is running work?":

Healthy      Deployments rolled out, pods ready, Services with endpoints...
Progressing  a rollout in progress (or stuck, until progressDeadlineSeconds runs out)
Degraded     failed: a Deployment past its deadline, a crash loop, a failed Job
Missing      the object is in git but not in the cluster
Suspended    paused (a suspended CronJob, a paused rollout)

They are independent, and the pair tells you where to look:

Synced + Degraded: git was applied correctly, and the version in git is broken. Fix it in git (roll forward or revert).
OutOfSync + Healthy: the cluster works but differs from git - drift, or a commit not synced yet.
Synced + Progressing for a long time: the rollout is stuck (readiness probe, image pull, quota - 18.28).

When does Argo CD notice a commit?

Two ways:

Polling: the controller re-reads git and re-renders every app every 3 minutes (the timeout.reconciliation setting, 180s by default).
Webhook: git calls Argo CD's API the moment you push (a webhook is an HTTP call one system makes to another when something happens). Then it is immediate.

"I pushed and nothing happened" is usually the 3 minutes. argocd app get APP --refresh makes it re-read git now; --hard-refresh also throws away its cache of rendered manifests. git.lab in this lab sends webhooks, so a push is seen within seconds.

Separately, the controller watches the cluster itself, so a change made with kubectl shows up as OutOfSync within seconds.

How it knows what it owns

When Argo CD applies an object it adds an annotation (15.26: a key-value note on an object) with the app's name:

argocd.argoproj.io/tracking-id: payments-dev:apps/Deployment:pay-dev/payments-api

(Older versions used the app.kubernetes.io/instance label instead, which clashed with Helm charts that set the same label.) An object with this annotation that is no longer in git is an orphan - left behind, a candidate for pruning (26.9). An object without it is not Argo CD's business.

The CLI

argocd is the command-line client. The shape is argocd app VERB APPNAME:

argocd admin initial-password -n argocd       # print the first admin password (kept in a Secret)
argocd login argocd.lab --username admin      # open a session with the server; asks for the password
argocd login --core                           # no server: talk to the Kubernetes API directly
argocd app list                               # every Application, one line each
argocd app get payments-dev [--refresh]       # summary, statuses, every resource with its status
argocd app diff payments-dev                  # git vs live, as a diff (exit code 1 = there are differences)
argocd app create NAME --repo URL --path DIR --dest-server URL --dest-namespace NS
                                              # register an Application (manual sync unless --sync-policy)
argocd app sync payments-dev [--prune]        # apply now; prints each resource's result
argocd app wait payments-dev --health --timeout 180   # block until Healthy (or give up after 180 s)
argocd app history payments-dev               # past syncs, each with an ID and the git commit
argocd app rollback payments-dev ID           # apply an older ID's content again (manual sync only)
argocd app set payments-dev --sync-policy automated --self-heal --auto-prune
                                              # change settings: here, turn on auto-sync (26.9)

argocd app rollback is refused while auto-sync is on (rollback cannot be initiated when auto-sync is enabled): the controller would immediately sync back to whatever git says. In GitOps the real rollback is git revert (25.2).

What you can now do

Describe what Argo CD's parts do and write an Application for a path in a repo.
Read sync status and health status together and say where the problem is.
Use argocd app get / diff / sync / history to inspect and deploy an app.

Why it helps

When the page says "payments-prod is Degraded", reading sync and health together tells you where to look. Synced plus Degraded means git was applied faithfully and the version in git is broken: revert in git or roll forward. OutOfSync plus Healthy means the cluster works but differs from git: drift or a pending change. Synced plus Progressing forever means a stuck rollout: readiness, image pull or quota.

"I pushed and nothing happened" is usually the 3-minute poll, and argocd app get --refresh is the answer. argocd app rollback being refused with auto-sync on is not a bug. Knowing the components (repo-server renders, controller reconciles) tells you which pod's logs to read when rendering or syncing fails. These are exactly the questions asked in Argo CD interviews and platform on-call handovers.

FAQ

What is the difference between sync status and health status?

Sync status compares what git renders with the live objects: Synced, OutOfSync, or Unknown when Argo CD cannot render or compare. Health says whether the running objects work: Healthy, Progressing, Degraded, Missing, Suspended. They are independent. An app can be Synced and Degraded (the version in git is broken) or OutOfSync and Healthy (the cluster works but has drifted or a change is not yet synced).

I pushed a commit and Argo CD didn't deploy it. Why?

Usually timing: without a git webhook Argo CD re-renders apps every 3 minutes (timeout.reconciliation). argocd app get APP --refresh forces it now, --hard-refresh also clears the manifest cache. Other causes: the Application tracks another branch or path (targetRevision, path), sync is manual so it only shows OutOfSync, a sync window denies it, or a previous sync of that commit failed and auto-sync will not retry without retry:.

How does Argo CD know which resources belong to an Application?

Argo CD 3.x marks every object it applies with the annotation argocd.argoproj.io/tracking-id, containing the app name, group, kind, namespace and name. An object with the annotation that is no longer rendered from git is an orphan to prune; an object without it is ignored. Older versions used the app.kubernetes.io/instance label by default, which clashed with Helm charts that set the same label.

Why is argocd app rollback refused?

Because the Application has auto-sync enabled. With auto-sync, the controller would immediately sync back to what git says, undoing the rollback, so Argo CD refuses with rollback cannot be initiated when auto-sync is enabled. In GitOps the real rollback is git revert of the bad commit. argocd app rollback is for manual-sync apps, and even then git and cluster disagree until git is fixed.

Which Argo CD component does what?

argocd-server serves the API, UI and CLI endpoint with auth and RBAC. argocd-repo-server clones git and renders manifests (helm template, kustomize build, plain YAML); rendering errors show up in its logs. argocd-application-controller compares desired and live state, computes sync and health and performs syncs. Redis caches, dex handles SSO, and the applicationset-controller generates Applications from templates.

In an interview Mid

Explain how Argo CD works.

Argo CD is a controller installed in the cluster (namespace argocd) whose desired state comes from git. The repo-server clones git and renders the manifests (helm template, kustomize build or plain YAML); the application controller compares them with the live objects and syncs (applies) the difference; argocd-server is the API, UI and what the argocd CLI talks to.

You give it Applications: "this path of this repo, kept applied to this namespace of this cluster". It notices commits by polling (every 3 minutes by default) or immediately via a webhook; argocd app get APP --refresh forces it. Objects it applies carry a tracking-id annotation, which is how it knows what it owns.

Every app has two independent statuses - read both:

Sync: Synced / OutOfSync (does the cluster match git?).
Health: Healthy, Progressing, Degraded, Missing, Suspended (does it work?).

Synced + Degraded = git was applied and the version in git is broken: fix it in git. OutOfSync + Healthy = drift or an unsynced commit. Rollback is git revert; argocd app rollback is refused while auto-sync is on.

Also asked: An Argo CD application shows Synced but Degraded. What does it mean and how do you proceed? · You pushed a commit and Argo CD did nothing for a few minutes. Why? · Why does argocd app rollback refuse to run when auto-sync is enabled?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.