OnCallReady

Lesson 26.16 · GitOps, Argo CD & Delivery · 10 min read

App of apps, ApplicationSet and projects

In plain words

Imagine a school where each class has a folder of rules. Instead of the head teacher handing out every folder by hand, there is one master folder that just says "these are the class folders". Give the school the master folder once, and it hands out the rest. Want a new class? Add its folder to the master folder. And each class has a rule sheet saying which rooms it may use, so class 3B cannot book the head teacher's office.

In Argo CD the master folder is the root Application of app-of-apps: its path apps/ contains only Application manifests. An ApplicationSet is a stamp that prints one Application per directory, cluster or pull request. An AppProject is the rule sheet: allowed source repos, destination namespaces like pay-*, and which resource kinds a team may create.

The problem

You now have payments-dev and payments-prod, each created with a hand-typed argocd app create. That command is not in git - so the Applications themselves are exactly the kind of unreviewed, unrecorded cluster state GitOps is meant to remove. And with 40 services x 3 environments, nobody wants to write 120 Applications by hand. Also, in a shared cluster, the payments team must not be able to deploy into someone else's namespace.

What you need to know already: 26.4 (the Application object), 26.9 (prune, waves, the resources finalizer), 17.30 (RBAC: roles and bindings), 15.26 (namespaces, labels), 2.16 (timers and cron schedules, for the cron format at the end).

Applications are just manifests

An Application is a Kubernetes object, so it can be a YAML file in git - and another Application can deploy it. The app-of-apps pattern:

payments-gitops/
  apps/                      <- the root app's path: nothing but Application / AppProject manifests
    project.yaml             AppProject payments (sync-wave -1: before the apps that use it)
    payments-dev.yaml        Application -> overlays/dev
    payments-prod.yaml       Application -> overlays/prod
  overlays/...
root.yaml                    the ONE thing you apply by hand: Application payments-root -> apps/

The root app (payments-root) points at apps/; its "manifests" are the child Applications. kubectl apply -f root.yaml once - the bootstrap, the single manual step that starts everything - and from then on adding an environment or a service is a commit to apps/.

With prune: true on the root, deleting a child's file deletes the child Application - and if the child has the resources finalizer (26.9), everything it deployed. Treat apps/ with the same review rules as prod.

Waves (26.9) work between Applications too: give the AppProject sync-wave: "-1", platform apps (the ingress controller from 16.19, cert-manager from 26.12) "0", the services "1". Since Argo CD 1.8 a child's health does not count toward its parent's health by default; teams that need "wait until the child is healthy" add a health check for argoproj.io/Application in argocd-cm.

ApplicationSet

When the Applications all follow one pattern, write the pattern once. An ApplicationSet is an object that generates Applications from a template:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payments
  namespace: argocd
spec:
  goTemplate: true                             # {{ }} placeholders in Go template syntax (25.24)
  generators:
    - git:                                     # one app per directory that matches
        repoURL: https://git.lab/pay/payments-gitops.git
        revision: HEAD
        directories:
          - path: overlays/*
  template:                                    # an Application, with placeholders
    metadata:
      name: 'payments-{{.path.basename}}'      # basename = last part of the path: dev, prod
    spec:
      project: payments
      source:
        repoURL: https://git.lab/pay/payments-gitops.git
        targetRevision: HEAD
        path: '{{.path.path}}'                 # overlays/dev, overlays/prod
      destination:
        server: https://kubernetes.default.svc
        namespace: 'pay-{{.path.basename}}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }

A generator produces a list of parameter sets; the template is filled once per set. The kinds:

Rule of thumb: app-of-apps when each app is hand-written and different, ApplicationSet when they follow a pattern.

AppProject: the tenancy boundary

A tenant is one team sharing a cluster with others. An AppProject is Argo CD's fence around a tenant's Applications: which repos they may use, where they may deploy, which kinds of objects, and who may press which buttons.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments
  namespace: argocd
spec:
  sourceRepos: [https://git.lab/pay/payments-gitops.git]    # where manifests may come FROM
  destinations:                                              # where they may go TO
    - server: https://kubernetes.default.svc
      namespace: pay-*
  clusterResourceWhitelist: []          # no cluster-scoped objects (no ClusterRoles, no CRDs)
  namespaceResourceBlacklist:           # kinds they may NOT create even in their namespaces
    - group: ''
      kind: ResourceQuota               # the platform owns quotas (17.9)
  roles:                                # Argo CD permissions, for groups from single sign-on
    - name: deployer
      policies: [p, proj:payments:deployer, applications, sync, payments/*, allow]
      groups: [payments-team]
  syncWindows:
    - kind: deny                        # no automated syncs during the Friday freeze
      schedule: '0 16 * * 5'            # cron format: minute hour day month weekday -> Fri 16:00
      duration: 64h                     # until Monday 08:00
      applications: ['*-prod']

The policy line reads: policy, for role proj:payments:deployer, on applications, action sync, objects payments/* (every app in the project), allow. A sync window is a time range when syncs are allowed or denied - a change freeze written as config.

An Application that points outside its project is refused with a condition (an error recorded in its status), not synced:

InvalidSpecError: application destination server 'https://kubernetes.default.svc' and namespace 'kube-system' do not match any of the allowed destinations in project 'payments'

The default project allows everything - fine for a lab, an audit finding in a bank. Argo CD-wide permissions live in argocd-rbac-cm (policy.csv with lines like the one above, and policy.default: role:readonly for everyone else); users and single sign-on in argocd-cm. All of that, and the root apps, can themselves live in a platform repo managed by an Argo CD Application: "Argo CD manages Argo CD".

What you can now do

Why it helps

When you join a platform team, the first thing to learn is how the Argo CD setup bootstraps: which root app you apply by hand and where everything else comes from. That tells you where to add a new environment and why deleting a file in apps/ can cascade-delete a whole service. ApplicationSets are how teams scale to many clusters and preview environments without hundreds of hand-written Applications.

AppProjects are the multi-tenancy control auditors ask about: a team's Application pointing at kube-system fails with InvalidSpecError ... do not match any of the allowed destinations, which is the control working. The default project allowing everything is a common audit finding, and sync windows are how you implement a Friday freeze. These patterns come up in senior platform interviews as "how do you manage Argo CD at scale?".

FAQ

What is the app-of-apps pattern?

A root Application whose path contains only other Application (and AppProject) manifests. You apply the root once by hand, the bootstrap, and it creates and manages all child Applications from git. Adding a service or environment becomes a commit to apps/. With prune on the root, deleting a child's file deletes that Application, and with the resources finalizer on the child, everything it deployed, so review apps/ like prod.

When should I use an ApplicationSet instead of app-of-apps?

Use an ApplicationSet when Applications follow a pattern: one per directory in overlays/*, one per registered cluster, one per open pull request, or the combination of environments and clusters with a matrix generator. Use app-of-apps when each Application is hand-written and different. They combine: a root app can deploy ApplicationSets. The ApplicationSet controller creates, updates and deletes Applications as generator results change.

What does an AppProject restrict?

Where manifests may come from (sourceRepos), where they may go (destinations: cluster and namespace patterns like pay-*), which cluster-scoped kinds are allowed (clusterResourceWhitelist, empty means none), which namespaced kinds are forbidden (namespaceResourceBlacklist, like ResourceQuota), project roles mapped to SSO groups, and sync windows. An Application that violates it gets an InvalidSpecError condition and is not synced.

Does a parent app wait for its child applications to be healthy?

Not by default since Argo CD 1.8: an Application's health is not aggregated into its parent, so waves between child apps only order their creation. If you need the parent to wait until a child app is actually healthy, for example platform components before services, add a custom health check for argoproj.io/Application in argocd-cm. Many teams do this for bootstrapping clusters.

How do sync windows work?

They are defined on an AppProject: kind: allow or deny, a cron schedule, a duration and which applications, namespaces or clusters they match. During a deny window, automated syncs do not happen, so a merge on Friday evening does not reach prod until the window ends. manualSync: true decides whether manual syncs are still allowed, for emergencies. They are how you implement change freezes in GitOps.

In an interview Mid

How do you manage many Argo CD Applications without creating each one by hand?

Applications are Kubernetes objects, so they belong in git too - an Application created with argocd app create is unreviewed cluster state.

Rule of thumb: app of apps when each app is different, ApplicationSet for a pattern (40 services x 3 environments).

And fence each team with an AppProject: allowed source repos, destinations (namespaces), resource kinds, roles, sync windows. An app pointing outside it gets InvalidSpecError instead of syncing. The default project allows everything.

Also asked: How do you provide multi-tenancy for several teams on one Argo CD instance? · How would you bootstrap a new cluster with Argo CD so it reaches the full desired state from git? · When would you use an ApplicationSet instead of app of apps?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.