OnCallReadyStart learning

Infrastructure as code Lesson 42.14 · Platform Engineering: Backstage & Crossplane · 15 min read

Changing a platform without destroying data

In plain words

A recipe change in a restaurant chain reaches every restaurant at once. If the new recipe renames "the soup" to "the starter", every kitchen throws away the soup it was keeping warm and starts a new pot. For soup that is a waste; for a database it is a disaster. This lesson is about the rules that stop a recipe change from throwing away what matters.

Changing a platform without destroying data

Creating resources is the easy part. A platform lives for years: Compositions change, XRDs gain fields, providers are upgraded, teams delete things. Every one of those changes reaches every XR that uses them - possibly hundreds of production databases at once. This lesson is about the mechanics that decide what happens, and the guard rails that stop a one-line change from replacing a database.

Composition revisions

Every time a Composition changes, Crossplane stores an immutable copy as a CompositionRevision named <composition>-<7 hex of a hash> and numbered:

$ kubectl get compositionrevisions
NAME                    REVISION   XR-KIND    XR-APIVERSION           AGE
database-rds-067ab94    2          Database   platform.lab/v1alpha1   4s
database-rds-a141dff    1          Database   platform.lab/v1alpha1   3m24s

Each XR records the revision it uses in spec.crossplane.compositionRevisionRef and follows a compositionUpdatePolicy:

  • Automatic (the default): the XR moves to the newest revision on its next reconcile. Change the Composition, and within a minute every XR is rebuilt from the new version.
  • Manual: the XR stays on its revision until someone changes compositionRevisionRef by hand.

Manual is how you roll out a risky change gradually: pin production XRs, let the change reach staging, render and compare, then move one production XR at a time. An XRD can set defaultCompositionUpdatePolicy: Manual for every new XR of its kind.

Garbage collection, and the rename trap

On every reconcile the pipeline returns the desired composed resources, by composition resource name. Anything Crossplane composed before that is not in the desired set any more is deleted. That is how removing a resource from a Composition removes it from the world - and it is exactly what makes this innocent-looking change dangerous:

       resources:
-      - name: db
+      - name: database
         base:
           apiVersion: rds.aws.m.upbound.io/v1beta1
           kind: Instance

To Crossplane, db is no longer desired and database is new. Composed names are <xr>-<hash of the XR's UID and the resource name>, so the new name gives a new object: Crossplane deletes the old Instance (and its finalizer makes the provider delete the RDS instance) and creates a new, empty one:

$ crossplane resource trace database orders
NAME                                        SYNCED   READY   STATUS
Database/orders (default)                   True     False   Creating: Unready resources: database
└─ Instance/orders-6c2fb8161588 (default)   True     False   Creating
$ kubectl get instances
NAME                  SYNCED   READY   EXTERNAL-NAME         AGE
orders-6c2fb8161588   True     False   orders-6c2fb8161588   4s
orders-f29bfca80b19   True     False   orders-f29bfca80b19   3m24s

With Automatic updates this happens to every XR of the kind, within a minute of kubectl apply. A composition resource name is an identity, not a label: never rename one that owns data. The same goes for changing a resource's kind (Crossplane refuses to change the kind of an existing composed resource; the usual "fix" of renaming it to get around that is the trap again).

Guard rail 1: never let Crossplane delete the data

For anything stateful, take Delete out of the managed resource's managementPolicies in the Composition's base:

        base:
          apiVersion: rds.aws.m.upbound.io/v1beta1
          kind: Instance
          spec:
            managementPolicies: ["Observe", "Create", "Update", "LateInitialize"]
            forProvider:
              deletionProtection: true

Now deleting the managed resource - by hand, by garbage collection, or by deleting the XR - only removes the Kubernetes object. The RDS instance stays and can be re-imported with its external name. Add the cloud's own protection too: deletionProtection: true makes AWS refuse the delete even if someone gets past Crossplane.

Guard rail 2: a Usage

A Usage (protection.crossplane.io/v1beta1) blocks the deletion of a resource while it is in use. It is enforced by Crossplane's admission webhook, nousages.protection.crossplane.io, so it stops kubectl delete and Crossplane's own garbage collection alike:

apiVersion: protection.crossplane.io/v1beta1
kind: Usage
metadata:
  name: protect-orders-db
  namespace: default
spec:
  of:
    apiVersion: rds.aws.m.upbound.io/v1beta1
    kind: Instance
    resourceRef:
      name: orders-f29bfca80b19
  reason: production database - never delete
$ kubectl get usages
NAME                DETAILS                                                             READY   AGE
protect-orders-db   Instance/orders-f29bfca80b19 - production database - never delete   True    5s
$ kubectl delete instance orders-f29bfca80b19
Error from server (This resource is in-use by 1 usage(s), including the *v1beta1.Usage "protect-orders-db" (in namespace "default") with reason: "production database - never delete".): admission webhook "nousages.protection.crossplane.io" denied the request: This resource is in-use by 1 usage(s), including the *v1beta1.Usage "protect-orders-db" (in namespace "default") with reason: "production database - never delete".

(Yes, the message appears twice: the webhook puts it in the status reason, and kubectl prints the reason in brackets before the message.) With the rename mistake from above, the XR now reports the refusal instead of losing the data:

Database/orders (default)   False   True   ReconcileError: cannot compose resources: cannot garbage collect composed resources that are no longer desired: cannot delete composed resource "db" (a Instance named orders-f29bfca80b19): admission webhook "nousages.protection.crossplane.io" denied the request: ...

A Usage can also say by another resource ("this bucket is used by that app"), so dependencies are deleted in the right order; ClusterUsage does the same for cluster-scoped resources.

Deleting XRs, XRDs and providers

  • Deleting an XR deletes its composed resources (it owns them); each managed resource deletes its external resource unless its policies lack Delete.
  • Deleting an XRD deletes its CRD, and with it every XR of that kind in every namespace. Treat XRDs like databases: no casual deletes.
  • Deleting a Composition does not touch existing XRs (they keep their revision), but they report errors on their next change.
  • Uninstalling a provider before its managed resources also removes the provider's CRDs (they are owned by its package, so Kubernetes garbage-collects them), and deleting a CRD deletes every object of that kind: all its managed resources, in every namespace, get a deletion timestamp - with a finalizer nobody will remove. They (and any namespace that contains them) hang in deletion; the external resources stay, unmanaged. The order is: delete the XRs and managed resources, wait until they are gone, then remove the provider. If you are already stuck: removing the finalizers by hand orphans the external resources; reinstalling the provider lets it finish the deletions - of every managed resource that was marked, not only the ones you meant. Crossplane 2.x has an alpha flag, --enable-provider-deletion-protection, that refuses to delete a provider while it still has managed resources. (simulator) The lab keeps the provider's CRDs when a provider is removed.

Rolling out a Composition change safely

  1. Render the change against a copy of a real XR (crossplane composition render), with --observed-resources for the composed resources it has today. Diff the output with the current one: a resource that disappears or gets a new name means a delete.
  2. Check managementPolicies and Usages on everything stateful.
  3. Pin production XRs with compositionUpdatePolicy: Manual (or a revision selector), apply, let staging follow, watch it.
  4. Move production XRs one at a time by setting compositionRevisionRef.
  5. Keep the change in git and review it like application code: a Composition is production code.

In an interview: Crossplane garbage-collects composed resources that leave the desired state, so renaming a resource in a Composition deletes the old one and creates a new one - for every XR on Automatic updates. Protect stateful resources with managementPolicies without Delete, the cloud's deletionProtection and a Usage, and roll out Composition changes with Manual update policy and crossplane composition render.

Why it helps

The most damaging Crossplane incidents are not outages but data loss from an innocent Composition change. Understanding revisions, update policies and garbage collection, and putting three independent guard rails on stateful resources, is what lets a platform team change its compositions every week without fear, and it is a common senior interview question about running Crossplane safely in real production clusters.

Commands in this lesson

kubectl crossplane

FAQ

What is a CompositionRevision?

An immutable copy of a Composition, created every time it changes, named after the composition and a short hash and numbered. XRs record which revision they use, so you can see exactly what changed between versions by comparing revisions.

What does compositionUpdatePolicy Manual do?

It keeps an XR on its current revision until someone changes compositionRevisionRef by hand. Automatic, the default, moves the XR to the newest revision on its next reconcile, so a Composition change reaches every XR within a minute.

Why is renaming a resource in a Composition dangerous?

Composed names come from the XR's UID and the resource name, so a new name is a new object. Crossplane garbage collects whatever is no longer desired, deleting the old resource and, through its finalizer, the real database, then creates a new empty one.

What are the guard rails for a database?

No Delete in the managed resource's managementPolicies, so deleting the object orphans the database; the cloud's deletion protection, so AWS refuses a delete; and a Usage, so the API server refuses to delete the object at all. Pin production XRs with Manual update policy as well.

Can I delete an XRD safely?

No. Deleting an XRD deletes its CRD and with it every composite resource of that kind in every namespace, which in turn deletes their composed resources unless their policies prevent it. Treat XRDs like databases.

In an interview Mid

How do you change a Crossplane Composition in production without losing data?

Composed resources that leave the desired state are garbage collected, so a renamed or removed resource is deleted for every XR on Automatic updates. I protect stateful resources with managementPolicies without Delete, the cloud's deletion protection and a Usage; pin production XRs with compositionUpdatePolicy Manual; render the change with crossplane composition render and observed resources and diff it; let staging follow first; then move production XRs one at a time by setting their revision.

Also asked: What is a Crossplane Usage? · What is the difference between Automatic and Manual update policy? · What happens when you uninstall a provider before its resources?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.