Changing a platform without destroying data
Creating resources is the easy part. A platform lives for years: Compositions change, XRDs gain fields, providers are upgraded, teams delete things. Every one of those changes reaches every XR that uses them - possibly hundreds of production databases at once. This lesson is about the mechanics that decide what happens, and the guard rails that stop a one-line change from replacing a database.
Composition revisions
Every time a Composition changes, Crossplane stores an immutable copy as a CompositionRevision named <composition>-<7 hex of a hash> and numbered:
$ kubectl get compositionrevisions
NAME REVISION XR-KIND XR-APIVERSION AGE
database-rds-067ab94 2 Database platform.lab/v1alpha1 4s
database-rds-a141dff 1 Database platform.lab/v1alpha1 3m24s
Each XR records the revision it uses in spec.crossplane.compositionRevisionRef and follows a compositionUpdatePolicy:
Automatic(the default): the XR moves to the newest revision on its next reconcile. Change the Composition, and within a minute every XR is rebuilt from the new version.Manual: the XR stays on its revision until someone changescompositionRevisionRefby hand.
Manual is how you roll out a risky change gradually: pin production XRs, let the change reach staging, render and compare, then move one production XR at a time. An XRD can set defaultCompositionUpdatePolicy: Manual for every new XR of its kind.
Garbage collection, and the rename trap
On every reconcile the pipeline returns the desired composed resources, by composition resource name. Anything Crossplane composed before that is not in the desired set any more is deleted. That is how removing a resource from a Composition removes it from the world - and it is exactly what makes this innocent-looking change dangerous:
resources:
- - name: db
+ - name: database
base:
apiVersion: rds.aws.m.upbound.io/v1beta1
kind: Instance
To Crossplane, db is no longer desired and database is new. Composed names are <xr>-<hash of the XR's UID and the resource name>, so the new name gives a new object: Crossplane deletes the old Instance (and its finalizer makes the provider delete the RDS instance) and creates a new, empty one:
$ crossplane resource trace database orders
NAME SYNCED READY STATUS
Database/orders (default) True False Creating: Unready resources: database
└─ Instance/orders-6c2fb8161588 (default) True False Creating
$ kubectl get instances
NAME SYNCED READY EXTERNAL-NAME AGE
orders-6c2fb8161588 True False orders-6c2fb8161588 4s
orders-f29bfca80b19 True False orders-f29bfca80b19 3m24s
With Automatic updates this happens to every XR of the kind, within a minute of kubectl apply. A composition resource name is an identity, not a label: never rename one that owns data. The same goes for changing a resource's kind (Crossplane refuses to change the kind of an existing composed resource; the usual "fix" of renaming it to get around that is the trap again).
Guard rail 1: never let Crossplane delete the data
For anything stateful, take Delete out of the managed resource's managementPolicies in the Composition's base:
base:
apiVersion: rds.aws.m.upbound.io/v1beta1
kind: Instance
spec:
managementPolicies: ["Observe", "Create", "Update", "LateInitialize"]
forProvider:
deletionProtection: true
Now deleting the managed resource - by hand, by garbage collection, or by deleting the XR - only removes the Kubernetes object. The RDS instance stays and can be re-imported with its external name. Add the cloud's own protection too: deletionProtection: true makes AWS refuse the delete even if someone gets past Crossplane.
Guard rail 2: a Usage
A Usage (protection.crossplane.io/v1beta1) blocks the deletion of a resource while it is in use. It is enforced by Crossplane's admission webhook, nousages.protection.crossplane.io, so it stops kubectl delete and Crossplane's own garbage collection alike:
apiVersion: protection.crossplane.io/v1beta1
kind: Usage
metadata:
name: protect-orders-db
namespace: default
spec:
of:
apiVersion: rds.aws.m.upbound.io/v1beta1
kind: Instance
resourceRef:
name: orders-f29bfca80b19
reason: production database - never delete
$ kubectl get usages
NAME DETAILS READY AGE
protect-orders-db Instance/orders-f29bfca80b19 - production database - never delete True 5s
$ kubectl delete instance orders-f29bfca80b19
Error from server (This resource is in-use by 1 usage(s), including the *v1beta1.Usage "protect-orders-db" (in namespace "default") with reason: "production database - never delete".): admission webhook "nousages.protection.crossplane.io" denied the request: This resource is in-use by 1 usage(s), including the *v1beta1.Usage "protect-orders-db" (in namespace "default") with reason: "production database - never delete".
(Yes, the message appears twice: the webhook puts it in the status reason, and kubectl prints the reason in brackets before the message.) With the rename mistake from above, the XR now reports the refusal instead of losing the data:
Database/orders (default) False True ReconcileError: cannot compose resources: cannot garbage collect composed resources that are no longer desired: cannot delete composed resource "db" (a Instance named orders-f29bfca80b19): admission webhook "nousages.protection.crossplane.io" denied the request: ...
A Usage can also say by another resource ("this bucket is used by that app"), so dependencies are deleted in the right order; ClusterUsage does the same for cluster-scoped resources.
Deleting XRs, XRDs and providers
- Deleting an XR deletes its composed resources (it owns them); each managed resource deletes its external resource unless its policies lack
Delete. - Deleting an XRD deletes its CRD, and with it every XR of that kind in every namespace. Treat XRDs like databases: no casual deletes.
- Deleting a Composition does not touch existing XRs (they keep their revision), but they report errors on their next change.
- Uninstalling a provider before its managed resources also removes the provider's CRDs (they are owned by its package, so Kubernetes garbage-collects them), and deleting a CRD deletes every object of that kind: all its managed resources, in every namespace, get a deletion timestamp - with a finalizer nobody will remove. They (and any namespace that contains them) hang in deletion; the external resources stay, unmanaged. The order is: delete the XRs and managed resources, wait until they are gone, then remove the provider. If you are already stuck: removing the finalizers by hand orphans the external resources; reinstalling the provider lets it finish the deletions - of every managed resource that was marked, not only the ones you meant. Crossplane 2.x has an alpha flag,
--enable-provider-deletion-protection, that refuses to delete a provider while it still has managed resources. (simulator) The lab keeps the provider's CRDs when a provider is removed.
Rolling out a Composition change safely
- Render the change against a copy of a real XR (
crossplane composition render), with--observed-resourcesfor the composed resources it has today. Diff the output with the current one: a resource that disappears or gets a new name means a delete. - Check
managementPoliciesand Usages on everything stateful. - Pin production XRs with
compositionUpdatePolicy: Manual(or a revision selector), apply, let staging follow, watch it. - Move production XRs one at a time by setting
compositionRevisionRef. - Keep the change in git and review it like application code: a Composition is production code.
In an interview: Crossplane garbage-collects composed resources that leave the desired state, so renaming a resource in a Composition deletes the old one and creates a new one - for every XR on Automatic updates. Protect stateful resources with managementPolicies without Delete, the cloud's deletionProtection and a Usage, and roll out Composition changes with Manual update policy and crossplane composition render.