"We requested a database twenty minutes ago and have nothing." In Crossplane that means a composite resource (XR) that is not Ready. The cause is rarely in the XR itself. An XR depends on a Composition, a function pipeline, the managed resources (MRs) it composes, a provider, a ProviderConfig and some credentials. The way to find it is to walk that chain from the top and stop at the first object that is not healthy.
The examples are Crossplane v2.4 (namespaced XRs, apiextensions.crossplane.io/v2 XRDs), the crossplane CLI v2.5 and function-patch-and-transform.
$ kubectl get database orders -n team-a
NAME SYNCED READY COMPOSITION AGE
orders False 19mStep 1: the tree
crossplane resource trace shows the XR and everything it composed, with each object's Synced, Ready and the reason for the first problem:
$ crossplane resource trace database orders -n team-a
NAME SYNCED READY STATUS
Database/orders (team-a) False - ReconcileError: cannot select Composition: no compatible Compositions found(Older guides use crossplane beta trace. The current CLI calls it crossplane resource trace.) How to read it:
- SYNCED False means Crossplane could not do its job on this object (select a Composition, run the pipeline, apply a resource, reach the cloud). READY False means the work is done but the result is not there yet.
- STATUS is the reason and message of the condition that is not healthy, cut to the last 64 characters by default (a leading
...).-o wideprints the whole message. - An XR rolls its children up:
Unsynced resources: dbandUnready resources: dbname the composed resources (by their name in the Composition) that are the problem.
Here there are no children at all. The XR never got as far as composing anything.
Problem 1: no compatible Compositions found
$ kubectl get compositions
NAME XR-KIND XR-APIVERSION AGE
database-rds Database platform.lab/v1alpha2 20m
$ kubectl get database orders -n team-a -o jsonpath='{.apiVersion}{"\n"}'
platform.lab/v1alpha1A Composition is only a candidate for an XR if its compositeTypeRef matches the XR's apiVersion and kind exactly. This one was written for v1alpha2, and the team's request is v1alpha1. Fixing the file and applying it does not work, though:
$ sed -i 's#platform.lab/v1alpha2#platform.lab/v1alpha1#' ~/oncall-lab/labs/platform/database-comp.yaml
$ kubectl apply -f ~/oncall-lab/labs/platform/database-comp.yaml
The Composition "database-rds" is invalid: spec.compositeTypeRef: Invalid value: "object": Value is immutable
$ kubectl delete composition database-rds && kubectl apply -f ~/oncall-lab/labs/platform/database-comp.yaml
composition.apiextensions.crossplane.io "database-rds" deleted
composition.apiextensions.crossplane.io/database-rds createdcompositeTypeRef is immutable (a CEL rule on the Composition CRD: self == oldSelf), so you delete the Composition and create it again.
The sibling error is cannot fetch Composition: ... not found. The XR selected a Composition once, recorded it in spec.crossplane.compositionRef, and that Composition has since been deleted or renamed. Creating a new one does not move existing XRs. You point the reference at the new one yourself.
Problem 2: Unsynced resources
$ crossplane resource trace database orders -n team-a
NAME SYNCED READY STATUS
Database/orders (team-a) False False ReconcileError: Unsynced resources: dbProgress: a Composition was selected and the pipeline ran, but the composed resource db could not be applied. The reason is in the XR's events:
$ kubectl describe database orders -n team-a | grep -A1 "cannot apply"
Warning ComposeResources 3s (x2 over 8s) defined/compositeresourcedefinition.apiextensions.crossplane.io cannot apply composed resource "db": Instance.rds.aws.m.upbound.io "orders-d71ae26354a3" is invalid: spec.forProvider.region: Required value
Normal ComposeResources 3s (x2 over 8s) defined/compositeresourcedefinition.apiextensions.crossplane.io Composed resource "db" is not yet validThe API server refused the RDS Instance: region is a required field and it is missing. But the Composition has a patch for it:
- fromFieldPath: spec.regoin
toFieldPath: spec.forProvider.regionspec.regoin is a typo. In function-patch-and-transform, a fromFieldPath that does not exist on the XR is not an error. The default policy is Optional, so the patch is skipped without a word, and the error shows up one level further down, as an invalid managed resource.
Prove it without the cluster: crossplane composition render
crossplane composition render runs the Composition's function pipeline locally (the functions run in Docker) against an XR file, and prints what Crossplane would compose:
$ crossplane composition render ~/oncall-lab/labs/platform/orders.yaml ~/oncall-lab/labs/platform/database-comp.yaml ~/oncall-lab/labs/platform/functions.yaml
...
apiVersion: rds.aws.m.upbound.io/v1beta1
kind: Instance
metadata:
annotations:
crossplane.io/composition-resource-name: db
generateName: orders-
...
spec:
forProvider:
allocatedStorage: 20
autoGeneratePassword: true
engine: postgres
engineVersion: "17"
instanceClass: db.t4g.micro
passwordSecretRef:
key: password
name: orders-db-password
skipFinalSnapshot: true
username: appNo region. Fix the path and render again before you touch the cluster:
$ sed -i 's#spec.regoin#spec.region#' ~/oncall-lab/labs/platform/database-comp.yaml && crossplane composition render ~/oncall-lab/labs/platform/orders.yaml ~/oncall-lab/labs/platform/database-comp.yaml ~/oncall-lab/labs/platform/functions.yaml | grep region
region: eu-central-1
$ kubectl apply -f ~/oncall-lab/labs/platform/database-comp.yaml
composition.apiextensions.crossplane.io/database-rds configured
$ crossplane resource trace database orders -n team-a
NAME SYNCED READY STATUS
Database/orders (team-a) True False Creating: Unready resources: db
└─ Instance/orders-d71ae26354a3 (team-a) True False CreatingSynced everywhere, and Ready False with Creating is now the honest state: RDS takes minutes. Later:
$ crossplane resource trace database orders -n team-a
NAME SYNCED READY STATUS
Database/orders (team-a) True True Available
└─ Instance/orders-d71ae26354a3 (team-a) True True AvailableIf a field must always arrive, say so in the Composition: policy: {fromFieldPath: Required} on the patch turns a silent skip into an error on the XR.
Problem 3: the provider cannot reach the cloud
A different morning: "our StorageBucket has said Creating for forty minutes, is S3 down?"
$ crossplane resource trace storagebucket invoices -n team-c
NAME SYNCED READY STATUS
StorageBucket/invoices (team-c) True False Creating: Unready resources: bucket
└─ Bucket/invoices-15370220dc8b (team-c) False - ReconcileError: ...iredToken: The security token included in the request is expiredThe XR is fine, and the managed resource below it is not Synced. With -o wide the full message reads connect failed: cannot get terraform setup: cannot get account id: cannot get the caller identity: GetCallerIdentity query failed: ... api error ExpiredToken: The security token included in the request is expired. Managed resource errors start with the step that failed: connect failed (ProviderConfig, Secret, credentials), observe failed, create failed, update failed, delete failed. Read them from the right for the cloud's own error code (the same codes the AWS CLI prints: AccessDenied, for example). Then look at the other teams' resources:
$ kubectl get buckets -A
NAMESPACE NAME SYNCED READY EXTERNAL-NAME AGE
team-a reports-f0d5f2fdfff0 False True reports-f0d5f2fdfff0 120m
team-b uploads-ea9584646c97 False True uploads-ea9584646c97 120m
team-c invoices-15370220dc8b False 40mEveryone is affected. The old buckets still say READY True because they exist, but SYNCED False: nobody is managing them any more. App monitoring sees nothing wrong, which is why you should alert on Synced=False that lasts. The credentials explain it:
$ kubectl get secret aws-creds -n crossplane-system -o jsonpath='{.data.credentials}' | base64 -d | cut -c1-24
[default]
aws_access_key_id = ASIA
aws_secret_access_key =
aws_session_token = IQoJ(cut keeps the secret values off your screen.) An access key ID starting with ASIA plus a session token is a temporary STS session, and someone pasted it into the provider's Secret. Replace it with credentials that do not expire, or better, stop using keys at all: on EKS, the AWS provider's IRSA or Pod Identity credential sources give it short-lived credentials that renew themselves.
The failure catalogue
| What trace / the conditions say | Where to look |
|---|---|
cannot select Composition: no compatible Compositions found | kubectl get compositions: XR-KIND and XR-APIVERSION must match the XR exactly |
cannot fetch Composition: ... not found | spec.crossplane.compositionRef points at a Composition that is gone |
cannot compose resources: cannot run Composition pipeline step ... | kubectl get functions: the function is missing or unhealthy |
... returned a fatal result: ... | the function's own message: patch index, transform index |
Unsynced resources: x | the XR's events: the API server's reason for refusing resource x |
Unready resources: x | the child x itself: often just "Creating", give it time |
child shows - in SYNCED and READY | no healthy provider for that kind: kubectl get providers (INSTALLED and HEALTHY) |
connect failed: ... | ProviderConfig, its Secret, the credentials |
create failed / observe failed: ... | the cloud's error code, at the end of the message |
Step by step, every time
crossplane resource trace <kind> <name> -n <ns>. Add-o widefor full messages.- Take the first row that is not
True True:kubectl describeit and read its events. - A child that has no conditions at all means the provider is not running.
- When the pipeline is the suspect, use
crossplane composition renderlocally. - After the fix, run trace again. Crossplane retries failures with backoff, so a fixed cause clears on its own within a minute or two.
A platform usually puts a form in front of these XRs. When the request comes from a Backstage template, check that its entity reached the catalog too: catalog-info.yaml validation errors.
Practise it
In the Platform Engineering: Backstage & Crossplane chapter, the lesson Debugging composite resources on call walks the whole catalogue. The lab The database that never comes is problems 1 and 2 above (fix both, rendering before you apply), and the incident new buckets never become Ready is problem 3, with all three teams waiting.