Debugging a composite resource
A team says "my database is not ready". Crossplane problems are rarely in one object: the XR depends on a Composition, a function, composed managed resources, a provider, a ProviderConfig and credentials. The skill is to walk that chain top-down and stop at the first thing that is not healthy.
Database/orders ──(Composition, CompositionRevision, Function)──▶ composed resources
│
Provider (Installed, Healthy) ◀── ProviderConfig ◀───┘ ──▶ AWS API
Step 1: the tree
crossplane resource trace shows the XR and everything it composed, with Synced, Ready and the reason (and the end of the message) for each:
$ crossplane resource trace database orders
NAME SYNCED READY STATUS
Database/orders (default) True False Creating: Unready resources: db
└─ Instance/orders-2cd4e17cab08 (default) False - ReconcileError: ...fig: clusterproviderconfigs.aws.m.upbound.io "default" not found
How to read it:
- The STATUS column shows the Synced reason and message when Synced is not True, otherwise the Ready one. A healthy row just says
Available. - The message is cut to its last 64 characters (that leading
...). Run it with-o widefor the full text and the composition resource name of each child. - The XR's own status rolls up its children:
Unready resources: dbnames the composed resources (by composition resource name) that are not ready yet;Unsynced resources: dbnames the ones Crossplane could not apply. -o yaml/-o jsonprint every object in the tree;-o dota graph.
Here the XR is fine; its database is not, and the reason is at the bottom of the tree: the AWS ClusterProviderConfig named default does not exist.
Step 2: conditions and events on the object that is wrong
kubectl describe instance orders-2cd4e17cab08 -n default
kubectl get events -n default --field-selector involvedObject.name=orders-2cd4e17cab08
Events give you the history (how long has it failed, did it ever work). The XR's events come from the component defined/compositeresourcedefinition.apiextensions.crossplane.io (Crossplane's definition controller runs every XR controller); a managed resource's from managed/<group>/<version>, kind=<kind>.
Step 3: the package and its config
If managed resources do not even get conditions, the provider is not running:
kubectl get providers # INSTALLED and HEALTHY must both be True
kubectl get functions
kubectl get clusterproviderconfigs.aws.m.upbound.io
A managed resource that nobody reconciles has no conditions at all (the SYNCED and READY columns are empty, trace shows -). That always means: no healthy provider for this kind.
The failure catalogue
The messages below are the real ones; learn to recognise them.
No Composition fits.
Database/orders (default) False - ReconcileError: cannot select Composition: no compatible Compositions found
No Composition has a compositeTypeRef with this XR's exact apiVersion and kind (or none carries the labels its compositionSelector asks for). Compare kubectl get compositions (columns XR-KIND and XR-APIVERSION) with the XR's apiVersion. A Composition for platform.lab/v1alpha2 does not serve v1alpha1.
The Composition the XR remembers is gone.
Database/orders (default) False False ReconcileError: cannot fetch Composition: cannot get Composition: compositions.apiextensions.crossplane.io "database-typo" not found
The choice of Composition is made once and written to spec.crossplane.compositionRef. Deleting that Composition and creating another does not move existing XRs; you set the reference yourself (kubectl patch or edit the XR).
The function is not installed (or not healthy).
cannot compose resources: cannot run Composition pipeline step "patch-and-transform": cannot find an active FunctionRevision (a FunctionRevision with spec.desiredState: Active)
The function refused its input or failed. The function's own message follows returned a fatal result::
cannot compose resources: pipeline step "patch-and-transform" returned a fatal result: cannot render composed resource "db" "FromCompositeFieldPath" patch at index 0: transform at index 0 returned error: map transform could not resolve: key huge is not found in map
The resource name, the patch index and the transform index point at the exact line of the Composition.
A composed resource is invalid. Crossplane applies it; the API server validates it against the provider's CRD and refuses:
Database/orders (default) False False ReconcileError: Unsynced resources: db
The XR's events carry the API server's message:
Warning ComposeResources 2s defined/compositeresourcedefinition.apiextensions.crossplane.io cannot apply composed resource "db": Instance.rds.aws.m.upbound.io "orders-f29bfca80b19" is invalid: spec.forProvider.region: Required value
A required field is missing - typically because a patch's fromFieldPath has a typo and was silently skipped, so the value never arrived.
Crossplane may not create that kind.
cannot compose resources: cannot apply composed resource "ingress": ingresses.networking.k8s.io "web-3f1a7c09be42" is forbidden: User "system:serviceaccount:crossplane-system:crossplane" cannot patch resource "ingresses" in API group "networking.k8s.io" in the namespace "team-a"
Grant it with a ClusterRole labelled rbac.crossplane.io/aggregate-to-crossplane: "true" (the compositions lesson).
The provider cannot authenticate or is refused by the cloud. These live on the managed resource, prefixed by the step: connect failed: ... (ProviderConfig, Secret, credentials), observe failed: ..., create failed: async create failed: .... Read them from the right for the cloud's error code: ExpiredToken, InvalidClientTokenId, AccessDenied, BucketAlreadyExists.
Step 4: reproduce the composition without the cluster
When the pipeline is the suspect, take the XR and the Composition out of the cluster and render them. You see exactly what the functions produce, without waiting for reconciles and without touching AWS:
kubectl get database orders -n default -o yaml > xr.yaml
kubectl get composition database-rds -o yaml > composition.yaml
kubectl get function function-patch-and-transform -o yaml > functions.yaml
crossplane composition render xr.yaml composition.yaml functions.yaml
A field that is missing from the rendered resource is a patch that did not apply. Add -r (--include-function-results) to see the functions' warnings, and -o observed.yaml (--observed-resources) to render as if certain composed resources already existed (needed for ToCompositeFieldPath patches).
Step 5: make it reconcile now
Crossplane re-checks healthy objects about once a minute and retries failures with backoff. After you fix the cause you can wait - or trigger a reconcile by changing the object, for example with an annotation:
kubectl annotate database orders -n default reconcile-now="$(date +%s)" --overwrite
To stop Crossplane from touching an XR or an MR while you investigate, pause it: kubectl annotate ... crossplane.io/paused=true. Synced turns False with reason ReconcilePaused. Remove the annotation to resume.
In an interview: debug Crossplane top-down: crossplane resource trace for the tree, then the conditions and events of the first object that is not healthy, then the Provider's Installed/Healthy and the ProviderConfig. An MR with no conditions means no healthy provider; Unsynced resources means the API server refused a composed resource; a skipped patch is usually a fromFieldPath typo, and crossplane composition render proves it.