OnCallReadyStart learning

Infrastructure as code Lesson 42.12 · Platform Engineering: Backstage & Crossplane · 20 min read

Debugging composite resources on call

In plain words

When a parcel does not arrive, you do not search the whole country. You follow the tracking: was the order received, was it handed to the courier, did the courier reach the warehouse. Debugging Crossplane is the same: follow the chain from the request to the composed resources to the provider, and stop at the first place where the tracking says something went wrong.

Debugging a composite resource

A team says "my database is not ready". Crossplane problems are rarely in one object: the XR depends on a Composition, a function, composed managed resources, a provider, a ProviderConfig and credentials. The skill is to walk that chain top-down and stop at the first thing that is not healthy.

  Database/orders  ──(Composition, CompositionRevision, Function)──▶  composed resources
                                                                          │
                     Provider (Installed, Healthy) ◀── ProviderConfig ◀───┘ ──▶ AWS API

Step 1: the tree

crossplane resource trace shows the XR and everything it composed, with Synced, Ready and the reason (and the end of the message) for each:

$ crossplane resource trace database orders
NAME                                        SYNCED   READY   STATUS
Database/orders (default)                   True     False   Creating: Unready resources: db
└─ Instance/orders-2cd4e17cab08 (default)   False    -       ReconcileError: ...fig: clusterproviderconfigs.aws.m.upbound.io "default" not found

How to read it:

  • The STATUS column shows the Synced reason and message when Synced is not True, otherwise the Ready one. A healthy row just says Available.
  • The message is cut to its last 64 characters (that leading ...). Run it with -o wide for the full text and the composition resource name of each child.
  • The XR's own status rolls up its children: Unready resources: db names the composed resources (by composition resource name) that are not ready yet; Unsynced resources: db names the ones Crossplane could not apply.
  • -o yaml / -o json print every object in the tree; -o dot a graph.

Here the XR is fine; its database is not, and the reason is at the bottom of the tree: the AWS ClusterProviderConfig named default does not exist.

Step 2: conditions and events on the object that is wrong

kubectl describe instance orders-2cd4e17cab08 -n default
kubectl get events -n default --field-selector involvedObject.name=orders-2cd4e17cab08

Events give you the history (how long has it failed, did it ever work). The XR's events come from the component defined/compositeresourcedefinition.apiextensions.crossplane.io (Crossplane's definition controller runs every XR controller); a managed resource's from managed/<group>/<version>, kind=<kind>.

Step 3: the package and its config

If managed resources do not even get conditions, the provider is not running:

kubectl get providers          # INSTALLED and HEALTHY must both be True
kubectl get functions
kubectl get clusterproviderconfigs.aws.m.upbound.io

A managed resource that nobody reconciles has no conditions at all (the SYNCED and READY columns are empty, trace shows -). That always means: no healthy provider for this kind.

The failure catalogue

The messages below are the real ones; learn to recognise them.

No Composition fits.

Database/orders (default)   False    -       ReconcileError: cannot select Composition: no compatible Compositions found

No Composition has a compositeTypeRef with this XR's exact apiVersion and kind (or none carries the labels its compositionSelector asks for). Compare kubectl get compositions (columns XR-KIND and XR-APIVERSION) with the XR's apiVersion. A Composition for platform.lab/v1alpha2 does not serve v1alpha1.

The Composition the XR remembers is gone.

Database/orders (default)   False   False   ReconcileError: cannot fetch Composition: cannot get Composition: compositions.apiextensions.crossplane.io "database-typo" not found

The choice of Composition is made once and written to spec.crossplane.compositionRef. Deleting that Composition and creating another does not move existing XRs; you set the reference yourself (kubectl patch or edit the XR).

The function is not installed (or not healthy).

cannot compose resources: cannot run Composition pipeline step "patch-and-transform": cannot find an active FunctionRevision (a FunctionRevision with spec.desiredState: Active)

The function refused its input or failed. The function's own message follows returned a fatal result::

cannot compose resources: pipeline step "patch-and-transform" returned a fatal result: cannot render composed resource "db" "FromCompositeFieldPath" patch at index 0: transform at index 0 returned error: map transform could not resolve: key huge is not found in map

The resource name, the patch index and the transform index point at the exact line of the Composition.

A composed resource is invalid. Crossplane applies it; the API server validates it against the provider's CRD and refuses:

Database/orders (default)   False    False   ReconcileError: Unsynced resources: db

The XR's events carry the API server's message:

  Warning  ComposeResources  2s   defined/compositeresourcedefinition.apiextensions.crossplane.io  cannot apply composed resource "db": Instance.rds.aws.m.upbound.io "orders-f29bfca80b19" is invalid: spec.forProvider.region: Required value

A required field is missing - typically because a patch's fromFieldPath has a typo and was silently skipped, so the value never arrived.

Crossplane may not create that kind.

cannot compose resources: cannot apply composed resource "ingress": ingresses.networking.k8s.io "web-3f1a7c09be42" is forbidden: User "system:serviceaccount:crossplane-system:crossplane" cannot patch resource "ingresses" in API group "networking.k8s.io" in the namespace "team-a"

Grant it with a ClusterRole labelled rbac.crossplane.io/aggregate-to-crossplane: "true" (the compositions lesson).

The provider cannot authenticate or is refused by the cloud. These live on the managed resource, prefixed by the step: connect failed: ... (ProviderConfig, Secret, credentials), observe failed: ..., create failed: async create failed: .... Read them from the right for the cloud's error code: ExpiredToken, InvalidClientTokenId, AccessDenied, BucketAlreadyExists.

Step 4: reproduce the composition without the cluster

When the pipeline is the suspect, take the XR and the Composition out of the cluster and render them. You see exactly what the functions produce, without waiting for reconciles and without touching AWS:

kubectl get database orders -n default -o yaml > xr.yaml
kubectl get composition database-rds -o yaml > composition.yaml
kubectl get function function-patch-and-transform -o yaml > functions.yaml
crossplane composition render xr.yaml composition.yaml functions.yaml

A field that is missing from the rendered resource is a patch that did not apply. Add -r (--include-function-results) to see the functions' warnings, and -o observed.yaml (--observed-resources) to render as if certain composed resources already existed (needed for ToCompositeFieldPath patches).

Step 5: make it reconcile now

Crossplane re-checks healthy objects about once a minute and retries failures with backoff. After you fix the cause you can wait - or trigger a reconcile by changing the object, for example with an annotation:

kubectl annotate database orders -n default reconcile-now="$(date +%s)" --overwrite

To stop Crossplane from touching an XR or an MR while you investigate, pause it: kubectl annotate ... crossplane.io/paused=true. Synced turns False with reason ReconcilePaused. Remove the annotation to resume.

In an interview: debug Crossplane top-down: crossplane resource trace for the tree, then the conditions and events of the first object that is not healthy, then the Provider's Installed/Healthy and the ProviderConfig. An MR with no conditions means no healthy provider; Unsynced resources means the API server refused a composed resource; a skipped patch is usually a fromFieldPath typo, and crossplane composition render proves it.

Why it helps

This is the on-call skill of the chapter. A team says their database is not ready, and you have minutes to say whether it is their request, the composition, a function, the provider, the credentials or the cloud. Learning the tree view, the failure catalogue and the render trick turns a vague ticket into a precise answer and a fix.

On the blog

Commands in this lesson

crossplane

FAQ

What does the STATUS column of trace show?

The Synced reason and message when Synced is not True, otherwise the Ready reason and message. The message is cut to its last 64 characters, so run trace with -o wide to see the full text and the composition resource name of each child.

What does Unsynced resources mean on an XR?

That the API server refused one or more composed resources, usually because a required field is missing or invalid. The XR's events carry the API server's validation message, which often points at a patch that was silently skipped.

Why does an XR keep a deleted Composition?

The choice of Composition is made once and written to spec.crossplane.compositionRef. Deleting that Composition and creating another does not move existing XRs; they report that the Composition cannot be fetched until you set the reference yourself.

How do I stop Crossplane touching something while I look?

Add the annotation crossplane.io/paused=true to the XR or managed resource. Synced turns False with reason ReconcilePaused and nothing is reconciled until you remove the annotation. Use it while you investigate or fix something by hand, and remember to take it off again.

How do I force a reconcile after a fix?

Crossplane re-checks healthy objects about once a minute and retries failures with backoff, so waiting works. To act now, change the object, for example by adding or updating any annotation, which triggers an immediate reconcile.

In an interview Mid

A team's Crossplane XR is not Ready. Walk me through debugging it.

I start with crossplane resource trace on the XR, -o wide for full messages, and stop at the first object that is not healthy. On the XR itself: composition selection (no compatible Compositions means compositeTypeRef does not match the apiVersion), the function, or a fatal function result. On a composed resource: connect failed means config or credentials, Unsynced means the API server refused it. Then provider Installed and Healthy, and crossplane composition render to reproduce the pipeline locally.

Also asked: What does a managed resource with no conditions tell you? · How would you find a typo in a patch? · How do you pause reconciliation?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.