OnCallReadyStart learning

Platform engineeringKubernetes · 7 min read

Crossplane resource not Ready: debug the XR top-down with crossplane resource trace

An XR stuck at SYNCED False or READY False: trace the tree, read the first unhealthy object, then fix the Composition, the patch or the credentials.

"We requested a database twenty minutes ago and have nothing." In Crossplane that means a composite resource (XR) that is not Ready. The cause is rarely in the XR itself. An XR depends on a Composition, a function pipeline, the managed resources (MRs) it composes, a provider, a ProviderConfig and some credentials. The way to find it is to walk that chain from the top and stop at the first object that is not healthy.

The examples are Crossplane v2.4 (namespaced XRs, apiextensions.crossplane.io/v2 XRDs), the crossplane CLI v2.5 and function-patch-and-transform.

terminal
$ kubectl get database orders -n team-a
NAME     SYNCED   READY   COMPOSITION   AGE
orders   False                          19m

Step 1: the tree

crossplane resource trace shows the XR and everything it composed, with each object's Synced, Ready and the reason for the first problem:

terminal
$ crossplane resource trace database orders -n team-a
NAME                       SYNCED   READY   STATUS
Database/orders (team-a)   False    -       ReconcileError: cannot select Composition: no compatible Compositions found

(Older guides use crossplane beta trace. The current CLI calls it crossplane resource trace.) How to read it:

  • SYNCED False means Crossplane could not do its job on this object (select a Composition, run the pipeline, apply a resource, reach the cloud). READY False means the work is done but the result is not there yet.
  • STATUS is the reason and message of the condition that is not healthy, cut to the last 64 characters by default (a leading ...). -o wide prints the whole message.
  • An XR rolls its children up: Unsynced resources: db and Unready resources: db name the composed resources (by their name in the Composition) that are the problem.

Here there are no children at all. The XR never got as far as composing anything.

Problem 1: no compatible Compositions found

terminal
$ kubectl get compositions
NAME           XR-KIND    XR-APIVERSION           AGE
database-rds   Database   platform.lab/v1alpha2   20m
$ kubectl get database orders -n team-a -o jsonpath='{.apiVersion}{"\n"}'
platform.lab/v1alpha1

A Composition is only a candidate for an XR if its compositeTypeRef matches the XR's apiVersion and kind exactly. This one was written for v1alpha2, and the team's request is v1alpha1. Fixing the file and applying it does not work, though:

terminal
$ sed -i 's#platform.lab/v1alpha2#platform.lab/v1alpha1#' ~/oncall-lab/labs/platform/database-comp.yaml
$ kubectl apply -f ~/oncall-lab/labs/platform/database-comp.yaml
The Composition "database-rds" is invalid: spec.compositeTypeRef: Invalid value: "object": Value is immutable
$ kubectl delete composition database-rds && kubectl apply -f ~/oncall-lab/labs/platform/database-comp.yaml
composition.apiextensions.crossplane.io "database-rds" deleted
composition.apiextensions.crossplane.io/database-rds created

compositeTypeRef is immutable (a CEL rule on the Composition CRD: self == oldSelf), so you delete the Composition and create it again.

The sibling error is cannot fetch Composition: ... not found. The XR selected a Composition once, recorded it in spec.crossplane.compositionRef, and that Composition has since been deleted or renamed. Creating a new one does not move existing XRs. You point the reference at the new one yourself.

Problem 2: Unsynced resources

terminal
$ crossplane resource trace database orders -n team-a
NAME                       SYNCED   READY   STATUS
Database/orders (team-a)   False    False   ReconcileError: Unsynced resources: db

Progress: a Composition was selected and the pipeline ran, but the composed resource db could not be applied. The reason is in the XR's events:

terminal
$ kubectl describe database orders -n team-a | grep -A1 "cannot apply"
  Warning  ComposeResources   3s (x2 over 8s)      defined/compositeresourcedefinition.apiextensions.crossplane.io  cannot apply composed resource "db": Instance.rds.aws.m.upbound.io "orders-d71ae26354a3" is invalid: spec.forProvider.region: Required value
  Normal   ComposeResources   3s (x2 over 8s)      defined/compositeresourcedefinition.apiextensions.crossplane.io  Composed resource "db" is not yet valid

The API server refused the RDS Instance: region is a required field and it is missing. But the Composition has a patch for it:

yaml
        - fromFieldPath: spec.regoin
          toFieldPath: spec.forProvider.region

spec.regoin is a typo. In function-patch-and-transform, a fromFieldPath that does not exist on the XR is not an error. The default policy is Optional, so the patch is skipped without a word, and the error shows up one level further down, as an invalid managed resource.

Prove it without the cluster: crossplane composition render

crossplane composition render runs the Composition's function pipeline locally (the functions run in Docker) against an XR file, and prints what Crossplane would compose:

terminal
$ crossplane composition render ~/oncall-lab/labs/platform/orders.yaml ~/oncall-lab/labs/platform/database-comp.yaml ~/oncall-lab/labs/platform/functions.yaml
...
apiVersion: rds.aws.m.upbound.io/v1beta1
kind: Instance
metadata:
  annotations:
    crossplane.io/composition-resource-name: db
  generateName: orders-
...
spec:
  forProvider:
    allocatedStorage: 20
    autoGeneratePassword: true
    engine: postgres
    engineVersion: "17"
    instanceClass: db.t4g.micro
    passwordSecretRef:
      key: password
      name: orders-db-password
    skipFinalSnapshot: true
    username: app

No region. Fix the path and render again before you touch the cluster:

terminal
$ sed -i 's#spec.regoin#spec.region#' ~/oncall-lab/labs/platform/database-comp.yaml && crossplane composition render ~/oncall-lab/labs/platform/orders.yaml ~/oncall-lab/labs/platform/database-comp.yaml ~/oncall-lab/labs/platform/functions.yaml | grep region
    region: eu-central-1
$ kubectl apply -f ~/oncall-lab/labs/platform/database-comp.yaml
composition.apiextensions.crossplane.io/database-rds configured
$ crossplane resource trace database orders -n team-a
NAME                                       SYNCED   READY   STATUS
Database/orders (team-a)                   True     False   Creating: Unready resources: db
└─ Instance/orders-d71ae26354a3 (team-a)   True     False   Creating

Synced everywhere, and Ready False with Creating is now the honest state: RDS takes minutes. Later:

terminal
$ crossplane resource trace database orders -n team-a
NAME                                       SYNCED   READY   STATUS
Database/orders (team-a)                   True     True    Available
└─ Instance/orders-d71ae26354a3 (team-a)   True     True    Available

If a field must always arrive, say so in the Composition: policy: {fromFieldPath: Required} on the patch turns a silent skip into an error on the XR.

Problem 3: the provider cannot reach the cloud

A different morning: "our StorageBucket has said Creating for forty minutes, is S3 down?"

terminal
$ crossplane resource trace storagebucket invoices -n team-c
NAME                                       SYNCED   READY   STATUS
StorageBucket/invoices (team-c)            True     False   Creating: Unready resources: bucket
└─ Bucket/invoices-15370220dc8b (team-c)   False    -       ReconcileError: ...iredToken: The security token included in the request is expired

The XR is fine, and the managed resource below it is not Synced. With -o wide the full message reads connect failed: cannot get terraform setup: cannot get account id: cannot get the caller identity: GetCallerIdentity query failed: ... api error ExpiredToken: The security token included in the request is expired. Managed resource errors start with the step that failed: connect failed (ProviderConfig, Secret, credentials), observe failed, create failed, update failed, delete failed. Read them from the right for the cloud's own error code (the same codes the AWS CLI prints: AccessDenied, for example). Then look at the other teams' resources:

terminal
$ kubectl get buckets -A
NAMESPACE   NAME                    SYNCED   READY   EXTERNAL-NAME          AGE
team-a      reports-f0d5f2fdfff0    False    True    reports-f0d5f2fdfff0   120m
team-b      uploads-ea9584646c97    False    True    uploads-ea9584646c97   120m
team-c      invoices-15370220dc8b   False                                   40m

Everyone is affected. The old buckets still say READY True because they exist, but SYNCED False: nobody is managing them any more. App monitoring sees nothing wrong, which is why you should alert on Synced=False that lasts. The credentials explain it:

terminal
$ kubectl get secret aws-creds -n crossplane-system -o jsonpath='{.data.credentials}' | base64 -d | cut -c1-24
[default]
aws_access_key_id = ASIA
aws_secret_access_key = 
aws_session_token = IQoJ

(cut keeps the secret values off your screen.) An access key ID starting with ASIA plus a session token is a temporary STS session, and someone pasted it into the provider's Secret. Replace it with credentials that do not expire, or better, stop using keys at all: on EKS, the AWS provider's IRSA or Pod Identity credential sources give it short-lived credentials that renew themselves.

The failure catalogue

What trace / the conditions sayWhere to look
cannot select Composition: no compatible Compositions foundkubectl get compositions: XR-KIND and XR-APIVERSION must match the XR exactly
cannot fetch Composition: ... not foundspec.crossplane.compositionRef points at a Composition that is gone
cannot compose resources: cannot run Composition pipeline step ...kubectl get functions: the function is missing or unhealthy
... returned a fatal result: ...the function's own message: patch index, transform index
Unsynced resources: xthe XR's events: the API server's reason for refusing resource x
Unready resources: xthe child x itself: often just "Creating", give it time
child shows - in SYNCED and READYno healthy provider for that kind: kubectl get providers (INSTALLED and HEALTHY)
connect failed: ...ProviderConfig, its Secret, the credentials
create failed / observe failed: ...the cloud's error code, at the end of the message

Step by step, every time

  1. crossplane resource trace <kind> <name> -n <ns>. Add -o wide for full messages.
  2. Take the first row that is not True True: kubectl describe it and read its events.
  3. A child that has no conditions at all means the provider is not running.
  4. When the pipeline is the suspect, use crossplane composition render locally.
  5. After the fix, run trace again. Crossplane retries failures with backoff, so a fixed cause clears on its own within a minute or two.

A platform usually puts a form in front of these XRs. When the request comes from a Backstage template, check that its entity reached the catalog too: catalog-info.yaml validation errors.

Practise it

In the Platform Engineering: Backstage & Crossplane chapter, the lesson Debugging composite resources on call walks the whole catalogue. The lab The database that never comes is problems 1 and 2 above (fix both, rendering before you apply), and the incident new buckets never become Ready is problem 3, with all three teams waiting.

OnCallReady is free, with no ads and no tracking. RSS · All posts