OnCallReadyStart learning

Infrastructure as code Lesson 42.6 · Platform Engineering: Backstage & Crossplane · 17 min read

Managed resources: Synced, Ready, external names and policies

In plain words

A managed resource is like a delivery note stuck to a parcel. The note says what should be inside (forProvider), the courier writes what was actually delivered (atProvider), and two stamps say whether the last check went fine (Synced) and whether the parcel is usable (Ready). If the courier loses his keys, the parcel is still there, but nobody can open or fix it any more.

Managed resources: one external thing, as an object

A managed resource (MR) is the smallest unit Crossplane builds: one external resource, represented as one Kubernetes object, reconciled by one provider. An S3 bucket from the AWS provider:

apiVersion: s3.aws.m.upbound.io/v1beta1
kind: Bucket
metadata:
  name: oncall-lab-reports-7f3a
  namespace: default
spec:
  forProvider:
    region: eu-central-1
    tags:
      team: platform

Every managed resource has the same skeleton, whichever provider it comes from:

FieldMeaning
spec.forProviderthe desired state: what you want the external resource to look like
spec.initProvidervalues used only at creation, then left alone (for fields something else changes later, like an autoscaled node count)
spec.providerConfigRefwhich (Cluster)ProviderConfig to authenticate with; default ClusterProviderConfig/default
spec.managementPolicieswhat Crossplane may do to the external resource; default ["*"]
metadata.annotations["crossplane.io/external-name"]the resource's name or ID in the external system
status.atProviderthe observed state, as the provider last read it
status.conditionsSynced and Ready (more below)

The AWS providers are generated from the Terraform AWS provider by a tool called upjet, so forProvider field names are the Terraform argument names in camel case (allocatedStorage for allocated_storage). The provider's documentation on the Upbound marketplace lists every field per kind.

Credentials: the ProviderConfig

The AWS providers authenticate through the family's config kinds in aws.m.upbound.io. In the lab the credentials are an IAM user's access key, stored in a Secret in the standard INI format of ~/.aws/credentials:

kubectl create secret generic aws-creds -n crossplane-system \
  --from-file=credentials=./crossplane-creds.ini
apiVersion: aws.m.upbound.io/v1beta1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: credentials

In production on EKS you would use source: IRSA or PodIdentity instead, so no long-lived key exists at all. Every reconcile starts by building an AWS client from this config: the provider reads the Secret, then calls STS GetCallerIdentity to learn the account ID. A bad, deleted or expired key fails right there, before any S3 call.

The two conditions

kubectl get buckets shows the two conditions every MR has:

NAME                      SYNCED   READY   EXTERNAL-NAME             AGE
oncall-lab-reports-7f3a   True     True    oncall-lab-reports-7f3a   45s
  • Synced: did the last reconcile succeed? True/ReconcileSuccess, or False/ReconcileError with the error as the message. Synced is about Crossplane's ability to talk to the API.
  • Ready: is the external resource usable? Available, or False with Creating, Deleting or Unavailable.

They are independent, and the combination tells you what kind of problem you have:

SyncedReadyMeaning
TrueTrueall good
TrueFalse (Creating)being created; wait (an RDS instance takes minutes)
False- (empty)it never got far enough to create anything: config, credentials, validation
FalseTrueit exists and worked before, but the last reconcile failed: credentials expired, permissions removed, an update the API refused

That last row is the classic quiet failure. The bucket is still there, the app still works, but Crossplane has stopped managing it: drift is not corrected and changes you push do nothing.

The Synced=False message starts with the step that failed - the managed reconciler's own prefixes:

PrefixStep
connect failed:building the client: ProviderConfig, Secret, credentials
observe failed:reading the external resource
create failed: / update failed: / delete failed:the write the provider attempted

Here is a bucket whose credentials were a temporary STS session that expired:

$ crossplane resource trace bucket oncall-lab-reports-7f3a
NAME                                       SYNCED   READY   STATUS
Bucket/oncall-lab-reports-7f3a (default)   False    True    ReconcileError: ...iredToken: The security token included in the request is expired

trace crops a long message to its last 64 characters; -o wide shows all of it, and so do the events:

  Warning  CannotConnectToProvider  9s     managed/s3.aws.m.upbound.io/v1beta1, kind=bucket  cannot get terraform setup: cannot get account id: cannot get the caller identity: GetCallerIdentity query failed: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: bcb6c141-295a-4721-a845-f89bca7c93af, api error ExpiredToken: The security token included in the request is expired

Read such an error from the right: the AWS error code (ExpiredToken) is the root cause; everything to the left is the path it travelled through the provider (Terraform setup, account ID lookup, STS call).

Events

Each reconcile step that does something writes an event on the MR, from the component managed/<group>/<version>, kind=<kind>:

ReasonWhen
CreatedExternalResource"Successfully requested creation of external resource"
UpdatedExternalResourcean update was sent
DeletedExternalResource"Successfully requested deletion of external resource"
CannotConnectToProviderthe connect step failed
CannotObserveExternalResourcethe read failed
CannotCreateExternalResource / CannotUpdateExternalResource / CannotDeleteExternalResourcethe write failed

kubectl describe bucket <name> shows them at the bottom. A healthy MR is re-checked about once a minute (the default poll interval); a failing one is retried sooner, with backoff.

Asynchronous operations

Creating a database takes minutes, so the AWS providers create asynchronously: Crossplane asks for the resource, marks it Ready=False/Creating, and the outcome arrives later. Two extra conditions show it:

  • AsyncOperation: False/Ongoing while it runs, True/Finished after;
  • LastAsyncOperation: True/Success, or False with AsyncCreateFailure / AsyncUpdateFailure / AsyncDeleteFailure and the error.

A failed asynchronous create shows up as create failed: async create failed: failed to create the resource: [...], where the part in brackets is the underlying Terraform diagnostic, including the AWS error code.

The external name

Crossplane needs to know which external resource an MR is. That is the crossplane.io/external-name annotation. For an S3 bucket it is the bucket name; if you do not set it, it defaults to the object's metadata.name and Crossplane writes it back after creation. For other kinds it is an ID the cloud assigns.

This is also how you import a resource that already exists: you create an MR whose external name is the existing resource. The next lesson's warning applies: if the MR is allowed to delete, deleting the object deletes the real thing.

Management policies

spec.managementPolicies lists what Crossplane may do:

PolicyAllows
Observeread the external resource into status.atProvider
Createcreate it if it does not exist
Updatechange it to match forProvider
Deletedelete it when the MR is deleted
LateInitializecopy defaults the cloud chose back into forProvider
*all of the above (the default)

Two combinations you will use:

  • ["Observe"]: import without control. Crossplane reads a resource someone else owns and exposes it (atProvider) to compositions, but never changes or deletes it.
  • Everything except Delete (["Observe", "Create", "Update", "LateInitialize"]): Crossplane manages the resource, but deleting the object orphans it - the external resource stays. In v1 this was deletionPolicy: Orphan; namespaced (v2) managed resources do not have deletionPolicy at all.
apiVersion: s3.aws.m.upbound.io/v1beta1
kind: Bucket
metadata:
  name: legacy-logs
  namespace: default
  annotations:
    crossplane.io/external-name: oncall-lab-legacy-logs
spec:
  managementPolicies: ["Observe"]
  forProvider:
    region: eu-central-1
$ kubectl get buckets
NAME                      SYNCED   READY   EXTERNAL-NAME             AGE
legacy-logs               True     True    oncall-lab-legacy-logs    10s
oncall-lab-reports-7f3a   True     True    oncall-lab-reports-7f3a   10s

Deleting legacy-logs now removes only the Kubernetes object. The bucket oncall-lab-legacy-logs is still in AWS.

The finalizer

From its first reconcile on, every managed resource carries the finalizer finalizer.managedresource.crossplane.io - one it created, one it imported, even an Observe-only one. When you delete the object, Kubernetes only sets deletionTimestamp; the provider deletes the external resource, then removes the finalizer, and only then the object disappears. Two consequences you will meet on call:

  • If the provider is not running, nobody removes the finalizer: the object hangs in deletion forever (and so does a namespace containing it).
  • Removing the finalizer by hand (kubectl patch ... finalizers: null) makes the object go away but leaves the external resource running and unmanaged. Sometimes that is what you want. Know which one you are doing.

In an interview: a managed resource has forProvider (desired) and atProvider (observed) and two conditions: Synced (did the last reconcile work) and Ready (is the thing usable). Synced=False with Ready=True means it exists but Crossplane can no longer manage it - read the message from the right to find the cloud's error code.

Why it helps

On call you read managed resources more than anything else in Crossplane. Synced and Ready, the error prefixes and the events tell you in one look whether the problem is credentials, permissions, validation or the cloud itself. Management policies and external names are also how you import existing resources safely and how you make sure deleting an object never deletes real data by accident.

On the blog

Commands in this lesson

crossplane kubectl

FAQ

What is the difference between Synced and Ready?

Synced says whether the last reconcile succeeded, so it is about Crossplane's ability to talk to the API. Ready says whether the external resource is usable. They are independent: Synced False with Ready True means the thing exists but Crossplane can no longer manage it.

How do I read a long provider error?

From the right. The cloud's error code, such as ExpiredToken or AccessDenied, is the root cause; everything to the left is the path through the provider. The prefix at the start, like connect failed or create failed, tells you which reconcile step broke.

What is the external name?

The annotation crossplane.io/external-name holds the resource's name or ID in the cloud. For an S3 bucket it defaults to the object's name. Setting it to an existing resource is how you import that resource into Crossplane.

How do I import without risk?

Create the managed resource with the external name of the existing resource and managementPolicies ["Observe"]. Crossplane then reads it into status.atProvider for compositions to use, but never changes or deletes it. With the default policies, deleting the object would delete the real resource.

What happens if I delete a managed resource whose provider is gone?

It hangs in deletion forever, because its finalizer is only removed by the provider after the external resource is deleted. Reinstalling the provider finishes the job; removing the finalizer by hand makes the object disappear but leaves the external resource running and unmanaged.

In an interview Mid

A managed resource shows SYNCED False and READY True. What does that mean?

The external resource exists and was usable at the last successful observation, but the last reconcile failed - often credentials, permissions or an update the API refused. Read the Synced message from the right to find the cloud's error code, such as ExpiredToken. It is dangerous because applications keep working, so nothing alerts, while Crossplane no longer corrects drift or applies changes; alert on Synced False that lasts, and check the ClusterProviderConfig and its Secret.

Also asked: What is the difference between forProvider and atProvider? · How do you import an existing bucket into Crossplane? · What does managementPolicies Observe do?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.