OnCallReadyStart learning

Infrastructure as code Lesson 42.4 · Platform Engineering: Backstage & Crossplane · 19 min read

Providers, functions and their credentials

In plain words

Crossplane is like a phone with no apps. A provider is an app that teaches it to talk to one service, say AWS S3, and a function is a small tool it calls while building something. You install them from a store by name and version, and like any app they need your login before they can do anything for you: that login is the ProviderConfig.

Packages: providers and functions

Crossplane core knows nothing about AWS or Kubernetes objects. Everything it can build arrives as a package: an OCI image (pulled from a registry such as xpkg.crossplane.io) that contains CRDs and, for most packages, a controller to run. Three package kinds exist:

KindAddsRuns
Providermanaged resource kinds + ProviderConfig kindsa controller that calls one external API
Functionnothing you kubectl geta gRPC server Crossplane calls during composition
ConfigurationXRDs and Compositions (a packaged platform)nothing; it depends on providers and functions

You install one with a small object in pkg.crossplane.io/v1:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-kubernetes
spec:
  package: xpkg.crossplane.io/crossplane-contrib/provider-kubernetes:v1.4.1

Pin the version. A tag like v1.4.1 (or a digest) is what you want in production; the package manager resolves it to a digest when it installs.

What happens when you apply it

The package manager (the packages/provider.pkg.crossplane.io controller in the core pod):

  1. pulls the package, resolves the tag to a digest;
  2. creates a ProviderRevision named <provider>-<first 12 hex of the digest>;
  3. installs the package's CRDs (in v2, as ManagedResourceDefinitions that an activation policy turns into real CRDs);
  4. starts the provider's controller as a Deployment of the same name in crossplane-system, with a ServiceAccount;
  5. sets two conditions on the Provider: Installed and Healthy.
$ kubectl get providers
NAME                  INSTALLED   HEALTHY   PACKAGE                                                            AGE
provider-kubernetes   True        True      xpkg.crossplane.io/crossplane-contrib/provider-kubernetes:v1.4.1   25s
$ kubectl get providerrevisions
NAME                                                  HEALTHY   RUNTIME-HEALTHY   RUNTIME-ACTIVE   IMAGE                                                              STATE    AGE
crossplane-contrib-provider-family-aws-07f560cef4ab   True      True              True             xpkg.crossplane.io/crossplane-contrib/provider-family-aws:v2.9.0   Active   28s
provider-aws-rds-3c9c61ffdbdb                         True      True              True             xpkg.crossplane.io/crossplane-contrib/provider-aws-rds:v2.9.0      Active   29s

Upgrading means changing spec.package to a new tag: a new revision becomes Active, the old one goes Inactive and its Deployment is removed. One inactive revision is kept (revisionHistoryLimit: 1), so a rollback is fast.

Read the conditions when something is wrong, never just the pod:

Condition / reasonMeans
Installed=False, UnpackingPackagethe package could not be pulled or resolved: typo, wrong tag, registry auth
Installed=True, ActivePackageRevisiona revision is installed and active
Healthy=False, UnhealthyPackageRevisionthe revision's runtime is not up yet (or crashing); the message says why
Healthy=True, HealthyPackageRevisionthe controller runs

A freshly installed provider shows Healthy=False for a few seconds with Package runtime health is "False" with message: post establish runtime hook failed for package: provider package deployment has no condition of type "Available" yet. That only becomes a problem if it stays.

kubectl get pkg lists Providers, Functions and Configurations together (they share the pkg category).

Families and dependencies

The AWS provider is split into service-scoped providers: provider-aws-s3, provider-aws-rds, provider-aws-ec2... one per AWS service, because one provider with every AWS kind would install close to a thousand CRDs. They share a parent, provider-family-aws, which owns the ProviderConfig kinds. Packages declare dependencies, and the package manager installs them for you. Install provider-aws-s3 and a second provider appears:

$ kubectl get providers
NAME                                     INSTALLED   HEALTHY   PACKAGE                                                            AGE
crossplane-contrib-provider-family-aws   True        True      xpkg.crossplane.io/crossplane-contrib/provider-family-aws:v2.9.0   29s
provider-aws-s3                          True        True      xpkg.crossplane.io/crossplane-contrib/provider-aws-s3:v2.9.0       30s

The dependency is named after its package path (<org>-<repository>). Do not delete it: the S3 provider cannot authenticate without the family's ProviderConfig kinds.

Activation: only the kinds you use

A provider package now ships ManagedResourceDefinitions (MRDs), one per kind. A ManagedResourceActivationPolicy (MRAP) says which MRDs become real CRDs. The Helm chart creates a default MRAP from the value provider.defaultActivations, which is ["*"]: everything is activated, as in v1. Large installations narrow it so the API server is not burdened with hundreds of unused CRDs.

$ kubectl get mrd
NAME                                                   STATE    ESTABLISHED   AGE
objects.kubernetes.crossplane.io                       Active   True          24s
objects.kubernetes.m.crossplane.io                     Active   True          24s
observedobjectcollections.kubernetes.m.crossplane.io   Active   True          24s

(The kinds without .m. in the group are the old cluster-scoped managed resources, kept for v1 users. (simulator) The lab's AWS providers ship only the few kinds this chapter uses; the real provider-aws-s3 has a few dozen per scope.)

DeploymentRuntimeConfig: shaping the provider's pod

The package manager creates the provider's Deployment for you. To change it (resources, node selectors, extra args, or a fixed ServiceAccount name) you attach a DeploymentRuntimeConfig:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-kubernetes
spec:
  serviceAccountTemplate:
    metadata:
      name: provider-kubernetes
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-kubernetes
spec:
  package: xpkg.crossplane.io/crossplane-contrib/provider-kubernetes:v1.4.1
  runtimeConfigRef:
    apiVersion: pkg.crossplane.io/v1beta1
    kind: DeploymentRuntimeConfig
    name: provider-kubernetes

Why name the ServiceAccount? Because by default it is named after the revision (provider-kubernetes-c258e2f2fc8d) and changes on every upgrade - any RoleBinding you wrote for the old name silently stops applying.

provider-kubernetes and its credentials

provider-kubernetes manages Kubernetes objects as managed resources: an Object wraps any manifest. It is the bridge when you want a composition to create objects in another cluster, or to get Crossplane's status and readiness handling for an object. Every provider needs to know how to authenticate: that is a ProviderConfig (namespaced) or ClusterProviderConfig (cluster-wide).

apiVersion: kubernetes.m.crossplane.io/v1alpha1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: InjectedIdentity

InjectedIdentity means "use the provider pod's own ServiceAccount" - talk to the cluster the provider runs in, with the RBAC that ServiceAccount has. Other sources are Secret (a kubeconfig for another cluster) and cloud workload identities.

A namespaced managed resource refers to its config in spec.providerConfigRef. If you leave it out, the CRD fills in {kind: ClusterProviderConfig, name: default}. If that object does not exist, nothing is created and the MR says so:

  Warning  CannotConnectToProvider  4s    managed/object.kubernetes.m.crossplane.io  cannot get provider config: clusterproviderconfigs.kubernetes.m.crossplane.io "default" not found

The RBAC trap

With InjectedIdentity the provider acts as its own ServiceAccount, which starts with no permissions on your objects. The first Object you create fails at the very first step, reading whether the object already exists:

$ crossplane resource trace object team-a-quota -o wide
NAME                            RESOURCE   SYNCED   READY   STATUS
Object/team-a-quota (default)              False    -       ReconcileError: observe failed: cannot get object: resourcequotas "team-a-quota" is forbidden: User "system:serviceaccount:crossplane-system:provider-kubernetes" cannot get resource "resourcequotas" in API group "" in the namespace "default"

That is the API server's own RBAC error, word for word, wrapped by Crossplane (observe failed: is the managed reconciler's prefix for the Observe step). The provider-kubernetes examples grant cluster-admin to that ServiceAccount, which is fine in a lab; in production grant only the kinds and namespaces your compositions need:

kubectl create clusterrolebinding provider-kubernetes-admin \
  --clusterrole=cluster-admin --serviceaccount=crossplane-system:provider-kubernetes

The provider retries on its own (errors back off; a healthy resource is re-checked every minute by default), so after the binding exists the Object becomes SYNCED True READY True without you touching it.

Functions

A Function is installed the same way and gets a FunctionRevision and a Deployment. You never create objects of a function's kinds: Crossplane calls the function over gRPC while it composes. The one used throughout this chapter is function-patch-and-transform:

apiVersion: pkg.crossplane.io/v1
kind: Function
metadata:
  name: function-patch-and-transform
spec:
  package: xpkg.crossplane.io/crossplane-contrib/function-patch-and-transform:v0.11.1
$ kubectl get functions
NAME                           INSTALLED   HEALTHY   PACKAGE                                                                      AGE
function-patch-and-transform   True        True      xpkg.crossplane.io/crossplane-contrib/function-patch-and-transform:v0.11.1   20s

(simulator) In the lab the provider and function pods are placeholders; their controllers run inside the simulator. Conditions, events, names and errors follow the real controllers' source.

In an interview: a Provider package adds managed resource kinds and runs the controller that reconciles them against one external API; a ProviderConfig says how it authenticates; with InjectedIdentity it acts as its own ServiceAccount, so it needs RBAC. Check a package with its Installed and Healthy conditions, not by looking at pods.

Why it helps

Most "nothing happens" tickets about Crossplane come down to a package that is not healthy, a ProviderConfig that does not exist, or a provider that lacks permissions. Knowing how packages become revisions, deployments and CRDs, and how they authenticate, lets you check the right conditions in seconds instead of reading pod logs, and avoid the ServiceAccount renaming trap on upgrades.

Commands in this lesson

kubectl crossplane

FAQ

What is a ProviderRevision?

An immutable record of one installed version of a provider, named after the provider and the first twelve hex characters of the package digest. Upgrading creates a new active revision and makes the old one inactive; one inactive revision is kept so a rollback is quick.

Why did a second provider appear when I installed provider-aws-s3?

The AWS providers are split per service and share a parent, provider-family-aws, which owns the ProviderConfig kinds. Packages declare dependencies and the package manager installs them, naming the dependency after its package path. Do not delete it: the S3 provider needs it to authenticate.

How do I check that a provider works?

Read its two conditions: Installed must be True with reason ActivePackageRevision and Healthy must be True with HealthyPackageRevision. If Healthy stays False, its message says why, for example that the provider's deployment has no Available condition yet. kubectl get providers shows both columns.

Why use a DeploymentRuntimeConfig?

It shapes the provider's Deployment: resources, node selectors, extra arguments and above all a fixed ServiceAccount name. By default the ServiceAccount is named after the revision and changes on every upgrade, so RoleBindings written for the old name silently stop applying.

Why does provider-kubernetes need RBAC?

With credentials source InjectedIdentity it talks to the cluster as its own ServiceAccount, which starts with no permissions on your objects. The API server refuses its first get, and the managed resource shows the RBAC error. Grant only the kinds and namespaces your compositions need.

In an interview Mid

A Crossplane managed resource never gets any status. How do you debug it?

No conditions at all means no controller is reconciling it, so check the provider first: kubectl get providers must show Installed and Healthy True, and its ProviderRevision and pod must be up. Then check the ProviderConfig the resource references - by default ClusterProviderConfig default - because a missing one gives a CannotConnectToProvider event. With InjectedIdentity, also check the provider ServiceAccount's RBAC, ideally fixed with a DeploymentRuntimeConfig.

Also asked: What is the difference between a Provider and a ProviderRevision? · What does a ProviderConfig do? · Why are AWS providers split into a family?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.