Lesson 30.23 · OpenShift · 19 min read

Operators, OperatorHub and OLM

Q: Why is my operator install stuck with no CSV?

Most often the Subscription has installPlanApproval: Manual and the InstallPlan is waiting: oc get ip shows APPROVED false, and the Subscription condition is InstallPlanPending with reason RequiresApproval. Approve the plan referenced in the Subscription's status.installPlanRef with oc patch installplan <name --type merge -p '{"spec":{"approved":true}}'. Also check the OperatorGroup and that the package and channel exist in the catalog.

In plain words

Imagine a phone's app store. You pick an app from the catalogue, choose whether it updates itself or asks you first, and the store installs it, checks it has the permissions it needs, and keeps it up to date. Some apps come with their own little helper that runs a whole service for you, like a game server that restarts itself when it crashes.

An Operator is that kind of helper for Kubernetes: a controller plus CRDs (custom resource definitions: new object kinds such as Kafka) that runs a product like Kafka. OLM is the app store: CatalogSources are the catalogues shown as OperatorHub, a Subscription says "install amq-streams from redhat-operators, channel stable, Manual approval", an InstallPlan is the plan waiting for approval, and the ClusterServiceVersion is the installed version. An OperatorGroup decides which namespaces it watches.

Why installing Kafka is not `helm install` here

The team wants Kafka. On AKS you would helm install a chart (25.19) and own every upgrade by hand. On OpenShift you install an operator from a catalog, and a small framework (OLM) decides when it may upgrade. When the install silently never happens, you need to know which of four objects is waiting.

What you need to know already: the reconciliation loop (15.9); CRDs and operators in one sentence (30.1); Deployments and ServiceAccounts (15.16, 17.33); RBAC (17.30); oc patch --type merge (15.40); jsonpath range (15.38); Helm charts and releases (25.19).

The words you need first

Operand - the thing the operator runs for you (the Kafka cluster); the operator is the manager, the operand is what it manages.
CR (custom resource) - one object of a CRD's kind, e.g. your Kafka named events.
OLM (Operator Lifecycle Manager) - the part of OpenShift that installs and upgrades operators. OperatorHub is its catalog page in the console.
Channel - an update stream of one operator (stable, stable-6.3): which versions you follow.
Approval - Automatic (OLM upgrades as soon as a new version appears) or Manual (a person approves each install and upgrade).

Software that runs software

An Operator is a controller that knows how to run one product - a Kafka cluster, a PostgreSQL database, Argo CD - plus the CRDs you use to ask for it. You create a Kafka object; the Strimzi operator creates the StatefulSets, Services, Secrets and certificates, and keeps fixing them. It is the reconcile loop from 15.9, written by the vendor. (Strimzi is the open-source Kafka operator; Red Hat sells it as "Streams for Apache Kafka", package amq-streams.)

OpenShift itself is built from operators (the ClusterOperators of lesson 30.1, owned by the Cluster Version Operator). Add-on operators - the ones you choose - are installed and upgraded by OLM, the Operator Lifecycle Manager, from catalogs shown in the console as OperatorHub.

The catalog

# as kubeadmin, in the operator mission
oc get catalogsources -n openshift-marketplace
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
certified-operators   Certified Operators   grpc   Red Hat     13d
community-operators   Community Operators   grpc   Red Hat     13d
redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     13d
redhat-operators      Red Hat Operators     grpc   Red Hat     13d
oc get packagemanifests -n openshift-marketplace
NAME                              CATALOG               AGE
amq-streams                       Red Hat Operators     13d
cluster-logging                   Red Hat Operators     13d
crunchy-postgres-operator         Certified Operators   13d
openshift-gitops-operator         Red Hat Operators     13d
openshift-pipelines-operator-rh   Red Hat Operators     13d
strimzi-kafka-operator            Community Operators   13d

A CatalogSource is an index image (an image that contains a list of operators and their versions) served over gRPC (a binary request/response protocol over HTTP/2) by a pod in openshift-marketplace. A PackageManifest is one product in a catalog, with its channels (update streams, like stable or stable-6.3) and the head version of each. (simulator) The lab catalog holds six packages; their versions are lab data - look up the real channel names on your cluster.

The jsonpath below prints each channel's name and its newest version (currentCSV); describe | grep -A10 InstallModes shows which install modes (next section) the operator supports:

oc get packagemanifest amq-streams -n openshift-marketplace -o jsonpath='{range .status.channels[*]}{.name}{" "}{.currentCSV}{"\n"}{end}'
stable amqstreams.v3.0.1
amq-streams-2.9.x amqstreams.v2.9.3
oc describe packagemanifest amq-streams -n openshift-marketplace | grep -A10 InstallModes
      InstallModes:
      - supported: true
        Type: OwnNamespace
      - supported: true
        Type: SingleNamespace
      - supported: true
        Type: MultiNamespace
      - supported: true
        Type: AllNamespaces

The four objects of an install

object	where	what it says
OperatorGroup (`og`)	the operator's namespace	which namespaces operators here watch (`targetNamespaces`); exactly one per namespace
Subscription (`sub`)	same namespace	"install package P from catalog C, follow channel X, approval Automatic/Manual"
InstallPlan (`ip`)	same namespace	OLM's plan to install a specific version: the CSV + CRDs + RBAC. Needs approval when the Subscription says Manual
ClusterServiceVersion (`csv`)	same namespace (copied to watched ones)	the installed operator version: its Deployment, CRDs it owns, permissions, phase

The console's Install button creates the first two for you. As YAML:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: kafka
  namespace: kafka
spec:
  targetNamespaces:
  - kafka
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: amq-streams
  namespace: kafka
spec:
  channel: stable
  name: amq-streams
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual

Install modes are what the OperatorGroup must match:

AllNamespaces - the operator watches every namespace. Installed into openshift-operators, which already has the global-operators OperatorGroup (no targetNamespaces = all). Most Red Hat platform operators (GitOps, Pipelines) are AllNamespaces only.
OwnNamespace - watches only the namespace it is installed in: targetNamespaces: [<its own namespace>]. Typical for "one team, one Kafka".
SingleNamespace / MultiNamespace - watch one / several other namespaces.

Following an install

oc apply -f amq-streams.yaml
operatorgroup.operators.coreos.com/kafka created
subscription.operators.coreos.com/amq-streams created
oc get sub,ip,csv -n kafka
NAME                                            PACKAGE       SOURCE             CHANNEL
subscription.operators.coreos.com/amq-streams   amq-streams   redhat-operators   stable

NAME                                             CSV                 APPROVAL   APPROVED
installplan.operators.coreos.com/install-2bzcl   amqstreams.v3.0.1   Manual     false

oc get sub,ip,csv lists three kinds at once. APPROVED false, no CSV: with installPlanApproval: Manual nothing happens until someone approves the plan. The Subscription says so too, in its conditions (the Reason / Status / Type lines):

oc describe sub amq-streams -n kafka | tail -12
    Reason:                RequiresApproval
    Status:                True
    Type:                  InstallPlanPending
  Current CSV:             amqstreams.v3.0.1
  Install Plan Ref:
    API Version:       operators.coreos.com/v1alpha1
    Kind:              InstallPlan
    Name:              install-2bzcl
    Namespace:         kafka
  State:                   UpgradePending

Approve the plan the Subscription points at (status.installPlanRef):

oc patch installplan install-2bzcl -n kafka --type merge -p '{"spec":{"approved":true}}'
installplan.operators.coreos.com/install-2bzcl patched
oc get csv -n kafka
NAME                DISPLAY                    VERSION   REPLACES   PHASE
amqstreams.v3.0.1   Streams for Apache Kafka   3.0.1                Succeeded

The CSV walks Pending -> InstallReady -> Installing -> Succeeded: requirements checked (OperatorGroup, CRDs, permissions), the operator Deployment created, the Deployment available. Then the CRDs exist and you can create the product:

oc apply -f kafka.yaml
kafka.kafka.strimzi.io/events created

Before the CSV succeeded, the same command fails - the API does not know the kind yet:

oc apply -f kafka.yaml
error: resource mapping not found for name: "events" namespace: "kafka" from "kafka.yaml": no matches for kind "Kafka" in version "kafka.strimzi.io/v1beta2"
ensure CRDs are installed first

Why Manual?

Automatic approval means OLM upgrades the operator as soon as the catalog has a new version in the channel - which in a bank means an operator changing behaviour in production on a random Tuesday. Most regulated shops run Manual and approve upgrade InstallPlans in a change window, testing them on a non-prod cluster first. The catch: an unapproved plan blocks the channel. Operators stuck at an old version, "the new CRD field does not exist", or a fresh install that never happened are almost always an InstallPlan waiting in RequiresApproval.

Only one InstallPlan matters: the one in the Subscription's status.installPlanRef. If the catalog moved on while a plan waited, OLM creates a new plan for the newer version; older unapproved plans are just left behind. Approving an old one does nothing useful.

When the CSV fails

oc get csv -n kafka
NAME                DISPLAY                    VERSION   REPLACES   PHASE
amqstreams.v3.0.1   Streams for Apache Kafka   3.0.1                Failed
oc describe csv amqstreams.v3.0.1 -n kafka | tail -5
  Reason: NoOperatorGroup
Events:
  Type     Reason           Age   From                        Message
  ----     ------           ----  ----                        -------
  Warning  NoOperatorGroup  4s    operator-lifecycle-manager  csv in namespace with no operatorgroups

The OperatorGroup reasons, the ones you will actually see:

reason	message	fix
`NoOperatorGroup`	`csv in namespace with no operatorgroups`	create one OperatorGroup
`TooManyOperatorGroups`	`csv created in namespace with multiple operatorgroups, can't pick one automatically`	delete the extra one
`UnsupportedOperatorGroup`	`OwnNamespace InstallModeType not supported, cannot configure to watch own namespace`	install where the mode is supported (often: `openshift-operators`)

OLM retries by itself once the OperatorGroup is right.

Upgrades

With a Subscription on channel stable and the operator at 3.0.0, a catalog update that adds 3.0.1 to stable produces a new InstallPlan (REPLACES amqstreams.v3.0.0). Automatic: it runs. Manual: it waits for you. Changing minor streams means changing the Subscription's channel (amq-streams-2.9.x -> stable), after reading the operator's upgrade notes. Uninstalling = delete the Subscription and the CSV (oc delete sub ... alone leaves the operator running); the CRDs and your CRs stay until you delete them.

OLM v1

OpenShift also ships OLM v1 (GA since 4.18): a ClusterExtension object (olm.operatorframework.io) that installs an operator bundle from a ClusterCatalog, with a ServiceAccount you provide for its permissions, and no OperatorGroups or InstallPlans. It is the direction of travel; the OperatorHub console and most existing clusters still use the v0 objects above, which is what you will be asked about.

What you can now do

Name the four OLM objects (OperatorGroup, Subscription, InstallPlan, CSV) and what each one says.
Install an operator: pick the channel and install mode, write the OperatorGroup and Subscription, approve a Manual InstallPlan.
Diagnose "the operator never arrived": a pending InstallPlan, NoOperatorGroup, "no matches for kind".

Why it helps

Platform teams on OpenShift install almost every add-on as an operator: GitOps, Pipelines, logging, Kafka, databases. The failures are specific and recurring. An install that "does nothing" is an InstallPlan waiting with APPROVED false because the Subscription says Manual. A CSV stuck Failed with NoOperatorGroup or TooManyOperatorGroups has a one-line fix. no matches for kind "Kafka" means the CSV has not succeeded yet.

Manual approval is what regulated shops use, so upgrades happen in change windows after testing, and you need to know that an unapproved plan blocks the channel and that only the plan in status.installPlanRef matters. Uninstalling requires deleting both Subscription and CSV. And OLM v1 with ClusterExtension is where things are heading, which a senior interviewer might ask about.

FAQ

What is the difference between an Operator and a Helm chart?

A Helm chart renders and applies manifests once per install or upgrade; after that, nothing watches them except Kubernetes itself. An Operator is a running controller with CRDs: you declare what you want, such as a Kafka cluster, and it continuously reconciles, handling day-2 tasks like scaling, upgrades, failover, certificate rotation and backups. Operators suit stateful, complex software; Helm suits packaging stateless apps.

Why is my operator install stuck with no CSV?

Most often the Subscription has installPlanApproval: Manual and the InstallPlan is waiting: oc get ip shows APPROVED false, and the Subscription condition is InstallPlanPending with reason RequiresApproval. Approve the plan referenced in the Subscription's status.installPlanRef with oc patch installplan <name> --type merge -p '{"spec":{"approved":true}}'. Also check the OperatorGroup and that the package and channel exist in the catalog.

What does an OperatorGroup do?

It defines which namespaces the operators installed in its namespace watch, through targetNamespaces; no target list means all namespaces. There must be exactly one OperatorGroup per namespace, and it must match an install mode the operator supports: OwnNamespace, SingleNamespace, MultiNamespace or AllNamespaces. AllNamespaces operators go into openshift-operators, which already has the global-operators group.

Should operator upgrades be automatic?

In production, usually not. Automatic approval upgrades the operator as soon as the catalog publishes a new version in the channel, which can change behaviour at any time. Regulated teams use Manual, test the upgrade on non-prod, and approve the InstallPlan in a change window. The catch is that a pending plan blocks further updates on the channel, so pending approvals must be tracked and not forgotten.

How do I uninstall an operator?

Delete both the Subscription and the ClusterServiceVersion: oc delete sub <name> alone stops future upgrades but leaves the installed operator running. Deleting the CSV removes the operator's Deployment. CRDs and the custom resources you created remain until you delete them, deliberately, since deleting a CRD deletes all its instances. Check the operator's documentation for cleanup of its data first.

In an interview Mid

An operator was installed but the CRD fields its documentation mentions do not exist. What is going on?

Almost always the operator is not at the version you think - an upgrade is waiting for approval in OLM (the Operator Lifecycle Manager).

The four objects in the operator's namespace:

OperatorGroup - which namespaces it watches (exactly one per namespace).
Subscription - package, catalog, channel, and approval Automatic or Manual.
InstallPlan - OLM's plan for one version (CSV + CRDs + RBAC).
ClusterServiceVersion (CSV) - the installed version and its phase.

Check: oc get sub,ip,csv -n <ns>. An InstallPlan with APPROVED false and the Subscription condition InstallPlanPending / RequiresApproval means a Manual upgrade (or install) never happened - the CRDs are still the old version's. Approve the plan in status.installPlanRef (oc patch installplan ... --type merge -p '{"spec":{"approved":true}}'), in a change window, after testing on non-prod. Older unapproved plans are just leftovers.

Other causes: the channel does not include that version yet (change channel after reading upgrade notes), or the CSV is Failed (NoOperatorGroup, TooManyOperatorGroups, UnsupportedOperatorGroup). Manual approval is the norm in banks - and the price is that a waiting plan blocks the channel.

Also asked: What is the Operator Lifecycle Manager? · Why would a bank set operator subscriptions to Manual approval? · How do you uninstall an operator installed through OLM?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.