Lesson 30.1 · OpenShift · 19 min read

What OpenShift adds to Kubernetes

Q: Can I deploy my apps into openshift-* or default namespaces?

No. openshift-, kube- and default are managed by the platform; operators reconcile them and may revert or delete your changes, and some are highly privileged, meaning SCC admission does not even run there. Deploy workloads only into your own projects. Read-only inspection of openshift- namespaces is fine and often necessary, for example looking at router pods in openshift-ingress.

Q: What are the extra API groups OpenShift adds?

route.openshift.io for Routes, build.openshift.io for BuildConfigs and Builds, image.openshift.io for ImageStreams, security.openshift.io for SCCs, project.openshift.io for Projects, apps.openshift.io for DeploymentConfigs, config.openshift.io for cluster configuration like ClusterVersion, and operators.coreos.com for OLM. They are normal API groups: oc explain, -o yaml, jsonpath and RBAC work on them like on any Kubernetes resource.

In plain words

Imagine a school that follows the national curriculum exactly, so everything you learned elsewhere still counts. But this school has its own house rules: you must sign in at a front desk to get a classroom, everyone wears a name badge with a random number instead of their own name, visitors come in through one official gate, and the caretaker repaints the whole building, walls and foundations included, on a timetable.

OpenShift is Kubernetes with house rules. The same Pods, Deployments and YAML work, plus Projects, SecurityContextConstraints (the random badge: a UID like 1000650000), Routes behind an HAProxy router, Builds and ImageStreams, Operators via OLM, and the Cluster Version Operator. On a new cluster you orient yourself with oc get clusterversion, oc get clusteroperators and oc get nodes.

Why this chapter exists

Many banks do not run plain Kubernetes: they run OpenShift. You join the team, every kubectl command you know still works - and the first image you deploy, one that ran fine on AKS, crash-loops with Permission denied. This chapter is the list of things that are different, and why.

What you need to know already: Pods, Deployments and Services (15.14, 15.16, 16.1); namespaces, labels and annotations (15.26); the reconciliation loop - a controller keeps making reality match the YAML (15.9); the apiserver's request path, including admission (15.5); -o yaml and jsonpath (15.38); Ingress (16.19); RBAC (17.30); securityContext and Pod Security Admission (17.37, 17.39); the kubeadm upgrade (18.16); a Dockerfile's USER (10.40); Helm (25.19).

The words you need first

Distribution - a packaged, supported build of an open-source project with extra pieces around it. Ubuntu is a Linux distribution; OpenShift is a Kubernetes one.
Operator - a controller (a program running in a pod, doing the 15.9 loop) that knows how to install, configure and repair one product: a database, the router, the image registry. You describe what you want in an object; the operator makes it so and keeps it so.
CRD (CustomResourceDefinition) - a way to add a new kind of object to the apiserver, as Argo CD did with Application (26.4). OpenShift's extra objects (Route, Project, ...) are new API kinds in the same way.
API group - the part before the / in apiVersion: apps in apps/v1. OpenShift's additions live in groups ending in openshift.io.

Same Kubernetes, different defaults

OpenShift is Red Hat's Kubernetes distribution. Underneath is ordinary Kubernetes - the same apiserver, the same Pods, Deployments, Services, ConfigMaps, RBAC, the same YAML. Everything you learned in chapters 15-19 is still true. What OpenShift adds is a set of opinions (defaults you cannot easily turn off) and a set of extra APIs (new kinds of object). Each row below gets its own lesson later; read it once as a map, not as something to memorise:

area	vanilla Kubernetes	OpenShift 4
namespaces	anyone with rights creates them	Projects: namespaces requested through an API that applies a template, a UID range, an admin binding
pod security	Pod Security Admission, opt-in labels	SecurityContextConstraints: every pod runs as a random non-root UID unless someone allows more
ingress	install a controller (ingress-nginx, a Gateway)	an HAProxy router is built in; Routes are its native API (Ingress still works)
images	push to an external registry	an internal registry and ImageStreams that track tags; Builds (S2I, Dockerfile) run on the cluster
add-ons	Helm charts, kubectl apply	Operators installed through OLM from OperatorHub catalogs
upgrades	kubeadm upgrade node by node	the Cluster Version Operator upgrades the whole cluster, OS included, from a release image
nodes	any Linux you like	RHCOS, an immutable OS managed by the cluster (Machine Config Operator)
CLI	kubectl	oc: kubectl plus the OpenShift commands
auth	client certs, OIDC you wire yourself	a built-in OAuth server (htpasswd, LDAP, OIDC identity providers), `oc login`

A few names in that table, in one line each (all come back later):

HAProxy - a widely used open-source load balancer / reverse proxy (reverse proxies: 9.23). OpenShift's "router" is HAProxy.
Internal registry - an image registry (10.54) that runs inside the cluster itself.
RHCOS (Red Hat Enterprise Linux CoreOS) - the node operating system. "Immutable" means you do not apt install on it: the cluster writes its configuration and updates it as a whole.
CRI-O - the container runtime the kubelet talks to (the "CRI" of 15.7), instead of containerd.
OAuth server - a login service inside the cluster: you give it a username and password, it gives you a token (a long random string that proves who you are). The identity provider is where it checks passwords: an htpasswd file (a file of usernames and password hashes), the company LDAP directory, or an OIDC login.

That table is this chapter. The rest of this lesson is how to recognise and orient yourself in a cluster.

The family

OCP - OpenShift Container Platform, the supported product you install yourself (bare metal, vSphere, Azure, AWS...). Most banks run this on-prem.
OKD - the community distribution OCP is built from. Same APIs, no support.
ARO - Azure Red Hat OpenShift, jointly run by Microsoft and Red Hat; the control plane is managed for you. ROSA is the same idea on AWS. OpenShift Dedicated is Red Hat-managed on either cloud.
Developer Sandbox - a free, time-limited project on a shared cluster. You get a project, not a cluster: no oc adm, no SCC grants, no operators. Good for practising the developer side of this chapter.

Versions are 4.<minor>.<z> (the same major.minor.patch idea as Kubernetes versions, 18.15). Each minor pins a Kubernetes minor: OpenShift 4.21 is Kubernetes 1.34 with CRI-O. Minors ship roughly every four months; even minors (4.18, 4.20, 4.22) are EUS (Extended Update Support: supported for longer) releases that big companies standardise on.

Orienting yourself on a cluster

oc is OpenShift's command-line tool: kubectl plus extra commands (next lesson). The first three commands on an unknown OpenShift cluster:

# as kubeadmin on the lab cluster - the next missions bring it up and log you in
oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.21.25   True        False         12d     Cluster version is 4.21.25

ClusterVersion is a cluster-scoped (not inside any namespace) object, and there is exactly one, called version. It is the thing the Cluster Version Operator (the operator that owns upgrades) reconciles. The columns: VERSION is what is running, AVAILABLE True means the cluster is up, PROGRESSING True means an upgrade is in flight, SINCE is how long it has been in this state, STATUS is the human summary.

oc get clusteroperators
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.21.25   True        False         False      13d
baremetal                                  4.21.25   True        False         False      13d
console                                    4.21.25   True        False         False      13d
dns                                        4.21.25   True        False         False      13d
etcd                                       4.21.25   True        False         False      13d
image-registry                             4.21.25   True        False         False      13d
ingress                                    4.21.25   True        False         False      13d
kube-apiserver                             4.21.25   True        False         False      13d
machine-config                             4.21.25   True        False         False      13d
network                                    4.21.25   True        False         False      13d
operator-lifecycle-manager                 4.21.25   True        False         False      13d
...

Every part of the platform - DNS, the router, the registry, the console, even etcd and the kube-apiserver - is run by an operator, and each operator reports its health in a ClusterOperator object. co is its short name (like deploy for deployments), so oc get co is the platform's health page. What you look for: AVAILABLE False (broken), DEGRADED True (working, but something is wrong), PROGRESSING True for a long time (stuck rolling out). During an upgrade the VERSION column changes operator by operator.

oc get nodes
NAME       STATUS   ROLES           AGE   VERSION
cp-1       Ready    control-plane   12d   v1.34.1
worker-1   Ready    <none>          12d   v1.34.1
worker-2   Ready    <none>          12d   v1.34.1

(simulator) The lab OpenShift API is layered on the same simulated control plane as the kubeadm lab cluster, so you see the kubeadm node names. A real cluster shows three control-plane nodes (ROLES control-plane,master) and workers labelled node-role.kubernetes.io/worker, and oc get nodes -o wide reports OS-IMAGE Red Hat Enterprise Linux CoreOS 9.6... and CONTAINER-RUNTIME cri-o://1.34....

The pieces you will meet in `openshift-*` namespaces

oc get pods -n openshift-ingress
NAME                              READY   STATUS    RESTARTS   AGE
router-default-7884t2w448-5pmj5   1/1     Running   0          13d
router-default-7884t2w448-bmbmh   1/1     Running   0          13d

The platform lives in ~60 openshift-* namespaces. The ones you will actually look into:

openshift-ingress - the router pods (HAProxy) that serve Routes and Ingresses
openshift-ingress-operator - the IngressController objects that configure them
openshift-image-registry - the internal registry
openshift-authentication - the OAuth server behind oc login
openshift-console - the web console
openshift-marketplace / openshift-operator-lifecycle-manager - OperatorHub and OLM
openshift-monitoring - a preinstalled Prometheus and Alertmanager (Ch 27-28)
openshift-cluster-version - the Cluster Version Operator itself

Rule of thumb: never deploy into, or edit things in, openshift-*, default, kube-*. They are managed and operators revert your changes. Some of them - default, kube-system, kube-public, openshift, openshift-infra and any namespace labelled openshift.io/run-level 0 or 1 - are what Red Hat's docs call highly privileged: admission plugins (the checks from 15.5 that may reject or change an object on its way in) such as SCC do not even run there.

The APIs OpenShift adds

oc api-resources lists every kind of object the apiserver knows (the same command exists in kubectl). The columns: NAME (what you type after get), SHORTNAMES, APIVERSION (group/version), NAMESPACED (true = lives inside a namespace), KIND (what you write in YAML). grep -E 'openshift|coreos' keeps only the OpenShift ones:

oc api-resources | grep -E 'openshift|coreos'
deploymentconfigs                   dc           apps.openshift.io/v1                true         DeploymentConfig
buildconfigs                        bc           build.openshift.io/v1               true         BuildConfig
builds                                           build.openshift.io/v1               true         Build
clusteroperators                    co           config.openshift.io/v1              false        ClusterOperator
clusterversions                                  config.openshift.io/v1              false        ClusterVersion
imagestreams                        is           image.openshift.io/v1               true         ImageStream
imagestreamtags                     istag        image.openshift.io/v1               true         ImageStreamTag
catalogsources                      catsrc       operators.coreos.com/v1alpha1       true         CatalogSource
clusterserviceversions              csv,csvs     operators.coreos.com/v1alpha1       true         ClusterServiceVersion
installplans                        ip           operators.coreos.com/v1alpha1       true         InstallPlan
operatorgroups                      og           operators.coreos.com/v1             true         OperatorGroup
subscriptions                       sub,subs     operators.coreos.com/v1alpha1       true         Subscription
projects                                         project.openshift.io/v1             false        Project
routes                                           route.openshift.io/v1               true         Route
securitycontextconstraints          scc          security.openshift.io/v1            false        SecurityContextConstraints

Read the groups: route.openshift.io, build.openshift.io, image.openshift.io, security.openshift.io, project.openshift.io, config.openshift.io (the cluster's own configuration), and operators.coreos.com (OLM, which came from CoreOS before Red Hat bought it). They are normal API groups: oc explain (prints the documentation of a field, as kubectl explain does), oc get -o yaml, RBAC and jsonpath all work on them exactly like on Deployments.

oc explain route.spec.tls.termination
GROUP:      route.openshift.io
KIND:       Route
VERSION:    v1

FIELD: termination <string>

DESCRIPTION:
    termination indicates termination type.

    * edge - TLS termination is done by the router and http is used to
    communicate with the backend (default) * passthrough - Traffic is sent
    straight to the destination without the router providing TLS termination *
    reencrypt - TLS termination is done by the router and https is used to
    communicate with the backend

Why a pod that works on AKS fails here

This is the question in every OpenShift interview, and chapter 30's spine. On AKS (or the kubeadm lab) a container runs as whatever USER its image says - very often root. On OpenShift a SecurityContextConstraints object (SCC: a rule set for what a pod may do) called restricted-v2 admits the pod but rewrites it first: a UID picked from a block reserved for the project (like 1000650000), group 0, every capability dropped (capabilities: 11.29), no privilege escalation, the default seccomp profile (17.37). An image that assumed root - to write under /var/cache, to chown files, to bind port 80 - now fails with Permission denied, and the pod crash-loops. Nothing is wrong with the cluster; the image was never portable. Lessons 30.7 and 30.9 take that apart, and the first incident (30.12) is exactly this.

The lab (simulator)

API: https://api.ocp.lab:6443, apps domain *.apps.ocp.lab -> 10.64.0.30
users: developer / developer and alice / alice (an htpasswd identity provider), kubeadmin with the password in ~/oncall-lab/labs/4-ocp/auth/kubeadmin-password
the installer's ~/oncall-lab/labs/4-ocp/auth/kubeconfig authenticates as system:admin
cat ~/oncall-lab/labs/4-ocp/README has all of it

(simulator) SCC admission applies to namespaces created through the OpenShift API (oc new-project, or oc create ns as an admin). On a real cluster every namespace gets the SCC annotations, whichever client created it.

What you can now do

Say what OpenShift is (Kubernetes plus opinions plus extra APIs) and name the family: OCP, OKD, ARO, ROSA.
Orient on an unknown cluster: oc get clusterversion, oc get co, oc get nodes.
Explain in one sentence why a root image that worked on AKS fails here.

Why it helps

On your first day on an OpenShift team, or during an incident on a cluster you do not know, the first three commands tell you what version you are on, whether an upgrade is running, and which platform component is unhealthy. oc get co with a DEGRADED True operator is often the whole diagnosis for "the console is down" or "Routes stopped working".

Knowing the map of openshift-* namespaces tells you where to look (router pods in openshift-ingress, OAuth in openshift-authentication) and where never to deploy or edit, since operators revert your changes. And the table of differences is the answer to the most common OpenShift interview question: "what does OpenShift add to Kubernetes, and why does a pod that runs on AKS fail here?" Being able to name SCC rewriting a pod, not just validating it, shows real understanding.

FAQ

What is a ClusterOperator?

A cluster-scoped object that reports the health of one platform component, such as ingress, dns, authentication, etcd or the image registry. Each component is run by an operator that sets AVAILABLE, PROGRESSING and DEGRADED conditions and the version. oc get co is the platform's health page: AVAILABLE False means broken, DEGRADED True means working but impaired, PROGRESSING True for a long time means a stuck rollout, often during an upgrade.

Can I deploy my apps into openshift-* or default namespaces?

No. openshift-*, kube-* and default are managed by the platform; operators reconcile them and may revert or delete your changes, and some are highly privileged, meaning SCC admission does not even run there. Deploy workloads only into your own projects. Read-only inspection of openshift-* namespaces is fine and often necessary, for example looking at router pods in openshift-ingress.

What is RHCOS?

Red Hat Enterprise Linux CoreOS, the immutable, image-based operating system OpenShift 4 runs on control-plane nodes and usually on workers. It is managed by the Machine Config Operator: configuration changes are declared as MachineConfig objects and rolled out by draining and rebooting nodes. You do not patch or configure nodes by hand; upgrading the cluster upgrades the OS too.

What are the extra API groups OpenShift adds?

route.openshift.io for Routes, build.openshift.io for BuildConfigs and Builds, image.openshift.io for ImageStreams, security.openshift.io for SCCs, project.openshift.io for Projects, apps.openshift.io for DeploymentConfigs, config.openshift.io for cluster configuration like ClusterVersion, and operators.coreos.com for OLM. They are normal API groups: oc explain, -o yaml, jsonpath and RBAC work on them like on any Kubernetes resource.

What does the Developer Sandbox let me practise?

It gives you a project on a shared Red Hat cluster for free for a limited time. You can log in with oc, deploy images, see SCC behaviour, run S2I builds, create Routes and view the console. You cannot use oc adm, grant SCCs, install operators or see cluster-level resources. It is ideal for the developer side of this chapter; the admin side needs a real or lab cluster.

In an interview Mid

What is OpenShift and how does it differ from vanilla Kubernetes?

OpenShift is Red Hat's Kubernetes distribution: ordinary Kubernetes underneath - same apiserver, Pods, Deployments, Services, RBAC, YAML - plus opinions (defaults you cannot easily turn off) and extra APIs:

Projects - namespaces requested self-service through a template, each with its own UID range.
SecurityContextConstraints - every pod runs as a random non-root UID (restricted-v2) unless someone grants more. The reason root images that work on AKS fail here.
Routes and a built-in HAProxy router (Ingress still works).
An internal registry, ImageStreams and Builds (S2I) on the cluster.
Operators everywhere: the platform itself (each reports in oc get co), and add-ons through OLM / OperatorHub.
Upgrades by the Cluster Version Operator, OS included: nodes run RHCOS, an immutable OS managed by the cluster.
A built-in OAuth server with oc login, and oc = kubectl plus OpenShift commands.

The family: OCP (self-installed), OKD (community), ARO / ROSA (managed on Azure / AWS). First commands on an unknown cluster: oc get clusterversion, oc get co, oc get nodes.

Also asked: How do you check the health of an OpenShift cluster? · Explain the operator-based architecture of OpenShift 4. · Why should you never deploy into openshift-* namespaces?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.