OnCallReady

Lesson 18.15 · Kubernetes: Cluster Operations & Troubleshooting · 7 min read

Version skew: what may run with what

In plain words

Imagine a relay team where the captain has to learn the new rules first. Runners can be a little behind on the new rules (they can still take the baton), but no runner may play by rules newer than the captain's, or the captain won't understand them. And the team only switches one rulebook edition at a time; nobody skips from edition 33 to 35.

That's the version skew policy. The kube-apiserver is the captain and goes first. Kubelets may be up to three minors older but never newer; the controller-manager and scheduler at most one older; kubectl within one minor either way. kubectl version shows client and server; the VERSION column of kubectl get nodes is each kubelet. And kubeadm refuses to skip a minor.

Every component has a version

The problem. Kubernetes ships a new minor version (1.34 -> 1.35) about every four months and supports each for about a year. Upgrading in the wrong order, or two steps at once, leaves components that cannot talk to each other. The version skew policy (the official rules for which versions may run together) tells you the order.

What you need to know already: the control plane and the kubelet (15.5, 15.7), kubeadm and the per-minor package repositories (18.6), versions as major.minor.patch.

A cluster is not "on 1.34". Each component has its own version and they are upgraded at different moments:

$ kubectl version
Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.34.1
$ kubectl get nodes
NAME       STATUS   ROLES           AGE   VERSION
cp-1       Ready    control-plane   12d   v1.34.1
worker-1   Ready    <none>          12d   v1.34.1
worker-2   Ready    <none>          12d   v1.34.1

The skew policy

From kubernetes.io "Version Skew Policy" (current as of the 1.3x releases). Numbers are minor versions (1.34 -> 1.35 is one minor); patch versions (1.34.1 -> 1.34.4) may always be mixed.

componentrule relative to kube-apiserver
kube-apiserver (HA, several instances)newest and oldest within 1 minor
kubeletnever newer; up to 3 minors older
kube-proxynever newer; up to 3 older (and within 3 of the kubelet next to it)
kube-controller-manager, kube-schedulernever newer; at most 1 older (normally equal)
kubectlwithin 1 minor, older or newer

Examples with the apiserver at 1.35:

(The kubelet window was two minors before 1.28; it is three now. The policy page is the source of truth - check it when you plan a real upgrade.)

What the rules imply

The control plane goes first. Every component must be not newer than the apiserver, so the apiserver is always the first thing upgraded and the kubelets the last. Inside the control plane: apiserver, then controller-manager and scheduler (kubeadm upgrade apply does exactly that order).

One minor at a time. kubeadm refuses to skip:

[upgrade/version] FATAL: the --version argument is invalid due to these errors:

	- Specified version to upgrade to "v1.36.1" is too high; kubeadm can upgrade only 1 minor version at a time

1.33 -> 1.35 means 1.33 -> 1.34 -> 1.35: two full upgrades. The generous kubelet skew is what makes this bearable - you can upgrade the control plane twice and roll the nodes once, as long as they never fall more than three behind.

Nodes can lag, briefly. The window is for during an upgrade and for node pools you replace rather than upgrade in place (managed clouds swap whole node images). It is not a place to live: features and bug fixes need the kubelet too, and the next control plane upgrade shrinks the window.

kubeadm itself goes first on each node. kubeadm can only upgrade to a version it knows; the command fails with Specified version to upgrade to "v1.35.3" is higher than the kubeadm version "v1.34.1". Upgrade kubeadm first using the tool you used to install kubeadm.

The order, for this cluster (1.34 -> 1.35)

  1. cp-1: repository line to v1.35, apt-get update, unhold + install kubeadm 1.35.x + hold.
  2. cp-1: kubeadm upgrade plan, then kubeadm upgrade apply v1.35.x - control-plane static pods, etcd if needed, CoreDNS, kube-proxy, certificates renewed. Now the apiserver says 1.35, every kubelet still 1.34 (allowed).
  3. cp-1: drain it, upgrade kubelet + kubectl packages, daemon-reload, restart kubelet, uncordon. get nodes shows cp-1 v1.35.
  4. each worker, one at a time: repository line, kubeadm package, kubeadm upgrade node, drain, kubelet + kubectl packages, daemon-reload, restart kubelet, uncordon. Wait for Ready before the next one.

The next lesson walks through each command and its output.

What you can now do

Why it helps

Every upgrade plan starts from these rules, so you need them to write one: the control plane first, then nodes; one minor at a time; nodes can lag during a rollout but not live there. When a teammate proposes "let's just upgrade the nodes first" or "jump straight from 1.33 to 1.35", you'll know why kubeadm (and managed services) won't allow it.

They also explain confusing situations: kubectl get nodes showing v1.34 after the control plane is on 1.35 (correct, the kubelets haven't been upgraded), a kubectl warning about exceeding the supported skew, or a node image left behind for too long that suddenly falls outside the window after the next control-plane upgrade. This is standard admin-exam and interview material.

Commands in this lesson

kubectl

FAQ

Does the VERSION column in kubectl get nodes show the control-plane version?

No, it shows each node's kubelet version. The control plane's version is "Server Version" in kubectl version (the API server), and the static pod images in /etc/kubernetes/manifests. After kubeadm upgrade apply, the server is on the new minor while every node still shows the old one until you upgrade and restart its kubelet.

Can a kubelet be newer than the API server?

No, never. The API server must understand everything the kubelet sends and does, so kubelets may be the same minor or up to three minors older (since 1.28; it was two before), but not newer. That's why the control plane is always upgraded first, and why new nodes must not join with a newer kubelet than the cluster.

Can I upgrade from 1.33 straight to 1.35?

Not with kubeadm: it only upgrades one minor at a time and fails with "kubeadm can upgrade only 1 minor version at a time". You go 1.33 to 1.34 to 1.35: two full control-plane upgrades. The kubelet skew lets you upgrade the control plane twice and roll the nodes once, as long as they never fall more than three minors behind.

How far may kubectl be from the cluster?

One minor, older or newer. With a 1.35 API server, kubectl 1.34, 1.35 and 1.36 are supported; 1.33 isn't, and kubectl prints a warning about exceeding the supported skew. In practice, if you work with several clusters on different versions, keep a kubectl within one minor of all of them, or use per-cluster versions.

Why do the controller-manager and scheduler have a tighter rule than kubelets?

They run next to the API server and use its newest APIs heavily, so they may only be at most one minor older, and normally equal. That window exists for the short moment during an upgrade when the API server has been replaced but they haven't yet; kubeadm upgrade apply upgrades them right after the API server.

In an interview Mid

Explain the Kubernetes version skew policy and what it means for upgrades.

Each component has its own version, relative to kube-apiserver:

What follows: the control plane goes first (nothing may be newer than the apiserver), then the nodes; and one minor at a time - 1.33 to 1.35 is two full upgrades, and kubeadm refuses to skip. The generous kubelet window lets you upgrade the control plane twice and roll the nodes once, but it is for during an upgrade, not for living there. Patch versions can always be mixed.

Read the versions: kubectl version (Server = apiserver), the VERSION column of kubectl get nodes (each kubelet), kubeadm version.

Also asked: How do you find out which Kubernetes versions are running in a cluster? · Why can you not skip a minor version with kubeadm? · Why must kubeadm be upgraded before you run kubeadm upgrade?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.