Lesson 18.16 · Kubernetes: Cluster Operations & Troubleshooting · 16 min read

The kubeadm upgrade, step by step

Q: What's the difference between kubeadm upgrade apply and upgrade node?

apply runs once, on the first control-plane node: it upgrades the cluster (static pods, etcd, CoreDNS, kube-proxy, cluster-wide config, certificates). upgrade node runs on every other node: on additional control-plane nodes it upgrades their static pods, and on workers it only refreshes the kubelet configuration from the kubelet-config ConfigMap that apply updated.

Q: What if kubeadm upgrade apply fails half way?

kubeadm backs up every manifest (and the etcd data) under /etc/kubernetes/tmp/kubeadm-backup- and tries to roll back the component that failed. Re-running kubeadm upgrade apply with the same version is safe, because it's idempotent. Read the error first: often it's a health check or an image that couldn't be pulled.

Q: What are the steps to upgrade a kubeadm cluster by one minor version?

First control-plane node: 1. Point apt at the next minor (/v1.35/deb/ in the repository line), apt-get update, unhold and install only kubeadm at the exact version, hold again. 2. sudo kubeadm upgrade plan - what will change, and that kubelets are your job. 3. sudo kubeadm upgrade apply v1.35.3 - the control-plane static pods one by one, etcd and CoreDNS, kube-proxy, certificates renewed, backups in /etc/kubernetes/tmp. 4. kubectl drain cp-1 --ignore-daemonsets, upgrade the kubelet and kubectl packages, systemctl daemon-reload, systemctl restart kubelet, kubectl uncordon cp-1. Each worker, one at a time: repository line, kubeadm package, sudo kubeadm upgrade node, drain, kubelet + kubectl packages, daemon-reload, restart kubelet, uncordon, wait for Ready. Mistakes to recognise: "version not found" (repository line not changed), a held package not upgraded, VERSION unchanged (kubelet not restarted), Ready,SchedulingDisabled (forgot to uncordon), a drain stuck on a PDB.

In plain words

Imagine updating the software on every computer in a school without stopping lessons. First you update the installer program itself on the head office computer. Then you ask it "what would you change?" and read the plan. Then you update the head office. Then, one classroom at a time, you move the pupils out, update that computer, check it works, and let the pupils back in, before moving to the next room.

That's a kubeadm upgrade. Switch the apt repository to the next minor and upgrade only kubeadm; kubeadm upgrade plan; kubeadm upgrade apply v1.35.x on the first control-plane node (static pods, etcd, CoreDNS, kube-proxy, certificates). Then per node: kubeadm upgrade node, drain, upgrade kubelet and kubectl, daemon-reload, restart the kubelet, uncordon.

1. kubeadm on the first control-plane node

The problem. The skew lesson (18.15) told you the order; this one gives you the commands, in that order, with the outputs you should expect - including the ones that mean you skipped a step.

What you need to know already: skew and order (18.15), apt pinning and holds (18.6), drain/uncordon (17.26), systemctl daemon-reload and restart (2.5), sed (7.6).

Point apt at the next minor and upgrade only kubeadm (the sed swaps /v1.34/deb/ for /v1.35/deb/ in the repository line; # is used as the separator because the text contains slashes):

# an illustration (no ▶): the upgrade mission walks this on cp-1 and the workers
learner@cp-1:~$ sudo sed -i 's#/v1.34/deb/#/v1.35/deb/#' /etc/apt/sources.list.d/kubernetes.list
learner@cp-1:~$ sudo apt-get update
learner@cp-1:~$ apt-cache madison kubeadm
 kubeadm | 1.35.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.35/deb  Packages
 kubeadm | 1.35.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.35/deb  Packages
 kubeadm | 1.35.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.35/deb  Packages

The package is held (you held it at install time), so:

# an illustration (no ▶): the upgrade mission walks this on cp-1 and the workers
learner@cp-1:~$ sudo apt-get install -y kubeadm=1.35.3-1.1
The following held packages will be changed:
  kubeadm
E: Held packages were changed and -y was used.
learner@cp-1:~$ sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.35.3-1.1 && sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
...
Setting up kubeadm (1.35.3-1.1) ...
kubeadm set on hold.
learner@cp-1:~$ kubeadm version -o short
v1.35.3

Forgetting to change the repository line gives E: Version '1.35.3-1.1' for 'kubeadm' was not found - the v1.34 repository simply has no 1.35 packages.

2. plan

# an illustration (no ▶): the upgrade mission walks this on cp-1 and the workers
learner@cp-1:~$ sudo kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.34.1
[upgrade/versions] kubeadm version: v1.35.3
[upgrade/versions] Target version: v1.35.3
[upgrade/versions] Latest version in the v1.34 series: v1.34.4

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE       CURRENT   TARGET
kubelet     cp-1       v1.34.1   v1.35.3
kubelet     worker-1   v1.34.1   v1.35.3
kubelet     worker-2   v1.34.1   v1.35.3

Upgrade to the latest stable version:

COMPONENT                 NODE   CURRENT   TARGET
kube-apiserver            cp-1   v1.34.1   v1.35.3
kube-controller-manager   cp-1   v1.34.1   v1.35.3
kube-scheduler            cp-1   v1.34.1   v1.35.3
kube-proxy                       1.34.1    v1.35.3
CoreDNS                          v1.12.1   v1.13.1
etcd                      cp-1   3.6.4-0   3.6.6-0

You can now apply the upgrade by executing the following command:

	kubeadm upgrade apply v1.35.3

It tells you two things: what apply will do (the second table) and what it will not do (the first: every kubelet is your job). etcd and CoreDNS move with the Kubernetes version - kubeadm pins which etcd each release runs.

3. apply

# an illustration (no ▶): the upgrade mission walks this on cp-1 and the workers
learner@cp-1:~$ sudo kubeadm upgrade apply v1.35.3
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
[upgrade/preflight] You have chosen to upgrade the cluster version to "v1.35.3"
[upgrade/versions] Cluster version: v1.34.1
[upgrade/versions] kubeadm version: v1.35.3
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/control-plane] Upgrading your static Pod-hosted control plane to version "v1.35.3" (timeout: 5m0s)...
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-09-22-20-00-06/etcd.yaml"
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.35.3".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.

Things worth noticing in that output:

it renews certificates as it goes ("Renewing apiserver certificate");
it backs up every manifest (and the etcd data dir) under /etc/kubernetes/tmp/kubeadm-backup-*: that is your rollback material;
it rewrites one static pod at a time and waits for the kubelet to restart it - the apiserver is briefly unavailable on a single control-plane cluster;
it does not touch any kubelet. kubectl get nodes still says v1.34.1 everywhere - correct, and allowed by the skew policy.

-y skips the prompt (for scripts). Without it, anything but y aborts: won't proceed; the user didn't answer (Y|y) in order to continue.

4. the kubelet on the control-plane node

# an illustration (no ▶): the upgrade mission walks this on cp-1 and the workers
$ kubectl drain cp-1 --ignore-daemonsets
$ ssh cp-1
learner@cp-1:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.35.3-1.1 kubectl=1.35.3-1.1 && sudo apt-mark hold kubelet kubectl
learner@cp-1:~$ sudo systemctl daemon-reload
learner@cp-1:~$ sudo systemctl restart kubelet
learner@cp-1:~$ exit
$ kubectl uncordon cp-1
$ kubectl get nodes
NAME       STATUS   ROLES           AGE   VERSION
cp-1       Ready    control-plane   12d   v1.35.3
worker-1   Ready    <none>          12d   v1.34.1
worker-2   Ready    <none>          12d   v1.34.1

Installing the package replaces the binary on disk; the running kubelet keeps the old binary in memory. The VERSION column only changes after the restart. The kubeadm docs include daemon-reload + restart kubelet explicitly; do not rely on the package to do it.

Why drain a node whose kubelet you are about to restart? Restarting the kubelet alone does not kill containers - but a new kubelet version may restart them anyway, and if anything goes wrong the node is out of rotation with nothing on it. Drain is cheap insurance.

5. each worker

# an illustration (no ▶): the upgrade mission walks this on cp-1 and the workers
learner@worker-1:~$ sudo sed -i 's#/v1.34/deb/#/v1.35/deb/#' /etc/apt/sources.list.d/kubernetes.list
learner@worker-1:~$ sudo apt-get update
learner@worker-1:~$ sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.35.3-1.1 && sudo apt-mark hold kubeadm
learner@worker-1:~$ sudo kubeadm upgrade node
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/preflight] Running pre-flight checks
[upgrade/preflight] Skipping prepull. Not a control plane node.
[upgrade/control-plane] Skipping phase. Not a control plane node.
[upgrade/kubeconfig] Skipping phase. Not a control plane node.
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config.../config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade] The upgrade for this node was successfully completed.

upgrade node on a worker only refreshes the kubelet configuration from the cluster (the kubelet-config ConfigMap that apply updated). On additional control-plane nodes it also upgrades their static pods - that is the difference between apply (first control plane, once) and node (everything else).

Then from oncall-lab, per worker:

# an illustration (no ▶): the upgrade mission walks this on cp-1 and the workers
$ kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
$ ssh worker-1 'sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.35.3-1.1 kubectl=1.35.3-1.1 && sudo apt-mark hold kubelet kubectl && sudo systemctl daemon-reload && sudo systemctl restart kubelet'
$ kubectl uncordon worker-1
$ kubectl get nodes worker-1        # Ready, v1.35.3 - only then the next worker

When it goes wrong

apply fails half way: the manifests are backed up; kubeadm tries to roll back the component that failed. Re-running kubeadm upgrade apply with the same version is safe (it is idempotent).
a worker will not drain: a PodDisruptionBudget (17.26). Fix the PDB or the replica count; never --disable-eviction in production without a reason.
you forgot to uncordon: the node shows Ready,SchedulingDisabled forever and the cluster slowly runs out of room.

What you can now do

Upgrade the first control-plane node: repository, kubeadm, upgrade plan, upgrade apply, then its kubelet.
Upgrade each worker with kubeadm upgrade node, drain, packages, restart, uncordon.
Recognise the "held package", "version not found" and "VERSION did not change" mistakes.

Why it helps

This is one of the highest-value exam tasks (a whole node upgraded in a few minutes, with exact commands) and one of the most common real maintenance jobs on self-managed clusters. The details are what trip people up: the held packages (E: Held packages were changed), the repository line that still points at the old minor (Version ... was not found), a node that still shows the old version because nobody restarted the kubelet, or a node left SchedulingDisabled because nobody uncordoned it.

Even on a managed cluster, where one upgrade command does the steps, the same sequence runs underneath, and when a node pool upgrade fails you'll recognise the phase: drain blocked by a PDB, a node not coming back Ready. Knowing what apply backs up gives you your rollback material.

Commands in this lesson

kubectl ssh

FAQ

Why does apt say "Held packages were changed and -y was used"?

Because you held kubeadm, kubelet and kubectl at install time with apt-mark hold, precisely so routine upgrades can't change them. For a deliberate upgrade: sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.35.3-1.1 && sudo apt-mark hold kubeadm. Same pattern later for kubelet and kubectl.

What's the difference between kubeadm upgrade apply and upgrade node?

apply runs once, on the first control-plane node: it upgrades the cluster (static pods, etcd, CoreDNS, kube-proxy, cluster-wide config, certificates). upgrade node runs on every other node: on additional control-plane nodes it upgrades their static pods, and on workers it only refreshes the kubelet configuration from the kubelet-config ConfigMap that apply updated.

I installed the new kubelet but get nodes still shows the old version. Why?

Installing the package replaces the binary on disk, but the running kubelet still uses the old one in memory. Run sudo systemctl daemon-reload and sudo systemctl restart kubelet; the VERSION column changes after the restart. The kubeadm docs include both steps explicitly; don't rely on the package to do it.

Why drain a node if restarting the kubelet doesn't kill containers?

A new kubelet version may restart containers anyway, and if something goes wrong the node is already out of rotation with nothing important on it. Draining moves workloads away safely, respecting PDBs. It's cheap insurance. Don't forget kubectl uncordon afterwards, or the node stays Ready,SchedulingDisabled and the cluster slowly runs out of room.

What if kubeadm upgrade apply fails half way?

kubeadm backs up every manifest (and the etcd data) under /etc/kubernetes/tmp/kubeadm-backup-* and tries to roll back the component that failed. Re-running kubeadm upgrade apply with the same version is safe, because it's idempotent. Read the error first: often it's a health check or an image that couldn't be pulled.

In an interview Mid

What are the steps to upgrade a kubeadm cluster by one minor version?

First control-plane node:

Point apt at the next minor (/v1.35/deb/ in the repository line), apt-get update, unhold and install only kubeadm at the exact version, hold again.
sudo kubeadm upgrade plan - what will change, and that kubelets are your job.
sudo kubeadm upgrade apply v1.35.3 - the control-plane static pods one by one, etcd and CoreDNS, kube-proxy, certificates renewed, backups in /etc/kubernetes/tmp.
kubectl drain cp-1 --ignore-daemonsets, upgrade the kubelet and kubectl packages, systemctl daemon-reload, systemctl restart kubelet, kubectl uncordon cp-1.

Each worker, one at a time: repository line, kubeadm package, sudo kubeadm upgrade node, drain, kubelet + kubectl packages, daemon-reload, restart kubelet, uncordon, wait for Ready.

Mistakes to recognise: "version not found" (repository line not changed), a held package not upgraded, VERSION unchanged (kubelet not restarted), Ready,SchedulingDisabled (forgot to uncordon), a drain stuck on a PDB.

Also asked: What does kubeadm upgrade apply change, and what does it leave to you? · What is the difference between kubeadm upgrade apply and kubeadm upgrade node? · How do you make cluster upgrades routine and low-risk?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.