OnCallReadyStart learning

Kubernetes Lesson 20.49 · Kubernetes: Cluster Operations & Troubleshooting · 9 min read

A highly-available control plane: three nodes, one endpoint, etcd quorum

In plain words

Imagine a club whose rulebook is kept by three secretaries. A new rule only counts when at least two of them have written it down. If one secretary is ill, the other two still agree and the club carries on. If two are ill, the last one can't decide anything alone - she keeps answering the door, but every request waits.

That's etcd's quorum: a majority of members must agree before anything is committed. The three API servers are three front doors, and the load balancer with one name, the control-plane endpoint, is the street sign that sends visitors to any door that's open. You set the endpoint at kubeadm init, carry the shared keys to the new nodes with --upload-certs, and add each door with kubeadm join --control-plane.

Why one control plane is not enough

The problem. The lab cluster has one control-plane node, cp-1. When it reboots for a kernel patch, nothing breaks for the users - pods keep running, Services keep routing - but nothing can change: no kubectl, no scheduling, no scaling, no self-healing of a crashed Deployment pod on another node. A production cluster that cannot change for twenty minutes during an incident is an outage of its own.

What you need to know already: what kubeadm init and join write (20.6), static pods (20.3), etcd and its certificates (20.20), the control-plane components (17.5).

High availability (HA) means running the control plane on three machines, so that losing any one of them changes nothing. Two pieces of it behave very differently:

componenthow it runs in HA
kube-apiserveractive-active: all three serve requests, a load balancer spreads clients over them
kube-controller-manager, kube-scheduleractive-passive: all three run, one holds a leader election Lease in kube-system and does the work; another takes over within seconds when it dies
etcda Raft cluster: one leader, the others followers; a write is committed only when a majority has it

Quorum: why three, and never two

etcd needs a quorum - more than half of its members - to commit anything. For n members the quorum is n/2 + 1 (rounded down), so:

membersquorumcan lose
110
220
321
431
532

Two members are worse than one: either failure stops the cluster, and there are twice as many machines to fail. Four tolerate no more than three. That is why etcd clusters are 3 or 5 (7 at most - every write waits for a majority, so more members means slower writes).

Without quorum the surviving member is still running, still accepts connections, and answers nothing that needs agreement: every write and every consistent read waits and then fails with etcdserver: request timed out. The API servers in front of it are up too - and every kubectl command times out. "The port answers" is not "the cluster works".

Two topologies

Stacked etcd - each control-plane node runs its own etcd member next to its API server (as static pods, like on cp-1). Three machines in total. It is kubeadm's default: kubeadm init creates the first member and every kubeadm join --control-plane adds one. The cost is coupling: losing a node loses an API server and an etcd member.

External etcd - etcd runs on its own machines; the control-plane nodes only point at it. Six machines at least (three of each), more to operate, but a dead control-plane node no longer costs an etcd vote. kubeadm needs a config file with the etcd endpoints under etcd.external. The kubernetes.io docs recommend at least three nodes for either.

One stable address: the control-plane endpoint

Every client - kubectl, every kubelet, kube-proxy, controllers in pods - reaches the API through one address. With three API servers that address must not be any one of them, or losing that node is still an outage. So a load balancer sits in front of the API servers on port 6443, and its DNS name is the control-plane endpoint:

               k8s-api.lab:6443  (HAProxy on ha-lb, TCP mode)
              /        |        \
     ha-cp-1:6443  ha-cp-2:6443  ha-cp-3:6443      API servers (active-active)
      etcd ------- etcd -------- etcd              stacked etcd, 3 members, quorum 2

The load balancer forwards TCP to every healthy backend (health check: a TCP connect to 6443, or HTTPS /healthz). HAProxy with keepalived for a floating IP is the classic choice, kube-vip (a static pod that holds a virtual IP) a popular one, a cloud load balancer the usual one in a cloud.

The endpoint is set once, at the very beginning, with kubeadm init --control-plane-endpoint k8s-api.lab:6443. It goes into admin.conf, every kubelet's kubelet.conf, the kubeadm-config ConfigMap and the API server certificate's SANs. Forget it and the cluster is single-node for good: a later kubeadm join --control-plane stops with unable to add a new control plane instance to a cluster that doesn't have a stable controlPlaneEndpoint address. Use a DNS name, not an IP - you can move a name to a new load balancer, the IP is in every certificate.

Sharing the certificates: --upload-certs

A new control-plane node needs the cluster's CAs and keys - ca.crt/key, sa.key/pub, front-proxy-ca, etcd/ca - and then generates its own leaf certificates. You can copy those files by hand, or let kubeadm carry them:

  • kubeadm init ... --upload-certs encrypts them into the Secret kubeadm-certs in kube-system and prints the certificate key (64 hex characters, an AES-256 key) that decrypts them.
  • kubeadm join k8s-api.lab:6443 --token ... --discovery-token-ca-cert-hash sha256:... --control-plane --certificate-key KEY downloads and decrypts them, writes the node's own certificates, its static pods, and adds a new etcd member.

The Secret and the key expire after two hours. Joining later? On a control-plane node: sudo kubeadm init phase upload-certs --upload-certs (a new key), then sudo kubeadm token create --print-join-command (a new token) and add --control-plane --certificate-key NEWKEY. The certificate key is a secret: with it and a valid token anyone on the network can become a control-plane node.

What failure looks like

With three stacked members:

  • One node down. kubectl through the endpoint just works: the load balancer stopped sending to the dead API server. kubectl get nodes shows it NotReady after the 50 s grace period. etcdctl endpoint health --cluster reports two healthy members and one unhealthy: failed to commit proposal: context deadline exceeded. etcdctl member list still shows three started members - membership is configuration, not health.
  • Two nodes down. Quorum lost. The last API server still accepts the TCP connection, so the load balancer still sends you there, and kubectl waits and fails: Error from server: etcdserver: request timed out. On the survivor, endpoint health is unhealthy and even member list times out - by default it is a linearizable read that needs the leader; member list --consistency=s answers from the local member. Running pods keep running; nothing changes until a majority is back.
  • They come back. Start the members and etcd elects a leader again; nothing to repair. A member that is gone for good is replaced, not waited for: etcdctl member remove ID, then a fresh node joins. Lose the majority for good and the way back is the snapshot (20.22) - which is why HA does not replace backups.

In an interview: "three control-plane nodes behind a load balancer, stacked etcd, so we survive one failure; quorum is a majority, so 3 or 5 members, never 2 or 4".

What you can now do

  • Explain stacked vs external etcd, and choose a member count from the quorum table.
  • Build an HA control plane with kubeadm init --control-plane-endpoint ... --upload-certs and kubeadm join --control-plane --certificate-key, and recover an expired key.
  • Predict what one and two failed control-plane nodes do, and prove it with etcdctl member list and endpoint health --cluster.

Why it helps

A single control-plane node turns every reboot, kernel patch or dead disk into twenty minutes where the cluster can't change: no deploys, no scaling, no rescheduling. Three control-plane nodes behind a load balancer make one failure a non-event, and knowing the quorum math tells you exactly how many failures that buys.

On the admin exam you'll explain topologies and join commands. On a platform team you'll build on-prem clusters, size etcd (3 or 5, never 2 or 4), and diagnose the confusing case where every machine is up and kubectl still times out because etcd lost its majority.

FAQ

Why does etcd need an odd number of members?

A write is committed only when more than half the members have it. Three members need two, so they survive one failure. Four need three, so they still survive only one - you added a machine that can fail and gained nothing. Two members need both, which is worse than one. That's why production etcd clusters are three or five members.

What's the difference between stacked and external etcd?

Stacked etcd runs an etcd member on each control-plane node next to its API server; it's the kubeadm default and needs three machines. External etcd runs on its own machines and the control plane points at it: at least six machines, more to operate, but losing a control-plane node no longer costs an etcd vote. External etcd needs a kubeadm config file with the endpoints.

I forgot --control-plane-endpoint at kubeadm init. Can I add it later?

Not cheaply. The endpoint is written into admin.conf, every kubelet.conf, the kubeadm-config ConfigMap and the API server certificate. A later kubeadm join --control-plane fails with "unable to add a new control plane instance to a cluster that doesn't have a stable controlPlaneEndpoint address". Set it on day one, as a DNS name, even with one node behind the load balancer.

My control-plane join fails with "Secret kubeadm-certs was not found". Why?

The certificates --upload-certs stored are deleted after two hours, together with the certificate key. On a working control-plane node run sudo kubeadm init phase upload-certs --upload-certs for a new key and sudo kubeadm token create --print-join-command for a fresh token, then join with --control-plane --certificate-key and the new key.

Two of my three control-plane nodes are down. Why does kubectl hang instead of failing fast?

The last API server is running and accepts the connection, so the load balancer still sends you there. Every request needs etcd, and one member of three is not a quorum, so it waits and fails with "etcdserver: request timed out". Running pods are unaffected. Bring a second member back and the cluster recovers by itself; etcdctl endpoint health --cluster confirms it.

In an interview Mid

How do you make a kubeadm control plane highly available, and what failures does it survive?

Three control-plane nodes behind a load balancer on port 6443, whose DNS name is the control-plane endpoint:

  • kubeadm init --control-plane-endpoint k8s-api.lab:6443 --upload-certs on the first node; the endpoint lands in every kubeconfig and the API server certificate.
  • kubeadm join ... --control-plane --certificate-key KEY on the other two: they download the shared CAs from the kubeadm-certs Secret (two hours) and add a stacked etcd member each.

API servers are active-active; controller-manager and scheduler use leader election. etcd needs a quorum, a majority: three members survive one failure, five survive two. Lose the majority and every request times out until members come back - so HA never replaces etcd backups.

Also asked: Why would you choose external etcd over stacked etcd? · What happens to running workloads when the control plane loses quorum? · How would you replace a control-plane node that is gone for good?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.