Why one control plane is not enough
The problem. The lab cluster has one control-plane node, cp-1. When it reboots for a kernel patch, nothing breaks for the users - pods keep running, Services keep routing - but nothing can change: no kubectl, no scheduling, no scaling, no self-healing of a crashed Deployment pod on another node. A production cluster that cannot change for twenty minutes during an incident is an outage of its own.
What you need to know already: what kubeadm init and join write (20.6), static pods (20.3), etcd and its certificates (20.20), the control-plane components (17.5).
High availability (HA) means running the control plane on three machines, so that losing any one of them changes nothing. Two pieces of it behave very differently:
| component | how it runs in HA |
|---|---|
| kube-apiserver | active-active: all three serve requests, a load balancer spreads clients over them |
| kube-controller-manager, kube-scheduler | active-passive: all three run, one holds a leader election Lease in kube-system and does the work; another takes over within seconds when it dies |
| etcd | a Raft cluster: one leader, the others followers; a write is committed only when a majority has it |
Quorum: why three, and never two
etcd needs a quorum - more than half of its members - to commit anything. For n members the quorum is n/2 + 1 (rounded down), so:
| members | quorum | can lose |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
Two members are worse than one: either failure stops the cluster, and there are twice as many machines to fail. Four tolerate no more than three. That is why etcd clusters are 3 or 5 (7 at most - every write waits for a majority, so more members means slower writes).
Without quorum the surviving member is still running, still accepts connections, and answers nothing that needs agreement: every write and every consistent read waits and then fails with etcdserver: request timed out. The API servers in front of it are up too - and every kubectl command times out. "The port answers" is not "the cluster works".
Two topologies
Stacked etcd - each control-plane node runs its own etcd member next to its API server (as static pods, like on cp-1). Three machines in total. It is kubeadm's default: kubeadm init creates the first member and every kubeadm join --control-plane adds one. The cost is coupling: losing a node loses an API server and an etcd member.
External etcd - etcd runs on its own machines; the control-plane nodes only point at it. Six machines at least (three of each), more to operate, but a dead control-plane node no longer costs an etcd vote. kubeadm needs a config file with the etcd endpoints under etcd.external. The kubernetes.io docs recommend at least three nodes for either.
One stable address: the control-plane endpoint
Every client - kubectl, every kubelet, kube-proxy, controllers in pods - reaches the API through one address. With three API servers that address must not be any one of them, or losing that node is still an outage. So a load balancer sits in front of the API servers on port 6443, and its DNS name is the control-plane endpoint:
k8s-api.lab:6443 (HAProxy on ha-lb, TCP mode)
/ | \
ha-cp-1:6443 ha-cp-2:6443 ha-cp-3:6443 API servers (active-active)
etcd ------- etcd -------- etcd stacked etcd, 3 members, quorum 2
The load balancer forwards TCP to every healthy backend (health check: a TCP connect to 6443, or HTTPS /healthz). HAProxy with keepalived for a floating IP is the classic choice, kube-vip (a static pod that holds a virtual IP) a popular one, a cloud load balancer the usual one in a cloud.
The endpoint is set once, at the very beginning, with kubeadm init --control-plane-endpoint k8s-api.lab:6443. It goes into admin.conf, every kubelet's kubelet.conf, the kubeadm-config ConfigMap and the API server certificate's SANs. Forget it and the cluster is single-node for good: a later kubeadm join --control-plane stops with unable to add a new control plane instance to a cluster that doesn't have a stable controlPlaneEndpoint address. Use a DNS name, not an IP - you can move a name to a new load balancer, the IP is in every certificate.
Sharing the certificates: --upload-certs
A new control-plane node needs the cluster's CAs and keys - ca.crt/key, sa.key/pub, front-proxy-ca, etcd/ca - and then generates its own leaf certificates. You can copy those files by hand, or let kubeadm carry them:
kubeadm init ... --upload-certsencrypts them into the Secret kubeadm-certs in kube-system and prints the certificate key (64 hex characters, an AES-256 key) that decrypts them.kubeadm join k8s-api.lab:6443 --token ... --discovery-token-ca-cert-hash sha256:... --control-plane --certificate-key KEYdownloads and decrypts them, writes the node's own certificates, its static pods, and adds a new etcd member.
The Secret and the key expire after two hours. Joining later? On a control-plane node: sudo kubeadm init phase upload-certs --upload-certs (a new key), then sudo kubeadm token create --print-join-command (a new token) and add --control-plane --certificate-key NEWKEY. The certificate key is a secret: with it and a valid token anyone on the network can become a control-plane node.
What failure looks like
With three stacked members:
- One node down. kubectl through the endpoint just works: the load balancer stopped sending to the dead API server.
kubectl get nodesshows it NotReady after the 50 s grace period.etcdctl endpoint health --clusterreports two healthy members and one unhealthy: failed to commit proposal: context deadline exceeded.etcdctl member liststill shows threestartedmembers - membership is configuration, not health. - Two nodes down. Quorum lost. The last API server still accepts the TCP connection, so the load balancer still sends you there, and kubectl waits and fails:
Error from server: etcdserver: request timed out. On the survivor,endpoint healthis unhealthy and evenmember listtimes out - by default it is a linearizable read that needs the leader;member list --consistency=sanswers from the local member. Running pods keep running; nothing changes until a majority is back. - They come back. Start the members and etcd elects a leader again; nothing to repair. A member that is gone for good is replaced, not waited for:
etcdctl member remove ID, then a fresh node joins. Lose the majority for good and the way back is the snapshot (20.22) - which is why HA does not replace backups.
In an interview: "three control-plane nodes behind a load balancer, stacked etcd, so we survive one failure; quorum is a majority, so 3 or 5 members, never 2 or 4".
What you can now do
- Explain stacked vs external etcd, and choose a member count from the quorum table.
- Build an HA control plane with
kubeadm init --control-plane-endpoint ... --upload-certsandkubeadm join --control-plane --certificate-key, and recover an expired key. - Predict what one and two failed control-plane nodes do, and prove it with
etcdctl member listandendpoint health --cluster.