Who decides: the controllers
Someone has to decide which broker leads each partition, notice that a broker died, and remember which topics exist. Up to Kafka 3.x that was ZooKeeper plus one broker acting as "the controller". Kafka 4.0 removed ZooKeeper. The cluster now keeps its metadata itself, in a replicated log called __cluster_metadata, managed by a small group of controllers that agree on it with the Raft consensus protocol. That mode is KRaft (Kafka Raft).
Every server has process.roles:
controller- takes part in the metadata quorum (this lab:kraft-1..3).broker- stores partitions and serves clients (this lab:kafka-1..3).broker,controller- both in one JVM ("combined"). Fine on a laptop; the Kafka documentation says combined mode is not recommended in critical environments, because you cannot restart or scale the controllers separately from the brokers.
$ ssh kraft-1 "grep -E '^(process.roles|node.id|controller.quorum|listeners|log.dirs)' /etc/kafka/server.properties"
process.roles=controller
node.id=101
controller.quorum.bootstrap.servers=kraft-1:9093,kraft-2:9093,kraft-3:9093
listeners=CONTROLLER://:9093
log.dirs=/var/lib/kafka/meta
$ ssh kafka-1 "grep -E '^(process.roles|node.id|controller.quorum|listeners|advertised|log.dirs)' /etc/kafka/server.properties"
process.roles=broker
node.id=1
controller.quorum.bootstrap.servers=kraft-1:9093,kraft-2:9093,kraft-3:9093
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafka-1:9092
log.dirs=/var/lib/kafka/data
node.id is unique across the whole cluster - controllers and brokers share the number space, which is why this lab gives the controllers 101-103. controller.quorum.bootstrap.servers is how every node finds the quorum: like a client's bootstrap.servers, it only has to reach one of them.
The quorum: a majority or nothing
The controllers elect one leader, the active controller. It writes every metadata change (a new topic, a broker fenced, a new leader for a partition) to __cluster_metadata; the other controllers are hot standbys that copy the log. A change counts once a majority of the voters have it. So:
| Controllers | Can lose | Why |
|---|---|---|
| 1 | 0 | no redundancy at all |
| 3 | 1 | 2 of 3 is a majority |
| 4 | 1 | 3 of 4 is needed - the fourth buys nothing |
| 5 | 2 | 3 of 5 is a majority |
That is why the documentation says "3 or 5". With the majority gone, nothing that changes metadata works: no new topics, no leader elections, no ISR changes. Brokers keep serving the partitions they already lead with the metadata they already have, which is why a lost quorum can look fine for a few minutes and then fall apart as soon as a broker has to fail over. The tools tell you in their own way: kafka-topics.sh --describe hangs for a minute and fails (it asks the controller which partitions are being reassigned), and so does kafka-metadata-quorum.sh describe. Every metadata question timing out while produce and fetch still work is the symptom; systemctl status kafka on the controllers is the next step.
The brokers are observers: they fetch the metadata log too, but do not vote.
kafka-metadata-quorum.sh
The tool to look at the quorum. describe --status is the summary:
$ kafka-metadata-quorum.sh --bootstrap-server kafka-1:9092 describe --status
ClusterId: Lp72heEcEHtV4Jrw77XtwQ
LeaderId: 101
LeaderEpoch: 4
HighWatermark: 4119
MaxFollowerLag: 0
MaxFollowerLagTimeMs: 0
CurrentVoters: [{"id": 101, "directoryId": "tSL9JiqDOEl9K6a4nENjVw", "endpoints": ["CONTROLLER://kraft-1:9093"]}, {"id": 102, "directoryId": "pkT2S34pSUTAqJagseUPNw", "endpoints": ["CONTROLLER://kraft-2:9093"]}, {"id": 103, "directoryId": "HGvV8dDzTwy1gz1oexGMtA", "endpoints": ["CONTROLLER://kraft-3:9093"]}]
CurrentObservers: [{"id": 1, "directoryId": "H-alcE5QAoVEmIhKA_x9zw"}, {"id": 2, "directoryId": "6ZLwf2YcgzAaMV49XG0PdA"}, {"id": 3, "directoryId": "wyW69merdO3rZOjUdCxFTQ"}]
- LeaderId / LeaderEpoch - the active controller and the election "term". The epoch goes up by one at every election: if you see it jump, controllers have been failing over.
- HighWatermark - the offset in
__cluster_metadatathat a majority has. It keeps growing even on a quiet cluster: the active controller writes a no-op record every 500 ms (metadata.max.idle.interval.ms) so the followers can tell it is alive. - MaxFollowerLag / MaxFollowerLagTimeMs - how far the slowest voter is behind. 0 or close to it is healthy; growing means a controller is down or cannot keep up.
- CurrentVoters - the controllers, each with its
directoryId(from itsmeta.properties) and endpoint; CurrentObservers - the brokers.
describe --replication shows the same per node:
$ kafka-metadata-quorum.sh --bootstrap-server kafka-1:9092 describe --replication
NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status
101 tSL9JiqDOEl9K6a4nENjVw 4121 0 1790107211300 1790107211300 Leader
102 pkT2S34pSUTAqJagseUPNw 4121 0 1790107211126 1790107211126 Follower
103 HGvV8dDzTwy1gz1oexGMtA 4121 0 1790107211089 1790107211089 Follower
1 H-alcE5QAoVEmIhKA_x9zw 4121 0 1790107211247 1790107211247 Observer
2 6ZLwf2YcgzAaMV49XG0PdA 4121 0 1790107211194 1790107211194 Observer
3 wyW69merdO3rZOjUdCxFTQ 4121 0 1790107211141 1790107211141 Observer
Lag is in records of the metadata log; the timestamps are milliseconds since 1970 (add --human-readable to get "120 ms ago"). A healthy voter has lag 0 and a last fetch a moment ago. A dead follower keeps its old LastFetchTimestamp while its lag grows; a voter that was down when the current leader was elected has never fetched from it and shows -1 everywhere (and MaxFollowerLagTimeMs in --status is then -1 too). You can ask the controllers directly with --bootstrap-controller kraft-1:9093 instead of going through a broker; that is the way in when every broker is down.
Formatting: why a new node does not just start
A KRaft node refuses to start on an empty data directory. Before the first start, its storage has to be formatted with kafka-storage.sh, which writes meta.properties: the node id, the cluster id and the directory's own id. Kafka did this automatically in the ZooKeeper days; the documentation explains why it no longer does - auto-formatting can hide a mistake (a disk that was not mounted, a node pointed at the wrong cluster), and for controllers an empty log could let a leader be elected without committed data.
$ ssh kafka-1 cat /var/lib/kafka/data/meta.properties
#
#Tue Sep 22 20:00:03 UTC 2026
cluster.id=Lp72heEcEHtV4Jrw77XtwQ
directory.id=H-alcE5QAoVEmIhKA_x9zw
node.id=1
version=1
$ ssh kafka-1 /opt/kafka/bin/kafka-storage.sh info -c /etc/kafka/server.properties
Found log directory:
/var/lib/kafka/data
Found metadata: {cluster.id=Lp72heEcEHtV4Jrw77XtwQ, directory.id=H-alcE5QAoVEmIhKA_x9zw, node.id=1, version=1}
The cluster id is a random 16-byte UUID in Kafka's base64 form (22 characters). A new cluster gets one from kafka-storage.sh random-uuid; every node of the same cluster must be formatted with the same id. A broker formatted with a fresh random id starts, then never joins: the controllers refuse its fetches of the metadata log with INCONSISTENT_CLUSTER_ID (an ERROR line in its server.log, many times a second), so its registration is never even sent. You get the existing id from the cluster itself:
$ kafka-cluster.sh cluster-id --bootstrap-server kafka-1:9092
Cluster ID: Lp72heEcEHtV4Jrw77XtwQ
The format command depends on the node:
| Node | Command |
|---|---|
| the first controllers of a new cluster | kafka-storage.sh format -t <id> -c <cfg> --initial-controllers "101@kraft-1:9093:<dirid>,..." |
| a single-node test cluster | kafka-storage.sh format -t <id> -c <cfg> --standalone |
| a broker (or a controller added later) | kafka-storage.sh format -t <id> -c <cfg> --no-initial-controllers |
Run it as the user that runs Kafka (sudo -u kafka ...): the files it creates belong to whoever ran it. A controller formatted without one of the three quorum flags is refused with a message that lists them.
$ kafka-storage.sh random-uuid
k75ivoGJ4QnvPP-V7nPl_g
When the active controller dies
If the leader crashes, the followers notice that it stopped sending (controller.quorum.fetch.timeout.ms, 2 s by default), wait a randomised election timeout, and vote; a candidate only gets a vote from a voter whose log is not ahead of its own, so the winner has every committed record. The epoch goes up by one. A clean stop (systemctl stop) is faster: the leader resigns as it shuts down and the others elect at once. In controller.log on the new leader:
[QuorumController id=102] Becoming the active controller at epoch 5, next write offset 4127.
Brokers find the new leader through controller.quorum.bootstrap.servers and carry on. From the outside, the cluster hiccups for a few seconds. The lab after this lesson makes you watch it happen.
In an interview: "KRaft replaced ZooKeeper in Kafka 4: three or five controllers keep the metadata in a Raft log and need a majority, so three survive one failure and five survive two. Brokers are observers that fetch that log. New nodes have to be formatted with the cluster's id before their first start."