OnCallReadyStart learning

Runtimes and data Lesson 28.3 · Kafka Operations · 16 min read

KRaft: controllers, the metadata quorum and formatting storage

In plain words

The controllers are the cluster's committee. They keep the minutes of every decision - which topics exist, who leads which partition - in a shared notebook, and one of them is the chair who writes. The others copy every line. If the chair leaves, the remaining members vote for a new one, but a vote only counts when more than half of the committee is in the room.

Who decides: the controllers

Someone has to decide which broker leads each partition, notice that a broker died, and remember which topics exist. Up to Kafka 3.x that was ZooKeeper plus one broker acting as "the controller". Kafka 4.0 removed ZooKeeper. The cluster now keeps its metadata itself, in a replicated log called __cluster_metadata, managed by a small group of controllers that agree on it with the Raft consensus protocol. That mode is KRaft (Kafka Raft).

Every server has process.roles:

  • controller - takes part in the metadata quorum (this lab: kraft-1..3).
  • broker - stores partitions and serves clients (this lab: kafka-1..3).
  • broker,controller - both in one JVM ("combined"). Fine on a laptop; the Kafka documentation says combined mode is not recommended in critical environments, because you cannot restart or scale the controllers separately from the brokers.
$ ssh kraft-1 "grep -E '^(process.roles|node.id|controller.quorum|listeners|log.dirs)' /etc/kafka/server.properties"
process.roles=controller
node.id=101
controller.quorum.bootstrap.servers=kraft-1:9093,kraft-2:9093,kraft-3:9093
listeners=CONTROLLER://:9093
log.dirs=/var/lib/kafka/meta
$ ssh kafka-1 "grep -E '^(process.roles|node.id|controller.quorum|listeners|advertised|log.dirs)' /etc/kafka/server.properties"
process.roles=broker
node.id=1
controller.quorum.bootstrap.servers=kraft-1:9093,kraft-2:9093,kraft-3:9093
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafka-1:9092
log.dirs=/var/lib/kafka/data

node.id is unique across the whole cluster - controllers and brokers share the number space, which is why this lab gives the controllers 101-103. controller.quorum.bootstrap.servers is how every node finds the quorum: like a client's bootstrap.servers, it only has to reach one of them.

The quorum: a majority or nothing

The controllers elect one leader, the active controller. It writes every metadata change (a new topic, a broker fenced, a new leader for a partition) to __cluster_metadata; the other controllers are hot standbys that copy the log. A change counts once a majority of the voters have it. So:

ControllersCan loseWhy
10no redundancy at all
312 of 3 is a majority
413 of 4 is needed - the fourth buys nothing
523 of 5 is a majority

That is why the documentation says "3 or 5". With the majority gone, nothing that changes metadata works: no new topics, no leader elections, no ISR changes. Brokers keep serving the partitions they already lead with the metadata they already have, which is why a lost quorum can look fine for a few minutes and then fall apart as soon as a broker has to fail over. The tools tell you in their own way: kafka-topics.sh --describe hangs for a minute and fails (it asks the controller which partitions are being reassigned), and so does kafka-metadata-quorum.sh describe. Every metadata question timing out while produce and fetch still work is the symptom; systemctl status kafka on the controllers is the next step.

The brokers are observers: they fetch the metadata log too, but do not vote.

kafka-metadata-quorum.sh

The tool to look at the quorum. describe --status is the summary:

$ kafka-metadata-quorum.sh --bootstrap-server kafka-1:9092 describe --status
ClusterId:              Lp72heEcEHtV4Jrw77XtwQ
LeaderId:               101
LeaderEpoch:            4
HighWatermark:          4119
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   0
CurrentVoters:          [{"id": 101, "directoryId": "tSL9JiqDOEl9K6a4nENjVw", "endpoints": ["CONTROLLER://kraft-1:9093"]}, {"id": 102, "directoryId": "pkT2S34pSUTAqJagseUPNw", "endpoints": ["CONTROLLER://kraft-2:9093"]}, {"id": 103, "directoryId": "HGvV8dDzTwy1gz1oexGMtA", "endpoints": ["CONTROLLER://kraft-3:9093"]}]
CurrentObservers:       [{"id": 1, "directoryId": "H-alcE5QAoVEmIhKA_x9zw"}, {"id": 2, "directoryId": "6ZLwf2YcgzAaMV49XG0PdA"}, {"id": 3, "directoryId": "wyW69merdO3rZOjUdCxFTQ"}]
  • LeaderId / LeaderEpoch - the active controller and the election "term". The epoch goes up by one at every election: if you see it jump, controllers have been failing over.
  • HighWatermark - the offset in __cluster_metadata that a majority has. It keeps growing even on a quiet cluster: the active controller writes a no-op record every 500 ms (metadata.max.idle.interval.ms) so the followers can tell it is alive.
  • MaxFollowerLag / MaxFollowerLagTimeMs - how far the slowest voter is behind. 0 or close to it is healthy; growing means a controller is down or cannot keep up.
  • CurrentVoters - the controllers, each with its directoryId (from its meta.properties) and endpoint; CurrentObservers - the brokers.

describe --replication shows the same per node:

$ kafka-metadata-quorum.sh --bootstrap-server kafka-1:9092 describe --replication
NodeId	DirectoryId           	LogEndOffset	Lag	LastFetchTimestamp	LastCaughtUpTimestamp	Status
101   	tSL9JiqDOEl9K6a4nENjVw	4121        	0  	1790107211300     	1790107211300        	Leader
102   	pkT2S34pSUTAqJagseUPNw	4121        	0  	1790107211126     	1790107211126        	Follower
103   	HGvV8dDzTwy1gz1oexGMtA	4121        	0  	1790107211089     	1790107211089        	Follower
1     	H-alcE5QAoVEmIhKA_x9zw	4121        	0  	1790107211247     	1790107211247        	Observer
2     	6ZLwf2YcgzAaMV49XG0PdA	4121        	0  	1790107211194     	1790107211194        	Observer
3     	wyW69merdO3rZOjUdCxFTQ	4121        	0  	1790107211141     	1790107211141        	Observer

Lag is in records of the metadata log; the timestamps are milliseconds since 1970 (add --human-readable to get "120 ms ago"). A healthy voter has lag 0 and a last fetch a moment ago. A dead follower keeps its old LastFetchTimestamp while its lag grows; a voter that was down when the current leader was elected has never fetched from it and shows -1 everywhere (and MaxFollowerLagTimeMs in --status is then -1 too). You can ask the controllers directly with --bootstrap-controller kraft-1:9093 instead of going through a broker; that is the way in when every broker is down.

Formatting: why a new node does not just start

A KRaft node refuses to start on an empty data directory. Before the first start, its storage has to be formatted with kafka-storage.sh, which writes meta.properties: the node id, the cluster id and the directory's own id. Kafka did this automatically in the ZooKeeper days; the documentation explains why it no longer does - auto-formatting can hide a mistake (a disk that was not mounted, a node pointed at the wrong cluster), and for controllers an empty log could let a leader be elected without committed data.

$ ssh kafka-1 cat /var/lib/kafka/data/meta.properties
#
#Tue Sep 22 20:00:03 UTC 2026
cluster.id=Lp72heEcEHtV4Jrw77XtwQ
directory.id=H-alcE5QAoVEmIhKA_x9zw
node.id=1
version=1
$ ssh kafka-1 /opt/kafka/bin/kafka-storage.sh info -c /etc/kafka/server.properties
Found log directory:
  /var/lib/kafka/data

Found metadata: {cluster.id=Lp72heEcEHtV4Jrw77XtwQ, directory.id=H-alcE5QAoVEmIhKA_x9zw, node.id=1, version=1}

The cluster id is a random 16-byte UUID in Kafka's base64 form (22 characters). A new cluster gets one from kafka-storage.sh random-uuid; every node of the same cluster must be formatted with the same id. A broker formatted with a fresh random id starts, then never joins: the controllers refuse its fetches of the metadata log with INCONSISTENT_CLUSTER_ID (an ERROR line in its server.log, many times a second), so its registration is never even sent. You get the existing id from the cluster itself:

$ kafka-cluster.sh cluster-id --bootstrap-server kafka-1:9092
Cluster ID: Lp72heEcEHtV4Jrw77XtwQ

The format command depends on the node:

NodeCommand
the first controllers of a new clusterkafka-storage.sh format -t <id> -c <cfg> --initial-controllers "101@kraft-1:9093:<dirid>,..."
a single-node test clusterkafka-storage.sh format -t <id> -c <cfg> --standalone
a broker (or a controller added later)kafka-storage.sh format -t <id> -c <cfg> --no-initial-controllers

Run it as the user that runs Kafka (sudo -u kafka ...): the files it creates belong to whoever ran it. A controller formatted without one of the three quorum flags is refused with a message that lists them.

$ kafka-storage.sh random-uuid
k75ivoGJ4QnvPP-V7nPl_g

When the active controller dies

If the leader crashes, the followers notice that it stopped sending (controller.quorum.fetch.timeout.ms, 2 s by default), wait a randomised election timeout, and vote; a candidate only gets a vote from a voter whose log is not ahead of its own, so the winner has every committed record. The epoch goes up by one. A clean stop (systemctl stop) is faster: the leader resigns as it shuts down and the others elect at once. In controller.log on the new leader:

[QuorumController id=102] Becoming the active controller at epoch 5, next write offset 4127.

Brokers find the new leader through controller.quorum.bootstrap.servers and carry on. From the outside, the cluster hiccups for a few seconds. The lab after this lesson makes you watch it happen.

In an interview: "KRaft replaced ZooKeeper in Kafka 4: three or five controllers keep the metadata in a Raft log and need a majority, so three survive one failure and five survive two. Brokers are observers that fetch that log. New nodes have to be formatted with the cluster's id before their first start."

Why it helps

When metadata changes stop working, or a broker cannot register, the cause is usually in the quorum or in how a node was formatted. Knowing majorities tells you how many controllers you may restart at once. Knowing kafka-storage.sh format and meta.properties explains the INCONSISTENT_CLUSTER_ID and node id errors you meet when adding or rebuilding nodes, and how to avoid them before they page you.

Commands in this lesson

ssh kafka-metadata-quorum.sh kafka-cluster.sh kafka-storage.sh

FAQ

How many controllers should a cluster have?

Three for most clusters, five for very large or critical ones. Three survive one failure, five survive two. An even number adds no safety: four controllers still only survive one failure, because a majority of four is three.

What does kafka-metadata-quorum.sh describe --status show?

The cluster id, the current leader (LeaderId) and its epoch, the high watermark of the metadata log, how far the slowest follower is behind (MaxFollowerLag and MaxFollowerLagTimeMs), and the voters and observers. Brokers are observers: they read the metadata log but do not vote.

What is meta.properties?

A small file that kafka-storage.sh format writes into every log directory. It holds the cluster.id, the node.id and the directory.id. At start the broker checks it against server.properties and refuses to run with a mismatched node id rather than pretend to be another node.

What do --standalone and --initial-controllers do?

They are for controllers with a dynamic quorum (controller.quorum.bootstrap.servers). --standalone formats a single first controller, --initial-controllers lists every first voter with its directory id, and --no-initial-controllers formats a node that joins an existing quorum. Brokers need none of them.

What happens when the active controller stops?

The followers stop receiving fetch responses, one of them starts an election, and with a majority of votes becomes the new leader with a higher epoch. Brokers keep serving meanwhile. The old leader rejoins as a follower when it comes back; there is no preferred controller.

In an interview Mid

How does KRaft replace ZooKeeper, and how many controller failures can a cluster survive?

The metadata lives in an internal Raft log, __cluster_metadata, replicated across the controllers; one is the active controller and writes, the others replicate, and a majority must agree. A quorum of 3 survives 1 failure, 5 survive 2. kafka-metadata-quorum.sh describe --status shows the LeaderId and how far the followers are behind; nodes are prepared with kafka-storage.sh format and the cluster id.

Also asked: What does process.roles configure? · Why is an even number of controllers no better? · What is in meta.properties?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.