Restarting brokers without an outage
Patches, a new JVM option, a static config change: they all mean restarting every broker. The rule is one broker at a time, and the next only when the cluster is whole again:
- Check the cluster is healthy: no under-replicated partitions, the quorum has a leader.
sudo systemctl restart kafkaon one broker. SIGTERM starts a controlled shutdown: the broker asks the controller to move its leaderships away, then closes its logs cleanly (no recovery at the next start). The lab's unit gives itTimeoutStopSec=180- a broker killed with SIGKILL by systemd after the default 90 s would lose the clean shutdown on a big disk.- Wait until it is back and caught up:
--under-replicated-partitionsempty again. Only then the next broker.
With RF 3 and min.insync.replicas 2, two brokers down at once means NOT_ENOUGH_REPLICAS for every partition they share. Rushing the second restart is how a routine patch turns into an incident.
$ ssh kafka-2 sudo systemctl restart kafka
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --under-replicated-partitions | wc -l
65
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --under-replicated-partitions | wc -l
0
$ ssh kafka-2 "grep -E 'controlled shutdown|Skipping recovery|Kafka Server started' /var/log/kafka/server.log" | tail -4
[2026-09-22 20:00:09,120] INFO [BrokerLifecycleManager id=2] Beginning controlled shutdown. (org.apache.kafka.server.BrokerLifecycleManager)
[2026-09-22 20:00:09,120] INFO [BrokerLifecycleManager id=2] The controller has asked us to exit controlled shutdown. (org.apache.kafka.server.BrokerLifecycleManager)
[2026-09-22 20:00:11,220] INFO Skipping recovery of 65 logs from /var/lib/kafka/data since clean shutdown file was found (kafka.log.LogManager)
[2026-09-22 20:00:13,000] INFO [KafkaRaftServer nodeId=2] Kafka Server started (kafka.server.KafkaRaftServer)
After a rolling restart the leaders are not where they were; the controller's imbalance check (or a preferred election, lesson 10) puts them back.
The JVM
A broker is a Java process: its heap is set by KAFKA_HEAP_OPTS (the lab's unit sets -Xms1g -Xmx1g; kafka-server-start.sh uses -Xmx1G -Xms1G when it is empty, and the tools use -Xmx256M). Kafka needs less heap than people expect, because it does not cache the data on the heap: it writes to files and reads them through the operating system's page cache. The Kafka documentation's example for a busy production broker is a 6 GB heap; the rest of the machine's memory should be left free for the page cache, which serves consumers that read the latest records without touching the disk.
The lab's unit adds KAFKA_OPTS=-XX:+ExitOnOutOfMemoryError: a JVM that runs out of heap exits at once (Terminating due to java.lang.OutOfMemoryError: Java heap space, exit code 3) and systemd restarts it, instead of limping on with broken threads. Check what a running broker really got:
$ ssh kafka-1 'ps -o rss=,args= -C java' | tr ' ' '\n' | grep -E '^-Xm|ExitOnOutOfMemory'
-Xms1g
-Xmx1g
-XX:+ExitOnOutOfMemoryError
$ ssh kafka-1 systemctl show kafka -p Environment -p LimitNOFILE
Environment="KAFKA_HEAP_OPTS=-Xms1g -Xmx1g" "KAFKA_OPTS=-XX:+ExitOnOutOfMemoryError" "LOG_DIR=/var/log/kafka" "JMX_PORT=9999"
LimitNOFILE=100000
Change the heap with a drop-in (sudo systemctl edit kafka) and a restart - one broker at a time.
File descriptors. Each segment is open files (log, index, timeindex), plus one socket per client connection. A broker with thousands of partitions needs tens of thousands of descriptors; the systemd default soft limit of 1024 is not enough, hence LimitNOFILE=100000 in the unit. Too many open files in server.log means the limit was not raised for the process that runs.
The metrics that should page you
Brokers and controllers publish metrics over JMX (the lab's unit sets JMX_PORT=9999). In production a JMX exporter feeds them to Prometheus or similar; kafka-jmx.sh reads them by hand:
$ kafka-jmx.sh --jmx-url service:jmx:rmi:///jndi/rmi://kafka-1:9999/jmxrmi --object-name kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions --one-time true
Trying to connect to JMX url: service:jmx:rmi:///jndi/rmi://kafka-1:9999/jmxrmi
"time","kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions:Value"
1790107244200,0
$ kafka-jmx.sh --jmx-url service:jmx:rmi:///jndi/rmi://kraft-1:9999/jmxrmi --object-name 'kafka.controller:type=KafkaController,name=*' --one-time true --report-format properties
Trying to connect to JMX url: service:jmx:rmi:///jndi/rmi://kraft-1:9999/jmxrmi
time=1790107245050
kafka.controller:type=KafkaController,name=ActiveBrokerCount:Value=3
kafka.controller:type=KafkaController,name=ActiveControllerCount:Value=1
kafka.controller:type=KafkaController,name=FencedBrokerCount:Value=0
kafka.controller:type=KafkaController,name=GlobalPartitionCount:Value=65
kafka.controller:type=KafkaController,name=GlobalTopicCount:Value=4
kafka.controller:type=KafkaController,name=OfflinePartitionsCount:Value=0
kafka.controller:type=KafkaController,name=PreferredReplicaImbalanceCount:Value=22
(Trying to connect to JMX url: ... is printed on stderr; the values on stdout.)
| Metric (MBean) | Where | Alert when |
|---|---|---|
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions | broker | > 0 for more than a few minutes |
kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount | broker | > 0: acks=all writes are failing |
kafka.controller:type=KafkaController,name=OfflinePartitionsCount | active controller | > 0: partitions without a leader |
kafka.controller:type=KafkaController,name=ActiveControllerCount | controllers | the sum over controllers is not exactly 1 |
kafka.controller:type=KafkaController,name=FencedBrokerCount | active controller | > 0 |
kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent | broker | below ~0.3: the broker is saturated |
| consumer lag (per group) | from the coordinator, via an exporter | growing for many minutes |
Lag is not a broker metric: it comes from comparing committed offsets with log-end offsets (what kafka-consumer-groups --describe does), so exporters such as Burrow or kafka-exporter compute it, or the consumers report records-lag-max themselves. Alert on lag growth or on lag in time (minutes behind), not on a fixed record count.
Kafka on Kubernetes
The same cluster can run on Kubernetes under an operator; the common open-source one is Strimzi (a CNCF project). You declare a Kafka resource and KafkaNodePools (controllers and brokers as separate pools - the same KRaft split as here), KafkaTopic and KafkaUser, and the operator runs the brokers as pods with persistent volumes and does the rolling restarts for you, waiting for the replicas to catch up between pods. Everything in this chapter still applies - the tools run inside the broker pods (kubectl exec), and a pod that will not start shows the same server.log lines in kubectl logs.
In an interview: "Roll one broker at a time and wait for under-replicated partitions to reach zero before the next. Keep the heap moderate and leave memory to the page cache. Alert on UnderMinIsrPartitionCount, OfflinePartitionsCount, the active controller count and consumer lag growth."