Why this chapter exists
Kafka sits in the middle of a lot of platforms: orders, payments, clicks and audit events are written to it by one set of services and read by others, sometimes hours later. When it is unhealthy, nothing crashes loudly at first. Producers start retrying, consumers fall behind, a disk fills up slowly - and an hour later the page says "payments are not being processed".
As the SRE you rarely write the producers or the consumers. You own the cluster: the brokers and controllers stay up, the data stays replicated, the disks do not fill, a consumer that falls behind is noticed, and a bad change can be undone. That is this chapter, on Apache Kafka 4.3.1 - the newest release on kafka.apache.org when this was written (25 June 2026). Kafka 4 runs only in KRaft mode: ZooKeeper was removed in 4.0, and the cluster keeps its own metadata.
What you need to know already: systemd units and journalctl (the systemd chapter), ssh to another machine, df/du (the filesystem chapter), and what a JVM heap is (the JVM chapter helps).
The words you need first
- Record (message, event) - a key, a value, a timestamp and optional headers. Kafka stores bytes; what the bytes mean is the applications' business.
- Topic - a named stream of records, like
orders. - Partition - a topic is split into partitions, numbered from 0. Each partition is an append-only log: records get increasing offsets (0, 1, 2 ...) and are never changed in place.
- Broker - a server that stores partitions and answers producers and consumers (port 9092 here).
- Controller - a server that manages the cluster's metadata: which topics exist, which broker leads each partition, who is alive. In KRaft mode the controllers form a quorum (port 9093).
- Replica - a copy of a partition on one broker. With a replication factor of 3, every partition lives on three brokers: one leader (all reads and writes go to it) and two followers that copy it.
- ISR (in-sync replicas) - the replicas that are caught up with the leader. Losing ISR members is what turns into producer errors.
- Producer - writes records to a topic. Consumer - reads them. A consumer group is a set of consumers that share the work: each partition is read by one member of the group.
- Offset commit / lag - a group remembers how far it got in each partition (its committed offset). Lag = the partition's end offset minus the committed offset: how much is waiting.
The lab
This chapter has its own cluster: six small Ubuntu VMs next to this box, built like a real tarball install of Apache Kafka (simulator: managed lab hosts).
| VM | Role | node.id | Listener |
|---|---|---|---|
kraft-1, kraft-2, kraft-3 | controllers (the KRaft quorum) | 101, 102, 103 | CONTROLLER://:9093 |
kafka-1, kafka-2, kafka-3 | brokers | 1, 2, 3 | PLAINTEXT://:9092 |
Kafka is unpacked in /opt/kafka_2.13-4.3.1 (the 2.13 is the Scala version it was built with) and /opt/kafka points at it. Each node has its configuration in /etc/kafka/server.properties, its data on its own disk mounted at /var/lib/kafka, its logs in /var/log/kafka, and runs as the unprivileged user kafka under a systemd unit called kafka.service. Apache ships no systemd unit: this one is the lab's, the kind every team writes. This box is your jump host: it has the same tarball, so all the bin/*.sh tools are here, and /etc/profile.d/kafka.sh puts /opt/kafka/bin on your PATH.
The shop's applications run on this box as services (simulator): orders-api and payments-api write to the orders and payments topics; billing and notifier read orders and notifications in the consumer groups billing and notifier.
A first look
Every Kafka command-line tool needs to know where the cluster is: --bootstrap-server takes one or more host:port of brokers. The tool asks that broker for the cluster's metadata and then talks to whichever brokers it needs. Start with the list of topics:
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --list
__consumer_offsets
notifications
orders
payments
__consumer_offsets is an internal topic: the consumer groups' committed offsets live there, in 50 partitions by default (offsets.topic.num.partitions). The other three are the shop's.
Now the brokers. kafka-cluster.sh list-endpoints asks the cluster which brokers are registered, and whether the controllers have fenced any (a fenced broker is cut off from leading anything):
$ kafka-cluster.sh list-endpoints --bootstrap-server kafka-1:9092
ID HOST PORT RACK STATE ENDPOINT_TYPE
1 kafka-1 9092 null unfenced broker
2 kafka-2 9092 null unfenced broker
3 kafka-3 9092 null unfenced broker
And on a broker itself, it is a normal systemd service:
$ ssh kafka-1 systemctl status kafka --no-pager --lines 0
● kafka.service - Apache Kafka broker (KRaft)
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; preset: enabled)
Active: active (running) since Tue 2026-09-22 20:00:03 UTC; 8s ago
Invocation: 227f06871c423f5a18a801ec8d6afdcd
Docs: https://kafka.apache.org/documentation/
Main PID: 1000 (java)
Tasks: 92 (limit: 4583)
Memory: 504.0M (peak: 504.0M)
CPU: 171ms
CGroup: /system.slice/kafka.service
└─1000 java -Xms1g -Xmx1g -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=…
The "Main PID" line says java: kafka-server-start.sh replaced itself (exec) with the JVM, so systemd watches the Java process. The full command line is long - the heap (-Xms1g -Xmx1g), the G1 garbage collector settings from kafka-run-class.sh, JMX on port 9999, and at the end the class kafka.Kafka with the configuration file.
The broker's own log is server.log. The line that matters after a start is the last one:
$ ssh kafka-1 "grep 'Kafka Server started' /var/log/kafka/server.log"
[2026-09-22 20:00:06,680] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)
The first five checks when Kafka pages you
- Is every node up?
kafka-cluster.sh list-endpoints(brokers,--include-fenced-brokersto see the fenced ones),systemctl status kafkaon the VM that is missing. - Does the controller quorum have a leader?
kafka-metadata-quorum.sh describe --status(next lesson). - Is anything under-replicated or offline?
kafka-topics.sh --describe --under-replicated-partitions,--under-min-isr-partitions,--unavailable-partitions(lesson 4). - Are consumers keeping up?
kafka-consumer-groups.sh --describe --group <g>and its LAG column (lesson 6). - Are the disks filling?
df -h /var/lib/kafkaon the brokers andkafka-log-dirs.sh(lesson 8).
Each of these gets its own lesson and lab. The chapter ends with four incidents: producers failing with NOT_ENOUGH_REPLICAS, a consumer group that falls further behind every minute, a broker disk that filled up, and an offset reset that made a service send yesterday's e-mails again.
In an interview: "A topic is split into partitions; a partition is an ordered, append-only log replicated to several brokers. One replica leads, the others follow, and the in-sync replicas are the ones caught up - that set decides whether a write with acks=all is accepted."