OnCallReadyStart learning

Runtimes and data Lesson 28.1 · Kafka Operations · 12 min read

Kafka for the on-call engineer: the log, the cluster, the lab

In plain words

Picture a post office where every letter is filed in order in a numbered drawer and never thrown away straight away. Writers drop letters at the end of a drawer, readers walk along the drawer at their own speed, and each reader remembers the number of the last letter read. Kafka is that post office: drawers are partitions, letter numbers are offsets, and the buildings that hold the drawers are brokers.

Why this chapter exists

Kafka sits in the middle of a lot of platforms: orders, payments, clicks and audit events are written to it by one set of services and read by others, sometimes hours later. When it is unhealthy, nothing crashes loudly at first. Producers start retrying, consumers fall behind, a disk fills up slowly - and an hour later the page says "payments are not being processed".

As the SRE you rarely write the producers or the consumers. You own the cluster: the brokers and controllers stay up, the data stays replicated, the disks do not fill, a consumer that falls behind is noticed, and a bad change can be undone. That is this chapter, on Apache Kafka 4.3.1 - the newest release on kafka.apache.org when this was written (25 June 2026). Kafka 4 runs only in KRaft mode: ZooKeeper was removed in 4.0, and the cluster keeps its own metadata.

What you need to know already: systemd units and journalctl (the systemd chapter), ssh to another machine, df/du (the filesystem chapter), and what a JVM heap is (the JVM chapter helps).

The words you need first

  • Record (message, event) - a key, a value, a timestamp and optional headers. Kafka stores bytes; what the bytes mean is the applications' business.
  • Topic - a named stream of records, like orders.
  • Partition - a topic is split into partitions, numbered from 0. Each partition is an append-only log: records get increasing offsets (0, 1, 2 ...) and are never changed in place.
  • Broker - a server that stores partitions and answers producers and consumers (port 9092 here).
  • Controller - a server that manages the cluster's metadata: which topics exist, which broker leads each partition, who is alive. In KRaft mode the controllers form a quorum (port 9093).
  • Replica - a copy of a partition on one broker. With a replication factor of 3, every partition lives on three brokers: one leader (all reads and writes go to it) and two followers that copy it.
  • ISR (in-sync replicas) - the replicas that are caught up with the leader. Losing ISR members is what turns into producer errors.
  • Producer - writes records to a topic. Consumer - reads them. A consumer group is a set of consumers that share the work: each partition is read by one member of the group.
  • Offset commit / lag - a group remembers how far it got in each partition (its committed offset). Lag = the partition's end offset minus the committed offset: how much is waiting.

The lab

This chapter has its own cluster: six small Ubuntu VMs next to this box, built like a real tarball install of Apache Kafka (simulator: managed lab hosts).

VMRolenode.idListener
kraft-1, kraft-2, kraft-3controllers (the KRaft quorum)101, 102, 103CONTROLLER://:9093
kafka-1, kafka-2, kafka-3brokers1, 2, 3PLAINTEXT://:9092

Kafka is unpacked in /opt/kafka_2.13-4.3.1 (the 2.13 is the Scala version it was built with) and /opt/kafka points at it. Each node has its configuration in /etc/kafka/server.properties, its data on its own disk mounted at /var/lib/kafka, its logs in /var/log/kafka, and runs as the unprivileged user kafka under a systemd unit called kafka.service. Apache ships no systemd unit: this one is the lab's, the kind every team writes. This box is your jump host: it has the same tarball, so all the bin/*.sh tools are here, and /etc/profile.d/kafka.sh puts /opt/kafka/bin on your PATH.

The shop's applications run on this box as services (simulator): orders-api and payments-api write to the orders and payments topics; billing and notifier read orders and notifications in the consumer groups billing and notifier.

A first look

Every Kafka command-line tool needs to know where the cluster is: --bootstrap-server takes one or more host:port of brokers. The tool asks that broker for the cluster's metadata and then talks to whichever brokers it needs. Start with the list of topics:

$ kafka-topics.sh --bootstrap-server kafka-1:9092 --list
__consumer_offsets
notifications
orders
payments

__consumer_offsets is an internal topic: the consumer groups' committed offsets live there, in 50 partitions by default (offsets.topic.num.partitions). The other three are the shop's.

Now the brokers. kafka-cluster.sh list-endpoints asks the cluster which brokers are registered, and whether the controllers have fenced any (a fenced broker is cut off from leading anything):

$ kafka-cluster.sh list-endpoints --bootstrap-server kafka-1:9092
ID         HOST    PORT       RACK       STATE      ENDPOINT_TYPE
1          kafka-1 9092       null       unfenced   broker
2          kafka-2 9092       null       unfenced   broker
3          kafka-3 9092       null       unfenced   broker

And on a broker itself, it is a normal systemd service:

$ ssh kafka-1 systemctl status kafka --no-pager --lines 0
● kafka.service - Apache Kafka broker (KRaft)
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-09-22 20:00:03 UTC; 8s ago
 Invocation: 227f06871c423f5a18a801ec8d6afdcd
       Docs: https://kafka.apache.org/documentation/
   Main PID: 1000 (java)
      Tasks: 92 (limit: 4583)
     Memory: 504.0M (peak: 504.0M)
        CPU: 171ms
     CGroup: /system.slice/kafka.service
             └─1000 java -Xms1g -Xmx1g -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=…

The "Main PID" line says java: kafka-server-start.sh replaced itself (exec) with the JVM, so systemd watches the Java process. The full command line is long - the heap (-Xms1g -Xmx1g), the G1 garbage collector settings from kafka-run-class.sh, JMX on port 9999, and at the end the class kafka.Kafka with the configuration file.

The broker's own log is server.log. The line that matters after a start is the last one:

$ ssh kafka-1 "grep 'Kafka Server started' /var/log/kafka/server.log"
[2026-09-22 20:00:06,680] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)

The first five checks when Kafka pages you

  1. Is every node up? kafka-cluster.sh list-endpoints (brokers, --include-fenced-brokers to see the fenced ones), systemctl status kafka on the VM that is missing.
  2. Does the controller quorum have a leader? kafka-metadata-quorum.sh describe --status (next lesson).
  3. Is anything under-replicated or offline? kafka-topics.sh --describe --under-replicated-partitions, --under-min-isr-partitions, --unavailable-partitions (lesson 4).
  4. Are consumers keeping up? kafka-consumer-groups.sh --describe --group <g> and its LAG column (lesson 6).
  5. Are the disks filling? df -h /var/lib/kafka on the brokers and kafka-log-dirs.sh (lesson 8).

Each of these gets its own lesson and lab. The chapter ends with four incidents: producers failing with NOT_ENOUGH_REPLICAS, a consumer group that falls further behind every minute, a broker disk that filled up, and an offset reset that made a service send yesterday's e-mails again.

In an interview: "A topic is split into partitions; a partition is an ordered, append-only log replicated to several brokers. One replica leads, the others follow, and the in-sync replicas are the ones caught up - that set decides whether a write with acks=all is accepted."

Why it helps

Every later lesson uses these words: record, topic, partition, offset, broker, controller, replica, consumer group and lag. Knowing what each one is - and which machine in the lab plays which role - lets you read a Kafka page quickly. The first five checks of an incident also start here: brokers, quorum, partitions, groups and the broker's own log, in that order, every time.

Commands in this lesson

kafka-topics.sh kafka-cluster.sh ssh

FAQ

What is the difference between a topic and a partition?

A topic is a named stream, like orders. It is split into partitions, and each partition is an ordered log of records with offsets 0, 1, 2 and so on. Ordering is guaranteed inside one partition only, not across the whole topic.

Is a Kafka consumer like a queue consumer that removes messages?

No. Reading does not delete anything. Records stay until retention removes them, and each consumer group keeps its own position (its committed offset), so several groups can read the same topic independently and a group can even go back and read again.

What are kraft-1 to kraft-3 in the lab?

They are the KRaft controllers, node ids 101 to 103. They hold the cluster's metadata (topics, partitions, leaders, configs) in a replicated log and elect one active controller. They do not store topic data; the brokers kafka-1 to kafka-3 do.

How was Kafka installed on the lab machines?

From the Apache tarball into /opt/kafka_2.13-4.3.1 with a symlink /opt/kafka, a kafka user, the config in /etc/kafka/server.properties and a systemd unit written by the team. Apache Kafka ships no unit file and no distribution package, so this layout is common.

Where are the logs of a broker?

In LOG_DIR, which the lab's unit sets to /var/log/kafka: server.log for the broker, controller.log for controller decisions, log-cleaner.log for compaction. The same server.log lines also reach the journal through standard output, so journalctl -u kafka shows them too.

In an interview Mid

Explain Kafka's main parts to someone who has never used it.

Producers write records to topics; each topic is split into partitions, ordered logs where every record gets an offset. Brokers store the partitions, with each partition copied to several brokers (replicas), one of them the leader. Controllers keep the metadata. Consumers read in consumer groups and commit their offsets, and lag is how far a group is behind the end of the log.

Also asked: What guarantees ordering in Kafka? · Does reading a record remove it? · What does a broker store?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.