Kafka Operations
Apache Kafka 4.3 (KRaft) from the on-call side: controllers and the metadata quorum, topics, replication and the ISR, acks and min.insync.replicas, consumer groups and lag, offset resets, retention and full disks, kafka-configs, reassignments and preferred leaders, rolling restarts and the metrics that page you - on a six-node lab cluster.
In plain words
Kafka is a row of very long notebooks that many people write into and read from at the same time. Each notebook (a partition) only ever grows at the end, every line gets a number (the offset), and three copies of every notebook are kept on three different shelves (brokers) in case one shelf falls over. Readers keep a bookmark of how far they got. Most of the on-call work is about the copies, the bookmarks and the shelf space.
Why it matters on call
Many companies put Kafka between their services: orders, payments and e-mails all flow through it. When it misbehaves, the symptoms show up elsewhere - producers retrying, consumers falling behind, disks filling - and the platform team is the one paged. This chapter teaches the operator's side of Kafka 4.3 with KRaft: reading the tools' output, changing configs at the right level, moving data and bringing brokers back without losing records.
Lessons
- 28.1Kafka for the on-call engineer: the log, the cluster, the lab
- 28.3KRaft: controllers, the metadata quorum and formatting storage
- 28.5Topics, partitions and replicas: reading kafka-topics --describe
- 28.7Durability: acks, the ISR, min.insync.replicas and ELR
- 28.9Producing and consuming from the command line
- 28.11Consumer groups, lag and rebalances
- 28.13Resetting offsets safely
- 28.15Storage: segments, retention, compaction and full disks
- 28.17kafka-configs: dynamic, static and default settings
- 28.19Moving partitions: reassignment, throttles and preferred leaders
- 28.23Running brokers: rolling restarts, the JVM and the metrics that page you
- 28.39The Kafka runbook
Open this chapter in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.
Questions people ask
Do I need to know how to write Kafka producers and consumers?
No. The chapter treats the applications as given and teaches what you see from the cluster's side: the console clients for checks, the consumer group tool for lag and offsets, and the broker logs. Knowing a few client settings (acks, max.poll.records) matters because they cause incidents.
Why Kafka 4.3 and KRaft only?
Kafka 4.0 removed ZooKeeper completely; every supported cluster now runs its metadata on KRaft controllers. Older guides that mention ZooKeeper, --zookeeper flags or broker ids in znodes describe a cluster you will no longer meet in a new installation.
What is simulated in the lab?
The cluster model, the brokers' logs and the tools' output follow Kafka 4.3.1, but no real JVM runs: network traffic, garbage collection and the exact timing of fetches are modelled. The applications on the box (orders-api, billing, notifier) are simulator programs with real client settings.
Is this the same as running Kafka on Kubernetes?
The concepts are the same. On Kubernetes an operator such as Strimzi runs the brokers as pods and does rolling restarts for you, but the tools, the logs and the failure modes in this chapter are identical; you run the tools inside a broker pod with kubectl exec.
Which tool will I use most on call?
kafka-topics.sh --describe and kafka-consumer-groups.sh --describe. The first tells you where every partition's leader and in-sync replicas are, the second how far each consumer group is behind. Together they answer most first questions of a Kafka page in a minute.