OnCallReadyStart learning

Runtimes and data Lesson 28.39 · Kafka Operations · 3 min read

The Kafka runbook

In plain words

This is the page you keep next to the phone at night. When something rings, you do not need to remember everything from the chapter - only which five questions to ask first, which answers point at which cause, and the few things you must never do even when they would make the alarm stop at once and everybody asks you to do it.

The Kafka runbook

One page to keep. Each line is a lesson of this chapter; the commands are the ones you used.

Paged: where to start

SymptomFirst commandWhat you look for
producers fail, NOT_ENOUGH_REPLICAS in their logskafka-topics.sh --describe --under-min-isr-partitionswhich brokers are missing from the ISR
LEADER_NOT_AVAILABLE, partitions with Leader: nonekafka-topics.sh --describe --unavailable-partitionsevery replica of those partitions down
a broker missingkafka-cluster.sh list-endpoints --include-fenced-brokers, then systemctl status kafka and server.log on itfenced, not starting, start errors
nothing changes (no topics created, no failover), --describe times outkafka-metadata-quorum.sh describe --status (or --bootstrap-controller); if that times out too, systemctl status kafka on kraft-1..3a leader, a majority of controllers running
consumers behindkafka-consumer-groups.sh --describe --group G twice, a minute apartlag growing or shrinking, owners -, state
a group rebalancing all the timejournalctl of the consumer: "poll timeout has expired"max.poll.records x time per record vs max.poll.interval.ms
disk fillingdf -h /var/lib/kafka, kafka-log-dirs.sh --describethe topic that grows; its retention
a broker died on a full diskserver.log: "No space left on device", "Shutdown broker because ... log dir ... has failed"free non-Kafka space, start, fix retention

The decisions

  • Writes refused or data at risk? NOT_ENOUGH_REPLICAS is Kafka protecting acknowledged data. Do not "fix" it by lowering min.insync.replicas or switching producers to acks=1 without the owners' agreement: bring the brokers back.
  • Unclean election (--election-type unclean) gets a partition back at the price of every record the new leader did not have. A data-loss decision, not a restart.
  • Offset resets: stop every consumer of the group, --dry-run, export --to-current, then --execute. --to-datetime needs milliseconds and is UTC. A replay creates duplicates downstream.
  • Retention changes act at the next retention check and never touch the active segment.
  • Reassignments: throttle, keep the rollback JSON, --verify until complete (it removes the throttle).
  • Rolling restarts: one broker, wait for zero under-replicated partitions, then the next.

Never

  • Delete segment files or partition directories by hand on a running broker.
  • Format a node with a new random-uuid when it should join an existing cluster.
  • Run kafka-storage.sh format as root for a broker that runs as kafka.
  • Restart a second broker while partitions are still under-replicated.
  • Point a debugging console consumer at a production group's name.

Versions to know

Kafka 4.x is KRaft only (ZooKeeper removed in 4.0); 4.0 made the new consumer rebalance protocol production-ready; 4.1 enables Eligible Leader Replicas on new clusters; Java 17, 21 and 25 are supported for brokers (the documentation recommends the newest LTS). This chapter used 4.3.1, the release current on kafka.apache.org in October 2026 - check the release notes for anything newer before you rely on a default.

When every lab of this chapter is done, opening this lesson removes the lab VMs (kraft-1..3, kafka-1..4) and the shop's services from this box.

Why it helps

Under pressure people skip steps and reach for the fix that makes the alert go away: lowering min.insync.replicas, resetting a lagging group to the end, deleting segment files. A short runbook of first commands, symptom-to-cause pairs and forbidden shortcuts is what turns the chapter into reliable incident handling, for you and for whoever is on call next, even on their first shift.

FAQ

What are the first commands on a Kafka page?

kafka-cluster.sh list-endpoints for the brokers, kafka-metadata-quorum.sh describe --status for the quorum, kafka-topics.sh --describe with --under-min-isr-partitions or --under-replicated-partitions, kafka-consumer-groups.sh --describe for lag, and the broker's server.log or journal. Together they separate a cluster problem from a client one.

What should I never do to make errors stop?

Never lower min.insync.replicas, never turn on unclean leader election for important data, never reset a lagging group to the latest offsets without the owners agreeing, and never delete segment files by hand on a running broker. Each one trades the alert for lost data.

What does NOT_ENOUGH_REPLICAS tell me?

That a partition's ISR is smaller than min.insync.replicas, so the leader refuses acks=all writes. Look for brokers that are down, fenced or too slow to stay in sync, fix them, and wait for the followers to catch up.

How do I tell a stuck consumer from a slow one?

Lag on one partition only points at that partition (a poison record, a hot key, one stuck consumer). Lag on every partition with a Stable group means the group is too slow; a group that keeps rebalancing usually has a max.poll.interval.ms problem.

What happens to the lab at the end of the chapter?

Once every mission, incident and drill of the chapter is done, opening this lesson removes the Kafka lab machines and the applications from the box. Redoing a lab later builds a fresh cluster again, with the same machines and applications.

In an interview Mid

Producers report errors and a consumer group is far behind at the same time. How do you approach it?

Check the cluster first: brokers registered (list-endpoints), quorum leader (describe --status), partitions under min ISR (kafka-topics --describe --under-min-isr-partitions). NOT_ENOUGH_REPLICAS points at brokers out of the ISR; fix them without lowering min.insync.replicas. Then kafka-consumer-groups --describe for the lag: if the producers were failing, the lag may simply be the backlog, so watch whether it drains before changing the consumers.

Also asked: What should you never do to silence Kafka errors? · How do you tell a cluster problem from a client problem? · What does an under-replicated partition mean?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.