Kafka Operations: interview questions
The question you are most likely to get for each topic, a model answer, and what else comes up.
You are paged because a Kafka-based service is slow. What do you check first? Mid
The five readings: are all brokers registered and unfenced (kafka-cluster.sh list-endpoints), does the metadata quorum have a leader (kafka-metadata-quorum.sh describe --status), are partitions under-replicated or under min.insync.replicas (kafka-topics.sh --describe with the report flags), how far behind the consumer groups are (kafka-consumer-groups.sh --describe), and the broker's server.log. That separates a cluster problem from a client one quickly.
Also asked: What is the difference between a broker and a controller? · What does consumer lag measure? · Why does Kafka keep three copies of each partition?
Explain Kafka's main parts to someone who has never used it. Mid
Producers write records to topics; each topic is split into partitions, ordered logs where every record gets an offset. Brokers store the partitions, with each partition copied to several brokers (replicas), one of them the leader. Controllers keep the metadata. Consumers read in consumer groups and commit their offsets, and lag is how far a group is behind the end of the log.
Also asked: What guarantees ordering in Kafka? · Does reading a record remove it? · What does a broker store?
Learn it: 28.1 Kafka for the on-call engineer: the log, the cluster, the lab
How does KRaft replace ZooKeeper, and how many controller failures can a cluster survive? Mid
The metadata lives in an internal Raft log, __cluster_metadata, replicated across the controllers; one is the active controller and writes, the others replicate, and a majority must agree. A quorum of 3 survives 1 failure, 5 survive 2. kafka-metadata-quorum.sh describe --status shows the LeaderId and how far the followers are behind; nodes are prepared with kafka-storage.sh format and the cluster id.
Also asked: What does process.roles configure? · Why is an even number of controllers no better? · What is in meta.properties?
Learn it: 28.3 KRaft: controllers, the metadata quorum and formatting storage
Read me a line of kafka-topics --describe and tell me if anything is wrong. Mid
Each partition line has Leader (the broker serving it), Replicas (the assignment, first one preferred) and Isr (replicas in sync). If Isr is shorter than Replicas the partition is under-replicated: a broker is down or behind. If the ISR is below min.insync.replicas, acks=all writes fail. Elr lists replicas that may still take over cleanly. --under-replicated-partitions shows only the problems.
Also asked: What is the difference between Replicas and Isr? · Why can you not remove partitions? · What does auto.create.topics.enable do?
Learn it: 28.5 Topics, partitions and replicas: reading kafka-topics --describe
What do acks=all and min.insync.replicas do together? Mid
acks=all makes the producer wait until every replica in the ISR has the record. min.insync.replicas is the broker-side minimum: if the ISR is smaller, the leader refuses the write with NOT_ENOUGH_REPLICAS. With RF 3 and min.insync.replicas 2, one broker can be down and writes still succeed with two copies; two down and acks=all producers get errors instead of single-copy writes.
Also asked: Why not set min.insync.replicas to 1 during an incident? · What is the difference between a controlled shutdown and a crash? · What does the ELR protect against?
Learn it: 28.7 Durability: acks, the ISR, min.insync.replicas and ELR
How do you check from the command line whether a topic receives data and what it looks like? Mid
kafka-get-offsets.sh shows the end offset of each partition - run it twice to see whether it grows. kafka-console-consumer.sh with --from-beginning, --max-messages and print.key, print.partition and print.offset shows real records. kafka-console-producer.sh with acks=all is a quick write test. A keyed record always lands in the same partition, which is what gives per-key ordering.
Also asked: Why does a new consumer start at the end? · How does a key choose its partition? · What happens to key ordering when partitions are added?
Learn it: 28.9 Producing and consuming from the command line
A consumer group's lag keeps growing. How do you find out why? Mid
kafka-consumer-groups --describe shows CURRENT-OFFSET, LOG-END-OFFSET and LAG per partition. Growing on every partition means the group is too slow: add consumers up to the partition count or speed them up. Growing on one partition means a hot key or a stuck consumer. --state shows rebalancing; a group that keeps rebalancing usually has max.poll.records times processing time above max.poll.interval.ms.
Also asked: Why can a group not use more consumers than partitions? · What triggers a rebalance? · What does the group coordinator do?
How do you replay a topic for a consumer group from a point in time? Mid
Stop every consumer of the group until --describe --state says Empty, export the current offsets with --reset-offsets --to-current --export, dry-run --reset-offsets --to-datetime with the time in UTC and the milliseconds, then run it with --execute and start the consumers. A replay means duplicates downstream unless the consumer is idempotent, so check with its owners first.
Also asked: Why must the group be inactive for a reset? · How do you undo a wrong reset? · What happens when a group's offsets expire?
Learn it: 28.13 Resetting offsets safely
How does Kafka retention work, and why might a topic use more disk than you expect? Mid
The log is split into segments and retention deletes whole closed segments when their newest record is older than retention.ms or while the partition exceeds retention.bytes, checked every log.retention.check.interval.ms. retention.bytes is per partition and per replica, and the active segment is never deleted, so a quiet topic keeps data until segment.ms rolls it. Compaction keeps the latest value per key instead.
Also asked: What is the difference between delete and compact? · What happens when a broker's data disk is full? · How do you size retention.bytes for a disk budget?
Learn it: 28.15 Storage: segments, retention, compaction and full disks
How do you change a broker setting without restarting the broker? Mid
With kafka-configs --alter --entity-type brokers, either --entity-default for every broker or --entity-name N for one, if the config is dynamically updatable; the value is stored in the metadata log and applies at once. Topic settings go on the topic with --entity-type topics. --describe --all shows each value's source; read-only configs like log.dirs need server.properties and a restart.
Also asked: What is the order of precedence for Kafka configs? · How do you remove a topic override? · What does a read-only broker config mean?
Learn it: 28.17 kafka-configs: dynamic, static and default settings
How do you move data onto a new Kafka broker without hurting production? Mid
kafka-reassign-partitions --generate with the topics and the new broker list gives a proposal and the current assignment; save the current one as rollback JSON. --execute with --throttle starts the copy at a bounded rate, --describe shows Adding Replicas and Removing Replicas, and --verify until every partition is completed, which also removes the throttles. Then check URP is 0 and run a preferred leader election if needed.
Also asked: Why does a new broker get no data? · What happens if you never run --verify? · What is the preferred leader?
Learn it: 28.19 Moving partitions: reassignment, throttles and preferred leaders
How do you restart a Kafka cluster's brokers safely, and what do you monitor? Mid
One broker at a time: check the cluster is healthy, restart it (SIGTERM gives a controlled shutdown, leaders move away first), and wait until under-replicated partitions are back to 0 before the next. Alert on UnderMinIsrPartitionCount and OfflinePartitionsCount above 0, the sum of ActiveControllerCount not being 1, UnderReplicatedPartitions above 0 for minutes, and consumer lag growth.
Also asked: Why keep the broker heap moderate? · What is a controlled shutdown? · Which Kafka metrics should page someone?
Learn it: 28.23 Running brokers: rolling restarts, the JVM and the metrics that page you
Producers report errors and a consumer group is far behind at the same time. How do you approach it? Mid
Check the cluster first: brokers registered (list-endpoints), quorum leader (describe --status), partitions under min ISR (kafka-topics --describe --under-min-isr-partitions). NOT_ENOUGH_REPLICAS points at brokers out of the ISR; fix them without lowering min.insync.replicas. Then kafka-consumer-groups --describe for the lag: if the producers were failing, the lag may simply be the backlog, so watch whether it drains before changing the consumers.
Also asked: What should you never do to silence Kafka errors? · How do you tell a cluster problem from a client problem? · What does an under-replicated partition mean?
Learn it: 28.39 The Kafka runbook
Practise these answers with flashcards and labs Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.