When is a write "written"?
A producer sends a batch to the partition's leader. What it waits for is the producer setting acks:
| acks | The leader answers when | Risk |
|---|---|---|
0 | never - the producer does not wait | records lost silently on any hiccup |
1 | it has written the batch to its own log | the leader dies before followers copied it: lost |
all (-1) | every replica in the ISR has it | none as long as one ISR member survives |
Since Kafka 3.0 the Java producer defaults to acks=all with enable.idempotence=true (retries cannot create duplicates), retries at the maximum and delivery.timeout.ms=120000: a record that cannot be written is retried for two minutes before the application hears about it.
The ISR and the high watermark
The in-sync replicas are the leader plus the followers that are caught up. A follower that stops fetching, or falls further behind than replica.lag.time.max.ms (30 s), is removed from the ISR by the leader. A broker that the controller fences (it stopped heartbeating for broker.session.timeout.ms, 9 s) is removed from every ISR at once.
The high watermark is the offset up to which every ISR member has the data. Consumers only see records below it, so a consumer never reads something that could still disappear in a failover.
min.insync.replicas: refusing writes on purpose
acks=all alone has a hole: if the ISR shrinks to the leader alone, "all ISR members have it" means "one copy". min.insync.replicas (a topic or cluster setting, default 1) closes it: with acks=all, the leader refuses a write while the ISR is smaller than this number, with NOT_ENOUGH_REPLICAS.
The standard production setting is RF 3, min.insync.replicas 2, acks=all: one broker can be down (maintenance, a crash) and writes continue with two copies; a second failure stops writes rather than accept data that exists only once. acks=1 producers are not affected by min.insync.replicas at all.
Let us take a broker away. A planned stop is a controlled shutdown: the broker tells the controller, its leaderships move to other replicas first, then it leaves:
$ ssh kafka-3 sudo systemctl stop kafka
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic payments
Topic: payments TopicId: F9uOvNBGcBs5FkImJ35bjg PartitionCount: 6 ReplicationFactor: 3 Configs: min.insync.replicas=2,segment.bytes=1073741824
Topic: payments Partition: 0 Leader: 1 Replicas: 3,1,2 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,1 Elr: LastKnownElr:
Topic: payments Partition: 3 Leader: 1 Replicas: 3,1,2 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 4 Leader: 1 Replicas: 1,2,3 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 5 Leader: 2 Replicas: 2,3,1 Isr: 2,1 Elr: LastKnownElr:
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --under-replicated-partitions --topic payments
Topic: payments Partition: 0 Leader: 1 Replicas: 3,1,2 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,1 Elr: LastKnownElr:
Topic: payments Partition: 3 Leader: 1 Replicas: 3,1,2 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 4 Leader: 1 Replicas: 1,2,3 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 5 Leader: 2 Replicas: 2,3,1 Isr: 2,1 Elr: LastKnownElr:
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --at-min-isr-partitions --topic payments
Topic: payments Partition: 0 Leader: 1 Replicas: 3,1,2 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,1 Elr: LastKnownElr:
Topic: payments Partition: 3 Leader: 1 Replicas: 3,1,2 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 4 Leader: 1 Replicas: 1,2,3 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 5 Leader: 2 Replicas: 2,3,1 Isr: 2,1 Elr: LastKnownElr:
Every partition is under-replicated (Isr has 2 of 3) and at min ISR: writes still work, but one more failure stops them. Now the second broker - not cleanly, its VM loses power (lab-vm is the lab's hypervisor console, simulator):
$ lab-vm stop kafka-2
kafka-2: powered off
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic payments | head -3
Topic: payments TopicId: F9uOvNBGcBs5FkImJ35bjg PartitionCount: 6 ReplicationFactor: 3 Configs: min.insync.replicas=2,segment.bytes=1073741824
Topic: payments Partition: 0 Leader: 1 Replicas: 3,1,2 Isr: 1,2 Elr: LastKnownElr:
Topic: payments Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2 Elr: LastKnownElr:
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic payments --under-min-isr-partitions
Topic: payments Partition: 0 Leader: 1 Replicas: 3,1,2 Isr: 1 Elr: 2 LastKnownElr:
Topic: payments Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1 Elr: 2 LastKnownElr:
Topic: payments Partition: 2 Leader: 1 Replicas: 2,3,1 Isr: 1 Elr: 2 LastKnownElr:
Topic: payments Partition: 3 Leader: 1 Replicas: 3,1,2 Isr: 1 Elr: 2 LastKnownElr:
Topic: payments Partition: 4 Leader: 1 Replicas: 1,2,3 Isr: 1 Elr: 2 LastKnownElr:
Topic: payments Partition: 5 Leader: 1 Replicas: 2,3,1 Isr: 1 Elr: 2 LastKnownElr:
Right after the power cut nothing changed: the controller still counts kafka-2 as alive until its session times out. Nine seconds later it is fenced: the ISR is 1, and kafka-2 moved to the Elr.
Producing with acks=all now fails (the console producer's default --request-required-acks is -1, and it retries 3 times):
$ echo 'refund 4711' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic payments
[2026-09-22 20:00:28,550] WARN [Producer clientId=console-producer] Got error produce response with correlation id 3 on topic-partition payments-3, retrying (2 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-09-22 20:00:28,650] WARN [Producer clientId=console-producer] Got error produce response with correlation id 4 on topic-partition payments-3, retrying (1 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-09-22 20:00:28,750] WARN [Producer clientId=console-producer] Got error produce response with correlation id 5 on topic-partition payments-3, retrying (0 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-09-22 20:00:28,850] ERROR Error when sending message to topic payments with key: null, value: 11 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.
The WARN ... Got error produce response ... Error: NOT_ENOUGH_REPLICAS lines are the producer's Sender retrying; real applications log exactly these lines, every few seconds, until delivery.timeout.ms runs out. With acks=1 the same write goes through - into a partition that now exists on one disk only:
$ echo 'refund 4711' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic payments --request-required-acks 1
The leader logs every refused append in its server.log, at ERROR, with the reason spelled out:
$ ssh kafka-1 "grep -A1 'Error processing append operation' /var/log/kafka/server.log" | tail -2
[2026-09-22 20:00:28,850] ERROR [ReplicaManager broker=1] Error processing append operation on partition F9uOvNBGcBs5FkImJ35bjg:payments-3 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR : 1 is insufficient to satisfy the min.isr requirement of 2 for partition payments-3, live replica(s) broker.id are : Set(1)
On a busy cluster that is thousands of lines a minute; the broker metric UnderMinIsrPartitionCount (lesson 11) is what should page you before anyone greps.
ELR: who may lead when the ISR is gone
If the ISR keeps shrinking down to nothing, who can become leader without losing acknowledged data? Kafka 4 tracks the Eligible Leader Replicas (KIP-966; on by default for clusters created with Kafka 4.1 or later). Because the high watermark cannot advance while the ISR is below min.insync.replicas, a replica that was in the ISR when it dropped below that size still has every committed record. Those replicas go into Elr, and the controller may elect one of them, in this order: someone in the ISR, then someone in the ELR. A broker that comes back after a crash (no clean shutdown) is taken out of the ELR when it registers, because it may have lost data that was not flushed to disk.
The last resort is an unclean election: a replica outside ISR and ELR becomes leader and everything it did not have is gone. unclean.leader.election.enable is false by default; kafka-leader-election.sh --election-type unclean forces one for a partition. It trades data for availability - a decision for the people who own the data, not a reflex at 3 a.m.
Bring both brokers back:
$ lab-vm start kafka-2
kafka-2: powering on (sshd answers in about 15 s)
$ ssh kafka-3 sudo systemctl start kafka
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic payments
Topic: payments TopicId: F9uOvNBGcBs5FkImJ35bjg PartitionCount: 6 ReplicationFactor: 3 Configs: min.insync.replicas=2,segment.bytes=1073741824
Topic: payments Partition: 0 Leader: 1 Replicas: 3,1,2 Isr: 3,1,2 Elr: LastKnownElr:
Topic: payments Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 Elr: LastKnownElr:
Topic: payments Partition: 2 Leader: 1 Replicas: 2,3,1 Isr: 2,3,1 Elr: LastKnownElr:
Topic: payments Partition: 3 Leader: 1 Replicas: 3,1,2 Isr: 3,1,2 Elr: LastKnownElr:
Topic: payments Partition: 4 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 Elr: LastKnownElr:
Topic: payments Partition: 5 Leader: 1 Replicas: 2,3,1 Isr: 2,3,1 Elr: LastKnownElr:
The ISRs fill up again as the followers catch up. Notice the leaders: they stayed where the failover put them. Moving leadership back to the preferred replicas is lesson 10.
In an interview: "With RF 3, min.insync.replicas 2 and acks=all you survive one broker down without losing writes or acknowledged data. A second failure makes producers get NOT_ENOUGH_REPLICAS - availability is traded for durability on purpose. acks=1 keeps writing, onto a single copy."