OnCallReadyStart learning

Runtimes and data Lesson 28.7 · Kafka Operations · 16 min read

Durability: acks, the ISR, min.insync.replicas and ELR

In plain words

When you hand over an important letter, you can walk away at once (acks=0), wait for the clerk to say "got it" (acks=1), or wait until the clerk has also made copies in the other offices (acks=all). min.insync.replicas is the office rule: "we only say got it when at least two offices have a copy - if not, we refuse the letter".

When is a write "written"?

A producer sends a batch to the partition's leader. What it waits for is the producer setting acks:

acksThe leader answers whenRisk
0never - the producer does not waitrecords lost silently on any hiccup
1it has written the batch to its own logthe leader dies before followers copied it: lost
all (-1)every replica in the ISR has itnone as long as one ISR member survives

Since Kafka 3.0 the Java producer defaults to acks=all with enable.idempotence=true (retries cannot create duplicates), retries at the maximum and delivery.timeout.ms=120000: a record that cannot be written is retried for two minutes before the application hears about it.

The ISR and the high watermark

The in-sync replicas are the leader plus the followers that are caught up. A follower that stops fetching, or falls further behind than replica.lag.time.max.ms (30 s), is removed from the ISR by the leader. A broker that the controller fences (it stopped heartbeating for broker.session.timeout.ms, 9 s) is removed from every ISR at once.

The high watermark is the offset up to which every ISR member has the data. Consumers only see records below it, so a consumer never reads something that could still disappear in a failover.

min.insync.replicas: refusing writes on purpose

acks=all alone has a hole: if the ISR shrinks to the leader alone, "all ISR members have it" means "one copy". min.insync.replicas (a topic or cluster setting, default 1) closes it: with acks=all, the leader refuses a write while the ISR is smaller than this number, with NOT_ENOUGH_REPLICAS.

The standard production setting is RF 3, min.insync.replicas 2, acks=all: one broker can be down (maintenance, a crash) and writes continue with two copies; a second failure stops writes rather than accept data that exists only once. acks=1 producers are not affected by min.insync.replicas at all.

Let us take a broker away. A planned stop is a controlled shutdown: the broker tells the controller, its leaderships move to other replicas first, then it leaves:

$ ssh kafka-3 sudo systemctl stop kafka
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic payments
Topic: payments	TopicId: F9uOvNBGcBs5FkImJ35bjg	PartitionCount: 6	ReplicationFactor: 3	Configs: min.insync.replicas=2,segment.bytes=1073741824
	Topic: payments	Partition: 0	Leader: 1	Replicas: 3,1,2	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 1	Leader: 1	Replicas: 1,2,3	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 2	Leader: 2	Replicas: 2,3,1	Isr: 2,1	Elr: 	LastKnownElr:
	Topic: payments	Partition: 3	Leader: 1	Replicas: 3,1,2	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 4	Leader: 1	Replicas: 1,2,3	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 5	Leader: 2	Replicas: 2,3,1	Isr: 2,1	Elr: 	LastKnownElr:
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --under-replicated-partitions --topic payments
	Topic: payments	Partition: 0	Leader: 1	Replicas: 3,1,2	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 1	Leader: 1	Replicas: 1,2,3	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 2	Leader: 2	Replicas: 2,3,1	Isr: 2,1	Elr: 	LastKnownElr:
	Topic: payments	Partition: 3	Leader: 1	Replicas: 3,1,2	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 4	Leader: 1	Replicas: 1,2,3	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 5	Leader: 2	Replicas: 2,3,1	Isr: 2,1	Elr: 	LastKnownElr:
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --at-min-isr-partitions --topic payments
	Topic: payments	Partition: 0	Leader: 1	Replicas: 3,1,2	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 1	Leader: 1	Replicas: 1,2,3	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 2	Leader: 2	Replicas: 2,3,1	Isr: 2,1	Elr: 	LastKnownElr:
	Topic: payments	Partition: 3	Leader: 1	Replicas: 3,1,2	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 4	Leader: 1	Replicas: 1,2,3	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 5	Leader: 2	Replicas: 2,3,1	Isr: 2,1	Elr: 	LastKnownElr:

Every partition is under-replicated (Isr has 2 of 3) and at min ISR: writes still work, but one more failure stops them. Now the second broker - not cleanly, its VM loses power (lab-vm is the lab's hypervisor console, simulator):

$ lab-vm stop kafka-2
kafka-2: powered off
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic payments | head -3
Topic: payments	TopicId: F9uOvNBGcBs5FkImJ35bjg	PartitionCount: 6	ReplicationFactor: 3	Configs: min.insync.replicas=2,segment.bytes=1073741824
	Topic: payments	Partition: 0	Leader: 1	Replicas: 3,1,2	Isr: 1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 1	Leader: 1	Replicas: 1,2,3	Isr: 1,2	Elr: 	LastKnownElr:
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic payments --under-min-isr-partitions
	Topic: payments	Partition: 0	Leader: 1	Replicas: 3,1,2	Isr: 1	Elr: 2	LastKnownElr:
	Topic: payments	Partition: 1	Leader: 1	Replicas: 1,2,3	Isr: 1	Elr: 2	LastKnownElr:
	Topic: payments	Partition: 2	Leader: 1	Replicas: 2,3,1	Isr: 1	Elr: 2	LastKnownElr:
	Topic: payments	Partition: 3	Leader: 1	Replicas: 3,1,2	Isr: 1	Elr: 2	LastKnownElr:
	Topic: payments	Partition: 4	Leader: 1	Replicas: 1,2,3	Isr: 1	Elr: 2	LastKnownElr:
	Topic: payments	Partition: 5	Leader: 1	Replicas: 2,3,1	Isr: 1	Elr: 2	LastKnownElr:

Right after the power cut nothing changed: the controller still counts kafka-2 as alive until its session times out. Nine seconds later it is fenced: the ISR is 1, and kafka-2 moved to the Elr.

Producing with acks=all now fails (the console producer's default --request-required-acks is -1, and it retries 3 times):

$ echo 'refund 4711' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic payments
[2026-09-22 20:00:28,550] WARN [Producer clientId=console-producer] Got error produce response with correlation id 3 on topic-partition payments-3, retrying (2 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-09-22 20:00:28,650] WARN [Producer clientId=console-producer] Got error produce response with correlation id 4 on topic-partition payments-3, retrying (1 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-09-22 20:00:28,750] WARN [Producer clientId=console-producer] Got error produce response with correlation id 5 on topic-partition payments-3, retrying (0 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-09-22 20:00:28,850] ERROR Error when sending message to topic payments with key: null, value: 11 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.

The WARN ... Got error produce response ... Error: NOT_ENOUGH_REPLICAS lines are the producer's Sender retrying; real applications log exactly these lines, every few seconds, until delivery.timeout.ms runs out. With acks=1 the same write goes through - into a partition that now exists on one disk only:

$ echo 'refund 4711' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic payments --request-required-acks 1

The leader logs every refused append in its server.log, at ERROR, with the reason spelled out:

$ ssh kafka-1 "grep -A1 'Error processing append operation' /var/log/kafka/server.log" | tail -2
[2026-09-22 20:00:28,850] ERROR [ReplicaManager broker=1] Error processing append operation on partition F9uOvNBGcBs5FkImJ35bjg:payments-3 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR : 1 is insufficient to satisfy the min.isr requirement of 2 for partition payments-3, live replica(s) broker.id are : Set(1)

On a busy cluster that is thousands of lines a minute; the broker metric UnderMinIsrPartitionCount (lesson 11) is what should page you before anyone greps.

ELR: who may lead when the ISR is gone

If the ISR keeps shrinking down to nothing, who can become leader without losing acknowledged data? Kafka 4 tracks the Eligible Leader Replicas (KIP-966; on by default for clusters created with Kafka 4.1 or later). Because the high watermark cannot advance while the ISR is below min.insync.replicas, a replica that was in the ISR when it dropped below that size still has every committed record. Those replicas go into Elr, and the controller may elect one of them, in this order: someone in the ISR, then someone in the ELR. A broker that comes back after a crash (no clean shutdown) is taken out of the ELR when it registers, because it may have lost data that was not flushed to disk.

The last resort is an unclean election: a replica outside ISR and ELR becomes leader and everything it did not have is gone. unclean.leader.election.enable is false by default; kafka-leader-election.sh --election-type unclean forces one for a partition. It trades data for availability - a decision for the people who own the data, not a reflex at 3 a.m.

Bring both brokers back:

$ lab-vm start kafka-2
kafka-2: powering on (sshd answers in about 15 s)
$ ssh kafka-3 sudo systemctl start kafka
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic payments
Topic: payments	TopicId: F9uOvNBGcBs5FkImJ35bjg	PartitionCount: 6	ReplicationFactor: 3	Configs: min.insync.replicas=2,segment.bytes=1073741824
	Topic: payments	Partition: 0	Leader: 1	Replicas: 3,1,2	Isr: 3,1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 1	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3	Elr: 	LastKnownElr:
	Topic: payments	Partition: 2	Leader: 1	Replicas: 2,3,1	Isr: 2,3,1	Elr: 	LastKnownElr:
	Topic: payments	Partition: 3	Leader: 1	Replicas: 3,1,2	Isr: 3,1,2	Elr: 	LastKnownElr:
	Topic: payments	Partition: 4	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3	Elr: 	LastKnownElr:
	Topic: payments	Partition: 5	Leader: 1	Replicas: 2,3,1	Isr: 2,3,1	Elr: 	LastKnownElr:

The ISRs fill up again as the followers catch up. Notice the leaders: they stayed where the failover put them. Moving leadership back to the preferred replicas is lesson 10.

In an interview: "With RF 3, min.insync.replicas 2 and acks=all you survive one broker down without losing writes or acknowledged data. A second failure makes producers get NOT_ENOUGH_REPLICAS - availability is traded for durability on purpose. acks=1 keeps writing, onto a single copy."

Why it helps

Durability settings decide whether a broker failure loses data or only causes errors. NOT_ENOUGH_REPLICAS is one of the most common Kafka pages, and understanding the ISR, min.insync.replicas, controlled shutdown and ELR lets you see whether the cluster is protecting your data or about to lose it. It also explains why lowering min.insync.replicas is the wrong fix, however much it is tempting during a page.

On the blog

Commands in this lesson

ssh kafka-topics.sh lab-vm echo

FAQ

When does a replica leave the ISR?

When it has not caught up with the leader for replica.lag.time.max.ms (30 seconds), or when its broker is fenced or shuts down. A controlled shutdown removes it at once; after a crash the controller notices when the broker's session times out.

What is the high watermark?

The offset up to which every ISR member has the records. Consumers only see records below it, so they never read something that could still disappear if the leader failed. kafka-get-offsets with the default time shows it as the end of each partition.

What is ELR?

Eligible leader replicas, on by default for new clusters since Kafka 4.1. When the ISR shrinks below min.insync.replicas, removed replicas that are known to have every committed record go into the ELR and may still become leader cleanly. LastKnownElr lists ones that may not.

What is unclean leader election?

Letting a replica that is not in the ISR become leader when no in-sync replica is available. The partition comes back sooner, but records the old leader had acknowledged can be lost. It is off by default and should stay off for data you cannot lose.

Is idempotence on by default?

Yes, for Java producers since Kafka 3.0 (with acks=all). The broker de-duplicates retried batches by producer id and sequence number, so a retry after a timeout does not write a duplicate record into the partition.

In an interview Mid

What do acks=all and min.insync.replicas do together?

acks=all makes the producer wait until every replica in the ISR has the record. min.insync.replicas is the broker-side minimum: if the ISR is smaller, the leader refuses the write with NOT_ENOUGH_REPLICAS. With RF 3 and min.insync.replicas 2, one broker can be down and writes still succeed with two copies; two down and acks=all producers get errors instead of single-copy writes.

Also asked: Why not set min.insync.replicas to 1 during an incident? · What is the difference between a controlled shutdown and a crash? · What does the ELR protect against?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.