The payment service cannot publish. Its log, and the console producer, say the same thing:
$ echo 'refund 4711' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic payments
[2026-09-22 20:00:33,050] WARN [Producer clientId=console-producer] Got error produce response with correlation id 3 on topic-partition payments-3, retrying (2 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-09-22 20:00:33,150] WARN [Producer clientId=console-producer] Got error produce response with correlation id 4 on topic-partition payments-3, retrying (1 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-09-22 20:00:33,250] WARN [Producer clientId=console-producer] Got error produce response with correlation id 5 on topic-partition payments-3, retrying (0 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-09-22 20:00:33,350] ERROR Error when sending message to topic payments with key: null, value: 11 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.NOT_ENOUGH_REPLICAS is Kafka error code 19, and that sentence is its message in the client's Errors table. It is not a client bug and not a network problem. The partition leader received the write and refused it on purpose. (The console producer retries 3 times by default. A Java application retries until delivery.timeout.ms, two minutes by default, and only then fails the send.)
The examples are Apache Kafka 4.3 (KRaft) with three brokers, topics with replication factor 3 and min.insync.replicas=2.
What the leader is checking
Each partition has a leader and followers. The ISR (in-sync replicas) is the leader plus the followers that are caught up. A producer with acks=all (the Java producer's default since Kafka 3.0) is told "written" only when every ISR member has the record.
The hole in that is a small ISR. If it shrinks to the leader alone, "every ISR member has it" means one copy. min.insync.replicas closes the hole: with acks=all, the leader refuses the write while the ISR is smaller than this number. You can see the exact check in the broker's log:
$ ssh kafka-1 "grep -A1 'Error processing append operation' /var/log/kafka/server.log" | tail -2
[2026-09-22 20:00:35,900] ERROR [ReplicaManager broker=1] Error processing append operation on partition F9uOvNBGcBs5FkImJ35bjg:payments-5 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR : 1 is insufficient to satisfy the min.isr requirement of 2 for partition payments-5, live replica(s) broker.id are : Set(1)ISR 1, required 2. So the question is never "why is Kafka refusing?" but "where did two of my three replicas go?"
Step 1: which partitions, which ISR
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic payments --under-min-isr-partitions
Topic: payments Partition: 0 Leader: 1 Replicas: 3,1,2 Isr: 1 Elr: 2 LastKnownElr:
Topic: payments Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1 Elr: 2 LastKnownElr:
Topic: payments Partition: 2 Leader: 1 Replicas: 2,3,1 Isr: 1 Elr: 2 LastKnownElr:
Topic: payments Partition: 3 Leader: 1 Replicas: 3,1,2 Isr: 1 Elr: 2 LastKnownElr:
Topic: payments Partition: 4 Leader: 1 Replicas: 1,2,3 Isr: 1 Elr: 2 LastKnownElr:
Topic: payments Partition: 5 Leader: 1 Replicas: 2,3,1 Isr: 1 Elr: 2 LastKnownElr: Three filters of --describe are worth knowing by heart:
| Filter | Shows partitions where | Meaning |
|---|---|---|
--under-replicated-partitions | ISR < replicas | a copy is missing, writes still work |
--at-min-isr-partitions | ISR = min.insync.replicas | writes still work, one more failure stops them |
--under-min-isr-partitions | ISR < min.insync.replicas | acks=all writes are refused |
Every partition has only broker 1 in its ISR. Leave out --topic and you see it on every topic, __consumer_offsets included, so consumer groups cannot store their committed offsets either. Elr: 2 is Kafka 4's eligible leader replicas (KIP-966, on by default for clusters created with 4.1 or later): broker 2 was in the ISR when it fell below the minimum, so it still has every committed record and may lead again without data loss.
Step 2: which brokers are gone
$ kafka-cluster.sh list-endpoints --bootstrap-server kafka-1:9092 --include-fenced-brokers
ID HOST PORT RACK STATE ENDPOINT_TYPE
1 kafka-1 9092 null unfenced broker
2 kafka-2 9092 null fenced broker
3 kafka-3 9092 null fenced broker Without --include-fenced-brokers you would only see broker 1. Fenced means the controller took the broker out of service: it stopped (a controlled shutdown) or stopped heartbeating (a crash, a power cut, a network split; broker.session.timeout.ms is 9 seconds). Now it is a normal outage: systemctl status kafka and the broker's server.log on each host. In the lab incident this post is built on, one broker was stopped for kernel patching and nobody started it again. The other was in a restart loop after a "capacity review" had cut its heap to 256 MB (java.lang.OutOfMemoryError in its journal; sizing a JVM's heap against its memory limit is its own topic: exit code 137 and the JVM).
Step 3: fix the brokers, not the setting
$ lab-vm start kafka-2
kafka-2: powering on (sshd answers in about 15 s)
$ ssh kafka-3 sudo systemctl start kafka
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic payments
Topic: payments TopicId: F9uOvNBGcBs5FkImJ35bjg PartitionCount: 6 ReplicationFactor: 3 Configs: min.insync.replicas=2,segment.bytes=1073741824
Topic: payments Partition: 0 Leader: 1 Replicas: 3,1,2 Isr: 3,1,2 Elr: LastKnownElr:
Topic: payments Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 Elr: LastKnownElr:
...
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --under-replicated-partitions(lab-vm is the lab's stand-in for your hypervisor or cloud console.) Once the followers have caught up, the ISRs are full again and the producer's retries succeed. Every leader is now on broker 1, because leadership does not move back by itself. A preferred leader election (kafka-leader-election.sh --election-type preferred) spreads it out again. Records whose delivery.timeout.ms ran out during the outage were never written: the producing team has to send them again. If a consumer then has to reprocess a window, reset its offsets to a time.
The two "fixes" that make it worse
Lowering min.insync.replicas to 1. The errors stop at once, and every payment is now acknowledged with a single copy on the one broker that is left. If that disk dies, acknowledged payments are gone. Before you change it, find out where the value comes from:
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --entity-type topics --entity-name payments --all | grep min.insync
min.insync.replicas=2 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:min.insync.replicas=2, STATIC_BROKER_CONFIG:min.insync.replicas=2, DEFAULT_CONFIG:min.insync.replicas=1}The synonyms are listed in the order the broker checks them: a cluster-wide dynamic default, the static server.properties, then Kafka's built-in default of 1. It is a cluster decision, not a per-incident knob.
Switching producers to acks=1. min.insync.replicas only applies to acks=all, so this write goes through:
$ echo 'refund 4711' | kafka-console-producer.sh --bootstrap-server kafka-1:9092 --topic payments --request-required-acks 1No error, and the refund now exists on one disk only. That is the outcome min.insync.replicas exists to prevent.
Choosing the numbers
| Replication factor | min.insync.replicas | Brokers that can be down with writes still working | Copies of every acknowledged write |
|---|---|---|---|
| 3 | 2 | 1 | at least 2 |
| 3 | 3 | 0 (any restart stops writes) | 3 |
| 2 | 2 | 0 | 2 |
| 3 | 1 (the default) | 2 | possibly only 1 |
RF 3, min.insync.replicas=2 and acks=all is the usual production choice. One broker can be down for patching or a crash, and a second failure stops writes rather than accepting data that exists only once. That is what happened here, on purpose.
A related error: NOT_ENOUGH_REPLICAS_AFTER_APPEND (code 20) means "Messages are written to the log, but to fewer in-sync replicas than required". The ISR shrank after the leader appended, so the write may exist but was not acknowledged, and the producer retries it. With idempotence on (the default), the retry does not create a duplicate.
Consumers suffer too: the group coordinator cannot store committed offsets, so the next symptom is often consumer lag that keeps growing.
What should page you
Do not page on producer errors. Page on the broker metric UnderMinIsrPartitionCount above 0, which you can see before anyone writes, and on UnderReplicatedPartitions staying above 0 for more than a few minutes. Most of these incidents start as an under-replicated partition that nobody looked at.
Practise it
In the Kafka Operations chapter, the lesson Durability: acks, the ISR, min.insync.replicas and ELR takes two brokers away from you one at a time. The lab Two brokers down: what acks=all does has you prove the difference between acks=all and acks=1, and the incident payments cannot write is the patching-plus-heap story above: get payments writing again without touching min.insync.replicas.