OnCallReadyStart learning

Kafka · 5 min read

Kafka consumer lag keeps growing: read the group, then find out why

Lag that only grows means the group is not committing. Read kafka-consumer-groups --describe and --state, spot a rebalance storm, and fix max.poll.records.

09:12, page: billing consumer lag above 10 000 and rising. Invoices are hours late. The billing team says nothing changed in the code, only some "throughput tuning" of its Kafka settings yesterday.

The examples are Apache Kafka 4.3 with the Java client's default (classic) group protocol, the same tools you have in /opt/kafka/bin.

Step 1: read the group

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing

Warning: Consumer group 'billing' is rebalancing.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
billing         orders          0          861000          864249          3249            billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2      billing-0
billing         orders          1          861000          864249          3249            billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2      billing-0
billing         orders          2          861000          864249          3249            billing-1-97c37fa3-5722-4635-807c-c5507524004d /10.64.0.2      billing-1
billing         orders          3          861000          864249          3249            billing-1-97c37fa3-5722-4635-807c-c5507524004d /10.64.0.2      billing-1
billing         orders          4          861000          864248          3248            billing-2-5e7a30ab-e98d-416a-8bb4-b25311f62429 /10.64.0.2      billing-2
billing         orders          5          861000          864248          3248            billing-2-5e7a30ab-e98d-416a-8bb4-b25311f62429 /10.64.0.2      billing-2

Per partition:

ColumnMeaning
CURRENT-OFFSETthe group's committed offset: the next record it will read after a restart
LOG-END-OFFSETthe end of the partition: where the next record will be written
LAGLOG-END-OFFSET minus CURRENT-OFFSET: records written but not yet processed (and committed)
CONSUMER-ID, HOST, CLIENT-IDthe member that owns the partition now, or - when nobody does

A single describe tells you very little. 3,249 records on each of six partitions is about 19,500 orders. At the ~40 orders a second this topic receives, that is about 8 minutes behind. Whether that is a problem depends on whether it is growing. So run it again a minute later:

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing

Warning: Consumer group 'billing' is rebalancing.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
billing         orders          0          861000          864662          3662            billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2      billing-0
...

LOG-END-OFFSET moved by 413 per partition. CURRENT-OFFSET did not move at all. That is not slow processing, because slow processing still commits something. Nothing at all is being committed.

Step 2: what the lag pattern tells you

What you see over a few minutesMost likely
CONSUMER-ID - everywhere, state Emptynobody is consuming: the app is down, or runs with another group.id
offsets move, but slower than LOG-END-OFFSETthe group is too slow: more consumers (up to the partition count) or faster processing
one partition lags, the rest are finea hot key, or one stuck member
offsets frozen, members present, "is rebalancing"the group never gets to commit: a rebalance storm, or a record that crashes the consumer every time (on Kubernetes: a CrashLoopBackOff)
huge lag that is shrinkinga group catching up after a restart: leave it alone

The "is rebalancing" warning on both describes is the clue. Check the group's state a few times:

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing --state

Warning: Consumer group 'billing' is rebalancing.

GROUP           COORDINATOR (ID)          ASSIGNMENT-STRATEGY  STATE                #MEMBERS
billing         kafka-2:9092  (2)         range                PreparingRebalance   3

PreparingRebalance again and again. With the classic protocol, nobody in the group consumes while a rebalance runs, and members whose partitions are revoked lose the work they had not committed.

Step 3: the consumer's own log says why

terminal
$ journalctl -u billing -o cat --no-pager | grep -m1 "poll timeout"
[2026-09-22 20:00:40,520] WARN [Consumer clientId=billing-1, groupId=billing] consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
$ journalctl -u billing --no-pager | grep -m1 "auto-commit"
Sep 22 20:00:40 oncall-lab java[17387]: [2026-09-22 20:00:40,520] WARN [Consumer clientId=billing-1, groupId=billing] Synchronous auto-commit of offsets {orders-2=OffsetAndMetadata{offset=863000, leaderEpoch=3, metadata=''}, orders-3=OffsetAndMetadata{offset=863000, leaderEpoch=3, metadata=''}} failed: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

That is the Java client's own wording (the AbstractCoordinator and ConsumerCoordinator classes). What happens, in order:

  1. poll() returns up to max.poll.records records. The app processes them, then calls poll() again.
  2. If that takes longer than max.poll.interval.ms, the client decides the app is stuck and leaves the group ("sending LeaveGroup request ... due to consumer poll timeout has expired").
  3. The commit for the batch it just processed fails: the consumer was kicked out.
  4. The group rebalances. Another member gets those partitions and starts again from the last committed offset, with the same batch, which takes too long again.

Everyone is busy all the time and the committed offsets never move. Downstream, the same records are processed again and again (here, the same invoices).

Step 4: the arithmetic

terminal
$ grep max.poll /etc/oncall/billing.properties
max.poll.records=2000
max.poll.interval.ms=30000

Each order takes billing about 20 ms (it calls a tax service). 2000 records x 20 ms = 40 s per batch, against a limit of 30 s. No batch could ever be committed. The client defaults are max.poll.records=500 and max.poll.interval.ms=300000 (5 minutes). The "tuning" raised the first and cut the second.

The rule: max.poll.records x worst-case time per record must stay well under max.poll.interval.ms, with a margin, because the tax service will have a slow day. Here 500 x 20 ms = 10 s, against 30 s:

terminal
$ sudo sed -i 's/^max.poll.records=2000/max.poll.records=500/' /etc/oncall/billing.properties
$ sudo systemctl restart billing
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing --state

GROUP           COORDINATOR (ID)          ASSIGNMENT-STRATEGY  STATE                #MEMBERS
billing         kafka-2:9092  (2)         range                Stable               3
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
billing         orders          0          862692          865212          2520            billing-0-2aa013bf-b7d1-4026-844a-b6a9d07881a5 /10.64.0.2      billing-0
...

Stable, and CURRENT-OFFSET moves. A few minutes later every partition shows LAG 0.

What not to do

  • Do not reset the group to the latest offset to make the lag go away. The graph turns green and those orders are never invoiced. --reset-offsets --to-latest skips data. It does not process it. (When you do need to move a group, do it safely.)
  • Do not just add consumers. More members than partitions sit idle, and in a storm every extra member means more rebalances.
  • Do not raise max.poll.interval.ms to an hour and call it fixed. That is also how long a consumer that is really stuck keeps its partitions before anyone notices.

If the producers are the ones failing, it is a different incident: NOT_ENOUGH_REPLICAS.

Watch the trend, not the number

Alert on lag that grows over several minutes, or on lag converted to time (lag divided by the produce rate), not on a fixed record count. On the client side the consumer reports records-lag-max (in consumer-fetch-manager-metrics). A rebalance storm shows up as a rebalance rate far above zero and committed offsets that do not move.

Kafka 4.0 also made the new consumer rebalance protocol (KIP-848, group.protocol=consumer) generally available. The broker computes the assignment and moves partitions one at a time, so members that keep their partitions do not stop. The Java client still defaults to group.protocol=classic in 4.3, so most groups you meet are still classic. The poll-interval rule applies to both.

Practise it

In the Kafka Operations chapter, the lesson Consumer groups, lag and rebalances reads these columns with you, the lab billing is falling behind and the drill Where is the lag? have you tell the patterns apart, and the incident billing never catches up is this exact rebalance storm. You have to fix it without skipping a single order.

OnCallReady is free, with no ads and no tracking. RSS · All posts