09:12, page: billing consumer lag above 10 000 and rising. Invoices are hours late. The billing team says nothing changed in the code, only some "throughput tuning" of its Kafka settings yesterday.
The examples are Apache Kafka 4.3 with the Java client's default (classic) group protocol, the same tools you have in /opt/kafka/bin.
Step 1: read the group
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing
Warning: Consumer group 'billing' is rebalancing.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
billing orders 0 861000 864249 3249 billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2 billing-0
billing orders 1 861000 864249 3249 billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2 billing-0
billing orders 2 861000 864249 3249 billing-1-97c37fa3-5722-4635-807c-c5507524004d /10.64.0.2 billing-1
billing orders 3 861000 864249 3249 billing-1-97c37fa3-5722-4635-807c-c5507524004d /10.64.0.2 billing-1
billing orders 4 861000 864248 3248 billing-2-5e7a30ab-e98d-416a-8bb4-b25311f62429 /10.64.0.2 billing-2
billing orders 5 861000 864248 3248 billing-2-5e7a30ab-e98d-416a-8bb4-b25311f62429 /10.64.0.2 billing-2Per partition:
| Column | Meaning |
|---|---|
| CURRENT-OFFSET | the group's committed offset: the next record it will read after a restart |
| LOG-END-OFFSET | the end of the partition: where the next record will be written |
| LAG | LOG-END-OFFSET minus CURRENT-OFFSET: records written but not yet processed (and committed) |
| CONSUMER-ID, HOST, CLIENT-ID | the member that owns the partition now, or - when nobody does |
A single describe tells you very little. 3,249 records on each of six partitions is about 19,500 orders. At the ~40 orders a second this topic receives, that is about 8 minutes behind. Whether that is a problem depends on whether it is growing. So run it again a minute later:
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing
Warning: Consumer group 'billing' is rebalancing.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
billing orders 0 861000 864662 3662 billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2 billing-0
...LOG-END-OFFSET moved by 413 per partition. CURRENT-OFFSET did not move at all. That is not slow processing, because slow processing still commits something. Nothing at all is being committed.
Step 2: what the lag pattern tells you
| What you see over a few minutes | Most likely |
|---|---|
CONSUMER-ID - everywhere, state Empty | nobody is consuming: the app is down, or runs with another group.id |
| offsets move, but slower than LOG-END-OFFSET | the group is too slow: more consumers (up to the partition count) or faster processing |
| one partition lags, the rest are fine | a hot key, or one stuck member |
| offsets frozen, members present, "is rebalancing" | the group never gets to commit: a rebalance storm, or a record that crashes the consumer every time (on Kubernetes: a CrashLoopBackOff) |
| huge lag that is shrinking | a group catching up after a restart: leave it alone |
The "is rebalancing" warning on both describes is the clue. Check the group's state a few times:
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing --state
Warning: Consumer group 'billing' is rebalancing.
GROUP COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS
billing kafka-2:9092 (2) range PreparingRebalance 3PreparingRebalance again and again. With the classic protocol, nobody in the group consumes while a rebalance runs, and members whose partitions are revoked lose the work they had not committed.
Step 3: the consumer's own log says why
$ journalctl -u billing -o cat --no-pager | grep -m1 "poll timeout"
[2026-09-22 20:00:40,520] WARN [Consumer clientId=billing-1, groupId=billing] consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
$ journalctl -u billing --no-pager | grep -m1 "auto-commit"
Sep 22 20:00:40 oncall-lab java[17387]: [2026-09-22 20:00:40,520] WARN [Consumer clientId=billing-1, groupId=billing] Synchronous auto-commit of offsets {orders-2=OffsetAndMetadata{offset=863000, leaderEpoch=3, metadata=''}, orders-3=OffsetAndMetadata{offset=863000, leaderEpoch=3, metadata=''}} failed: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)That is the Java client's own wording (the AbstractCoordinator and ConsumerCoordinator classes). What happens, in order:
poll()returns up tomax.poll.recordsrecords. The app processes them, then callspoll()again.- If that takes longer than
max.poll.interval.ms, the client decides the app is stuck and leaves the group ("sending LeaveGroup request ... due to consumer poll timeout has expired"). - The commit for the batch it just processed fails: the consumer was kicked out.
- The group rebalances. Another member gets those partitions and starts again from the last committed offset, with the same batch, which takes too long again.
Everyone is busy all the time and the committed offsets never move. Downstream, the same records are processed again and again (here, the same invoices).
Step 4: the arithmetic
$ grep max.poll /etc/oncall/billing.properties
max.poll.records=2000
max.poll.interval.ms=30000Each order takes billing about 20 ms (it calls a tax service). 2000 records x 20 ms = 40 s per batch, against a limit of 30 s. No batch could ever be committed. The client defaults are max.poll.records=500 and max.poll.interval.ms=300000 (5 minutes). The "tuning" raised the first and cut the second.
The rule: max.poll.records x worst-case time per record must stay well under max.poll.interval.ms, with a margin, because the tax service will have a slow day. Here 500 x 20 ms = 10 s, against 30 s:
$ sudo sed -i 's/^max.poll.records=2000/max.poll.records=500/' /etc/oncall/billing.properties
$ sudo systemctl restart billing
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing --state
GROUP COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS
billing kafka-2:9092 (2) range Stable 3
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
billing orders 0 862692 865212 2520 billing-0-2aa013bf-b7d1-4026-844a-b6a9d07881a5 /10.64.0.2 billing-0
...Stable, and CURRENT-OFFSET moves. A few minutes later every partition shows LAG 0.
What not to do
- Do not reset the group to the latest offset to make the lag go away. The graph turns green and those orders are never invoiced.
--reset-offsets --to-latestskips data. It does not process it. (When you do need to move a group, do it safely.) - Do not just add consumers. More members than partitions sit idle, and in a storm every extra member means more rebalances.
- Do not raise
max.poll.interval.msto an hour and call it fixed. That is also how long a consumer that is really stuck keeps its partitions before anyone notices.
If the producers are the ones failing, it is a different incident: NOT_ENOUGH_REPLICAS.
Watch the trend, not the number
Alert on lag that grows over several minutes, or on lag converted to time (lag divided by the produce rate), not on a fixed record count. On the client side the consumer reports records-lag-max (in consumer-fetch-manager-metrics). A rebalance storm shows up as a rebalance rate far above zero and committed offsets that do not move.
Kafka 4.0 also made the new consumer rebalance protocol (KIP-848, group.protocol=consumer) generally available. The broker computes the assignment and moves partitions one at a time, so members that keep their partitions do not stop. The Java client still defaults to group.protocol=classic in 4.3, so most groups you meet are still classic. The poll-interval rule applies to both.
Practise it
In the Kafka Operations chapter, the lesson Consumer groups, lag and rebalances reads these columns with you, the lab billing is falling behind and the drill Where is the lag? have you tell the patterns apart, and the incident billing never catches up is this exact rebalance storm. You have to fix it without skipping a single order.