OnCallReadyStart learning

Runtimes and data Lesson 28.11 · Kafka Operations · 14 min read

Consumer groups, lag and rebalances

In plain words

A consumer group is a team of workers sharing a pile of folders. Each folder (partition) belongs to exactly one worker, and every worker writes down how far they got. If a worker leaves or a new one joins, the team stops and hands out the folders again - that pause is a rebalance. A team with more workers than folders leaves some workers with nothing to do.

A consumer group shares the work

Consumers that use the same group.id form a consumer group. The group splits the partitions of the topics it reads among its members: every partition is read by exactly one member at a time. Six partitions and three consumers: two partitions each. Six partitions and eight consumers: two consumers sit idle. That is why the partition count is the ceiling of a group's parallelism.

Each group has a group coordinator: the broker that leads the group's partition of __consumer_offsets (partition = abs(hash(group.id)) % 50). Members heartbeat to it, it runs the rebalances, and it stores the committed offsets - per partition, the offset of the next record the group will read. The Java consumer commits automatically every 5 s (auto.commit.interval.ms) by default.

kafka-consumer-groups.sh

$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --list
billing
notifier
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
billing         orders          2          864025          864025          0               billing-1-97c37fa3-5722-4635-807c-c5507524004d /10.64.0.2      billing-1
billing         orders          3          864025          864025          0               billing-1-97c37fa3-5722-4635-807c-c5507524004d /10.64.0.2      billing-1
billing         orders          0          864026          864026          0               billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2      billing-0
billing         orders          1          864026          864026          0               billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2      billing-0
billing         orders          4          864025          864025          0               billing-2-5e7a30ab-e98d-416a-8bb4-b25311f62429 /10.64.0.2      billing-2
billing         orders          5          864025          864025          0               billing-2-5e7a30ab-e98d-416a-8bb4-b25311f62429 /10.64.0.2      billing-2

The output starts with an empty line (the tool puts a newline before every row). The rows come member by member (the member with the most partitions first), so partition numbers can appear out of order. The columns:

ColumnMeaning
CURRENT-OFFSETthe group's committed offset for the partition
LOG-END-OFFSETthe partition's high watermark: where the next record will be written
LAGLOG-END-OFFSET - CURRENT-OFFSET: records produced but not yet processed
CONSUMER-IDthe member that owns the partition now (<client.id>-<UUID>), or - when nobody does
HOSTthe member's address, as the coordinator saw the connection
CLIENT-IDthe client.id the application set

A partition with a committed offset but no owner shows - in the last three columns: the group has a position there, but nothing is reading it.

Lag is a trend, not a number. 40 records of lag on a topic that receives 40 records a second is one second: healthy. Lag that keeps growing between two describes a minute apart means the group processes slower than producers write, or not at all. Lag that is huge but shrinking is a group catching up after a restart.

Two other views of the same group:

$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing --members

GROUP           CONSUMER-ID                                    HOST            CLIENT-ID       #PARTITIONS
billing         billing-1-97c37fa3-5722-4635-807c-c5507524004d /10.64.0.2      billing-1       2
billing         billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2      billing-0       2
billing         billing-2-5e7a30ab-e98d-416a-8bb4-b25311f62429 /10.64.0.2      billing-2       2
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing --state

GROUP           COORDINATOR (ID)          ASSIGNMENT-STRATEGY  STATE                #MEMBERS
billing         kafka-2:9092  (2)         range                Stable               3

--members lists the members and how many partitions each has (--verbose adds the assignment); --state shows the coordinator, the assignment strategy and the group state:

StateMeaning
Stablemembers joined, partitions assigned, consuming
PreparingRebalance / CompletingRebalancea rebalance is running: nobody consumes
Emptyno members, but committed offsets are kept (offsets.retention.minutes, 7 days)
Deadno members and no offsets left

A describe during a rebalance warns on stderr (Warning: Consumer group 'billing' is rebalancing.).

Rebalances

Whenever the membership changes - a consumer starts, stops cleanly (it sends a LeaveGroup), crashes (the coordinator notices after session.timeout.ms, 45 s), or is kicked out - the group rebalances and the partitions are assigned again. With the classic protocol, every member stops consuming until the rebalance completes.

The quiet killer is max.poll.interval.ms (default 5 minutes). The consumer gets records in batches of up to max.poll.records (500) and has to call poll() again within max.poll.interval.ms, or the client concludes it is stuck, leaves the group, and logs:

consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer
than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much
time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing
the maximum size of batches returned in poll() with max.poll.records.

Its commit for that batch then fails ("the consumer was kicked out of the group"), so the batch is processed again after the rebalance, takes too long again, and so on: a rebalance storm. The group spends its life in PreparingRebalance, lag grows, and the downstream system sees duplicates. The fix is in the message: smaller batches (max.poll.records) or more time (max.poll.interval.ms), so that max.poll.records x time per record stays well under the interval.

Watch a group go away and come back:

$ sudo systemctl stop billing
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing

Consumer group 'billing' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
billing         orders          0          864040          864047          7               -               -               -
billing         orders          1          864040          864047          7               -               -               -
billing         orders          2          864039          864047          8               -               -               -
billing         orders          3          864039          864047          8               -               -               -
billing         orders          4          864039          864047          8               -               -               -
billing         orders          5          864039          864047          8               -               -               -
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing

Consumer group 'billing' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
billing         orders          0          864040          864254          214             -               -               -
billing         orders          1          864040          864254          214             -               -               -
billing         orders          2          864039          864254          215             -               -               -
billing         orders          3          864039          864254          215             -               -               -
billing         orders          4          864039          864254          215             -               -               -
billing         orders          5          864039          864254          215             -               -               -
$ sudo systemctl start billing
$ journalctl -u billing --no-pager | tail -6
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Successfully joined group with generation Generation{generationId=3, memberId='billing-2-c0a6b4ec-6e4f-46b9-b636-58b55e0bc14c', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Successfully synced group in generation Generation{generationId=3, memberId='billing-2-c0a6b4ec-6e4f-46b9-b636-58b55e0bc14c', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Notifying assignor about the new Assignment(partitions=[orders-4, orders-5]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Adding newly assigned partitions: orders-4, orders-5 (org.apache.kafka.clients.consumer.internals.ConsumerRebalanceListenerInvoker)
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Setting offset for partition orders-4 to the committed offset FetchPosition{offset=864039, offsetEpoch=Optional[3], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3:9092 (id: 3 rack: null isFenced: false)], epoch=3}} (org.apache.kafka.clients.consumer.internals.ConsumerUtils)
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Setting offset for partition orders-5 to the committed offset FetchPosition{offset=864039, offsetEpoch=Optional[3], currentLeader=LeaderAndEpoch{leader=Optional[kafka-1:9092 (id: 1 rack: null isFenced: false)], epoch=3}} (org.apache.kafka.clients.consumer.internals.ConsumerUtils)

Stopped cleanly, the members left at once (state Empty, no owners, lag growing by about 40 a second); started again, they rejoined, got their partitions and resumed from the committed offsets - the lines "Successfully joined group", "Adding newly assigned partitions" and "Setting offset for partition ... to the committed offset" in the application's log are the Java client's own.

The new consumer group protocol (KIP-848)

Kafka 4.0 made the consumer rebalance protocol production-ready (group.protocol=consumer in the client): the coordinator computes the assignment itself and moves partitions one at a time, so members that keep their partitions never stop consuming. Clients still default to group.protocol=classic in 4.3, which is what this lab's applications use; kafka-consumer-groups.sh --list --type shows each group's type (Classic or Consumer).

In an interview: "Lag is log-end offset minus the committed offset, per partition; I watch whether it grows. A consumer that takes longer than max.poll.interval.ms between polls is kicked out, its commit fails and the group rebalances again and again - fix it with fewer records per poll or a longer interval."

Why it helps

Consumer lag is the most common Kafka symptom that reaches the platform team. kafka-consumer-groups --describe tells you whether one partition is stuck or the whole group is too slow, and whether the group keeps rebalancing. Understanding max.poll.interval.ms and session timeouts explains the rebalance storms that make lag grow while consumers look busy, a pattern that confuses many teams and wastes hours of their time.

On the blog

Commands in this lesson

kafka-consumer-groups.sh systemctl journalctl

FAQ

What is the group coordinator?

The broker that leads the __consumer_offsets partition the group's id hashes to. It tracks the members, runs rebalances and stores the committed offsets. --describe --state shows it in the COORDINATOR (ID) column, with its host and broker id.

What does the group state mean?

Stable means members are consuming. PreparingRebalance and CompletingRebalance mean a rebalance is running and nobody consumes. Empty means no members but committed offsets remain. Dead means the group has no members and no offsets left.

What is the difference between session.timeout.ms and max.poll.interval.ms?

The session timeout (45 seconds by default) is about heartbeats: a crashed consumer is removed after it. max.poll.interval.ms (5 minutes) is about progress: a consumer that does not call poll() again within it is removed even though it still sends heartbeats.

What is the new consumer group protocol?

KIP-848, production-ready since Kafka 4.0: the broker computes assignments and rebalances incrementally without stopping the whole group. Clients opt in with group.protocol=consumer; the console consumer prints a notice about it when it uses the classic protocol.

How should I alert on lag?

On lag that keeps growing, or on lag measured in time (how many minutes behind), not on a fixed record count. A busy topic can have thousands of records of lag that are only seconds of work.

In an interview Mid

A consumer group's lag keeps growing. How do you find out why?

kafka-consumer-groups --describe shows CURRENT-OFFSET, LOG-END-OFFSET and LAG per partition. Growing on every partition means the group is too slow: add consumers up to the partition count or speed them up. Growing on one partition means a hot key or a stuck consumer. --state shows rebalancing; a group that keeps rebalancing usually has max.poll.records times processing time above max.poll.interval.ms.

Also asked: Why can a group not use more consumers than partitions? · What triggers a rebalance? · What does the group coordinator do?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.