A consumer group shares the work
Consumers that use the same group.id form a consumer group. The group splits the partitions of the topics it reads among its members: every partition is read by exactly one member at a time. Six partitions and three consumers: two partitions each. Six partitions and eight consumers: two consumers sit idle. That is why the partition count is the ceiling of a group's parallelism.
Each group has a group coordinator: the broker that leads the group's partition of __consumer_offsets (partition = abs(hash(group.id)) % 50). Members heartbeat to it, it runs the rebalances, and it stores the committed offsets - per partition, the offset of the next record the group will read. The Java consumer commits automatically every 5 s (auto.commit.interval.ms) by default.
kafka-consumer-groups.sh
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --list
billing
notifier
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
billing orders 2 864025 864025 0 billing-1-97c37fa3-5722-4635-807c-c5507524004d /10.64.0.2 billing-1
billing orders 3 864025 864025 0 billing-1-97c37fa3-5722-4635-807c-c5507524004d /10.64.0.2 billing-1
billing orders 0 864026 864026 0 billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2 billing-0
billing orders 1 864026 864026 0 billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2 billing-0
billing orders 4 864025 864025 0 billing-2-5e7a30ab-e98d-416a-8bb4-b25311f62429 /10.64.0.2 billing-2
billing orders 5 864025 864025 0 billing-2-5e7a30ab-e98d-416a-8bb4-b25311f62429 /10.64.0.2 billing-2
The output starts with an empty line (the tool puts a newline before every row). The rows come member by member (the member with the most partitions first), so partition numbers can appear out of order. The columns:
| Column | Meaning |
|---|---|
| CURRENT-OFFSET | the group's committed offset for the partition |
| LOG-END-OFFSET | the partition's high watermark: where the next record will be written |
| LAG | LOG-END-OFFSET - CURRENT-OFFSET: records produced but not yet processed |
| CONSUMER-ID | the member that owns the partition now (<client.id>-<UUID>), or - when nobody does |
| HOST | the member's address, as the coordinator saw the connection |
| CLIENT-ID | the client.id the application set |
A partition with a committed offset but no owner shows - in the last three columns: the group has a position there, but nothing is reading it.
Lag is a trend, not a number. 40 records of lag on a topic that receives 40 records a second is one second: healthy. Lag that keeps growing between two describes a minute apart means the group processes slower than producers write, or not at all. Lag that is huge but shrinking is a group catching up after a restart.
Two other views of the same group:
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing --members
GROUP CONSUMER-ID HOST CLIENT-ID #PARTITIONS
billing billing-1-97c37fa3-5722-4635-807c-c5507524004d /10.64.0.2 billing-1 2
billing billing-0-dfd9e64d-dcf6-42d6-a7ae-6d0aa37c98a0 /10.64.0.2 billing-0 2
billing billing-2-5e7a30ab-e98d-416a-8bb4-b25311f62429 /10.64.0.2 billing-2 2
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing --state
GROUP COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS
billing kafka-2:9092 (2) range Stable 3
--members lists the members and how many partitions each has (--verbose adds the assignment); --state shows the coordinator, the assignment strategy and the group state:
| State | Meaning |
|---|---|
| Stable | members joined, partitions assigned, consuming |
| PreparingRebalance / CompletingRebalance | a rebalance is running: nobody consumes |
| Empty | no members, but committed offsets are kept (offsets.retention.minutes, 7 days) |
| Dead | no members and no offsets left |
A describe during a rebalance warns on stderr (Warning: Consumer group 'billing' is rebalancing.).
Rebalances
Whenever the membership changes - a consumer starts, stops cleanly (it sends a LeaveGroup), crashes (the coordinator notices after session.timeout.ms, 45 s), or is kicked out - the group rebalances and the partitions are assigned again. With the classic protocol, every member stops consuming until the rebalance completes.
The quiet killer is max.poll.interval.ms (default 5 minutes). The consumer gets records in batches of up to max.poll.records (500) and has to call poll() again within max.poll.interval.ms, or the client concludes it is stuck, leaves the group, and logs:
consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer
than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much
time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing
the maximum size of batches returned in poll() with max.poll.records.
Its commit for that batch then fails ("the consumer was kicked out of the group"), so the batch is processed again after the rebalance, takes too long again, and so on: a rebalance storm. The group spends its life in PreparingRebalance, lag grows, and the downstream system sees duplicates. The fix is in the message: smaller batches (max.poll.records) or more time (max.poll.interval.ms), so that max.poll.records x time per record stays well under the interval.
Watch a group go away and come back:
$ sudo systemctl stop billing
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing
Consumer group 'billing' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
billing orders 0 864040 864047 7 - - -
billing orders 1 864040 864047 7 - - -
billing orders 2 864039 864047 8 - - -
billing orders 3 864039 864047 8 - - -
billing orders 4 864039 864047 8 - - -
billing orders 5 864039 864047 8 - - -
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group billing
Consumer group 'billing' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
billing orders 0 864040 864254 214 - - -
billing orders 1 864040 864254 214 - - -
billing orders 2 864039 864254 215 - - -
billing orders 3 864039 864254 215 - - -
billing orders 4 864039 864254 215 - - -
billing orders 5 864039 864254 215 - - -
$ sudo systemctl start billing
$ journalctl -u billing --no-pager | tail -6
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Successfully joined group with generation Generation{generationId=3, memberId='billing-2-c0a6b4ec-6e4f-46b9-b636-58b55e0bc14c', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Successfully synced group in generation Generation{generationId=3, memberId='billing-2-c0a6b4ec-6e4f-46b9-b636-58b55e0bc14c', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Notifying assignor about the new Assignment(partitions=[orders-4, orders-5]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Adding newly assigned partitions: orders-4, orders-5 (org.apache.kafka.clients.consumer.internals.ConsumerRebalanceListenerInvoker)
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Setting offset for partition orders-4 to the committed offset FetchPosition{offset=864039, offsetEpoch=Optional[3], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3:9092 (id: 3 rack: null isFenced: false)], epoch=3}} (org.apache.kafka.clients.consumer.internals.ConsumerUtils)
Sep 22 20:00:45 oncall-lab java[17395]: [2026-09-22 20:00:45,900] INFO [Consumer clientId=billing-2, groupId=billing] Setting offset for partition orders-5 to the committed offset FetchPosition{offset=864039, offsetEpoch=Optional[3], currentLeader=LeaderAndEpoch{leader=Optional[kafka-1:9092 (id: 1 rack: null isFenced: false)], epoch=3}} (org.apache.kafka.clients.consumer.internals.ConsumerUtils)
Stopped cleanly, the members left at once (state Empty, no owners, lag growing by about 40 a second); started again, they rejoined, got their partitions and resumed from the committed offsets - the lines "Successfully joined group", "Adding newly assigned partitions" and "Setting offset for partition ... to the committed offset" in the application's log are the Java client's own.
The new consumer group protocol (KIP-848)
Kafka 4.0 made the consumer rebalance protocol production-ready (group.protocol=consumer in the client): the coordinator computes the assignment itself and moves partitions one at a time, so members that keep their partitions never stop consuming. Clients still default to group.protocol=classic in 4.3, which is what this lab's applications use; kafka-consumer-groups.sh --list --type shows each group's type (Classic or Consumer).
In an interview: "Lag is log-end offset minus the committed offset, per partition; I watch whether it grows. A consumer that takes longer than max.poll.interval.ms between polls is kicked out, its commit fails and the group rebalances again and again - fix it with fewer records per poll or a longer interval."