OnCallReadyStart learning

Kafka · 6 min read

Reset Kafka consumer offsets to a date and time, safely

kafka-consumer-groups --reset-offsets --to-datetime: stop the group, mind the milliseconds and UTC, dry-run, export a backup, then execute, and undo it.

A bug in the notifier sent broken e-mails for the last hour. It is fixed now, and you want the group to process everything since 19:00 again. Or the opposite: someone reset the group by mistake, customers are getting yesterday's e-mails, and you need to put it back to where it was. Both are kafka-consumer-groups.sh --reset-offsets, the Kafka command that does the most damage when used wrong, because it does exactly what it is told.

The examples are Apache Kafka 4.3. The group is notifier, reading the 3-partition topic notifications.

Rule 1: the group must be empty

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --group notifier --reset-offsets --topic notifications --to-datetime 2026-09-22T20:00:00.000 --execute

Error: Assignments can only be reset if the group 'notifier' is inactive, but the current state is Stable.

GROUP           TOPIC           PARTITION  NEW-OFFSET

While members are connected, their own commits would overwrite yours, so the tool refuses. Stop every instance of the consumer, on every host and in every pod, then check:

terminal
$ sudo systemctl stop notifier
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group notifier --state

Consumer group 'notifier' has no active members.

GROUP           COORDINATOR (ID)          ASSIGNMENT-STRATEGY  STATE                #MEMBERS
notifier        kafka-3:9092  (3)         -                    Empty                0

Empty is what you want. On Kubernetes that means scaling the Deployment to 0 and waiting for the pods to be gone, not just restarting it.

Rule 2: save where the group is now

Before you move anything, write the current offsets to a file. --to-current --export prints them as CSV, and --from-file can put them back later:

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --group notifier --reset-offsets --topic notifications --to-current --export --dry-run > /tmp/notifier-before.csv
$ cat /tmp/notifier-before.csv
notifications,0,518400
notifications,1,518400
notifications,2,518400

One line per partition: topic,partition,offset. With several groups in one export, the group name comes first. This file is your undo button.

Rule 3: the datetime format has two traps

The option's own help gives the format: YYYY-MM-DDThh:mm:ss.sss. The milliseconds are not optional:

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --group notifier --reset-offsets --topic notifications --to-datetime 2026-09-22T19:00:00 --dry-run

Error: Executing consumer group command failed due to java.text.ParseException: Unparseable date: "2026-09-22T19:00:00Z"
java.lang.RuntimeException: java.text.ParseException: Unparseable date: "2026-09-22T19:00:00Z"
	at org.apache.kafka.tools.OffsetsUtils.resetToDateTime(OffsetsUtils.java:313)
...

Notice the Z in the message. You did not type one. Kafka's Utils.getDateTime adds Z (UTC) when the time has no zone, then parses strictly with yyyy-MM-dd'T'HH:mm:ss.SSSXXX, which needs exactly three digits of milliseconds. That gives you the second trap: a time without a zone is UTC, not your local time. With the milliseconds:

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --group notifier --reset-offsets --topic notifications --to-datetime 2026-09-22T19:00:00.000 --dry-run

GROUP           TOPIC           PARTITION  NEW-OFFSET
notifier        notifications   0          503988
notifier        notifications   1          503988
notifier        notifications   2          503988
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --group notifier --reset-offsets --topic notifications --to-datetime 2026-09-22T21:00:00.000+02:00 --dry-run

GROUP           TOPIC           PARTITION  NEW-OFFSET
notifier        notifications   0          503988
notifier        notifications   1          503988
notifier        notifications   2          503988

21:00 at +02:00 is 19:00 UTC, so the offsets are the same. If you think in Amsterdam time, write the offset explicitly. For each partition, the new offset is the first record whose timestamp is at or after that moment. A time before the oldest record gives the oldest offset, and a time in the future gives the end. (The timestamp is the record's own: the producer's send time by default, or the broker's append time on topics with message.timestamp.type=LogAppendTime.)

"The last hour" without doing the arithmetic is --by-duration, an ISO-8601 duration counted back from now:

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --group notifier --reset-offsets --topic notifications --by-duration PT1H --dry-run

GROUP           TOPIC           PARTITION  NEW-OFFSET
notifier        notifications   0          504112
notifier        notifications   1          504112
notifier        notifications   2          504112

Rule 4: nothing happens without --execute

Every command above was a dry run. Without either flag, Kafka 4.3 also does a dry run, and says so:

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --group notifier --reset-offsets --topic notifications --to-datetime 2026-09-22T20:00:05.000
WARN: No action will be performed as the --execute option is missing. In version 5.0, this command will require either --dry-run or --execute to be specified. You should add the --dry-run option explicitly if you are scripting this command and want to keep the current default behavior without prompting.

GROUP           TOPIC           PARTITION  NEW-OFFSET
notifier        notifications   0          518400
notifier        notifications   1          518400
notifier        notifications   2          518400

Write --dry-run explicitly in scripts and runbooks. When the plan looks right:

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --group notifier --reset-offsets --topic notifications --to-datetime 2026-09-22T19:00:00.000 --execute

GROUP           TOPIC           PARTITION  NEW-OFFSET
notifier        notifications   0          503988
notifier        notifications   1          503988
notifier        notifications   2          503988
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group notifier

Consumer group 'notifier' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
notifier        notifications   0          503988          518400          14412           -               -               -
notifier        notifications   1          503988          518400          14412           -               -               -
notifier        notifications   2          503988          518400          14412           -               -               -

43,236 records will be processed again when you start the consumer. Each --execute only moves the offsets you scoped: --topic notifications is every partition of the topic, --topic notifications:0,2 two of them, --all-topics every topic the group has offsets for.

Changed your mind? The undo file:

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --group notifier --reset-offsets --from-file /tmp/notifier-before.csv --execute

GROUP           TOPIC           PARTITION  NEW-OFFSET
notifier        notifications   0          518400
notifier        notifications   1          518400
notifier        notifications   2          518400
$ sudo systemctl start notifier

Undoing someone else's reset

Customers are getting old e-mails again. The group is at offset 652 of 518,400: it was moved back to the start of the log and has been re-sending from there ever since:

terminal
$ kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group notifier

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
notifier        notifications   0          652             518400          517748          notifier-0-c5de7624-2ee3-4142-bc12-730df028e732 /10.64.0.2      notifier-0
notifier        notifications   1          652             518400          517748          notifier-0-c5de7624-2ee3-4142-bc12-730df028e732 /10.64.0.2      notifier-0
notifier        notifications   2          1272            518400          517128          notifier-1-7401d0d3-f651-4777-9961-25020deb21cf /10.64.0.2      notifier-1

A reset needs an empty group, so whoever did it had to stop the consumer first. The stop and the start bracket the moment, and sudo logs who did it:

terminal
$ journalctl -t sudo --no-pager | tail -4
Sep 22 20:00:12 oncall-lab sudo[17390]:   deploy : TTY=pts/3 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/bin/systemctl stop notifier
Sep 22 20:00:16 oncall-lab sudo[17391]:   deploy : TTY=pts/3 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/bin/systemctl start notifier

So stop the flood first (sudo systemctl stop notifier), then reset to the moment the consumer was stopped, 20:00:12 UTC. Everything older was already sent, and everything newer still has to go out. Dry-run, check, --execute, start it again, and tell support when the duplicates stopped (a status update template helps). On a real cluster you would also give each application's principal ACLs on its own group only, so that this command cannot be run against it from just anywhere.

Before you replay: what duplicates do downstream

If the lag you want to fix comes from a slow or stuck group, a reset is the wrong tool: find out why the lag grows first. Kafka has no exactly-once delivery to the outside world. Going back an hour means an hour of e-mails, invoices or webhooks again, unless the consumer is idempotent (it remembers what it already did). Ask the consumer's owners before you replay. (The same question comes up after a broker outage, for example one where producers failed with NOT_ENOUGH_REPLICAS and records had to be sent again.) Skipping forward (--to-latest) has the opposite risk: records nobody will ever process.

Committed offsets also expire. A group that stays Empty for longer than offsets.retention.minutes (7 days by default) loses them, and when it starts again auto.offset.reset decides where it begins. The default is latest, which silently skips everything in between.

The checklist

  1. Stop every consumer in the group. --describe --state must say Empty.
  2. --to-current --export --dry-run > before.csv: the undo file.
  3. Dry-run the reset. Use milliseconds, and add a zone (Z or +02:00) so you know which clock you mean.
  4. Read the NEW-OFFSET column, then --execute.
  5. --describe (the lag is what will be reprocessed), then start the consumer.

Practise it

In the Kafka Operations chapter, the lesson Resetting offsets safely covers every reset option, the lab Replay two hours of orders has you replay a window without double-billing, the drill Move the offsets, exactly gives you random targets to hit, and the incident customers get yesterday's e-mails again is the accidental reset above.

OnCallReady is free, with no ads and no tracking. RSS · All posts