Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
How to Reset Kafka Consumer Group Offset

Kafka 0.11.0.0 (Confluent 3.3.0) added support to manipulate offsets for a consumer group via cli kafka-consumer-groups command.

  1. List the topics to which the group is subscribed
kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --describe

Note the values under "CURRENT-OFFSET" and "LOG-END-OFFSET". "CURRENT-OFFSET" is the offset where this consumer group is currently at in each of the partitions.

  1. Reset the consumer offset for a topic (preview)
kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> --reset-offsets --to-earliest

This will print the expected result of the reset, but not actually run it.

  1. Reset the consumer offset for a topic (execute)
kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> --reset-offsets --to-earliest --execute

This will execute the reset and reset the consumer group offset for the specified topic back to 0.

  1. Repeat 1 to check if the reset is successful

Note

  • The consumer group must have no running instance when performing the reset. Otherwise the reset will be rejected.
  • There are many other resetting options, run kafka-consumer-groups for details
    • --shift-by <positive_or_negative_integer>
    • --to-current
    • --to-latest
    • --to-offset <offset_integer>
    • --to-datetime <datetime_string>
    • --by-duration <duration_string>
  • The command also provides an option to reset offsets for all topics the consumer group subscribes to: --all-topics
@daniilyar

This comment has been minimized.

Copy link

commented Feb 22, 2018

For ones who want to do the same for Kafka 0.10 you may download Kafka 1.0 binaries which include the script with this new offset reset capabilities, it will work for Kafka 0.10 as well:

wget http://apache-mirror.rbc.ru/pub/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz
tar -xvf kafka_2.11-1.0.0.tgz
cd kafka_2.11-1.0.0/bin

# Check what offsets is flume-archive currently on:
./kafka-consumer-groups.sh --bootstrap-server <kafka:port> --group <group> --describe

# Reset the offset
./kafka-consumer-groups.sh --bootstrap-server <kafka:port>--group <group>--topic <topic> --reset-offsets --to-earliest --execute

# Check what offsets is flume-archive currently on again to make sure that offset is now resetted properly:
./kafka-consumer-groups.sh --bootstrap-server <kafka:port> --group <group> --describe

# Current offset for all topic partitions should be now reduced. I.e. LOG-END-OFFSET should be bigger than CURRENT-OFFSET
@michalkarolik

This comment has been minimized.

Copy link

commented Mar 21, 2018

thx for instruction :) Do you know why it is so hard to achieve in most libraries? Did you do any research about it? I have checked that in kafka-go, sarama ( both golang) and spring-kafka - there is no easy way to reset offset while using consumer groups. I just have problem with understanding if its dangerous to reset offset or there are some other technical issues.

More description:
https://stackoverflow.com/questions/49360325/kafka-consumer-group-loses-not-commited-messages

@Joginder22

This comment has been minimized.

Copy link

commented Apr 17, 2018

thanks daniilyar. but we're facing below error:

Error: Executing consumer group command failed due to The broker only supports OffsetFetchRequest v1, but we need v2 or newer to request all topic partitions.

Seems like version incompatibility here to execute commands with 1.0 binaries.

@yasha-podeswa-hs

This comment has been minimized.

Copy link

commented Apr 25, 2018

Anyone know if there's a way to do this on Kafka 0.9.0?

@dkurzaj

This comment has been minimized.

Copy link

commented Jul 31, 2018

Apparently my latests and my earliests offsets have the same values, and are equal to LOG-END-OFFSET. I can not manage to go back to the beginning of the messages.

@mduhan

This comment has been minimized.

Copy link

commented Aug 18, 2018

kafka .10 does not support --rest-offsets but this can be achieved using java apis.
Please have a look.

https://gist.github.com/mduhan/0e0a4b08694f50d8a646d2adf02542fc

@manigeeth-narendula

This comment has been minimized.

Copy link

commented Apr 1, 2019

Can anyone tell me how to do this for kafka 1.0.0

@cricket007

This comment has been minimized.

Copy link

commented Apr 22, 2019

The API is all the same since 0.11. This includes 1.0.0, 2.0, and above until anyone mentions otherwise.

@yuranos

This comment has been minimized.

Copy link

commented Jul 15, 2019

Applying 1.0 to 0.10 didn't work for me either. Same as @Joginder22:
Error: Executing consumer group command failed due to The broker only supports OffsetFetchRequest v1, but we need v2 or newer to request all topic partitions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.