Skip to content

Instantly share code, notes, and snippets.

@marwei
Created November 9, 2017 23:39
Star You must be signed in to star a gist
Save marwei/cd40657c481f94ebe273ecc16601674b to your computer and use it in GitHub Desktop.
How to Reset Kafka Consumer Group Offset

Kafka 0.11.0.0 (Confluent 3.3.0) added support to manipulate offsets for a consumer group via cli kafka-consumer-groups command.

  1. List the topics to which the group is subscribed
kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --describe

Note the values under "CURRENT-OFFSET" and "LOG-END-OFFSET". "CURRENT-OFFSET" is the offset where this consumer group is currently at in each of the partitions.

  1. Reset the consumer offset for a topic (preview)
kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> --reset-offsets --to-earliest

This will print the expected result of the reset, but not actually run it.

  1. Reset the consumer offset for a topic (execute)
kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> --reset-offsets --to-earliest --execute

This will execute the reset and reset the consumer group offset for the specified topic back to 0.

  1. Repeat 1 to check if the reset is successful

Note

  • The consumer group must have no running instance when performing the reset. Otherwise the reset will be rejected.
  • There are many other resetting options, run kafka-consumer-groups for details
    • --shift-by <positive_or_negative_integer>
    • --to-current
    • --to-latest
    • --to-offset <offset_integer>
    • --to-datetime <datetime_string>
    • --by-duration <duration_string>
  • The command also provides an option to reset offsets for all topics the consumer group subscribes to: --all-topics
@zonedoutfella
Copy link

Apparently my latests and my earliests offsets have the same values, and are equal to LOG-END-OFFSET. I can not manage to go back to the beginning of the messages.

@dkurzaj that is because your topic doesn't contain messages, and the offset can only be reset to the first message present in the topic at max

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment