Skip to content

Instantly share code, notes, and snippets.

@AnanthaRajuC

AnanthaRajuC/Delete message after consuming it in KAFKA.md

Forked from slgithub/Delete message after consuming it in KAFKA.md
Created August 23, 2022 06:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AnanthaRajuC/0318e2314707733dbeb891e3509f713e to your computer and use it in GitHub Desktop.
Save AnanthaRajuC/0318e2314707733dbeb891e3509f713e to your computer and use it in GitHub Desktop.
Download ZIP
Delete message after consuming it in KAFKA
Raw
Delete message after consuming it in KAFKA.md

Delete message after consuming it in KAFKA

In Kafka, the responsibility of what has been consumed is the responsibility of the consumer and this is also one of the main reasons why Kafka has such great horizontal scalability.

Using the high level consumer API will automatically do this for you by committing consumed offsets in Zookeeper (or a more recent configuration option is using by a special Kafka topic to keep track of consumed messages).

The simple consumer API make you deal with how and where to keep track of consumed messages yourself.

Purging of messages in Kafka is done automatically by either specifying a retention time for a topic or by defining a disk quota for it so for your case of one 5GB file, this file will be deleted after the retention period you define has passed, regardless of if it has been consumed or not.

Temporarily update the retention time on the topic to one second: kafka-topics.sh --zookeeper localhost:13003 --alter --topic MyTopic --config retention.ms=1000

then wait for the purge to take effect (about one minute). Once purged, restore the previous retention.ms value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment