Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Quick command reference for Apache Kafka

Kafka Topics

List existing topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

Describe a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic

Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --delete-config retention.ms

Delete a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic

Get number of messages in a topic ???

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'

Get the earliest offset still in a topic

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2

Get the latest offset still in a topic

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1

Consume messages with the console consumer

bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic mytopic --from-beginning

Get the consumer offsets for a topic

bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=mytopic --group=my_consumer_group

Read from __consumer_offsets

Add the following property to config/consumer.properties: exclude.internal.topics=false

bin/kafka-console-consumer.sh --consumer.config config/consumer.properties --from-beginning --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"

Kafka Consumer Groups

List the consumer groups known to Kafka

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list (old api)

bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list (new api)

View the details of a consumer group

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group <group name>

kafkacat

Getting the last five message of a topic

kafkacat -C -b localhost:9092 -t mytopic -p 0 -o -5 -e

Zookeeper

Starting the Zookeeper Shell

bin/zookeeper-shell.sh localhost:2181

@leeforaday

This comment has been minimized.

Copy link

leeforaday commented Apr 17, 2018

The command for "Get number of messages in a topic ???" will only work if our earliest offsets are zero, correct?
If we have a topic, whose message retention period already passed (meaning some messages were discarded and new ones were added), we would have to get the earliest and latest offsets, subtract them for each partition accordingly and then add them, right?

@davewat

This comment has been minimized.

Copy link

davewat commented Apr 26, 2018

You just need to wrap the two calculations in a script, and then calculate the difference. An example:
https://gist.github.com/davewat/b3c5c4a383d1d9d26df33ecbb6ddd668

@ghost

This comment has been minimized.

Copy link

ghost commented Apr 30, 2018

View the details of a consumer group: passing --zookeeper shows old api consumer groups only (as for List Consumer Groups)
For new API's consumers use --bootstrap-server
It might be obvious, but it is worth mentioning it explicitly :)
(long time no see @ursuad :) )

@vinayakmkmishra1792

This comment has been minimized.

Copy link

vinayakmkmishra1792 commented Jul 22, 2018

Is their something that would list latest offsets by partitions?
Thanks in advance.

@aswinjoseroy

This comment has been minimized.

Copy link

aswinjoseroy commented Aug 3, 2018

If someone wants to see a consumer group's progress, you may use this.

@gunjanarora

This comment has been minimized.

Copy link

gunjanarora commented Sep 14, 2018

Some of these commands are not working. For example --zookeeper is not a valid option for listing consumer groups. Instead, need to pass broker as argument.
Working command:
kafka-consumer-groups --bootstrap-server {Broker_List} --list

@kucera-jan-cz

This comment has been minimized.

Copy link

kucera-jan-cz commented Sep 18, 2018

One important note to following scripts and gist mentioned by @davewat - these counts does not reflect deleted messages in compacted topic. We learned this the hard way...

@claudioaltamura

This comment has been minimized.

Copy link

claudioaltamura commented Jan 9, 2019

List the consumer groups known to Kafka
new api
you can omit the option --new-consumer (kafka 1.1.0)

The [new-consumer] option is deprecated and will be removed in a future major release...

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list (new api)

@tfalcao

This comment has been minimized.

Copy link

tfalcao commented Apr 16, 2019

Offsets for a group

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group mygroup

@cabib

This comment has been minimized.

Copy link

cabib commented Jun 26, 2019

........kafka_2.12-2.1.0\bin\windows>kafka-consumer-groups.bat --bootstrap-server localhost:9092 --describe

@Manish-Aman

This comment has been minimized.

Copy link

Manish-Aman commented Oct 21, 2019

some commands need to be modified

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.