Skip to content

Instantly share code, notes, and snippets.

@ursuad
Last active March 14, 2024 10:32
Show Gist options
  • Save ursuad/e5b8542024a15e4db601f34906b30bb5 to your computer and use it in GitHub Desktop.
Save ursuad/e5b8542024a15e4db601f34906b30bb5 to your computer and use it in GitHub Desktop.
Quick command reference for Apache Kafka

Kafka Topics

List existing topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

Describe a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic

Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --delete-config retention.ms

Delete a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic

Get number of messages in a topic ???

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'

Get the earliest offset still in a topic

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2

Get the latest offset still in a topic

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1

Consume messages with the console consumer

bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic mytopic --from-beginning

Get the consumer offsets for a topic

bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=mytopic --group=my_consumer_group

Read from __consumer_offsets

Add the following property to config/consumer.properties: exclude.internal.topics=false

bin/kafka-console-consumer.sh --consumer.config config/consumer.properties --from-beginning --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"

Kafka Consumer Groups

List the consumer groups known to Kafka

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list (old api)

bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list (new api)

View the details of a consumer group

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group <group name>

kafkacat

Getting the last five message of a topic

kafkacat -C -b localhost:9092 -t mytopic -p 0 -o -5 -e

Zookeeper

Starting the Zookeeper Shell

bin/zookeeper-shell.sh localhost:2181

@Manish-Aman
Copy link

some commands need to be modified

@vicmerlis
Copy link

Get number of messages in a topic

echo "$(($(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1 | awk -F ":" '{sum += $3} END {print sum}')-$(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2 | awk -F ":" '{sum += $3} END {print sum}')))"

@vicmerlis
Copy link

Topic size:
kafka-log-dirs.sh --describe --bootstrap-server $kafkaHost:9092 --topic-list $topicName

@saikrishna-chunchu
Copy link

https://gist.github.com/ursuad/e5b8542024a15e4db601f34906b30bb5#read-from-__consumer_offsets

For Kafka 11 onwards, the formatter to be used is kafka.coordinator.group.GroupMetadataManager$OffsetsMessageFormatter

@Dongzhenpu
Copy link

Get number of messages in a Topic
GetOffsetShell only worked in PLAINTEXT, how to achieve this in SASL_PALINTEXT?

@stfclv
Copy link

stfclv commented Mar 16, 2023

--command-config /path/to/file.properties with the files.properties containing the ```security.protocol=SASL_SSL````

PLAIN versus PLAINTEXT: Do not confuse the SASL mechanism PLAIN with the no TLS/SSL encryption option, which is called PLAINTEXT. Configuration parameters such as sasl.enabled.mechanisms or sasl.mechanism.inter.broker.protocol may be configured to use the SASL mechanism PLAIN, whereas security.inter.broker.protocol or listeners may be configured to use the no TLS/SSL encryption option, SASL_PLAINTEXT.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment