kafka-cheat-sheet

kafka-cheat-sheet.md

Kafka Cheat Sheet

1. Start kafka server

# Using docker
Docker —rm -d —name Kafka -e KAFKA_CREATE_TOPICS=test:1:1 wurstmeister/kafka -p 9092 9092 
# Using OS
nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
nohup bin/kafka-server-start.sh config/server.properties &

2. Woring with topics

# List topic
kafka-topics --list --zookeeper localhost:2181
# Create topic
kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 3 --topic test_topic
# Describe topic
kafka-topics --describe --bootstrap-server localhost:9092 --topic test_topic
# Alter Topic to change partitions and replications
kafka-topics --alter --bootstrap-server localhost:9092 --topic test_topic --partitions 5

3. Producer and Consumer

# Start Provider
kafka-console-producer --broker-list localhost:9092 --topic test_topic
# Start producer with key:value separator
kafka-console-producer --broker-list localhost:9092 --topic test_topic --property "parse.key=true" --property "key.separator=:"
# Start Consumer
kafka-console-consumer --bootstrap-server localhost:9092 --topic test_topic --from-beginning

4. Consumer Group

# Create new and describe consumer group
kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --describe --group group_name
# Describe consumer group
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group group_name
# Reset offsets of consumer group [--to-earliest, --to-latest]
kafka-consumer-groups --bootstrap-server localhost:9092 --group group_name --topic test_topic --reset-offsets --to-earliest --execute
# Seek to offset
kafka-consumer-groups --bootstrap-server localhost:9092 --group group_name --topic test_topic --reset-offsets --to-offset 12345 --execute
# Shift by offsets
kafka-consumer-groups --bootstrap-server localhost:9092 --group group_name --topic test_topic --reset-offsets --shift-by -1000 --execute

5. Unset JMX_PORT to delete topic in kafka

unset JMX_PORT; kafka-streams-application-reset.sh  --input-topics order_topic   --application-id KafkaManagerOffsetCache --bootstrap-servers 172.16.8.98:9095
unset JMX_PORT; kafka-topics.sh --zookeeper 172.16.8.98:2184 --delete --topic order_topic