Some kafka operations

kafka commands.md

Commands to check various states of Kafka

NOTE: Code blocks indicate input on prompt with a '$'. Everything else is output

Create a new topic

$ /opt/kafka/bin/kafka-topics.sh --topic my_topic --zookeeper localhost:2181 --create --partitions 1 --replication-factor 1
Created topic "my_topic".

List all topics

$ /opt/kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181__consumer_offsets
bro_raw
my_topic

List topic details

$ /opt/kafka/bin/kafka-topics.sh --topic my_topic --zookeeper localhost:2181 --describe
Topic:my_topic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: my_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

Change number of partitions

$ /opt/kafka/bin/kafka-topics.sh --topic my_topic --zookeeper localhost:2181 --alter --partitions 4
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

$ /opt/kafka/bin/kafka-topics.sh --topic my_topic --zookeeper localhost:2181 --describe
Topic:my_topic	PartitionCount:4	ReplicationFactor:1	Configs:
	Topic: my_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
	Topic: my_topic	Partition: 1	Leader: 0	Replicas: 0	Isr: 0
	Topic: my_topic	Partition: 2	Leader: 0	Replicas: 0	Isr: 0
	Topic: my_topic	Partition: 3	Leader: 0	Replicas: 0	Isr: 0

Check topic offsets

Ideally the lag is 0 or close to it. The Offset field indicates the last record that the consumer read (in this case, logstash). The logSize indicates how many total records are in the topic partition. The lag indicates the difference between the two and should approach 0.

$ /opt/kafka/bin/kafka-consumer-offset-checker.sh --zookeeper 127.0.0.1:2181 --topic bro_raw --group logstash
Group           Topic                          Pid Offset          logSize         Lag             Owner
logstash        bro_raw                        0   2642920         2642920         0               logstash_lab003.ds.mocyber.org-1448462427435-5d63162f-0
logstash        bro_raw                        1   221536          221536          0               logstash_lab003.ds.mocyber.org-1448462427435-5d63162f-0
logstash        bro_raw                        2   86              86              0               logstash_lab003.ds.mocyber.org-1448462427435-5d63162f-0
logstash        bro_raw                        3   438             438             0               logstash_lab003.ds.mocyber.org-1448462427435-5d63162f-0
logstash        bro_raw                        4   0               0               0               logstash_lab003.ds.mocyber.org-1448462427435-5d63162f-0
logstash        bro_raw                        5   0               0               0               logstash_lab003.ds.mocyber.org-1448462427435-5d63162f-0