Skip to content

Instantly share code, notes, and snippets.

@xsqian
Last active October 11, 2023 19:04
Show Gist options
  • Save xsqian/0d0ed946891b9e9375953380b5da3385 to your computer and use it in GitHub Desktop.
Save xsqian/0d0ed946891b9e9375953380b5da3385 to your computer and use it in GitHub Desktop.
How to alter kafka partitions
# create a topic with specified partitons
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --create --partitions 2 --topic test-partitions
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --describe --topic test-partitions
# create a topic with default partitions
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --create --topic test-def-partitions
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --describe --topic test-def-partitions
# alter the number of partitions of an existing topic
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --alter --topic test-def-partitions --partitions 3
kafka-topics --bootstrap-server xx.xx.xx.xx:39092 --describe --topic test-def-partitions
# Topic: test-def-partitions PartitionCount: 3 ReplicationFactor: 1 Configs:
# Topic: test-def-partitions Partition: 0 Leader: 1 Replicas: 1 Isr: 1 Offline:
# Topic: test-def-partitions Partition: 1 Leader: 1 Replicas: 1 Isr: 1 Offline:
# Topic: test-def-partitions Partition: 2 Leader: 1 Replicas: 1 Isr: 1 Offline:
# check the number of message in a topic
kafka-run-class kafka.tools.GetOffsetShell --broker-list 20.150.137.17:39092 --topic test-partitions
# create a topic mytopic1 with 2 partitions
kafka-topics --bootstrap-server 20.150.137.17:39092 --create --partitions 2 --topic mytopic1
# start producing message for the topics mytopic1
kafka-console-producer \
--bootstrap-server 20.150.137.17:39092 \
--topic mytopic1 \
--property parse.key=true \
--property key.separator=,
# enter the following messages:
a,{"column1": "1"}
b,{"column1": "1"}
c,{"column1": "1"}
d,{"column1": "1"}
a,{"column1": "2"}
b,{"column1": "2"}
c,{"column1": "2"}
d,{"column1": "2"}
a,{"column1": "3"}
b,{"column1": "3"}
c,{"column1": "3"}
d,{"column1": "3"}
# see the messages in the topic mytopic1 and partition 0
kafka-console-consumer \
--bootstrap-server 20.150.137.17:39092 \
--topic mytopic1 \
--property print.key=true \
--property key.separator=, \
--partition 0 \
--from-beginning
# see the messages in the topic mytopic1 and partition 1
kafka-console-consumer \
--bootstrap-server 20.150.137.17:39092 \
--topic mytopic1 \
--property print.key=true \
--property key.separator=, \
--partition 1 \
--from-beginning
# show the number of messages in echo of the partitions in the topic mytopic1
# https://stackoverflow.com/questions/28579948/java-how-to-get-number-of-messages-in-a-topic-in-apache-kafka
kafka-run-class kafka.tools.GetOffsetShell \
--broker-list 20.150.137.17:39092 --topic mytopic1
mytopic1:0:9
mytopic1:1:3
kafka-run-class kafka.admin.ConsumerGroupCommand \
--all-groups \
--bootstrap-server 20.150.137.17:39092 \
--describe | grep mytopic1
Consumer group '_confluent-controlcenter-6-2-0-lastProduceTimeConsumer' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
_confluent-controlcenter-6-2-0-lastProduceTimeConsumer mytopic1 1 3 3 0 - - -
_confluent-controlcenter-6-2-0-lastProduceTimeConsumer mytopic1 0 9 9 0 - - -
# ###########ConsumerOffsetChecker################# has been deprecated. Going forward,
# please use kafka-consumer-groups.sh (kafka.admin.ConsumerGroupCommand) for this functionality.
# kafka-run-class kafka.admin.ConsumerOffsetChecker --topic mytopic1 --broker-list 20.150.137.17:39092 --all-groups
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment