Skip to content

Instantly share code, notes, and snippets.

@philipz
Last active March 25, 2020 09:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save philipz/18ede9ca92b28ddb280de1e1a057b7e0 to your computer and use it in GitHub Desktop.
Save philipz/18ede9ca92b28ddb280de1e1a057b7e0 to your computer and use it in GitHub Desktop.
Kafka Partitions and Consumer Group

A. Consumer > Partitions

consumer.properties

group.id=group1
  1. ./kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic test
  2. ./kafka-console-producer.sh --broker-list kafka:9092 --topic test
  3. ./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --consumer.config ./consumer.properties
  4. ./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --consumer.config ./consumer.properties

Result

  1. 1,a 1,b 1,c
  2. 1,a 1,b 1,c

B. Consumer = Partitions

  1. ./kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 2 --topic test1
  2. ./kafka-console-producer.sh --broker-list kafka:9092 --topic test1
  3. ./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test1 --consumer.config ./consumer.properties
  4. ./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test1 --consumer.config ./consumer.properties

Result

  1. 1,a 1,b 1,c 1,d
  2. 1,a 1,c
  3. 1,b 1,d

C. Multi-Consumer ID

consumer1.properties

group.id=group2
  1. ./kafka-console-producer.sh --broker-list kafka:9092 --topic test1
  2. ./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test1 --consumer.config ./consumer.properties
  3. ./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test1 --consumer.config ./consumer1.properties

Result

  1. 1,a 1,b 1,c 1,d
  2. 1,a 1,b 1,c 1,d
  3. 1,a 1,b 1,c 1,d

D. Consumer < Partitions

Increase topic partition number

./kafka-topics.sh --alter --zookeeper zookeeper:2181 --partitions 3 --topic test

  1. ./kafka-console-producer.sh --broker-list kafka:9092 --topic test
  2. ./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --consumer.config ./consumer.properties
  3. ./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --consumer.config ./consumer.properties

Result

  1. 1,a 1,b 1,c 1,d 1,e 1,f
  2. 1,a 1,c 1,d 1,f
  3. 1,b 1,e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment