Skip to content

Instantly share code, notes, and snippets.

  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
What would you like to do?
Create Kafka Multi Node, Multi Broker Cluster

How to create a MultiNode - MultiBroker Cluster for Kafka on AWS

PreRequisites :

  1. Kafka Binary files :

  2. Atleast 2 AWS machines : AWS EMR or EC2 will be preferable

  3. A Kafka Manager Utility to watch up the Cluster :


  1. Now first download the kafka Tarball or binaries on your AWS instances and extract them

    tar -xzvf kafka_2.10- mv kafka_2.10- kafka

  2. On Both the Instances, you only need two properties to be changed i.e. &

a) Going with the first one edit "" on both the instances to

vi ~/kafka/config/
clientPort=2080 #Changing the Port from default "2181" to "2080"
#add here more servers if you want

b) Now edit both instances "" and update the following this

vi ~/kafka/config/ 
port=9092<IP1> #for 2nd EC2 instance it'll be "ec2-<IP2>"
zookeeper.connect=ec2-<IP1>,ec2-<IP2> #Add all the Instances here for each Instance
  1. After this go to the /tmp of every instance and create following things :

    cd /tmp/ mkdir zookeeper #Zookeeper temp dir cd zookeeper touch myid #Zookeeper temp file echo '1' >> myid #Add Server ID for Respective Instances i.e. "server.1 and server.2 etc"

  2. Now all is done, Need to start zookeeper and kafka-server on both instances

    ##Start on both Instances

    ~/kafka/bin/ ~/kafka/config/

    ~/kafka/bin/ ~/kafka/config/

  3. Now go to KafkaTool and add both of your AWS instances, You'll see multi brokers with multple partitions in it.

Post your questions here for more help

Copy link

rahuludumatt commented Jul 1, 2018

How to create consumer in java which connect to zookeeper instead of kafka broker url?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment