Skip to content

Instantly share code, notes, and snippets.

@mkanchwala
Last active July 2, 2022 10:44
Show Gist options
  • Save mkanchwala/fbfdd5ef866a58a77f6e to your computer and use it in GitHub Desktop.
Save mkanchwala/fbfdd5ef866a58a77f6e to your computer and use it in GitHub Desktop.
Create Kafka Multi Node, Multi Broker Cluster

How to create a MultiNode - MultiBroker Cluster for Kafka on AWS

PreRequisites :

  1. Kafka Binary files : http://kafka.apache.org/downloads.html

  2. Atleast 2 AWS machines : AWS EMR or EC2 will be preferable

  3. A Kafka Manager Utility to watch up the Cluster : https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem

Installation

  1. Now first download the kafka Tarball or binaries on your AWS instances and extract them

    tar -xzvf kafka_2.10-0.8.2.1.tgz mv kafka_2.10-0.8.2.1 kafka

  2. On Both the Instances, you only need two properties to be changed i.e. zookeeper.properties & server.properties

a) Going with the first one edit "zookeeper.properties" on both the instances to

vi ~/kafka/config/zookeeper.properties
clientPort=2080 #Changing the Port from default "2181" to "2080"
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
#add here more servers if you want
initLimit=5
syncLimit=2

b) Now edit both instances "server.properties" and update the following this

vi ~/kafka/config/server.properties
broker.id=1 
port=9092
host.name=ec2-<IP1>.amazonaws.com #for 2nd EC2 instance it'll be "ec2-<IP2>.amazonaws.com"
num.partitions=4
zookeeper.connect=ec2-<IP1>.amazonaws.com:2080,ec2-<IP2>.amazonaws.com:2080 #Add all the Instances here for each Instance
  1. After this go to the /tmp of every instance and create following things :

    cd /tmp/ mkdir zookeeper #Zookeeper temp dir cd zookeeper touch myid #Zookeeper temp file echo '1' >> myid #Add Server ID for Respective Instances i.e. "server.1 and server.2 etc"

  2. Now all is done, Need to start zookeeper and kafka-server on both instances

    ##Start on both Instances

    ~/kafka/bin/zookeeper-server-start.sh ~/kafka/config/zookeeper.properties

    ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties

  3. Now go to KafkaTool and add both of your AWS instances, You'll see multi brokers with multple partitions in it.

Post your questions here for more help

@t6nand
Copy link

t6nand commented Dec 4, 2015

How to test for multiple instances of zookeeper with kafka on same machine?

@Liamhanninen
Copy link

Do both instances have zookeeper and Kafka? Or does one instance of Kafka and one zookeeper. I'm trying to do one on each. How would I configure server.1 and server.2 even if zookeeper is only in one instance?

@ciasom
Copy link

ciasom commented Oct 3, 2016

Hi, how can you run 2 zookeepers? I got an warning when I started it: You need at least 3 servers.

@kjarrad
Copy link

kjarrad commented Oct 4, 2016

The 'E' in EC2 is 'elastic'.
That is, AWS can add/remove nodes to the cluster to achieve horizontal scalability.
Can this Gist be expanded to include this?

Note: I'm only asking about configuration of the cluster. Until that is achieved there is no need to consider issues such as re-sharding or reassigning partition leaders.

@sachinyeola
Copy link

I have 2 different servers, I want to fetch logs in a kafka topic from 1st server and load these logs on to elastic which is running on 2nd server. ELK and kafka is running on 2nd server. what configurations are required for it ?

@Ritesh-Sharma
Copy link

Ritesh-Sharma commented Mar 22, 2017

Hi @mkanchwala , can you tell me what are these port range

server.1=ec2-.amazonaws.com:2888:3888
server.2=ec2-.amazonaws.com:2888:3888

what is running on these ports ?

@avinash-mishra
Copy link

I am getting this error:

[2017-03-22 11:32:25,038] INFO Notification time out: 1600 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2017-03-22 11:32:31,644] WARN Cannot open channel to 2 at election address /<IP>:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
	at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)

Any Idea how to fix it?

@amitk-abk
Copy link

amitk-abk commented May 9, 2017

Thanks for this write up! Its definitely helpful.
Just wanted to know if zookeeper cluster that is set up already is to be used, is there any need to alter zookeeper.properties?
I think just editing server.properties with that already present zookeeper set up should be sufficient.
Hope this is the case.

@Tiyadong
Copy link

thank you for this very good post. I have couple of questions here.

  1. I installed kafka with 3 brokers by ambari.
  2. I have few zookeeper.properties and server.properties files on these 3 servers.
    /etc/kafka/conf.backup/zookeeper.properties
    /etc/kafka/2.5.5.0-157/0/zookeeper.properties
    /usr/hdp/2.5.5.0-157/etc/kafka/conf.default/zookeeper.properties
  3. I guess is should be --> /usr/hdp/2.5.5.0-157/etc/kafka/conf.default/zookeeper.properties --> please confirm it.

For the server.propertied, I have these files on these 3 broker machines
4. /etc/kafka/conf.backup/server.properties
/etc/kafka/2.5.5.0-157/0/server.properties
/usr/hdp/2.5.5.0-157/etc/kafka/conf.default/server.properties
5. I guess is should be --> /usr/hdp/2.5.5.0-157/etc/kafka/conf.default/server.properties -- please confirm it
6. finally, what is the relationship between these zookeepers on broker server and zookeeper on the maters servers? kafka zookeeper seemed have its won dormitory, these zookeeper still work with other zookeepers in the cluster?

your post is very detailed, learned so much from it.
thank you for your help.

@Tiyadong
Copy link

I have figured out the file I need to modify, however, when I start zookeeper-server-start.sh, I got an error:
Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException:
any ideas? please help.

@Tiyadong
Copy link

hi, at your step #5, you said "Now go to KafkaTool and add both of your AWS instances, You'll see multi brokers with multple partitions in it."
I had a different approach, I have installed kafka with 3 brokers on which zookeepers already installed, then I came back to following your steps to configure, you think this would make difference? this would cause my errors I have here? Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)

please respond.
thank you so much for your help.

@Tiyadong
Copy link

please discard all my questions, I fixed it.
thanks,

@Parth6288
Copy link

Trying to configure multinode instance using the steps mentioned however i am getting below error :
"Invalid config, exiting abnormally"

ubuntu@kafka2:~/kafka/kafka_2.11-1.0.0$ bin/zookeeper-server-start.sh config/zookeeper.properties
[2018-01-04 11:50:07,623] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-01-04 11:50:07,647] INFO Resolved hostname: x.x.x.189 to address: /x.x.x.189 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2018-01-04 11:50:07,647] INFO Resolved hostname: x.x.x.231 to address: /x.x.x.231 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2018-01-04 11:50:07,647] WARN No server failure will be tolerated. You need at least 3 servers. (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-01-04 11:50:07,648] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing config/zookeeper.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:154)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.lang.IllegalArgumentException: initLimit is not set
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:355)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:150)
... 2 more
Invalid config, exiting abnormally

Here is my zookeeper.properties file:
https://pastebin.com/zzQ3cUSD
i also created myid file inside the datadir and gave the server id (231 and 189 respectively in both the servers)

@Parth6288
Copy link

@Tiyadong How did you fix it? i am getting the exact same error.

@sdarwin
Copy link

sdarwin commented Apr 4, 2018

Notice the message: "You need at least 3 servers." An odd number of zookeeper servers is recommended.

@DivsDibs
Copy link

DivsDibs commented May 4, 2018

@avinash-mishra I fixed it with having correct Inbound rule specified in Security Group attached to the EC2, while accessing internally and not over internet.

@fredespo
Copy link

Why do you need to specify '2888:3888' in zookeeper.properties for server.1 and server.2 ?

@sarat532
Copy link

sarat532 commented Jul 2, 2022

Hi @avinash-mishra I'm also getting the same error.

[2022-07-02 10:27:51,573] WARN Cannot open channel to 2 at election address ip-172-31-39-106.ap-south-1.compute.internal/172.31.39.106:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
Please let me know how you resolved this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment