Skip to content

Instantly share code, notes, and snippets.

@mkanchwala
Last active July 2, 2022 10:44
Show Gist options
  • Save mkanchwala/fbfdd5ef866a58a77f6e to your computer and use it in GitHub Desktop.
Save mkanchwala/fbfdd5ef866a58a77f6e to your computer and use it in GitHub Desktop.
Create Kafka Multi Node, Multi Broker Cluster

How to create a MultiNode - MultiBroker Cluster for Kafka on AWS

PreRequisites :

  1. Kafka Binary files : http://kafka.apache.org/downloads.html

  2. Atleast 2 AWS machines : AWS EMR or EC2 will be preferable

  3. A Kafka Manager Utility to watch up the Cluster : https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem

Installation

  1. Now first download the kafka Tarball or binaries on your AWS instances and extract them

    tar -xzvf kafka_2.10-0.8.2.1.tgz mv kafka_2.10-0.8.2.1 kafka

  2. On Both the Instances, you only need two properties to be changed i.e. zookeeper.properties & server.properties

a) Going with the first one edit "zookeeper.properties" on both the instances to

vi ~/kafka/config/zookeeper.properties
clientPort=2080 #Changing the Port from default "2181" to "2080"
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
#add here more servers if you want
initLimit=5
syncLimit=2

b) Now edit both instances "server.properties" and update the following this

vi ~/kafka/config/server.properties
broker.id=1 
port=9092
host.name=ec2-<IP1>.amazonaws.com #for 2nd EC2 instance it'll be "ec2-<IP2>.amazonaws.com"
num.partitions=4
zookeeper.connect=ec2-<IP1>.amazonaws.com:2080,ec2-<IP2>.amazonaws.com:2080 #Add all the Instances here for each Instance
  1. After this go to the /tmp of every instance and create following things :

    cd /tmp/ mkdir zookeeper #Zookeeper temp dir cd zookeeper touch myid #Zookeeper temp file echo '1' >> myid #Add Server ID for Respective Instances i.e. "server.1 and server.2 etc"

  2. Now all is done, Need to start zookeeper and kafka-server on both instances

    ##Start on both Instances

    ~/kafka/bin/zookeeper-server-start.sh ~/kafka/config/zookeeper.properties

    ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties

  3. Now go to KafkaTool and add both of your AWS instances, You'll see multi brokers with multple partitions in it.

Post your questions here for more help

@Parth6288
Copy link

Trying to configure multinode instance using the steps mentioned however i am getting below error :
"Invalid config, exiting abnormally"

ubuntu@kafka2:~/kafka/kafka_2.11-1.0.0$ bin/zookeeper-server-start.sh config/zookeeper.properties
[2018-01-04 11:50:07,623] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-01-04 11:50:07,647] INFO Resolved hostname: x.x.x.189 to address: /x.x.x.189 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2018-01-04 11:50:07,647] INFO Resolved hostname: x.x.x.231 to address: /x.x.x.231 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2018-01-04 11:50:07,647] WARN No server failure will be tolerated. You need at least 3 servers. (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-01-04 11:50:07,648] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing config/zookeeper.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:154)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.lang.IllegalArgumentException: initLimit is not set
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:355)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:150)
... 2 more
Invalid config, exiting abnormally

Here is my zookeeper.properties file:
https://pastebin.com/zzQ3cUSD
i also created myid file inside the datadir and gave the server id (231 and 189 respectively in both the servers)

@Parth6288
Copy link

@Tiyadong How did you fix it? i am getting the exact same error.

@sdarwin
Copy link

sdarwin commented Apr 4, 2018

Notice the message: "You need at least 3 servers." An odd number of zookeeper servers is recommended.

@DivsDibs
Copy link

DivsDibs commented May 4, 2018

@avinash-mishra I fixed it with having correct Inbound rule specified in Security Group attached to the EC2, while accessing internally and not over internet.

@fredespo
Copy link

Why do you need to specify '2888:3888' in zookeeper.properties for server.1 and server.2 ?

@sarat532
Copy link

sarat532 commented Jul 2, 2022

Hi @avinash-mishra I'm also getting the same error.

[2022-07-02 10:27:51,573] WARN Cannot open channel to 2 at election address ip-172-31-39-106.ap-south-1.compute.internal/172.31.39.106:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
Please let me know how you resolved this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment