Create Kafka Multi Node, Multi Broker Cluster

Kafka MultiNode - MultiBroker Cluster.md

How to create a MultiNode - MultiBroker Cluster for Kafka on AWS

PreRequisites :

1. Kafka Binary files : http://kafka.apache.org/downloads.html

2. Atleast 2 AWS machines : AWS EMR or EC2 will be preferable

3. A Kafka Manager Utility to watch up the Cluster : https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem

Installation

1. Now first download the kafka Tarball or binaries on your AWS instances and extract them

   tar -xzvf kafka_2.10-0.8.2.1.tgz mv kafka_2.10-0.8.2.1 kafka

2. On Both the Instances, you only need two properties to be changed i.e. zookeeper.properties & server.properties

a) Going with the first one edit "zookeeper.properties" on both the instances to

vi ~/kafka/config/zookeeper.properties
clientPort=2080 #Changing the Port from default "2181" to "2080"
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
#add here more servers if you want
initLimit=5
syncLimit=2

b) Now edit both instances "server.properties" and update the following this

vi ~/kafka/config/server.properties
broker.id=1 
port=9092
host.name=ec2-<IP1>.amazonaws.com #for 2nd EC2 instance it'll be "ec2-<IP2>.amazonaws.com"
num.partitions=4
zookeeper.connect=ec2-<IP1>.amazonaws.com:2080,ec2-<IP2>.amazonaws.com:2080 #Add all the Instances here for each Instance

3. After this go to the /tmp of every instance and create following things :

   cd /tmp/ mkdir zookeeper #Zookeeper temp dir cd zookeeper touch myid #Zookeeper temp file echo '1' >> myid #Add Server ID for Respective Instances i.e. "server.1 and server.2 etc"

4. Now all is done, Need to start zookeeper and kafka-server on both instances

   ##Start on both Instances

   ~/kafka/bin/zookeeper-server-start.sh ~/kafka/config/zookeeper.properties

   ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties

5. Now go to KafkaTool and add both of your AWS instances, You'll see multi brokers with multple partitions in it.

Post your questions here for more help

@Tiyadong How did you fix it? i am getting the exact same error.

Notice the message: "You need at least 3 servers." An odd number of zookeeper servers is recommended.

@avinash-mishra I fixed it with having correct Inbound rule specified in Security Group attached to the EC2, while accessing internally and not over internet.

Why do you need to specify '2888:3888' in zookeeper.properties for server.1 and server.2 ?

Hi @avinash-mishra I'm also getting the same error.

[2022-07-02 10:27:51,573] WARN Cannot open channel to 2 at election address ip-172-31-39-106.ap-south-1.compute.internal/172.31.39.106:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
Please let me know how you resolved this.