Skip to content

Instantly share code, notes, and snippets.

@vipmax
Last active April 17, 2024 21:21
Show Gist options
  • Star 84 You must be signed in to star a gist
  • Fork 25 You must be signed in to fork a gist
  • Save vipmax/9ceeaa02932ba276fa810c923dbcbd4f to your computer and use it in GitHub Desktop.
Save vipmax/9ceeaa02932ba276fa810c923dbcbd4f to your computer and use it in GitHub Desktop.

Kafka installation with systemd

0. Create kafka user

sudo adduser kafka
sudo adduser kafka sudo
su -l kafka

1. Download and Install kafka archive

cd /opt                                                              # go to /opt folder

curl  https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz -O   # download archive to folder
tar -zxvf kafka*.tgz                                                 # extract archive 
rm kafka*.tgz                                                        # remove archive 
ln -s kafka_2.13-3.2.0/ kafka                                        # create symbolic link  

2. Create zookeeper service file replace {{ }} with the correct value in your case

# just creates a file with the following content  
cat <<EOF > /etc/systemd/system/kafka-zookeeper.service            

[Unit]
Description=Apache Zookeeper server (Kafka)
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User={{USER || kafka}}
Environment=JAVA_HOME={{must be JAVA_HOME here || /usr/lib/jvm/java-1.8.0-openjdk}}
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh

[Install]
WantedBy=multi-user.target

EOF

3. Create kafka service file

# just creates a file with the following content  
cat <<EOF > /etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target
After=network.target remote-fs.target kafka-zookeeper.service

[Service]
Type=simple
User={{USER || kafka}}
Environment=JAVA_HOME={{must be JAVA_HOME here || /usr/lib/jvm/java-1.8.0-openjdk}}
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

EOF

3 * Edit service files

nano /etc/systemd/system/kafka-zookeeper.service
nano /etc/systemd/system/kafka.service

4. Edit kafka settings

nano /opt/kafka/config/server.properties
listeners=PLAINTEXT://{{IP || 192.168.0.103}}:9092

5. Reload and start the systemd services

systemctl daemon-reload
systemctl enable kafka-zookeeper.service
systemctl enable kafka.service
systemctl start kafka-zookeeper.service
systemctl start kafka.service

systemctl status kafka-zookeeper.service
systemctl status kafka.service

6. Check services state

systemctl status kafka-zookeeper.service
systemctl status kafka.service
@bmshukla
Copy link

User=nano & Group=nano
There is no "nano" found.
Should I create login "nano" in /etc/passwd and group "nano" in /etc/group ?

@vipmax
Copy link
Author

vipmax commented Nov 6, 2019

User=nano & Group=nano
There is no "nano" found.
Should I create login "nano" in /etc/passwd and group "nano" in /etc/group ?

no
change it to your username
nano its just in my case

@ptebraak
Copy link

Thanks for posting this. I wonder how you add additional brokers to the service? Could you include an example of a service with 2 brokers?

@hillal20
Copy link

@training1-rbcalonzo
Copy link

the "http://apache-mirror.rbc.ru/pub/apache/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz" is not found

you can choose the version available in the mirror by visiting this link: http://apache-mirror.rbc.ru/pub/apache/kafka/

@izghitu
Copy link

izghitu commented Jan 21, 2021

thank you

@pythonzm
Copy link

thank you

@glmonarch
Copy link

Thx a lot!

@Torrakanor611
Copy link

is good, but if you are trying to setup a kafka broker in a remote machine were o only have a ssh conection to that machine, u should create another user and give him permissions to sudo, like explained in this guide:
https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-20-04

@minalhk
Copy link

minalhk commented Jun 3, 2021

kafka.service - Apache Kafka Server
Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2021-06-03 16:57:10 IST; 20s ago
Docs: http://kafka.apache.org/documentation.html
Process: 5376 ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties (code=exited, status=1/FAILURE)
Main PID: 5376 (code=exited, status=1/FAILURE)

Jun 03 16:57:10 minal systemd[1]: Started Apache Kafka Server.
Jun 03 16:57:10 minal kafka-server-start.sh[5376]: Classpath is empty. Please build the project first e.g. by running './gradlew jar -PscalaVe
Jun 03 16:57:10 minal systemd[1]: kafka.service: Main process exited, code=exited, status=1/FAILURE
Jun 03 16:57:10 minal systemd[1]: kafka.service: Failed with result 'exit-code'.

server is not starting and I'm getting this response when I check kafka status
If you could help

@weenxin
Copy link

weenxin commented Dec 15, 2021

thanks

@mlugaliki
Copy link

Very helpful. Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment