kafka setup on rhel7

kafka.setup.md

Rough notes on setup

other notes

https://gist.github.com/jkreps/c7ddb4041ef62a900e6c

Update box

• apt-get update
• apt-get upgrade
• apt-get java

Setup user

• create kafka user and group (check this on all servers)
• useradd kafka
• passwd kafka
• chage -I -1 -m 0 -M 99999 -E -1 kafka
• id kafka

Visudo

• Uncomment Software, Processes, Services
• All kafka user to have access to these
• Uncomment line and give kafka access to restart

Download Latest Kafka

• wget http://apache.cs.utah.edu/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz -e use_proxy=yes -e http_proxy=proxy:port
• tar -xvzf /opt/isv/kafka/

Update configs

/opt/isv/var/kafka/zookeeper path of ID files

• echo "1" > /opt/isv/var/kafka/zookeeper/myid
• echo "2" > /opt/isv/var/kafka/zookeeper/myid
• echo "3" > /opt/isv/var/kafka/zookeeper/myid

config/zookeeper.properties

• initLimit=5
• syncLimit=2
• server.1=<>:2888:3888
• server.2=<>:2888:3888
• server.3=<>:2888:3888

update config/server.properties

• zookeeper.connect=:2181,:2181,:2181

misc

• create /opt/isv/var/kafka/kafka-logs

• chown -R kafka:kafka /opt/isv/kafka

• chown -R kafka:kafka /opt/isv/var/kafka

• /etc/profile

• export JAVA_HOME=$JAVA_HOME:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-0.b14.el7_2.x86_64/jre/bin/java

• export PATH=$JAVA_HOME:$PATH

• create /opt/isv/var/kafka/kafka-logs

Setup Services

• /usr/lib/systemd/system/zookeeper.service

• /usr/lib/systemd/system/kafka.service

####symbolic link from kafka install dir to /usr/libexec/kafka

• ln -sf /opt/isv/kafka/kafka_2.11-0.10.0.0 /usr/libexec/kafka

• systemctl daemon-reload

• systemctl enable zookeeper

• systemctl enable kafka

• systemctl start zookeeper

• systemctl status zookeeper

• systemctl start kafka

• systemctl status kafka