Skip to content

Instantly share code, notes, and snippets.

Last active February 6, 2023 07:06
Show Gist options
  • Save monkut/07cd1618102cbae8d587811654c92902 to your computer and use it in GitHub Desktop.
Save monkut/07cd1618102cbae8d587811654c92902 to your computer and use it in GitHub Desktop.
kafka installation on ubuntu 16.04
# referecing:
1. Add 'kafka' user::
$ sudo useradd kafka -m
2. Install Java::
$ sudo apt-get update
$ sudo apt-get install default-jre
3. Install zookeeper::
$ sudo apt-get install zookeeperd
.. note::
After the installation completes, ZooKeeper will be started as a daemon automatically. By default, it will listen on port 2181.
4. Confirm zookeeper is running on expected port::
$ telnet localhost 2181
Trying ::1...
Connected to localhost.
Escape character is '^]'.
ruok <-- Type at empty prompt!
imokConnection closed by foreign host.
.. note::
if after typing 'ruok' once connected to 'localhost', zookeeper will respond with 'imok' and close the session.
5. Download kafka from
# with cntlm proxy installed and running if necessary
$ export http_proxy=
$ export https_proxy=
# grab latest stable
$ wget
6. untar and move binaries to /usr/local/kafka::
$ tar xvf kafka_2.11-
$ sudo mv kafka_2.11- /usr/local/kafka
7. Configure Kafka Server::
# turn on topic delete
$ vi /usr/local/kafka/config/server.properites
#>> At end of file add:
delete.topic.enable = true
# save and quit
8. Test Server::
$ /usr/local/kafka/bin/ /usr/local/kafka/config/
[2016-08-06 01:22:00,000] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
.. note::
This only starts the server temporarily for intial testing, the service should be registered later...
9. With the kafka sever running, open another session, and create a topic::
$ /usr/local/kafka/bin/ --create --topic topic-test --zookeeper localhost:2181 --partitions 1 --replication-factor 1
Created topic "topic-test".
10. List available topics::
$ /usr/local/kafka/bin/ --list --zookeeper localhost:2181
.. note::
You should see the created, 'topic-test' topic listed.
11. Send message to topic as a producer via the ''::
echo "hello world" | /usr/local/kafka/bin/ --broker-list localhost:9092 --topic topic-test
12. *Consume* the send message::
$ /usr/local/kafka/bin/ --zookeeper localhost:2181 --topic topic-test --from-beginning
.. note::
The '--from-beginning' flag given to start a consumer with the earliest message present in the log, rather than the latest message. (see */usr/local/kafka/bin/* help for more option details)
# for install of scala (sbt):
Spark installation
1. Install Scala Build Tool (sbt) [Make sure https_proxy is set if needed]::
1.1 get Scala Build Tool ubuntu repository info::
1.2 Install sbt repostory info::
sudo dpkg -i sbt-0.13.11.deb
1.3 Update repository info and install 'sbt::
sudo apt-get update
sudo apt-get install sbt
2. download spark binary (Grab latest stable from:
3. untar and move::
tar xvf spark-2.0.0-bin-hadoop2.7.tgz
sudo mv spark-2.0.0-bin-hadoop2.7 /usr/local/spark
4. Add spark configuraiton to your profile (or appropriate ENV configuration)::
vi ~/.profile
(Add the following to .profile)
# set PATH so it includes user's private bin directories
export PYSPARK_PYTHON=python3
5. Apply to current ENV::
source ~/.profile
5. Test configuration::
--> Should open the pyspark console
Copy link

cabecada commented Jun 28, 2017

this be /usr/local/kafka/config/ i guess :)

Copy link


Copy link


Copy link

RARain commented Nov 23, 2018


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment