kafka installation on ubuntu 16.04

Apache Kafka on Ubuntu.txt

Install and configure Apache Kafka on Ubuntu 16.04

1. Install Java (JDK8)::

Add the repository

  $ sudo add-apt-repository -y ppa:webupd8team/java

Update the metadata of the new repository and install JDK

  $ sudo apt update && apt install oracle-java8-installer -y

Verify that JDK 8 is installed properly

  $ java -version

You should see the output something like this

  java version "1.8.0_111"
  Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
  Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

2. Install zookeeper::

  $ sudo apt install zookeeperd
  
After the installation completes, ZooKeeper will be started as a daemon automatically. By default, it will listen on port 2181.

Confirm zookeeper is running on expected port::

  $ netstat -ant | grep :2181

If everything's fine, you should see the following output:

  tcp6       0      0 :::2181                 :::*                    LISTEN

if after typing 'ruok' once connected to 'localhost', zookeeper will respond with 'imok' and close the session.

  $ telnet localhost 2181
  Trying ::1...
  Connected to localhost.
  Escape character is '^]'.
  ruok <-- Type at empty prompt!
  imokConnection closed by foreign host.
    
3. Download kafka from http://kafka.apache.org/downloads.html::

  $ curl -O http://apache-mirror.rbc.ru/pub/apache/kafka/0.10.1.1/kafka_2.11-0.10.1.1.tgz

Untar and move binaries to /opt/kafka/:

  $ sudo mkdir -p /opt/kafka && sudo tar -xvf kafka_2.11-0.10.1.1.tgz -C /opt/kafka/
  
4. Configure and run Kafka Server::

Turn on topic delete
  
  $ sudo sed -i 's/#delete.topic.enable=true/delete.topic.enable=true/' /opt/kafka/kafka_2.11-0.10.1.1/config/server.properties
  
Start Kafka by running kafka-server-start.sh script

  $ sudo /opt/kafka/kafka_2.11-0.10.1.1/bin/kafka-server-start.sh /opt/kafka/kafka_2.11-0.10.1.1/config/server.properties

Now, we can check listening ports:
- ZooKeeper : 2181
- Kafka 

  $ netstat -ant | grep -E ':2181|:9092'

If everything's fine, you should see the following output:

  tcp6       0      0 :::9092                 :::*                    LISTEN      1367/java       
  tcp6       0      0 :::2181                 :::*                    LISTEN      -               
  
5.  Test Server::

Open another session, and create a topic:

  $ /opt/kafka/kafka_2.11-0.10.1.1/bin/kafka-topics.sh --create --topic topic-test --zookeeper localhost:2181 --partitions 1 --replication-factor 1

List available topics:

  $ /opt/kafka/kafka_2.11-0.10.1.1/bin/kafka-topics.sh --list --zookeeper localhost:2181
    
Send message to topic as a producer via the 'kafka-console-producer.sh'::
  
    echo "hello world" | /opt/kafka/kafka_2.11-0.10.1.1/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic-test
    
*Consume* the send message::

   $ /opt/kafka/kafka_2.11-0.10.1.1/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-test --from-beginning

# referecing:
# https://gist.github.com/monkut/07cd1618102cbae8d587811654c92902
# https://devops.profitbricks.com/tutorials/install-and-configure-apache-kafka-on-ubuntu-1604-1/