Setting up Zookeeper, Kafka, Elasticsearch, Kibana, and Logstash on the UTD Server

0-README.md

UTD Server Kafka & ELK Stack

This tutorial outlines how to set up Zookeeper, Kafka, Elasticsearch, Kibana, and Logstash on the UTD server.

Table of Contents:

1. Setup
2. Install Zookeper
3. Install Kafka (depends on Zookeeper)
4. Install PCRE
5. Install NGINX (depends on PCRE)
6. Install Elasticsearch
7. Install Kibana (depends on NGINX and Elasticsearch)
8. Install Logstash

1-SETUP.md

Setup

Firstly, the configurations from Lab 1's EnvVariables file can cause some issues in setup. Provided below is a modification of the original EnvVariables with almost everything commented out and Java being updated to point to JDK8.

#! /bin/sh
export JAVA_HOME=/usr/local/jdk1.8.0_172
# export PIG_HOME=/usr/local/pig-0.13.0
# export PIG_CLASSPATH=/usr/local/pig-0.13.0/lib
# export HADOOP_INSTALL=/usr/local/hadoop-2.4.1
# export HADOOP_CONF_DIR=/usr/local/hadoop-2.4.1/etc/hadoop
# export HADOOP_HOME_WARN_SUPPRESS=1
# export HADOOP_HOME=/usr/local/hadoop-2.4.1
# export HADOOP_PREFIX=/usr/local/hadoop-2.4.1
# export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
# export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
# source $HADOOP_PREFIX/etc/hadoop/hadoop-env.sh
# source $HADOOP_PREFIX/etc/hadoop/yarn-env.sh
#
# export HADOOP_MAPRED_HOME=/usr/local/hadoop-2.4.1
# export YARN_HOME=/usr/local/hadoop-2.4.1
#
# export HIVE_HOME=/usr/local/apache-hive-0.13.1
# export MAVEN_HOME=/usr/local/apache-maven-3.2.2
# export ANT_HOME=/usr/local/apache-ant-1.9.4
# export SPARK_HOME=/usr/local/spark-1.6.1-bin-hadoop2.4
# export MAHOUT_HOME=/usr/local/mahout-1.0u
# export MASTER=spark://cshadoop1.utdallas.edu:7077
# beware of accidental line wrap on the remaining lines
# export PYTHONHOME=/opt/python-2.7.11
# export PYTHONPATH=/opt/python-2.7.11
# export PATH=$MAVEN_HOME/bin:$MAHOUT_HOME/bin:$PATH
# export PATH=$HIVE_HOME/bin:$PIG_HOME/bin:$PATH
# export PATH=$ANT_HOME/bin:$HADOOP_HOME/bin:$PATH
# export PATH=$SPARK_HOME/bin:$GIRAPH_HOME/bin:$PATH
# export PATH=$PYTHONHOME/bin:$PATH
export PATH=$JAVA_HOME/bin:$PATH
# export PATH=/usr/local/apache-cassandra-2.0.5/bin:$PATH

To log in to the CS6350 server, you need to connect with the host YOURNETID@cs6360.utdallas.edu. In order to connect outside of UTD, a VPN has to be set up. Instructions can be found here.

Lastly, when logging into the CS6350 server, create a programs folder. This folder will be the folder we will install all of the software in.

2-ZOOKEEPER.md

Install Zookeeper

Note: These instructions are for Zookeeper 3.7.1 which, at the time of this writing, was the latest version.

Inside the programs folder you made, do the command:

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.7.1/apache-zookeeper-3.7.1-bin.tar.gz

Then extract the tar:

tar -zxf apache-zookeeper-3.7.1-bin.tar.gz

Then go inside the folder it makes and do:

mkdir data

Then go inside the data folder, do pwd, copy the output into some notepad or something on your computer. This is the full path to the data folder. Then go back cd ../ and do

vi conf/zoo.cfg

Inside of it, add this stuff:

tickTime=2000
dataDir=/path/to/zookeeper/data
clientPort=2181
initLimit=5
syncLimit=2

Replace /path/to/zookeeper/data with the path you copied before into your notepad. After saving the file, do

bin/zkServer.sh start

It should say:

$ JMX enabled by default
$ Using config: /Users/../zookeeper-3.7.1/bin/../conf/zoo.cfg
$ Starting zookeeper ... STARTED

So now Zookeeper is set up. Now we can set up Kafka.

3-KAFKA.md

Install Kafka

Note: These instructions are for Kafka 2.13-3.5.1 which, at the time of this writing, was the latest version.

Go back to your programs folder, do this:

wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz

Then extract the tar again:

tar -zxf kafka_2.13-3.5.1.tgz

Then go inside the folder it made for kafka and do:

bin/kafka-server-start.sh config/server.properties

This should now start your kafka server finally. The -daemon flag can be added to allow it to run in the background.

4-PCRE.md

Install PCRE

Note: These instructions are for PCRE2-10.42 which, at the time of this writing, was the latest version.

In order to install NGINX, you need to install PCRE. Go to the programs folder you made. First things first and do pwd here and copy the full path to some notepad or something. Then, do:

wget https://github.com/PCRE2Project/pcre2/releases/download/pcre2-10.42/pcre2-10.42.tar.gz

Then extract it:

tar -xzf pcre2-10.42.tar.gz

cd into the pcre folder and type:

./configure --prefix=/path/to/your/programs/pcre-install

Remember to replace /path/to/your/programs with the path you copied to your notepad. After that, type the following commands:

make
make install

Now you're done installing PCRE.

5-NGINX.md

Install NGINX

Note: These instructions are for NGINX-1.25.1 which, at the time of this writing, was the latest version.

Go to that programs folder you made and do:

wget https://nginx.org/download/nginx-1.25.1.tar.gz

Then extract it:

tar -xzf nginx-1.25.1.tar.gz

Next, cd into the nginx folder and type:

./configure --prefix=/path/to/your/programs/nginx-install --with-pcre=/path/to/your/programs/pcre2-10.42

Remember to replace both of the "/path/to/your/programs" stuff with the path you copied to your notepad. After that, type the following commands:

make
make install

Now we need to change the port of NGINX to a different port since, because we have don't have root access, we can't use port 80. Note this isn't required if you are installing on your own computer. I change it to port 3200 but you can change it to any other port number. So, go back to the programs folder and cd into nginx-install. Then, do:

vi conf/nginx.conf

Inside it, change this line:

listen       80;

to:

listen       3200;

Now we need to make the sites-available and sites-enabled folders for Kibana in the future. So, in that same nginx.conf file, find:

include       mime.types;

below that line, type:

include       /path/to/your/programs/nginx-install/sites-enabled/*;

Remember to replace /path/to/your/programs with the path you copied to your notepad. Now you're done with the nginx.conf file. Next, do:

mkdir sites-available
mkdir sites-enabled

Now do:

nano sites-available/file.com

Inside it, add this:

server { 
listen 3200; 
server_name file.com; 

auth_basic "Restricted Access"; 
auth_basic_user_file /etc/nginx/htpasswd.users; 
location / { 
   proxy_pass http://localhost:5601; 
   proxy_http_version 1.1; 
   proxy_set_header Upgrade $http_upgrade; 
   prxy_set_header Connection 'upgrade'; 
   proxy_set_header Host $host;  
   proxy_cache_bypass $http_upgrade; 
}

If you didn't use port 3200 you need to change that in this file. Now do:

ln -s sites-available/file.com sites-enabled/file.com

Finally we are ready to run NGINX. Run this:

./sbin/nginx

Now you're done installing NGINX.

Verify NGINX Install

Let's verify that it runs.

On your computer (not UTD server), run the following command:

ssh -L 3200:localhost:3200 yournetid@cs6360.utdallas.edu

Once you type in your password, go to the browser on your computer and go to localhost:3200. It should show a "Welcome to nginx!" page. If so, you've installed NGINX successfully.

6-ELASTICSEARCH.md

Install Elasticsearch

Note: These instructions are for Elasticsearch 8.9.0 which, at the time of this writing, was the latest version.

Go to that programs folder you made and do:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.9.0-linux-x86_64.tar.gz

Then extract it:

tar -xzf elasticsearch-8.9.0-linux-x86_64.tar.gz

Next, cd into the elastic search folder it made. Then, do:

vi config/elasticsearch.yml

Go all the way to the bottom and replace everything that says true with false to disable security.

Finally, do:

./bin/elasticsearch

After a bit, you're done installing Elasticsearch.

Verify Elasticsearch Install

Let's verify that it runs.

On your computer (not UTD server), run the following command:

ssh -L 9200:localhost:9200 yournetid@cs6360.utdallas.edu

Once you type in your password, go to the browser on your computer and go to localhost:9200. It should show some json with some text saying elasticsearch on it. If so, you've installed Elasticsearch successfully. Open a new window to the UTD CS6360 server so you don't terminate Elasticsearch running.

7-KIBANA.md

Install Kibana

Note: These instructions are for Kibana 8.9.0 which, at the time of this writing, was the latest version.

Go to that programs folder you made and do:

wget https://artifacts.elastic.co/downloads/kibana/kibana-8.9.0-linux-x86_64.tar.gz

Now we're gonna extract it. WARNING this will take a LONG time (like 30 minutes). The v flag in the command is included so you can see the entire progress.

tar -xzvf kibana-8.9.0-linux-x86_64.tar.gz

Next, cd into the kibana folder and type:

vi config/kibana.yml

Now remove the # from all these lines:

...
#server.port: 5601
...
#server.host: "localhost"
...
#elasticsearch.hosts: ["http://localhost:9200"]

So it should now be:

...
server.port: 5601
...
server.host: "localhost"
...
elasticsearch.hosts: ["http://localhost:9200"]

Save the file. Now do:

./bin/kibana

It will take quite a bit to see anything output. After some time, 1 line will output saying config has been initialized. After that, you still have to wait a bit. After you see a lot more output and things become stable, you're done installing Kibana.

Verify Kibana Install

Let's verify that it runs.

On your computer (not UTD server), run the following command:

ssh -L 5601:localhost:5601 yournetid@cs6360.utdallas.edu

Once you type in your password, go to the browser on your computer and go to localhost:5601. It should show some fancy UI with the Elastic logo. This is where we can do the visualization stuff for the project. If you see that, you've installed Kibana successfully. Open a new window to the UTD CS6360 server so you don't terminate Kibana running.

8-LOGSTASH.md

Install Logstash

Note: These instructions are for Logstash 8.9.0 which, at the time of this writing, was the latest version.

Go to that programs folder you made and do:

wget https://artifacts.elastic.co/downloads/logstash/logstash-8.9.0-linux-x86_64.tar.gz

Now we're gonna extract it. WARNING this will take a bit of time (like 10 minutes). The v flag in the command is included so you can see the entire progress.

tar -xzvf logstash-8.9.0-linux-x86_64.tar.gz

Next, cd into the logstash folder. Luckily, logstash doesn't really need any special configs to test out. Run the following command:

bin/logstash -e 'input { stdin { } } output { stdout {} }'

After some time, you'll see the output stabilize.

Verify Logstash Install

To verify it's installed, simply type hello world in the console then click enter. It will output some JSON. If so, you've installed Logstash successfully.