Elasticsearch, Logstash, Kibana, Centos 7, Firewalld

ELK.md

Install ELK(Elasticsearch, Logstatsh, and Kibana) on Centos 7

Install java 1.8

• download and install java

cd /opt
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u102-b14/jre-8u102-linux-x64.rpm"
rpm -Uvh jre-8u102-linux-x64.rpm

• look java version

java -version

Install Elasticsearch

• import Elasticsearch PGP key

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

• create a new repository file Elasticsearch

cat > /etc/yum.repos.d/elasticsearch.repo<<EOF
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

• install the Elasticsearch package

yum -y install elasticsearch

• Start and enable the service

systemctl daemon-reload
systemctl enable elasticsearch
systemctl start elasticsearch

• Allow traffic through TCP port 9200 in your firewall

firewall-cmd --add-port=9200/tcp
firewall-cmd --add-port=9200/tcp --permanent

• Check if Elasticsearch responds

curl -X GET http://localhost:9200

Install logstash

• create a new repository file logstash

cat > /etc/yum.repos.d/logstash.repo<<EOF
[logstash-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

• Install the Logstash

yum -y install logstash

• Add a SSL certificate based on the IP address of the ELK server

vim /etc/pki/tls/openssl.cnf

• add below [ v3_ca ]

subjectAltName = IP: 192.168.0.29 

• Generate a self-signed certificate valid for 365 days

cd /etc/pki/tls
openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

• Configure Logstash input, output, and filter files

cat > /etc/logstash/conf.d/logstash.conf<<EOF
# input file
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}
# output file
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
# filter file
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGLINE}" }
    }
    date {
      match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
EOF

• Verify the Logstash configuration files

service logstash configtest

• Start and enable logstash

systemctl daemon-reload
systemctl start logstash
systemctl enable logstash

• Configure the firewall to allow Logstash

firewall-cmd --add-port=5044/tcp
firewall-cmd --add-port=5044/tcp --permanent

Install Kibana

• create a new repository file Kibana

cat > /etc/yum.repos.d/kibana.repo<<EOF
[kibana]
name=Kibana repository
baseurl=http://packages.elastic.co/kibana/4.4/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
EOF

• Install the Kibana package

yum -y install kibana

• Start and enable Kibana

systemctl daemon-reload
systemctl start kibana
systemctl enable kibana

• Configure the firewall to allow kibana

firewall-cmd --add-port=5601/tcp
firewall-cmd --add-port=5601/tcp --permanent

Install Filebeat on the Client Servers

• add folder

mkdir -p /etc/pki/tls/certs

• Copy the SSL certificate from the server to the clients

scp /etc/pki/tls/certs/logstash-forwarder.crt user@client_server_private_address:/etc/pki/tls/certs/

• import the elastic key on the client1 server

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

-Import the Elasticsearch public GPG key to the rpm package manager on the client1 server

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

• Download Filebeat and install it with rpm

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.4-x86_64.rpm
rpm -vi filebeat-5.6.4-x86_64.rpm

• edit Filebeat configuration file

vim /etc/filebeat/filebeat.yml

• We will add two files '/var/log/secure' for ssh activity and '/var/log/messages' for the server log, and comment out the - /var/log/*.log file

paths:
   - /var/log/secure
   - /var/log/messages
 # - /var/log/*.log

• Then find the line that specifies document_type:, uncomment it and change its value to "syslog"

document_type: syslog

• Filebeat is using Elasticsearch as the output target by default. In this tutorial, we will change it to Logshtash. Disable Elasticsearch output by adding comments on the lines 83 and 85

#output.elasticsearch:
 # Array of hosts to connect to.
 #  hosts: ["localhost:9200"]

• Now add the new logstash output configuration. Uncomment the logstash output configuration and change all value to the configuration that is shown below

### Logstash as output
logstash:
    # The Logstash hosts
    hosts: ["ELK_server_private_IP:5044"]
    bulk_max_size: 1024
    tls:
      # List of root certificates for HTTPS server verifications
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

Save the file and exit vim

• Now start and enable Filebeat to put our changes into place

systemctl enable filebeat
systemctl start filebeat

• Test Filebeat Installation, On your ELK Server

curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

• Launch Kibana (http://ELK_server_private_IP:5601)