Skip to content

Instantly share code, notes, and snippets.

@amanualt amanualt/ELK.md
Last active Sep 9, 2019

Embed
What would you like to do?
Elasticsearch, Logstash, Kibana, Centos 7, Firewalld

Install ELK(Elasticsearch, Logstatsh, and Kibana) on Centos 7

Install java 1.8

  • download and install java
cd /opt
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u102-b14/jre-8u102-linux-x64.rpm"
rpm -Uvh jre-8u102-linux-x64.rpm
  • look java version
java -version

Install Elasticsearch

  • import Elasticsearch PGP key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
  • create a new repository file Elasticsearch
cat > /etc/yum.repos.d/elasticsearch.repo<<EOF
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
  • install the Elasticsearch package
yum -y install elasticsearch
  • Start and enable the service
systemctl daemon-reload
systemctl enable elasticsearch
systemctl start elasticsearch
  • Allow traffic through TCP port 9200 in your firewall
firewall-cmd --add-port=9200/tcp
firewall-cmd --add-port=9200/tcp --permanent
  • Check if Elasticsearch responds
curl -X GET http://localhost:9200

Install logstash

  • create a new repository file logstash
cat > /etc/yum.repos.d/logstash.repo<<EOF
[logstash-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
  • Install the Logstash
yum -y install logstash
  • Add a SSL certificate based on the IP address of the ELK server
vim /etc/pki/tls/openssl.cnf
  • add below [ v3_ca ]
subjectAltName = IP: 192.168.0.29 
  • Generate a self-signed certificate valid for 365 days
cd /etc/pki/tls
openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
  • Configure Logstash input, output, and filter files
cat > /etc/logstash/conf.d/logstash.conf<<EOF
# input file
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}
# output file
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
# filter file
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGLINE}" }
    }
    date {
      match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
EOF
  • Verify the Logstash configuration files
service logstash configtest
  • Start and enable logstash
systemctl daemon-reload
systemctl start logstash
systemctl enable logstash
  • Configure the firewall to allow Logstash
firewall-cmd --add-port=5044/tcp
firewall-cmd --add-port=5044/tcp --permanent

Install Kibana

  • create a new repository file Kibana
cat > /etc/yum.repos.d/kibana.repo<<EOF
[kibana]
name=Kibana repository
baseurl=http://packages.elastic.co/kibana/4.4/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
EOF
  • Install the Kibana package
yum -y install kibana
  • Start and enable Kibana
systemctl daemon-reload
systemctl start kibana
systemctl enable kibana
  • Configure the firewall to allow kibana
firewall-cmd --add-port=5601/tcp
firewall-cmd --add-port=5601/tcp --permanent

Install Filebeat on the Client Servers

  • add folder
mkdir -p /etc/pki/tls/certs
  • Copy the SSL certificate from the server to the clients
scp /etc/pki/tls/certs/logstash-forwarder.crt user@client_server_private_address:/etc/pki/tls/certs/
  • import the elastic key on the client1 server
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

-Import the Elasticsearch public GPG key to the rpm package manager on the client1 server

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
  • Download Filebeat and install it with rpm
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.4-x86_64.rpm
rpm -vi filebeat-5.6.4-x86_64.rpm
  • edit Filebeat configuration file
vim /etc/filebeat/filebeat.yml
  • We will add two files '/var/log/secure' for ssh activity and '/var/log/messages' for the server log, and comment out the - /var/log/*.log file
paths:
   - /var/log/secure
   - /var/log/messages
 # - /var/log/*.log
  • Then find the line that specifies document_type:, uncomment it and change its value to "syslog"
document_type: syslog
  • Filebeat is using Elasticsearch as the output target by default. In this tutorial, we will change it to Logshtash. Disable Elasticsearch output by adding comments on the lines 83 and 85
#output.elasticsearch:
 # Array of hosts to connect to.
 #  hosts: ["localhost:9200"]
  • Now add the new logstash output configuration. Uncomment the logstash output configuration and change all value to the configuration that is shown below
### Logstash as output
logstash:
    # The Logstash hosts
    hosts: ["ELK_server_private_IP:5044"]
    bulk_max_size: 1024
    tls:
      # List of root certificates for HTTPS server verifications
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

Save the file and exit vim

  • Now start and enable Filebeat to put our changes into place
systemctl enable filebeat
systemctl start filebeat
  • Test Filebeat Installation, On your ELK Server
curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
  • Launch Kibana (http://ELK_server_private_IP:5601)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.