Stack ELK

ELK

Stack ELK
=========

 The ELK stack contains Elasticsearch, Logstash, and Kibana:
    - Elasticsearch for deep search and data analytics
    - Logstash for centralized logging, log enrichment and parsing
    - Kibana for powerful and beautiful data visualizations

 The architecture ELK:
 
           send data              storage data                           reads data                             
  Data/Log ----------> [Logstash] ------------> [ElasticSearch ip:port] <----------- [{nginx:80}/Kibana]
  
Logstash
  Managing events
  Collect data
  Parse data
  Enrich data
  Store data
  
  Logstash architecture:
    Input  ------------> Filter -------------> Output
    
    File pattern:
    Input  [logs file, stream, datastore, files, monitoring, queues, network]
    Filter [parse, enrich, tag, drop]
    Output  [email, datastore, files, monitoring, queues, API, paper]
   
Elasticsearch
  Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine. Distributed RESTful search and analytics.
  
  
Kibana
  visualize logs and time-stamped data.
  Elasticsearch works seamlessly with Kibana to let you see and interact with your data.
  
  
Implemetation:

1 - Install java
  # 


2 - Install Elasticsearch
  # wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.tar.gz
  # tar zxvf elasticsearch-1.1.1.tar.gz
  # cd elasticsearch-1.1.1/
  # ./bin/elasticsearch 

3 - Install Logstash
  # wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.1.tar.gz
  # tar -zxvf logstash-1.4.1.tar.gz 
   
  Teste: 
  # cd logstash-1.4.1/bin
  # ./logstash -e 'input { stdin { } } output { stdout {} }' 
   
  Then type a message in the terminal, for example "hello world". See the output as follows:
  # hello world
      output: "2013-11-21T01:22:14.405+0000 0.0.0.0 hello world" 
   
  # CTRL +C (to exit)
   
  # vim first-log.conf
      ------------------------------------------------------------------------------------------ 
      input {
        file {
          path => ["/var/log/messages", "/var/log/*", "/d2/java/jboss-eap-6.3/standalone/log/*"] 
        }
      }
      output {
        elasticsearch { host => localhost }  # user "localhost" or "elasticsearch_ip_address".
        stdout { codec => rubydebug }
      }
      ------------------------------------------------------------------------------------------
      
  # ./logstash -f first-log.conf &
  
  You will see something like:
     ------------------------------------------------------------------------------------------
     {
       "message" => "Dec 23 15:10:01 srv238 CROND[28897]: (root) CMD (/usr/lib64/sa/sa1 1 1)",
       "@version" => "1",
       "@timestamp" => "2014-12-23T17:10:02.060Z",
          "host" => "srv238.cnpq.br",
          "path" => "/var/log/cron"
     }
     {
      "message" => "Dec 23 15:20:01 srv238 systemd: Created slice user-0.slice.",
      "@version" => "1",
      "@timestamp" => "2014-12-23T17:20:02.479Z",
          "host" => "srv238.xxx.xx",
          "path" => "/var/log/messages"
     }
     ------------------------------------------------------------------------------------------
     
4 - Install Kibana (on APACHE or NGINX) 
  
  with apache:
  # cd /var/www/html
  # wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz
  # tar -xzvf kibana-3.1.0.tar.gz
  # mv kibana-3.1.0 kibana 
  
  with nginx:
  # cd /usr/share/nginx/html
  # wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz
  # tar -xzvf kibana-3.1.0.tar.gz
  # mv kibana-3.1.0 kibana 

  Access URL: <your_ip_address>/kibana