Ubuntu 14.04. Docker -> ELK (Logstash + Elasticsearch + Kibana for your Apache Logs)

ubuntu_elk_apache.md

Ubuntu info

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.4 LTS
Release:	14.04
Codename:	trusty

Installing docker

Make sure this file exists:

cat /etc/apt/sources.list.d/docker.list 
deb https://apt.dockerproject.org/repo ubuntu-trusty main

Installation:

apt-get purge lxc-docker
apt-cache policy docker-engine
apt-get update && apt-get install docker-engine

Make sure that docker is running:

service docker start
docker run hello-world

Add a non-root user to the docker group:

usermod -aG docker <user>

Now install docker-compose:

curl -L https://github.com/docker/compose/releases/download/1.6.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Docker containers

As the user you just added to the docker group. Elasticsearch and Kibana docker compose recipe:

elasticsearch-int:
  image: elasticsearch
  ports:
    - 9200:9200

kibana-int:
  image: kibana
  ports:
    - 5601:5601
  environment:
    - ELASTICSEARCH_URL=http://elasticsearch-int:9200
  links:
    - elasticsearch-int

Both app will be running on your public IP. If you wanna restrict it to your host "localhost", modify ports pointing to your server internal IP:

elasticsearch-int:
  ports:
    - 10.132.136.138:9200:9200
...
kibana-int:
...
    - 10.132.136.138:5601:5601
...

Logstash Container

Create the following configuration file in your current directory (copied from this guide):

# logstash.conf
input {
    file {
        path => '/var/log/apache2/access.log'
    }
}

filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
}

output {
    elasticsearch {
        hosts => ["10.132.136.138:9200"]
    }
}

Run the container:

docker run -d -it -v "$PWD":/config-dir logstash logstash -f /config-dir/logstash.conf