Route Docker Logs to ELK Stack

Instructions.md

• With Docker 1.8.0 shipped new log-driver for GELF via UDP, this means that the logs from Docker Container(s) can be shipped directly to the ELK stack for further analysis.
• This tutorial will illustrate how to use the GELF log-driver with Docker engine.
• Step 1: Setup ELK Stack:
  • docker run -d --name es elasticsearch
  • docker run -d --name logstash --link es:elasticsearch logstash -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf
  • Note the config for Logstash can be found at this link
  • docker run --link es:elasticsearch -d kibana
• Once the ELK stack is up now let's fire up our nginx container which ships its logs to ELK stack.

• LOGSTASH_ADDRESS=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' logstash)
• docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt gelf-tag="fe" nginx
• All logs from the nginx container will be shipped to our ELK stack for slicing and dicing.
• To verify that logs are being passed in visit http://<kibana-container-ip>:5601 follow through the setup and you should see the logs in Kibana.

I am unable to view the logs on Kibana and also my logstash container shuts down immediately. The following is the output i get on kibana:

Try without double logstash:
docker run -d --name logstash --link es:elasticsearch -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf
also config should be:

input {
  gelf {}
}
output {
  elasticsearch {
    hosts => ["elasticsearch"]
    workers=> 10
  }
  stdout {
  }
}

i tried worked but my attempt was:

docker run --name es elasticsearch
docker run --name ls --link es:elasticsearch -v /home/sasol/Projects/betelgeuse/ELK/logstash.conf:/config-dir/logstash.conf  logstash logstash -f /config-dir/logstash.conf
docker run --link es:elasticsearch -p 5601:5601 kibana
docker run  --log-driver=gelf --log-opt gelf-address=udp://$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' ls):12201 --log-opt tag="test" alpine /bin/sh -c "while truedo echo  My message \$RANDOM; sleep 1; done;"

i changed it but still my logstash container shuts down. the minute i refresh kibana

try the --verbose or even --debug switches, i dont think its kibana related.

could you please explain what 2 logstash in -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf do?

@z-vr It mounts config inside container, then first logstash is name of image which we run, and then logstash -f /config-dir/logstash.conf is command which we run inside container

This is works for me

docker run -d --name elastic elasticsearch
docker run -d --name logstash --link elastic:elasticsearch -v /tmp/logstash.conf:/config-dir/logstash.conf logstash -f /config-dir/logstash.conf
docker run -d --name kibana --link elastic:elasticsearch -p 5601:5601 kibana
docker run --rm --log-driver=gelf --log-opt gelf-address=udp://$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' logstash):12201 --log-opt tag="test" alpine /bin/sh -c "while true; do echo My Message \$RANDOM; sleep 1; done;"

the most important is logstash.conf, do not use worker, since it is not supported anymore.

input {
  gelf { }
}

output {
  elasticsearch {
    hosts => ["elasticsearch"]
  }
  stdout { }
}

while trying to run:

docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt gelf-tag="fe" nginx

I'm getting following error:

docker: Error response from daemon: unknown log opt "gelf-tag" for gelf log driver.

$ docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Thu Oct 27 00:09:21 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Thu Oct 27 00:09:21 2016
 OS/Arch:      linux/amd64
 Experimental: true
$ 

Please advise, Thank you!

@a1exus gelf log-driver supports tags by providing tag flag, so you should alter the command in the following manner:
docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt tag="fe" nginx

This works for me:
docker run -d --name es elasticsearch

docker run -d --name logstash --link es:elasticsearch logstash -v "$PWD":/config-dir -f /config-dir/logstash.conf

docker run --link es:elasticsearch -d kibana

docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://127.0.0.1:12201 --log-opt tag="fe" nginx

logstash.conf:

input {
gelf { }
}

output {
elasticsearch {
hosts => ["elasticsearch"]
}
stdout { }
}

Can someone explain, how not to hard-code elastic container ip in logstash.conf?

logstash container is restarting.

Logstatsh process inside the container

root@4e14335c9d93:/# ps -ef | grep logstash
logstash     1     0 47 13:25 ?        00:00:08 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb -v /home/vagrant:/config-dir -f /config-dir/logstash-gelf.conf

I queried the docker logs. I see logstash is not recognizing -v option

vagrant@PerfQual-host:~$ docker logs 4e14335c9d937874715d63c9a6413a33749913f1339f964c5dac8ef0cdb78426
ERROR: Unrecognised option '-v'

See: 'bin/logstash --help'
ERROR: Unrecognised option '-v'

See: 'bin/logstash --help'
vagrant@PerfQual-host:~$

I have written the docker-elk-deployment project to simplify these steps, it supports the Elastic Stack 5.2.0+ on swarm mode cluster, and use gelf logging driver to gathering logs from docker containers, anyone who has problems with these steps can go to https://github.com/HackerWilson/docker-elk-deployment and have a try.

Is there any way to increase the size of logs logstash can take? Logstash is breaking apart my logs into multiple messages which then fails to parse.

@AlecBruns see logstash-plugins/logstash-input-gelf#37 and moby/moby#22920 and moby/moby#22979

You can't use docker logging if you want to parse multiple lines.

I found that this does not work when using docker-compose has anyone else had luck? https://forums.docker.com/t/docker-loading-in-stack/33051

Can I use this to send logs directly in ELK and keep them in the journal of the host?

Here's a working verision which only requires docker-compose.
https://github.com/amalic/nginxelk

Please let me know what you think.