Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Route Docker Logs to ELK Stack
  • With Docker 1.8.0 shipped new log-driver for GELF via UDP, this means that the logs from Docker Container(s) can be shipped directly to the ELK stack for further analysis.
  • This tutorial will illustrate how to use the GELF log-driver with Docker engine.
  • Step 1: Setup ELK Stack:
    • docker run -d --name es elasticsearch
    • docker run -d --name logstash --link es:elasticsearch logstash -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf
    • Note the config for Logstash can be found at this link
    • docker run --link es:elasticsearch -d kibana
  • Once the ELK stack is up now let's fire up our nginx container which ships its logs to ELK stack.
  • LOGSTASH_ADDRESS=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' logstash)
  • docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt gelf-tag="fe" nginx
  • All logs from the nginx container will be shipped to our ELK stack for slicing and dicing.
  • To verify that logs are being passed in visit http://<kibana-container-ip>:5601 follow through the setup and you should see the logs in Kibana. alt text

I am unable to view the logs on Kibana and also my logstash container shuts down immediately. The following is the output i get on kibana:

image

s4s0l commented Aug 30, 2016

Try without double logstash:
docker run -d --name logstash --link es:elasticsearch -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf
also config should be:

input {
  gelf {}
}
output {
  elasticsearch {
    hosts => ["elasticsearch"]
    workers=> 10
  }
  stdout {
  }
}

s4s0l commented Aug 30, 2016

i tried worked but my attempt was:

docker run --name es elasticsearch
docker run --name ls --link es:elasticsearch -v /home/sasol/Projects/betelgeuse/ELK/logstash.conf:/config-dir/logstash.conf  logstash logstash -f /config-dir/logstash.conf
docker run --link es:elasticsearch -p 5601:5601 kibana
docker run  --log-driver=gelf --log-opt gelf-address=udp://$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' ls):12201 --log-opt tag="test" alpine /bin/sh -c "while truedo echo  My message \$RANDOM; sleep 1; done;"

i changed it but still my logstash container shuts down. the minute i refresh kibana

image

s4s0l commented Aug 30, 2016

try the --verbose or even --debug switches, i dont think its kibana related.

z-vr commented Oct 23, 2016 edited

could you please explain what 2 logstash in -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf do?

girokon commented Oct 28, 2016

@z-vr It mounts config inside container, then first logstash is name of image which we run, and then logstash -f /config-dir/logstash.conf is command which we run inside container

uudashr commented Nov 4, 2016

This is works for me

docker run -d --name elastic elasticsearch
docker run -d --name logstash --link elastic:elasticsearch -v /tmp/logstash.conf:/config-dir/logstash.conf logstash -f /config-dir/logstash.conf
docker run -d --name kibana --link elastic:elasticsearch -p 5601:5601 kibana
docker run --rm --log-driver=gelf --log-opt gelf-address=udp://$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' logstash):12201 --log-opt tag="test" alpine /bin/sh -c "while true; do echo My Message \$RANDOM; sleep 1; done;"

the most important is logstash.conf, do not use worker, since it is not supported anymore.

input {
  gelf { }
}

output {
  elasticsearch {
    hosts => ["elasticsearch"]
  }
  stdout { }
}

a1exus commented Nov 16, 2016

while trying to run:

docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt gelf-tag="fe" nginx

I'm getting following error:

docker: Error response from daemon: unknown log opt "gelf-tag" for gelf log driver.

$ docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Thu Oct 27 00:09:21 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Thu Oct 27 00:09:21 2016
 OS/Arch:      linux/amd64
 Experimental: true
$ 

Please advise, Thank you!

@a1exus gelf log-driver supports tags by providing tag flag, so you should alter the command in the following manner:
docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt tag="fe" nginx

estabij commented Dec 16, 2016

This works for me:
docker run -d --name es elasticsearch

docker run -d --name logstash --link es:elasticsearch logstash -v "$PWD":/config-dir -f /config-dir/logstash.conf

docker run --link es:elasticsearch -d kibana

docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://127.0.0.1:12201 --log-opt tag="fe" nginx

logstash.conf:

input {
gelf { }
}

output {
elasticsearch {
hosts => ["elasticsearch"]
}
stdout { }
}

tol182 commented Dec 28, 2016

Can someone explain, how not to hard-code elastic container ip in logstash.conf?

logstash container is restarting.

Logstatsh process inside the container

root@4e14335c9d93:/# ps -ef | grep logstash
logstash     1     0 47 13:25 ?        00:00:08 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb -v /home/vagrant:/config-dir -f /config-dir/logstash-gelf.conf

I queried the docker logs. I see logstash is not recognizing -v option

vagrant@PerfQual-host:~$ docker logs 4e14335c9d937874715d63c9a6413a33749913f1339f964c5dac8ef0cdb78426
ERROR: Unrecognised option '-v'

See: 'bin/logstash --help'
ERROR: Unrecognised option '-v'

See: 'bin/logstash --help'
vagrant@PerfQual-host:~$

HackerWilson commented Feb 24, 2017 edited

I have written the docker-elk-deployment project to simplify these steps, it supports the Elastic Stack 5.2.0+ on swarm mode cluster, and use gelf logging driver to gathering logs from docker containers, anyone who has problems with these steps can go to https://github.com/HackerWilson/docker-elk-deployment and have a try.

Is there any way to increase the size of logs logstash can take? Logstash is breaking apart my logs into multiple messages which then fails to parse.

@AlecBruns see logstash-plugins/logstash-input-gelf#37 and docker/docker#22920 and docker/docker#22979

You can't use docker logging if you want to parse multiple lines.

trajano commented Jun 2, 2017

I found that this does not work when using docker-compose has anyone else had luck? https://forums.docker.com/t/docker-loading-in-stack/33051

qubusp commented Jun 12, 2017 edited

Can I use this to send logs directly in ELK and keep them in the journal of the host?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment