Create a gist now

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Route Docker Logs to ELK Stack
  • With Docker 1.8.0 shipped new log-driver for GELF via UDP, this means that the logs from Docker Container(s) can be shipped directly to the ELK stack for further analysis.
  • This tutorial will illustrate how to use the GELF log-driver with Docker engine.
  • Step 1: Setup ELK Stack:
    • docker run -d --name es elasticsearch
    • docker run -d --name logstash --link es:elasticsearch logstash -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf
    • Note the config for Logstash can be found at this link
    • docker run --link es:elasticsearch -d kibana
  • Once the ELK stack is up now let's fire up our nginx container which ships its logs to ELK stack.
  • LOGSTASH_ADDRESS=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' logstash)
  • docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt gelf-tag="fe" nginx
  • All logs from the nginx container will be shipped to our ELK stack for slicing and dicing.
  • To verify that logs are being passed in visit http://<kibana-container-ip>:5601 follow through the setup and you should see the logs in Kibana. alt text
@taniabhatia

This comment has been minimized.

Show comment
Hide comment
@taniabhatia

taniabhatia Aug 30, 2016

I am unable to view the logs on Kibana and also my logstash container shuts down immediately. The following is the output i get on kibana:

image

I am unable to view the logs on Kibana and also my logstash container shuts down immediately. The following is the output i get on kibana:

image

@s4s0l

This comment has been minimized.

Show comment
Hide comment
@s4s0l

s4s0l Aug 30, 2016

Try without double logstash:
docker run -d --name logstash --link es:elasticsearch -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf
also config should be:

input {
  gelf {}
}
output {
  elasticsearch {
    hosts => ["elasticsearch"]
    workers=> 10
  }
  stdout {
  }
}

s4s0l commented Aug 30, 2016

Try without double logstash:
docker run -d --name logstash --link es:elasticsearch -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf
also config should be:

input {
  gelf {}
}
output {
  elasticsearch {
    hosts => ["elasticsearch"]
    workers=> 10
  }
  stdout {
  }
}
@s4s0l

This comment has been minimized.

Show comment
Hide comment
@s4s0l

s4s0l Aug 30, 2016

i tried worked but my attempt was:

docker run --name es elasticsearch
docker run --name ls --link es:elasticsearch -v /home/sasol/Projects/betelgeuse/ELK/logstash.conf:/config-dir/logstash.conf  logstash logstash -f /config-dir/logstash.conf
docker run --link es:elasticsearch -p 5601:5601 kibana
docker run  --log-driver=gelf --log-opt gelf-address=udp://$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' ls):12201 --log-opt tag="test" alpine /bin/sh -c "while truedo echo  My message \$RANDOM; sleep 1; done;"

s4s0l commented Aug 30, 2016

i tried worked but my attempt was:

docker run --name es elasticsearch
docker run --name ls --link es:elasticsearch -v /home/sasol/Projects/betelgeuse/ELK/logstash.conf:/config-dir/logstash.conf  logstash logstash -f /config-dir/logstash.conf
docker run --link es:elasticsearch -p 5601:5601 kibana
docker run  --log-driver=gelf --log-opt gelf-address=udp://$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' ls):12201 --log-opt tag="test" alpine /bin/sh -c "while truedo echo  My message \$RANDOM; sleep 1; done;"

@taniabhatia

This comment has been minimized.

Show comment
Hide comment
@taniabhatia

taniabhatia Aug 30, 2016

i changed it but still my logstash container shuts down. the minute i refresh kibana

image

i changed it but still my logstash container shuts down. the minute i refresh kibana

image

@s4s0l

This comment has been minimized.

Show comment
Hide comment
@s4s0l

s4s0l Aug 30, 2016

try the --verbose or even --debug switches, i dont think its kibana related.

s4s0l commented Aug 30, 2016

try the --verbose or even --debug switches, i dont think its kibana related.

@z-vr

This comment has been minimized.

Show comment
Hide comment
@z-vr

z-vr Oct 23, 2016

could you please explain what 2 logstash in -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf do?

z-vr commented Oct 23, 2016

could you please explain what 2 logstash in -v /tmp/logstash.conf:/config-dir/logstash.conf logstash logstash -f /config-dir/logstash.conf do?

@girokon

This comment has been minimized.

Show comment
Hide comment
@girokon

girokon Oct 28, 2016

@z-vr It mounts config inside container, then first logstash is name of image which we run, and then logstash -f /config-dir/logstash.conf is command which we run inside container

girokon commented Oct 28, 2016

@z-vr It mounts config inside container, then first logstash is name of image which we run, and then logstash -f /config-dir/logstash.conf is command which we run inside container

@uudashr

This comment has been minimized.

Show comment
Hide comment
@uudashr

uudashr Nov 4, 2016

This is works for me

docker run -d --name elastic elasticsearch
docker run -d --name logstash --link elastic:elasticsearch -v /tmp/logstash.conf:/config-dir/logstash.conf logstash -f /config-dir/logstash.conf
docker run -d --name kibana --link elastic:elasticsearch -p 5601:5601 kibana
docker run --rm --log-driver=gelf --log-opt gelf-address=udp://$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' logstash):12201 --log-opt tag="test" alpine /bin/sh -c "while true; do echo My Message \$RANDOM; sleep 1; done;"

the most important is logstash.conf, do not use worker, since it is not supported anymore.

input {
  gelf { }
}

output {
  elasticsearch {
    hosts => ["elasticsearch"]
  }
  stdout { }
}

uudashr commented Nov 4, 2016

This is works for me

docker run -d --name elastic elasticsearch
docker run -d --name logstash --link elastic:elasticsearch -v /tmp/logstash.conf:/config-dir/logstash.conf logstash -f /config-dir/logstash.conf
docker run -d --name kibana --link elastic:elasticsearch -p 5601:5601 kibana
docker run --rm --log-driver=gelf --log-opt gelf-address=udp://$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' logstash):12201 --log-opt tag="test" alpine /bin/sh -c "while true; do echo My Message \$RANDOM; sleep 1; done;"

the most important is logstash.conf, do not use worker, since it is not supported anymore.

input {
  gelf { }
}

output {
  elasticsearch {
    hosts => ["elasticsearch"]
  }
  stdout { }
}
@a1exus

This comment has been minimized.

Show comment
Hide comment
@a1exus

a1exus Nov 16, 2016

while trying to run:

docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt gelf-tag="fe" nginx

I'm getting following error:

docker: Error response from daemon: unknown log opt "gelf-tag" for gelf log driver.

$ docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Thu Oct 27 00:09:21 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Thu Oct 27 00:09:21 2016
 OS/Arch:      linux/amd64
 Experimental: true
$ 

Please advise, Thank you!

a1exus commented Nov 16, 2016

while trying to run:

docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt gelf-tag="fe" nginx

I'm getting following error:

docker: Error response from daemon: unknown log opt "gelf-tag" for gelf log driver.

$ docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Thu Oct 27 00:09:21 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Thu Oct 27 00:09:21 2016
 OS/Arch:      linux/amd64
 Experimental: true
$ 

Please advise, Thank you!

@reflectiondm

This comment has been minimized.

Show comment
Hide comment
@reflectiondm

reflectiondm Nov 25, 2016

@a1exus gelf log-driver supports tags by providing tag flag, so you should alter the command in the following manner:
docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt tag="fe" nginx

@a1exus gelf log-driver supports tags by providing tag flag, so you should alter the command in the following manner:
docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://$LOGSTASH_ADDRESS:12201 --log-opt tag="fe" nginx

@estabij

This comment has been minimized.

Show comment
Hide comment
@estabij

estabij Dec 16, 2016

This works for me:
docker run -d --name es elasticsearch

docker run -d --name logstash --link es:elasticsearch logstash -v "$PWD":/config-dir -f /config-dir/logstash.conf

docker run --link es:elasticsearch -d kibana

docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://127.0.0.1:12201 --log-opt tag="fe" nginx

logstash.conf:

input {
gelf { }
}

output {
elasticsearch {
hosts => ["elasticsearch"]
}
stdout { }
}

estabij commented Dec 16, 2016

This works for me:
docker run -d --name es elasticsearch

docker run -d --name logstash --link es:elasticsearch logstash -v "$PWD":/config-dir -f /config-dir/logstash.conf

docker run --link es:elasticsearch -d kibana

docker run -d --net=host --log-driver=gelf --log-opt gelf-address=udp://127.0.0.1:12201 --log-opt tag="fe" nginx

logstash.conf:

input {
gelf { }
}

output {
elasticsearch {
hosts => ["elasticsearch"]
}
stdout { }
}

@tol182

This comment has been minimized.

Show comment
Hide comment
@tol182

tol182 Dec 28, 2016

Can someone explain, how not to hard-code elastic container ip in logstash.conf?

tol182 commented Dec 28, 2016

Can someone explain, how not to hard-code elastic container ip in logstash.conf?

@nsphaniraj

This comment has been minimized.

Show comment
Hide comment
@nsphaniraj

nsphaniraj Dec 28, 2016

logstash container is restarting.

Logstatsh process inside the container

root@4e14335c9d93:/# ps -ef | grep logstash
logstash     1     0 47 13:25 ?        00:00:08 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb -v /home/vagrant:/config-dir -f /config-dir/logstash-gelf.conf

I queried the docker logs. I see logstash is not recognizing -v option

vagrant@PerfQual-host:~$ docker logs 4e14335c9d937874715d63c9a6413a33749913f1339f964c5dac8ef0cdb78426
ERROR: Unrecognised option '-v'

See: 'bin/logstash --help'
ERROR: Unrecognised option '-v'

See: 'bin/logstash --help'
vagrant@PerfQual-host:~$

logstash container is restarting.

Logstatsh process inside the container

root@4e14335c9d93:/# ps -ef | grep logstash
logstash     1     0 47 13:25 ?        00:00:08 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb -v /home/vagrant:/config-dir -f /config-dir/logstash-gelf.conf

I queried the docker logs. I see logstash is not recognizing -v option

vagrant@PerfQual-host:~$ docker logs 4e14335c9d937874715d63c9a6413a33749913f1339f964c5dac8ef0cdb78426
ERROR: Unrecognised option '-v'

See: 'bin/logstash --help'
ERROR: Unrecognised option '-v'

See: 'bin/logstash --help'
vagrant@PerfQual-host:~$
@HackerWilson

This comment has been minimized.

Show comment
Hide comment
@HackerWilson

HackerWilson Feb 24, 2017

I have written the docker-elk-deployment project to simplify these steps, it supports the Elastic Stack 5.2.0+ on swarm mode cluster, and use gelf logging driver to gathering logs from docker containers, anyone who has problems with these steps can go to https://github.com/HackerWilson/docker-elk-deployment and have a try.

HackerWilson commented Feb 24, 2017

I have written the docker-elk-deployment project to simplify these steps, it supports the Elastic Stack 5.2.0+ on swarm mode cluster, and use gelf logging driver to gathering logs from docker containers, anyone who has problems with these steps can go to https://github.com/HackerWilson/docker-elk-deployment and have a try.

@AlecBruns

This comment has been minimized.

Show comment
Hide comment
@AlecBruns

AlecBruns Mar 13, 2017

Is there any way to increase the size of logs logstash can take? Logstash is breaking apart my logs into multiple messages which then fails to parse.

Is there any way to increase the size of logs logstash can take? Logstash is breaking apart my logs into multiple messages which then fails to parse.

@caduvieira

This comment has been minimized.

Show comment
Hide comment
@caduvieira

caduvieira Apr 5, 2017

@AlecBruns see logstash-plugins/logstash-input-gelf#37 and docker/docker#22920 and docker/docker#22979

You can't use docker logging if you want to parse multiple lines.

@AlecBruns see logstash-plugins/logstash-input-gelf#37 and docker/docker#22920 and docker/docker#22979

You can't use docker logging if you want to parse multiple lines.

@trajano

This comment has been minimized.

Show comment
Hide comment
@trajano

trajano Jun 2, 2017

I found that this does not work when using docker-compose has anyone else had luck? https://forums.docker.com/t/docker-loading-in-stack/33051

trajano commented Jun 2, 2017

I found that this does not work when using docker-compose has anyone else had luck? https://forums.docker.com/t/docker-loading-in-stack/33051

@qubusp

This comment has been minimized.

Show comment
Hide comment
@qubusp

qubusp Jun 12, 2017

Can I use this to send logs directly in ELK and keep them in the journal of the host?

qubusp commented Jun 12, 2017

Can I use this to send logs directly in ELK and keep them in the journal of the host?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment