Skip to content

Instantly share code, notes, and snippets.

@jonbrouse

jonbrouse/logslogslogs.md

Last active April 7, 2017 13:58
Show Gist options
  • Save jonbrouse/f7cbfb28f969936e92401d7cdadf7847 to your computer and use it in GitHub Desktop.
Save jonbrouse/f7cbfb28f969936e92401d7cdadf7847 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
logslogslogs.md

Shipping a Docker Container's Logs to ELK

Logstash

  • Without leveraging a broker, Logstash shouldn't be used as a shipper but it still has a place in the stack.
  • Logtash acts as an ETL agent to normalize and parse your events.
  • Additionally, Logstash can be used to pull from sources such as S3 buckets and CloudWatch logs.

Sidecars, Agents, and Drivers

  • Filebeat is the way to go if you plan to use a sidecar or ship from a mount point on your host machine.
    • Filebeat uses a back pressure-sensitive protocol that prevents your stack from being over loaded and thus events from being dropped.
  • You could leverage one of the Docker logging drivers.
    • Since your logs are streamed, the file system won't fill up with log files.
    • I'm PoCing a solution to stream logs from an AWS ECS task via the awslogs Docker logging drivers to Sumplogic.
    • Docker Container ~> CloudWatch Log Group <~ Sumo Lambda ~> Sumologic
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment