Skip to content

Instantly share code, notes, and snippets.

@AlexZeitler
Last active July 13, 2018 12:23
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save AlexZeitler/58cb5d30b004bcb0b199b0b51384b86c to your computer and use it in GitHub Desktop.
EFK
sed -e "s/?ELASTICSEARCH?/$ELASTICSEARCH/g" -e "s/?ELASTICINDEXPREFIX?/$ELASTICINDEXPREFIX/g" < /fluentd/etc/elastic.conf.template > /fluentd/etc/elastic.conf
version: "2"
services:
fluentd:
build: ../fluentd
ports:
- "24224:24224"
environment:
- ELASTICSEARCH
- ELASTICINDEXPREFIX
command: /bin/sh -c "/fluentd/config.sh && fluentd -c /fluentd/etc/fluent.conf"
service1:
depends_on:
- fluentd
image: someimage:tag
environment:
- ELASTICSEARCH
- ELASTICINDEXPREFIX
logging:
driver: fluentd
options:
fluentd-address: localhost:24224
tag: 'docker.{{.ImageName}}.{{.Name}}.{{.ID}}'
FROM fluent/fluentd:v0.12.34
USER root
COPY ./etc/* /fluentd/etc/
COPY config.sh /fluentd/
RUN apk add --update --virtual .build-deps \
sudo build-base ruby-dev \
# cutomize following instruction as you wish
&& sudo -u fluent gem install \
fluent-plugin-elasticsearch \
&& sudo -u fluent gem sources --clear-all \
&& apk del .build-deps \
&& rm -rf /var/cache/apk/* \
/home/fluent/.gem/ruby/2.3.0/cache/*.gem
USER fluent
EXPOSE 24224
@type elasticsearch
logstash_format true
hosts ?ELASTICSEARCH? #(optional; default="localhost")
#port #(optional; default=9200)
logstash_prefix ?ELASTICINDEXPREFIX? #(optional; default=logstash)
include_tag_key true
flush_interval 5
<source>
@type forward
port 24224
</source>
<filter fluent.**>
@type record_transformer
<record>
fields.module fluentd
</record>
</filter>
<match fluent.**>
@include elastic.conf
</match>
# JSON-Parse
<filter **>
@type parser
@log_level trace
format json
key_name log
hash_value_field fields
</filter>
#Don't override anything if no further parts of the tag are given or they say 'no'!
#docker.<UID> is the default tag. No override!
<match logging logging.no logging.no.no docker.** *>
@include elastic.conf
</match>
<filter logging.no.*>
@include overrideuuid.conf
</filter>
<match logging.no.*>
@include elastic.conf
</match>
<filter logging.* logging.*.no>
@include overridemodule.conf
</filter>
<match logging.* logging.*.no>
@include elastic.conf
</match>
<filter logging.**>
@include overridemodule.conf
@include overrideuuid.conf
</filter>
<match logging.**>
@include elastic.conf
</match>
@type record_transformer
<record>
fields.module ${tag_parts[1]}
</record>
@type record_transformer
<record>
fields.uuid ${tag_parts[2]}
</record>
Folder structure:
- fluentd
- etc
- elastic.conf.template
- fluent.conf
- overridemodule
- overrideuuid.conf
- Dockerfile
- config.sh
- deploy
- docker-compose.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment