Skip to content

Instantly share code, notes, and snippets.

@iMartyn
Created June 14, 2017 13:51
Show Gist options
  • Save iMartyn/dc385c99493a2b853f033bf3455b1ac4 to your computer and use it in GitHub Desktop.
Save iMartyn/dc385c99493a2b853f033bf3455b1ac4 to your computer and use it in GitHub Desktop.
fluentd config based on tectonic's config
#general
<match fluent.**>
type null
</match>
<source>
@type http
port 9880
bind 0.0.0.0
</source>
<source>
@type monitor_agent
bind 0.0.0.0
port 24220
tag fluentd.monitor.metrics
</source>
#prometheus
<source>
@type prometheus
bind 0.0.0.0
metrics_path /metrics
</source>
<source>
@type prometheus_monitor
# update the metrics every 5 seconds
interval 5
</source>
<source>
@type prometheus_output_monitor
interval 5
</source>
<source>
@type prometheus_tail_monitor
interval 5
</source>
#systemd-input
<source>
@type systemd
pos_file /var/log/fluentd-journald-systemd.pos
read_from_head true
strip_underscores true
tag systemd
</source>
#systemd-filter
<match systemd>
@type rewrite_tag_filter
rewriterule1 SYSTEMD_UNIT ^(.+).service$ systemd.$1
rewriterule2 SYSTEMD_UNIT !^(.+).service$ systemd.unmatched
</match>
#siem-filter
<filter systemd.sshd>
@type grep
regexp1 SYSTEMD_UNIT (sshd@.*\.service)
</filter>
<filter systemd.sshd>
@type record_transformer
<record>
siem_event true
</record>
</filter>
#extra
<filter **>
@type record_transformer
<record>
cluster "${name}"
</record>
</filter>
#output
<match systemd.sshd**>
@type kinesis_streams
region ${aws-region}
stream_name ${kinesis-stream}
</match>
<match **>
type aws-elasticsearch-service
log_level info
include_tag_key true
<endpoint>
url ${es-endpoint}
region eu-west-1
</endpoint>
logstash_format true
logstash_prefix k8s-${name}
template_file /fluentd/etc/elasticsearch-template-es5x.json
template_name elasticsearch-template-es5x.json
buffer_chunk_limit 2M
buffer_queue_limit 32
flush_interval 10s
max_retry_wait 30
disable_retry_limit
num_threads 8
</match>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment