Created
October 10, 2017 03:17
-
-
Save wbchn/88b6ffdc0425466877068a6eee762ed8 to your computer and use it in GitHub Desktop.
fluentd: k8s log to logzio
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# This configuration file for Fluentd / td-agent is used | |
# to watch changes to Docker log files that live in the | |
# directory /var/lib/docker/containers/ and are symbolically | |
# linked to from the /var/log directory using names that capture the | |
# pod name and container name. These logs are then submitted to | |
# Google Cloud Logging which assumes the installation of the cloud-logging plug-in. | |
# | |
# This configuration is almost identical to google-fluentd-journal.conf, with | |
# the one difference being that this doesn't try to collect systemd journal | |
# logs. | |
# | |
# Example | |
# ======= | |
# A line in the Docker log file might like like this JSON: | |
# | |
# {"log":"2014/09/25 21:15:03 Got request with path wombat\n", | |
# "stream":"stderr", | |
# "time":"2014-09-25T21:15:03.499185026Z"} | |
# | |
# The record reformer is used to write the tag to focus on the pod name | |
# and the Kubernetes container name. For example a Docker container's logs | |
# might be in the directory: | |
# /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b | |
# and in the file: | |
# 997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log | |
# where 997599971ee6... is the Docker ID of the running container. | |
# The Kubernetes kubelet makes a symbolic link to this file on the host machine | |
# in the /var/log/containers directory which includes the pod name and the Kubernetes | |
# container name: | |
# synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log | |
# -> | |
# /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log | |
# The /var/log directory on the host is mapped to the /var/log directory in the container | |
# running this instance of Fluentd and we end up collecting the file: | |
# /var/log/containers/synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log | |
# This results in the tag: | |
# var.log.containers.synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log | |
# The record reformer is used is discard the var.log.containers prefix and | |
# the Docker container ID suffix and "kubernetes." is pre-pended giving the | |
# final tag which is ingested into Elasticsearch: | |
# kubernetes.synthetic-logger-0.25lps-pod_default-synth-lgr | |
# This makes it easier for users to search for logs by pod name or by | |
# the name of the Kubernetes container regardless of how many times the | |
# Kubernetes pod has been restarted (resulting in a several Docker container IDs). | |
# Do not directly collect fluentd's own logs to avoid infinite loops. | |
<match fluent.**> | |
type null | |
</match> | |
<source> | |
type tail | |
path "#{ENV['LOG_PATH']}" | |
pos_file /var/log/containers/logz-pos-containers.log.pos | |
time_format %Y-%m-%dT%H:%M:%S.%NZ | |
keep_time_key true | |
tag kubernetes.* | |
format json | |
</source> | |
<filter kubernetes.**> | |
type kubernetes_metadata | |
</filter> | |
<match kubernetes.**> | |
type logzio_buffered | |
endpoint_url "#{ENV['LOGZIO_URL']}?token=#{ENV['LOGZIO_TOKEN']}&type=#{ENV['LOGZIO_TYPE']}" | |
output_include_time true | |
output_include_tags true | |
buffer_type file | |
buffer_path /tmp/logz-io-buffer | |
flush_interval 10s | |
buffer_chunk_limit 1m | |
</match> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment