Skip to content

Instantly share code, notes, and snippets.

@cou929
Created November 18, 2015 21:42
Show Gist options
  • Save cou929/145866169d58d0827f8a to your computer and use it in GitHub Desktop.
Save cou929/145866169d58d0827f8a to your computer and use it in GitHub Desktop.
<match mtburn.ad.*>
type forest
subtype copy
<template>
##
# S3
<store>
type s3
aws_key_id xxxx
aws_sec_key xxxx
s3_bucket mtburn-log
path ad/
buffer_type file
buffer_chunk_limit 8m
buffer_queue_limit 16
buffer_path /mtburn/tmp/td-agent/s3-buffer/${tag}
flush_interval 1m
time_slice_format %Y%m%d/%H/%Y%m%d%H_${tag}
format json
include_time_key false
include_tag_key false
</store>
##
# BigQuery
<store>
type bigquery
method insert
auth_method private_key
email xxxx
private_key_path xxxx
project xxxx
dataset mtburnlog
table ad
fetch_schema true
flush_interval 3s
try_flush_interval 0.05
queued_chunk_flush_interval 0.01
buffer_type file
buffer_chunk_limit 512k
buffer_queue_limit 1000
buffer_chunk_records_limit 300
retry_limit 10
max_retry_wait 1s
num_threads 10
buffer_path /mtburn/tmp/td-agent/bq-buffer/${tag}
</store>
##
# local
<store>
type rewrite
add_prefix local
</store>
</template>
</match>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment