Skip to content

Instantly share code, notes, and snippets.

@rickt
Created November 9, 2018 00:02
Show Gist options
  • Save rickt/641e086d37ff7453b7ea202dc4266aa5 to your computer and use it in GitHub Desktop.
Save rickt/641e086d37ff7453b7ea202dc4266aa5 to your computer and use it in GitHub Desktop.
#
# Example fluentd config file for nginx access log injection into Google Bigquery
# collecting nginx access log
<source>
@type tail
tag accesslogs-rickts-dev-box
path /var/log/nginx/access.log
format apache
buffer_type file
buffer_path /tmp/tdagent.rickts-dev-box
pos_file /var/log/td-agent/nginx.pos
</source>
# add servername
<filter accesslogs-rickts-dev-box>
@type record_transformer
<record>
hostname ${hostname}
</record>
</filter>
# add vhost
<filter accesslogs-rickts-dev-box>
@type record_transformer
<record>
vhost rickts-dev-box.fix8r.com
</record>
</filter>
# forwarding to bigquery plugin
<match accesslogs-rickts-dev-box>
@type bigquery_insert
auth_method json_key
email 0123456789123456-compute@developer.gserviceaccount.com
json_key /path/to/your/service/account/credentials.json
project "rickts-dev-project"
dataset nginxweblogs
tables nginxweblogtable
time_format %s
time_field time
fetch_schema true
field_integer time
</match>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment