Skip to content

Instantly share code, notes, and snippets.

@cdgraff
Last active May 15, 2019 21:40
Show Gist options
  • Save cdgraff/b261154a0e75be80c02325badefd94bc to your computer and use it in GitHub Desktop.
Save cdgraff/b261154a0e75be80c02325badefd94bc to your computer and use it in GitHub Desktop.
BigQuery + Icecast2 logs
[ { "type": "TIMESTAMP", "name": "time" },
{ "type": "STRING", "name": "host" },
{ "type": "STRING", "name": "user" },
{ "type": "STRING", "name": "method" },
{ "type": "STRING", "name": "path" },
{ "type": "STRING", "name": "code" },
{ "type": "INTEGER", "name": "size" },
{ "type": "STRING", "name": "referer" },
{ "type": "STRING", "name": "agent" },
{ "type": "STRING", "name": "duration" } ]
<source>
@type tail
# Need this to load older logs
read_from_head true
path /var/log/icecast/access.log
pos_file /var/log/td-agent/icecast2.access_log.pos
<parse>
@type regexp
expression ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")? (?<duration>[^ ]*)$
time_format %d/%b/%Y:%H:%M:%S %z
</parse>
tag bq.icecast2.access
</source>
<match bq.icecast2.access>
@type bigquery_insert
# Authenticate with BigQuery using the VM's service account, this run inside GCP vm
auth_method compute_engine
project [MY_PROJECT_NAME]
dataset fluentd
table icecast2
fetch_schema true
<inject>
# Convert fluentd timestamp into TIMESTAMP string
time_key time
time_type string
time_format %Y-%m-%dT%H:%M:%S.%NZ
</inject>
</match>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment