Skip to content

Instantly share code, notes, and snippets.

@yukiyan
Created July 31, 2016 11:16
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yukiyan/3c5dd521bcfe0be74ec629297c4e3d5e to your computer and use it in GitHub Desktop.
Save yukiyan/3c5dd521bcfe0be74ec629297c4e3d5e to your computer and use it in GitHub Desktop.
use typecast
[
{
"name": "boolean_value1",
"type": "BOOLEAN"
},
{
"name": "boolean_value2",
"type": "BOOLEAN"
}
]
version: '2'
services:
fluentd:
build: .
container_name: fluentd
volumes:
- ./:/fluentd/etc/
FROM fluent/fluentd:v0.12.26-onbuild
USER root
RUN apk --update add build-base ruby-dev
USER fluent
RUN gem install \
fluent-plugin-bigquery \
fluent-plugin-typecast \
fluent-plugin-parser &&\
fluent-gem install bigdecimal
<source>
@type dummy
dummy {"message":"boolean_value1:true\tboolean_value2:false"}
rate 1
tag bigquery
</source>
<filter bigquery>
@type parser
format ltsv
key_name message
</filter>
<match bigquery>
@type typecast
item_types boolean_value1:bool,boolean_value2:bool
prefix typed
</match>
<match typed.bigquery>
@type bigquery
method insert
auth_method private_key
email xxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxx@developer.gserviceaccount.com
private_key_path 00000000000000000000000000000000-privatekey.p12
project yukiyan-development
dataset sample
table sample
auto_create_table true
schema_path /fluentd/etc/bq_schema.json
</match>
@yukiyan
Copy link
Author

yukiyan commented Jul 31, 2016

2016-07-31 20 13 04

@yukiyan
Copy link
Author

yukiyan commented Jul 31, 2016

apk --update add build-base ruby-dev とか fluent-gem install bigdecimal はコンテナでfluent-plugin-bigquery動かす上で必要なやつなのであまり気にしなくて良い。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment