Skip to content

Instantly share code, notes, and snippets.

@yokozawa
Last active June 23, 2017 16:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yokozawa/9b9bedcdd5eb8dbe54e4aa0721e7c06b to your computer and use it in GitHub Desktop.
Save yokozawa/9b9bedcdd5eb8dbe54e4aa0721e7c06b to your computer and use it in GitHub Desktop.
_export:
host: 'localhost'
user: 'me'
password: 'my-pass'
database: 'my-database'
project_id: 'my-projectid'
dataset: 'my-dataset'
timezone: UTC
schedule:
daily>: 10:00:00
+main:
!include : my-db.dig
+step1:
for_each>:
table: [
users
, products
]
_do:
embulk>: mysql_to_bigquery.yml
+step2:
embulk>: mysql_to_bigquery_orders.yml
in:
type: mysql
host: ${host}
user: ${user}
password: '${password}'
database: ${database}
table: ${table}
out:
type: bigquery
mode: replace
auth_method: json_key
json_keyfile: ./my-bigquery.json
project: ${project_id}
dataset: ${dataset}
auto_create_table: true
table: ${table}
allow_quoted_newlines: true
in:
type: mysql
host: ${host}
user: ${user}
password: '${password}'
database: ${database}
query: |
select id,user_id,product_id from orders
out:
type: bigquery
mode: replace
auth_method: json_key
json_keyfile: ./my-bigquery.json
project: ${project_id}
dataset: ${dataset}
auto_create_table: true
table: orders
allow_quoted_newlines: true
@hiroyuki-sato
Copy link

hiroyuki-sato commented Jun 23, 2017

ブログ拝見しました。
Digdagクライアントでもシークレット使えるはずですが
これじゃダメですか?
http://docs.digdag.io/command_reference.html#secrets

追記
もしかして、ローカルモードではなく、スケジューラでシークレットが使えないと言う話ですか?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment