embulk-filter-hash example

example.md

embulk-filter-hashの利用例

サンプルデータはembulk exampleで生成されるデータを利用

例1

commentフィールドをマスクする

filters:
  - type: hash
    columns:
     - { name: comment }

実行

*************************** 1 ***************************
      id (     long) : 1
 account (     long) : 32,864
    time (timestamp) : 2015-01-27 19:23:49 UTC
purchase (timestamp) : 2015-01-27 00:00:00 UTC
 comment (   string) : a5200d9ab77875014a3a8d7056ec2300ab9912725d7fb2991c0a6407d6887ae0
*************************** 2 ***************************
      id (     long) : 2
 account (     long) : 14,824
    time (timestamp) : 2015-01-27 19:01:23 UTC
purchase (timestamp) : 2015-01-27 00:00:00 UTC
 comment (   string) : 7d9042617062cee04213e56a3edf080a2392fbfd27f81f5bb987bccdf4b91b28
*************************** 3 ***************************
      id (     long) : 3
 account (     long) : 27,559
    time (timestamp) : 2015-01-28 02:20:02 UTC
purchase (timestamp) : 2015-01-28 00:00:00 UTC
 comment (   string) : 845f9ce2fbe32336095bb6d37f2520f507300de04309fd164d9199f4ec6a5291
*************************** 4 ***************************
      id (     long) : 4
 account (     long) : 11,270
    time (timestamp) : 2015-01-29 11:54:36 UTC
purchase (timestamp) : 2015-01-29 00:00:00 UTC
 comment (   string) : fb329000228cc5a24c264c57139de8bf854fc86fc18bf1c04ab61a2b5cb4b921

例2

commentフィールドをMD5でマスクして、カラム名をmasked_commentに変更

filters:
  - type: hash
    columns:
     - { name: comment, algorithm: md5, new_name: masked_comment }

実行

embulk preview -G config.yml
*************************** 1 ***************************
            id (     long) : 1
       account (     long) : 32,864
          time (timestamp) : 2015-01-27 19:23:49 UTC
      purchase (timestamp) : 2015-01-27 00:00:00 UTC
masked_comment (   string) : 6df79df4b40c508e4ddd9f0a71c96a55
*************************** 2 ***************************
            id (     long) : 2
       account (     long) : 14,824
          time (timestamp) : 2015-01-27 19:01:23 UTC
      purchase (timestamp) : 2015-01-27 00:00:00 UTC
masked_comment (   string) : 89525b9ed30a6625bd86bc7df1ebf24e
*************************** 3 ***************************
            id (     long) : 3
       account (     long) : 27,559
          time (timestamp) : 2015-01-28 02:20:02 UTC
      purchase (timestamp) : 2015-01-28 00:00:00 UTC
masked_comment (   string) : ecb703a8ad1afaa18a3a65720aedc167
*************************** 4 ***************************
            id (     long) : 4
       account (     long) : 11,270
          time (timestamp) : 2015-01-29 11:54:36 UTC
      purchase (timestamp) : 2015-01-29 00:00:00 UTC
masked_comment (   string) : 6c3e226b4d4795d518ab341b0824ec29