Skip to content

Instantly share code, notes, and snippets.

@hiroyuki-sato
Last active August 29, 2015 14:17
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hiroyuki-sato/01026212009af027a0e2 to your computer and use it in GitHub Desktop.
Save hiroyuki-sato/01026212009af027a0e2 to your computer and use it in GitHub Desktop.
Embulk and SQlite3
embulk example hoge2
embulk run config.yaml

config.yml

in:
  type: file
  path_prefix: /private/tmp/hoge2/csv/sample_
  decoders:
  - {type: gzip}
  parser:
    charset: UTF-8
    newline: CRLF
    type: csv
    delimiter: ','
    quote: '"'
    escape: ''
    skip_header_lines: 1
    columns:
    - {name: id, type: long}
    - {name: account, type: long}
    - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S'}
    - {name: purchase, type: timestamp, format: '%Y%m%d'}
    - {name: comment, type: string}
exec: {}
out: 
  type: jdbc
  driver_path: /usr/local/jdbc/sqlite-jdbc-3.8.7.jar
  driver_class: org.sqlite.JDBC
  url: 'jdbc:sqlite:/tmp/sample.db'
  mode: insert
  table: test
2015-03-24 21:05:43.334 +0900: Embulk v0.5.3
2015-03-24 21:05:45.255 +0900 [INFO] (transaction): Listing local files at directory '/private/tmp/hoge2/csv' filtering filename by prefix 'sample_'
2015-03-24 21:05:45.263 +0900 [INFO] (transaction): Loading files [/private/tmp/hoge2/csv/sample_01.csv.gz]
2015-03-24 21:05:45.510 +0900 [INFO] (transaction): SQL: CREATE TABLE IF NOT EXISTS  test  ( id  BIGINT,  account  BIGINT,  time  TIMESTAMP,  purchase  TIMESTAMP,  comment  CLOB)
2015-03-24 21:05:45.512 +0900 [INFO] (transaction): > 0.00 seconds
2015-03-24 21:05:45.530 +0900 [INFO] (transaction): {done:  0 / 1, running: 0}
2015-03-24 21:05:45.559 +0900 [INFO] (transaction): {done:  0 / 1, running: 0}
java.lang.UnsupportedOperationException: Unsupported type BIGINT (sqlType=4, size=2000000000, scale=10)
	at org.embulk.output.jdbc.setter.ColumnSetterFactory.unsupportedOperationException(org/embulk/output/jdbc/setter/ColumnSetterFactory.java:133)
	at org.embulk.output.jdbc.setter.ColumnSetterFactory.newColumnSetter(org/embulk/output/jdbc/setter/ColumnSetterFactory.java:116)
	at org.embulk.output.jdbc.AbstractJdbcOutputPlugin.open(org/embulk/output/jdbc/AbstractJdbcOutputPlugin.java:484)
	at org.embulk.exec.LocalExecutor$6.call(org/embulk/exec/LocalExecutor.java:625)
	at org.embulk.exec.LocalExecutor$6.call(org/embulk/exec/LocalExecutor.java:616)
	at java.util.concurrent.FutureTask.run(java/util/concurrent/FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(java/lang/Thread.java:745)

Error: Unsupported type BIGINT (sqlType=4, size=2000000000, scale=10)

longの代わりにString

in:
  type: file
  path_prefix: /private/tmp/hoge2/csv/sample_
  decoders:
  - {type: gzip}
  parser:
    charset: UTF-8
    newline: CRLF
    type: csv
    delimiter: ','
    quote: '"'
    escape: ''
    skip_header_lines: 1
    columns:
    - {name: id, type: string}
    - {name: account, type: string}
    - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S'}
    - {name: purchase, type: timestamp, format: '%Y%m%d'}
    - {name: comment, type: string}
exec: {}
out: 
  type: jdbc
  driver_path: /usr/local/jdbc/sqlite-jdbc-3.8.7.jar
  driver_class: org.sqlite.JDBC
  url: 'jdbc:sqlite:/tmp/sample.db'
  mode: insert
  table: test
2015-03-24 21:06:50.793 +0900: Embulk v0.5.3
2015-03-24 21:06:52.847 +0900 [INFO] (transaction): Listing local files at directory '/private/tmp/hoge2/csv' filtering filename by prefix 'sample_'
2015-03-24 21:06:52.855 +0900 [INFO] (transaction): Loading files [/private/tmp/hoge2/csv/sample_01.csv.gz]
2015-03-24 21:06:53.066 +0900 [INFO] (transaction): SQL: CREATE TABLE IF NOT EXISTS  test  ( id  CLOB,  account  CLOB,  time  TIMESTAMP,  purchase  TIMESTAMP,  comment  CLOB)
2015-03-24 21:06:53.069 +0900 [INFO] (transaction): > 0.00 seconds
2015-03-24 21:06:53.085 +0900 [INFO] (transaction): {done:  0 / 1, running: 0}
2015-03-24 21:06:53.111 +0900 [INFO] (transaction): {done:  0 / 1, running: 0}
java.lang.UnsupportedOperationException: Unsupported type BIGINT (sqlType=4, size=2000000000, scale=10)
	at org.embulk.output.jdbc.setter.ColumnSetterFactory.unsupportedOperationException(org/embulk/output/jdbc/setter/ColumnSetterFactory.java:133)
	at org.embulk.output.jdbc.setter.ColumnSetterFactory.newColumnSetter(org/embulk/output/jdbc/setter/ColumnSetterFactory.java:116)
	at org.embulk.output.jdbc.AbstractJdbcOutputPlugin.open(org/embulk/output/jdbc/AbstractJdbcOutputPlugin.java:484)
	at org.embulk.exec.LocalExecutor$6.call(org/embulk/exec/LocalExecutor.java:625)
	at org.embulk.exec.LocalExecutor$6.call(org/embulk/exec/LocalExecutor.java:616)
	at java.util.concurrent.FutureTask.run(java/util/concurrent/FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(java/lang/Thread.java:745)

Error: Unsupported type BIGINT (sqlType=4, size=2000000000, scale=10)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment