Skip to content

Instantly share code, notes, and snippets.

@kyontan
Last active February 27, 2019 15:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kyontan/8573272d6a79d3c2b4b577592da08d4d to your computer and use it in GitHub Desktop.
Save kyontan/8573272d6a79d3c2b4b577592da08d4d to your computer and use it in GitHub Desktop.
$ embulk run -b ./ -l debug test.yaml.liquid -c test_diff.yaml
2019-02-27 15:22:24.626 +0000: Embulk v0.8.18
2019-02-27 15:22:26.115 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-mysql (0.9.3)
2019-02-27 15:22:26.144 +0000 [INFO] (0001:transaction): Loaded plugin embulk-output-gcs (0.4.3)
2019-02-27 15:22:26.175 +0000 [INFO] (0001:transaction): JDBC Driver = /some_path/vendor/bundle/jruby/2.3.0/gems/embulk-input-mysql-0.9.3/default_jdbc_driver/mysql-connector-java-5.1.44.jar
2019-02-27 15:22:26.183 +0000 [INFO] (0001:transaction): Fetch size is 10000. Using server-side prepared statement.
2019-02-27 15:22:26.184 +0000 [INFO] (0001:transaction): Connecting to jdbc:mysql://xxx options {useCompression=true, socketTimeout=3600000, useSSL=..., user=..., useLegacyDatetimeCode=..., tcpKeepAlive=..., useCursorFetch=..., connectTimeout=..., password=***, zeroDateTimeBehavior=convertToNull}
2019-02-27 15:22:26.436 +0000 [INFO] (0001:transaction): Using JDBC Driver mysql-connector-java-5.1.44 ( Revision: b3cda4f864902ffdde495b9df93937c3e20009be )
2019-02-27 15:22:26.436 +0000 [WARN] (0001:transaction): embulk-input-mysql 0.9.0 upgraded the bundled MySQL Connector/J version from 5.1.34 to 5.1.44 .
2019-02-27 15:22:26.436 +0000 [WARN] (0001:transaction): And set useLegacyDatetimeCode=false by default in order to get correct datetime value when the server timezone and the client timezone are different.
2019-02-27 15:22:26.436 +0000 [WARN] (0001:transaction): Set useLegacyDatetimeCode=true if you need to get datetime value same as older embulk-input-mysql.
2019-02-27 15:22:26.512 +0000 [INFO] (0001:transaction): Using local thread executor with max_threads=16 / output tasks 8 = input tasks 1 * 8
2019-02-27 15:22:26.533 +0000 [INFO] (0001:transaction): Loaded plugin embulk-formatter-fast_jsonl (0.1.2)
org.embulk.exec.PartialExecutionException: org.embulk.config.ConfigException: com.fasterxml.jackson.databind.JsonMappingException: Setting null to a task field is not allowed. Use Optional<T> (com.google.common.base.Optional) to represent null.
at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(org/embulk/exec/BulkLoader.java:373)
at org.embulk.exec.BulkLoader.doRun(org/embulk/exec/BulkLoader.java:591)
at org.embulk.exec.BulkLoader.access$000(org/embulk/exec/BulkLoader.java:33)
at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:389)
at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:385)
at org.embulk.spi.Exec.doWith(org/embulk/spi/Exec.java:25)
at org.embulk.exec.BulkLoader.run(org/embulk/exec/BulkLoader.java:385)
at org.embulk.EmbulkEmbed.run(org/embulk/EmbulkEmbed.java:180)
at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453)
at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314)
at RUBY.run(uri:classloader:/embulk/runner.rb:84)
at RUBY.run(uri:classloader:/embulk/command/embulk_run.rb:307)
at RUBY.<main>(uri:classloader:/embulk/command/embulk_main.rb:2)
at org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:850)
at org.jruby.Ruby.loadFile(org/jruby/Ruby.java:2976)
at org.jruby.RubyKernel.requireCommon(org/jruby/RubyKernel.java:963)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:956)
at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(org/jruby/RubyKernel$INVOKER$s$1$0$require19.gen)
at some_path.embulk.command.embulk_bundle.invokeOther35:require(some_path/$_dot_embulk/bin/embulk/embulk/command/file:/some_path/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:30)
at some_path.embulk.command.embulk_bundle.<main>(file:/some_path/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:30)
at java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)
at org.jruby.Ruby.runScript(org/jruby/Ruby.java:834)
at org.jruby.Ruby.runNormally(org/jruby/Ruby.java:749)
at org.jruby.Ruby.runNormally(org/jruby/Ruby.java:767)
at org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:580)
at org.jruby.Main.doRunFromMain(org/jruby/Main.java:425)
at org.jruby.Main.internalRun(org/jruby/Main.java:313)
at org.jruby.Main.run(org/jruby/Main.java:242)
at org.jruby.Main.main(org/jruby/Main.java:204)
at org.embulk.cli.Main.main(org/embulk/cli/Main.java:23)
Suppressed: java.lang.NullPointerException
at org.embulk.exec.BulkLoader.doCleanup(BulkLoader.java:494)
at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:425)
at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:421)
at org.embulk.spi.Exec.doWith(Exec.java:25)
at org.embulk.exec.BulkLoader.cleanup(BulkLoader.java:421)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:184)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:453)
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:314)
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:46)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:90)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:205)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:358)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:84)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
at org.jruby.Ruby.runInterpreter(Ruby.java:850)
at org.jruby.Ruby.loadFile(Ruby.java:2976)
at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:243)
at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:885)
at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:525)
at org.jruby.runtime.load.LoadService.require(LoadService.java:396)
at org.jruby.RubyKernel.requireCommon(RubyKernel.java:963)
at org.jruby.RubyKernel.require19(RubyKernel.java:956)
at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:383)
at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
at some_path.embulk.command.embulk_bundle.invokeOther35:require(file:/some_path/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:30)
at some_path.embulk.command.embulk_bundle.RUBY$script(file:/some_path/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:30)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.jruby.ir.Compiler$1.load(Compiler.java:111)
at org.jruby.Ruby.runScript(Ruby.java:834)
at org.jruby.Ruby.runNormally(Ruby.java:749)
at org.jruby.Ruby.runNormally(Ruby.java:767)
at org.jruby.Ruby.runFromMain(Ruby.java:580)
at org.jruby.Main.doRunFromMain(Main.java:425)
at org.jruby.Main.internalRun(Main.java:313)
at org.jruby.Main.run(Main.java:242)
at org.jruby.Main.main(Main.java:204)
at org.embulk.cli.Main.main(Main.java:23)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Setting null to a task field is not allowed. Use Optional<T> (com.google.common.base.Optional) to represent null.
at org.embulk.config.TaskSerDe$TaskDeserializer.deserialize(TaskSerDe.java:163)
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3534)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1870)
at org.embulk.config.ModelManager.readObjectWithConfigSerDe(ModelManager.java:72)
... 76 more
Error: org.embulk.config.ConfigException: com.fasterxml.jackson.databind.JsonMappingException: Setting null to a task field is not allowed. Use Optional<T> (com.google.common.base.Optional) to represent null.
/embulk!/embulk/command/embulk_bundle.rb:30)
at some_path.embulk.embulk.command.embulk_bundle.RUBY$script(file:/some_path/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:30)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.jruby.ir.Compiler$1.load(Compiler.java:111)
at org.jruby.Ruby.runScript(Ruby.java:834)
at org.jruby.Ruby.runNormally(Ruby.java:749)
at org.jruby.Ruby.runNormally(Ruby.java:767)
at org.jruby.Ruby.runFromMain(Ruby.java:580)
at org.jruby.Main.doRunFromMain(Main.java:425)
at org.jruby.Main.internalRun(Main.java:313)
at org.jruby.Main.run(Main.java:242)
at org.jruby.Main.main(Main.java:204)
at org.embulk.cli.Main.main(Main.java:23)
Caused by: org.embulk.config.ConfigException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "content" (class java.util.Optional), not marked as ignorable (0 known properties: ])
at [Source: N/A; line: -1, column: -1] (through reference chain: java.util.Optional["content"])
at org.embulk.config.ModelManager.readObjectWithConfigSerDe(ModelManager.java:75)
at org.embulk.config.DataSourceImpl.loadConfig(DataSourceImpl.java:220)
at org.embulk.output.GcsOutputPlugin.transaction(GcsOutputPlugin.java:28)
at org.embulk.spi.FileOutputRunner.transaction(FileOutputRunner.java:64)
at org.embulk.exec.BulkLoader$4$1$1.transaction(BulkLoader.java:544)
at org.embulk.exec.LocalExecutorPlugin.transaction(LocalExecutorPlugin.java:54)
at org.embulk.exec.BulkLoader$4$1.run(BulkLoader.java:539)
at org.embulk.spi.util.Filters$RecursiveControl.transaction(Filters.java:96)
at org.embulk.spi.util.Filters.transaction(Filters.java:49)
at org.embulk.exec.BulkLoader$4.run(BulkLoader.java:533)
at org.embulk.input.jdbc.AbstractJdbcInputPlugin.transaction(AbstractJdbcInputPlugin.java:206)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:528)
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:33)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:389)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:385)
at org.embulk.spi.Exec.doWith(Exec.java:25)
at org.embulk.exec.BulkLoader.run(BulkLoader.java:385)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:453)
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:314)
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:46)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:90)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:205)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:358)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:84)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
at org.jruby.Ruby.runInterpreter(Ruby.java:850)
at org.jruby.Ruby.loadFile(Ruby.java:2976)
at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:243)
at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:885)
at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:525)
at org.jruby.runtime.load.LoadService.require(LoadService.java:396)
at org.jruby.RubyKernel.requireCommon(RubyKernel.java:963)
at org.jruby.RubyKernel.require19(RubyKernel.java:956)
at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:383)
at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
at some_path.embulk.embulk.command.embulk_bundle.invokeOther35:require(file:/some_path/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:30)
at some_path.embulk.embulk.command.embulk_bundle.RUBY$script(file:/some_path/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:30)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.jruby.ir.Compiler$1.load(Compiler.java:111)
at org.jruby.Ruby.runScript(Ruby.java:834)
at org.jruby.Ruby.runNormally(Ruby.java:749)
at org.jruby.Ruby.runNormally(Ruby.java:767)
at org.jruby.Ruby.runFromMain(Ruby.java:580)
at org.jruby.Main.doRunFromMain(Main.java:425)
at org.jruby.Main.internalRun(Main.java:313)
at org.jruby.Main.run(Main.java:242)
at org.jruby.Main.main(Main.java:204)
at org.embulk.cli.Main.main(Main.java:23)
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "content" (class java.util.Optional), not marked as ignorable (0 known properties: ])
at [Source: N/A; line: -1, column: -1] (through reference chain: java.util.Optional["content"])
at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:51)
at com.fasterxml.jackson.databind.DeserializationContext.reportUnknownProperty(DeserializationContext.java:817)
at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:958)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1324)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1302)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:249)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:136)
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3534)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1894)
at org.embulk.config.TaskSerDe$TaskDeserializer.deserialize(TaskSerDe.java:161)
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3534)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1870)
at org.embulk.config.ModelManager.readObjectWithConfigSerDe(ModelManager.java:72)
... 76 more
Error: org.embulk.config.ConfigException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "content" (class java.util.Optional), not marked as ignorable (0 known properties: ])
at [Source: N/A; line: -1, column: -1] (through reference chain: java.util.Optional["content"])
in:
type: mysql
host: xxx
user: ...
password: ...
database: ...
table: ...
connect_timeout: 1800
socket_timeout: 3600
options: {useLegacyDatetimeCode: false}
incremental: true
incremental_columns: ["updated_at"]
out:
type: gcs
auth_method: json_key
json_keyfile: /xxx/yyy.json
# json_keyfile:
# content: |
# { ...
bucket: ...
path_prefix: dir_name
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment