Skip to content

Instantly share code, notes, and snippets.

View embulk.txt
embulk gem install -V embulk-output-bigquery
2021-04-21 19:14:42.166 +0900: Embulk v0.9.23
Gem plugin path is: /Users/hsato/.embulk/lib/gems
HEAD https://api.rubygems.org/api/v1/dependencies
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=embulk-output-bigquery
200 OK
Getting SRV record failed: DNS result has no information for _rubygems._tcp.api.rubygems.org
View embulk.txt
testRuntimeClasspath - Runtime classpath of source set 'test'.
+--- org.jruby.joni:joni:2.1.41
| \--- org.jruby.jcodings:jcodings:1.0.55
+--- org.jruby.jcodings:jcodings:1.0.55
+--- org.embulk:embulk-util-config:0.2.1
+--- org.embulk:embulk-util-timestamp:0.2.1
| \--- org.embulk:embulk-util-rubytime:0.3.2
+--- org.embulk:embulk-util-json:0.1.0
| \--- com.fasterxml.jackson.core:jackson-core:2.6.7
+--- org.embulk:embulk-util-text:0.1.0
View embulk_error.txt
2021-03-21 16:58:34.196 +0900: Embulk v0.9.23
2021-03-21 16:58:35.233 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2021-03-21 16:58:38.277 +0900 [INFO] (main): Gem's home and path are set by default: "/Users/hsato/.embulk/lib/gems"
2021-03-21 16:58:39.609 +0900 [INFO] (main): Started Embulk v0.9.23
2021-03-21 16:58:39.768 +0900 [INFO] (0001:transaction): Loaded plugin embulk/parser/jsonpath from a load path
2021-03-21 16:58:39.817 +0900 [INFO] (0001:transaction): Listing local files at directory 'example' filtering filename by prefix 'input'
2021-03-21 16:58:39.819 +0900 [INFO] (0001:transaction): "follow_symlinks" is set false. Note that symbolic links to directories are skipped.
2021-03-21 16:58:39.822 +0900 [INFO] (0001:transaction): Loading files [example/input.json, example/input3.json, example/input2.json]
2021-03-21 16:58:39.908 +0900 [INFO] (0001:transaction): Using local thread executor with max_threads=8 / output tasks 6 = input tasks 3 * 2
2021-03
View embulk_log.txt
******************************************************************
*
*
* embulk v0.10.20
*
*
******************************************************************
%%%%% try: 1 %%%%%
2021-03-15 09:53:48.509 +0900 [INFO] (main): embulk_home is set from command-line: /Users/usr/.embulk-dev/0.10.20
2021-03-15 09:53:49.011 +0900 [INFO] (main): Started Embulk v0.10.20
View embulk_error.txt
org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: java.lang.ClassFormatError: Truncated class file
at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:344)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:584)
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:37)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:357)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:354)
at org.embulk.spi.ExecInternal.doWith(ExecInternal.java:26)
at org.embulk.exec.BulkLoader.run(BulkLoader.java:354)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:324)
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:288)
@hiroyuki-sato
hiroyuki-sato / cdata.tsv
Last active Mar 4, 2021
CData Drivers 20210303
View cdata.tsv
No category name note
1 BIG DATA & NOSQL Amazon Athena
2 BIG DATA & NOSQL Amazon DynamoDB
3 BIG DATA & NOSQL Azure Data Catalog
4 BIG DATA & NOSQL Azure Data Lake Storage
5 BIG DATA & NOSQL BigQuery
6 BIG DATA & NOSQL Cassandra
7 BIG DATA & NOSQL Cloudant
8 BIG DATA & NOSQL CockroachDB
9 BIG DATA & NOSQL Cosmos DB
@hiroyuki-sato
hiroyuki-sato / epsg.java
Created Feb 26, 2021
EPSG:32654 -> EPSG:6677変換
View epsg.java
/*
* This Java source file was generated by the Gradle 'init' task.
*/
package test;
import org.locationtech.proj4j.CRSFactory;
import org.locationtech.proj4j.CoordinateReferenceSystem;
import org.locationtech.proj4j.CoordinateTransform;
import org.locationtech.proj4j.CoordinateTransformFactory;
import org.locationtech.proj4j.ProjCoordinate;
@hiroyuki-sato
hiroyuki-sato / embulk.diff
Created Feb 16, 2021
embulk-filter-flatten_json patch
View embulk.diff
diff --git a/src/main/java/org/embulk/filter/flatten_json/FlattenJsonFilterPlugin.java b/src/main/java/org/embulk/filter/flatten_json/FlattenJsonFilterPlugin.java
index 8caaf97..7898bc5 100644
--- a/src/main/java/org/embulk/filter/flatten_json/FlattenJsonFilterPlugin.java
+++ b/src/main/java/org/embulk/filter/flatten_json/FlattenJsonFilterPlugin.java
@@ -85,11 +85,12 @@ public class FlattenJsonFilterPlugin
return new PageOutput()
{
private PageReader pageReader = new PageReader(inputSchema);
+ private PageBuilder pageBuilder = new PageBuilder(Exec.getBufferAllocator(), outputSchema, output);
@hiroyuki-sato
hiroyuki-sato / digdag.sql
Created Feb 15, 2021
digdag slow attempts check
View digdag.sql
select
p.name,
sa.attempt_name,
sa.created_at,
sa.finished_at,
(sa.finished_at - sa.created_at) as est
from
session_attempts sa,
projects p
where
@hiroyuki-sato
hiroyuki-sato / embulk.txt
Created Jan 20, 2021
embulk 0.10.25-SNAPSHOT test
View embulk.txt
/path/to/embulk-0.10.25-SNAPSHOT.jar -l debug -X embulk_home=/Users/user/.embulk-dev guess hoge/seed.yml
2021-01-20 16:28:15.815 +0900 [INFO] (main): embulk_home is set from command-line: /Users/user/.embulk-dev
2021-01-20 16:28:15.817 +0900 [DEBUG] (main): Command-line arguments: [hoge/seed.yml]
2021-01-20 16:28:15.817 +0900 [DEBUG] (main): Embulk system properties: {m2_repo=/Users/user/.embulk-dev/lib/m2/repository, gem_path=, gem_home=/Users/user/.embulk-dev/lib/gems, embulk_home=/Users/user/.embulk-dev, log_level=debug, jruby=file:///Users/user/.embulk/jars/jruby-complete-9.2.14.0.jar}
2021-01-20 16:28:16.374 +0900 [INFO] (main): Started Embulk v0.10.25-SNAPSHOT
2021-01-20 16:28:16.519 +0900 [INFO] (0001:guess): Listing local files at directory '/private/tmp/hoge/csv' filtering filename by prefix 'sample_'
2021-01-20 16:28:16.519 +0900 [INFO] (0001:guess): "follow_symlinks" is set false. Note that symbolic links to directories are skipped.