Skip to content

Instantly share code, notes, and snippets.

View suztomo's full-sized avatar
🐢

Tomo Suzuki suztomo

🐢
80 followers · 23 following
View GitHub Profile
@suztomo
suztomo / beam-dataflow-dependencies
Created November 7, 2019 18:34
Apache Bream Dataflow Dependencies
org.example:word-count-beam:jar:0.1
+- org.apache.beam:beam-sdks-java-core:jar:2.16.0:compile
| +- org.apache.beam:beam-model-pipeline:jar:2.16.0:compile
| | +- org.apache.beam:beam-vendor-grpc-1_21_0:jar:0.1:compile
| | \- com.google.errorprone:error_prone_annotations:jar:2.3.2:compile
| +- org.apache.beam:beam-model-job-management:jar:2.16.0:compile
| | +- (org.apache.beam:beam-vendor-grpc-1_21_0:jar:0.1:compile - omitted for duplicate)
| | \- (com.google.errorprone:error_prone_annotations:jar:2.3.2:compile - omitted for duplicate)
| +- org.apache.beam:beam-vendor-bytebuddy-1_9_3:jar:0.1:compile
| +- org.apache.beam:beam-vendor-guava-26_0-jre:jar:0.1:compile
@suztomo
suztomo / beam-dataflow-dependencies-with-libraries-bom-2.8.0
Created November 7, 2019 19:40
Apache Bream Dataflow Dependencies with Google Libraries BOM 2.8.0
org.example:word-count-beam:jar:0.1
+- org.apache.beam:beam-sdks-java-core:jar:2.16.0:compile
| +- org.apache.beam:beam-model-pipeline:jar:2.16.0:compile
| | +- org.apache.beam:beam-vendor-grpc-1_21_0:jar:0.1:compile
| | \- com.google.errorprone:error_prone_annotations:jar:2.3.2:compile
| +- org.apache.beam:beam-model-job-management:jar:2.16.0:compile
| | +- (org.apache.beam:beam-vendor-grpc-1_21_0:jar:0.1:compile - omitted for duplicate)
| | \- (com.google.errorprone:error_prone_annotations:jar:2.3.2:compile - omitted for duplicate)
| +- org.apache.beam:beam-vendor-bytebuddy-1_9_3:jar:0.1:compile
| +- org.apache.beam:beam-vendor-guava-26_0-jre:jar:0.1:compile
@suztomo
suztomo / dependencyUpdates_report.txt
Created November 12, 2019 17:28
Beam dependencyUpdates report
------------------------------------------------------------
: Project Dependency Updates (report to plain text file)
------------------------------------------------------------
The following dependencies are using the latest milestone version:
- args4j:args4j:2.33
- com.amazonaws:amazon-kinesis-producer:0.13.1
- com.amazonaws:aws-java-sdk-cloudwatch:1.11.519
- com.amazonaws:aws-java-sdk-core:1.11.519
- com.amazonaws:aws-java-sdk-dynamodb:1.11.519
@suztomo
suztomo / gradle-output.
Last active November 12, 2019 19:15
Output of runBeamDependencyCheck
suztomo@suxtomo24:~/beam-workspace/src$ pwd
/usr/local/google/home/suztomo/beam-workspace/src
suztomo@suxtomo24:~/beam-workspace/src$ ls -alt
total 212
drwxr-xr-x 8 suztomo primarygroup 4096 Nov 12 12:20 .git
drwx------ 5 suztomo primarygroup 4096 Nov 12 12:20 .gradle
drwxr-xr-x 18 suztomo primarygroup 4096 Nov 12 11:37 .
drwxr-xr-x 3 suztomo primarygroup 4096 Nov 12 11:37 build
drwxr-xr-x 5 suztomo primarygroup 4096 Nov 12 11:35 buildSrc
-rw-r--r-- 1 suztomo primarygroup 6173 Nov 12 11:33 settings.gradle
@suztomo
suztomo / gist:39d00252a1e158479f44cc0baa59b6ad
Created November 13, 2019 17:10
org.apache.beam.sdk.io.clickhouse.AtomicInsertTest.html#testIdempotentInsert failure
ru.yandex.clickhouse.except.ClickHouseException: ClickHouse exception, code: 999, host: localhost, port: 32780; Code: 999, e.displayText() = Coordination::Exception: All connection tries failed while connecting to ZooKeeper. Addresses: 172.18.0.2:2181
Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Timeout: connect timed out: 172.18.0.2:2181, e.what() = Timeout, 172.18.0.2:2181
Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Timeout: connect timed out: 172.18.0.2:2181, e.what() = Timeout, 172.18.0.2:2181
Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Timeout: connect timed out: 172.18.0.2:2181, e.what() = Timeout, 172.18.0.2:2181
(Connection loss), e.what() = Coordination::Exception
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:58)
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:28)
at ru.yandex.clickhouse.ClickHouseStatementImpl.checkForErrorAndThrow(Click
@suztomo
suztomo / gist:13473e6b9765c0e96c22aeffab18ef66
Created November 13, 2019 20:12
Gradle version plugin release version working fine
suztomo@suxtomo24:~/hello-gradle-version-plugin$ cat build.gradle
plugins {
id 'java'
id "com.github.ben-manes.versions" version "0.27.0"
}
group 'suztomo'
version '1.0-SNAPSHOT'
sourceCompatibility = 1.8
@suztomo
suztomo / gist:ea1a80efd312d99a0080bdf09923bb32
Created November 14, 2019 21:39
BaseClickHouseTest.java to log ClickHouse messages
suztomo@suxtomo24:~/beam4$ git diff
diff --git a/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java b/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java
index 8c39688fb4..fdb3db337f 100644
--- a/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java
+++ b/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java
@@ -125,6 +125,12 @@ public class BaseClickHouseTest {
@AfterClass
public static void tearDown() {
+ System.out.println("ClickHouse log ---");
@suztomo
suztomo / gist:67d8f3e82889e1fa74952a2a7c4d8274
Created November 19, 2019 19:57
Apache Beam with bigtable-client-core latest 1.12.1
suztomo@suxtomo24:~/beam6$ ./gradlew -p ./sdks/java/io/google-cloud-platform dependencies
Configuration on demand is an incubating feature.
> Task :sdks:java:io:google-cloud-platform:dependencies
------------------------------------------------------------
Project :sdks:java:io:google-cloud-platform - Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform
------------------------------------------------------------
annotationProcessor - Annotation processors and their dependencies for source set 'main'.
@suztomo
suztomo / gist:91a8514d7e867495175f5d0a21821836
Created November 19, 2019 19:59
Apache Beam with bigtable-client-core 1.8.0
suztomo@suxtomo24:~/beam6$ ./gradlew -p ./sdks/java/io/google-cloud-platform dependencies
Configuration on demand is an incubating feature.
> Task :sdks:java:io:google-cloud-platform:dependencies
------------------------------------------------------------
Project :sdks:java:io:google-cloud-platform - Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform
------------------------------------------------------------
annotationProcessor - Annotation processors and their dependencies for source set 'main'.
@suztomo
suztomo / gist:1aa9bf01b9765b6af0652f1c251ca573
Created November 20, 2019 15:35
Gradle lint for Apache Beam sdks/java/io/google-cloud-platform
> Task :autoLintGradle
This project contains lint violations. A complete listing of the violations follows.
Because none were serious, the build's overall status was unaffected.
warning unused-dependency one or more classes in io.grpc:grpc-api:1.21.0 are required by your code directly (no auto-fix available)
warning unused-dependency one or more classes in io.grpc:grpc-api:1.21.0 are required by your code directly (no auto-fix available)
warning unused-dependency one or more classes in io.grpc:grpc-api:1.21.0 are required by your code directly (no auto-fix available)