Skip to content

Instantly share code, notes, and snippets.

View suztomo's full-sized avatar
🐢

Tomo Suzuki suztomo

🐢
82 followers · 23 following
View GitHub Profile
@suztomo
suztomo / gist:1f7030940448dc08bb60e7508e9074a2
Created December 4, 2019 21:13
Dependency Paths to com.google.protobuf:protobuf-lite:3.0.1
io.grpc:grpc-protobuf:1.25.0 (compile) / io.grpc:grpc-protobuf-lite:1.25.0 (compile) / com.google.protobuf:protobuf-lite:3.0.1 (compile)
io.grpc:grpc-alts:1.25.0 (compile) / io.grpc:grpc-protobuf:1.25.0 (compile) / io.grpc:grpc-protobuf-lite:1.25.0 (compile) / com.google.protobuf:protobuf-lite:3.0.1 (compile)
io.grpc:grpc-grpclb:1.25.0 (compile) / io.grpc:grpc-protobuf:1.25.0 (compile) / io.grpc:grpc-protobuf-lite:1.25.0 (compile) / com.google.protobuf:protobuf-lite:3.0.1 (compile)
com.google.api.grpc:grpc-google-common-protos:1.17.0 (compile) / io.grpc:grpc-protobuf:1.23.0 (compile) / io.grpc:grpc-protobuf-lite:1.23.0 (compile) / com.google.protobuf:protobuf-lite:3.0.1 (compile)
com.google.api.grpc:grpc-google-iam-v1:0.13.0 (compile) / io.grpc:grpc-protobuf:1.23.0 (compile) / io.grpc:grpc-protobuf-lite:1.23.0 (compile) / com.google.protobuf:protobuf-lite:3.0.1 (compile)
com.google.cloud:google-cloud-bigtable:1.6.0 (compile) / io.grpc:grpc-protobuf:1.24.1 (compile) / io.grpc:grpc-protobuf-lite:1.24.1 (compile
@suztomo
suztomo / gist:a9e5d46dc64bd19967c6762a468c9b66
Created December 6, 2019 19:23
LinkageChecker output for Apache Beam beam-sdks-java-io-google-cloud-platform
suztomo@suxtomo24:~/beam6$ ./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-sdks-java-io-google-cloud-platform :checkJavaLinkage
...
> Task :checkJavaLinkage
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Class org.brotli.dec.BrotliInputStream is not found;
referenced by 1 class file
@suztomo
suztomo / gist:c58e1e6149e92774e7b8a1249dee749e
Created December 9, 2019 17:23
suztomo@suxtomo24:~/beam6$ cat /tmp/sql-2.18.txt
Configuration on demand is an incubating feature.
> Task :sdks:java:extensions:sql:dependencies
...
compile - Dependencies for source set 'main' (deprecated, use 'implementation' instead).
+--- project :sdks:java:core
| +--- org.antlr:antlr4:4.7
| | +--- org.antlr:antlr4-runtime:4.7
@suztomo
suztomo / gist:e3fa9e5ce9eda5265082c1db8401d6db
Created December 9, 2019 21:26
Beam Java SQL Extension undeclared dependency
suztomo@suxtomo24:~/beam6$ ./gradlew -p sdks/java/extensions/sql generateGradleLintReport
Configuration on demand is an incubating feature.
> Task :sdks:java:extensions:sql:generateGradleLintReport
Generated a report containing information about 37 lint violations in this project
> Task :autoLintGradle
This project contains lint violations. A complete listing of the violations follows.
Because none were serious, the build's overall status was unaffected.
@suztomo
suztomo / gist:bd2771d73fa26ea16c0dae8870b3d633
Created December 10, 2019 17:00
Linkage Checker on beam-sdks-java-io-google-cloud-platform
suztomo@suxtomo24:~/beam6$ git checkout -b linkage-checker origin/master
Branch 'linkage-checker' set up to track remote branch 'master' from 'origin'.
Switched to a new branch 'linkage-checker'
suztomo@suxtomo24:~/beam6$ ./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-sdks-java-io-google-cloud-platform :checkJavaLinkage
Starting a Gradle Daemon, 1 stopped Daemon could not be reused, use --status for details
Configuration on demand is an incubating feature.
...
@suztomo
suztomo / gist:67af33764c1a42d53b00793fb279e77d
Created December 10, 2019 19:53
Linkage Checker (1.1.1-SNAPSHOT) beam-sdks-java-io-google-cloud-platform without reportOnlyReachable
suztomo@suxtomo24:~/beam6$ ./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-sdks-java-io-google-cloud-platform :checkJavaLinkage
Configuration on demand is an incubating feature.
> Task :runners:apex:buildDependencyTree
See the report at: file:///usr/local/google/home/suztomo/beam6/runners/apex/build/classes/java/main/org/apache/beam/runners/apex/dependency-tree
> Task :checkJavaLinkage
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
@suztomo
suztomo / gist:031fb54137564447b45239538ad7a7cf
Created December 10, 2019 19:55
Linkage Checker (1.1.1-SNAPSHOT) beam-sdks-java-io-google-cloud-platform with reportOnlyReachable
suztomo@suxtomo24:~/beam6$ ./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-sdks-java-io-google-cloud-platform :checkJavaLinkage
Configuration on demand is an incubating feature.
> Task :runners:apex:buildDependencyTree
See the report at: file:///usr/local/google/home/suztomo/beam6/runners/apex/build/classes/java/main/org/apache/beam/runners/apex/dependency-tree
> Task :checkJavaLinkage
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
@suztomo
suztomo / gist:70e4cfcdfb3a5416c3cb853098df8a06
Created December 12, 2019 17:57
Linkage Checker for org.apache.cassandra:cassandra-all:3.11.5 and com.google.guava:guava:26.0-jre
(guava-26.0-jre.jar) com.google.common.net.HostAndPort's method getHostText() is not found;
referenced by 1 class file
org.apache.cassandra.locator.InetAddressAndPort (cassandra-all-3.11.5.jar)
(guava-26.0-jre.jar) com.google.common.util.concurrent.Futures's method transform(com.google.common.util.concurrent.ListenableFuture arg1, com.google.common.util.concurrent.AsyncFunction arg2, java.util.concurrent.Executor arg3) is not found;
referenced by 3 class files
org.apache.cassandra.repair.RepairJob (cassandra-all-3.11.5.jar)
com.datastax.driver.core.Connection (cassandra-driver-core-3.0.1-shaded.jar)
com.datastax.driver.core.SessionManager (cassandra-driver-core-3.0.1-shaded.jar)
(guava-26.0-jre.jar) com.google.common.util.concurrent.Futures's method addCallback(com.google.common.util.concurrent.ListenableFuture arg1, com.google.common.util.concurrent.FutureCallback arg2) is not found;
referenced by 10 class files
@suztomo
suztomo / gist:6fe16f6bda526aab97e879feac70309d
Last active December 16, 2019 17:40
Attempt to upgrade Beam's cassandra-all's dependency to 4.0-alpha2

Upgrading cassandra-all to version 4.0-alpha2 caused org.apache.beam.sdk.io.hadoop.format.HadoopFormatIOCassandraTest failure:

org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml. Those properties [thrift_prepared_statements_cache_size_mb] are not valid
	at org.apache.cassandra.config.YamlConfigurationLoader$PropertiesChecker.check(YamlConfigurationLoader.java:234)
	at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:125)
	at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:101)

I then removed thrift_prepared_statements_cache_size_mb parameter from cassandra.yaml. I got:

@suztomo
suztomo / gist:986391176c08f668eedddf69464fa2b0
Created December 17, 2019 21:59
Dependency path to google-auth-library-oauth2-http-0.17.1.jar
google-auth-library-oauth2-http-0.17.1.jar is at:
io.grpc:grpc-alts:1.25.0 (compile) / com.google.auth:google-auth-library-oauth2-http:0.17.1 (compile)
com.google.api:gax-grpc:1.52.0 (compile) / io.grpc:grpc-alts:1.25.0 (compile) / com.google.auth:google-auth-library-oauth2-http:0.17.1 (compile)
com.google.cloud:google-cloud-bigtable:1.7.1 (compile) / io.grpc:grpc-alts:1.25.0 (runtime) / com.google.auth:google-auth-library-oauth2-http:0.17.1 (compile)
com.google.cloud:google-cloud-automl:0.115.1-beta (compile) / com.google.api:gax-grpc:1.50.1 (compile) / io.grpc:grpc-alts:1.25.0 (compile) / com.google.auth:google-auth-library-oauth2-http:0.17.1 (compile)
com.google.cloud:google-cloud-bigtable:1.7.1 (compile) / com.google.api:gax-grpc:1.50.1 (compile) / io.grpc:grpc-alts:1.25.0 (compile) / com.google.auth:google-auth-library-oauth2-http:0.17.1 (compile)
com.google.cloud:google-cloud-billingbudgets:0.1.1 (compile) / com.google.api:gax-grpc:1.50.1 (compile) / io.grpc:grpc-alts:1.25.0 (compile) / com.