Skip to content

Instantly share code, notes, and snippets.

View suztomo's full-sized avatar
🐢

Tomo Suzuki suztomo

🐢
View GitHub Profile
@suztomo
suztomo / gist:6f5ce8a174bcb1a90302b6f4c1d13f87
Created November 21, 2019 15:22
sdks/java/io/kinesis dependencies with Guava 20.0
compile - Dependencies for source set 'main' (deprecated, use 'implementation' instead).
+--- project :sdks:java:core
| +--- project :model:pipeline
| | +--- org.apache.beam:beam-vendor-grpc-1_21_0:0.1
| | \--- com.google.errorprone:error_prone_annotations:2.3.2 -> 2.0.15
| +--- project :model:job-management
| | +--- org.apache.beam:beam-vendor-grpc-1_21_0:0.1
| | \--- com.google.errorprone:error_prone_annotations:2.3.2 -> 2.0.15
| +--- org.apache.beam:beam-vendor-bytebuddy-1_9_3:0.1
| +--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
@suztomo
suztomo / gist:22a5d15e21b4d65507c3be3f38139bb9
Created November 21, 2019 15:24
sdks/java/io/kinesis dependencies without explicit Guava dependency
compile - Dependencies for source set 'main' (deprecated, use 'implementation' instead).
+--- project :sdks:java:core
| +--- project :model:pipeline
| | +--- org.apache.beam:beam-vendor-grpc-1_21_0:0.1
| | \--- com.google.errorprone:error_prone_annotations:2.3.2 -> 2.0.15
| +--- project :model:job-management
| | +--- org.apache.beam:beam-vendor-grpc-1_21_0:0.1
| | \--- com.google.errorprone:error_prone_annotations:2.3.2 -> 2.0.15
| +--- org.apache.beam:beam-vendor-bytebuddy-1_9_3:0.1
| +--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
@suztomo
suztomo / gist:3c9454c9ffa9dbbfb6e0158cf5a19b81
Created November 21, 2019 16:07
Apache Beam checkJavaLinkage output for beam-sdks-java-io-google-cloud-platform
suztomo@suxtomo24:~/luke-beam$ ./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-sdks-java-io-google-cloud-platform :checkJavaLinkage
Configuration on demand is an incubating feature.
> Task :runners:apex:buildDependencyTree
See the report at: file:///usr/local/google/home/suztomo/luke-beam/runners/apex/build/classes/java/main/org/apache/beam/runners/apex/dependency-tree
> Task :checkJavaLinkage
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
@suztomo
suztomo / gist:ebfc110652b8ffaf7fede64276d7a053
Created November 22, 2019 02:28
:sdks:python:test-suites:tox:py36:testPy36Gcp failure
Ran 2955 tests in 648.661s
OK (SKIP=315)
py35-gcp run-test-post: commands[0] | /usr/local/google/home/suztomo/beam4/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
___________________________________ summary ____________________________________
py35-gcp: commands succeeded
congratulations :)
FAILURE: Build failed with an exception.
@suztomo
suztomo / gist:c4a7357c7237aebb7dbe7bfe1e4185f0
Created November 22, 2019 18:45
Apache Beam Linkage Check beam-sdks-java-io-google-cloud-platform
suztomo@suxtomo24:~/beam6$ ./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-sdks-java-io-google-cloud-platform :checkJavaLinkage |tee linkage.log
...
> Task :checkJavaLinkage
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: /usr/local/google/home/suztomo/beam6/sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderTest.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /usr/local/google/home/suztomo/beam6/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java uses unchecked or unsafe operations.
suztomo@suxtomo24:~/beam6$ ./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-sdks-java-io-hadoop-common :checkJavaLinkage |tee linkage.log
...
Exception in thread "main" com.google.cloud.tools.opensource.dependencies.AggregatedRepositoryException: There were failure(s) in dependency resolution
org.apache.beam:beam-sdks-java-io-hadoop-common:jar:2.18.0-SNAPSHOT (compile) / org.apache.hadoop:hadoop-common:jar:2.7.3 (provided) / org.apache.hadoop:hadoop-auth:jar:2.7.3 (compile) / org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 (compile) / net.sf.ehcache:ehcache-core:jar:2.4.4 (compile) / org.hibernate:hibernate-core:jar:3.5.1-Final (provided) / org.jboss.javaee:jboss-jacc-api_JDK4:jar:1.1.0 (provided) / apache-xerces:xml-apis:jar:2.7.1 (compile): org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact org.jboss.javaee:jboss-servlet-api_3.0:jar:1.0-SNAPSHOT in central (http://repo1.maven.org/maven2/)
org.apache.beam:beam-sdks-java-io-hadoop-common:jar:2.18.0-SNA
suztomo@suxtomo24:~/beam6$ git diff 95e1a77f
diff --git a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 2414880b02..6ef367709b 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -374,8 +374,8 @@ class BeamModulePlugin implements Plugin<Project> {
def google_cloud_core_version = "1.91.1"
def google_cloud_spanner_version = "1.38.0"
def grpc_version = "1.23.0"
- def guava_version = "28.1-jre"
@suztomo
suztomo / gist:a31a6f0f5fb6b1712923f78a1cd9c3ad
Created November 25, 2019 17:14
Google HTTP API Client DateTime Parsing Behavior Change Between 1.30.0 and 1.31.0
suztomo@suxtomo24:~/datetime-in-google-http-client$ cat pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>google-http-client-date</groupId>
<artifactId>google-http-client-date</artifactId>
<version>1.0-SNAPSHOT</version>
@suztomo
suztomo / gist:ed53227909bbdaba51e0e8ced95bbce9
Created November 26, 2019 15:53
org/elasticsearch/hadoop/util/Version in elasticsearch-hadoop-5.0.0.jar
suztomo@suxtomo24:~/beam6$ javap -v -cp /usr/local/google/home/suztomo/.gradle/caches/modules-2/files-2.1/org.elasticsearch/elasticsearch-hadoop/5.0.0/7c2d400119b089314ed2f4b9672a0d307ce49db7/elasticsearch-hadoop-5.0.0.jar org/elasticsearch/hadoop/util/Version
Classfile jar:file:/usr/local/google/home/suztomo/.gradle/caches/modules-2/files-2.1/org.elasticsearch/elasticsearch-hadoop/5.0.0/7c2d400119b089314ed2f4b9672a0d307ce49db7/elasticsearch-hadoop-5.0.0.jar!/org/elasticsearch/hadoop/util/Version.class
Last modified Oct 26, 2016; size 4215 bytes
MD5 checksum 6ed196dd125a4bcf38e82913ebc1fe60
Compiled from "Version.java"
public abstract class org.elasticsearch.hadoop.util.Version
minor version: 0
major version: 50
flags: ACC_PUBLIC, ACC_SUPER, ACC_ABSTRACT
Constant pool:
@suztomo
suztomo / CorrespondenceUsage.java
Created December 3, 2019 22:02
Usage of Truth's Correspondence
@Test
public void truth() {
Truth.assertThat(ImmutableList.of(1, 2, 3)).comparingElementsUsing(
Correspondence.transforming((Integer x) -> (x % 2) == 0, "is even")).contains(true);
}