I hereby claim:
- I am iemejia on github.
- I am iemejia (https://keybase.io/iemejia) on keybase.
- I have a public key ASABEFbLoiwceZydGgeedvWNUOszYQUfnpTm0Nu8SfNJkQo
To claim this, I am signing this object:
Ismaël Mejía iemejia
iemejia
/ InvokeDoFn.java
Created
September 21, 2020 15:08
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.talend.labs.beam.classification; | |
| import static org.apache.beam.runners.core.construction.BeamUrns.getUrn; | |
| import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull; | |
| import java.io.ByteArrayOutputStream; | |
| import java.io.IOException; | |
| import java.util.Arrays; | |
| import java.util.Collection; | |
| import java.util.Collections; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| BEAM_RC_VERSION=2.22.0_rc1 | |
| BEAM_VERSION=2.22.0 | |
| docker pull apache/beam_spark_job_server:$BEAM_RC_VERSION | |
| docker pull apache/beam_flink1.10_job_server:$BEAM_RC_VERSION | |
| docker pull apache/beam_flink1.9_job_server:$BEAM_RC_VERSION | |
| docker pull apache/beam_java_sdk:$BEAM_RC_VERSION | |
| docker pull apache/beam_python3.6_sdk:$BEAM_RC_VERSION |
iemejia
/ beam-versions-compatibility-analysis.sh
Last active
March 9, 2020 21:03
Apache Beam versions compatibility analysis
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| mkdir -p "$HOME/releases/japi" | |
| # create exclusions file for annotations to be skipped | |
| cat <<EOF > "$HOME/releases/japi/beam-skip-annotations" | |
| org.apache.beam.sdk.annotations.Experimental | |
| org.apache.beam.sdk.annotations.Internal | |
| EOF |
iemejia
/ flink-job-server
Created
March 2, 2020 21:37
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ismael@ismael-dell ~/workspace/beam-release-2.19.0 release-2.19.0 python -m apache_beam.examples.wordcount --input LICENSE --output counts --runner PortableRunner --job_endpoint localhost:8099 --environment_type DOCKER --environment_config apache/beam_python2.7_sdk:2.19.0 | |
| /usr/bin/python: No module named apache_beam.examples | |
| ✘ ismael@ismael-dell ~/workspace/beam-release-2.19.0 release-2.19.0 docker run --net=host apache/beam_flink1.9_job_server:2.19.0 | |
| [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - ArtifactStagingService started on localhost:8098 | |
| [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java ExpansionService started on localhost:8097 | |
| [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - JobService started on localhost:8099 | |
| [grpc-default-executor-0] INFO org.apache.beam.runners.flink.FlinkJobInvoker - Invoking job BeamApp-root-0302213432-bb11b12c_c2d3ea57-3e6e-4cf7-9cb4-ff296688854b with pipeline |
iemejia
/ BEAM-9352-jackson-transitive-consistency
Created
February 22, 2020 12:38
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| sdks/java/build-tools/beam-linkage-check.sh "beam-sdks-java-io-kafka beam-sdks-java-io-rabbitmq beam-sdks-java-extensions-sql" | |
| Sat Feb 22 12:30:52 CET 2020: Installing artifacts of BEAM-9352-jackson-transitive-consistency(83961374) to Maven local repository. | |
| Sat Feb 22 12:36:53 CET 2020: Running linkage check for beam-sdks-java-io-kafka in BEAM-9352-jackson-transitive-consistency | |
| SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". | |
| SLF4J: Defaulting to no-operation (NOP) logger implementation | |
| SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. | |
| Sat Feb 22 12:37:21 CET 2020: Done: build/linkagecheck/83961374-beam-sdks-java-io-kafka | |
| Sat Feb 22 12:37:21 CET 2020: Running linkage check for beam-sdks-java-io-rabbitmq in BEAM-9352-jackson-transitive-consistency | |
| SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". | |
| SLF4J: Defaulting to no-operation (NOP) logger implementation |
iemejia
/ linkage-hcatalog-issue
Created
February 21, 2020 15:46
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ sdks/java/build-tools/beam-linkage-check.sh "beam-sdks-java-io-hcatalog" | |
| Fri Feb 21 15:19:18 CET 2020: Installing artifacts of BEAM-9162-jackson-update(cef0d682) to Maven local repository. | |
| ^C% ✘ ismael@ismael-dell ~/workspace/beam-master4 BEAM-9162-jackson-update sdks/java/build-tools/beam-linkage-check.sh "beam-sdks-java-io-hcatalog" | |
| Fri Feb 21 15:20:41 CET 2020: Installing artifacts of BEAM-9162-jackson-update(cef0d682) to Maven local repository. | |
| Fri Feb 21 15:25:08 CET 2020: Running linkage check for beam-sdks-java-io-hcatalog in BEAM-9162-jackson-update | |
| SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". | |
| SLF4J: Defaulting to no-operation (NOP) logger implementation | |
| SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. | |
| Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded | |
| at com.google.cloud.tools.opensource.dependencies.DependencyPath.hashCode(D |
iemejia
/ gist:1b840e8ca248d51dd649b83612d8b330
Created
February 21, 2020 15:44
BEAM-9162-jackson-upgrade-linkage-checks
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| INITIAL GOAL | |
| sdks/java/build-tools/beam-linkage-check.sh "beam-runners-apex beam-runners-core-construction-java beam-runners-core-java beam-runners-direct-java beam-runners-extensions-java-metrics beam-runners-flink-1.7 beam-runners-flink-1.7-job-server beam-runners-flink-1.8 beam-runners-flink-1.8-job-server beam-runners-flink-1.9 beam-runners-flink-1.9-job-server beam-runners-gearpump beam-runners-google-cloud-dataflow-java beam-runners-java-fn-execution beam-runners-jet beam-runners-local-java-core beam-runners-portability-java beam-runners-samza beam-runners-samza-job-server beam-runners-spark beam-runners-spark-job-server beam-sdks-java-bom beam-sdks-java-core beam-sdks-java-extensions-euphoria beam-sdks-java-extensions-google-cloud-platform-core beam-sdks-java-extensions-join-library beam-sdks-java-extensions-json-jackson beam-sdks-java-extensions-kryo beam-sdks-java-extensions-protobuf beam-sdks-java-extensions-sketching beam-sdks-java-extensions-sorter beam-sdks-java-extensions-sql beam-sdks-java-exte |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ ./beam-linkage-check.sh | |
| [Tue Feb 11 15:55:33 CET 2020] Installing artifacts (d04f7067) to Maven local repository | |
| [Tue Feb 11 16:02:50 CET 2020] Running linkage check for beam-sdks-java-core in BEAM-9280-upgrade-commons-compress | |
| SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". | |
| SLF4J: Defaulting to no-operation (NOP) logger implementation | |
| SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. | |
| [Tue Feb 11 16:03:09 CET 2020] Done: /home/ismael/linkagecheck/d04f7067-beam-sdks-java-core | |
| [Tue Feb 11 16:03:09 CET 2020] Running linkage check for beam-sdks-java-io-solr in BEAM-9280-upgrade-commons-compress | |
| SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". | |
| SLF4J: Defaulting to no-operation (NOP) logger implementation |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ ./beam-linkage-check.sh | |
| [Tue Feb 11 11:30:50 CET 2020] Installing artifacts (edc6c5f8) to Maven local repository | |
| [Tue Feb 11 11:31:02 CET 2020] Running linkage check for beam-sdks-java-io-amazon-web-services in BEAM-9160-upgrade-aws-sdk | |
| SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". | |
| SLF4J: Defaulting to no-operation (NOP) logger implementation | |
| SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. | |
| [Tue Feb 11 11:31:17 CET 2020] Done: /home/ismael/linkagecheck/edc6c5f8-beam-sdks-java-io-amazon-web-services | |
| [Tue Feb 11 11:31:17 CET 2020] Running linkage check for beam-sdks-java-io-amazon-web-services2 in BEAM-9160-upgrade-aws-sdk | |
| SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". | |
| SLF4J: Defaulting to no-operation (NOP) logger implementation |
NewerOlder