Skip to content

Instantly share code, notes, and snippets.

@suztomo

suztomo/offline-dependencies-plugin.md

Last active June 1, 2020 15:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save suztomo/7a29e5bd3204e148eeefb1468c30358c to your computer and use it in GitHub Desktop.
Save suztomo/7a29e5bd3204e148eeefb1468c30358c to your computer and use it in GitHub Desktop.
Download ZIP
Trying to run offline-dependencies-plugin with master
Raw
offline-dependencies-plugin.md

With v2.19.0

./gradlew \
    :model:fn-execution:updateOfflineRepository \
    :model:job-management:updateOfflineRepository \
    :model:pipeline:updateOfflineRepository \
    :sdks:java:core:updateOfflineRepository \
    :sdks:java:extensions:google-cloud-platform-core:updateOfflineRepository \
    :sdks:java:extensions:join-library:updateOfflineRepository \
    :sdks:java:extensions:protobuf:updateOfflineRepository \
    :sdks:java:extensions:sql:updateOfflineRepository \
    :sdks:java:extensions:sql:datacatalog:updateOfflineRepository \
    :sdks:java:extensions:sql:zetasql:updateOfflineRepository \
    :sdks:java:fn-execution:updateOfflineRepository \
    :sdks:java:harness:updateOfflineRepository \
    :sdks:java:io:google-cloud-platform:updateOfflineRepository \
    :sdks:java:io:kafka:updateOfflineRepository \
    :sdks:java:io:mongodb:updateOfflineRepository \
    :sdks:java:io:parquet:updateOfflineRepository \
    :sdks:java:testing:nexmark:updateOfflineRepository \
    :runners:core-construction-java:updateOfflineRepository \
    :runners:core-java:updateOfflineRepository \
    :runners:direct-java:updateOfflineRepository \
    :runners:google-cloud-dataflow-java:updateOfflineRepository \
    :runners:google-cloud-dataflow-java:worker:updateOfflineRepository \
    :runners:java-fn-execution:updateOfflineRepository \
    :runners:local-java:updateOfflineRepository \
    :examples:java:updateOfflineRepository \
    :vendor:bytebuddy-1_9_3:updateOfflineRepository\
    :vendor:grpc-1_26_0:updateOfflineRepository \
    :vendor:guava-26_0-jre:updateOfflineRepository\
    :vendor:sdks-java-extensions-protobuf:updateOfflineRepository \
    --no-parallel \
    -PofflineRepositoryRoot=/tmp/offline-plugin

The build failed:

suztomo-macbookpro44% ./gradlew --offline :runners:google-cloud-dataflow-java:compileJava
Configuration on demand is an incubating feature.
> Task :model:pipeline:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':model:pipeline:compileJava'.
> Could not resolve all files for configuration ':model:pipeline:errorprone'.
   > Could not find any matches for com.google.errorprone:error_prone_core:latest.release as no versions of com.google.errorprone:error_prone_core are available.
     Searched in the following locations:
       - file:/tmp/offline-plugin/com/google/errorprone/error_prone_core/maven-metadata.xml
       - file:/tmp/offline-plugin/com/google/errorprone/error_prone_core/
     Required by:
         project :model:pipeline

Gradle dependencies shows "latest.release" under errorprone configuration

errorprone
\--- com.google.errorprone:error_prone_core:latest.release -> 2.4.0
     +--- com.google.errorprone:error_prone_annotation:2.4.0
     |    \--- com.google.guava:guava:27.0.1-jre
     |         +--- com.google.guava:failureaccess:1.0.1
...

With master

It's complaining flogger error.

suztomo-macbookpro44% ./gradlew \
    :model:fn-execution:updateOfflineRepository \
    :model:job-management:updateOfflineRepository \
    :model:pipeline:updateOfflineRepository \
    :sdks:java:core:updateOfflineRepository \
    :sdks:java:extensions:google-cloud-platform-core:updateOfflineRepository \
    :sdks:java:extensions:join-library:updateOfflineRepository \
    :sdks:java:extensions:protobuf:updateOfflineRepository \
    :sdks:java:extensions:sql:updateOfflineRepository \
    :sdks:java:extensions:sql:datacatalog:updateOfflineRepository \
    :sdks:java:extensions:sql:zetasql:updateOfflineRepository \
    :sdks:java:fn-execution:updateOfflineRepository \
    :sdks:java:harness:updateOfflineRepository \
    :sdks:java:io:google-cloud-platform:updateOfflineRepository \
    :sdks:java:io:kafka:updateOfflineRepository \
    :sdks:java:io:mongodb:updateOfflineRepository \
    :sdks:java:io:parquet:updateOfflineRepository \
    :sdks:java:testing:nexmark:updateOfflineRepository \
    :runners:core-construction-java:updateOfflineRepository \
    :runners:core-java:updateOfflineRepository \
    :runners:direct-java:updateOfflineRepository \
    :runners:google-cloud-dataflow-java:updateOfflineRepository \
    :runners:google-cloud-dataflow-java:worker:updateOfflineRepository \
    :runners:java-fn-execution:updateOfflineRepository \
    :runners:local-java:updateOfflineRepository \
    :examples:java:updateOfflineRepository \
    :vendor:bytebuddy-1_10_8:updateOfflineRepository\
    :vendor:grpc-1_26_0:updateOfflineRepository \
    :vendor:guava-26_0-jre:updateOfflineRepository\
    :vendor:sdks-java-extensions-protobuf:updateOfflineRepository \
    --no-parallel \
    -PofflineRepositoryRoot=/tmp/offline-plugin

> Task :sdks:java:extensions:google-cloud-platform-core:updateOfflineRepository
1 problem was encountered while building the effective model for com.google.flogger:flogger:0.5.1
[FATAL] 'dependencies.dependency. com.google.flogger:flogger:0.5.1' for com.google.flogger:flogger:0.5.1 is referencing itself. @ 
: [[FATAL] 'dependencies.dependency. com.google.flogger:flogger:0.5.1' for com.google.flogger:flogger:0.5.1 is referencing itself. @ com.google.flogger:flogger:0.5.1]

Changed build.gradle to use /tmp/offline-plugin

suztomo-macbookpro44% git diff d87fc525e820f78c4546f3f0f5bd5218e3ce323f
diff --git a/gradle.properties b/gradle.properties
index 22f2c251bd..a5cc583a91 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -19,7 +19,7 @@ org.gradle.caching=true
 org.gradle.parallel=true
 org.gradle.configureondemand=true
 org.gradle.jvmargs=-Xss10240k
-offlineRepositoryRoot=offline-repository
+offlineRepositoryRoot=/tmp/offline-plugin
 signing.gnupg.executable=gpg
 signing.gnupg.useLegacyGpg=true
 
diff --git a/settings.gradle b/settings.gradle
index 058213798e..07e7ecb0b3 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -16,6 +16,14 @@
  * limitations under the License.
  */
 
+buildscript {
+  repositories {
+    maven {
+      url '/tmp/offline-plugin'
+    }
+  }
+}
+
 rootProject.name = "beam"
 
 include ":release"

suztomo-macbookpro44% ./gradlew --offline :runners:google-cloud-dataflow-java:compileJava
Configuration on demand is an incubating feature.

> Task :sdks:java:core:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :sdks:java:extensions:google-cloud-platform-core:compileJava
Note: /Users/suztomo/beam/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :vendor:sdks-java-extensions-protobuf:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :sdks:java:fn-execution:compileJava
Note: /Users/suztomo/beam/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataSizeBasedBufferingOutboundObserver.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :sdks:java:extensions:protobuf:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :runners:core-construction-java:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :runners:core-java:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :sdks:java:harness:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :runners:java-fn-execution:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :sdks:java:expansion-service:compileJava
Note: /Users/suztomo/beam/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /Users/suztomo/beam/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :sdks:java:io:google-cloud-platform:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':sdks:java:io:google-cloud-platform:compileJava'.
> Could not resolve all files for configuration ':sdks:java:io:google-cloud-platform:compileClasspath'.
   > Could not find com.google.cloud:google-cloud-bigquery:1.108.0.
     Searched in the following locations:
       - file:/tmp/offline-plugin/com/google/cloud/google-cloud-bigquery/1.108.0/google-cloud-bigquery-1.108.0.pom
       - file:/tmp/offline-plugin/com/google/cloud/google-cloud-bigquery/1.108.0/google-cloud-bigquery-1.108.0.jar
     Required by:
         project :sdks:java:io:google-cloud-platform > com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta
   > Could not find com.google.http-client:google-http-client-protobuf:1.34.0.
     Searched in the following locations:
       - file:/tmp/offline-plugin/com/google/http-client/google-http-client-protobuf/1.34.0/google-http-client-protobuf-1.34.0.pom
       - file:/tmp/offline-plugin/com/google/http-client/google-http-client-protobuf/1.34.0/google-http-client-protobuf-1.34.0.jar
     Required by:
         project :sdks:java:io:google-cloud-platform > com.google.cloud.datastore:datastore-v1-proto-client:1.6.3

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1m 43s
45 actionable tasks: 33 executed, 12 up-to-date
suztomo-macbookpro44% find /tmp/offline-plugin - name 'google-cloud-bigquery-*'
find: -: unknown primary or operator
suztomo-macbookpro44% find /tmp/offline-plugin -name 'google-cloud-bigquery-*' 
/tmp/offline-plugin/com/google/cloud/google-cloud-bigquery-parent
/tmp/offline-plugin/com/google/cloud/google-cloud-bigquery-parent/1.106.0/google-cloud-bigquery-parent-1.106.0.pom
/tmp/offline-plugin/com/google/cloud/google-cloud-bigquery/1.106.0/google-cloud-bigquery-1.106.0.jar
/tmp/offline-plugin/com/google/cloud/google-cloud-bigquery/1.106.0/google-cloud-bigquery-1.106.0.pom
suztomo-macbookpro44%
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment