Skip to content

Instantly share code, notes, and snippets.

@beccasaurus
Last active July 29, 2019 21:09
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save beccasaurus/487a6b089d0bfc97c8b48f77c45f5c2e to your computer and use it in GitHub Desktop.
Save beccasaurus/487a6b089d0bfc97c8b48f77c45f5c2e to your computer and use it in GitHub Desktop.
Generating Java code samples and sample-tester manifest and running sample-tester - pom generator

tl;dr

# clone Java google cloud client libraries repo and cd into the google cloud language library folder
git clone https://github.com/googleapis/google-cloud-java.git
cd google-cloud-java/
cd google-cloud-clients/google-cloud/language/

# update synth.py so it generates samples/v1/ directory
https://gist.githubusercontent.com/beccasaurus/487a6b089d0bfc97c8b48f77c45f5c2e/raw/2e3da0eaff96921adf6b1f6d38af9a0bd10e11f1/synth.py

# install and run synthtool
python3 -m pip install --user --upgrade git+https://github.com/googleapis/synthtool.git
python3 -m synthtool

cd samples/v1/

# generate pom.xml which has execution profiles for running each sample via its region tag
# if you choose to use this, it only works with Ruby 2.5+, sorry, Mac has a lower version
# brew install rbenv && rbenv install 2.5.5 && gem install httparty <== dependencies (not intending this to be used for long!)
curl -O https://gist.githubusercontent.com/beccasaurus/487a6b089d0bfc97c8b48f77c45f5c2e/raw/2e3da0eaff96921adf6b1f6d38af9a0bd10e11f1/generate-pom
curl -O https://gist.githubusercontent.com/beccasaurus/487a6b089d0bfc97c8b48f77c45f5c2e/raw/2e3da0eaff96921adf6b1f6d38af9a0bd10e11f1/pom.xml.erb
chmod +x generate-pom
./generate-pom language v1 src/ > pom.xml
mvn clean package # compile

# generate samples.manifest.yaml which maps region tags to how to invoke samples using mvn exec:java to run the profiles
python3 -m pip install --user --upgrade git+https://github.com/beccasaurus/sample-tester.git@add-invocation-to-manifest
sample-tester gen-manifest --env java --invocation "mvn exec:java -q -D{sample} -Dexec.args='@args'" --output samples.manifest.yaml "src/**/*.java"

# download tests from googleapis
curl -O https://raw.githubusercontent.com/googleapis/googleapis/master/google/cloud/language/v1/language.tests.yaml

# run tests
sample-tester *.yaml

RUNNING: Test environment: "java"
  RUNNING: Test suite: "Natural Language V1"
    PASSED: Test case: "Analyze Syntax"
    PASSED: Test case: "Analyze Syntax – GCS"
    PASSED: Test case: "Analyze Sentiment"
    PASSED: Test case: "Analyze Sentiment – Negative"
    PASSED: Test case: "Analyze Sentiment – GCS"
    PASSED: Test case: "Analyze Sentiment – GCS – Negative"
    PASSED: Test case: "Analyze Entities"
    PASSED: Test case: "Analyze Entities – GCS"
    PASSED: Test case: "Analyze Entity Sentiment"
    PASSED: Test case: "Analyze Entity Sentiment – GCS"
    PASSED: Test case: "Classify Text"
    PASSED: Test case: "Classify Text – GCS"

Let's use Natural Language which already has (a) samples (b) tests

You will need https://github.com/googleapis/google-cloud-java

You will also need the latest version of https://github.com/googleapis/synthtool

  • python3 -m pip install --user --upgrade git+https://github.com/googleapis/synthtool.git

And the latest version of https://github.com/googleapis/sample-tester

  • python3 -m pip install --user --upgrade git+https://github.com/hzyi-google/sample-tester.git@python-manifest

Actually, for right now...

  • python3 -m pip install --user --upgrade git+https://github.com/beccasaurus/sample-tester.git@add-invocation-to-manifest

cd google-cloud-java
cd google-cloud-clients/google-cloud-language/

Update the synth.py

# Add this to java_library call
generator_args=["--dev_samples"]

# Add this below, in the versions loop
s.copy(library / f'gapic-google-cloud-{service}-{version}/samples', f'samples/{version}')

Run synth

  • python3 -m synthtool

This generates a samples/v1/ directory

samples/
└── v1
   ├── build.gradle
   └── src
       └── main
           └── java
               └── com
                   └── google
                       └── cloud
                           └── examples
                               └── language
                                   └── v1
                                       ├── LanguageClassifyGcs.java
                                       ├── LanguageClassifyText.java
                                       ├── LanguageEntitiesGcs.java
                                       ├── LanguageEntitiesText.java
                                       ├── LanguageEntitySentimentGcs.java
                                       ├── LanguageEntitySentimentText.java
                                       ├── LanguageSentimentGcs.java
                                       ├── LanguageSentimentText.java
                                       ├── LanguageSyntaxGcs.java
                                       └── LanguageSyntaxText.java

Let's create a pom.xml that knows how to run these.

And then setup sample-tester so it knows how to execute them via the pom.xml.

And then run samples-tester.

This could also work using the build.gradle.

cd samples/v1/

We're going to create a pom.xml here in samples/v1/

I have a Ruby script which:

  • Reads all Java files in a src directory and maps their region tags to class names and uses those for Maven execution profiles
  • Finds the most recent Maven package to include for the given version of the API

Download generate-pom and pom.xml.erb into the samples/v1/ directory

  • curl -O https://gist.githubusercontent.com/beccasaurus/487a6b089d0bfc97c8b48f77c45f5c2e/raw/d8df94ca17e5779ec028215884747333889602cb/generate-pom
  • curl -O https://gist.githubusercontent.com/beccasaurus/487a6b089d0bfc97c8b48f77c45f5c2e/raw/d8df94ca17e5779ec028215884747333889602cb/pom.xml.erb

Generate the pom.xml by running this from samples/v1/

  • ruby generate-pom language v1 src/ > pom.xml

Now compile

  • mvn clean package

Now try running a sample (with -q so only the STDOUT printed from the sample prints out)

  • mvn exec:java -q -Dlanguage_syntax_text
  •  Part of speech: DET
     Text: This
     Part of speech: VERB
     Text: is
     Part of speech: DET
     Text: a
     Part of speech: ADJ
     Text: short
     Part of speech: NOUN
     Text: sentence
     Part of speech: PUNCT
     Text: .
    

Now try passing an argument to customize the text passed to the sample

  • mvn exec:java -q -Dlanguage_syntax_text -Dexec.arguments='--text_content="Hello, world."'
  •  Part of speech: PUNCT
     Text: "
     Part of speech: X
     Text: Hello
    

Sweet. Now we need to generate a sample.manifest.yaml for sample-tester so that, when a test tries to invoke region tag anguage_syntax_text, it runs the above command.

We'll save it as samples/v1/samples.manifest.yaml

  • sample-tester gen-manifest --env java                                 \
      --invocation "mvn exec:java -q -D{sample} -Dexec.arguments=@args" \
      --output samples.manifest.yaml                                       \
      "src/**/*.java"
    

Using -Dexec.arguments=@args is probably problematic with multiple params, will test to see... using -Dexec.arguments='@args' messes up string processing by the Java CLI

If you look at samples.manifest.yaml, it should include a list of region tags and how to invoke them (in addition to the paths to the individual .java files, which we don't actually need for running samples)

  • cat samples.manifest.yaml

version: 2 sets:

  • environment: java invocation: mvn exec:java -q -D{sample} -Dexec.arguments=@args path: /Users/rebeccataylor/code/beccasaurus/google-cloud-java/google-cloud-clients/google-cloud-language/samples/v1/ items:
    • path: src/main/java/com/google/cloud/examples/language/v1/LanguageEntitySentimentText.java sample: language_entity_sentiment_text
    • path: src/main/java/com/google/cloud/examples/language/v1/LanguageSentimentGcs.java sample: language_sentiment_gcs
    • path: src/main/java/com/google/cloud/examples/language/v1/LanguageEntitySentimentGcs.java sample: language_entity_sentiment_gcs
    • path: src/main/java/com/google/cloud/examples/language/v1/LanguageEntitiesText.java sample: language_entities_text
    • path: src/main/java/com/google/cloud/examples/language/v1/LanguageSentimentText.java sample: language_sentiment_text
    • path: src/main/java/com/google/cloud/examples/language/v1/LanguageSyntaxGcs.java sample: language_syntax_gcs
    • path: src/main/java/com/google/cloud/examples/language/v1/LanguageSyntaxText.java sample: language_syntax_text
    • path: src/main/java/com/google/cloud/examples/language/v1/LanguageClassifyGcs.java sample: language_classify_gcs
    • path: src/main/java/com/google/cloud/examples/language/v1/LanguageEntitiesGcs.java sample: language_entities_gcs
    • path: src/main/java/com/google/cloud/examples/language/v1/AnalyzeTextSentiment.java sample: analyze_text_sentiment
  • path: src/main/java/com/google/cloud/examples/language/v1/LanguageClassifyText.java sample: language_classify_text

Cool, now copy the tests YAML into samples/v1/ from googleapis: https://github.com/googleapis/googleapis/blob/master/google/cloud/language/v1/language.tests.yaml

- `curl -O https://raw.githubusercontent.com/googleapis/googleapis/master/google/cloud/language/v1/language.tests.yaml`

Now let's try to run the tests!

- `sample-tester *.yaml`
- ```
RUNNING: Test environment: "java"
   RUNNING: Test suite: "Natural Language V1"
     PASSED: Test case: "Analyze Syntax"
     PASSED: Test case: "Analyze Syntax – GCS"
     PASSED: Test case: "Analyze Sentiment"
     PASSED: Test case: "Analyze Sentiment – Negative"
     PASSED: Test case: "Analyze Sentiment – GCS"
     PASSED: Test case: "Analyze Sentiment – GCS – Negative"
     PASSED: Test case: "Analyze Entities"
     PASSED: Test case: "Analyze Entities – GCS"
     PASSED: Test case: "Analyze Entity Sentiment"
     PASSED: Test case: "Analyze Entity Sentiment – GCS"
     PASSED: Test case: "Classify Text"
     PASSED: Test case: "Classify Text – GCS"

Tests passe

Annnndddddd.... Done :)

You should have this, in your final state (not showing the compiled .class files)

.
├── build.gradle
├── generate-pom
├── language.tests.yaml
├── pom.xml
├── pom.xml.erb
├── samples.manifest.yaml
├── src
│   └── main
│       └── java
│           └── com
│               └── google
│                   └── cloud
│                       └── examples
│                           └── language
│                               └── v1
│                                   ├── LanguageClassifyGcs.java
│                                   ├── LanguageClassifyText.java
│                                   ├── LanguageEntitiesGcs.java
│                                   ├── LanguageEntitiesText.java
│                                   ├── LanguageEntitySentimentGcs.java
│                                   ├── LanguageEntitySentimentText.java
│                                   ├── LanguageSentimentGcs.java
│                                   ├── LanguageSentimentText.java
│                                   ├── LanguageSyntaxGcs.java
│                                   └── LanguageSyntaxText.java
#! /usr/bin/env ruby
def usage!
puts "Usage: $0 api_name api_version path/to/java/dir/containing/a/src/dir"
exit 1
end
usage! unless ARGV.length == 3
POM_TEMPLATE = "pom.xml.erb"
API_NAME = ARGV.shift
API_VERSION = ARGV.shift
SRC_DIR = ARGV.shift
usage! unless File.directory? SRC_DIR
library_name = "google-cloud-#{API_NAME}"
##
# Create a mapping of region_tags to their class names (based on file names)
#
# "language_analyze_text" => "LanguageAnalyzeText"
#
# This could lazily simply presume every JavaClass.java has a java_class region tag
##
region_tags_to_class_names = {}
java_files = Dir.glob File.join(SRC_DIR, "**", "*.java")
java_files.each do |path|
class_name = File.basename path, ".java"
java_source = File.read path
region_tags = java_source.scan(/\[START ([^\]]+)\]/).flatten.
reject { |region_tag| region_tag.end_with? "_core" } # ignore _core region tags
region_tags.each do |region_tag|
region_tags_to_class_names[region_tag] = class_name
end
end
##
# Lookup the latest version of the Maven packagbe
##
require "httparty" # gem install httparty (could use backed in Net::HTTP but I'm lazy)
url = %{http://search.maven.org/solrsearch/select?q=g:"com.google.cloud"+AND+a:"#{library_name}"&rows=20&core=gav}
response = HTTParty.get url
packages = response["response"]["docs"]
first_package = packages.first
if first_package && (first_package["a"] == library_name)
library_version = first_package["v"]
else
STDERR.puts "Couldn't find package: #{library_name}"
exit
end
##
# Render pom.xml template (prints it to STDOUT)
##
template_variables = {
api_name: API_NAME,
api_version: API_VERSION,
library_name: library_name,
library_version: library_version,
region_tags_to_class_names: region_tags_to_class_names
}
require "erb"
puts ERB.new(File.read POM_TEMPLATE).result_with_hash(template_variables)
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>google-cloud-language</artifactId>
<version>1.65.1-SNAPSHOT</version><!-- {x-version-update:google-cloud-language:current} -->
<packaging>jar</packaging>
<name>Google Cloud Natural Language</name>
<url>https://github.com/googleapis/google-cloud-java/tree/master/google-cloud-clients/google-cloud-language
</url>
<description>
Java idiomatic client for Google Cloud Natural Language.
</description>
<parent>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-clients</artifactId>
<version>0.83.1-alpha-SNAPSHOT</version><!-- {x-version-update:google-cloud-clients:current} -->
</parent>
<properties>
<site.installationModule>google-cloud-language</site.installationModule>
</properties>
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>google-cloud-core</artifactId>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>google-cloud-core-grpc</artifactId>
</dependency>
<dependency>
<groupId>com.google.api.grpc</groupId>
<artifactId>proto-google-cloud-language-v1</artifactId>
</dependency>
<dependency>
<groupId>com.google.api.grpc</groupId>
<artifactId>proto-google-cloud-language-v1beta2</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty-shaded</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-auth</artifactId>
</dependency>
<!-- Test dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.easymock</groupId>
<artifactId>easymock</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.api.grpc</groupId>
<artifactId>grpc-google-cloud-language-v1</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.api.grpc</groupId>
<artifactId>grpc-google-cloud-language-v1beta2</artifactId>
<scope>test</scope>
</dependency>
<!-- Need testing utility classes for generated gRPC clients tests -->
<dependency>
<groupId>com.google.api</groupId>
<artifactId>gax-grpc</artifactId>
<classifier>testlib</classifier>
<scope>test</scope>
</dependency>
</dependencies>
</project>
<!--
Copyright 2019, Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>ccom.google.cloud.examples.language.v1h</groupId>
<artifactId>language-v1-google-cloud-samples</artifactId>
<packaging>jar</packaging>
<parent>
<groupId>com.google.cloud.samples</groupId>
<artifactId>shared-configuration</artifactId>
<version>1.0.10</version>
</parent>
<properties>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>1.8</maven.compiler.source>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<!-- Published GAPIC Client Library -->
<!-- Latest version via Maven Central -->
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId><%= library_name %></artifactId>
<version><%= library_version %></version>
</dependency>
<!-- Command Line Arguments for Samples -->
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.4</version>
</dependency>
</dependencies>
<profiles>
<!-- Profiles for `mvn exec:java -D[profile name]` for each sample -->
<% region_tags_to_class_names.each do |region_tag, main_class| %>
<profile>
<id><%= region_tag %></id>
<activation>
<property>
<name><%= region_tag %></name>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<executions>
<execution>
<goals>
<goal>java</goal>
</goals>
</execution>
</executions>
<configuration>
<mainClass>com.google.cloud.examples.<%= api_name %>.<%= api_version %>.<%= main_class %></mainClass>
<cleanupDaemonThreads>false</cleanupDaemonThreads>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<% end %>
</profiles>
</project>
# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""This script is used to synthesize generated parts of this library."""
import synthtool as s
import synthtool.gcp as gcp
import synthtool.languages.java as java
gapic = gcp.GAPICGenerator()
service = 'language'
versions = ['v1', 'v1beta2']
config_pattern = '/google/cloud/language/artman_language_{version}.yaml'
for version in versions:
library = gapic.java_library(
service=service,
version=version,
config_path=config_pattern.format(version=version),
artman_output_name='', generator_args=["--dev_samples"])
s.copy(library / f'gapic-google-cloud-{service}-{version}/src', 'src')
s.copy(library / f'gapic-google-cloud-{service}-{version}/samples', f'samples/{version}')
s.copy(library / f'grpc-google-cloud-{service}-{version}/src', f'../../google-api-grpc/grpc-google-cloud-{service}-{version}/src')
s.copy(library / f'proto-google-cloud-{service}-{version}/src', f'../../google-api-grpc/proto-google-cloud-{service}-{version}/src')
java.format_code('./src')
java.format_code(f'../../google-api-grpc/grpc-google-cloud-{service}-{version}/src')
java.format_code(f'../../google-api-grpc/proto-google-cloud-{service}-{version}/src')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment