Skip to content

Instantly share code, notes, and snippets.

@beccasaurus

beccasaurus/LICENSE

Last active September 30, 2019 12:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save beccasaurus/66e22078a456c73e65abcc0f1af7baa7 to your computer and use it in GitHub Desktop.
Save beccasaurus/66e22078a456c73e65abcc0f1af7baa7 to your computer and use it in GitHub Desktop.
Download ZIP
$ sample-composer
Raw
README.md

$ sample-composer

sample-composer [*.yaml] [-o output/] [-r region_tag]

Define GAPIC code samples using templates for reusability.

curl -LO https://gist.github.com/beccasaurus/66e22078a456c73e65abcc0f1af7baa7/raw/sample-composer

Options


  *                   Sample composition files (YAML)
                      
                      Only files ending with .yml and .yaml

                      .samples.yaml,     Automatically loaded if found
                       samples.yaml      in the current directory
                                         (when no arguments passed)
                      
                      Directories are searched recursively


  -o, --output        The location to output samples.
                        -o dir/          output one sample per file in dir/
                        -o file.yaml     output samples in file.yaml
                        -o STDOUT        output samples to STDOUT
                        -o x_gapic.yaml  output into legacy _gapic config file

                      The --stdout option defaults to STDOUT

  -r, --region-tag    Specific region tag(s) to output.
                        -r tag_name      output region tag matching exactly
                        -r /tag.*/       output region tags matching pattern

                      May be defined multiple times.
                        -r tag_one -r tag_two -r /pattern/

  -t, --test-suite    Filter name of test suites output.
                        -t "Sweet name"      output test suites matching exactly
                        -t /partial.*/       output test suites matching pattern

                      May be defined multiple times.
                        -t "Suite One" -t "Sweet Two" -t /pattern/

  -h, --help          Print this help message

  -v, --version       Print sample-composer version

Reference

Sample composition YAML files may contain any of the following top-level keys:

  • composer: One or more sample-composition actions to perform
  • samples: Sample configurations (each must have unique region_tag)
  • templates: Template configurations (each must have unique name)
  • tests: Test suite definitions

Composer Configuration

Defines your sample set.

sample_compositions:
- output: samples/
  region_tags: ["specific_tag", /^tag_pattern/]
  paths:
  - ../sample_definitions/
  - ../sample_templates/
  - ../../common_templates

Sample compositions may contain multiple inputs/outputs configurations.

For example, you might want to output GAPIC code sample configuration files but also inject legacy formatted samples into a _gapic.yaml config:

sample_compositions:
- output: ..
  paths: _templates/
- output: ../../product_gapic.yaml
  paths: _templates/

Sample Configuration

Defines samples.

samples:
- region_tag: my_full_region_tag
  title: my sample
  service: servicename
  rpc: methodname
  request:
  - field: config.some_field
    value: "field value"
    comment: |
      this comment described how this field is used
    input_parameter: some_field
  response:
  - print: ["hello, world."]

Every sample must have a unique region_tag (cannot be composed)

Request fields can be configured to read content from a local file.

region_tag: my_full_region_tag
request:
- field: config.content_bytes
  value: "local_file.mp3"
  value_is_file: true
  input_parameter: file_path

Samples can be composed of one of more templates which can:

  • provide the service: and/or rpc:
  • provide a prefix and/or suffix which will be added to title:
  • provide request: fields
  • provide response: body
region_tag: my_full_region_tag
template:
- template_one
- template_two

Samples can override the values of request: fields included by templates:

region_tag: my_full_region_tag
template:
- (template which adds request fields A and B)
request:
- field: A
  value: "changed the value"

Samples can include the response: of any template and provide snippets to include in placeholders of the body of that template's response. Samples can provide snippets to include in response: included by template:

region_tag: my_full_region_tag
response:
- print: ["Hello, world!"]
- include: (name of template that has a response:)
- print: ["Goodbye"]

To replace placeholder values in template response bodies:

region_tag: my_full_region_tag
response:
- print: ["Hello, world!"]
- include: (name of template that has a response:)
  with:
    variable_name:
    # The placeholder in the template body will be replaced by
    # these response items.
    - comment: ["..."]
    - print: ["..."]
- print: ["Goodbye"]

Template Configuration

Defines templates.

templates:
- name: my_template
  service: (including the template will add this service)
  rpc: (including the template will add this rpc)
  request:
  - field: (including the template will include these request fields)
    value: ...
  - field: ...
  response:
  # can be included in sample response via `- include: [template name]`
  - print: [""]
  - loop:
      collection: x
      variable: y
      body:
      # can be replaced in sample response via `with: [placeholder name]`
      - yield: placehodler name

Test Configuration

Define test suites.

tests:
- suite: My Group of Tests
  filename: my_tests                   # (Optional) define filename
  test_template: TemplateToImportFrom  # Full name of the test template to import
  # Provide values for template placeholders
  test_data:
    # Define names of samples
    create_sample: automl_natural_language_create_dataset
    list_sample: automl_natural_language_list_datasets
    get_sample: automl_natural_language_get_dataset
    delete_sample: automl_natural_language_delete_dataset
    # Define a block of code
    create_dataset_assertions: &assert_classification_type_printed
    - assert_contains:
      - literal: "Text Classification Type: MULTILABEL"
    # All three of these snippets (with placeholders in the template)
    # can have the same value, so you can use YAML anchors/aliases.
    list_datasets_assertions: *assert_classification_type_printed
    get_dataset_assertions: *assert_classification_type_printed

Define test suite templates.

templates:

- test_template: AutoML Dataset Management Test Suite
  setup:
  - call:
    $sample: $create_sample
  - teardown:
    - log: ["Teardown"]
    - yield: teardown_block
Raw
LICENSE
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Raw
sample-composer
#! /usr/bin/env ruby
##
# 👋 This code is intentionally lazy and redundant ~ all the more refreshing to refactor 🌷
##
## I can't wait to refactor this into something elegant 💖
# TODO - say *which* required fields are missing
# TODO - refactor (duh)
# TODO - validation against proto ~ and using proto field comments
#
# TODO - PREVIEW - CLI
# TODO - PREVIEW - Paged Responses (will need .proto)
#
# TODO - YIELD params to call:, need to be able to have them be optional too (like SOME can add extra param)
#
# TODO – shorthand for service/rpc, rpc can be AutoML #foo. OR, heck, it'll FIND THE SERVICE.
#
# TODO - Polyfill shared test steps
# TODO - Automatically add project to `call` for samples which take project (and allow other defaults, like location)
#
# Personal OSS: the proto parser (should use official w/ C bindings if possible instead)
# but I'd like to have it for funsies
require "fileutils"
require "optparse"
require "tmpdir"
require "yaml"
def sample_composer_error *messages, error: nil, source_path: nil, raise_error: false
unless error
error = messages.find { |msg| msg.is_a? Exception }
messages.delete_if { |msg| msg.is_a? Exception } if error
end
messages.push "Message: #{error.message}" if error
messages.push "[#{source_path}]" if source_path
message = messages.join " "
STDERR.puts message
if raise_error
raise error if error
raise message
end
end
class SampleComposer
VERSION = "0.2.0"
AUTOLOAD_CONFIGS = %w[ .samples.yaml .samples.yml samples.yaml samples.yml _samples.yaml _samples.yml ]
TOPLEVEL_COMPOSITIONS_KEY = "sample_compositions"
TOPLEVEL_SAMPLES_KEY = "samples"
TOPLEVEL_TEMPLATES_KEY = "templates"
TOPLEVEL_TESTS_KEY = "tests"
# If true (-l, --list) region tags and template names are printed (no file output)
$LIST_ONLY = false
# If true (-p, --preview) outputs pseudocode representation of samples / tests
$PREVIEW_ONLY = false
# If true (--yaml) outputs YAML sample representation of samples / tests
# when combined with -p, --preview
$PREVIEW_YAML = false
# For -w, --watch which will re-run the compositions every {watch_seconds} seconds.
$WATCH_SECONDS = nil
# Compositions loaded from files.
# @return [Array<CompositionDefinition>]
attr_reader :compositions
# Template definitions loaded from files.
# @return [Array<TemplateDefinition>]
attr_reader :templates
# Sample definitions loaded from files.
# @return [Array<SampleDefinition>]
attr_reader :samples
# Test definitions loaded from files.
# @return [Array<TestDefinition>]
attr_reader :tests
# The configured output value, e.g. directory to save samples or _gapic.yaml
# @return [String]
attr_accessor :output
# If true (--skip-compositions) then discovered compositions are not executed.
attr_accessor :skip_compositions
# Filter(s) on the region tag(s) to ouput
# @return [Array<String,Regexp>]
def region_tags
@region_tags.map! { |tag| regexify tag }
end
# Filter(s) on the test suites(s) to ouput
# @return [Array<String,Regexp>]
def test_suite_filters
@test_suite_filters.map! { |tag| regexify tag }
end
# Supports /xxx/ and /xxx/i for case insensitive regexp
def regexify text
return text unless text.is_a? String
if text.start_with?("/") && text.end_with?("/i")
Regexp.new text[1..-3], "i"
elsif text.start_with?("/") && text.end_with?("/")
Regexp.new text[1..-2]
else
text
end
end
# Paths to loaded YAML files
# Each composition can only load a given YAML file once.
attr_reader :loaded_yaml_files
def list_only
$LIST_ONLY
end
def preview_only
$PREVIEW_ONLY
end
def preview_yaml
$PREVIEW_YAML
end
def watch_seconds
$WATCH_SECONDS
end
# If true (default), generated files include a comment describing that they
# were generated via sample-composer (so folks know not to edit them directly)
#
# To disable, set the environment variable GENERATED_COMMENT=false
def output_generated_notice_comment
! (ENV["GENERATED_COMMENT"] == "false")
end
# If true (default), generated files include an Apache 2.0 license header.
#
# To disable, set the environment variable LICENSE_COMMENT=false
def output_google_license_header_comment
! (ENV["LICENSE_COMMENT"] == "false")
end
attr_accessor :source_path # should probably refactor this class + CompositionTemplate
def initialize
@compositions = []
@templates = []
@samples = []
@tests = []
@region_tags = []
@test_suite_filters = []
@loaded_yaml_files = []
@list_only = false
@skip_compositions = false
end
def self.run
original_args = ARGV.dup
composer = SampleComposer.new
ARGV.options do |opts|
opts.on("-o", "--output=value") { |value| composer.output = value }
opts.on("-r", "--region-tag=value") { |value| composer.region_tags.push value }
opts.on("-t", "--test-suite=value") { |value| composer.test_suite_filters.push value }
opts.on("-l", "--list") { $LIST_ONLY = true }
opts.on("-p", "--preview") { $PREVIEW_ONLY = true }
opts.on("-s", "--skip-compositions") { composer.skip_compositions = true }
opts.on( "--stdout") { composer.output = "STDOUT" }
opts.on("-u", "--usage") { usage! }
opts.on("-u", "--usage") { usage! }
opts.on("-h", "--help") { help! }
opts.on("-v", "--version") { version! }
opts.on( "--yaml") { $PREVIEW_YAML = true }
# Shorthand for common preview setup
opts.on("--pp") { composer.output = "STDOUT"
$PREVIEW_ONLY = true
$WATCH_SECONDS = 1 }
# Even more specific, prescriptive setup ~ use "@" in region tags to mark them for preview
opts.on("--ppp") { composer.output = "STDOUT"
$PREVIEW_ONLY = true
$WATCH_SECONDS = 1
composer.region_tags.push "/@/"
composer.test_suite_filters.push "/@/" }
opts.parse!
end
file_paths = ARGV.map { |arg| Dir.glob(arg) }.flatten.uniq
if $WATCH_SECONDS
while true
sample_composer = make_copy composer
system "clear"
begin
sample_composer.compose! *file_paths.clone, run_compositions: true
rescue StandardError => ex
puts ex.message
end
puts
puts
puts
puts "$ sample-composer #{original_args.join(' ')}"
$WATCH_SECONDS.times do
print "."
sleep 1
end
end
else
composer.compose! *file_paths.clone, run_compositions: true
end
end
def self.help! message = nil
full_usage = DATA.read # read usage documentation from __END__
puts full_usage.lines.take_while { |line| ! line.start_with?("Reference") }.join
puts "\n#{message}" if message
exit 1
end
def self.usage! message = nil
puts DATA.read # read usage documentation from __END__
puts "\n#{message}" if message
exit 1
end
def self.version!
puts "sample-composer #{VERSION}"
exit 1
end
# Run this sample composition using the provided files / dirs / glob paths
#
# If `run_compositions` is set to true, this will detect and execute
# sample compositions defined in sample composition files.
def compose! *files, run_compositions: false
run_compositions = false if skip_compositions
if files.any?
load_files *files
else
if default_sample_yaml
load_files default_sample_yaml
puts "Loaded #{compositions.size} compositions, #{samples.size} samples, #{templates.size} templates"
else
SampleComposer.usage! "No files provided"
end
end
if compositions.none? && samples.none? && templates.none?
puts "Compositions none? #{compositions.size}"
SampleComposer.usage! "No files or compositions provided"
end
filter_by_region_tag! if region_tags.any?
filter_test_suites! if test_suite_filters.any?
if list_only
list_loaded_definitions!
else
templatize_templates!
templatize_samples!
templatize_tests!
if preview_only
preview_loaded_definitions!
else
# The main entrypoint composition might not have a -o / may not be valid.
# In this case, the other loaded compositions may be run.
# ie. only valid compositions with -o and paths are actually run.
if validate_compose!(*files)
output_samples! unless test_suite_filters.any? && region_tags.none?
output_tests! unless region_tags.any? && test_suite_filters.none?
end
end
end
if run_compositions
compositions.each { |composition| composition.compose! }
end
end
def colorize text, color
Pseudocoder.colorize text, color
end
def list_loaded_definitions!
if templates.any?
puts "[#{ colorize 'Templates', :light_blue }]"
templates.sort_by(&:name).each { |template| puts colorize(" #{template.name}", :dark_grey) }
puts
end
if samples.any?
puts "[#{ colorize 'Samples', :light_blue }]"
samples.sort_by(&:region_tag).each do |sample|
puts sample.region_tag
if sample.template_names.any?
sample.template_names.each { |template_name| puts colorize(" -> #{template_name}", :dark_grey) }
end
end
puts
end
if tests.any?
puts "[#{ colorize 'Tests', :light_blue }]"
tests.sort_by(&:name).each do |test|
puts test.name
puts colorize(" -> #{test.test_template_name}", :dark_grey)
end
puts
end
if compositions.any?
puts "[#{ colorize 'Compositions', :light_blue }]"
compositions.each { |composition| puts " #{composition.sample_composer_command}" }
puts
end
end
def preview_loaded_definitions!
preview_loaded_samples! unless test_suite_filters.any? && region_tags.none?
preview_loaded_tests! unless region_tags.any? && test_suite_filters.none?
end
def preview_loaded_samples!
if preview_yaml
puts samples.first.to_sample_yaml if samples.any?
else
puts samples.first.pseudocode if samples.any?
end
end
def preview_loaded_tests!
if preview_yaml
puts tests.first.to_test_yaml if tests.any?
else
puts tests.first.pseudocode if tests.any?
end
end
def validate_compose! *files
missing_fields = []
missing_fields.push "output" if output.nil? || output.empty?
missing_fields.push "[file paths]" if files.empty?
if missing_fields.any?
# puts "Note: composition not saved output:(#{output.inspect}) paths:(#{files.inspect})"
false
else
true
end
end
private
# Default .y[a]ml file, if any, to be auto-loaded.
def default_sample_yaml
AUTOLOAD_CONFIGS.find { |filename| File.exist? filename }
end
# Remove all loaded samples which don't match the provided filters, if any.
def filter_by_region_tag!
if region_tags.any?
samples.select! { |sample|
region_tags.all? { |matcher| matcher.match sample.raw_region_tag }
}
end
end
# Remove all loaded samples which don't match the provided filters, if any.
def filter_test_suites!
if test_suite_filters.any?
tests.select! { |test|
test_suite_filters.all? { |matcher| matcher.match test.name }
}
end
end
# Transform all templates based on their definitions + templates
def templatize_templates!
templates.each do |template|
begin
if template.is_a? TestTemplateDefinition
next # Tests don't go through this step.
else
templatize_template! template
end
rescue StandardError => ex
sample_composer_error(
"Failed to apply template(s) for template #{template.name}.",
source_path: template.source_path,
error: ex
)
raise
end
end
end
# Transform all samples based on their definitions + templates
def templatize_samples!
samples.each do |sample|
begin
templatize_sample! sample
rescue StandardError => ex
sample_composer_error(
"Failed to apply template(s) for sample #{sample.region_tag}.",
source_path: sample.source_path,
error: ex
)
raise
end
end
end
# Transform all tests based on their definitions + templates
def templatize_tests!
tests.each do |test|
begin
templatize_test! test
rescue StandardError => ex
sample_composer_error(
"Failed to apply template(s) for test #{test.name}.",
source_path: test.source_path,
error: ex
)
raise
end
end
end
def templatize_test! test
test_template = get_template test.test_template_name
if test_template.nil?
sample_composer_error "Template(s) not found #{test.test_template_name}",
source_path: test.source_path,
raise_error: true
end
TestTemplatizer.new(test, test_template.test_suite_template).modify! if test_template
end
# Output the templatized samples :)
def output_samples!
samples.each &:validate!
if output == "STDOUT"
puts SampleDefinition.sample_file_hash(*samples).to_yaml
elsif File.directory? output
output_samples_to_directory output
elsif output.end_with? "_gapic.yaml"
puts "Write samples to GAPIC config in legacy v1.1 format"
puts "GAPIC config: #{output}"
`which gapicify-samples &>/dev/null`
if ! $?.success?
puts "Not found: gapicify-samples"
puts
puts "Saving code samples to _gapic.yaml files in v1.1 format"
puts "uses the `gapicify-samples` command line utility."
puts
puts "You can download and read more here:"
puts "https://gist.github.com/beccasaurus/e548b579656c36787b986f26df7d2b1b"
puts
puts "To download:"
puts "curl -LO https://gist.github.com/beccasaurus/e548b579656c36787b986f26df7d2b1b/raw/gapicify-samples"
puts "chmod +x gapicify-samples"
exit 1
end
Dir.mktmpdir do |dir|
# Write all samples to a temporary directory and then use gapicify-samples
# to write them to _gapic.yaml config. Temp dir cleaned up after block.
output_samples_to_directory dir, verbose: false
command = %{gapicify-samples "#{output}" #{dir}}
puts
puts "=============================="
puts "$ #{command}"
system command
puts "=============================="
puts
if $?.success?
puts "Wrote samples to #{output}"
else
puts "gapicify-samples failed (exit #{$?.exitstatus})"
exit $?.exitstatus
end
end
elsif output.end_with?(".yaml") or output.end_with?(".yml")
yaml = SampleDefinition.sample_file_hash(*samples).to_yaml
yaml = output_header_comment + yaml
File.write output, yaml
elsif output.end_with? "/"
FileUtils.mkdir_p output
output_samples_to_directory output
else
sample_composer_error "Unknown output #{output.inspect}", source_path: source_path
end
end
def output_tests_to_directory directory_path, verbose: true
tests.each do |test|
test_filename = test.filename
unless test_filename.end_with?(".yaml") || test.filename.end_with?(".yml")
test_filename += ".test.yaml"
end
filename = File.join directory_path, test_filename
yaml = TestDefinition.test_file_hash(test).to_yaml
yaml = output_header_comment + yaml
File.write filename, yaml
puts "Wrote #{filename}" if verbose
end
end
# Output templatized tests (will merge with sample output above)
def output_tests!
tests.each &:validate!
if output == "STDOUT"
puts TestDefinition.test_file_hash(*tests).to_yaml
elsif File.directory? output
output_tests_to_directory output
elsif output.end_with? "_gapic.yaml"
# Tests are not written to _gapic.yaml
elsif output.end_with?(".yaml") or output.end_with?(".yml")
yaml = TestDefinition.test_file_hash(*tests).to_yaml
yaml = output_header_comment + yaml
File.write output, yaml
elsif output.end_with? "/"
FileUtils.mkdir_p output
output_tests_to_directory output
else
sample_composer_error "Unknown output #{output.inspect}", source_path: source_path
end
end
def output_header_comment
comment = ""
if output_google_license_header_comment
comment << %{# Copyright #{Time.now.year} Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
}
end
if output_generated_notice_comment
comment << "# DO NOT EDIT! This is a generated sample configuration.\n\n"
end
comment
end
# Get a loaded template by name, if present.
# @return [TemplateDefinition,nil]
def get_template template_name
templates.find { |template| template_name == template.name }
end
# Processes all of the non-option arguments passed to sample-composer.
# Searches all passed directories recursively for .y[a]ml files.
# Treats all other arguments as file globs and filters out non .y[a]ml files.
# Files which do not end with .y[a]ml are ignored (no warning/message
# Each .y[a]ml file is then processed via #load_yaml_file
def load_files *misc_files
yaml_files = []
misc_files.each do |path_or_glob|
if File.directory? path_or_glob
yaml_files.concat Dir.glob(File.join path_or_glob, "**", "*.yml")
yaml_files.concat Dir.glob(File.join path_or_glob, "**", "*.yaml")
elsif File.file?(path_or_glob) || path_or_glob.include?("*")
files_from_pattern = Dir.glob path_or_glob
yaml_files.concat files_from_pattern.find_all { |path| path.end_with? ".yml" }
yaml_files.concat files_from_pattern.find_all { |path| path.end_with? ".yaml" }
else
sample_composer_error(
"File not found: #{path_or_glob}",
source_path: source_path,
raise_error: true
)
end
end
yaml_files.each { |yaml_file| load_yaml_file yaml_file }
end
# Load the given YAML file :)
#
# YAML files are loaded if:
# (1) they parse OK as valid YAML
# (2) they contain any of these top-level keys: samples, templates, sample_composer
#
# YAML files which do not contain these keys are ignored (no message/warning)
# Messages are printed for YAML files which are invalid (for debugging)
def load_yaml_file yaml_file_path
if loaded_yaml_files.include? yaml_file_path
STDERR.puts "Already loaded #{yaml_file_path}"
return
end
loaded_yaml_files.push yaml_file_path
structure = YAML.load_file yaml_file_path
if structure.respond_to? :[]
if structure[TOPLEVEL_COMPOSITIONS_KEY] && structure[TOPLEVEL_COMPOSITIONS_KEY].respond_to?(:each)
compositions_count = structure[TOPLEVEL_COMPOSITIONS_KEY].size
structure[TOPLEVEL_COMPOSITIONS_KEY].each { |sub_structure| load_composition sub_structure, yaml_file_path }
end
if structure[TOPLEVEL_TEMPLATES_KEY] && structure[TOPLEVEL_TEMPLATES_KEY].respond_to?(:each)
templates_count = structure[TOPLEVEL_TEMPLATES_KEY].size
structure[TOPLEVEL_TEMPLATES_KEY].each { |sub_structure| load_template sub_structure, yaml_file_path }
end
if structure[TOPLEVEL_SAMPLES_KEY] && structure[TOPLEVEL_SAMPLES_KEY].respond_to?(:each)
samples_count = structure[TOPLEVEL_SAMPLES_KEY].size
structure[TOPLEVEL_SAMPLES_KEY].each { |sub_structure| load_sample sub_structure, yaml_file_path }
end
if structure[TOPLEVEL_TESTS_KEY] && structure[TOPLEVEL_TESTS_KEY].respond_to?(:each)
tests_count = structure[TOPLEVEL_TESTS_KEY].size
structure[TOPLEVEL_TESTS_KEY].each { |sub_structure| load_test sub_structure, yaml_file_path }
end
end
rescue StandardError => ex
sample_composer_error(
"Failed to parse as YAML with error",
source_path: yaml_file_path,
error: ex
)
raise
end
def load_sample structure, source_path
sample = SampleDefinition.from_hash structure
sample.source_path = source_path
samples.push sample
rescue StandardError => ex
sample_composer_error "Error loading sample.", source_path: source_path, error: ex
end
def load_template structure, source_path
# Allow "test_suite" instead of "test_template" – change everything later ;)
structure["test_template"] = structure.delete("test_suite") if structure.has_key?("test_suite")
if structure.has_key? "test_template"
test_template = TestTemplateDefinition.from_hash structure
test_template.source_path = source_path
templates.push test_template
else
template = TemplateDefinition.from_hash structure
template.source_path = source_path
templates.push template
end
rescue StandardError => ex
sample_composer_error "Error loading template.", source_path: source_path, error: ex
raise
end
def load_composition structure, source_path
composition = CompositionDefinition.from_hash structure
composition.source_path = source_path
composition.output = output if output && ! output.empty?
composition.region_tags.concat region_tags.clone
composition.test_suite_filters.concat test_suite_filters.clone
compositions.push composition
rescue StandardError => ex
sample_composer_error "Error loading composition.", source_path: source_path, error: ex
end
def load_test structure, source_path
test = TestDefinition.from_hash structure
test.source_path = source_path
tests.push test
rescue StandardError => ex
sample_composer_error "Error loading test.", source_path: source_path, error: ex
raise
end
def templatize_template! template
templates = []
not_found = []
template_names = template.required_template_names
while template_names.any?
name = template_names.shift
x_template = get_template name
if x_template
templates.push x_template
else
not_found.push name
end
end
if not_found.any?
sample_composer_error "Template(s) not found #{not_found.map(&:inspect).join ', '}",
source_path: template.source_path,
raise_error: true
end
SampleTemplatizer.new(template, templates).modify! if templates.any?
end
def templatize_sample! sample
templates = []
not_found = []
template_names = sample.required_template_names
while template_names.any?
name = template_names.shift
template = get_template name
if template
templates.push template
else
not_found.push name
end
end
if not_found.any?
sample_composer_error "Template(s) not found #{not_found.map(&:inspect).join ', '}",
source_path: sample.source_path,
raise_error: true
end
SampleTemplatizer.new(sample, templates).modify! if templates.any?
end
def output_samples_to_directory directory_path, verbose: true
samples.each do |sample|
filename = File.join directory_path, "#{sample.region_tag}.yaml"
yaml = SampleDefinition.sample_file_hash(sample).to_yaml
yaml = output_header_comment + yaml
File.write filename, yaml
puts "Wrote #{filename}" if verbose
end
end
def self.make_copy obj
Marshal.load Marshal.dump(obj)
end
end
class CompositionDefinition
# @return Array<string> Paths and globs for samples
attr_accessor :paths
# @return Array<String> Filters to apply to samples for output
attr_reader :region_tags
# @return Array<String> Filters to apply to tests for output
attr_reader :test_suite_filters
attr_accessor :output
# Track the path where this sample is defined (for debugging)
attr_accessor :source_path
def initialize
@paths = []
@region_tags = []
@test_suite_filters = []
end
def self.from_hash hash
composition = CompositionDefinition.new
composition.output = hash["output"]
composition.paths = Array(hash["paths"])
composition.region_tags.concat Array(hash["region_tags"]) if hash["region_tags"]
composition.test_suite_filters.concat Array(hash["test_suites"]) if hash["test_suites"]
unless composition.output == "STDOUT"
composition.output = File.expand_path composition.output
if hash["output"].end_with?("/") && ! composition.output.end_with?("/")
composition.output.concat "/"
end
end
composition
end
def compose!
validate!
puts
puts "$ #{sample_composer_command}"
puts
# THIS class and SampleComposer are pretty much the same, TODO refactor
composer = SampleComposer.new
composer.output = output
composer.region_tags.concat region_tags
composer.test_suite_filters.concat test_suite_filters
composer.source_path = source_path
composer.compose! *paths, run_compositions: false
end
# Shell command representation of composition, as a sample-composer command
def sample_composer_command
command = ["sample-composer"]
command.concat ["--output", output] if output
region_tags.each { |region_tag| command.concat ["--region-tag", (region_tag.is_a?(Regexp) ? "/#{region_tag.source}/" : region_tag)] }
test_suite_filters.each { |filter| command.concat ["--test-suite", (filter.is_a?(Regexp) ? "/#{filter.source}/" : filter)] }
command.concat paths
command.join " "
end
def validate!
missing_fields = []
missing_fields.push "output" if output.nil? || output.empty?
missing_fields.push "paths" if paths.empty?
raise "Composition missing required fields: #{missing_fields.join ', '}" if missing_fields.any?
end
end
# Represents a definition of a code sample template.
class TemplateDefinition
# Template name (identifier)
attr_accessor :name
attr_accessor :title
attr_accessor :title_prefix
attr_accessor :title_suffix
attr_accessor :description
attr_accessor :description_prefix
attr_accessor :description_suffix
attr_accessor :service_name
attr_accessor :rpc_name
attr_reader :request_fields
attr_reader :response_body
attr_reader :calling_forms
# Track the path where this sample is defined (for debugging)
attr_accessor :source_path
# List of template names to apply (in order)
attr_reader :template_names
def self.from_hash hash
return TestTemplateDefinition.from_hash hash if hash.has_key?("test_template")
raise "Missing required key 'template' for template definition" unless hash["template"] || hash["name"] # allow 'name'
template = TemplateDefinition.new
template.name = hash["template"] || hash["name"]
template.title = hash["title"]
template.title_prefix = hash["title_prefix"]
template.title_suffix = hash["title_suffix"]
template.description = hash["description"]
template.description_prefix = hash["description_prefix"]
template.description_suffix = hash["description_suffix"]
template.service_name = hash["service"]
template.rpc_name = hash["rpc"]
template.request_fields.concat hash["request"] if hash["request"]
template.response_body.concat hash["response"] if hash["response"]
template.calling_forms.concat Array(hash["calling_form"]) if hash["calling_form"]
template.calling_forms.concat Array(hash["calling_forms"]) if hash["calling_forms"]
template.template_names.concat Array(hash["templates"]) if hash["templates"] # may be array or single item :)
template.template_names.concat Array(hash["include"]) if hash["include"] # actually, let's allow include/mixin
template.template_names.concat Array(hash["mixin"]) if hash["mixin"] # as names too (while figuring out what we like)
template.template_names.concat Array(hash["mixins"]) if hash["mixins"] # as names too (while figuring out what we like)
template
end
# # # template.template_names.concat Array(hash["template"]) if hash["template"] # be forgiving... template(s) key # lol nope!
def initialize
@request_fields = []
@response_body = []
@template_names = []
@calling_forms = []
end
def required_template_names
template_names.clone + response_template_names
end
private
def response_template_names body = response_body
template_names = []
body.each do |item|
if item["include"]
template_names.push item["include"]
if item["with"]
item["with"].each do |variable_name, inner_body|
template_names.concat response_template_names(inner_body)
end
end
elsif item["loop"]
template_names.concat response_template_names(item["loop"]["body"])
end
end
template_names
end
end
# Represents a definition of a test template.
class TestTemplateDefinition
# Identifier of the template representing a test suite
attr_accessor :name
# Hash of the full test suite template, with setup/teardown/cases etc
attr_accessor :test_suite_template
# Track the path where this sample is defined (for debugging)
attr_accessor :source_path
def self.from_hash hash
raise "Missing required key 'test_suite' for test definition" unless hash["test_template"]
raise "Test template missing one of: setup, teardown, cases" unless valid_suite_keys(hash)
test_template = TestTemplateDefinition.new
test_template.name = hash.delete "test_template"
test_template.test_suite_template["name"] = test_template.name
test_template.test_suite_template.merge! hash
test_template
end
def initialize
@test_suite_template = {}
end
def self.valid_suite_keys hash
hash.has_key?("setup") || hash.has_key?("teardown") || hash.has_key?("cases")
end
end
# Represents a definition of a code sample (before and after templating).
class SampleDefinition
REQUIRED_HASH_KEYS = %w[ region_tag title description service rpc ]
# This sample's unique "region tag" (identifier)
# Cannot be templatized.
attr_accessor :region_tag
# Descriptive title summary for sample.
# Cannot be fully templatized, but templates can add prefix / suffix.
attr_accessor :title
# Description of sample.
# Cannot be templatized.
attr_accessor :description
attr_accessor :service_name
attr_accessor :rpc_name
attr_reader :request_fields
attr_reader :response_body
# Which calling forms should this sample render to?
attr_reader :calling_forms
# List of template names to apply (in order)
# Not included in the final output of sample YAML, for templating only.
attr_reader :template_names
# Track the path where this sample is defined (for debugging)
attr_accessor :source_path
# Raw, unchanged region tag.
# Used for filtering.
# Hack :)
# Filtering can use non-\w characters defined in the region_tag :P
attr_accessor :raw_region_tag
def self.from_hash hash
raise "Missing required key 'region_tag' for sample definition" unless hash["region_tag"]
sample = SampleDefinition.new
sample.raw_region_tag = hash["region_tag"]
sample.region_tag = hash["region_tag"].gsub(/[^\w]/, "")
sample.title = hash["title"] if hash["title"] && ! hash["title"].empty?
sample.description = hash["description"] if hash["description"] && ! hash["description"].empty?
sample.service_name = hash["service"]
sample.rpc_name = hash["rpc"]
sample.request_fields.concat hash["request"] if hash["request"]
sample.response_body.concat hash["response"] if hash["response"]
sample.calling_forms.concat Array(hash["calling_form"]) if hash["calling_form"]
sample.calling_forms.concat Array(hash["calling_forms"]) if hash["calling_forms"]
sample.calling_forms.concat Array(hash["calling_patterns"]) if hash["calling_patterns"]
sample.template_names.concat Array(hash["template"]) if hash["template"] # be forgiving... template(s) key
sample.template_names.concat Array(hash["templates"]) if hash["templates"] # may be array or single item :)
sample.template_names.concat Array(hash["include"]) if hash["include"] # actually, let's allow include/mixin
sample.template_names.concat Array(hash["mixin"]) if hash["mixin"] # as names too (while figuring out what we like)
sample.template_names.concat Array(hash["mixins"]) if hash["mixins"] # as names too (while figuring out what we like)
sample
end
def humanize_rpc_name rpc_name
rpc_name.gsub(/[A-Z]/, ' \0').strip if rpc_name
end
def name
region_tag
end
# Returns structure of sample(s) as a valid GAPIC code sample file structure.
def self.sample_file_hash *samples
{
"type" => "com.google.api.codegen.samplegen.v1p2.SampleConfigProto",
"schema_version" => "1.2.0",
"samples" => samples.map(&:to_sample_hash)
}
end
def initialize
@request_fields = []
@response_body = []
@template_names = []
@calling_forms = []
end
def required_template_names
template_names.clone + response_template_names
end
# Returns structure of this sample as a valid GAPIC code sample structure.
def to_sample_hash
hash = {
"region_tag" => region_tag,
"title" => title,
"description" => description,
"service" => service_name,
"rpc" => rpc_name,
"request" => request_fields,
"response" => response_body
}
hash["calling_patterns"] = calling_forms if calling_forms.any?
hash
end
def to_sample_yaml
# finalize with validate! #hack #fixme
validate!
SampleDefinition.sample_file_hash(self).to_yaml
end
# Is the resulting sample valid?
# Presumes all templates has been applied.
# Validation of the end result (all req'd fields should be present)
def valid?
validate!
true
rescue
false
end
def validate!
fill_in_title_and_description!
fix_print_and_comment_single_elements!
remove_unused_yields!
apply_string_interpolation!
check_required_fields!
raise "Unapplied templates #{template_names.join ', '}" if template_names.any?
end
def pseudocode
Pseudocoder.sample_pseudocode self
end
private
def fill_in_title_and_description!
if title.nil? || title.empty?
self.title = [description, humanize_rpc_name(rpc_name)].compact.detect {|str| ! str.empty? }
end
if description.nil? || description.empty?
self.description = [title, humanize_rpc_name(rpc_name)].compact.detect {|str| ! str.empty? }
end
end
# If any `print:` or `comment:` used a single item instead of an array value, fix it for the user.
def fix_print_and_comment_single_elements! body = response_body
body.each do |item|
item["print"] = Array(item["print"]) if item["print"]
item["comment"] = Array(item["comment"]) if item["comment"]
if item["loop"]
fix_print_and_comment_single_elements! item["loop"]["body"]
end
end
end
# This should be part of TEMPLATIZING.
# Let's replace `$print: "foooo bar {hi} there"` with `print: "foo bar %s" and 'hi' as second array item
def apply_string_interpolation! body = response_body
body.each do |item|
item.keys.each do |key|
value = item[key]
if key.start_with?("$") && value.is_a?(String)
variable_names = []
value.gsub!(/\{(?<var_name>[^\}]+)\}/) do |match|
variable_names.push Regexp.last_match[:var_name]
"%s"
end
item.delete key
item[key[1..-1]] = [value, *variable_names]
apply_string_interpolation! body
break
end
end
apply_string_interpolation! item["loop"]["body"] if item["loop"]
end
end
def remove_unused_yields! body = response_body
body.each_with_index do |item, i|
if item["yield"]
body.delete_at i
remove_unused_yields! body
break
end
if item["loop"]
remove_unused_yields! item["loop"]["body"]
end
end
end
def response_template_names body = response_body
template_names = []
body.each do |item|
if item["include"]
template_names.push item["include"]
if item["with"]
item["with"].each do |variable_name, inner_body|
template_names.concat response_template_names(inner_body)
end
end
elsif item["loop"]
template_names.concat response_template_names(item["loop"]["body"])
end
end
template_names
end
def check_required_fields!
sample_hash = to_sample_hash
missing_fields = []
REQUIRED_HASH_KEYS.each do |key|
missing_fields.push key unless sample_hash[key] && ! sample_hash[key].empty?
end
if missing_fields.any?
sample_composer_error "Sample missing required final hash keys: #{missing_fields.join ', '}, region_tag: #{region_tag.inspect}\nSample: #{to_sample_hash.inspect}",
raise_error: true,
source_path: source_path
end
end
end
# Represents a definition of a code sample system test (sample-tester).
#
# Currently, this is limited to configuring variables to render a
# template of a test suite (defined under templates: as a test_suite:)
# and can provide variables, e.g. for sample names or assertion code snippets
#
# Templates for tests are defined like so:
#
# templates:
# - test_suite: Identifier for this templatized test suite
# setup:
# teardown: # <=== regular body
# cases:
# - $name: Interpolate $test_name from defined test configuration
# spec:
# - call:
# $sample: $list_sample_name
# - assert_contains:
# - literal: "Cool things."
# - yield: list_assertions
#
# There are 2 templating functions:
# 1. You can interpolate `test_data` values by putting $ before the keyname
# and then any test data variable with a name found in the value will replace
# that value, e.g. test_data: foo: X would replace $foo in the value of a $print
# 2. You can use `yield:` to insert blocks of code (in cases or setup or teardown)
#
class TestDefinition
# Test suite name (identifier)
attr_accessor :suite_name
# Track the path where this sample is defined (for debugging)
attr_accessor :source_path
# The name of the base template to render.
# This test provides the view model variables to render this templated test.
attr_accessor :test_template_name
# Hash of named view model variables.
# These may be simple values, e.g. the region tag of a test (interpolated by $name)
# Or these may be arrays of test body snippets (inserted via yield:)
attr_accessor :test_data
# Representation of the final test, created by TestTemplatizer
attr_accessor :rendered_test
# The filename for saving this test, when sample-composer is configured to
# save tests to a directory. May be provided by YAML, else inferred by name.
attr_accessor :filename
def self.from_hash hash
raise "Missing required key 'suite' for test definition" unless hash["suite"]
raise "Missing required key 'test_template' for test definition" unless hash["test_template"] || hash["template"]
test = TestDefinition.new
test.suite_name = hash["suite"]
test.test_template_name = hash["test_template"] || hash["template"]
test.test_data.merge! hash["test_data"] if hash["test_data"]
if hash["filename"]
test.filename = hash["filename"]
else
test.filename = test.suite_name.downcase.gsub(/[^a-zA-Z0-9]/, "_").gsub(/_+/, "_")
end
test
end
# Returns structure of test(s) as a valid sample-tester v1.2 code sample test structure.
def self.test_file_hash *tests
{
"type" => "test/samples",
"schema_version" => 1,
"test" => {
"suites" => tests.map(&:rendered_test)
}
}
end
def test_hash
TestDefinition.test_file_hash self
end
alias to_test_hash test_hash
def initialize
@test_data = {}
end
def name
suite_name
end
# Assert final rendered test has cases and a suite name
def validate!
unless rendered_test["name"]
sample_composer_error "Test missing required field 'suite' for #{name} [Template: #{test_template_name}]\nRendered test content:\n#{rendered_test.to_yaml}",
raise_error: true,
source_path: source_path
end
unless rendered_test["cases"] && rendered_test["cases"].size > 0
sample_composer_error "Test missing required test cases (must have at least 1 test case) #{name} [Template: #{test_template_name}]",
raise_error: true,
source_path: source_path
end
end
def pseudocode
Pseudocoder.test_pseudocode self
end
end
# Includes the logic to mutate samples by merging templates into them.
class SampleTemplatizer
attr_reader :sample
attr_reader :the_template
attr_reader :templates
def target_object
sample || the_template
end
def initialize sample_or_template, templates
if sample_or_template.is_a? SampleDefinition
@sample = sample_or_template
elsif sample_or_template.is_a? TemplateDefinition
@the_template = sample_or_template
end
@templates = templates
end
# Applies templates to sample (in order)
def modify!
while target_object.template_names.any?
template_name = target_object.template_names.shift
if template = templates.find { |template| template.name == template_name }
apply_template! template
else
raise "Required template #{template_name} not provided (for target: #{target_object.name})"
end
end
# Process once (response uses 0-N templates)
replace_includes_in_response! target_object.response_body
end
private
def get_template template_name
templates.find { |template| template_name == template.name }
end
def apply_template! template
target_object.rpc_name = template.rpc_name if template.rpc_name
target_object.service_name = template.service_name if template.service_name
target_object.calling_forms.concat template.calling_forms if template.calling_forms.any?
if target_object.title && ! target_object.title.empty?
target_object.title = [template.title_prefix, target_object.title, template.title_suffix].compact.map(&:strip).join(' ')
else
target_object.title = [template.title_prefix, template.title, template.title_suffix].compact.map(&:strip).join(' ')
end
if target_object.description && ! target_object.description.empty?
target_object.description = [template.description_prefix, target_object.description, template.description_suffix].compact.map(&:strip).join(' ')
else
target_object.description = [template.description_prefix, template.description, template.description_suffix].compact.map(&:strip).join(' ')
end
apply_request_template! template if template.request_fields.any?
apply_response_template! template if template.response_body.any?
end
def apply_request_template! template
template.request_fields.each do |field_template|
field_name = field_template["field"]
matching_field = target_object.request_fields.find { |f| f["field"] == field_name }
if matching_field
# Sample has its own definition of this field
# Add template's field values (except those overriden by target_object)
field_template.each do |key, value|
unless matching_field.has_key? key
matching_field[key] = value
end
end
else
# Use the template's field definition in its entirety
target_object.request_fields.push field_template
end
end
end
def apply_response_template! template
if target_object.response_body.empty?
# Use the entire template response (if none defined by the target_object)
target_object.response_body.concat template.response_body
else
# Otherwise, walk through the target_object's response body and replace
# all `include` response items with the body of the correct template.
# replace_includes_in_response! <== this is run once via modify_target_object!
end
end
def replace_includes_in_response! response_body
response_body.each_with_index do |item, i|
if item["include"]
template = get_template item["include"]
template_response_body = replace_yields_in_response!(
make_copy(template.response_body),
yield_data: item["with"]
)
response_body.delete_at i
response_body.insert i, *template_response_body
replace_includes_in_response! response_body
break
elsif item["loop"]
replace_includes_in_response! item["loop"]["body"]
end
end
end
def replace_yields_in_response! response_body, yield_data:
response_body.each_with_index do |item, i|
if item["yield"]
yield_variable_name = item["yield"]
# If yield present without matching with, keep it there (might be replaced by another template down the road)
if yield_data && yield_data.has_key?(yield_variable_name)
response_body.delete_at i
if yield_data && yield_data[yield_variable_name]
response_body.insert i, *yield_data[yield_variable_name]
end
replace_yields_in_response! response_body, yield_data: yield_data
end
break
elsif item["loop"]
replace_yields_in_response! item["loop"]["body"], yield_data: yield_data
end
end
response_body
end
def make_copy obj
Marshal.load Marshal.dump(obj)
end
end
# Includes the logic to merging template data for individual tests into a test template.
class TestTemplatizer
# The TestDefinition (from under tests: => suite:)
attr_accessor :test
def initialize test, template
@test = test
@test.rendered_test = make_copy template
@test.rendered_test["name"] = @test.name
end
def test_data
test.test_data
end
def template
test.rendered_test
end
def modify! items = template
items.each_with_index do |item, i|
# Replace yield:
if item.is_a?(Hash) && item["yield"]
yield_variable_name = item["yield"]
if test_data.has_key? yield_variable_name
yield_data = test_data[yield_variable_name]
items.delete_at i
items.insert i, *yield_data
else
# Add verbosity level for this level of detail:
#STDERR.puts "Test definition missing yield value for template '#{yield_variable_name}'" +
# " #{test.source_path}"
items.delete_at i # Remove the yield, but don't replace it with any value.
end
modify! items
break
# Replace $key: some $value for Hashes
# Besides that, just modify! the value if it's iterable
elsif item.is_a?(Hash) && item.keys.any? { |key| key.to_s.start_with? "$" }
item.keys.each do |key|
if key.to_s.start_with? "$"
test_data.each do |data_key, data_value|
placeholder_variable = "$#{data_key}"
case item[key]
when String
item[key].gsub!(placeholder_variable, data_value) if item[key].include?(placeholder_variable)
when Array
# only one level deep, add recursion later
item[key].each_with_index do |value, index|
# Only strings, first level deep of array
if value.is_a?(String) && value.include?(placeholder_variable)
item[key][index] = value.gsub(placeholder_variable, data_value)
end
end
when Hash
# only one level deep, add recursion later
item[key].each do |subkey, value|
# Only strings, first level deep of array
if value.is_a?(String) && value.include?(placeholder_variable)
item[key][subkey] = value.gsub(placeholder_variable, data_value)
end
end
# Only strings, first level deep of Hash
else
# LOG skipped
end
end
item[key[1..-1]] = item.delete key
end
end
elsif item.respond_to?(:each)
modify! item
end
end
end
private
def make_copy obj
Marshal.load Marshal.dump(obj)
end
end
class Pseudocoder
def self.sample_pseudocode sample
SamplePseudocoder.new(sample).to_s
end
def self.test_pseudocode test
TestPseudocoder.new(test).to_s
end
def self.colorize text, color, bold = false
color ||= :reset
text = [COLORS[color.to_sym], text, COLORS[:reset]].join
text.sub! /\[(\d+)m/, '[1;\1m' if bold
text
end
COLORS = {
reset: "\e[0m",
black: "\e[30m",
white: "\e[97m",
red: "\e[31m",
light_red: "\e[91m",
green: "\e[32m",
light_green: "\e[92m",
yellow: "\e[33m",
light_yellow: "\e[93m",
blue: "\e[34m",
light_blue: "\e[94m",
magenta: "\e[35m",
light_magenta: "\e[95m",
cyan: "\e[36m",
light_cyan: "\e[96m",
grey: "\e[37m",
light_grey: "\e[37m",
dark_grey: "\e[90m",
}
attr_reader :coder_object
attr_reader :code_lines
def initialize coder_object
@coder_object = coder_object
@code_lines = []
end
def method_missing name, *args, &block
if coder_object.respond_to? name
coder_object.send name, *args, &block
else
super
end
end
def respond_to? name
return true if coder_object.respond_to?(name)
super
end
def to_s
render!
code_lines.join "\n"
end
def write text, color: nil, indent: 0, bold: false
text.lines.each do |line|
line.prepend " " * indent
line.chomp!
if color
code_lines.append colorize(line, color, bold)
else
code_lines.append line
end
end
end
def append text, color: nil, indent: nil, bold: false
# By default, no indent is added.
# But you can!
text.prepend " " * indent if indent
code_lines.append "" if code_lines.nil? || code_lines.empty?
code_lines.last << colorize(text, color, bold)
end
## def wrap character, text, bold = false, color: nil, indent: nil
## end
def br
code_lines << ""
end
def underscore text
text.gsub(/[A-Z]/, '_\0').downcase.sub(/^_/, '')
end
def colorize text, color, bold
Pseudocoder.colorize text, color, bold
end
class TestPseudocoder < Pseudocoder
def render!
if test_hash["test"] && test_hash["test"]["suites"]
test_hash["test"]["suites"].each do |suite_hash|
render_test_suite suite_hash
end
end
end
alias test coder_object
def test_hash
test.test_hash
end
def note_test_variable variable_name
@pseudo_variable_table ||= []
@pseudo_variable_table << variable_name.to_s
end
def test_variable_defined? varname
@pseudo_variable_table.include? varname.to_s
end
def test_variables
@pseudo_variable_table ||= []
@pseudo_variable_table
end
def render_test_suite suite_hash
@pseudo_variable_table = []
append "["
append suite_hash["name"], color: :blue, bold: true
append "]\n"
render_setup suite_hash["setup"] if suite_hash["setup"]
render_teardown suite_hash["teardown"] if suite_hash["teardown"]
suite_hash["cases"].each {|test_case_hash| render_test_case test_case_hash } if suite_hash["cases"]
end
def render_setup setup_hash
write "setUp() {\n", color: :blue, bold: true
setup_hash.each {|step| render_step step, indent: 1 }
write "}\n\n", color: :blue, bold: true
end
def render_teardown teardown_hash
write "tearDown() {\n", color: :blue, bold: true
teardown_hash.each {|step| render_step step, indent: 1 }
write "}\n\n", color: :blue, bold: true
end
def render_test_case case_hash
case_name = case_hash["name"]
case_steps = case_hash["spec"]
write "test#{case_name.gsub /[^\w]*/, ''}() {\n", color: :blue, bold: true
case_steps.each {|step| render_step step, indent: 1 } if case_steps&.any?
write "}\n\n", color: :blue, bold: true
end
def render_step step, indent: 0
# Most steps are defined under 1 directive name which is the key, like :assert_contains
if step.keys.length == 1
step_name = step.keys.first.to_sym
step_data = step[step_name.to_s]
case step_name
when :log
render_log_step step_data, indent: indent
when :env
render_step_env step_data, indent: indent
# when :uuid
when :code
render_step_code step_data, indent: indent
when :extract_match
render_extract_match step_data, indent: indent
# when :assert_success
# when :assert_failure
when :assert_contains
render_assert_contains step_data, indent: indent
when :assert_not_contains
render_assert_not_contains step_data, indent: indent
when :assert_contains_any
render_assert_contains_any step_data, indent: indent
# when :assert_excludes_all
# when :assert_excludes_any
when :call
render_step_call step_data, indent: indent
when :call_may_fail
render_step_call_may_fail step_data, indent: indent
# when :shell
else
write "TODO: ", color: :light_red, indent: indent
append "Support for #{step_name}", color: :red, indent: indent, bold: true
end
else
write "TODO: ", color: :light_yellow, indent: indent
append "Unsupported Step #{step.to_yaml}", color: :yellow, indent: indent, bold: true
end
end
# TODO - update colorize to use named params instaed...
def render_log_step log_messages, indent: 0
text = ""
log_messages.each do |log_message|
if test_variable_defined? log_message
# This is a test variable to include in the log
varname = log_message
text << Pseudocoder.colorize(" $\{#{varname}\}", :yellow)
else
# This is a regular text message
text << " #{Pseudocoder.colorize log_message, :cyan}"
end
end
text = Pseudocoder.colorize("print('", :blue) + "#{text.strip}" + Pseudocoder.colorize("')", :blue)
write text, indent: indent
end
def render_step_env step, indent: 0
variable_name = step["variable"]
environment_variable = step["name"]
note_test_variable variable_name
write "@#{variable_name}", color: :yellow, indent: indent
append " = "
append_get_environment_variable(variable_name, environment_variable)
end
def render_step_code code, indent: 0
# Note probable variable names and add them to our table of test locals.
code.scan(/^\s*([\w]+)\s+=/).flatten.each {|varname| note_test_variable varname }
write "eval('", color: :blue, indent: indent
append code.strip, color: :cyan
append "')", color: :blue
end
def render_step_call call, indent: 0, newline: true
region_tag = call["sample"]
raise "call without sample: not currently supported by pseudocoded preview" if region_tag.nil? || region_tag.empty?
params = call["params"]
newline ? write("\n") : write("")
write "# Run Sample [", color: :dark_grey, indent: indent
append region_tag.split("_").map(&:capitalize).join(" "), color: :light_grey, bold: true
append "]", color: :dark_grey
write "#{region_tag}", color: :blue, indent: indent, bold: true
append "(", color: :blue
# TODO support args
if params && params.any?
params.each_with_index do |(name, value), index|
append "#{name}", color: :light_blue
append ": "
if value.keys.first == "variable"
varname = value["variable"]
append "@#{varname}", color: :yellow
elsif value.keys.first == "literal"
append test_native_value(value["literal"])
else
raise "Don't understand this call w/ params in the test + how to render it as pseudocode"
end
unless index >= params.length - 1
append ", "
end
end
end
append ")", color: :blue
end
def render_step_call_may_fail call, indent: 0
write "try {", indent: indent, color: :light_blue
render_step_call call, indent: indent + 1, newline: false
write "} catch (e) {}", indent: indent, color: :light_blue
append " # Call is allowed to fail OK", color: :dark_grey
end
def render_extract_match call, indent: 0
pattern = call["pattern"]
variable = call["variable"]
groups = call["groups"]
if variable
note_test_variable variable
write "@#{variable}", indent: indent, color: :yellow
append " = "
append "output.Match(", color: :blue
append "/#{pattern}/", color: :cyan
append ")[", color: :blue
append "0", color: :cyan
append "]", color: :blue
end
if groups&.any?
groups.each_with_index do |variable_name_for_group, index|
note_test_variable variable_name_for_group
write "@#{variable_name_for_group}", indent: indent, color: :yellow
append " = "
append "output.Match(", color: :blue
append "/#{pattern}/", color: :cyan
append ")[", color: :blue
append "#{index}", color: :cyan
append "]", color: :blue
end
end
end
def render_assert_contains values, indent: 0, negative_assertion: false
values.each do |value|
write "assert ", color: :green, indent: indent
append "output.#{ "Not" if negative_assertion }Contains(", color: :green
if value["variable"]
append "@#{value["variable"]}", color: :yellow
elsif value["literal"]
append value["literal"].inspect, color: :cyan
else
raise "? #{values.inspect} ?"
end
append ")", color: :green
end
end
def render_assert_not_contains values, indent: 0
render_assert_contains values, indent: indent, negative_assertion: true
end
def render_assert_contains_any values, indent: 0, negative_assertion: false
return if values.nil? || values.none?
write "assert ", color: :green, indent: indent
append "output.#{ "Not" if negative_assertion }ContainsAny(", color: :green
values.each_with_index do |value, index|
if value["variable"]
append "@#{value["variable"]}", color: :yellow
elsif value["literal"]
append value["literal"].inspect, color: :cyan
else
raise "? #{values.inspect} ?"
end
append ", " unless index >= values.size - 1
end
append ")", color: :green
end
def append_get_environment_variable local_variable, env_variable
append "GetEnvironmentVariable('", color: :blue
append env_variable, color: :cyan
append "')", color: :blue
end
# meh.
def append_varname varname, color: :yellow, indent: 0, bold: false
append varname, color: color, indent: indent
end
end
class SamplePseudocoder < Pseudocoder
alias sample coder_object
def initialize sample
super
sample.validate! # hack ... this actual does some template finalization ... gross :)
end
def render!
generated_comment
br
start_region_tag
import_statements
br
sample_function
end_region_tag
br
main_function_with_cli_parsing
end
def generated_comment
write "# Generated Sample - #{title}", color: :dark_grey
end
def start_region_tag
write "# [START #{region_tag}]", color: :dark_grey
end
def end_region_tag
write "# [END #{region_tag}]", color: :dark_grey
end
def import_statements
write "import #{service_name}"
end
def sample_function
write comment_code(description) if description
sample_function_definition
br
instantiate_client
br
build_request
br
call_api
br
print_response
write "}"
end
def sample_function_definition
parameter_function_comments
write "function demo_#{underscore rpc_name}("
append sample_function_parameter_definitions
append ") {"
end
def parameter_function_comments
write comment_code("") if request_fields.any? { |field| field["input_parameter"] }
request_fields.select { |field| field["input_parameter"] }.each do |field|
comment_text = field["comment"]
write comment_code("@param #{field["input_parameter"]} #{comment_text}".strip, wrapping_indent: 1)
end
end
def comment_code text, wrapping_indent: 0, indent: 0
indent_spaces = " " * indent
wrapping_spaces = " " * wrapping_indent
comment_text = text.chomp.gsub("\n", "\n#{indent_spaces}# #{wrapping_spaces}")
colorize "# #{comment_text}", :dark_grey
end
def build_request indent: 1
write "#{var_keyword} #{param_name_code "request"} = new #{rpc_name}Request()", indent: indent
request_fields.each do |field|
field_name = field["field"].gsub("%", ".")
field_value = field["value"].is_a?(String) ? string_code(field["value"]) : field["value"].inspect
if field["comment"] && ! field["input_parameter"]
comment_text = field["comment"]
br
write comment_code(comment_text, indent: indent), indent: indent
end
if var_name = field["input_parameter"]
write "#{param_name_code "request"}.#{field_name} = #{param_name_code var_name}", indent: indent
else
write "#{param_name_code "request"}.#{field_name} = #{field_value}", indent: indent
end
end
end
def param_name_code param_name
colorize param_name, :light_cyan
end
def call_api
write "#{var_keyword} #{param_name_code "response"} = client.#{rpc_name}(request)", indent: 1
end
def print_response body = response_body, indent: 1
body.each do |item|
if item["loop"]
print_loop item, indent: indent
elsif item["define"]
print_define item, indent: indent
elsif item["comment"]
br
print_comment item, indent: indent
elsif item["print"]
print_print item, indent: indent
else
write "TODO #{item.keys.inspect}", indent: indent
end
end
end
# define:
def print_define hash, indent: 1
definition = hash["define"]
equals_index = definition.index "="
var_name = definition[0..equals_index-1].strip
value = definition[equals_index+1..-1].strip
value.sub! "$resp", "response"
write "#{var_keyword} #{param_name_code var_name} = #{value}", indent: indent
br
end
# comment:
def print_comment hash, indent: 1, color: :dark_grey
comment = hash["comment"]
text = comment.shift
while comment.any?
variable_name = comment.shift
variable_name = variable_name.split("_").map(&:capitalize).join
text.sub! "%s", "#{variable_name}"
end
text.lines.each do |line|
write "# #{line.chomp}", indent: indent, color: color
end
end
# print:
def print_print hash, indent: 1
print_out = hash["print"]
text = print_out.shift
interpolated = false
while print_out.any?
variable_name = print_out.shift
text.sub! "%s", "{#{variable_name}}"
interpolated = true
end
if interpolated
write %{print($#{string_code text})}, indent: indent
else
write %{print(#{string_code text})}, indent: indent
end
end
# loop:
def print_loop hash, indent: 1
collection_name = hash["loop"]["collection"]
variable_name = hash["loop"]["variable"]
write "for (#{variable_name} in #{collection_name}) {", indent: indent
print_response hash["loop"]["body"], indent: indent + 1
write "}", indent: indent
end
def string_code text
colorize %{"#{text}"}, :light_blue
end
def var_keyword
colorize("var", :light_)
end
def instantiate_client
write "#{var_keyword} #{param_name_code "client"} = new #{service_name}Client()", indent: 1
end
def sample_function_parameter_definitions
params = []
request_fields.select { |field| field.has_key? "input_parameter" }.each do |field|
value = field["value"]
if value.is_a? String
value = string_code value
else
value = value.inspect
end
params << "#{param_name_code field["input_parameter"]} = #{value}"
end
params.join(' ')
end
def main_function_with_cli_parsing
write "main() {"
write "# TODO", color: :dark_grey
write "}"
end
end
end
# sample_composer.rb
SampleComposer.run if $PROGRAM_NAME == __FILE__
__END__
sample-composer [*.yaml] [-o output/] [-r region_tag]
Define GAPIC code samples using templates for reusability.
Options:
* Sample composition files (YAML)
Only files ending with .yml and .yaml
.samples.yaml, Automatically loaded if found
samples.yaml in the current directory
(when no arguments passed)
Directories are searched recursively
-o, --output The location to output samples.
-o dir/ output one sample per file in dir/
-o file.yaml output samples in file.yaml
-o STDOUT output samples to STDOUT
-o x_gapic.yaml output into legacy _gapic config file
The --stdout option defaults to STDOUT
-r, --region_tag Specific region tag(s) to output.
-r tag_name output region tag
-r /tag.*/ output region tags matching pattern
May be defined multiple times.
-r tag_one -r tag_two -r /pattern/
-t, --test-suite Filter name of test suites output.
-t "Sweet name" output test suites matching exactly
-t /partial.*/ output test suites matching pattern
May be defined multiple times.
-t "Suite One" -t "Sweet Two" -t /pattern/
-l, --list Print available templates + samples + tests
-r and -t can be used to filter list
-p, --preview Renders pseudocode representation of the tests and samples
-o STDOUT is useful for previewing samples
--pp Renders pseudocode representation of the tests and samples
to STDOUT and refreshes every second
--ppp Renders pseudocode representation of the tests and samples
which include '@' in the name of the sample region tag
or test suite name to STDOUT and refreshes every second
-u, --usage Print full usage information with syntax reference
-h, --help Print this help message
-v, --version Print sample-composer version
Reference:
Sample composition YAML files may contain any of the following top-level keys:
composer: One or more sample-composition actions to perform
samples: Sample configurations (each must have unique region_tag)
templates: Template configurations (each must have unique name)
tests: TODO
[Composer Configuration]
Defines your sample set.
```
sample_compositions:
- output: samples/
region_tags: ["specific_tag", /^tag_pattern/]
paths:
- ../sample_definitions/
- ../sample_templates/
- ../../common_templates
```
Sample compositions may contain multiple inputs/outputs configurations.
For example, you might want to output GAPIC code sample configuration files
but also inject legacy formatted samples into a _gapic.yaml config:
```
sample_compositions:
- output: ..
paths: _templates/
- output: ../../product_gapic.yaml
paths: _templates/
```
[Sample Configuration]
Defines samples.
```
samples:
- region_tag: my_full_region_tag
title: my sample
service: servicename
rpc: methodname
request:
- field: config.some_field
value: "field value"
comment: |
this comment described how this field is used
input_parameter: some_field
response:
- print: ["hello, world."]
```
Every sample must have a unique region_tag (cannot be composed)
Request fields can be configured to read content from a local file.
```
region_tag: my_full_region_tag
request:
- field: config.content_bytes
value: "local_file.mp3"
value_is_file: true
input_parameter: file_path
```
Samples can be composed of one of more templates which can:
- provide the service: and/or rpc:
- provide a prefix and/or suffix which will be added to title:
- provide request: fields
- provide response: body
```
region_tag: my_full_region_tag
template:
- template_one
- template_two
```
Samples can override the values of request: fields included by templates:
```
region_tag: my_full_region_tag
template:
- (template which adds request fields A and B)
request:
- field: A
value: "changed the value"
```
Samples can include the `response:` of any template and provide snippets
to include in placeholders of the body of that template's response.
Samples can provide snippets to include in response: included by template:
```
region_tag: my_full_region_tag
response:
- print: ["Hello, world!"]
- include: (name of template that has a response:)
- print: ["Goodbye"]
```
To replace placeholder values in template response bodies:
```
region_tag: my_full_region_tag
response:
- print: ["Hello, world!"]
- include: (name of template that has a response:)
with:
variable_name:
# The placeholder in the template body will be replaced by
# these response items.
- comment: ["..."]
- print: ["..."]
- print: ["Goodbye"]
```
[Template Configuration]
Defines templates.
```
templates:
- name: my_template
service: (including the template will add this service)
rpc: (including the template will add this rpc)
request:
- field: (including the template will include these request fields)
value: ...
- field: ...
response:
# can be included in sample response via `- include: [template name]`
- print: [""]
- loop:
collection: x
variable: y
body:
# can be replaced in sample response via `with: [placeholder name]`
- yield: placehodler name
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment