Skip to content

Instantly share code, notes, and snippets.

@p5k6

p5k6/test-precondition-transform.yml

Created May 19, 2016 16:35
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save p5k6/e32242ad1345838229025239c72c68b0 to your computer and use it in GitHub Desktop.
Save p5k6/e32242ad1345838229025239c72c68b0 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
test-precondition-transform.yml
name: qa-test-transform
frequency: daily
load_time: 08:00
description: |
testing out new transform step with precondition
steps:
- step_type: extract-s3
name: ops-input
directory_uri: s3://my-s3-bucket-name/app/qa/full/#{format(@scheduledStartTime, 'YYYY-MM-dd')}/
- step_type: transform-s3-prefix
name: cat_files
output_path: s3://my-s3-bucket-name/app/qa/data-pipeline/line-count.out
input_node: ops-input
command: cat $INPUT1_STAGING_DIR/part* | wc -l
s3_precondition: s3://my-s3-bucket-name/app/qa/full/#{format(@scheduledStartTime, 'YYYY-MM-dd')}/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment