Skip to content

Instantly share code, notes, and snippets.

@thirteen37
Created December 5, 2019 06:09
Show Gist options
  • Save thirteen37/5ea8b7e59a201158b42c4ec1b5b570b6 to your computer and use it in GitHub Desktop.
Save thirteen37/5ea8b7e59a201158b42c4ec1b5b570b6 to your computer and use it in GitHub Desktop.
Example Clockwork pipeline spec
# name of the pipeline
name: mypipeline
# default Docker image for each task
image: alpine
# default resources required for each task
resources:
cpu: 2
memory: 4GB
disk: 80GB
# default working directory for each task, can reference a Git repo
workdir:
type: git
src: https://source.golabs.io/ds/example
ref: master
# default variables for each task
variables:
# user-specified environment variable
KEY_FROM_USER: value_from_user
# empty value means the variable will be retrieved from Airflow
KEY_FROM_RUNTIME:
# list of tasks that make up this pipeline
tasks:
- # unique name to identify the task in the pipeline
name: task1
# Docker image for this task runtime
image: debian
# command to run in the task
command: echo "task1 using default resources"
- name: task2
command: echo "task2 using more resources"
# specifies which tasks should succeed before running the current task
dependsOn: ["task1"]
# resources for this task
resources:
cpu: 8
memory: 120GB
disk: 80GB
# additional variables for this task
variables:
mykey: myvalue
# working directory for this task
workdir:
type: git
src: https://source.golabs.io/ds/example
ref: mybranch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment