Skip to content

Instantly share code, notes, and snippets.

@apeckham
Last active April 27, 2019 22:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save apeckham/97d6cda56ecb759bd6c964d40ec261a1 to your computer and use it in GitHub Desktop.
Save apeckham/97d6cda56ecb759bd6c964d40ec261a1 to your computer and use it in GitHub Desktop.
Airflow DAG in Hy, synced from git repo with google cloud build
(import [airflow [DAG]]
[airflow.operators.dummy_operator [DummyOperator]]
[airflow.operators.python_operator [PythonOperator]]
[airflow.hooks.base_hook [BaseHook]]
[airflow.contrib.operators.slack_webhook_operator [SlackWebhookOperator]]
[datetime [datetime]])
(defn print-hello [] "Hello world")
(with [dag (DAG "hello_worl2" :description "Simple DAG"
:schedule_interval "0 12 * * *"
:start_date (datetime 2017 3 20)
:catchup False)]
(>> (DummyOperator :task_id "dummy_task" :retries 3)
(PythonOperator :task_id "hello_task" :python_callable print-hello)
(SlackWebhookOperator :task_id "hello_slack"
:http_conn_id "slack"
:webhook_token (. (.get_connection BaseHook "slack") password)
:message "Hey from Airflow"
:username "airflow")))
# Based on https://engineering.adwerx.com/sync-a-github-repo-to-your-gcp-composer-airflow-dags-folder-2b87eb065915
steps:
- name: hylang:0.13.1
args: ['bash', '-c', 'for i in */*.hy; do hy2py $i >${i/.hy/.py}; done']
- name: ubuntu
args: ['bash', '-c', "echo '$COMMIT_SHA' > REVISION.txt"]
- name: gcr.io/cloud-builders/gsutil
args:
- '-m'
- 'rsync'
- '-x'
- '.*\.hy$'
- '-d'
- '-r'
- 'dags'
- 'gs://${_GCS_BUCKET}/dags'
- name: gcr.io/cloud-builders/gsutil
args:
- '-m'
- 'rsync'
- '-x'
- '.*\.hy$'
- '-d'
- '-r'
- 'plugins'
- 'gs://${_GCS_BUCKET}/plugins'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment