Skip to content

Instantly share code, notes, and snippets.

@peacing

peacing/airflow-dag-simple.py

Last active May 25, 2021 11:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save peacing/58b0774b9d9192747824708f0587cc17 to your computer and use it in GitHub Desktop.
Save peacing/58b0774b9d9192747824708f0587cc17 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
airflow-dag-simple.py
from airflow.decorators import dag
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
@dag()
def workflow():
spark_submit = SparkSubmitOperator(
task_id='spark_submit',
application='path/to/job/script.py',
application_args=['s3://example-bucket/path/to/input',
's3://example-bucket/path/to/output' ] )
sample_dag = workflow()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment