Skip to content

Instantly share code, notes, and snippets.

@nehiljain

nehiljain/jinja-example-airflow-part-2.py

Created June 14, 2018 14:05
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save nehiljain/12d1c90fcf9fe53068a7fddf774558ce to your computer and use it in GitHub Desktop.
Save nehiljain/12d1c90fcf9fe53068a7fddf774558ce to your computer and use it in GitHub Desktop.
Download ZIP
Code Sample for Blog
Raw
jinja-example-airflow-part-2.py
# suppose my data file name has the following format "datatfile_YYYY_MM_DD.csv"; this file arrives in S3 every day.
file_suffix = "{{ execution_date.strftime('%Y-%m-%d') }}"
bucket_key_template = 's3://[bucket_name]/datatfile_{}.csv'.format(file_suffix)
file_sensor = S3KeySensor(
 task_id='s3_key_sensor_task',
 poke_interval=60 * 30, # (seconds); checking file every half an hour
 timeout=60 * 60 * 12, # timeout in 12 hours
 bucket_key=bucket_key_template,
 bucket_name=None,
 wildcard_match=False,
 dag=dag
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment