Skip to content

Instantly share code, notes, and snippets.

@ilanrosenfeld7
Last active February 4, 2020 18:00
Show Gist options
  • Save ilanrosenfeld7/c653bb92d94f4e8bf8ef7b62b1415f9a to your computer and use it in GitHub Desktop.
Save ilanrosenfeld7/c653bb92d94f4e8bf8ef7b62b1415f9a to your computer and use it in GitHub Desktop.
project_id = '$PROJECT_ID'
dag_id = '$DAG_ID'
origin_bucket = '$ORIGIN_BUCKET'
target_bucket = '$TARGET_BUCKET'
yesterday = datetime.datetime.now() - datetime.timedelta(days=1)
default_dag_args = {
'start_date': yesterday,
'retries': 1,
'retry_delay': datetime.timedelta(seconds=5),
'project_id': project_id,
}
file_copy_dag = models.DAG(
dag_id,
description='File copy DAG',
catchup=False,
default_args=default_dag_args,
schedule_interval=None,
)
file_copy_task = PythonOperator(
task_id='file_copy',
python_callable=move_files_between_buckets,
op_args=[origin_bucket, target_bucket],
provide_context=True,
email_on_failure=False,
dag=file_copy_dag
)
file_copy_task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment