Skip to content

Instantly share code, notes, and snippets.

@naturalett

naturalett/airflow_bash.py

Last active June 5, 2023 21:56
Download ZIP

Revisions

  1. Lidor Ettinger revised this gist Jun 5, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion airflow_bash.py
    View file
    Original file line number Diff line number Diff line change
    @@ -3,7 +3,7 @@
    from airflow import DAG
    from airflow.operators.bash import BashOperator

    NUM_TASKS = 100
    NUM_TASKS = 500

    default_args = {
    'owner': 'airflow',
  2. Lidor Ettinger created this gist Jun 5, 2023.
    29 changes: 29 additions & 0 deletions airflow_bash.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,29 @@
    from datetime import datetime, timedelta

    from airflow import DAG
    from airflow.operators.bash import BashOperator

    NUM_TASKS = 100

    default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['admin@admin.com'],
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
    }

    with DAG(
    'bash_tasks',
    default_args=default_args,
    start_date=datetime(2021, 1, 1),
    schedule_interval=None,
    catchup=False
    ) as dag:
    for i in range(0, NUM_TASKS):
    BashOperator(
    task_id=f'task_{i}',
    bash_command='date'
    )