dylanbstorey

Pattern is useful when you want to trigger many copies of the same DAG with different parameters.

Driver DAG

This dag is run on a schedule, fetches the appropriate configurations needed and then triggers the external DAG as a configuration. There is a final "wait" step that is optional that simply ensures there is a blocking step at the end of the submissions to ensure that concurrency is kept at "1" for the parameterized DAG. This pattern uses the dynamic task mapping paradigm to determine how many external dags should be triggered during execution. As a general note, the generate_list task could be replaced with any operator even ones that use external service calls to determine if a specific configuration should be run at this execution cycle.

dylanbstorey

 breeze start-airflow --verbose                                                                                                                                                                                                                                                                                                   ─╯
The output of the asset compilation is stored in: /home/dstorey/airflow/.build/www/asset_compile.out


Running command: <docker info>


Working directory /home/dstorey/airflow

dylanbstorey

use git2::{Cred, RemoteCallbacks, Repository};

use git2;
use git2_credentials::CredentialHandler;
use tempfile;
use std::path::Path;

/// works after running ╰─❯ ssh-keygen -t ed25519 -C "dylan.storey@gmail.com.com"       
/// and uploading to github/gitlab