Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@ericwong3
ericwong3 / airflow-vs-prefect.md
Last active July 19, 2022 19:32
Airflow vs Prefect

(This is written after I implemented a simple, distributed pipeline on Airflow, and after spending hours on reading Prefect docs and looking for equivalent functions. The comparison might be slightly biased towards Airflow due to being more familiar with it, but best effort is made to ensure information about Prefect is accurate)

DAG and Tasks-related

Dynamic/Mapped Tasks

This refers to some tasks that requires unknown number of parallel runs unknown before execution. For example, to query the database and do "for each row, execute another task". This is important for parallelization of large tasks.

Prefect (+2) - Supported, it has dedicated Mapped Task feature