simplified-airflow

simplified-airflow.md

Idea

https://airflow.apache.org/docs/apache-airflow/stable/index.html

Py-dag

https://github.com/thieman/py-dag

Problem Statement

Create a simple DAG using py-dag package.

Each node to have custom function/logic/lambda for evaluation. Each node outputs success/failure

Build workflow executor which traverses the graph and executes the workflow.

Improvements

• WorkflowRun to organise runtime data (max_retries) and results
• Add method to check if the entire workflow is successful

Example data:

https://stackoverflow.com/questions/54903222/implementing-a-dag-in-python

graph = {
  'root': ['a'], 
  'a': ['b', 'e'], 
  'b': ['c', 'd'], 
  'd': ['e'], 
  'c': [], 
  'e': [] 
}

processes = { 
  "root": lambda: print("root"), 
  "a":    lambda: print("A"), 
  "b":    lambda: print("B"), 
  "c":    lambda: print("C"), 
  "d":    lambda: print("D"), 
  "e":    lambda: print("E") 
}