Skip to content

Instantly share code, notes, and snippets.

@kaxil
Last active December 10, 2021 09:39
Show Gist options
  • Save kaxil/61f41dd87a69230d1a637dc3a1d2fa2c to your computer and use it in GitHub Desktop.
Save kaxil/61f41dd87a69230d1a637dc3a1d2fa2c to your computer and use it in GitHub Desktop.
Using Airflow Json Variables
from airflow.models import Variable
# Common (Not-so-nice way)
# 3 DB connections when the file is parsed
var1 = Variable.get("var1")
var2 = Variable.get("var2")
var3 = Variable.get("var3")
# Recommended Way
# Just 1 Database call
dag_config = Variable.get("dag1_config", deserialize_json=True)
dag_config["var1"]
dag_config["var2"]
dag_config["var3"]
# You can directly use it Templated arguments {{ var.json.my_var.path }}
bash_task = BashOperator(
task_id="bash_task",
bash_command='{{ var.json.dag1_config.var1 }} ',
dag=dag,
)
@kaxil
Copy link
Author

kaxil commented Dec 10, 2021

@saveriogzz The connections to db outside of dag and task objects should be as minimum as possible.

You can also use secrets backend i.e store Airflow Variable in environment variable - https://airflow.apache.org/docs/apache-airflow/1.10.10/concepts.html#storing-variables-in-environment-variables and then you can have as many variable outside dag objects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment