airflow_download instructions

airflow_instructions.md

airflow_complete_tutorial

The right way to initialize the airflow from the beginning

1.build your workplace

  mkdir xx/airflow_test

2. build the virtual environment

  python -m venv venv_test(name)
  source venv_test/bin/activate

3. download the airflow

   (venv_test) pip install apache-airflow

4. export the environment

(venv_test) $ cd xx/airflow_test
(venv_test) $ mkdir airflow_home
(venv_test) $ export AIRFLOW_HOME=$('pwd')/airflow_home

5. initialize the airflow

(venv_test) $ airflow initdb
(venv_test) $ airflow webserver

6. build new dags

   mkdir dags

build new dag under this folder

7. update the airflow UI

modify the max_threads in airflow.cfg from 2 to 1.

Open a new terminator

   cd xxx/airflow_test
   source venv_test/bin/activate
(venv_test) export AIRFLOW_HOME=$('pwd')/airflow_home
(venv_test) airflow initdb
(venv_test) airflow scheduler

If there is no updation, restart webserver and scheduler(export airflow_home for both of them!)

8. (Optional) Add the path locally.

vim ~/.bashrc

add the line:

export AIRFLOW_HOME = /wordspace/airflow_home

Additional

1.If you want to delete all the historical data:

airflow resetdb