Skip to content

Instantly share code, notes, and snippets.

@osule

osule/airflow-debug-container.md

Created July 1, 2021 11:08
Show Gist options
  • Save osule/c4b5f364e7556d4e4130aeacd26ffe2d to your computer and use it in GitHub Desktop.
Save osule/c4b5f364e7556d4e4130aeacd26ffe2d to your computer and use it in GitHub Desktop.
Download ZIP
Airflow container task instance profiling
Raw
airflow-debug-container.md

Install procps

docker exec -u 0 -ti <container-id> bash -c "apt-get update && apt-get install -y procps"

docker exec -u 0 -ti cbe1c06b96c7 bash -c "apt-get update && apt-get install -y procps"

Install py-spy

docker exec -ti <container-id> pip install py-spy

docker exec -ti cbe1c06b96c7 pip install py-spy

Needs PSYS_TRACE enabled for the running container

Profile the worker

docker exec -ti <container-id> py-spy top -p <process-id> -s

docker exec -ti cbe1c06b96c7 py-spy top -p 75 -s

Run task instance

    docker exec -ti <container-id> bash  -c "bash /bin/pre-container-run.sh backfill -s 2021-05-31T10:00 -e 2021-05-31T11:00 -t add_distance_utilization drt_data_import"
    
docker exec -ti cbe1c06b96c7 bash  -c "bash /bin/pre-container-run.sh backfill -s 2021-05-31T10:00 -e 2021-05-31T11:00 -t add_distance_utilization drt_data_import"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment