Skip to content

Instantly share code, notes, and snippets.

@philihp

philihp/ec2-airflow.md

Last active April 14, 2023 00:34
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save philihp/1825691f7533c8435a766a1d0b0452bb to your computer and use it in GitHub Desktop.
Save philihp/1825691f7533c8435a766a1d0b0452bb to your computer and use it in GitHub Desktop.
Download ZIP
How to get Apache Airflow running on a bare AWS EC2 node
Raw
ec2-airflow.md

How to get Apache Airflow running on a bare AWS EC2 node

Amazon Managed Workflows for Apache Airflow (AWS MWAA) is prohibitively expensive for someone tinkering around on a personal account, and unprepared to pay a few hundred dollars a month at a minimum for their smallest environment. Here are the commands I used to get it running on Ubuntu 20.

Prepare the box

sudo apt-get update

Get Python

sudo apt-get install python3-pip
pip3 install --upgrade pip==20.2.4

Download Airflow

This home directory is actually the default, but I figured I'd put it here since it's probably relevant.

export AIRFLOW_HOME=~/airflow
pip3 install apache-airflow

Then disconnect and reconnect to have your shell restarted.

Initialize Airflow

airflow db init
airflow users create \
     --username philihp \
     --firstname Philihp \
     --lastname Busby \
     --role Admin \
     --email philihp@philihp.com

Start Processes

Do these on two separate terminals.

airflow webserver --port 8080
airflow scheduler
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment