Skip to content

Instantly share code, notes, and snippets.

View jleto's full-sized avatar

Justin Leto jleto

  • AWS
  • Manhattan, New York
  • X @letojj
View GitHub Profile
@sebradloff
sebradloff / Dockerfile
Created March 3, 2018 18:26
Setup for remote logs s3, airflow 1.9.0
# Showing the necessary part of adding the logging config to the python path
COPY config/log_config.py config/log_config.py
COPY config/__init__.py config/__init__.py
RUN chown -R airflow: ${AIRFLOW_HOME}
ENV PYTHONPATH ${PYTHONPATH}:/usr/lib/python2.7/site-packages/:${AIRFLOW_HOME}/config/
#!/usr/bin/env bash
sudo apt-get update -y
sudo apt-get autoremove -y
sudo apt-get upgrade -y
sudo apt-get install python-pip -y
pip install -U pip
sudo apt-get install python-virtualenv -y
sudo apt-get install python-dev -y
sudo apt-get install libpq-dev -y
sudo apt-get install build-essential -y
@rosiehoyem
rosiehoyem / getting-started-with-airflow.md
Last active January 14, 2023 18:09
Getting Started With Airflow

Install Airflow

1. Install Airflow

Follow the installation instructions on the Airflow website.

Update Airflow Configurations

To configure Airflow to use Postgres rather than the default Sqlite3, go to airflow.cfg and update this configuration to LocalExecutor:

# The executor class that airflow should use. Choices include
@zacgx
zacgx / airflow-python3.sh
Last active July 19, 2022 15:15
Installing Airflow with CeleryExcuter, using PostgreSQL as metadata database and Redis for Celery message broker
# this script has been tested and worked in a freshly installed Ubuntu 16.04 and 16.10
# it assumes that you are running airflow in a private netowrk and no need to be worry about outside access
# if that's not the case, the lines for PostgreSQL and Redis in this script need to be updated accordingly
# run as root
sudo su
# initial system updates and installs
apt-get update && apt-get upgrade -y && apt-get autoremove && apt-get autoclean
apt-get -y install build-essential binutils gcc make git htop nethogs tmux
@kozikow
kozikow / airflow_config.go
Last active March 8, 2018 01:53
Airflow config
// Generate airflow config to stdout.
// To start service the from scratch, put this in src/airflow_config/airflow_config.go and run:
// export GOPATH=$(pwd)
// export PATH=$PATH:$GOPATH/bin
// go get k8s.io/client-go/1.4/kubernetes
// go install airflow_config && airflow_config > airflow.yaml
// kubectl create -f airflow.yaml
package main
import (