Skip to content

Instantly share code, notes, and snippets.

View rampage644's full-sized avatar

Sergei Turukin rampage644

53 followers · 0 following
View GitHub Profile
@rampage644
rampage644 / airflow_deploy_design.md
Created October 6, 2015 20:53
Airflow flows deployment

Introduction

This document describes how Airflow jobs (or workflows) get deployed onto production system.

Directory structure

  • HOME directory:/home/airflow
  • DAG directory: $HOME/airflow-git-dir/dags/
  • Config directory: $HOME/airflow-git-dir/configs/
  • Unittest directore: $HOME/airflow-git-dir/tests/. Preferable, discoverable by both nose and py.test
  • Credentials should be accessed by by some library
@rampage644
rampage644 / talk2907.md
Last active August 18, 2016 08:08
Shub talk

My Shub Talks

29/07 - Introduce workflow manager

Brief intro

First, I'd like to say hello to everyone and thank for coming.

@rampage644
rampage644 / ds-dev.md
Created August 2, 2016 13:21
DS dev process comments

Dataservices spider development process

Disclaimer: Everything described in this document is my personal opinion that doesn't have to be true for everyone.

Common

Key information