Skip to content

Instantly share code, notes, and snippets.

View ascii1011's full-sized avatar

Christopher Harty ascii1011

View GitHub Profile
@ascii1011
ascii1011 / airflow.cfg
Created March 1, 2019 15:05
Airflow 1.10.1 + Cent7.3
[core]
# The home folder for airflow, default is ~/airflow
airflow_home = /opt/airflow
# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository
# This path must be absolute
dags_folder = /opt/airflow/dags
sudo apt-get update -y && apt-get upgrade
sudo apt-get install -y virtualbox, vagrant, ansible, python-pip
pip install awscli --upgrade --user
# aws cred file "~/.aws/credentials" should look like this:
[default]
aws_access_key_id = some_hash1
aws_secret_access_key = some_hash2
[some_profile_name]
#!/usr/bin/bash
PACKAGES_INSTALLED="~/packages_installed.txt"
sudo apt-get update -y >> $PACKAGES_INSTALLED
sudo apt-get upgrade >> $PACKAGES_INSTALLED
sudo apt-get install -y screen git wget >> $PACKAGES_INSTALLED
sudo apt-get install -y emacs >> $PACKAGES_INSTALLED
@ascii1011
ascii1011 / airflow_xcom_global_pull_dag.py
Last active September 29, 2017 18:41
airflow1.7 passing global xcom values
"""
Intended use with Airflow 1.7.1.3
Please use at your own risk. This file is for research only and not real data.
"""
from datetime import datetime
import logging
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
"""
### Working Minimum Airflow 1.8 install on CentOS 7.3.1611 ###
# Christopher R. Harty
# 5-17-2017
# Airflow src: /usr/lib/python2.7/site-packages/airflow
# Airflow dags: ~/airflow/dags
# with:
# - epel(x86_64) 7.9
# - Python 2.7.13
# - PostgreSQL 9.6 Server
# - setuptools 3.4.4
# aws - ec2 - ubuntu 16.04 LTS
# (HVM) / ami-f4cc1de2
HOSTS_FILE="/etc/hosts"
PUBLIC_ADAPTER="eth0"
echo "Add appropriate mapping for local IP, since EC2 instance. So if IP = 172.30.1.5, then..."
echo "172.30.1.5 ip-172-30-1-5 >> $HOSTS_FILE"
echo ""
echo "Attempting to add IP to Hosts file..."
IP=$(ip addr show $PUBLIC_ADAPTER | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)
---
- name: SSH tightening
hosts: all
sudo: True
tasks:
- name: Disable root's ssh account
action: >
lineinfile
dest=/etc/ssh/sshd_config
(function($) {
$.fn.levenshteinDistance = function(u, v) {
var m = u.length;
var n = v.length;
var D = [];
for(var i = 0; i <= m; i++) {
D.push([]);
for(var j = 0; j <= n; j++) {
D[i][j] = 0;
}