Skip to content

Instantly share code, notes, and snippets.

View tdhopper's full-sized avatar
©️
𝔀𝓸𝓻𝓴𝓲𝓷𝓰 𝓱𝓪𝓻𝓭

Tim Hopper tdhopper

©️
𝔀𝓸𝓻𝓴𝓲𝓷𝓰 𝓱𝓪𝓻𝓭
View GitHub Profile
@xofer
xofer / 0-storm-proxy-readme.md
Last active June 28, 2021 11:43
Storm UI Nginx conf for Nimbus with logviewer proxy

We have a setup that I assume is quite common: A publicly accessible Nimbus running Storm UI. The worker nodes can only be accessed from the Nimbus (via the LAN). All the nodes have internal DNS names (i.e. node.lan.example.com), which is set in the configuration files; they use these DNS names to reach each other. The Nimbus has an external DNS name (storm.example.com) for public access. The Nimbus's UI is behind an Nginx proxy, which provides HTTP Auth and HTTPS.

Because of this setup, the logviewer links in the UI do not work. In order to fix this, we employ an elaborate hack shown in the conf file below. It uses ngx_http_substitutions_filter_module to rewrite content returned by the Storm UI and some complicated URL rewrite tricks to proxy the workers' logviewers through through the Nimbus.

@topiaruss
topiaruss / pytest.ini
Last active June 21, 2016 15:57
Experimenting with minimal test frameworks for spout and bolt
[pytest]
python_paths = src/bolts src/spouts
testpaths = src/tests
@bishboria
bishboria / springer-free-maths-books.md
Last active March 22, 2024 11:19
Springer made a bunch of books available for free, these were the direct links
@rampage644
rampage644 / airflow_deploy_design.md
Created October 6, 2015 20:53
Airflow flows deployment

Introduction

This document describes how Airflow jobs (or workflows) get deployed onto production system.

Directory structure

  • HOME directory:/home/airflow
  • DAG directory: $HOME/airflow-git-dir/dags/
  • Config directory: $HOME/airflow-git-dir/configs/
  • Unittest directore: $HOME/airflow-git-dir/tests/. Preferable, discoverable by both nose and py.test
  • Credentials should be accessed by by some library
@lojic
lojic / WestminsterShorterCatechism.txt
Created July 24, 2015 13:53
A plain text version of the Westminster Shorter Catechism with original proof texts.
The Westminster Shorter Catechism
=================================
The original text of 1647, with the Assembly's proof texts.
1. What is the chief end of man?
Man's chief end is to glorify God,(1) and to enjoy him for ever.(2)
(1) I Cor. 10:31; Rom. 11:36. [ Ps 86; Is 60:21; Rev 4:11 ]
@hadley
hadley / ds-training.md
Created March 13, 2015 18:49
My advise on what you need to do to become a data scientist...

If you were to give recommendations to your "little brother/sister" on things that they need to do to become a data scientist, what would those things be?

I think the "Data Science Venn Diagram" (http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram) is a great place to start. You need three things to be a good data scientist:

  • Statistical knowledge
  • Programming/hacking skills
  • Domain expertise

Statistical knowledge

@xofer
xofer / hostkey.sh
Last active August 29, 2015 14:13
bash function to update host keys
hostkey () {
for HOST in "$@"; do
LIST=$HOST
PARTS=(${HOST//./ })
if [[ ${#PARTS[*]} -gt 2 && "${PARTS[0]}" != "$HOST" ]]; then
LIST="$LIST ${PARTS[0]}"
fi
IP=$(dig +short $HOST)
if [ -n "$IP" ]; then
LIST="$LIST $IP"
@wrobstory
wrobstory / gist:98033556dd590132852e
Created September 8, 2014 04:17
PIP ALL THE THINGS
#!/bin/bash
virtualenv .env &&
source .env/bin/activate &&
brew install libevent &&
brew install libyaml &&
pip install cython &&
pip install numpy &&
pip install scipy &&
pip install pandas &&
pip install matplotlib &&
@amontalenti
amontalenti / tb.py
Created August 8, 2014 05:31
extract Python Traceback from log files (based on https://gist.github.com/originell/1923003)
"""
Extract unique Python-Exceptions with their Traceback from a log/text file.
Usage::
python extract_exceptions.py -f logfile.txt
Furthermore it supports excluding exceptions you don't want to have::