Skip to content

Instantly share code, notes, and snippets.

View Markbnj's full-sized avatar

Mark Betz Markbnj

View GitHub Profile
@Markbnj
Markbnj / getch.py
Created August 26, 2015 17:05
Get a single key press from the terminal in python
import termios
import sys
import tty
def _getch():
fd = sys.stdin.fileno()
old_settings = termios.tcgetattr(fd)
try:
tty.setraw(fd)
ch = sys.stdin.read(1)
@Markbnj
Markbnj / environment-specific-settings.py
Last active August 29, 2015 14:27
Environment-specific settings for python 2 modules
# Environment-specific settings for python 2 modules
# This code placed in __init__.py will allow a module to maintain
# environment-specific settings in files located in a settings folder
# within the module.
#
# Example:
#
# ├── mymodule.py
# ├── __init__.py
@Markbnj
Markbnj / ec2-discover-instances.py
Last active August 29, 2015 14:27
Discover ec2 instances by tag and tag value
from boto import ec2
class Resource(object):
"""
Represents a single resource returned from the Discovery class's
get_resources method.
"""
def __init__(self, host_name, private_host_name, ip, private_ip, tags):
self.__host_name = host_name
self.__private_host_name = private_host_name
@Markbnj
Markbnj / debug-scrapyd-ipdb.py
Last active August 29, 2015 14:27
User ipdb to debug a running spider in scrapyd
# The scrapyd daemon adds a lot of power and flexibility to scrapy, but it also adds a layer of indirection
# (another process) that can make debugging a running spider troublesome. Use this code to launch scrapyd
# in ipdb (or pdb). You'll be able to breakpoint and catch exceptions inside the spiders it launches.
import scrapyd.runner
import ipdb
ipdb.runcall(scrapyd.runner.main)
@Markbnj
Markbnj / simple-google-places-wrapper.py
Created August 11, 2015 19:43
A simple google places wrapper using requests
import requests
import json
GOOGLE_API_KEY = "<A google API key>"
TEXT_SEARCH_TEMPLATE = "https://maps.googleapis.com/maps/api/place/textsearch/json?key={}&query={}"
NEARBY_SEARCH_TEMPLATE = "https://maps.googleapis.com/maps/api/place/nearbysearch/json?key={}&location={},{}&radius={}"
DETAIL_SEARCH_ID_TEMPLATE = "https://maps.googleapis.com/maps/api/place/details/json?key={}&placeid={}"
@Markbnj
Markbnj / set-proxy.sh
Created August 11, 2015 20:26
Set http/https proxy address on a console session
# sets the http/s proxy for the current shell to
# the passed argument, which should be in the form
# http://xxx.xxx.xxx.xxx:xxxx
export http_proxy=$1
export https_proxy=$http_proxy
export no_proxy="localhost,127.0.0.1"
@Markbnj
Markbnj / set-up-phantomjs.py
Created August 11, 2015 20:45
Set up phantomjs driver for selenium, example
# An example showing how to set up phantomjs for use with Selenium
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
# These arguments will be passed on the phantomjs command line
# when webdriver launches it. See the reference page at
# http://phantomjs.org/api/command-line.html for more settings.
#
@Markbnj
Markbnj / update-launch-es.sh
Created August 11, 2015 21:06
Update elasticsearch.yml and launch the service
# Script to update elasticsearch.yml and launch the service. In this
# scenario the service is being launched inside a container on ec2
# but the technique is easily generalizable.
#
# Expects the following environment vars to be set inside the container
#
# HOST_IP - the IP address of the host running the container
# AWS_ACCESS_KEY - the AWS account access key
# AWS_SECRET_KEY - the AWS account secret key
# CLUSTER_NAME - the name of the ES cluster to join
@Markbnj
Markbnj / Dockerfile
Created August 11, 2015 21:12
Simple dockerfile for an Ubuntu-based base image for python services
# DockerFile for building a base image for python services
FROM ubuntu:15.04
MAINTAINER Someone <someone@some.where>
# common stuff
RUN echo "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d && \
mkdir /home/root && \
dpkg-reconfigure locales && \
locale-gen en_US en_US.UTF-8 && \
@Markbnj
Markbnj / haproxy-install.sh
Created August 11, 2015 21:13
Install haproxy
sudo apt-add-repository ppa:vbernat/haproxy-1.5
sudo apt-get update
sudo apt-get -qy install haproxy
sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig
# To enable logging edit /etc/rsyslog.conf to match the following:
#
# provides UDP syslog reception
# $ModLoad imudp
# $UDPServerRun 514