Skip to content

Instantly share code, notes, and snippets.

Avatar

Mark Betz Markbnj

View GitHub Profile
@Markbnj
Markbnj / environment-specific-settings.py
Last active Aug 29, 2015
Environment-specific settings for python 2 modules
View environment-specific-settings.py
# Environment-specific settings for python 2 modules
# This code placed in __init__.py will allow a module to maintain
# environment-specific settings in files located in a settings folder
# within the module.
#
# Example:
#
# ├── mymodule.py
# ├── __init__.py
@Markbnj
Markbnj / ec2-discover-instances.py
Last active Aug 29, 2015
Discover ec2 instances by tag and tag value
View ec2-discover-instances.py
from boto import ec2
class Resource(object):
"""
Represents a single resource returned from the Discovery class's
get_resources method.
"""
def __init__(self, host_name, private_host_name, ip, private_ip, tags):
self.__host_name = host_name
self.__private_host_name = private_host_name
@Markbnj
Markbnj / debug-scrapyd-ipdb.py
Last active Aug 29, 2015
User ipdb to debug a running spider in scrapyd
View debug-scrapyd-ipdb.py
# The scrapyd daemon adds a lot of power and flexibility to scrapy, but it also adds a layer of indirection
# (another process) that can make debugging a running spider troublesome. Use this code to launch scrapyd
# in ipdb (or pdb). You'll be able to breakpoint and catch exceptions inside the spiders it launches.
import scrapyd.runner
import ipdb
ipdb.runcall(scrapyd.runner.main)
@Markbnj
Markbnj / simple-google-places-wrapper.py
Created Aug 11, 2015
A simple google places wrapper using requests
View simple-google-places-wrapper.py
import requests
import json
GOOGLE_API_KEY = "<A google API key>"
TEXT_SEARCH_TEMPLATE = "https://maps.googleapis.com/maps/api/place/textsearch/json?key={}&query={}"
NEARBY_SEARCH_TEMPLATE = "https://maps.googleapis.com/maps/api/place/nearbysearch/json?key={}&location={},{}&radius={}"
DETAIL_SEARCH_ID_TEMPLATE = "https://maps.googleapis.com/maps/api/place/details/json?key={}&placeid={}"
@Markbnj
Markbnj / geonames-org-wrapper.py
Created Aug 11, 2015
A simple geonames.org wrapper in python
View geonames-org-wrapper.py
import requests
import json
# For information on endpoints and arguments see the geonames
# API documentation at:
#
# http://www.geonames.org/export/web-services.html
class Geonames(object):
@Markbnj
Markbnj / set-proxy.sh
Created Aug 11, 2015
Set http/https proxy address on a console session
View set-proxy.sh
# sets the http/s proxy for the current shell to
# the passed argument, which should be in the form
# http://xxx.xxx.xxx.xxx:xxxx
export http_proxy=$1
export https_proxy=$http_proxy
export no_proxy="localhost,127.0.0.1"
@Markbnj
Markbnj / set-up-phantomjs.py
Created Aug 11, 2015
Set up phantomjs driver for selenium, example
View set-up-phantomjs.py
# An example showing how to set up phantomjs for use with Selenium
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
# These arguments will be passed on the phantomjs command line
# when webdriver launches it. See the reference page at
# http://phantomjs.org/api/command-line.html for more settings.
#
@Markbnj
Markbnj / docker-inject-ec2-host-ip.sh
Created Aug 11, 2015
Inject the ec2 instance private IP into a docker container at start
View docker-inject-ec2-host-ip.sh
# Sometimes you need to get the IP of the ec2 instance a container
# is running on. This script will get the IP from the local metadata
# service and inject it as an environment var.
localip=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
export HOST_IP=$localip
sudo docker run -i -t -e "HOST_IP=$localip" --name="test" -h="test" some-repo
@Markbnj
Markbnj / update-launch-es.sh
Created Aug 11, 2015
Update elasticsearch.yml and launch the service
View update-launch-es.sh
# Script to update elasticsearch.yml and launch the service. In this
# scenario the service is being launched inside a container on ec2
# but the technique is easily generalizable.
#
# Expects the following environment vars to be set inside the container
#
# HOST_IP - the IP address of the host running the container
# AWS_ACCESS_KEY - the AWS account access key
# AWS_SECRET_KEY - the AWS account secret key
# CLUSTER_NAME - the name of the ES cluster to join
@Markbnj
Markbnj / Dockerfile
Created Aug 11, 2015
Simple dockerfile for an Ubuntu-based base image for python services
View Dockerfile
# DockerFile for building a base image for python services
FROM ubuntu:15.04
MAINTAINER Someone <someone@some.where>
# common stuff
RUN echo "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d && \
mkdir /home/root && \
dpkg-reconfigure locales && \
locale-gen en_US en_US.UTF-8 && \
You can’t perform that action at this time.