Skip to content

Instantly share code, notes, and snippets.

View danielrmeyer's full-sized avatar

Daniel Meyer danielrmeyer

View GitHub Profile
@danielrmeyer
danielrmeyer / supervisor
Created June 27, 2016 17:03
supervisor config to keep a python script running in home directory.
[program:test]
command=/usr/bin/python /home/ubuntu/test.py
directory=/home/ubuntu
autostart=true
autorestart=true
startretries=3
stderr_logfile=/home/ubuntu/test.err.log
stdout_logfile=/home/ubuntu/test.out.log
user=ubuntu
@danielrmeyer
danielrmeyer / spark_cassandra.scala
Created June 13, 2016 19:00
Connect to a cassandra cluster using spark hosted on Amazon EMR
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark.connector._
val conf = new SparkConf(true)
.set("spark.cassandra.connection.host", "<cassandra_host")
.set("spark.cassandra.auth.username", "cassandra")
.set("spark.cassandra.auth.password", "########")
@danielrmeyer
danielrmeyer / .emacs
Last active February 23, 2017 04:18
emacs stuff
(setq-default indent-tabs-mode nil)
(setq scroll-step 1)
(custom-set-variables
;; custom-set-variables was added by Custom.
;; If you edit it by hand, you could mess it up, so be careful.
;; Your init file should contain only one such instance.
;; If there is more than one, they won't work right.
'(custom-safe-themes
@danielrmeyer
danielrmeyer / Vagrantfile
Created May 10, 2016 20:32
Get docker running on ubuntu trusty64
# -*- mode: ruby -*-
# vi: set ft=ruby :
$setup = <<SCRIPT
sudo apt-get update
sudo apt-get install -y emacs24
sudo apt-get install apt-transport-https ca-certificates
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo "deb https://apt.dockerproject.org/repo ubuntu-trusty main" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
@danielrmeyer
danielrmeyer / tcpwatch2json.py
Created April 27, 2016 17:34
convert a recorded http interaction from tcpwatch to a json object.
# -*- coding: utf-8 -*-
import os
import glob
import codecs
import re
import json
path2recording = os.path.join(os.getcwd(), 'recording')
requests = glob.glob(os.path.join(path2recording, "*.request"))
######
## Kong configuration file. All commented values are default values.
## Uncomment and update a value to configure Kong to your needs.
##
## Lines starting with `##` are comments.
## Lines starting with `#` are properties that can be updated.
## Beware of YAML formatting for nested properties.
######
## Additional plugins that this node needs to load.
init_config:
instances:
- proc_name: gunicorn
@danielrmeyer
danielrmeyer / run_indexing_test.sh
Created June 27, 2015 00:47
Start monitor_indexing.py on a remote client inside a screen session
TEST_DIR=/Users/dmeyer/Sources/automaton_solr_multimech/resources/tests/loadtests/multimech_solr
function poll_client_for_screen_session {
result=$(ctool --provider=existing run solr_perf_client 0 "screen -list" | grep 'Detached')
while [ "$result" != "" ]
do
sleep 10
result=$(ctool --provider=existing run solr_perf_client 0 "screen -list" | grep 'Detached')
done
@danielrmeyer
danielrmeyer / monitor_indexing.py
Created June 26, 2015 21:10
Monitor indexing status and log elapsed time and doc count
import argparse
import time
import requests
import re
parser = argparse.ArgumentParser(description="Monitor and time indexing.")
parser.add_argument('-l', '--hosts', type=str, help='comma seperated list of hosts to connect to.', default='127.0.0.1')
arguments = parser.parse_args()
indexing_status_patt = re.compile(r'<bool name="indexing">([a-z]+)</bool>')
num_docs_patt = re.compile(r'<int name="numDocs">([0-9]*)</int>')
from cassandra.cluster import Cluster
from cassandra.policies import RetryPolicy
import sys
import argparse
import datetime
from itertools import cycle, product
import csv
import time
class AlwaysRetryPolicy(RetryPolicy):