Skip to content

Instantly share code, notes, and snippets.

@jehiah
jehiah / simulate_memcached.py
Created August 20, 2011 01:43
test script to simulate a partially reachable memcached instance
#!/bin/env python
"""
This is a test script to simulate a memcached instance on a server
that has gone south and is accepting connections, but generally not
responding.
The goal of this script is to help test/develop correct client
side settings for timeouts/failure scenarios
@jehiah
jehiah / nsq_testing.go
Last active June 30, 2020 08:13
NSQ Producer testing abstraction
package nsqutils
import (
"sync"
"time"
nsq "github.com/nsqio/go-nsq"
)
// Producer is an interface that nsq.Producer fulfills
@jehiah
jehiah / README.md
Created June 4, 2020 15:23
BigTable ApplyReadModifyWrite load script

Performance metrics for cloud.google.com/go/bigtable v1.1.0 vs v1.3.0 which show a stall at 2k ops/sec. tested from n1-highcpu-8 against a 6 node bigtable cluster.

$ for VERSION in v1.1.0 v1.3.0; do git reset --hard; go get cloud.google.com/go/bigtable@${VERSION}; go get cloud.google.com/go@v0.57.0; go build; go list -m all | egrep 'cloud.google.com/go|google.golang.org'; for C in 10 25 50 75 100 125; do ./run_bt_load.sh --concurrency=$C --duration=30s 2>&1 | grep -- '- finished'; done; done
HEAD is now at 4299de614e bigtable v1.1.0 - reproduce breaking performance
go: downloading google.golang.org/genproto v0.0.0-20200603110839-e855014d5736
go: downloading github.com/golang/protobuf v1.4.1
go: downloading google.golang.org/api v0.26.0
go: downloading google.golang.org/protobuf v1.24.0
go: downloading google.golang.org/appengine v1.6.6
@jehiah
jehiah / tcp_throttle.sh
Created April 23, 2020 20:31
tcp_throttle uses packet shaping to add delay to specific packets
#!/bin/bash
# this uses packet shaping to add delay to specific packets.
# for more informatino about "netem" the network emulation module
# see http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
# or http://lartc.org/howto/index.html section 9 and 12
# please add more options to this script as needed.
function usage() {
echo "USAGE: $(basename $0) --interface=<ethX> (--dport=<PORT> | --host=<IP>) --delay=<DELAY_SPEC> --start"
@jehiah
jehiah / corona_virus_daily_change.sh
Created April 4, 2020 02:40
Generate CSV of daily deltas from nychealth/coronavirus-data
#!/bin/zsh
#####
# This script provides a delta history of all updates to case-hosp-death.csv
# based on the AS_OF from summary.csv
# https://github.com/nychealth/coronavirus-data/blob/master/case-hosp-death.csv
# https://github.com/nychealth/coronavirus-data/blob/master/summary.csv
#
# By: Jehiah Czebotar
@jehiah
jehiah / FOIL-2018-858-00481.csv
Created January 2, 2020 18:48
NYC 311 Service Request Satisfaction Survey Data
We can't make this file beautiful and searchable because it's too large.
Date,Complaint Type,Descriptor 1,Agency,"Overall, I am satisfied with the way my Service Request was handled.",Why were you dissatisfied with how your Service Request was handled?
2018-04-06,Street Light Condition,Street Light Out,DOT,0 - Strongly Disagree,The Agency did not correct the issue.
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-04-04,DOF Property - Reduction Issue,Personal STAR Exemption,DOF,0 - Strongly Disagree,"Status updates were unhelpful, inaccurate, incomplete, and/or confusing."
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-31,CFC Recovery,17 CFC Appliance,DSNY,75 - Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,0 - Strongly Disagree,The Agency did not correct the issue.
2018-04-03,DOF Property - Reduction Issue,Personal SCHE Exemption,DOF,0 - Strongly Disagree,Other (please specify)
@jehiah
jehiah / xls_to_csv.py
Last active November 15, 2019 14:54
Convert xls to CSV
#!/usr/local/bin/python
import openpyxl
from openpyxl.utils.exceptions import InvalidFileException
import tornado.options
import os.path
import csv
import sys
import xlrd
import datetime
@jehiah
jehiah / BufferedMessageHandler.py
Created July 15, 2013 02:05
Buffered NSQ reader (python)
import tornado.ioloop
import logging
import time
class BufferedMessageHandler(object):
"""
Usage:
buffered_handler = BufferedMessageHandler(my_handler)
r = Reader(message_handler=buffered_handler)
We can't make this file beautiful and searchable because it's too large.
Date,Complaint Type,Descriptor 1,Agency,"Overall, I am satisfied with the way my Service Request was handled.",Why were you dissatisfied with how your Service Request was handled?
2018-04-06,Street Light Condition,Street Light Out,DOT,0 - Strongly Disagree,The Agency did not correct the issue.
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-04-04,DOF Property - Reduction Issue,Personal STAR Exemption,DOF,0 - Strongly Disagree,"Status updates were unhelpful, inaccurate, incomplete, and/or confusing."
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-31,CFC Recovery,17 CFC Appliance,DSNY,75 - Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,0 - Strongly Disagree,The Agency did not correct the issue.
2018-04-03,DOF Property - Reduction Issue,Personal SCHE Exemption,DOF,0 - Strongly Disagree,Other (please specify)
@jehiah
jehiah / twitter_archiver.py
Created December 17, 2010 05:00
python script to archive your tweets
#!/usr/bin/env python
"""
twitter_archiver.py written by Jehiah Czebotar 2010 <jehiah@gmail.com> http://jehiah.cz/
this uses the great 'python twitter tools' library by Mike Verdone
http://mike.verdone.ca/twitter/
usage:
$ pip install twitter