Skip to content

Instantly share code, notes, and snippets.

Avatar

Jehiah Czebotar jehiah

View GitHub Profile
@jehiah
jehiah / README.md
Created Jun 4, 2020
BigTable ApplyReadModifyWrite load script
View README.md

Performance metrics for cloud.google.com/go/bigtable v1.1.0 vs v1.3.0 which show a stall at 2k ops/sec. tested from n1-highcpu-8 against a 6 node bigtable cluster.

$ for VERSION in v1.1.0 v1.3.0; do git reset --hard; go get cloud.google.com/go/bigtable@${VERSION}; go get cloud.google.com/go@v0.57.0; go build; go list -m all | egrep 'cloud.google.com/go|google.golang.org'; for C in 10 25 50 75 100 125; do ./run_bt_load.sh --concurrency=$C --duration=30s 2>&1 | grep -- '- finished'; done; done
HEAD is now at 4299de614e bigtable v1.1.0 - reproduce breaking performance
go: downloading google.golang.org/genproto v0.0.0-20200603110839-e855014d5736
go: downloading github.com/golang/protobuf v1.4.1
go: downloading google.golang.org/api v0.26.0
go: downloading google.golang.org/protobuf v1.24.0
go: downloading google.golang.org/appengine v1.6.6
@jehiah
jehiah / tcp_throttle.sh
Created Apr 23, 2020
tcp_throttle uses packet shaping to add delay to specific packets
View tcp_throttle.sh
#!/bin/bash
# this uses packet shaping to add delay to specific packets.
# for more informatino about "netem" the network emulation module
# see http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
# or http://lartc.org/howto/index.html section 9 and 12
# please add more options to this script as needed.
function usage() {
echo "USAGE: $(basename $0) --interface=<ethX> (--dport=<PORT> | --host=<IP>) --delay=<DELAY_SPEC> --start"
@jehiah
jehiah / corona_virus_daily_change.sh
Created Apr 4, 2020
Generate CSV of daily deltas from nychealth/coronavirus-data
View corona_virus_daily_change.sh
#!/bin/zsh
#####
# This script provides a delta history of all updates to case-hosp-death.csv
# based on the AS_OF from summary.csv
# https://github.com/nychealth/coronavirus-data/blob/master/case-hosp-death.csv
# https://github.com/nychealth/coronavirus-data/blob/master/summary.csv
#
# By: Jehiah Czebotar
@jehiah
jehiah / FOIL-2018-858-00481.csv
Created Jan 2, 2020
NYC 311 Service Request Satisfaction Survey Data
View FOIL-2018-858-00481.csv
We can't make this file beautiful and searchable because it's too large.
Date,Complaint Type,Descriptor 1,Agency,"Overall, I am satisfied with the way my Service Request was handled.",Why were you dissatisfied with how your Service Request was handled?
2018-04-06,Street Light Condition,Street Light Out,DOT,0 - Strongly Disagree,The Agency did not correct the issue.
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-04-04,DOF Property - Reduction Issue,Personal STAR Exemption,DOF,0 - Strongly Disagree,"Status updates were unhelpful, inaccurate, incomplete, and/or confusing."
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-31,CFC Recovery,17 CFC Appliance,DSNY,75 - Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,0 - Strongly Disagree,The Agency did not correct the issue.
View FOIL-2018-858-00481.csv
We can't make this file beautiful and searchable because it's too large.
Date,Complaint Type,Descriptor 1,Agency,"Overall, I am satisfied with the way my Service Request was handled.",Why were you dissatisfied with how your Service Request was handled?
2018-04-06,Street Light Condition,Street Light Out,DOT,0 - Strongly Disagree,The Agency did not correct the issue.
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-04-04,DOF Property - Reduction Issue,Personal STAR Exemption,DOF,0 - Strongly Disagree,"Status updates were unhelpful, inaccurate, incomplete, and/or confusing."
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-31,CFC Recovery,17 CFC Appliance,DSNY,75 - Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,0 - Strongly Disagree,The Agency did not correct the issue.
@jehiah
jehiah / over.py
Created May 9, 2018
Script to filter output from `uniq -c` to only items that appear more than `--frequency=n` times.
View over.py
#!/bitly/local/bin/python
import tornado.options
import logging
import sys
if __name__ == "__main__":
tornado.options.define("frequency", type=int)
tornado.options.parse_command_line()
for line in sys.stdin:
@jehiah
jehiah / xls_to_csv.py
Last active Nov 15, 2019
Convert xls to CSV
View xls_to_csv.py
#!/usr/local/bin/python
import openpyxl
from openpyxl.utils.exceptions import InvalidFileException
import tornado.options
import os.path
import csv
import sys
import xlrd
import datetime
@jehiah
jehiah / pizza.py
Last active Jan 13, 2019
Calculate the ratio of ingredients needed for the perfect quantity of pizza dough
View pizza.py
#!/usr/bin/env python
# Calculate the ratio of ingredients needed for the perfect quantity of pizza dough
#
# $ pizza.py --target-oz=49
# target: 49.00oz 1389gram
# flour: 800 grams (400 x 2)
# water: 520 grams
# salt: 16 grams
# sugar: 12 grams
@jehiah
jehiah / semver.py
Created Apr 10, 2017
Semver 2.0.0 compatible version comparison w/o regexes
View semver.py
def compare(a, b):
"""
Compare two version strings.
:return -1, 0, 1
:rtype: int
"""
a = map(_cast_int, a.split('+', 2)[0].split('.'))
b = map(_cast_int, b.split('+', 2)[0].split('.'))
@jehiah
jehiah / bitly_datastream.py
Created Feb 3, 2017
pynsq example for connecting to Bitly Datastream
View bitly_datastream.py
import json
import logging
import nsq
import sys
import tornado
import tornado.options
def handle_message(message):
print json.loads(message.body)['u']
message.finish()
You can’t perform that action at this time.