Skip to content

Instantly share code, notes, and snippets.

@jehiah
jehiah / gist:b747bf4ac069fccc3b89aec5cc8a1307
Created May 25, 2023 19:11
NYPD Parking Facilities geojson
cat City\ Owned\ and\ Leased\ Property\ \(COLP\)\ \ \ Shapefile.geojson | jq -c '{type:"FeatureCollection", features:[(.features[] | select(.properties.agency == "NYPD") | select(.properties.usecode | in({"0100": true, "0130":true, "0510":true, "0110":true, "0120":true, "0690":true}) )) ]}' > NYPD_parking.geojson
@jehiah
jehiah / minimal_cidr_list.py
Created July 6, 2021 18:58
minify a list of CIDrs to the largest CID listed
#!/usr/bin/env python
import sys
from ipaddress import IPv4Network
from collections import defaultdict
if __name__ == "__main__":
inputs = defaultdict(list)
for line in sys.stdin:
cidr = line.strip().decode('utf-8')
#!/bin/bash
# run https://github.com/client9/misspell linter
SCRIPT=$(readlink -f "$0")
SCRIPTPATH=`dirname "$SCRIPT"`
# get to repo root
cd $SCRIPTPATH/../../..
WRITE=""
while [ "$1" != "" ]; do
PARAM=${1%%=*}
@jehiah
jehiah / svg_abs_to_rel.py
Created February 9, 2021 23:00
Resize SVG images
from decimal import Decimal
from xml.dom import minidom
import sys
import os
import re
import simplejson as json
import string
import tornado.options
import logging
@jehiah
jehiah / README.md
Created June 4, 2020 15:23
BigTable ApplyReadModifyWrite load script

Performance metrics for cloud.google.com/go/bigtable v1.1.0 vs v1.3.0 which show a stall at 2k ops/sec. tested from n1-highcpu-8 against a 6 node bigtable cluster.

$ for VERSION in v1.1.0 v1.3.0; do git reset --hard; go get cloud.google.com/go/bigtable@${VERSION}; go get cloud.google.com/go@v0.57.0; go build; go list -m all | egrep 'cloud.google.com/go|google.golang.org'; for C in 10 25 50 75 100 125; do ./run_bt_load.sh --concurrency=$C --duration=30s 2>&1 | grep -- '- finished'; done; done
HEAD is now at 4299de614e bigtable v1.1.0 - reproduce breaking performance
go: downloading google.golang.org/genproto v0.0.0-20200603110839-e855014d5736
go: downloading github.com/golang/protobuf v1.4.1
go: downloading google.golang.org/api v0.26.0
go: downloading google.golang.org/protobuf v1.24.0
go: downloading google.golang.org/appengine v1.6.6
@jehiah
jehiah / tcp_throttle.sh
Created April 23, 2020 20:31
tcp_throttle uses packet shaping to add delay to specific packets
#!/bin/bash
# this uses packet shaping to add delay to specific packets.
# for more informatino about "netem" the network emulation module
# see http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
# or http://lartc.org/howto/index.html section 9 and 12
# please add more options to this script as needed.
function usage() {
echo "USAGE: $(basename $0) --interface=<ethX> (--dport=<PORT> | --host=<IP>) --delay=<DELAY_SPEC> --start"
@jehiah
jehiah / corona_virus_daily_change.sh
Created April 4, 2020 02:40
Generate CSV of daily deltas from nychealth/coronavirus-data
#!/bin/zsh
#####
# This script provides a delta history of all updates to case-hosp-death.csv
# based on the AS_OF from summary.csv
# https://github.com/nychealth/coronavirus-data/blob/master/case-hosp-death.csv
# https://github.com/nychealth/coronavirus-data/blob/master/summary.csv
#
# By: Jehiah Czebotar
@jehiah
jehiah / FOIL-2018-858-00481.csv
Created January 2, 2020 18:48
NYC 311 Service Request Satisfaction Survey Data
We can't make this file beautiful and searchable because it's too large.
Date,Complaint Type,Descriptor 1,Agency,"Overall, I am satisfied with the way my Service Request was handled.",Why were you dissatisfied with how your Service Request was handled?
2018-04-06,Street Light Condition,Street Light Out,DOT,0 - Strongly Disagree,The Agency did not correct the issue.
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-04-04,DOF Property - Reduction Issue,Personal STAR Exemption,DOF,0 - Strongly Disagree,"Status updates were unhelpful, inaccurate, incomplete, and/or confusing."
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-31,CFC Recovery,17 CFC Appliance,DSNY,75 - Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,0 - Strongly Disagree,The Agency did not correct the issue.
We can't make this file beautiful and searchable because it's too large.
Date,Complaint Type,Descriptor 1,Agency,"Overall, I am satisfied with the way my Service Request was handled.",Why were you dissatisfied with how your Service Request was handled?
2018-04-06,Street Light Condition,Street Light Out,DOT,0 - Strongly Disagree,The Agency did not correct the issue.
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-04-04,DOF Property - Reduction Issue,Personal STAR Exemption,DOF,0 - Strongly Disagree,"Status updates were unhelpful, inaccurate, incomplete, and/or confusing."
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,100 - Strongly Agree,
2018-03-31,CFC Recovery,17 CFC Appliance,DSNY,75 - Agree,
2018-03-30,CFC Recovery,17 CFC Appliance,DSNY,0 - Strongly Disagree,The Agency did not correct the issue.
@jehiah
jehiah / over.py
Created May 9, 2018 00:54
Script to filter output from `uniq -c` to only items that appear more than `--frequency=n` times.
#!/bitly/local/bin/python
import tornado.options
import logging
import sys
if __name__ == "__main__":
tornado.options.define("frequency", type=int)
tornado.options.parse_command_line()
for line in sys.stdin: