Skip to content

Instantly share code, notes, and snippets.

Avatar

Chris Herwig hrwgc

  • Google
  • San Francisco
View GitHub Profile
@hrwgc
hrwgc / mercantile-tiles-from-geojson.sh
Last active Aug 29, 2015
mercantile find web mercator tiles for a given zoom level based on geojson feature collection
View mercantile-tiles-from-geojson.sh
#!/bin/bash
function to_tiles(){
geojson=$1;
zoom=$2;
jq -c .features[] < "$geojson" | mercantile tiles $zoom | sort | uniq
}
@hrwgc
hrwgc / aws-cli-s3cmd-du.sh
Last active Apr 22, 2020
aws-cli get total size of all objects within s3 prefix. (mimic behavior of `s3cmd du` with aws-cli)
View aws-cli-s3cmd-du.sh
#!/bin/bash
function s3du(){
bucket=`cut -d/ -f3 <<< $1`
prefix=`awk -F/ '{for (i=4; i<NF; i++) printf $i"/"; print $NF}' <<< $1`
aws s3api list-objects --bucket $bucket --prefix=$prefix --output json --query '[sum(Contents[].Size), length(Contents[])]' | jq '. |{ size:.[0],num_objects: .[1]}'
}
s3du $1;
@hrwgc
hrwgc / xml2json.js
Created Sep 17, 2014
convert xml to json nodejs command line utility
View xml2json.js
#!/usr/bin/env node
var parser = require('xml2json');
var fs = require('fs');
function onFile(err, data) {
if (err) {
console.log('Uh oh: ' + err);
return;
}
var json = parser.toJson(data);
View perf-guide.md

perf guide

Profiling tools are critical for trying to understand performance bottlenecks in code.

Without real data about what a program is doing while running, detecting bottlenecks is at best a process of trial and error for both users and developers. While thoughtful testing of various program configurations along with measuring time elapsed for decrete runs can often be enough - why not learn faster and funner ways to rapidly collect real data about what a program is doing?

Actual data about program execution like which functions are being called while a program is active helps point to hot parts of the code where most time may be being spent. While users of applications may not easily understand the output of profiling tools, being equipped to generate profiling output can be extremely useful for sharing with developers, since the time to set up robust test cases for developers is can be greater than the time it takes to understand and optimize slow code paths. Therefore it can be invaluable to get

@hrwgc
hrwgc / 1_how_to_fix.md
Last active Dec 29, 2015
postgres / postgis randomly stops working on osx mavericks - fix
View 1_how_to_fix.md

Following advice of Psql: could not connect...:

Running the following command resets PostgreSQL's transaction log, which resolves the problem.

 pg_resetxlog -f /usr/local/Cellar/postgresql/9.3.1/data

you should replace /usr/local/Cellar/postgresql/9.3.1/data with your own pg_datadir (eg., pg_ctl -D $DATADIR start)

@hrwgc
hrwgc / validate.sh
Created Nov 13, 2013
bash wget - check if file exists at url before downloading
View validate.sh
#!/bin/bash
# simple function to check http response code before downloading a remote file
# example usage:
# if `validate_url $url >/dev/null`; then dosomething; else echo "does not exist"; fi
function validate_url(){
if [[ `wget -S --spider $1 2>&1 | grep 'HTTP/1.1 200 OK'` ]]; then echo "true"; fi
}
View landsat-band-mappings.md
Landsat 4-5 Wavelength Landsat 7 Wavelength Landsat 8 Wavelength (micrometers)
Band 1 - Coastal aerosol 0.43 - 0.45
Band 1 0.45-0.52 Band 1 0.45-0.52 Band 2 - Blue 0.45 - 0.51
Band 2 0.52-0.60 Band 2 0.52-0.60 Band 3 - Green 0.53 - 0.59
Band 3 0.63-0.69 Band 3 0.63-0.69 Band 4 - Red 0.64 - 0.67
Band 4 0.76-0.90 Band 4 0.77-0.90 Band 5 - Near Infrared (NIR) 0.85 - 0.88
Band 9 - Cirrus 1.36 - 1.38
Band 5 1.55-1.75 Band 5 1.55-1.75 Band 6 - SWIR 1 1.57 - 1.65
Band 7 2.08-2.35 Band 7 2.09-2.35 Band 7 - SWIR 2 2.11 - 2.29
@hrwgc
hrwgc / clean_git.sh
Created Oct 22, 2013
remove all non master branches from git repo
View clean_git.sh
function clean_git_branches(){
git checkout master;
BRANCHES=`git branch | grep -Ev '^\s{0,}\*{0,}\s{0,}master$'`
git branch -D ${BRANCHES[@]}
}
View clipboard.py
#!/usr/bin/env python
import os
def CopyToClipboard(text):
cmd = "echo '%s' | tr -d \"\\\n\" | pbcopy" % text
os.system(cmd)
@hrwgc
hrwgc / nearest_neighbor.sql
Last active Dec 19, 2015
postgis select across polygon layers based on nearest adjacent polygon from other layer.
View nearest_neighbor.sql
create table
closest_point as
select distinct on (a.featureid) a.featureid as image_id,
(select b.eventid from fl_events as b order by st_distance(a.wkb_geometry,b.wkb_geometry) limit 1) as event_id,
a.wkb_geometry from images as a,
fl_events as b;
update images set eventid = closest_point.event_id from closest_point where images.featureid = closest_point.image_id;
You can’t perform that action at this time.