Skip to content

Instantly share code, notes, and snippets.

Jesse Crocker JesseCrocker

Block or report user

Report or block JesseCrocker

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@JesseCrocker
JesseCrocker / cleanup.js
Created May 3, 2018
Aws lambda function that allows a user to hit a url and get added to an aws security group, and another function to cleanup old entries in the group
View cleanup.js
"use strict";
var AWS = require("aws-sdk");
var ec2 = new AWS.EC2();
var securityGroup = "sg-XXX";
exports.handler = (event, context, callback) => {
ec2.describeSecurityGroups({ GroupIds: [securityGroup] }, function(
err,
data
@JesseCrocker
JesseCrocker / worldclim_to_cog.md
Created Mar 23, 2018
Convert worldclim climate data to cloud optimized geotif
View worldclim_to_cog.md

http://www.worldclim.org/current

wget http://biogeo.ucdavis.edu/data/climate/worldclim/1_4/grid/cur/tmin_30s_bil.zip
wget http://biogeo.ucdavis.edu/data/climate/worldclim/1_4/grid/cur/tmax_30s_bil.zip
wget http://biogeo.ucdavis.edu/data/climate/worldclim/1_4/grid/cur/tmean_30s_bil.zip
wget http://biogeo.ucdavis.edu/data/climate/worldclim/1_4/grid/cur/prec_30s_bil.zip

unzip tmin_30s_bil.zip
@JesseCrocker
JesseCrocker / nlcd_to_cog.md
Created Mar 21, 2018
Convert NLCD to Cloud Optimized GeoTiff
View nlcd_to_cog.md

NLCD data is published in 3x3 degree chunks, as tiff files inside a zip file. For processing this data it's far more convienent if the files are available directly as tiff files without having to unzip.

s3cmd get --skip-existing -r s3://prd-tnm/StagedProducts/NLCD/data/2011/landcover/3x3/
for i in *.zip; do unzip $i '*.tif'; done

mkdir tmp
mkdir clouded

for i in *.tif; do
@JesseCrocker
JesseCrocker / download_tiles.py
Created Feb 26, 2018
Download TMS tiles for a bbox, from http or s3
View download_tiles.py
#!/usr/bin/env python3
import logging
from optparse import OptionParser
import os
import mercantile
from urllib.parse import urlparse
import requests
from boto.s3.connection import S3Connection
View Trailhead research.txt
Executive summary:
There are 1261 points with a name ending in Trailhead. A decent % of them have no tags other than name.
There are 544 points tagged highway = trailhead. A bunch of them are in the Phillipenes.
There are 821 polygons with a name ending in Trailhead. There are almost all tagged as parking lots.
There are 64 polygons tagged highway=Trailhead. They are all tagged amenity=parking.
osm_rendering=> select name, osm_id, "natural", "amenity", "highway", "tourism", leisure, tags from planet_osm_point where name ilike '% Trailhead';
name | osm_id | natural | amenity | highway | tourism | leisure | tags
----------------------------------------------------------------------+------------+---------+-----------------+-------------------+-------------+----------------+----------------
@JesseCrocker
JesseCrocker / test-mapnik-sql.py
Created Jan 23, 2018
Replace tokens in a mapnik sql query with values for a specified tile
View test-mapnik-sql.py
#!/usr/bin/env python3
import logging
from optparse import OptionParser
import os
import sys
import mercantile
def _main():
usage = "usage: %prog"
@JesseCrocker
JesseCrocker / README.md
Last active May 21, 2018
Simple ECS service deploy script
View README.md

Deploy updates to aws ecs services based on github commits. Posts deploy notifications to slack.

Assumptions:

  • Task definitions map 1:1 with services, and they have the same name.
  • Code is stored on github
  • You want to deploy the latest commit in a branch
  • Docker images are tagged with commit SHA
  • Docker images are stored on AWS ECR
@JesseCrocker
JesseCrocker / upload_tiles.py
Created Jan 9, 2018
Upload valhalla routing tiles to s3
View upload_tiles.py
#!/usr/bin/env python3
import os
from urllib.parse import urlparse
from multiprocessing.pool import ThreadPool
from functools import partial
import gzip
from boto.s3.connection import S3Connection
from boto.s3.key import Key
View valhalla diary.md

Ultimate Goal: valhalla running on AWS ECS, behind a load balancer, in multiple regions, with auto scaling, with data updating once a week.

Diary

First goal: Build with docker, then cut tiles from a small extract and get server to run locally.

jesse:projects/ $ mkdir ~/valhalla-data
jesse:projects/ $ cd ~/valhalla-data
jesse:projects/ $ wget http://download.geofabrik.de/north-america/us/california-latest.osm.pbf
[output abbreviated]
jesse:projects/ $ cd ~/projects
@JesseCrocker
JesseCrocker / split.py
Created Dec 14, 2017
Split a gpx file into multiple files
View split.py
#!/usr/bin/env python
import logging
from optparse import OptionParser
import os
import gpxpy
import gpxpy.gpx
def split_gpx(source, dest_dir, max_segment_points=500):
You can’t perform that action at this time.