Skip to content

Instantly share code, notes, and snippets.

@chriswhong
chriswhong / gist:762ceac7fb8a1420e7e7adceb770b707
Last active March 3, 2022 09:01
Using ST_AsMVT() and ST_AsMVTGeom() in express to build a vector tile endpoint
/* GET /tiles/:z/:x/:y.mvt */
/* Retreive a vector tile by tileid */
router.get('/tiles/:z/:x/:y.mvt', async (req, res) => {
const { z, x, y } = req.params;
// calculate the bounding polygon for this tile
const bbox = mercator.bbox(x, y, z, false);
// Query the database, using ST_AsMVTGeom() to clip the geometries
// Wrap the whole query with ST_AsMVT(), which will create a protocol buffer
@chriswhong
chriswhong / transform.star
Created July 7, 2021 20:57
qri transform for storing mta elevator statuses
load("encoding/csv.star", "csv")
load("encoding/json.star", "json")
load("http.star", "http")
def download(ctx):
# send an http get request to the elevator outages page
raw = http.get('https://traintime.lirr.org/api/ElevEsc?api_key=dwrxfpcsp42pcwxfbpd90zde5fjeb05pcdaqjw3j').body()
data = json.loads(raw)
stations = data['Stations']
Sub Auto_Open()
Dim oToolbar As CommandBar
Dim oButton As CommandBarButton
Dim MyToolbar As String
' Give the toolbar a name
MyToolbar = "Kewl Tools"
On Error Resume Next
' so that it doesn't stop on the next line if the toolbar's already there
@chriswhong
chriswhong / gist:ee6a44f8c0b7c3706d4161a408c8ac4f
Last active March 9, 2021 19:46
Using mapshaper.js to convert geojson in memory into shapefile download

include zip.js and mapshaper.js

<script src="js/zip.js"></script>
<script src="js/mapshaper.js"></script>

serve deflate.js, inflate.js, and z-worker.js somewhere, and reference them in your code with zip.workerScriptsPath = '{path}', make sure the path ends with a slash!


  //create a mapshaper dataset from geojson FeatureCollection
@chriswhong
chriswhong / gist:ff86b482273ce20cc8dc2391a6b30c4c
Created May 11, 2020 16:47
Pulling a Qri versioned dataset from the distributed web into a pandas dataframe (python)
# an example of pulling in a CSV string from a local qri repo,
# and parsing it into a dataframe with pandas.read_csv()
import pandas as pd
import subprocess
from io import StringIO
# this Qri class should eventually have all the methods we would need for reading,
# writing, pushing, pulling, committing, etc.
class Qri:
@chriswhong
chriswhong / gist:93ed4660ff23fdf483b41654290f0b18
Created April 7, 2020 17:57
Socrata Download URL generates different files when accessed multiple times
## Problem
When troubleshooting another bug at qri.io, we noticed that hitting the same download URL from the NYC Open Data portal (powered by Socrata) generated csv files with different md5 hashes. This was causing our version control software to consider the csv "changed". On further investigation, the actual data was the same, but the ordering of the rows was different. This is likely due to the fact that the files are being generated on the fly from a database query (PostgreSQL in the case of Socrata open data portals) returning unpredictable row orders when no `ORDER BY` is specified in the query.
To reproduce:
- curl the same download URL several times and save to csv, such as this one for popular baby names:
```
$curl https://data.cityofnewyork.us/resource/25th-nujf.csv > 0.csv
$curl https://data.cityofnewyork.us/resource/25th-nujf.csv > 1.csv
@chriswhong
chriswhong / convert csv with WKT to GeoJson
Last active March 5, 2020 13:19
How to parse a CSV with a well-known text geometry column and convert to geojson FeatureCollection with the 'wellknown' package
// Use papa parse to download and parse a CSV
Papa.parse('data/nyc_zoning_lots.csv', {
download: true,
header: true,
complete: ({ data }) => {
// transform array of objects to geojson FeatureCollection
const FC = {
type: 'FeatureCollection',
features: data.map((row) => { // map csv rows to geojson features
const geometry = wellknown.parse(row.geom) // WKT to geojson geometry
@chriswhong
chriswhong / Qmf6Vfs78CeKHsWus2gXjBkvha93acBXN3FB7DtEFha4oj.csv
Created January 23, 2020 14:57
NYC Plaza Program Sites from Qri Dataset chriswhong/nyc_plaza_program_sites@/ipfs/Qmf6Vfs78CeKHsWus2gXjBkvha93acBXN3FB7DtEFha4oj.csv
We can make this file beautiful and searchable if this error is corrected: It looks like row 5 should actually have 13 columns, instead of 9. in line 4.
plaza,boro_community_district_1,boro_community_district_2,level,area_sf,on_street,cross_street_1,cross_street_2,partner,partner_info_url,longitude,latitude,cecm_map
Albee Square Plaza,BK02,,C,25200,Dekalb Ave,Bond Street,Fulton Mall,Fulton Mall Improvement Association,http://downtownbrooklyn.com/about/fulton-mall-improvement-association,-73.983171,40.689973,https://www1.nyc.gov/assets/cecm/downloads/plaza-maps/bk-albee-square.pdf
Ave C Plaza,BK12,,D,5600,McDonald Ave,Avenue C,Church Ave,Kensington Stewards,https://bklyner.com/want-a-new-pedestrian-plaza-in-kensington-attend-this-public-workshop-on-wednesday-april-29-kensington/,-73.979094,40.640571,https://www1.nyc.gov/assets/cecm/downloads/plaza-maps/bk-avenue-c-plaza.pdf
Brooklyn Plaza,BK02,,B,50000,Jay Street,High Street,Manhattan Bridge,DOT,https://www1.nyc.gov/html/dot/html/home/home.shtml,-73.986569,40.699203,https://www1.nyc.gov/assets/cecm/downloads/plaza-maps/bk-brooklyn-plaza.pdf
Cadman Plaza East,BK02,,,,Cadman Plaza East,Tillary St,Johnson St,Down
@chriswhong
chriswhong / index.html
Last active September 9, 2019 15:57
Qri blob loading spinner prototype
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/ >
<title>Qri Loading Spinner Prototype</title>
<script src="https://d3js.org/d3.v3.min.js"></script>
<!-- jQuery -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<!-- Bootstrap 3 -->
@chriswhong
chriswhong / readme.md
Last active September 8, 2019 12:03
Run open trip planner docker container for NYC

This set of commands is for setting up an open trip planner instance for New York City. OTP requires GTFS data and OSM streets data to build a graph, which it uses for trip planning.

Lucky for us, someone here dockerhub has left a nice CLI command to build the graph and run the container, but we need to get the data first.

The data are downloaded on the host machine. For me, this is a digitalocean droplet running ubuntu 14.

First, get GTFS for the New York City Subway from the MTA's downloads page: wget http://web.mta.info/developers/data/nyct/subway/google_transit.zip Next, get OSM city extract for NYC. Thanks Mapzen! https://s3.amazonaws.com/metro-extracts.mapzen.com/new-york_new-york.osm.pbf

Finally, run the following docker command: docker run -it -v $(pwd):/var/otp/graphs opentripplanner/opentripplanner --build /var/otp/graphs --analyst