Skip to content

Instantly share code, notes, and snippets.

View chriswhong's full-sized avatar

Chris Whong chriswhong

687 followers · 47 following
View GitHub Profile
@chriswhong
chriswhong / choropleth.js
Created March 23, 2017 11:53
Choropleth for mapboxGL
// choropleth.js
// given polygon geojson data and options, returns a GL fill style
// adaptation of Tim Wisniewskis' Leaflet Choropleth
import chroma from 'chroma-js';
const Choropleth = (geojson, opts) => {
opts = opts || {};
@chriswhong
chriswhong / README.md
Last active June 21, 2017 12:06
Logging Maryland MTA Real-time bus data

Logging Maryland MTA Real-time bus data

Technical considerations for logging real-time bus data

Overview

The Maryland MTA has made available real-time bus location data in GTFS-RT format, available at http://mta.maryland.gov/content/developer-resources. Various stakeholders wish to use historical real-time data to assess the timeliness of the bus system. Successful analysis would require a consistent dataset of archived real time locations.

Database vs Static Files

A major consideration is whether to use a database to store the data, or to store it in a well-organized hierarchical tree of static files.

Storing the data in database means a slightly more complex hosting environment. New data are committed to the database, but the data would not be easily accessible without more work to expose certain queries as bulk downloads, or allow users to build their own queries and retreive data on demand (basically, we must build a web api on top of the database in order to make it accessible).

@chriswhong
chriswhong / idea.md
Created July 1, 2016 20:08
Idea for git-powered distributed dataset management

The Problem:

If you follow the open data scene, you'll often hear about how the "feedback loop" for making corrections, comments, or asking questions about datasets is either fuzzy, disjointed, or nonexistent. If I know for a fact that something in a government dataset is wrong, how do I get that record fixed? Do I call 311? Will the operator even know what I am talking about if I say I want to make a correction to a single record in a public dataset? There's DAT. There's storing your data as a CSV in github. These approaches work, but are very much developer-centric. (pull requests and diffs are hard to wrap your head around if you spend your day analyzing data in excel or desktop GIS. The fact of the matter is that most of the people managing datasets in government organizations are not DBAs, data scientists, or programmers.

Idea:

It's basically git for data plus a simple UI for exploration, management, and editing. Users would have to use Github SSO to edit in the UI, and behind the scenes

@chriswhong
chriswhong / index.html
Last active October 11, 2017 11:09
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>carto-mapboxgl-demo</title>
<link href='https://api.mapbox.com/mapbox-gl-js/v0.26.0/mapbox-gl.css' rel='stylesheet' />
<style>
body {
@chriswhong
chriswhong / geosupport-autocomplete.md
Last active November 6, 2017 14:51
An autocomplete web API for geosupport

Problem

DCP maintains geosupport, the city's official geocoder, which can be accessed by using the libraries directly, or via various client-facing application/services including GOAT, GeoClient, GeoBat, etc.

The NYC Planning Labs team would like to incorporate geosupport results into web mapping applications, but geosupport's architecture does not work well with the "autocomplete" search style that is popular in modern web applications. Traditionally, to use geosupport, the user must provide at least a house number, street name, and borough, and will either git a hit, or a list of possible alternatives.

Below we propose some high-level specifications for a GeoSupport Autocomplete Web API that would be useful for search in modern web applications.

A Model to Emulate: Mapzen Search's Autocomplete API

Mapzen Search has an autocomplete API, which is well-documente, open source, and provides the type of experience developers want in a modern g

@chriswhong
chriswhong / pluto-carto.md
Last active June 15, 2018 23:22
Loading PLUTO into Carto

To load MapPLUTO into carto, the best approach is to upload the five borough shapefiles together, then UNION ALL them together.

  • Upload all five zipped borough shapefiles from Bytes of the Big Apple. Be sure to uncheck 'Allow Carto to guess column types" when uploading, or you'll get column type mismatches
  • UNION ALL the tables together with the following query. We can't just SELECT * because we'd have duplicate cartodb_ids in the result set, and saving as a new table would fail.
SELECT the_geom,the_geom_webmercator,borough,block,lot,cd,ct2010,cb2010,schooldist,council,zipcode,firecomp,policeprct,healthcent,healtharea,sanitboro,sanitdistr,sanitsub,address,zonedist1,zonedist2,zonedist3,zonedist4,overlay1,overlay2,spdist1,spdist2,spdist3,ltdheight,splitzone,bldgclass,landuse,easements,ownertype,ownername,lotarea,bldgarea,comarea,resarea,officearea,retailarea,garagearea,strgearea,factryarea,otherarea,areasource,numbldgs,numfloors,unitsres,unitstotal,lotfront,lotdepth,bldgfront,bldgdepth,ext,pro
@chriswhong
chriswhong / downloadroute.js
Created July 10, 2018 05:02
Streaming from ps-promise to json2csv to express response
// This code queries postgresql and streams the results into a csv
// Works like a charm for a download button, no hangup for large files!
const Json2csvTransform = require('json2csv').Transform;
const QueryStream = require('pg-query-stream');
const JSONStream = require('JSONStream');
router.get('/download.csv', async (req, res) => {
const { query } = req;
@chriswhong
chriswhong / gist:11e2b2d68505e239f89f46e36aaa03d2
Created July 20, 2018 18:04
Convenience Redirect in express.js. Build URLs using a known unique id
const express = require('express');
const router = express.Router();
router.get('/:ulurpnumber', async (req, res) => {
const { app, params } = req;
const { ulurpnumber } = params;
// find projectid for this ceqrnumber
// http://localhost:3000/projects/ulurp/170358ZMM
@chriswhong
chriswhong / chart.js
Last active July 27, 2018 20:14
A D3 Stacked Bar Chart showing NYC Planning Labs' Weekly Commits on Major Projects between July 2017 and July 2018
// based on https://bl.ocks.org/mbostock/3886208
const svg = d3.select('svg');
const margin = {
top: 20, right: 20, bottom: 100, left: 40,
};
@chriswhong
chriswhong / README.md
Created February 27, 2018 06:41
Consuming USGS OrthoImagery in MapboxGL

MapboxGL Layer Definition for consuming tiles from the USGS Imagery MapServer

{
  id: 'usgs-aerials',
  type: 'raster',
  source: {
    type: 'raster',
    tiles: [
      'https://basemap.nationalmap.gov/arcgis/rest/services/USGSImageryOnly/MapServer/tile/{z}/{y}/{x}',
 ],