Skip to content

Instantly share code, notes, and snippets.

Avatar

Chris Whong chriswhong

View GitHub Profile
@chriswhong
chriswhong / gist:ff86b482273ce20cc8dc2391a6b30c4c
Created May 11, 2020
Pulling a Qri versioned dataset from the distributed web into a pandas dataframe (python)
View gist:ff86b482273ce20cc8dc2391a6b30c4c
# an example of pulling in a CSV string from a local qri repo,
# and parsing it into a dataframe with pandas.read_csv()
import pandas as pd
import subprocess
from io import StringIO
# this Qri class should eventually have all the methods we would need for reading,
# writing, pushing, pulling, committing, etc.
class Qri:
@chriswhong
chriswhong / gist:93ed4660ff23fdf483b41654290f0b18
Created Apr 7, 2020
Socrata Download URL generates different files when accessed multiple times
View gist:93ed4660ff23fdf483b41654290f0b18
## Problem
When troubleshooting another bug at qri.io, we noticed that hitting the same download URL from the NYC Open Data portal (powered by Socrata) generated csv files with different md5 hashes. This was causing our version control software to consider the csv "changed". On further investigation, the actual data was the same, but the ordering of the rows was different. This is likely due to the fact that the files are being generated on the fly from a database query (PostgreSQL in the case of Socrata open data portals) returning unpredictable row orders when no `ORDER BY` is specified in the query.
To reproduce:
- curl the same download URL several times and save to csv, such as this one for popular baby names:
```
$curl https://data.cityofnewyork.us/resource/25th-nujf.csv > 0.csv
$curl https://data.cityofnewyork.us/resource/25th-nujf.csv > 1.csv
@chriswhong
chriswhong / convert csv with WKT to GeoJson
Last active Mar 5, 2020
How to parse a CSV with a well-known text geometry column and convert to geojson FeatureCollection with the 'wellknown' package
View convert csv with WKT to GeoJson
// Use papa parse to download and parse a CSV
Papa.parse('data/nyc_zoning_lots.csv', {
download: true,
header: true,
complete: ({ data }) => {
// transform array of objects to geojson FeatureCollection
const FC = {
type: 'FeatureCollection',
features: data.map((row) => { // map csv rows to geojson features
const geometry = wellknown.parse(row.geom) // WKT to geojson geometry
@chriswhong
chriswhong / README.md
Created Feb 28, 2020
Using ogr2ogr to load a CSV into Postgres
View README.md

The absolute easiest way to get a CSV into a postgresql table is to use ogr2ogr with AUTODETECT_TYPE=YES.

I learned a while back that this is what cartoDB uses to import your CSV into postgis (with a lot of other parameters added)

ogr2ogr -f PostgreSQL PG:"host=localhost user=postgres dbname=postgres password=password"  docs.csv -oo AUTODETECT_TYPE=YES
@chriswhong
chriswhong / Qmf6Vfs78CeKHsWus2gXjBkvha93acBXN3FB7DtEFha4oj.csv
Created Jan 23, 2020
NYC Plaza Program Sites from Qri Dataset chriswhong/nyc_plaza_program_sites@/ipfs/Qmf6Vfs78CeKHsWus2gXjBkvha93acBXN3FB7DtEFha4oj.csv
View Qmf6Vfs78CeKHsWus2gXjBkvha93acBXN3FB7DtEFha4oj.csv
We can make this file beautiful and searchable if this error is corrected: It looks like row 5 should actually have 13 columns, instead of 9. in line 4.
plaza,boro_community_district_1,boro_community_district_2,level,area_sf,on_street,cross_street_1,cross_street_2,partner,partner_info_url,longitude,latitude,cecm_map
Albee Square Plaza,BK02,,C,25200,Dekalb Ave,Bond Street,Fulton Mall,Fulton Mall Improvement Association,http://downtownbrooklyn.com/about/fulton-mall-improvement-association,-73.983171,40.689973,https://www1.nyc.gov/assets/cecm/downloads/plaza-maps/bk-albee-square.pdf
Ave C Plaza,BK12,,D,5600,McDonald Ave,Avenue C,Church Ave,Kensington Stewards,https://bklyner.com/want-a-new-pedestrian-plaza-in-kensington-attend-this-public-workshop-on-wednesday-april-29-kensington/,-73.979094,40.640571,https://www1.nyc.gov/assets/cecm/downloads/plaza-maps/bk-avenue-c-plaza.pdf
Brooklyn Plaza,BK02,,B,50000,Jay Street,High Street,Manhattan Bridge,DOT,https://www1.nyc.gov/html/dot/html/home/home.shtml,-73.986569,40.699203,https://www1.nyc.gov/assets/cecm/downloads/plaza-maps/bk-brooklyn-plaza.pdf
Cadman Plaza East,BK02,,,,Cadman Plaza East,Tillary St,Johnson St,Down
@chriswhong
chriswhong / index.html
Last active Sep 9, 2019
Qri blob loading spinner prototype
View index.html
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/ >
<title>Qri Loading Spinner Prototype</title>
<script src="https://d3js.org/d3.v3.min.js"></script>
<!-- jQuery -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<!-- Bootstrap 3 -->
View keybase.md

Keybase proof

I hereby claim:

  • I am chriswhong on github.
  • I am cwhong (https://keybase.io/cwhong) on keybase.
  • I have a public key ASCScIKKKJdJewGrjgHOzYmnx7P7j7d2Bxdxs4E3qVopBQo

To claim this, I am signing this object:

@chriswhong
chriswhong / README.md
Last active May 30, 2019
Clickable Markers in MapboxGL
View README.md

This example shows how to extend mapboxGL's Marker class to add custom functionality on click.

Why?

MapboxGL has a very convenient Marker class that can be used to quickly get markers on the map with a few lines of code (versus the more complex method of adding sources and layers). They behave a bit differently from the rest of the map features because they are actually HTML elements overlaid on the map canvas.

The stock markers are great, and they are SVG so you can color them by passing in a color option! However, the only interactivity you can easily set up is a popup. When you google 'clickable markers', the examples you find are all using symbol layers with queryRenderedFeatures.

I wanted to trigger navigation in a single page app using the stock Markers and determined a simple extension of the class would help me accomplish this. You can also see these markers in action at https://paintthetown.chriswhong.com

@chriswhong
chriswhong / scrape.js
Created Apr 22, 2019
Decrypting Amtrak's real-time train location geoJSON feed
View scrape.js
// decrypting Amtrak's real-time train location geoJSON feed
// based on https://github.com/Vivalize/Amtrak-Train-Stats
const fetch = require('node-fetch');
const CryptoJS = require('crypto-js');
// this is the xhr call done by https://www.amtrak.com/track-your-train.html containing encrypted train location data
const dataUrl = 'https://maps.amtrak.com/services/MapDataService/trains/getTrainsData';
// these constants are pulled from RoutesList.v.json, which is an object with keys 'arr', 's', and 'v'
const sValue = '9a3686ac'; // found at s[8]
@chriswhong
chriswhong / Maputnik.png
Last active Feb 7, 2019
Custom buffersize param on carto maps api tile instantiation
You can’t perform that action at this time.