Skip to content

Instantly share code, notes, and snippets.

@johan
Created January 24, 2017 06:15
Show Gist options
  • Star 10 You must be signed in to star a gist
  • Fork 5 You must be signed in to fork a gist
  • Save johan/db11e7bd04f030031dae209fa1a6c3e4 to your computer and use it in GitHub Desktop.
Save johan/db11e7bd04f030031dae209fa1a6c3e4 to your computer and use it in GitHub Desktop.
Makefile of the steps in Mike Bostock's command-line cartography tutorial, parts 1-4 https://medium.com/@mbostock/command-line-cartography-part-1-897aa8f8ca2c
# request one at http://api.census.gov/data/key_signup.html and paste it below
CENSUS_API_KEY=YOUR_CODE_HERE
# a factor 1609.34 squared
SQ_M_TO_SQ_MI=2589975.2356
#prereqs:
# npm install -g shapefile # 0.6.1
# npm install -g d3-geo-projection # 1.2.1
# npm install -g ndjson-cli # 0.3.0
# npm install -g d3 # 4.4.4
#nice-to-have:
# npm install -g d3-dsv # 1.0.3
# Part 1:
# https://medium.com/@mbostock/command-line-cartography-part-1-897aa8f8ca2c
# FIPS codes for US states, used for these file names:
# 01 AL ALABAMA
# 02 AK ALASKA
# 04 AZ ARIZONA
# 05 AR ARKANSAS
# 06 CA CALIFORNIA
# 08 CO COLORADO
# 09 CT CONNECTICUT
# 10 DE DELAWARE
# 11 DC DISTRICT OF COLUMBIA
# 12 FL FLORIDA
# 13 GA GEORGIA
# 15 HI HAWAII
# 16 ID IDAHO
# 17 IL ILLINOIS
# 18 IN INDIANA
# 19 IA IOWA
# 20 KS KANSAS
# 21 KY KENTUCKY
# 22 LA LOUISIANA
# 23 ME MAINE
# 24 MD MARYLAND
# 25 MA MASSACHUSETTS
# 26 MI MICHIGAN
# 27 MN MINNESOTA
# 28 MS MISSISSIPPI
# 29 MO MISSOURI
# 30 MT MONTANA
# 31 NE NEBRASKA
# 32 NV NEVADA
# 33 NH NEW HAMPSHIRE
# 34 NJ NEW JERSEY
# 35 NM NEW MEXICO
# 36 NY NEW YORK
# 37 NC NORTH CAROLINA
# 38 ND NORTH DAKOTA
# 39 OH OHIO
# 40 OK OKLAHOMA
# 41 OR OREGON
# 42 PA PENNSYLVANIA
# 44 RI RHODE ISLAND
# 45 SC SOUTH CAROLINA
# 46 SD SOUTH DAKOTA
# 47 TN TENNESSEE
# 48 TX TEXAS
# 49 UT UTAH
# 50 VT VERMONT
# 51 VA VIRGINIA
# 53 WA WASHINGTON
# 54 WV WEST VIRGINIA
# 55 WI WISCONSIN
# 56 WY WYOMING
# 60 AS AMERICAN SAMOA
# 66 GU GUAM
# 72 PR PUERTO RICO
# 78 VI VIRGIN ISLANDS
# geometries
# cb_2014_06_tract_500k.zip: CA
%.zip:
curl "http://www2.census.gov/geo/tiger/GENZ2014/shp/$@" -o "$@"
# extracting the shapefile and its dBASE features (.shp / .dbf)
cb_2014_06_tract_500k.%: cb_2014_06_tract_500k.zip
unzip -o "$<" "$@" ; touch "$@"
# populate a California shapefile of geometry and features
ca.json: cb_2014_06_tract_500k.shp cb_2014_06_tract_500k.dbf
shp2json "$<" -o "$@"
# pre-project coordinates for us, to go easier on mobile devices
ca-albers.json: ca.json
geoproject 'd3.geoConicEqualArea().parallels([34, 40.5]).rotate([120, 0]).fitSize([960, 960], d)' < "$<" > "$@"
# Part 2:
# https://medium.com/@mbostock/command-line-cartography-part-2-c3a82c5c0f3
# one feature per line
ca-albers.ndjson: ca-albers.json
ndjson-split 'd.features' < "$<" > "$@"
# set id of each feature
ca-albers-id.ndjson: ca-albers.ndjson
ndjson-map 'd.id = d.properties.GEOID.slice(2), d' < "$<" > "$@"
# The magic trait B01003 means population estimate below for census.gov data:
# get census info for every Californian (in=state:06) census tract (for=tract:*)
cb_2014_06_tract_B01003.json:
curl "http://api.census.gov/data/2014/acs5?key=${CENSUS_API_KEY}&get=B01003_001E&for=tract:*&in=state:06" -o "$@"
# chop header line, and map to objects with id and B01003 properties
cb_2014_06_tract_B01003.ndjson: cb_2014_06_tract_B01003.json
ndjson-cat "$<" \
| ndjson-split 'd.slice(1)' \
| ndjson-map '{id: d[2] + d[3], B01003: +d[0]}' \
> "$@"
# tuple them up on every line
ca-albers-join.ndjson: ca-albers-id.ndjson cb_2014_06_tract_B01003.ndjson
ndjson-join 'd.id' $^ > "$@"
# set the .properties.density on the first item of each line, and return just it
ca-albers-density.ndjson: ca-albers-join.ndjson
ndjson-map "d[0].properties = {density: Math.floor(d[1].B01003 / d[0].properties.ALAND * ${SQ_M_TO_SQ_MI})}, d[0]" \
< "$<" \
> "$@"
# compose it back to geoJSON again, if we want to verify looks
ca-albers-density.json: ca-albers-density.ndjson
ndjson-reduce < "$<" \
| ndjson-map '{type: "FeatureCollection", features: d}' \
> "$@"
# or, equivalently:
# ndjson-reduce 'p.features.push(d), p' \
# '{type: "FeatureCollection", features: []}' \
# < "$<" > "$@"
# set fill property by a sequential scale with the Viridis color scheme with D3
ca-albers-color.ndjson: ca-albers-density.ndjson
ndjson-map -r d3 \
'(d.properties.fill = d3.scaleSequential(d3.interpolateViridis).domain([0, 4000])(d.properties.density), d)' \
< "$<" > "$@"
# geo2svg (d3-geo-projection) composes this back to a density-skewed honest map
ca-albers-color.svg: ca-albers-color.ndjson Makefile
geo2svg -n --stroke none -p 1 -w 960 -h 960 < "$<" > "$@"
# Part 3:
# https://medium.com/@mbostock/command-line-cartography-part-3-1158e4c55a1e
#prereqs:
# npm install -g topojson # 2.2.0
# topojson:ize!
ca-tracts-topo.json: ca-albers-density.ndjson
geo2topo -n tracts="$<" > "$@"
# simplify, to a precision of 1 square pixel; we have conic-equal-area projected
ca-simple-topo.json: ca-tracts-topo.json
toposimplify -p 1 -f < "$<" > "$@"
# integer-quantize and delta-encode topology
ca-quantized-topo.json: ca-simple-topo.json
topoquantize 1e5 < "$<" > "$@"
# add in counties in the topology (composed from their sub-tracts)
# - join tracts by the id's first three digits
ca-merge-topo.json: ca-quantized-topo.json
topomerge -k 'd.id.slice(0, 3)' counties=tracts < "$<" > "$@"
# state-internal county borders, sans state outline
# (polygons a and b are the same for exterior arcs)
ca-topo.json: ca-merge-topo.json
topomerge --mesh -f 'a !== b' counties=counties < "$<" > "$@"
# Part 4:
# https://medium.com/@mbostock/command-line-cartography-part-4-82d0d26df0cf
#prereq:
# npm install -g d3-scale-chromatic # 1.1.0
# compose back into an svg again for perusal
ca-tracts-color.svg: ca-topo.json
topo2geo tracts=- < "$<" \
| ndjson-map -r d3 'z = d3.scaleSequential(d3.interpolateViridis).domain([0, 4000]), d.features.forEach(f => f.properties.fill = z(f.properties.density)), d' \
| ndjson-split 'd.features' \
| geo2svg -n --stroke none -p 1 -w 960 -h 960 \
> "$@"
# map population with a non-linear transform - let's try square root
ca-tracts-sqrt.svg: ca-topo.json
topo2geo tracts=- < "$<" \
| ndjson-map -r d3 'z = d3.scaleSequential(d3.interpolateViridis).domain([0, 100]), d.features.forEach(f => f.properties.fill = z(Math.sqrt(f.properties.density))), d' \
| ndjson-split 'd.features' \
| geo2svg -n --stroke none -p 1 -w 960 -h 960 \
> "$@"
# or, logarithmic:
ca-tracts-log.svg: ca-topo.json
topo2geo tracts=- < "$<" \
| ndjson-map -r d3 'z = d3.scaleLog().domain(d3.extent(d.features.filter(f => f.properties.density), f => f.properties.density)).interpolate(() => d3.interpolateViridis), d.features.forEach(f => f.properties.fill = z(f.properties.density)), d' \
| ndjson-split 'd.features' \
| geo2svg -n --stroke none -p 1 -w 960 -h 960 \
> "$@"
# quantile scale
ca-tracts-quantile.svg: ca-topo.json
topo2geo tracts=- < "$<" \
| ndjson-map -r d3 'z = d3.scaleQuantile().domain(d.features.map(f => f.properties.density)).range(d3.quantize(d3.interpolateViridis, 256)), d.features.forEach(f => f.properties.fill = z(f.properties.density)), d' \
| ndjson-split 'd.features' \
| geo2svg -n --stroke none -p 1 -w 960 -h 960 \
> "$@"
# chromatic custom threshold scale
ca-tracts-threshold.svg: ca-topo.json
topo2geo tracts=- < "$<" \
| ndjson-map -r d3 -r d3=d3-scale-chromatic 'z = d3.scaleThreshold().domain([1, 10, 50, 200, 500, 1000, 2000, 4000]).range(d3.schemeOrRd[9]), d.features.forEach(f => f.properties.fill = z(f.properties.density)), d' \
| ndjson-split 'd.features' \
| geo2svg -n --stroke none -p 1 -w 960 -h 960 \
> "$@"
# and merge in the county borders again
ca.svg: ca-topo.json
(topo2geo tracts=- \
< "$<" \
| ndjson-map -r d3 -r d3=d3-scale-chromatic 'z = d3.scaleThreshold().domain([1, 10, 50, 200, 500, 1000, 2000, 4000]).range(d3.schemeOrRd[9]), d.features.forEach(f => f.properties.fill = z(f.properties.density)), d' \
| ndjson-split 'd.features' \
;topo2geo counties=- \
< "$<" \
| ndjson-map 'd.properties = {"stroke": "#000", "stroke-opacity": 0.3}, d' \
) | geo2svg -n --stroke none -p 1 -w 960 -h 960 \
> "$@"
@JaimeStill
Copy link

had to add the -L parameter to the initial curl command because of a redirect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment