Skip to content

Instantly share code, notes, and snippets.

@pstuffa

pstuffa/source.js

Created August 16, 2018 15:28
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pstuffa/b875906769690583545195c0133d5bb5 to your computer and use it in GitHub Desktop.
Save pstuffa/b875906769690583545195c0133d5bb5 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
source.js
// URL: https://beta.observablehq.com/@pstuffa/making-maps-with-nyc-open-data
// Title: 3D Topographical Maps with NYC Open Data + D3
// Author: pstuffa (@pstuffa)
// Version: 1469
// Runtime version: 1
const m0 = {
id: "bd35dbda7bea48d1@1469",
variables: [
{
inputs: ["md"],
value: (function(md){return(
md`# 3D Topographical Maps with NYC Open Data + D3`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`This walkthrough is part of a class for ITP's Summer Program on [Making Data Tangible](https://itp.nyu.edu/camp2018/session/24). We'll walk through how to make a 3D topographical map using NYC Open Data + D3 for generating the design.`
)})
},
{
name: "PrintedMap",
value: (async function(){return(
await new Promise((resolve, reject) => {
let i = new Image();
i.onload = () => resolve(i);
i.onerror = reject;
// i.crossOrigin = 'anynomous';
i.src = 'https://user-images.githubusercontent.com/8422826/41239520-7c66230a-6d66-11e8-8d35-52bb1508b93a.jpg';
i.width = 600;
i.height = 600;
})
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`## NYC Open Data`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`New York City has a lot of [data](https://opendata.cityofnewyork.us/), and they do a great job making this data easily available. Here, we'll pull the NYC Census data, [available here](https://data.cityofnewyork.us/City-Government/Census-Demographics-at-the-Neighborhood-Tabulation/rnsn-acs2), so we can use population metrics to build our 3D map. Let's load it into our notebook and look at it a bit.`
)})
},
{
name: "nycOpenData",
inputs: ["d3","nycOpenDataLookup","datasets"],
value: (function(d3,nycOpenDataLookup,datasets){return(
d3.json(nycOpenDataLookup[datasets])
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`## Designing a 3D Map`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`## What's a Topographical Map?`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`Edward Tufte calls the Swiss national mountain maps "a standard of excellence for serious information displays." These maps are topographical maps, which essentially is any map with large details and a quantitative representation. Often, the quantiative representation is elevation using contours areas. In the image below, you can see the elevation noted as contours using red lines.`
)})
},
{
name: "cImg",
value: (async function(){return(
await new Promise((resolve, reject) => {
let i = new Image();
i.onload = () => resolve(i);
i.onerror = reject;
// i.crossOrigin = 'anynomous';
i.src = 'http://4.bp.blogspot.com/-km2kDrF4gy0/UkH5V5qZttI/AAAAAAAACd0/MRaQtB8Q8_8/s1600/swissmap.png';
i.width = 600;
i.height = 600;
})
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`Why is this such a great example of data visualization? Well, as Tufte notes, it's 100% about the content and 0% about "chart junk." Everything you see on the map is meaningful in some way, from the names of the towns and cities, the labels for the elevation contours, to the light shading of the colors to give it a three dimensional depth.`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md` As we design our map for printing, let's keep these ideas in mind. While we want to do something feasible for printing, let's make sure we're still representing the data effectively. Something as detailed as this Swiss map will be hard to print, as there are many details that will be lost in the process. But to note, the Swiss map is not 100% accurate - no mountain ranges are that perfectly smooth. They too are making approximations and generalizations so the visual works better for its medium, which is print. So, like they, we will try to find the right level of approximiation that doesn't degrade the data but works well for our medium.`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`## Let's Make a Map`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`I have this JSON file of a NYC topojson map. In this format, it's pretty simple to create a map in D3. We'll use a custom projection that takes this object, a width and height, and a map projection, and combine that with d3.geoPath(), which serves as our SVG Path generator. When we then create the path object, we bind our data and call the Path generator, which draws it on the SVG. Then, we add some simple styling to show the object.`
)})
},
{
name: "path",
inputs: ["d3","width","height","nyc"],
value: (function(d3,width,height,nyc){return(
d3.geoPath()
.projection(d3.geoConicConformal()
.parallels([33, 45])
.rotate([96, -39])
.fitSize([width, height], nyc))
)})
},
{
name: "nyc",
inputs: ["d3"],
value: (function(d3){return(
d3.json("https://gist.githubusercontent.com/pstuffa/928a2a31f352e59edef5ef56fa767e20/raw/7ba0230c627237c12cc1b3809f85d99486621756/nyc.json")
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`You can see in the object above there is an array of 310 objects in the 'features' section, which is an object for each NYC neighbordhood. In the rendering below, it's technically one single path object.`
)})
},
{
name: "BasicMap",
inputs: ["d3","DOM","width","height","nyc","path"],
value: (function(d3,DOM,width,height,nyc,path)
{
const svg = d3.select(DOM.svg(width, height))
.style("width", "100%")
.style("height", "auto");
svg.selectAll("path")
.data(nyc.features)
.enter().append("path")
.attr("d", path)
.style("fill", "none")
.style("stroke", "#000")
.style("stroke-width", .15)
.attr("id", "nycPath");
svg.append("clipPath")
.attr("id", "clipPathID")
.append("use")
.attr("xlink:href","#nycPath");
return svg.node();
}
)
},
{
inputs: ["md"],
value: (function(md){return(
md`Now we want to layer in our data. Let's look at our data again:`
)})
},
{
inputs: ["nycOpenData"],
value: (function(nycOpenData){return(
nycOpenData
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`An array of 197 objects. Arrays of objects are our friends, as they work very well with D3.`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`## Choropleth Map`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`A Choropleth map is a standard choice for layering data on top of a map. Here we are matching the neighborhoods from our geojson with the census data from NYC Open Data. We use color then to represent the number of people living in that neighborhood. Notice there is an OK match rate.`
)})
},
{
name: "color",
inputs: ["d3","colorScheme"],
value: (function(d3,colorScheme){return(
d3.scaleSequential(d3["interpolate" + colorScheme])
)})
},
{
name: "Choropleth",
inputs: ["d3","DOM","width","height","color","nycOpenData","nyc","path","neighborhoodLookup","nycNeighborhoodLookupMap"],
value: (function(d3,DOM,width,height,color,nycOpenData,nyc,path,neighborhoodLookup,nycNeighborhoodLookupMap)
{
const svg = d3.select(DOM.svg(width, height))
.style("width", "100%")
.style("height", "auto");
color.domain(d3.extent(nycOpenData, d => +d["total_population_2010_number"]));
svg.selectAll("path")
.data(nyc.features)
.enter().append("path")
.attr("d", path)
.style("fill", d => {
let val = neighborhoodLookup[ "$" + nycNeighborhoodLookupMap[d["properties"]["neighborhood"]]];
return val > 0 ? color(val) : "none";
})
.style("stroke", "#000")
.style("stroke-width", .05)
.attr("id", "nycPath");
return svg.node();
}
)
},
{
inputs: ["md"],
value: (function(md){return(
md`## Bubble Map`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`A slight variation on the Choropleth, a Bubble Map layers circles ontop of the neighborhood centroids, sized by a radius scale, which is based on the metric we're using. Just to emphasize, I added the color scale to the circles so they stand out even more. I think its' pretty interesting that with this approach, it's much less noticeable the low match rate.`
)})
},
{
name: "BubbleMap",
inputs: ["d3","DOM","width","height","nyc","path","color","nycOpenData","neighborhoodLookup","nycNeighborhoodLookupMap"],
value: (function(d3,DOM,width,height,nyc,path,color,nycOpenData,neighborhoodLookup,nycNeighborhoodLookupMap)
{
const svg = d3.select(DOM.svg(width, height))
.style("width", "100%")
.style("height", "auto");
svg.selectAll("path")
.data(nyc.features)
.enter().append("path")
.attr("d", path)
.style("fill", "#fff")
.style("stroke", "#000")
.style("stroke-width", .05)
.attr("id", "nycPath");
color.domain(d3.extent(nycOpenData, d => +d["total_population_2010_number"]));
const radiusScale = d3.scaleLinear()
.range([0, 300])
.domain(d3.extent(nycOpenData, d => +d["total_population_2010_number"]));
svg.selectAll("circle")
.data(nyc.features)
.enter().append("circle")
.attr("cx", d => path.centroid(d)[0])
.attr("cy", d => path.centroid(d)[1])
.style("fill", "steelblue")
.style("fill-opacity", .75)
.attr("r", d => {
let val = neighborhoodLookup[ "$" + nycNeighborhoodLookupMap[d["properties"]["neighborhood"]]];
return val == undefined ? 0 : Math.sqrt(radiusScale(val))/Math.PI;
})
.style("fill", d => {
let val = neighborhoodLookup[ "$" + nycNeighborhoodLookupMap[d["properties"]["neighborhood"]]];
return val == undefined ? 0 : color(val);
})
return svg.node();
}
)
},
{
inputs: ["md"],
value: (function(md){return(
md`To note on the radius scale, since the radius represents our metric, as to not over emphasize it's weight, we must run the √ and divide it by π`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`## Density Contours`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`If our goal is to 3D print this, we might want something that will represent the information similarily, but work for a 3D printing design. While the Choropleth map works for 2D, we might want something more like a topigraphical map for 3D. That's where we can use Density Contours to smooth out our data into more easily-3D-printable areas.`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`### Approximating Density`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`The density contour function is looking for a points to calculate density, and we have is a metric. What we can do to use the function then is to convert our metric into a number of points. Below, you'll see I have created a new dataset based on the metric - for every hundred, it will create a new point with the same location. (I added some jitter to the graph below so that you can see the points seperated a bit, otherwise it would look like a single point.)`
)})
},
{
name: "contoursData",
inputs: ["nyc","nycNeighborhoodLookupMap","neighborhoodLookup","d3"],
value: (function(nyc,nycNeighborhoodLookupMap,neighborhoodLookup,d3)
{
let values = [];
nyc.features.map(neighborhood => {
const lookupVal = "$" + nycNeighborhoodLookupMap[neighborhood["properties"]["neighborhood"]];
const numPopulation = neighborhoodLookup[lookupVal];
d3.range(numPopulation/1000).map(Object).forEach((d,i) => {
values.push({"neighborHoodD": neighborhood, popID: i});
})
})
return values;
}
)
},
{
name: "FauxDensityMap",
inputs: ["d3","DOM","width","height","nyc","path","contoursData","jitter"],
value: (function(d3,DOM,width,height,nyc,path,contoursData,jitter)
{
const svg = d3.select(DOM.svg(width, height))
.style("width", "100%")
.style("height", "auto");
svg.selectAll("path")
.data(nyc.features)
.enter().append("path")
.attr("d", path)
.style("fill", "#fff")
.style("stroke", "#000")
.style("stroke-width", .05)
.attr("id", "nycPath");
svg.selectAll("circle")
.data(contoursData)
.enter().append("circle")
.attr("cx", d => path.centroid(d.neighborHoodD)[0] + jitter(3))
.attr("cy", d => path.centroid(d.neighborHoodD)[1] + jitter(3))
.style("fill", "#000")
.style("fill-opacity", .05)
.attr("r", 2)
return svg.node();
}
)
},
{
name: "viewof bandWidth",
inputs: ["html"],
value: (function(html){return(
html`<input min=1 max=20 value=4 type=range>`
)})
},
{
name: "bandWidth",
inputs: ["Generators","viewof bandWidth"],
value: (G, _) => G.input(_)
},
{
name: "viewof contourCellSize",
inputs: ["html"],
value: (function(html){return(
html`<input min=1 max=20 value=3 type=range>`
)})
},
{
name: "contourCellSize",
inputs: ["Generators","viewof contourCellSize"],
value: (G, _) => G.input(_)
},
{
name: "densityContours",
inputs: ["d3","path","width","height","contourCellSize","bandWidth","contoursData"],
value: (function(d3,path,width,height,contourCellSize,bandWidth,contoursData){return(
d3.contourDensity()
.x(d => path.centroid(d.neighborHoodD)[0])
.y(d => path.centroid(d.neighborHoodD)[1])
.size([width, height])
.cellSize(contourCellSize)
.bandwidth(bandWidth)
(contoursData)
)})
},
{
name: "DensityContourMap",
inputs: ["d3","DOM","width","height","nyc","path","densityContours"],
value: (function(d3,DOM,width,height,nyc,path,densityContours)
{
const svg = d3.select(DOM.svg(width, height))
.style("width", "100%")
.style("height", "auto");
svg.append("path")
.datum(nyc)
.attr("d", path)
.style("fill", "none")
.attr("id", "nycPath");
const contours = svg.append("g")
.attr("fill", "none")
.selectAll(".contour")
.data(densityContours)
.enter().append("g");
contours.append("path")
.attr("class", "contour")
.attr("d", d3.geoPath())
.style("stroke-width",.25)
.style("stroke", "#000")
return svg.node();
}
)
},
{
inputs: ["md"],
value: (function(md){return(
md`Color can help define the levels of the contours to show the areas of highest density. Using D3's sequentialScale, we can input various color ranges and apply it to the density contours.`
)})
},
{
name: "viewof colorScheme",
inputs: ["dropdown","interpolators"],
value: (function(dropdown,interpolators){return(
dropdown(interpolators)
)})
},
{
name: "colorScheme",
inputs: ["Generators","viewof colorScheme"],
value: (G, _) => G.input(_)
},
{
name: "ColoredContourMap",
inputs: ["d3","DOM","width","height","color","densityContours","nyc","path"],
value: (function(d3,DOM,width,height,color,densityContours,nyc,path)
{
const svg = d3.select(DOM.svg(width, height))
.style("width", "100%")
.style("height", "auto");
color.domain(d3.extent(densityContours, d => d.value ));
svg.append("path")
.datum(nyc)
.attr("d", path)
.style("fill", "none")
.attr("id", "nycPath");
const contours = svg.append("g")
.attr("fill", "none")
.selectAll(".contour")
.data(densityContours)
.enter().append("g");
contours.append("path")
.attr("class", "contour")
.attr("d", d3.geoPath())
.style("stroke", "#000")
.style("stroke-width", .1)
.style("fill", d => color(d.value))
return svg.node();
}
)
},
{
inputs: ["md"],
value: (function(md){return(
md`We can add a clipPath object to the path, which cuts at outline for the path shape we created.`
)})
},
{
name: "ClipContourMap",
inputs: ["d3","DOM","width","height","color","densityContours","nyc","path","contourCellSize","bandWidth","contoursData"],
value: (function(d3,DOM,width,height,color,densityContours,nyc,path,contourCellSize,bandWidth,contoursData)
{
const svg = d3.select(DOM.svg(width, height))
.style("width", "100%")
.style("height", "auto");
color.domain(d3.extent(densityContours, d => d.value ));
svg.append("path")
.datum(nyc)
.attr("d", path)
.style("fill", "none")
.style("stroke", "#000")
.style("stroke-width", .1)
.attr("id", "nycPathTwo");
svg.append("clipPath")
.attr("id", "nyClipPathTwo")
.append("use")
.attr("xlink:href","#nycPathTwo");
const newContour = d3.contourDensity()
.x(d => path.centroid(d.neighborHoodD)[0])
.y(d => path.centroid(d.neighborHoodD)[1])
.size([width, height])
.cellSize(contourCellSize)
.bandwidth(bandWidth)
(contoursData)
const contours = svg.append("g")
.attr("fill", "none")
.selectAll(".contour")
.data(newContour)
.enter().append("g");
contours.append("path")
.attr("clip-path","url(#nyClipPathTwo)")
.attr("class", "contour")
.attr("d", d3.geoPath())
.style("stroke", "#000")
.style("stroke-width", .1)
.style("fill", d => color(d.value))
return svg.node();
}
)
},
{
inputs: ["md"],
value: (function(md){return(
md`If we compare this map with our Choropleth, how well does it represent the data?`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`## Export for Printing`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`For printing, we'll want to have each layer as separate as possible. This section creates an group for each layer, so it's easy to print them separately. I added a little expand scroller so you can see how this works.`
)})
},
{
name: "viewof ExpandHeight",
inputs: ["html"],
value: (function(html){return(
html`<input min=200 max=2200 value=200 step=50 type=range>`
)})
},
{
name: "ExpandHeight",
inputs: ["Generators","viewof ExpandHeight"],
value: (G, _) => G.input(_)
},
{
name: "ContourTiles",
inputs: ["d3","DOM","width","ExpandHeight","color","densityContours"],
value: (function(d3,DOM,width,ExpandHeight,color,densityContours)
{
const svg = d3.select(DOM.svg(width, ExpandHeight))
.style("width", "100%")
.style("height", "auto");
color.domain(d3.extent(densityContours, d => d.value ));
const exapandScale = d3.scaleLinear()
.range([0, ExpandHeight])
.domain([200, 2200]);
const yScale = d3.scaleBand()
.domain(d3.range(densityContours.length).map(Object))
.rangeRound([0, exapandScale(ExpandHeight)])
.paddingInner([.01]);
const contours = svg.append("g")
.attr("fill", "none")
.selectAll(".contour")
.data(densityContours)
.enter().append("g")
.attr("transform", (d,i) => `translate(0,${yScale(i)})`)
if(ExpandHeight > 1000) {
contours.append("rect")
.attr("height", yScale.bandwidth())
.attr("width", width)
.style("stroke", "#000")
.style("stroke-width", .25)
}
contours.append("path")
.attr("class", "contour")
.attr("d", d3.geoPath())
.style("stroke", "#000")
.style("stroke-width", .1)
.style("fill", d => color(d.value))
const fontScale = d3.scaleLinear()
.range([0, 16])
.domain([200, 2200]);
contours.append("text")
.style("fill","#000")
.style("font-size", fontScale(ExpandHeight)+"px")
.attr("dy", 20)
.attr("dx", 60)
.text((d,i) => `Tile #${i+1}`)
return svg.node();
}
)
},
{
inputs: ["md"],
value: (function(md){return(
md`Finally, we remove the color (you can always add this in by using different colored materials), and add the following layer to the base, which will help with the later construction.`
)})
},
{
name: "PrintableTiles",
inputs: ["d3","DOM","width","ExpandHeight","color","densityContours"],
value: (function(d3,DOM,width,ExpandHeight,color,densityContours)
{
const svg = d3.select(DOM.svg(width, ExpandHeight))
.style("width", "100%")
.style("height", "auto");
color.domain(d3.extent(densityContours, d => d.value ));
const exapandScale = d3.scaleLinear()
.range([0, ExpandHeight])
.domain([200, 2200]);
const yScale = d3.scaleBand()
.domain(d3.range(densityContours.length).map(Object))
.rangeRound([0, exapandScale(ExpandHeight)])
.paddingInner([.01]);
const contours = svg.append("g")
.attr("fill", "none")
.selectAll(".contour")
.data(densityContours)
.enter().append("g")
.attr("transform", (d,i) => `translate(0,${yScale(i)})`)
const nextContour = svg.append("g")
.attr("fill", "none")
.selectAll(".contour")
.data(densityContours)
.enter().append("g")
.attr("transform", (d,i) => `translate(0,${yScale(i) - yScale.bandwidth() })`)
if(ExpandHeight > 1500) {
contours.append("rect")
.attr("height", yScale.bandwidth())
.attr("width", width)
.style("stroke", "#000")
.style("stroke-width", .25)
}
nextContour.append("path")
.attr("class", "contour")
.attr("d", d3.geoPath())
.style("stroke", "#000")
.style("stroke-width", .1)
.style("fill", "none")
contours.append("path")
.attr("class", "contour")
.attr("d", d3.geoPath())
.style("stroke", "#000")
.style("stroke-width", .1)
.style("fill", "none")
const fontScale = d3.scaleLinear()
.range([0, 16])
.domain([200, 2200]);
contours.append("text")
.style("fill","#000")
.style("font-size", fontScale(ExpandHeight)+"px")
.attr("dy", 20)
.attr("dx", 60)
.text((d,i) => `Tile #${i+1}`)
return svg.node();
}
)
},
{
inputs: ["md"],
value: (function(md){return(
md`You can dowload the file here, which will be used for the printing process.`
)})
},
{
inputs: ["DOM","serialize","PrintableTiles"],
value: (function(DOM,serialize,PrintableTiles){return(
DOM.download(serialize(PrintableTiles), null, "Download Printable Tiles")
)})
},
{
inputs: ["DOM","serialize","BasicMap"],
value: (function(DOM,serialize,BasicMap){return(
DOM.download(serialize(BasicMap), null, "Download Basic Map")
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`And now the printing begins!
We picked 1/8" transparent acrylic sheets so that you can see throuh the model and get a sense of depth. You can pick anything handy and safe to laser cut.
Note that I peeled the top protective film and left the bottom layer on. In the process of laser cutting, the honeycomb mesh could leave marks on the material, which will be visiable in you use transparent material.`
)})
},
{
name: "printingOne",
value: (async function(){return(
await new Promise((resolve, reject) => {
let i = new Image();
i.onload = () => resolve(i);
i.onerror = reject;
// i.crossOrigin = 'anynomous';
i.src = 'https://user-images.githubusercontent.com/8422826/41239571-927334c6-6d66-11e8-9c1a-467fe04baf3a.jpg';
i.width = 600;
i.height = 600;
})
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`The smallest cuts might fall through the honey comb. Some blue tape would reduce your need of searching for them in the catch-all tray in the laser cutter.`
)})
},
{
name: "printingTwo",
value: (async function(){return(
await new Promise((resolve, reject) => {
let i = new Image();
i.onload = () => resolve(i);
i.onerror = reject;
// i.crossOrigin = 'anynomous';
i.src = 'https://user-images.githubusercontent.com/8422826/41239588-953a2ce6-6d66-11e8-8a36-da476ecd2474.jpg';
i.width = 600;
i.height = 600;
})
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`Peel off the protection layer off your pieces and start stacking them from layer 1 up, adding acrylic cement as you go. Group the small pieces with the tile so that you don't get them confused!`
)})
},
{
name: "printingThree",
value: (async function(){return(
await new Promise((resolve, reject) => {
let i = new Image();
i.onload = () => resolve(i);
i.onerror = reject;
// i.crossOrigin = 'anynomous';
i.src = 'https://user-images.githubusercontent.com/8422826/41239559-8d175c50-6d66-11e8-957c-4c0f5d956fa9.jpg';
i.width = 600;
i.height = 600;
})
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`The negative comes handy in case you want to cast the model or simply make a transportation case.`
)})
},
{
name: "printingFour",
value: (async function(){return(
await new Promise((resolve, reject) => {
let i = new Image();
i.onload = () => resolve(i);
i.onerror = reject;
// i.crossOrigin = 'anynomous';
i.src = 'https://user-images.githubusercontent.com/8422826/41247156-2859ed90-6d7b-11e8-8309-c116931192ef.jpg';
i.width = 600;
i.height = 600;
})
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`## Other Considerations...`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`### ThreeJS Implementation
There are (somewhat) simple ways of importing SVG paths into Three.js. Take a look at this [notebook](https://beta.observablehq.com/@pstuffa/hello-three-js-countour-map) if you want to learn more.`
)})
},
{
from: "@pstuffa/hello-three-js-countour-map",
name: "ThreeJSDensityContourMap",
remote: "ThreeJSDensityContourMap"
},
{
inputs: ["ThreeJSDensityContourMap"],
value: (function(ThreeJSDensityContourMap){return(
ThreeJSDensityContourMap
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`### Contours with Inverse Weighted Distance
Another approach is to use inverse weighted distance instead of density contours, which will preserve the original datapoints better than the density contours. You can read more about it [here](https://en.wikipedia.org/wiki/Inverse_distance_weighting). I hope to have a working Observable example of this soon.`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`### A True Topographical Map
NYC Open Data also has the files for a true NYC contour Map. Using Mapshaper.org, one can simplify the file so it's not too large for hosting. (If you're interested in Line Simplification, there's [a great blog post](https://bost.ocks.org/mike/simplify/) by Mike Bostock about it.) The link to the contour files is [here](https://data.cityofnewyork.us/City-Government/Contours/3cdm-p29e). I'm still working on getting this the right size for Observable, so stay tuned.`
)})
},
{
inputs: ["md"],
value: (function(md){return(
md`### The Least Fun Work
A lot of the work comes down to getting the data in a good spot. You'll notice in this section below that a good chunk of the time was spent wrangling the data in useable state. Since we're using files from different sources, many of the identifiers don't match up, so I had to go in and semi-manually correct them. If you want to try a different dataset, you'll need to figure out how to format it and get it working in a similar way. I have a few other datasets available via that drop down, you would just need to handle the metrics and the lookups accordingly.`
)})
},
{
name: "viewof datasets",
inputs: ["dropdown"],
value: (function(dropdown){return(
dropdown(["Census Data","Population Data","Housing Data","Some Other Data"])
)})
},
{
name: "datasets",
inputs: ["Generators","viewof datasets"],
value: (G, _) => G.input(_)
},
{
inputs: ["DOM","serialize","DensityContourMap"],
value: (function(DOM,serialize,DensityContourMap){return(
DOM.download(serialize(DensityContourMap), null, "Download Density Contour")
)})
},
{
inputs: ["nycOpenData"],
value: (function(nycOpenData){return(
nycOpenData
)})
},
{
inputs: ["nyc"],
value: (function(nyc){return(
nyc
)})
},
{
name: "nycOpenDataLookup",
value: (function(){return(
{
"Population Data" : "https://data.cityofnewyork.us/resource/y7yy-gq65.json",
"Housing Data": "https://data.cityofnewyork.us/resource/wwhg-3wy7.json",
"Census Data": "https://data.cityofnewyork.us/resource/w5g7-dwbx.json"
}
)})
},
{
name: "neighborhoodLookup",
inputs: ["d3","nycOpenData"],
value: (function(d3,nycOpenData){return(
d3.nest()
.key(d => d["geographic_area_neighborhood_tabulation_area_nta_name"])
.rollup(leaves => {
return d3.sum(leaves, d => +d["total_population_2010_number"])
})
.map(nycOpenData.filter(d => d["geographic_area_neighborhood_tabulation_area_nta_name"] != undefined ))
)})
},
{
name: "populationData",
inputs: ["nyc","nycNeighborhoodLookupMap","neighborhoodLookup"],
value: (function(nyc,nycNeighborhoodLookupMap,neighborhoodLookup){return(
nyc.features.map(neighborhood => {
const lookupVal = "$" + nycNeighborhoodLookupMap[neighborhood["properties"]["neighborhood"]];
const numPopulation = neighborhoodLookup[lookupVal];
return {"neighborHoodD": neighborhood, popID: numPopulation}
})
)})
},
{
name: "height",
value: (function(){return(
200
)})
},
{
name: "width",
value: (function(){return(
300
)})
},
{
name: "nycNeighborhoodLookup",
inputs: ["d3"],
value: (function(d3){return(
d3.csv("https://gist.githubusercontent.com/pstuffa/02b58268d9879492b22ef7cdfdd7930a/raw/dca951d78abe5a141afb1c999a191f2274fbcaaa/nycLookup.csv")
)})
},
{
name: "nycNeighborhoodLookupMap",
inputs: ["nycNeighborhoodLookup"],
value: (function(nycNeighborhoodLookup)
{
let map = {};
nycNeighborhoodLookup.forEach(d => {
if(d.geographic_area_neighborhood_tabulation_area_nta_name != "") {
map[d.MapNeighborHoods] = d.geographic_area_neighborhood_tabulation_area_nta_name
}
})
return map;
}
)
},
{
name: "interpolators",
value: (function(){return(
[
"Viridis",
"Inferno",
"Magma",
"Plasma",
"Warm",
"Cool",
"Rainbow",
"CubehelixDefault",
"Blues",
"Greens",
"Greys",
"Oranges",
"Purples",
"Reds",
"BuGn",
"BuPu",
"GnBu",
"OrRd",
"PuBuGn",
"PuBu",
"PuRd",
"RdPu",
"YlGnBu",
"YlGn",
"YlOrBr",
"YlOrRd"
]
)})
},
{
name: "d3",
inputs: ["require"],
value: (function(require){return(
require("d3@5")
)})
},
{
name: "dropdown",
inputs: ["html"],
value: (function(html){return(
function dropdown(options) {
let dd = html `<select>`
for (var i = 0; i < options.length; i++) {
var option = document.createElement("option");
option.value = options[i];
option.text = options[i];
dd.appendChild(option);
}
return dd;
}
)})
},
{
name: "serialize",
value: (function()
{
const xmlns = "http://www.w3.org/2000/xmlns/";
const xlinkns = "http://www.w3.org/1999/xlink";
const svgns = "http://www.w3.org/2000/svg";
return function serialize(svg) {
svg = svg.cloneNode(true);
svg.setAttributeNS(xmlns, "xmlns", svgns);
svg.setAttributeNS(xmlns, "xmlns:xlink", xlinkns);
const serializer = new window.XMLSerializer;
const string = serializer.serializeToString(svg);
return new Blob([string], {type: "image/svg+xml"});
};
}
)
},
{
name: "jitter",
value: (function()
{
return function jitter(deg) {
const firstVal = Math.random()*deg;
const secondVal = Math.random();
const thirdVal = (secondVal > .5 ? -1 : 1)
return firstVal * thirdVal;
}
}
)
}
]
};
const m1 = {
id: "@pstuffa/hello-three-js-countour-map",
variables: [
{
name: "ThreeJSDensityContourMap",
inputs: ["THREE","width","height","camera","group","scene"],
value: (function*(THREE,width,height,camera,group,scene)
{
const renderer = new THREE.WebGLRenderer({antialias: true});
renderer.setSize(width, height);
renderer.setPixelRatio(devicePixelRatio);
const controls = new THREE.OrbitControls( camera, renderer.domElement );
try {
while (true) {
group.rotation.z += 0.01;
group.rotation.y += 0.01;
renderer.render(scene, camera);
yield renderer.domElement;
}
} finally {
renderer.dispose();
}
}
)
},
{
name: "THREE",
inputs: ["require"],
value: (async function(require)
{
const THREE = await require("three@0.82.1/build/three.min.js");
if (!THREE.OrbitControls) { // If not previously loaded.
const module = window.module = {exports: undefined};
THREE.OrbitControls = await require("three-orbit-controls@82.1.0/index.js").catch(() => module.exports(THREE));
}
return THREE;
}
)
},
{
name: "width",
value: (function(){return(
800
)})
},
{
name: "height",
value: (function(){return(
800
)})
},
{
name: "camera",
inputs: ["width","height","THREE"],
value: (function(width,height,THREE)
{
const fov = 50;
const aspect = width / height;
const near = 1;
const far = 100000;
const camera = new THREE.PerspectiveCamera(fov, aspect, near, far);
camera.position.set( 0, 0, 400);
return camera;
}
)
},
{
name: "group",
inputs: ["THREE","initSVGObject","addGeoObject"],
value: (function(THREE,initSVGObject,addGeoObject)
{
const group = new THREE.Group();
const obj = initSVGObject();
addGeoObject( group, obj );
return group;
}
)
},
{
name: "scene",
inputs: ["THREE","group"],
value: (function(THREE,group)
{
const scene = new THREE.Scene();
scene.background = new THREE.Color(0xffffff);
scene.add(group);
return scene;
}
)
},
{
name: "initSVGObject",
inputs: ["densityContourPaths","colorScale"],
value: (function(densityContourPaths,colorScale){return(
() => {
const obj = {};
obj.paths = densityContourPaths;
obj.depths = densityContourPaths.map((d,i) => i * 5)
obj.colors = densityContourPaths.map((d,i) => colorScale(i))
obj.center = { x: 150, y: 100 };
return obj;
}
)})
},
{
name: "addGeoObject",
inputs: ["transformSVGPath","THREE"],
value: (function(transformSVGPath,THREE){return(
( group, svgObject ) => {
const paths = svgObject.paths;
const depths = svgObject.depths;
const colors = svgObject.colors;
const center = svgObject.center;
for ( var i = 0; i < paths.length; i ++ ) {
const path = transformSVGPath( paths[ i ] );
const color = new THREE.Color( colors[ i ] );
const material = new THREE.MeshLambertMaterial( {
color: color,
emissive: color
} );
const depth = depths[ i ];
const simpleShapes = path.toShapes( true );
for ( var j = 0; j < simpleShapes.length; j ++ ) {
const simpleShape = simpleShapes[ j ];
const shape3d = new THREE.ExtrudeGeometry( simpleShape, {
depth: depth,
bevelEnabled: false
} );
const mesh = new THREE.Mesh( shape3d, material );
mesh.rotation.x = Math.PI;
mesh.translateZ( - depth - 1 );
mesh.translateX( - center.x );
mesh.translateY( - center.y );
group.add( mesh );
}
}
}
)})
},
{
name: "densityContourPaths",
inputs: ["densityContours","d3"],
value: (function(densityContours,d3)
{
let paths = [];
densityContours.forEach((d) => {
paths.push(d3.geoPath()(d));
});
return paths;
}
)
},
{
name: "colorScale",
inputs: ["d3","densityContourPaths"],
value: (function(d3,densityContourPaths){return(
d3.scaleSequential(d3.interpolateViridis)
.domain([0, densityContourPaths.length])
)})
},
{
name: "transformSVGPath",
inputs: ["THREE","COMMA","SPACE","MINUS","DIGIT_0","DIGIT_9","PERIOD","DEGS_TO_RADS"],
value: (function(THREE,COMMA,SPACE,MINUS,DIGIT_0,DIGIT_9,PERIOD,DEGS_TO_RADS){return(
( pathStr ) => {
var path = new THREE.ShapePath();
var idx = 1, len = pathStr.length, activeCmd,
x = 0, y = 0, nx = 0, ny = 0, firstX = null, firstY = null,
x1 = 0, x2 = 0, y1 = 0, y2 = 0,
rx = 0, ry = 0, xar = 0, laf = 0, sf = 0, cx, cy;
function eatNum() {
var sidx, c, isFloat = false, s;
// eat delims
while ( idx < len ) {
c = pathStr.charCodeAt( idx );
if ( c !== COMMA && c !== SPACE ) break;
idx ++;
}
if ( c === MINUS ) {
sidx = idx ++;
} else {
sidx = idx;
}
// eat number
while ( idx < len ) {
c = pathStr.charCodeAt( idx );
if ( DIGIT_0 <= c && c <= DIGIT_9 ) {
idx ++;
continue;
} else if ( c === PERIOD ) {
idx ++;
isFloat = true;
continue;
}
s = pathStr.substring( sidx, idx );
return isFloat ? parseFloat( s ) : parseInt( s );
}
s = pathStr.substring( sidx );
return isFloat ? parseFloat( s ) : parseInt( s );
}
function nextIsNum() {
var c;
// do permanently eat any delims...
while ( idx < len ) {
c = pathStr.charCodeAt( idx );
if ( c !== COMMA && c !== SPACE ) break;
idx ++;
}
c = pathStr.charCodeAt( idx );
return ( c === MINUS || ( DIGIT_0 <= c && c <= DIGIT_9 ) );
}
var canRepeat;
activeCmd = pathStr[ 0 ];
while ( idx <= len ) {
canRepeat = true;
switch ( activeCmd ) {
// moveto commands, become lineto's if repeated
case 'M':
x = eatNum();
y = eatNum();
path.moveTo( x, y );
activeCmd = 'L';
firstX = x;
firstY = y;
break;
case 'm':
x += eatNum();
y += eatNum();
path.moveTo( x, y );
activeCmd = 'l';
firstX = x;
firstY = y;
break;
case 'Z':
case 'z':
canRepeat = false;
if ( x !== firstX || y !== firstY ) path.lineTo( firstX, firstY );
break;
// - lines!
case 'L':
case 'H':
case 'V':
nx = ( activeCmd === 'V' ) ? x : eatNum();
ny = ( activeCmd === 'H' ) ? y : eatNum();
path.lineTo( nx, ny );
x = nx;
y = ny;
break;
case 'l':
case 'h':
case 'v':
nx = ( activeCmd === 'v' ) ? x : ( x + eatNum() );
ny = ( activeCmd === 'h' ) ? y : ( y + eatNum() );
path.lineTo( nx, ny );
x = nx;
y = ny;
break;
// - cubic bezier
case 'C':
x1 = eatNum(); y1 = eatNum();
case 'S':
if ( activeCmd === 'S' ) {
x1 = 2 * x - x2;
y1 = 2 * y - y2;
}
x2 = eatNum();
y2 = eatNum();
nx = eatNum();
ny = eatNum();
path.bezierCurveTo( x1, y1, x2, y2, nx, ny );
x = nx; y = ny;
break;
case 'c':
x1 = x + eatNum();
y1 = y + eatNum();
case 's':
if ( activeCmd === 's' ) {
x1 = 2 * x - x2;
y1 = 2 * y - y2;
}
x2 = x + eatNum();
y2 = y + eatNum();
nx = x + eatNum();
ny = y + eatNum();
path.bezierCurveTo( x1, y1, x2, y2, nx, ny );
x = nx; y = ny;
break;
// - quadratic bezier
case 'Q':
x1 = eatNum(); y1 = eatNum();
case 'T':
if ( activeCmd === 'T' ) {
x1 = 2 * x - x1;
y1 = 2 * y - y1;
}
nx = eatNum();
ny = eatNum();
path.quadraticCurveTo( x1, y1, nx, ny );
x = nx;
y = ny;
break;
case 'q':
x1 = x + eatNum();
y1 = y + eatNum();
case 't':
if ( activeCmd === 't' ) {
x1 = 2 * x - x1;
y1 = 2 * y - y1;
}
nx = x + eatNum();
ny = y + eatNum();
path.quadraticCurveTo( x1, y1, nx, ny );
x = nx; y = ny;
break;
// - elliptical arc
case 'A':
rx = eatNum();
ry = eatNum();
xar = eatNum() * DEGS_TO_RADS;
laf = eatNum();
sf = eatNum();
nx = eatNum();
ny = eatNum();
if ( rx !== ry ) console.warn( 'Forcing elliptical arc to be a circular one:', rx, ry );
// SVG implementation notes does all the math for us! woo!
// http://www.w3.org/TR/SVG/implnote.html#ArcImplementationNotes
// step1, using x1 as x1'
x1 = Math.cos( xar ) * ( x - nx ) / 2 + Math.sin( xar ) * ( y - ny ) / 2;
y1 = - Math.sin( xar ) * ( x - nx ) / 2 + Math.cos( xar ) * ( y - ny ) / 2;
// step 2, using x2 as cx'
var norm = Math.sqrt( ( rx * rx * ry * ry - rx * rx * y1 * y1 - ry * ry * x1 * x1 ) /
( rx * rx * y1 * y1 + ry * ry * x1 * x1 ) );
if ( laf === sf ) norm = - norm;
x2 = norm * rx * y1 / ry;
y2 = norm * -ry * x1 / rx;
// step 3
cx = Math.cos( xar ) * x2 - Math.sin( xar ) * y2 + ( x + nx ) / 2;
cy = Math.sin( xar ) * x2 + Math.cos( xar ) * y2 + ( y + ny ) / 2;
var u = new THREE.Vector2( 1, 0 );
var v = new THREE.Vector2( ( x1 - x2 ) / rx, ( y1 - y2 ) / ry );
var startAng = Math.acos( u.dot( v ) / u.length() / v.length() );
if ( ( ( u.x * v.y ) - ( u.y * v.x ) ) < 0 ) startAng = - startAng;
// we can reuse 'v' from start angle as our 'u' for delta angle
u.x = ( - x1 - x2 ) / rx;
u.y = ( - y1 - y2 ) / ry;
var deltaAng = Math.acos( v.dot( u ) / v.length() / u.length() );
// This normalization ends up making our curves fail to triangulate...
if ( ( ( v.x * u.y ) - ( v.y * u.x ) ) < 0 ) deltaAng = - deltaAng;
if ( ! sf && deltaAng > 0 ) deltaAng -= Math.PI * 2;
if ( sf && deltaAng < 0 ) deltaAng += Math.PI * 2;
path.absarc( cx, cy, rx, startAng, startAng + deltaAng, sf );
x = nx;
y = ny;
break;
default:
throw new Error( 'Wrong path command: ' + activeCmd );
}
// just reissue the command
if ( canRepeat && nextIsNum() ) continue;
activeCmd = pathStr[ idx ++ ];
}
return path;
}
)})
},
{
name: "densityContours",
inputs: ["d3","path","width","height","contoursData"],
value: (function(d3,path,width,height,contoursData){return(
d3.contourDensity()
.x(d => path.centroid(d.neighborHoodD)[0])
.y(d => path.centroid(d.neighborHoodD)[1])
.size([width, height])
.cellSize(2)
.bandwidth(7)
(contoursData)
)})
},
{
name: "d3",
inputs: ["require"],
value: (function(require){return(
require("d3@5")
)})
},
{
name: "COMMA",
value: (function(){return(
44
)})
},
{
name: "SPACE",
value: (function(){return(
32
)})
},
{
name: "MINUS",
value: (function(){return(
45
)})
},
{
name: "DIGIT_0",
value: (function(){return(
48
)})
},
{
name: "DIGIT_9",
value: (function(){return(
57
)})
},
{
name: "PERIOD",
value: (function(){return(
46
)})
},
{
name: "DEGS_TO_RADS",
value: (function(){return(
Math.PI / 180
)})
},
{
from: "@pstuffa/making-maps-with-nyc-open-data",
name: "path",
remote: "path"
},
{
from: "@pstuffa/making-maps-with-nyc-open-data",
name: "contoursData",
remote: "contoursData"
}
]
};
const m2 = {
id: "@pstuffa/making-maps-with-nyc-open-data",
variables: [
{
name: "path",
inputs: ["d3","width","height","nyc"],
value: (function(d3,width,height,nyc){return(
d3.geoPath()
.projection(d3.geoConicConformal()
.parallels([33, 45])
.rotate([96, -39])
.fitSize([width, height], nyc))
)})
},
{
name: "contoursData",
inputs: ["nyc","nycNeighborhoodLookupMap","neighborhoodLookup","d3"],
value: (function(nyc,nycNeighborhoodLookupMap,neighborhoodLookup,d3)
{
let values = [];
nyc.features.map(neighborhood => {
const lookupVal = "$" + nycNeighborhoodLookupMap[neighborhood["properties"]["neighborhood"]];
const numPopulation = neighborhoodLookup[lookupVal];
d3.range(numPopulation/1000).map(Object).forEach((d,i) => {
values.push({"neighborHoodD": neighborhood, popID: i});
})
})
return values;
}
)
},
{
name: "d3",
inputs: ["require"],
value: (function(require){return(
require("d3@5")
)})
},
{
name: "width",
value: (function(){return(
300
)})
},
{
name: "height",
value: (function(){return(
200
)})
},
{
name: "nyc",
inputs: ["d3"],
value: (function(d3){return(
d3.json("https://gist.githubusercontent.com/pstuffa/928a2a31f352e59edef5ef56fa767e20/raw/7ba0230c627237c12cc1b3809f85d99486621756/nyc.json")
)})
},
{
name: "nycNeighborhoodLookupMap",
inputs: ["nycNeighborhoodLookup"],
value: (function(nycNeighborhoodLookup)
{
let map = {};
nycNeighborhoodLookup.forEach(d => {
if(d.geographic_area_neighborhood_tabulation_area_nta_name != "") {
map[d.MapNeighborHoods] = d.geographic_area_neighborhood_tabulation_area_nta_name
}
})
return map;
}
)
},
{
name: "neighborhoodLookup",
inputs: ["d3","nycOpenData"],
value: (function(d3,nycOpenData){return(
d3.nest()
.key(d => d["geographic_area_neighborhood_tabulation_area_nta_name"])
.rollup(leaves => {
return d3.sum(leaves, d => +d["total_population_2010_number"])
})
.map(nycOpenData.filter(d => d["geographic_area_neighborhood_tabulation_area_nta_name"] != undefined ))
)})
},
{
name: "nycNeighborhoodLookup",
inputs: ["d3"],
value: (function(d3){return(
d3.csv("https://gist.githubusercontent.com/pstuffa/02b58268d9879492b22ef7cdfdd7930a/raw/dca951d78abe5a141afb1c999a191f2274fbcaaa/nycLookup.csv")
)})
},
{
name: "nycOpenData",
inputs: ["d3","nycOpenDataLookup","datasets"],
value: (function(d3,nycOpenDataLookup,datasets){return(
d3.json(nycOpenDataLookup[datasets])
)})
},
{
name: "nycOpenDataLookup",
value: (function(){return(
{
"Population Data" : "https://data.cityofnewyork.us/resource/y7yy-gq65.json",
"Housing Data": "https://data.cityofnewyork.us/resource/wwhg-3wy7.json",
"Census Data": "https://data.cityofnewyork.us/resource/w5g7-dwbx.json"
}
)})
},
{
name: "viewof datasets",
inputs: ["dropdown"],
value: (function(dropdown){return(
dropdown(["Census Data","Population Data","Housing Data","Some Other Data"])
)})
},
{
name: "datasets",
inputs: ["Generators","viewof datasets"],
value: (G, _) => G.input(_)
},
{
name: "dropdown",
inputs: ["html"],
value: (function(html){return(
function dropdown(options) {
let dd = html `<select>`
for (var i = 0; i < options.length; i++) {
var option = document.createElement("option");
option.value = options[i];
option.text = options[i];
dd.appendChild(option);
}
return dd;
}
)})
}
]
};
const notebook = {
id: "bd35dbda7bea48d1@1469",
modules: [m0,m1,m2]
};
export default notebook;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment