almccon/README.md Secret

## README.md

      
    Raw
  

              README.md
            
          
American Panorama: A New Atlas of American History
NACIS2015 October 15, 2015
Alan McConchie // @mappingmashups // Stamen Design // @stamen
Slides: http://sta.mn/f54  // Notes (this document): http://sta.mn/6gk
Full URL: https://speakerdeck.com/almccon/nacis2015-a-new-atlas-of-american-history


Atlas of the Historical Geography of the United States
Charles O. Paullin John K. Wright Published 1932
Our story starts with this atlas


This was state-of-the-art at the time.


It contains nearly 700 maps, and took dozens of researchers almost 20 years to research and assemble.


“The ideal historical atlas might well be a collection of motion-picture maps, if these could be displayed on the pages of a book without the paraphernalia of projector, reel and screen.”
– John K. Wright
Paullin and Wright did some groundbreaking things, especially with how they represented time. And of course, when you're mapping history, showing time is just as important as showing space.
In the introduction, Wright imagined an ideal historical atlas, which sounds a lot like a present-day tablet computer.
With hindsight, we might argue that ideally you want more than just motion-pictures, you want the ability to pause, rewind, and explore through time with as much flexibility as you can explore space on a map.


The Digital Scholarship Lab at the University of Richmond is trying to make Paullin and Wright's dream come true, to give the atlas the interactivity and dynamism the subject matter deserves.
Funded by a grant from the Mellon Foundation, they scanned every map from the Paullin atlas and put them online.


http://dsl.richmond.edu/historicalatlas/


http://dsl.richmond.edu/historicalatlas/


These maps add animation and interactivity without changing the look of the original map (with the exception of optional reprojection).
More info about this project: http://www.nytimes.com/2013/12/26/arts/lab-in-richmond-animates-comprehensive-1932-us-atlas.html


American Panorama: An Atlas of United States History
Our goal is to make a historical atlas for contemporary audiences, a Paullin atlas for the 21st century. We want it to be as state-of-the-art for today as Paullin's atlas was in the 1930's.
Despite the historical subject matter, our goal was not to create maps with a faux-historical aesthetic.


In progress: four interactive maps
Our design approach: instead of building a bunch of components, we planned from the beginning to dive into this "the Stamen way"


In progress: four interactive maps
Plus a toolkit for hundreds more...
Through this process, we will create components that have been tested in real-world situations. From these we abstract reusable components that the Digital Scholarship Lab (and anyone else) can use to build more maps.
This is will be a multi-year process, so we have to strike a balance between using the latest tools, but also making sure this code will be reusable over fairly long timeframes (at least in internet time!)


Forced Migration of Enslaved People in the United States 1810-1860
The first map we took on.
The data is the most complex with this one, and it uses some new statistical analyses that the Digital Scholarhship Lab has been doing based on census data, that's never been shown before.
I'll spend the most amount of time talking about this first map, and I'll use this one to illustrate the technical infrastructure we're using.


Whiteboarding the forced migration map
Context: transatlantic slave trade banned in the US in 1808. Slave owners in the Eastern part of the South were selling slaves to western states. The slave owners' desire for new markets was a significant driver of western expansion of the US, etc.
There are no good records of sales of enslaved people between states, but the DSL was able to estimate the number of forced migrations in each decade using census data.
This is heavy, heavy stuff to work with... and there's no way to spend the time to dwell on it in a 20 minute presentation in the way it deserves.
But this is an important story to tell, and an important story for us to as Americans to never forget. But we know that these maps are only part of the story.


Animated gif: [http://imgur.com/wJr4zHP]
Timeline
I'm using animated gifs to show the interactivity so I don't have to pause to do a live demo.
This map is pretty much finished. (Meaning there may be more tweaks before launch, but hopefully not many)
You can see that the main theme of the map (the repeating red and gray colors) shows which regions were exporting and importing slaves.
At the upper left is the map, at the upper right is a diverging bar chart listing states, and at the lower right is a bubble chart showing counties.


Animated gif: [http://imgur.com/ovthNPg]
Crops


Animated gif: [http://imgur.com/Cuz8CuQ]
County selection
Note how all the views are linked, so you can select a state or county in one pane of the visualization, and see that entity selected in the other views. And we can follow a county through time.


Animated gif: [http://imgur.com/5VG8cAV]
Narratives:
This is a chilling topic, and the map is very data-heavy. We risk losing sight of the individual human tragedy. So the visualization also includes qualitative data as told by people who experienced these forced migrations.


Animated gif: [http://imgur.com/bRRGDJF]
Walkthrough:
A walkthrough introduces the user to the various parts of the map, and explains some of the history during the process. At any point the user can drop out of the walkthrough and start exploring, and they can come back to it whenever they want.


Animated gif: [http://imgur.com/K5m2mUU]
Hash:
Notice the "stateful hash": The URL changes as you pan around and interact with the map. This means you can copy and save that link, and always come back to the view as you left it. Even more importantly, you can send that link to someone else and they'll see the map exactly as you were looking at it.
This is important, because we expect this map to be used in educational settings (college undergraduate, maybe advanced high schoolers) where the class might only focus on a specific state, or decade.


Architecture diagram:
The data lives in CartoDB, and we use it for some of the mapping functionality, but not all.
The interactivity happens in javascript, which pulls data from the CartoDB API.
We also have some tables synced from Google Sheets. For example, the DSL had a bunch of research assistants transcribing journal entries into spreadsheets without needing to interact with CartoDB itself.


We use a custom basemap in a non-Mercator map projection.
We need a custom basemap anyway because we can't use any off-the-shelf base tiles which would show present day features like freeways or cities like Las Vegas which didn't exist in the 19th Century.
This is Natural Earth data, uploaded to CartoDB, and styled using CartoCSS. So this is a tiled map.


EPSG: 2163 US National Atlas Equal-Area (Lambert Azimuthal)
And because all the maps we're building (at least the first ones) only show the lower 48 states, there's no reason to use Mercator. Specifically, we chose the US National Atlas Equal-Area projection (which is technically a specific kind of Lambert Azimuthal.


And we generate these tiles from CartoDB! How do we get CartoDB to understand non-Mercator projections? We use the power of PostGIS, which is the database that CartoDB uses under the hood. In the CartoDB SQL window, you can transform your layer's geometry into any projection you like.
SELECT cartodb_id, ST_Transform(the_geom_webmercator, 2163) AS the_geom_webmercator, ...


But what about that terrain?


OpenTerrain tiles
Our leaflet code requests the basemap tiles from CartoDB, but we also add transparent terrain overlay tiles from our own OpenTerrain servers.
OpenTerrain is also very much in progress. Note that the terrain still has a bit of a "sandpaper" effect at this zoom level. We will have a more downsampled terrain layer soon.
Note: I made the transparent parts of the terrain tile look gray in this slide.
More about the OpenTerrain project can be found at openterrain.tumblr.com


This is a choropleth of forced migrations per square miles, mapped using 1850 county boundaries.
If you tried to animate this, you'd notice that the county boundaries are constantly changing. We used the NHGIS historical boundary files (nhgis.org) for
this. But we needed a better way to show change over time.


So instead we convert the data into a standard spatial unit, like hexagons. Here we intersect a grid of hexagons with the county data.


The resulting hexagons.
We decided this size of hexagon was too large, so we used a smaller hexagon size for the final map.


The final size of the hexagons we used.
Here the hexagons are colored according to the number of counties they intersect. You can see that most hexagons are a light color, meaning they only intersect one or two counties.


Animated gif: [http://imgur.com/PBERDbO]
Here's what those hexagons look like if we take their centroid and draw proportional symbols instead.
This is one of Kai Chang's prototype animations, which differs a bit from the final product.


Also notice that the hexagon grid was in our National Atlas projection too. Here's what the original hexagons looked like if you view them in Mercator. This is before I intersected them with the counties. I generated enough of them to cover the bounding box of the US in the National Atlas projection. https://www.flickr.com/photos/almccon/16260286439/


Animated gif: [http://imgur.com/wJr4zHP] (same link as before)
Back to the bubble plot. It's perhaps the most difficult to wrap your head around, but it also shows this phenomenon the best: how counties grow in enslaved population, and eventually shift from inmigration to outmigrations.


Animated gif: [http://imgur.com/LPcDizk]
Here's one of Kai Chang's prototypes, using slightly different scales on the axes. Note that it helped us debug an error in one county's 1850 data: one county shoots up to the upper right, then drops down again.


Hans Rosling: gapminder.org
Of course our inspiration for the animated bubble charts came from Hans Rosling's gapminder.org


Overland Trails 1840-1860
Our second map was about the overland trails. Note that all the first four maps have something to do with migration or transportation, but eventually the atlas will include maps with all kinds of themes.
For the remaining maps I won't spend as much time on them.


Whiteboarding overland trails
We reused some of the layout from the first map: A map at the upper left, a timeline chart at the bottom, and supplemental charts and narratives as the right. We're developing a template for these maps, but it's going to be very flexible. We make sure to let the needs of each map drive the design.


Animated gif: [http://imgur.com/cN1l2xE]
Timeline. The map only shows a small number of paths where we have fully georeferenced diaries. We have aggregate numbers of migrants per year at the lower left.
Sorry, some of these animated GIFs have compression artifacts, where some of the colors get washed out.


Animated gif: [http://imgur.com/v6KjPRy]
Toggles to focus on just the Oregon, California, or Mormon trails


Animated gif: [http://imgur.com/nuBD48S]
Narratives: Instead of a single point for each narrative (as in the last map), we have coordinates for each diary entry, for dozens of diarists


E J Marey Paris to Lyon train schedule (1885)
Our inspiration for the chart in the timeline is Marey's diagram of train schedules.
As the overland trails became well-traveled, we should see the later travelers moving faster than earlier ones. So lines at the right side of the timeline should be steeper than at the left. (it's subtle)


Foreign-born Population 1850-2010
Map #3: Foreign-born population


Whiteboarding the foreign born map
At first we thought of doing a dorling cartogram


New York Times Map of Olympic Medals
Inspiration: New York Times Dorling cartogram http://www.nytimes.com/interactive/2008/08/04/sports/olympics/20080804_MEDALCOUNT_MAP.html
However, we found that the jostling of the countries between years was too distracting, so we used overlapping proportional symbols


Moritz Stefaner Map Your Moves
Inspiration: http://moritz.stefaner.eu/projects/map%20your%20moves/
We especially like how the region names are just there for context, but that the dots don't need to be geographically accurate


Andy Woodruff et al. Value-by-alpha map
Inspiration: http://andywoodruff.com/blog/value-by-alpha-maps/


Animated gif: [http://imgur.com/GThJ4UM]
Timeline
The map layout uses D3: each continent has its own projection. The continents don't change size; they're there only for orientation and context. Their sizes relative to each other are adjusted for aesthetic considerations and placement.
Note the legend for the proportional circles, and then wait until the end of the animation. The numbers for Mexico in 2000 and 2010 are so large that they'd make all the other circles too small to see... so we let Mexico go well beyond the largest legend circle.
Trivia: can you spot anything unusual going on with this map? Hawaii is duplicated, because at first it's a foreign country, and later it's a territory, then a state.


Animated gif: [http://imgur.com/n97iDaP]
Selecting countries highlights their circle, and changes the value-by-alpha map (notice how the legend changes too). The timeline chart becomes a dual- axis chart, showing the rise and fall of immigration from the selected country.


Animated gif: [http://imgur.com/SplsgRH]
County selection. Note that county-level immigration data by country is not available for 1850 and 1860.


Canals 1820-1890
Map #4: Canals


Canals map is still in progress, but here's the latest design, showing a snapshot of real data.
Here our challenge is dealing with the heterogeneity of the data. Some canals have rich data about the types of commodities they carried, and the volume of each kind each year. Other canals don't have this at all, or only for some years. Sometimes the categories and units of measurement change.
And for all the canals, we can't fully trust the data. The source material is the annual reports of the canal companies, and there no way to know how much exaggeration and outright fabrication they contain.
More than any of the other maps, this map will need to communicate this "meta-narrative" about the difficulty of data collection and the unknowability of the past.


Two weeks ago, we started the process of moving components out into a public repo. You are welcome to follow along and participate! The maps themselves will launch in the next few months.
http://github.com/americanpanorama


Thank you!
Alan McConchie // @mappingmashups // Stamen Design // @stamen Digital Scholarship Lab // dsl.richmond.edu
Presentation slides // sta.mn/f54
Thanks to everyone at University of Richmond Digital Scholarship Lab, Rob Nelson, Nate Ayers, Justin Madron, and Ed Ayers. They've been a pleasure to work with!
And thanks to the Mellon foundation for supporting this work!


Oregon Trail: You made it!