###IPAM - UCLA
Aurelia Moser, @auremoser, aurelia@mozillafoundation.org
May 30th, 2016
Instagram @chazhutton
Find this document here:
- Script: http://tinyurl.com/ipam-16-maps
- Github: https://github.com/auremoser/ipam-16
- Slides: <http://tinyurl.com/ipam-16-slides >
This workshop will be a light introduction to mapping with open source tools, featuring Open Street Map Data, CartoDB, Leaflet and other libraries for storing, analyzing and visualizing geospatial data.
- Mapping Basics
- You are Here.
- Types + Topics
- Anatomy of a Webmap
- Mapping Data
- Formats + Free Sources
- Collecting, Sync, Storage
- Mapping Tools
- Toolbox: Google, Mapbox, CartoDB, Leaflet
- Mapping in a GUI
- Data Import
- Customizing UI + CartoCSS
- Querying in SQL
- Customizing Basemaps
- Mapping in Code
- APIs / JS Libs
- Python
- R
- Mapping Movement
- Tell Time/Stories: Odyssey + Torque
- Partner Graphics: Graphs + Charts
- Building a Narrative
- Part 0: Geology Data: OSM
- Part 1: Earthquake Data
- Part 2: Elephant Migration Data
- Sharing your Map
- Seeking inspiration
- Resources
YOU ARE HERE.
Where are you? If I gave you directions here, you could say you were at:
460 Portola Plaza, Los Angeles, CA 90095, USA
But if I wanted to describe the location as a point on a map, I might describe the same location using latitude and longitude:
34° 4' 12.9288'' N 118° 26' 30.6636'' W
This reads as "34 degrees, 4 minutes and 12.93 seconds North, 118 degrees, 26 minutes and 30.66 seconds West."
We describe locations with a spatiotemporal notation, it's also somewhat obscure. So let's describe the same place using longitude and longitude but using decimal degrees instead of minutes and seconds. There are a number of conversion tools available online to do this:
34.070258, -118.441851
This workshop's content was subtly inspired by the "ring of fire" an area skirting the Pacific Ocean that is associated with an almost continuous series of oceanic trenches, volcanic arcs, and volcanic belts and/or plate movements. We'll be mapping geological features and some random animal migration data.
You can read more about this topic in the New Yorker article, The Really Big One: an earthquake will destroy a sizable portion of the coastal Northwest. The question is when..
All across the region, seismologists are looking at their watches, wondering how long we have, and what we will do, before geological time catches up to our own.
####Types + Topics
There are many different types of map you can make, browse this gallery to view some options. Most maps illustrate some kind of geospatial change.
- Simple -- most basic visualization
- Cluster -- counts number of points within a certain binned region
- Choropleth -- makes a histogram of your data and gives bins different colors depending on the color ramp chosen
- Category -- color data based on unique category (works best for a handful of unique types)
- Bubble -- size markers based on column values
- Intensity -- colors by density
- Density -- data aggregated by number of points within a hexagon
- Torque -- temporal visualization of data
- Heat -- more fluid map of concentration; emphasis on far over near-view
DYNAMIC MAPS
Source: The Data Visualization Catalogue.
Source: Time Travel Between Counties, Carto.JS
DATA SEARCH TOOLS
Source: GDELT Geographic News Search Tool
Journalists have used GDELT data to track wildlife crime, and the spread of the Islamic State in the Middle East among other things.
You can fork the GDELT hourly synced data set from the CartoDB DataLibrary and add it as a layer on your map or use the Geographic Search Tool linked above to search for tags of interest.
CHART GRAPHICS
Source: Geogia County Car Crash Counts, C3.JS
COMPARISON NARRATIVE
Map population by relative density
- Maps give you more context than most visualizations.
- They allow you to apply data to a recognizable topography.
Sources
- CartoDB: light open source library and graphical user interface application for hosting and visualizing geospatial data
- ChartJS: light library for creating charts and graphs
- GDELT: the global database of events languages and tones
Source: Open Street Map
Maps are composed of:
####1. Tiles (often)
Tiles are 256x256 pixels squares that plot one next to the other to stitch together a pseudo-seamless image; the tiles draw dynamically as you focus and zoom in on your map to progressively build you a seemingly continuous canvas.
####2. Feature Layers
Data layers that populate on top of your basemap.
####3. Javascript/HTML/CSS for rendering on the web
With these languages, you can publish your map with the base tiles loaded and your data layers appropriately geocoded; with Javascript you can also add to the interactivity of your map, revealing metadata in the tooltips, for example.
We'll be working with an assortment of vector point/line/polygon data:
- POLYGONS: California Counties
- POINTS: Earthquake Centroids
- LINES: Elephant Migration Patterns
Here are additional geo-spatial datasets you might want to play with:
- Columbia University GeoSpatial Datasets
- World GeoSpatial Datasets: countries, cities, codes, flags, languages, latitude/longitude, etc.
- Center for National Geographic Information
- IDEE Metadata Catalog: spatial data from multiple agencies
- Geogig: distributed geospatial data
- International GIS Data - Penn State
We'll be collecting data from multiple sources. Most of which are available in the repository where our data will be stored.
OUTCROPS IN OSM
Outcrops are places where the bedrock or superficial deposits have become locally exposed and are directly accessible to analysis in OSM; let's map them!
- OSM: Open StreetMap + Overpass Turbo
You can read more about Overpass on the Open Street Map Wiki.
Steps:
- Pan manually to an area in OverPass Turbo
- Go to the "Wizard"
- Look up the appropriate OSM tag in the OSM wiki
- Search for "geological=outcrop" or whatever
- Export your data as GeoJson or KML
- Upload into CartoDB or another interface
Notes:
- Only nodes, only ways, only relations can save you processing time/power (-> limit to only map what you want)
- Searching for a large OSM dataset can freeze Overpass Turbo (-> define your bounding box)
There are lots of geospatial data formats that may or may not be easy to parse. Here's a short list of ones that are commonly imported into CartoDB so you can get familiar with a few you might encounter.
So let's take a look at this outcrop data. You want to lead people to all of the outcrop stations globally. You can use the kml file here in the data folder.
You can get the libraries from OSM or NYC Open Data but let's go with the KML from OSM that we extracted with OverPass.
.kml which is a notation for XML that first made popular by Google Earth before becoming standardized. Google Mapping Products happily use kml files. Check to see how the locations are described:
<?xml version="1.0" encoding="UTF-8"?><kml xmlns="http://www.opengis.net/kml/2.2"><Document><name>overpass-turbo.eu export</name><description>Filtered OSM data converted to KML by overpass turbo.
Copyright: The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.
Timestamp: 2015-09-14T14:05:02Z</description><Placemark><name>Sonnberg</name><Point><coordinates>9.43865,50.6228007</coordinates></Point><ExtendedData><Data name="@id"><value>node/428215780</value></Data><Data name="geological"><value>outcrop</value></Data><Data name="name"><value>Sonnberg</value></Data><Data name="note"><value>Rocky promontory, subject to erosion. Middle Triassic shallow marine limestone, Jena Formation (U.-Muschelkalk, Wellenkalk). Small outcrop within E-W trending graben structure. Contact to MIttlerer Buntsandstein exposed in river bed southwest of crag.</value></Data></ExtendedData></Placemark><Placemark><name>Monkey rock</name><Point><coordinates>177.127952,-17.7323069</coordinates></Point>
...Notice that the point coordinates are in the format <coordinates>longitude, latitude</coordinates>. Some geoformats are lat, long and others are long, lat. This makes everyone sad.
When coordinates cannot easily be parsed they go to "null island," a fictional place at coordinates (0,0).
In a bit, we'll show how you can convert, process and use a file like this to plot data on your map.
CartoDB is a Postgres database in the cloud, which means it handles a lot of your backend data needs and allows you to query for data and pull those data and basemap tiles into your front-end code, we'll be using this for part of the workshop to manage hosting easily. It's appropriate to deal with databases when you have multiple datasets you'd like to layer on your map. For single or simple datasets, you can load a file with JQuery/Leaflet. We'll demo both.
You can also store your data in Github, or in another service that makes it web accessible. Read more about that here and here.
There are loads of ways to approach a map here are a few approaches to mapping the same dataset.
We can make a map with the outcrop data, or other data in the data folder for this workshop.
Using Google Maps
- https://www.google.com/maps/d/
- create account if you don't already have a Gmail account
- click on Import Map in top left hand menu (or My Maps -> Create map in some Google Maps UIs)
- upload
outcrop.kml - explore changing the map features if you would like
Using Mapbox
- https://mapbox.com/
- create account if you don't already have a Mapbox account
- click on the Data tab at the top right hand corner of the screen
- click on import
- upload
outcrop.kml - select map features if you would like then click on Import Features
- explore changing the map features if you would like
Using CartoDB
- https://cartodb.com/
- create account if you don't already have a CartoDB account, use this URL to get boosted features
- click on Create Map; select Map View at the top of the screen
- click on the '+' or Add Layer option at the top of the right side menu
- upload
outcrop.kml - explore changing the map features, if you like
Using Leaflet
You can also make a map from scratch using Leaflet.js to attach a set of points to a map made of tiles provided by OpenStreetMap.
You will first need to convert your kml file into GeoJSON (although I have both in the data folder) for this workshop.
GeoJSON is a file format that is easily digestable by JavaScript. If you have a data format (shp, kml) that is not geojson you can convert it to the right format for your code with GeoJSON.io/
- go to http://geojson.io/
- from the menu Open select File and upload our kml file: https://github.com/copystar/foss4g-cdb/blob/master/outcrop.kml
- notice how GeoJSON looks like in the side-menu
{
"type": "FeatureCollection",
"generator": "overpass-turbo",
"copyright": "The data included in this document is from www. openstreetmap.org. The data is made available under ODbL.",
"timestamp": "2015-09-14T14:05:02Z",
"features": [
{
"type": "Feature",
"id": "node/428215780",
"properties": {
"@id": "node/428215780",
"geological": "outcrop",
"name": "Sonnberg",
"note": "Rocky promontory, subject to erosion. Middle Triassic shallow marine limestone, Jena Formation (U.- Muschelkalk, Wellenkalk). Small outcrop within E-W trending graben structure. Contact to MIttlerer Buntsandstein exposed in river bed southwest of crag."
},
"geometry": {
"type": "Point",
"coordinates": [
9.43865,
50.6228007
]
}
},
{
"type": "Feature",
"id": "node/568331113",
"properties": {
"@id": "node/568331113",
"geological": "outcrop",
"name": "Monkey rock"
},
"geometry": {
"type": "Point",
"coordinates": [
177.127952,
-17.7323069
]
}
},
- after the map is drawn, from the menu Save, select GeoJSON"
- refer to "Adding GeoJSON to Leaflet with Link Relations" : http://lyzidiamond.com/posts/osgeo-august-meeting to find the HTML that use can use as a template that will import GeoJSON into a map created by Leaflet.js
- use http://{s}.tile.osm.org/{z}/{x}/{y}.png for your map tiles
- use
outcrop.geojsonfor your geojson layer: https://github.com/auremoser/ipam-16/blob/master/data/outcrop.geojson - explore changing the map features if you would like using Leaflet.js: http://leafletjs.com/
Quick outcrop map by type.
- EuroBird Portal
- Alcatraz Escape Revisited
- LA Sheriff Election Results
- Starwars Galaxy Map
- Demonstrations in Brazil
- Global Forest Watch
- Urban Reviewer
CartoDB is a Postgres database in the cloud, which means it handles a lot of your backend data needs and allows you to query for data and pull those data and basemap tiles into your front-end code; we'll be using this for the next half of the workshop.
#####You can sign-up for a CartoDB Academy Account via this URL.
The GUI interface for CartoDB looks as follows:
You have myriad customization options in the in-browser editor:
sql- run sql and postgis functions across your datawizard- adjust the type, colors and fills in your mapinfowindow- create hovers, tooltips with information from your datatablescss- customize the css and style of your map outside the wizardlegends- create keys for your mapfilters- filter the data without sql
DATA IMPORT
Let's say I import the 50 state polygons from the CartoDB Data Library. I navigate to Datasets > Data Library and search for states:
It gives me a table I can fork called ne_50m_admin_1_states.
CUSTOMIZING UI + CARTOCSS
You can customize how those polygons appear with CartoCSS, a CSS flavored language for styling cartographic projects.
There are also a series of presets in the Wizard that write your css for you:
- Marker Fill: change the size, color, and opacity of all markers
- Marker Stroke: change the width, color, and opacity of every marker's border
- Composite Operation: change the color of markers when they overlap
- Label Text: Text appearing by a marker (can be from columns)
QUERYING IN SQL
Say you want to move Alaska and Hawaii into view because the typical Mercator Projection just isn't cutting it.
CartoDB supports PostGIS and SQL functional manipulation of your data in the cloud.
CUSTOMIZING BASEMAPS
For the above, I made my basemap background one color (#90cccb), and added country polygons from the dataset in our common Data Library.
Our Positron and Dark Matter basemaps are available for free in Leaflet and OpenLayers maps, as well as core CartoDB maps. Read more about that here, and check out documentation on the available CartoDB Basemaps can be found here.
JS
You can read more about the CartoDB APIs and JS Library here
- CartoJS - JS library for interacting with CartoDB
- Maps API - generate public/private maps with data hosted on your CDB account
- SQL API - run sql in your code to dynamically filter/request/query your mapped data stored in CartoDB via http calls
- Import API - push data to your CartoDB Account
Use also the CartoDB Uploader package available in NPM for working in Node.js
PYTHON
Use the CartoDB Python module to create and update tables; check out Andy's Jupyter demos.
R
Use Kyle Walker's R2CartoDB project to create and update tables in R.
MAPS THAT TELL TIME - Torque
Source: Traffic Change Data for SF Demo: Municipal Traffic in SF App Blog: CartoDB + Firebase
- Demonstrations in Brazil
- Animal migration patterns
- Diwali Celebrated
- Ramadan Tweets w/OdysseyJS
- Alcatraz Escapees
- Lynching and the Press
Election response in Argentina post-primary election.
MAPS THAT TELL STORIES - Odyssey JS
You can use CartoDB's SQL API to query your data and pull it into any charting library of your choosing.
You can easily wire up a chart of earthquake depth data for example, set it as a sync table to update continuously, check the code here.
Learn more about it here!
Here are some examples:
| Type | Title | Link/Demo | BlogPost |
|---|---|---|---|
| NVD3 | NYC Elementary Schools | Chris' Github | |
| Chart.js Bar Graph | Traffic Data | Aurelia's Block | |
| Highcharts | Sensor Data | Github / Demo | MOW Post |
| Highcharts | Weather Data | Aurelia's Block | Tutorial |
| Chart.js Line Graph | Tornado Data | Andrew's Block | |
| Plot.ly | Earthquake Data | Plotly Tutorial | CartoDB Blog |
Part 1: Earthquake Mapping Data
We're going to visualize earthquake data to explore some of the mapping options available to us with a variety of data types.
Copy the link to the data here (don't download):
http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csvImport it into your account like this:
EXPLORE
We wrote some algorithms to analyze the data you import to suggest maps to make. When you choose one of these, it updates the wizard with all of the styling information.
MAKE A MULTILAYER MAP
Now that we've made a map with our earthquakes, let's put a layer of earthquake faults below as a reference.
The data comes from here, and they made a Leaflet.js map with the dataset here.
Copy the following link and add it as a layer to your map:
http://earthquake.usgs.gov/hazards/qfaults/qfaults.zipLooking at the metadata, we could visualize the layers using the slip code attribute that shows the age of the faults, like they do in the map above.
SPATIAL PROCESSING + SQL
CartoDB is a tool for geospatial data manipulation and visualization.
SQL is a language for data manipulation. PostGIS is an extension to SQL that has the power to manipulate geospatial data.
The most basic statement is:
SELECT * FROM table_nameA more detailed query is like this:
SELECT
name,
height,
age
FROM
class_list
WHERE
name IN ('Aurelia', 'Jorgito')
OR (
height < 1.8
AND
height > 1.6
)SELECTis what you're requesting (required)FROMis where the data is located (optional -- you can make a table on the fly)WHEREis the filter on the data you're requesting (optional)
You can optionally add LIMIT n (where n is an integer >= 0), which gives you only n entries, and ORDER BY column_name ASC, which sorts in ascending order (DESC is another option).
Using the earthquakes data, you can combine them to give you the top 20 largest earthquakes ordered by strength (mag) and, for equal strength, by time.
SELECT
place,
mag,
time,
the_geom_webmercator
FROM
all_month
ORDER BY
mag DESC, time DESC
LIMIT
20
There are two special columns in CartoDB:
the_geomthe_geom_webmercator
The first of these is in the units of standard latitude/longitude, while the second is a projection based on the original Mercator projection but optimized for the web.
If you want to run on-the-fly SQL commands and see your map update, make sure to SELECT the the_geom_webmercator because this is the column that's used for mapping--the other is more of a convenience column since most datasets use lat/long.
If you want to enable interaction on your maps (click events, hover boxes, etc.), you also need to SELECT the column called cartodb_id.
For instance, the following statement will produce a map with click features:
SELECT
the_geom_webmercator,
place,
mag,
cartodb_id
FROM
all_monthSeveral are aggregate functions, meaning that they need to be grouped by a certain column.
Find the average of the earthquake magnitude as reported by various stations:
SELECT
avg(mag) AS avg_mag,
ST_Transform(ST_Centroid(ST_Union(the_geom)),3857) As the_geom_webmercator,
net
FROM
all_month
GROUP BY
netBuild a tool to execute SQL and CSS instructions on button click, like this one for earthquake data.
Check out the Map Academy: Lesson 3 to learn how to make this selector map!
Part 2: Elephant Tracking Data
Movebank.org has a lots animal tracking data. To get our data, I went to Tracking Data Map, then searched for "Loxodonta africana". But you can easily import it into your account by copying the link below.
We'll be making this map:
- Create a new Map
- Import the data into your account from this link:
http://aureliamoser.cartodb.com/api/v2/sql?q=SELECT%20*%20FROM%20elephant_movements&format=geojson&filename=elephant_movementsWe can see our elephant move around with a Torque visualization.
Let's visualize roughly the paths that the elephants take from point to point. These will be best-guess lines as we don't know the intermediate points. They represent linear interpolations between consecutive points.
We can easily do this with some SQL and PostGIS by pasting in the following command and running it:
SELECT
ST_Transform(
ST_MakeLine(
the_geom ORDER BY timestamp),
3857
) As the_geom_webmercator
FROM
elephant_movementsand apply the following CartoCSS to visualize our lines better:
#elephant_movements{
line-color: #41c491;
line-width: 1;
line-opacity: 0.7;
line-smooth: 1.4;
line-clip: true;
line-dasharray: 2, 3, 2;
line-comp-op: multiply;
}
We can get a more nuanced view of the data by using composite operations
SELECT
ST_Transform(
ST_MakeLine(
a.the_geom,
b.the_geom)
,3857
) As the_geom_webmercator,
to_char(a.timestamp,'HH12:MI AM, DD Mon') || '--' || to_char(b.timestamp,'HH12:MI AM, DD Mon') As date_range,
a.cartodb_id
FROM
elephant_movements a
JOIN
elephant_movements b
ON
a.cartodb_id + 1 = b.cartodb_idAnd apply the following CartoCSS to show intensity when lines cross:
#elephant_movements{
line-color: #41c491;
line-width: 2;
line-opacity: 0.7;
line-comp-op: multiply;
line-clip: true;
line-smooth: 1.4;
}
You can publish your map as is via the "Share" button in the interface, but if you would like to build a more custom version you can use the JS libraries and APIs mentioned above, and use CartoDB as your Data Store.
In the workshop repo, you'll find a template for making your map as a site online:
- index.html: setting up your map div and canvas
- main.js: pulling in your basemap, cartodb feature layers and other details
If you run this locally, you'll get a map that looks like this:
We worked on a little tool to study your dataset in CartoDB and suggest "interesting maps" to you in the GUI. It'll be a little modal in the lower left corner and will produce demo maps with pre-fab SQL and CSS like this:
###RESOURCES
- Map Academy
- CartoDB.js -- build a web app to visualize your data, allowing for user interaction
- SQL and PostGIS
- CartoDB Tutorials
- CartoDB Trainings
- CartoDB Editor Documentation
- CartoDB APIs
- Community help on StackExchange
- CartoDB Map Gallery
- CartoDB Bootstrap Template by Chris Wong
- Friendly Geo Intro by Joey Lee
- Lyzi Diamond tutorial on Leaflet
- World GeoSpatial Datasets: countries, cities, codes, flags, languages, latitude/longitude, etc.
- Center for National Geographic Information
- IDEE Metadata Catalog: spatial data from multiple agencies
- Geogig: distributed geospatial data
- International GIS Data - Penn State
- Charting Tools Repository
- Recommended tools for Visualizations
- Perception Concerns
- Gestalt Theory
- Color Brewer or Geocolor
If you want to chat with me, or send me some cool maps you've made, just email aurelia@mozillafoundation.org.
