Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?

Building Dynamic Maps with Open Source Tools

###IPAM - UCLA

Aurelia Moser, @auremoser, aurelia@mozillafoundation.org

May 30th, 2016

EveryCity

Instagram @chazhutton

Find this document here:

This workshop will be a light introduction to mapping with open source tools, featuring Open Street Map Data, CartoDB, Leaflet and other libraries for storing, analyzing and visualizing geospatial data.

Outline

  1. Mapping Basics
    • You are Here.
    • Types + Topics
    • Anatomy of a Webmap
  2. Mapping Data
    • Formats + Free Sources
    • Collecting, Sync, Storage
  3. Mapping Tools
    • Toolbox: Google, Mapbox, CartoDB, Leaflet
    • Mapping in a GUI
      • Data Import
      • Customizing UI + CartoCSS
      • Querying in SQL
      • Customizing Basemaps
    • Mapping in Code
      • APIs / JS Libs
      • Python
      • R
    • Mapping Movement
      • Tell Time/Stories: Odyssey + Torque
      • Partner Graphics: Graphs + Charts
  4. Building a Narrative
  5. Resources

Mapping Basics

YOU ARE HERE.

LA

Where are you? If I gave you directions here, you could say you were at:

460 Portola Plaza, Los Angeles, CA 90095, USA

But if I wanted to describe the location as a point on a map, I might describe the same location using latitude and longitude:

34° 4' 12.9288'' N 118° 26' 30.6636'' W

This reads as "34 degrees, 4 minutes and 12.93 seconds North, 118 degrees, 26 minutes and 30.66 seconds West."

We describe locations with a spatiotemporal notation, it's also somewhat obscure. So let's describe the same place using longitude and longitude but using decimal degrees instead of minutes and seconds. There are a number of conversion tools available online to do this:

34.070258, -118.441851

This workshop's content was subtly inspired by the "ring of fire" an area skirting the Pacific Ocean that is associated with an almost continuous series of oceanic trenches, volcanic arcs, and volcanic belts and/or plate movements. We'll be mapping geological features and some random animal migration data.

You can read more about this topic in the New Yorker article, The Really Big One: an earthquake will destroy a sizable portion of the coastal Northwest. The question is when..

All across the region, seismologists are looking at their watches, wondering how long we have, and what we will do, before geological time catches up to our own.

####Types + Topics

Gallery of maps

There are many different types of map you can make, browse this gallery to view some options. Most maps illustrate some kind of geospatial change.

  • Simple -- most basic visualization
  • Cluster -- counts number of points within a certain binned region
  • Choropleth -- makes a histogram of your data and gives bins different colors depending on the color ramp chosen
  • Category -- color data based on unique category (works best for a handful of unique types)
  • Bubble -- size markers based on column values
  • Intensity -- colors by density
  • Density -- data aggregated by number of points within a hexagon
  • Torque -- temporal visualization of data
  • Heat -- more fluid map of concentration; emphasis on far over near-view

DYNAMIC MAPS

Types of Visualizations

Source: The Data Visualization Catalogue.

Time Travel Map

Source: Time Travel Between Counties, Carto.JS

DATA SEARCH TOOLS GDELT Geographic News Search Tool

Source: GDELT Geographic News Search Tool

Journalists have used GDELT data to track wildlife crime, and the spread of the Islamic State in the Middle East among other things.

You can fork the GDELT hourly synced data set from the CartoDB DataLibrary and add it as a layer on your map or use the Geographic Search Tool linked above to search for tags of interest.

CHART GRAPHICS

County Chart

Source: Geogia County Car Crash Counts, C3.JS

COMPARISON NARRATIVE

Michael's Syrian Refugee visualization Map population by relative density

  • Maps give you more context than most visualizations.
  • They allow you to apply data to a recognizable topography.

Sources

  • CartoDB: light open source library and graphical user interface application for hosting and visualizing geospatial data
  • ChartJS: light library for creating charts and graphs
  • GDELT: the global database of events languages and tones

Anatomy of a WebMap

Webmap Gif

Source: Open Street Map

Maps are composed of:

####1. Tiles (often)

Stamen Tiles

Tiles are 256x256 pixels squares that plot one next to the other to stitch together a pseudo-seamless image; the tiles draw dynamically as you focus and zoom in on your map to progressively build you a seemingly continuous canvas.

####2. Feature Layers

Data layers that populate on top of your basemap.

features

####3. Javascript/HTML/CSS for rendering on the web

With these languages, you can publish your map with the base tiles loaded and your data layers appropriately geocoded; with Javascript you can also add to the interactivity of your map, revealing metadata in the tooltips, for example.

Mapping Data

We'll be working with an assortment of vector point/line/polygon data:

cali-data

Here are additional geo-spatial datasets you might want to play with:

Collecting + Sync Tables

We'll be collecting data from multiple sources. Most of which are available in the repository where our data will be stored.

OUTCROPS IN OSM

Outcrops are places where the bedrock or superficial deposits have become locally exposed and are directly accessible to analysis in OSM; let's map them!

OSM Geo

Overpass

You can read more about Overpass on the Open Street Map Wiki.

Steps:

  • Pan manually to an area in OverPass Turbo
  • Go to the "Wizard"
  • Look up the appropriate OSM tag in the OSM wiki
  • Search for "geological=outcrop" or whatever
  • Export your data as GeoJson or KML
  • Upload into CartoDB or another interface

outcrop search

Notes:

  • Only nodes, only ways, only relations can save you processing time/power (-> limit to only map what you want)
  • Searching for a large OSM dataset can freeze Overpass Turbo (-> define your bounding box)

Turbo Output Interface

There are lots of geospatial data formats that may or may not be easy to parse. Here's a short list of ones that are commonly imported into CartoDB so you can get familiar with a few you might encounter.

Part 0: Outcrops

Outcrop Data

So let's take a look at this outcrop data. You want to lead people to all of the outcrop stations globally. You can use the kml file here in the data folder.

You can get the libraries from OSM or NYC Open Data but let's go with the KML from OSM that we extracted with OverPass.

.kml which is a notation for XML that first made popular by Google Earth before becoming standardized. Google Mapping Products happily use kml files. Check to see how the locations are described:

<?xml version="1.0" encoding="UTF-8"?><kml xmlns="http://www.opengis.net/kml/2.2"><Document><name>overpass-turbo.eu export</name><description>Filtered OSM data converted to KML by overpass turbo.
Copyright: The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.
Timestamp: 2015-09-14T14:05:02Z</description><Placemark><name>Sonnberg</name><Point><coordinates>9.43865,50.6228007</coordinates></Point><ExtendedData><Data name="@id"><value>node/428215780</value></Data><Data name="geological"><value>outcrop</value></Data><Data name="name"><value>Sonnberg</value></Data><Data name="note"><value>Rocky promontory, subject to erosion. Middle Triassic shallow marine limestone, Jena Formation (U.-Muschelkalk, Wellenkalk). Small outcrop within E-W trending graben structure. Contact to MIttlerer Buntsandstein exposed in river bed southwest of crag.</value></Data></ExtendedData></Placemark><Placemark><name>Monkey rock</name><Point><coordinates>177.127952,-17.7323069</coordinates></Point>
...

Notice that the point coordinates are in the format <coordinates>longitude, latitude</coordinates>. Some geoformats are lat, long and others are long, lat. This makes everyone sad.

When coordinates cannot easily be parsed they go to "null island," a fictional place at coordinates (0,0).

In a bit, we'll show how you can convert, process and use a file like this to plot data on your map.

Storing

CartoDB is a Postgres database in the cloud, which means it handles a lot of your backend data needs and allows you to query for data and pull those data and basemap tiles into your front-end code, we'll be using this for part of the workshop to manage hosting easily. It's appropriate to deal with databases when you have multiple datasets you'd like to layer on your map. For single or simple datasets, you can load a file with JQuery/Leaflet. We'll demo both.

architecture

You can also store your data in Github, or in another service that makes it web accessible. Read more about that here and here.

github

Mapping Tools

Logos

There are loads of ways to approach a map here are a few approaches to mapping the same dataset.

Map Making Exercise

We can make a map with the outcrop data, or other data in the data folder for this workshop.

Using Google Maps

  • https://www.google.com/maps/d/
  • create account if you don't already have a Gmail account
  • click on Import Map in top left hand menu (or My Maps -> Create map in some Google Maps UIs)
  • upload outcrop.kml
  • explore changing the map features if you would like

Using Mapbox

  • https://mapbox.com/
  • create account if you don't already have a Mapbox account
  • click on the Data tab at the top right hand corner of the screen
  • click on import
  • upload outcrop.kml
  • select map features if you would like then click on Import Features
  • explore changing the map features if you would like

Using CartoDB

  • https://cartodb.com/
  • create account if you don't already have a CartoDB account, use this URL to get boosted features
  • click on Create Map; select Map View at the top of the screen
  • click on the '+' or Add Layer option at the top of the right side menu
  • upload outcrop.kml
  • explore changing the map features, if you like

Using Leaflet

You can also make a map from scratch using Leaflet.js to attach a set of points to a map made of tiles provided by OpenStreetMap.

You will first need to convert your kml file into GeoJSON (although I have both in the data folder) for this workshop.

GeoJSON is a file format that is easily digestable by JavaScript. If you have a data format (shp, kml) that is not geojson you can convert it to the right format for your code with GeoJSON.io/

	{
	  "type": "FeatureCollection",
	  "generator": "overpass-turbo",
	  "copyright": "The data included in this document is from www.	openstreetmap.org. The data is made available under ODbL.",
	  "timestamp": "2015-09-14T14:05:02Z",
	  "features": [
	    {
	      "type": "Feature",
	      "id": "node/428215780",
	      "properties": {
	        "@id": "node/428215780",
	        "geological": "outcrop",
	        "name": "Sonnberg",
	        "note": "Rocky promontory, subject to erosion. Middle 	Triassic shallow marine limestone, Jena Formation (U.-	Muschelkalk, Wellenkalk). Small outcrop within E-W 	trending graben structure. Contact to MIttlerer 	Buntsandstein exposed in river bed southwest of crag."
	      },
	      "geometry": {
	        "type": "Point",
	        "coordinates": [
	          9.43865,
	          50.6228007
	        ]
	      }
	    },
	    {
	      "type": "Feature",
	      "id": "node/568331113",
	      "properties": {
	        "@id": "node/568331113",
	        "geological": "outcrop",
	        "name": "Monkey rock"
	      },
	      "geometry": {
	        "type": "Point",
	        "coordinates": [
	          177.127952,
	          -17.7323069
	        ]
	      }
	    },

outcrop

Quick outcrop map by type.

Map Examples in CartoDB

urban-interface

urban-reviewer

Mapping Tools

CartoDB is a Postgres database in the cloud, which means it handles a lot of your backend data needs and allows you to query for data and pull those data and basemap tiles into your front-end code; we'll be using this for the next half of the workshop.

Basemaps

Mapping in a GUI

#####You can sign-up for a CartoDB Academy Account via this URL.

The GUI interface for CartoDB looks as follows: dashboard data view query view map view css view share

You have myriad customization options in the in-browser editor:

  • sql - run sql and postgis functions across your data
  • wizard - adjust the type, colors and fills in your map
  • infowindow - create hovers, tooltips with information from your datatables
  • css - customize the css and style of your map outside the wizard
  • legends - create keys for your map
  • filters - filter the data without sql

DATA IMPORT

Let's say I import the 50 state polygons from the CartoDB Data Library. I navigate to Datasets > Data Library and search for states:

Data Library

It gives me a table I can fork called ne_50m_admin_1_states.

CUSTOMIZING UI + CARTOCSS

You can customize how those polygons appear with CartoCSS, a CSS flavored language for styling cartographic projects.

There are also a series of presets in the Wizard that write your css for you:

  • Marker Fill: change the size, color, and opacity of all markers
  • Marker Stroke: change the width, color, and opacity of every marker's border
  • Composite Operation: change the color of markers when they overlap
  • Label Text: Text appearing by a marker (can be from columns)

css

QUERYING IN SQL

Say you want to move Alaska and Hawaii into view because the typical Mercator Projection just isn't cutting it.

CartoDB supports PostGIS and SQL functional manipulation of your data in the cloud.

sql

CUSTOMIZING BASEMAPS

For the above, I made my basemap background one color (#90cccb), and added country polygons from the dataset in our common Data Library.

base

Our Positron and Dark Matter basemaps are available for free in Leaflet and OpenLayers maps, as well as core CartoDB maps. Read more about that here, and check out documentation on the available CartoDB Basemaps can be found here.

Mapping in Code

Plotly

JS

You can read more about the CartoDB APIs and JS Library here

  • CartoJS - JS library for interacting with CartoDB
  • Maps API - generate public/private maps with data hosted on your CDB account
  • SQL API - run sql in your code to dynamically filter/request/query your mapped data stored in CartoDB via http calls
  • Import API - push data to your CartoDB Account

Use also the CartoDB Uploader package available in NPM for working in Node.js

PYTHON

Use the CartoDB Python module to create and update tables; check out Andy's Jupyter demos.

R

Use Kyle Walker's R2CartoDB project to create and update tables in R.

Mapping Movement

MAPS THAT TELL TIME - Torque

Realtime Traffic Map

Source: Traffic Change Data for SF Demo: Municipal Traffic in SF App Blog: CartoDB + Firebase

  1. Demonstrations in Brazil
  2. Animal migration patterns
  3. Diwali Celebrated
  4. Ramadan Tweets w/OdysseyJS
  5. Alcatraz Escapees
  6. Lynching and the Press

Elecciones Argentinas

Election response in Argentina post-primary election.

MAPS THAT TELL STORIES - Odyssey JS

  1. Al Jazeera: Israeli-Palestinian Conflict by Tweets
  2. The Sounds of 11M
  3. Berlin Wall Historic Tour
GRAPHS + CHARTS

You can use CartoDB's SQL API to query your data and pull it into any charting library of your choosing.

You can easily wire up a chart of earthquake depth data for example, set it as a sync table to update continuously, check the code here.

chart

Learn more about it here!

Here are some examples:

Type Title Link/Demo BlogPost
NVD3 NYC Elementary Schools Chris' Github
Chart.js Bar Graph Traffic Data Aurelia's Block
Highcharts Sensor Data Github / Demo MOW Post
Highcharts Weather Data Aurelia's Block Tutorial
Chart.js Line Graph Tornado Data Andrew's Block
Plot.ly Earthquake Data Plotly Tutorial CartoDB Blog

Building a Narrative

Part 1: Earthquake Mapping Data

We're going to visualize earthquake data to explore some of the mapping options available to us with a variety of data types.

Earthquakes + Faults

Copy the link to the data here (don't download):

http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv

Import it into your account like this:

add layer

EXPLORE

analyzing data

We wrote some algorithms to analyze the data you import to suggest maps to make. When you choose one of these, it updates the wizard with all of the styling information.

MAKE A MULTILAYER MAP

Now that we've made a map with our earthquakes, let's put a layer of earthquake faults below as a reference.

The data comes from here, and they made a Leaflet.js map with the dataset here.

Copy the following link and add it as a layer to your map:

http://earthquake.usgs.gov/hazards/qfaults/qfaults.zip

Looking at the metadata, we could visualize the layers using the slip code attribute that shows the age of the faults, like they do in the map above.

Multilayer

SPATIAL PROCESSING + SQL

CartoDB is a tool for geospatial data manipulation and visualization.

SQL is a language for data manipulation. PostGIS is an extension to SQL that has the power to manipulate geospatial data.

Basic Syntax

The most basic statement is:

SELECT * FROM table_name

A more detailed query is like this:

SELECT
  name,
  height,
  age
FROM
  class_list
WHERE
  name IN ('Aurelia', 'Jorgito')
  OR (
    height < 1.8
   AND
    height > 1.6
  )
  1. SELECT is what you're requesting (required)
  2. FROM is where the data is located (optional -- you can make a table on the fly)
  3. WHERE is the filter on the data you're requesting (optional)

You can optionally add LIMIT n (where n is an integer >= 0), which gives you only n entries, and ORDER BY column_name ASC, which sorts in ascending order (DESC is another option).

Using the earthquakes data, you can combine them to give you the top 20 largest earthquakes ordered by strength (mag) and, for equal strength, by time.

SELECT
  place,
  mag,
  time,
  the_geom_webmercator
FROM
  all_month
ORDER BY
  mag DESC, time DESC
LIMIT
  20

Sql

the_geom, the_geom_webmercator, and cartodb_id

There are two special columns in CartoDB:

  1. the_geom
  2. the_geom_webmercator

The first of these is in the units of standard latitude/longitude, while the second is a projection based on the original Mercator projection but optimized for the web.

If you want to run on-the-fly SQL commands and see your map update, make sure to SELECT the the_geom_webmercator because this is the column that's used for mapping--the other is more of a convenience column since most datasets use lat/long.

If you want to enable interaction on your maps (click events, hover boxes, etc.), you also need to SELECT the column called cartodb_id.

For instance, the following statement will produce a map with click features:

SELECT
  the_geom_webmercator,
  place,
  mag,
  cartodb_id
FROM
  all_month
Functions

Several are aggregate functions, meaning that they need to be grouped by a certain column.

Find the average of the earthquake magnitude as reported by various stations:

SELECT
  avg(mag) AS avg_mag,
  ST_Transform(ST_Centroid(ST_Union(the_geom)),3857) As the_geom_webmercator,
  net
FROM
  all_month
GROUP BY
  net
Go Further

Build a tool to execute SQL and CSS instructions on button click, like this one for earthquake data.

Check out the Map Academy: Lesson 3 to learn how to make this selector map!

Part 2: Elephant Tracking Data

Movebank.org has a lots animal tracking data. To get our data, I went to Tracking Data Map, then searched for "Loxodonta africana". But you can easily import it into your account by copying the link below.

We'll be making this map:

elephant map

  1. Create a new Map
  2. Import the data into your account from this link:
http://aureliamoser.cartodb.com/api/v2/sql?q=SELECT%20*%20FROM%20elephant_movements&format=geojson&filename=elephant_movements

Animated Elephant Movements

We can see our elephant move around with a Torque visualization.

Multilayer map with the same data used differently

Let's visualize roughly the paths that the elephants take from point to point. These will be best-guess lines as we don't know the intermediate points. They represent linear interpolations between consecutive points.

We can easily do this with some SQL and PostGIS by pasting in the following command and running it:

SELECT 
  ST_Transform(
    ST_MakeLine(
      the_geom ORDER BY timestamp),
    3857
  ) As the_geom_webmercator  
FROM 
  elephant_movements

and apply the following CartoCSS to visualize our lines better:

#elephant_movements{
  line-color: #41c491;
  line-width: 1;
  line-opacity: 0.7;
  line-smooth: 1.4;
  line-clip: true;
  line-dasharray: 2, 3, 2;
  line-comp-op: multiply;
}

elephant map

We can get a more nuanced view of the data by using composite operations

SELECT 
  ST_Transform(
    ST_MakeLine(
      a.the_geom,
      b.the_geom)
    ,3857
  ) As the_geom_webmercator,
  to_char(a.timestamp,'HH12:MI AM, DD Mon') || '--' || to_char(b.timestamp,'HH12:MI AM, DD Mon') As date_range,
  a.cartodb_id
FROM 
  elephant_movements a
JOIN 
  elephant_movements b
ON
  a.cartodb_id + 1 = b.cartodb_id

And apply the following CartoCSS to show intensity when lines cross:

#elephant_movements{
  line-color: #41c491;
  line-width: 2;
  line-opacity: 0.7;
  line-comp-op: multiply;
  line-clip: true;
  line-smooth: 1.4;
}

elephant map

Sharing Your Map

You can publish your map as is via the "Share" button in the interface, but if you would like to build a more custom version you can use the JS libraries and APIs mentioned above, and use CartoDB as your Data Store.

share

In the workshop repo, you'll find a template for making your map as a site online:

  • index.html: setting up your map div and canvas
  • main.js: pulling in your basemap, cartodb feature layers and other details

If you run this locally, you'll get a map that looks like this:

LA MAP

Seeking Inspiration

We worked on a little tool to study your dataset in CartoDB and suggest "interesting maps" to you in the GUI. It'll be a little modal in the lower left corner and will produce demo maps with pre-fab SQL and CSS like this:

Pecan

###RESOURCES

Mapping + CartoDB

  1. Map Academy
  2. CartoDB Tutorials
  3. CartoDB Trainings
  4. CartoDB Editor Documentation
  5. CartoDB APIs
  6. Community help on StackExchange
  7. CartoDB Map Gallery
  8. CartoDB Bootstrap Template by Chris Wong
  9. Friendly Geo Intro by Joey Lee
  10. Lyzi Diamond tutorial on Leaflet

Data

  1. World GeoSpatial Datasets: countries, cities, codes, flags, languages, latitude/longitude, etc.
  2. Center for National Geographic Information
  3. IDEE Metadata Catalog: spatial data from multiple agencies
  4. Geogig: distributed geospatial data
  5. International GIS Data - Penn State

Visualization

  1. Charting Tools Repository
  2. Recommended tools for Visualizations
  3. Perception Concerns
  4. Gestalt Theory
  5. Color Brewer or Geocolor

If you want to chat with me, or send me some cool maps you've made, just email aurelia@mozillafoundation.org.

SELECT
CASE
WHEN state_name != 'Alaska' AND state_name != 'Hawaii'
THEN ST_Transform(the_geom_webmercator,42303)
WHEN state_name = 'Alaska'
THEN
ST_Rotate(ST_Scale(
ST_Transform(
ST_Translate(
the_geom,90,-50
)
,3857
)
, 0.4
, 0.5
),0)
WHEN state_name = 'Hawaii'
THEN
ST_Scale(
ST_Transform(
ST_Translate(
the_geom,55,3
)
,42303
)
, 1.5
, 1.5
)
END the_geom_webmercator
, cartodb_id
FROM states
INSERT into spatial_ref_sys (srid, auth_name, auth_srid, proj4text, srtext) values ( 42303, 'EPSG', 42303, '+proj=aea +lat_1=29.5 +lat_2=45.5 +lat_0=23 +lon_0=-96 +x_0=0 +y_0=0 +datum=NAD83 +units=m +no_defs ', 'PROJCS["NAD83 / Albers NorthAm",GEOGCS["NAD83",DATUM["North_American_Datum_1983",SPHEROID["GRS_1980",6378137,298.257222101]],PRIMEM["Greenwich",0],UNIT["Decimal_Degree",0.0174532925199433]],PROJECTION["Albers_conic_equal_area"],PARAMETER["central_meridian",-96.0],PARAMETER["latitude_of_origin",23],PARAMETER["standard_parallel_1",29.5],PARAMETER["standard_parallel_2",45.5],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["Meter",1],AUTHORITY["EPSG","42303"]]');
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment