Skip to content

Instantly share code, notes, and snippets.

@Kenyat1989

Kenyat1989/thumbnail.png

Forked from auremoser/thumbnail.png
Created October 11, 2015 09:59
Show Gist options
  • Save Kenyat1989/7b1f1b04e96b28ea0bc5 to your computer and use it in GitHub Desktop.
Save Kenyat1989/7b1f1b04e96b28ea0bc5 to your computer and use it in GitHub Desktop.
Download ZIP
FOSS4G: Mapping Workshop
Raw
workshop.md

Building Dynamic Maps with CartoDB

###FOSS4G Seoul

Aurelia Moser, @auremoser, aurelia@cartodb.com

Andy Eschbacher, @MrEPhysics, eschbacher@cartodb.com

September 15th, 2015 WS22 14h30-18h30

SupermanVS Oil Drilling

Find this document here:

Outline

  1. Mapping Basics
    • You are Here.
    • Types + Topics
    • Anatomy of a Webmap
  2. Mapping Data
    • Formats + Free Sources
    • Collecting + Sync Tables
    • Storing
  3. Mapping Tools
    • Examples
    • Tour of the interface
    • APIs / JS Libs
    • Data Import + Sync Tables
    • Customizing UI + CartoCSS
  4. Creating Custom Maps
    • Mapping in the Editor
    • Mapping in Code
      • Python
      • Node
      • R
  5. Building a Narrative
  6. Resources

SupermanVS Oil Drilling 2

MAPPING BASICS

YOU ARE HERE.

seoul

Where are you? If I gave you directions here, you could say you were at

70 Baumoe-ro 12-gil Seocho-gu, Seoul South Korea

But if I wanted to describe the location as a point on a map, I might describe the same location using latitude and longitude:

37° 46' 63.55", 127° 03' 29.27"

It reads as "37 degrees, 46 minutes and 63.55 seconds, 127 degrees, 03 minutes and 29.27 seconds"

We describe locations with a spatiotemporal notation, it's also somewhat obscure. So let's describe the same place using longitude and longitude but using decimal degrees instead of minutes and seconds. There are a number of conversion tools available online to do this:

37.46635575024666, 127.03292727470398

This workshop's content was subtly inspired by the "ring of fire" an area skirting the Pacific Ocean that is associated with an almost continuous series of oceanic trenches, volcanic arcs, and volcanic belts and/or plate movements.

You can see some of the most epic geological phenomena in Korea and environs, like Halla Mountain, a one of the world’s few shield volcanoes located over a magma “hotspot” on a stationary plate of continental crust. The singularity of its peculiar volcanic features combined with its instructiveness about geological processes has earned the site World Heritage status.

cheju

###Types + Topics

Gallery of maps

There are many different types of map you can make, browse this gallery to view some options. Most maps illustrate some kind of geospatial change.

DYNAMIC MAPSs

Types of Visualizations

Source: The Data Visualization Catalogue.

Time Travel Map

Source: Time Travel Between Counties, Carto.JS

DATA SEARCH TOOLS GDELT Geographic News Search Tool

Source: GDELT Geographic News Search Tool

Journalists have used the GDELT data to track wildlife crime, and the spread of the Islamic State in the Middle East among other things.

You can fork the GDELT hourly synced data set from the CartoDB DataLibrary and add it as a layer on your map or use the Geographic Search Tool linked above to search for tags of interest.

CHART GRAPHICS

County Chart

Source: Geogia County Car Crash Counts, C3.JS

COMPARISON NARRATIVE

Michael's Syrian Refugee visualization Map population by relative density

  • Maps give you more context than most visualizations.
  • They allow you to apply data to a recognizable topography.

Sources

  • CartoDB: light open source library and graphical user interface application for hosting and visualizing geospatial data
  • ChartJS: light library for creating charts and graphs
  • GDELT: the global database of events languages and tones

Anatomy of a WebMap

Webmap Gif

Maps are composed of:

####1. Tiles (often)

Stamen Tiles

Tiles are 256x256 pixels squares that plot one next to the other to stitch together a pseudo-seamless image; the tiles draw dynamically as you focus and zoom in on your map so that the map draws on a continuous canvas

zoom pyramid

####2. Feature Layers

Map layers that populate on top of your basemap

####3. Javascript/HTML/CSS for rendering on the web

With these languages you can publish your map with the basetiles loaded and your data layers appropriately geocoded; with javascript you can also add to the interactivity of your map, revealing metadata in the tooltips

Mapping Data

Formats + Free Sources

You can get geospatial data for your feature layers in lots of places. Here are some examples.

Collecting + Sync Tables

We'll be collecting data from multiple sources. Most of which are available in the repository where our data will be stored.

OUTCROPS IN OSM

Outcrops are placse where the bedrock or superficial deposits have become locally exposed and are directly accessible to analysis in OSM; let's map them!

OSM Geo

Overpass

You can read more about Overpass on the Open Street Map Wiki.

Steps:

  • Pan manually to an area in OverPass Turbo
  • Go to the "Wizard"
  • Look up the appropriate OSM tag in the OSM wiki
  • Search for "geological=outcrop" or whatever
  • Export your data as GeoJson or KML
  • Upload into CartoDB or another interface

outcrop search

Notes:

  • only nodes, only ways, only relations (-> limit so it doesn't map data you're not interested in)
  • search for something that is too large it freezes (-> define your bounding box)

Turbo Output Interface

Geological Features

  • CartoDB Data Library

Data Library

There are lots of geospatial data formats that may or may not be easy to parse. Here's a short list of ones that are commonly imported into CartoDB.

Case Study 1: Outcrops

Outcrop Data

So let's take a look at this outcrop data. You want to lead people to all of the outcrop stations globally. You can use the kml file here in the data folder.

You can get the libraries from OSM or NYC Open Data but let's go with the KML from OSM that we extracted with OverPass.

.kml which is a notation for XML that first made popular by Google Earth before becoming standardized. Google Mapping Products happily use kml files. Check to see how the locations are described:

<?xml version="1.0" encoding="UTF-8"?><kml xmlns="http://www.opengis.net/kml/2.2"><Document><name>overpass-turbo.eu export</name><description>Filtered OSM data converted to KML by overpass turbo.
Copyright: The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.
Timestamp: 2015-09-14T14:05:02Z</description><Placemark><name>Sonnberg</name><Point><coordinates>9.43865,50.6228007</coordinates></Point><ExtendedData><Data name="@id"><value>node/428215780</value></Data><Data name="geological"><value>outcrop</value></Data><Data name="name"><value>Sonnberg</value></Data><Data name="note"><value>Rocky promontory, subject to erosion. Middle Triassic shallow marine limestone, Jena Formation (U.-Muschelkalk, Wellenkalk). Small outcrop within E-W trending graben structure. Contact to MIttlerer Buntsandstein exposed in river bed southwest of crag.</value></Data></ExtendedData></Placemark><Placemark><name>Monkey rock</name><Point><coordinates>177.127952,-17.7323069</coordinates></Point>
...

Notice that the point coordinates are in the format longitude, latitude. Some geoformats are long, lat and others are long, lat. This makes everyone sad.

When coordinates cannot easily be parsed they go to "null island," a fictional place at coordinates (0,0).

Null-island

In a bit, we'll show how you can convert, process and use a file like this to plot data on your map!

Storing

CartoDB is a PostGres database in the cloud, which means it handles a lot of your backend data needs and allows you to query for data and pull those data and basemap tiles into your front-end code.

You can also store your data in Github, or in another service that makes it web accessible. Read more about that here and here.

Mapping Tools

Logos

There are loads of ways to approach a map here are a few approaches to mapping the same dataset.

Map Making Exercise

We can make a map with the outcrop data, or other data in the data folder for this workshop.

Using Google Maps

  • https://www.google.com/maps/d/
  • create account if you don't already have a Gmail account
  • click on Import Map in top left hand menu (or My Maps -> Create map in some Google Maps UIs)
  • upload outcrop.kml
  • explore changing the map features if you would like

Using Mapbox

  • https://mapbox.com/
  • create account if you don't already have a Mapbox account
  • click on the Data tab at the top right hand corner of the screen; * click on import
  • upload outcrop.kml
  • select map features if you would like then click on Import Features explore changing the map features if you would like

Using CartoDB

  • https://cartodb.com/
  • create account if you don't already have a CartoDB account, use this URL to get boosted features
  • click on Create Map; select Map View at the top of the screen
  • click on the '+' or Add Layer option at the top of the right side menu
  • upload outcrop.kml
  • explore changing the map features if you would like

Using Leaflet

You can also make a map from scratch using Leaflet.js to attach a set of points to a map made of tiles provided by OpenStreetMap.

You will first need to convert your kml file into GeoJSON (although I have both in the data folder) for this workshop.

GeoJSON is a file format that is easily digestable by JavaScript. If you have a data format (shp, kml) that is not geojson you can convert it to the right format for your code with GeoJSON.io/

	{
	  "type": "FeatureCollection",
	  "generator": "overpass-turbo",
	  "copyright": "The data included in this document is from www.	openstreetmap.org. The data is made available under ODbL.",
	  "timestamp": "2015-09-14T14:05:02Z",
	  "features": [
	    {
	      "type": "Feature",
	      "id": "node/428215780",
	      "properties": {
	        "@id": "node/428215780",
	        "geological": "outcrop",
	        "name": "Sonnberg",
	        "note": "Rocky promontory, subject to erosion. Middle 	Triassic shallow marine limestone, Jena Formation (U.-	Muschelkalk, Wellenkalk). Small outcrop within E-W 	trending graben structure. Contact to MIttlerer 	Buntsandstein exposed in river bed southwest of crag."
	      },
	      "geometry": {
	        "type": "Point",
	        "coordinates": [
	          9.43865,
	          50.6228007
	        ]
	      }
	    },
	    {
	      "type": "Feature",
	      "id": "node/568331113",
	      "properties": {
	        "@id": "node/568331113",
	        "geological": "outcrop",
	        "name": "Monkey rock"
	      },
	      "geometry": {
	        "type": "Point",
	        "coordinates": [
	          177.127952,
	          -17.7323069
	        ]
	      }
	    },

outcrop

Quick outcrop map by type.

Map Examples in CartoDB

urban-interface

urban-reviewer

Tour of the interface

We'll use CartoDB for the subsequent exercises because of it's flexibility, support for SQL and JS, and backend handling for data management and geocoding.

int-data int-upload int-vis int-share

APIs / JS Libs

You can read more about the CartoDB APIs and JS Library here

  • CartoJS - JS library for interacting with CartoDB
  • Maps API - generate public/private maps with data hosted on your CDB account
  • SQL API - run sql in your code to dynamically filter/request/query your mapped data stored in CartoDB via http calls
  • Import API - push data to your CartoDB Account

Data Import + Sync Tables

Geospatial data is info that ids a geolocation and its characteristic features/frontiers, typically represented by points, lines, polygons, and/or complex geographic features.

Issues:

Geocoding + SQL/PostGIS

The most basic SQL statement is:

SELECT * FROM table_name

The CartoDB editor features a SQL "tray" that supports PostGIS functions to filter, modify, and analyze your data on the fly, updating your tables, merging them, changing projections and all spatial operations you might need to build awesome maps.

SQL

Andy will followup with more SQL in Part 2 of this workshop.

Sync Tables

Sync Tables

The Editor is especially setup to process realtime data updates via sync tables.

You can import data that lives online via a URL, and set it to pull and update your map at regular intervals.

Check the file types supported in sync tables; keep in mind that it also works with dropbox + google drive.

Notes:

  • to auto-geocode a sync table, verify that it contains the following:
    • country column, a latitude column and a separate longitude column
    • a column of IP addresses
  • to edit sync tables you need to be disconnected from the datasource, so during syncing, you can use SQL to manipulate the dataset on the fly
  • you can create sync tables in both the Editor and via the Import API

Customizing UI + CartoCSS

Basemaps

You have myriad customization options in the in-browser editor:

  • sql - run sql and postgis functions across your data
  • wizard - adjust the type, colors and fills in your map
  • infowindow - create hovers, tooltips with information from your datatables
  • css - customize the css and style of your map outside the wizard
  • legends - create keys for your map
  • filters - filter the data without sql

Likewise, many types of visualizations:

  • Simple -- most basic visualization
  • Cluster -- counts number of points within a certain binned region
  • Choropleth -- makes a histogram of your data and gives bins different colors depending on the color ramp chosen
  • Category -- color data based on unique category (works best for a handful of unique types)
  • Bubble -- size markers based on column values
  • Intensity -- colors by density
  • Density -- data aggregated by number of points within a hexagon
  • Torque -- temporal visualization of data
  • Heat -- more fluid map of concentration; emphasis on far over near-view

Creating Custom Maps

Pluto Editor

Mapping in the Editor

The Editor GUI allows you to select your visualization "type" and customize color palettes, design details, and otherwise set the tone for your map. It also gives you ways to run sql and set your cartocss in the browser before you input those things as templates in your code.

Styles
  • Marker Fill: change the size, color, and opacity of all markers
  • Marker Stroke: change the width, color, and opacity of every marker's border
  • Composite Operation: change the color of markers when they overlap
  • Label Text: Text appearing by a marker (can be from columns)
Infowindows/hovers

Infowindow options

  • Select which column data appear in infowindow by toggling column on
  • Customize further by selecting HTML-view
Basemaps

Basemap options

Check out the base CartoDB Basemaps from Stamen.

Mapping in Code

Plotly

Python

Use the CartoDB Python module to create and update tables; check out Andy's IPythonNB demos.

Node

Use the CartoDB Uploader package available in NPM for working in Node.js

R

Use Kyle Walker's R2CartoDB project to create and update tables in R.

Building a Narrative: Earthquake + Elephant Data

We already made a simple map of outcrop points with OSM data, but usually building a narrative on a map requires more work.

Andy worked on a little tool to study your dataset in CartoDB and suggest "interesting maps" to you in the GUI. It'll be a little modal in the lower left corner and will produce demo maps with pre-fab SQL and CSS like this:

Pecan

Part 1 + 2: Earthquake Mapping Data

The data folder includes some outcrop data as well as provinces for S.Korea. You can make a map with any one of these, or a dataset of your choosing.

We'll be working with earthquake data, making something like this.

Earthquake selector

Check out the Map Academy: Lesson 3 to learn how to make this selector map!

PROCEED TO

Part 3: Elephant Tracking Data

Let's try the Movepick elephant data, making something like this.

PROCEED TO

Sharing Your Map

You can publish your map as is via the "Share" button in the interface, but if you would like to build a more custom version you can use the JS libraries and APIs mentioned above, and use CartoDB as your Data Store.

In the workshop repo, you'll find a template for making your map as a site online:

  • index.html: setting up your map div and canvas
  • main.js: pulling in your basemap, cartodb feature layers and other details

If you run this locally, you'll get a map that looks like this:

Province Map

Tell Time/Stories: Odyssey + Torque

SupermanVS in Flight

Outside of the CartoJS library, we have others to help you build dynamic narrative with your data.

Maps that tell Time - Torque

Realtime Traffic Map

Source: Traffic Change Data for SF Demo: Municipal Traffic in SF App Blog: CartoDB + Firebase

  1. Demonstrations in Brazil
  2. Animal migration patterns
  3. Diwali Celebrated
  4. Ramadan Tweets w/OdysseyJS
  5. Alcatraz Escapees
  6. Lynching and the Press

Twitter Import

Maps that tell Stories - Odyssey JS

  1. Al Jazeera: Israeli-Palestinian Conflict by Tweets
  2. The Sounds of 11M
  3. Berlin Wall Historic Tour

Partner Graphics: Graphs + Charts

You can use CartoDB's SQL API to query your data and pull it into any charting library of your choosing.

You can easily wire up a chart of earthquake depth data for example, set it as a sync table to update continuously, check the code here.

chart

Learn more about it here!

Here are some examples:

Type Title Link/Demo BlogPost
NVD3 NYC Elementary Schools Chris' Github
Chart.js Bar Graph Traffic Data Aurelia's Block
Highcharts Sensor Data Github / Demo MOW Post
Highcharts Weather Data Aurelia's Block Tutorial
Chart.js Line Graph Tornado Data Andrew's Block
Plot.ly Earthquake Data Plotly Tutorial CartoDB Blog

###RESOURCES Circumnavigation

Mapping + CartoDB

  1. Map Academy
  2. CartoDB Tutorials
  3. CartoDB Trainings
  4. CartoDB Editor Documentation
  5. CartoDB APIs
  6. Community help on StackExchange
  7. CartoDB Map Gallery
  8. CartoDB Bootstrap Template by Chris Wong

Data

  1. World GeoSpatial Datasets: countries, cities, codes, flags, languages, latitude/longitude, etc.
  2. Center for National Geographic Information
  3. IDEE Metadata Catalog: spatial data from multiple agencies
  4. Geogig: distributed geospatial data
  5. International GIS Data - Penn State
  6. South Korea Shapefiles
  7. GIS research guides and Seoul City - University of Michigan

Visualization

  1. Charting Tools Repository
  2. Workshops @ CartoDB
  3. Recommended tools for Visualizations
  4. Perception Concerns
  5. Gestalt Theory
  6. Color Brewer or Geocolor

If you want to chat with us, or send us some cool maps you've made, just email Aurelia at aurelia@cartodb.com or Andy at eschbacher@cartodb.com.

Please give us feedback on the workshop here!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment