DSI - Timeseries Class

a_timeseries.md

Intro to Time Series Data

###SVA-DSI ###Metrics and Data Visualiation I October 18th, 2016

Find this document here:

• Gist: http://tinyurl.com/dsi-timeseries

Outline

1. Time Series Basics
   • What is time-series?
   • Types of Visualizations
2. Time-Series Data
   • Data + Date Formats
   • Tools for Converting your Data
3. Charting Tools
   • Time-Series Libraries
   • D3 - Drawing with Data
   • Diagram a Time-series Project
     • (IFES Election Monitoring)
     • (Ushahidi's Downloads Graph)
4. Building A Narrative
   • Case Study 1: NYC 311 Graffiti (Highcharts)
   • Case Study 2: Task Calendar (D3)
   • Bl.ocks for Time-series display
5. Resources

Time-Series Basics

What is time-series?

Time Series Data is a series of data points ordered in time, or data that has a timestamp associated with it (this can include date and time information). Often, time-series data is most interesting for journalism, because it illustrates change, and difference from precedent.

Types of Visualizations

Data Visualization Catalogue is a great place to get started to define the type of visualization that is most appropriate for your data.

Take a look at it, and spend some time deciding which visualizations are most appropriate for the data you would like to work with in this course. You can sort by format and the function you desire in your resulting visualization.

If you want to see how something is implemented, you might search for a library or language in Nerdy Data, a search engine for source code. You can also use it to find out how your favorite technologies where built.

Time-Series Data

Time series data comes in various formats.

It also comes from a variety of sources, and often you will have to process it (change it to the format you need for your visualization library to understand it). Sometimes, you might want to convert all of your date formats to the same format and then reconvert when you visualize, if so you can convert to unix time, which is a series of numbers that the computer will read as a date that a human might recognize.

so the following are the same date/time, with different codes

• 09/20/2015 8:25PM (UTC)
• 1443644745 (Unix/Epoch)
• 2015-09-30T20:25:45+00:00 (ISO 8601)

Typical Formats

Formats

• JSON - Javascript Object Notation, data objects made of attribute-value pairs
• CSV - Comma-Separated Values, tabular data delimted by , s to suggest different fields, you can convert excel spreadsheets to this easily, and it can be read into your code
• TSV - Tab-Separated Values, same as above, but tab is the delimiting
• XML - eXtensible Markup Language

You can tell the format of a file by its extension, or the letters that follow the . after the file name.

• aurelia.txt would be a text file
• aurelia.csv would be a CSV file
• aurelia.xml would be an XML file
• aurelia.json might look like this...

{
  "id": 1,
  "name": "Aurelia",
  "height": 123,
  "tags": [
    "Teacher",
    "Nerd"
  ],
  "students": {
    "onsite": 16,
    "remote": 4
  }
}

Types

• Dates | "Friday, September 24th, 2004" or
• Integers/Numbers (int/num) | "1"
• Booleans | "TRUE"
• Characters (char) | "a"
• Floats | "1.2435"
• Strings (str) - alphanumeric collection of characters | "aurelia"

#####Considerations

• What do you want it to look like?
• What do you want to happen when the user interacts with it?
• Type of dataset and date format?
• Type of visualization?

If you're wanting to get started with a dataset that is not your own, you can follow the tutorial from Michelle Minkoff at ForJournalism: http://forjournalism.github.io/courses/charting-and-visualization/

It provides a great overview of the process of coding, complete with debugging, working with dirty datasets (in this case, crime data for Georgia), and visualizing them with Raphael.JS, a Javascript library for visualizations.

Data + Date Formats

• Quandl - the "wikipedia" of time-series data, they provide datasets and format parsers for conversion to what you need
• Data.gov - almost every government/city has an "open data portal" intiative where you can download data of interest to you, and search through for differnt formats
• Federal Data Listing
• Enigma.io - loads more open data, larger datasets, and tools for correlating multple datasets
• Exversion - similar data catalog, they also have a fabulous newsletter (subscribe for cool datasets in your inbox)
• NYC Open Data - loads of cities have "socrata" portals where you can download data
• NASA Data
• IRE Database Library - IRE also provides a lot of open data to investigative journos, along with data dictionaries telling you how to read it

For more complex datasets, there are guides online like the Journalists' Guide to datsets which help you parse and read commonly used but pretty obscure data releases from an investigative perspective.

Likewise, some time-series data is not easy to find, and you have to request it from government agencies. If you would like to explore this, check out FOIA Machine. For the purposes of this class though, you might opt for something that's easier to find.

Tools for converting data

• Mr. Data Converter - an online tool for converting data from excel to other formats (HTML/JSON/XML)
• Open Refine - not unlike excel but way more powerful for large datasets, you can also convert formats in refine (ie. from JSON to CSV or vice-versa)
• DSV - a parser and formatter for delimiter-separated values
• Tabula - a tool for extracting data from PDFs

Since this is not a class for data cleaning, but almost every dataset requires it, take a look at some examples for the data visualization you might want to build, and try to make your data match that style. If you're dealing with a large dataset that would take too long to manually edit, come chat with me and I can help, you can also take a look at the Data Wrangling Handbook which features nice tutorials on how to deal with data.

Charting Tools

Time-Series Libraries

There are loads of time-series data libraries, you can google for your language of preference and "time-series" to discover them. Most of the popular ones are frameworks built for D3.

Here are a few:

Javascript

• Rickshaw
• Metrics Graphics
• C3JS
• HighCharts
• Miso Project

All are pretty customizable with CSS, so don't feel bound to their default designs.

Python

• Seaborn
• Plot.ly
• Bokeh

D3 - Drawing with Data

D3.js is a physics based animation library for drawing with data. It is incredibly flexible, but has a non-trivial syntax and math prerequisite, and can is not supported by all browsers (like I.E.). It's great for dynamism and very popular so I encourage you to check it out.

• Tutorials
• Documentation

Diagram a Time-series Project

Here is an example D3 project, we'll go through it together:

Code: https://github.com/auremoser/ushaverse Demo: http://auremoser.github.io/ushaverse/

Here is what the data looked like:

What if I wanted to make it more dynamic/complex? I could try adapting it to this. Usually when I want to try something new, I search through Mike Bostock's blocks for inspiration.

Building A Narrative

Case Study 1: Graffiti Charts

Graffiti Chart: http://auremoser.github.io/graphitiTime/

This project used a CSV to create small Bar Charts using HighCharts, from data about NYC cleanup rates for Graffiti 311 complaints.

Github Repo Here: https://github.com/auremoser/graphitiTime

Case Study 2: Task Calendar

Project Tracker: <auremoser.github.io/pirateplotr/>

This project made project management data into a Gantt Chart using D3.js, a kind of timeseries graph updated every few minutes, using Google Sheets as a data store.

Github Repo Here

Blocks for Time

There are lots of ways you can approache a time-series visualization, and most libraries will have examples/demos that you can modify.

bl.ocks are also a fantastic resource for other examples and demos, so search through those to learn how code creates visualizations. Mike Bostock has some great examples.

Here are some other charts and graphs that you can take a look at, not specificially time-series, but temporal and chart-related:

Type	Title	Link/Demo	BlogPost
Chart.js Bar Graph	Traffic Data	Aurelia's Block
Highcharts	Sensor Data	Github / Demo	MOW Post
Highcharts	Weather Data	Aurelia's Block	Tutorial
Chart.js Line Graph	Tornado Data	Andrew's Block
Plot.ly	Earthquake Data	Plotly Tutorial	CartoDB Blog

###Resources

1. Charting Tools Repository
2. Recommended tools for Visualizations
3. Perception Concerns
4. Gestalt Theory
5. Color Brewer or Geocolor
6. ForJournalism
7. Data Wrangling Handbook

index.html

<!DOCTYPE html>
<html>
	<title>Blackout Map</title>
	<link rel="shortcut icon" href="https://raw.githubusercontent.com/auremoser/images/master/lightning-favicon.ico" type="image/x-icon">
	<meta name="viewport" content="initial-scale=1.0, user-scalable=no" />
	<meta http-equiv="content-type" content="text/html; charset=UTF-8"/>
	<!-- add cartoCSS stylesheet -->
	<link rel="stylesheet" href="http://libs.cartocdn.com/cartodb.js/v3/themes/css/cartodb.css"/>


	<style>
		html, body {
		  height: 100%;
		  margin: 0;
		  padding: 0;
		}
		header {
			background-color: #fff;
			color: #252525;
			font-family: 'Unica One', sans-serif;
			font-size: 45px;
			padding: 0;
			text-align: center;
			text-transform: uppercase;
		}
		#map {
			bottom: 0;
			left: 0;
			position: absolute;
			right: 0;
      		top: 50px;
		}
		#temp {
			bottom: 0;
			display:none;
			height: 300px;
			left: 0;
			position: absolute;
			right: 0;
			width: 100%;
		}
	</style>
	<body>
		<header>
     		<strong>Blackout Map</strong> - Lebanon
     	</header>

		<div id="map"></div>
		<div id="temp"></div>

		<!-- include jquery and highcharts -->
    <script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
		<script src="http://code.highcharts.com/highcharts.js"></script>
		<script src="https://rawgit.com/highslide-software/highcharts.com/master/js/themes/dark-unica.js"></script>
    <!-- include google maps library *before* load cartodb.js -->
    <script src="https://maps.googleapis.com/maps/api/js?sensor=false"></script>
    <!-- include cartodb.js library -->
    <script src="http://libs.cartocdn.com/cartodb.js/v3/cartodb.js"></script>
    <!-- include your javascripts -->
    <script src="main.js"></script>
    <script src="temp.js"></script>
	</body>
</html>

main.js

function main() {
  var vis;
  // add if you want to select on click and trigger a function
  var CLICKLAYER = 1;

  // add your url ID here between viz/ and /viz.json
  var layerUrl = 'https://team.cartodb.com/u/aureliamoser/api/v2/viz/67f88c2a-b77d-11e4-aae0-0e853d047bba/viz.json';

  var options = {
    // zoom projection to adjust starting point zoom
    zoom: 5,
    // set lat/long to center on lebanon
    center_lat: 29.90817309627726,
    center_lon: 35.5365673828125,
    cartodb_logo: false,
    layer_selector: true,
  }
  // layers from your visualization, set them up as vars
  var layerIds = {
	'pop2000':0,
	'epl2009':1,
};

// create map with layers from cartodb editor
  cartodb.createVis('map', layerUrl, options)
    .done(function(vis, layers) {
      	console.log(vis, 'layers', layers)
      	window.vis = vis;
      	window.overlay = vis.getLayers()[1];
      	window.layers = layers;

      ///////////
      // pt. 2 continue if you want to select on click to trigger a function
      var sublayer = layers[1].getSubLayer(CLICKLAYER);

      sublayer.on('featureClick', function(e, pos, latlng, data) {
        console.log(e, pos, latlng, data);
        $('#temp').show()

      })
      ///////////
    })

    .on('error', function() {
      cartodb.log.log("some error occurred");
    });
}

$(main);

temp.js

// load some temperature data -> high temperature ==> blackouts
// SOURCE:
// http://sdwebx.worldbank.org/climateportal/index.cfm?page=country_historical_climate&ThisRegion=Middle%20East&ThisCCode=LBN#%23

////////////////
// PULLING DATA
////////////////
// You need to patch your jquery request
jQuery.ajaxPrefilter(function(options) {
    if (options.crossDomain && jQuery.support.cors) {
        options.url = 'http://guarded-ocean-2049.herokuapp.com/' + options.url;
    }
});

//Simple helper function to manipulate strings as templates.
function GTemplate(){}

GTemplate.compile = function(template, otag, ctag) {
    if(!template) template = '';
    var compiled = function(context){

        function replaceFn() {
            var prop = arguments[1];
            return (prop in context) ? context[prop] : '';
        }
        otag = otag || '{{';
        ctag = ctag || '}}';

        return template.replace(new RegExp(otag+'(\\w+)'+ctag,'g'), replaceFn);
    };
    return compiled;
};

// this is the string template we use to construct our requests.
var templateString = "http://{{host}}/climateportal/components/getData.cfc?method=getCCKHistoricalMonthlyCRUPixChart&_cf_ajaxproxytoken={{token}}&returnFormat=json&argumentCollection={{query}}&_cf_nodebug=true&_cf_nocache=true&_cf_rc=31";

// here we are compiling the templateString, we do this only once.
var urlTemplate = GTemplate.compile(templateString);

////////////////
// EXAMPLE USE:
////////////////

// This is the payload for your request, on each click on the map, you would need to
// build this object, from the click you get lat/long, sYear and eYear you get from
// the pull down menu, and type is 1/2 rain/temp.
var payload = JSON.stringify({
                lat: 34.19817309627726,
                long: 36.1065673828125,
                sYear:"1900",
                eYear:"2009",
                type:2
    });

// build url options from payload.
var urlOptions = getURLOptions(payload);
// build url from comiled template.
var url = urlTemplate(urlOptions);

var temp;

function updateTemperature(data){
    if(!data || !data.length) return console.error('NO VALID DATA SET');
    // make all numbers take one decimal
    data = data.map(function(tmp){
        return Math.round(tmp * 10) / 10;
    });

    window.temp.series[0].setData(data);
}

function getURLOptions(payload){
  var urlOptions =
  {
    host:'sdwebx.worldbank.org',
    token:"54BF96C2F7AB082C7E975889405CA3D9871063A2CA50D4BAACAD95848B5C73C977",
    query: payload
  };
    return urlOptions;
}

////////////////
// BUILD CHART
////////////////
function setupChart(){
    temp = new Highcharts.Chart({
        chart: {
            renderTo: 'temp'
        },
        title: {
            text: 'Monthly Average Temperature',
            x: -20 //center
        },
        subtitle: {
            text: 'Source: WorldBank.org',
            x: -20
        },
        xAxis: {
            categories: ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
                'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
        },
        yAxis: {
            title: {
                text: 'Temperature (°C)'
            },
            plotLines: [{
                value: 0,
                width: 1,
                color: '#808080'
            }]
        },
        tooltip: {
            valueSuffix: '°C'
        },
        legend: {
            layout: 'vertical',
            align: 'right',
            verticalAlign: 'middle',
            borderWidth: 0
        },
        series: [{
            name: 'Lebanon',
            data: [],
            color: {
                linearGradient: { x1: 0, x2: 0, y1: 0, y2: 1 },
            stops: [
                [0, '#808080'],
                [1, '#1e1e1e']
            ]
            }
        }]
    });
}

function loadData(){
    // make ajax request using url
    $.ajax({
        url:url,
        type:'GET',
        // you need to add this crossDomain property
        crossDomain:true,
        dataType:'json',
        success:function(data){
            console.log(data);
            updateTemperature(data);
        // add some jquery animations to the rendering of the chart
        if(! $('#temp').is(":visible")){
              $('#temp').show(function(){
                // $('#map').animate({height:'58%'});
                $('#map').animate({bottom: "36%"});
                $('#temp').animate({top: "64%"});
              });

              google.maps.event.trigger(map, 'resize');
            }
        },

        error:function(xhr, er){
            console.log(er);
        }
    });
}

$(function () {
    setupChart();
    loadData();
});

thumbnail.png