README.md

Summary:

This project is part of the Udacity Data Analyst Nanodegree Project 5, with the aim of sharing knowledge with others. This is a dimplejs example based on the flight data from RITA that you can find at:

http://stat-computing.org/dataexpo/2009/the-data.html

The graph represents the evolution in terms of number of flights per year versus total distance accumulated per airline per year for the period between 1988 to 2008.

Check it out in b.locks: http://bl.ocks.org/kingkastle/raw/7188565a417968331fda/

Story:

This graph represents the evolution of the commercial aviation, in particular US airlines during the period 1988-2008, the main parameters represented are:

• Total flights per year
• Average Flight Distance per year
• Total Airlines per year

Based on these variables and its evolution during the period, the following conclusions can be made:

1. Constant increase of the Average flight distance: While years pass there is a constant increase of the average flight distance for all airlines, a probable reason for this would be the technological evolution of the aircrafts. It seems airlines compete each other to reach new destinations further away or create new further appart connections.
2. Increase of the Number of Flights: In the overall, the amount of flights per year gradually increases except for two particular years: 2002 and 2008. As we know, 11-September 2002 and economical crisis in 2008 could be the events to explain this decrease in the overall number of flights.
3. Market fragmentation: What happend after 2002? there are a bunch of new airlines into play, after the aviation crisis in 2002 new investors invested in new airlines?... These questions go beyond the scope of this graph.

Older Versions

The files: · output_previous.tsv · index_previous.html

Correspond to the previous dimple graph submmited, but after instructors' notes, this version was disregarded.

Design

This current version is much simpler, in terms of coding, than the previous version. However it is much easier to interpretate.

This current version is motivated by the comment of the last instructor's review: "I do think this visualization (Previous visualization, in this case) is explanatory and that you've found a really interesting story to tell (average flight distance is increasing and airlines are flying more flights). My only criticism is on the design choices and how the information is displayed. This project can be very subjective to grade, but I think there is a clearer way to encode the information so that your story really comes through. I made a rough draft plot in R that I'd like you to consider. http://postimg.org/image/dbgq31bb9/ My version is quite over plotted, but I've changed the encoding so that year is on the x-axis, plot type is a line chart, average flight is on the y-axis, and point radius is the number of flights. Your main conclusions about the data have to do with time (story section 1, 2 and 5). It's important to feature the time variable as a position encoding because it's so important to your story (position is usually the most important encoding for a reader). To avoid over plotting, you could actually make multiple plots based on market strategy: one plot for the big legacy airlines, one plot for regional short distance airlines. Or categorize airline by market type and give airlines with the same market type the same color. Right now, it's really hard for me to follow the trends over time. I have to mentally choose an airline, and with each new year try to locate its color as it skips around. But I can't really follow two airlines at once to compare them or see general trends over time because so many airlines come in and out of the data set. But again, I like the story you're telling."

So, starting from instructor's example, I decided to focus on the main bullets that the graph wants to convey:

1. Evolution of the flight per year
2. Evolution of the average flight distance per year
3. Evolution in the number of airlines

Based on these points, I created this simple graph which clearly shows the evolution of this points. At this stage, I don't try to create a fancy graph, like the previous one, but a graph that conveys the right message in the simplest possible way. So in the x-axis is represented the time variable, the first y-axis includes the total flights per year computed for all airlines, secondary y-axis includes the averaged flight distance averaged for all airlines per year and the bar chart includes the total number of airlines per year.

To end, contrary to the instructor's recomendation of representing all the airlines, I prefer to represent the resulting numbers of all of them, as I think this is a better way to convey the right message and of course, avoid overplotting issues that may confuse readers.

How to use it

1. Download the data from the RITA portal: http://stat-computing.org/dataexpo/2009/the-data.html
2. Download the carriers.csv file from: http://stat-computing.org/dataexpo/2009/carriers.csv
3. In the Python file, on line 4, change the path to use the one where all your downloaded files from the previous points are
4. The Python file creates the file: "output3.tsv", place this file where index.html is
5. Open index.html and enjoy!

Resources

Main resource used is: http://dimplejs.org/advanced_examples_viewer.html?id=advanced_storyboard_control

Feedback

Here are some comments I received from some colleagues:

Colleague 1: What do you notice in the visualization? Every year is quite different for each airline. What questions do you have about the data? It looks there is a lot of variation from year to year What relationships do you notice? Flights increase per year What do you think is the main takeaway from this visualization? Everything changes each year. Is there something you don’t understand in the graphic? Everything is clear for me, all is easy to see in this graph.

Colleague 2: What do you notice in the visualization? There is an almost linear increase of the flights per year except for 2002, probably due to September 11? … What questions do you have about the data? None, it is clear info. … What relationships do you notice?

1. Seems to be a linear correlation between flight number and distance per airline
2. Seems not to be any correlation between average delay per flight and number of flights per year or distance. … What do you think is the main takeaway from this visualization? Basically, the selection of these 3 variables. … Is there something you don’t understand in the graphic? No

Colleague 3: What do you notice in the visualization? You can see how airlines increase their business (number of flights) and how airlines evolve through the years. What questions do you have about the data? I can find a correlation between delay and distance or number of flights. What relationships do you notice? Number of flights and distance are correlated. What do you think is the main takeaway from this visualization? See the market evolution Is there something you don’t understand in the graphic? Everything is clear

Udacity Coach: He inspired most of the changes since the first version.

dataformat.py

import bz2, math
import json, csv
import pandas
import numpy as np

data_path = r"C:\Users\sesa219988\Desktop\TRABAJO\Udacity\Pt5\data\\"

###    Define the variables of interest:

Year = "Year"
DepDelay = "DepDelay"
UniqueCarrier = "UniqueCarrier"
Distance = "Distance"
Cancelled = "Cancelled"
CancellationCode = "CancellationCode"

###    Create the object stores all the data in the right format:

per_year = {}

def carrierfinder(UniqueCarrier):
    with open(data_path + "carriers.csv") as carrier_file:
        for line in carrier_file:
            if UniqueCarrier in line: return line.split(",")[1].strip()[1:-1]
    return "Carrier Not found"

def fill_Data(Year,UniqueCarrier,DepDelay,Distance,Cancelled,CancellationCode):
    # Next two fields could be "NA", in that cesa, these are transformed to 0:
    if DepDelay == "NA": DepDelay = 0
    if Distance == "NA": Distance = 0
    if Cancelled == "NA": Cancelled = 0
    if Year not in per_year: per_year[Year] = {}
    if UniqueCarrier in per_year[Year]:
        per_year[Year][UniqueCarrier]["Accumulated_Delay"] += int(DepDelay)
        per_year[Year][UniqueCarrier]["Distance"] += int(Distance)
        per_year[Year][UniqueCarrier]["Number_of_Flights"] += 1
        per_year[Year][UniqueCarrier]["Cancelled"] += int(Cancelled)
        #if Cancelled != "0": print "OK",Cancelled,per_year[Year][UniqueCarrier]["Cancelled"],Year,UniqueCarrier
            
    else:
        per_year[Year][UniqueCarrier] = {"Cancelled": 0, "Accumulated_Delay": int(DepDelay), "Distance": int(Distance),"Number_of_Flights": 1}
    return per_year
"""
for year in range(1987,2009):
    input_file = bz2.BZ2File(data_path + "{0}.csv.bz2".format(str(year)), 'rb')    
    for i,line in enumerate(input_file):
        #if i > 3000: break
        valores = line.split(",")
        if i == 0:
            idx_year = valores.index(Year)
            idx_DepDelay = valores.index(DepDelay)
            idx_UniqueCarrier = valores.index(UniqueCarrier)
            idx_Distance = valores.index(Distance)
            idx_Cancelled = valores.index(Cancelled)
            idx_CancellationCode = valores.index(CancellationCode)
            continue
        fill_Data(valores[idx_year],carrierfinder(valores[idx_UniqueCarrier]),valores[idx_DepDelay],valores[idx_Distance],valores[idx_Cancelled],valores[idx_CancellationCode])
#        print str(i) + " " + "Year: {0}, DepDelay: {1}, UniqueCarrier: {2}, Distance: {3}, Cancelled: {4}, CancelationCode: {5}".\
#        format(valores[idx_year],valores[idx_DepDelay],carrierfinder(valores[idx_UniqueCarrier]),valores[idx_Distance],valores[idx_Cancelled],valores[idx_CancellationCode])
    #break  

with open(data_path + 'data.txt', 'w') as outfile:
    json.dump(per_year, outfile)
"""
with open(data_path + 'data.txt','r') as data_file:    
    data = json.load(data_file)

  
field_names = ['Year', 'UniqueCarrier', 'Delay_per_Flight (minutes)', 'Distance', 'Number_of_Flights', 'Accumulated_Delay',"Cancelled", 'Cancel_per_Flight', 'Distance_per_Flight','Distance/Flights']
with open(data_path + 'output2.tsv', 'w') as output_file:
    dw = csv.DictWriter(output_file, fieldnames = field_names, delimiter='\t')
    dw.writeheader()
    for year in data.keys():
        for UniqueCarrier in data[year].keys():
            temp_row = data[year][UniqueCarrier]
            temp_row['Year'] = year
            temp_row['UniqueCarrier'] = UniqueCarrier
            temp_row['Delay_per_Flight (minutes)'] = round(float(temp_row['Accumulated_Delay'])/float(temp_row['Number_of_Flights']),2)
            temp_row['Cancel_per_Flight'] = round(float(temp_row['Cancelled'])/float(temp_row['Number_of_Flights']),2)
            temp_row['Distance_per_Flight'] = round(float(temp_row['Distance'])/float(temp_row['Number_of_Flights']),2)
            temp_row['Distance/Flights'] = round(float(temp_row['Distance'])/float(temp_row['Number_of_Flights']),2)
            dw.writerow(temp_row)
            
df = pandas.read_csv(data_path + 'output2.tsv',sep = "\t")
df_final =  df.groupby('Year').agg({'Distance_per_Flight' : np.mean, 'Number_of_Flights' : np.sum, 'UniqueCarrier': np.size})
df_final['Distance_per_Flight'] = df_final['Distance_per_Flight'].apply(lambda x: int(x))
df_final['Number_of_Flights'] = df_final['Number_of_Flights'].apply(lambda x: int(x))
df_final = df_final.reset_index()
df_final.to_json(data_path + 'output3.tsv',orient = 'records')

index.html

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
  </head>
<body>
  <div>
	<h1 style="text-align: center;"><span style="color:#696969;"><em><span style="font-size:18px;"><span style="font-family:comic sans ms,cursive;"><span style="background-color:#FFFFFF;">Commercial Aviation Evolution:</span><br />
	<span style="background-color:#FFFFFF;">Average Flight Distance and Number of Flights per Year</span></span></span></em></span></h1>
  </div>
  <div id="chartContainer">
  <script src="http://d3js.org/d3.v3.min.js"></script>
  <script src="http://dimplejs.org/dist/dimple.v2.1.2.min.js"></script>
  <script type="text/javascript">
      var svg = dimple.newSvg("#chartContainer", 1300, 550);
      d3.json("output3.tsv", function (data) {
			var myChart = new dimple.chart(svg, data);
			var x = myChart.addCategoryAxis("x", "Year");
			var y1 = myChart.addMeasureAxis("y", "Number_of_Flights");
			y1.overrideMin = 3000000;
			var y2 = myChart.addMeasureAxis("y", "Distance_per_Flight");
			y2.overrideMin = 500;
			// Plot the bars first - the order of series determines their dom position
			// from back to front, this means bars are at the back.
			var y3 = myChart.addMeasureAxis("y", "UniqueCarrier");
			var s3 = myChart.addSeries("Number of Airlines", dimple.plot.bar, [x, y3]);
			
			s3.afterDraw = function (shape, data) {
				var s = d3.select(shape),
					rect = {
						x: parseFloat(s.attr("x")),
						y: parseFloat(s.attr("y")),
						width: parseFloat(s.attr("width")),
						height: parseFloat(s.attr("height"))
					};
				if (rect.height >= 1) {
					svg.append("text")
						.attr("x", rect.x + rect.width / 2)
						.attr("y", rect.y + rect.height / 2)
						.attr("dy", "0.35em")
						.style("text-anchor", "middle")
						.style("font-size", "11px")
						.style("font-family", "sans-serif")
						.attr("fill", "white")
						.style("opacity", 0.8)
						.attr("dy", "0.35em")
						.text(d3.format(",")(data.yValue));
				}
			};
			
			var s1 = myChart.addSeries("Total Flights", dimple.plot.line, [x, y1]);
			var s2 = myChart.addSeries("Averaged Flight Distance", dimple.plot.line, [x, y2]);
			myChart.addLegend(100, 10, 1200, 60);
			myChart.assignColor("Number of Airlines", "teal", "teal", 0.50);
			myChart.assignColor("Total Flights", "red", "red", 0.95);
			myChart.assignColor("Averaged Flight Distance", "green", "green", 0.95);
			myChart.draw();
			d3.select("text[y = '275']").text("Total Flights (1m = 1000000)");
			d3.select("text[x = '1217']").text("Averaged Flight Distance in miles");
			d3.select("text[x = '1190']").remove();
			// Set some custom display elements for each series shape

      });
  </script>
</div>
</body>
</html>

index_previous.html

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
  </head>
<body>
  <div>
	<h1 style="text-align: center;"><span style="color:#696969;"><em><span style="font-size:18px;"><span style="font-family:comic sans ms,cursive;"><span style="background-color:#FFFFFF;">Commercial Aviation Evolution:</span><br />
	<span style="background-color:#FFFFFF;">Average Flight Distance and Number of Flights per Airline per Year</span></span></span></em></span></h1>
  </div>
  <div id="chartContainer">
  <script src="http://d3js.org/d3.v3.min.js"></script>
  <script src="http://dimplejs.org/dist/dimple.v2.1.2.min.js"></script>
  <script type="text/javascript">
      var svg = dimple.newSvg("#chartContainer", 1400, 650);
      d3.tsv("output_previous.tsv", function (data) {

          // Filter for 1 year
          data = dimple.filterData(data, "Year", [
              "1988", "1989", "1990", "1991", "1992",
              "1993", "1994", "1995", "1996", "1997", "1998",
			  "1999", "2000", "2001", "2002", "2003", "2004",
			  "2005", "2006", "2007", "2008"
          ]);
		  // console.table(data);
          // Create the indicator chart on the right of the main chart
          var indicator = new dimple.chart(svg, data);

          // Pick blue as the default and orange for the selected month
          var defaultColor = indicator.defaultColors[0];
          var indicatorColor = indicator.defaultColors[2];

          // The frame duration for the animation in milliseconds
		  var firstclick = false;

          // Place the indicator bar chart to the right
          indicator.setBounds(980, 150, 200, 311);

          // Add dates along the y axis
          var y = indicator.addCategoryAxis("y", "Year");
          y.addOrderRule("Date", "Desc");

          // Use sales for bar size and hide the axis
          var x = indicator.addMeasureAxis("x", "Distance");
          x.hidden = true;

          // Add the bars to the indicator and add event handlers
          var s = indicator.addSeries(null, dimple.plot.bar);
          s.addEventHandler("click", onClick);
          // Draw the side chart
          indicator.draw();

          // Remove the title from the y axis
          y.titleShape.remove();

          // Remove the lines from the y axis
          y.shapes.selectAll("line,path").remove();

          // Move the y axis text inside the plot area
          y.shapes.selectAll("text")
                  .style("text-anchor", "start")
                  .style("font-size", "11px")
                  .attr("transform", "translate(18, 0.5)");

          // This block simply adds the legend title. I put it into a d3 data
          // object to split it onto 2 lines.  This technique works with any
          // number of lines, it isn't dimple specific.
          svg.selectAll("title_text")
                  .data(["Click bar to select",
                      "and stop animation."])
                  .enter()
                  .append("text")
                  .attr("x", 980)
                  .attr("y", function (d, i) { return 110 + i * 12; })
                  .style("font-family", "sans-serif")
                  .style("font-size", "10px")
                  .style("color", "Black")
                  .text(function (d) { return d; });

          // Manually set the bar colors
          s.shapes
                  .attr("rx", 10)
                  .attr("ry", 10)
                  .style("fill", defaultColor.fill)
                  .style("stroke", defaultColor.stroke)
                  .style("opacity", 0.4);

		  // Draw the initial plot for the first year in the serie: 1988
		  var Year = 1988;
		  draw_airlines(Year.toString());
		  s.shapes
				  .style("fill", function (d) { return (d.y != Year ? defaultColor.fill : indicatorColor.fill) })
				  .style("stroke", function (d) { return (d.y != Year ? defaultColor.stroke : indicatorColor.stroke) });
				  
		  // Update plot every 2 seconds while user doesn't click on the bars:
		  setInterval(run_update, 2000);
		  
		  // Define update runner function:
		  function run_update(){
			if(!firstclick){
				Year++;
				if (Year > 2008){Year = 1988};
				update_chart(Year.toString());
				s.shapes
				  .style("fill", function (d) { return (d.y != Year ? defaultColor.fill : indicatorColor.fill) })
				  .style("stroke", function (d) { return (d.y != Year ? defaultColor.stroke : indicatorColor.stroke) });
			};
		  };
		  
          // Draw the main chart   Number of Flights   Delay_per_Flight (minutes)  Cancelled   Distance_per_Flight  Cancel_per_Flight
		  function draw_airlines(Year){
			  datay = dimple.filterData(data, "Year", [Year]);
			  bubbles = new dimple.chart(svg, datay);
			  bubbles.setBounds(70, 80, 900, 500)
			  var myXAxis = bubbles.addMeasureAxis("x", "Distance_per_Flight");	  
			  var myYAxis = bubbles.addMeasureAxis("y", "Number of Flights");
			  myXAxis.overrideMax = 1500 //1000000000;
			  myYAxis.overrideMax = 1300000;
			  //bubbles.addMeasureAxis("z", "Delay_per_Flight (minutes)");
			  bubbles.addSeries("UniqueCarrier", dimple.plot.bubble);
			  bubbles.addLegend(10, 10, 1200, 60,"UniqueCarrier");
			  bubbles.draw();
			  d3.select("text[y = '330']").text("Number of Flights (1m = 1000000)");
			  d3.select("text[x = '520']").text("Average Distance per Flight in k·miles ( 1k = 1000)");
			  //console.table(datay);
			  update_color();
		  };
		  
		  // Define update function:
	      function update_chart(Year){
		      d3.selectAll("g.dimple-legend").remove();
			  d3.selectAll("circle.dimple-bubble").remove();
			  d3.selectAll(".dimple-custom-axis-line").remove();
			  d3.selectAll("text[y='0'].dimple-custom-axis-label").remove()
			  d3.selectAll("text[x='0'].dimple-custom-axis-label").remove()
			  d3.selectAll(".dimple-gridline").remove();
			  d3.selectAll(".dimple-title").remove();
			  d3.selectAll(".dimple-tooltip").remove();
			  d3.selectAll("circle").remove();
			  d3.selectAll("line").remove();
			  draw_airlines(Year)
		      //bubbles.series.forEach(function(series){series.shapes.remove();});
			  //datay = dimple.filterData(data, "Year", [Year]);
			  //bubbles.data = datay;
			  //bubbles.addSeries("UniqueCarrier", dimple.plot.bubble);
			  //bubbles.draw();
			  //console.table(datay);
			  //update_color();
		  };

          // On click of the side chart
          function onClick(e) {
		    firstclick = true;
			s.shapes
				  .style("fill", function (d) { return (d.y != e.yValue ? defaultColor.fill : indicatorColor.fill) })
				  .style("stroke", function (d) { return (d.y != e.yValue ? defaultColor.stroke : indicatorColor.stroke) });
			update_chart(e.yValue.toString());
          };
		  
		  // Since d3 color palets are up to 20 elements and there are 29 Airlines, we need to create a custom palet:
		  function update_color(){
		    var airlines = ["dimple-arizona-pacific-inc-", "dimple-custom-air-service", "dimple-manu-a-air-transport", "dimple-dawn-air", "dimple-pan-american-world-airways--1-", "dimple-stateswest-airlines", "dimple-business-express-airlines", "dimple-air-cargo-enterprises-inc-", "dimple-american-airlines-inc-", "dimple-delta-air-lines-inc-", "dimple-alaska-airlines-inc-", "dimple-freedom-airlines-d-b-a-hp-expr", "dimple-northwest-airlines-inc-", "dimple-psa-airlines-inc-", "dimple-midway-airlines-inc---1-", "dimple-advance-air-charters", "dimple-multi-aer", "dimple-independence-air", "dimple-atlantic-southeast-airlines", "dimple-airtran-airways-corporation", "dimple-jetblue-airways", "dimple-skywest-airlines-inc-", "dimple-air-america-inc-", "dimple-trans-air--hawaii-", "dimple-acm-air-charter-gmbh", "dimple-comair-inc-", "dimple-frontier-airlines-inc-", "dimple-mesa-airlines-inc-", "dimple-pinnacle-airlines-inc-"];
		    var colors = d3.scale.category20().range().concat(d3.scale.category20b().range());
			var current_airlines_obj = d3.selectAll("g.dimple-legend")[0];
			var current_airlines = [];
			current_airlines_obj.forEach(function getairline(element){
													var new_airline = (element['classList'][1]);
													if (current_airlines.indexOf(new_airline) == -1){current_airlines.push(new_airline);}
													});
			for (var i = 0; i < current_airlines.length ; i++){
				var airline = current_airlines[i];
				var color = colors[airlines.indexOf(airline)];
				d3.select("rect." + airline).style('fill', color);
				d3.select("circle." + airline).attr('fill',color);
				d3.select("rect." + airline).style('stroke', color);
				d3.select("circle." + airline).attr('stroke',color);
				d3.select("circle." + airline).attr('r',40);
			};
		  };
		  

      });
  </script>
</div>
</body>
</html>

output3.tsv

[{"Year":1988,"Number_of_Flights":5202096,"UniqueCarrier":14,"Distance_per_Flight":561},{"Year":1989,"Number_of_Flights":5041200,"UniqueCarrier":13,"Distance_per_Flight":594},{"Year":1990,"Number_of_Flights":5270893,"UniqueCarrier":12,"Distance_per_Flight":622},{"Year":1991,"Number_of_Flights":5076925,"UniqueCarrier":12,"Distance_per_Flight":646},{"Year":1992,"Number_of_Flights":5092157,"UniqueCarrier":10,"Distance_per_Flight":648},{"Year":1993,"Number_of_Flights":5070501,"UniqueCarrier":10,"Distance_per_Flight":660},{"Year":1994,"Number_of_Flights":5180048,"UniqueCarrier":10,"Distance_per_Flight":670},{"Year":1995,"Number_of_Flights":5327435,"UniqueCarrier":10,"Distance_per_Flight":701},{"Year":1996,"Number_of_Flights":5351983,"UniqueCarrier":10,"Distance_per_Flight":722},{"Year":1997,"Number_of_Flights":5411843,"UniqueCarrier":10,"Distance_per_Flight":738},{"Year":1998,"Number_of_Flights":5384721,"UniqueCarrier":10,"Distance_per_Flight":756},{"Year":1999,"Number_of_Flights":5527884,"UniqueCarrier":10,"Distance_per_Flight":773},{"Year":2000,"Number_of_Flights":5683047,"UniqueCarrier":11,"Distance_per_Flight":732},{"Year":2001,"Number_of_Flights":5967780,"UniqueCarrier":12,"Distance_per_Flight":713},{"Year":2002,"Number_of_Flights":5271359,"UniqueCarrier":10,"Distance_per_Flight":775},{"Year":2003,"Number_of_Flights":6488540,"UniqueCarrier":18,"Distance_per_Flight":738},{"Year":2004,"Number_of_Flights":7129270,"UniqueCarrier":19,"Distance_per_Flight":743},{"Year":2005,"Number_of_Flights":7140596,"UniqueCarrier":20,"Distance_per_Flight":766},{"Year":2006,"Number_of_Flights":7141922,"UniqueCarrier":20,"Distance_per_Flight":760},{"Year":2007,"Number_of_Flights":7453215,"UniqueCarrier":20,"Distance_per_Flight":715},{"Year":2008,"Number_of_Flights":7009728,"UniqueCarrier":20,"Distance_per_Flight":717}]

output_previous.tsv