Map where Amazon orders go

README.md

Map where Amazon orders go

The Friends of the Alum Rock library sells books via Amazon to raise money for library programs. We thought it would be fun to see where we send the books.

fetch orders

Amazon makes it surprisingly hard to get programmatic access to order information. They do have an API (Amazon Marketplace Web Service), but it's only available to business sellers.

I wrote a Python script using mechanize that logs in to our seller account, goes to the order page, and clicks each of the orders to get to the order detail page. From there, I used regular expressions to extract the relevant order info (shipping address, date, price, and title). The HTML is not very well marked up, so this is likely to break. I use the MapQuest geocoding API to convert the mailing address to latitude/longitude so it can be mapped easily. I save this data in GeoJSON format, and update it whenever an order comes in. The orders.json file contains some sample data but is not updated automatically.

display orders

I used MapBox and Leaflet to display the GeoJSON orders data on a map, with a jQuery UI slider to filter orders by price.

I wanted custom markers created from GeoJSON data that can be filtered by the user. This was harder than I expected:

• adding markers with L.geoJson doesn't add them to a L.mapbox.featureLayer that can respond to events
• creating markers using L.mapbox.featureLayer doesn't allow customizing the markers
• L.mapbox.featureLayer.setFilter doesn't run for markers added manually to the feature layer

Here's how I got it working:

• create a feature layer with no data: var featureLayer = L.mapbox.featureLayer(null);
• load the geoJSON data with jQuery $.getJSON
• iterate through the data and create a CircleMarker for each feature, with color and size a function of age and sale price
• add the marker to the feature layer
• save the CircleMarker objects in an array keyed by orderID
• in the price filter slider's change function, clear all the markers from the feature layer
• for each feature that matches the selected price criteria, lookup the saved layer and add it to the feature layer

fetch_orders.py

import mechanize 
import json
import re
import requests
import datetime
from datetime import timedelta
from datetime import date

# scrape recent order data from Amazon Seller Central site

# load config
# { "geojson_fn" : "orders.json", "username" : "amazon_login", 
#   "password" : "amazon_password", "mapquest_key" : "mapquest API key" }
with open("config.json") as cf:
    config = json.load(cf)

MQ_URL = "http://www.mapquestapi.com/geocoding/v1/address?key="+config["mapquest_key"]

# load orders.json data
with open(config["geojson_fn"]) as jf:    
    orders = json.load(jf)

br = mechanize.Browser()  
br.set_handle_robots(False)  
br.addheaders = [("User-agent", "Mozilla/5.0")]  

# log in
sign_in = br.open("https://sellercentral.amazon.com/gp/homepage.html")  
br.select_form(name="signinWidget")  
br["username"] = config["username"]
br["password"] = config["password"]
logged_in = br.submit() 

# get orders for last 7 days
days = 7
week = timedelta(days=days)
today = date.today()
beginDate = (today-week).strftime("%m%%2F%d%%2F%y")
endDate = today.strftime("%m%%2F%d%%2F%y")
dayRange = str(days)

url = "https://sellercentral.amazon.com/gp/orders-v2/list?ie=UTF8&ajaxBelowTheFoldRows=0&byDate=orderDate&currentPage=1&exactDateBegin="+beginDate+"&exactDateEnd="+endDate+"&highlightOrderID=&isBelowTheFold=1&isDebug=0&isSearch=0&itemsPerPage=100&paymentFilter=Default&preSelectedRange="+dayRange+"&searchDateOption=preSelected&searchFulfillers=all&searchKeyword=&searchLanguage=en_US&searchType=0&shipExactDateBegin="+beginDate+"&shipExactDateEnd="+endDate+"&shipSearchDateOption=shipPreSelected&shipSelectedRange=7&shipmentFilter=Default&showCancelled=0&showPending=0&sortBy=OrderStatusDescending&statusFilter=Default"
orders_html = br.open(url)

# click order links to get to detail page
for l in br.links(url_regex='orders-v2/detail'):
    # get orderID from the URL
    m = re.search(".*?orderID=(\d+-\d+-\d+)", l.url)
    order_id = m.group(1)
    print order_id
    # continue if order is already in file
    if any(x for x in orders["features"] if x["properties"]["orderID"] == order_id):
        print "\talready have order_id %s" % order_id
        continue
    # go to order detail
    br.follow_link(l)
    data = br.response().get_data()
    # get address
    m = re.search('<td.*?>.*?Shipping Address:.*?<br>(.*)</td>', data, re.M)
    addr = m.group(1).replace('&nbsp;', ' ')
    lines = addr.split('<br>')
    address = ",".join(lines[1:])
    params = { "location" : address }
    # geocode
    r = requests.get(MQ_URL, params=params)
    geocode = json.loads(r.text)
    coord = geocode["results"][0]["locations"][0]["displayLatLng"]
    lng_lat = [coord["lng"], coord["lat"]]
    # get order date
    m = re.search('<td.*?>.*?Purchase Date:</td>.*?<td.*?>(.*?)</td>', data, re.M|re.S)
    dt = m.group(1)
    dtstr = datetime.datetime.strptime(dt, "%B %d, %Y %H:%M:%S %p %Z").strftime("%Y-%m-%d")
    # get price
    m = re.search('<td.*?>.*?Items total:</td>.*?<td.*?>\$(.*?)</td>', data, re.M|re.S)
    price = float(m.group(1))
    # get title
    for l in br.links(url_regex='/gp/product/'):
        title = l.text
    order = {
        "type" : "Feature",
        "geometry" : {
            "type" : "Point",
            "coordinates" : lng_lat
        },
        "properties" : {
            "orderID" : order_id,
            "orderDate" : dtstr,
            "price" : price,
            "title" : title
        }
    }
    # write to file
    orders["features"].append(order)

with open(config["geojson_fn"], "w") as jf:    
    json.dump(orders, jf, indent=2)

print "wrote %i orders to %s" % (len(orders), config["geojson_fn"])

index.html

<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<script src="http://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.6.0/underscore-min.js"></script>
<script src="http://cdnjs.cloudflare.com/ajax/libs/d3/3.4.2/d3.js"></script>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/2.0.3/jquery.min.js"></script>
<link rel="stylesheet" href="http://ajax.googleapis.com/ajax/libs/jqueryui/1.10.4/themes/smoothness/jquery-ui.css" />
<script src="http://ajax.googleapis.com/ajax/libs/jqueryui/1.10.4/jquery-ui.min.js"></script>
<script src="https://api.tiles.mapbox.com/mapbox.js/v1.6.1/mapbox.js"></script>
<link href="https://api.tiles.mapbox.com/mapbox.js/v1.6.1/mapbox.css" rel="stylesheet" />
<script src="http://cdnjs.cloudflare.com/ajax/libs/moment.js/2.5.1/moment.min.js"></script>
<style>
#map { 
  width: 600px;
  height: 400px;
}
.popupText {
  font-size: 11px;
}
.legend {
    font-size: 8px;
}
.legend i {
    float: left;
    height: 16px;
    margin-right: 5px;
    margin-top: 4px;
    opacity: 0.7;
    width: 16px;
}
.legend i.circle-4 {
  border-radius: 50%;
  width: 4px;
  height: 4px;
  margin-top: 8px;
}
.legend i.circle-6 {
  border-radius: 50%;
  width: 6px;
  height: 6px;
  margin-top: 8px;
}
.legend i.circle-8 {
  border-radius: 50%;
  width: 8px;
  height: 8px;
  margin-top: 8px;
}
.legend i.circle-10 {
  border-radius: 50%;
  width: 10px;
  height: 10px;
  margin-top: 8px;
}
.legend-block {
  float: left;
  padding-right: 5px;
}
.legend-header {
  font-weight: bold;
  height: 14px;
}
.legend-row {
  height: 20px;
}
#price-slider {
  width: 300px;
  margin-bottom: 0.5em;
  margin-left: 0.5em;
}
#price {
  border: 0; 
  font-weight: bold;
}
</style>
<script type="text/javascript">
jQuery(function($) {
  // want: custom markers from geoJSON data that can be filtered
  // 
  // adding markers with L.geoJson doesn't add them to a featureLayer that
  // can respond to events

  // creating them via featureLayer doesn't have a way to customize the markers

  // setFilter doesn't run for markers added manually to the feature layer

  // solution: keep a reference to the marker layers; clear layers on filter
  // and re-populate using saved layers

  var map = L.mapbox.map('map')
      .setView([37.8, -96], 4)
      .addLayer(L.mapbox.tileLayer("kielni.heic2ki4"));
  // add markers manually so they can be customized
  var featureLayer = L.mapbox.featureLayer(null);
  // hold references to layers since there's no way to hide/show them
  // other than add/remove
  var markers = {};

  var colors = ["#238b45", "#66c2a4", "#b2e2e2", "#edf8fb" ];
  var ageThresholds = [ 0, 7, 30, 90, 9999 ];
  var sizeReduce = [ 6, 4, 2, 0 ];
  var pricePoints = [ 0, 10, 30, 50, 9999 ];
  var today = moment();

  // load JSON data
  $.getJSON("orders.json", function(data) {
    // create a CircleMarker for each feature
    // color depends on age in days
    // size depends on sale price
    for (i = data.features.length - 1; i >= 0; i--) {
      var feature = data.features[i];
      var days = today.diff(moment(feature.properties.orderDate), 'days');
      var idx = 0;
      _.find(ageThresholds, function(age) {
        idx++;
        return days < age; 
      });
      idx = -1;
      _.find(pricePoints, function(pp) {
        idx++;
        return feature.properties.price < pp; 
      });
      var size = 10 - sizeReduce[idx-1];
      console.log("price="+feature.properties.price+" idx="+(idx-1)+" size="+size);
      var coords = feature.geometry.coordinates;
      var latlng = [coords[1], coords[0]];
      var marker = L.circleMarker(latlng, { 
        radius: size,
        fillColor: colors[idx-1],
        color: colors[idx-1],
        weight: 1,
        color: "#000",
        fillOpacity: 0.8
      });
      // popup with title, price, and date
      var popup = '<div class="popupText">'+feature.properties.title+"<br>"+
        moment(feature.properties.orderDate).format("MM/DD/YY")+
        " $"+feature.properties.price.toFixed(2)+"</div>";
      feature.properties.popupText = popup;
      marker.bindPopup(popup);
      // save the marker
      markers[feature.properties.orderID] = marker;
      featureLayer.addLayer(marker);
    }
    featureLayer.addTo(map);

    // legend
    var legendControl = L.control({position: "bottomleft"});
    legendControl.onAdd = function(map) {
      var div = L.DomUtil.create("div", "info legend");
      var text = '<div class="legend-header">Days ago</div>';
      for (var i = 0; i < colors.length; i++) {        
        text += '<div class="legend-row">';
        text += '<i style="background:' + colors[i] + '"></i> '+ageThresholds[i];
        if (i == ageThresholds.length-2) {
          text += "+";
        } else {
          text += " - "+ageThresholds[i+1];
        }
        text += " days</div>";
      }
      div.innerHTML = '<div class="legend-block">'+text+'</div>';
      text = '<div class="legend-header">Sale price</div>';
      var numLegendPrices = pricePoints.length-1;
      for (var i = 0; i < numLegendPrices; i++) {
        var size = 10-sizeReduce[i];
        text += '<div class="legend-row">';
        text += '<i class="legend-row circle-'+size+'" style="background:' + colors[0] + '"></i> $'+pricePoints[i].toFixed(2);
        if (i == numLegendPrices-1) {
          text += "+";
        } else {
          text += " - $"+pricePoints[i+1].toFixed(2);
        }
        text += "</div>";
      }
      div.innerHTML += '<div class="legend-block">'+text+'</div>';
      return div;
    };
    legendControl.addTo(map);

    // price range slider
    var minPrice = d3.min(data.features, function(d) { 
      return d.properties.price });
    var maxPrice = d3.max(data.features, function(d) { 
      return d.properties.price });
    $("#price-range").slider({
      range: true,
      min: Math.round(minPrice),
      max: Math.round(maxPrice),
      values: [ minPrice, maxPrice ],
      change: function(e, ui) {
        var minPrice = ui.values[0];
        var maxPrice = ui.values[1];
        $("#price").val("$"+minPrice+" - $"+maxPrice);
        // clear all
        featureLayer.clearLayers();
        _.each(data.features, function(feature) {
          var price = feature.properties.price;
          // add back from saved markers if price meets criteria
          if (price >= minPrice && price <= maxPrice) { 
            featureLayer.addLayer(markers[feature.properties.orderID] );
          }
        });
      }
    });
    var minPriceStr = "$"+$("#price-range").slider("values", 0);
    var maxPriceStr = "$"+$("#price-range").slider("values", 1);
    $("#price").val(minPriceStr+" - "+maxPriceStr);
  });

});
</script>
</head>
<body>
<h2>Where do our Amazon orders go?</h2>
<div id="price-slider">
  <label for="price-range">Show sale prices:</label>
  <input type="text" id="price">
  <div id="price-range"></div>
</div>
<div id="map"></div>
</body>
</html>

orders.json