server_survey.ipynb

{
 "metadata": {
  "name": "server_survey"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Survey of ECN support in web servers\n",
      "====================================\n",
      "\n",
      "© Electronics Research Group 2013.\n",
      "\n",
      "This notebook contains the Python code for performing a survey of ECN support in web servers.\n",
      "\n",
      "Dependencies\n",
      "------------\n",
      "\n",
      "- pydns (installable via pip)\n",
      "- tcpdump (must be able to capture packets, see http://www.stev.org/post/2012/01/19/Getting-tcpdump-to-run-as-non-root.aspx)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Configuration\n",
      "-------------\n",
      "\n",
      "In order to capture the correct packets, please set the interface that your default route is on and the IP address on that interface and then run the following block:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "NET_INTERFACE = \"eth0\"\n",
      "NET_ADDRESS = \"192.168.1.74\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": "*"
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Getting Ready\n",
      "-------------\n",
      "\n",
      "We'll begin by importing the required libraries and reading in the hosts we want to survey."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import DNS, socket, os, shlex\n",
      "from urllib2 import urlopen\n",
      "from IPython.display import HTML\n",
      "from subprocess import Popen\n",
      "from time import sleep\n",
      "\n",
      "hosts = []\n",
      "\n",
      "with open('hosts.txt') as f:\n",
      "    hosts = f.readlines()\n",
      "\n",
      "hosts = [x.strip() for x in hosts] # strip newlines"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "DNS Lookups\n",
      "-----------\n",
      "\n",
      "Now things are set up, we'll look up IP addresses for the hosts in DNS. We only care about IPv4 in this experiment."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "addresses = {}\n",
      "dnsFailures = 0\n",
      "\n",
      "for host in hosts:\n",
      "    if len(addresses.keys()) % 50 == 0:\n",
      "        sleep(0.2) # Let's not kill the network or the DNS server\n",
      "    try:\n",
      "        addresses[host] = DNS.dnslookup(host, 'A')[0]\n",
      "    except:\n",
      "        addresses[host] = '0.0.0.0'\n",
      "        dnsFailures += 1\n",
      "\n",
      "uniqueAddresses = set(addresses.values())\n",
      "\n",
      "if '0.0.0.0' in uniqueAddresses:\n",
      "    uniqueAddresses.remove('0.0.0.0')\n",
      "\n",
      "dnsTable = \"\"\"<table>\n",
      "    <tr><th>Total Domains</th><td>%s</td></tr>\n",
      "    <tr><th>Successfully Resolved</th><td>%d</td></tr>\n",
      "    <tr><th>Failures</th><td>%d</td></tr>\n",
      "    <tr><th>Unique Addresses</th><td>%d</td>\n",
      "    </table>\"\"\" % \\\n",
      "  ( len(hosts), len(hosts) - dnsFailures, dnsFailures, len(uniqueAddresses) )\n",
      "\n",
      "HTML(dnsTable)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We now have the IP addresses for the servers we'll be working with. It is likely that the last step is not *entirely* deterministic due to the ever changing nature of the Internet. In fact, it's unlikely that any two runs would produce the same list of addresses. The overall results should be similar though as it would be expected that all the servers that run a particular website identified by its DNS name would be configured in the same way."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Connecting without ECN\n",
      "----------------------\n",
      "\n",
      "Now let's see how many of these servers we can connect to successfully on TCP port 80 with ECN disabled. ECN client support should be disabled on the host before running this next step. On Linux:\n",
      "\n",
      "`echo 0 | sudo tee /proc/sys/net/ipv4/tcp_ecn`"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "noECNSuccess = {}\n",
      "noECNSuccesses = 0\n",
      "\n",
      "for address in uniqueAddresses:\n",
      "    if len(noECNSuccess.values()) % 50 == 0:\n",
      "        sleep(0.2) # Let's not kill the network\n",
      "    try:\n",
      "        s = socket.create_connection((address, 80))\n",
      "        noECNSuccess[address] = True\n",
      "        noECNSuccesses += 1\n",
      "    except:\n",
      "        noECNSuccess[address] = False\n",
      "    finally:\n",
      "        try:\n",
      "            s.close()\n",
      "        except:\n",
      "            pass\n",
      "\n",
      "noECNTable = \"\"\"<table>\n",
      "    <tr><th>Total Addresses</th><td>%d</td></tr>\n",
      "    <tr><th>Connection Succeeded</th><td>%d</td></tr>\n",
      "    <tr><th>Connection Failed</th><td>%d</td></tr>\n",
      "    </table>\"\"\" % \\\n",
      "    ( len(uniqueAddresses), noECNSuccesses, len(uniqueAddresses) - noECNSuccesses )\n",
      "    \n",
      "HTML(noECNTable)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Connecting with ECN\n",
      "-------------------\n",
      "\n",
      "In this next step is the heart of the experiment. For this next step, ECN support should be enabled. In Linux:\n",
      "\n",
      "`echo 1 | sudo tee /proc/sys/net/ipv4/tcp_ecn`\n",
      "\n",
      "__Warning:__ In the previous steps, dubious hosts (such as adult sites) may have been accessed, especially if you're using a list of top websites or a list of random unchecked websites, but no content was actually retrieved from them. In this step, it is necessary to retrieve content in order to have packets containing data (ECN doesn't set the ECT codepoint on TCP control packets). Make sure you've checked with your network operator that this is not going to get you fired.\n",
      "\n",
      "__Warning:__ In this step a packet capture will be performed. The filter will be set such that only HTTP traffic originating from the host you're running the script on will be captured but again, check with your network operator (especially if this is a shared host) that this is not going to get you fired."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ECNSuccess = {}\n",
      "ECNSuccesses = 0\n",
      "\n",
      "tcpdumpCmd = \"/usr/sbin/tcpdump -w survey.pcap -i %s '(src host %s and dst port 80) or (dst host %s and src port 80)'\" % \\\n",
      "    ( NET_INTERFACE, NET_ADDRESS, NET_ADDRESS )\n",
      "    \n",
      "tcpdump = Popen(shlex.split(tcpdumpCmd))\n",
      "\n",
      "for address in uniqueAddresses:\n",
      "    if len(ECNSuccess.values()) % 50 == 0:\n",
      "        sleep(0.3) # Let's not kill the network\n",
      "    try:\n",
      "        url = \"http://%s/\" % (address,)\n",
      "        u = urlopen(url)\n",
      "        u.read()\n",
      "        ECNSuccess[address] = True\n",
      "        ECNSuccesses += 1\n",
      "    except:\n",
      "        ECNSuccess[address] = False\n",
      "    finally:\n",
      "        try:\n",
      "            u.close()\n",
      "        except:\n",
      "            pass\n",
      "\n",
      "tcpdump.terminate()\n",
      "        \n",
      "ECNTable = \"\"\"<table>\n",
      "    <tr><th>Total Addresses</th><td>%d</td></tr>\n",
      "    <tr><th>Connection Succeeded</th><td>%d</td></tr>\n",
      "    <tr><th>Connection Failed</th><td>%d</td></tr>\n",
      "    </table>\"\"\" % \\\n",
      "    ( len(uniqueAddresses), ECNSuccesses, len(uniqueAddresses) - ECNSuccesses )\n",
      "    \n",
      "HTML(ECNTable)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Saving the data\n",
      "---------------\n",
      "\n",
      "After the last step you'll have in the current working directory a new file named `survey.pcap` which contains the packets sent and recieved with ECN enabled. Analysis of this will be able to determine how many servers provided ECN when it was requested. The final step here will write out the data collected during the experiment to a CSV file named `survey.csv`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "with open('survey.csv', 'w') as csv:\n",
      "\n",
      "    csv.write(\"Hostname,IP Address,NoECN,ECN\\n\")\n",
      "\n",
      "    for host in hosts:\n",
      "        if addresses[host] == '0.0.0.0':\n",
      "            csv.write(\"%s,0.0.0.0,0,0\\n\" % (host,))\n",
      "        else:\n",
      "            csv.write(\"%s,%s,%s,%s\\n\" % \\\n",
      "                (host, \\\n",
      "                 addresses[host], \\\n",
      "                 \"1\" if ECNSuccess[addresses[host]] else \"0\", \\\n",
      "                 \"1\" if noECNSuccess[addresses[host]] else \"0\"))\n",
      " "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The CSV file contains one row for each hostname provided at the start of the experiment. The columns are:\n",
      "\n",
      "- *Hostname*: The hostname\n",
      "- *IP Address*: The IP address that the hostname resolved to and was used for this experiment\n",
      "- *NoECN*: Whether or not a TCP connection could be established with ECN disabled (1 if it could, otherwise 0)\n",
      "- *ECN*: Whether or not a TCP connection could be established with ECN enabled (1 if it could, otherwise 0)"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}