abhigrover101/Project Amazon Fine Food Reviews.ipynb

## Project Amazon Fine Food Reviews.ipynb
{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import sqlite3\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk\n",
    "import string\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import scipy.sparse as sparse\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn import metrics\n",
    "from nltk.corpus import stopwords\n",
    "from sklearn.cross_validation import StratifiedKFold\n",
    "from sklearn.naive_bayes import BernoulliNB\n",
    "from sklearn import linear_model\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.metrics import recall_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Reading the data from the Sqlite file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## download dataset from this location : https://www.kaggle.com/snap/amazon-fine-food-reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sqlobject = sqlite3.connect('./database.sqlite')\n",
    "\n",
    "reviews = pd.read_sql_query(\"\"\"SELECT Score, Summary FROM Reviews\"\"\", sqlobject)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "original = reviews.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reviews = original.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(568454, 2)\n"
     ]
    }
   ],
   "source": [
    "print(reviews.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "reviews = reviews.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(568454, 2)\n"
     ]
    }
   ],
   "source": [
    "print (reviews.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## this is how loaded data looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Score                                            Summary\n",
      "0       5                              Good Quality Dog Food\n",
      "1       1                                  Not as Advertised\n",
      "2       4                              \"Delight\" says it all\n",
      "3       2                                     Cough Medicine\n",
      "4       5                                        Great taffy\n",
      "5       4                                         Nice Taffy\n",
      "6       5      Great!  Just as good as the expensive brands!\n",
      "7       5                             Wonderful, tasty taffy\n",
      "8       5                                         Yay Barley\n",
      "9       5                                   Healthy Dog Food\n",
      "10      5                    The Best Hot Sauce in the World\n",
      "11      5  My cats LOVE this \"diet\" food better than thei...\n",
      "12      1               My Cats Are Not Fans of the New Food\n",
      "13      4                                  fresh and greasy!\n",
      "14      5                       Strawberry Twizzlers - Yummy\n",
      "15      5           Lots of twizzlers, just what you expect.\n",
      "16      2                                         poor taste\n",
      "17      5                                           Love it!\n",
      "18      5                                 GREAT SWEET CANDY!\n",
      "19      5                            Home delivered twizlers\n",
      "20      5                                       Always fresh\n",
      "21      5                                          TWIZZLERS\n",
      "22      5                                 Delicious product!\n",
      "23      5                                          Twizzlers\n",
      "24      5                      Please sell these in Mexico!!\n"
     ]
    }
   ],
   "source": [
    "print (reviews.head(25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We can see above that there are no missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Encoding score to Positive or negative based on value of each sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "scores = reviews['Score']\n",
    "reviews['Score'] = reviews['Score'].apply(lambda x : 'pos' if x > 3 else 'neg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.183198640523243"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Distribution of labels in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Score\n",
       "neg    124677\n",
       "pos    443777\n",
       "Name: Summary, dtype: int64"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.groupby('Score')['Summary'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1cd2d019cf8>"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmYAAAGSCAYAAABJxw0YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+UX3V95/HnCxDQChjamtgEdK2JQrGLoUBb23W0W4Ja\nIXu2xVSt7JruejZYrLUewZ6SobRbYU8tS3tgu0pLoHgiZbe1FjYElbHVgiYCxTYYsl2DJDRRCYT6\no5aE9/7xvcGb6SQzk4yZz0yej3Pm5M77fj53Pt+ck5lX3vdzv5OqQpIkSdPviOlegCRJkgYMZpIk\nSY0wmEmSJDXCYCZJktQIg5kkSVIjDGaSJEmNMJhJmnZJ7krytkM9t5v/E0kePND5Y1zv9iS/0B1f\nmOSvpvDab0qyZqquJ6k9BjNJUybJl5K8ZrrXsUeSlUn+OcnO7uOLSX4vybw9Y6rq01V1ygSvdeN4\n46rqdVV1U790gGt/YZKnkzzzfbqqPlxV5x7I9STNDAYzSbPd6qo6ATgR+HfAPODzSeZO9RdKkqm8\nHINQN5XXlNQ4g5mk77okz0vysSRfSfJYdzx/1LCXJPls19n60yTP683/0SSfSfJ4kvuSvGqya6iq\n3VX1IPBG4KvAu7trvyrJI72v9d4kW5I8meTBJK9OsgR4H/DGJP+Y5L5u7F1JfjPJp5N8A/hXY9xa\nPaLr0j2RZEO/ozi6wziqK/ep7s8nurWcPfrWaJIfT/K57u/ls0l+rHfuriS/0a3tySRrkpw42b83\nSYeWwUzSoXAE8IfAScDJwDeB3x815heA/8Cgo7Ub+D2ALsD9BfAbVTUH+FXgfyX53gNZSFU9DXwU\n+Ml+uftai4CLgDOq6nhgCbC5qu4A/ivwkao6rqpe0Zv7FuAXgeOAL4/xJc8GNgHfCwwD/7sfOvfj\n33R/Hl9Vx1fVZ0etdQ6Dv5eru2v/LnBbV9/j54ELge8HjmHwdyepYQYzSd91VbWjqv60qr5dVd8A\nfpvvBI89bqqqB6vqW8CvAz/X3Rp8M3BbF46oqk8A64HXHcSSHmVwa3O03cDRwGlJjqqqL1fVl8a5\n1g1V9cWqerqqdo1xfntVXdN17G4BNgKvn8Ra93Ur8/XAQ92+s6erajXwReANvTF/VFV/X1XfBm4B\nTp/E15U0DQxmkr7rkjw7yR8k2ZzkCQa36Z43ak/WI73jh4FnAd8HvBC4IMmO7uNx4JUMOmsHaj6w\nY3Sxqv4e+GUGna3tST7cf1BgHx4Z5/zWUZ8/DPzABNe5Pz/QXWv0tfu3iLf1jr8JPHcKvq6k7yKD\nmaRD4d3AQuDMqnoe3+mW9YPZSb3jFwJPAV9jEHxurKoTu4853e3E/3YgC+nC4BuAvxzrfFWtrqqf\n7NYAcOWeU/u45HhPXY7eS3cyg44dwDeA5/TO9UPgeNd9FHjRGNceHQQlzSAGM0lT7egkx/Q+jmSw\n/+pbwJPdBvThMea9JcnLkjwHuBz4k6oq4I+BNyQ5J8kRSY7tNuxPtOsUgCRHJjkFWA3MZbAna++B\nyaJus//RwD93a366O70deNEBPHk5N8kvJTkqyc8BLwNu787dDyzrzv0I8LO9eV/tvvYP7uO6twML\nkyzrXtsbgVOAj01yfZIaYjCTNNVuY3Db7FvdnysZhKDnMOiA/TXfCSZ7FHATsIpBJ+ho4J0AVbUF\nOJ/BU5FfZXC77lf5zvev8TpLFyR5EngC+LPuGmdU1bYxxh4DvL8b8yiDTfOXduf+hEHIeyzJ+v18\n7dG1exh0C78GXAH8+6p6vDv368BLGNxWXQnc/MxFBnvtfgv4THcL96y9vkjVDuBnGPxdfK378/W9\nax/Q+6dJml4Z/Id0AgMHb3L4eeCRqjovyUrgPwFf6Ya8r6rWdGMvBd4G7ALeWVVru/pi4AbgWOD2\nqvrlrn40cCNwBoNvMG+sqi935y4Efo3BN5nfqqpx3+BRkiRpJppMx+ydwN+Nqn2gqhZ3H3tC2SnA\nBQxa6q8Fru21/q8DllfVImBR995AAMuBHVW1kMGj31d115oDXAacyeCR85VJTpjsi5QkSZoJJhTM\nkixg8Gj6h0afGmP4+QzeaXtXVW1m8P49Z3VPNh1XVeu6cTcCS3tzVnXHtwJ73nBxCbC2qnZW1RPA\nWsBfRyJJkmaliXbMfhd4D/9yz8I7ktyf5EO9TtZ89n58fGtXmw9s6dW38J2nlZ6ZU1W7gZ3dBuF9\nXUuSJGnWGTeYJXk9gzdIvJ+9O2TXAi+uqtMZvFfO70zhuvzdcJIk6bBz1ATGvBI4L8nrgGcDxyW5\nsare2hvzQb7ziPZW9n4/ogVdbV/1/pxHu0frj6+qHUm2AkOj5tw1eoFJfPpIkiTNGFU1ZhNqwk9l\nwuCX/QLv7p7KnLfncfMk72LwxpFvSnIqg0e+z2Zw2/FOYGFVVZJ7gIuBdQweqb+mqtYkWQGcVlUr\nkiwDllbVsm7z/3pgMYPu3noGj7k/MWpdNZnXocPb8PAww8PD070MSbOM31s0UUn2Gcwm0jHbl6uS\nnM7gDRA3A28HqKoNSW4BNjB45+4VvdR0EXu/Xcaarn49cFOSTcBjwLLuWo8nuYJBICvg8tGhTJIk\nabaYVDCrqk8x+B13jLqVOXrcbzP4JcWj658HXj5G/dsM3mJjrGvdwCDMSZIkzWq+878OO0NDQ9O9\nBEmzkN9bNBUmtcesVe4xk6RDZ96CeWzfun26l6EZYu78uWzbMtZvQDt87W+PmcFMkjQpScb+NfTS\nWIbBn9F7218w81amJElSIwxmkiRJjTCYSZIkNcJgJkmS1AiDmSRJUiMMZpIkSY0wmEmSJDXCYCZJ\nktQIg5kkSVIjDGaSJEmNMJhJkiQ1wmAmSZLUCIOZJElSIwxmkiRJjTCYSZIkNcJgJkmS1AiDmSRJ\nUiMMZpIkSY0wmEmSJDXCYCZJktQIg5kkSVIjDGaSJEmNMJhJkiQ1wmAmSZLUCIOZJElSIyYczJIc\nkeTeJH/efT4nydokG5PckeSE3thLk2xK8mCSc3r1xUkeSPJQkqt79aOTrO7m3J3k5N65C7vxG5O8\n9eBfsiRJUpsm0zF7J7Ch9/klwMer6qXAJ4FLAZKcClwAnAK8Frg2Sbo51wHLq2oRsCjJkq6+HNhR\nVQuBq4GrumvNAS4DzgTOBlb2A6AkSdJsMqFglmQB8DrgQ73y+cCq7ngVsLQ7Pg9YXVW7qmozsAk4\nK8k84LiqWteNu7E3p3+tW4HXdMdLgLVVtbOqngDWAudO/OVJkiTNHBPtmP0u8B6gerW5VbUdoKq2\nAc/v6vOBR3rjtna1+cCWXn1LV9trTlXtBnYmOXE/15IkSZp1xg1mSV4PbK+q+4HsZ2jt59xk7e/r\nSJIkzUpHTWDMK4HzkrwOeDZwXJKbgG1J5lbV9u425Ve68VuBk3rzF3S1fdX7cx5NciRwfFXtSLIV\nGBo1566xFjk8PPzM8dDQEENDQ2MNkyRJOqRGRkYYGRmZ0NhUTbzRleRVwLur6rwkVwGPVdWVSd4L\nzKmqS7rN/zcz2Kw/H7gTWFhVleQe4GJgHXAbcE1VrUmyAjitqlYkWQYsrapl3eb/9cBiBt299cAZ\n3X6z/rpqMq9DknTgksDwdK9CM8Yw+DN6b0moqjHvDk6kY7Yv7wduSfI24GEGT2JSVRuS3MLgCc6n\ngBW91HQRcANwLHB7Va3p6tcDNyXZBDwGLOuu9XiSKxgEsgIuHx3KJEmSZotJdcxaZcdMkg4dO2aa\nlGE7ZqPtr2PmO/9LkiQ1wmAmSZLUCIOZJElSIwxmkiRJjTCYSZIkNcJgJkmS1AiDmSRJUiMMZpIk\nSY0wmEmSJDXCYCZJktQIg5kkSVIjDGaSJEmNMJhJkiQ1wmAmSZLUCIOZJElSIwxmkiRJjTCYSZIk\nNcJgJkmS1AiDmSRJUiMMZpIkSY0wmEmSJDXCYCZJktQIg5kkSVIjDGaSJEmNMJhJkiQ1wmAmSZLU\nCIOZJElSIwxmkiRJjTCYSZIkNWLcYJbkmCSfTXJfki8kWdnVVybZkuTe7uPc3pxLk2xK8mCSc3r1\nxUkeSPJQkqt79aOTrO7m3J3k5N65C7vxG5O8depeuiRJUluOGm9AVX07yaur6ptJjgQ+k+T/dKc/\nUFUf6I9PcgpwAXAKsAD4eJKFVVXAdcDyqlqX5PYkS6rqDmA5sKOqFiZ5I3AVsCzJHOAyYDEQ4PNJ\nPlpVO6fm5UuSJLVjQrcyq+qb3eExDMJcdZ9njOHnA6uraldVbQY2AWclmQccV1XrunE3Akt7c1Z1\nx7cCr+mOlwBrq2pnVT0BrAWe6cxJkiTNJhMKZkmOSHIfsA24sxeu3pHk/iQfSnJCV5sPPNKbvrWr\nzQe29Opbutpec6pqN7AzyYn7uZYkSdKsM9GO2dNV9QoGtybPSnIqcC3w4qo6nUFg+50pXNdYnThJ\nkqRZbdw9Zn1V9WSSEeDcUXvLPgh8rDveCpzUO7egq+2r3p/zaLeP7fiq2pFkKzA0as5dY61teHj4\nmeOhoSGGhobGGiZJknRIjYyMMDIyMqGxGezJ38+A5PuAp6pqZ5JnA3cA7wfurapt3Zh3AWdW1Zu6\nbtrNwNkMbjveCSysqkpyD3AxsA64DbimqtYkWQGcVlUrkiwDllbVns3/6xls/j+iOz6j22/WX2ON\n9zokSVMjCQxP9yo0YwyDP6P3loSqGvPu4EQ6Zi8AViU5gkE4+khV3Z7kxiSnA08Dm4G3A1TVhiS3\nABuAp4AVvdR0EXADcCxwe1Wt6erXAzcl2QQ8BizrrvV4kisYBLICLh8dyiRJkmaLcTtmM4EdM0k6\ndOyYaVKG7ZiNtr+Ome/8L0mS1AiDmSRJUiMMZpIkSY0wmEmSJDXCYCZJktQIg5kkSVIjDGaSJEmN\nMJhJkiQ1wmAmSZLUCIOZJElSIwxmkiRJjTCYSZIkNcJgJkmS1AiDmSRJUiMMZpIkSY0wmEmSJDXC\nYCZJktQIg5kkSVIjDGaSJEmNMJhJkiQ1wmAmSZLUCIOZJElSIwxmkiRJjTCYSZIkNcJgJkmS1AiD\nmSRJUiMMZpIkSY0wmEmSJDVi3GCW5Jgkn01yX5IvJFnZ1eckWZtkY5I7kpzQm3Npkk1JHkxyTq++\nOMkDSR5KcnWvfnSS1d2cu5Oc3Dt3YTd+Y5K3Tt1LlyRJasu4wayqvg28uqpeAZwOvDbJWcAlwMer\n6qXAJ4FLAZKcClwAnAK8Frg2SbrLXQcsr6pFwKIkS7r6cmBHVS0Ergau6q41B7gMOBM4G1jZD4CS\nJEmzyYRuZVbVN7vDY4CjgALOB1Z19VXA0u74PGB1Ve2qqs3AJuCsJPOA46pqXTfuxt6c/rVuBV7T\nHS8B1lbVzqp6AlgLnDupVyhJkjRDTCiYJTkiyX3ANuDOLlzNrartAFW1DXh+N3w+8Ehv+tauNh/Y\n0qtv6Wp7zamq3cDOJCfu51qSJEmzzkQ7Zk93tzIXMOh+/RCDrtlew6ZwXRl/iCRJ0uxy1GQGV9WT\nSUYY3E7cnmRuVW3vblN+pRu2FTipN21BV9tXvT/n0SRHAsdX1Y4kW4GhUXPuGmttw8PDzxwPDQ0x\nNDQ01jBJkqRDamRkhJGRkQmNTdX+G11Jvg94qqp2Jnk2cAfwfuBVDDbsX5nkvcCcqrqk2/x/M4PN\n+vOBO4GFVVVJ7gEuBtYBtwHXVNWaJCuA06pqRZJlwNKqWtZt/l8PLGbQ3VsPnNHtN+uvscZ7HZKk\nqZEEhqd7FZoxhsGf0XtLQlWNeXdwIh2zFwCrkhzBIBx9pKpu70LWLUneBjzM4ElMqmpDkluADcBT\nwIpearoIuAE4Fri9qtZ09euBm5JsAh4DlnXXejzJFQwCWQGXjw5lkiRJs8W4HbOZwI6ZJB06dsw0\nKcN2zEbbX8fMd/6XJElqhMFMkiSpEQYzSZKkRhjMJEmSGmEwkyRJaoTBTJIkqREGM0mSpEYYzCRJ\nkhphMJMkSWqEwUySJKkRBjNJkqRGGMwkSZIaYTCTJElqhMFMkiSpEQYzSZKkRhjMJEmSGmEwkyRJ\naoTBTJIkqREGM0mSpEYYzCRJkhphMJMkSWqEwUySJKkRBjNJkqRGGMwkSZIaYTCTJElqhMFMkiSp\nEQYzSZKkRhjMJEmSGjFuMEuyIMknk/xdki8k+aWuvjLJliT3dh/n9uZcmmRTkgeTnNOrL07yQJKH\nklzdqx+dZHU35+4kJ/fOXdiN35jkrVP30iVJktpy1ATG7AJ+paruT/Jc4PNJ7uzOfaCqPtAfnOQU\n4ALgFGAB8PEkC6uqgOuA5VW1LsntSZZU1R3AcmBHVS1M8kbgKmBZkjnAZcBiIN3X/mhV7Tz4ly5J\nktSWcTtmVbWtqu7vjr8OPAjM705njCnnA6uraldVbQY2AWclmQccV1XrunE3Akt7c1Z1x7cCr+mO\nlwBrq2pnVT0BrAWe6cxJkiTNJpPaY5bkRcDpwGe70juS3J/kQ0lO6GrzgUd607Z2tfnAll59C98J\neM/MqardwM4kJ+7nWpIkSbPOhINZdxvzVuCdXefsWuDFVXU6sA34nSlc11idOEmSpFltInvMSHIU\ng1B2U1V9FKCqvtob8kHgY93xVuCk3rkFXW1f9f6cR5McCRxfVTuSbAWGRs25a6w1Dg8PP3M8NDTE\n0NDQWMMkSZIOqZGREUZGRiY0NoM9+eMMSm4EvlZVv9Krzauqbd3xu4Azq+pNSU4FbgbOZnDb8U5g\nYVVVknuAi4F1wG3ANVW1JskK4LSqWpFkGbC0qvZs/l/PYPP/Ed3xGd1+s/76aiKvQ5J08JLA8HSv\nQjPGMPgzem9JqKox7w6O2zFL8krgzcAXktwHFPA+4E1JTgeeBjYDbweoqg1JbgE2AE8BK3qp6SLg\nBuBY4PaqWtPVrwduSrIJeAxY1l3r8SRXMAhkBVw+OpRJkiTNFhPqmLXOjpkkHTp2zDQpw3bMRttf\nx8x3/pckSWqEwUySJKkRBjNJkqRGGMwkSZIaYTCTJElqhMFMkiSpEQYzSZKkRhjMJEmSGmEwkyRJ\naoTBTJIkqREGM0mSpEYYzCRJkhphMJMkSWqEwUySJKkRBjNJkqRGGMwkSZIaYTCTJElqhMFMkiSp\nEQYzSZKkRhjMJEmSGmEwkyRJaoTBTJIkqREGM0mSpEYYzCRJkhphMJMkSWqEwUySJKkRBjNJkqRG\nGMwkSZIaMW4wS7IgySeT/F2SLyS5uKvPSbI2ycYkdyQ5oTfn0iSbkjyY5JxefXGSB5I8lOTqXv3o\nJKu7OXcnObl37sJu/MYkb526ly5JktSWiXTMdgG/UlU/BPwYcFGSlwGXAB+vqpcCnwQuBUhyKnAB\ncArwWuDaJOmudR2wvKoWAYuSLOnqy4EdVbUQuBq4qrvWHOAy4EzgbGBlPwBKkiTNJuMGs6raVlX3\nd8dfBx4EFgDnA6u6YauApd3xecDqqtpVVZuBTcBZSeYBx1XVum7cjb05/WvdCrymO14CrK2qnVX1\nBLAWOPdAXqgkSVLrJrXHLMmLgNOBe4C5VbUdBuENeH43bD7wSG/a1q42H9jSq2/panvNqardwM4k\nJ+7nWpIkSbPOhINZkucy6Ga9s+uc1aghoz8/GBl/iCRJ0uxy1EQGJTmKQSi7qao+2pW3J5lbVdu7\n25Rf6epbgZN60xd0tX3V+3MeTXIkcHxV7UiyFRgaNeeusdY4PDz8zPHQ0BBDQ0NjDZMkSTqkRkZG\nGBkZmdDYVI3f6EpyI/C1qvqVXu1KBhv2r0zyXmBOVV3Sbf6/mcFm/fnAncDCqqok9wAXA+uA24Br\nqmpNkhXAaVW1IskyYGlVLes2/68HFjPo7q0Hzuj2m/XXVxN5HZKkg5cEhqd7FZoxhsGf0XtLQlWN\neXdw3I5ZklcCbwa+kOQ+Brcs3wdcCdyS5G3AwwyexKSqNiS5BdgAPAWs6KWmi4AbgGOB26tqTVe/\nHrgpySbgMWBZd63Hk1zBIJAVcPnoUCZJkjRbTKhj1jo7ZpJ06Ngx06QM2zEbbX8dM9/5X5IkqREG\nM0mSpEYYzCRJkhphMJMkSWqEwUySJKkRBjNJkqRGGMwkSZIaYTCTJElqhMFMkiSpEQYzSZKkRhjM\nJEmSGmEwkyRJaoTBTJIkqREGM0mSpEYYzCRJkhphMJMkSWqEwUySJKkRBjNJkqRGGMwkSZIaYTCT\nJElqhMFMkiSpEQYzSZKkRhjMJEmSGmEwkyRJaoTBTJIkqREGM0mSpEYYzCRJkhphMJMkSWrEuMEs\nyfVJtid5oFdbmWRLknu7j3N75y5NsinJg0nO6dUXJ3kgyUNJru7Vj06yuptzd5KTe+cu7MZvTPLW\nqXnJkiRJbZpIx+yPgCVj1D9QVYu7jzUASU4BLgBOAV4LXJsk3fjrgOVVtQhYlGTPNZcDO6pqIXA1\ncFV3rTnAZcCZwNnAyiQnHMiLlCRJmgnGDWZV9Wng8TFOZYza+cDqqtpVVZuBTcBZSeYBx1XVum7c\njcDS3pxV3fGtwGu64yXA2qraWVVPAGuBZzpzkiRJs83B7DF7R5L7k3yo18maDzzSG7O1q80HtvTq\nW7raXnOqajewM8mJ+7mWJEnSrHSgwexa4MVVdTqwDfidqVvSmJ04SZKkWe+oA5lUVV/tffpB4GPd\n8VbgpN65BV1tX/X+nEeTHAkcX1U7kmwFhkbNuWtfaxoeHn7meGhoiKGhoX0NlSRJOmRGRkYYGRmZ\n0NhU1fiDkhcBH6uql3efz6uqbd3xu4Azq+pNSU4FbmawWX8+cCewsKoqyT3AxcA64Dbgmqpak2QF\ncFpVrUiyDFhaVcu6zf/rgcUMOnvrgTO6/Waj11cTeR2SpIOXBIanexWaMYbBn9F7S0JVjXmHcNyO\nWZIPM+hcfW+SLwMrgVcnOR14GtgMvB2gqjYkuQXYADwFrOglpouAG4Bjgdv3PMkJXA/clGQT8Biw\nrLvW40muYBDICrh8rFAmSZI0W0yoY9Y6O2aSdOjYMdOkDNsxG21/HTPf+V+SJKkRBjNJkqRGGMwk\nSZIaYTCTJElqhMFMkiSpEQYzSZKkRhjMJEmSGmEwkyRJaoTBTJIkqREGM0mSpEYYzCRJkhphMJMk\nSWqEwUySJKkRBjNJkqRGGMwkSZIacdR0L0DfPS+aN4+Ht2+f7mVohnjh3Lls3rZtupchSYc1g9ks\n9vD27dR0L0IzRgzxkjTtvJUpSZLUCIOZJElSIwxmkiRJjTCYSZIkNcJgJkmS1AiDmSRJUiMMZpIk\nSY0wmEmSJDXCYCZJktQIg5kkSVIjDGaSJEmNMJhJkiQ1YtxgluT6JNuTPNCrzUmyNsnGJHckOaF3\n7tIkm5I8mOScXn1xkgeSPJTk6l796CSruzl3Jzm5d+7CbvzGJG+dmpcsSZLUpol0zP4IWDKqdgnw\n8ap6KfBJ4FKAJKcCFwCnAK8Frk2Sbs51wPKqWgQsSrLnmsuBHVW1ELgauKq71hzgMuBM4GxgZT8A\nSpIkzTbjBrOq+jTw+Kjy+cCq7ngVsLQ7Pg9YXVW7qmozsAk4K8k84LiqWteNu7E3p3+tW4HXdMdL\ngLVVtbOqngDWAudO4rVJkiTNKAe6x+z5VbUdoKq2Ac/v6vOBR3rjtna1+cCWXn1LV9trTlXtBnYm\nOXE/15IkSZqVpmrzf03RdQAy/hBJkqTZ56gDnLc9ydyq2t7dpvxKV98KnNQbt6Cr7aven/NokiOB\n46tqR5KtwNCoOXfta0HDw8PPHA8NDTE0NLSvoZIkSYfMyMgIIyMjExqbqvGbXUleBHysql7efX4l\ngw37VyZ5LzCnqi7pNv/fzGCz/nzgTmBhVVWSe4CLgXXAbcA1VbUmyQrgtKpakWQZsLSqlnWb/9cD\nixl09tYDZ3T7zUavrybyOg43Saa0lanZLYD/jjQRSWB4ulehGWPY7y2jJaGqxrxDOG7HLMmHGXSu\nvjfJl4GVwPuBP0nyNuBhBk9iUlUbktwCbACeAlb0EtNFwA3AscDtVbWmq18P3JRkE/AYsKy71uNJ\nrmAQyAq4fKxQJkmSNFtMqGPWOjtmY7NjpsmwY6aJsmOmSRn2e8to++uY+c7/kiRJjTCYSZIkNcJg\nJkmS1AiDmSRJUiMMZpIkSY0wmEmSJDXCYCZJktQIg5kkSVIjDGaSJEmNMJhJkiQ1wmAmSZLUCIOZ\nJElSIwxmkiRJjTCYSZIkNcJgJkmS1AiDmSRJUiMMZpIkSY0wmEmSJDXCYCZJktQIg5kkSVIjDGaS\nJEmNMJhJkiQ1wmAmSZLUCIOZJElSIwxmkiRJjTCYSZIkNcJgJkmS1AiDmSRJUiMOKpgl2Zzkb5Lc\nl+RzXW1OkrVJNia5I8kJvfGXJtmU5MEk5/Tqi5M8kOShJFf36kcnWd3NuTvJyQezXkmSpJYdbMfs\naWCoql5RVWd1tUuAj1fVS4FPApcCJDkVuAA4BXgtcG2SdHOuA5ZX1SJgUZIlXX05sKOqFgJXA1cd\n5HolSZKadbDBLGNc43xgVXe8CljaHZ8HrK6qXVW1GdgEnJVkHnBcVa3rxt3Ym9O/1q3ATx3keiVJ\nkpp1sMGsgDuTrEvyi11tblVtB6iqbcDzu/p84JHe3K1dbT6wpVff0tX2mlNVu4Enkpx4kGuWJElq\n0lEHOf+VVfUPSb4fWJtkI4Ow1jf684OR8YdIkiTNTAcVzKrqH7o/v5rkz4CzgO1J5lbV9u425Ve6\n4VuBk3rTF3S1fdX7cx5NciRwfFXtGGstw8PDzxwPDQ0xNDR0MC9NkiRpSoyMjDAyMjKhsak6sIZW\nkucAR1TV15N8D7AWuJzBPrAdVXVlkvcCc6rqkm7z/83A2QxuUd4JLKyqSnIPcDGwDrgNuKaq1iRZ\nAZxWVSuSLAOWVtWyMdZSB/o6ZrMkU9qu1OwWwH9HmogkMDzdq9CMMez3ltGSUFVj3gU8mI7ZXOBP\nk1R3nZuram2S9cAtSd4GPMzgSUyqakOSW4ANwFPAil6augi4ATgWuL2q1nT164GbkmwCHgP+RSiT\nJEmaLQ64Y9YSO2Zjs2OmybBjpomyY6ZJGfZ7y2j765j5zv+SJEmNMJhJkiQ1wmAmSZLUCIOZJElS\nIwxmkiRFoPgUAAAF7ElEQVRJjTCYSZIkNcJgJkmS1AiDmSRJUiMMZpIkSY0wmEmSJDXCYCZJktQI\ng5kkSVIjDGaSJEmNMJhJkiQ1wmAmSZLUCIOZJElSIwxmkiRJjTCYSZIkNcJgJkmS1AiDmSRJUiMM\nZpIkSY0wmEmSJDXCYCZJktQIg5kkSVIjDGaSJEmNMJhJkiQ1wmAmSZLUCIOZJElSI2ZEMEtybpIv\nJnkoyXunez2SJEnfDc0HsyRHAL8PLAF+CPj5JC+b3lVpJhuZ7gVImp2+NN0L0GzQfDADzgI2VdXD\nVfUUsBo4f5rXpBlsZLoXIGl22jzdC9BsMBOC2Xzgkd7nW7qaJEnSrDITgpkkSdJh4ajpXsAEbAVO\n7n2+oKvtJckhW9BM4t/K2C6f7gU0yn9HmrDh6V5Aoz413Qtok99bJi5VNd1r2K8kRwIbgZ8C/gH4\nHPDzVfXgtC5MkiRpijXfMauq3UneAaxlcOv1ekOZJEmajZrvmEmSJB0u3PwvSZLUCIOZJElSIwxm\nkiQdoCRXJTk+ybOSfCLJV5O8ZbrXpZnLYKbDQpJ/TPLkqI9HkvxpkhdP9/okzVjnVNWTwM8weO//\nlwDvmdYVaUZr/qlMaYpczeC3RnyYwdu7LQN+ELgX+ENgaNpWJmkm2/Nz9PXAn1TVTt+zSwfDpzJ1\nWEjyN1X1r0fV7q+q08c6J0kTkeT9wFLgWwx+t/PzgL+oqrOndWGasbyVqcPFN5NckOSI7uMC4J+6\nc/7vRNIBqapLgB8HfqSqngK+AZw/vavSTGbHTIeFbh/Zfwd+jEEQuwd4F4Nf73VGVX16GpcnaYZK\n8izgvwD/pit9CvgfXUiTJs1gJknSAUryIeBZwKqu9AvA7qr6xelblWYyg5kOC0kWAdcBc6vqtCQ/\nDJxXVb85zUuTNIPtY/+q+1Z1wNxjpsPFB4FLgacAquoBBk9mStLB2J3kB/d80m2b2D2N69EM59tl\n6HDxnKr63KjH2HdN12IkzRrvAe5K8v+6z18E/MfpW45mOjtmOlx8rftfbQEk+VngH6Z3SZJmgc8A\nfwA8Dezoju+e1hVpRnOPmQ4L3e2F/8ngsfbHgS8Bb66qh6d1YZJmtCS3AE8CN3elNwHPq6qfm75V\naSYzmOmwkOQY4GcZ3GY4kcE30qqq35jOdUma2ZJsqKpTx6tJE+WtTB0uPgq8gcHm/0eBrzN4I0hJ\nOhj3JvnRPZ8kORtYP43r0Qxnx0yHhSR/W1WnTfc6JM0uSR4EXgp8uSudDGxk8HBRVdUPT9faNDP5\nVKYOF3+d5OVV9YXpXoikWeXc6V6AZhc7ZjosJNkAvITBpv9vA8H/zUqSGmMw02EhyQvHqvtUpiSp\nJQYzSZKkRvhUpiRJUiMMZpIkSY0wmEmSJDXCYCbpsJHk15L8bZK/SXJvkjOne02S1Of7mEk6LHTv\nzv464PSq2pXkRODog7jekVW1e8oWKEnYMZN0+HgB8LWq2gVQVTuqaluSM5N8Jsn9Se5J8j1Jjkny\nh0keSPL5JEMASS5M8tEknwA+3tV+Ncnnuvkrp+3VSZoV7JhJOlysBS5L8kXgE8BHgLuB1cDPVdW9\nSZ4L/BPwTuDpqvrhJC8F1iZZ2F3nFcDLq2pnkp8GFlbVWUkC/HmSn6iqTx/qFydpdrBjJumwUFXf\nABYD/xn4KoNA9nbg0aq6txvz9e725E8Af9zVNgKbgUXdpe6sqp3d8TnATye5F7iXwe9M3BPgJGnS\n7JhJOmzU4B21/xL4yyRfAC6a4NT0jr8xqv7bVfXBKVqipMOcHTNJh4Uki5K8pFc6HdgAvCDJj3Rj\nnpvkSOCvgDfvmQecBGwc47J3AG9L8j3d2B9I8v3fxZchaZazYybpcPFc4PeSnADsAv4vg9uafwT8\nfpJnA98E/i1wLXBdkgeAp4ALq+qpwTay76iqO5O8DLi7O/ePwFsY3CqVpEnzd2VKkiQ1wluZkiRJ\njTCYSZIkNcJgJkmS1AiDmSRJUiMMZpIkSY0wmEmSJDXCYCZJktQIg5kkSVIj/j//dYobTjU0EwAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd2d019518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "reviews.groupby('Score')['Summary'].count().plot(kind='bar',color=['r','g'],title='Label Distribution',figsize=(10,6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of negative reviews 21.93 %\n",
      "Percentage of positive reviews 78.07 %\n"
     ]
    }
   ],
   "source": [
    "print ('Percentage of negative reviews %.2f %%' % ((reviews.groupby('Score')['Summary'].count()['neg'])*100.0/len(reviews)))\n",
    "print ('Percentage of positive reviews %.2f %%' % ((reviews.groupby('Score')['Summary'].count()['pos'])*100.0/len(reviews)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### As we can see the data is pretty skewed. We are able to already decide that we must use Stratified Split for evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Splitting the dataset based on labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def splitPosNeg(Summaries):\n",
    "    neg = reviews.loc[Summaries['Score']=='neg']\n",
    "    pos = reviews.loc[Summaries['Score']=='pos']\n",
    "    return [pos,neg]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "[pos,neg] = splitPosNeg(reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 5. Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using lemmatization as it was giving better results than stemming. Other steps include removing punctuation amd upper case to lower case conversion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#preprocessing steps\n",
    "\n",
    "#stemmer = PorterStemmer()\n",
    "lemmatizer = nltk.WordNetLemmatizer()\n",
    "stop = stopwords.words('english')\n",
    "translation = str.maketrans(string.punctuation,' '*len(string.punctuation))\n",
    "\n",
    "def preprocessing(line):\n",
    "    tokens=[]\n",
    "    line = line.translate(translation)\n",
    "    line = nltk.word_tokenize(line.lower())\n",
    "    for t in line:\n",
    "        #if(t not in stop):\n",
    "            #stemmed = stemmer.stem(t)\n",
    "        stemmed = lemmatizer.lemmatize(t)\n",
    "        tokens.append(stemmed)\n",
    "    return ' '.join(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pos_data = []\n",
    "neg_data = []\n",
    "for p in pos['Summary']:\n",
    "    pos_data.append(preprocessing(p))\n",
    "\n",
    "for n in neg['Summary']:\n",
    "    neg_data.append(preprocessing(n))    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = pos_data + neg_data\n",
    "labels = np.concatenate((pos['Score'].values,neg['Score'].values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Splitting the data into train and test using a stratified strategy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "[Data_train,Data_test,Train_labels,Test_labels] = train_test_split(data,labels , test_size=0.25, random_state=20160121,stratify=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## tokenizing the training data to find frequency of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "t = []\n",
    "for line in Data_train:\n",
    "    l = nltk.word_tokenize(line)\n",
    "    for w in l:\n",
    "        t.append(w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27048\n"
     ]
    }
   ],
   "source": [
    "word_features = nltk.FreqDist(t)\n",
    "print (len(word_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We can see that the number of unique words in the dataset is pretty huge. When doing text classification the vocabulary of the data set becomes the featureset.\n",
    "### We would want to reduce the size of this feature set. This can be done in severals ways.\n",
    "### One of them is feature reduction using PCA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Feature Reduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a. Using PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vec_all = CountVectorizer()\n",
    "ctr_features_all = vec_all.fit_transform(Data_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf_vec_all = TfidfTransformer()\n",
    "tr_features_all = tf_vec_all.fit_transform(ctr_features_all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cte_features_all = vec_all.transform(Data_test)\n",
    "te_features_all = tf_vec_all.transform(cte_features_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pca doesn't apply to sparse matrices so we use a different version of PCA called Truncated SVD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "svd = TruncatedSVD(n_components=200)\n",
    "tr_features_truncated = svd.fit_transform(tr_features_all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "te_features_truncated = svd.transform(te_features_all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "svd = TruncatedSVD(n_components=200)\n",
    "ctr_features_truncated = svd.fit_transform(ctr_features_all)\n",
    "cte_features_truncated = svd.transform(cte_features_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running 4 classification algorithms : Naive Bayes, Logistic Regression, Decision Trees and Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "models = {'BernoulliNB':BernoulliNB(binarize=0.5)\n",
    "          ,'Logistic' : linear_model.LogisticRegression(C=1e5),'Decision Tree' : DecisionTreeClassifier(random_state=20160121, criterion='entropy'),\n",
    "                                                                                                        'Perceptron': linear_model.Perceptron(n_iter=1000)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "           Model  TF-IDF Accuracy\n",
      "0  Decision Tree         0.858473\n",
      "1    BernoulliNB         0.793455\n",
      "2       Logistic         0.844667\n",
      "3     Perceptron         0.814593\n"
     ]
    }
   ],
   "source": [
    "results_svd = pd.DataFrame()\n",
    "\n",
    "foldnum = 0\n",
    "tfprediction = {}\n",
    "cprediction = {}\n",
    "for name,model in models.items():\n",
    "        model.fit(tr_features_truncated, Train_labels)\n",
    "        tfprediction[name] = model.predict(te_features_truncated)\n",
    "        tfaccuracy = metrics.accuracy_score(tfprediction[name],Test_labels)\n",
    "        \n",
    "        #model.fit(ctr_features_truncated,Train_labels)\n",
    "        #cprediction[name] = model.predict(cte_features_truncated)\n",
    "        #caccuracy = metrics.accuracy_score(cprediction[name],Test_labels)\n",
    "\n",
    "        results_svd.loc[foldnum,'Model']=name\n",
    "        results_svd.loc[foldnum,'TF-IDF Accuracy']=tfaccuracy\n",
    "        #results_svd.loc[foldnum,'Count Accuracy']=caccuracy\n",
    "        foldnum = foldnum+1\n",
    "print (results_svd)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for  Decision Tree\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.68      0.68      0.68     31169\n",
      "        pos       0.91      0.91      0.91    110945\n",
      "\n",
      "avg / total       0.86      0.86      0.86    142114\n",
      "\n",
      "\n",
      "\n",
      "Classification report for  BernoulliNB\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.88      0.07      0.13     31169\n",
      "        pos       0.79      1.00      0.88    110945\n",
      "\n",
      "avg / total       0.81      0.79      0.72    142114\n",
      "\n",
      "\n",
      "\n",
      "Classification report for  Logistic\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.76      0.42      0.54     31169\n",
      "        pos       0.86      0.96      0.91    110945\n",
      "\n",
      "avg / total       0.84      0.84      0.83    142114\n",
      "\n",
      "\n",
      "\n",
      "Classification report for  Perceptron\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.84      0.19      0.31     31169\n",
      "        pos       0.81      0.99      0.89    110945\n",
      "\n",
      "avg / total       0.82      0.81      0.77    142114\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for name,model in models.items():\n",
    "    print (\"Classification report for \",name)\n",
    "    print(metrics.classification_report(Test_labels, tfprediction[name]))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1cd69c1ac18>"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAHDCAYAAADr+CQ1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucZVV95/3Pt8ELKqAMBh7BhggJKFEiKvGJjhYyCpoL\n6kSFeEkgY4iRiWOcCWqcpIhGjeNkzIgmkBAH9RmIjknAS5BoutRMQkS5aJJuIYotdxUiooC0ze/5\n4+yCQ3m66yy6uvY+3Z/361Uvzt57nX1+VXU4/a211l47VYUkSZKmt6bvAiRJkmaNAUqSJKmRAUqS\nJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUpSsyQHJLkrybKfIUl+IclnVqmuByb5cJJvJfmz1XhNSTsn\nA5S0g0vy1SR3JNlryf5LuxC09j6eumURuYltx4LYt7uvryQ59T7WA/BzwMOBh1XVi7bhPJK0VQYo\nacdXwFXACYs7kvwYsBttIWh7KWDPqtoD+Hngt5I8q/UkXW/YAcAVdR9WCE6yS+tzJO28DFDSzuF9\nwC+Mbf8CcPZ4gyR7JHlvkq8nuSrJb44dW5Pk7Um+keRfgJ+a8Nw/SXJdkquTvDFJGuoLQFVdBPwT\n8GPdeQ9NcmGSm5KsT/KCsdd8T5J3J/lokluBTwG/BRzf9WadmJE3dL1wNyT5X0n26J6/2Pt1UpKN\nwCfH9v1ikq91r3tykicmuTzJzUneOVbDo5J8Msk3u5/b+xfP3x2/Kslruuf+a5Jzktx/7PhxXU/g\nLUmuXAyOK/DzlLSdGaCkncNFwO5JDul6al4EvJ8uuHROB3YHDgTmgJclObE79svAc4DDgScyGiob\ndzZwJ/Ao4PHAM4H/0FBfAJI8BXgMcEmSBwEXdnXuDRwPvDvJoWPPOwF4Y1XtDhwNvBk4t6r2qKr3\nACcCLwOe3tW2e/d9jnsacChwzNi+I4GDGf2c3gG8HngGo2D3wiT/dqzuNwP7Ao8G9gfml5z/BcCz\ngB9m9PP7xe57PZLRz+01VbVnV8dXu+ds689T0nZmgJJ2Hou9UM8E1gPXLR4YC1Wvrarbqmoj8N+B\nl3ZNXgC8o6quq6pvAW8Ze+4+wLOBV1fVHVX1TUah4+4hw2UE+EaSm4AzgVOragH4aeCqqnpvjVwO\nfKirZdF5Xa8VVXXnhHP/PPD7VbWxqm4DXseoh2rxs6+A366q26vqe2P7fqeq7qyqTwDfBc6pqpuq\n6jrgM4xCDVX15ar6ZFV9v6puAv4Ho7A27g+q6sbu5/Zh4Me7/ScBZ1XV33Tnur6qrkjyQ9v485S0\nCnbtuwBJq+b9wKcZ9YS8d8mxvRl9HnxtbN9GYL/u8SOAq5ccW7QWuB9wfTfKlO5r/FxbU8C/mTBv\n6QDgyUlu7rYD7LKk9qvZukcsqXUjo+9zn7F910x43tfHHt8O3Lhk+yEAXdj5A+Dfdvt2AW7m3saf\nexvw/3SPHwl8dMJrH8C2/TwlrQIDlLSTqKqvJbmKUe/GSUsOfxPYxOgf7w3dvgOAa7vH1zP6B5+x\nY4uuBu5gcgiaVvjBCe1XAwtVdcyE9ouWe73ruHetBzD6Pm/knu9nWybSvxm4Czisqm5JchzwzmWe\ns+hq4KAt7N/Wn6ek7cwhPGnnchLwjKq6fXxnVd0FfAD43SQPSXIA8GpGw350x34tyX5JHgacOvbc\nGxjNVfofSXbvJm4/KsnTpqxpS5OjPwL8aJKXJNk1yf26ydyHTP3dwjnAq5McmOQhwO8ymiN111Ze\nu2Wy9u7Ad4Bbk+wH/JeG554FnJjkqO5n9ogkh6zAz1PSKjBASTu+u3sxquqqqrpk0jHg1xgNMX2F\n0VDf+7uJ2AB/DHwcuBz4HKO5SONeBtwf+GdGQ1gfZDSxuqm+e+2s+g6jydfHM+pJug54K/CAKc8L\n8KeMQuCngS8z+v5+bZnXXrpva9unAU8AFuc3Lf25bLEHqaouZjTJ/R3ALcACo+FQ2Lafp6RVkGl6\niJMcy+h/8jWMJj3+3pLjD2X0QXUQo/kBJ1XVP698uZIkSf2b5jYMaxhd9nsMcBhwwpLLiGF0ie+l\nVXU4o6t8/udKFypJkjQU0wzhHQlc2V0GvAk4FzhuSZvHAIuX4n4JODDJw1e0UkmSpIGYJkDtx70v\nFb6Gey5tXnQ58Hy4e3G4tYwWlJMkSdrhrNQk8rcCD0tyCfBK4FJg8wqdW5IkaVCmWQfqWu65MgRG\nPUvXjjeoqlsZW1emW2vmK0tPlMQ1TSRJ0syoqolLm0wToC4GDu7Whbme0SXF97qlQJI9gduqalOS\nlwOf6i5BnlRIU+E7g/n5eebn5/suQzPA94pa+H7RtHyvTLa1e3gvG6CqanOSUxgt7La4jMH6JCeP\nDteZjG6ieXaSuxjdSf2XVqRySZKkAZrqVi5VdQFwyJJ9Z4w9vmjpcUmSpB2VK5EPwNzcXN8laEb4\nXlEL3y+alu+VdlOtRL5iL5Z4b0xJkjQTkmxxErk9UJIkSY0MUJIkSY0MUJIkSY0MUJIkSY0MUJIk\nSY0MUJIkSY0MUJIkSY0MUJIkSY0MUJIkSY2muhfejujAffdl44039l3G4Bywzz589YYb+i5DkqRB\n22lv5ZKEYVQyLAGG8juSJKlP3spFkiRpBRmgJEmSGhmgJEmSGu20k8glaXvYd98DufHGjX2XMTj7\n7HMAN9zw1b7LkFaMk8h1L04il7ZNEvDTZYL42aKZ4yRySZKkFWSAkiRJamSAkiRJamSAkiRJamSA\nkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJajRVgEpybJINSa5IcuqE43skOT/JZUm+\nmOQXV7xSSZJ2MGvX7ksSv8a+1q7dt+9fy1SWvRdekjXAFcDRwHXAxcDxVbVhrM3rgD2q6nVJ9ga+\nBOxTVd9fci7vhTdw3gtP2jbeC29LvBfeJElYt67vKoblqKOG8+/Qtt4L70jgyqraWFWbgHOB45a0\nKWD37vHuwE1Lw5MkSdKOYpoAtR9w9dj2Nd2+cacDj0lyHXA58KqVKU+SJGl4VmoS+THApVX1CODx\nwLuSPGSFzi1JkjQou07R5lpg7dj2/t2+cScCbwGoqi8nuQo4FPjc0pPNz8/f/Xhubo65ubmmgiVJ\nkraHhYUFFhYWpmo7zSTyXRhNCj8auB74LHBCVa0fa/Mu4OtVdVqSfRgFp8Or6uYl53IS+cA5iVza\nNk4i3xInkU/iJPIfNCuTyJftgaqqzUlOAS5kNOR3VlWtT3Ly6HCdCbwJ+F9JvtA97TeWhidplu27\n/77ceO2NfZcxKPvstw83XHND32VIUi+W7YFa0RezB2rw7IGaLAnM913FwMz7XpnEHqgtsQdqEnug\nftCs9EC5ErkkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5Qk\nSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIj\nA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5Qk\nSVIjA5QkSVKjqQJUkmOTbEhyRZJTJxz/z0kuTXJJki8m+X6Sh658uZIkSf1bNkAlWQOcDhwDHAac\nkOTQ8TZV9faqenxVHQG8Dlioqm9tj4IlSZL6Nk0P1JHAlVW1sao2AecCx22l/QnAOStRnCRJ0hBN\nE6D2A64e276m2/cDkuwGHAt8aNtLkyRJGqZdV/h8PwP87daG7+bn5+9+PDc3x9zc3AqXIEmS1G5h\nYYGFhYWp2qaqtt4geTIwX1XHdtuvBaqqfm9C2z8HPlBV527hXLXc662WJAyjkmEJMJTf0ZAkgfm+\nqxiYed8rkyQBP10miO+XCZKwbl3fVQzLUUcN57MlCVWVScemGcK7GDg4yQFJ7g8cD5w/4UX2BJ4O\nnLctxUqSJA3dskN4VbU5ySnAhYwC11lVtT7JyaPDdWbX9LnAx6vq9u1XriRJUv+mmgNVVRcAhyzZ\nd8aS7bOBs1euNEmSpGFyJXJJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRG\nBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJ\nkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRG\nBihJkqRGBihJkqRGUwWoJMcm2ZDkiiSnbqHNXJJLk/xjknUrW6YkSdJw7LpcgyRrgNOBo4HrgIuT\nnFdVG8ba7Am8C3hWVV2bZO/tVbAkSVLfpumBOhK4sqo2VtUm4FzguCVtfh74UFVdC1BV31zZMiVJ\nkoZjmgC1H3D12PY13b5xPwrslWRdkouTvHSlCpQkSRqaZYfwGs5zBPAM4MHA3yf5+6r6l6UN5+fn\n7348NzfH3NzcCpUgSZJ03y0sLLCwsDBV22kC1LXA2rHt/bt9464BvllVdwB3JPk0cDiw1QAlSZI0\nFEs7dk477bQttp1mCO9i4OAkByS5P3A8cP6SNucBT02yS5IHAT8BrG+sW5IkaSYs2wNVVZuTnAJc\nyChwnVVV65OcPDpcZ1bVhiQfB74AbAbOrKp/3q6VS5Ik9WSqOVBVdQFwyJJ9ZyzZfjvw9pUrTZIk\naZhciVySJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmR\nAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqS\nJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKnR\nVAEqybFJNiS5IsmpE44/Pcm3klzSfb1h5UuVJEkahl2Xa5BkDXA6cDRwHXBxkvOqasOSpp+uqp/d\nDjVKkiQNyjQ9UEcCV1bVxqraBJwLHDehXVa0MkmSpIGaJkDtB1w9tn1Nt2+p/zfJZUk+muQxK1Kd\nJEnSAC07hDelzwNrq+q2JM8G/hL40RU6tyRJ0qBME6CuBdaObe/f7btbVX1n7PFfJXl3kr2q6ual\nJ5ufn7/78dzcHHNzc40lS5IkrbyFhQUWFhamapuq2nqDZBfgS4wmkV8PfBY4oarWj7XZp6pu7B4f\nCXygqg6ccK5a7vVWSxKGUcmwBBjK72hIksB831UMzLzvlUmSgJ8uE8T3ywRJWLeu7yqG5aijhvPZ\nkoSqmjjHe9keqKranOQU4EJGc6bOqqr1SU4eHa4zgZ9L8gpgE3A78KKVK1+SJGlYppoDVVUXAIcs\n2XfG2ON3Ae9a2dIkSZKGyZXIJUmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmS\nGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmg\nJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmSGhmgJEmS\nGhmgJEmSGhmgJEmSGk0VoJIcm2RDkiuSnLqVdk9KsinJ81euREmSpGFZNkAlWQOcDhwDHAackOTQ\nLbR7K/DxlS5SkiRpSKbpgToSuLKqNlbVJuBc4LgJ7f4j8H+Ar69gfZIkSYMzTYDaD7h6bPuabt/d\nkjwCeG5V/SGQlStPkiRpeFZqEvk7gPG5UYYoSZK0w9p1ijbXAmvHtvfv9o17InBukgB7A89Osqmq\nzl96svn5+bsfz83NMTc311iyJEnSyltYWGBhYWGqtqmqrTdIdgG+BBwNXA98FjihqtZvof17gA9X\n1Z9POFbLvd5qScIwKhmWAEP5HQ1JEpjvu4qBmfe9Msno70h/Lj8ovl8mSMK6dX1XMSxHHTWcz5Yk\nVNXEUbVle6CqanOSU4ALGQ35nVVV65OcPDpcZy59yjZXLEmSNGDTDOFRVRcAhyzZd8YW2p60AnVJ\nkiQNliuRS5IkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIk\nNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJA\nSZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIkNTJASZIk\nNZoqQCU5NsmGJFckOXXC8Z9NcnmSS5N8NslTVr5USZKkYdh1uQZJ1gCnA0cD1wEXJzmvqjaMNftE\nVZ3ftX8s8AHg0duhXkmSpN5N0wN1JHBlVW2sqk3AucBx4w2q6raxzYcAd61ciZIkScMyTYDaD7h6\nbPuabt+9JHlukvXAh4GTVqY8SZKk4VmxSeRV9ZdV9WjgucCbVuq8kiRJQ7PsHCjgWmDt2Pb+3b6J\nqupvkzwqyV5VdfPS4/Pz83c/npubY25ubupiJUmStpeFhQUWFhamapuq2nqDZBfgS4wmkV8PfBY4\noarWj7U5qKq+3D0+Ajivqh454Vy13OutliQMo5JhCTCU39GQJIH5vqsYmHnfK5MkAT9dJojvlwmS\nsG5d31UMy1FHDeezJQlVlUnHlu2BqqrNSU4BLmQ05HdWVa1PcvLocJ0J/PskLwPuBG4HXrhy5UuS\nJA3LNEN4VNUFwCFL9p0x9vhtwNtWtjRJkqRhciVySZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKk\nRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYo\nSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKk\nRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRlMFqCTHJtmQ5Iokp044/vNJLu++/jbJY1e+\nVEmSpGFYNkAlWQOcDhwDHAackOTQJc2+Ajytqg4H3gT88UoXKkmSNBTT9EAdCVxZVRurahNwLnDc\neIOquqiqbuk2LwL2W9kyJUmShmOaALUfcPXY9jVsPSD9B+CvtqUoSZKkIdt1JU+W5CjgROCpW2oz\nPz9/9+O5uTnm5uZWsgRJkqT7ZGFhgYWFhanapqq23iB5MjBfVcd2268Fqqp+b0m7xwEfAo6tqi9v\n4Vy13OutliQMo5JhCTCU39GQJIH5vqsYmHnfK5MkAT9dJojvlwmSsG5d31UMy1FHDeezJQlVlUnH\nphnCuxg4OMkBSe4PHA+cv+QF1jIKTy/dUniSJEnaUSw7hFdVm5OcAlzIKHCdVVXrk5w8OlxnAv8V\n2At4d0Z/fm2qqiO3Z+GSJEl9mWoOVFVdAByyZN8ZY49fDrx8ZUuTJEkaJlcilyRJamSAkiRJamSA\nkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJ\namSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSA\nkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJamSAkiRJajRVgEpybJINSa5IcuqE44ck\n+bskdyT59ZUvU5IkaTh2Xa5BkjXA6cDRwHXAxUnOq6oNY81uAv4j8NztUqUkSdKATNMDdSRwZVVt\nrKpNwLnAceMNquqbVfV54PvboUZJkqRBmSZA7QdcPbZ9TbdPkiRpp7TsEN5Km5+fv/vx3Nwcc3Nz\nq12CJEnSD1hYWGBhYWGqttMEqGuBtWPb+3f77pPxACVJkjQUSzt2TjvttC22nWYI72Lg4CQHJLk/\ncDxw/lbaZ7oyJUmSZtOyPVBVtTnJKcCFjALXWVW1PsnJo8N1ZpJ9gM8BuwN3JXkV8Jiq+s72LF6S\nJKkPU82BqqoLgEOW7Dtj7PGNwCNXtjRJkqRhciVySZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKk\nRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYo\nSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKk\nRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRlMFqCTHJtmQ5Iokp26hzf9McmWSy5L8+MqWKUmSNBzL\nBqgka4DTgWOAw4ATkhy6pM2zgYOq6keAk4E/2g617rAW+i5As+OqvgvQbFnouwDNiMsu67uC2TNN\nD9SRwJVVtbGqNgHnAsctaXMc8F6AqvoHYM8k+6xopTuwhb4L0Oz4at8FaLYs9F2AZoQBqt00AWo/\n4Oqx7Wu6fVtrc+2ENpIkSTsEJ5FLkiQ1SlVtvUHyZGC+qo7ttl8LVFX93libPwLWVdWfddsbgKdX\n1Y1LzrX1F5MkSRqQqsqk/btO8dyLgYOTHABcDxwPnLCkzfnAK4E/6wLXt5aGp60VIUmSNEuWDVBV\ntTnJKcCFjIb8zqqq9UlOHh2uM6vqY0mek+RfgO8CJ27fsiVJkvqz7BCeJEmS7s1J5JIkSY0MUD1I\nsluS13WT70lycLcYqSRJmgEGqH78KRDgqd32dcCb+ytHsyLJ3km8GENblOSVSR46tv2wJL/aZ03S\njsg5UD1I8rmqemKSS6vq8d2+y6rKewjqbt0VrW8FbgbeCLwP2JvRHz4vq6oLeixPAzXps2T8s0Za\nlOThwMuBAxm7qKyqTuqrplkyzTIGWnl3JnkgUABJfhi4s9+SNECnA68H9gT+Bnh2VV3U3YvyHMAA\npUl2SZLq/jpOsgtw/55r0jCdB3wG+ASwuedaZo4Bqh+/w+gfv/2TnA08HfilfkvSAO1aVRcCJPmd\nqroIoKo2OIqnrbiA0Zp8Z3TbJ2PY1mQPqqpT+y5iVhmgelBVFyT5PPCTjOZC/Zeq+nrPZWl47hp7\nfPuSY469a0tOZRSaXtFt/zXwJ/2VowH7SJLnVNXH+i5kFjkHqidJjgcOqqrfTfJI4Ieq6vN916Xh\nSLKZ0cK0AXYDbls8BDywqu7XV22SZl+SW4EHM5pCsqnbXVW1R39VzQ4DVA+SnA7cD3haVT06yV7A\nx6vqST2XJmlGJflAVb0wyReZ0ENZVY/roSxph+UQXj9+sqqOSHIpQFXdnMRJnpK2xau6//50r1Vo\npiT5WeBp3eZCVX2kz3pmietA9WNTkjXccxXev+He810kktya5Nvdf28d274tyff7rk/DUlXXdw9/\ntao2jn8BrgOlH5DkrYyC9z93X69K8pZ+q5odDuH1IMnLgOcBT2S0qOYLgdOq6txeC9OgJXkI8EpG\nE4T/oqpe03NJGqAkl1TVEUv2fcEhPC2V5AvAj1fVXd32LsClvlem4xBeD6rqvd1VeP+O0YTgF1TV\nP/ZclgaqW1X6PwEvA/438KSquqnfqjQ0SV7BqKfpoO4fxkW7A/+3n6o0Ax7KaLFeGK05pykZoFZZ\nl/C/UFWHAf/Udz0ariR7A68BXsSop/LxVXVLv1VpwP438FfAW4DXju2/tapunvwU7eTeAlyaZB2j\nP+afxr3fO9oKh/B6kOTDwK9U1bV916LhSvJd4BvAe4Bblx6vqt9f9aI0eEkOAq6pqu8lmQMeB7y3\nqr7Vb2Uaku6emvsD3wcWrwD/bFXd0F9Vs8UA1YMu7T8B+HtG6/wAUFXP760oDU6SebayYGZVnbZ6\n1WhWJLmM0fzKA4GPMbpdx2FV9Zw+69LwJPliVT227zpmlUN4/XhT3wVo+Kpqvu8aNJPuqqrvJ3k+\n8M6qeufikinSEpckeVJVXdx3IbPIALWKklxYVc+qqk/2XYuGL8lvVNXbkryTyQsj/loPZWn4NiU5\ngdFFBz/T7XPVek3yE8CLk2zknrselFfhTccAtboe3ncBminru/9+rtcqNGtOBH4F+N2quirJDwPv\n67kmDdMxfRcwy5wDtYqSfAX4z1s6XlV/vorlSJJ2YkneV1UvXW6fJrMHanXtyeg2C5lwrAADlO7W\nXa25tUnkP7uK5WjgvBee7oPDxje6ZXae0FMtM8cAtbo2VtVJfRehmfH2vgvQTPFeeJpKktcBrwd2\nS/Jt7vmj/k7gzN4KmzEO4a2iJJdW1eP7rkOSpCRvqarX9V3HrDJAraIkP+YtWzStLQ3FLHJIRpMk\nuZUffN/cwuhihNdU1VdWvyoNUbeY5vOApzJ6z3ymqv6y36pmhwFKGqgkB2zteFVtXK1aNDuSvBG4\nhtGtXQIcDxwEXAK8oqrm+qtOQ5Lk3cDBwDndrhcBX66qV/ZX1ewwQEnSDiTJ5VV1+JJ9l1XVj086\npp1Xkg3Ao6sLAknWAP9UVY/ut7LZsKbvAiRtXZJbk3y7+7ojyeZu4qc0yW1JXphkTff1QuCO7ph/\nMWvcvwBrx7Yf2e3TFLwKrwdJngLMAwcw+h0srv76qD7r0jBV1e6Lj7s5C8cBT+6vIg3ci4E/AN7d\nbf898JIkuwGn9FaVhmh3YH2SzzIK10cCn0tyPrhUynIcwutB1236auDzwObF/VV1U29FaaZ4Raek\nbZXk6Vs7XlWfWq1aZpE9UP24par+qu8iNBu6m8IuWgM8kXuGZKR7SbI/8E7gKd2uzwCvqqpr+qtK\nQ1RVn+ouVvmRqvpE10u5a1Xd2ndts8AA1Y91Sf4bo5XHv7e4s6ou6a8kDdjPjD3+PvBVRsN40iTv\nYXQF3gu67Zd0+57ZW0UapCQvB34Z2IvRlZr7A38EHN1nXbPCIbweJFk3YXdV1TNWvRhJO5TFK+6W\n2ycluYzRvKd/WJwSkOSLVfXYfiubDfZA9aCqjuq7Bs0Oh2TU6KYkL+GetX1OAJxfqUm+V1V3jq5N\ngSS74pXp/xaYAAAHhUlEQVSaU3MZgx4k2TPJ7yf5XPf135Ps2XddGqz3AOcDj+i+PtztkyY5CXgh\ncANwPfBzwC/2WZAG61NJFu+J90zgg4w+XzQFh/B6kORDwD8CZ3e7XgocXlXP3/KztLNySEbbKsl/\nqqp39F2HhqVbOPOXgGcxWk7n48CflMFgKgaoHvgPolok+SSjHqfxIZkTq8qJnppKkq9V1drlW2pn\nkuTBwB1Vtbnb3gV4QFXd1m9ls8EhvH7cnuSpixvdwpq391iPhm3SkMyJvVakWZO+C9AgfRLYbWx7\nN+ATPdUyc5xE3o9XAGd3854C3IxzFLQF3U2DXRFY28KhBk3ywKr6zuJGVX0nyYP6LGiWGKB6UFWX\nAYcn2aPb9r5m2qIkDwdeDhzI2P+zVXVSXzVpeJLcyuSgFO7dyyAt+m6SIxbXIEzyBBwNmZoBahUl\neUlVvT/Jry/ZD0BV/X4vhWnozmO0dMEnGLv1jzRu/J6J0pReBXwwyXWMgva+wIv6LWl2GKBW14O7\n//pBpxYPqqpT+y5C0o6juwLv/sChwCHd7i9V1ab+qpotXoUnDVySNwF/V1Uf67sWSTsOb0q+bbwK\nrwdJ3pZkjyT3S/LJJN/oVg6WJnkV8JEktyf5dpJbkzhvTtK2+mSSf5/FeSRqYg9UDxbXfEryPOCn\ngV8HPl1Vh/dcmgam+2B7ZFV9re9aJO1YugsPHsxobuXtjOZBVVXt0WthM8I5UP1Y/Ln/FPDBqrrF\nPwA0SVVVko8C3txT0orywoNt4xBePz6SZAPwBEZdqA8H7ui5Jg3XJUme1HcRknYsGXlJkv/abT8y\nyZF91zUrHMLrSZK9gFuqanO3cNkeVXVD33VpeLqwfTCwEfgu93SzP67XwiTNtCR/CNwFPKOqHp3k\nYcCFVeUfbFNwCG8VJXlGVf1NkueP7Rtv8uerX5VmwDF9FyBph/QTVXVEkksBqupfk9y/76JmhQFq\ndT0d+BvgZyYcKwxQmqCqNnb3TvyRqnpPN+T7kL7rkjTzNnU3EC64+64Hd/Vb0uxwCE8auCS/DTwR\nOKSqfjTJIxhdfPCUnkuTNMOSvJjRyuNHAGczulH5G6rqg70WNiOcRN6DJG9O8tCx7Yd1iyVKkzyP\n0c2EvwtQVdfhavaStlFV/X/AbwBvAa4Hnmt4mp5DeP14dlW9fnGjG3d+DvCGHmvScN3ZLWew2M3+\n4OWeIElbkuSBwK8wujjli8AZVfX9fquaPfZA9WOXJA9Y3EiyG/CArbTXzu0DSc4AHprk5YxuKvzH\nPdckaXadzWhawBeBZwNv77ec2eQcqB4kOZXRRPL3dLtOBM6vqrf1V5WGLMkzgWcxWsLg41X11z2X\nJGlGJfliVT22e7wr8NmqOqLnsmaOAaonSY4F/l23+ddV9fE+69FsSLI3cFP5P66k+yjJJeOBaem2\npmOA6kmSAxhdlv6JbiHNXarq1r7r0nAkeTLwVuBm4I3A+4C9GQ29v6yqLuixPEkzKslmuotSGPVq\n7wbchvfCa2KA6kE3j+WXgb2q6qAkPwL8UVUd3XNpGpAknwNeD+wJnMno4oOLkhwKnFNVj++1QEna\niTmJvB+vBJ4CfBugqq4EfqjXijREu1bVhd1lxTdU1UUAVbWh57okaadngOrH96rqzsWNbhKfXYFa\nanxF4NuXHPP9Ikk9ch2ofnwqyeuB3bqrq34V+HDPNWl4Dk/ybbo5Ct1juu0H9leWJMk5UD1Isgb4\nJcYuSwf+xCurJEmaDQaonnQ3baSqvtF3LZIkqY1zoFZRRuaTfBP4EvClJN9I8lt91yZJkqZngFpd\nr2Z09d2TqmqvqtoL+AngKUle3W9pkiRpWg7hraIklwLPrKpvLtn/cOBC1/WRJGk22AO1uu63NDzB\n3fOg7tdDPZIk6T4wQK2uO+/jMUmSNCAO4a2iJfcfutch4IFVZS+UJEkzwAAlSZLUyCE8SZKkRgYo\nSZKkRgYoSZKkRgYoSZKkRgYoSYOR5K4k7x3b3qW73dH5jee5Ksle29pGkrbEACVpSL4L/FiSB3Tb\nzwSuvg/nmebyYi9BlnSfGaAkDc3HgJ/qHp8AnLN4IMnDkvxFksuT/F2Sx3b790ry8SRfTPLHjNZW\nW3zOi5P8Q5JLkvxhksVjd7eRpFYGKElDUsC5wAldL9TjgH8YO34acElVHQ78JrA43PfbwGeq6rHA\nXwBrAZIcCrwI+MmqOgK4C3jxanwjknZsu/ZdgCSNq6p/THIgo96nj3LvnqKnAs/v2q3rep52B54G\nPK/b/7Ek/9q1Pxo4Ari463l6IHDDanwfknZsBihJQ3Q+8N+AOWDvZdpOmss0Pkx3dlX95sqVJkkO\n4UkalsXg86fAaVX1T0uOfwZ4CUCSOeCbVfUd4NN0Q3NJng08tGv/SeDnkjy8O/awJGu363cgaadg\nD5SkISmAqroWOH3C8XngT5NczuiKvV/o9p8GnJPkeODvgK9151mf5A3AhUnWAHcCr+yOexWepPvM\nmwlLkiQ1cghPkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSp0f8P\nkJVgSVQUMUAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd667a69e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results_svd.plot(kind='bar',color=['r','g','b','y'],title='Model Performance',figsize=(10,6),x='Model',legend=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## comparing negative samples accuracies\n",
    "negcom = pd.DataFrame()\n",
    "for name,model in models.items():\n",
    "    p= recall_score(Test_labels,tfprediction[name],pos_label='neg')\n",
    "    negcom.loc[name,'recall'] = p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAHDCAYAAAAa83+4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYJmV95//3ZwAFkYMjCAoiiiYYAygqxsPqKAooRgKe\nQIlRs8YY3CXqbgiuu47GBFez/vJTTBRPiyaKsuoGiAqCjEcQFBBMGMUTclYEcYIgw/DdP6oamqa7\n55npfqqe6nm/rquv6aqn+unPXAw9n7nvu+5KVSFJkqRuLOs7gCRJ0qbE8iVJktQhy5ckSVKHLF+S\nJEkdsnxJkiR1yPIlSZLUIcuXpCUhyXeTPLXvHJK0PpYvSRMjyR1JHjbj3JuTfGx9X1tVv1tVXxlD\npj9KcnuSXyX5ZZILkhw87fVtkvxdksvbay5L8q4ky2e8z6okNyTZYrEzShoWy5ekSTLXrs997wb9\njaratqq2Bz4MfCrJdm2R+hLwSOCAqtoWeCJwPbDf1BcneQjwFOAO4Hmdp5c0USxfkiZJ5n0xuX+S\nU5PcmOQXSb487bUfJ3lG+/mbk3wyyYntaNQlSfaddu2+7QjWTUk+leSkJG8dMeOHga2APYA/AnYF\n/qCqvgdQVddX1d9U1Remfc3LgHOA/w28fMTvI2mJsnxJGpI3AFcA9wceALxxnmt/H/g4sB1wKvBe\ngHa06jM0JWo58Ang0FG+eZLNgVcBa4DLgP2BL1TVLev50pcB/9jmOTDJjqN8P0lLk+VL0pCsBR4I\nPLSq1lXV1+e59mtVdXo1D7D9GLB3e/6JwGZVdXz7Hp8FzlvP931ikhuAq4EX04x0raEpgdfM94VJ\nngLsBnyqqi4AfgC8ZD3fT9ISZvmSNEnWATMXpG9BU7oA3gn8EDgjyQ+SHDPPe1077fNfA1smWUZT\n3q6ace0V68l1TlUtr6oHVNWTqurs9vwv2vebz8uAM6rqxvb4EzTTlZI2UZYvSZPkp8DuM849FLgc\noKr+var+S1XtQbNw/fVJnr6B3+MaYJcZ5x68EVkBzqSZRtxqtheTbAm8CHhakmuSXAP8ObBPkr02\n8ntKGjjLl6RJ8kngTUl2SeOZwHOBkwGSHJxkj/baNcDtNKNlo5hazH8OsC7JUUk2S3II0+5M3EAf\noxk1+3SS324z3z/JsUkOollLdjvN3ZD7tB+PBL6Ko1/SJsvyJWmSvBX4BvA14Abg7cBLqurS9vVH\nAGcmWQN8HXjvtL291rcdRQFU1VrgMOA/AjfSrL86FfjNhoatqtuAZwKrgS8CNwHn0qwF+ybNlOOH\nq+qqqvrZ1AfN4v+XtNOgkjYxadaijunNkw/R/Kv1uqrae45r3g08G7gZeHlVXTS2QJI0iyTnAv9Q\nVSf2nUXS0jfuf3V9BDhwrheTPBvYo6oeAbwaeN+Y80gSSZ6aZKd22vGPgL2AL6zv6yRpMWw+zjev\nqq+1OzvP5RDgo+2132x3jN6pqq4bZy5Jm7zfBj4F3Af4EfB8f+5I6spYy9cIduHut3hf1Z7zh6Ck\nsamqDwAf6DuHpE2Tiz0lSZI61PfI11XcfX+dXbnn5ocAJOn7wbqSJEkjq6pZn1fbRfkKcz8s9xTg\nKOCTSX4P+OV86y7GeWfmUK1cuZKVK1f2HUMD4J8VbQj/vGhU/lmZXTJX9Rlz+UrycWAFcP8kPwXe\nDNwLqKo6oao+l+Q5SX5As9XEK8aZR5IkqW/jvttxvQ+PrarXjjODJEnSJHHB/cCtWLGi7wgaCP+s\naEP450Wj8s/KhhvrDveLKUkNJaskSdq0JZlzwb0jX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+S\nJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmS\nJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS\n1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElS\nhyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkd\nsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI\n8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElSh8ZevpIclGR1ku8nOWaW17dNckqS\ni5JckuTl484kSZLUl1TV+N48WQZ8H9gfuBo4Hzi8qlZPu+ZYYNuqOjbJDsD3gJ2q6vYZ71XjzCpJ\nkrRYklBVme21cY987QdcVlWXV9Va4CTgkBnXFLBN+/k2wC9mFi9JkqSlYtzlaxfgimnHV7bnpjse\n+J0kVwPfAY4ecyZJkqTeTMKC+wOBC6vqQcBjgPcmuW/PmSRJksZi8zG//1XAbtOOd23PTfcK4DiA\nqvphkh8DewLfmvlmK1euvPPzFStWsGLFisVNK0mStBFWrVrFqlWrRrp23AvuN6NZQL8/cA1wHnBE\nVV067Zr3Aj+rqrck2YmmdO1TVTfMeC8X3EuSpEGYb8H9WEe+qmpdktcCZ9BMcX6oqi5N8urm5ToB\neBvwv5Nc3H7ZX8wsXpIkSUvFWEe+FpMjX5IkaSj63GpCkiRJ01i+JEmSOmT5kiRJ6pDlS5IkqUOW\nL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+\nJEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmS\nJEnqkOVLkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUMbVL6S3C/J3uMKI0mStNStt3wlWZVk\n2yTLgQuADyR51/ijSZIkLT2jjHxtV1W/Ag4DPlpVTwCeOd5YkiRJS9Mo5WvzJA8EXgScNuY8kiRJ\nS9oo5eutwOnAD6vq/CQPAy4bbyxJkqSlKVXVd4aRJKmhZJUkSZu2JFRVZnttlAX3v5XkrCTfbY/3\nTvKmxQ4pSZK0KRhl2vEDwLHAWoCquhg4fJyhJt3uO+9MEj9mfOy+8859/6eRJGnibT7CNfepqvOS\nu42c3T6mPINw+XXX4QToPeW66/qOIEnSxBtl5Ov6JHtA0zeSvAC4ZqypJEmSlqj1Lrhv7248AXgS\ncCPwY+DIqvrJ2NPdPcfELLhP4sjXLAJMyn8jSZL6NN+C+5HvdkyyNbCsqtYsZrhRWb4mn+VLkqTG\nfOVrzjVfSV4/15sBVJWPGJIkSdpA8y2436azFJIkSZsIN1ndCE47zs5pR0mSGgvdZPVhSU5N8vMk\nP0vyz+0ifEmSJG2gUbaa+DjwKeCBwIOAk4FPjDOUJEnSUjXKVhMXV9XeM859p6r2GWuye+Zw2nHC\nOe0oSVJjo+52nObzSf4SOIlmo9UXA59Lshygqm5YtKSSJElL3CgjXz+e5+Wqqk7WfznyNfkc+ZIk\nqbEom6z2zfI1+SxfkiQ1FjTtmGQz4GBg9+nXu8mqJEnShhtlzdepwK3AJcAd440jSZK0tI1Svnad\nebejJEmSNs4o+3x9PskBY08iSZK0CRhl5Otc4LNJlgFruXNddW071mSSJElL0KhbTRwCXNLn7Ybe\n7Tj5vNtRkqTGgp7tCFwBfHdimo8kSdKAjTLt+CNgVZLPA7+ZOulWE5IkSRtulPL14/bjXu2HJEmS\nNtLYd7hPchDwdzRTnB+qqv85yzUrgP8P2AL4eVU9fZZrJmbm0zVfs3PNlyRJjQU9XijJjsBfAI8C\ntpw6X1XPGOEbLwO+D+wPXA2cDxxeVaunXbMd8A3ggKq6KskOVXX9LO9l+Zpwli9JkhoLXXD/T8Bq\n4KHAW4Cf0JSoUewHXFZVl1fVWuAkmjsnp3sJ8OmqugpgtuIlSZK0VIxSvu5fVR8C1lbVl6vqlcB6\nR71au9DcLTnlyvbcdL8FLE9ydpLzk/zhiO8tSZI0OKMsuF/b/npNkoNppg+XL3KGfWkK3dbAOUnO\nqaofLOL3kCRJmgijlK+3teuy3gC8B9gWeN2I738VsNu0413bc9NdCVxfVbcCtyb5CrAPcI/ytXLl\nyjs/X7FiBStWrBgxhiRJ0visWrWKVatWjXTtWO92TLIZ8D2aBffXAOcBR1TVpdOu2ZOm1B0E3Bv4\nJvDiqvq3Ge/lgvsJ54J7SZIaC1pwn+QdSbZNskWSs5L8PMmRo3zjqloHvBY4A/hX4KSqujTJq5P8\nSXvNauB04GKa50ieMLN4SZIkLRWjbDVxUVU9OsmhwHOB1wNfqap9ugg4LYcjXxPOkS9JkhoL3Wpi\nal3YwcDJVXXToiWTJEnaxIyy4P60JKuBW4DXtJuu3jreWJIkSUvTSAvukywHbqqqdUm2BrapqmvH\nnu7uGZx2nHBOO0qS1FjQ44UmheVr8lm+JElqLHTNlyRJkhaJ5UuSJKlDo+zzlSRHJvkf7fFuSfYb\nfzRJkqSlZ5SRr78Hnggc0R6vAd47tkSSJElL2ChbTTyhqvZNciFAVd2Y5F5jziVJkrQkjTLytbZ9\nRmMBtPt83THWVJIkSUvUKOXr3cBngQck+Wvga8DfjDWVJEnSEjXqJqt7AvvTbOV0VlVdOu5gs2Rw\nn68J5z5fkiQ1FrTJapJ3AydV1TfGEW5Ulq/JZ/mSJKmx0E1Wvw28KckPk/xtksctbjxJkqRNx8iP\nF2qf7/h84HBgt6p6xDiDzfL9HfmacI58SZLUWKzHCz0c2BN4CLB6MYJJkiRtakZZ8/UO4FDgh8An\ngc9W1S87yDYzhyNfE86RL0mSGvONfI2yyeoPgSdW1fWLG0uSJGnTM+fIV5I9q2p1kn1ne72qLhhr\nsnvmceRrwjnyJUlSY6O2mkhyQlX9SZKzZ3m5quoZixlyfSxfk8/yJUlSY6H7fG1ZVbeu79y4Wb4m\nn+VLkqTGQu92nG1z1V43XJUkSRqqORfcJ9kZ2AXYKsljaAY2ALYF7tNBNkmSpCVnvrsdDwReDuwK\nvGva+TXAG8eYSZIkackaZc3X86vq0x3lmS+Ha74mnGu+JElqLGjBffsGBwOPAracOldVb120hCOw\nfE0+y5ckSY0FLbhP8j7gxcB/ovn79YU0jxiSJEnSBhpl2vHiqtp72q/3BT5fVf+hm4h35nDka8I5\n8iVJUmOhW03c0v766yQPAtYCD1yscJIkSZuSUZ7teFqS7YF3AhcABXxwrKkkSZKWqJEW3N95cXJv\nYMuquml8keb83k47TjinHSVJasw37bjeka8kh81y7ibgkqr62SLkkyRJ2mSMMu34x8ATgakHbK8A\nvg08NMlbq+pjY8omSZK05IxSvjYHHllV1wEk2Qn4KPAE4CuA5UuSJGlEo9zt+OCp4tX6WXvuBpo7\nHyVJkjSiUUa+ViU5DTi5PX5Be25r4JdjSyZJkrQEjbLJaoDDgKe0p74OfLrrWw+923HyebejJEmN\nBd3tWFWV5FvATVV1ZpL7APcF1ixyTkmSpCVvlGc7vgr4P8D721O7AP93nKEkSZKWqlEW3B8FPBn4\nFUBVXQY8YJyhJEmSlqpRytdvquq2qYMkm4NLniRJkjbGKOXry0neCGyV5Fk0dz2eOt5YkiRJS9Mo\ndzsuo9nl/gCaG9pOBz7o3Y6aybsdJUlqzHe34wY9WLtPlq/JZ/mSJKmx0AdrPxlYCTykvb79O7Ye\ntpghJUmSNgWjTDuuBl5H8zDtdVPnq+oX4412jxyOfE04R74kSWosaOSLZnPVzy9yJkmSpE3SKCNf\nbwc2Az4D/GbqfFVdMN5o98jhyNeEc+RLkqTGghbcJzl7ltNVVc9YjHCjsnxNPsuXJEkN73ZcZJav\n2Vm+JElqzFe+RtlkVZIkSYvE8iVJktShectXkmVJntRVGEmSpKVu3vJVVXcA7+0oiyRJ0pI3yrTj\nWUmen2TWRWOSJEka3ShbTawBtqbZ3f4W7nq80Lbjj3e3HN7tOOG821FamJ133p3rrru87xgTZ6ed\nHsK11/6k7xjSBnGriUVm+Zqd5UtamGaCwf+H7in+bNHgLPTxQiR5HvDU9nBVVZ22WOEkSZI2Jetd\n89U+Xuho4N/aj6OTHDfqN0hyUJLVSb6f5Jh5rnt8krVJDhv1vSVJkoZmlDVfFwOPbu98JMlmwIVV\ntfd63zxZBnwf2B+4GjgfOLyqVs9y3Rdp1pR9uKo+M8t7Oe044Zx2lBbGace5OO2o4VmMHe63n/b5\ndhvwvfcDLquqy6tqLXAScMgs1/0n4P8AP9uA95YkSRqcUdZ8HQdc2D5gOzRrv/5yxPffBbhi2vGV\nNIXsTkkeBPxBVT09yd1ekyRJWmrmLF9JnlxVXwc+A6wCHt++dExVXbuIGf4OmL4WzP3EJEnSkjXf\nyNe7gccC51TVvsApG/H+VwG7TTvetT033eOAk9pNXHcAnp1kbVXd4/utXLnyzs9XrFjBihUrNiKS\nJEnS4lq1ahWrVq0a6do5F9wnORe4GPgDmrVad1NV/3m9b94szv8ezYL7a4DzgCOq6tI5rv8IcKoL\n7ofJBffSwrjgfi4uuNfwbOw+X88FngkcCHx7Y75xVa1L8lrgDJrF/R+qqkuTvLp5uU6Y+SUb830k\nSZKGYpStJvapqu90lGe+HI58TThHvqSFceRrLo58aXh8vNAis3zNzvIlLYzlay6WLw3PYuzzJUmS\npEVg+ZIkSerQfPt8vX6+L6yqdy1+HEmSpKVtvrsdt+kshSRJ0ibCBfcbwQX3s3PBvbQwLrifiwvu\nNTwbtc9XknfP96ajbLIqSZKku5tv2nGjNlaVJEnS3Jx23AhOO87OaUdpYZx2nIvTjhqejX280NQX\n7wgcA/wOsOXU+ap6xqIllCRJ2kSMss/XPwGXAg8F3gL8BDh/jJkkSZKWrFGe7fjtqnpskourau/2\n3PlV9fhOEt6Vw2nHCee0o7QwTjvOxWlHDc+Cph2Bte2v1yQ5GLgaWL5Y4SRJkjYlo5SvtyXZDngD\n8B5gW+B1Y00lSZK0RHm340Zw2nF2TjtKC+O041ycdtTwzDftuN4F90lOTLL9tOP7JfnwYgaUJEna\nVIxyt+PeVfXLqYOquhF4zPgiSZIkLV2jlK9lSe43dZBkOaOtFZMkSdIMo5So/wWck+Tk9viFwF+P\nL5IkSdLSNdKC+yS/A0ztaP+lqvq3saaaPYML7iecC+6lhXHB/VxccK/hWdCC+9Zy4OaqOh74eZKH\nLlo6SZKkTcgodzu+mebZjse2p7YA/nGcoSRJkpaqUUa+DgWeB9wMUFVXA9uMM5QkSdJSNUr5uq1d\nbFUASbYebyRJkqSla5Ty9akk7we2T/Iq4Ezgg+ONJUmStDSNerfjs4ADaG5oO72qvjjuYLNk8G7H\nCefdjtLCeLfjXLzbUcMz392OG/xsxyTLgCOq6p8WI9wGfF/L14SzfEkLY/mai+VLw7NRW00k2TbJ\nsUmOT3JAGq8FfgS8aFxhJUmSlrI5R76S/DNwI3AOsD/wAJrBjaOr6qLOEt6Vx5GvCefIl7QwjnzN\nxZEvDc9GTTsmuaSq9mo/3wy4Btitqm4dW9J5WL4mn+VLWhjL11wsXxqejd3hfu3UJ1W1Driyr+Il\nSZK0VMw38rWOdmNVmkGNrYBfc+cAR23bScK78jjyNeEc+ZIWxpGvuTjypeGZb+Rr87m+qKo2G18k\nSZKkTdOoD9aWJEnSIrB8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKH\nLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2y\nfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHRp7+UpyUJLV\nSb6f5JhZXn9Jku+0H19Lste4M0mSJPVlrOUryTLgeOBA4FHAEUn2nHHZj4CnVtU+wNuAD4wzkyRJ\nUp/GPfK1H3BZVV1eVWuBk4BDpl9QVedW1U3t4bnALmPOJEmS1Jtxl69dgCumHV/J/OXqPwKfH2si\nSZKkHm3ed4ApSZ4OvAJ4St9ZJEmSxmXc5esqYLdpx7u25+4myd7ACcBBVXXjXG+2cuXKOz9fsWIF\nK1asWKyckiRJG23VqlWsWrVqpGtTVWMLkmQz4HvA/sA1wHnAEVV16bRrdgPOAv6wqs6d571qnFk3\nRBImI8lkCTAp/42kIUoC/nSZRfzZosFJQlVlttfGOvJVVeuSvBY4g2Z92Yeq6tIkr25erhOA/w4s\nB/4+zU+etVW13zhzSZIk9WWsI1+LyZGvyefIl7QwjnzNxZGv2ey2285cccV1fceYKA9+8E789KfX\n9h0DmH/ky/K1ESxfs7N8SQtj+ZqL5Ws2STj77L5TTJanP31y/h6ar3z5eCFJkqQOWb4kSZI6ZPmS\nJEnqkOVLkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ5UuS\nJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUOWL0mS\npA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+JEmS\nOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnq\n0OZ9B5CWup133Znrrrqu7xgTZaddduLaK6/tO4Yk9cLyJY3ZdVddByv7TjFZrltpGZW06XLaUZIk\nqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKk\nDlm+JEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSerQ2MtXkoOSrE7y/STHzHHNu5Nc\nluSiJI8edyZJkqS+jLV8JVkGHA8cCDwKOCLJnjOueTawR1U9Ang18L5xZlpqVvUdQMPx474DaFhW\n9R1AA3HRRX0nGJ5xj3ztB1xWVZdX1VrgJOCQGdccAnwUoKq+CWyXZKcx51oyVvUdQMPxk74DaFhW\n9R1AA2H52nDjLl+7AFdMO76yPTffNVfNco0kSdKS4IJ7SZKkDqWqxvfmye8BK6vqoPb4L4Gqqv85\n7Zr3AWdX1Sfb49XA06rquhnvNb6gkiRJi6yqMtv5zcf8fc8HHp7kIcA1wOHAETOuOQU4CvhkW9Z+\nObN4wdy/AUmSpCEZa/mqqnVJXgucQTPF+aGqujTJq5uX64Sq+lyS5yT5AXAz8IpxZpIkSerTWKcd\nJUmSdHcuuJckSeqQ5WtgkmyV5Nj2RgWSPLzdqFaSJA2A5Wt4PgwEeEp7fDXwN/3F0VAk2SGJN65o\nTkmOSrL9tOP7JfmzPjNJS5FrvgYmybeq6nFJLqyqx7TnLqoqn4mpO7V3Dr8duAH4K+BjwA40/+B6\nWVV9ocd4mlCz/SyZ/rNGmpJkR+BVwO5Mu3mvql7ZV6YhGfdWE1p8tyXZEiiAJA8Fbus3kibQ8cAb\nge2ALwHPrqpz22erfgKwfGk2myVJtf8qT7IZcK+eM2ky/TPwVeBMYF3PWQbH8jU8b6X5i3PXJCcC\nTwP+uN9ImkCbV9UZAEneWlXnAlTVamceNY8v0Oy5+P72+NVY1DW7+1TVMX2HGCrL18BU1ReSfBt4\nEs3ar/9aVT/rOZYmzx3TPr9lxmuuNdBcjqEpXK9pj78IfLC/OJpgpyV5TlV9ru8gQ+SarwFKcjiw\nR1X9dZIHAw+oqm/3nUuTI8k6mk2LA2wF/HrqJWDLqtqir2yShi/JGmBrmmUva9vTVVXb9pdqOCxf\nA5PkeGAL4KlV9cgky4HTq+rxPUeTNFBJPlVVL0pyCbOMjFbV3j3EkpYspx2H50lVtW+SCwGq6oYk\nLoiVtBBHt78+t9cUGpQkzwOe2h6uqqrT+swzJO7zNTxrkyzjrrsd78/d1/dIJFmT5Fftr2umHf86\nye1959Nkqapr2k//rKoun/4BuM+X7iHJ22lK+7+1H0cnOa7fVMPhtOPAJHkZcCjwOJoNV18EvKWq\nTuo1mCZakvsCR9Espv5sVb2h50iaQEkuqKp9Z5y72GlHzZTkYuDRVXVHe7wZcKF/VkbjtOPAVNVH\n27sdn0mzePqFVfXdnmNpQrW7lf858DLg48Djq+oX/abSpEnyGpoRrj3av1SnbAN8vZ9UGoDtaTZy\nhmZPQY3I8jUg7b8sLq6qRwH/2nceTa4kOwBvAF5MM0L6mKq6qd9UmmAfBz4PHAf85bTza6rqhtm/\nRJu444ALk5xNMxDwVO7+Z0fzcNpxYJKcCvxpVV3VdxZNriQ3Az8HPgKsmfl6Vb2r81CaeEn2AK6s\nqt8kWQHsDXy0qn7ZbzJNkvYZsbsCtwNTd9qfV1XX9pdqWCxfA9P+K+OxwDk0+zgBUFWH9RZKEyfJ\nSubZTLWq3tJdGg1Fkoto1pPuDnyO5hEyj6qq5/SZS5MnySVVtVffOYbKacfheVvfATT5qmpl3xk0\nSHdU1e1JDgPeU1XvmdrWRprhgiSPr6rz+w4yRJavgUhyRlUdUFVn9Z1Fky/JX1TVO5K8h9k3zfzP\nPcTS5Fub5AiaGzR+vz3n0xA0mycAL01yOXc9TaO823E0lq/h2LHvABqUS9tfv9VrCg3NK4A/Bf66\nqn6c5KHAx3rOpMl0YN8Bhsw1XwOR5EfAf5nr9ar6TIdxJEmbsCQfq6o/XN85zc6Rr+HYjubRH5nl\ntQIsX7pTe1fsfAvun9dhHE04n+2ojfCo6QftVkiP7SnL4Fi+huPyqnpl3yE0GH/bdwANis921EiS\nHAu8Edgqya+4a0DgNuCE3oINjNOOA5Hkwqp6TN85JElKclxVHdt3jqGyfA1Ekt/1MUIa1VzTR1Oc\nRtJskqwcTI/VAAAJ3ElEQVThnn9ubqK5ceMNVfWj7lNpErUbrR4KPIXmz8xXq+r/9ptqOCxf0hKU\n5CHzvV5Vl3eVRcOR5K+AK2keNxTgcGAP4ALgNVW1or90miRJ/h54OPCJ9tSLgR9W1VH9pRoOy5ck\nCYAk36mqfWacu6iqHj3ba9p0JVkNPLLaEpFkGfCvVfXIfpMNw7K+A0ganyRrkvyq/bg1ybp2kaw0\nm18neVGSZe3Hi4Bb29f8l7qm+wGw27TjB7fnNALvdhyYJE8GVgIPofnvN7Wr8MP6zKXJVFXbTH3e\nrtE4BPi9/hJpwr0U+P+Bv2+PzwGOTLIV8NreUmkSbQNcmuQ8mmK+H/CtJKeA29msj9OOA9MO9b4O\n+Dawbup8Vf2it1AaFO+clbRQSZ423+tV9eWusgyRI1/Dc1NVfb7vEBqG9gHJU5YBj+OuaSTpbpLs\nCrwHeHJ76qvA0VV1ZX+pNImq6svtjT2PqKoz29HRzatqTd/ZhsDyNTxnJ3knzY72v5k6WVUX9BdJ\nE+z3p31+O/ATmqlHaTYfobnT8YXt8ZHtuWf1lkgTKcmrgD8BltPcEbsr8D5g/z5zDYXTjgOT5OxZ\nTldVPaPzMJKWlKk7G9d3TkpyEc06r29OLWNIcklV7dVvsmFw5GtgqurpfWfQcDiNpA30iyRHctfe\nTUcArifVbH5TVbc19/FAks3xjtiRudXEwCTZLsm7knyr/fhfSbbrO5cm1keAU4AHtR+ntuek2bwS\neBFwLXAN8ALg5X0G0sT6cpKpZzw+CziZ5ueLRuC048Ak+TTwXeDE9tQfAvtU1WFzf5U2VU4jaaGS\n/HlV/V3fOTRZ2k1V/xg4gGbLo9OBD5alYiSWr4HxL1NtiCRn0Yx0TZ9GekVVuShWI0ny06rabf1X\nalOSZGvg1qpa1x5vBty7qn7db7JhcNpxeG5J8pSpg3bT1Vt6zKPJNts00it6TaShSd8BNJHOAraa\ndrwVcGZPWQbHBffD8xrgxHadV4AbcE2G5tA+QNudprUQTo9oNltW1b9PHVTVvye5T5+BhsTyNTBV\ndRGwT5Jt22Of06c5JdkReBWwO9P+f6+qV/aVSZMnyRpmL1nh7qMb0pSbk+w7tcdkksfiLMzILF8D\nkeTIqvrHJK+fcR6AqnpXL8E06f6ZZnuJM5n2OCppuunPAJVGdDRwcpKraUr6zsCL+400HJav4di6\n/dUfktoQ96mqY/oOIWnpaO90vBewJ/Db7envVdXa/lINi3c7SktYkrcB36iqz/WdRdLSkeTCqZ3t\nteG823FgkrwjybZJtkhyVpKftztSS7M5GjgtyS1JfpVkTRLXCUpaqLOSPD9Ta1+0QRz5GpipPb2S\nHAo8F3g98JWq2qfnaJow7Q/FB1fVT/vOImlpaW/S2JpmLektNOu+qqq27TXYQLjma3im/psdDJxc\nVTf5Dw/Npqoqyb8APuhW0qLyJo2FcdpxeE5Lshp4LM2w747ArT1n0uS6IMnj+w4haWlJ48gk/709\nfnCS/frONRROOw5QkuXATVW1rt3UbtuqurbvXJo8bVF/OHA5cDN3TQ3s3WswSYOW5B+AO4BnVNUj\nk9wPOKOq/MfeCJx2HIgkz6iqLyU5bNq56Zd8pvtUGoAD+w4gaUl6QlXtm+RCgKq6Mcm9+g41FJav\n4Xga8CXg92d5rbB8aRZVdXn7LNBHVNVH2mnq+/adS9LgrW0fpl1w59M07ug30nA47SgtYUneDDwO\n+O2q+q0kD6K5UePJPUeTNGBJXkqzo/2+wInAC4A3VdXJvQYbCBfcD0ySv0my/bTj+7UbaUqzOZTm\nwdo3A1TV1fiUBEkLVFX/BPwFcBxwDfAHFq/ROe04PM+uqjdOHbTz7M8B3tRjJk2u29otJ6amBrZe\n3xdI0lySbAn8Kc2NPJcA76+q2/tNNTyOfA3PZknuPXWQZCvg3vNcr03bp5K8H9g+yatoHrD9gZ4z\nSRquE2mWMlwCPBv4237jDJNrvgYmyTE0i+4/0p56BXBKVb2jv1SaZEmeBRxAs83E6VX1xZ4jSRqo\nJJdU1V7t55sD51XVvj3HGhzL1wAlOQh4Znv4xao6vc88GoYkOwC/KP+nl7SRklwwvWzNPNZoLF8D\nlOQhNFsHnNlusrpZVa3pO5cmR5LfA94O3AD8FfAxYAeapQYvq6ov9BhP0kAlWUd7Aw/NaPpWwK/x\n2Y4bxPI1MO26nT8BllfVHkkeAbyvqvbvOZomSJJvAW8EtgNOoLlR49wkewKfqKrH9BpQkjZhLrgf\nnqOAJwO/Aqiqy4AH9JpIk2jzqjqjvfX72qo6F6CqVvecS5I2eZav4flNVd02ddAueHT4UjNN32n6\nlhmv+edFknrkPl/D8+UkbwS2au9i+zPg1J4zafLsk+RXtGsy2s9pj7fsL5YkyTVfA5NkGfDHTNs6\nAPigd7BJkjQMlq8Bah9gSlX9vO8skiRpw7jmayDSWJnkeuB7wPeS/DzJ/+g7myRJGp3lazheR3OX\n4+OranlVLQeeADw5yev6jSZJkkbltONAJLkQeFZVXT/j/I7AGe7bJEnSMDjyNRxbzCxecOe6ry16\nyCNJkjaC5Ws4btvI1yRJ0gRx2nEgZjxP624vAVtWlaNfkiQNgOVLkiSpQ047SpIkdcjyJUmS1CHL\nlyRJUocsX5IkSR2yfEkatCR3JPnotOPN2kdvnbKB7/PjJMsXeo0krY/lS9LQ3Qz8bpJ7t8fPAq7Y\niPcZ5dZvbw+XtGCWL0lLweeAg9vPjwA+MfVCkvsl+WyS7yT5RpK92vPLk5ye5JIkH6DZM2/qa16a\n5JtJLkjyD0mmXkv7+n2SnJbkwiQXJ3lhJ79LSUuC5UvS0BVwEnBEO/q1N/DNaa+/BbigqvYB/hsw\nNUX5ZuCrVbUX8FlgN4AkewIvBp5UVfsCdwAvnfE9DwKuqqrHVNXewBfG8juTtCRt3ncASVqoqvpu\nkt1pRr3+hWmjWMBTgMPa685uR7y2AZ4KHNqe/1ySG9vr9wf2Bc5vR7y2BK6d8S0vAf42yXHAv1TV\n18byG5O0JFm+JC0VpwDvBFYAO6zn2tnWbk2fWjyxqv7bnF9cdVmSfYHnAG9LcmZVvW3DI0vaFDnt\nKGnopkrTh4G3VNW/znj9q8CRAElWANdX1b8DX6GdTkzybGD79vqzgBck2bF97X5JdrvbN0weCNxS\nVR+nKXz7LvZvStLS5ciXpKErgKq6Cjh+ltdXAh9O8h2aOyP/qD3/FuATSQ4HvgH8tH2fS5O8CTgj\nyTLgNuCo9vWpEbO9gHcmuaN9/TVj+H1JWqJ8sLYkSVKHnHaUJEnqkOVLkiSpQ5YvSZKkDlm+JEmS\nOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjr0/wCpvqoNLqXZ9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd661f9be0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pl = negcom.plot(kind='bar',color=['r','g','b','y'],title='Using PCA',figsize=(10,6),legend=False)\n",
    "pl.set(xlabel='Models',ylabel='Recall for negative samples')\n",
    "lim = plt.ylim([0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The performance seems to be okay but we can try improving it further"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Another way to reduce feature set is to have better feature selection. We can use the most frequent 5000 features as the featureset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Feature Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "topwords = [fpair[0] for fpair in list(word_features.most_common(5000))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('great', 55078), ('the', 41540), ('good', 38876), ('a', 38079), ('for', 30448), ('love', 26982), ('and', 26492), ('not', 25274), ('it', 25120), ('best', 25075), ('my', 21948), ('coffee', 19082), ('i', 19041), ('but', 18943), ('this', 18389), ('taste', 18227), ('tea', 17046), ('to', 15942), ('dog', 15760), ('product', 15523), ('of', 15457), ('is', 14242), ('delicious', 14189), ('s', 13046), ('very', 11988)]\n"
     ]
    }
   ],
   "source": [
    "print (word_features.most_common(25))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word_his = pd.DataFrame(word_features.most_common(25),columns=['words','count'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        words  count\n",
      "0       great  55078\n",
      "1         the  41540\n",
      "2        good  38876\n",
      "3           a  38079\n",
      "4         for  30448\n",
      "5        love  26982\n",
      "6         and  26492\n",
      "7         not  25274\n",
      "8          it  25120\n",
      "9        best  25075\n",
      "10         my  21948\n",
      "11     coffee  19082\n",
      "12          i  19041\n",
      "13        but  18943\n",
      "14       this  18389\n",
      "15      taste  18227\n",
      "16        tea  17046\n",
      "17         to  15942\n",
      "18        dog  15760\n",
      "19    product  15523\n",
      "20         of  15457\n",
      "21         is  14242\n",
      "22  delicious  14189\n",
      "23          s  13046\n",
      "24       very  11988\n"
     ]
    }
   ],
   "source": [
    "print (word_his)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1cd64f15400>"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmAAAAGrCAYAAABnrCs6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcZFV99/HPF1BBNiEKoyiMGkBxBQU3jGMEcUWiETGu\ngUQjRjEmRjBRB+OTCNGIy6PRSGSJhqDRqFERFEdQUZBdQeCJgoAyiojiEgX8PX/cW0xN0zNT01X3\ndlfP5/169aurbt361anq7upvnXPuuakqJEmS1J+N5rsBkiRJGxoDmCRJUs8MYJIkST0zgEmSJPXM\nACZJktQzA5gkSVLPDGCStA5JNk/y2SQ3Jjl+vtszKUl2TXLzfLdD2hAZwKQNQJKbkvys/bo1yS+H\ntj13wo/1jiT/L8lPk3wzyUEzbt8zyflJfp7ka0keMMnHX19J9ktyxTp2ey5w56q6S1W9qI929cjF\nIKV5YACTNgBVtWVVbVVVWwFXAU8d2vbvE364nwL7VdXWwEuBf06yO0CSTYH/Av4Z2Ab4KPDxJPP5\nXhTWHUJ2Ai5bY4Fk44m2qAPT0EZpQ2IAkzY8ab9WbUg2TfJ/k3w/yfeSHD34hz3oIUqyPMmPk/xP\nkj9cU/GqekNV/U97+SvA14FHtjc/EfhVVb2vqm4G3gZsCew9a0OTf09yTJJT2x6z05PcrW3rT5Jc\nPNyDluRBSc5ob7sgyZOGbntGkkvbXr+rkvx5km2BjwH3GeoR3GZGG94C/DXw4kGPYZKXJvlCkncn\nuQF4bbvvS5N8O8n1ST6V5B5DdZ6a5PIkNyR5W5KzkvzRLM958yS/SrJFe/3vkvxvkju1149O8vft\n5W2SfDjJD9ufy2uG6tyujUk2bnsor09yObDvjMf+0yTfbZ/n/0vyrDX9nCWNxwAmCeBNwAOBBwAP\nA5bRhI6BpcAmwPbAS4Djk+y0rqJtiNgD+Ga7aTfgwsHt1ZwL7eL2cdfkQODVwO8AdwC+BnwR2Bb4\nLPCP7WPdCfhvmkB117b9Hxlq57HA89tewIcCZ1bVDcAfAN8Z6hH8yfCDV9XhwD8Bx83oMXwscF7b\nrrcleQ7wSuCp7et0PvBvbdvuDvwH8BfA3YAf0bzOt1NVv2hfo8e2m34PuJpVIfZxwIr28vuAjWl6\n6J4IvGzGkPJqbQReQfOzfQDwKOA5gx2T3AU4Gnh8+xrtzaqfm6QJM4BJAvgj4A1V9ZOq+hHwZuAF\nQ7ffDLypqm6pqi8AnwfW2As25APAGVV1Znt9C5ohymE/o+kFW5OPVNU3q+rXwCeAG6vqo214O5km\nTEETVH5bVcdU1a1VdSpwGqtCxi3AA5Ns0T7PC2c+0Hr6TlX9azV+TTPc+uaq+p+quhX4O2DvJHcD\nngacXVWfbm87GvjJmktzBvC4JHcEdgbe217fAngw8JX2tmcCf11Vv2p7HY9h9Z/bzDY+G3hbVa2s\nqh+37RhWwIOS3KmqrquqNQ67ShqPAUwSwBLge0PXrwJ2GLr+o6r6zYzb78FaJHkXcC9WDwQ/B7aa\nsevWwE1rKbVy6PKvZrm+RXv57qz+HAbtHDyPZ9CExu8l+XySh6+t/SO4esb1nWjmu93QDvn9EPgN\ncE+a1+q2/avqt8C1a6n9JeDxwCOAc4DTaXquHgNc1PaSLaEZSh5ux8yf28w23oPb7z9o043A84DD\ngOuS/FeS+66ljZLGYACTBPADmgAxsBOrB4S7tj0uAzsC319TsSRHAY8GnlRVvxy66VvAQ4b2C83Q\n57fm3vTbfL9t17AdaZ9HVX29qp4ObEfTM/bhdp+5HgU4837fA15cVdu2X9tU1RZVdT7N63tb29rn\nvQNr9mWa1+mpNGHsQuB+NMOMX2r3uQ74Las/59ue7xra+AOaUDyw2jByVX22qvahCbNXA+9ZSxsl\njcEAJgngJOCNSbZNsh3wOuDEodvvCLw+yR2S/D6wD/CfsxVKciTwdOCJVTWzZ+s0YLMkL2kD3V/R\n9H59eYy2Dw4oOBPYKMkr28nm+9JMMj85yZ2TPCfJlsCtND1xt7b3Wwlsl2TzMdoAzXys1yfZBW6b\nIP/M9rZPAg9P8pQkm9DMT9tmDXWoqp/ShNI/A77UDrd+A/gT2gDW9kh+HPj79vndl6b36sTZqwLN\nkO1fJFmS5K7A8KT9e7Tt24xmyPnnNAFPUgcMYNKGZ7YenzcAl9D80z+PJsz849Dt36WZQ3Udzbyu\nF1fVlTOLtKHq9cC9ge8OHVn4KoCq+l+aocCX0cyB+kPggHZIbtS2zrpPO8fpaTTznH4MvBU4cKid\nBwNXto/7fOCF7f0upAlIV7XDh3cZ4TFv34iqk4B3AR9LciPN67hPe9t1wEHAO2iGJu9GE6jWZtDT\ndd7Q9Tuzelh9KU0AvYpmXt7717GsyLtpfrbfAs6iOTBgYGPgcJpesh8BDwf+fB1tlDRHaT5YrWOn\nZGuaN90H0nwiOhi4nOaPdyeaN7UD209tJDmi3ecW4LB2MixJ9gCOAzYFPlNVr2q33xE4geaooOuB\n51TVzLkckuZBkv2Ad1XVLvPdlsUkyVk0r+uH17mzpEVn1B6wd9AEpvvTzEv4Ns0npc9X1a40E0SP\nAEiyG81h4/cHngy8p53vAM2RPIe0b+S7tG/sAIcAN1TVzjRH8cw8MkeSJGnRWGcAS7IV8Niq+iBA\nexj6T2mGEQbnRDseOKC9vD9wUrvflcAVwF5JlgBbVtU57X4nDN1nuNZHgSeM9awkaeHzFEDSBmyT\nEfa5N3B9kg/S9H59A3gVsH1VrYRmfkM7cReaI3vOGrr/te22W4BrhrZfw6qjgHagPTS6qm5Nc8Lb\nbdtFEiXNo6r6HODw44RV1aPnuw2S5s8oQ5Cb0Kxk/X+rag/gFzTDjzM/vU3y01zWvYskSdJ0GqUH\n7Brg6qoaHLHznzQBbGWS7atqZTu8+MP29mtZfZ2Ze7bb1rR9+D7fT3P+ua1m6/1KYpe9JEmaGlU1\na6fSOnvA2mHGqwdr29DMz/oWzWHbL263vYjmFCG02w9Kcsck9wZ+l+YUHNcBP02yVzsp/4Uz7vOi\n9vKzaSb1r6k9I3298Y1vHHnf9f3qqva01Z3GNk9b3Wlss6+Fr4Wvha/FfNddKG1em1F6wKA5weyH\nktwB+A7wxzRrxpyc5GCaNWgObAPSJUlOpllT6Gbg0FrVipez+jIUp7TbjwVOTHIFzfo9B43YLkmS\npKkzUgCrZqHCPWe5aZ817P8PwD/Msv1c4EGzbP81bYCTJEla7DZevnz5fLdhZEceeeTy9Wnv0qVL\nO2tLV7WnrW6Xta3bfe1pq9tl7Wmr22XtaavbZe1pq9tl7Wmr22XtUeseeeSRLF++/MjZbhtpJfyF\nIklNU3slSdKGKwk110n4kiRJmiwDmCRJUs8MYJIkST0zgEmSJPXMACZJktQzA5gkSVLPDGCSJEk9\nM4BJkiT1zAAmSZLUMwOYJElSzwxgkiRJPTOASZIk9cwAJkmS1DMDmCRJUs8MYJIkST0zgEmSJPXM\nACZJktQzA5gkSVLPDGCSJEk9M4BJkiT1zAAmSZLUs0URwJYsWUqSkb6WLFk6382VJEkbuFTVfLdh\nZElqtvYmAUZ9HmGanrMkSZpOSaiqzHbbougBkyRJmiYGMEmSpJ4ZwCRJknpmAJMkSeqZAUySJKln\nBjBJkqSeGcAkSZJ6ZgCTJEnqmQFMkiSpZwYwSZKknhnAJEmSemYAkyRJ6pkBTJIkqWcGMEmSpJ4Z\nwCRJknpmAJMkSeqZAUySJKlnBjBJkqSeGcAkSZJ6ZgCTJEnqmQFMkiSpZwYwSZKkno0UwJJcmeTC\nJOcnObvdtk2SU5NcluRzSbYe2v+IJFckuTTJE4e275HkoiSXJzlmaPsdk5zU3uesJDtO8klKkiQt\nJKP2gP0WWFZVu1fVXu22w4HPV9WuwOnAEQBJdgMOBO4PPBl4T5K093kvcEhV7QLskmS/dvshwA1V\ntTNwDHD0mM9LkiRpwRo1gGWWfZ8BHN9ePh44oL28P3BSVd1SVVcCVwB7JVkCbFlV57T7nTB0n+Fa\nHwWesD5PQpIkaZqMGsAKOC3JOUn+pN22fVWtBKiq64Dt2u07AFcP3ffadtsOwDVD269pt612n6q6\nFbgxybbr+VwkSZKmwiYj7veYqvpBkrsBpya5jCaUDZt5fRxZ9y6SJEnTaaQAVlU/aL//KMl/AXsB\nK5NsX1Ur2+HFH7a7Xwvca+ju92y3rWn78H2+n2RjYKuqumG2tixfvvy2y8uWLWPZsmWjPAVJkqRO\nrVixghUrVoy0b6rW3nGV5M7ARlX18ySbA6cCR9LM07qhqo5K8lpgm6o6vJ2E/yHgETRDi6cBO1dV\nJfka8ErgHODTwDur6pQkhwIPrKpDkxwEHFBVB83Slpqtvc0c/1E74MK6nrMkSdK4klBVs47qjdID\ntj3w8STV7v+hqjo1yTeAk5McDFxFc+QjVXVJkpOBS4CbgUOHUtPLgeOATYHPVNUp7fZjgROTXAH8\nGLhd+JIkSVos1tkDtpDYAyZJkqbF2nrAXAlfkiSpZwYwSZKknhnAJEmSemYAkyRJ6pkBTJIkqWcG\nMEmSpJ4ZwCRJknpmAJMkSeqZAUySJKlnBjBJkqSeGcAkSZJ6ZgCTJEnqmQFMkiSpZwYwSZKknhnA\nJEmSemYAkyRJ6pkBTJIkqWcGMEmSpJ4ZwCRJknpmAJMkSeqZAUySJKlnBjBJkqSeGcAkSZJ6ZgCT\nJEnqmQFMkiSpZwYwSZKknhnAJEmSemYAkyRJ6pkBTJIkqWcGMEmSpJ4ZwCRJknpmAJMkSeqZAUyS\nJKlnBjBJkqSeGcAkSZJ6ZgCTJEnqmQFMkiSpZwYwSZKknhnAJEmSemYAkyRJ6pkBTJIkqWcGMEmS\npJ4ZwCRJknpmAFuLJUuWkmSkryVLls53cyVJ0pRIVc13G0aWpGZrbxJg1OcRRn3OXdWVJEmLXxKq\nKrPdZg+YJElSzwxgkiRJPRs5gCXZKMl5ST7ZXt8myalJLkvyuSRbD+17RJIrklya5IlD2/dIclGS\ny5McM7T9jklOau9zVpIdJ/UEJUmSFpr16QE7DLhk6PrhwOeralfgdOAIgCS7AQcC9weeDLwnzWQq\ngPcCh1TVLsAuSfZrtx8C3FBVOwPHAEfP8flIkiQteCMFsCT3BJ4CfGBo8zOA49vLxwMHtJf3B06q\nqluq6krgCmCvJEuALavqnHa/E4buM1zro8AT1v+pSJIkTYdRe8DeDryG1Q8J3L6qVgJU1XXAdu32\nHYCrh/a7tt22A3DN0PZr2m2r3aeqbgVuTLLt6E9DkiRpemyyrh2SPBVYWVUXJFm2ll0nuQbDrIds\nAixfvvy2y8uWLWPZsmUTfFhJkqS5WbFiBStWrBhp33WuA5bk74HnA7cAmwFbAh8HHg4sq6qV7fDi\nF6vq/kkOB6qqjmrvfwrwRuCqwT7t9oOAx1XVywb7VNXXk2wM/KCqtpvRFNcBkyRJU2OsdcCq6nVV\ntWNV3Qc4CDi9ql4AfAp4cbvbi4BPtJc/CRzUHtl4b+B3gbPbYcqfJtmrnZT/whn3eVF7+dk0k/ol\nSZIWpXUOQa7FW4CTkxxM07t1IEBVXZLkZJojJm8GDh3qtno5cBywKfCZqjql3X4scGKSK4Af0wQ9\nSZKkRclTEa398TqpK0mSFj9PRSRJkrSAGMAkSZJ6ZgCbB0uWLCXJSF9Lliyd7+ZKkqQJcw7Y2h9v\nqupKkqSFwzlgkiRJC4gBTJIkqWcGsEXG+WWSJC18zgFb++NNVd2ua0uSpNE5B0ySJGkBMYBJkiT1\nzAAmSZLUMwOYJElSzwxgkiRJPTOASZIk9cwAJkmS1DMDmCRJUs8MYJIkST0zgEmSJPXMACZJktQz\nA5gkSVLPDGCSJEk9M4BJkiT1zAAmSZLUMwOYJElSzwxgkiRJPTOASZIk9cwAJkmS1DMDmCRJUs8M\nYJIkST0zgEmSJPXMACZJktQzA5gkSVLPDGCSJEk9M4BJkiT1zAAmSZLUMwOYJElSzwxgkiRJPTOA\nSZIk9cwAJkmS1DMDmCRJUs8MYJIkST0zgEmSJPXMACZJktQzA5gkSVLPDGCSJEk9M4BJkiT1zAAm\nSZLUs3UGsCR3SvL1JOcnuTjJG9vt2yQ5NcllST6XZOuh+xyR5IoklyZ54tD2PZJclOTyJMcMbb9j\nkpPa+5yVZMdJP1FJkqSFYp0BrKp+DTy+qnYHHgo8OclewOHA56tqV+B04AiAJLsBBwL3B54MvCdJ\n2nLvBQ6pql2AXZLs124/BLihqnYGjgGOntQTlCRJWmhGGoKsql+2F+8EbAIU8Azg+Hb78cAB7eX9\ngZOq6paquhK4AtgryRJgy6o6p93vhKH7DNf6KPCEOT0bdWbJkqUkGelryZKl891cSZIWtJECWJKN\nkpwPXAec1oao7atqJUBVXQds1+6+A3D10N2vbbftAFwztP2adttq96mqW4Ebk2w7p2ekTqxceRVN\n7l73V7OvJElak01G2amqfgvsnmQr4ONJHkDz33a13SbYrqzphuXLl992edmyZSxbtmyCDytJkjQ3\nK1asYMWKFSPtm6r1y01JXg/8EvgTYFlVrWyHF79YVfdPcjhQVXVUu/8pwBuBqwb7tNsPAh5XVS8b\n7FNVX0+yMfCDqtpulseu2drbTDEb9XmEUZ/ztNXtsnaXbZYkaTFKQlXN2qk0ylGQdx0c4ZhkM2Bf\n4FLgk8CL291eBHyivfxJ4KD2yMZ7A78LnN0OU/40yV7tpPwXzrjPi9rLz6aZ1C9JkrQojTIEeXfg\n+CQb0QS2/6iqzyT5GnBykoNpercOBKiqS5KcDFwC3AwcOtRt9XLgOGBT4DNVdUq7/VjgxCRXAD8G\nDprIs5MkSVqA1nsIcj45BDl/tR2ClCRp/Yw1BClJkqTJMoBJkiT1zAAmSZLUMwOYJElSzwxgkiRJ\nPTOASZIk9cwAJkmS1DMDmCRJUs8MYJIkST0zgEmSJPXMACZJktQzA5gkSVLPDGCaV0uWLCXJSF9L\nliyd7+ZKkjQRqar5bsPIktRs7U0CjPo8wqjPedrqdll72upKkjTfklBVme02e8AkSZJ6ZgDTouXw\npiRpoXIIcu2PN1V1u6w9bXW7ri1J0ro4BClJkrSAGMAkSZJ6ZgCTJEnqmQFMkiSpZwYwSZKknhnA\npPXU1fIWLpshSRsOl6FY++NNVd0ua09b3S5rT1tdaMLdypVXjbTv9tvvxHXXXTlybUnS7Na2DIUB\nbO2PN1V1u6w9bXW7rD1tdbuuLUmaneuASZIkLSAGMEmSpJ4ZwCRJknpmAJMkSeqZAUySJKlnBjBJ\nkqSeGcAkSZJ6ZgCTJEnqmQFMkiSpZwYwSZKknhnAJEmSemYAkyRJ6pkBTJIkqWcGMEmSpJ4ZwCRJ\nknpmAJM0Z0uWLCXJSF9Lliyd7+ZK0oKRqprvNowsSc3W3iTAqM8jjPqcp61ul7WnrW6Xtaetbpe1\nu2yzJE27JFRVZrvNHjBJkqSeGcAkSZJ6ZgCTJEnqmQFMkiSpZ+sMYEnumeT0JN9KcnGSV7bbt0ly\napLLknwuydZD9zkiyRVJLk3yxKHteyS5KMnlSY4Z2n7HJCe19zkryY6TfqKSJEkLxSg9YLcAr66q\nBwCPAl6e5H7A4cDnq2pX4HTgCIAkuwEHAvcHngy8J82hUgDvBQ6pql2AXZLs124/BLihqnYGjgGO\nnsizkyRJWoDWGcCq6rqquqC9/HPgUuCewDOA49vdjgcOaC/vD5xUVbdU1ZXAFcBeSZYAW1bVOe1+\nJwzdZ7jWR4EnjPOkJEmSFrL1mgOWZCnwUOBrwPZVtRKakAZs1+62A3D10N2ubbftAFwztP2adttq\n96mqW4Ebk2y7Pm2TJEmaFiMHsCRb0PROHdb2hM1cUXGSKyzOumiZJEnSYrDJKDsl2YQmfJ1YVZ9o\nN69Msn1VrWyHF3/Ybr8WuNfQ3e/ZblvT9uH7fD/JxsBWVXXDbG1Zvnz5bZeXLVvGsmXLRnkKkiRJ\nnVqxYgUrVqwYad+RTkWU5ATg+qp69dC2o2gmzh+V5LXANlV1eDsJ/0PAI2iGFk8Ddq6qSvI14JXA\nOcCngXdW1SlJDgUeWFWHJjkIOKCqDpqlHZ6KaJ5qT1vdLmtPW90ua3sqIklas7WdimidASzJY4Az\ngItp3mkLeB1wNnAyTc/VVcCBVXVje58jaI5svJlmyPLUdvvDgOOATYHPVNVh7fY7AScCuwM/Bg5q\nJ/DPbIsBbJ5qT1vdLmtPW90uaxvAJGnNxgpgC4kBbP5qT1vdLmtPW90uaxvAJGnNPBm3JEnSAmIA\nk7TgLFmylCQjfS1ZsrSTuutbW5LWh0OQa3+8qarbZe1pq9tl7Wmr22XtxV13/WovWbKUlSuvGmnf\n7bffieuuu3I92iFpGq1tCHKkZSgkSWvXhK/RwtrKlS51KG3oHIKUJEnqmQFMkhawrubDSZpfBjBJ\nWsBWDW2u+2vUOWjgAQnSfHMOmCRtgNZnzlqzv/PWpEmyB0ySJKlnBjBJkqSeGcAkSZJ6ZgCTJEnq\nmQFMkjRRLp0hrZtHQUqSJsqzAkjrZg+YJElSzwxgkiRJPTOASZIk9cwAJkmS1DMDmCRJUs8MYJKk\nqeDyFlpMDGCSpKmwanmLdX81+45mfYKd4U6TYgCTJG3Q1ifYdRnu1ifY2Rs4/VI12mJ5C0GSmq29\nSRh10T8Ioz7naavbZe1pq9tl7Wmr22XtxV23y9rTVrfL2tNWt8va819Xk5WEqpp1tWF7wCRJknpm\nAJMkSeqZAUySJKlnBjBJkqSeGcAkSZJ6ZgCTJElAt2uiuXTG6lyGYu2PN1V1u6w9bXW7rD1tdbus\nvbjrdll72up2WXva6nZZe9rqdll7cSyd4TIUkiRJC4gBTJIkqWcGMEmSpJ4ZwCRJknpmAJMkSeqZ\nAUySJE2taV3eYpP5boAkSdJcrVx5FaMub7Fy5awrQswLe8AkSZJ6ZgCTJEnqmQFMkiSpZwYwSZKk\nnhnAJEmSZuj66EqPgpQkSZqh66Mr7QGTJEnqmQFMkiSpZwYwSZKknhnAJEmSerbOAJbk2CQrk1w0\ntG2bJKcmuSzJ55JsPXTbEUmuSHJpkicObd8jyUVJLk9yzND2OyY5qb3PWUl2nOQTlCRJWmhG6QH7\nILDfjG2HA5+vql2B04EjAJLsBhwI3B94MvCeJINDA94LHFJVuwC7JBnUPAS4oap2Bo4Bjh7j+UiS\nJC146wxgVfVl4CczNj8DOL69fDxwQHt5f+Ckqrqlqq4ErgD2SrIE2LKqzmn3O2HoPsO1Pgo8YQ7P\nQ5IkaWrMdQ7YdlW1EqCqrgO2a7fvAFw9tN+17bYdgGuGtl/TblvtPlV1K3Bjkm3n2C5JkqQFb1KT\n8EdbqWw067+amSRJ0hSZ60r4K5NsX1Ur2+HFH7bbrwXuNbTfPdtta9o+fJ/vJ9kY2KqqbljTAy9f\nvvy2y8uWLWPZsmVzfAqSJEmTs2LFClasWDHSvqlad+dVkqXAp6rqQe31o2gmzh+V5LXANlV1eDsJ\n/0PAI2iGFk8Ddq6qSvI14JXAOcCngXdW1SlJDgUeWFWHJjkIOKCqDlpDO2q29jbz/EfthAujPOdp\nrNtl7Wmr22XtaavbZe3FXbfL2tNWt8va01a3y9rTVrfL2oujbhKqataRvXX2gCX5MLAM+J0k3wPe\nCLwF+EiSg4GraI58pKouSXIycAlwM3DoUGJ6OXAcsCnwmao6pd1+LHBikiuAHwOzhi9JkqTFYqQe\nsIXCHrD5qz1tdbusPW11u6y9uOt2WXva6nZZe9rqdll72up2WXtx1F1bD5gr4UuSJPXMACZJktQz\nA5gkSVLPDGCSJEk9M4BJkiT1zAAmSZLUMwOYJElSzwxgkiRJPTOASZIk9cwAJkmS1DMDmCRJUs8M\nYJIkST0zgEmSJPXMACZJktQzA5gkSVLPDGCSJEk9M4BJkiT1zAAmSZLUMwOYJElSzwxgkiRJPTOA\nSZIk9cwAJkmS1DMDmCRJUs8MYJIkST0zgEmSJPXMACZJktQzA5gkSVLPDGCSJEk9M4BJkiT1zAAm\nSZLUMwOYJElSzwxgkiRJPTOASZIk9cwAJkmS1DMDmCRJUs8MYJIkST0zgEmSJPXMACZJktQzA5gk\nSVLPDGCSJEk9M4BJkiT1zAAmSZLUMwOYJElSzwxgkiRJPTOASZIk9cwAJkmS1DMDmCRJUs8WTABL\n8qQk305yeZLXznd7JEmSurIgAliSjYB3A/sBDwCem+R+41VdMXa7+q89bXW7rG3d7mtPW90ua09b\n3S5rT1vdLmtPW90ua09b3S5rT6bugghgwF7AFVV1VVXdDJwEPGO8kivGb1Xvtaetbpe1rdt97Wmr\n22XtaavbZe1pq9tl7Wmr22XtaavbZe3J1F0oAWwH4Oqh69e02yRJkhadhRLAJEmSNhipqvluA0ke\nCSyvqie11w8HqqqOmrHf/DdWkiRpRFWV2bYvlAC2MXAZ8ATgB8DZwHOr6tJ5bZgkSVIHNpnvBgBU\n1a1J/hw4lWZY9FjDlyRJWqwWRA+YJEnShsRJ+JIkST0zgC1iSbZJsleS3xt8zXebFpMkdxpl20KS\n5DGjbJtD3d8Zt8Zi0dVrrOnm70X3kjw7yZbt5b9N8rEke8x3u9ZkUQWwJM8eZdsc6m6f5Ngkn22v\n75bkkHHrdinJnwBnAJ8Djmy/Lx+z5sZJPjR+69b6GDsl2ae9vNngj2mBOmvEbestyWGjbJuDd424\nbX19LclHkjwlyaxH/MxVkr2T/HF7+W5J7j3J+h3o5DVOcliSrdI4Nsl5SZ44gbp3TvL6JP/SXt85\nydPGrTtUv5MPgu378tPar+0mUbNjE/29SHJi+30S7wtreozHJNm8vfz8JP+UZKcJ1O0qKL2+qm5K\nsjewD3As8N5xiyZ5W5IHjN26GRbEJPwJOgL4yAjb1tdxwAeBv2mvXw78B80Pd70luRhY4+S7qnrw\nXOrOcBiwJ/C1qnp8e2qnvx+nYHuwxE5J7lhVv5lAG1eT5E+BlwDbAvcF7gn8M83RsXOpdxNrf523\nmmPdJTScnoFpAAAWN0lEQVQLBW+WZHdgEDi2Au48l5qzeBHwjhnbXjzLtpEkeRTwaOBuSV49dNNW\nwMZzqTnDLjRveAcD70xyMnBcVV0+TtEkbwQeDuxK8zd4B+DfgDn3HCT5clXtPcvvR2iWv5nr70XX\nr/HBVfWOJPsB2wAvAE6kOXhpHB8EzgUe1V6/luY987/HrDv4IHgYzd/yBcAjaT6k/P6YdQ8E/pFm\nSfIA70rymqr66Jh1Z3tv/inwDeDNVfXjOdTs6vfiYUnuARyc5ARWvQ8BUFU3jFF74L3AQ5I8BPhL\n4APACcDjxqz7+qr6yFBQ+sf2sR4xZt1b2+9PBd5fVZ9O8uYxawJcCrw/ySY0fy//XlU/Hbfooghg\nSZ4MPAXYIck7h27aCrhlAg9x16o6OckRAFV1S5Jb13WntRh8unx5+/3E9vvzxqg50/9W1f8mIcmd\nqurbSXadQN3vAF9J8kngF4ONVfVPE6j9cprTUn29rXnFOJ9sq2rwCevvaJY3OZHmTep5wN3HaOd+\nNGHonsDw874JeN0YdUnyXOCPgHu3r/HAlsA4b6h3BLag+Zsf7lX8GfCHY9QFmtQCnAacluTxNCHp\n0CQXAodX1Vx7Bv8A2B04r32c74/bK1pVe7ffJ9272ulrzKp/sE8BTqyqb02ot/G+VfWc9nePqvrl\nBHsxJ/5BsPU3wJ5V9UNoekaBzwNjBTDgszT/xD/cXj+I5kPVdTQfxJ8+h5pd/V78M/AF4D40AXog\nNCHyPmPUHrilqirJM4B3V9WxExr96SooXZvkfcC+wFFppoSMPdJXVR8APtD+D/1j4KIkXwH+paq+\nONe6iyKAAd+n+YSyP6v/It4E/MUE6v8izRyXgtsWjp1z+q2qq9o6+1bV7kM3HZ7kPODwcRrbuibJ\nXYD/ovmn+BPgqgnU/Z/2ayNWfzOZhF9X1W8G7/3tp41JHKa7f1U9ZOj6e9tg8Ia5FKuq44Hjkzyr\nqv5zAu0b9lWasHhX4G1D228CLppr0ar6EvClJMcN/f5tBGxRVT8bo720tX4HeD7wQpp/Vq8APgk8\nlKY3Za7Dhr9p/wEM/vY2H7etXZntNZ6wc5OcSvNaHtEG0d9OoO5vkmzGqve3+wK/nkBd6O6D4EaD\n8NX6MZOZUrNPVQ0PhV2c5Lyq2iPJ8+dSsKvfi6p6J01v83tpwthgaPeMqrpwQg9zU9vx8Hzg99r3\njDtMoG4nQQk4EHgS8NaqujHJ3YHXTKDuYL3S+7Vf1wMXAq9O8tKqOmhORatq0XwBd+io7h7AV2hC\n11dohiAfPIG6FwCPGbr+aOCCDtr/OJpwescJ1tyC5p/3JNt5NE0P0rdp/jA/DvyfCdT9Kk2v18Y0\nf+TPA746gbp3oekB+0b79TZg6y5+Byf4Gn+Ypmd4c+ASmvOuvmYCdS8HXg/sMMttrx2j7l8B76Pp\nef1TmuGrV8z367iONn8ROH3m1wTqbtS+F92lvf47E3ofeiLwJeBHwIeAK4HHT+i1+Hj7d7KcZk7q\nJ4DPTKDu0TTzWl/cfn0WOGoCdS8E9hq6vidwYXv5/DFrnzb42bXXtwE+N4E2HwZcTDPX9000H9Qm\n8jcCLAFeDTy2vb4j8MIJ1L0z8Exg5/b63YEnTqLNXXwBbweuaN+L9ppx22Vzrbuo1gFLsjPwD8Bu\nwKaD7VU1dlds2xuzK0337mVVdfMEaj4M+Fdg67buT2jmeZw3bu2uJHkgzVDetu2m62n+IL81gdob\nAYfQ/EMIzRvsB2rMX9IkS2nmTj2G5lP+V4BXVdWVY9b9T+CbwPHtphcAD6mqZ45Rs5P5SUP1L6iq\nhyZ5Hs0/88OBc2vMeYdJ9qQJzzsx1LM+bt229r4M/U5U1Wnj1uxS+3c9sCnwLJqhnL+eY737VdNz\nNOsk5Um8X7Q9mI+keY2/VlXXj1tzlsd4HM173Sk15hzSJEfRTFXYu910JvDIqnrtmHX3pHlP3oLm\ntfgZzXvSJcBTq+rkMWqfX6uPeMy6bQ51LwIeVVW/aK9vDpw1ib+9riTZcbbtVfW9vtuyLu1w/N8C\n/zR4jWfcvnXNcT7YYgtgXwbeSJNWn04zVrtRVc1pqGlG7UcDS1n9n8sJ49Zta2/d1ht7Ul/XknwV\n+Jtqx72TLAP+vqoePYHazwQ+XVWTGv7o1CDMrGvbQpLkWzTDgh+mmdPxpSQX1upDtHOpexlNb9U3\nGRoWqwkMuaQ56mrnqvp8kjsDG1fVTePW7VOSs6tqrzne9/1V9ZIks801qaoad0L7F6rqCevaNkb9\nvWl+fh9s52ptUVXfHbPmebX6UCFJLppU6OjiPTnJucAfDEJG+3v98ZnPYw51L6aZD/e/7fVNgXOq\n6kFj1Oz6g+DgYIfQfEi5N03HxsSPNJyEJBeP83quyWKZAzawWVV9IUnaN/7l7S/9WAEszeG+96UZ\nMhxMHiyao0HGqbs1TWD8vfb6l4A3LfAgtnkNTTqsqhUTnJfzdODtSc6gOcr0lKoa+yCK9k3/T7l9\ngD54zNK/SrJ3VX25fZzHAL8as2bX3kczxHQhcEb7T2DsOWDAj6rqUxOos5rc/sjYHRjjyNg+JNl2\n6OpGNEdxbj3XelX1kvb748ds2mraf9R3Bu6aZBtWP5p3hwk9xkSPYk3yMuBQ4D5tz8/AljQ922Pp\n+D35b4AvtzUDPJbmd3tcHwS+nuTj7fUDmOMR+gPV3YEqg/qrhZm2d/fQLh5rQs5LsmdVnTPJoout\nB+yrNF3SH6WZd3Et8JaqGmvSZ5JLgd3GHQqbpe7Eh7C61v6Rn8eqIzefDzysqv5gQvXvADwZeA7N\nz/K0qvqTMWt+lWaI4lxWBWhqzAn0SR5K87Mb/HP9CfCiqprzZPn5kGSTcYNukicAz6U5Kuu2Hsyq\n+tiYdS+gPTJ2MFTT1afRSUnyXVb1GtxCE3jfNAjqY9aeWE98mvWjXgXcg+a9chDAfkZzdNe7x2os\nt/38dgfOG/r5zbmnqg1I29BMNRk+WOmmmsCyC12/Jye5K81QL0xwqLcNMLcNx1bV+ZOo26eF/Hed\n5NvA79IcyPYLVvUEjtXjuth6wA6j+UT3SuDvgMfTrKc0rm/STEb8wQRqDbtvVT1r6PqR7RvWgpPk\nxKp6AU2QWQoM/rGeQbP200RU1c1pFrwtYDOaT3NjBTDgzuPODVmDS2kmA9+XZqLxT2nau2ADWJLt\naZYBuEdVPTnJbjTrP431iZlmuP9+ND0cgyHIYtXvyVx1dWRsl3aj+TS/N01bz6Q5SGMsk+6Jr6p3\nAO9I8oqqmsRivLOZ6FGsbU/UT2nCfhc6e0/OqgVoBz3OuyWhqs4Yt3Y7D3DBzh2eKauvhzY4uOT7\n89ScUezXRdFFFcAG3YNJfltVfzxuvSSfonmD2xK4JMnZrP7pfv8xH2KahrAGi/69iCbYDtaagRkL\nAM5VmvXcngMso1lg8QM0hxWP67+TPKWqPjOBWsM+AdxI88Z37YRrd+U4Jrio8JA9x+1pXoMvJXkd\nzaK3+9IEm4kPdU7Y8TT/ZAdrEv4RTY/xuGfleDgd9MQD1yXZspoVxP+W5p/hmyd0MNDJaZYbuEs7\nnHww8C8TqNuVLt+Th5dD2JSmZ/dcxlyUdkoND23eAnwamPSSPhNTVVfNNpdx3LqLbQhy8El+i6ra\nMc3qvS+tqjmNLac5aifAUcDwEUyhOeR5rFV7ZwxhhWahzQU5hJXklcDLaBb3Gw4bg67YSRxp+u80\nYeCzk5yI304k3ZwmPN/M5CaSfrOqHjiBJvYmyTlVtWeGjr6axIEDST4I/GNVXTKRhq6q28mRsV1K\ncklV7baubXOo+xHglVU10Z74wZBg+w/mzTSrkr9h3Pe3tvZRNAukDv/89umoR3ps7f+ME+hhWkGS\newHHzOhx26Ak2QKgqn4+321Zm+G5jFW1S9sZ8ZGqGutcnouqBww4hqar8JMAVXVhxjjvWDUL6JHk\nDoPLA2kWLhxLVV1Ac5qHrdrrk5gM3YkaWvSvql7W0WM8tx0i27cdcjq7Vl9sca51t2wnRu/M0PIk\nE/DVJA+qqosnWLNrE11UeMgjgQva+U+/Zsw5Ell1FN4/tP+sF3KvyUznJXlkVX0NIMkjGGMIsoee\n+K5WJQfYt/353bZ0SJK3AQsqgM0YEjuB5gMbNPN99qGbaQXXAPfvoO6ClxnLGSW5nibofnNeG7Zm\nEz8jByy+AEZVXZ3Vz6Ix51MGTfkRN53oKnwBpDlx+luZ/PndZjsf3VcZ/0i6vYEXTyp09OTVNB9Q\n7pPmVBp3YzKnyXnSBGoMu3s74Xz/JCcxY5h7QsNjE5VVh9bfgSacf6+9vhPN4sJz9VZW9cQfMPyQ\n7bZxTXxV8q7fOzsw+Ge6K83iq5+geX2fD5w9iQdI8i5WTdvYiGY5mAX3e9yT9wOvrtWXM3o/zWLk\nC1EnZ+RYbEOQH6VZmfzdNCf1PAx4eM3xNAHTfsTNtElzeqB9a8b53Wr8NaouZtX56B6a9nx0477O\naZZwuJ3q5jQ0E5Fm6YE/p+kpvolmZfl3VbuG0EKR5A9phh735va9R1Vjrn3VhTX9PgyM+3uRjta+\nSrO22pOAi6s5/+rdgQdV1ZxP8t31e2dX0iyB89Rq15lrezk+XVVzHkkZqj18QNgtwJVVtRDDaOcy\ny9qDs21bKJL8Fc0Iyr40v9MHAx8e9+CVxRbA7kqz4vk+NJ9eTgUOqzmcwb4Ps829mcR8nGmVGYch\nt/N/Lhz30OSheU8XAI+oql8n+VYt0EX/upTkZJoJ4h9qN/0RzelRxp0gPlFJHlNVX0nyhqp603y3\nZz4N9ybRnId1YEvgK1U1p3MUzvI427H6GUQW3KrkXUuzoPCDB3NQ297Aizo6wGSDlY6XM5q0JH8J\n/BAYfNg5tSZwRo5FMwSZ5kSZL6iq5813W9bDNB0F2YdTknwO+Pf2+nOASRy52NWJyafRA2dMBv9i\nkolOnJ+QdwIPoxly26ADGM1ZCz5Ldz3x+9Ocx/QeNP9kdqQZMt3gPqDQzP86O6svanrcOAWHhqZn\ntcCnLHTlYJpzVw6WqTmTCS5n1IEtaNp3A82BYhOZE7jYesDOqao957sdo+rziJtpkeRZrFol+8yq\n+vja9p9D/Ymdj24aJfk3mlMQDU8Qf3lVvXB+W7a6JF+jeZN7Bs0b3mqq6pW9N2qRaof+f59muH/3\nJI8Hnl9Vh8xz0+ZFmkVNH9tePaPGXNR0aGj65e334V6fqqrDb38vLURJHkzTMfAs4Jqq2meseoss\ngL2dZgLsf9AcvQIszAm7sNqRN4P1RH5Oc0Taue0RktJEzJggviuw2gTxcZdImLR2OsE+NJPMb3cq\nsao6/nZ30pwk+UZVPbwNYrtX1W8X8nycaZXZT8Z9u3l9i1mSY6rqVUNH9q5mAkf0dirJEpr1/A4C\nthy393LRDEG2BnOnjmy/DxYLXXATdlsPb78+SdPW59F86v+zJB+pqqPns3F9ye1P+HrbTUxgvS4B\n8LT5bsD6qOYULSclubSqLpzv9ixyN7brMZ0BfCjJD2k+DGqyMpjb2F55NGMebTqFBr1/b53XVqyn\nJIfSLAp+N+AjwJ9OYs3DxRbA/ptVZ1invfyzJA9doD1K9wT2GCxCl2axt0/TLEtxLs1pbha96uiE\nr1plIR+ZuQ4/bufj3DYsTXNgzTXz2KbF5kLgl8Bf0HwI3JoJrPKt2zkE+Nf2CNHQTDlZyPOeJq6q\nzm0vfgP4VVX9Fm6bw32neWvYut0LeNWkc8RiG4L8MKv3KD2NpkdpKc2qtQsq0KQ5weeDqurm9vqd\naI76u99s3dXShibJaTST0IfnzTyvqvadv1YtLl0tb6HZtQFscF7LDVI7x3Ofoc6HLWiOLFyo64B1\nYrH1gE1bj9KHgK8n+UR7/enAh9tF3hbikWlS37arqg8OXT8uyavmrTWLyNDyFvedksVSp9o0Lrzd\noU1r6PRDVfXzdj26DcpiC2DbMXSKDprz/m1fVb9KMrFzC05KVf1dks+yanjlz6pqsOjkNC2nIXXl\n+iTPZ9XSJM8FFuS6flOo0+UtdDv/SrPw9oHt9RcAHwQ2xIW3f5Fkj8EBckkexga4BNNiG4J8Pc05\nm4Z7lD5Js8bN+6dsjTBpg9cewv8u4FE0czq/Cryiqq6e14ZJ68mFt1dJsidwEvB9mulCS4DnDM0R\n2yAsqgAGkOThrOpR+spQj5KkKZPkeJrJrz9pr28LvLWqNqjJy5p+Sc4CXjNj4e23VtWj5rdl8yPJ\nYEkcgMsGc6E3JIsugElaPNawdpIHqGjquPA2JPn9qjo9yazDrlX1sdm2L1aLbQ6YpMVloyTbzOgB\n831LU6U9r+2uVfWQJFsBVNXP5rlZ8+FxwOk004NmKladmmiDYA+YpAUryQuB19EsfgjNKtT/p6pO\nXPO9pIVncMaB+W6HFg4DmKQFLclurDqbxemTWIFa6luStwDXc/tT5W0wR5wOnX5vVlX1T321ZSGw\nK1/SgtYGLkOXpt1zaIbZDp2x/T7z0Jb54llPhtgDJklSx5JsRhO+9qYJYmcC/1xVG9z6V2psaCcC\nlSRpPhwP3B94J83adru12zY4SXZJ8oUk32yvPzjJ3853u/pmD5gkSR1LcklV7baubRuC9jRMrwHe\nN1hSJsk3q+qB89uyftkDJklS985L8sjBlSSPADbUhcLvXFVnz9h2y7y0ZB45CV+SpO49DPhqku+1\n13cELktyMVBV9eD5a1rvrk9yX5q5cCT5Q+AH89uk/jkEKUlSx9rzmq5RVV3VV1vmW5L7AO8HHk1z\nRoDvAs/bkF4DMIBJkqQezLIO2GY0U6F+Aa4DJkmS1IXBOmC7AnsCnwACvACYOSds0bMHTJIk9SbJ\nGcBTq+qm9vqWwKer6vfmt2X98ihISZLUp+2B3wxd/027bYPiEKQkSerTCcDZST7eXj8AOG7+mjM/\nHIKUJEm9SrIH8Nj26hlVdf58tmc+GMAkSZJ65hwwSZKknhnAJEmSemYAkyRJ6pkBTJLWIsnjknxq\nvtshaXExgEnSkCSzvS96tJKkiTKASVo0kvxVkj9vL789yRfay49P8m9JDkpyUfv1lqH73ZTkrUnO\nBx6Z5ElJLk3yDeCZQ/s9Lsn5Sc5Lcm6Szft+jpIWBwOYpMXkTFatLfQwYPMkG7fbLgfeAiwDHgrs\nmWT/dt/NgbOqanfgXOD9NKdKeTiwZKj+XwKHVtVgDaNfdft0JC1WBjBJi8m5wMPac8v9GjiL5qS/\njwV+Aqyoqhuq6rfAh4DBueduBT7WXr4f8J2q+k57/d+G6n8FeHuSVwDbtHUkab0ZwCQtGlV1C3Al\n8GKasHQm8Hjgvu32rOGuv6rVV6Wedb+qOgo4BNgM+EqSXSbRbkkbHgOYpMXmTOCvgDOALwN/BpwP\nnAP8XpJt22HJ5wIr2vsMB65vAzsluXd7/bmDG5Lcp6q+VVVHt/Xu1+UTkbR4GcAkLTZn0szbOquq\nfkgzT+uMqroOOJwmdJ0PfKOq/ru9z229X1X1a+AlwGfaSfgrh2q/KsnFSS4AfgN8tusnI2lx8lyQ\nkiRJPbMHTJIkqWcGMEmSpJ4ZwCRJknpmAJMkSeqZAUySJKlnBjBJkqSeGcAkSZJ6ZgCTJEnq2f8H\nAxEZbn5mlpoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd69d7d2e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "word_his.plot(kind='bar',x=word_his['words'],legend=False,title='Top 20 most freq words',figsize=(10,6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vec = CountVectorizer()\n",
    "c_fit = vec.fit_transform([' '.join(topwords)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf_vec = TfidfTransformer()\n",
    "tf_fit = tf_vec.fit_transform(c_fit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ctr_features = vec.transform(Data_train)\n",
    "tr_features = tf_vec.transform(ctr_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cte_features = vec.transform(Data_test)\n",
    "te_features = tf_vec.transform(cte_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "models = {'BernoulliNB':BernoulliNB()\n",
    "          ,'Logistic' : linear_model.LogisticRegression(C=1e5),'Decision Tree' : DecisionTreeClassifier(random_state=20160121, criterion='entropy'),\n",
    "          'Perceptron': linear_model.Perceptron(n_iter=1000)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   TF-IDF Accuracy          Model\n",
      "0         0.915047  Decision Tree\n",
      "1         0.869049    BernoulliNB\n",
      "2         0.888955       Logistic\n",
      "3         0.851753     Perceptron\n"
     ]
    }
   ],
   "source": [
    "results = pd.DataFrame()\n",
    "\n",
    "foldnum = 0\n",
    "tfprediction = {}\n",
    "cprediction = {}\n",
    "for name,model in models.items():\n",
    "        \n",
    "        model.fit(tr_features, Train_labels)\n",
    "        tfprediction[name] = model.predict(te_features)\n",
    "        tfaccuracy = metrics.accuracy_score(tfprediction[name],Test_labels)\n",
    "        \n",
    "        #model.fit(ctr_features,Train_labels)\n",
    "        #cprediction[name] = model.predict(cte_features)\n",
    "        #caccuracy = metrics.accuracy_score(cprediction[name],Test_labels)\n",
    "        \n",
    "        results.loc[foldnum,'TF-IDF Accuracy']=tfaccuracy\n",
    "        #results.loc[foldnum,'Count Accuracy']=caccuracy\n",
    "        results.loc[foldnum,'Model']=name\n",
    "        foldnum = foldnum+1\n",
    "print (results)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for  Decision Tree\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.82      0.78      0.80     31169\n",
      "        pos       0.94      0.95      0.95    110945\n",
      "\n",
      "avg / total       0.91      0.92      0.91    142114\n",
      "\n",
      "\n",
      "\n",
      "Classification report for  BernoulliNB\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.72      0.67      0.69     31169\n",
      "        pos       0.91      0.93      0.92    110945\n",
      "\n",
      "avg / total       0.87      0.87      0.87    142114\n",
      "\n",
      "\n",
      "\n",
      "Classification report for  Logistic\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.80      0.66      0.72     31169\n",
      "        pos       0.91      0.95      0.93    110945\n",
      "\n",
      "avg / total       0.88      0.89      0.88    142114\n",
      "\n",
      "\n",
      "\n",
      "Classification report for  Perceptron\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.70      0.57      0.63     31169\n",
      "        pos       0.89      0.93      0.91    110945\n",
      "\n",
      "avg / total       0.84      0.85      0.85    142114\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for name,model in models.items():\n",
    "    print (\"Classification report for \",name)\n",
    "    print(metrics.classification_report(Test_labels, tfprediction[name]))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1cd6bad33c8>"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAHDCAYAAADr+CQ1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYZFV97vHvC6igAkogcASBiAYMKopKfKLRRqKAuRA9\nURnFCySGGDwa40mIxsQ2Xk9ijInEKIoE9QSMJ8kRlQARp9WciJdwEQ0oKg7DVQFF5CIIv/NH7Yaa\ntma6F9P03jXz/TxPP1N77V1Vv66uqX57rbXXTlUhSZKkpdui7wIkSZKmjQFKkiSpkQFKkiSpkQFK\nkiSpkQFKkiSpkQFKkiSpkQFKUrMkeyS5I8minyFJXpTksytU19ZJPpbk+0k+vBLPKWnzZICSNnFJ\nvp3kliQ7LGg/twtBu9/Nh25ZRG7isWNB7Afd17eSHHs36wH4DWAn4IFV9dyNeBxJ2iADlLTpK+AS\nYNV8Q5JHANvQFoLuKQVsX1XbAc8D/jTJ01sfpOsN2wP4et2NFYKTbNl6H0mbLwOUtHn4IPCise0X\nASeNH5BkuyQfSPKdJJck+eOxfVskeVuS7yb5BvDLE+77viRXJFmb5A1J0lBfAKrqbOCrwCO6x90n\nyZlJrk1yYZJnjz3niUneleQTSW4APg38KXB415t1ZEZe2/XCXZXk75Ns191/vvfrqCRrgLPG2l6c\n5NLueY9O8rgk5ye5Lsk7x2p4SJKzklzTvW4fmn/8bv8lSV7V3fd7SU5Ocu+x/Yd1PYHXJ7l4Pjgu\nw+sp6R5mgJI2D2cD2ybZu+upeS7wIbrg0jkO2BbYE5gBXpjkyG7fbwPPAPYDHsdoqGzcScCtwEOA\nxwBPA36rob4AJHki8HPAOUnuC5zZ1bkjcDjwriT7jN1vFfCGqtoWOAh4M3BKVW1XVScCRwIvBJ7S\n1bZt932OezKwD3DwWNsBwEMZvU7vAF4DPJVRsHtOkl8cq/vNwC7Aw4HdgNkFj/9s4OnAzzB6/V7c\nfa8HMHrdXlVV23d1fLu7z8a+npLuYQYoafMx3wv1NOBC4Ir5HWOh6o+q6qaqWgP8JfCC7pBnA++o\nqiuq6vvAW8buuzNwKPDKqrqlqq5hFDruHDJcRIDvJrkWOB44tqrmgF8BLqmqD9TI+cA/dbXM+2jX\na0VV3TrhsZ8HvL2q1lTVTcCrGfVQzX/2FfC6qrq5qn401vZnVXVrVX0SuBE4uaquraorgM8yCjVU\n1Ter6qyq+nFVXQv8FaOwNu6vq+rq7nX7GPDorv0o4ISq+lT3WFdW1deT/PRGvp6SVsBWfRcgacV8\nCPgMo56QDyzYtyOjz4NLx9rWALt2tx8ErF2wb97uwL2AK7tRpnRf44+1IQX81IR5S3sAT0hyXbcd\nYMsFta9lwx60oNY1jL7PncfaLptwv++M3b4ZuHrB9v0BurDz18Avdm1bAtexrvH73gT8t+72g4FP\nTHjuPdi411PSCjBASZuJqro0ySWMejeOWrD7GuA2Rr+8L+ra9gAu725fyegXPmP75q0FbmFyCFqq\n8JMT2tcCc1V18ITj5y32fFewbq17MPo+r+au72djJtK/GbgD2Leqrk9yGPDORe4zby2w13raN/b1\nlHQPcwhP2rwcBTy1qm4eb6yqO4B/BN6U5P5J9gBeyWjYj27fy5PsmuSBwLFj972K0Vylv0qybTdx\n+yFJnrzEmtY3OfrjwM8mOSLJVknu1U3m3nvJ3y2cDLwyyZ5J7g+8idEcqTs28Nwtk7W3BX4I3JBk\nV+APGu57AnBkkgO71+xBSfZehtdT0gowQEmbvjt7Marqkqo6Z9I+4OWMhpi+xWio70PdRGyA9wJn\nAOcDX2I0F2ncC4F7A//FaAjrI4wmVjfVt05j1Q8ZTb4+nFFP0hXAW4H7LPFxAd7PKAR+Bvgmo+/v\n5Ys898K2DW2/HngsMD+/aeHrst4epKr6IqNJ7u8ArgfmGA2Hwsa9npJWQBbrIU5yAqPJnFdX1aPW\nc8zfMBoWuBF4cVWdt9yFSpIkDcVSeqBOZN3Te9eR5FBgr6p6GHA08O5lqk2SJGmQFg1QVfXvwPc2\ncMhhdGfFVNXnge2705olSZI2ScsxB2pX1j2V+HLuOvVZkiRpk+MkckmSpEbLsQ7U5ay7Psxu3LV2\nzDqSuKaJJEmaGlU1cWmTpQao+ZVwJzkVOAb4cJInAN+vqqvXcyyuC/eTZmdnmZ2d7bsMTQHfK2rh\n+0VL5Xtlsg1dw3vRAJXkHxhdWPSnklwKvI7R+iRVVcdX1WlJnpHRFdpvZLSuiSRJ0iZr0QBVVc9b\nwjEvW55yJEmShs9J5AMwMzPTdwmaEr5X1ML3i5bK90q7RVciX9YnS7w2piRJmgpJ1juJ3B4oSZKk\nRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYo\nSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKkRgYoSZKk\nRgYoSZKkRgYoSZKkRlv1XUBf9txlF9ZcfXXfZQzOHjvvzLevuqrvMiRJGrRU1co9WVIr+XwbkoRh\nVDIsAYbyM5IkqU9JqKpM2ucQniRJUiMDlCRJUiMDlCRJUiMDlCRJUiMDlCRJUiMDlCRJUiMDlCRJ\nUqPNdiFNSbon7LLLnlx99Zq+yxicnXfeg6uu+nbfZUjLxoU0tQ4X0pQ2ThLw02WC+NmiqeNCmpIk\nScvIACVJktTIACVJktTIACVJktTIACVJktTIZQykJdhlt124+vKr+y5jUHbedWeuuuyqvsuQpF64\njIHW4TIGkyWB2b6rGJhZ3yuTuIzB+riMgaaPyxhIkiQtIwOUJElSIwOUJElSIyeRS5LUk91334W1\naz1BZdyDH7wzl146/BNUDFCSJPVk7dqrWb267yqG5cADpyNQOoQnSZLUyAAlSZLUyAAlSZLUyAAl\nSZLUyAAlSZLUyAAlSZLUyAAlSZLUyAAlSZLUyAAlSZLUyAAlSZLUyAAlSZLUaEkBKskhSS5K8vUk\nx07Yv12SU5Ocl+SCJC9e9kolSZIGYtEAlWQL4DjgYGBfYFWSfRYcdgzw1ap6NHAg8JdJvFCxJEna\nJC2lB+oA4OKqWlNVtwGnAIctOKaAbbvb2wLXVtWPl69MSZKk4VhKgNoVWDu2fVnXNu444OeSXAGc\nD7xiecqTJEkanuWaRH4wcG5VPQh4DPC3Se6/TI8tSZI0KEuZp3Q5sPvY9m5d27gjgbcAVNU3k1wC\n7AN8aeGDzc7O3nl7ZmaGmZmZpoIlSZLuCXNzc8zNzS3p2FTVhg9ItgS+BhwEXAl8AVhVVReOHfO3\nwHeq6vVJdmYUnParqusWPFYt9nwrJQnDqGRYAgzlZzQkSWC27yoGZtb3yiRJwE+XCeL7ZYIkrF7d\ndxXDcuCBw/lsSUJVZdK+RXugqur2JC8DzmQ05HdCVV2Y5OjR7joeeCPw90m+3N3tDxeGJ0mSpE3F\nkpYaqKrTgb0XtL1n7PaVjOZBSZIkbfJciVySJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqS\nJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmR\nAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqS\nJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmR\nAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqS\nJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmR\nAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKmRAUqSJKnRkgJUkkOSXJTk60mOXc8xM0nOTfKVJKuXt0xJ\nkqTh2GqxA5JsARwHHARcAXwxyUer6qKxY7YH/hZ4elVdnmTHe6pgSZKkvi2lB+oA4OKqWlNVtwGn\nAIctOOZ5wD9V1eUAVXXN8pYpSZI0HEsJULsCa8e2L+vaxv0ssEOS1Um+mOQFy1WgJEnS0Cw6hNfw\nOPsDTwXuB3wuyeeq6hvL9PiSJEmDsZQAdTmw+9j2bl3buMuAa6rqFuCWJJ8B9gN+IkDNzs7eeXtm\nZoaZmZm2iiVJku4Bc3NzzM3NLenYVNWGD0i2BL7GaBL5lcAXgFVVdeHYMfsA7wQOAe4DfB54blX9\n14LHqsWeb6UkYRiVDEuAofyMhiQJzPZdxcDM+l6ZJAn46TJBfL9MkITVnre+jgMPHM5nSxKqKpP2\nLdoDVVW3J3kZcCajOVMnVNWFSY4e7a7jq+qiJGcAXwZuB45fGJ4kSZI2FUuaA1VVpwN7L2h7z4Lt\ntwFvW77SJEmShsmVyCVJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJ\nkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZ\noCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJ\nkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZ\noCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJ\nkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZoCRJkhoZ\noCRJkhoZoCRJkhoZoCRJkhotKUAlOSTJRUm+nuTYDRz3+CS3JXnW8pUoSZI0LIsGqCRbAMcBBwP7\nAquS7LOe494KnLHcRUqSJA3JUnqgDgAurqo1VXUbcApw2ITj/gfwf4DvLGN9kiRJg7OUALUrsHZs\n+7Ku7U5JHgT8elX9HZDlK0+SJGl4lmsS+TuA8blRhihJkrTJ2moJx1wO7D62vVvXNu5xwClJAuwI\nHJrktqo6deGDzc7O3nl7ZmaGmZmZxpIlSZKW39zcHHNzc0s6NlW14QOSLYGvAQcBVwJfAFZV1YXr\nOf5E4GNV9c8T9tViz7dSkjCMSoYlwFB+RkOSBGb7rmJgZn2vTDL6O9LX5SfF98sESVi9uu8qhuXA\nA4fz2ZKEqpo4qrZoD1RV3Z7kZcCZjIb8TqiqC5McPdpdxy+8y0ZXLEmSNGBLGcKjqk4H9l7Q9p71\nHHvUMtQlSZI0WK5ELkmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS\n1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgA\nJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS\n1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgA\nJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS\n1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgA\nJUmS1MgAJUmS1MgAJUmS1GhJASrJIUkuSvL1JMdO2P+8JOd3X/+e5JHLX6okSdIwLBqgkmwBHAcc\nDOwLrEqyz4LDvgU8uar2A94IvHe5C5UkSRqKpfRAHQBcXFVrquo24BTgsPEDqursqrq+2zwb2HV5\ny5QkSRqOpQSoXYG1Y9uXseGA9FvAv25MUZIkSUO21XI+WJIDgSOBJy3n40qSJA3JUgLU5cDuY9u7\ndW3rSPIo4HjgkKr63voebHZ29s7bMzMzzMzMLLFUSZKke87c3Bxzc3NLOjZVteEDki2BrwEHAVcC\nXwBWVdWFY8fsDpwFvKCqzt7AY9Viz7dSkjCMSoYlwFB+RkOSBGb7rmJgZn2vTJIE/HSZIL5fJkjC\n6tV9VzEsBx44nM+WJFRVJu1btAeqqm5P8jLgTEZzpk6oqguTHD3aXccDfwLsALwro0+P26rqgOX7\nFiRJkoZjSXOgqup0YO8Fbe8Zu/0S4CXLW5okSdIwuRK5JElSIwOUJElSIwOUJElSIwOUJElSIwOU\nJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElS\nIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOU\nJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElS\nIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOU\nJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElS\nIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSIwOUJElSoyUFqCSHJLkoydeTHLueY/4mycVJ\nzkvy6OUtU5IkaTgWDVBJtgCOAw4G9gVWJdlnwTGHAntV1cOAo4F33wO1brLm+i5A0+OSvgvQdJnr\nuwBNifPO67uC6bOUHqgDgIurak1V3QacAhy24JjDgA8AVNXnge2T7LyslW7C5vouQNPj230XoOky\n13cBmhIGqHZLCVC7AmvHti/r2jZ0zOUTjpEkSdokOIlckiSpUapqwwckTwBmq+qQbvuPgKqq/zV2\nzLuB1VX14W77IuApVXX1gsfa8JNJkiQNSFVlUvtWS7jvF4GHJtkDuBI4HFi14JhTgWOAD3eB6/sL\nw9OGipAkSZomiwaoqro9ycuAMxkN+Z1QVRcmOXq0u46vqtOSPCPJN4AbgSPv2bIlSZL6s+gQniRJ\nktblJHJJkqRGBqgeJNkmyau7yfckeWi3GKkkSZoCBqh+vB8I8KRu+wrgzf2Vo2mRZMcknoyh9Upy\nTJIHjG0/MMnv9lmTtClyDlQPknypqh6X5NyqekzXdl5VeQ1B3ak7o/WtwHXAG4APAjsy+sPnhVV1\neo/laaAmfZaMf9ZI85LsBLwE2JOxk8qq6qi+apomS1nGQMvv1iRbAwWQ5GeAW/stSQN0HPAaYHvg\nU8ChVXV2dy3KkwEDlCbZMkmq++s4yZbAvXuuScP0UeCzwCeB23uuZeoYoPrxZ4x++e2W5CTgKcBv\n9luSBmirqjoTIMmfVdXZAFV1kaN42oDTGa3J955u+2gM25rsvlV1bN9FTCsDVA+q6vQk/wn8AqO5\nUH9QVd/puSwNzx1jt29esM+xd63PsYxC00u77X8D3tdfORqwjyd5RlWd1nch08g5UD1JcjiwV1W9\nKcmDgZ+uqv/suy4NR5LbGS1MG2Ab4Kb5XcDWVXWvvmqTNP2S3ADcj9EUktu65qqq7fqranoYoHqQ\n5DjgXsCTq+rhSXYAzqiqx/dcmqQpleQfq+o5SS5gQg9lVT2qh7KkTZZDeP34haraP8m5AFV1XRIn\neUraGK/o/v2VXqvQVEnya8CTu825qvp4n/VME9eB6sdtSbbgrrPwfop157tIJLkhyQ+6f28Y274p\nyY/7rk/DUlVXdjd/t6rWjH8BrgOln5DkrYyC9391X69I8pZ+q5oeDuH1IMkLgWcCj2O0qOZzgNdX\n1Sm9FqZBS3J/4BhGE4T/pape1XNJGqAk51TV/gvavuwQnhZK8mXg0VV1R7e9JXCu75WlcQivB1X1\nge4svF9iNCH42VX1lZ7L0kB1q0r/HvBC4B+Ax1fVtf1WpaFJ8lJGPU17db8Y520L/L9+qtIUeACj\nxXphtOaclsgAtcK6hP/lqtoX+Grf9Wi4kuwIvAp4LqOeysdU1fX9VqUB+wfgX4G3AH801n5DVV03\n+S7azL0FODfJakZ/zD+Zdd872gCH8HqQ5GPA71TV5X3XouFKciPwXeBE4IaF+6vq7StelAYvyV7A\nZVX1oyQzwKOAD1TV9/utTEPSXVNzN+DHwPwZ4F+oqqv6q2q6GKB60KX9xwKfY7TODwBV9azeitLg\nJJllAwtmVtXrV64aTYsk5zGaX7kncBqjy3XsW1XP6LMuDU+SC6rqkX3XMa0cwuvHG/suQMNXVbN9\n16CpdEdV/TjJs4B3VtU755dMkRY4J8njq+qLfRcyjQxQKyjJmVX19Ko6q+9aNHxJ/rCq/jzJO5m8\nMOLLeyhLw3dbklWMTjr41a7NVes1yc8Dz0+yhruuelCehbc0BqiVtVPfBWiqXNj9+6Veq9C0ORL4\nHeBNVXVJkp8BPthzTRqmg/suYJo5B2oFJfkW8D/Xt7+q/nkFy5EkbcaSfLCqXrBYmyazB2plbc/o\nMguZsK8AA5Tu1J2tuaFJ5L+2guVo4LwWnu6Gfcc3umV2HttTLVPHALWy1lTVUX0Xoanxtr4L0FTx\nWnhakiSvBl4DbJPkB9z1R/2twPG9FTZlHMJbQUnOrarH9F2HJElJ3lJVr+67jmllgFpBSR7hJVu0\nVOsbipnnkIwmSXIDP/m+uZ7RyQivqqpvrXxVGqJuMc1nAk9i9J75bFX9336rmh4GKGmgkuyxof1V\ntWalatH0SPIG4DJGl3YJcDiwF3AO8NKqmumvOg1JkncBDwVO7pqeC3yzqo7pr6rpYYCSpE1IkvOr\nar8FbedV1aMn7dPmK8lFwMOrCwJJtgC+WlUP77ey6bBF3wVI2rAkNyT5Qfd1S5Lbu4mf0iQ3JXlO\nki26r+cAt3T7/ItZ474B7D62/eCuTUvgWXg9SPJEYBbYg9HPYH7114f0WZeGqaq2nb/dzVk4DHhC\nfxVp4J4P/DXwrm77c8ARSbYBXtZbVRqibYELk3yBUbg+APhSklPBpVIW4xBeD7pu01cC/wncPt9e\nVdf2VpSmimd0StpYSZ6yof1V9emVqmUa2QPVj+ur6l/7LkLTobso7LwtgMdx15CMtI4kuwHvBJ7Y\nNX0WeEVVXdZfVRqiqvp0d7LKw6rqk10v5VZVdUPftU0DA1Q/Vif5C0Yrj/9ovrGqzumvJA3Yr47d\n/jHwbUbDeNIkJzI6A+/Z3fYRXdvTeqtIg5TkJcBvAzswOlNzN+DdwEF91jUtHMLrQZLVE5qrqp66\n4sVI2qTMn3G3WJuU5DxG854+Pz8lIMkFVfXIfiubDvZA9aCqDuy7Bk0Ph2TU6NokR3DX2j6rAOdX\napIfVdWto3NTIMlWeKbmkrmMQQ+SbJ/k7Um+1H39ZZLt+65Lg3UicCrwoO7rY12bNMlRwHOAq4Ar\ngd8AXtxnQRqsTyeZvybe04CPMPp80RI4hNeDJP8EfAU4qWt6AbBfVT1r/ffS5sohGW2sJL9XVe/o\nuw4NS7dw5m8CT2e0nM4ZwPvKYLAkBqge+AtRLZKcxajHaXxI5siqcqKnliTJpVW1++JHanOS5H7A\nLVV1e7e9JXCfqrqp38qmg0N4/bg5yZPmN7qFNW/usR4N26QhmSN7rUjTJn0XoEE6C9hmbHsb4JM9\n1TJ1nETej5cCJ3XzngJch3MUtB7dRYNdEVgbw6EGTbJ1Vf1wfqOqfpjkvn0WNE0MUD2oqvOA/ZJs\n1217XTOtV5KdgJcAezL2f7aqjuqrJg1PkhuYHJTCur0M0rwbk+w/vwZhksfiaMiSGaBWUJIjqupD\nSX5/QTsAVfX2XgrT0H2U0dIFn2Ts0j/SuPFrJkpL9ArgI0muYBS0dwGe229J08MAtbLu1/3rB51a\n3Leqju27CEmbju4MvHsD+wB7d81fq6rb+qtqungWnjRwSd4I/EdVndZ3LZI2HV6UfON4Fl4Pkvx5\nku2S3CvJWUm+260cLE3yCuDjSW5O8oMkNyRx3pykjXVWkv+e+XkkamIPVA/m13xK8kzgV4DfBz5T\nVfv1XJoGpvtge3BVXdp3LZI2Ld2JB/djNLfyZkbzoKqqtuu1sCnhHKh+zL/uvwx8pKqu9w8ATVJV\nleQTgBf3lLSsPPFg4ziE14+PJ7kIeCyjLtSdgFt6rknDdU6Sx/ddhKRNS0aOSPIn3faDkxzQd13T\nwiG8niTZAbi+qm7vFi7brqqu6rsuDU8Xth8KrAFu5K5u9kf1WpikqZbk74A7gKdW1cOTPBA4s6r8\ng20JHMJbQUmeWlWfSvKssbbxQ/555avSFDi47wIkbZJ+vqr2T3IuQFV9L8m9+y5qWhigVtZTgE8B\nvzphX2GA0gRVtaa7duLDqurEbsj3/n3XJWnq3dZdQLjgzqse3NFvSdPDITxp4JK8DngcsHdV/WyS\nBzE6+eCJPZcmaYoleT6jlcf3B05idKHy11bVR3otbEo4ibwHSd6c5AFj2w/sFkuUJnkmo4sJ3whQ\nVVfgavaSNlJV/W/gD4G3AFcCv254WjqH8PpxaFW9Zn6jG3d+BvDaHmvScN3aLWcw381+v8XuIEnr\nk2Rr4HcYnZxyAfCeqvpxv1VNH3ug+rFlkvvMbyTZBrjPBo7X5u0fk7wHeECSlzC6qPB7e65J0vQ6\nidG0gAuAQ4G39VvOdHIOVA+SHMtoIvmJXdORwKlV9ef9VaUhS/I04OmMljA4o6r+reeSJE2pJBdU\n1SO721sBX6iq/Xsua+oYoHqS5BDgl7rNf6uqM/qsR9MhyY7AteV/XEl3U5JzxgPTwm0tjQGqJ0n2\nYHRa+ie7hTS3rKob+q5Lw5HkCcBbgeuANwAfBHZkNPT+wqo6vcfyJE2pJLfTnZTCqFd7G+AmvBZe\nEwNUD7ovamBDAAADFElEQVR5LL8N7FBVeyV5GPDuqjqo59I0IEm+BLwG2B44ntHJB2cn2Qc4uaoe\n02uBkrQZcxJ5P44Bngj8AKCqLgZ+uteKNERbVdWZ3WnFV1XV2QBVdVHPdUnSZs8A1Y8fVdWt8xvd\nJD67ArXQ+IrANy/Y5/tFknrkOlD9+HSS1wDbdGdX/S7wsZ5r0vDsl+QHdHMUutt021v3V5YkyTlQ\nPUiyBfCbjJ2WDrzPM6skSZoOBqiedBdtpKq+23ctkiSpjXOgVlBGZpNcA3wN+FqS7yb5075rkyRJ\nS2eAWlmvZHT23eOraoeq2gH4eeCJSV7Zb2mSJGmpHMJbQUnOBZ5WVdcsaN8JONN1fSRJmg72QK2s\ney0MT3DnPKh79VCPJEm6GwxQK+vWu7lPkiQNiEN4K2jB9YfW2QVsXVX2QkmSNAUMUJIkSY0cwpMk\nSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJI0GEnuSPKBse0tu8sdndr4OJck2WFjj5Gk9TFASRqSG4FH\nJLlPt/00YO3deJylnF7sKciS7jYDlKShOQ345e72KuDk+R1JHpjkX5Kcn+Q/kjyya98hyRlJLkjy\nXkZrq83f5/lJPp/knCR/l2R+353HSFIrA5SkISngFGBV1wv1KODzY/tfD5xTVfsBfwzMD/e9Dvhs\nVT0S+Bdgd4Ak+wDPBX6hqvYH7gCevxLfiKRN21Z9FyBJ46rqK0n2ZNT79AnW7Sl6EvCs7rjVXc/T\ntsCTgWd27acl+V53/EHA/sAXu56nrYGrVuL7kLRpM0BJGqJTgb8AZoAdFzl20lym8WG6k6rqj5ev\nNElyCE/SsMwHn/cDr6+qry7Y/1ngCIAkM8A1VfVD4DN0Q3NJDgUe0B1/FvAbSXbq9j0wye736Hcg\nabNgD5SkISmAqrocOG7C/lng/UnOZ3TG3ou69tcDJyc5HPgP4NLucS5M8lrgzCRbALcCx3T7PQtP\n0t3mxYQlSZIaOYQnSZLUyAAlSZLUyAAlSZLUyAAlSZLUyAAlSZLUyAAlSZLUyAAlSZLUyAAlSZLU\n6P8Dcml6HpCoThkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd6bac5a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results.plot(kind='bar',color=['r','g','b','y'],title='Model Performance',figsize=(10,6),x='Model',legend=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "## comparing negative samples accuracies\n",
    "negcom = pd.DataFrame()\n",
    "for name,model in models.items():\n",
    "    p= recall_score(Test_labels,tfprediction[name],pos_label='neg')\n",
    "    negcom.loc[name,'recall'] = p\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAHDCAYAAAAa83+4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUZWV97vHvA6igMogQiCCoOGAMEFHRG11aSFDQRKLX\nCWdIiBNXo96IGq8WEaNJjPEqGkWRgBPqjYk4AaI0ahQRAUVtFCMiswMIiCAt/O4fZxccyuru6uG8\n++zq72etXnX2UKeeqj4UT7/vu/dJVSFJkqQ2Nuo7gCRJ0obE8iVJktSQ5UuSJKkhy5ckSVJDli9J\nkqSGLF+SJEkNWb4kSZIasnxJWqkkP05yQ5Kt5+0/O8nNSXZax+e/Ocm9VnH8uUl+m+SaJNd2H9++\nLl9zDbItS3L9vK/90HV8zp2779nfvdIGzF8AklalgAuAA+d2JPlDYLPu2Pp4/tX5alVtUVWbdx9f\nstBJEyg0Bbxo3tf++jo+Z7rnzVo/QbLWnytpOli+JK3OB4Dnjm0/Fzh2/IQkWyQ5LslPk1yQ5G/H\nju3SjSL9sjv+kW7/aYxKyLe7UaWnrEmoJMckeVeSzyS5FphJcvskb0lyYZLLuuN3GPucv0lyaZKL\nkxy0upE3VlKSkuya5OQkv0iyfDx7ksclOSvJ1V2O14996mndx1/OjaQleX2SD4x9/m1Gx5KcmuSI\nJF9Jch1wz+7nfXT3vVyU5A1zpWxlP29J08PyJWl1Tgc2T3K/rhA8Dfggty0mRwKbA/cAZoDnJDmo\nO/YG4KSq2grYEXgHQFU9qju+Wzeq9PG1yHYg8Iaq2hz4L+AfgHsDu3cfdwBeB5BkP+DlwD7AfYA/\nYS1G75LcETiZ0c9gG+DpwDuT7Nqd8ivg2VW1JfB44AVJntAde2T3cYt5I2nzc8zffhbwl4x+xj9h\nVH5/A9wLeCCwb3ccVvLzljQ9LF+SFmNu9GtfYDlw6dyBsUL2qqr6dVVdCPwz8OzulBXAzkl2qKob\nq+qr8557ddNo/yPJlUmu6j7uNXbsk1V1OkBV/QY4BHhZVV1dVdcBb+bWKdOnAMdU1fKquh6YXcT3\n/faxr31mt+9PgQuq6rga+Rbwie75qaovVdV3u8ffAY4HHjXvedd06vDfquq8qroZ2BrYv/s+b6iq\nnwNvY1QCYfU/b0k9s3xJWowPAs8AngccN+/YNsAmjEZk5lzIaNQJ4JWMfteckeTcsRGxxfpaVW1d\nVXfpPp4xduyiuQdJtgXuCHyzK0xXAp8D7tqdcrfx87uMqytBLxn72g/u9u0MPGzuayS5itHPZrsu\nx0OTfLGb8vsl8HxGP6N1MZ57Z+B2wGVjX//dwLbd8b9h3X7ekiZsk74DSJp+VfWTJBcwGnE5eN7h\nn9ONtgDndft2Bi7pPvcK4K8AkjwcOCXJaVX1o/URbV6OXwMPqKrLFjj3MuDuY9s7s3YXDVwELKuq\nx67k+IeAtwOPraoVSf6FWwvgQl/vOkalcc7vL3DO+OddBNwA3LWqfuf5quqnTO7nLWk9cORL0mId\nDDy6m7K7RTcV9jHgjUnunGRn4GWMpipJ8uQkc6NgvwRu7v4AXM5o3dI664rIe4G3daNgJNkhyWO6\nUz4GPC/J/bt1W69byy/1aeC+SZ6VZJMkt0vy4CT3647fGbiqK157MRoVm/MzRt/7LmP7zgEemeTu\nSbYEXrWa7/NyRmvO/iXJ5hm5V5JHdt/zqn7ekqaA5UvSqtwyslJVF1TVWQsdA17CaNTpR8CXgA9W\n1THdsYcAX09yDfCfjKbyftwdmwWO66bPnry22cYcBvwQOL2b8jsZuG+X/0RGa6O+CPwA+MJaPD9V\n9SvgMYzWWF3a/XkzMHdV5YuANyS5Gngt8NGxz70eeCPwX3Pr16rqlO6cbwPfAD61iBzPAW4PfA+4\nEvg4sH13bFU/b0lTIAuMWq+/J0+OZrQ49Yqq2n0l57yd0VTGdcDzquqciQWSpDFJbgbu7ZScpJYm\nPfJ1DLCydREk2R/Yparuw2hR6rsnnEeSJKlXEy1fVfUV4KpVnHIA3ZVT3f1utkyy3SQzSdKYyQ39\nS9JK9H214w7c9hLqS7p9V/QTR9KGpKo27juDpA2PC+4lSZIa6nvk6xJue9+dHbt9vyOJ0wOSJGkw\nqmrBGzm3KF9h5XeRPgF4MfDRJA8DftndkHFBk7wyc6hmZ2eZnZ3tO4YGwNeK1oSvFy2Wr5WFde91\nv6CJlq8kH2b0Jrt3TfIT4PWM7k1TVXVUVX02yeOS/JDRrSZ8GwxJkrSkTbR8VdUzFnHOoZPMIEmS\nNE1ccD9wMzMzfUfQQPha0Zrw9aLF8rWy5iZ6h/v1KclC7yErSZI0dZKsdMG9I1+SJEkNWb4kSZIa\nsnxJkiQ1ZPmSJElqyPIlSZLUkOVLkiSpIcuXJElSQ5YvSZKkhixfkiRJDVm+JEmSGrJ8SZIkNWT5\nkiRJasjyJUmS1JDlS5IkqSHLlyRJUkOWL0mSpIYsX5IkSQ1ZviRJkhqyfEmSJDVk+ZIkSWrI8iVJ\nktSQ5UuSJKkhy5ckSVJDli9JkqSGLF+SJEkNWb4kSZIasnxJkiQ1ZPmSJElqyPIlSZLUkOVLkiSp\nIcuXJElSQ5YvSZKkhixfkiRJDVm+JEmSGrJ8SZIkNWT5kiRJasjyJUmS1JDlS5IkqSHLlyRJUkOW\nL0mSpIYsX5IkSQ1ZviRJkhqyfEmSJDVk+ZIkSWrI8iVJktSQ5UuSJKkhy5ckSVJDli9JkqSGLF+S\nJEkNWb4kSZIasnxJkiQ1ZPmSJElqyPIlSZLUkOVLkiSpIcuXJElSQ5YvSZKkhiZevpLsl+S8JD9I\nctgCx7dIckKSc5Kcm+R5k84kSZLUl1TV5J482Qj4AbAPcCnwDeDpVXXe2DmvBraoqlcn2Qb4PrBd\nVf123nPVJLNKkiStL0moqix0bNIjX3sB51fVhVW1AjgeOGDeOQVs3j3eHPjF/OIlSZK0VEy6fO0A\nXDS2fXG3b9yRwB8kuRT4FvDSCWeSJEnqzTQsuH8scHZV3Q14IPDOJHfuOZMkSdJEbDLh578E2Gls\ne8du37iDgDcBVNV/J7kA2BU4c/6Tzc7O3vJ4ZmaGmZmZ9ZtWkiRpLSxbtoxly5Yt6txJL7jfmNEC\n+n2Ay4AzgAOravnYOe8EflpVhyfZjlHp2qOqrpz3XC64lyRJg7CqBfcTHfmqqpuSHAqczGiK8+iq\nWp7k+aPDdRRwBPBvSb7dfdor5xcvSZKkpWKiI1/rkyNfkiRpKPq81YQkSZLGWL4kSZIamvTVjkvS\nPbbfnguvuKLvGFNn5+2248eXX953DEmSppprvtZCEqYjyXQJMC1/R5Ik9ck1X5IkSVPC8iVJktSQ\n5UuSJKkhy5ckSVJDli9JkqSGLF+SJEkNWb4kSZIasnxJkiQ1ZPmSJElqyPIlSZLUkOVLkiSpIcuX\nJElSQ5YvSZKkhixfkiRJDVm+JEmSGrJ8SZIkNWT5kiRJasjyJUmS1JDlS5IkqSHLlyRJUkOWL0mS\npIYsX5IkSQ1ZviRJkhqyfEmSJDVk+ZIkSWrI8iVJktSQ5UuSJKkhy5ckSVJDli9JkqSGLF+SJEkN\nWb4kSZIasnxJkiQ1ZPmSJElqyPIlSZLUkOVLkiSpIcuXJElSQ5v0HUBa6rbfcXuuuOSKvmNMle12\n2I7LL7687xiS1ItUVd8ZFiVJTUvWJExHkukSYFr+jqZJEpjtO8WUmfW1ImlpS0JVZaFjTjtKkiQ1\n5LSjJE2J7be/B1dccWHfMabOdtvtzOWX/7jvGNJ6Y/mSpCkxKl5Ox853xRULztxIg+W0oyRJUkOW\nL0mSpIYsX5IkSQ1ZviRJkhqyfEmSJDVk+ZIkSWrI8iVJktSQ5UuSJKkhy5ckSVJDli9JkqSGLF+S\nJEkNWb4kSZIasnxJkiQ1NPHylWS/JOcl+UGSw1ZyzkySs5N8J8mpk84kSZLUl00m+eRJNgKOBPYB\nLgW+keSTVXXe2DlbAu8EHlNVlyTZZpKZJEmS+jTpka+9gPOr6sKqWgEcDxww75xnAP9eVZcAVNXP\nJ5xJkiSpN5MuXzsAF41tX9ztG3dfYOskpyb5RpJnTziTJElSbyY67bhImwB7Ao8G7gR8LcnXquqH\n/caSJEla/yZdvi4Bdhrb3rHbN+5i4OdVdQNwQ5IvAXsAv1O+Zmdnb3k8MzPDzMzMeo4rSZK05pYt\nW8ayZcsWdW6qamJBkmwMfJ/RgvvLgDOAA6tq+dg5uwLvAPYD7gB8HXhaVX1v3nPVJLOuiSRMR5Lp\nEmBa/o6mSRKY7TvFlJn1tbKQJOBvlwXE14sGJwlVlYWOTXTkq6puSnIocDKj9WVHV9XyJM8fHa6j\nquq8JCcB3wZuAo6aX7wkSZKWiomOfK1PjnxNP0e+FubI1wJmfa0sxJGvlXHkS8OzqpEv73AvSZLU\nkOVLkiSpIcuXJElSQ5YvSZKkhqbhJquSJGkN7bTT9lx00RV9x5gqd7/7dvzkJ5f3HWO1LF+SJA3Q\nRRddwamn9p1iuuy99zDKqNOOkiRJDVm+JEmSGrJ8SZIkNWT5kiRJasjyJUmS1JDlS5IkqSHLlyRJ\nUkOWL0mSpIYsX5IkSQ1ZviRJkhqyfEmSJDW0yvd2TPLyVR2vqreu3ziSJElL2+reWHvzJikkSZI2\nEKssX1V1eKsgkiRJG4LVTTu+fVXHq+ol6zeOJEnS0ra6acdvNkkhSZK0gVjdtOOxrYJIkiRtCFY3\n8gVAkm2Bw4A/ADad219Vj55QLkmSpCVpsff5+hCwHLgncDjwY+AbE8okSZK0ZC22fN21qo4GVlTV\naVV1MOColyRJ0hpa1LQjsKL7eFmSxwOXAltPJpIkSdLStdjydUSSLYFXAO8AtgBeNrFUkiRJS9Si\nyldVfbp7eDWw9+TiSJIkLW2LWvOV5NgkW41t3yXJ+ycXS5IkaWla7IL73avql3MbVXUV8MDJRJIk\nSVq6Flu+Nkpyl7mNJFuz+PVikiRJ6iy2QP0z8LUkH++2nwK8cTKRJEmSlq7FLrg/LsmZ3HpvrydV\n1fcmF0uSJGlpWuy0I4zu63VdVR0J/CzJPSeUSZIkacla7NWOr2f03o6v7nbdDvjgpEJJkiQtVYsd\n+Xoi8ATgOoCquhTYfFKhJEmSlqrFlq8bq6qAAkhyp8lFkiRJWroWW74+luQ9wFZJDgFOAd43uViS\nJElL02KvdnxLkn2Ba4D7Aa+rqs9PNJkkSdIStOgbpXZl6/MASTZK8syq+tDEkkmSJC1Bq5x2TLJF\nklcnOTLJYzJyKPAj4KltIkqSJC0dqxv5+gBwFfA14C+B1wAB/ryqzplwNkmSpCVndeXrXlW1G0CS\n9wGXATtV1Q0TTyZJkrQEre5qxxVzD6rqJuBii5ckSdLaW93I1x5JrukeB9is2w5QVbXFRNNJkiQt\nMassX1W1casgkiRJG4I1eWNtSZIkrSPLlyRJUkOWL0mSpIYsX5IkSQ1ZviRJkhqyfEmSJDVk+ZIk\nSWrI8iVJktSQ5UuSJKkhy5ckSVJDli9JkqSGJl6+kuyX5LwkP0hy2CrOe0iSFUmeNOlMkiRJfZlo\n+UqyEXAk8FjgAcCBSXZdyXlvBk6aZB5JkqS+TXrkay/g/Kq6sKpWAMcDByxw3v8C/h/w0wnnkSRJ\n6tWky9cOwEVj2xd3+26R5G7An1fVvwKZcB5JkqReTcOC+7cB42vBLGCSJGnJ2mTCz38JsNPY9o7d\nvnEPBo5PEmAbYP8kK6rqhPlPNjs7e8vjmZkZZmZm1ndeSZKkNbZs2TKWLVu2qHNTVRMLkmRj4PvA\nPsBlwBnAgVW1fCXnHwN8qqo+scCxmmTWNZGE6UgyXQJMy9/RNEkCs32nmDKzvlYWMvo3qD+X3xVf\nLwtIwqmn9p1iuuy99/T8bklCVS04mzfRka+quinJocDJjKY4j66q5UmePzpcR83/lEnmkSRJ6tuk\npx2pqhOB+83b956VnHvwpPNIkiT1aRoW3EuSJG0wLF+SJEkNWb4kSZIasnxJkiQ1ZPmSJElqyPIl\nSZLUkOVLkiSpIcuXJElSQ5YvSZKkhixfkiRJDVm+JEmSGrJ8SZIkNWT5kiRJasjyJUmS1JDlS5Ik\nqSHLlyRJUkOWL0mSpIYsX5IkSQ1ZviRJkhqyfEmSJDVk+ZIkSWrI8iVJktSQ5UuSJKkhy5ckSVJD\nli9JkqSGLF+SJEkNWb4kSZIasnxJkiQ1ZPmSJElqyPIlSZLUkOVLkiSpIcuXJElSQ5YvSZKkhixf\nkiRJDVm+JEmSGrJ8SZIkNWT5kiRJasjyJUmS1JDlS5IkqSHLlyRJUkOWL0mSpIYsX5IkSQ1ZviRJ\nkhqyfEmSJDVk+ZIkSWrI8iVJktSQ5UuSJKkhy5ckSVJDli9JkqSGLF+SJEkNWb4kSZIasnxJkiQ1\nZPmSJElqyPIlSZLUkOVLkiSpIcuXJElSQ5YvSZKkhixfkiRJDU28fCXZL8l5SX6Q5LAFjj8jybe6\nP19JstukM0mSJPVlouUryUbAkcBjgQcABybZdd5pPwIeWVV7AEcA751kJkmSpD5NeuRrL+D8qrqw\nqlYAxwMHjJ9QVadX1dXd5unADhPOJEmS1JtJl68dgIvGti9m1eXqL4HPTTSRJElSjzbpO8CcJHsD\nBwGP6DuLJEnSpEy6fF0C7DS2vWO37zaS7A4cBexXVVet7MlmZ2dveTwzM8PMzMz6yilJkrTWli1b\nxrJlyxZ1bqpqYkGSbAx8H9gHuAw4AziwqpaPnbMT8AXg2VV1+iqeqyaZdU0kYTqSTJcA0/J3NE2S\nwGzfKabMrK+VhSQBf7ssIL5eFpCEU0/tO8V02Xvv6fndkoSqykLHJjryVVU3JTkUOJnR+rKjq2p5\nkuePDtdRwP8BtgbeldFvnhVVtdckc0mSJPVl4mu+qupE4H7z9r1n7PEhwCGTziFJkjQNvMO9JElS\nQ5YvSZKkhixfkiRJDVm+JEmSGrJ8SZIkNWT5kiRJasjyJUmS1JDlS5IkqSHLlyRJUkOWL0mSpIYs\nX5IkSQ1ZviRJkhqyfEmSJDVk+ZIkSWrI8iVJktSQ5UuSJKkhy5ckSVJDli9JkqSGLF+SJEkNWb4k\nSZIasnxJkiQ1ZPmSJElqyPIlSZLUkOVLkiSpIcuXJElSQ5YvSZKkhixfkiRJDVm+JEmSGrJ8SZIk\nNWT5kiRJasjyJUmS1JDlS5IkqSHLlyRJUkOWL0mSpIYsX5IkSQ1ZviRJkhqyfEmSJDVk+ZIkSWrI\n8iVJktSQ5UuSJKkhy5ckSVJDli9JkqSGLF+SJEkNWb4kSZIasnxJkiQ1ZPmSJElqyPIlSZLUkOVL\nkiSpIcuXJElSQ5YvSZKkhixfkiRJDVm+JEmSGrJ8SZIkNWT5kiRJasjyJUmS1JDlS5IkqSHLlyRJ\nUkOWL0mSpIYmXr6S7JfkvCQ/SHLYSs55e5Lzk5yT5I8mnUmSJKkvEy1fSTYCjgQeCzwAODDJrvPO\n2R/YparuAzwfePckMy01y/oOoOG4oO8AGpZlfQfQQJxzTt8JhmfSI197AedX1YVVtQI4Hjhg3jkH\nAMcBVNXXgS2TbDfhXEvGsr4DaDh+3HcADcuyvgNoICxfa27S5WsH4KKx7Yu7fas655IFzpEkSVoS\nXHAvSZLUUKpqck+ePAyYrar9uu1XAVVV/zB2zruBU6vqo932ecCjquqKec81uaCSJEnrWVVlof2b\nTPjrfgO4d5KdgcuApwMHzjvnBODFwEe7svbL+cULVv4NSJIkDclEy1dV3ZTkUOBkRlOcR1fV8iTP\nHx2uo6rqs0kel+SHwHXAQZPMJEmS1KeJTjtKkiTptlxwL0mS1JDla2CSbJbk1d2FCiS5d3ejWkmS\nNACWr+F5PxDgEd32pcDf9xdHQ5FkmyReuKKVSvLiJFuNbd8lyYv6zCQtRa75GpgkZ1bVg5OcXVUP\n7PadU1W+J6Zu0V05/GbgSuANwAeAbRj9g+s5VXVij/E0pRb6XTL+u0aak2Rb4BDgHoxdvFdVB/eV\naUgmfasJrX83JtkUKIAk9wRu7DeSptCRwGuALYEvAvtX1ende6t+BLB8aSEbJ0l1/ypPsjFw+54z\naTp9EvgycApwU89ZBsfyNTx/x+h/nDsmORZ4FPAX/UbSFNqkqk4GSPJ3VXU6QFWd58yjVuFERvdc\nfE+3/Xws6lrYHavqsL5DDJXla2Cq6sQk3wT+mNHar7+pqp/2HEvT5+axx9fPO+ZaA63MYYwK1wu7\n7c8D7+svjqbYp5M8rqo+23eQIXLN1wAleTqwS1W9Mcndgd+rqm/2nUvTI8lNjG5aHGAz4Ndzh4BN\nq+p2fWWTNHxJrgXuxGjZy4pud1XVFv2lGg7L18AkORK4HfDIqrp/kq2Bk6rqIT1HkzRQST5WVU9N\nci4LjIxW1e49xJKWLKcdh+ePq2rPJGcDVNWVSVwQK2ldvLT7+Ke9ptCgJHkC8Mhuc1lVfbrPPEPi\nfb6GZ0WSjbj1ase7ctv1PRJJrk1yTffx2rHtXyf5bd/5NF2q6rLu4Yuq6sLxP4D3+dLvSPJmRqX9\ne92flyZ5U7+phsNpx4FJ8hzgicCDGd1w9anA4VV1fK/BNNWS3Bl4MaPF1P9RVa/oOZKmUJKzqmrP\nefu+7bSj5kvybeCPqurmbntj4GxfK4vjtOPAVNVx3dWOf8Jo8fRTquo7PcfSlOruVv7XwHOADwMP\nqapf9JtK0ybJCxmNcO3S/U91zubAf/WTSgOwFaMbOcPonoJaJMvXgHT/svh2VT0A+G7feTS9kmwD\nvAJ4GqMR0gdW1dX9ptIU+zDwOeBNwKvG9l9bVVcu/CnawL0JODvJqYwGAh7JbV87WgWnHQcmyaeA\nF1TVJX1n0fRKch3wM+AY4Nr5x6vqrc1Daeol2QW4uKp+k2QG2B04rqp+2W8yTZPuPWJ3BH4LzF1p\nf0ZVXd5fqmGxfA1M96+MBwFfY3QfJwCq6km9hdLUSTLLKm6mWlWHt0ujoUhyDqP1pPcAPsvoLWQe\nUFWP6zOXpk+Sc6tqt75zDJXTjsNzRN8BNP2qarbvDBqkm6vqt0meBLyjqt4xd1sbaZ6zkjykqr7R\nd5AhsnwNRJKTq+oxVfWFvrNo+iV5ZVX9Y5J3sPBNM1/SQyxNvxVJDmR0gcafdft8NwQt5KHAM5Nc\nyK3vplFe7bg4lq/h2LbvABqU5d3HM3tNoaE5CHgB8MaquiDJPYEP9JxJ0+mxfQcYMtd8DUSSHwH/\ne2XHq+oTDeNIkjZgST5QVc9e3T4tzJGv4diS0Vt/ZIFjBVi+dIvuqthVLbh/QsM4mnK+t6PWwgPG\nN7pbIT2opyyDY/kajgur6uC+Q2gw3tJ3AA2K7+2oRUnyauA1wGZJruHWAYEbgaN6CzYwTjsORJKz\nq+qBfeeQJCnJm6rq1X3nGCrL10Ak+UPfRkiLtbLpozlOI2khSa7ld183VzO6cOMVVfWj9qk0jbob\nrT4ReASj18yXq+o/+001HJYvaQlKsvOqjlfVha2yaDiSvAG4mNHbDQV4OrALcBbwwqqa6S+dpkmS\ndwH3Bj7S7Xoa8N9V9eL+Ug2H5UuSBECSb1XVHvP2nVNVf7TQMW24kpwH3L+6EpFkI+C7VXX/fpMN\nw0Z9B5A0OUmuTXJN9+eGJDd1i2Slhfw6yVOTbNT9eSpwQ3fMf6lr3A+Bnca2797t0yJ4tePAJHk4\nMAvszOjvb+6uwvfqM5emU1VtPve4W6NxAPCw/hJpyj0T+L/Au7rtrwHPSrIZcGhvqTSNNgeWJzmD\nUTHfCzgzyQng7WxWx2nHgemGel8GfBO4aW5/Vf2it1AaFK+clbSukjxqVcer6rRWWYbIka/hubqq\nPtd3CA1D9wbJczYCHsyt00jSbSTZEXgH8PBu15eBl1bVxf2l0jSqqtO6C3vuU1WndKOjm1TVtX1n\nGwLL1/CcmuSfGN3R/jdzO6vqrP4iaYr92djj3wI/ZjT1KC3kGEZXOj6l235Wt2/f3hJpKiU5BPgr\nYGtGV8TuCLwb2KfPXEPhtOPAJDl1gd1VVY9uHkbSkjJ3ZePq9klJzmG0zuvrc8sYkpxbVbv1m2wY\nHPkamKrau+8MGg6nkbSGfpHkWdx676YDAdeTaiG/qaobR9fxQJJN8IrYRfNWEwOTZMskb01yZvfn\nn5Ns2XcuTa1jgBOAu3V/PtXtkxZyMPBU4HLgMuDJwPP6DKSpdVqSufd43Bf4OKPfL1oEpx0HJsm/\nA98Bju12PRvYo6qetPLP0obKaSStqyR/XVVv6zuHpkt3U9W/AB7D6JZHJwHvK0vFoli+Bsb/mWpN\nJPkCo5Gu8Wmkg6rKRbFalCQ/qaqdVn+mNiRJ7gTcUFU3ddsbA3eoql/3m2wYnHYcnuuTPGJuo7vp\n6vU95tF0W2ga6aBeE2lo0ncATaUvAJuNbW8GnNJTlsFxwf3wvBA4tlvnFeBKXJOhlejeQNs7TWtd\nOD2ihWxaVb+a26iqXyW5Y5+BhsTyNTBVdQ6wR5Itum3fp08rlWRb4BDgHoz9915VB/eVSdMnybUs\nXLLCbUc3pDnXJdlz7h6TSR6EszCLZvkaiCTPqqoPJnn5vP0AVNVbewmmafdJRreXOIWxt6OSxo2/\nB6i0SC8FPp7kUkYlfXvgaf1GGg7L13DcqfvoL0mtiTtW1WF9h5C0dHRXOt4e2BW4X7f7+1W1or9U\nw+LVjtISluQI4KtV9dm+s0haOpKcPXdne605r3YcmCT/mGSLJLdL8oUkP+vuSC0t5KXAp5Ncn+Sa\nJNcmcZ2gpHX1hST/M3NrX7RGHPkamLl7eiV5IvCnwMuBL1XVHj1H05Tpfinevap+0ncWSUtLd5HG\nnRitJb2e0bqvqqoteg02EK75Gp65v7PHAx+vqqv9h4cWUlWV5DOAb3Qrab3yIo1147Tj8Hw6yXnA\ngxgN+24L3NBzJk2vs5I8pO8QkpaWjDwryf/ptu+eZK++cw2F044DlGRr4Oqquqm7qd0WVXV537k0\nfbqifm/gQuA6bp0a2L3XYJIGLcm/AjcDj66q+ye5C3ByVfmPvUVw2nEgkjy6qr6Y5Elj+8ZP+UT7\nVBqAx/bd/WUXAAAFHklEQVQdQNKS9NCq2jPJ2QBVdVWS2/cdaigsX8PxKOCLwJ8tcKywfGkBVXVh\n916g96mqY7pp6jv3nUvS4K3o3ky74JZ307i530jD4bSjtIQleT3wYOB+VXXfJHdjdKHGw3uOJmnA\nkjyT0R3t9wSOBZ4MvLaqPt5rsIFwwf3AJPn7JFuNbd+lu5GmtJAnMnpj7esAqupSfJcESeuoqj4E\nvBJ4E3AZ8OcWr8Vz2nF49q+q18xtdPPsjwNe22MmTa8bu1tOzE0N3Gl1nyBJK5NkU+AFjC7kORd4\nT1X9tt9Uw+PI1/BsnOQOcxtJNgPusIrztWH7WJL3AFslOYTRG2y/t+dMkobrWEZLGc4F9gfe0m+c\nYXLN18AkOYzRovtjul0HASdU1T/2l0rTLMm+wGMY3WbipKr6fM+RJA1UknOrarfu8SbAGVW1Z8+x\nBsfyNUBJ9gP+pNv8fFWd1GceDUOSbYBflP/RS1pLSc4aL1vzt7U4lq8BSrIzo1sHnNLdZHXjqrq2\n71yaHkkeBrwZuBJ4A/ABYBtGSw2eU1Un9hhP0kAluYnuAh5Go+mbAb/G93ZcI5avgenW7fwVsHVV\n7ZLkPsC7q2qfnqNpiiQ5E3gNsCVwFKMLNU5Psivwkap6YK8BJWkD5oL74Xkx8HDgGoCqOh/4vV4T\naRptUlUnd5d+X15VpwNU1Xk955KkDZ7la3h+U1U3zm10Cx4dvtR843eavn7eMV8vktQj7/M1PKcl\neQ2wWXcV24uAT/WcSdNnjyTX0K3J6B7TbW/aXyxJkmu+BibJRsBfMHbrAOB9XsEmSdIwWL4GqHsD\nU6rqZ31nkSRJa8Y1XwORkdkkPwe+D3w/yc+SvK7vbJIkafEsX8PxMkZXOT6kqrauqq2BhwIPT/Ky\nfqNJkqTFctpxIJKcDexbVT+ft39b4GTv2yRJ0jA48jUct5tfvOCWdV+36yGPJElaC5av4bhxLY9J\nkqQp4rTjQMx7P63bHAI2rSpHvyRJGgDLlyRJUkNOO0qSJDVk+ZIkSWrI8iVJktSQ5UuSJKkhy5ek\nQUtyc5LjxrY37t5664Q1fJ4Lkmy9rudI0upYviQN3XXAHya5Q7e9L3DRWjzPYi799vJwSevM8iVp\nKfgs8Pju8YHAR+YOJLlLkv9I8q0kX02yW7d/6yQnJTk3yXsZ3TNv7nOemeTrSc5K8q9J5o6lO37H\nJJ9OcnaSbyd5SpPvUtKSYPmSNHQFHA8c2I1+7Q58fez44cBZVbUH8LfA3BTl64EvV9VuwH8AOwEk\n2RV4GvDHVbUncDPwzHlfcz/gkqp6YFXtDpw4ke9M0pK0Sd8BJGldVdV3ktyD0ajXZxgbxQIeATyp\nO+/UbsRrc+CRwBO7/Z9NclV3/j7AnsA3uhGvTYHL533Jc4G3JHkT8Jmq+spEvjFJS5LlS9JScQLw\nT8AMsM1qzl1o7db41OKxVfW3K/3kqvOT7Ak8DjgiySlVdcSaR5a0IXLaUdLQzZWm9wOHV9V35x3/\nMvAsgCQzwM+r6lfAl+imE5PsD2zVnf8F4MlJtu2O3SXJTrf5gsnvA9dX1YcZFb491/c3JWnpcuRL\n0tAVQFVdAhy5wPFZ4P1JvsXoysjndvsPBz6S5OnAV4GfdM+zPMlrgZOTbATcCLy4Oz43YrYb8E9J\nbu6Ov3AC35ekJco31pYkSWrIaUdJkqSGLF+SJEkNWb4kSZIasnxJkiQ1ZPmSJElqyPIlSZLUkOVL\nkiSpIcuXJElSQ/8fxdZgGY5vrMEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd2d4981d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pl = negcom.plot(kind='bar',color=['r','g','b','y'],title='Most Freq Features',figsize=(10,6),legend=False)\n",
    "pl.set(xlabel='Models',ylabel='Recall')\n",
    "lim = plt.ylim([0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 9. Finally we can try by running it on our entire featureset. One can guess it will take a lot of time with Decision Tree Classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We can make use of group of words to get better results. Sometimes sequence of words might have different effect on the prediction. Sequences like \"not good\" or \"not bad\" affect the prediction in way different than when used individually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a. Unigram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### using countvectorizer to generate count vectors for the training data.\n",
    "vec_all = CountVectorizer()\n",
    "ctr_features_all = vec_all.fit_transform(Data_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf_vec_all = TfidfTransformer()\n",
    "tr_features_all = tf_vec_all.fit_transform(ctr_features_all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cte_features_all = vec_all.transform(Data_test)\n",
    "te_features_all = tf_vec_all.transform(cte_features_all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "models = {'BernoulliNB':BernoulliNB()\n",
    "          ,'Logistic' : linear_model.LogisticRegression(C=1e5),'Perceptron': linear_model.Perceptron(n_iter=1000)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   TF-IDF Accuracy        Model\n",
      "0         0.873644  BernoulliNB\n",
      "1         0.893339     Logistic\n",
      "2         0.861090   Perceptron\n"
     ]
    }
   ],
   "source": [
    "results_all_uni = pd.DataFrame()\n",
    "\n",
    "#stratified = StratifiedKFold(labels, n_folds=5,random_state=20160121, shuffle=True)\n",
    "\n",
    "foldnum = 0\n",
    "tfprediction = {}\n",
    "cprediction = {}\n",
    "for name,model in models.items():\n",
    "        \n",
    "        model.fit(tr_features_all, Train_labels)\n",
    "        tfprediction[name] = model.predict(te_features_all)\n",
    "        tfaccuracy = metrics.accuracy_score(tfprediction[name],Test_labels)\n",
    "        \n",
    "        #model.fit(ctr_features_all,Train_labels)\n",
    "        #cprediction[name] = model.predict(cte_features_all)\n",
    "        #caccuracy = metrics.accuracy_score(cprediction[name],Test_labels)\n",
    "        \n",
    "        results_all_uni.loc[foldnum,'TF-IDF Accuracy']=tfaccuracy\n",
    "        #results_all_uni.loc[foldnum,'Count Accuracy']=caccuracy\n",
    "        results_all_uni.loc[foldnum,'Model']=name\n",
    "        foldnum = foldnum+1\n",
    "print (results_all_uni)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for  BernoulliNB\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.73      0.68      0.70     31169\n",
      "        pos       0.91      0.93      0.92    110945\n",
      "\n",
      "avg / total       0.87      0.87      0.87    142114\n",
      "\n",
      "Classification report for  Logistic\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.80      0.69      0.74     31169\n",
      "        pos       0.92      0.95      0.93    110945\n",
      "\n",
      "avg / total       0.89      0.89      0.89    142114\n",
      "\n",
      "Classification report for  Perceptron\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.70      0.64      0.67     31169\n",
      "        pos       0.90      0.92      0.91    110945\n",
      "\n",
      "avg / total       0.86      0.86      0.86    142114\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for name,model in models.items():\n",
    "    print (\"Classification report for \",name)\n",
    "    print(metrics.classification_report(Test_labels, tfprediction[name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1cd0a0b92b0>"
      ]
     },
     "execution_count": 201,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAG6CAYAAADH1Xg8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu85XVd7/HXe2a8kALq0TAZB1JT1LymZMeynTwUsJOY\neRuvQRl1JK08J9LKNsdrJytNtBxD89KB9Jh5KyXN7eUYgoHkZQZQcRwGGBVvKCDj8Dl/rN+GxWbP\n7PVl1uzfWmtez8djP/xdvuu3Pnvxc+33fL/f3++XqkKSJEmjW9N3AZIkSdPGACVJktTIACVJktTI\nACVJktTIACVJktTIACVJktTIACWpWZLDklyXZMXvkCTPSvLxVarr1knem+TbSf5hNd5T0v7JACXN\nuCRfSXJNkjss2X5eF4I23MxDt9xEbtm2Q0Hsu93Pl5OcfDPrAXgCcCfg9lX15L04jiTtkQFKmn0F\nXAxsXNyQ5CeBA2gLQftKAQdX1UHAU4EXJXl060G63rDDgAvrZtwhOMna1tdI2n8ZoKT9w1uBZw2t\nPwt483CDJAcleUuSryW5OMkfDu1bk+SVSb6e5IvALy7z2r9NcmmSbUlenCQN9QWgqs4CPg/8ZHfc\nI5KcmeSKJJuTPHHoPd+U5HVJ3p/kSuCjwIuAp3S9Wcdn4I+6XrjLk/xdkoO61y/2fp2QZCvw4aFt\nv5rkq937npjkIUnOT/LNJK8ZquFuST6c5Bvd5/a2xeN3+y9O8vzutd9KcnqSWw7tP67rCfxOkosW\ng+MYPk9J+5gBSto/nAUcmOReXU/Nk4G30QWXzqnAgcDhwBzwzCTHd/t+A3gM8ADgIQyGyoa9GbgW\nuBvwIOBRwK831BeAJA8H7gOcm+RHgDO7Ou8IPAV4XZIjhl63EXhxVR0IHAW8DDijqg6qqjcBxwPP\nBH6+q+3A7vcc9gjgCODooW1HAvdg8Dm9Cngh8EgGwe5JSX5uqO6XAXcG7g2sB+aXHP+JwKOBH2fw\n+f1q97seyeBze35VHdzV8ZXuNXv7eUraxwxQ0v5jsRfqUcBm4NLFHUOh6g+q6qqq2gr8OfCMrskT\ngVdV1aVV9W3g5UOvPQQ4Fvjdqrqmqr7BIHRcP2S4ggBfT3IFsAk4uaoWgP8GXFxVb6mB84F3drUs\nenfXa0VVXbvMsZ8K/EVVba2qq4AXMOihWvzuK+BPqurqqvrB0Lb/VVXXVtWHgO8Dp1fVFVV1KfBx\nBqGGqvpSVX24qn5YVVcAf8kgrA17dVXt6D639wIP7LafAJxWVf/WHeuyqrowyY/u5ecpaRWs67sA\nSavmbcDHGPSEvGXJvjsy+D746tC2rcCh3fJdgG1L9i3aANwCuKwbZUr3M3ysPSngvywzb+kw4GFJ\nvtmtB1i7pPZt7NldltS6lcHvecjQtkuWed3XhpavBnYsWb8tQBd2Xg38XLdtLfBNbmz4tVcBP9Yt\n3xV4/zLvfRh793lKWgUGKGk/UVVfTXIxg96NE5bs/gawk8Ef7y3dtsOA7d3yZQz+4DO0b9E24BqW\nD0GjCjed0L4NWKiqo5dpv2il97uUG9d6GIPfcwc3/D57M5H+ZcB1wH2r6jtJjgNes8JrFm0D7r6b\n7Xv7eUraxxzCk/YvJwCPrKqrhzdW1XXA24GXJrltksOA32Uw7Ee377lJDk1ye+DkoddezmCu0l8m\nObCbuH23JI8YsabdTY5+H3DPJE9Psi7JLbrJ3Pca+beF04HfTXJ4ktsCL2UwR+q6Pbx3y2TtA4Hv\nAVcmORT4nw2vPQ04PskvdJ/ZXZLcawyfp6RVYICSZt/1vRhVdXFVnbvcPuC5DIaYvsxgqO9t3URs\ngDcAHwTOBz7NYC7SsGcCtwS+wGAI6x0MJlY31XejjVXfYzD5+ikMepIuBV4B3GrE4wK8kUEI/Bjw\nJQa/33NXeO+l2/a0fgrwU8Di/Kaln8tue5Cq6hwGk9xfBXwHWGAwHAp793lKWgUZpYc4yTEM/k++\nhsGkxz9dsv92DL6o7s5gfsAJVfWF8ZcrSZLUv1Eew7CGwWW/RwP3BTYuuYwYBpf4nldVD2Bwlc9f\njbtQSZKkSTHKEN6RwEXdZcA7gTOA45a0uQ+weCnuBcDhSe401kolSZImxCgB6lBufKnwJdxwafOi\n84HHw/U3h9vA4IZykiRJM2dck8hfAdw+ybnAc4DzgF1jOrYkSdJEGeU+UNu54coQGPQsbR9uUFVX\nMnRfme5eM19eeqAk3tNEkiRNjapa9tYmowSoc4B7dPeFuYzBJcU3eqRAkoOBq6pqZ5JnAx/tLkFe\nrpCmwrV35ufnmZ+f77sMaZ/yPNf+wPN89e3pGd4rBqiq2pXkJAY3dlu8jcHmJCcOdtcmBg/RfHOS\n6xg8Sf3XxlK5JEnSBBrpUS5V9QHgXku2vX5o+ayl+yVNnzuvvzM7tu9YueEEOuWUU/ou4WY55NBD\nuPySy/suQ1Ijn4U34+bm5vouQVNkx/YdMN93FTfDxQwekTyFdsxPZ2DV6vP7fLKMdCfysb1Z4rMx\npQmWZDoD1DSbd26oNKmS7HYSuc/CkyRJamSAkiRJamSAkiRJamSAkiRJauRVeCM6/M53ZusOr5ZZ\nTYcdcghfudzLuyVJk8cANaKtO3bgdTKrKwZWSdKEcghPkiSpkQFKkiSpkQFKkiSpkXOgJEn7lTvf\n+XB27Njadxn7lUMOOYzLL/9K32WMlQFKkrRfGYQnLwtaTTt2LPs0lKnmEJ4kSVIjA5QkSVIjA5Qk\nSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIj\nA5QkSVIjA5QkSVIjA5QkSVKjkQJUkmOSbElyYZKTl9l/UJL3JPlMks8m+dWxVypJkjQhVgxQSdYA\npwJHA/cFNiY5Ykmz5wCfr6oHAr8A/HmSdeMuVpIkaRKM0gN1JHBRVW2tqp3AGcBxS9oUcGC3fCBw\nRVX9cHxlSpIkTY5RAtShwLah9Uu6bcNOBe6T5FLgfOB54ylPkiRp8oxrEvnRwHlVdRfgQcBrk9x2\nTMeWJEmaKKPMU9oObBhaX99tG3Y88HKAqvpSkouBI4BPLz3Y/Pz89ctzc3PMzc01FSxJkrQvLCws\nsLCwMFLbVNWeGyRrgQuAo4DLgLOBjVW1eajNa4GvVdUpSQ5hEJweUFXfXHKsWun9JlUSprPy6RVg\nWs+XaZUE5vuuYj8z73m+2pKA3+irLFN5niehqrLcvhV7oKpqV5KTgDMZDPmdVlWbk5w42F2bgJcA\nf5fkP7uX/f7S8CRJkjQrVuyBGuub2QOlBvZArT57oHow73m+2uyB6sPs9UB5J3JJkqRGBihJkqRG\nBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJ\nkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRG\nBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGBihJkqRGIwWoJMck2ZLk\nwiQnL7P/fyQ5L8m5ST6b5IdJbjf+ciVJkvq3YoBKsgY4FTgauC+wMckRw22q6pVV9aCqejDwAmCh\nqr69LwqWJEnq2yg9UEcCF1XV1qraCZwBHLeH9huB08dRnCRJ0iQaJUAdCmwbWr+k23YTSQ4AjgHe\nufelSZIkTaZxTyL/JeATDt9JkqRZtm6ENtuBDUPr67tty3kKKwzfzc/PX788NzfH3NzcCCVIkiTt\nWwsLCywsLIzUNlW15wbJWuAC4CjgMuBsYGNVbV7S7mDgy8D6qrp6N8eqld5vUiVhOiufXgGm9XyZ\nVklgvu8q9jPznuerLQn4jb7KMpXneRKqKsvtW7EHqqp2JTkJOJPBkN9pVbU5yYmD3bWpa/o44IO7\nC0+SJEmzYsUeqLG+mT1QamAP1OqzB6oH857nq80eqD7MXg+UdyKXJElqZICSJElqZICSJElqZICS\nJElqZICSJElqZICSJElqZICSJElqZICSJElqZICSJElqZICSJElqZICSJElqZICSJElqZICSJElq\nZICSJElqZICSJElqZICSJElqZICSJElqZICSJElqZICSJElqZICSJElqZICSJElqZICSJElqZICS\nJElqZICSJElqZICSJElqZICSJElqZICSJElqZICSJElqNFKASnJMki1JLkxy8m7azCU5L8nnknxk\nvGVKkiRNjnUrNUiyBjgVOAq4FDgnyburastQm4OB1wKPrqrtSe64rwqWJEnq2yg9UEcCF1XV1qra\nCZwBHLekzVOBd1bVdoCq+sZ4y5QkSZocowSoQ4FtQ+uXdNuG3RO4Q5KPJDknyTPGVaAkSdKkWXEI\nr+E4DwYeCdwG+Pck/15VXxzT8SVJkibGKAFqO7BhaH19t23YJcA3quoa4JokHwMeANwkQM3Pz1+/\nPDc3x9zcXFvFkiRJ+8DCwgILCwsjtU1V7blBsha4gMEk8suAs4GNVbV5qM0RwGuAY4BbAZ8CnlxV\nX1hyrFrp/SZVEqaz8ukVYFrPl2mVBOb7rmI/M+95vtqSgN/oqyxTeZ4noaqy3L4Ve6CqaleSk4Az\nGcyZOq2qNic5cbC7NlXVliQfBP4T2AVsWhqeJEmSZsWKPVBjfTN7oNTAHqjVZw9UD+Y9z1ebPVB9\nmL0eKO9ELkmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS\n1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgA\nJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS1MgAJUmS\n1GikAJXkmCRbklyY5ORl9v98km8nObf7+aPxlypJkjQZ1q3UIMka4FTgKOBS4Jwk766qLUuafqyq\nHrsPapQkSZooo/RAHQlcVFVbq2oncAZw3DLtMtbKJEmSJtQoAepQYNvQ+iXdtqV+Jslnkrw/yX3G\nUp0kSdIEWnEIb0T/AWyoqquSHAv8E3DPMR1bkiRpoowSoLYDG4bW13fbrldV3xta/pckr0tyh6r6\n5tKDzc/PX788NzfH3NxcY8mSJEnjt7CwwMLCwkhtU1V7bpCsBS5gMIn8MuBsYGNVbR5qc0hV7eiW\njwTeXlWHL3OsWun9JlUSprPy6RVgWs+XaZUE5vuuYj8z73m+2pKA3+irLFN5niehqpad471iD1RV\n7UpyEnAmgzlTp1XV5iQnDnbXJuAJSX4L2AlcDTx5fOVLkiRNlhV7oMb6ZvZAqYE9UKvPHqgezHue\nrzZ7oPowez1Q3olckiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSp\nkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFK\nkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSpkQFKkiSp\nkQFKkiSp0UgBKskxSbYkuTDJyXto99AkO5M8fnwlSpIkTZYVA1SSNcCpwNHAfYGNSY7YTbtXAB8c\nd5GSJEmTZJQeqCOBi6pqa1XtBM4Ajlum3W8D/xf42hjrkyRJmjijBKhDgW1D65d0266X5C7A46rq\nr4GMrzxJkqTJM65J5K8ChudGGaIkSdLMWjdCm+3AhqH19d22YQ8BzkgS4I7AsUl2VtV7lh5sfn7+\n+uW5uTnm5uYaS5YkSRq/hYUFFhYWRmqbqtpzg2QtcAFwFHAZcDawsao276b9m4D3VtU/LrOvVnq/\nSZWE6ax8egWY1vNlWiWB+b6r2M/Me56vtsG/9f3MV1em8jxPQlUtO6q2Yg9UVe1KchJwJoMhv9Oq\nanOSEwe7a9PSl+x1xZIkSRNsxR6osb6ZPVBqYA/U6rMHqgfznuerzR6oPsxeD5R3IpckSWpkgJIk\nSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpk\ngJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIk\nSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWo0UoBKckySLUkuTHLy\nMvsfm+T8JOclOTvJw8dfqiRJ0mRYt1KDJGuAU4GjgEuBc5K8u6q2DDX7UFW9p2t/P+DtwL33Qb2S\nJEm9G6UH6kjgoqraWlU7gTOA44YbVNVVQ6u3Ba4bX4mSJEmTZZQAdSiwbWj9km7bjSR5XJLNwHuB\nE8ZTniRJ0uQZ2yTyqvqnqro38DjgJeM6riRJ0qRZcQ4UsB3YMLS+vtu2rKr6RJK7JblDVX1z6f75\n+fnrl+fm5pibmxu5WEmSpH1lYWGBhYWFkdqmqvbcIFkLXMBgEvllwNnAxqraPNTm7lX1pW75wcC7\nq+quyxyrVnq/SZWE6ax8egWY1vNlWiWB+b6r2M/Me56vtiTgN/oqy1Se50moqiy3b8UeqKraleQk\n4EwGQ36nVdXmJCcOdtcm4FeSPBO4FrgaeNL4ypckSZosK/ZAjfXN7IFSA3ugVp89UD2Y9zxfbfZA\n9WH2eqC8E7kkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5Qk\nSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIj\nA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5QkSVIjA5Qk\nSVIjA5QkSVKjkQJUkmOSbElyYZKTl9n/1CTndz+fSHK/8ZcqSZI0GVYMUEnWAKcCRwP3BTYmOWJJ\nsy8Dj6iqBwAvAd4w7kIlSZImxSg9UEcCF1XV1qraCZwBHDfcoKrOqqrvdKtnAYeOt0xJkqTJMUqA\nOhTYNrR+CXsOSL8O/MveFCVJkjTJ1o3zYEl+ATge+NlxHleSJGmSjBKgtgMbhtbXd9tuJMn9gU3A\nMVX1rd0dbH5+/vrlubk55ubmRixVkiRp31lYWGBhYWGktqmqPTdI1gIXAEcBlwFnAxuravNQmw3A\nh4FnVNVZezhWrfR+kyoJ01n59AowrefLtEoC831XsZ+Z9zxfbUnAb/RVlqk8z5NQVVlu34o9UFW1\nK8lJwJkM5kydVlWbk5w42F2bgD8G7gC8LoMzc2dVHTm+X0GSJGlyrNgDNdY3swdKDeyBWn32QPVg\n3vN8tdkD1YfZ64HyTuSSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCS\nJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmN\nDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCSJEmNDFCS\nJEmNDFCSJEmNRgpQSY5JsiXJhUlOXmb/vZJ8Msk1SX5v/GVKkiRNjnUrNUiyBjgVOAq4FDgnybur\nastQsyuA3wYet0+qlCRJmiCj9EAdCVxUVVuraidwBnDccIOq+kZV/Qfww31QoyRJ0kQZJUAdCmwb\nWr+k2yZJkrRfchK5JElSoxXnQAHbgQ1D6+u7bTfL/Pz89ctzc3PMzc3d3ENJkiSNzcLCAgsLCyO1\nTVXtuUGyFriAwSTyy4CzgY1VtXmZtn8CfK+q/nw3x6qV3m9SJWE6K59eAab1fJlWSWC+7yr2M/Oe\n56stCfiNvsoyled5Eqoqy+1bsQeqqnYlOQk4k8GQ32lVtTnJiYPdtSnJIcCngQOB65I8D7hPVX1v\nfL+GJEnSZBhlCI+q+gBwryXbXj+0vAO463hLkyRJmkxOIpckSWpkgJIkSWpkgJIkSWpkgJIkSWpk\ngJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIk\nSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpk\ngJIkSWpkgJIkSWpkgJIkSWpkgJIkSWpkgJIkSWo0UoBKckySLUkuTHLybtr8VZKLknwmyQPHW6Yk\nSdLkWDFAJVkDnAocDdwX2JjkiCVtjgXuXlU/AZwI/M0+qFU3w0LfBUir4eK+C5BWw0LfBWjIKD1Q\nRwIXVdXWqtoJnAEct6TNccBbAKrqU8DBSQ4Za6W6WRb6LkBaDV/puwBpNSz0XYCGjBKgDgW2Da1f\n0m3bU5vty7SRJEmaCU4ilyRJarRuhDbbgQ1D6+u7bUvb3HWFNgAkaalvokxr5af0XcBemObzZWrN\n913AzfTRvgu4+TzP+zCtn/n0fqPP2nk+SoA6B7hHksOAy4CnABuXtHkP8BzgH5I8DPh2Ve1YeqCq\nmq1PT5Ik7ZdWDFBVtSvJScCZDIb8TquqzUlOHOyuTVX1z0kek+SLwPeB4/dt2ZIkSf1JVfVdgyRJ\n0lRxErkkSVIjA5QkSVKjUSaRa4oluSNwRTlWqxmT5DnA31fVt7v12wMbq+p1/VYmjU+SOwHPBg5n\n6G92VZ3QV00acA7UDOmugHwF8E3gxcBbgTsy6Gl8ZlV9oMfypLFK8pmqeuCSbedV1YP6qkkatySf\nBD4O/Aewa3F7Vb2zt6IE2AM1a04FXggcDPwbcGxVndU9u/B0wAClWbI2SRZ7V5OsBW7Zc03SuP1I\nVZ3cdxG6KedAzZZ1VXVmVb0DuLyqzgKoqi091yXtCx9gcO+5o5Ichf9I0Gx6X5LH9F2EbsohvBmS\n5NyqevDS5eXWpWmXZA1wInBUt+lfgb+tql27f5U0XZJcCdwGuBbY2W2uqjqov6oEBqiZkmQXgxuZ\nBjgAuGpxF3DrqrpFX7VJkjRLDFCSpkqSt1fVk5J8FrjJF1hV3b+HsqR9JsljgUd0qwtV9b4+69GA\nAUrSVEnyY1V1Wfd8zpuoqq2rXZO0ryR5BfBQ4O+7TRuBT1fVC/qrSmCAmindWHlx48eMF4OrLW9Z\nVV51qZmR5E+XXp203DZpmiX5T+CBVXVdt74WOM+e1v55Fd4MqaoDq+qg7n8PBH4MeClwOfDqfquT\nxu5Ry2w7dtWrkPa92w0tH9xbFboReyRmUJLbAb8DPBP4P8BDq+qKfquSxiPJbwH/Hbh796/zRQcC\n/6+fqqR95uXAeUk+wmB04RHAH/RbksAhvJnSPbbl+cCTgTcCr6mq7/RblTReSQ4Gbs/gD8vwH5Ir\nq+qb/VQljV+SAOuBHzKYBwVwdlVd3l9VWmSAmiFJvg98HXgTcOXS/VX1F6telLSPJLk7cElV/SDJ\nHHB/4C2Lz8aTZkGSz1bV/fquQzflHKjZ8mcMwhMMhjOW/kiz5J3AriT3ADYBd2UwZC3NknOTPHTl\nZlpt9kBJmkqLd9dP8vvA1VX1Gh8mrFmTZAtwD2ArN9woubwKr39OIp8hSX6/qv53ktew/A0Gn9tD\nWdK+sjPJRgYXS/xSt8277WvWHN13AVqeAWq2bO7+99O9ViGtjuOB3wReWlUXJ/lx4K091ySN20uq\n6hnDG5K8FXjGbtprlTiEJ0nShFrmwfBrgc9W1X16LEvYAzVTkryXZYbuFlXVY1exHGmf8Fl42h8k\neQHwQuCAJN/lhidMXMvgogn1zB6oGZLk5/e0v6o+ulq1SPuKz8LT/iTJy33u3WQyQEmSNKG6m2n+\nMvCzDHpcP15V/9RvVQID1EzZ3ZDGIoc2NEuGHp497DsMLqJ4flV9efWrksYryesY3Mbg9G7Tk4Ev\nVdVz+qtKYICaKbsb0ljk0IZmSZIXA5cwuHlmgKcAdwfOBX6rqub6q04aj+4+UPeu7o91kjXA56vq\n3v1WJieRzxADkvYzj62qBwytb0rymao6OckLe6tKGq8vAhsY3EgTBnfc/2J/5WiRj3KZQUmuTPLd\n7ueaJLu6qzikWXJVkiclWdP9PAm4pttn17pmxYHA5iQLST4CfAE4KMl7kryn59r2a/ZAzaCquv65\nd90ExOOAh/VXkbRPPA14NfC6bv3fgacnOQA4qbeqpPF6Ud8FaHnOgdpP+IwwSZpO3fzWn6iqD3X/\nQFhXVVf2Xdf+zh6oGZTk8UOra4CHcMPQhjQTkqwHXgM8vNv0ceB5VXVJf1VJ45Xk2cBvAHdgcJHE\neuBvgKP6rEsGqFn1S0PLPwS+wmAYT5olb2JwBd4Tu/Wnd9se1VtF0vg9BzgS+BRAVV2U5Ef7LUlg\ngJpJVXV83zVIq+BOVfWmofW/S/I7vVUj7Rs/qKprB9NZIck6vEhiIngV3gxKsj7Ju5J8rft5Zzfc\nIc2SK5I8Pcna7ufpwBV9FyWN2Ue723IckORRwDuA9/Zck3AS+UxK8q8Mhjbe2m16OvC0qnJoQzOj\nm1j7GuBnGPyL/JPAb1fVtl4Lk8aou3HmrwGPZnDD2A8Cf1v+8e6dAWoGdTcTfOBK26RZk+R3qupV\nfdchjUuS2wDXVNWubn0tcKuquqrfyuQQ3mxyaEP7q9/ruwBpzD4MHDC0fgDwoZ5q0RAD1Gw6AXgS\ncDlwGfAEwInl2h+k7wKkMbt1VX1vcaVb/pEe61HHq/BmUPdMvMf2XYfUA+ckaNZ8P8mDq+pcgCQ/\nBVzdc03CADWTktwJeDZwOEP/javqhL5qksYlyZUsH5TCjYc6pFnwPOAdSS5lcI7fGXhyvyUJDFCz\n6t0M7sr8IWBXz7VIYzX8rEdplnVX4N0SOAK4V7f5gqra2V9VWuRVeDPIK+4kaTb4HNPJ5STy2fS+\nJI/puwhl7g0sAAAEF0lEQVRJ0l77cJJfyeKtyDUx7IGaQd0ckdsAPwB2Mhg3r6o6qNfCJElNhr7P\ndzGYPO73+YQwQM2Y7l8pd62qr/ZdiyRJs8ohvBnT3d7//X3XIUnaexl4epI/7tbvmuTIvuuSAWpW\nnZvkoX0XIUnaa69j8LzHp3br3wNe2185WuRtDGbTTwNPS7IV+D43jJnfv9+yJEmNfrqqHpzkPICq\n+laSW/ZdlAxQs+rovguQJI3Fzu4BwgXX3yj5un5LEjiEN5O6R7ncFXhkt3wV/reWpGn0V8C7gB9N\n8lLgE8DL+i1J4FV4MynJnwAPAe5VVfdMchfgHVX18J5LkyQ1SnIEcBSD6RgfrqrNPZckHMKbVb8M\nPAg4F6CqLk3i4y8kaUokuTXwm8A9gM8Cr6+qH/ZblYY5rDObru1uZ7A4Zn6bnuuRJLV5M4ORhM8C\nxwKv7LccLWUP1Gx6e5LXA7dL8mzgBOANPdckSRrdfarqfgBJTgPO7rkeLWGAmkFV9cokjwK+y+AJ\n3i+qqn/tuSxJ0uh2Li5U1Q99FN7kcRL5jEtyR+CK8j+0JE2NJLsY3McPBpPHD2BwRbXPwpsQzoGa\nIUkelmQhyT8meVCSzwGfA3YkOabv+iRJo6mqtVV1UPdzYFWtG1o2PE0Ae6BmSJJPAy8EDgY2AcdW\n1VndJbCnV9WDei1QkqQZYQ/UbFlXVWdW1TuAy6vqLICq2tJzXZIkzRQD1GwZvr3/1Uv22dUoSdKY\nOIQ3Q4YmHQ5POKRbv3VV3aKv2iRJmiUGKEmSpEYO4UmSJDUyQEmSJDUyQEmSJDUyQEmaGEmuS/KW\nofW1Sb6e5D2Nx7k4yR32to0k7Y4BStIk+T7wk0lu1a0/Cth2M44zytUxXkEj6WYzQEmaNP8M/GK3\nvBE4fXFHktsneVeS85N8Msni0+rvkOSDST6b5A0Mbt2x+JqnJflUknOT/HVueCqrT2eVdLMZoCRN\nkgLOADZ2vVD3Bz41tP8U4NyqegDwh8DicN+fAB+vqvsB7wI2AHSPMXoy8F+r6sEMbjb7tNX4RSTN\ntnV9FyBJw6rqc0kOZ9D79H5u3FP0s8Dju3Yf6XqeDgQeAfxyt/2fk3yra38U8GDgnK7n6dbA5avx\ne0iabQYoSZPoPcCfAXPAHVdou9xcpuFhujdX1R+OrzRJcghP0mRZDD5vBE6pqs8v2f9x4OkASeaA\nb1TV94CP0Q3NJTkWuF3X/sPAE5Lcqdt3+yQb9ulvIGm/YA+UpElSAFW1HTh1mf3zwBuTnM/gir1n\nddtPAU5P8hTgk8BXu+NsTvJHwJlJ1gDXAs/p9nsVnqSbzWfhSZIkNXIIT5IkqZEBSpIkqZEBSpIk\nqZEBSpIkqZEBSpIkqZEBSpIkqZEBSpIkqZEBSpIkqdH/B0slNzu6vX04AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd0a090908>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results_all_uni.plot(kind='bar',color=['r','g','b'],title='Model Performance',figsize=(10,6),x='Model',legend=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## comparing negative samples accuracies\n",
    "negcom = pd.DataFrame()\n",
    "for name,model in models.items():\n",
    "    p= recall_score(Test_labels,tfprediction[name],pos_label='neg')\n",
    "    negcom.loc[name,'recall'] = p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAG6CAYAAAA23iOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUZWV97vHv0yCCyBgUFMQxSjSCEMRxmVJUBhNR44Ri\nrpooMZBodC1R442FYtRovAmgURQNGhVFNCJOqKEcoigKKiqtOCEzCgiIIm3zu3+cXXC6urr6NNTe\nu2vX97NWrT57qFNP9zqr++l3v/vdqSokSZLUjRV9B5AkSVpOLF+SJEkdsnxJkiR1yPIlSZLUIcuX\nJElShyxfkiRJHbJ8SRqsJP+R5B/7ziFJ4+I6X5I2VkluBO5VVT8Z2/eqZt+z+ksmSbecI1+SNmbr\n+t9h6/9rTOLfj5Ja4V8ukjZmWfBg8qdJLkjy4iSXJbkoybPHjr87yavHtl+a5OIkFyb5qyQ3JrnH\n2LlvTfKJJNcCU0kOTHJWkquTnN+Mus2+112b7392kp8nuSLJoUn2TvLtJFcmOWbx/0gkLXWb9h1A\nkm6lnYCtgDsDjwU+nOSjVXX1+ElJ9gdeBDwK+BnwDtYeQTsYOKCqzkiyGfBg4FlV9b0kfwx8NsnZ\nVXXK2PfsA9wLeATwceBTzc+4LXB2kg9V1ZcW9XcsaUlz5EvSUncD8JqqWl1VnwJ+DdxnnvOeAry7\nqlZW1fXA9DznfKyqzgCoqhuq6otV9b1m+7vAicCfjp1fwKubcz8HXAd8oKquqKqLgS8Bey7Ob1PS\nUFi+JG3MVgO3mbPvNsCqse0rqurGse3fALef573uDFwwtn0Ba1/WHD9Okn2S/E+Sy5P8CjgU2GHO\n91w+9vq3wGVztufLImkZs3xJ2pj9HLjbnH13B86/Be91CbDL2PaurH3Zce72+4H/Bnauqm2Bt7Oe\neWiStD6WL0kbsw8Cr0yyc0YeDfwZ8OFb8F4fAp6TZLcktwNeOcH33B64qqpWJdkHeMac4xYxSRvM\n8iVpY/Zq4CvAl4ErgdcDz6iq7y/wPfMuQ1FVnwaOBk4Hfgh8tTn0uwXe62+B1yS5mlFZ++B6ftb6\ntiWp3UVWkxzP6H+pl1XV7us452jgAEYTVZ9dVd9qLZAkNZLsBpwD3HbOnDFJalXbI1/vBvZb18Ek\nBwD3rKo/ZDSR9W0t55G0jCV5QpLNkmwHvAE4xeIlqWutlq+q+jJw1QKnHAS8pzn3a8A2SXZsM5Ok\nZe1QRncnnsfojsm/7TeOpOWo70VWd2bNW7svavZdNv/pknTLVdUBfWeQJCfcS5Ikdajvka+LgLuM\nbe/S7FtLEu8akiRJS0ZVzbscTRflK6x7LZxTgMOADyZ5MPCrqlrnJcc278zU2qanp5menu47htQq\nP+daDvycdy9Z9zKArZavJO8HpoA/SPJz4FXAZkBV1XFV9ckkByb5EaOlJp7TZh5JkqS+tVq+qmru\natDznXN4mxkkSZI2Jk641zpNTU31HUFqnZ9zLQd+zjcura5wv5iS1FLJKkmSlrck65xw78iXJElS\nhyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkd\nsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI\n8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHL\nlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxf\nkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJ\nkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJ\nktSh1stXkv2TrEzywyRHzHN86ySnJPlWknOSPLvtTJIkSX1JVbX35skK4IfAvsDFwJnA06tq5dg5\nLwe2rqqXJ9kB+AGwY1X9fs57VZtZJUmSFksSqirzHWt75Gsf4LyqOr+qVgEnAgfNOaeArZrXWwFX\nzC1ekiRJQ9F2+doZuGBs+8Jm37hjgfsmuRj4NvDCljNJkiT1ZmOYcL8fcHZV3RnYE3hLktv3nEmS\nJKkVm7b8/hcBu45t79LsG/cc4HUAVfXjJD8FdgO+MffNpqenb3o9NTXF1NTU4qaVJEm6BWZmZpiZ\nmZno3LYn3G/CaAL9vsAlwNeBg6vq3LFz3gJcXlVHJtmRUenao6qunPNeTriXJElLwkIT7lsd+aqq\n1UkOB05jdInz+Ko6N8mho8N1HHAU8J9JvtN820vnFi9JkqShaHXkazE58iVJkpaKPpeakCRJ0hjL\nlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxf\nkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJ\nkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHVog8pX\nku2S7N5WGEmSpKFbb/lKMpNk6yTbA2cB70jy5vajSZIkDc8kI1/bVNU1wJOA91TVg4BHtxtLkiRp\nmCYpX5smuRPwVODUlvNIkiQN2iTl69XAZ4AfV9WZSe4BnNduLEmSpGFKVfWdYSJJaqlklZajnXbZ\nicsuuqzvGMvKjjvvyKUXXtp3DEnzSEJVZd5j6ys0Se4N/AewY1X9cXO34+Or6qjFj7pgDsuXtBFL\nAtN9p1hmpsG/F6WN00Lla5LLju8AXg6sAqiq7wBPX7x4kiRJy8ck5et2VfX1Oft+30YYSZKkoZuk\nfP0yyT2BAkjyZOCSVlNJkiQN1KYTnHMYcBywW5KLgJ8Ch7SaamDuttNOnH+ZE5G7dNcdd+RnlzoR\nWZK08Vlv+aqqnwCPTrIlsKKqrm0/1rCcf9llOCW2W7HsSpI2UussX0levI79AFSVjxiSJEnaQAuN\nfG3VWQpJkqRlYp3lq6qO7DKIJEnScrDeux2T3CPJx5P8IsnlST7WPGJIkiRJG2iSpSbeD3wIuBNw\nZ+Ak4ANthpIkSRqqSRdZfW9V/b75+i9g87aDSZIkDdEk63x9KsnLgBMZLbT6NOCTSbYHqKorW8wn\nSZI0KJOUr6c2vx46Z//TGZUx539JkiRNaJJFVu/eRRBJkqTlYL3lK8kmwOOAu42f7yKrkqTlZqed\n7sZll53fd4xlZccd78qll/6s7xiLapLLjh8HrgfOAW5sN44kSRuvUfHygXFduuyy9B1h0U1Svnap\nqt1bTyJJkrQMTLLUxKeSPLb1JJIkScvAJCNfZwAfTbICWAUEqKrautVkkiRJAzRJ+Xoz8BDgnKry\nQrckSdKtMMllxwuA71q8JEmSbr1JRr5+Aswk+RTwu9mdLjUhSZK04SYpXz9tvjZrviRJknQLTbLC\n/ZG35gck2R/4N0aXOI+vqjfMc84U8P+A2wC/qKpH3pqfKUmStLGaZIX7OwAvBe4HbD67v6oeNcH3\nrgCOBfYFLgbOTPKxqlo5ds42wFuAx1bVRUl22ODfhSRJ0hIxyYT79wErgbsDRwI/A86c8P33Ac6r\nqvOrahVwInDQnHOeAZxcVRcBVNUvJ3xvSZKkJWeS8vUHVXU8sKqqvlBVzwXWO+rV2JnR3ZKzLmz2\njbs3sH2S05OcmeRZE763JEnSkjPJhPtVza+XJHkco8uH2y9yhr0YFbotga8m+WpV/WgRf4YkSdJG\nYZLydVQzL+slwDHA1sA/TPj+FwG7jm3v0uwbdyHwy6q6Hrg+yReBPYC1ytf09PRNr6emppiampow\nhiRJUntmZmaYmZmZ6Ny0uXZqkk2AHzCacH8J8HXg4Ko6d+yc3RiVuv2B2wJfA55WVd+f815Ldp3X\nJCzN5EtX8wysvmMsK0lguu8Uy8y0n/OuJQH/Ru9YluTnPAlVlfmOrXfOV5J/SbJ1ktsk+XySXyQ5\nZJIfXFWrgcOB04DvASdW1blJDk3y/OaclcBngO8weo7kcXOLlyRJ0lCsd+Qrybeq6gFJngj8GfBi\n4ItVtUcXAcdyOPKliTny1T1Hvnow7ee8a4589WEZjnxx87ywxwEnVdXVi5ZMkiRpmZlkwv2pSVYC\nvwVe0Cy6en27sSRJkoZpvSNfVfUy4KHA3s1Cqb9h7YVSJUmSNIFJRr6oqivHXl8HXNdaIkmSpAGb\nZM6XJEmSFonlS5IkqUOTrPOVJIck+adme9ck+7QfTZIkaXgmGfl6K/AQ4OBm+1rgLa0lkiRJGrBJ\nJtw/qKr2SnI2QFVdlWSzlnNJkiQN0iQjX6uaZzQWQLPO142tppIkSRqoScrX0cBHgTsmeS3wZeCf\nW00lSZI0UOu97FhV70vyTWBfRo/Me0JVndt6MkmSpAFab/lKcjRwYlU5yV6SJOlWmuSy4zeBVyb5\ncZI3Jdm77VCSJElDNcmzHU+oqgOBBwI/AN6Q5LzWk0mSJA3Qhqxwfy9gN+CuwMp24kiSJA3bJCvc\n/0sz0vVq4LvA3lX1560nkyRJGqBJFln9MfCQqvpl22EkSZKGbp3lK8luVbUSOBPYNcmu48er6qy2\nw0mSJA3NQiNfLwaeD/zrPMcKeFQriSRJkgZsneWrqp7fvDygqq4fP5Zk81ZTSZIkDdQkdzt+ZcJ9\nkiRJWo+F5nztBOwMbJFkT0aPFgLYGrhdB9kkSZIGZ6E5X/sBzwZ2Ad48tv9a4BUtZpIkSRqsheZ8\nnQCckOQvqurkDjNJkiQN1nrX+aqqk5M8DrgfsPnY/le3GUySJGmIJlnh/m3A04C/YzTv6ymMHjEk\nSZKkDTTJ3Y4Praq/BK6qqiOBhwD3bjeWJEnSME1Svn7b/PqbJHcGVgF3ai+SJEnScE3ybMdTk2wL\nvBE4i9Hq9u9sNZUkSdJATTLh/jXNy5OTnApsXlVXtxtLkiRpmNZbvpI8aZ59VwPnVNXlraSSJEka\nqEkuO/4Vo0n2pzfbU8A3gbsneXVVvbelbJIkSYMzSfnaFPijqroMIMmOwHuABwFfBCxfkiRJE5rk\nbse7zBavxuXNvisZ3fkoSZKkCU0y8jXTTLQ/qdl+crNvS+BXrSWTJEkaoEnK12HAk4CHN9snACdX\nVQGPbCuYJEnSEE2y1EQl+QZwdVV9LsntgNsD17aeTpIkaWAmebbj84APA29vdu0M/HeboSRJkoZq\nkgn3hwEPA64BqKrzgDu2GUqSJGmoJilfv6uqG2Y3kmzK6BFDkiRJ2kCTlK8vJHkFsEWSxzC66/Hj\n7caSJEkapknK18uAXwDnAIcCnwRe2WYoSZKkoZrkbscbgXc0X5IkSboVJnmw9sOAaeCuzflhtALF\nPdqNJkmSNDyTLLJ6PPAPjB6mvbrdOJIkScM2Sfm6uqo+1XoSSZKkZWCS8nV6kjcCHwF+N7uzqs5q\nLZUkSdJATVK+HtT8uvfYvgIetfhxJEmShm2Sux19eLYkSdIimWSdL0mSJC0Sy5ckSVKHFixfSVYk\neWhXYSRJkoZuwfLVrG7/lo6ySJIkDd4klx0/n+QvkqT1NJIkSQM3Sfk6FDgJuCHJNUmuTXJNy7kk\nSZIGaZKlJrbqIogkSdJyMMkiqyR5PPCIZnOmqk5tL5IkSdJwrfeyY5LXAy8Evt98vTDJ6yb9AUn2\nT7IyyQ+THLHAeQ9MsirJkyZ9b0mSpKVmkpGvA4EHNHc+kuQE4Gzg5ev7xiQrgGOBfYGLgTOTfKyq\nVs5z3uuBz2xYfEmSpKVl0kVWtx17vc0GvP8+wHlVdX5VrQJOBA6a57y/Az4MXL4B7y1JkrTkTDLy\n9Trg7CSnA2E09+tlE77/zsAFY9sXMipkN0lyZ+AJVfXIJGsckyRJGpp1lq8kD6uq/wU+AswAD2wO\nHVFVly5ihn8DxueCuZ6YJEkarIVGvo4G/gT4alXtBZxyC97/ImDXse1dmn3j9gZObBZx3QE4IMmq\nqlrr501PT9/0empqiqmpqVsQSZIkaXHNzMwwMzMz0bmpqvkPJGcA3wGewGiu1hqq6u/X++bJJsAP\nGE24vwT4OnBwVZ27jvPfDXy8qj4yz7FaV9aNXRKWZvKlK8BS/bwsVUlguu8Uy8y0n/OujcYJ/DPv\nVpbk5zwJVTXv1byFRr7+DHg0sB/wzVvyg6tqdZLDgdMYTe4/vqrOTXLo6HAdN/dbbsnPkSRJWirW\nOfJ10wnJHlX17Y7yLJTDkS9NzJGv7jny1YNpP+ddc+SrD8Mb+VrvUhMbQ/GSJEkaiknX+ZIkSdIi\nsHxJkiR1aKF1vl680DdW1ZsXP44kSdKwLXS341adpZAkSVom1lm+qurILoNIkiQtBwtddjx6oW+c\nZJFVSZIkrWmhy463aGFVSZIkrdtClx1P6DKIJEnScrDQyBcASe4AHAHcF9h8dn9VParFXJIkSYM0\nyTpf7wPOBe4OHAn8DDizxUySJEmDNUn5+oOqOh5YVVVfqKrnAo56SZIk3QLrvewIrGp+vSTJ44CL\nge3biyRJkjRck5Svo5JsA7wEOAbYGviHVlNJkiQN1HrLV1Wd2ry8Gnhku3EkSZKGbb1zvpKckGTb\nse3tkryr3ViSJEnDNMmE+92r6lezG1V1FbBne5EkSZKGa5LytSLJdrMbSbZnsrlikiRJmmOSEvWv\nwFeTnNRsPwV4bXuRJEmShmuSCffvSfINbl7b60lV9f12Y0mSJA3TJJcdYbSu13VVdSzwiyR3bzGT\nJEnSYE1yt+OrGD3b8eXNrtsA/9VmKEmSpKGaZOTricDjgesAqupiYKs2Q0mSJA3VJOXrhqoqoACS\nbNluJEmSpOGapHx9KMnbgW2TPA/4HPDOdmNJkiQN0yR3O74pyWOAa4D7AP9UVZ9tPZkkSdIATbRY\nalO2PguQZEWSZ1bV+1pNJkmSNEDrvOyYZOskL09ybJLHZuRw4CfAU7uLKEmSNBwLjXy9F7gK+Crw\n18ArgABPqKpvdZBNkiRpcBYqX/eoqvsDJHkncAmwa1Vd30kySZKkAVrobsdVsy+qajVwocVLkiTp\n1llo5GuPJNc0rwNs0WwHqKrauvV0kiRJA7PO8lVVm3QZRJIkaTmY9MHakiRJWgSWL0mSpA5ZviRJ\nkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ\n6pDlS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSp\nQ5YvSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUOtl68k+ydZmeSHSY6Y5/gzkny7+fpykvu3nUmSJKkv\nrZavJCuAY4H9gPsBByfZbc5pPwEeUVV7AEcB72gzkyRJUp/aHvnaBzivqs6vqlXAicBB4ydU1RlV\ndXWzeQawc8uZJEmSetN2+doZuGBs+0IWLld/DXyq1USSJEk92rTvALOSPBJ4DvDwvrNIkiS1pe3y\ndRGw69j2Ls2+NSTZHTgO2L+qrlrXm01PT9/0empqiqmpqcXKKUmSdIvNzMwwMzMz0bmpqtaCJNkE\n+AGwL3AJ8HXg4Ko6d+ycXYHPA8+qqjMWeK9qM2ubkrA0ky9dAZbq52WpSgLTfadYZqb9nHctCfg3\neseyJD/nSaiqzHes1ZGvqlqd5HDgNEbzy46vqnOTHDo6XMcB/xfYHnhrRp/qVVW1T5u5JEmS+tLq\nyNdicuRLG8KRr+458tWDaT/nXXPkqw/DG/lyhXtJkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKk\nDlm+JEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6\nZPmSJEnqkOVLkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ\n5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUOW\nL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+\nJEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmS\nJEnqkOVLkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjrUevlKsn+SlUl+\nmOSIdZxzdJLzknwryQPaziRJktSXVstXkhXAscB+wP2Ag5PsNuecA4B7VtUfAocCb2szkyY303cA\nqQs/7TuA1IWZvgNoTNsjX/sA51XV+VW1CjgROGjOOQcB7wGoqq8B2yTZseVcmsBM3wGkLvys7wBS\nF2b6DqAxbZevnYELxrYvbPYtdM5F85wjSZI0CE64lyRJ6tCmLb//RcCuY9u7NPvmnnOX9ZwDQJJF\nDdelpZr8yL4D3ApL+fOyZE33HeAW+kLfAW45P+d9WKp/5kv3b/Shfc7bLl9nAvdKclfgEuDpwMFz\nzjkFOAz4YJIHA7+qqsvmvlFVDetPXpIkLUutlq+qWp3kcOA0Rpc4j6+qc5McOjpcx1XVJ5McmORH\nwHXAc9rMJEmS1KdUVd8ZJEmSlg0n3EuSJHXI8iVJktShtifcawlLsgNwRXltWgOT5DDgfVX1q2Z7\nO+Dgqnprv8mkxZPkDsDzgLsx9u99VT23r0wacc6XAGjuNH09cCXwGuC9wA6MRkf/sqo+3WM8aVEl\n+VZVPWDOvrOras++MkmLLclXgC8B3wRWz+6vqpN7CyXAkS/d7FjgFcA2wP8AB1TVGc2zOD8AWL40\nJJskyeyobpJNgM16ziQttttV1RF9h9DanPOlWZtW1WlVdRJwaVWdAVBVK3vOJbXh04zWFtw3yb74\nHwwN06lJDuw7hNbmZUcBkOSsqtpr7uv5tqWlLskK4FBg32bXZ4F3VtXqdX+XtLQkuRbYErgBWNXs\nrqraur9UAsuXGklWM1rkNsAWwG9mDwGbV9Vt+somSdKQWL4kLRtJPlRVT01yDrDWX35VtXsPsaTW\nJHk88Ihmc6aqTu0zj0YsX5KWjSR3qqpLmufNrqWqzu86k9SWJK8HHgi8r9l1MPCNqnp5f6kEli81\nmrkBxegy46xidEfsZlXlnbEajCRvmHsX2Hz7pKUsyXeAB1TVjc32JsDZjvD2z7sdBUBVbVVVWze/\nbgXcCXgtcCnw7/2mkxbdY+bZd0DnKaT2bTv2epveUmgNjmZoDUm2BV4E/CXwfuCBVXVFv6mkxZHk\nBcDfAvdsRgVmbQX8bz+ppNa8Djg7yemMrmo8AnhZv5EEXnZUo3mU0EuApwHvAo6pqqv7TSUtriTb\nANsx+kdp/B+ha6vqyn5SSYsvSYBdgN8zmvcF8PWqurS/VJpl+RIASa4DfgG8G7h27vGqenPnoaSW\nJLkncGFV/S7JFLA78J7ZZz1KQ5DknKq6f985tDbnfGnWGxkVLxhdgpn7JQ3JycDqJPcCjgPuwugy\nuzQkZyV54PpPU9cc+ZK07Mw+tSHJS4HfVtUxPlhbQ5NkJXAv4HxuXkS7vNuxf064FwBJXlpV/5Lk\nGOZffPLve4gltWVVkoMZ3Vjy580+n+Kgodmv7wCan+VLs85tfv1GrymkbjwH+BvgtVX10yR3B97b\ncyZpsR1VVc8a35HkvcCz1nG+OuJlR0mSBmj28vrY9ibAOVV13x5jCUe+1Ejycea53Dirqh7fYRyp\nFT7bUctBkpcDrwC2SHINNz+55AZGN5ioZ458CYAkf7rQ8ar6QldZpLb4bEctJ0le53McN06WL0mS\nBqhZaPWJwMMZjfR+qar+u99UAsuXGuu6DDPLyzEakrEHyY+7mtENJy+pqp90n0paXEneymipiQ80\nu54G/LiqDusvlcDypca6LsPM8nKMhiTJa4ALGS2sGuDpwD2Bs4AXVNVUf+mkxdGs8/VH1fxDn2QF\n8L2q+qN+k8kJ9wIsV1p2Hl9Ve4xtH5fkW1V1RJJX9JZKWlw/AnZltMgqjJ7k8KP+4miWjxfSGpJc\nm+Sa5uv6JKubu2WkIflNkqcmWdF8PRW4vjnm5QANxVbAuUlmkpwOfB/YOskpSU7pOduy5siX1lBV\nNz3HsZmseRDw4P4SSa14JvDvwFub7a8ChyTZAji8t1TS4vqnvgNofs750nr5zDtJWpqa+bx/WFWf\na/5zsWlVXdt3ruXOkS+tIcmTxjZXAHtz8+UYaRCS7AIcAzys2fUl4IVVdWF/qaTFleR5wPOB7Rnd\nULIL8DZg3z5zyfKltf352OvfAz9jdOlRGpJ3M7rT8SnN9iHNvsf0lkhafIcB+wBfA6iq85Lcsd9I\nAsuX5qiq5/SdQerAHarq3WPb/5nkRb2lkdrxu6q6YTR9F5JsijeUbBS821FrSLJLko8mubz5Orm5\nRCMNyRVJDkmySfN1CHBF36GkRfaFZumULZI8BjgJ+HjPmYQT7jVHks8yuhzz3mbXIcAzq8rLMRqM\nZhLyMcBDGI0EfAX4u6q6oNdg0iJqFlX9K+CxjBYT/gzwzvIf/t5ZvrSGZqHJB6xvnzQ0SV5UVf/W\ndw5psSTZEri+qlY325sAt62q3/SbTF521FxejtFy9eK+A0iL7PPAFmPbWwCf6ymLxli+NNdzgacC\nlwKXAE8GnISv5SB9B5AW2eZV9evZjeb17XrMo4Z3O2oNzTMeH993DqkHzsHQ0FyXZK+qOgsgyZ8A\nv+05k7B8aY4kdwCeB9yNsc9HVT23r0zSYklyLfOXrLDm5RlpCF4InJTkYkaf8Z2Ap/UbSWD50to+\nxmi1788Bq3vOIi2q8WeXSkPW3Om4GbAbcJ9m9w+qalV/qTTLux21Bu9slKRh8Lm8Gy8n3GuuU5Mc\n2HcISdIwtcYNAAAEbklEQVSt9vkkf5HZJe610XDkS2to5sRsCfwOWMVonkBV1da9BpMkbZCxv89X\nM5po79/nGwnLl27S/O/oLlX1876zSJI0VF521E2aR058ou8ckqRbLyOHJPm/zfZdkuzTdy5ZvrS2\ns5I8sO8QkqRb7a2Mnl/6jGb718Bb+oujWS41obkeBDwzyfnAddw8R2D3fmNJkjbQg6pqryRnA1TV\nVUk26zuULF9a2359B5AkLYpVzcO0C25aRPvGfiMJvOyoOZrHC90FeFTz+jf4OZGkpeho4KPAHZO8\nFvgy8M/9RhJ4t6PmSPIqYG/gPlV17yR3Bk6qqof1HE2StIGS7Absy2gKyeer6tyeIwkvO2ptTwT2\nBM4CqKqLk/hIFklaIpJsDvwNcC/gHODtVfX7flNpnJeTNNcNzZITs3MEtuw5jyRpw5zA6ArGOcAB\nwJv6jaO5HPnSXB9K8nZg2yTPA54LvKPnTJKkyd23qu4PkOR44Os959Ecli+toarelOQxwDXAfYB/\nqqrP9hxLkjS5VbMvqur3Ptpx4+OEe61Tkh2AK8oPiSQtGUlWM1qnEUYT7bdgdOe6z3bcSDjnSwAk\neXCSmSQfSbJnku8C3wUuS7J/3/kkSZOpqk2qauvma6uq2nTstcVrI+DIlwBI8g3gFcA2wHHAAVV1\nRnOb8geqas9eA0qSNBCOfGnWplV1WlWdBFxaVWcAVNXKnnNJkjQoli/NGn/kxG/nHHN4VJKkReJl\nRwFrTNAcn5xJs715Vd2mr2ySJA2J5UuSJKlDXnaUJEnqkOVLkiSpQ5YvSZKkDlm+JC1pSW5M8p6x\n7U2S/CLJKRv4Pj9Nsv2tPUeS1sfyJWmpuw744yS3bbYfA1xwC95nkruPvENJ0q1m+ZI0BJ8EHte8\nPhj4wOyBJNsl+WiSbyf5SpL7N/u3T/KZJOckeQejZVVmv+eZSb6W5Kwk/5Gbn0yc5vjtkpya5Owk\n30nylE5+l5IGwfIlaakr4ETg4Gb0a3fga2PHjwTOqqo9gH8EZi9Rvgr4UlXdH/gosCtA80itpwEP\nraq9GC1A/Mw5P3N/4KKq2rOqdgc+3crvTNIgbdp3AEm6tarqu0nuxmjU6xOMjWIBDwee1Jx3ejPi\ntRXwCOCJzf5PJrmqOX9fYC/gzGbEa3Pg0jk/8hzgTUleB3yiqr7cym9M0iBZviQNxSnAG4EpYIf1\nnDvf3K3xS4snVNU/rvObq85LshdwIHBUks9V1VEbHlnScuRlR0lL3WxpehdwZFV9b87xLwGHACSZ\nAn5ZVb8GvkhzOTHJAcC2zfmfB56c5A7Nse2S7LrGD0zuBPy2qt7PqPDttdi/KUnD5ciXpKWuAKrq\nIuDYeY5PA+9K8m1Gd0b+n2b/kcAHkjwd+Arw8+Z9zk3ySuC0JCuAG4DDmuOzI2b3B96Y5Mbm+Ata\n+H1JGiif7ShJktQhLztKkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocs\nX5IkSR36/6TB4wAd0ee+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd2aaef588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pl = negcom.plot(kind='bar',color=['r','g','b','y'],title='Unigram',figsize=(10,6),legend=False)\n",
    "pl.set(xlabel='Models',ylabel='Recall for negative samples')\n",
    "lim = plt.ylim([0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b. Using Bigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "## using countvectorizer to generate count vectors for the training data.\n",
    "vec_all = CountVectorizer(ngram_range=(1,2))\n",
    "ctr_features_all = vec_all.fit_transform(Data_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf_vec_all = TfidfTransformer()\n",
    "tr_features_all = tf_vec_all.fit_transform(ctr_features_all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cte_features_all = vec_all.transform(Data_test)\n",
    "te_features_all = tf_vec_all.transform(cte_features_all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(426340, 263567)\n"
     ]
    }
   ],
   "source": [
    "print ((ctr_features_all.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   TF-IDF Accuracy        Model\n",
      "0         0.887231  BernoulliNB\n",
      "1         0.928550     Logistic\n",
      "2         0.903760   Perceptron\n"
     ]
    }
   ],
   "source": [
    "results_all_bi = pd.DataFrame()\n",
    "\n",
    "#stratified = StratifiedKFold(labels, n_folds=5,random_state=20160121, shuffle=True)\n",
    "tfprediction = {}\n",
    "cprediction = {}\n",
    "foldnum = 0\n",
    "for name,model in models.items():\n",
    "        \n",
    "        model.fit(tr_features_all, Train_labels)\n",
    "        tfprediction[name] = model.predict(te_features_all)\n",
    "        tfaccuracy = metrics.accuracy_score(tfprediction[name],Test_labels)\n",
    "        \n",
    "        #model.fit(ctr_features_all,Train_labels)\n",
    "        #cprediction[name] = model.predict(cte_features_all)\n",
    "        #caccuracy = metrics.accuracy_score(cprediction[name],Test_labels)\n",
    "        \n",
    "        results_all_bi.loc[foldnum,'TF-IDF Accuracy']=tfaccuracy\n",
    "        #results_all_bi.loc[foldnum,'Count Accuracy']=caccuracy\n",
    "        results_all_bi.loc[foldnum,'Model']=name\n",
    "        foldnum = foldnum+1\n",
    "print (results_all_bi)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for  BernoulliNB\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.79      0.67      0.72     31169\n",
      "        pos       0.91      0.95      0.93    110945\n",
      "\n",
      "avg / total       0.88      0.89      0.88    142114\n",
      "\n",
      "\n",
      "\n",
      "Classification report for  Logistic\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.86      0.80      0.83     31169\n",
      "        pos       0.95      0.96      0.95    110945\n",
      "\n",
      "avg / total       0.93      0.93      0.93    142114\n",
      "\n",
      "\n",
      "\n",
      "Classification report for  Perceptron\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.78      0.78      0.78     31169\n",
      "        pos       0.94      0.94      0.94    110945\n",
      "\n",
      "avg / total       0.90      0.90      0.90    142114\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for name,model in models.items():\n",
    "    print (\"Classification report for \",name)\n",
    "    print(metrics.classification_report(Test_labels, tfprediction[name]))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1cd08116cf8>"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAG6CAYAAADH1Xg8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHz1JREFUeJzt3XmUdGddJ/DvLwmbkAQwGCQhQUGJIKsQmVGxJQdCcCSj\nI0vYw4jRAVGHM0ZRoRlkmXEDCajBgCwKyiAj20gk0oCDYZkAAiasIWQHwh4IhPCbP6o6VJpOup+k\n367uej+fc/q8de996tav6r2n+tvP89x7q7sDAMDm7TPvAgAAdhsBCgBgkAAFADBIgAIAGCRAAQAM\nEqAAAAYJUMCwqjq8qr5VVRt+h1TVo6vq7dtU1w2r6nVV9YWq+pvteE1g7yRAwYKrqk9W1WVVdfM1\n6987DUGHXctdj1xEbt22M0HsS9OfT1TVideyniT5+SS3SHKz7n7IddgPwDUSoGDxdZKzkxy3uqKq\nfjjJjTIWgvaUTnJgdx+Q5GFJnlJV9xvdybQ37PAkH+lrcYXgqtp39DnA3kuAgr3Dy5I8emb50Ule\nMtugqg6oqpdW1aer6uyq+u2ZbftU1R9U1Weq6mNJfnqd5/5FVV1QVedW1dOrqgbqqyTp7tOTfCjJ\nD0/3e0RVnVpVl1TVmVX1oJnXfHFVvaCq3lBVX07y1iRPSfLQaW/W8TXxO9NeuIuq6i+r6oDp81d7\nvx5bVeckOW1m3WOq6lPT1z2hqu5RVe+vqs9V1fNmavj+qjqtqj47/dxevrr/6fazq+pJ0+d+vqpe\nUVXXn9l+7LQn8ItV9dHV4LgFnyewhwlQsHc4Pcn+VXX7aU/NQ5K8PNPgMnVSkv2T3CbJUpJHVdXx\n022/mOQBSe6S5B6ZDJXNekmSbyT5/iR3S3LfJL8wUF8lSVX9WJI7JDmjqr4ryanTOg9K8tAkL6iq\nI2aed1ySp3f3/kmOSvLMJK/s7gO6+8VJjk/yqCQ/Oa1t/+n7nHXvJEckOXpm3ZFJbpfJ5/ScJE9O\ncp9Mgt2Dq+onZup+ZpJbJvmhJIcmWV6z/wcluV+S78vk83vM9L0emcnn9qTuPnBaxyenz7munyew\nhwlQsPdY7YW6b5Izk1ywumEmVP1md3+1u89J8odJHjlt8qAkz+nuC7r7C0meNfPcg5Mck+TXu/uy\n7v5sJqHjyiHDDVSSz1TVJUlOTnJid68k+Q9Jzu7ul/bE+5O8elrLqr+f9lqlu7+xzr4fluSPuvuc\n7v5qkt/KpIdq9buvkzy1u7/W3V+fWfffu/sb3f3mJJcmeUV3X9LdFyR5eyahJt398e4+rbu/2d2X\nJPnjTMLarOd298XTz+11Se46Xf/YJKd09z9N93Vhd3+kqr7nOn6ewDbYb94FANvm5UnelklPyEvX\nbDsok++DT82sOyfJIdPHt0py7pptqw5Lcr0kF05HmWr6M7uva9JJvnudeUuHJ7lXVX1uulxJ9l1T\n+7m5ZrdaU+s5mbzPg2fWnbfO8z498/hrSS5es3yTJJmGnecm+Ynpun2TfC5XNfvcryb53unjWyd5\nwzqvfXiu2+cJbAMBCvYS3f2pqjo7k96Nx67Z/Nkkl2fyy/us6brDk5w/fXxhJr/wM7Nt1blJLsv6\nIWizKt85of3cJCvdffQ67Vdt9HoX5Kq1Hp7J+7w4334/12Ui/TOTfCvJHbv7i1V1bJLnbfCcVecm\nue3VrL+unyewhxnCg73LY5Pcp7u/Nruyu7+V5G+TPKOqblJVhyf59UyG/TLd9sSqOqSqbpbkxJnn\nXpTJXKU/rqr9pxO3v7+q7r3Jmq5ucvTrk/xgVT2iqvarqutNJ3PfftPvNnlFkl+vqttU1U2SPCOT\nOVLfuobXHpmsvX+SryT5clUdkuS/DTz3lCTHV9VPTT+zW1XV7bfg8wS2gQAFi+/KXozuPru7z1hv\nW5InZjLE9IlMhvpePp2InSQvTPKmJO9P8p5M5iLNelSS6yf5t0yGsF6VycTqofqusrL7K5lMvn5o\nJj1JFyR5dpIbbHK/SfKiTELg25J8PJP398QNXnvtumtaflqSH0myOr9p7edytT1I3f3uTCa5PyfJ\nF5OsZDIcmly3zxPYBrVRD3FVnZLJZM6Lu/vOV9PmTzIZFrg0yWO6+31bXSgAwE6xmR6oF+eqp/de\nRVUdk+S23f0DSU5I8mdbVBsAwI60YYDq7n9O8vlraHJspmfFdPc7kxw4Pa0ZAGAhbcUcqENy1VOJ\nz8+3T30GAFg4JpEDAAzaiutAnZ+rXh/m0Hz72jFXUVWuaQIA7Brdve6lTTYboFavhLue1yZ5fJK/\nqap7JflCd198NW3junDba3l5OcvLy/MuA/Yoxzl7A8f59rume3hvGKCq6q8zubHod1fVp5I8NZPr\nk3R3n9zdb6yqB9TkDu2XZnJdEwCAhbVhgOruh22izRO2phwAgJ3PJPIFt7S0NO8SYI9znLM3cJzv\nLBteiXxLX6zKvTEBgF2hqq52ErkeKACAQQIUAMAgAQoAYJAABQAwSIACABgkQAEADBKgAAAGCVAA\nAIMEKACAQQIUAMAgAQoAYJAABQAwSIACABgkQAEADBKgAAAGCVAAAIMEKACAQQIUAMCg/eZdALBz\n3PLQW+bi8y+edxl7lYMPOTgXnXfRvMsABlV3b9+LVfV2vh4wpqqS5XlXsZdZTnwvws5UVenuWm+b\nITwAgEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAxyHSgA9iq3vOVtcvHF58y7jL3KwQcfnosu+uS8\ny9hSAhQAe5VJeHLtre108cXrXkppVzOEBwAwSIACABgkQAEADBKgAAAGmUS+Sbe55S1zzsXuUr+d\nDj/44HzyInepB2DnEaA26ZyLL3bOxjYrgRWAHcoQHgDAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAA\nBglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABglQAACDBCgAgEECFADAIAEK\nAGCQAAUAMEiAAgAYJEABAAwSoAAABm0qQFXV/avqrKr6SFWduM72A6rqtVX1vqr6QFU9ZssrBQDY\nITYMUFW1T5KTkhyd5I5JjquqI9Y0e3ySD3X3XZP8VJI/rKr9trpYAICdYDM9UEcm+Wh3n9Pdlyd5\nZZJj17TpJPtPH++f5JLu/ubWlQkAsHNsJkAdkuTcmeXzputmnZTkDlV1QZL3J/nVrSkPAGDn2apJ\n5EcneW933yrJ3ZI8v6puskX7BgDYUTYzT+n8JIfNLB86XTfr+CTPSpLu/nhVnZ3kiCTvWbuz5eXl\nKx8vLS1laWlpqGAAgD1hZWUlKysrm2pb3X3NDar2TfLhJEcluTDJu5Ic191nzrR5fpJPd/fTqurg\nTILTXbr7c2v21Ru93k5VVdmdle9elWS3Hi+7VVUly/OuYi+z7DjfblWV+EbfZrUrj/OqSnfXets2\n7IHq7iuq6glJTs1kyO+U7j6zqk6YbO6Tk/xekr+sqn+dPu031oYnAIBFsWEP1Ja+mB4oBuiB2n56\noOZg2XG+3fRAzcPi9UC5EjkAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQA\nwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRA\nAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBB\nAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIA\nGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQo\nAIBBAhQAwCABCgBg0KYCVFXdv6rOqqqPVNWJV9NmqareW1UfrKq3bG2ZAAA7x34bNaiqfZKclOSo\nJBckeXdV/X13nzXT5sAkz09yv+4+v6oO2lMFAwDM22Z6oI5M8tHuPqe7L0/yyiTHrmnzsCSv7u7z\nk6S7P7u1ZQIA7BybCVCHJDl3Zvm86bpZP5jk5lX1lqp6d1U9cqsKBADYaTYcwhvYz92T3CfJjZP8\nS1X9S3d/bIv2DwCwY2wmQJ2f5LCZ5UOn62adl+Sz3X1Zksuq6m1J7pLkOwLU8vLylY+XlpaytLQ0\nVjEAwB6wsrKSlZWVTbWt7r7mBlX7JvlwJpPIL0zyriTHdfeZM22OSPK8JPdPcoMk70zykO7+tzX7\n6o1eb6eqquzOynevSrJbj5fdqqqS5XlXsZdZdpxvt6pKfKNvs9qVx3lVpbtrvW0b9kB19xVV9YQk\np2YyZ+qU7j6zqk6YbO6Tu/usqnpTkn9NckWSk9eGJwCARbFhD9SWvpgeKAbogdp+eqDmYNlxvt30\nQM3D4vVAuRI5AMAgAQoAYJAABQAwSIACABgkQAEADBKgAAAGCVAAAIMEKACAQQIUAMAgAQoAYJAA\nBQAwSIACABgkQAEADBKgAAAGCVAAAIMEKACAQQIUAMAgAQoAYJAABQAwSIACABgkQAEADBKgAAAG\nCVAAAIMEKACAQQIUAMAgAQoAYJAABQAwSIACABgkQAEADBKgAAAGCVAAAIMEKACAQQIUAMAgAQoA\nYJAABQAwSIACABgkQAEADBKgAAAGCVAAAIMEKACAQQIUAMAgAQoAYJAABQAwSIACABgkQAEADBKg\nAAAGCVAAAIMEKACAQQIUAMAgAQoAYJAABQAwSIACABgkQAEADBKgAAAGCVAAAIMEKACAQQIUAMAg\nAQoAYNCmAlRV3b+qzqqqj1TVidfQ7p5VdXlV/dzWlQgAsLNsGKCqap8kJyU5OskdkxxXVUdcTbtn\nJ3nTVhcJALCTbKYH6sgkH+3uc7r78iSvTHLsOu1+Jcn/SvLpLawPAGDH2UyAOiTJuTPL503XXamq\nbpXkP3b3nyaprSsPAGDn2apJ5M9JMjs3SogCABbWfptoc36Sw2aWD52um3WPJK+sqkpyUJJjqury\n7n7t2p0tLy9f+XhpaSlLS0uDJQMAbL2VlZWsrKxsqm119zU3qNo3yYeTHJXkwiTvSnJcd595Ne1f\nnOR13f1362zrjV5vp6qq7M7Kd69KsluPl92qqpLleVexl1l2nG+3yd/6PvPtVbvyOK+qdPe6o2ob\n9kB19xVV9YQkp2Yy5HdKd59ZVSdMNvfJa59ynSsGANjBNuyB2tIX0wPFAD1Q208P1BwsO863mx6o\neVi8HihXIgcAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAA\nAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCAB\nCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAM\nEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQA\nwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRA\nAQAM2lSAqqr7V9VZVfWRqjpxne0Pq6r3T3/+uarutPWlAgDsDBsGqKraJ8lJSY5Ocsckx1XVEWua\nfSLJvbv7Lkl+L8kLt7pQAICdYjM9UEcm+Wh3n9Pdlyd5ZZJjZxt09+nd/cXp4ulJDtnaMgEAdo7N\nBKhDkpw7s3xerjkg/UKS/3NdigIA2Mn228qdVdVPJTk+yY9v5X4BAHaSzQSo85McNrN86HTdVVTV\nnZOcnOT+3f35q9vZ8vLylY+XlpaytLS0yVIBAPaclZWVrKysbKptdfc1N6jaN8mHkxyV5MIk70py\nXHefOdPmsCSnJXlkd59+DfvqjV5vp6qq7M7Kd69KsluPl92qqpLleVexl1l2nG+3qkp8o2+z2pXH\neVWlu2u9bRv2QHX3FVX1hCSnZjJn6pTuPrOqTphs7pOT/G6Smyd5QU2OzMu7+8itewsAADvHhj1Q\nW/pieqAYoAdq++mBmoNlx/l20wM1D4vXA+VK5AAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRA\nAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBB\nAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIA\nGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQo\nAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBI\ngAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBmwpQVXX/qjqrqj5SVSdeTZs/qaqPVtX7ququW1smAMDO\nsWGAqqp9kpyU5Ogkd0xyXFUdsabNMUlu290/kOSEJH+2B2rlWliZdwGwHc6edwGwHVbmXQAzNtMD\ndWSSj3b3Od19eZJXJjl2TZtjk7w0Sbr7nUkOrKqDt7RSrpWVeRcA2+GT8y4AtsPKvAtgxmYC1CFJ\nzp1ZPm+67pranL9OGwCAhWASOQDAoP020eb8JIfNLB86Xbe2za03aJMkqaqR+naU3Vr50+ZdwHWw\nm4+XXWt53gVcS2+ddwHXnuN8HnbrZ757v9EX7TjfTIB6d5LbVdXhSS5M8tAkx61p89okj0/yN1V1\nryRf6O6L1+6ouxfr0wMA9kobBqjuvqKqnpDk1EyG/E7p7jOr6oTJ5j65u99YVQ+oqo8luTTJ8Xu2\nbACA+anunncNAAC7iknkAACDBCgAgEGbmUTOLlZVByW5pI3VsmCq6vFJ/qq7vzBdvlmS47r7BfOt\nDLZOVd0iyeOS3CYzv7O7+7HzqokJc6AWyPQMyGcn+VySpyd5WZKDMulpfFR3/8Mcy4MtVVXv6+67\nrln33u6+27xqgq1WVe9I8vYk/y/JFavru/vVcyuKJHqgFs1JSZ6c5MAk/5TkmO4+fXrvwlckEaBY\nJPtWVa32rlbVvkmuP+eaYKt9V3efOO8i+E7mQC2W/br71O5+VZKLuvv0JOnus+ZcF+wJ/5DJteeO\nqqqj4o8EFtPrq+oB8y6C72QIb4FU1Rndffe1j9dbht2uqvZJckKSo6ar/jHJX3T3FVf/LNhdqurL\nSW6c5BtJLp+u7u4+YH5VkQhQC6WqrsjkQqaV5EZJvrq6KckNu/t686oNABaJAAXsKlX1t9394Kr6\nQJLv+ALr7jvPoSzYY6rqgUnuPV1c6e7Xz7MeJgQoYFepqu/t7gun9+f8Dt19znbXBHtKVT07yT2T\n/NV01XFJ3tPdvzW/qkgEqIUyHSvvXPU2453J2ZbX725nXbIwqup/rD07ab11sJtV1b8muWt3f2u6\nvG+S9+ppnT9n4S2Q7t6/uw+Y/rt/ku9N8owkFyV57nyrgy1333XWHbPtVcCed9OZxwfOrQquQo/E\nAqqqmyb5tSSPSvLXSe7Z3ZfMtyrYGlX1y0n+S5LbTv86X7V/kv87n6pgj3lWkvdW1VsyGV24d5Lf\nnG9JJIbwFsr0ti1PSvKQJC9K8rzu/uJ8q4KtVVUHJrlZJr9YZn+RfLm7PzefqmDrVVUlOTTJNzOZ\nB5Uk7+rui+ZXFasEqAVSVZcm+UySFyf58trt3f1H214U7CFVddsk53X316tqKcmdk7x09d54sAiq\n6gPdfad518F3Mgdqsfx+JuEpmQxnrP2BRfLqJFdU1e2SnJzk1pkMWcMiOaOq7rlxM7abHihgV1q9\nun5V/UaSr3X389xMmEVTVWcluV2Sc/LtCyW3s/DmzyTyBVJVv9Hd/7Oqnpf1LzD4xDmUBXvK5VV1\nXCYnS/zMdJ2r7bNojp53AaxPgFosZ07/fc9cq4DtcXySX0ryjO4+u6q+L8nL5lwTbLXf6+5Hzq6o\nqpcleeTVtGebGMIDgB1qnRvD75vkA919hzmWRfRALZSqel3WGbpb1d0P3MZyYI9wLzz2BlX1W0me\nnORGVfWlfPsOE9/I5KQJ5kwP1AKpqp+8pu3d/dbtqgX2FPfCY29SVc9y37udSYACgB1qejHNn03y\n45n0uL69u//3fKsiEaAWytUNaawytMEimbl59qwvZnISxZO6+xPbXxVsrap6QSaXMXjFdNVDkny8\nux8/v6pIBKiFcnVDGqsMbbBIqurpSc7L5OKZleShSW6b5Iwkv9zdS/OrDrbG9DpQP9TTX9ZVtU+S\nD3X3D823MkwiXyACEnuZB3b3XWaWT66q93X3iVX15LlVBVvrY0kOy+RCmsnkivsfm185rHIrlwVU\nVV+uqi9Nfy6rqiumZ3HAIvlqVT24qvaZ/jw4yWXTbbrWWRT7Jzmzqlaq6i1J/i3JAVX12qp67Zxr\n26vpgVpA3X3lfe+mExCPTXKv+VUEe8TDkzw3yQumy/+S5BFVdaMkT5hbVbC1njLvAlifOVB7CfcI\nA9idpvNbf6C73zz9A2G/7v7yvOva2+mBWkBV9XMzi/skuUe+PbQBC6GqDk3yvCQ/Nl319iS/2t3n\nza8q2FpV9bgkv5jk5pmcJHFokj9LctQ860KAWlQ/M/P4m0k+mckwHiySF2dyBt6DpsuPmK6779wq\ngq33+CRHJnlnknT3R6vqe+ZbEokAtZC6+/h51wDb4Bbd/eKZ5b+sql+bWzWwZ3y9u78xmc6aVNV+\ncZLEjuAsvAVUVYdW1Wuq6tPTn1dPhztgkVxSVY+oqn2nP49Icsm8i4It9tbpZTluVFX3TfKqJK+b\nc03EJPKFVFX/mMnQxsumqx6R5OHdbWiDhTGdWPu8JP8uk7/I35HkV7r73LkWBltoeuHM/5zkfplc\nMPZNSf6i/fKeOwFqAU0vJnjXjdbBoqmqX+vu58y7DtgqVXXjJJd19xXT5X2T3KC7vzrfyjCEt5gM\nbbC3+q/zLgC22GlJbjSzfKMkb55TLcwQoBbTY5M8OMlFSS5M8vNJTCxnb1DzLgC22A27+yurC9PH\n3zXHephyFt4Cmt4T74HzrgPmwJwEFs2lVXX37j4jSarqR5J8bc41EQFqIVXVLZI8LsltMvN/3N2P\nnVdNsFWq6stZPyhVrjrUAYvgV5O8qqouyOQYv2WSh8y3JBIBalH9fSZXZX5zkivmXAtsqdl7PcIi\nm56Bd/0kRyS5/XT1h7v78vlVxSpn4S0gZ9wBLAb3Md25TCJfTK+vqgfMuwgArrPTquo/1eqlyNkx\n9EAtoOkckRsn+XqSyzMZN+/uPmCuhQEwZOb7/IpMJo/7Pt8hBKgFM/0r5dbd/al51wIAi8oQ3oKZ\nXt7/DfOuA4DrriYeUVW/O12+dVUdOe+6EKAW1RlVdc95FwHAdfaCTO73+LDp8leSPH9+5bDKZQwW\n048meXhVnZPk0nx7zPzO8y0LgEE/2t13r6r3Jkl3f76qrj/vohCgFtXR8y4AgC1x+fQGwp1ceaHk\nb823JBJDeAtpeiuXWye5z/TxV+P/GmA3+pMkr0nyPVX1jCT/nOSZ8y2JxFl4C6mqnprkHklu390/\nWFW3SvKq7v6xOZcGwKCqOiLJUZlMxzitu8+cc0nEEN6i+tkkd0tyRpJ09wVV5fYXALtEVd0wyS8l\nuV2SDyT58+7+5nyrYpZhncX0jenlDFbHzG8853oAGPOSTEYSPpDkmCR/MN9yWEsP1GL626r68yQ3\nrarHJXlskhfOuSYANu8O3X2nJKmqU5K8a871sIYAtYC6+w+q6r5JvpTJHbyf0t3/OOeyANi8y1cf\ndPc33Qpv5zGJfMFV1UFJLmn/0QC7RlVdkcl1/JLJ5PEbZXJGtXvh7RDmQC2QqrpXVa1U1d9V1d2q\n6oNJPpjk4qq6/7zrA2Bzunvf7j5g+rN/d+8381h42gH0QC2QqnpPkicnOTDJyUmO6e7Tp6fAvqK7\n7zbXAgFgQeiBWiz7dfep3f2qJBd19+lJ0t1nzbkuAFgoAtRimb28/9fWbNPVCABbxBDeApmZdDg7\n4TDT5Rt29/XmVRsALBIBCgBgkCE8AIBBAhQAwCABCgBgkAAF7BhV9a2qeunM8r5V9Zmqeu3gfs6u\nqptf1zYAV0eAAnaSS5P8cFXdYLp83yTnXov9bObsGGfQANeaAAXsNG9M8tPTx8clecXqhqq6WVW9\npqreX1XvqKrVu9XfvKreVFUfqKoXZnLpjtXnPLyq3llVZ1TVn9a378rq7qzAtSZAATtJJ3llkuOm\nvVB3TvLOme1PS3JGd98lyW8nWR3ue2qSt3f3nZK8JslhSTK9jdFDkvz77r57Jhebffh2vBFgse03\n7wIAZnX3B6vqNpn0Pr0hV+0p+vEkPzdt95Zpz9P+Se6d5Gen699YVZ+ftj8qyd2TvHva83TDJBdt\nx/sAFpsABexEr03y+0mWkhy0Qdv15jLNDtO9pLt/e+tKAzCEB+wsq8HnRUme1t0fWrP97UkekSRV\ntZTks939lSRvy3RorqqOSXLTafvTkvx8Vd1iuu1mVXXYHn0HwF5BDxSwk3SSdPf5SU5aZ/tykhdV\n1fszOWPv0dP1T0vyiqp6aJJ3JPnUdD9nVtXvJDm1qvZJ8o0kj59udxYecK25Fx4AwCBDeAAAgwQo\nAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAb9f/7JGK/vGvBnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd08116390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results_all_bi.plot(kind='bar',color=['r','g','b'],title='Model Performance',figsize=(10,6),x='Model',legend=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## comparing negative samples accuracies\n",
    "negcom = pd.DataFrame()\n",
    "for name,model in models.items():\n",
    "    p= recall_score(Test_labels,tfprediction[name],pos_label='neg')\n",
    "    negcom.loc[name,'recall'] = p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAG6CAYAAAA23iOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUZWV97vHvA4ggYQwKCuIYJRpBCOK4vKWoDCaiRkUU\nvWqixIDR6FoixsTCIWo03gTQKIoGjUokOABOKKEcoigKKmq34gAyNgoIiCJt87t/nF1wuno6Tdfe\nu2vX97NWrT57qFNPs86qfnj3u9+dqkKSJEnd2KTvAJIkSYuJ5UuSJKlDli9JkqQOWb4kSZI6ZPmS\nJEnqkOVLkiSpQ5YvSYOR5N+T/H3fOSRpbeI6X5IWkiQXAXcBfg8sB74K/HVVXdZnLkmalCNfkhaa\nAp5YVdsAdwWuAo7b0DdN4u9DSZ3wl42khSgAVXUz8N/AAwCSvD/J6249KXllksuTXJrkL5PckuTe\nY+e+M8mnktwATCU5KMl5Sa5LcnGS14691z2a739ekp8nuTrJ4Un2SfKdJNck2eASKGn4Nus7gCTd\nXknuBBwCfG01xw4AXgY8FrgIeA+jUbNxhwIHVtU5STYHHgY8p6q+n+RPgM8nOb+qThv7nn2B+wKP\nBk4HPtP8jDsC5yf5aFV9eR7/mpIGxpEvSQvRJ5JcA/wKeBzwttWc83Tg/VW1tKpuAqZXc84nq+oc\nGI2iVdWXqur7zfb3gJOB/zN2fgGva879AnAj8JGqurqqLge+DOw1P39FSUNl+ZK0EB1cVTswGm16\nCfClJHeZc87dgEvGti+huVw5Z9+tkuyb5H+SXJXkV8DhwI5zvueqsde/BZbN2f6D9fqbSFp0LF+S\nFqLZOV9VVR8HVgCPmnPOFcCuY9u7seplx7nbHwY+AexSVdsB72bVwiZJG8TyJWlBS3IwsB2wZM6h\njwLPT7J7MzfsNRO83R8A11bV8iT7As+a++M2OLCkRc/yJWkhOj3J9UmuA14PPLeqVipfVfVZ4Fjg\nbOBH3DYp/3dred+/AV7fvO9rgP+ac3xdI2cunChpnVpdZDXJicCfAcuqao81nHMscCCjiavPq6pv\ntxZI0qKVZHfgAuCOVXVL33kkLV5tj3y9H9h/TQeTHAjcp6r+iNHE1ne1nEfSIpLkyUk2T7I98Bbg\nNIuXpL61Wr6q6ivAtWs55WDgA825Xwe2TbJTm5kkLSqHM7o78UJGjyL6m37jSFL/i6zuwsq3el/W\n7Fu2+tMlaXJVdWDfGSRpLifcS5Ikdajvka/LgLuPbe/a7FtFEu8ikiRJC0ZVrXZ5mi7KV1jz2jin\nAUcA/5XkYcCvqmqNlxzbvDNTq5qenmZ6errvGFKr/JxrMfBz3r1kzcsCtlq+knwYmAL+MMnPgdcC\nmzNamPqEqvp0koOS/JjRUhPPbzOPJElS31otX1U1d3Xo1Z1zZJsZJEmSNiZOuNcaTU1N9R1Bap2f\ncy0Gfs43Lq2ucD+fktRCySpJkha3JGuccO/IlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKH\nLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2y\nfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjy\nJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuX\nJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+S\nJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmS\nJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUodbLV5IDkixN8qMkR63m+DZJTkvy7SQX\nJHle25kkSZL6kqpq782TTYAfAfsBlwPnAs+sqqVj5xwNbFNVRyfZEfghsFNV/X7Oe1WbWSVJkuZL\nEqoqqzvW9sjXvsCFVXVxVS0HTgYOnnNOAVs3r7cGrp5bvCRJkoai7fK1C3DJ2Palzb5xxwMPSHI5\n8B3gpS1nkiRJ6s1mfQcA9gfOr6rHJrkP8Pkke1TVr/sOJmlyO++6M8suW9Z3jEVlp1124spLr+w7\nhqT11Hb5ugzYbWx712bfuOcDbwKoqp8k+RmwO/DNuW82PT196+upqSmmpqbmN62k223ZZctguu8U\ni8uyacuutLGYmZlhZmZmonPbnnC/KaMJ9PsBVwDfAA6tqiVj57wDuKqqjkmyE6PStWdVXTPnvZxw\nL23Ekli+ujYN/l6UNk5rm3Df6shXVa1IciRwJqP5ZSdW1ZIkh48O1wnAG4D/SPLd5tteObd4SZIk\nDUWrI1/zyZEvaePmyFcPph35kjZWfS41IUmSpDGWL0mSpA5ZviRJkjpk+ZIkSerQxrDIqiRJC8LO\nO9+TZcsu7jvGorLTTvfgyisv6jvGvLJ8SZI0oVHx8g7TLi1bttobBhc0LztKkiR1yPIlSZLUIcuX\nJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+S\nJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmS\nJHXI8iVJktSh9SpfSbZPskdbYSRJkoZuneUryUySbZLsAJwHvCfJ29uPJkmSNDyTjHxtW1XXA08F\nPlBVDwUe124sSZKkYZqkfG2W5K7AM4AzWs4jSZI0aJOUr9cBnwN+UlXnJrk3cGG7sSRJkoYpVdV3\nhokkqYWSVVqMksB03ykWmWnw92K3kgD+N+9WFuTnPAlVldUdm2TC/f2SnJXke832HkleM98hJUmS\nFoNJLju+BzgaWA5QVd8FntlmKEmSpKGapHzdqaq+MWff79sII0mSNHSTlK9fJrkPzUXuJE8Drmg1\nlSRJ0kBtNsE5RwAnALsnuQz4GXBYq6kkSZIGap3lq6p+CjwuyVbAJlV1Q/uxJEmShmmN5SvJy9ew\nH4Cq8hFDE7rnzjtz8bJlfcdYVO6x005cdOWVfceQJGkVaxv52rqzFAN38bJlrgrTsVh2JUkbqTWW\nr6o6pssgkiRJi8Eki6zeO8npSX6R5Kokn2weMSRJkqT1NMlSEx8GPgrcFbgbcArwkTZDSZIkDdWk\ni6x+sKp+33z9J7BF28EkSZKGaJJ1vj6T5FXAyYwWWj0E+HSSHQCq6poW80mSJA3KJOXrGc2fh8/Z\n/0xGZcz5X5IkSROaZJHVe3URRJIkaTFYZ/lKsinwROCe4+e7yKokSdL6m+Sy4+nATcAFwC3txpEk\nSRq2ScrXrlW1R+tJJEmSFoFJlpr4TJIntJ5EkiRpEZhk5Osc4ONJNgGWAwGqqrZpNZkkSdIATVK+\n3g48HLigqnw+tCRJ0gaY5LLjJcD3LF6SJEkbbpKRr58CM0k+A/xudqdLTUiSJK2/ScrXz5qvzZsv\nSZIk3U6TrHB/zIb8gCQHAP/K6BLniVX1ltWcMwX8P+AOwC+q6jEb8jMlSZI2VpOscH9n4JXAA4Et\nZvdX1WMn+N5NgOOB/YDLgXOTfLKqlo6dsy3wDuAJVXVZkh3X+28hSZK0QEwy4f5DwFLgXsAxwEXA\nuRO+/77AhVV1cVUtB04GDp5zzrOAU6vqMoCq+uWE7y1JkrTgTFK+/rCqTgSWV9UXq+oFwDpHvRq7\nMLpbctalzb5x9wN2SHJ2knOTPGfC95YkSVpwJplwv7z584okT2R0+XCHec6wN6NCtxXwtSRfq6of\nz+PPkCRJ2ihMUr7e0MzLegVwHLAN8HcTvv9lwG5j27s2+8ZdCvyyqm4CbkryJWBPYJXyNT09fevr\nqakppqamJowhSZLUnpmZGWZmZiY6N22unZpkU+CHjCbcXwF8Azi0qpaMnbM7o1J3AHBH4OvAIVX1\ngznvtWDXeU3Cwky+cDXPwOo7xqKSBKb7TrHITPs571oS8Dd6x7IgP+dJqKqs7tg653wl+eck2yS5\nQ5KzkvwiyWGT/OCqWgEcCZwJfB84uaqWJDk8yYuac5YCnwO+y+g5kifMLV6SJElDsc6RryTfrqoH\nJ3kK8GfAy4EvVdWeXQQcy+HIlybmyFf3HPnqwbSf86458tWHRTjyxW3zwp4InFJV181bMkmSpEVm\nkgn3ZyRZCvwWeHGz6OpN7caSJEkapnWOfFXVq4BHAPs0C6X+hlUXSpUkSdIEJhn5oqquGXt9I3Bj\na4kkSZIGbJI5X5IkSZonli9JkqQOTbLOV5IcluQfm+3dkuzbfjRJkqThmWTk653Aw4FDm+0bgHe0\nlkiSJGnAJplw/9Cq2jvJ+QBVdW2SzVvOJUmSNEiTjHwtb57RWADNOl+3tJpKkiRpoCYpX8cCHwfu\nkuSNwFeAf2o1lSRJ0kCt87JjVX0oybeA/Rg9Mu/JVbWk9WSSJEkDtM7yleRY4OSqcpK9JEnSBprk\nsuO3gNck+UmStyXZp+1QkiRJQzXJsx1PqqqDgIcAPwTekuTC1pNJkiQN0PqscH9fYHfgHsDSduJI\nkiQN2yQr3P9zM9L1OuB7wD5V9eetJ5MkSRqgSRZZ/Qnw8Kr6ZdthJEmShm6N5SvJ7lW1FDgX2C3J\nbuPHq+q8tsNJkiQNzdpGvl4OvAj4l9UcK+CxrSSSJEkasDWWr6p6UfPywKq6afxYki1aTSVJkjRQ\nk9zt+NUJ90mSJGkd1jbna2dgF2DLJHsxerQQwDbAnTrIJkmSNDhrm/O1P/A8YFfg7WP7bwBe3WIm\nSZKkwVrbnK+TgJOS/EVVndphJkmSpMFa5zpfVXVqkicCDwS2GNv/ujaDSZIkDdEkK9y/CzgEeAmj\neV9PZ/SIIUmSJK2nSe52fERVPRe4tqqOAR4O3K/dWJIkScM0Sfn6bfPnb5LcDVgO3LW9SJIkScM1\nybMdz0iyHfBW4DxGq9u/t9VUkiRJAzXJhPvXNy9PTXIGsEVVXdduLEmSpGFaZ/lK8tTV7LsOuKCq\nrmollSRJ0kBNctnxLxlNsj+72Z4CvgXcK8nrquqDLWWTJEkanEnK12bAH1fVMoAkOwEfAB4KfAmw\nfEmSJE1okrsd7z5bvBpXNfuuYXTnoyRJkiY0ycjXTDPR/pRm+2nNvq2AX7WWTJIkaYAmKV9HAE8F\nHtVsnwScWlUFPKatYJIkSUM0yVITleSbwHVV9YUkdwL+ALih9XSSJEkDM8mzHV8I/Dfw7mbXLsAn\n2gwlSZI0VJNMuD8CeCRwPUBVXQjcpc1QkiRJQzVJ+fpdVd08u5FkM0aPGJIkSdJ6mqR8fTHJq4Et\nkzye0V2Pp7cbS5IkaZgmKV+vAn4BXAAcDnwaeE2boSRJkoZqkrsdbwHe03xJkiRpA0zyYO1HAtPA\nPZrzw2gFinu3G02SJGl4Jllk9UTg7xg9THtFu3EkSZKGbZLydV1Vfab1JJIkSYvAJOXr7CRvBT4G\n/G52Z1Wd11oqSZKkgZqkfD20+XOfsX0FPHb+40iSJA3bJHc7+vBsSZKkeTLJOl+SJEmaJ5YvSZKk\nDq21fCXZJMkjugojSZI0dGstX83q9u/oKIskSdLgTXLZ8awkf5EkraeRJEkauEnK1+HAKcDNSa5P\nckOS61vOJUmSNEiTLDWxdRdBJEmSFoNJFlklyZOARzebM1V1RnuRJEmShmudlx2TvBl4KfCD5uul\nSd406Q9IckCSpUl+lOSotZz3kCTLkzx10veWJElaaCYZ+ToIeHBz5yNJTgLOB45e1zcm2QQ4HtgP\nuBw4N8knq2rpas57M/C59YsvSZK0sEy6yOp2Y6+3XY/33xe4sKourqrlwMnAwas57yXAfwNXrcd7\nS5IkLTiTjHy9CTg/ydlAGM39etWE778LcMnY9qWMCtmtktwNeHJVPSbJSsckSZKGZo3lK8kjq+p/\ngY8BM8BDmkNHVdWV85jhX4HxuWCuJyZJkgZrbSNfxwJ/CnytqvYGTrsd738ZsNvY9q7NvnH7ACc3\ni7juCByYZHlVrfLzpqenb309NTXF1NTU7YgkSZI0v2ZmZpiZmZno3FTV6g8k5wDfBZ7MaK7WSqrq\nb9f55smmwA8ZTbi/AvgGcGhVLVnD+e8HTq+qj63mWK0p68YuCQsz+cIVYKF+XhaqJDDdd4pFZtrP\neddG4wT+N+9WFuTnPAlVtdqreWsb+foz4HHA/sC3bs8PrqoVSY4EzmQ0uf/EqlqS5PDR4Tph7rfc\nnp8jSZK0UKxx5OvWE5I9q+o7HeVZWw5HvjQxR76658hXD6b9nHfNka8+DG/ka51LTWwMxUuSJGko\nJl3nS5IkSfPA8iVJktShta3z9fK1fWNVvX3+40iSJA3b2u523LqzFJIkSYvEGstXVR3TZRBJkqTF\nYG2XHY9d2zdOssiqJEmSVra2y463a2FVSZIkrdnaLjue1GUQSZKkxWBtI18AJLkzcBTwAGCL2f1V\n9dgWc0mSJA3SJOt8fQhYAtwLOAa4CDi3xUySJEmDNUn5+sOqOhFYXlVfrKoXAI56SZIk3Q7rvOwI\nLG/+vCLJE4HLgR3aiyRJkjRck5SvNyTZFngFcBywDfB3raaSJEkaqHWWr6o6o3l5HfCYduNIkiQN\n2zrnfCU5Kcl2Y9vbJ3lfu7EkSZKGaZIJ93tU1a9mN6rqWmCv9iJJkiQN1yTla5Mk289uJNmByeaK\nSZIkaY5JStS/AF9Lckqz/XTgje1FkiRJGq5JJtx/IMk3uW1tr6dW1Q/ajSVJkjRMk1x2hNG6XjdW\n1fHAL5Lcq8VMkiRJgzXJ3Y6vZfRsx6ObXXcA/rPNUJIkSUM1ycjXU4AnATcCVNXlwNZthpIkSRqq\nScrXzVVVQAEk2ardSJIkScM1Sfn6aJJ3A9sleSHwBeC97caSJEkapknudnxbkscD1wP3B/6xqj7f\nejJJkqQBmmix1KZsfR4gySZJnl1VH2o1mSRJ0gCt8bJjkm2SHJ3k+CRPyMiRwE+BZ3QXUZIkaTjW\nNvL1QeBa4GvAXwGvBgI8uaq+3UE2SZKkwVlb+bp3VT0IIMl7gSuA3arqpk6SSZIkDdDa7nZcPvui\nqlYAl1q8JEmSNszaRr72THJ98zrAls12gKqqbVpPJ0mSNDBrLF9VtWmXQSRJkhaDSR+sLUmSpHlg\n+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ6pDl\nS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5Yv\nSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjrUevlKckCSpUl+lOSo1Rx/VpLvNF9f\nSfKgtjNJkiT1pdXylWQT4Hhgf+CBwKFJdp9z2k+BR1fVnsAbgPe0mUmSJKlPbY987QtcWFUXV9Vy\n4GTg4PETquqcqrqu2TwH2KXlTJIkSb1pu3ztAlwytn0pay9XfwV8ptVEkiRJPdqs7wCzkjwGeD7w\nqL6zSJIktaXt8nUZsNvY9q7NvpUk2QM4ATigqq5d05tNT0/f+npqaoqpqan5yilJknS7zczMMDMz\nM9G5qarWgiTZFPghsB9wBfAN4NCqWjJ2zm7AWcBzquqctbxXtZm1TUlYmMkXrgAL9fOyUCWB6b5T\nLDLTfs67lgT8jd6xLMjPeRKqKqs71urIV1WtSHIkcCaj+WUnVtWSJIePDtcJwD8AOwDvzOhTvbyq\n9m0zlyRJUl9aHfmaT458aX048tU9R756MO3nvGuOfPVheCNfrnAvSZLUIcuXJElShyxfkiRJHbJ8\nSZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIl\nSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ck\nSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5Ik\nSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIk\ndcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLU\nIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKH\nWi9fSQ5IsjTJj5IctYZzjk1yYZJvJ3lw25kkSZL60mr5SrIJcDywP/BA4NAku88550DgPlX1R8Dh\nwLvazKTJzfQdQOrCz/oOIHVhpu8AGtP2yNe+wIVVdXFVLQdOBg6ec87BwAcAqurrwLZJdmo5lyYw\n03cAqQsX9R1A6sJM3wE0pu3ytQtwydj2pc2+tZ1z2WrOkSRJGgQn3EuSJHVos5bf/zJgt7HtXZt9\nc8+5+zrOASDJvIbr0kJNfkzfATbAQv68LFjTfQe4nb7Yd4Dbz895Hxbqf/OF+xt9aJ/ztsvXucB9\nk9wDuAJ4JnDonHNOA44A/ivJw4BfVdWyuW9UVcP6Ly9JkhalVstXVa1IciRwJqNLnCdW1ZIkh48O\n1wlV9ekkByX5MXAj8Pw2M0mSJPUpVdV3BkmSpEXDCfeSJEkdsnxJkiR1qO0J91rAkuwIXF1em9bA\nJDkC+FBV/arZ3h44tKre2W8yaf4kuTPwQuCejP17X1Uv6CuTRpzzJQCaO03fDFwDvB74ILAjo9HR\n51bVZ3uMJ82rJN+uqgfP2Xd+Ve3VVyZpviX5KvBl4FvAitn9VXVqb6EEOPKl2xwPvBrYFvgf4MCq\nOqd5FudHAMuXhmTTJJkd1U2yKbB5z5mk+Xanqjqq7xBalXO+NGuzqjqzqk4BrqyqcwCqamnPuaQ2\nfJbR2oL7JdkP/wdDw3RGkoP6DqFVedlRACQ5r6r2nvt6ddvSQpdkE+BwYL9m1+eB91bVijV/l7Sw\nJLkB2Aq4GVje7K6q2qa/VALLlxpJVjBa5DbAlsBvZg8BW1TVHfrKJknSkFi+JC0aST5aVc9IcgGw\nyi+/qtqjh1hSa5I8CXh0szlTVWf0mUcjli9Ji0aSu1bVFc3zZldRVRd3nUlqS5I3Aw8BPtTsOhT4\nZlUd3V8qgeVLjWZuQDG6zDirGN0Ru3lVeWesBiPJW+beBba6fdJCluS7wIOr6pZme1PgfEd4++fd\njgKgqrauqm2aP7cG7gq8EbgS+Ld+00nz7vGr2Xdg5ymk9m039nrb3lJoJY5maCVJtgNeBjwX+DDw\nkKq6ut9U0vxI8mLgb4D7NKMCs7YG/refVFJr3gScn+RsRlc1Hg28qt9IAi87qtE8SugVwCHA+4Dj\nquq6flNJ8yvJtsD2jP5RGv9H6IaquqafVNL8SxJgV+D3jOZ9AXyjqq7sL5VmWb4EQJIbgV8A7wdu\nmHu8qt7eeSipJUnuA1xaVb9LMgXsAXxg9lmP0hAkuaCqHtR3Dq3KOV+a9VZGxQtGl2DmfklDciqw\nIsl9gROAuzO6zC4NyXlJHrLu09Q1R74kLTqzT21I8krgt1V1nA/W1tAkWQrcF7iY2xbRLu927J8T\n7gVAkldW1T8nOY7VLz75tz3EktqyPMmhjG4s+fNmn09x0NDs33cArZ7lS7OWNH9+s9cUUjeeD/w1\n8Maq+lmSewEf7DmTNN/eUFXPGd+R5IPAc9ZwvjriZUdJkgZo9vL62PamwAVV9YAeYwlHvtRIcjqr\nudw4q6qe1GEcqRU+21GLQZKjgVcDWya5ntueXHIzoxtM1DNHvgRAkv+ztuNV9cWuskht8dmOWkyS\nvMnnOG6cLF+SJA1Qs9DqU4BHMRrp/XJVfaLfVALLlxprugwzy8sxGpKxB8mPu47RDSevqKqfdp9K\nml9J3sloqYmPNLsOAX5SVUf0l0pg+VJjTZdhZnk5RkOS5PXApYwWVg3wTOA+wHnAi6tqqr900vxo\n1vn642r+oU+yCfD9qvrjfpPJCfcCLFdadJ5UVXuObZ+Q5NtVdVSSV/eWSppfPwZ2Y7TIKoye5PDj\n/uJolo8X0kqS3JDk+ubrpiQrmrtlpCH5TZJnJNmk+XoGcFNzzMsBGoqtgSVJZpKcDfwA2CbJaUlO\n6znboubIl1ZSVbc+x7GZrHkw8LD+EkmteDbwb8A7m+2vAYcl2RI4srdU0vz6x74DaPWc86V18pl3\nkrQwNfN5/6iqvtD8z8VmVXVD37kWO0e+tJIkTx3b3ATYh9sux0iDkGRX4Djgkc2uLwMvrapL+0sl\nza8kLwReBOzA6IaSXYF3Afv1mUuWL63qz8de/x64iNGlR2lI3s/oTsenN9uHNfse31siaf4dAewL\nfB2gqi5Mcpd+IwksX5qjqp7fdwapA3euqvePbf9Hkpf1lkZqx++q6ubR9F1IshneULJR8G5HrSTJ\nrkk+nuSq5uvU5hKNNCRXJzksyabN12HA1X2HkubZF5ulU7ZM8njgFOD0njMJJ9xrjiSfZ3Q55oPN\nrsOAZ1eVl2M0GM0k5OOAhzMaCfgq8JKquqTXYNI8ahZV/UvgCYwWE/4c8N7yH/7eWb60kmahyQev\na580NEleVlX/2ncOab4k2Qq4qapWNNubAnesqt/0m0xedtRcXo7RYvXyvgNI8+wsYMux7S2BL/SU\nRWMsX5rrBcAzgCuBK4CnAU7C12KQvgNI82yLqvr17Ebz+k495lHDux21kuYZj0/qO4fUA+dgaGhu\nTLJ3VZ0HkORPgd/2nElYvjRHkjsDLwTuydjno6pe0Fcmab4kuYHVl6yw8uUZaQheCpyS5HJGn/Gd\ngUP6jSSwfGlVn2S02vcXgBU9Z5Hm1fizS6Uha+503BzYHbh/s/uHVbW8v1Sa5d2OWol3NkrSMPhc\n3o2XE+6ThoqyAAAEd0lEQVQ11xlJDuo7hCRpg52V5C8yu8S9NhqOfGklzZyYrYDfAcsZzROoqtqm\n12CSpPUy9vt8BaOJ9v4+30hYvnSr5v+O7l5VP+87iyRJQ+VlR92qeeTEp/rOIUnacBk5LMk/NNt3\nT7Jv37lk+dKqzkvykL5DSJI22DsZPb/0Wc32r4F39BdHs1xqQnM9FHh2kouBG7ltjsAe/caSJK2n\nh1bV3knOB6iqa5Ns3ncoWb60qv37DiBJmhfLm4dpF9y6iPYt/UYSeNlRczSPF7o78Njm9W/wcyJJ\nC9GxwMeBuyR5I/AV4J/6jSTwbkfNkeS1wD7A/avqfknuBpxSVY/sOZokaT0l2R3Yj9EUkrOqaknP\nkYSXHbWqpwB7AecBVNXlSXwkiyQtEEm2AP4auC9wAfDuqvp9v6k0zstJmuvmZsmJ2TkCW/WcR5K0\nfk5idAXjAuBA4G39xtFcjnxpro8meTewXZIXAi8A3tNzJknS5B5QVQ8CSHIi8I2e82gOy5dWUlVv\nS/J44Hrg/sA/VtXne44lSZrc8tkXVfV7H+248XHCvdYoyY7A1eWHRJIWjCQrGK3TCKOJ9lsyunPd\nZztuJJzzJQCSPCzJTJKPJdkryfeA7wHLkhzQdz5J0mSqatOq2qb52rqqNht7bfHaCDjyJQCSfBN4\nNbAtcAJwYFWd09ym/JGq2qvXgJIkDYQjX5q1WVWdWVWnAFdW1TkAVbW051ySJA2K5Uuzxh858ds5\nxxwelSRpnnjZUcBKEzTHJ2fSbG9RVXfoK5skSUNi+ZIkSeqQlx0lSZI6ZPmSJEnqkOVLkiSpQ5Yv\nSQtakluSfGBse9Mkv0hy2nq+z8+S7LCh50jSuli+JC10NwJ/kuSOzfbjgUtux/tMcveRdyhJ2mCW\nL0lD8Gngic3rQ4GPzB5Isn2Sjyf5TpKvJnlQs3+HJJ9LckGS9zBaVmX2e56d5OtJzkvy77ntycRp\njt8pyRlJzk/y3SRP7+RvKWkQLF+SFroCTgYObUa/9gC+Pnb8GOC8qtoT+Htg9hLla4EvV9WDgI8D\nuwE0j9Q6BHhEVe3NaAHiZ8/5mQcAl1XVXlW1B/DZVv5mkgZps74DSNKGqqrvJbkno1GvTzE2igU8\nCnhqc97ZzYjX1sCjgac0+z+d5Nrm/P2AvYFzmxGvLYAr5/zIC4C3JXkT8Kmq+korfzFJg2T5kjQU\npwFvBaaAHddx7urmbo1fWjypqv5+jd9cdWGSvYGDgDck+UJVvWH9I0tajLzsKGmhmy1N7wOOqarv\nzzn+ZeAwgCRTwC+r6tfAl2guJyY5ENiuOf8s4GlJ7twc2z7Jbiv9wOSuwG+r6sOMCt/e8/2XkjRc\njnxJWugKoKouA45fzfFp4H1JvsPozsj/2+w/BvhIkmcCXwV+3rzPkiSvAc5MsglwM3BEc3x2xOxB\nwFuT3NIcf3ELfy9JA+WzHSVJkjrkZUdJkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+JEmS\nOmT5kiRJ6pDlS5IkqUP/H7te4qlbUDBHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd08121940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pl = negcom.plot(kind='bar',color=['r','g','b','y'],title='Bigram',figsize=(10,6),legend=False)\n",
    "pl.set(xlabel='Models',ylabel='Recall for negative samples')\n",
    "lim = plt.ylim([0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### c. Using Trigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "## using countvectorizer to generate count vectors for the training data.\n",
    "vec_all = CountVectorizer(ngram_range=(1,3))\n",
    "ctr_features_all = vec_all.fit_transform(Data_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf_vec_all = TfidfTransformer()\n",
    "tr_features_all = tf_vec_all.fit_transform(ctr_features_all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cte_features_all = vec_all.transform(Data_test)\n",
    "te_features_all = tf_vec_all.transform(cte_features_all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   TF-IDF Accuracy        Model\n",
      "0         0.870041  BernoulliNB\n",
      "1         0.932456     Logistic\n",
      "2         0.919227   Perceptron\n"
     ]
    }
   ],
   "source": [
    "results_all_tri = pd.DataFrame()\n",
    "\n",
    "#stratified = StratifiedKFold(labels, n_folds=5,random_state=20160121, shuffle=True)\n",
    "tfprediction = {}\n",
    "cprediction = {}\n",
    "foldnum = 0\n",
    "for name,model in models.items():\n",
    "        \n",
    "        model.fit(tr_features_all, Train_labels)\n",
    "        tfprediction[name] = model.predict(te_features_all)\n",
    "        tfaccuracy = metrics.accuracy_score(tfprediction[name],Test_labels)\n",
    "        \n",
    "        #model.fit(ctr_features_all,Train_labels)\n",
    "        #cprediction[name] = model.predict(cte_features_all)\n",
    "        #caccuracy = metrics.accuracy_score(cprediction[name],Test_labels)\n",
    "        \n",
    "        results_all_tri.loc[foldnum,'TF-IDF Accuracy']=tfaccuracy\n",
    "        #results_all_tri.loc[foldnum,'Count Accuracy']=caccuracy\n",
    "        results_all_tri.loc[foldnum,'Model']=name\n",
    "        foldnum = foldnum+1\n",
    "print (results_all_tri)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1cd2d54ee48>"
      ]
     },
     "execution_count": 217,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAG6CAYAAADH1Xg8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH0RJREFUeJzt3XuUbGdZJ+DfmwQEIYkgGCQhQUGJIlchMqNiaxaG4EhG\nR4FwNYwaHRB1XGMUb80owow3lIgajQiooAw6gjoSRRpwNFwmgIgJ1xByPUi4yFVCeOePqg6VppPu\nL+nTu7vO86zV69Te+6tdb9XZq/rX3/ftvau7AwDA9h02dQEAAPuNAAUAMEiAAgAYJEABAAwSoAAA\nBglQAACDBChgWFWdUFWfqaotv0Oq6glV9ZpdqutWVfWyqvpQVf3RbrwmcGgSoGDJVdV7quqTVXX7\nDevfOA9Bx9/EXY9cRG7TtgtB7F/nP++uqrNuYj1J8h1J7pjkdt39yJuxH4AbJUDB8uskFyc5fX1F\nVX1VkltnLAQdLJ3k6O4+Ksmjk/x0VX3z6E7mvWEnJHl734QrBFfV4aPPAQ5dAhQcGl6Q5AkLy09I\n8rzFBlV1VFU9v6reV1UXV9VPLGw7rKp+sar+paremeRbNnnu71TVFVV1aVX9bFXVQH2VJN19fpK3\nJvmq+X5PrKrzqurqqrqwqr5z4TWfW1XPqaq/qKqPJHlVkp9O8qh5b9YZNfOT8164q6rq96rqqPnz\n13u/nlhVlyR5xcK676qq985f98yqekBVvbmqPlBVz16o4Uur6hVV9f755/b76/ufb7+4qn5k/twP\nVtULq+qWC9tPm/cEfriq3rEeHHfg8wQOMgEKDg3nJzmyqu4x76l5ZJLfzzy4zJ2d5Mgkd02ykuTx\nVXXGfNv3JnlYkvskeUBmQ2WLnpfkU0m+NMn9kjwkyXcP1FdJUlVfm+Qrk1xQVZ+f5Lx5nXdI8qgk\nz6mqExeed3qSn+3uI5OcnOTnk7you4/q7ucmOSPJ45N8w7y2I+fvc9GDk5yY5JSFdScluXtmn9Oz\nkjw1yTdlFuweUVVfv1D3zye5U5KvSHJcktUN+//OJN+c5Esy+/y+a/5eT8rsc/uR7j56Xsd75s+5\nuZ8ncJAJUHDoWO+FekiSC5Ncsb5hIVT9WHd/vLsvSfJLSR43b/KdSZ7V3Vd094eSPGPhucckOTXJ\nD3f3J7v7/ZmFjuuGDLdQSf6lqq5Ock6Ss7p7Lcl/SHJxdz+/Z96c5CXzWtb92bzXKt39qU32/egk\nv9zdl3T3x5P8eGY9VOvffZ3kZ7r7E939bwvr/nt3f6q7/ybJx5K8sLuv7u4rkrwms1CT7n5Xd7+i\nuz/d3Vcn+ZXMwtqiX+3uA/PP7WVJ7jtf/8Qk53b33873dWV3v72qvuhmfp7ALjhi6gKAXfP7SV6d\nWU/I8zdsu0Nm3wfvXVh3SZJj54/vnOTSDdvWHZ/kFkmunI8y1fxncV83ppN84Sbzlk5I8qCq+sB8\nuZIcvqH2S3Pj7ryh1ksye5/HLKy7bJPnvW/h8SeSHNiwfNskmYedX03y9fN1hyf5QK5v8bkfT/LF\n88d3SfIXm7z2Cbl5nyewCwQoOER093ur6uLMejeeuGHz+5Nck9kv74vm605Icvn88ZWZ/cLPwrZ1\nlyb5ZDYPQdtV+dwJ7ZcmWevuUzZpv26r17si16/1hMze54F89v3cnIn0P5/kM0nu2d0frqrTkjx7\ni+esuzTJ3W5g/c39PIGDzBAeHFqemOSbuvsTiyu7+zNJ/jjJ06vqtlV1QpIfzmzYL/NtT6mqY6vq\ndknOWnjuVZnNVfqVqjpyPnH7S6vqwdus6YYmR/95ki+vqsdW1RFVdYv5ZO57bPvdJi9M8sNVddeq\num2Sp2c2R+ozN/LaI5O1j0zy0SQfqapjk/y3geeem+SMqvrG+Wd256q6xw58nsAuEKBg+V3Xi9Hd\nF3f3BZttS/KUzIaY3p3ZUN/vzydiJ8lvJ3l5kjcneUNmc5EWPT7JLZP8c2ZDWC/ObGL1UH3XW9n9\n0cwmXz8qs56kK5I8M8nnbXO/SfK7mYXAVyd5V2bv7ylbvPbGdTe2/LQkX51kfX7Txs/lBnuQuvv1\nmU1yf1aSDydZy2w4NLl5nyewC2qrHuKqOjezyZwHuvveN9Dm1zIbFvhYku/q7jftdKEAAHvFdnqg\nnpvrn957PVV1apK7dfeXJTkzyW/uUG0AAHvSlgGqu/8uyQdvpMlpmZ8V092vTXL0/LRmAICltBNz\noI7N9U8lvjyfPfUZAGDpmEQOADBoJ64DdXmuf32Y4/LZa8dcT1W5pgkAsG9096aXNtlugFq/Eu5m\nXprkSUn+qKoelORD3X3gBtrGdeF21+rqalZXV6cuAw4qxzmHAsf57ruxe3hvGaCq6g8zu7HoF1bV\ne5P8TGbXJ+nuPqe7/7KqHlazO7R/LLPrmgAALK0tA1R3P3obbZ68M+UAAOx9JpEvuZWVlalLgIPO\ncc6hwHG+t2x5JfIdfbEq98YEAPaFqrrBSeR6oAAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiA\nAgAYJEABAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABh0xdQHA\n3nGn4+6UA5cfmLqMQ8oxxx6Tqy67auoygEHV3bv3YlW9m68HjKmqZHXqKg4xq4nvRdibqirdXZtt\nM4QHADBIgAIAGGQOFACHlDvd6a45cOCSqcs4pBxzzAm56qr3TF3GjhKgADikzMKTeWe76cCBTacR\n7WuG8AAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABglQAACDBCgAgEEC\nFADAIAEKAGCQAAUAMEiAAgAYdMTUBewXd73TnXLJgQNTl3FIOeGYY/Keq66augwA+BwC1DZdcuBA\neuoiDjElsAKwRxnCAwAYJEABAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAYJEAB\nAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMGhbAaqqHlpVF1XV26vqrE22H1VVL62qN1XV\nW6rqu3a8UgCAPWLLAFVVhyU5O8kpSe6Z5PSqOnFDsycleWt33zfJNyb5pao6YqeLBQDYC7bTA3VS\nknd09yXdfU2SFyU5bUObTnLk/PGRSa7u7k/vXJkAAHvHdgLUsUkuXVi+bL5u0dlJvrKqrkjy5iQ/\nuDPlAQDsPTs1ifyUJG/s7jsnuV+SX6+q2+7QvgEA9pTtzFO6PMnxC8vHzdctOiPJM5Kku99VVRcn\nOTHJGzbubHV19brHKysrWVlZGSoYAOBgWFtby9ra2rbaVnffeIOqw5O8LcnJSa5M8rokp3f3hQtt\nfj3J+7r7aVV1TGbB6T7d/YEN++qtXm+vqqrsz8r3r0qyX4+X/aqqktWpqzjErDrOd1tVJb7Rd1nt\ny+O8qtLdtdm2LXuguvvaqnpykvMyG/I7t7svrKozZ5v7nCQ/l+T3quof50/70Y3hCQBgWWzZA7Wj\nL6YHigF6oHafHqgJrDrOd5seqCksXw+UK5EDAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQAAUA\nMEiAAgAYJEABAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABglQ\nAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQ\nAAUAMEiAAgAYJEABAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAA\nBglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABglQAACDBCgAgEECFADAIAEK\nAGCQAAUAMEiAAgAYJEABAAwSoAAABm0rQFXVQ6vqoqp6e1WddQNtVqrqjVX1T1X1yp0tEwBg7zhi\nqwZVdViSs5OcnOSKJK+vqj/r7osW2hyd5NeTfHN3X15VdzhYBQMATG07PVAnJXlHd1/S3dckeVGS\n0za0eXSSl3T35UnS3e/f2TIBAPaO7QSoY5NcurB82Xzdoi9PcvuqemVVvb6qHrdTBQIA7DVbDuEN\n7Of+Sb4pyW2S/ENV/UN3v3OH9g8AsGdsJ0BdnuT4heXj5usWXZbk/d39ySSfrKpXJ7lPks8JUKur\nq9c9XllZycrKyljFAAAHwdraWtbW1rbVtrr7xhtUHZ7kbZlNIr8yyeuSnN7dFy60OTHJs5M8NMnn\nJXltkkd29z9v2Fdv9Xp7VVVlf1a+f1WS/Xq87FdVlaxOXcUhZtVxvtuqKvGNvstqXx7nVZXurs22\nbdkD1d3XVtWTk5yX2Zypc7v7wqo6c7a5z+nui6rq5Un+Mcm1Sc7ZGJ4AAJbFlj1QO/pieqAYoAdq\n9+mBmsCq43y36YGawvL1QLkSOQDAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABglQAACDBCgAgEEC\nFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAY\nJEABAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABglQAACDBCgA\ngEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiA\nAgAYJEABAAwSoAAABglQAACDBCgAgEECFADAIAEKAGCQAAUAMEiAAgAYJEABAAwSoAAABglQAACD\nBCgAgEECFADAIAEKAGDQtgJUVT20qi6qqrdX1Vk30u6BVXVNVX37zpUIALC3bBmgquqwJGcnOSXJ\nPZOcXlUn3kC7ZyZ5+U4XCQCwl2ynB+qkJO/o7ku6+5okL0py2ibtfiDJ/0ryvh2sDwBgz9lOgDo2\nyaULy5fN112nqu6c5D92928kqZ0rDwBg79mpSeTPSrI4N0qIAgCW1hHbaHN5kuMXlo+br1v0gCQv\nqqpKcockp1bVNd390o07W11dve7xyspKVlZWBksGANh5a2trWVtb21bb6u4bb1B1eJK3JTk5yZVJ\nXpfk9O6+8AbaPzfJy7r7TzbZ1lu93l5VVdmfle9flWS/Hi/7VVUlq1NXcYhZdZzvttnf+j7z3VX7\n8jivqnT3pqNqW/ZAdfe1VfXkJOdlNuR3bndfWFVnzjb3ORufcrMrBgDYw7bsgdrRF9MDxQA9ULtP\nD9QEVh3nu00P1BSWrwfKlcgBAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAA\nAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCAB\nCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAM\nEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQA\nwCABCgBgkAAFADBIgAIAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADBIgAIAGCRA\nAQAMEqAAAAYJUAAAg7YVoKrqoVV1UVW9varO2mT7o6vqzfOfv6uqe+18qQAAe8OWAaqqDktydpJT\nktwzyelVdeKGZu9O8uDuvk+Sn0vy2ztdKADAXrGdHqiTkryjuy/p7muSvCjJaYsNuvv87v7wfPH8\nJMfubJkAAHvHdgLUsUkuXVi+LDcekL47yf+5OUUBAOxlR+zkzqrqG5OckeTrdnK/AAB7yXYC1OVJ\njl9YPm6+7nqq6t5Jzkny0O7+4A3tbHV19brHKysrWVlZ2WapAAAHz9raWtbW1rbVtrr7xhtUHZ7k\nbUlOTnJlktclOb27L1xoc3ySVyR5XHeffyP76q1eb6+qquzPyvevSrJfj5f9qqqS1amrOMSsOs53\nW1UlvtF3We3L47yq0t212bYte6C6+9qqenKS8zKbM3Vud19YVWfONvc5SX4qye2TPKdmR+Y13X3S\nzr0FAIC9Y8seqB19MT1QDNADtfv0QE1g1XG+2/RATWH5eqBciRwAYJAABQAwSIACABgkQAEADBKg\nAAAGCVAAAIMEKACAQQIUAMAgAQoAYJAABQAwSIACABgkQAEADBKgAAAGCVAAAIMEKACAQQIUAMAg\nAQoAYJAABQAwSIACABgkQAEADBKgAAAGCVAAAIMEKACAQQIUAMAgAQoAYJAABQAwSIACABgkQAEA\nDBKgAAAGCVAAAIMEKACAQQIUAMAgAQoAYJAABQAwSIACABgkQAEADBKgAAAGCVAAAIMEKACAQQIU\nAMAgAQoAYJAABQAwSIACABgkQAEADBKgAAAGCVAAAIMEKACAQQIUAMAgAQoAYJAABQAwSIACABgk\nQAEADBKgAAAGCVAAAIMEKACAQQIUAMAgAQoAYJAABQAwaFsBqqoeWlUXVdXbq+qsG2jza1X1jqp6\nU1Xdd2fLBADYO7YMUFV1WJKzk5yS5J5JTq+qEze0OTXJ3br7y5KcmeQ3D0Kt3ARrUxcAu+HiqQuA\n3bA2dQEs2E4P1ElJ3tHdl3T3NUlelOS0DW1OS/L8JOnu1yY5uqqO2dFKuUnWpi4AdsN7pi4AdsPa\n1AWwYDsB6tgkly4sXzZfd2NtLt+kDQDAUjCJHABg0BHbaHN5kuMXlo+br9vY5i5btEmSVNVIfXvK\nfq38aVMXcDPs5+Nl31qduoCb6FVTF3DTOc6nsF8/8/37jb5sx/l2AtTrk9y9qk5IcmWSRyU5fUOb\nlyZ5UpI/qqoHJflQdx/YuKPuXq5PDwA4JG0ZoLr72qp6cpLzMhvyO7e7L6yqM2eb+5zu/suqelhV\nvTPJx5KccXDLBgCYTnX31DUAAOwrJpEDAAwSoAAABm1nEjn7WFXdIcnVbayWJVNVT0ryB939ofny\n7ZKc3t3PmbYy2DlVdcck35Pkrln4nd3dT5yqJmbMgVoi8zMgn5nkA0l+NskLktwhs57Gx3f3X01Y\nHuyoqnpTd993w7o3dvf9pqoJdlpV/X2S1yT5f0muXV/f3S+ZrCiS6IFaNmcneWqSo5P8bZJTu/v8\n+b0LX5hEgGKZHF5Vtd67WlWHJ7nlxDXBTvv87j5r6iL4XOZALZcjuvu87n5xkqu6+/wk6e6LJq4L\nDoa/yuzacydX1cnxRwLL6c+r6mFTF8HnMoS3RKrqgu6+/8bHmy3DfldVhyU5M8nJ81V/neR3uvva\nG34W7C9V9ZEkt0nyqSTXzFd3dx81XVUkAtRSqaprM7uQaSW5dZKPr29KcqvuvsVUtQHAMhGggH2l\nqv64ux9RVW9J8jlfYN197wnKgoOmqh6e5MHzxbXu/vMp62FGgAL2lar64u6+cn5/zs/R3Zfsdk1w\nsFTVM5M8MMkfzFednuQN3f3j01VFIkAtlflYeef6txnvzM62vGV3O+uSpVFV/2Pj2UmbrYP9rKr+\nMcl9u/sz8+XDk7xRT+v0nIW3RLr7yO4+av7vkUm+OMnTk1yV5FenrQ523EM2WXfqrlcBB98XLDw+\nerIquB49Ekuoqr4gyQ8leXySP0zywO6+etqqYGdU1fcn+S9J7jb/63zdkUn+7zRVwUHzjCRvrKpX\nZja68OAkPzZtSSSG8JbK/LYtP5LkkUl+N8mzu/vD01YFO6uqjk5yu8x+sSz+IvlId39gmqpg51VV\nJTkuyaczmweVJK/r7qumq4p1AtQSqaqPJfmXJM9N8pGN27v7l3e9KDhIqupuSS7r7n+rqpUk907y\n/PV748EyqKq3dPe9pq6Dz2UO1HL5hczCUzIbztj4A8vkJUmuraq7JzknyV0yG7KGZXJBVT1w62bs\nNj1QwL60fnX9qvrRJJ/o7me7mTDLpqouSnL3JJfksxdKbmfhTc8k8iVSVT/a3f+zqp6dzS8w+JQJ\nyoKD5ZqqOj2zkyW+db7O1fZZNqdMXQCbE6CWy4Xzf98waRWwO85I8n1Jnt7dF1fVlyR5wcQ1wU77\nue5+3OKKqnpBksfdQHt2iSE8ANijNrkx/OFJ3tLdXzlhWUQP1FKpqpdlk6G7dd398F0sBw4K98Lj\nUFBVP57kqUluXVX/ms/eYeJTmZ00wcT0QC2RqvqGG9ve3a/arVrgYHEvPA4lVfUM973bmwQoANij\n5hfT/LYkX5dZj+truvt/T1sViQC1VG5oSGOdoQ2WycLNsxd9OLOTKH6ku9+9+1XBzqqq52R2GYMX\nzlc9Msm7uvtJ01VFIkAtlRsa0lhnaINlUlU/m+SyzC6eWUkeleRuSS5I8v3dvTJddbAz5teB+oqe\n/7KuqsOSvLW7v2LayjCJfIkISBxiHt7d91lYPqeq3tTdZ1XVUyerCnbWO5Mcn9mFNJPZFfffOV05\nrHMrlyVUVR+pqn+d/3yyqq6dn8UBy+TjVfWIqjps/vOIJJ+cb9O1zrI4MsmFVbVWVa9M8s9Jjqqq\nl1bVSyeu7ZCmB2oJdfd1972bT0A8LcmDpqsIDorHJPnVJM+ZL/9DksdW1a2TPHmyqmBn/fTUBbA5\nc6AOEe4RBrA/zee3fll3/838D4QjuvsjU9d1qNMDtYSq6tsXFg9L8oB8dmgDlkJVHZfk2Um+dr7q\nNUl+sLsvm64q2FlV9T1JvjfJ7TM7SeK4JL+Z5OQp60KAWlbfuvD400nek9kwHiyT52Z2Bt53zpcf\nO1/3kMkqgp33pCQnJXltknT3O6rqi6YtiUSAWkrdfcbUNcAuuGN3P3dh+feq6ocmqwYOjn/r7k/N\nprMmVXVEnCSxJzgLbwlV1XFV9adV9b75z0vmwx2wTK6uqsdW1eHzn8cmuXrqomCHvWp+WY5bV9VD\nkrw4ycsmromYRL6UquqvMxvaeMF81WOTPKa7DW2wNOYTa5+d5N9l9hf53yf5ge6+dNLCYAfNL5z5\nn5N8c2YXjH15kt9pv7wnJ0AtofnFBO+71TpYNlX1Q939rKnrgJ1SVbdJ8snuvna+fHiSz+vuj09b\nGYbwlpOhDQ5V/3XqAmCHvSLJrReWb53kbyaqhQUC1HJ6YpJHJLkqyZVJviOJieUcCmrqAmCH3aq7\nP7q+MH/8+RPWw5yz8JbQ/J54D5+6DpiAOQksm49V1f27+4IkqaqvTvKJiWsiAtRSqqo7JvmeJHfN\nwv9xdz9xqppgp1TVR7J5UKpcf6gDlsEPJnlxVV2R2TF+pySPnLYkEgFqWf1ZZldl/psk105cC+yo\nxXs9wjKbn4F3yyQnJrnHfPXbuvua6apinbPwlpAz7gCWg/uY7l0mkS+nP6+qh01dBAA32yuq6j/V\n+qXI2TP0QC2h+RyR2yT5tyTXZDZu3t191KSFATBk4fv82swmj/s+3yMEqCUz/yvlLt393qlrAYBl\nZQhvycwv7/8XU9cBwM1XM4+tqp+aL9+lqk6aui4EqGV1QVU9cOoiALjZnpPZ/R4fPV/+aJJfn64c\n1rmMwXL6miSPqapLknwsnx0zv/e0ZQEw6Gu6+/5V9cYk6e4PVtUtpy4KAWpZnTJ1AQDsiGvmNxDu\n5LoLJX9m2pJIDOEtpfmtXO6S5Jvmjz8e/9cA+9GvJfnTJF9UVU9P8ndJfn7akkichbeUqupnkjwg\nyT26+8ur6s5JXtzdXztxaQAMqqoTk5yc2XSMV3T3hROXRAzhLatvS3K/JBckSXdfUVVufwGwT1TV\nrZJ8X5K7J3lLkt/q7k9PWxWLDOssp0/NL2ewPmZ+m4nrAWDM8zIbSXhLklOT/OK05bCRHqjl9MdV\n9VtJvqCqvifJE5P89sQ1AbB9X9nd90qSqjo3yesmrocNBKgl1N2/WFUPSfKvmd3B+6e7+68nLguA\n7btm/UF3f9qt8PYek8iXXFXdIcnV7T8aYN+oqmszu45fMps8fuvMzqh2L7w9whyoJVJVD6qqtar6\nk6q6X1X9U5J/SnKgqh46dX0AbE93H97dR81/juzuIxYeC097gB6oJVJVb0jy1CRHJzknyandff78\nFNgXdvf9Ji0QAJaEHqjlckR3n9fdL05yVXefnyTdfdHEdQHAUhGglsvi5f0/sWGbrkYA2CGG8JbI\nwqTDxQmHmS/fqrtvMVVtALBMBCgAgEGG8AAABglQAACDBCgAgEECFLBnVNVnqur5C8uHV9W/VNVL\nB/dzcVXd/ua2AbghAhSwl3wsyVdV1efNlx+S5NKbsJ/tnB3jDBrgJhOggL3mL5N8y/zx6UleuL6h\nqm5XVX9aVW+uqr+vqvW71d++ql5eVW+pqt/O7NId6895TFW9tqouqKrfqM/eldXdWYGbTIAC9pJO\n8qIkp897oe6d5LUL25+W5ILuvk+Sn0iyPtz3M0le0933SvKnSY5PkvltjB6Z5N939/0zu9jsY3bj\njQDL7YipCwBY1N3/VFV3zaz36S9y/Z6ir0vy7fN2r5z3PB2Z5MFJvm2+/i+r6oPz9icnuX+S1897\nnm6V5KrdeB/AchOggL3opUl+IclKkjts0XazuUyLw3TP6+6f2LnSAAzhAXvLevD53SRP6+63btj+\nmiSPTZKqWkny/u7+aJJXZz40V1WnJvmCeftXJPmOqrrjfNvtqur4g/oOgEOCHihgL+kk6e7Lk5y9\nyfbVJL9bVW/O7Iy9J8zXPy3JC6vqUUn+Psl75/u5sKp+Msl5VXVYkk8ledJ8u7PwgJvMvfAAAAYZ\nwgMAGCRAAQAMEqAAAAYJUAAAgwQoAIBBAhQAwCABCgBgkAAFADDo/wNGHxivynEDMAAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd2d54e4e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results_all_tri.plot(kind='bar',color=['r','g','b'],title='Model Performance',figsize=(10,6),x='Model',legend=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for  BernoulliNB\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.83      0.51      0.63     31169\n",
      "        pos       0.88      0.97      0.92    110945\n",
      "\n",
      "avg / total       0.87      0.87      0.86    142114\n",
      "\n",
      "\n",
      "\n",
      "Classification report for  Logistic\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.87      0.82      0.84     31169\n",
      "        pos       0.95      0.96      0.96    110945\n",
      "\n",
      "avg / total       0.93      0.93      0.93    142114\n",
      "\n",
      "\n",
      "\n",
      "Classification report for  Perceptron\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.83      0.80      0.81     31169\n",
      "        pos       0.94      0.95      0.95    110945\n",
      "\n",
      "avg / total       0.92      0.92      0.92    142114\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for name,model in models.items():\n",
    "    print (\"Classification report for \",name)\n",
    "    print(metrics.classification_report(Test_labels, tfprediction[name]))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## comparing negative samples accuracies\n",
    "negcom = pd.DataFrame()\n",
    "for name,model in models.items():\n",
    "    p= recall_score(Test_labels,tfprediction[name],pos_label='neg')\n",
    "    negcom.loc[name,'recall'] = p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAG6CAYAAAA23iOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUZWV97vHv040GJIxRQUEco6gXEKI4LlOKCmgUNQZF\nMTeSKPFCYtS1nGJM4RA1DtcBjaJocCQSNCpxQLmUQ5RBQUVtDE4IzagiEMa2/d0/zi44Xd1dfbq7\n9t5du76ftWr12fvsc+qpXmdVP/3ud787VYUkSZK6sazvAJIkSUuJ5UuSJKlDli9JkqQOWb4kSZI6\nZPmSJEnqkOVLkiSpQ5YvSYOT5JVJ3t13Dklal7jOl6QtWZJrgdlfVNsCNwGrm31HVtXH+8omSZvC\n8iVp0UjyU+Avq+r0eY5ZXlWrN+N7bNbrJWlDPO0oaTFJ83XrjuQ1SU5M8rEkVwPPavZ9cOyY5yS5\nMMkVSV6e5KIkj5zn9Q9J8s0kVyVZmeTtSZY3xy9P8rskf53kx0muTvKqJPdqXvObJB+dPV6S5rJ8\nSRqCJwMfqaodgE80+wogyV7A24FDgd2AOwC7zPP6fwNWAX8L7Aw8HDgQOHLOax4D7N08//fAu5rv\ncVdgv+axJK3F8iVpCL5eVZ8DqKob5zz3NOBTVXVmVa0CXsmc0bM5r7+pqr5dVWfXyM+B9wF/POc1\nb6iq66vq+8AK4AtVdVFVXQ18Edh3QX9CSYOxVd8BJGkBXDTPc3cef76qrk9y1XyvT3If4C3AHwG3\nA5YDZ855zRVjj28ALp+zvcNEySUtOY58SRqC+a4cuhTYfXYjybbATht4/XuB84B7NKci/5G1R8sk\naZNYviQN3UnAk5Psn+Q2wKuZv6wBbAdcXVU3JLkva8/3kqRNZvmStJhs9No4VXUe8ELg34GVwJXA\nrxitF7Y+Lwb+Isk1wL8AJ24gh2v2SJpYq+t8JTke+BPg8qraez3HvAM4GLgO+Iuq+k5rgSQteUm2\nA34D7FFVK/vOI2npaXvk64OMLtFepyQHA/esqj9kNKz/npbzSFqCkjwxyTZJfh94K/Bti5ekvrRa\nvqrq68Dcq4rGHQJ8qDn2TGCHJHPX35GkzfUU4BLgF8AewGH9xpG0lPW91MRurHmJ98pm3+XrPlyS\nNl5VHQEc0XcOSQIn3EuSJHWq75GvlcBdxrZ3b/atJYlXE0mSpEWjqta5PmAX5WutG+GO+QxwFPBv\nSR4C/Kaq1nvKsc0rM7W26elppqen+44htcrPuZYCP+fdS9a/LnOr5SvJx4Ap4A+S/ILRKtG3Baqq\njquqzyV5fJIfM1pq4jlt5pEkSepbq+Wrqp45wTFHt5lBkiRpS+KEe63X1NRU3xGk1vk511Lg53zL\n0uoK9wspSS2WrJIkaWlLst4J9458SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQh\ny5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocs\nX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8\nSZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIl\nSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ck\nSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5Ik\nSR2yfEmSJHXI8iVJktQhy5ckSVKHtuo7gKRh2HX3Xbl85eV9x1hSdtltFy67+LK+Y0jaSKmqdr9B\nchDwNkajbMdX1RvnPL898BFgD2A58Jaq+td1vE+1nVXSpksC032nWGKmwd+L0pYpCVWVdT3X6mnH\nJMuAY4EDgfsDhyXZc85hRwE/qKoHAI8C3pLEETlJkjRIbc/52h+4oKourKpVwInAIXOOKWC75vF2\nwK+q6rct55IkSepF2+VrN+Cise2Lm33jjgXul+QS4LvAC1rOJEmS1Jst4WrHA4Fzq+rOwL7Au5L8\nfs+ZJEmSWtH23KqVjCbSz9q92TfuOcDrAarqJ0l+BuwJfGvum01PT9/yeGpqiqmpqYVNK0mStAlm\nZmaYmZmZ6NhWr3ZMshz4EXAAcClwFnBYVa0YO+ZdwBVVdUySXRiVrn2q6tdz3surHaUtmFc79mDa\nqx2lLdV8Vzu2OvJVVauTHA2cyq1LTaxIcuTo6ToOeC3wr0m+17zsJXOLlyRJW4Jdd70bl19+Yd8x\nlpRddrkrl132875jLKjW1/laKI58SVs2R756MO3IV9eSMLpIX93Jovyc97bOlyRJktZk+ZIkSeqQ\n5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUOW\nL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+\nJEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9JkqQObVT5SrJTkr3b\nCiNJkjR0GyxfSWaSbJ9kZ+Ac4H1J3tp+NEmSpOGZZORrh6q6Bngq8KGqejDwmHZjSZIkDdMk5Wur\nJHcCDgVOaTmPJEnSoE1Svl4NfBH4SVWdneQewAXtxpIkSRqmVFXfGSaSpBZLVmkpSgLTfadYYqbB\n34vdSgL4d96tLMrPeRKqKut6bpIJ9/dOclqS7zfbeyd55UKHlCRJWgomOe34PuDlwCqAqvoe8Iw2\nQ0mSJA3VJOXrdlV11px9v20jjCRJ0tBNUr5+meSeNCe5kzwNuLTVVJIkSQO11QTHHAUcB+yZZCXw\nM+DwVlNJkiQN1AbLV1X9FHhMkm2BZVV1bfuxJEmShmm95SvJi9azH4Cq8hZDkiRJG2m+ka/tOksh\nSZK0RKy3fFXVMV0GkSRJWgomWWT1Hkk+m+TKJFck+XRziyFJkiRtpEmWmvgY8AngTsCdgZOAj7cZ\nSpIkaagmXWT1w1X12+brI8DWbQeTJEkaoknW+fp8kpcBJzJaaPXpwOeS7AxQVb9uMZ8kSdKgTFK+\nDm3+PHLO/mcwKmPO/5IkSZrQJIus3r2LIJIkSUvBBstXkuXAE4C7jR/vIquSJEkbb5LTjp8FbgTO\nA37XbhxJkqRhm6R87V5Ve7eeRJIkaQmYZKmJzyd5XOtJJEmSloBJRr7OAD6VZBmwCghQVbV9q8kk\nSZIGaJLy9VbgocB5VVUt55EkSRq0SU47XgR83+IlSZK0+SYZ+fopMJPk88BNsztdakKSJGnjTVK+\nftZ83bb5kiRJ0iaaZIX7YzbnGyQ5CHgbo1Ocx1fVG9dxzBTwf4HbAFdW1aM253tKkiRtqSZZ4f4O\nwEuA+wNbz+6vqkdP8NplwLHAAcAlwNlJPl1V548dswPwLuBxVbUyye03+qeQJElaJCaZcP9R4Hzg\n7sAxwM+Bsyd8//2BC6rqwqpaBZwIHDLnmGcCJ1fVSoCq+uWE7y1JkrToTFK+/qCqjgdWVdVXquoI\nYIOjXo3dGF0tOeviZt+4ewM7Jzk9ydlJnj3he0uSJC06k0y4X9X8eWmSJzA6fbjzAmfYj1Gh2xb4\nZpJvVtWPF/B7SJIkbREmKV+vbeZlvRh4J7A98MIJ338lsMfY9u7NvnEXA7+sqhuBG5N8FdgHWKt8\nTU9P3/J4amqKqampCWNIkiS1Z2ZmhpmZmYmOTZtrpyZZDvyI0YT7S4GzgMOqasXYMXsyKnUHAb8H\nnAk8vap+OOe9XOdV2oIlgem+Uywx0+DvxW4lAfw771YW5ec8CVWVdT23wTlfSf45yfZJbpPktCRX\nJjl8km9cVauBo4FTgR8AJ1bViiRHJnlec8z5wBeB7zG6j+Rxc4uXJEnSUGxw5CvJd6rqAUmeAvwJ\n8CLgq1W1TxcBx3I48iVtwRz56sG0I19dc+SrD0tw5Itb54U9ATipqq5esGSSJElLzCQT7k9Jcj5w\nA/D8ZtHVG9uNJUmSNEwbHPmqqpcBDwMe2CyUej1rL5QqSZKkCUwy8kVV/Xrs8XXAda0lkiRJGrBJ\n5nxJkiRpgVi+JEmSOjTJOl9JcniSVzXbeyTZv/1okiRJwzPJyNe7gYcChzXb1wLvai2RJEnSgE0y\n4f7BVbVfknMBquqqJLdtOZckSdIgTTLytaq5R2MBNOt8/a7VVJIkSQM1Sfl6B/Ap4I5JXgd8Hfin\nVlNJkiQN1AZPO1bVR5N8GzgACPDkqlrRejJJkqQB2mD5SvIO4MSqcpK9JEnSZprktOO3gVcm+UmS\nNyd5YNuhJEmShmqSezueUFWPBx4E/Ah4Y5ILWk8mSZI0QBuzwv29gD2BuwLntxNHkiRp2CZZ4f6f\nm5GuVwPfBx5YVU9sPZkkSdIATbLI6k+Ah1bVL9sOI0mSNHTrLV9J9qyq84GzgT2S7DH+fFWd03Y4\nSZKkoZlv5OtFwPOAt6zjuQIe3UoiSZKkAVtv+aqq5zUPD66qG8efS7J1q6kkSZIGapKrHb8x4T5J\nkiRtwHxzvnYFdgO2SbIvo1sLAWwP3K6DbJIkSYMz35yvA4G/AHYH3jq2/1rgFS1mkiRJGqz55nyd\nAJyQ5E+r6uQOM0mSJA3WBtf5qqqTkzwBuD+w9dj+V7cZbEjutuuuXHj55X3HWFLuussu/Pyyy/qO\nIUnSWjZYvpK8h9Ecr0cB7weeBpzVcq5BufDyy6m+QywxsexKkrZQk1zt+LCq+nPgqqo6BngocO92\nY0mSJA3TJOXrhubP65PcGVgF3Km9SJIkScM1yb0dT0myI/Am4BxGq9u/v9VUkiRJAzXJhPvXNA9P\nTnIKsHVVXd1uLEmSpGGaZML9U9ex72rgvKq6opVUkiRJAzXJace/ZDTJ/vRmewr4NnD3JK+uqg+3\nlE2SJGlwJilfWwH3rarLAZLsAnwIeDDwVcDyJUmSNKFJrna8y2zxalzR7Ps1oysfJUmSNKFJRr5m\nmon2JzXbT2v2bQv8prVkkiRJAzRJ+ToKeCrwiGb7BODkqipGq95LkiRpQpMsNVFJvgVcXVVfTnI7\n4PeBa1tPJ0mSNDAbnPOV5LnAvwPvbXbtBvxHm6EkSZKGapIJ90cBDweuAaiqC4A7thlKkiRpqCYp\nXzdV1c2zG0m2YnSLIUmSJG2kScrXV5K8AtgmyWMZXfX42XZjSZIkDdMk5etlwJXAecCRwOeAV7YZ\nSpIkaagmudrxd8D7mi9JkiRthklurP1wYBq4a3N8GK1AcY92o0mSJA3PJIusHg+8kNHNtFe3G0eS\nJGnYJilfV1fV51tPIkmStARMUr5OT/Im4JPATbM7q+qc1lJJkiQN1CTl68HNnw8c21fAoxc+jiRJ\n0rBNcrWjN8+WJElaIJOs8yVJkqQFYvmSJEnq0LzlK8myJA/rKowkSdLQzVu+mtXt39VRFkmSpMGb\n5LTjaUn+NElaTyNJkjRwk5SvI4GTgJuTXJPk2iTXtJxLkiRpkCZZamK7LoJIkiQtBZMsskqSJwGP\nbDZnquqU9iJJkiQN1wZPOyZ5A/AC4IfN1wuSvH7Sb5DkoCTnJ/nvJC+d57gHJVmV5KmTvrckSdJi\nM8nI1+OBBzRXPpLkBOBc4OUbemGSZcCxwAHAJcDZST5dVeev47g3AF/cuPiSJEmLy6SLrO449niH\njXj//YELqurCqloFnAgcso7j/gb4d+CKjXhvSZKkRWeSka/XA+cmOR0Io7lfL5vw/XcDLhrbvphR\nIbtFkjsDT66qRyVZ4zlJkqShWW/5SvLwqvov4JPADPCg5qmXVtVlC5jhbcD4XDDXE5MkSYM138jX\nO4A/Ar5ZVfsBn9mE918J7DG2vXuzb9wDgRObRVxvDxycZFVVrfX9pqenb3k8NTXF1NTUJkSSJEla\nWDMzM8zMzEx0bKpq3U8kZwDfA57MaK7WGqrqbzf45sly4EeMJtxfCpwFHFZVK9Zz/AeBz1bVJ9fx\nXK0v65YuCYsz+eIVYLF+XharJDDdd4olZtrPeddG4wT+nXcri/JznoSqWufZvPlGvv4EeAxwIPDt\nTfnGVbU6ydHAqYwm9x9fVSuSHDl6uo6b+5JN+T6SJEmLxXpHvm45INmnqr7bUZ75cjjypYk58tU9\nR756MO3nvGuOfPVheCNfG1xqYksoXpIkSUMx6TpfkiRJWgCWL0mSpA7Nt87Xi+Z7YVW9deHjSJIk\nDdt8Vztu11kKSZKkJWK95auqjukyiCRJ0lIw32nHd8z3wkkWWZUkSdKa5jvtuEkLq0qSJGn95jvt\neEKXQSRJkpaC+Ua+AEhyB+ClwP2ArWf3V9WjW8wlSZI0SJOs8/VRYAVwd+AY4OfA2S1mkiRJGqxJ\nytcfVNXxwKqq+kpVHQE46iVJkrQJNnjaEVjV/HlpkicAlwA7txdJkiRpuCYpX69NsgPwYuCdwPbA\nC1tNJUmSNFAbLF9VdUrz8GrgUe3GkSRJGrYNzvlKckKSHce2d0rygXZjSZIkDdMkE+73rqrfzG5U\n1VXAvu1FkiRJGq5JyteyJDvNbiTZmcnmikmSJGmOSUrUW4BvJjmp2f4z4HXtRZIkSRquSSbcfyjJ\nt7h1ba+nVtUP240lSZI0TJOcdoTRul7XVdWxwJVJ7t5iJkmSpMGa5GrHf2R0b8eXN7tuA3ykzVCS\nJElDNcnI11OAJwHXAVTVJcB2bYaSJEkaqknK181VVUABJNm23UiSJEnDNUn5+kSS9wI7Jnku8GXg\n/e3GkiRJGqZJrnZ8c5LHAtcA9wFeVVVfaj2ZJEnSAE20WGpTtr4EkGRZkmdV1UdbTSZJkjRA6z3t\nmGT7JC9PcmySx2XkaOCnwKHdRZQkSRqO+Ua+PgxcBXwT+CvgFUCAJ1fVdzrIJkmSNDjzla97VNVe\nAEneD1wK7FFVN3aSTJIkaYDmu9px1eyDqloNXGzxkiRJ2jzzjXztk+Sa5nGAbZrtAFVV27eeTpIk\naWDWW76qanmXQSRJkpaCSW+sLUmSpAVg+ZIkSeqQ5UuSJKlDli9JkqQOWb4kSZI6ZPmSJEnqkOVL\nkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJkjpk+ZIkSeqQ5UuSJKlDli9J\nkqQOWb4kSZI6ZPmSJEnqkOVLkiSpQ5YvSZKkDlm+JEmSOmT5kiRJ6pDlS5IkqUOWL0mSpA5ZviRJ\nkjrUevlKclCS85P8d5KXruP5Zyb5bvP19SR7tZ1JkiSpL62WryTLgGOBA4H7A4cl2XPOYT8FHllV\n+wCvBd7XZiZJkqQ+tT3ytT9wQVVdWFWrgBOBQ8YPqKozqurqZvMMYLeWM0mSJPWm7fK1G3DR2PbF\nzF+u/gr4fKuJJEmSerRV3wFmJXkU8BzgEX1nkSRJakvb5WslsMfY9u7NvjUk2Rs4Djioqq5a35tN\nT0/f8nhqaoqpqamFyilJkrTJZmZmmJmZmejYVFVrQZIsB34EHABcCpwFHFZVK8aO2QM4DXh2VZ0x\nz3tVm1nblITFmXzxCrBYPy+LVRKY7jvFEjPt57xrScDf6B3LovycJ6Gqsq7nWh35qqrVSY4GTmU0\nv+z4qlqR5MjR03Uc8A/AzsC7M/pUr6qq/dvMJUmS1JdWR74WkiNf2hiOfHXPka8eTPs575ojX30Y\n3siXK9xLkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmS\nJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS\n1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElS\nhyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkd\nsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHLlyRJUocsX5IkSR2yfEmSJHXI\n8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR1yPIlSZLUIcuXJElShyxfkiRJHbJ8SZIkdcjyJUmS1CHL\nlyRJUocsX5IkSR2yfEmSJHXI8iVJktSh1stXkoOSnJ/kv5O8dD3HvCPJBUm+k+QBbWeSJEnqS6vl\nK8ky4FjgQOD+wGFJ9pxzzMHAPavqD4Ejgfe0mUmTm+k7gNSFn/UdQOrCTN8BNKbtka/9gQuq6sKq\nWgWcCBwy55hDgA8BVNWZwA5Jdmk5lyYw03cAqQs/7zuA1IWZvgNoTNvlazfgorHti5t98x2zch3H\nSJIkDYIT7iVJkjq0VcvvvxLYY2x792bf3GPusoFjAEiyoOG6tFiTH9N3gM2wmD8vi9Z03wE20Vf6\nDrDp/Jz3YbH+nS/e3+hD+5y3Xb7OBu6V5K7ApcAzgMPmHPMZ4Cjg35I8BPhNVV0+942qalh/85Ik\naUlqtXxV1eokRwOnMjrFeXxVrUhy5OjpOq6qPpfk8Ul+DFwHPKfNTJIkSX1KVfWdQZIkaclwwr0k\nSVKHLF+SJEkdanvCvRaxJLcHflWem9bAJDkK+GhV/abZ3gk4rKre3W8yaeEkuQPwXOBujP17X1VH\n9JVJI875EgDNlaZvAH4NvAb4MHB7RqOjf15VX+gxnrSgknynqh4wZ9+5VbVvX5mkhZbkG8DXgG8D\nq2f3V9XJvYUS4MiXbnUs8ApgB+D/AQdX1RnNvTg/Dli+NCTLk2R2VDfJcuC2PWeSFtrtquqlfYfQ\n2pzzpVlbVdWpVXUScFlVnQFQVef3nEtqwxcYrS14QJID8D8YGqZTkjy+7xBam6cdBUCSc6pqv7mP\n17UtLXZJlgFHAgc0u74EvL+qVq//VdLikuRaYFvgZmBVs7uqavv+UgksX2okWc1okdsA2wDXzz4F\nbF1Vt+krmyRJQ2L5krRkJPlEVR2a5DxgrV9+VbV3D7Gk1iR5EvDIZnOmqk7pM49GLF+Slowkd6qq\nS5v7za6lqi7sOpPUliRvAB4EfLTZdRjwrap6eX+pBJYvNZq5AcXoNOOsYnRF7G2ryitjNRhJ3jj3\nKrB17ZMWsyTfAx5QVb9rtpcD5zrC2z+vdhQAVbVdVW3f/LkdcCfgdcBlwNv7TSctuMeuY9/BnaeQ\n2rfj2OMdekuhNTiaoTUk2RH4O+DPgY8BD6qqX/WbSloYSZ4P/B/gns2owKztgP/qJ5XUmtcD5yY5\nndFZjUcCL+s3ksDTjmo0txJ6MfB04APAO6vq6n5TSQsryQ7AToz+URr/R+jaqvp1P6mkhZckwO7A\nbxnN+wI4q6ou6y+VZlm+BECS64ArgQ8C1859vqre2nkoqSVJ7glcXFU3JZkC9gY+NHuvR2kIkpxX\nVXv1nUNrc86XZr2JUfGC0SmYuV/SkJwMrE5yL+A44C6MTrNLQ3JOkgdt+DB1zZEvSUvO7F0bkrwE\nuKGq3umNtTU0Sc4H7gVcyK2LaJdXO/bPCfcCIMlLquqfk7yTdS8++bc9xJLasirJYYwuLHlis8+7\nOGhoDuw7gNbN8qVZK5o/v9VrCqkbzwH+GnhdVf0syd2BD/ecSVpor62qZ4/vSPJh4NnrOV4d8bSj\nJEkDNHt6fWx7OXBeVd2vx1jCkS81knyWdZxunFVVT+owjtQK7+2opSDJy4FXANskuYZb71xyM6ML\nTNQzR74EQJI/nu/5qvpKV1mktnhvRy0lSV7vfRy3TJYvSZIGqFlo9SnAIxiN9H6tqv6j31QCy5ca\n6zsNM8vTMRqSsRvJj7ua0QUnL66qn3afSlpYSd7NaKmJjze7ng78pKqO6i+VwPKlxvpOw8zydIyG\nJMlrgIsZLawa4BnAPYFzgOdX1VR/6aSF0azzdd9q/qFPsgz4QVXdt99kcsK9AMuVlpwnVdU+Y9vH\nJflOVb00ySt6SyUtrB8DezBaZBVGd3L4cX9xNMvbC2kNSa5Nck3zdWOS1c3VMtKQXJ/k0CTLmq9D\ngRub5zwdoKHYDliRZCbJ6cAPge2TfCbJZ3rOtqQ58qU1VNUt93FsJmseAjykv0RSK54FvB14d7P9\nTeDwJNsAR/eWSlpYr+o7gNbNOV/aIO95J0mLUzOf9w+r6svNfy62qqpr+8611DnypTUkeerY5jLg\ngdx6OkYahCS7A+8EHt7s+hrwgqq6uL9U0sJK8lzgecDOjC4o2R14D3BAn7lk+dLanjj2+LfAzxmd\nepSG5IOMrnT8s2b78GbfY3tLJC28o4D9gTMBquqCJHfsN5LA8qU5quo5fWeQOnCHqvrg2Pa/Jvm7\n3tJI7bipqm4eTd+FJFvhBSVbBK921BqS7J7kU0muaL5Obk7RSEPyqySHJ1nefB0O/KrvUNIC+0qz\ndMo2SR4LnAR8tudMwgn3miPJlxidjvlws+tw4FlV5ekYDUYzCfmdwEMZjQR8A/ibqrqo12DSAmoW\nVf1L4HGMFhP+IvD+8h/+3lm+tIZmockHbGifNDRJ/q6q3tZ3DmmhJNkWuLGqVjfby4Hfq6rr+00m\nTztqLk/HaKl6Ud8BpAV2GrDN2PY2wJd7yqIxli/NdQRwKHAZcCnwNMBJ+FoK0ncAaYFtXVX/M7vR\nPL5dj3nU8GpHraG5x+OT+s4h9cA5GBqa65LsV1XnACT5I+CGnjMJy5fmSHIH4LnA3Rj7fFTVEX1l\nkhZKkmtZd8kKa56ekYbgBcBJSS5h9BnfFXh6v5EEli+t7dOMVvv+MrC65yzSghq/d6k0ZM2VjrcF\n9gTu0+z+UVWt6i+VZnm1o9bglY2SNAzel3fL5YR7zXVKksf3HUKStNlOS/KnmV3iXlsMR760hmZO\nzLbATcAqRvMEqqq27zWYJGmjjP0+X81oor2/z7cQli/dovnf0V2q6hd9Z5Ekaag87ahbNLec+M++\nc0iSNl9rwoRRAAAEH0lEQVRGDk/yD832XZLs33cuWb60tnOSPKjvEJKkzfZuRvcvfWaz/T/Au/qL\no1kuNaG5Hgw8K8mFwHXcOkdg735jSZI20oOrar8k5wJU1VVJbtt3KFm+tLYD+w4gSVoQq5qbaRfc\nsoj27/qNJPC0o+Zobi90F+DRzePr8XMiSYvRO4BPAXdM8jrg68A/9RtJ4NWOmiPJPwIPBO5TVfdO\ncmfgpKp6eM/RJEkbKcmewAGMppCcVlUreo4kPO2otT0F2Bc4B6CqLkniLVkkaZFIsjXw18C9gPOA\n91bVb/tNpXGeTtJcNzdLTszOEdi25zySpI1zAqMzGOcBBwNv7jeO5nLkS3N9Isl7gR2TPBc4Anhf\nz5kkSZO7X1XtBZDkeOCsnvNoDsuX1lBVb07yWOAa4D7Aq6rqSz3HkiRNbtXsg6r6rbd23PI44V7r\nleT2wK/KD4kkLRpJVjNapxFGE+23YXTluvd23EI450sAJHlIkpkkn0yyb5LvA98HLk9yUN/5JEmT\nqarlVbV987VdVW019tjitQVw5EsAJPkW8ApgB+A44OCqOqO5TPnjVbVvrwElSRoIR740a6uqOrWq\nTgIuq6ozAKrq/J5zSZI0KJYvzRq/5cQNc55zeFSSpAXiaUcBa0zQHJ+cSbO9dVXdpq9skiQNieVL\nkiSpQ552lCRJ6pDlS5IkqUOWL0mSpA5ZviQtakl+l+RDY9vLk1yZ5DMb+T4/S7Lz5h4jSRti+ZK0\n2F0H/K8kv9dsPxa4aBPeZ5Krj7xCSdJms3xJGoLPAU9oHh8GfHz2iSQ7JflUku8m+UaSvZr9Oyf5\nYpLzkryP0bIqs695VpIzk5yT5F9y652J0zx/uySnJDk3yfeS/FknP6WkQbB8SVrsCjgROKwZ/dob\nOHPs+WOAc6pqH+DvgdlTlP8IfK2q9gI+BewB0NxS6+nAw6pqP0YLED9rzvc8CFhZVftW1d7AF1r5\nySQN0lZ9B5CkzVVV309yN0ajXv/J2CgW8Ajgqc1xpzcjXtsBjwSe0uz/XJKrmuMPAPYDzm5GvLYG\nLpvzLc8D3pzk9cB/VtXXW/nBJA2S5UvSUHwGeBMwBdx+A8eua+7W+KnFE6rq79f74qoLkuwHPB54\nbZIvV9VrNz6ypKXI046SFrvZ0vQB4Jiq+sGc578GHA6QZAr4ZVX9D/BVmtOJSQ4GdmyOPw14WpI7\nNM/tlGSPNb5hcifghqr6GKPCt99C/1CShsuRL0mLXQFU1Urg2HU8Pw18IMl3GV0Z+b+b/ccAH0/y\nDOAbwC+a91mR5JXAqUmWATcDRzXPz46Y7QW8Kcnvmuef38LPJWmgvLejJElShzztKEmS1CHLlyRJ\nUocsX5IkSR2yfEmSJHXI8iVJktQhy5ckSVKHLF+SJEkdsnxJkiR16P8DFsvGJ/LA5X0AAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd2d3ad0f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pl = negcom.plot(kind='bar',color=['r','g','b','y'],title='Trigram',figsize=(10,6),legend=False)\n",
    "pl.set(xlabel='Models',ylabel='Recall for negative samples')\n",
    "lim = plt.ylim([0,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## this code is taken from: https://www.kaggle.com/gpayen/d/snap/amazon-fine-food-reviews/building-a-prediction-model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWIAAAEpCAYAAACpwyjVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+8FVW9//HX++AP1ID8TYFQKhiaiqhopmKZmFlqfsO0\nblDSzVL7obe6Upbmj0j6lqSm3ZIQTEOyW2qRv7L8UQgoEiqkhKGCgiWKloUIn/vHrIPD4ZzNPvuc\nzcxhv5895nH2XrPWzBrIz/mwZs0aRQRmZlacpqI7YGbW6ByIzcwK5kBsZlYwB2Izs4I5EJuZFcyB\n2MysYA7Etg5J3SXdIulFSTd04DgfkXRrZ/atKJIOlTS/6H7YpkueR9w1SfoIcBbwNuAlYA7wzYj4\nQweP+x/AmcA7ogH+zyFpDbB7RDxRdF+scTkj7oIknQ18F7gI2AnoB3wf+EAnHL4/8HgjBOGk4nVK\n6raxOmINLCK8daEN6Am8DJxYoc4WwHhgCbAYuBTYPO0bBjwNnA0sS3VGpX3nAyuBV8my7E8A5wHX\n5o7dH1gDNKXvHwcWpvoLgVNS+Sjg3ly7Q4CZwAvADLKMu3nf74ALgPvScW4Ftmvj2pr7/6Vc/48H\njgEeA/4OjMnVPxD4YzrvEuByYLO07+50Lf9I5x2RO/6XgWeBSc1lqc2uwPPA4PT9zcBzwOFF/3/D\nW9fdnBF3Pe8AtgR+WaHOucBQYB9g3/T53Nz+3kAPsiDySeBKSb0i4nzgm8CUiOgZERNT/ZZZYwBI\n2hr4HnB0RPQkC7ZzWqm3LfArsl8O25P9Yvh1Km92Clnw3jFd3xcrXF9vsl82byb7RfEj4KPAfsDh\nwNck9U91VwNfALYj+7N7N3A6QEQMS3X2Ttf7s9zx30j2L41P5a8lsiGMLwM/kbQVMBGYGBH3VOiv\nWUUOxF3P9sDfI2JNhTofAb4REc9HxPPAN4CP5fa/ClwYEasj4jdkGeEeNfZnNbC3pO4RsSwiWrup\ndSzZcMf1EbEmIqYAf2bdoZSJEbEwIlYCU4HBFc75Ktl4+GpgCrADMD4iXomIecA8sl9ARMTsiJgZ\nmaeAH5JluHlq5ZrOi4hVqT/riIgJwF/IMvudWfeXnFm7ORB3Pc8DO0iq9Hf3ZuCp3PcnU9naY7QI\n5K8Ab2hvRyLiFeDDwGeAZ9Nsi9YC+ptTH/KeBPrkvi9tR3+ej4jmLP1f6edzuf3/am4vaUDq17OS\nXgQuJgvclfwtIlZtoM7VwF7A5VXUNavIgbjrmU42jntChTpLyMZym/UHnqnxfP8Ets59f1N+Z0Tc\nERHDyf45/xhZxtnSM8BbWpT1S/2st6uA+cBuEfFG4KusnwG3tKEbeNuQDbNMAM6X9MbO6Kg1Lgfi\nLiYiXiIbF/2+pOMlbSVpM0nHSPpWqjYFOFfSDpJ2AL4GXFvjKecAh0vaRVIv4JzmHZJ2knRcGite\nRTbE0dqQyTRggKSTJXWT9GFgEHBLjX1qjx7ASxHxiqS3kWXveUvJbsC1x2XAzIj4FNm1/U/Hu2mN\nzIG4C4qI75LNejiX7J/kT5HdgGq+gXcR8AAwF/hT+nxxpUNWONedwA3pWLNYN3g2pX4sIZutcDjr\nBzoiYjnwfrIbcH9PP4+NiBc2dP4qtXozMfki8FFJL5EFzCkt6p4PTJa0XNKHNnQiSccBw0k3/Miu\nfz9Jp9TScTPwAx1mZoVzRmxmVjAHYjOzgjkQm5kVzIHYzKxoRT9jXctGdlfcmzdvJdo69b/xzXu0\n59yL2ogTE8jWI5mbK9sWuJ1szvttQK9U3p/sQaLZabsy12YI2ayhx8me4Gwu34JsFs4Csvn9/XL7\nRqX6jwEjN3S9XXLWhKSYvWhF0d2oqx9cOpZPnzWm6G7U3YDe7X6gr8v55oXf4CtfO6/obtRVj+7d\niIgNPShTNUnRfb/PVlX33w9d3uq5JR1KNrd9ckTsk8ouIXsyc5yk/wa2jYhz0toktzTXa3GcGcCZ\nETFL0jTgexFxm6TPkK1TcnqaG//BiDg5raHyAFkAF/AgMCQi2gxaHpows3KSqtvaEBH3ka26l3c8\n2Yp6pJ/5J1RbC+a9gR4RMSsVTc61yR/rRrIFpQCOBm6PiBUR8SJZBv7eSpfqQGxm5aSm6rb22Ski\nlgFExFKy9bybvUXSbEm/S9k0ZOuhLM7VWczra6T0IVsylcgWoFohabt8ebKEdddVWc9m7b0K2zgO\nOPjQDVeyLuGww4cV3YWuqY1sd/VLT7Pm5cWt7qtB89jss2RjvC9IGgL8UtKe7TxWzUMzDsQldcA7\nDiu6C9ZJDht2RNFd6JrayHa79epPt179135f/ez97TnqMkk7R8SyNOzwHEBEvEq2vCoRMVvSQmAg\nWTa7S659X15frKp53zPpTS49I2K5pCXAES3a/K5Spzw0YWbl1MEx4uajsG6mejPZW2Ugm9lwU3Yq\nrV1aVtKuwO7AE2n4YoWkoZIEjGxuk441Kn0eAdyVPt8GHCWpV7pxd1Qqa5MzYjMrp6aOvS5Q0vVk\nmen2kp4iW7XwW8DPJJ1Ktib2San64cAFkl4lW0HwtHSjDeAM4BqgOzAtIprfTj4BuFbSArJ1wk8G\nSMMbF5LNnAiylzQ0H6v1vnr6mhWpEaavNYK6TF97R3XTN/89fWynnrsIzojNrJw2POywyXAgNrNy\nav/UtC7LgdjMyskZsZlZwZwRm5kVzIHYzKxgTR6aMDMrVgfnEXclDsRmVk4emjAzK5hnTZiZFcwZ\nsZlZwZwRm5kVzBmxmVnBnBGbmRXMGbGZWcGcEZuZFaypccJT41ypmXUtzojNzArmMWIzs4I5IzYz\nK5gzYjOzgjkjNjMrlhyIzcyK5UBsZlYwNdAbOhpnNNzMuhRJVW0V2n9e0sNp+1wq21bS7ZIek3Sb\npF65+mMkLZA0X9LwXPkQSXMlPS5pfK58C0lTUpvpkvrVeq0OxGZWSh0JxJL2AkYDBwCDgfdL2g04\nB7gzIvYA7gLGpPp7AicBg4BjgCv1+sGvAkZHxEBgoKSjU/loYHlEDADGA+NqvVYHYjMrpQ5mxIOA\nGRGxMiJWA/cAJwLHAZNSnUnACenzccCUiHgtIhYBC4ChknoDPSJiVqo3Odfm+NyxbgSOrPVaHYjN\nrJxU5da6R4DD0lDE1sD7gF2AnSNiGUBELAV2SvX7AE/n2i9JZX2AxbnyxalsnTYp2L8oabtaLtU3\n68yslNrKdl9bNp/Xls2v2DYi/izpEuAO4B/AQ8Dq1qp2sJt5Nd9ddCA2s1JqKxBv3ntPNu+959rv\nKx/5Rav1ImIiMDEd62Ky7HWZpJ0jYlkadnguVV9CljE365vK2irPt3lGUjegZ0Qsb8clruWhCTMr\npU6YNbFj+tkP+CBwPXAz8PFUZRRwU/p8M3BymgnxVmB3YGYavlghaWi6eTeyRZtR6fMIspt/NXFG\nbGal1AnziH+exmxXAadHxEtpuGKqpFOBJ8lmShAR8yRNBebl6jcPW5wBXAN0B6ZFxK2pfAJwraQF\nwPPAybV21IHYzEqpo0/WRcThrZQtB97TRv2xwNhWyh8E9m6lfCUpkHeUA7GZlZIfcTYzK5gDsZlZ\n0RonDjsQm1k5OSM2MyuYA7GZWcEciM3MCuZAbGZWsEZaGN6B2MxKyRmxmVnBHIjNzIrWOHHYgdjM\nyqmRMuK6LoMpqb+keZJ+KOkRSbdK2lLSrpJ+I2mWpLslDUz1d00v4fuTpAslvVzP/plZeXV0Gcyu\nZGOsR7w7cHlEvB14EfgQ8EPgzIg4EPgS2cv5AL4HXBoR+5K9kqQzV883sy6kkQLxxhia+GtEPJw+\nzwbeAhwC/Cz3ltTN0893kL2QD7JFnL+9EfpnZiW0qQTZamyMQLwy93k1sDPwQkQMaaVuPgOu+Lfw\ng0tfXzb0gIMP5YB3HNaRPppZO9x79++5956763oOzyPuXC3/NF8C/irpQxFxI4CkfSJiLnA/2dDF\nVDaw2v2nzxpTj76aWRUOG3YEhw07Yu33sRdf0OnnaKSMeGOMEbcc5w3go8BoSXMkPQIcl/adBZwt\naQ6wG7BiI/TPzEpIqm7bFNQ1I46IJ4F9ct+/k9t9TCtNlkTEwQCSPgwMrGf/zKy8GikjLts84v0l\nXUE2nPECcGrB/TGzgjRQHC5XII6I+4DBRffDzIrnjNjMrGANFIcdiM2snJo8fc3MrFiNFIg3xvQ1\nM7N268j0NUkDJT0kaXb6uULS5ySdJ2lxKp8t6b25NmMkLZA0X9LwXPkQSXMlPS5pfK58C0lTUpvp\nkvrVeq0OxGZWSh1ZayIiHo+I/dITvPsD/wR+kXZ/NyKGpO3WdK5BwEnAILKptVfmlmC4ChgdEQOB\ngZKOTuWjgeURMQAYD4yr9VodiM2slDpx0Z/3AAsj4unmQ7dS53hgSkS8FhGLgAXAUEm9gR4RMSvV\nmwyckGszKX2+ETiy/VeZcSA2s1LqxCfrPgz8NPf9zPRU79WSeqWyPsDTuTpLUlkfspUgmy1OZeu0\niYjVwIuStmvvdYIDsZmVVGdkxJI2J1tC4Wep6Epg14gYDCwFvtNW21q6XGtDz5ows1JqK8a+/Nc5\n/GPRn6o9zDHAgxHxN4Dmn8mPgFvS5yXALrl9fVNZW+X5Ns9I6gb0jIjl1XYsz4HYzEqprWy35677\n0XPX/dZ+X/r7yZUOcwq5YQlJvSNiafp6IvBI+nwzcJ2kS8mGHHYHZkZEpBkXQ4FZwEjgslybUcAM\nYARwV3uuL8+B2MxKqaNP1knamuxG3adyxeMkDQbWAIuA0wAiYp6kqcA8YBVwekQ0rxx5BnAN0B2Y\n1jzTApgAXCtpAfA8G1i6txIHYjMrpY4+0BERrwA7tigbWaH+WGBsK+UPAnu3Ur6SbMpbhzkQm1kp\nedEfM7OCNVAcdiA2s3JyRmxmVrAGisMOxGZWTs6IzcwK1kBx2IHYzMrJGbGZWcEaaWF4B2IzKyVn\nxGZmBWugOOxAbGbl5IzYzKxgDRSHHYjNrJyaGigSOxCbWSk1UBx2IDazcvIYMSCpZ6WGEfFS53fH\nzCzTzfOIAXgUCNZ9IV7z9wD61bFfZtbgGighbjsQR8Qube0zM6s31f5S5C6nqZpKkk6W9JX0ua+k\n/evbLTNrdE2qbtsUbDAQS7oCeBfwsVT0CvCDenbKzExSVdumoJpZE4dExBBJDwFExHJJW9S5X2bW\n4DaRGFuVagLxKklNZDfokLQ92auozczqppEe6KhmjPj7wM+BHSV9A7gPuKSuvTKzhidVt20KNpgR\nR8RkSQ8C70lFIyLikfp2y8wa3aYy/luNqmZNAN2AVcCr7WhjZlazbk2qamuLpF6SfiZpvqRHJR0k\naVtJt0t6TNJtknrl6o+RtCDVH54rHyJprqTHJY3PlW8haUpqM11Szc9WVDNr4qvAT4E3A32B6yWN\nqfWEZmbVUJVbBd8DpkXEIGBf4M/AOcCdEbEHcBcwBkDSnsBJwCDgGOBKvZ6SXwWMjoiBwEBJR6fy\n0cDyiBgAjAfG1Xqt1WS3I4EDI+LciPgqMBT4eK0nNDOrRkemr6UlGg6LiIkAEfFaRKwAjgcmpWqT\ngBPS5+OAKaneImABMFRSb6BHRMxK9Sbn2uSPdSNwZK3XWk0gfpZ1x5I3S2VmZnXTwQc63gr8XdJE\nSbMl/VDS1sDOEbEMICKWAjul+n2Ap3Ptl6SyPsDiXPniVLZOm4hYDbwoabtarrXSoj+Xkk1ZWw48\nKum29H04MKutdmZmnaGDN+s2A4YAZ0TEAymenUOahpvT8ntH1NzhSrMmmmdGPAr8Old+f60nMzOr\nVltxeOm8WSyb/8CGmi8Gno6I5oo/JwvEyyTtHBHL0rDDc2n/EiC/vk7fVNZWeb7NM5K6AT0jYnkV\nl7aeSov+TKjlgGZmnaGtjPhNew3lTXsNXfv94V/8z3p1UqB9WtLAiHicbPz20bR9nOxZiFHATanJ\nzcB1KXPuA+wOzIyIkLRC0lCykYCRwGW5NqOAGcAIspt/NdngPGJJuwEXA3sC3XMXOrDWk5qZbUgn\nLOjzObLgujnwBPAJsqm4UyWdCjxJNlOCiJgnaSowj2yq7ukR0TxscQZwDVn8mxYRt6byCcC1khYA\nzwMn19rRah5xvga4CPj/ZNM6PkHnjquYma2no484R8SfgANb2fWeVsqIiLHA2FbKHwT2bqV8JSmQ\nd1Q1sya2jojb0okXRsS5ZAHZzKxumqSqtk1BNRnxyrToz0JJnyYboO5R326ZWaPbRGJsVaoJxGcB\n25CNt1wM9AJOrWenzMwaaa2Jahb9mZE+vszri8ObmdVVA8Xhig90/IIKN+Ui4sS69MjMjMZaj7hS\nRnzFRuuFmVkLDRSHKz7Q8duN2ZH2GtSnZ9FdsE6w7YFnFt0FKymPEZuZFaybA7GZWbE64cm6LqPq\nQCxpy/QkiZlZ3TVSIK7mDR1DJT1MtlAykvaVdHnde2ZmDa0jC8N3NdU84nwZ8H6yRS2an99+Vz07\nZWbWwYXhu5RqhiaaIuLJFr95VtepP2ZmgKevtfR0Wosz0uLHnwUer2+3zKzR+YGOdX2GbHiiH7AM\nuDOVmZnVTTXjppuKataaeI4OLHhsZlaLBkqIq3pDx49oZc2JiPhUXXpkZgZ021TuxFWhmqGJO3Of\nuwMfZN3XTpuZdboGisNVDU3ckP8u6Vrgvrr1yMwM36zbkLcCO3d2R8zM8hooDlc1RvwCr48RNwHL\ngXPq2SkzMw9NJMqe4tiX7D11AGtyr5g2M6sb0TiRuOJUvRR0p0XE6rQ5CJvZRtFIjzhXM2d6jqT9\n6t4TM7OcRgrEld5Zt1lEvAbsB8yStBD4JyCyZHnIRuqjmTWgRppHXCkjnpl+HgfsAbwPGAF8KP00\nM6sbqbqt8jHUJOkhSTen7+dJWixpdtrem6s7RtICSfMlDc+VD5E0V9LjksbnyreQNCW1mS6pX63X\nWulmnQAiYmGtBzczq1UnzSP+PPAokH/J5Xcj4rv5SpIGAScBg4C+wJ2SBqT7YlcBoyNilqRpko6O\niNuA0cDyiBgg6cPAOGpcDqJSIN5R0tlt7Wx5IWZmnamjIxOS+pL9S/5iIB/LWjvy8cCUNBy7SNIC\nYKikJ4EeETEr1ZsMnADcltqcl8pvpANvvq80NNENeAPQo43NzKxuOmFo4lLgS6y/Vs6ZkuZIulpS\nr1TWh3WXbliSyvoAi3Pli1PZOm0iYjXwoqTtarjUihnxsxFxQS0HNTPrqKY25hE/Pvt+Fjx0f8W2\nko4FlkXEHElH5HZdCVwQESHpIuA7wCc7p8e1T3ze4BixmVkR2sp299j/YPbY/+C1338z8XutVXsn\ncJyk9wFbAT0kTY6Ikbk6PwJuSZ+XALvk9vVNZW2V59s8k16a0TMilldzbS1VGpo4spYDmpl1ho7M\nI46Ir0REv4jYlewG2l0RMVJS71y1E4FH0uebgZPTTIi3ArsDMyNiKbAivURZwEjgplybUenzCOCu\nWq+1zYy41shuZtYZ6jSPeJykwcAaYBFwGkBEzJM0FZgHrAJOzz1JfAZwDdkywNMi4tZUPgG4Nt3Y\ne54OvECjltXXzMzqrrOWwYyIu4G70+eRFeqNBca2Uv4gsHcr5SvJprx1mAOxmZWSl8E0MyuYXx5q\nZlYwNVBK7EBsZqXUOGHYgdjMSsrvrDMzK1jjhGEHYjMrqQZKiB2IzaycujVQJHYgNrNS8qwJM7OC\nNU4YdiA2s5JyRmxmVjA/WWdmVjBnxGZmBWucMOxAbGYl1UAJsQOxmZWT5xGbmRVMDTQ44UBsZqXU\nQAmxA7GZlVOTM2Izs2I5IzYzK5gDsZlZwXyzzsysYE2NE4cdiM2snBrpVUmNtK6GmXUhqvJ/rbaV\ntpQ0Q9JDkh6WdF4q31bS7ZIek3SbpF65NmMkLZA0X9LwXPkQSXMlPS5pfK58C0lTUpvpkvrVeq0O\nxGZWSk2qbmtNRKwE3hUR+wGDgWMkDQXOAe6MiD2Au4AxAJL2BE4CBgHHAFfq9VWHrgJGR8RAYKCk\no1P5aGB5RAwAxgPjar7WWhuamdVTRzJigIh4JX3ckmwYNoDjgUmpfBJwQvp8HDAlIl6LiEXAAmCo\npN5Aj4iYlepNzrXJH+tG4Mhar7XugVhS/5Tq/0TSPElTJXWXdKSk2ZL+JOlqSZun+t+S9IikOZJq\n/g1jZl2bVN3Wdns1SXoIWArckYLpzhGxDCAilgI7pep9gKdzzZeksj7A4lz54lS2TpuIWA28KGm7\nWq51Y2XEewBXRMSewEvAfwETgRERsS+wOfCZdBEnRMTbI2IwcNFG6p+ZlYyq3NoSEWvS0ERfsux2\nL7KseJ1qndzlmmysWRNPRcT96fN1wNeAJyJiYSqbBJwOfB/4l6SrgV8Dv2rrgBddcP7az4cPO4LD\nhx3R+b02s1atfnkJa/6xpK7naGvWxOwZ9zF7xn1VHyciXpL0e+C9wDJJO0fEsjTs8FyqtgTYJdes\nbyprqzzf5hlJ3YCeEbG86o7lFDV97UVgvRQ+IlanAfUjgRHAmbQx7nLu18+vZ//MrIJuPfrQrUef\ntd9XL5tVoXZt2kov9z/oUPY/6NC13398+SXrt5V2AFZFxApJWwFHAd8CbgY+DlwCjAJuSk1uBq6T\ndCnZkMPuwMyICEkrUlyaBYwELsu1GQXMIItXd9V6rRsrEPeTdFBEzAA+QnZBp0naNSKeAD4G3C1p\na2CbiLhV0nTgLxupf2ZWNh2bRvwmYJKkJrIh2BsiYpqk+4Gpkk4FniSbKUFEzJM0FZgHrAJOj4jm\nYYszgGuA7sC0iLg1lU8ArpW0AHgeOLnWzm6sQPwYcIakicCjwKXA/cCNKaWfBfwA2B64SVL31O6s\njdQ/MyuZjjzQEREPA0NaKV8OvKeNNmOBsa2UPwjs3Ur5SlIg76iNFYhfi4iRLcp+x/p/UEuBgzZO\nl8yszBrnubqNF4g7886kmTWCBorEdQ/EEfEksE+9z2NmmxavvmZmVrAGWvPHgdjMyqmB4rADsZmV\nVANFYgdiMysljxGbmRXMb+gwMyuaA7GZWbE8NGFmVjBPXzMzK1gDxWEHYjMrqQaKxA7EZlZKHiM2\nMyuYx4jNzArmQGxmVjAPTZiZFcwZsZlZwRooDjsQm1lJNVAkdiA2s1LyGLGZWcE8RmxmVrAGisMO\nxGZWUg0UiZuK7oCZWWuapKq2tkiaIGmZpLm5svMkLZY0O23vze0bI2mBpPmShufKh0iaK+lxSeNz\n5VtImpLaTJfUr+ZrrbWhmVk9qcqtgonA0a2UfzcihqTtVgBJg4CTgEHAMcCV0toofxUwOiIGAgMl\nNR9zNLA8IgYA44FxtV2pA7GZlVUHI3FE3Ae80MaRWzoemBIRr0XEImABMFRSb6BHRMxK9SYDJ+Ta\nTEqfbwSOrPLK1uNAbGalpCr/V4MzJc2RdLWkXqmsD/B0rs6SVNYHWJwrX5zK1mkTEauBFyVtV0uH\nHIjNrJSk6rZ2uhLYNSIGA0uB73Rml2tt6FkTZlZKbUW16ffdzfQ/3FPTMSPib7mvPwJuSZ+XALvk\n9vVNZW2V59s8I6kb0DMiltfSLwdiMyultrLdQw4bxiGHDVv7ffy4iysehlxMl9Q7IpamrycCj6TP\nNwPXSbqUbMhhd2BmRISkFZKGArOAkcBluTajgBnACOCudlzeOhyIzaykOjaRWNL1wBHA9pKeAs4D\n3iVpMLAGWAScBhAR8yRNBeYBq4DTIyLSoc4ArgG6A9OaZ1oAE4BrJS0AngdOrrmvr5+r65AU/1rV\n9fpt69v2wDOL7oJ1gn/P+T4R0WmPYEiKJS+srKpun2237NRzF8EZsZmVkteaMDMrmFdfMzMrWuPE\nYQdiMyunBorDDsRmVk4eIzYzK5jHiM3MitY4cdiB2MzKqcmB2MysWB6aMDMrWCPdrPMymGZmBXNG\nbGal1EgZsQOxmZWSx4jNzArmjNjMrGANFIcdiM2spBooEnvWREndc/fvi+6CdZLVLy/ZcCVbT5NU\n1bYpcCAuKQfiTceafzgQ10JVbpsCD02YWTltKlG2Cg7EZlZKjTR9rcu+PLToPpjZujr55aGLgP5V\nVn8yIt7SWecuQpcMxGZmmxLfrDMzK5gDsZlZwRyIzcwK5kBsZlYwB+IuQMoeH2r+aV2b/x6tJQfi\nrmEgQESE/yPuuiTtJWnn8FQla8GBuOQkDQBmSboCHIy7KknHAVcBb8mV+e/RAM8jLrX0H+9Hgb8C\nHwNuiYhPp31yZtU1SNoL+ClwYkT8RdIOwNYR8ZSkpohYU3AXrWDOiEtK0jbA2cD1EXEO8HbgXZIu\nA2fGXUHu72dn4DlgJ0lfByYBcyUNdhA2cCAus1fIMuHFABHxAvB54BOSLkxlzojLbfv08/fAA8D3\ngCeAk4FxwF7FdMvKxoG4ZCTtIWkXYBtgJnCdpK3T7n8APwSGSzq8qD7ahkl6L3C9pMnA+cC3IuLA\niPgJWQAeSfaL1syrr5WJpGOAS4AbgVPIhiP2Au6V9FvgI8BxwOq0WQmlMeErgE8APYH9gR9I+iJZ\nljwJ+K+I+GNxvbQycUZcEpJ2B84DPgj8BVhDdkPnTOBLwD3Au8ky5eHAswV11TZsS+COiLgX+A3w\nY+BlYA/gYeCDEfErj/FbM2fE5fECcB1Z9vQF4PiIeFnScOD+iHgpZVrfBkZFxBMF9tVaIemdwFuB\nzYERkm6JiGnAYkmvAf3Tzbl54DF+e50DccEkDQMGkd3EOYvs72S3iFgl6WDgHOA/gZfIbtwdGxHP\nF9Vfa52kQ4CrgQeBZcBTwNfTeP+jwCHA5OJ6aGXmecQFknQQ2T9bHwPmA1uR3cS5GHgNOBU4PyJu\nKqyTtkGShpKN7Y+JiPsl7Uo2ln8IsB3wJNkc8F8W2E0rMWfEBUn/8X4DOCUi5kr6GNkbCW4gu0H3\nCPDliLjDD2+UXi/gcLIx/PuBp8n+hdMXOLl5rrD/Hq0tvllXnDcC7wGOSt9/SvYf78vAwxExPiLu\nAI8lll36ezoROFXSKRGxClgBDAN2aL4p579Ha4sz4oJExO2STgTGSnomIn4q6Ya0+09F9s3aLyJu\nkrSGbN4DtUuSAAADwklEQVT3/yOb9XJhRDxXcNesC/AYccEkvQ+4ELgsIiYV3R/rmLQ+yAXAdRHx\nbWfDVg1nxAWLiGmSNgO+JekOYKnXH+i6IuJmSf8GfixpYUT8b9F9svJzRlwSknaMiL8V3Q/rHJKO\nAhZ6vrdVw4HYzKxgnjVhZlYwB2Izs4I5EJuZFcyB2MysYA7EZmYFcyBuIJJWS5ot6WFJN0jq3oFj\nDZN0S/r8AUlfrlC3l6TP1HCO8ySdXW15izoT05OL1Z6rv6SH29tHs87gQNxY/hkRQyJib2AV8OmW\nFdq5WHkARMQtETGuQr1tgdPb1dNieC6nFcKBuHHdC+yeMsE/S5qUMsK+ko6S9EdJD6TMeWvI3sMm\nab6kB8gWuSGVj5J0efq8k6T/lTRH0kNpTeWxwG4pG78k1fuipJmp3nm5Y31V0mOS7iF7o0VFkj6Z\njvOQpJ+1yPKPkjQrXd+xqX6TpHGSZqRz/2eH/yTNOsiBuLEIID1SfQzZa3sABgBXpEz5FeBc4MiI\nOIBsofOzJW1J9uLSY1N57xbHbs4mLwN+HxGDgSFki6KfA/wlZeP/nZ46GxARQ4H9gAMkHSppCHAS\nsA9wLHBgFdf084gYGhH7AX8GRuf29Y+IA4H3k70zbou0/8WIOAgYCnxKUv8qzmNWN15rorFsJWl2\n+nwvMAHoAyyKiFmp/GBgT+APaZhic2A68Dbgidwjuz8he3NIS+8GPgZrF7p5WdJ2LeoMJ8tWZ5P9\nctiG7JdBT+AXEbESWCnp5iquaR9JF5ItK7oNcFtu39TUj79IWpiuYTiwt6QRqU7PdO4FVZzLrC4c\niBvLKxExJF+QhoT/mS8Cbo+Ij7aot2/atyHVjLMKGBsRP2pxjs9X0balicBxEfGIpFFkawC31hel\n7wI+27zWc+7czoqtMB6aaCxtBdJ8+f3AOyXtBiBpa0kDyP7Z31/SW1O9U9o41m9JN+bSeGxPssXu\ne+Tq3Ea2iPo2qd6bJe1I9qbqEyRtKakH8IEqrukNwFJJmwMfbbFvhDK7kb3U87F07tPT8AySBkja\nqpU/B7ONxhlxY2krW11bHhF/l/Rx4KdpXDiAcyNigaTTgGmS/kk2tPGGVo71BeCHkkaTvXfvMxEx\nI938mwv8Jo0TDwKmp4z8ZeA/IuIhSVOBuWQv4JxZxTV9PdV7DpjBugH/qbSvB3BaRLwq6WrgLcDs\nNPTyHHDCBv58zOrKq6+ZmRXMQxNmZgVzIDYzK5gDsZlZwRyIzcwK5kBsZlYwB2Izs4I5EJuZFcyB\n2MysYP8HUQahZr6LKhIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd302b9f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUwAAAEpCAYAAAD4Vxu2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu8XdO5//HPdyfuSVzqUoKoeyjKIS7toWg1fupSp06F\nojiq2vSmSnuqpbSlRW9CS6lDFXVUi1bRqjuRECJIxK1BBCVRITiRPL8/5tjJzLL3WnNn7b3X2Nnf\n9+u1XpmXseZ81tp7PxlzzDHHUERgZmaNtbU6ADOzvsIJ08ysIidMM7OKnDDNzCpywjQzq8gJ08ys\nIifMJYSkkyT9Ji2vI+k1SermczwtabfuPGaFcx4j6YX0eVZu4jizJa3XfZG1jqSHJe3c6jj6IyfM\niiT9Q9KLkpYrbTtS0i2tjKtGAETEsxExJPp4J1tJA4GzgI+kzzNrcY8VEYMj4h/dFlwPkHSRpFMa\nlYuI90fE7b0Rky3KCbO6oPi+vtLB9i7r7trfEuq9wDLA5FYHkgNJA1odQ3/nhNk1ZwBfkzSko52S\ndpI0TtIsSfdK2rG07xZJ35N0p6Q3gPelbadKuitdMl4jaRVJl0r6VzrGuqVj/FTSM2nfeEkf6iSO\nYZLmS2qTtEM69mvp9aakp1I5SfqGpCck/VPSFZJWKh3nkFSz/qek/673xUhaVtJZqfwsSbdLWibt\n2yddRs6U9HdJm5be97Skr0mamN53uaSlJW0ETEnFZkn6W/lz1XyvR6TlDSTdKulVSS9JurxUbr6k\n9dPyEEmXpDJPS/pWqdxhku6QdEaK90lJI+t87qclHZfiny3pV5JWl3R9+r5vkrRiqfyVkmakz3qr\npOFp+1HAwcDx6X3XlI5/vKSJwOuSBqjUNCLpz5LOLB3/CkkX1PtZWRMiwq8KL+BpYDfgKuDUtO1I\n4O9peWVgJnAQxX9EB6b1ldP+W4B/AJum/QPTtqnAesBg4BGKJLFrKnMxcGEphoOAldK+rwIzgKXT\nvpOAS9LyMGAe0FbzGQYCtwLfS+tfBu4G1gSWAn4BXJb2bQbMBj6Y9p0F/B+wWyffzznA3ylqhQJ2\nSO/bGHg9fXcDgK8DjwMDS9/rWGCN9NkeBT5b8znU2edK3+ERafky4JtpeWlgp1K5ecD6afkS4A/A\n8umYjwGHp32HAW8DR6TP8TlgeoPfi7uBVdP3+CJwH7BliuFm4Nul8p9J510K+DHwQGnfRcApHRx/\nArAWsEz5dzEtrwG8AHyYIuE+ASzf6r+XJfXV8gD6youFCXNzYBbwHhZNmJ8Gxta8527g0LR8C3By\nzf5b2v/A0/qZwJ9L6x8HJtSJaSawRVqukjB/AVxbWn8U2LW0viZFUmwDvk1Knmnf8imRvCthpsQy\nB3h/B/tOBK6oKfscsHPpex1V2v9D4NyOPkdHn4tFE+bFwC+BoR3EMR9YP322t4FNSvs+W/o5HgZM\nLe1bLp1z9Tq/F+X4rwLOKa2PBq7u5L0rpbgGp/XOEuZhHf0ultY/ATwDvATs2Oq/lSX55UvyLoqI\nR4A/Ad+s2bUWMK1m2zRgaGn92Q4O+WJp+c0O1ge1r6RLv0fT5dwsYAhFzaYhSUcDO1PUUtsNA/6Q\nLj1nUiTQuRS1lrXK8UbEHOCVTg6/KkVb41Md7Fvke4niL/xZFv1eyp95DqXP3EVfp0iI4yRNknR4\nJ7EOpEgw7Wp/Ti+U4n2TIsnXi6nSzzA1kZyemkBepUh8QeOf4XMN9v+Jovb+WETc06CsNcEJc/Gc\nDBzFon9kz1NcWpetC0wvrS/2XWtJ/06RED4ZEStHxMrAaxR/zFXe+11gn4h4vbTrGWDPiFglvVaO\niBUiYgbF5f46pWMsT1Gr7sjLwFvABh3se54iMZetQ+Mk0JE30r/Ll7a9t30hIl6KiM9GxFCKS+lz\n29sta2KdWxPTMBb9OfWUg4G9KWqHK1H8voiFP8POfj8a/d78gOI/uzUlHdgNcVonnDAXQ0Q8CfwO\n+FJp8/XARpIOTA3znwKGA9d102kHUfyhv5JuinyHot2zM4KiT2aK9dAUd9l5wA/abyxJWk3SPmnf\nVcDHVdzIWgo4hU6Sc6o1/hr4saQ1SzeblgKuBPaStKukgZKOo0iuVWtCC84ZES9TJLZPp3McQSlJ\nS/qkpPb/xF6luNydXxPr/BTT9yUNkjSMoj34NxXjacYgiuaAWZJWAE5j0WT4IkWzQWUq+mMeBhxC\n0T56tqQ1uyVaexcnzOpq/5c/haKm0973cSZFm+NxFLWY44C9YmHfwY5qCV2pcd6YXlMpLuXm0PEl\nfu2xdwNWB65Kd19nS5qU9v0MuAa4SdK/KNpcR6TP8yjwBeByilriK9SvFR4HTALGp7KnU7Q1TqVo\n3x0D/BPYC9g7It6pibPR52h3FHA8xXc8HLirtG874F5JrwF/BL4UC/telo/zJYrv7yngduDSiLio\nCzHU21ev7CUUtfrpwMMU33fZhcDmqYnk6jrHKxqDpcEU7bZfiIgXIuJO4AKKtlDrAe13H83MrAHX\nMM3MKnLCNDOryAnTzKwiJ0wzs4oGtjqAxSHJd6rMMhMR3TagjJYeEsyd3ZW3TIuI9brr/J3pk3fJ\nJcWQUZe0Oowe9dakq1l2i/1bHUaPu/esJf8zjjnz+4w+7luNC/Zhw9ca1L0JU4plt/5i5fJvPXB2\nt56/M32yhmlm/UCGIyA6YZpZnpTfLRYnzEwNXH14q0OwbjJip39vdQh9k2uYVtXANZwwlxQjdvL0\nO4vFNUwzs4pcwzQzq6gtvymMnDDNLE++JDczq8iX5GZmFWVYw8wvIjMzKGqYVV8dvl0jJU2RNFXS\nCR3sX0nS1WmK5LGSNmsUkhOmmeVJbdVftW8t5q4fA3yMYqbXUZI2rSn23xTTHG9FMc3HzxuF5IRp\nZnlqImFSTLXyeERMi4i5wBXAvjVlNgP+DhARjwHrSVqtXkhOmGaWpzZVf73bUBad8+o5Fp3lFWAi\nsD+ApBEUs7yuXS8k3/Qxszz1fD/M04GfSZpAMYHfA8C8em9wwjSzPNW5Sz5v1lPMn/VUvXdPp6gx\ntlubmrnnI2I2cMSC00lPU8wk2iknTDPLU51+mANW2YABqyyYkp55T99cW2Q8sGGad34GcCAwatHD\na0VgTkTMlXQUcFtEvF4vJCdMM8tTE/0wI2KepNHATRT3ai6MiMmSji52x/kU89pfLGk+8AhwZKPj\nOmGaWZ6afNInIm4ANqnZdl5peWzt/kacMM0sTxk+6eOEaWZ58rPkZmYVuYZpZlaRa5hmZhW15Zee\n8ovIzAxcwzQzq8xtmGZmFbmGaWZWkWuYZmYVuYZpZlaNnDDNzKpxwjQzq0gdj6TeUk6YZpYl1zDN\nzCpywjQzqyjHhJlfRyczMwB14dXR26WRkqZImirphA72D5F0raQHJU2S9JlGITlhmlmWJFV+dfDe\nNmAM8DFgc2CUpE1rin0BeCQiPgDsCpwlqe5VtxOmmWWpmYQJjAAej4hpETEXuALYt6ZMAIPT8mDg\nlYh4p15MbsM0syw12YY5FHi2tP4cRRItGwNcK+l5YBDwqUYHdcI0syzV64c594VHmfvCo82e4mPA\nAxGxm6QNgL9K2rLeVLtOmGaWpXo1zKXX3Jyl19x8wfpbD11dW2Q6sG5pfe20rexw4DSAiHhS0tPA\npsB9nZ3XbZhmlqUm2zDHAxtKGiZpaeBA4NqaMtOAj6RzrQFsDDxVLybXMM0sS820YUbEPEmjgZso\nKoYXRsRkSUcXu+N84HvA/0h6KL3t+IiYWe+4Tphmlqcm+61HxA3AJjXbzistz6Box6zMCdPMspTj\nkz5OmGaWJSdMM7OKnDDNzCpywjQzq8gDCJuZVeQapplZRU6YZmZV5ZcvnTDNLE851jB79Fny9Bzn\no5LOl/SwpBskLSNpfUl/kTRe0m2SNk7l15d0j6SJkk6VNLsn4zOzfDX5LHmP6I3BNzYEzo6I9wOv\nAp8EzgdGR8R2wNeBX6SyPwN+EhFbUYxfF70Qn5llKMeE2RuX5E9HxKS0PAFYD9gJ+F8t/KRLpX93\nZOGoyJcBZ/RCfGaWoRwvyXsjYb5dWp4HrAHMiohtOihbrlHW/bbemrRw/LuBqw9n4BrDm4nRzLpg\n3N23M+7uO3r0HP21H2btp34NeFrSJyPiKoA0yvFDwFiKS/YrKcav69SyW+zfE7GaWQUjdtqZETvt\nvGD9nB+f1u3nyLGG2RttmLXtkAEcDByZprd8GNgn7fsqcKykB4ENgH/1QnxmliGp+qu39GgNMyKm\nAVuW1s8q7d6zg7dMj4gdACR9imIEZDPrh5qtYUoaCfyUhQMI/7Bm/3EUlbeguI8yHFg1Il7t7Ji5\n9cP8N0ljKC7jZwFHtDgeM2uRZvJlaV7y3YHngfGSromIKe1lIuJM4MxU/uPAV+olS8gsYUbEncAH\nWh2HmbVekzXMBfOSp2O1z0s+pZPyo4DLGx3Uk6CZWZaabMPsaF7yoR2fR8sBI4HfN4opqxqmmVm7\ntt7rVrQ3cGejy3FwwjSzTNVLmG9Mm8gb0ybWe3uVecnbHUiFy3FwwjSzTNVrwhy03lYMWm+rBev/\nvOPS2iIL5iUHZlAkxVHvPodWBHahuFvekBOmmWWpF+YlB9gPuDEi3qxyXCdMM8tSs/0wG81LntYv\nBi6uekwnTDPLUoZPRjphmlmecnyW3AnTzLKUYb50wjSzPLmGaWZWUYb50gnTzPLUi0/6VOaEaWZZ\n8iW5mVlFGeZLJ0wzy5NrmGZmFWWYL50wzSxPrmGamVWUYb50wjSzPLmGaWZWkfthmplV5BqmmVlF\nGeZLzxppZnmSVPnVyftHSpoiaaqkEzop82FJD0h6WNItjWJyDdPMstRMDVNSGzAG2B14Hhgv6ZqI\nmFIqsyJwDrBHREyXtGqj4zphmlmW2pq7Jh8BPB4R0wAkXQHsC0wplTkI+H1ETAeIiJcbxtRMRGZm\nPUWq/urAUODZ0vpzaVvZxsAqkm6RNF7SIY1icg3TzLJU7y75zKn3M/PxCc2eYiCwDbAbsAJwj6R7\nIuKJem/okKQh9c4UEa8tbpRmZo0MqNMPc7VNt2W1TbddsP7U9RfWFpkOrFtaXzttK3sOeDki3gLe\nknQ7sBXQ9YQJPAIEUI66fT1qgjEz61ZNdisaD2woaRgwAzgQGFVT5hrgbEkDgGWA7YEf1ztopwkz\nItZpKlwzsyaIxc+YETFP0mjgJop7NRdGxGRJRxe74/yImCLpRuAhYB5wfkQ8Wu+4ldowJR0IrB8R\nP5C0NrBGRNy/2J/GzKyBZp+MjIgbgE1qtp1Xs34mcGblmBoVkDQG2BVov4M0B/hl1ROYmS2OZjuu\n94QqNcydImIbSQ8ARMRMSUv3cFxm1s/l+GhklYQ5N/WaDwBJ7wHm92hUZtbvNdlxvUdU6bh+DvB7\nYDVJ3wXuBH7Yo1GZWb/XZMf1HtGwhhkRl0i6H/hI2nRARDzcs2GZWX/Xl4d3GwDMpbgs9+OUZtbj\n6nVcb5Uqd8m/BVwOrEXRW/4ySd/s6cDMrH9TF169pUoN81Bg64iYAyDp+8ADwGk9GZiZ9W999ZJ8\nRk25gWmbmVmPyfCKvO7gGz+haLOcCTySHiEKYA+K5zTNzHpMX6thtt8JfwT4c2n72J4Lx8yskGG+\nrDv4xrvGSzIz6y19rYYJgKQNgO8DmwHLtm+PiI17MC4z6+dybMOs0qfyf4CLKO7e7wlcCfyuB2My\nM6NNqvzqtZgqlFk+Im4EiIgnI+JEisRpZtZj+mrCfDsNvvGkpM9J2hsY3MNxmVk/1+yz5I3mJZe0\ni6RXJU1IrxMbxVSlH+ZXKSYI+hJFW+aKwBEV3mdmttiauelTZV7y5PaI2KfqcasMvnFvWpzNwkGE\nzcx6VJNX2lXmJYcuPllZr+P6H0hjYHYkIvbvyonMzLqiybbJjuYlH9FBuR0lPUgxo+TXm5nTZ0yX\nQzQz6ya9cC/nfmDdiJgjaU/gj0Dd7pL1Oq7f3M3BdasXL3HrwJJg5e1GtzoEy1S9NszpD49j+sPj\n6r294bzkEfF6afkvks6VtEpEzOzsoFXHwzQz61UD6iTMdbfYnnW32H7B+n1XnltbpOG85JLWiIgX\n0/IIQPWSJThhmlmmmnnSp8q85MAnJR1DMTj6m8CnGh23csKUtExEvL144ZuZdU1Pz0seEedQzFlW\nPaZGBSSNkDQJeDytbyXp7K6cxMysq3Kcl7zKkz4/Bz4OvAIQEROBXXsyKDOzNlV/9ZYql+RtETGt\nJovP66F4zMyAPjYeZsmz6Q5SSBoAfBGY2rNhmVl/15uDalRVJWEeQ3FZvi7wIvC3tM3MrMfkOJ93\nlWfJX6Low2Rm1msyrGBWGnH9V3TwTHlEfLZHIjIzAwZkOOR6lUvyv5WWlwU+waIPtZuZdbsM82Wl\nS/JFpqOQ9Bvgzh6LyMyMvnvTp9b7gDW6OxAzs7IM82WlNsxZLGzDbANmAt/oyaDMzPrcJbmK3upb\nsXBYpPkR0emgwmZm3UVdGwy9V9Tt6pSS4/URMS+9nCzNrFfk+Ghklb6hD0rauscjMTMryTFh1pvT\nZ2BEvANsTTHj2pPAGxSTBkVEbNNLMZpZP9TX+mGOA7YBKk9BaWbWXXK8S17vklwAEfFkR69eis/M\n+qk2qfKrI5JGSpoiaaqkEzo7j6TtJM2V1HAm3Ho1zNUkHdvZzoj4caODm5ktrmauyCW1Ucx8uzvw\nPEWz4jURMaWDcqcDN1Y5br2EOQAYRBcnOjcz6w5NXpKPAB6PiGnFsXQFsC8wpabcF4GrgO2qHLRe\nwpwREacsRqBmZk1ra66uNpRFx7x4jiKJLiBpLWC/iNg1jfnbUL2E6ZqlmbVML9z0+SlQbttseMZ6\nCXP3psMxM1tM9dowp04Yy9QJY+u9fTrFoOft1mbhE4vttgWuSE80rgrsKWluRFzb2UE7TZiNJjQ3\nM+tJ9fphDt92R4Zvu+OC9T//+me1RcYDG0oaBsygGAR9VLlARKzfvizpIuC6eskSFm+0IjOzHtfM\n8G4RMU/SaOAmiu6TF0bEZElHF7vj/Nq3VDmuE6aZZanZNsyIuAHYpGbbeZ2UPaLKMZ0wzSxLfXIS\nNDOzVlCGz0Y6YZpZlvJLl06YZpapJWVOHzOzHpdfunTCNLNMZVjBdMI0szwNyDBjOmGaWZZ8l9zM\nrKL80qUTppllyjVMM7OK/KSPmVlFrmGamVWUX7p0wjSzTGVYwXTCNLM8uR+mmVlFyvCiPMcbUWZm\nSNVfHb9fIyVNkTRV0gkd7N9H0kRJD0gaJ+mDjWJyDdPMstTMNLuS2oAxFJM5Pg+Ml3RNRJTnJf9b\n+xw+krYArgSG14/JzCxDTdYwRwCPR8S0iJgLXAHsWy4QEXNKq4OA+Y1icsI0syw1mTCHAs+W1p9L\n22rOof0kTQauAxrO6+NLcjPLUr2bPhPH3cXEcXc1fY6I+CPwR0kfAr4HfLReeSdMM8tSnWnJ2Xr7\nD7L19gvv0Vx67hm1RaYD65bW107bOhQRd0paX9IqETGz05jqh2xm1hptUuVXB8YDG0oaJmlp4EDg\n2nIBSRuUlrcBlq6XLME1TDPLVDP9MCNinqTRwE0UFcMLI2KypKOL3XE+8B+SDgX+D3gT+M9Gx3XC\nNLMs1bskryIibgA2qdl2Xmn5R8CPunJMJ0wzy1K/fNIntSFMlnSppEclXSlpWUm7S5qQetpfIGmp\nVP50SQ9LelBSl7K/mS05mn3Spyf01k2fTYAxEbEZ8BrwNeAi4ICI2ApYCjhG0irAfhHx/oj4AMVt\nfjPrh9SFV2/prUvyZyJibFr+LfBt4KmIeDJtuxj4PHAO8KakC4A/A3/q7IDfO+XkBcs77/Jhdt7l\nw90ftZl1aN7s6cx/vdNeOt2ik7vfLdWqNsxXgVVqN6Y7WyMonv88ABidlt/lxO+c3JPxmVkdAwYP\nZcDghQ/OzHtxfLefI7902XuX5OtK2j4tH0TRR2o9SeunbYcAt0laHlgp3d06Ftiyl+Izs9xkeE3e\nWzXMx4AvSLoIeAT4CTAWuErSAIoE+kvgPcA1kpZN7/tqL8VnZpnpz5fk70TEoTXbbgG2qdn2ArA9\nZtbv5Zcuey9hRi+dx8yWFBlmzB5PmBExDbdFmlkX5dhx3U/6mFmWMmzCdMI0szxlmC+dMM0sUxlm\nTCdMM8uS2zDNzCpqdni3nuAR180sT00+6VNhXvKD0mhpEyXdmabarcs1TDPLUjOX5BXnJX8K2Dki\n/iVpJPArYId6x3XCNLMsNdmtaMG85MWx1D4v+YKEWRpBDYpHtd81DW8tX5KbWZaavCKvNC95yX8B\nf2kUk2uYZpanXrrpI2lX4HDgQ43KOmGaWZbqtWGOu/t2xt9zR723V5qXXNKWwPnAyIiY1TCmiL43\nLoakeHNu34vb3m3l7Ua3OgTrBm89eA4R0W11QknxyPTXK5fffOigRc6fho18jOKmzwxgHDAqIiaX\nyqwL3AwcUtOe2SnXMM0sS83c9Kk4L/m3KWZ+OFeSgLkRMaLecZ0wzSxLzT7pU2Fe8qOAo7pyTCdM\nM8uSRysyM6sow3zphGlmmcowYzphmlmWPFqRmVlFbsM0M6sow3zphGlmmcowYzphmlmW2jK8JnfC\nNLMs5ZcunTDNLFcZZkwnTDPLkrsVmZlVlGETphOmmeUpw3zphGlmeXIN08yssvwyphOmmWWpLb98\n6VkjzSxPUvVXx+/XSElTJE2VdEIH+zeRdLektyQdWyUm1zDNLEvNdCuS1AaMoZjT53lgvKRrImJK\nqdgrwBeB/aoe1zVMM8tTcxOTjwAej4hpETEXuALYt1wgIl6OiPuBd6qG5IRpZllqLl8yFHi2tP5c\n2tYUX5KbWZbcrcjMrKJ6bZh33XEbd91xW723TwfWLa2vnbY1F1NENHuMXicp3pzb9+K2d1t5u9Gt\nDsG6wVsPnkNEdFudUFK8NHtu5fKrD15qkfNLGgA8RnHTZwYwDhgVEZM7ONdJwOsRcVaj87iGaWZZ\naqYfZkTMkzQauIniXs2FETFZ0tHF7jhf0hrAfcBgYL6kLwObRcTrnR3XCdPMstTsaEURcQOwSc22\n80rLLwLrdOWYTphmlqUcb/q4W5GZWUWuYZpZlnKsYTphmlmWPOK6mVlFrmGamVWUYb50wjSzTGWY\nMX2XPFO333Zrq0OwbjJvdtNP5PVLbVLlV6/F1Gtnsi5xwlxyzH/dCXNxNDlaUY/wJbmZ5SnDS3In\nTDPLUo7divrsaEWtjsHMFtXNoxX9AxjWhbdMi4j1uuv8nemTCdPMrBV808fMrCInTDOzipwwzcwq\ncsI0M6vICbMPkIpHGdr/tb7NP8e+ywmzb9gYiolI/MfWd0naXNIa4a4pfZYTZuYkbQSMlzQGnDT7\nKkn7AL8A1itt88+xj3E/zIylP7KDgaeBQ4DrIuJzaZ9cU+kbJG0OXA7sHxFPSFoVWD4inpHUFhHz\nWxyiVeQaZqYkrQAcC1wWEd8A3g/sKunn4JpmX1D6+awBvASsLuk7wMXAQ5I+4GTZtzhh5msORc3y\nOYCImAV8GThc0qlpm2uYeXtP+vdWivmvfwY8BRwI/AjYvDVh2eJywsyMpE0krQOsAIwDfitp+bT7\ndeB8YA9JO7cqRmtM0kjgMkmXACcDp0fEdhFxKUWiPJTiP0TrQzxaUUYk7Qn8ELgKGEVxGb45cIek\nm4GDgH2AeellGUptlmOAw4EhwL8Bv5R0HEWt82LgaxFxd+uitMXhGmYmJG0InAR8AngCmE9xY2A0\n8HXgdmA3iprnHsCMFoVqjS0D/DUi7gD+AvwamA1sAkwCPhERf3IbdN/jGmY+ZgG/paiNfAXYNyJm\nS9oDGBsRr6WayxnAYRHxVAtjtQ5I+iDwPmAp4ABJ10XE9cBzkt4BhqWbPI+C26D7IifMFpO0CzCc\n4mbAVyl+JhtExFxJOwDfAI4CXqO4AbRXRLzSqnitY5J2Ai4A7gdeBJ4BvpPaox8BdgIuaV2E1h3c\nD7OFJG1Pcbn2GDAZWI7iZsD3gXeAI4CTI+KalgVpDUkaQdH2/M2IGCtpfYq25p2AVYBpFH1o/9jC\nMK0buIbZIumP7LvAqIh4SNIhFCNM/47iRs/DwPER8Vd3Us/eisDOFG3MY4FnKa4Y1gYObO9r6Z9j\n3+ebPq2zEvAR4KNp/XKKP7LZwKSI+GlE/BXc1pW79HPaHzhC0qiImAv8C9gFWLX95o5/jn2fa5gt\nEhE3SdofOE3S8xFxuaTfpd0TWxmbdV1EXCNpPkW/2f+g6OVwakS81OLQrBu5DbPFJP0/4FTg5xFx\ncavjseak5/9PAX4bEWe4drlkcQ2zxSLiekkDgdMl/RV4wc8X910Rca2kt4BfS3oyIq5udUzWfVzD\nzISk1SLin62Ow7qHpI8CT7q/7JLFCdPMrCLfJTczq8gJ08ysIidMM7OKnDDNzCpywjQzq8gJsx+R\nNE/SBEmTJP1O0rJNHGsXSdel5b0lHV+n7IqSjlmMc5wk6diq22vKXJSepKp6rmGSJnU1RutfnDD7\nlzciYpuI2AKYC3yutkAXB7UNgIi4LiJ+VKfcysDnuxRpa7iPndXlhNl/3QFsmGpWUyRdnGpYa0v6\nqKS7Jd2XaqLLQzFPjaTJku6jGGyCtP0wSWen5dUlXS3pQUkPpDE9TwM2SLXbH6Zyx0kal8qdVDrW\ntyQ9Jul2ihHK65L0X+k4D0j635pa80cljU+fb69Uvk3SjyTdm859VNPfpPUbTpj9iwDSo5h7UkyX\nALARMCbVPOcAJwK7R8S2FAPiHitpGYoJ2PZK299bc+z22tnPgVsj4gPANhSD534DeCLVbk9IT8Fs\nFBEjgK2BbSV9SNI2wH8CWwJ7AdtV+Ey/j4gREbE1MAU4srRvWERsB3ycYk6dpdP+VyNie2AE8FlJ\nwyqcx8zPkvczy0makJbvAC4EhgL/iIjxafsOwGbAXenyfCngHmBT4KnSo36XUowEX2s34BBYMODE\nbEmr1JTZg6L2N4Eiia9AkbSHAH+IiLeBtyVdW+EzbZmmHV4pHefG0r4rUxxPSHoyfYY9gC0kHZDK\nDEnnfryH9dhoAAABYklEQVTCuayfc8LsX+ZExDblDanJ8o3yJuCmiDi4ptxWaV8jVdoBBZwWEb+q\nOceXK7y31kXAPhHxsKTDKMag7CgWpXUBX2wfa7R0btcyrSFfkvcvnSW88vaxwAclbQAgaXlJG1Fc\n7g6T9L5UblQnx7qZdIMntRcOoRgUeXCpzI0Ug+2ukMqtJWk1ipkx95O0jKTBwN4VPtMg4AVJSwEH\n1+w7QIUNKCYneyyd+/OpWQJJG0laroPvwexdXMPsXzqr/S3YHhEvS/oMcHlqtwzgxIh4XNLRwPWS\n3qC4pB/UwbG+Apwv6UiKeYmOiYh7002kh4C/pHbM4cA9qYY7G/h0RDwg6UrgIYqJxMZV+EzfSeVe\nAu5l0cT8TNo3GDg6Iv5P0gXAesCE1OTwErBfg+/HDPBoRWZmlfmS3MysIidMM7OKnDDNzCpywjQz\nq8gJ08ysIidMM7OKnDDNzCpywjQzq+j/AwMXkH1IiNWrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cd0a63e940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_confusion_matrix(cm, title='Confusion matrix', cmap=plt.cm.Blues):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(set(labels)))\n",
    "    plt.xticks(tick_marks, ['neg','pos'], rotation=45)\n",
    "    plt.yticks(tick_marks, ['neg','pos'])\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    \n",
    "# Compute confusion matrix\n",
    "cm = confusion_matrix(Test_labels, tfprediction['Logistic'])\n",
    "np.set_printoptions(precision=2)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm)    \n",
    "\n",
    "cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm_normalized, title='Normalized confusion matrix')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "## Thank you"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

## Report
Amazon Fine Food Reviews: A Sentiment Classification Problem

Abhishek Grover
1503611
abgrover@ucsc.edu


Instructor: Jay Pujara
jpujara+tim245@ucsc.edu
1.	Abstract
The internet is full of websites that provide the ability to write reviews for products and services available online and offline. The websites like yelp, zomato, imdb etc got successful only through the authenticity and accuracy of the reviews they make available. Success of product selling websites such as Amazon, ebay etc also gets affected by the quality of the reviews they have for their products. All these sites provide a way to the reviewer to write his/her comments about the service or product and give a rating for it. Based on these comments one can classify each review as good or bad. From this data a model can be trained that can identify the sentiment hidden in a review. This has many possible applications: the learned model can be used to identify sentiments in reviews or data that doesn’t have any sentiment information like score or rating eg. People post comments about restaurants on facebook and twitter which do not provide any rating mechanism. This project intends to tackle this problem by employing text classification techniques and learning several models based on different algorithms such as Decision Tree, Perceptron, Naïve Bayes and Logistic regression. This paper will discuss the problems that were faced while performing sentiment classification on a large dataset and what can be done to solve those problems


2.	Introduction
The main goal of the project is to analyze some large dataset and perform sentiment classification on it. Sentiment classification is a type of text classification in which a given text is classified according to the sentimental polarity of the opinion it contains. For the purpose of this project the Amazon Fine Food Reviews dataset, which is available on Kaggle, is being used. [1][4]

Following sections describe the important phases of Sentiment Classification: the Exploratory Data Analysis for the dataset, the preprocessing steps done on the data, learning algorithms applied and the results they gave and finally the analysis from those results.

3.	Exploratory Data Analysis
The Amazon Fine Food Reviews dataset is ~300 MB large dataset which consists of around 568k reviews about amazon food products written by reviewers between 1999 and 2012. Each review has the following 10 features:
•	Id
•	ProductId - unique identifier for the product
•	UserId - unqiue identifier for the user
•	ProfileName
•	HelpfulnessNumerator - number of users who found the review helpful
•	HelpfulnessDenominator - number of users who indicated whether they found the review helpful
•	Score - rating between 1 and 5
•	Time - timestamp for the review
•	Summary - brief summary of the review
•	Text - text of the review

So out of the 10 features for the reviews it can be seen that ‘score’, ‘summary’ and ‘text’ are the ones having some kind of predictive value. Also ‘text’ is kind of redundant as summary is sufficient to extract the sentiment hidden in the review. So it’s sufficient to load only these two from the sqlite data file. Score has a value between 1 and 5. So for the purpose of the project all reviews having score above 3 are encoded as positive and below or equal to 3 are encoded as negative. The mean of scores is 4.18. One should expect a distribution which has more positive than negative reviews. The data looks some thing like this.


(Figure 1)

After loading the data it is found that there are exactly 568454 number of reviews in the dataset. As expected after encoding the score the dataset got split into 124677 negative reviews and 443777 positive reviews.


(Figure 2)

From the label distribution one can conclude that the dataset is skewed as it has a large number of positive reviews and very few negative reviews. Positive reviews form 21.93 % of the dataset and negative reviews form 78.07 % of the dataset. This is an important piece of information as it already enables one to decide that a stratified strategy needs to be used for splitting data for evaluation.

4.	Preprocessing
To make the data more useful a number of preprocessing techniques are applied, most of them very common in text classification.

•	Stop words removal: stop words refer to the most common words in any language. They usually don’t have any predictive value and just increase the size of the feature set. Removing such words from the dataset would be very beneficial.
•	Lemmatization:  lemmatization is chosen over stemming. Although the goal of both stemming and lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form, better results were observed when using lemmatization instead of stemming. Explaining the difference between the two is a little out of the scope for this paper.
•	Punctuation Removal: refers to removing common punctuation marks such as !,?,”” etc. For eg: ‘Hi!’ and ‘Hi’ will be considered as two different words although they refer to the same thing.
•	Upper Case to Lower Case: convert all upper case letters to lower case letters.
•	Feature Reduction/Selection: This is the most important preprocessing step for sentiment classification. Classification algorithms are run on subset of the features, so selecting the right features becomes important. This step will be discussed in detail later in the report.

After applying all preprocessing steps except feature reduction/selection, 27048 unique words were obtained from the dataset which form the feature set. The next step is to try and reduce the size of the feature set by applying various Feature Reduction/Selection techniques. After preprocessing, the dataset is split into train and test, with test consisting of 25% samples of the entire dataset.

5.	Feature Reduction/Selection
The size of the dataset is essentially 568454*27048 which is quite a large number to be running any algorithm. Thus it becomes important to somehow reduce the size of the feature set. There are a number of ways this can be done.

a.	PCA
One can make use of application of principal component analysis (PCA) to reduce the feature set [3]. PCA is a procedure which uses orthogonal transformation to convert a set of variables in n-dimensional space to a smaller dimensional space. Consider an example in which points are distributed in a 2-d plane having maximum variance along the x-axis. One can fit these points in 1-d by squeezing all the points on the x axis. The x axis is the first principal component and the data has maximum variance along it. Something similar can be done for higher dimensions too.
For the purpose of the project, the feature set is reduced to 200 components using Truncated SVD which is a variant of PCA and works on sparse matrices.

b.	Most Frequent Features
Another way to reduce the number of features is to use a subset of the most frequent words occurring in the dataset as the feature set.
	Find the frequency of all words in the training data and select the most common 5000 words as features. The logic behind this approach is that all reviews must contain certain critical words that define the sentiment of the review and since it’s a reviews dataset these must occur very frequently. 5000 words are still quite a lot of features but it reduces the feature set to about 1/5th of the original which is still a workable problem. The frequency distribution for the dataset looks something like below.

(Figure 3)


From figure it is visible that words such as great, good, best, love, delicious etc occur most frequently in the dataset and these are the words that usually have maximum predictive value for sentiment analysis. This also proves that the dataset is not corrupt or irrelevant to the problem statement.

6.	Learning the models
4 models are trained on the training set and evaluated against the test set. Since the number of samples in the training set is huge it’s clear that it won’t be possible to run some inefficient classification algorithms like KNearest Neighbors or Random Forests etc. The 4 classifiers used in the project are:
•	Decision Tree Classifier
•	Naïve Bayes Classifier
•	Logistic Regression
•	Perceptron
The first problem that needs to be tackled is that most of the classification algorithms expect inputs in the form of feature vectors having numerical values and having fixed size instead of raw text documents (reviews in this case) of variable size. This can be tackled by using the Bag-of-Words strategy[2]. This strategy involves 3 steps:
•	Tokenization: breaking the document into tokens where each token represents a single word.
•	Counting: counting the frequency of each word in the document.
•	Normalization: weighing down or reducing importance of the words that occur the most in the corpus.

The reviews can be represented in the form of vectors of numerical values where each numerical value reflects the frequency of a word in that review.  These vectors are then normalized based on the frequency of tokens/words occurring in the entire corpus. Thus the entire set of reviews can be represented as a single matrix of rows where each row represents a review and each column represents a word in the corpus. This process is called Vectorization.

6.1	Using PCA
After applying vectorization and before applying any kind of feature reduction/selection the size of the input matrix is 426340*27048. After applying PCA to reduce features, the input matrix size reduces to 426340*200. The decision to choose 200 components is a consequence of running and testing the algorithms with different number of components.

6.1.1	Results
The models are trained on the input matrix generated above. Test data is also transformed in a similar fashion to get a test matrix. Following are the results:


(Figure 4)

From the results it can be seen that Decision Tree Classifier works best for the Dataset. This implies that the dataset splits pretty well on words, which is kind of obvious as meaning of words affects the sentiment of the review. Note that although the accuracy of Perceptron and BernoulliNB does not look that bad but if one considers that the dataset is skewed and contains 78% positive reviews, predicting the majority class will always give at least 78% accuracy. So compared to that perceptron and BernoulliNB doesn’t work that well in this case.
	Note that for skewed data recall is the best measure for performance of a model. The performance of all four models is compared below.
(Figure 5)
 As claimed earlier Perceptron and Naïve Bayes are predicting positive for almost all the elements, hence the recall and precision values are pretty low for negative samples precision/recall.
(Figure 6)

6.2	Using Max Frequency Words
The most important 5000 words are vectorized using Tf-idf transformer. Using the same transformer, the train and the test data are also vectorized. This essentially means that only those words of the training and testing data, which are among the most frequent 5000 words, will have numerical value in the generated matrices. These matrices are then used for training and evaluating the models.

6.2.1	Results
There is significant improvement in all the models. Following is a result summary.


(Figure 7)

One important thing to note about Perceptron is that it only converges when data is linearly separable. Since the number of features are so large one cannot tell if Perceptron will converge on this dataset. Thus restricting the maximum iterations for it is important. Following is a comparison of recall for negative samples.

(Figure 7)


(Figure 8)

6.3	Without Feature Reduction/Selection
Lastly the models are trained without doing any feature reduction/selection step. Decision Tree Classifier runs pretty inefficiently for datasets having large number of features, so training the Decision Tree Classifier is avoided.
	Since the entire feature set is being used, the sequence of words (relative order) can be utilized to do a better prediction. For example : some words when used together have a different meaning compared to their meaning when considered alone like “not good” or “not bad”.
	The models are trained for 3 strategies called Unigram, Bigram and Trigram.


6.3.1	Unigram
Unigram is the normal case, when each word is considered as a separate feature. The entire feature set is vectorized and the model is trained on the generated matrix. The size of the training matrix is 426340*27048 and testing matrix is 142114*27048.

6.3.1.1	Results
As expected accuracies obtained are better than after applying feature reduction or selection but the number of computations done is also way higher. Following are the accuracies:

(Figure 9)


(Figure 10)

All the classifiers perform pretty well and even have good precision and recall values for negative samples. Following shows a visual comparison of recall for negative samples:


(Figure 11)

6.3.2	Bigram
In this approach all sequence of adjacent words are also considered as features apart from Unigrams. So now 2 word phrases like “not good”, “not bad”, “pretty bad” etc will also have a predictive value which wasn’t there when using Unigrams. The entire feature set is vectorized and the model is trained on the generated matrix.  The size of the training matrix is 426340*263567 and testing matrix is 142114*263567

6.3.2.1	Results
The accuracies improved even further. The algorithms being used run well on sparse data which is the format of the input that is generated after vectorization. Following are the results:

(Figure 12)

There is a significant improvement on the recall of negative instances which might infer that many reviewers would have used 2 word phrases like “not good” or “not great” to imply a negative review. Following is the visual representation of the negative samples accuracy:


(Figure 13)


(Figure 14)

6.3.3	Trigram
In this all sequences of 3 adjacent words are considered as a separate feature apart from Bigrams and Trigrams.
The entire feature set is again vectorized and the model is trained on the generated matrix.  The size of the training matrix is 426340* 653393 and testing matrix is 142114* 653393.


6.3.3.1	Results
Trigrams give the best results.

(Figure 15)

Logistic Regression gives accuracy as high as 93.2 % and even perceptron accuracy is very high. The recall/precision values for negative samples are higher than ever.

(Figure 16)


(Figure 17)


6.4	Analysis
Since logistic regression performs best in all three cases, let’s do a little more analysis of it with the help of a confusion matrix. A confusion matrix plots the True labels against predicted labels. It is just a good way to visualize the classification report.


(Figure 18)

From the first matrix it is evident that a large number of samples were predicted to be positive and their actual label was also positive. Whereas very few negative samples which were predicted negative were also truly negative. But this matrix is not indicative of the performance because in testing data the negative samples were very less, so it is expected to see the predicted label vs true label part of the matrix for negative labels as lightly shaded. To visualize the performance better, it is better to look at the normalized confusion matrix. The normalized confusion matrix represents the ratio of predicted labels and true labels. Now one can see that logistic regression predicted negative samples accurately too.

7.	Future Work
It is evident that for the purpose of sentiment classification, feature reduction and selection are very important. Apart from the methods discussed in this paper there are other ways which can be explored to select features more smartly.
	One can utilize POS tagging mechanism to tag words in the training data and extract the important words based on the tags. For sentiment classification adjectives are the critical tags. One must take care of other tags too which might have some predictive value.
	Other advanced strategies such as using Word2Vec can also be utilized. Using Word2Vec, one can find similar words in the dataset and essentially find their relation with labels. There are other ways too in which one can use Word2Vec to improve the models.

8.	Conclusion
As a conclusion it can be said that bag-of-words is a pretty efficient method if one can compromise a little with accuracy. Also for datasets of such a large size it is advisable to use algorithms that run in linear time (like naïve bayes, although they might not give a very high accuracy).
	Finally, utilizing sequence of words is a good approach when the main goal is to improve accuracy of the model.


9.	References
[1] https://www.kaggle.com/snap/amazon-fine-food-reviews
[2] http://scikit-learn.org/stable/modules/feature_extraction.html
[3] https://en.wikipedia.org/wiki/Principal_component_analysis
[4] J. McAuley and J. Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. WWW, 2013

10.	Github link
You can find this paper and code for the project at the following github link.
	Amazon Fine Food Reviews: A Sentiment Classification Problem

	Abhishek Grover
	1503611
	abgrover@ucsc.edu



	Instructor: Jay Pujara
	jpujara+tim245@ucsc.edu
	1. Abstract
	The internet is full of websites that provide the ability to write reviews for products and services available online and offline. The websites like yelp, zomato, imdb etc got successful only through the authenticity and accuracy of the reviews they make available. Success of product selling websites such as Amazon, ebay etc also gets affected by the quality of the reviews they have for their products. All these sites provide a way to the reviewer to write his/her comments about the service or product and give a rating for it. Based on these comments one can classify each review as good or bad. From this data a model can be trained that can identify the sentiment hidden in a review. This has many possible applications: the learned model can be used to identify sentiments in reviews or data that doesn’t have any sentiment information like score or rating eg. People post comments about restaurants on facebook and twitter which do not provide any rating mechanism. This project intends to tackle this problem by employing text classification techniques and learning several models based on different algorithms such as Decision Tree, Perceptron, Naïve Bayes and Logistic regression. This paper will discuss the problems that were faced while performing sentiment classification on a large dataset and what can be done to solve those problems



	2. Introduction
	The main goal of the project is to analyze some large dataset and perform sentiment classification on it. Sentiment classification is a type of text classification in which a given text is classified according to the sentimental polarity of the opinion it contains. For the purpose of this project the Amazon Fine Food Reviews dataset, which is available on Kaggle, is being used. [1][4]

	Following sections describe the important phases of Sentiment Classification: the Exploratory Data Analysis for the dataset, the preprocessing steps done on the data, learning algorithms applied and the results they gave and finally the analysis from those results.

	3. Exploratory Data Analysis
	The Amazon Fine Food Reviews dataset is ~300 MB large dataset which consists of around 568k reviews about amazon food products written by reviewers between 1999 and 2012. Each review has the following 10 features:
	• Id
	• ProductId - unique identifier for the product
	• UserId - unqiue identifier for the user
	• ProfileName
	• HelpfulnessNumerator - number of users who found the review helpful
	• HelpfulnessDenominator - number of users who indicated whether they found the review helpful
	• Score - rating between 1 and 5
	• Time - timestamp for the review
	• Summary - brief summary of the review
	• Text - text of the review

	So out of the 10 features for the reviews it can be seen that ‘score’, ‘summary’ and ‘text’ are the ones having some kind of predictive value. Also ‘text’ is kind of redundant as summary is sufficient to extract the sentiment hidden in the review. So it’s sufficient to load only these two from the sqlite data file. Score has a value between 1 and 5. So for the purpose of the project all reviews having score above 3 are encoded as positive and below or equal to 3 are encoded as negative. The mean of scores is 4.18. One should expect a distribution which has more positive than negative reviews. The data looks some thing like this.



	(Figure 1)

	After loading the data it is found that there are exactly 568454 number of reviews in the dataset. As expected after encoding the score the dataset got split into 124677 negative reviews and 443777 positive reviews.


	(Figure 2)

	From the label distribution one can conclude that the dataset is skewed as it has a large number of positive reviews and very few negative reviews. Positive reviews form 21.93 % of the dataset and negative reviews form 78.07 % of the dataset. This is an important piece of information as it already enables one to decide that a stratified strategy needs to be used for splitting data for evaluation.

	4. Preprocessing
	To make the data more useful a number of preprocessing techniques are applied, most of them very common in text classification.

	• Stop words removal: stop words refer to the most common words in any language. They usually don’t have any predictive value and just increase the size of the feature set. Removing such words from the dataset would be very beneficial.
	• Lemmatization: lemmatization is chosen over stemming. Although the goal of both stemming and lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form, better results were observed when using lemmatization instead of stemming. Explaining the difference between the two is a little out of the scope for this paper.
	• Punctuation Removal: refers to removing common punctuation marks such as !,?,”” etc. For eg: ‘Hi!’ and ‘Hi’ will be considered as two different words although they refer to the same thing.
	• Upper Case to Lower Case: convert all upper case letters to lower case letters.
	• Feature Reduction/Selection: This is the most important preprocessing step for sentiment classification. Classification algorithms are run on subset of the features, so selecting the right features becomes important. This step will be discussed in detail later in the report.

	After applying all preprocessing steps except feature reduction/selection, 27048 unique words were obtained from the dataset which form the feature set. The next step is to try and reduce the size of the feature set by applying various Feature Reduction/Selection techniques. After preprocessing, the dataset is split into train and test, with test consisting of 25% samples of the entire dataset.

	5. Feature Reduction/Selection
	The size of the dataset is essentially 568454*27048 which is quite a large number to be running any algorithm. Thus it becomes important to somehow reduce the size of the feature set. There are a number of ways this can be done.

	a. PCA
	One can make use of application of principal component analysis (PCA) to reduce the feature set [3]. PCA is a procedure which uses orthogonal transformation to convert a set of variables in n-dimensional space to a smaller dimensional space. Consider an example in which points are distributed in a 2-d plane having maximum variance along the x-axis. One can fit these points in 1-d by squeezing all the points on the x axis. The x axis is the first principal component and the data has maximum variance along it. Something similar can be done for higher dimensions too.
	For the purpose of the project, the feature set is reduced to 200 components using Truncated SVD which is a variant of PCA and works on sparse matrices.

	b. Most Frequent Features
	Another way to reduce the number of features is to use a subset of the most frequent words occurring in the dataset as the feature set.
	Find the frequency of all words in the training data and select the most common 5000 words as features. The logic behind this approach is that all reviews must contain certain critical words that define the sentiment of the review and since it’s a reviews dataset these must occur very frequently. 5000 words are still quite a lot of features but it reduces the feature set to about 1/5th of the original which is still a workable problem. The frequency distribution for the dataset looks something like below.

	(Figure 3)


	From figure it is visible that words such as great, good, best, love, delicious etc occur most frequently in the dataset and these are the words that usually have maximum predictive value for sentiment analysis. This also proves that the dataset is not corrupt or irrelevant to the problem statement.

	6. Learning the models
	4 models are trained on the training set and evaluated against the test set. Since the number of samples in the training set is huge it’s clear that it won’t be possible to run some inefficient classification algorithms like KNearest Neighbors or Random Forests etc. The 4 classifiers used in the project are:
	• Decision Tree Classifier
	• Naïve Bayes Classifier
	• Logistic Regression
	• Perceptron
	The first problem that needs to be tackled is that most of the classification algorithms expect inputs in the form of feature vectors having numerical values and having fixed size instead of raw text documents (reviews in this case) of variable size. This can be tackled by using the Bag-of-Words strategy[2]. This strategy involves 3 steps:
	• Tokenization: breaking the document into tokens where each token represents a single word.
	• Counting: counting the frequency of each word in the document.
	• Normalization: weighing down or reducing importance of the words that occur the most in the corpus.

	The reviews can be represented in the form of vectors of numerical values where each numerical value reflects the frequency of a word in that review. These vectors are then normalized based on the frequency of tokens/words occurring in the entire corpus. Thus the entire set of reviews can be represented as a single matrix of rows where each row represents a review and each column represents a word in the corpus. This process is called Vectorization.

	6.1 Using PCA
	After applying vectorization and before applying any kind of feature reduction/selection the size of the input matrix is 42634027048. After applying PCA to reduce features, the input matrix size reduces to 426340200. The decision to choose 200 components is a consequence of running and testing the algorithms with different number of components.

	6.1.1 Results
	The models are trained on the input matrix generated above. Test data is also transformed in a similar fashion to get a test matrix. Following are the results:


	(Figure 4)

	From the results it can be seen that Decision Tree Classifier works best for the Dataset. This implies that the dataset splits pretty well on words, which is kind of obvious as meaning of words affects the sentiment of the review. Note that although the accuracy of Perceptron and BernoulliNB does not look that bad but if one considers that the dataset is skewed and contains 78% positive reviews, predicting the majority class will always give at least 78% accuracy. So compared to that perceptron and BernoulliNB doesn’t work that well in this case.
	Note that for skewed data recall is the best measure for performance of a model. The performance of all four models is compared below.
	(Figure 5)
	As claimed earlier Perceptron and Naïve Bayes are predicting positive for almost all the elements, hence the recall and precision values are pretty low for negative samples precision/recall.
	(Figure 6)

	6.2 Using Max Frequency Words
	The most important 5000 words are vectorized using Tf-idf transformer. Using the same transformer, the train and the test data are also vectorized. This essentially means that only those words of the training and testing data, which are among the most frequent 5000 words, will have numerical value in the generated matrices. These matrices are then used for training and evaluating the models.

	6.2.1 Results
	There is significant improvement in all the models. Following is a result summary.


	(Figure 7)

	One important thing to note about Perceptron is that it only converges when data is linearly separable. Since the number of features are so large one cannot tell if Perceptron will converge on this dataset. Thus restricting the maximum iterations for it is important. Following is a comparison of recall for negative samples.

	(Figure 7)


	(Figure 8)

	6.3 Without Feature Reduction/Selection
	Lastly the models are trained without doing any feature reduction/selection step. Decision Tree Classifier runs pretty inefficiently for datasets having large number of features, so training the Decision Tree Classifier is avoided.
	Since the entire feature set is being used, the sequence of words (relative order) can be utilized to do a better prediction. For example : some words when used together have a different meaning compared to their meaning when considered alone like “not good” or “not bad”.
	The models are trained for 3 strategies called Unigram, Bigram and Trigram.




	6.3.1 Unigram
	Unigram is the normal case, when each word is considered as a separate feature. The entire feature set is vectorized and the model is trained on the generated matrix. The size of the training matrix is 42634027048 and testing matrix is 14211427048.

	6.3.1.1 Results
	As expected accuracies obtained are better than after applying feature reduction or selection but the number of computations done is also way higher. Following are the accuracies:

	(Figure 9)



	(Figure 10)

	All the classifiers perform pretty well and even have good precision and recall values for negative samples. Following shows a visual comparison of recall for negative samples:



	(Figure 11)

	6.3.2 Bigram
	In this approach all sequence of adjacent words are also considered as features apart from Unigrams. So now 2 word phrases like “not good”, “not bad”, “pretty bad” etc will also have a predictive value which wasn’t there when using Unigrams. The entire feature set is vectorized and the model is trained on the generated matrix. The size of the training matrix is 426340263567 and testing matrix is 142114263567

	6.3.2.1 Results
	The accuracies improved even further. The algorithms being used run well on sparse data which is the format of the input that is generated after vectorization. Following are the results:

	(Figure 12)

	There is a significant improvement on the recall of negative instances which might infer that many reviewers would have used 2 word phrases like “not good” or “not great” to imply a negative review. Following is the visual representation of the negative samples accuracy:


	(Figure 13)


	(Figure 14)

	6.3.3 Trigram
	In this all sequences of 3 adjacent words are considered as a separate feature apart from Bigrams and Trigrams.
	The entire feature set is again vectorized and the model is trained on the generated matrix. The size of the training matrix is 426340* 653393 and testing matrix is 142114* 653393.


	6.3.3.1 Results
	Trigrams give the best results.

	(Figure 15)

	Logistic Regression gives accuracy as high as 93.2 % and even perceptron accuracy is very high. The recall/precision values for negative samples are higher than ever.

	(Figure 16)


	(Figure 17)



	6.4 Analysis
	Since logistic regression performs best in all three cases, let’s do a little more analysis of it with the help of a confusion matrix. A confusion matrix plots the True labels against predicted labels. It is just a good way to visualize the classification report.


	(Figure 18)

	From the first matrix it is evident that a large number of samples were predicted to be positive and their actual label was also positive. Whereas very few negative samples which were predicted negative were also truly negative. But this matrix is not indicative of the performance because in testing data the negative samples were very less, so it is expected to see the predicted label vs true label part of the matrix for negative labels as lightly shaded. To visualize the performance better, it is better to look at the normalized confusion matrix. The normalized confusion matrix represents the ratio of predicted labels and true labels. Now one can see that logistic regression predicted negative samples accurately too.

	7. Future Work
	It is evident that for the purpose of sentiment classification, feature reduction and selection are very important. Apart from the methods discussed in this paper there are other ways which can be explored to select features more smartly.
	One can utilize POS tagging mechanism to tag words in the training data and extract the important words based on the tags. For sentiment classification adjectives are the critical tags. One must take care of other tags too which might have some predictive value.
	Other advanced strategies such as using Word2Vec can also be utilized. Using Word2Vec, one can find similar words in the dataset and essentially find their relation with labels. There are other ways too in which one can use Word2Vec to improve the models.

	8. Conclusion
	As a conclusion it can be said that bag-of-words is a pretty efficient method if one can compromise a little with accuracy. Also for datasets of such a large size it is advisable to use algorithms that run in linear time (like naïve bayes, although they might not give a very high accuracy).
	Finally, utilizing sequence of words is a good approach when the main goal is to improve accuracy of the model.


	9. References
	[1] https://www.kaggle.com/snap/amazon-fine-food-reviews
	[2] http://scikit-learn.org/stable/modules/feature_extraction.html
	[3] https://en.wikipedia.org/wiki/Principal_component_analysis
	[4] J. McAuley and J. Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. WWW, 2013

	10. Github link
	You can find this paper and code for the project at the following github link.