Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression\n",
    "### Author: _Calvin Chi_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regression analysis is a statistical procedure of building a model to estimate the relationship between variables. In this tutorial, we will cover two types of regression models - linear regression and logistic regression. We will start with linear regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression\n",
    "In linear regression, we assume that the true relationship between independent variables and the dependent variable follows the relationship:\n",
    "\n",
    "$$y = X\\beta + \\epsilon$$\n",
    "\n",
    "Where $y \\in \\mathbb{R}^{m x 1}$, $X \\in \\mathbb{R}^{m x n}$, $\\beta \\in \\mathbb{R}^{n x 1}$, and $\\epsilon  \\stackrel{iid}{\\sim} N(0, \\sigma)$. Typically we do not know what the standard deviation $\\sigma$ of the residual is, but it is also not required for estimating the coefficients $\\beta$ relating the relationship between $X$ and $y$. \n",
    "\n",
    "This model has a few statistical properties: \n",
    "\n",
    "(1) $E(y) = E(X\\beta) + E(\\epsilon) = X\\beta$    \n",
    "(2) $y \\sim N(X\\beta, \\sigma)$ because $Var(y) \\sim Var(X\\beta) + Var(\\epsilon) = Var(\\epsilon) = \\sigma^{2}$. $Var(X\\beta) = 0$ because $X\\beta$ are fixed points, not random variables.     \n",
    "(3) $\\hat{\\beta}$ is a consistent estimator according to $P( |\\hat{\\beta} - \\beta| ) > \\epsilon) \\rightarrow 0$\n",
    "\n",
    "#### _Consistency Proof_\n",
    "(1) $E(\\hat{\\beta}) = \\beta $   \n",
    "(2) In general, If $E[(X_{n} - X)^{2}] \\rightarrow 0$, then $P(|X_{n} - X| > \\epsilon) \\rightarrow 0$ because $P(|X_{n} - X|^{2} > \\epsilon^{2}) \\le \\frac{E[(X_{n} - X)^{2}]}{\\epsilon^{2}} = 0$ according to Markov's inequality.   \n",
    "(3) $Cov(\\hat{B}) = (X^{T}X)^{-1}\\sigma^{2} = Var(\\hat{B})$ because:   \n",
    "> If the loss function is defined as $l = \\sum_{i=1}^{n}(y_{i} - \\beta x_{i})^{2}$   \n",
    "> Then $\\hat{B} = (X^{T}X)^{-1}X^{T}y$   \n",
    "> $Cov(\\hat{B}) = (X^{T}X)^{-1}X^{T}Cov(\\epsilon)X(X^{T}X)^{-1} = (X^{T}X)^{-1}(X^{T}X)Cov(\\epsilon)(X^{T}X)^{-1} = \\sigma^{2}(X^{T}X)^{-1}$    \n",
    "> $Cov(\\hat{B}) = Var(\\hat{B})$ because $\\sigma^{2}$ cross-diagonal terms are zero. $\\sigma^{2}$'s cross-diagonal terms are zero because $\\epsilon$ are independently and identically distributed.   \n",
    "\n",
    "(4) If $Z = c + AY$, where only $Y$ is random, then $Cov(Z) = ACov(Y)A^{T}$\n",
    "\n",
    "$E[(\\hat{B} - B)^{2}] = E[(\\hat{B} - E(\\hat{B}))^{2}] = Var(\\hat{B})$ by definition of variance.   \n",
    "Since $X \\in \\mathbb{R}^{nx1}$, $X^{T}X$ is a scalar, and $\\rightarrow \\infty$ as $n \\rightarrow \\infty$, hence   \n",
    "$Cov(\\hat{B}) = Var(\\hat{B}) = (X^{T}X)^{-1}\\sigma^{2} \\rightarrow 0$, and $E[(\\hat{B} - B)^{2}] \\rightarrow 0$ as $n \\rightarrow \\infty$, and thus $P( |\\hat{\\beta} - \\beta| ) > \\epsilon) \\rightarrow 0$. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization\n",
    "An intuitive way of evaluating the fitness of our model is to quanitfy the difference between predicted value and actual value $y - \\hat{y}$. A common way to do so is to sum the squared differences of all the predicted output, defined as \n",
    "\n",
    "$$L = \\sum_{i=1}^{n}(y^{i} - \\hat{y}^{i})^{2}$$\n",
    "\n",
    "Where $y^{i}$ represents the actual value of sample $i$. Our goal is to find $\\beta$ that minimizes loss, which is sometimes called the objective function. One way to minimize this loss is by finding $\\beta$ such that the first derivative of the loss is equal to zero. Let us do that now in matrix notation:\n",
    "\n",
    "$$L = (y - \\hat{y})^{T}(y - \\hat{y})$$\n",
    "$$\\frac{d(L)}{d\\beta} = -2X^{T}(y - X\\beta) = -X^{T}y + X^{T}X\\beta = 0$$\n",
    "$$\\hat{\\beta} = (X^{T}X)^{-1}X^{T}y$$\n",
    "\n",
    "We can verify that the $\\hat{\\beta}$ that we have found is the $\\hat{\\beta}$ that minimizes $L$ by the second derivative test:\n",
    "\n",
    "$$\\frac{d^{2}(L)}{d\\beta^{2}} = X^{T}X $$\n",
    "\n",
    "Which ensures that the diagonal terms, and thus all the appropriate second derivative terms, are positive. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In our implementation below we will use linear regression to predict median home value in a given Census area. There are a total of 8 features for each data point. First, let us load up this dataset and examine the design matrix $X$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(19440, 8)\n",
      "(19440, 1)\n"
     ]
    }
   ],
   "source": [
    "from scipy import io\n",
    "import numpy as np\n",
    "from numpy.linalg import inv\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "data = io.loadmat('datasets/housing_data.mat')\n",
    "xtrain = data['Xtrain']\n",
    "ytrain = data['Ytrain']\n",
    "xvalidate = data['Xvalidate']\n",
    "yvalidate = data['Yvalidate']\n",
    "print(xtrain.shape)\n",
    "print(ytrain.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The design matrix $X$ is a $19440 x 8$ matrix. We now add the constant terms of one so that the model estimates $\\beta_{0}$ values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(19440, 9)\n",
      "(1200, 9)\n"
     ]
    }
   ],
   "source": [
    "m1 = xtrain.shape[0]\n",
    "m2 = xvalidate.shape[0]\n",
    "xtrain = np.hstack((np.ones((m1, 1)), xtrain))\n",
    "xvalidate = np.hstack((np.ones((m2, 1)), xvalidate))\n",
    "print(xtrain.shape)\n",
    "print(xvalidate.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now find our $\\beta$ values using our least squares solution: \n",
    "\n",
    "$$\\hat{\\beta} = (X^{T}X)^{-1}X^{T}y$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ -3.56686439e+06]\n",
      " [  4.05879986e+04]\n",
      " [  1.19561189e+03]\n",
      " [ -8.50145688e+00]\n",
      " [  1.18352188e+02]\n",
      " [ -3.77900280e+01]\n",
      " [  4.30562637e+01]\n",
      " [ -4.21794075e+04]\n",
      " [ -4.24573474e+04]]\n"
     ]
    }
   ],
   "source": [
    "coef = np.dot(np.dot(inv(np.dot(np.transpose(xtrain), xtrain)), \n",
    "        np.transpose(xtrain)), ytrain)\n",
    "print(coef)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEZCAYAAABB4IgrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYXWV99vHvzfkgiWAFagICQgQURVCsoDIFSUA0gFSM\nVYlKtRZaUKMvRL0grX2vCopGXwWLIgQKRsRSUTGTIIwWy1k5GZSogAmHWAgElIhJuN8/1jOwGWYm\ne2f2mr0zuT/Xta9Z+1mn316E/dvPYT1LtomIiKjTBp0OICIixr4km4iIqF2STURE1C7JJiIiapdk\nExERtUuyiYiI2iXZRLSRpJmSzul0HK2S9K+S/lfS/eX9UZJ+J+kxSXtLukPSG5s4zuOSdqo73lj3\nKPfZRDeRdA+wLbAK+APQC5xg+4lOxtUNJG0P/CvwZmBL4D7gW8AZtleM4Lg7AL8CdrD9cCn7NfBh\n298fceBrF9N5wGLbp3bi/NF+qdlEtzFwuO1xwN7Aq4CZdZxI0jrz71/S1sC1wKbAa22PBw4BxgMv\nGeHhXww81J9oGsoWjvC4EU9bZ/5ni/WKAGz/nqpms/fTK6RNJH1O0r2SHpB0lqRNG9b/H0n3S1oi\n6ThJT0napaw7r2z/A0mPAz3DHU/SCyR9T9Ijkh6W9OOG85xczvGYpDsl/XUpP03ShQ3bTS1NUMsk\nXSVp94Z1d0uaIenWco5vStpkiGsyA3jM9ntsLy7X5z7bH7V9Rzne/pJuKMe6XtLrGs41TtLXy7VZ\nLOnTqhwMzAdeVD7LReXabADcJmlRQ6wHleUNJH1C0q8lLZd0o6QJZV3j9R7u2h5Y4viopKWS7pP0\n3rLuA8C7gP9TYvrucNc81g1JNtG1JE0EDgMWNRSfDuwKvKL8nQCcWrY/FPgwcFBZ10NVU2r0TuDT\ntrcCfjrc8ai+4BcDL6Bq2vtEOc8k4ARg31IDmwLc03AON2x3MXAi8ELgh8D3JG3UsO3bgcnAzsAr\ngfcOcTkOBv5ziHX9NZ/vA7NLvF8AflDKAeYAfwZ2oaotHgL8ne0fUV3j+22Ps/2ucm0E7GV7t0FO\nNwN4B3BoqWG9H+hv5my83sNdW4Dtga2AFwF/B3xF0njbXwMuomoeHGf7iCaueXS5JJvoRv8l6THg\nd8BSYFbDug8AH7G93PYfgc9QJRCovrjPs/1L238asF+/79q+DsD2k2s43krgL4Gdba+2/dNSvhrY\nBHi5pI1s/8723YOc6xjg+7avsr0a+BywObB/wzZftL3U9qPA92ioxQ3wAuCBIdYBHA7cZfti20/Z\nngv8EnirpG2pEspHbP/J9kNUSemdwxwPSg1zEMcBn7T9awDbt9t+ZJB9hru2UCW/T5dr+0OqPrqX\nDnHOZq95dKkkm+hGR5RfrwcCuwN/ASDphcAWwM2lWWoZVW3hBWW/F1HVRPot5rlfmE+vb+J4nwV+\nA8wvTUYnA9j+DVUNahawVNLFpfN+oBcB9/a/cTUaZzHVL/x+SxuWnwCeN8Q1eZgq8Q3lWecq7i3n\nejGwMfBA+ZyPAF+lXNe1sAPw2+E2aOLaAjxs+6mG90N+/iGu+XDXI7pMkk10o/4+m/+mav45s5Q/\nRPWF9DLb25TX80tTDlS//Cc2HGdHntuM1vh+2OPZ/oPtj9l+CTAV+Gh/P4HtubbfQPVFDlWT0UD3\nN6zvtwOwpIlrMNCVwFHDrL8f2GlA2Y5UI9YWA38CXlA+49blc75iLeKgHG9NgxLW9N9qTZ4zTHaQ\na/6ZpiOOjkuyiW43GzhE0l6lZvA1YHb55YykCZIml20vAd4naXdJWwCfGu7AazqepMMl9X+pPk41\nHPspSZMk/XXpzP8zsAJ46rln4BLg8LLtRpI+RvWlf+1aXIfPA+MkzZG0Y0OsZ0p6OXAFsJukaZI2\nlPQOYA+qZrwHqQYBfEHSVmVgwC5q4r6ZIXwd+LSkXUscezX0DQFrvrZNWErVv0TZt9lrHl0qySa6\nzbN+0Zb+hTk807F8CvBr4DpJj1J9iU4q284DvgRcDdzFM1/qTw5zvpOHOh6wG3BlGZ31U+Artn9M\nNfz4M8D/UtUoXsggw7Nt3wW8G/hy2fZw4K22Vw32WYdT+kT2p+pHul7ScmAB8Cjwa9vLgLcAH6Oq\nVXyMagj5snKIY6n6PBYCy4BvU3XQD3nKYd5/niqRzi9xfJ2qL2rgdsNd2zWd81zgZaUJ7j9L7Gu8\n5tG9On5Tp6TxVP9YX071S+X9VF8U36KqLt8DHGN7edl+ZtlmFXCS7fmlfB/gfGAz4ArbHy7lmwAX\nAPtS/U/4Dtu/G6WPFx2kapjx7cCmA/oGImKUdUPN5otUyWEPqqGfv6T69Xql7ZcCV1F+wUjak2qE\nzx5Uo2vOktTfAXw2cJztScAkSVNK+XHAsjKEczZwxuh8rOgESUeW+zu2pupHuTyJJqLzOppsJI0D\n3mD7PADbq0oN5giqphPK3yPL8lRgbtnuHqr7L/YrI4G2sn1j2e6Chn0aj3Up1f0KMXb9PfB7qn8b\nK4HjOxtORABstOZNarUz8JCqeZBeCdxENbxxO9tLAWw/WO4TgGoYZ2Pn6n2lbBXPHuGzhGeGl06g\nDHe1vVrSo5K2aWjLjjHE9mGdjiEinqvTzWgbAftQdbzuA/yRqgltuM7JkRrqRrWIiKhJp2s2S6hm\ndr2pvP8OVbJZKmk720tLE9nvy/r7qO5T6DexlA1V3rjP/ZI2BMYNVquRlOmvIyLWgu01/ojvaM2m\nNJUtLvMeQdWf8gvgcp6ZI2o68N2yfDkwrXQA70w139IN5T6C5ZL2KwMGjh2wz/Sy/HaqAQdDxdN1\nr9NOO63jMSSmxLQ+xpWYmns1q9M1G6gmKbxI0sZUU2C8D9gQuETS+6mm3DgGwPZCSZdQ3SuwEjje\nz3zaE3j20Od5pfxc4EJVs9c+DEwblU8VERFP63iysX0r8JpBVr1piO3/Dfi3QcpvBvYapPxJSrKK\niIjO6PQAgViDnp6eTofwHImpOYmped0YV2Jqr47PINAtJDnXIiKiNZJwtw8QiIiI9UOSTURE1C7J\nJiIiapdkExERtUuyiYiI2iXZRERE7ZJsIiKidkk2ERFRuySbiIioXZJNRETULskmIiJql2QTERG1\nS7KJiIjaJdlERETtkmwiIqJ2STYREVG7JJuIiKhdkk1ERNQuySYiImqXZBMREbVLsomIiNol2URE\nRO2SbCIionZJNhERUbskm4iIqF2STURE1K4rko2kDST9TNLl5f3WkuZL+pWkXknjG7adKWmRpDsl\nTW4o30fSbZLukjS7oXwTSXPLPtdK2nF0P11ERHRFsgFOAhY2vD8FuNL2S4GrgJkAkvYEjgH2AA4D\nzpKkss/ZwHG2JwGTJE0p5ccBy2zvBswGzqj7w7RDb28vkycfzeTJR9Pb29vpcCIiRqTjyUbSRODN\nwNcbio8A5pTlOcCRZXkqMNf2Ktv3AIuA/SRtD2xl+8ay3QUN+zQe61Lg4Do+Rzv19vZy1FHTWbBg\nKgsWTOWoo6Yn4UTEOq3jyQb4AvBxwA1l29leCmD7QWDbUj4BWNyw3X2lbAKwpKF8SSl71j62VwOP\nStqmzZ+hrc488xxWrDgdmA5MZ8WK0znzzHM6HVZExFrraLKRdDiw1PYtgIbZ1MOsa/m0bTxWREQ0\nYaMOn/8AYKqkNwObA1tJuhB4UNJ2tpeWJrLfl+3vA3Zo2H9iKRuqvHGf+yVtCIyzvWywYGbNmvX0\nck9PDz09PSP7dGtpxowPcs0101mxonq/+eYnM2PGnOF3iogYBX19ffT19bW8n+x2VhrWnqQDgRm2\np0o6A3jY9umSTga2tn1KGSBwEfBaquaxBcButi3pOuBE4EbgB8CXbM+TdDzwctvHS5oGHGl72iDn\nd7dcC6j6bfqbzmbM+CBTpkxZwx4REaNPErbX2GLUrclmG+ASqhrJvcAxth8t282kGmG2EjjJ9vxS\nvi9wPrAZcIXtk0r5psCFwKuAh4FpZXDBwPN3VbKJiFgXrHPJptOSbCIiWtdssumG0WgRETHGJdlE\nRETtkmwiIqJ2STYREVG7JJuIiKhdkk1ERNQuySYiImqXZBMREbVLsomIiNol2URERO2SbCIionZJ\nNhERUbskm4iIqF1LyUbSBpLG1RVMRESMTWtMNpIuljRO0pbAHcBCSR+vP7SIiBgrmqnZ7Gn7MeBI\n4IfAzsB7ao0qIiLGlGaSzcaSNqZKNpfbXllzTBERMcY0k2z+HbgH2BL4iaQXA8vrDCoiIsaWNT4W\nWtLOtu9ueC9gV9uL6g5uNOWx0BERrWvnY6G/0/imfCPPXdvAIiJi/bPRUCsk7Q68DBgv6W0Nq8YB\nm9UdWEREjB1DJhvgpcBbgOcDb20ofxz4QJ1BRUTE2NJMn83rbF87SvF0TPpsIiJa184+m4cl/UjS\nHeXAr5D0qRFHGBER641mks3XgJnASgDbtwHT6gwqIiLGlmaSzRa2bxhQtqqOYCIiYmxqJtk8JOkl\ngAEk/Q3wQK1RRUTEmNLMAIFdgHOA/YFHgLuBd9u+p/boRlEGCEREtK7ZAQJrTDYNB9wS2MD24yMN\nrhsl2UREtK5to9EkbSrpb4GTgI9IOlXSqW0KcqKkqyT9QtLtkk4s5VtLmi/pV5J6JY1v2GempEWS\n7pQ0uaF8H0m3SbpL0uyG8k0kzS37XCtpx3bEHhERzWumz+a7wBFUgwL+2PBqh1XAR22/DHgdcEKZ\nueAU4ErbLwWuohoNh6Q9gWOAPYDDgLPKXG0AZwPH2Z4ETJI0pZQfByyzvRswGzijTbFHRESThptB\noN9E24fWcXLbDwIPluU/SLoTmEiV3A4sm80B+qgS0FRgru1VwD2SFgH7SboX2Mr2jWWfC6geidBb\njnVaKb8U+HIdnyUiIobWTM3mfyTtVXcgknYC9gauA7azvRSeTkjbls0mAIsbdruvlE0AljSULyll\nz9rH9mrgUUnb1PIhIiJiUM3UbF4PvFfS3cCTgKgmf35Fu4KQ9DyqWsdJpYYzsKe+nT33Q3ZkzZo1\n6+nlnp4eenp62njaiIh1X19fH319fS3v18zQ5xcPVm773pbPNvjxNwK+D/zQ9hdL2Z1Aj+2lkrYH\nrra9h6RTqlP79LLdPKomsnv7tynl04ADbf9D/za2r5e0IfCA7W0HiSOj0SIiWjTi0WiSxpXFx4d4\ntcs3gIX9iaa4HHhvWZ5ONUihv3xaGWG2M7ArcENpalsuab8yYODYAftML8tvpxpwEBERo2jImo2k\n79t+S2k+M89ufrLtXUZ8cukA4CfA7eUcBj4B3ABcAuxAVWs5xvajZZ+ZVCPMVlI1u80v5fsC51M9\na+cK2yeV8k2BC4FXAQ8D0wa7ITU1m4iI1rX9ps6xLskmIqJ17byp86gBN1U+X9KRIw0wIiLWH80M\nELjF9t4Dyn5u+1W1RjbKUrOJiGhdOx+eNtg2zQyZjoiIAJpLNjdJ+rykl5TX54Gb6w4sIiLGjmaS\nzT8Bfwa+VV5PAifUGVRERIwtGY1WpM8mIqJ1zfbZDNn3Imm27Q9L+h6DTBdje+oIY4yIiPXEcB39\nF5S/nxuNQCIiYuwaLtl8FjgYeLPtk0cpnoiIGIOGSzZ/KWl/YKqkuQyYLdn2z2qNLCIixozh5kb7\nG6o5yF4P3Mhz50Y7qP7wRk8GCEREtG7EAwSopuI/TNKptv+ljbFFRMR6Zrj7bL5U/mYetIiIGJHh\najYrJZ0DTJD0pYErbZ9YX1gRETGWDJds3gK8CZhCpqeJiIgRaGbW51favnWU4umYDBCIiGhdO2d9\nXiHpR5LuKAd+haRPjTjCiIhYbzSTbL4GzKR6DDO2bwOm1RlURESMLc0kmy1s3zCgbFUdwURExNjU\nTLJ5SNJLKJNxlps9H6g1qoiIGFOaGSCwC3AOsD/wCHA38C7b99Yf3ujJAIGIiNY1O0Cg6efZSNoS\n2MD24yMNrhsl2UREtK5to9EkjS+Pgv4xcLWkMyWNb0eQERGxfmimz+YbwOPAMeX1GHBenUFFRMTY\n0kyfzS22915T2bouzWgREa1r902dr2848AHAipEEFxER65fh5kbr9yHggoZ+mkeA99YWUUREjDmt\njEYbB2D7sVoj6pA0o61Zb28vZ555DgAzZnyQKVOmdDiiiOi0EQ99lvRRYLntcweUHwdsZXt2WyId\nJZIOBWZTNR2ea/v0AeuTbIbR29vLUUdNZ8WK6rJtvvnJXHbZnCSciPVcO/ps3gVcMEj5hcD71zaw\nTpC0AfBlqsclvAx4p6TdOxvVuuXMM88piWY6UCWd/lpOJ/X29jJ58tFMnnw0vb29nQ4HSEyt6Na4\noga2B30Btw6z7vah1nXjC/gr4IcN708BTh6wjWNohxzyNsP5BpfX+T7kkLd1NKZ58+Z58823K3Gd\n7803387z5s1LTOtATN0cV7SmfHeu+Xt4yBVwO7DdIOXbrYPJ5mjgnIb37wa+NGCbkV3xMa4bvxi6\nMQEmpnU/rmhNs8lmuNFonwV+IGkG8LNStm8p/1w7alXdZtasWU8v9/T00NPT07FYus2UKVO47LI5\nDQME0l8TsT7q6+ujr6+v9R2Hy0TAYVTT1DwMPFSWD2smi3XTi6oZbV7D+zSjjQHdWNtKTOt+XNEa\nmqzZND30eV0maUPgV8DBVI9HuAF4p+07G7bx+nAtxppuHI6dmJrXrXFF89o+6/O6rgx9/iLPDH3+\nzID1STYRES1KsmlRkk1EROvaOTdaRETEiKxxbjRJm1INHd6pcXvb/1JfWBERMZY0MxHnd4HlwM3A\nk/WGExERY1EzyWai7UNrjyQiIsasZvps/kfSXrVHEhERY1YzT+pcCOwK3E3VjCaqm3heUX94oyej\n0SIiWtfsaLRmmtEOa0M8ERGxHmvqPhtJrwTeUN7+t+1ba42qA1KziYhoXdvus5F0EnARsG15/Yek\nfxp5iBERsb5ops/mNuB1tv9Y3m8JXJs+m4iIaOcMAgJWN7xfXcoiIiKa0swAgfOA6yVdVt4fCZxb\nX0gRETHWNDtAYB/g9eXtf9v+ea1RdUCa0SIiWjfiWZ8ljbP9mKRtBltve9kIY+wqSTYREa1rR7L5\nvu23SLobaNyo/6bOXdoTandIsomIaF2eZ9OiJJuIiNa18z6bA8pwZyS9W9LnJe3YjiAjImL90MzQ\n57OBJ8osAjOA3wAX1hpVRESMKc0km1WlfekI4Mu2vwJsVW9YERExljRzn83jkmYC7wbeKGkDYON6\nw4qIiLGkmZrNO6geLXCc7QeBicBna40qIiLGlGbmRtsS+JPt1ZImAbsDP7S9cjQCHC0ZjRYR0bp2\nzo32E2BTSROA+cB7gPNHFl5ERKxPmpqI0/YTwNuAs2y/HXh5vWFFRMRY0lSykfQ64F3AD1rYLyIi\nAmguaXwYmAlcZvsXknYBrq43rIiIGEuanq5G0halOW1MygCBiIjWtXO6mtdJWgj8srx/paSz2hBj\nRESsJ5ppRpsNTAEeBrB9K/DGkZ5Y0hmS7pR0i6TvSBrXsG6mpEVl/eSG8n0k3SbpLkmzG8o3kTS3\n7HNt49xtkqaX7X8l6diRxh0REa1rqqPf9uIBRasH3bA184GX2d4bWETVL4SkPYFjgD2Aw4CzJPVX\n0c6murl0EjBJ0pRSfhywzPZuVMnxjHKsrYFTgdcArwVOkzS+DbFHREQLmkk2iyXtD1jSxpI+Btw5\n0hPbvtL2U+XtdVQzEwBMBebaXmX7HqpEtJ+k7YGtbN9YtruA6hHVUM3bNqcsXwocVJanAPNtL7f9\nKFWCO3SksUdERGuaSTYfAk4AJgD3AXuX9+30fuCKsjwBaKxJ3VfKJgBLGsqXlLJn7WN7NbC8PGF0\nqGNFRMQoGnYiTkkbAu+x/a61ObikBcB2jUVUT/38pO3vlW0+Cay0/c21OcdQp16bnWbNmvX0ck9P\nDz09PW0KJyJibOjr66Ovr6/l/ZqZG+1G269Zy7jWdOz3Ah8ADrL9ZCk7heqx06eX9/OA04B7gatt\n71HKpwEH2v6H/m1sX18S5AO2ty3b9Nj+UNnnq+UY3xoklgx9johoUTvnRrtG0pclvaGMBttH0j5t\nCPBQ4OPA1P5EU1wOTCsjzHYGdgVuKDNOL5e0XxkwcCzw3YZ9ppfltwNXleVe4BBJ48tggUNKWURE\njKJmajaDzRZg2wcNUt78iaVFwCaUIdXAdbaPL+tmUo0wWwmcZHt+Kd+XahLQzYArbJ9Uyjelenro\nq8rxppXBBf21p09SNd/9q+0LhognNZuIiBY1W7NpegaBsS7JJiKidc0mmzU+qVPSRwcpXg7cbPuW\ntQkuIiLWL800o10MvBr4Xil6C3AbsBPwbdtn1BngaEnNJiKidW1rRpP0E+DNtv9Q3j+P6lEDh1LV\nbvZsQ7wdl2QTEdG6do5G2xZoHC22EtjO9ooB5REREYNaY58NcBFwvaT+YcZvBS6WtCWwsLbIIiJi\nzGhqNJqkVwMHlLc/tX1TrVF1QJrRIiJa185mNKjua3nM9heBe8vNlhEREU1pZoDAaVSj0V5qe5Kk\nF1GNQjtg2B3XManZRES0rp01m6Oopv3/I4Dt+4GtRhZeRESsT5pJNn8uP/kNUAYGRERENK2ZZHOJ\npH8Hni/pA8CVwNfrDSsiIsaSZkejHQJMpnpOTK/tBXUHNtrSZxMR0braJuKUtAHwTtsXrW1w3SjJ\nJiKidSMeICBpnKSZ5Vk2k1X5R+C3wDHtDDYiIsa2IWs2ZcaAR4BrgYOppq0R1fNlxtxsz6nZRES0\nbsTNaJJut71XWd4QeADY0faf2hppl0iyiYhoXTvus1nZv2B7NbBkrCaaiIio13A1m9WUGzmpms82\nB54oy7Y9blQiHCWp2UREtG7ET+q0vWF7Q4qIiPVVsxNxRkRErLUkm4iIqF2STURE1C7JJiIiapdk\nExERtUuyiYiI2iXZRERE7ZJsIiKidkk2ERFRu44nG0kzJD0laZuGspmSFkm6U9LkhvJ9JN0m6S5J\nsxvKN5E0t+xzraQdG9ZNL9v/StKxo/fJIiKiX0eTjaSJwCHAvQ1le1A9L2cP4DDgLEn98+6cDRxn\nexIwSdKUUn4csMz2bsBs4IxyrK2BU4HXAK8FTpM0vvYPFhERz9Lpms0XgI8PKDsCmGt7le17gEXA\nfpK2B7ayfWPZ7gLgyIZ95pTlS4GDyvIUYL7t5bYfBeYDh9bySSIiYkgdSzaSpgKLbd8+YNUEYHHD\n+/tK2QRgSUP5klL2rH3K4xCWl2a5oY4VERGjaMhZn9tB0gJgu8YiwMCngE9QNaHVcuqajhsREWuh\n1mRje9BkIunlwE7AraU/ZiLwM0n7UdU+dmzYfGIpuw/YYZByGtbdX54qOs72Mkn3AT0D9rl6qHhn\nzZr19HJPTw89PT1DbRoRsV7q6+ujr6+v5f2GfHjaaJJ0N7CP7Uck7QlcRNWhPwFYAOxm25KuA04E\nbgR+AHzJ9jxJxwMvt328pGnAkbanlQECNwH7UDUZ3gTsW/pvBsaQh6dFBL29vZx55jkAzJjxQaZM\nmbKGPerXjTH1G/HD00aZKU1fthdKugRYSPVo6uMbssAJwPnAZsAVtueV8nOBCyUtAh4GppVjPSLp\n01RJxsA/D5ZoIiKg+lI/6qjprFhxOgDXXDOdyy6b09Ev926MaW10Rc2mG6RmExGTJx/NggVTgeml\nZA6HHHI58+d/JzENodmaTaeHPkdExHqgW5rRIiI6bsaMD3LNNdNZsaJ6v/nmJzNjxpzhd1oPY1ob\naUYr0owWEdCdnfHdGFO/ZpvRkmyKJJuIiNalzyYiIrpGkk1ERNQuySYiImqXZBMREbVLsomIiNol\n2URERO2SbCIionZJNhERUbskm4iIqF2STURE1C7JJiIiapdkExERtUuyiYiI2iXZRERE7ZJsIiKi\ndkk2ERFRuySbiIioXZJNRETULskmIiJql2QTERG1S7KJiIjaJdlERETtkmwiIqJ2STYREVG7jiYb\nSf8k6U5Jt0v6TEP5TEmLyrrJDeX7SLpN0l2SZjeUbyJpbtnnWkk7NqybXrb/laRjR+/TRUREv44l\nG0k9wFuBvWzvBXyulO8BHAPsARwGnCVJZbezgeNsTwImSZpSyo8DltneDZgNnFGOtTVwKvAa4LXA\naZLGj8LHa5u+vr5Oh/Aciak5ial53RhXYmqvTtZs/gH4jO1VALYfKuVHAHNtr7J9D7AI2E/S9sBW\ntm8s210AHNmwz5yyfClwUFmeAsy3vdz2o8B84NAaP1PbdeM/rsTUnMTUvG6MKzG1VyeTzSTgjZKu\nk3S1pH1L+QRgccN295WyCcCShvIlpexZ+9heDSyXtM0wx4qIiFG0UZ0Hl7QA2K6xCDDwqXLurW3/\nlaTXAN8GdmnXqdt0nIiIaAfbHXkBVwAHNrxfBLwAOAU4paF8HlV/y/bAnQ3l04CzG7cpyxsCv2/Y\n5qsN+3wVeMcQ8TivvPLKK6/WX81859das1mD/6LqW/mxpEnAJrYflnQ5cJGkz1M1ee0K3GDbkpZL\n2g+4ETgW+FI51uXAdOB64O3AVaW8F/i/ZVDABsAhVMnsOWynNhQRUZNOJpvzgG9Iuh14kip5YHuh\npEuAhcBK4HiXqgdwAnA+sBlwhe15pfxc4EJJi4CHqWo02H5E0qeBm6gy8D+XgQIRETGK9Mz3eERE\nRD0ygwAg6VBJvyw3f57c6XgAJJ0raamk2zodC4CkiZKukvSLchPuiZ2OCUDSppKul/TzEtdpnY4J\nQNIGkn5WmoW7gqR7JN1artUNnY4HQNJ4Sd8uN3D/QtJrOxzPpHJ9flb+Lu+Gf+uSPiLpjnJT+0WS\nNul0TACSTir/363xO2G9r9lI2gC4CzgYuJ+qP2ia7V92OK7XA38ALrD9ik7GUuLZHtje9i2Sngfc\nDBzR6esEIGkL209I2hD4KXCi7Y5+mUr6CLAvMM721E7G0k/Sb4F9bT/S6Vj6STof+LHt8yRtBGxh\n+7EOhwU8/d2whGrw0eI1bV9jHC8CrgF2t/1nSd8CfmD7gk7FVOJ6GfBNqpvmVwE/BD5k+7eDbZ+a\nDewHLLLS+LscAAAEWElEQVR9r+2VwFyqm0Q7yvY1QNd8Kdh+0PYtZfkPwJ10yT1Ltp8oi5tS9UN2\n9BeUpInAm4GvdzKOQYgu+n9e0jjgDbbPAyg3cndFoineBPymk4mmwYbAlv0JmeqHcaftAVxv+8ly\nf+NPgLcNtXHX/MProIE3fjbeLBqDkLQTsDfV6L+OK01WPwceBBY0zDLRKV8APk6Hk94gDCyQdKOk\nD3Q6GGBn4CFJ55Vmq3Mkbd7poBq8g+qXe0fZvh84E/gd1Y3pj9q+srNRAXAH8AZJW0vaguoH1g5D\nbZxkEy0pTWiXAieVGk7H2X7K9quAicBrJe3ZqVgkHQ4sLbVA0V03GB9gex+qL4UTSlNtJ20E7AN8\npcT1BEPcmjDaJG0MTKW62bzTsTyfqrXlxcCLgOdJ+tvORgWlCf10YAHVfZM/B1YPtX2STfVLYceG\n9xNLWQxQqvCXAhfa/m6n4xmoNMFcTWfnvzsAmFr6R74J/LWkjrat97P9QPn7v8BlVE3InbQEWGz7\npvL+Uqrk0w0OA24u16rT3gT81vay0lz1n8D+HY4JANvn2X617R7gUar+70El2VQDAnaV9OIywmMa\n1U2i3aDbfhl/A1ho+4udDqSfpL/on8m7NMEcAnRs0ILtT9je0fYuVP+WrrLd8UdbSNqi1EqRtCUw\nmaoZpGNsLwUWl5u6oRqks7CDITV6J13QhFb8DvgrSZuVGfAPpuoz7ThJLyx/dwSOAi4eattO3tTZ\nFWyvlvSPVDNCbwCca7vj/yElXQz0AC+Q9DvgtP6O1A7FcwDwLuD20j9i4BMNN9Z2yl8Cc8rIoQ2A\nb9m+osMxdaPtgMskmer/+4tsz+9wTAAnUs0YsjHwW+B9HY6H0v/wJuCDnY4FwPYNki6laqZaWf6e\n09monvadMulx/w34Qw7wWO+HPkdERP3SjBYREbVLsomIiNol2URERO2SbCIionZJNhERUbskm4iI\nqF2STcQok/R4i9sfKOl7dcUTMRqSbCJG39rc3JYb4mKdlmQT0SGlxnJ1w8PDLmxYd2gpu4mGadvL\ntDPnSrpO0s2S3lrKPyzp3LK8V3mY1Waj/qEihpBkE9FZe1NN2bIn8BJJ+0valGo6ksNtvxrYvmH7\nTwI/sv1XwEHA58qccF8s+x9JNYfdB2z/aTQ/SMRwkmwiOusG2w+4mjfqFmAnYHeqWX77n3j4Hw3b\nTwZOKfPT9QGbADuW/d8HXAj02b5ulOKPaMp6PxFnRIc92bC8mmf+nxxqtm8BR9teNMi6ScDjVM88\niegqqdlEjL41PTbil8CLJe1c3r+zYV0vVbNbdSBp7/J3PFVT2hupZgo/un3hRoxckk3E6BtqZJkB\nbD8J/D1wRRkgsLRhm08DG0u6TdIdwL+U8s8D/8/2r4G/A/5N0l/UEn3EWsgjBiIionap2URERO2S\nbCIionZJNhERUbskm4iIqF2STURE1C7JJiIiapdkExERtUuyiYiI2v1/mR91LOenEV4AAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x109786c88>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "xplot = np.array(range(1, 9))\n",
    "yplot = coef[1:len(coef)]\n",
    "plt.scatter(xplot, yplot)\n",
    "plt.title(\"Regression Coefficients\")\n",
    "plt.xlabel('Index')\n",
    "plt.ylabel('Regression Coefficinets')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now predict on our validation set via the equation:\n",
    "\n",
    "$$y = X_{validation}\\hat{\\beta}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ypredict = np.dot(xvalidate, coef)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And calculate the residual sum of squares error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The residual sum of squares is 5.79495379765e+12\n"
     ]
    }
   ],
   "source": [
    "error = ypredict - yvalidate\n",
    "RSS = np.sum(error**2)\n",
    "print(\"The residual sum of squares is\", RSS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now evaluate how well our assumption that the errors are independently and identically distributed holds: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ4AAAEZCAYAAACnyUNvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvXucX1V56P19JsOQgcltEi6DIQFHESm0BAvkNB6TWpJg\ne15UUpXjhaFyXtrq5621owLqK3BI66WlYm2FWi+JWEXUwzGeFyfBQk6PPaJYsVBAiDfkIiCCsVZe\nuT3nj73W7LXXXvt3m9915vl+Pr/P7NnXtfdeez3ruaxniapiGIZhGN1iqNcFMAzDMBYWJngMwzCM\nrmKCxzAMw+gqJngMwzCMrmKCxzAMw+gqJngMwzCMrmKCxzCMlhGRtSLyjIh0tS3p1XWN9mAvzeg5\nIvIDEfmFiPxMRP7N/f2rXpernxGRZSLyURH5kYjsF5Fvi8jbelSclgYDishFInJVt69r9J7hXhfA\nMMgakN9R1Rvr7Sgii1T16Xrrmj1HOxER0c6PzH4/cBDwPFX9mYgcAxzf4Wt2AhMeCxDTeIx+QZIr\nRaZE5Csi8pci8ghwUcU6EZF3Ou3pQRHZISJL3Tm8Web1InIP8A+J69whIr8d/L9IRB4WkRPd/+tF\n5J9E5DERuUVENgb73igi212Z/h04WkTOEZHvOu3tuyLyn92+hV5+bDKqOi7BycCnVPVnAKp6t6r+\nt+C8l4vID502dLOIvDDYdpGIXCMiV7nr/IuIPFdELhCRh0TkHhHZHN3fn4nI19z5rhWR5RXva6mI\nfEREHhCRe0XkUhFJvtvEsc+IyO+LyN0i8qiI/HWwbUhE/kJEfiwi3wF+p9HrisiHRORzwb7vFZHr\nGymT0RlM8BiDwKnAd4BDgT+tWPd7wNnARuDZwBLgr6PzvAg4FtiauMangFcH/58O/FhVvyUizwL+\nB/BfVXUF8Bbg8yKyMtj/tcB/cdd9BPgAsFVVlwK/AXwr2Dfu5SuAiBxU57iQm4A/c4LqOYntXwd+\nFVjh7u2zIjISbP9PwE5gubvGbjLhfwRwKfC30fleB5wDHA48DXywolw7gSfI3sE6YDPZc2mU3wFe\nAPwa8EoR2eLWnwf8tlv/68DvNnHdaeB4ETlbRP4jeV0xeoWq2s9+Pf0B3wd+BjwKPOb+nuu2TQE/\niPZPrfsy8AfB/8eQNURDwFqyxnJtjTJMujIsdv9/EninW34bsDPafwZ4nVu+Ebg42HaQu4eX+/MF\n2y4CPhH878s2VOu4RHkPBC4AbgZ+CdwNnF5j/0eBE4Iy7A62/Sd37+L+HwOeAZYG9/dnwf7Pd9eU\nqPyHAf8/cGCw71nADRVlip/FM8B/CP7/DPA2t/wPwHnBts3NXJdMQ/yJq2uv7HWdX+g/03iMfuGl\nqjquqivc348G2+5N7B+vOwK4J/j/HjIf5mHBuvuqLq6q3wXuAP4vERkFzgD+3m1eS9b7ftT9HgM2\nkPX+S+VR1V8ArwL+EPiRiHzR+WBqUnHc8yr2/aWqvkdVTwZWAp8l02qWA4jIW5z58DFX3qXAquAU\nDwXLjwOPqGuh3f+QCaDS/ZE92wOi8wGscet/FDynKxP71SIs1y+CMhyRKEPD11XVm4HvkQnLzzZR\nHqMDmOAx+oVafoCUAzpe9wCZgPCsBZ6k2JDVc2RfTWZueylwu6p+362/l6xnPh4IxyWq+udV51bV\n61V1C5lwugv4O7fp38k0G89Eg8dVoqo/B/4MOJjMv/RC4K3A77qyrsBpNPXOVYMjg+W1ZNrkI9E+\n95JpHiuD57RcVX91Dtf1/ChRhoavKyJvBEbI6sn5bSiPMQdM8BjzhU8DbxaRo0RkjMzvc7WqPuO2\nN9LoXg1sIdM4PhWs/ySZJrTFObkXi8hGETkidRIROVREznA+myeBn5OZkSDzp7xIRI4UkWVk5rJa\nxyWj7yQLpPh1ETlARA4E/pjMTHkXmZ/pSeAnIjIiIu9y6+bCa0XkWFe2S4DPBhqSAKjqg8Ae4P0i\nskQyni0iL5rjtQGuAf5IRJ4lIisIhEe96zpt81LgNWS+nbeKSDuEodEiJniMfuGLLsLK/z7f5PEf\nA64C/hH4LpmZ5o+C7XXDdl0D9lVgPZl/wa+/j0wLejvwYzIzz1vIv5/43EPAnwD3k2kFLyITZqjq\nl925byXzz3yxkeNSxQU+7spzP/BbZCHpvyALFNhN5vf5PtmzSJkraxHf01VkDvwHyDSHN1Xse7bb\nfgeZX+mzFE2SzVwz/P/vyO7pX4BvAHH9SF5XRBa5sr9bVf9VVb8DvAO4SkQOaLBcRpuRvNPSowKI\n/ADYT9YjfFJVT3E9ms+QqdM/IHMG7nf7Xwi8HngKeJOq7nHrTwJ2AIuB61T1j936EeATZJEyjwCv\nUtUfum1TZJVQgT9V1U904ZYNY6AQkRuBq1T1Y70uizE/6AeN5xlgk6quU9VT3LoLgC+r6vOAG4AL\nAUTkOOCVZFE1LwE+FIwRuIIsEuoY4BgR8SGz5wKPqupzgcuB97lzrQDeRRbtcirZWJBlnb1VwzAM\nox8Ej1Aux0vJ1Hrc35e55TPI7PZPqeoPgH3AKSJyOLDERa5ApuG8LHGuzwEvdstbgT2qul9Vf0pm\nIz69bXdlGPMHyy5gtJV+SJmjwPUi8jTwt6r6EeAwVX0IMru7iBzq9n0WmQ3ec79b9xTFUNn73Hp/\nzL3uXE+7kdfj4froXIZhBKjqi+vvZRiN0w+CZ4Oq/khEDgH2iMhd1HYyzpW5hJQahmEYc6TngkdV\nf+T+/lhE/jtwCvCQiBymqg85M9rDbvf7Kcbyr3brqtaHxzzgIlyWquqjInI/sCk6JpmkUkTM1GAY\nhtECqlrq7PfUxyMiB7kxF4jIwWRjKG4DdpHlhYIsPcoX3PIu4Cw3NuFo4DnA110Y7H4ROcUFG5wd\nHTPlll9BFqwAWWjmZsnSy68gS8Gxu6qsvU4xoapcdNFFPS/DQiu7ldvKPZ/L3emyV9Frjecw4Fqn\nUQwDf6+qe0TkG8A1IvJ6sjETrwRQ1TtE5BqyWP0ngTdofndvpBhOPePWf5QsZn8fWa6ms9y5HhOR\nS8nGBChwiWZBBoZhGEYH6ang0SwlyYmJ9Y8Cp1Uc827g3Yn1/wyckFj/S5zgSmzbQSasDMMwjC7R\nD+HURoNs2rSp10VomUEtu5W7u1i5u08vyt7zzAWDgHRlQknDMIz5hYig/RZcYBiGYSw8TPAYhmEY\nXcUEj2EYhtFVTPAYhmEYXcUEj2EYhtFVTPAYhmEYXcUEj2EYhtFVTPAYhmEYXcUEj2EYhtFVTPAY\nhmEYXcUEj2EYhtFVTPAYhmE0ye7du9myZRtbtmxj9+7KabyMCixJaANYklDDMDy7d+/m5S+f4vHH\n3wvA6Oj5XHvtTrZu3drjkvUfVUlCTfA0gAkewzA8W7Zs4/rrzyCf2HgnmzfvYs+ez/eyWH1JX2en\nFpEhEfmmiOxy/68QkT0icpeI7BaRZcG+F4rIPhG5U0S2BOtPEpFbReRuEbk8WD8iIle7Y74qImuC\nbVNu/7tE5Oxu3a9hGMZCpi8ED/AmsumsPRcAX1bV5wE3ABcCiMhxZLOJPh94CfAhEfHS9ArgXFU9\nBjhGRLzeey7wqKo+F7gceJ871wrgXcDJwKnARaGAMwzDSDE9fR6jo+cDO4GdjI6ez/T0eb0u1kDR\nc8EjIquB3wY+Eqx+Kdlbxf19mVs+A7haVZ9S1R8A+4BTRORwYImq3uz2+0RwTHiuzwEvdstbgT2q\nul9VfwrsAU5v570ZhjH/2Lp1K9dem5nXNm/eZf6dFhjudQGA9wNvBUJt4zBVfQhAVR8UkUPd+mcB\nXw32u9+tewq4L1h/n1vvj7nXnetpEdkvIuPh+uhchmEYNdm6dasJmznQU8EjIr8DPKSq3xKRTTV2\nbadnv+ToaoSLL754dnnTpk0DPce6YRhGJ9i7dy979+6tu1+vNZ4NwBki8tvAKLBERK4CHhSRw1T1\nIWdGe9jtfz9wZHD8areuan14zAMisghYqqqPisj9wKbomBurChoKHsMwDKNM3Cm/5JJLkvv11Mej\nqm9X1TWq+mzgLOAGVX0d8EXgHLfbFPAFt7wLOMtFqh0NPAf4uqo+COwXkVNcsMHZ0TE+7vEVZMEK\nALuBzSKyzAUabHbrDMMwjA7Sa42nivcA14jI64F7yCLZUNU7ROQasgi4J4E3BANs3gjsABYD16nq\njFv/UeAqEdkH/IRMwKGqj4nIpcA3yEx5l7ggA8MwDKOD2ADSBrABpIZhGM3T1wNIDcMwjIWDCR7D\nMAyjq5jgMQzDMLqKCR7DMAyjq5jgMQzDMLqKCR7DMAyjq5jgMQzDMLqKCR7DMAyjq5jgMQzDMLqK\nCR7DMAyjq5jgMQzDMLqKCR7DMAyjq5jgMQzDMLqKCR7DMAyjq5jgMQzDMLqKCR7DMAyjq/RU8IjI\ngSLyNRG5RURuE5GL3PoVIrJHRO4Skd0isiw45kIR2Scid4rIlmD9SSJyq4jcLSKXB+tHRORqd8xX\nRWRNsG3K7X+XiJzdrfs2DMNYyPRU8KjqL4HfVNV1wInAS0TkFOAC4Muq+jzgBuBCABE5jmwa7OcD\nLwE+JCJ+drsrgHNV9RjgGBHZ6tafCzyqqs8FLgfe5861AngXcDJwKnBRKOAMwzCMztBzU5uq/sIt\nHggMAwq8FNjp1u8EXuaWzwCuVtWnVPUHwD7gFBE5HFiiqje7/T4RHBOe63PAi93yVmCPqu5X1Z8C\ne4DT23x7hmEYRkTPBY+IDInILcCDwPVOeBymqg8BqOqDwKFu92cB9waH3+/WPQu4L1h/n1tXOEZV\nnwb2i8h4jXMZhmEYHWS41wVQ1WeAdSKyFLhWRH6FTOsp7NbGS0r9XcpcfPHFs8ubNm1i06ZNbSqO\nYRjG/GDv3r3s3bu37n49FzweVf2ZiOwlM3c9JCKHqepDzoz2sNvtfuDI4LDVbl3V+vCYB0RkEbBU\nVR8VkfuBTdExN1aVLxQ8hmEYRpm4U37JJZck9+t1VNsq79AXkVFgM3AnsAs4x+02BXzBLe8CznKR\nakcDzwG+7sxx+0XkFBdscHZ0zJRbfgVZsALAbmCziCxzgQab3TrDMAyjg/Ra45kAdorIEJkQ/Iyq\nXiciNwHXiMjrgXvIItlQ1TtE5BrgDuBJ4A2q6s1wbwR2AIuB61R1xq3/KHCViOwDfgKc5c71mIhc\nCnyDzJR3iQsyMAzDMDqI5O22UYWIqD0nwzCM5hARVLXkV+95VJthGIaxsDDBYxgLgN27d7Nlyza2\nbNnG7t3myjR6i5naGsBMbcYgs3v3bl7+8ikef/y9AIyOns+11+5k69atdY40jLlRZWozwdMAJniM\nQWbLlm1cf/0Z5MGdO9m8eRd79ny+l8UyFgDm4zEMwzD6gl6HUxuG0WGmp8/jK1+Z4vHHs/9HR89n\nenpn7YMMo4OYqa0BzNRmDDq7d+/msss+DGSCyPw7RjcwH88cMMFjGIbRPObjMQzDMPoCEzyGYRhG\nVzHBYxiGYXQVEzyGYRhGVzHBYxiGYXQVEzyGYRhGVzHBYxiGMeAMWhJYG8fTADaOxzCMfqWfk8D2\n5TgeEVktIjeIyO0icpuI/JFbv0JE9ojIXSKy20+P7bZdKCL7ROROEdkSrD9JRG4VkbtF5PJg/YiI\nXO2O+aqIrAm2Tbn97xKRs7t134ZhGO3isss+7ITOFJAJIJ+lol/ptantKeBPVPVXgP8AvFFEjgUu\nAL6sqs8DbgAuBBCR48imwX4+8BLgQyLipekVwLmqegxwjIh4cX8u8KiqPhe4HHifO9cK4F3AycCp\nwEWhgDMMwzA6Q08Fj6o+qKrfcss/B+4EVgMvBXwWw53Ay9zyGcDVqvqUqv4A2AecIiKHA0tU9Wa3\n3yeCY8JzfQ54sVveCuxR1f2q+lNgD3B6++/SMAyjc0xPn8fo6PlkzdxOlwT2vF4Xqya91nhmEZGj\ngBOBm4DDVPUhyIQTcKjb7VnAvcFh97t1zwLuC9bf59YVjlHVp4H9IjJe41wLlkFzUBoLF6urOVu3\nbuXaa7M5ljZv3tU3/p1a9MW0CCIyRqaNvElVfy4isSe/nZ79kqOrES6++OLZ5U2bNrFp06Y2Fac/\niB2UX/nK1EBU4FaxbM2Dy0KsqxdeeCn33PMga9eu5t3vvrB0r1u3bu2L+9+7dy979+6tv6Oq9vRH\nJvxmyISOX3cnmdYDcDhwp1u+ADg/2G+GzD8zu49bfxZwRbiPW14EPBzsc2VwzJXAqyrKqPOdzZvP\nVNihoO63QzdvPrPXxeoIMzMzOjp6mLvfHTo6epjOzMz0ulhGgyy0ujoyslxh1Wx9HRk5ZGDqq2s7\nS21qP5jaPgbcoaofCNbtAs5xy1PAF4L1Z7lItaOB5wBf18wct19ETnHBBmdHx/g5f19BFqwAsBvY\nLCLLXKDBZrfOmOcMYhSQsTC57LIP88QTxwJ/ga+vTzzx5wNfX3tqahORDcBrgNtE5BYyk9rbgfcC\n14jI64F7yCLZUNU7ROQa4A7gSeANTqoCvBHYASwGrlPVGbf+o8BVIrIP+AmZpoOqPiYilwLfcNe9\nRLMggwWJzVJpDApWVwcfG0DaAAtlAOlC8Xv084A7ozEWUl0944yzeOKJYTKtB0ZG3squXVcNxD3b\nDKRzYKEInn6lE43MQmm4jMGnkeCCfsUEzxwwwdM7TDsxamEdiP6mL1PmGEY9LBBg4VJvrI7vlFx/\n/Rlcf/0ZvPzlUwt+TM+gYILHMIyG6OagzUaEinVKBhcTPEZfM4jpQOZCv47I77Z2YUJlfmOCx+hr\nBjEdSKv0g+moSvD1oyBYaJ2SeUVqVKn9+i9zwczMjG7efKZu3nzmwIxarsd8vKe50OsR+bUyOnS7\nbI1ml7A61N9Qkbmg5436IPx6LXjmY4qX+XhPc6XXgqfW9XvxvkyoDD4meAZY8PS6QeoE8/Ge5kqv\nhXG9d2KCwGiWKsHTF9mpDcPI/Vn5uJTu+rPqpaLplwzIxuBjA0gboNcDSOfTIEo/4O+RRx7i9tvv\n5okn/hwY7HuaT9iATKOdWOaCOdBrwQPzo0GIBejIyB9z5JFH8thjv+hKKpD58AwXIvbeBpcqwdNz\n/8kg/OiDqLb5QNmHMK1DQyuSPo12+xNy/8m0wnoVWaKTkyeav6LP6bXfy5gbWHCBCZ5eUxY865PO\n7E40Ntm1pxW88FnVd41ZM8J2Pjv6w3tbt27jvApCmc/vLYUJnnkmeAaxAscCZWhoZbJR6UTEW3ZO\nL+j6L6KuGWE7n7WARutIq+eOv5lufkfz+b1V0beCh2yitoeAW4N1K4A9wF1ks4IuC7ZdCOwjmx57\nS7D+JOBW4G7g8mD9CHC1O+arwJpg25Tb/y7g7BplbOe7mDODXIHDD3379u3J++iE4JmZmQkasf4T\nPM3cc6P7drtRbce1qs2xmYl0aGilbt++venrpb6ZYv2b1qGhlbpu3caOPatuDSHop05pPwueFwIn\nRoLnvcDb3PL5wHvc8nHALWQzpx4FfIc8QOJrwMlu+Tpgq1v+Q+BDbvlVwNWaC7fvAsuA5X65oozt\nfRtzpKoC91OFa4SZmRldt26Djo9PFj74eoK1kftM7bN9+/agEatvauvm82y34Olm56Sd10rd2+Tk\ncQVf4MjIch0ZOaSp66XOOz4+6dbNaGaC7eyz6lSHKvyGqjpzvaJvBU9WNtZGgufbwGFu+XDg2275\nAuD8YL8vAae6fe4I1p8FXOGWZ4BT3fIi4OF4H/f/FcCrKsrXthfRDlIVeN26DU1VuE40qs36KFoR\nLo00crX28eddt26Drlu3sdLs0u0PuNH78mWv1/B2uncdPqvJyRPbdq3Ucyj7eWLf4LSOj0/WrHe1\nBU/3NJF21qmZmRkdGVle6ESJjHflXhpl0ATPo9H2R93fDwKvDtZ/BDgTeAGwJ1j/QmCXW74NOCLY\ntg8YB6aBtwfr3wn8SUX52vQa2kNjH2d1hetEb7jZc6ZMKrUaD9/Q5Y1F9X220uh20rcQ30O9e0xt\nj8s3MrK8JDjn+gyauY/QRAXL23qt+DnUDkqZ0Vh73b59e9KXU21qSwe5dIJWtfXUPtm3EJe9e/fS\nCIMueH6i7RM83xl0waPayMc5dx9BMzR7zuL+5cajWsup/2G1YopsNOKuUeJrzVXYN/t829m5qF3X\nwmhBL7RXzKlRrXcvRVNbWftpNETfm6nGxiZUZHlbntVcaVRbz+5/fXD/Mwob3He0rC/uRXXwBM+d\nkantTrccm9pmAlPbncH6Rk1tVwbHXFnL1HbRRRfN/m688cZ2vZeGaaS33KijtBnndMoHk6LZcxY/\n9tqNfDNCqvwssn2mpqYqG6R0+asbsHrvJTOB5KawkZFD5hwW3KoWFze0c2v0s3q1ZMmaSPD4hi+L\nHFy3bkMD56qvtTVyL2UNuLEOw1zKUq9szewfa2e1Ok3lzteMZtrmUvfz38W0ioyXvtlu+CxvvPHG\nQlvZ74LnKOC24P/3egFDOrhgBDiaYnDBTcApgJAFF5zu1r+BPLjgLNLBBX55eUX52vt2mqSqF1TV\ng6vXYDbqT4jtxyMjh9TsyTZ/zuwDKTZk9QRPJhTq2fTjj7vKdOb3m5hYo2FPcWRkuU5OnlBT6KYE\nTPYOykKmfI/bdHj4UB0fn9Tt27c3XQeabShb1YDyZx864KddQ1debrzDU78D0QjVJtLagnCuWn8z\n/rbq8obPrrbJvKxlhs+xvrmwE+b1RmhZ8Dgt4S/q7dfqD/gU8ADwS+CHwO85QfBlsjDnPaFAIAun\n/g7lcOoXOLPaPuADwfoDgWvc+puAo4Jt57j1d9PH4dTNBBM0+kFt375dx8cnKxu+7Dz1K3MzPeqq\ncxbvpdx4NaPNVT+/Wtfd5hqAabffwbpoUX1tJ9VIrFu30fXCp10DkZmixsYmgnvw19vhfksbFj71\nGruqd9BIvUgdmx8XazbH6pIlR84K9kY6PNWNZ/MNf1W5t2/f3lCHqfjuZhTW6/j4ZN3n6Lc1Y/pt\n5hk09l0XhXYmbJuxGszteTfDnDQe4KZG9puvv34UPFVO9kYbmHq9n3qCp5UeVK1z1tPWUtsb7fVn\n1y37IfLGJ3yWMwqr3f4b3Lbjk73m1DsYG5vQxYvHS4JlcvK42QZkePjQqBFfr2NjEw3Vhfx+aplj\nimNeah3jqdKoiv6Eal9OI/Vu+/btdc2rcYeoFfNQMcouLVRy4VQOra+KaPRlWbLkyJrCI/xGqgW5\nP7bYQanyRdbSeLdv367DwweX6lzYmRlUwXMFsAt4HZkz/0zgzEaOnQ+/XgueVCNfpZKn9g3tyP7D\nbkQ41eo5tupzaPWcMzMzkSaxQWFc48ah9vMrNsj59eLxHMdq0Wa+Q+FAXbJkTUFDXLduQ7TPUtew\nHl9q+JYsWTNbvvw+DguOXdakBlc2PabO64VDPXNtsU4Ue9Te7AgrtOr91DOH5u9+qWZCxz/jvDxT\nU1NR43mQDg+vLOyTiliL33XR3Fbrvc9osdNR3anLNZHpoN6Vn5UvY7GTNK0iS3TJkjU6MXFUIHxj\nzfdAHR09QsfGJpJ5BGv5XLP2oCzEyt/AgJjasmP5eOL3sUaOnQ+/Xgse1eaipEKTzOTkCYUPIP/w\n6wuNmZkZN3hvmS5adIhOTp7YlOBJOVF9mcbHJ3Vy8oSCxlK/J39s8JE3Z+IIn0nYUyya2vw5N0bn\n366p3mTemGbRRSL+Oaf8Ivk7yhrY1kOQq/0E67Welho+g8nJ44JGMBSW6fdQK0Cilu9iZGS583HV\nHn9THNB5pmaaZ3H/VBaDkKJ2e7zW1nTT95oSPMUxP8Vziowl6lQcABBqV96kG9YBX8eqBzfX+ubL\npriilpdqQ7pBXwcX9PuvHwRPisbs0OHHXrtBTJ2/SkOp14Mqbi/27MrmjMxvMzl5gg4PL5ttQEdG\nlkcCKW4wqnt4Vc8rFBQjI8tnBeLk5HG6aJH3m8QmwXRDFL+DvEGrdvjmpqu4kc2d4CmBner9ls0+\n27WWVlJ8L2HPPWwcq4V6LVNeWKa09lRbKKqmNMGw8faCqHbodv4O/LMoC7uxsYlAgK3WODhicvKE\nkg8tP++ZwfknFVbr5ORxs9cv+8TCTkwsHMLnf2iwf6NaZcpM3V8JcOeq8awGrgUedr/PA6sbOXY+\n/PpF8NRStWPSTuG4QajtUG3Ez1PVKBZ7r3FE2XTk56jqKR8SmAano/KntZBalE1jqwq+m2LDGpra\nwp539tyGhw+dNamk05Vs05RWkwuKuLHP7rcY9u3LkZ7KIW1W2qZwsGba4WoVWVFh6w/fbSgws8CB\neFxLKKDrjXkpmn3CelC83+HhZQWNd/v27dEzm4neQ32NLu8oeW1nZXQ+f64NCgdproF405/XRkZ1\ndPSIgq8pXTeKoctFM95yzYRLSvD4jpMvrxeq5W/Fa4UpjbOYTig2A5aFe7eZq+C53kWbDbvfOcD1\njRw7H379IHjq+Udi8g9ge/CBhY1hbTW+2jxSP7ItO4833cTCK+wBpz7GlGllgxbNEL6nWv8DC8s3\nOnpE6RivucT7r1u3QScmjtLh4UN1ZGRZ4vrHBs+1KDjGxibctqWl7cWw6rghmdaixhKadco92cyJ\nHpqVfIO3LNh3m0IWsl70b4QNX7mxWrduQ0HrSo8hKYcrZ8LwQC2a/8J3f6wODS3T0dFDkgKsnH4n\nvK/jtCzMp53g3xA821CAhVrg+mhdbA2oPW1GWcssWw6Kz8oLv1QgQyz0p9x+oUZXNlkWw/eXJ1Lk\nVGur3Tazqc5d8HyrkXXz9dcPgietfVSPZykKgPADyHrCsHy2J7du3YZAAwmFg+8953ZpkfHZHnR1\n0EPYoMXRUGHDFZuj4obYHxv2An2jUv8Di815+Tid/PktWbKmzvPzvfODnSluOih32Ch64bjUrQ8b\nudwkU9S6UulO4ueREtCxkAob1zOjaxe1wqmpqaDnHmoGYaDDYh0dPUSHhw/ViYlnJwaMls1d27dv\n18nJ4xSWaC3ne954pgVY/NwzjS585huC9xhqhL4zM60wobFWAivcffj6FD/r8LnVDpQoWxOKAttr\nwflz8/X3UvQ2AAAgAElEQVRjtY6MLNXR0UN0aGiZiizTYgctNr8eFv1/rI6OrprVsLNvLa5D29zz\nyc2h+Tsva7CdFkRzFTz/ALzWjelZ5Jb/oZFj58Ov14InN13FDu/aJo/MdLHCVe7QMZ+p5CMjSzUX\nLLV6f6EdPA/lLPoXsg8vL2co6A5251ilxR79jNvuG5KwjDs0F1xhI1WdUTgOgy0OGvX35NeVB+/V\njvJar4sWheMlNmjeQ/XRWuNabNjCZ7heYUwnJp6tImOaiuzKghPCY/w5U41ceP5QeITPKx3u7RvG\niYmjAr9W6EcJTVBhgx5HE4ZC0Gs5q7XcGKbMRb6Dko7ACzsP+Xs8QfMO0erEszktKENR6B9wwJgT\n+l7TCTWRHcH/8bMuCs7h4YN1dPQQzeqsP48Pu1/rtLjcNBpG5Y2MLHc+zFAbPcjdR9wB8OUJ68N4\ncO7xSOMN63TRslEePN1YVo52MFfBs9aFU//Y+Xj+O8G8NvP910vBk7Yt17fl5sIqrNxh79ifI9ZA\nvKCp0kRmapYh6/Uuj84bfmzlBn9qakrHxyed1lX8kPOy+kYiPG/Wi1yy5MiEfT3ULELB48tW9tuM\njU0EwQ2rg3N5TScUBCcE/3vtxzf6ocaXMt9k4bWjo4fo6Oghs2G2Re0ybDzLTuO8MdmuuTD19+3P\nU44Kq9YWvLAPTVD+eU5pUXMKNUdfb/y1QoHiBW7uZ8qjztImvpRZKNeiQ8G8Q4vaQlgfY6EW3uth\n0d/VCmM6NLRKx8cP17yT5O8x/A68YPPnPECL5tZ0Z8P75coaii/3wRpbFrLrhwIkdU9jBUFXu5NS\n6//O+YHmmrngzfX2m8+/XgqedGM6XrPyFCPa/H6+d+QbKa/a+57WmZrZ0H2FXxFcN9Ycwo89Faoa\nHl9tIpycPC7oPfqPqeg3GRo6yH1csZM6H8MzPJxFV2UaWJVNPezRhh+97wXHGpdvrMLGzTfCyzRv\naI9xx40nntuRieNVM2FxsKbLljKzHKewSoeGls02YsUR+nG5x3RoaFngm/ICI77/ouAdH590GlAo\neGJTp2ox04MXvqGgXqnFRjPPNpENIvXPaGN03nQWi0wo59GI5Q6Nv1ZYhlCopfw329zfsL6F9W86\neK/+vcfaevwd+rqe1uSK5nKvvcf3dKa7ru8A+PoTWjxC0+o29T683D9W5StNfQd9KniyY/l6I/vN\n11/vBU9sg49NMqleZVxB4w9lg8JizU0kcY/Nf8xVPppQIOY2+vzavncY29Kzip5pRqkxRcdX7Ovv\n+UyFw0sfViacfEOR1oqKjuvtievHpqmwFx03Xv7vEvecfGMcNrihMzx8Xis13WCFjdrS4Jf2seX3\n44Xxse59rlZYrcPDB89qk5mgD/dPDyLOnrVvfEPTVPiufQTWhObmUx9MsU1zrSHUGH0nYVnwTENz\naSZkRkcnEtt8AxwKwY1avJfjtRi2H49LioMUvH+l6j2EFoBYuIb7psyQsQnu+Fn/VT6INrQaxO9+\ntWbafqgxT2jRlBt/36H/rKjFejN0/m7LQS/9amp7P/DXwH8km2L6JOCkRo6dD79em9qKNtrQL+Ib\nuaxR8uMqig7+gzRv/MJxEpNaNqmlPtCwkfCNTmyi2aYwposWHaITE8/WoaGDgn3S0V+543VjcH3V\nlJDKTYa+UfEN35lRmTcGjUlsllgRjc/wDVUYVBE2ZlnDMjIyHjUSoUALe6WxsPMNctzTXq9lYRx3\nLvy2jYl7yZ355XrhBVXesx0dPXzWb7Ju3QbnX0oNZhzVkZFxHRpappnw8u8wjswKy+frk7/PNVoW\ntKn3WzYZ5SHg8SBc/17iDtHqGttXa9ap8pp0+J14S4BvwKvqf3z91Vqsg+vd/4u1GIZ9oLufYuMu\nsnz2PZTHXsU+Ne/7O1Dz73zM7Rf6qGJNNI9GjCc6zNuR8PsuRyW2m7kKnhsTvxsaOXY+/HodXFAe\nFOc/2LBH5D+wFa7xSPX8vXnHm4TiHp8PWKgyV3nfwjbNGh3fC/ONlG9Q40GM0zo0tKww/igfmxM7\neMMGI0zDH5bTa0teA/T7+8iyMS0KiWKPO0v/EjYYKVNLvn/u9wm1HS+8/D3E7yL0E2132xcH+8X3\nHTZeoZaV1hjTaXdSUW5xGG6q579NyyanWCsJNQhfT0Ihn5ofJvT9xB2LvOEbHV2lxSCG1Mj/pZpH\nZI4FHYIdmpsa/fMbcxFj/n9fp0Mzs7/WwZoLy22aWwBigRRqtF6wxYLZ339Kk68aj+M7DKG51AtS\nr734DuK0Zubb0BpR1lpTZJaItdpNM5vqHAQPMAS8st5+8/nXa8GTq+ixMzYO2w17Y3EvfDJa9o3N\n4ui4VE/Qq/8rA6HmG7txLUejlUfPL1mypnA/eRBCSjB6TanYe8/PGSeB9L1L3wsd07xB3qjlhm5C\ni+a67eqjpCYmjimVPexFFsdNzATlXa1lH4p/xv4Xaxq5Y3ti4tkVqXwOTrz3aRdhV7yvPPQ47gnH\nZqe4cU/NZBkHX6SCLY6N3qEXhl7IeW0mfKdjmgu77JkPD4cZI3xdntBi5FjeeRBZHvi4vIBPacD+\nXvy7DuvCSi027l5A+et5oRzWxbzMeT3xPplUNGaqnsTRbqFZ7tDgnPG3G2rOqSi42hPwZYInNPP5\nsi7t63DqbzSy33z99VrwqGpFzH7o/AwFi2/El2puakvZsn0Pyn+0/hzxtUKhFvsGDtdio3SmFk0+\nWeX2aUUyE9GKxLliP0fsIN2gRVNbvF9o/op73CnNYHW0X6jRVfcIywMcp3V09BCXhiUWSv4jj02a\n3lxaNkH6RiB/Tl5AhUI4bafPc8DF14qzAcTRTynBEw7aXa9wsA4NpRKghqZXf0+59psFh3jTXZg1\nIBX04Msaa17pnn042DccWJkHL/gyeuG3TYvaTeq5hIIuNNFl35rPV1gM+vGBJZPuF2dFWK1lgbxY\nFy06xHXkwlD80Kwbm2ljDS4UmiuSmTu8iTUb1Bx+G76zsqyj7dZcBc97gLcAR5JNGz0OjDdy7Hz4\n9YPgqQ4y8D38VdF2b7YKP+p4jEDY+PiPz3+gYUMRNvSx4PGhp+GH5Xu2+WDViYk1wTiKUFCEafLD\nBmZj9JGEvVpvmtmhxai8DUEZ/PnDEOhcWBRNJX5/b8LLG/l4JsdURuqiphabz3yPPG48Y7NUHgSR\nziCeTRyXjyXyveQs7NwPLMydy6HJL+6Rx3Uift++1+5Nsr5BH6vI/Ow7N9NaHqTrzxHef3xvvgMV\nltu/99DvU+2XCIVQcayMrx/epOa1jlBrCbWK5Vr8NuJvbnE0ANQ/q2JHa3z8MKfVe63N17PtmvuA\nvPk2FubHqoj354Tr17vM1Sc4X10u4IeHy9nNy9lOyuPuJidP7Gi7NVfB8/3E73uNHDsffv0geFIJ\nGk877TT3QY1p2dQTN26q3qQ0NLRMx8ZWaGocR54qJ2zYwg8gbnjHtdiD9UIjFoKhDyi0iWfmmjxB\n6Kpg/zD6J9RQQpu/t8lPB+viUOW4MdzhUuJ4Z7hvmOLzlPOVlSd5i/1kM64hCHPRHazphj18FimB\nFXcIfCcgDp+O/TOxv8hfxwvnte6Y7PmLrNCJiaNmsxVMTp5YY1S87+SEAR1+v3hwaZiapkrweK3J\nO9JDP0/e2BffaTHPWzgKv5ioNezoxJqqT9gajg/zdT7UCGPfVqjRrXbnjzs2mWm57Jv1ZtA4y3ps\n1gyFZrHDlFsL6gcJlLOdzATlyOpGIxMQzgXLTl0tVE4Hvk02C+n5Ffu0+NjbSzyiO/towg8yNf98\nSl0vhuiGg9zyAXvhedYG1/C9Y6/NLNbiAFXVstlsbfAhT2sWghwKoqWz4abF9D2hANumxQZ5RrMe\nnP/fj2qPn4M3eZWzFORaZGh6OzP6G3/0RUFdjBQqNjzlxnXJ7PQS+ftLjQWJNYpQ6E5oUWM5UssC\nIndkF8f7hHUg12SGh1eWkrzm2mkosLxWcrAWG9LQVxTWN9+5CE1Pvg6FQn6VFk3FZf/V6GjokwvN\nhf4cXiuL66EvR1k4FAVVaE72giHUyL3A9NtCU105b2AxZc6ZmguoWNCEUW2xsAl9Q6FZPby3ah9P\nVr/jurlNwyneO529uiXBA7wtWH5FtO3Pah07CD8XOPEdl5nhAOBbwLGJ/eby7NtKMaVLqrcUfpDh\nyOawYU3bzFX9LJG+V+vP4xsc3xj5j87/TaXk8R/llJb9LitK51+0KG/8coHgx2eEQituoNZrXsZU\nwxVqMpl9/LTTTguSPfpjV9e4zpnBB+6f8/rZzMUiS7Sozaxyg2PL00aUc8kdrkVhGmfgjsdS+Wfr\n/4+PL/d+0wI97v1vnH3/ea869POEwQi+Q7AyKE+oBc1o7vuL649PLTOqZaEVCtNY6wkFwMaK5Uzo\nlmc53Vg6X3FQZ/xMZjQPXQ6PWx3c00otvqdypvTMr+K1Ka9Bh0LfC5C1mmvZcWBOaAaNtZe0MPVk\nKbMO1FBz73b26lYFzzdTy6n/B/EHrAe+FPx/QUrr6RfBU0yg6LWFuIfkc1WlQpKrBE+e4TfvMYY+\nGN9T9g2g79muDM4Z9nDXKxzgGoDYtBB+eKHfKW+csw8mHowZawKhmcpH+YRlOcStiwVzGC0X+ha8\ngAob23igX2hiy+f/KU4DsEHh+FmhFAqZfByNt8sf7MyLYRBIKERDgRI3dL5R9Npj9UBTT97IpucX\nKo8ZizWGWOv05isfhebL4e/Bv/dYWwnNTKGZriol1Ey0LUyFVPZbjI6uioIOYl/aKl206JDZTAqZ\nRhgHGsQh3d7PmM8XFW9btOiQxAy1ofnXv+NQ6wjNz3GIvP9Wwmvl30sxiCL/jsM5eoqWjfFAc6w2\n0bWTVgXPLanl1P+D+AO2AR8O/n8t8FeJ/Vp/8m2kPMug/9DDhirujWUfY3Hq3jhzbxhOHfoDUhN6\nhYMgwx5ueYKwLMoqzLWWm9aKwiQXCsPDh0ZCMjYVhlpY2LsOhY0XJHEalTgbdmiKmNHiAEjVvBcc\njuPINTof4VT0vxVHjBe3p8JuU1phKp/cRi2+03AMzXLNzZG5CS02veTlKCZ99VGHZZ+A/z/MGuA1\nF+/cLkbXDQ8vc+/v2GB7GPkWdwRSgQneFxGWwwu4FcFzjOtypm2OjU0U7rk4SdqYxuXNwuvDc8Zj\nivLnmflVvdYfp3daUhq0mYd8ZwE3Pj+f/2VpjWKNJuz0xf6xXNvO527ypriiOTn31eadpSwfYTh2\nqQ8zFywAjWeABY/OVqSxsQkdHT0kyLNV9jnE4ad+rpliduI4Q0CsGYWNdEqTGZ9N2KnqZ5Qsh+WO\njR3qtKHwQ/MBA2GDG2o5WYTTkiVHRmNdwsbHl93b0tdq0RmcEnZ++w5Np1CJtcSylua1meLUEvlz\nr9Y0wo5CqnxhBvLwvcRzscS+n7x+xNNm5CPnwwZpgy5adEiQpDWOfgwTgMaCMBaI4eBW739cpVlC\nTa8thM/fN+KrtKjdhp2VVH3wjWcY1Zgd582G9c2aceMem1jLnSnVMOt7qmNW7nSk56uKNepMMOWR\ncvG3UQydzzS1MOQ77DBmzyEfmJsL5YmJoxJz+PRZrjbgaeBnwL8BT7ll//+TtY4dhJ8ztc0E/1ea\n2i666KLZ34033ji3t9Eiqflv8p6VD6OtTteRPk/Yswt7zLGpLVPVy41i+aP3H1veEE9pNjguDlVe\nEgigVILOsgYRm6/ysOMwgsr7nLx57Dgt+n1ic4XvxYe+E18OH0jhG8WNwXkyYVj2KVQJnriRDk0p\nsR/F349vlMOxMtl4kuJspWu1bJ7y/+eNVXEiM9Vi5ultGnYS8sjJMFy/qnOQ33MeLZZfvxgQEWZR\nCO87NOVNa3HcSxzp559lWA/Xqw+Uicf2FAM5Us+6yo9SnKXXk2W/iJ395ecQp68pTqFdri+Tk8cF\nmdrDcmQDuP2EfuWpuMuz6w4Px/VhfZQEtv2C58Ybbyy0lRbVlhY8i4LgghEXXPD8xH4tv4h2k5pJ\nsBiFFiaCzCr35OQJhXOUTXbhmI14RHRsYgqXiyGuZad6sZdeFdacCY84Mmibwko3buHESMvJBVH5\nA/Q9xDD1jxdsedl9tuTi8eXUK3mZvQ8jHm8SN1S1TG2hiTATBsU5VbxgOSi6TkoYZSPoTzvtNGfL\nXxmVK56kbpXm461SvpTaPfzMNxXOnZTqHOTPtZiQNB6/5d9J2NBv1GJIc9l/kQ3SDWcTDTtMy6Oy\npPwzvg6E4dKhhrVB84wJxXcYk/v1UvVA3bPwnRovNONcdFXHpL61orabWyqqOjXhhHnxLKxlE2Jf\nmdoWws+FU98F7AMuqNinxcfeGWLhUxxs6D+ishPcUzbZrY0aoNjBHDasvmcYjrnw19qWMDdl67II\nn7SKn5kuQqGUnuSuXO7QdxULAN+I5r1eOLhgg/fPsjiFRHg/sWkxzpAcP5+s8UiZt7yAnZw8QScn\nT5jNW1c0Ga530wAcE5UnDOaIhflBumjRCi2P6/CCxZukQlOeb2QP1VqaS+ikHhubCBo7/4zDBnVM\nw6msyzOHxkJUNQuB9wI2DnmOG9JyecoJRcN7iP2Dq4PrhINU4zKV36GPUPMzsq5btzHStCcVVgVm\nsjADRujgj//Glon4+5iezSxe7tykfFLhPqEPLtZ88s5V2MHoBCZ45iacWnjknSFlbsvmK0mbS2C5\nTk1N1ZgS2pubwoobDrYLP2b/EaX8HSlzSN5gpDIvFENaw15rWkClBI9vHPJpFvIGudHQ0XLm5rgh\n8GX3jUnYKKa1nPDcoclneHhZ4f/UNMT5s/Jh2v6ZhCYuL2DCRjQUsmNaTJbqyxxPvR2at4rPanLy\nuOiZhO895a8qNpplwRAe5zWfcIR+nAQ3Hy8msmJ2qva87saZpDcG5w7HIaXm4UmZfMvvMOsU+brk\n62hYvvSg1lyTnozK6OtMnoWgOJV1vXofa/Ypwb8+KNfG5LuJO0edwgTPPBE8qca36CwuZy0WGSvZ\nu8sV3pva4p5YqOanesfZR5RrOsXMyGVtJTcB+jDO4rZwQGRx31o+rnJGgenk1Ny1bNlhb9rPCloM\ngkjPBuqnAve2/NCmX87tVr43H1FWdkCHjXQ4a2z4fsszqQ4PHxo4wL22EY+1CYVm3BjvcJFPcYfE\nazlxDzsleHYEz79WB8W/8+I9LF48rqlcduXMznGmi7DuVmVfz8rhzYJVQQC5nzIePuDvsRzEE/p1\nsm8uNc1Cui4W63dx8ryiT7b4rLwPqjjGL8y7WGwPUpp/pzDBM68ET7GBLZvaqhoF3xPMTQJ5csk4\n3NXvu1pHR1e5/FCxPyAXALnNO524sEpo+G25YAxNIv56WYPrP8LQYVv+wPMPu6wFLa1rViiHRnu/\nS9E/NDl5Qil0NhVSXda6Yht70X6fjpILtcV4NH3ZqewFeu4/CgMrYmFczHJQzMLtzX2pxrZYB4pT\nL+T3UdWQps1HWb3JBuSOJ669IzouL38u5ELBFAoa/200pgHn5V4VlKWesN2mxW9ruftuYv9T3lmJ\nG/9i+He5vsVBEykNO9fe49Dz9VqcdqRPMxfYr/8ETz64MjelnXbaaYUPPJ2zzTdc9dK7hNpHHN0V\n2otzYeAb3tSAtiVL1pR6kamPLR/v4ENA47DW4keYf3xxjz7uHZf9XbHZsWzmSvVQa5snqo+Lx+4s\n1fQYFd8Apxz4sZa0KjomTz80MrI8IdBDYVHMCxcmQS1nxSg/1/K0EOUed+zfqnrnxTo7quVBlanO\nxAmFDoxvwHOBFJulvCYfagtlrTv9LlUzs/NaLWuasSUh9d2VhzE0Ijzy66etB40+U5Elzi/nv4E4\nl1618G0XJnjmieApR9NktvSwB14cWBb2IHdoeezBei2aEHzvM+zNhVpU9Udbzk9W7fvwlM1kfnCi\nPzYlCBsrW+1ghFiTqYqSa+wDrR0yfbCGgiE9BXW1o7sYzVX2SVT1nv3zzXvQseZbPEdZmIfPacVs\njrkqzdVfL2WyqurZj41NROZM/05DH1dRmwsb8mLDH0fZxfWinEA2zj5efJcaHBcOdM21+bKwrl9n\nqnyVxe0p0161hpaXofgdFKNMU1qbCZ6+/fWT4CnOxlk2eYVO0byxCtNrxFE2ZXNPnqE61bjnDWKs\nARSnDKgvqKpH9B+kufAJo7GqtIR0Ly5lM89Hc6ca+VgwlcOEq3ry1feyKunDKSfuTN1n8V4yE0ru\no0g1mlWkox+rzEZlrXpqamr2PD5CL7520dwYT3iWGne2o+K+faelPDi03ECX62QtzaIo9MsDgdMC\nzQscn2vu+FJdzv2k9U279QRPZtWIhXFaSFSPycvK4KdJzwVT90KpVU3wzAvBk5mzvPbiw2zTjXvq\noyyOK4jHzRQFSdkUkkq9n2pQwtHm/qNd7xqisYqe5cZEo+OvcayWP8KiNjU8fHBlVFLZZh72+qob\ned8ATU6eUBDgZd9F/SCHsLHw5y4GPsxoObVP+V7KgQp5cs969abcoNbyA6bvoZafLn+fYcO2Pnme\nYoBASqPN/SPxs6/WTKob5kZNqeG1Qi2yOFC3rGHm9X48eb+p9xGH0IcCKns+jc0wmh6Tl34m9a7b\nCUzwzAPBk3/Y4bzwtUKUy2q3b4CKGk39j9Y7vIvp3tOV218jzosVOr7z+6mnzeRhp/U+/pRJp3gd\n1aLDubqRTx+b3Wst4VIviCIdnRQ3fGkNK9WojI9P1qwzcXnCCLw40jGfDynd027MRBQemw58SAcW\nNKZZNnJvtaK18mPqR+WlOgwpLSoXpBtrniMktkiEnZfiGKiiqbN+/Syb0kMfXJW22ilM8MwbwRNW\n8BmtsgPn4w+KudRCR3Kt3mtj5civ6X0q4UdfFlDTOjx8aJ1IsOoGoBnndXV5Y4dz5iOr+hBT95rK\ngRVmh/BRW2Gm4rKpq+wDiwMdyuUom/FqZRYuX7P4PH05fXaIdPhz7QG8ceNczG5dboi90IsDLnwO\nvmYbw1oCoepc+ZQFqQGftYVGdUdkh1ZNjxCWs6x5pYR7LHTK1oXw/suWifJEhlnHYmXwf+cSg4aY\n4JkHgifdWyt/3HkFToe+1ouOabwceaVOffTpMRfpMjQa8RNS1WusXd7GIpvKx+bmiZGRcjLQJUuO\nTD6bommuvp+siux9btN85tD1CgdVmkqqe/a5HyTP75cS+sXectW9xeUuvpN0J6Joki1G4rVKI2a3\n8j1UjZWprku1A1bKpsWsXm8o1dOU2bSoVfr6WZ5krsoykWviqeEFqTFHG+f0zBvBBM88EDyq6Tj/\nOLGj7znX82W0cu3Q9OaXq0Zcp53I9cvQiEAs964bO2crY3tiAVfL5FW7R1zftFfr2ecmmNo+hGI5\n4rD4OPN4bIas/WwafTf1tJBmOj1V+4amo0ZMx8XnUt6v0Xur7d8LhcaxiYi9TPjHWTK8BhKfv5k6\nnn/zYdYIf1x6ltROY4JnnggeT/iRxM5P38suzydfv8Gqdb2iTyBX1VP+JK9FFHtjrX/wMWV/QmMC\nNTdlNP4syo1VdfRSbcGjWivjcfy842dSJeDrl9mPDwobsSr/2tzqSb37aPZd19JQ8vodC9VylFo5\nlU3rnbBagrBYpqqOX22tO+7gNWoSzzsn3jwXmjNjf2vr5s1mMMEzzwRPSLmxy80pk5MnuPQjzfXy\nY1IfbDjvSfGDK9uQa/UUU6apWo1T3rtsPjS0UZNMvWN8+vrQj1O+z9SA1/qaTjPPqur5pLXiqmwB\n9f1r7aAVv2Jac/OaQLXpsph8tZgxoF6uvFbuyx+fHqNVHAjajBbjzx8mma0VQJGnSioKtsWLx6NJ\n6ar9Ru3EBM88FjxFoVDO7NxoL69WbzSlsYyNTSQ+uOprpM6fEpq1/DblgITmQkObabxb6XX6Y2O7\nfiMRV565mINSgi8zQ3n/UHnGzFb8a63QiraRP4uU1lAtKKuPKwbCNDsgtvbzjrNye4EfpgMa16mp\nqabrU7HOb1M/LYYfXxVSFXbfiPWh3ZjgmQeCJ0zPHoZWFudWT/sf6pl6avWyN28+00UBFVX1chr8\n5itzuZGt3esum5Dqm63Ce4xNL400JnGvuGo5PFeVE7rKRBOur9JgU+Wtf6x//3FW6yxMfa6msEap\n5ZOrdc1ikERoBtxW6ScpPv9GQsDT76kRQZx6V3lC1zO1nBuv/j3HFO8lbeaNO0pxRvSwvrei9beK\nCZ4BFzzl8OgdOjy8MqhIfkBpo6Pxix9S1QdYDtX0g0Fjdf7Ywqj6eDyGv4dUmHEzztRWP5pmzDz1\nNI7iM0mbLBrV5KqEXCPXSB2bNommc3RlU6a3FuzQLHkdLY+qr1eGmZnUtBdLdWpqqsEsCtWpm2q/\np/qmx1RdGRlZoe00XxYFTzlIoPge6ycVbVaDnwsmeAZc8KRzQcWhsdUDSotRN+WPoL5TPL9eblor\nJ03M5gYqZ7/Nk5sWe2qqtc1aqYGirXw0zWgRtfZNj7qvJ6BCgVo/n1zYM232nRU7C/nzyTOQ58+/\nEfNouyj6aurff/33V9+U24iGW7vjUxbiqWvG9bUYUVads7BRracoRMsdiDyLeTlqslbEaSc025i+\nEzzA7wL/CjwNnBRtu9DNCHonsCVYfxJwK3A3cHmwfgS42h3zVWBNsG3K7X8XcHaw/ijgJrft08Bw\njbK29WW0QlnwFCt0McVI2u+R9zobG3FfO0w6njFSNR3GuSPRgGYmsuHhQ2v6V6pMHY3Y3Wv7kmon\nMK2l1RQbptrCLCxHnty1mEKont+j1jurJ7Ti5xNrnHMxWzZLVYehXYKn1Q6JP7b4nnwj7gdjljtL\ntY+vP16r2fL6a0xMrCmVKe9ANNJ57GxS0Jh+FDzPA54L3BAKHuD5wC3AsBMO3wHEbfsacLJbvg7Y\n6pb/EPiQW34VcLVbXgF8F1gGLPfLbttngFe45SuA369R1ja/juYpm9qqB+dVNcq1tA7VWokvqxqL\nuMpLB4wAABjUSURBVNGsdvi2MpalFbNaLV9VeRBp9Xl9I12eyns6ES1W3yRWlUIozx6cdhjX0xTn\nYjJp1BxVVaea7TWn9m/0HurtN1e/Rbmz4yeRazy8fPv27bpoURhc4C0B5QGytXxL9Z5n3IEo+sEa\n04C7Rd8JntkCwI2R4LkAOD/4/0vAqcDhwB3B+rOAK9zyDHCqW14EPBzvo7mAeZVb/jEw5JbXAzM1\nyti2F9Eq3s49NLRMFy06REdHj2j6Q2u1Ia/WHlIhrmlTX96AlkdQV5WhlSioWsfUMl9VRzqlP2bf\nSPmZXKsGMBYbzMNL+4yPTybNYFNTU3VNbVXvpxnqmWAbE+Rza9DqBReE76Vqv7kInpS5LJvHpnH/\nTF6/44nXxnVy8rjS/o36ABvFd26q/IjdMKulGCTB80Hg1cH/HwHOBF4A7AnWvxDY5ZZvA44Itu0D\nxoFp4O3B+ncCfwKsBO4O1q8Gbq1Rxra9iFao74BudkxE8x9ndXnyQILiTKbFdPqq6ranTXGpa9Qb\nG5Q6ppHoulRDE5r0iucomzRTTtuq6xafeVkoFicxyxuhzE82rY2kTJkrtepF1bZ21aUqajWkVfu3\nKghT95JFcRYHYaY0F09xIrpmMlTXrj/N0kshk6JK8AzTQUTkeuCwcBWgwDtU9YudvHSb9pnl4osv\nnl3etGkTmzZtaq5Ec+Cyyz7M44+/l8xdBY8/Dv/zf+7i2mt3ctllHwZgenonW7durXme6enz+MpX\npnj88ez/0dHzmZ7e2XR5tm7dGl3779m6dStbtmwD/m/g+8ARwAYeeOD7s8c98MC/Aa8HzifrK/wT\ncBcbN04n7/mJJy4nU3Q/DDzAr/zKMTXv8bLLPswzz5zjzp8xNPRmpqc/XbP8jzzya9xyy+/hn+8z\nz1wZ7g1MMT5+KWvXHs7ttx/g9r0S+PPgmNsYGnozzzyTHeWfrb9GxoXAq2f/Gxl5K+9+91W8+tVv\njO7kn4D/AnwSeC3wltktrb6zWrSrXrSL3bt38/KXT/H440cD7yes95dd9uFkHSjXyfrfQy2e+9xn\nc/vtt/PEE6uAdyLy71x66VsaOOd5rrzvBaqfZbkOHs8tt7Rc3MJ553Lfc2Xv3r3s3bu3/o4padTN\nH/VNbTPkprY7g/WNmtquDI65ktzU9jBFU9uXapSxXR2Almhn77Id9vqqfeuVM9+entJ3rvdcNAHW\nTwBafa202aO4X2PZDFLaVRxhVfbjpOYf6pzj35czDE1OT4rWGVNbTP6cmwueqHVv9YJRao1ha+Q6\nxXeYBZGkpjGoVcZe+mI6BX1uantB8P9xZMEFI8DRFIMLbgJOIdNWrgNOd+vfQB5ccBbp4AK/vNxt\n+0wghK4A/qBGGdv7Npqk05WyynTWrJO3XjlrOUHrmcMauedWn1OVKbOZyLiqDN3+/PUasNBhnOXe\na4/ppVHiOhDPSJsKR+6UWafKh1jPxDzX4IW53ktqnFozxMLfBE9nBM7LgHuBx4EfhRoHmU3iO5TD\nqV9AZqPZB3wgWH8gcI1bfxNwVLDtHLf+borh1EeTRcnd7YTQATXK2t630QKdtN3W+tCb1UQa6V02\nGuLZyj23+pwaOa6W9tKO5JMhjU750C7qaXOdFHoxtTpCVfWvSsB02hfVTubawWyHNaPd9J3gGaRf\nPwieTlLPtJHet/UPuZkPrNfO0vj6rZoZ23HtTtKK4OlkD73Z59yrIIh23JOnmbKm6mWnow9bwQTP\nAhU8zfXk648Nale6jVY0jG5/NM0KyH6y0bfiB6kytVWlsWk26rAdNKvZ9Mt7aaQcjQqe1LnqTyzX\nG8FrgmcBCp5mG85U+GpK0Mw1jXyj9Pqjafb6rTT2KY1hrppOqxplI8lPPdmz6ew0ClU068vptdas\n2njKn0beWyoYpjxMYXrW32SCZ0B/gyp4Wmm444+0l43/XEwP3b5+s1RpDO3QKOfSc270Wv0meGqt\n7weaeSf17qF8rjhHY2MZNbqBCR4TPJW9rHbZndtNMxFJnTCndNJMU9Vwt6OHmuoRpxKizuXdtsPU\n1mrwSD+YzpqlneWOA0/KCWjjQcfVCXE7jQmeBSh46lX2Rj6GWhFG3bqH5nuA7cvA26ledCcFT6P+\nmrl2KuYSXNBqQ9xr8+tcaFd9zPPrZYmA44nluh2KXwsTPAtQ8KjWruzNqP/NpC/pNqn76HVyxHp0\n0tTmz795c+1cb73UHloVIIMkeLpp/u1EAFA7MMGzQAVPLZr5iPv5g081oO0aV9NJv0GnggtC6r23\nXvlFWq1PnRaW7XoenSpnO31F3cAEjwmeEs18HP0seFQ7ExQxqP6EkH69h7mUq5FGtdf+o059L/36\nPqswwWOCJ0mjH+igVfh2lLcXwraZBrPWvlVh0v30zjrVK+8H/1GnIyJbEaq9qAMmeEzwzJlOV952\nn3+u5+u24Gmmway17yB2Etr53vvBf9Std9Co9ter+mCCxwRPX9OPtvtuf7Dt8rn1u1k0pBPPuF/8\nR93oqLU24LR79aFK8HR0Ph7DaJTUnENVc680Sz6/SzZHyle+MsW119afr6Xdc7wYZTrx3ludX6jd\n77vTc+N08pvpOClpZL+Fq/EMYpRTq+N8+o2FaGrrpBO+H/1a7aSZCDcztQ3gb6EInl5W0Fau3c+m\nhlYbvnYGFwzC3C6DJCT7jWY7KhZcMGC/hSJ4eq0ZNBsm2+hYnW43bv3QmNbTiPpJG+i38gwS/f7s\nTPCY4KlLrwVPPeLGtJnUIN38QOfyHNtVzqoy9INQNBYOfSd4gPeRzTD6LeDzwNJg24Vks4bGM5Ce\nBNxKNmvo5cH6EeBqd8xXgTXBtim3/10UZyA9imy20ruBTwPDNcra3rfRJjoRftzPjVK5MZ3uyzQ+\n/RBVVVWGfu9cGPOLfhQ8pwFDbvk9wLvd8nHALcCwEw7fAcRt+xpwslu+Dtjqlv8Q+JBbfhVwtVte\nAXwXWAYs98tu22eAV7jlK4Dfr1HWNr+OuZNqpNoxULCfVfdUo7lu3Ya+u+d+GMBYVQYTPEY36TvB\nUygEvAy4yi1fAJwfbPsScCpwOHBHsP4s4Aq3PAOc6pYXAQ/H+2guYF7lln8cCL71wEyN8rXtRbSL\nQen9t5NOaGSd0vJaEWbtFgqpMvS7VmvML/pd8OwC/rNb/iDw6mDbR4AzgRcAe4L1LwR2ueXbgCOC\nbfuAcWAaeHuw/p3AnwArgbuD9auBW2uUr20vol2UG6neTMrVbfpllHsn6KfR7obRDqoET0cHkIrI\n9cBh4SpAgXeo6hfdPu8AnlTVT7fz0m3aZ5aLL754dnnTpk1s2rSpuRK1kd27d/PIIw8xNPRmnnkm\nWzc0tG92edDYvXt3MGjvvJoD4Do9KK+XdGvA6nx+hkZv2bt3L3v37q2/Y0oadesHnAP8E3BgsC42\ntc2Qm9ruDNY3amq7MjjmSnJT28MUTW1fqlHOdnUA5kyxV5xPzNZPc3A0Q69NP72+vmF0AxvHkzfm\npwO3Ayuj9T64YAQ4mmJwwU3AKWTaynXA6W79G8iDC84iHVzgl5e7bZ8JhNAVwB/UKGubX0fr1DIN\nDaIJpR9MXYP43Pode6b9Qz9mLuhlrrYPkgmX60UE4CZVfYOq3iEi1wB3AE8Cb3A3APBGYAewGLhO\nVWfc+o8CV4nIPuAnZMIHVX1MRC4FvkFm4rtEVX/qjrkAuNptv8WdY6Dx5pPLLvvwrLnGTCr1MdNT\ne2k1N57RGfoyp1tKGtmvfzWe+ZKjyzOIZTZq0w9arJFj2amNOVPLAd2XPZs6WAZow+gsrWbr7iQm\neAaQdpuGmokq6wRm6ppf9GNDt5Dpx86dd9obNRARHYTnFNvWR0fPr2tbb+UYw6hHrzszRn8gIqhq\naeiKCZ4GGBTBA81/8Fu2bOP668/Am+dgJ5s372LPns93tqCGYcx7qgSPmdrmGWa2Mgyj3xnqdQGM\n3jI9fR6jo+cDO4Gdzh5/XlPn2L17N1u2bGPLlm3s3r27I+U0DGP+YKa2BhgkU1srzMUebz4iwzCq\nMB/PHJjvgmcumI/IMIwqqgSPmdoMYx4wCObOQSij0R0suMCYEzZmo/cMQoqaQSij0T3M1NYAZmqr\nTa/HbPT6+r1mEMydg1BGo/1YOLXRNI026L0M4baetGEMIKkEbvbr3ySh3WJQkndaQsrBeFeDUEaj\n/WBJQo1mGMSEowuVfszFFTMIZTS6hwkeY6Cx4IaMQchYMQhlXCj02i9qwQUNsBCDCwZpYGivPyLD\nGCS6+W333QBSEfmvwEuBZ4CHgHNU9UG37ULg9cBTwJtUdY9bfxLFGUj/2K0fAT4BvAB4hGxK6x+6\nbVPAO8hmIP1TVf2EW38UcDUwDvwz8DpVfaqirAtO8IA16IYxH+lmhGE/DiB9n6r+mqquA/4/4CIA\nETkOeCXwfOAlwIfEzY0NXAGcq6rHAMeIiG8JzwUeVdXnApcD73PnWgG8CzgZOBW4SESWuWPeC1zm\nzvVTdw4jYOvWrezZ83n27Pm8CR3DMNpGzwSPqv48+PdgMs0H4AzgalV9SlV/AOwDThGRw4Elqnqz\n2+8TwMvc8kvJslwCfA54sVveCuxR1f2q+lNgD3C62/ZiwIv4ncDL23VvhmEY/Uo7EgPPlZ4GF4jI\nduBsMo3jN93qZwFfDXa73617CrgvWH+fW++PuRdAVZ8Wkf0iMh6uD88lIiuBx1T1meBcR7TrvgzD\nMPqVfogw7KjgEZHrgcPCVWS+lneo6hdV9Z3AO0XkfOD/AS5u16XbtM8sF1988ezypk2b2LRpU3Ml\nMgzD6BM6FWG4d+9e9u7dW3e/jgoeVd3c4K6fIvPzXEymlRwZbFvt1lWtJ9j2gIgsApaq6qMicj+w\nKTrmRlX9iYgsE5Ehp/WE50oSCh7DMAyjTNwpv+SSS5L79czHIyLPCf59GfBtt7wLOEtERkTkaOA5\nwNddxNt+ETnFBRucDXwhOMaHaLwCuMEt7wY2OyGzAtjs1gHc6PbFHevPZRiGYXSQXoZTfw44hiyo\n4B7gD1T1R27bhWRRZk9SDKd+AcVw6je59QcCVwHrgJ8AZ7nABETkHPJw6u1BOPXRZOHUK4BbgNeq\n6pMVZV2Q4dSGYRhzoe/G8QwSJngMwzCapx/H8RiGscCwyeAMMI2nIUzjMYy5M0hpmIz2YKa2OWCC\nxzDmjk0Gt/AwU5sxLzHTjWEMHjYtgjGw2Oyjg4VNYWF4zNTWAGZq60/MdDN4WMbzhUWVqc00HsMw\nuoZNBmeACR5jgDHTjWEMJmZqawAztfUvZroxjP7FwqnngAkewzCM5rFwasMwDKMvMMFjGIZhdBUT\nPIZhGEZXMcFjGIZhdBUTPIZhGEZX6bngEZFpEXlGRMaDdReKyD4RuVNEtgTrTxKRW0XkbhG5PFg/\nIiJXu2O+KiJrgm1Tbv+7ROTsYP1RInKT2/ZpEbExTYZhGF2gp4JHRFaTTUd9T7Du+cArgecDLwE+\n5Ka6BrgCOFdVjwGOERE/aONc4FFVfS5wOfA+d64VwLuAk4FTgYtEZJk75r3AZe5cP3Xn6Gv27t3b\n6yK0zKCW3crdXazc3acXZe+1xvN+4K3RupcCV6vqU2766n3AKSJyOLBEVW92+30CeFlwjB+y/jng\nxW55K7BHVfer6k+BPcDpbtuLAZ/Uayfw8rbdVYewyt19rNzdxcrdfRaU4BGRM4B7VfW2aNOzgHuD\n/+93654F3Besv8+tKxyjqk8D+53pLnkuEVkJPKaqzwTnOmLON2UYhmHUpaN+DRG5HjgsXAUo8E7g\n7WRmto5cuk37GIZhGO1GVbv+A44HHgS+B3wfeBL4AXAocAFwQbDvDJl/5nDgzmD9WcAV4T5ueRHw\ncLDPlcExVwKvcssPA0NueT3wpRrlVfvZz372s1/zv1Sb2pNILlX9VzJBAoCIfB84SVUfE5FdwN+L\nyF+SmcqeA3xdVVVE9ovIKcDNwNnAX7lT7CKblOVrwCuAG9z63cCfuoCCITIN6wK37Ua372fcsV+o\nUV7TjgzDMNpEv4QQK870pap3iMg1wB1kmtAbggydbwR2AIuB61R1xq3/KHCViOwDfkKm6eAE2aXA\nN9w1LnFBBpAJoKvd9lvcOQzDMIwOY9mpDcMwjK7S63Bqo0FE5HQR+bYb8Hp+r8sTIiKrReQGEbld\nRG4TkT9y61eIyB43eHd3MIaqcpBwLxCRIRH5pjPzDkS5RWSZiHzWleN2ETl1QMr9ZhH5VzcQ/O/d\n4O++LLeIfFREHhKRW4N1TZe1auB7l8v9Pleub4nI50VkaU/L3YvgAvs1HYwxBHwHWAscAHwLOLbX\n5QrKdzhwolseA+4CjiUbpPs2t/584D1u+Tgy8+YwcJS7N+lh+d8MfBLY5f7v+3KTmZx/zy0PA8v6\nvdxkQxa+B4y4/71/tS/LDbwQOBG4NVjXdFnJfM8nu+XrgK09KPdp5MFU7wHe3ctym8YzGJwC7FPV\ne1T1SeBqskGzfYGqPqiq33LLPwfuBFZTHNi7k3zA7xkkBgl3tdAOlz3jt4GPBKv7utyut/ofVfXj\nAK48++nzcjsWAQe7FFWjZGPr+rLcqvoV4LFodVNlrTPwvWvlVtUvaz5u8Say77Nn5TbBMxjEA2HD\nwbN9hYgcRdbbugk4TFUfgkw4kYXLQ/Ug4V7gs2eEzs5+L/fRwCMi8nFnIvywiBxEn5dbVR8ALgN+\n6MqwX1W/TJ+XO+LQJstaa+B7r3g9mQYDPSq3CR6jbYjIGFnKojc5zSeOXOmrSBYR+R3gIaet1QqZ\n76tyk5lFTgL+RlVPAv6dLEqz35/3cjKNYS2Z2e1gEXkNfV7uOgxSWRGRdwBPquqne1kOEzyDwf3A\nmuD/1W5d3+BMJ58DrlJVPybqIRE5zG0/nGzQLmRlPzI4vFf3swE4Q0S+B3waeLGIXAU82Oflvo8s\n3dQ33P+fJxNE/f68TwO+p6qPapba6lrgN+j/coc0W9a+uQcROYfMrPzqYHVPym2CZzC4GXiOiKwV\nkRGycUq7elymmI8Bd6jqB4J1u4Bz3HI4SHcXcJaLaDoaN0i4WwX1qOrbVXWNqj6b7JneoKqvA75I\nf5f7IeBeETnGrfot4Hb6/HmTmdjWi8hiERGyct9Bf5dbKGrDTZXVmeP2i8gp7p7PpsZg9U6VW0RO\nJzMpn6Gqvwz26025OxldYb+2RqqcThYtto8gpVA//Mg0h6fJou1uAb7pyjsOfNmVew+wPDjmQrII\nmjuBLX1wDxvJo9r6vtzAr5F1SL4F/DeyqLZBKPdFrgy3kjnnD+jXcgOfAh4AfkkmNH8PWNFsWYEX\nALe5b/cDPSr3PrLpZ77pfh/qZbltAKlhGIbRVczUZhiGYXQVEzyGYRhGVzHBYxiGYXQVEzyGYRhG\nVzHBYxiGYXQVEzyGYRhGVzHBYxgtIiLvcCn+/8XlTDu5w9e7UUROamL/j4vImU1e4/siMt586Qyj\ncfplBlLDGChEZD1Z+pETVfUp11iP9LhY7cAG9hkdxzQew2iNCeARVX0KQLP8Yw8CiMj/KyJfc5No\nXekPcBrLX4rIzZJN3vbrblKuuySbgh2XFulOEfmkiNwhIteIyOL44iKyWUT+t4h8Q0Q+47JTV+I0\nmYtF5J+dhnaMWz/uJjS7TUT+jmKalde4+/imiFwhGWvcxGDj7v9/FJHT2vFAjYWDCR7DaI09wBrJ\nZoX9GxF5UbDtg6p6qqr+KnCQy4Lt+aWqngz8LVnuqz8ETgDOEZEVbp/nAX+tqscB/wa8IbywiKwE\n3gn8lqr+OvDPwHQDZX5YVV8AXAm8xa27CPhfqnoCWdLONe4axwKvAn5DswzYzwCvUdUfkk0kdqW7\n5u2aTW1gGA1jgscwWkBV/50sI/R5wI+Bq0XkbLf5t0TkJsmmHv5N4FeCQ31y19uAf1XVh1X1CeC7\n5NmAf6iqN7nlT5LNKBmynmzmyH8SkVvIEjiuoT7Xur//TDbbJMCL3DVQ1evIJxD7LXd/N7trvBh4\nttvvY8BS4PfJBZhhNIz5eAyjRTRLdPiPwD+KyG3A2SLyGeBvgJNU9QERuQgITWU+M/AzwTJkvpWq\n7zH2uwiwR1Vf02SR/fWernEtCf7uVNV3lHYQGSWfwXKMbD4gw2gY03gMowVE5BgReU6w6kSy7L+L\nyQTFT9zEeL/bwunXiMipbvnVwP+Ktt8EbBCRSVeWg0TkuS1cBzLB+Rp3npcAy936fwB+V0QOcdtW\niIjXqt5LpiW9i+KU4YbREKbxGEZrjAEfFJFlwFNkaeXPU9X9zkl/O/AjivPH1IoYC7fdBbxRRD7u\nznNluI+qPuIm9fq0iBzo1r+TLH191Tmrrn2JO89ZwP8mS6OPqt4pIu8E9ojIEPCEK9NRwK8DG1RV\nRWSbiEyp6s4a92YYBWxaBMPoI0RkLfA/nLPfMOYlZmozjP7DeoPGvMY0HsMwDKOrmMZjGIZhdBUT\nPIZhGEZXMcFjGIZhdBUTPIZhGEZXMcFjGIZhdBUTPIZhGEZX+T/dRedc+lAb/gAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1096f81d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(range(m2), (ypredict - yvalidate))\n",
    "plt.xlim(-150, 1300)\n",
    "plt.title(\"Error versus Sample Index\")\n",
    "plt.xlabel('Sample Index')\n",
    "plt.ylabel('Error')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And indeed, the errors appear to be identically and independently distributed across samples. We can check whether the errors do distribute according to a normal distribution with mean zero and unknown standard deviation:\n",
    "\n",
    "$$\\epsilon \\stackrel{iid}{\\sim} N(0, \\sigma)$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEZCAYAAAC0HgObAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGtRJREFUeJzt3Xu0JWV55/Hvj6sX5CJKHwWlvbAUDYoaUIes8WhGJOMo\n6hhFncH7OLK8jImjoAx0spJRnImXZdSJiTpKVERnGSUrIk30aEIUVERQbq2hm4t0y6CCtyFAP/NH\n1aF3b/Y5Z5/u2pdz+vtZa69T+63aVc+uU7ueXe/71rtTVUiS1JXdJh2AJGl1MbFIkjplYpEkdcrE\nIknqlIlFktQpE4skqVMmFk2VJN9P8q8nHcckJXlukmuT3JrksZOOR1ouE4vGJsk1SZ7WV/bSJP8w\n/7yqfquqvr7Eeg5NsjXJaj1+/wdwUlXtW1Xf65/ZvvdftIln/u+bJxCnNNAekw5AApZ7l27a12QE\nsZBk96q6cxTrHtKhwOWLzC/gMVV1zVIrGvRelvv+pmB/aIVZrd/4tEL1XtUkOSrJt5LckuTGJP+z\nXexr7d+ft9/Wn5jGqUk2Jtmc5H8n2bdnvSe2825ql+vdzulJPpvkzCQ/B17abvufkvwsyQ1J3p9k\nj571bU3y2iRXt/H9cZKHJrkgyc+TnNW7fN97HBTrfZLsleQXNJ/LS5NsWGg3sUBSXeC9DCrbK8l7\n2/d2fZL3JNmzXcdTklyX5C1JbgQ+muTAJOe0++PmJF8btH0JTCyavMWuOt4HvLeq9gMeBpzdls+3\nwezbVhddCLwcOBF4CvBQ4D7AnwMkeRTwAeBFwAOA/YAH9m3r2cDZVbU/8EngDuC/APcFngw8DTip\n7zXHAo8DngS8BfgL4MXAg4Aj2u0NMijWD1TVv1TVfdp9ckRVHbbIvllM/3vpL/sUcCpwNPAY4LHt\n9Kk965gB9gceDPwn4A+B64ADgYOAt+1gbNoFmFg0bn+T5KfzD5oT/kL+BXh4kgOr6tdVdVHf/N6k\n9GLg3VW1qap+DZwCvLBth/n3wBer6htVdQdw2oBtfaOqzgGoqtuq6rtVdVE1rgU+TJMIep1RVb+q\nqiuA7wPntdv/BfAlmqQzyKBYT+hrM1qqmu/idh/+rP379IXey4Cy/9fG8EdVdXNV3Qz8EfAfe9Zx\nJ3B6Vd3eruN2mqT8kKq6s6ouWCI+7cJMLBq346vqvvMP7n4V0OuVwCOAK5NcmOSZiyz7QGBTz/NN\nNG2Ia9p5183PqKrfADf3vf663idJDmurfm5sq4/+FLhf32t+0jP9G2BL3/N9diDWYT2u3YcHtH/X\nL/ReFih7IHBtXwy9V3E3VdXtPc/fBfwIOC/JD5O8dRmxahdjYtG4Dd3gXlU/qqoXV9X9aU5sn0ty\nTwY39v+YptF73qE01VlbgBuBQ+4KoFnHgf2b63v+IeAK4GFt9dHblxP7EgbFejvbJ6alLBbLoP3T\nX3bDgBh+vNDy7ZXZm6vqYTTVan+Q5KnLiFe7EBOLplaSlySZv0q4heZktxW4qf37sJ7FPw28Kcna\nJPvQXGGcVVVbgc8Bz0rypLaBet0Qm78PcGtV/TrJI4HXdvKmlo51XM4CTk1yv3Yf/zfgzIUWTvLM\nJPP7+xc0SXuc8WoFMbFonIbpVty7zHHAD5LcCrwHeGHb/vEbmpPxBW37wtHAR2lOjF+nqbL5NfAG\ngKq6HHg98Bmab+W30lRj3cbC3gy8pN32X9CciBd7L8vpMr1grEOuq4Dv9d3H8u5lbB/gT4BvA5cC\n32un/3SR5Q8Dzm97rV1A09nAnmEaKKP8oa8khwCfoKk73gp8uKren+R04NVsq6N+W1Wd277mFOAV\nNN+I3lhV540sQO2Sktwb+Dnw8KratNTykpZn1IllBpipqkvaS/7vAMcDLwR+UVXv7lv+cJqukEfR\n1ImfDxxW/syldlKSfwf8Pc1V+p8BR1XVEyYblbQ6jbQqrKo2V9Ul7fQvaRpDD25nD2p8PJ6mrvmO\nqtoIbKDpXy/trONpqsGup2mbOWGy4Uir19jaWJKsBY4ELmyLXpfkkiR/lWS/tuxgtu8WeQPbEpG0\nw6rq1W3X3AOq6ulVtdBd7ZJ20lgSS1sN9jmaNpNfAh8EHlpVRwKbaaomJEmrwMgHoWzHS/occGZV\nfQGgqm7qWeQvgXPa6RtohsOYd0hb1r9O21wkaQdU1UgGb+01jiuWjwKXV9X75gvaRv15z6MZDgPg\nizRDW+yV5CHAw4H+YTwAqKqpe5x++ukTj8GYjGlXjMuYhnuMy0ivWJIcA7wEuCzJd2n6378NeHGS\nI2m6IG8EXgPN/QZJzqYZMvx2mt+k8OpEklaQkSaWagaq233ArHMXec07gHeMLChJ0kh5532HZmdn\nJx3C3RjTcIxpeNMYlzFNl5HeIDkqSawhk6RlSkKtksZ7SdIuxMQiSeqUiUWS1CkTiySpUyYWaZlm\nZtaSZNHHzMzaSYcpTYy9wqRlSsLSv8WVsd7pLA3DXmGSpBXJxCJJ6pSJRZLUKROLJKlTJhZJUqdM\nLJKkTplYJEmdMrFIkjplYpEkdcrEIknqlIlFktQpE4skqVMmFklSp0wskqROmVgkSZ0ysUiSOmVi\nkSR1ysQiSeqUiUWS1CkTiySpUyYWSVKnTCxSj5mZtSRZ9CFpcamqScewbElqJcat6dckjqWOreGW\n8RjVtElCVY3825FXLJKkTplYJEmdMrFIkjplYpEkdcrEIknq1EgTS5JDknwlyQ+SXJbkDW35AUnO\nS3JVki8n2a/nNack2ZDkiiTHjjI+SVL3RtrdOMkMMFNVlyTZB/gOcDzwcuDmqnpXkrcCB1TVyUke\nBXwSOAo4BDgfOKy/b7HdjTUqdjfWarYquhtX1eaquqSd/iVwBU3COB74eLvYx4HntNPPBs6qqjuq\naiOwATh6lDFKkro1tjaWJGuBI4FvAmuqags0yQc4qF3sYOC6npfd0JZJklaIPcaxkbYa7HPAG6vq\nl0n66wiWXWewbt26u6ZnZ2eZnZ3dmRAladWZm5tjbm5u7Nsd+ZAuSfYA/hb4UlW9ry27Apitqi1t\nO8xXq+rwJCcDVVVntMudC5xeVRf2rdM2Fo2EbSxazVZFG0vro8Dl80ml9UXgZe30S4Ev9JSfkGSv\nJA8BHg5cNIYYJUkdGXWvsGOArwOX0XzFK+BtNMnibOBBwCbgBVX18/Y1pwCvBG6nqTo7b8B6vWLR\nSHjFotVsXFcsjm4s9TCxaDVbTVVhkqRdiIlFktQpE4skqVMmFklSp0wskqROmVgkSZ0ysUiSOmVi\nkSR1ysQiSeqUiUWS1CkTiySpUyYWSVKnTCySpE6ZWCRJnTKxSJI6ZWKRJHXKxCJJ6pSJRZLUKROL\ndhkzM2tJsuhD0s7zN++1y+jy9+z9zXutRP7mvSRpRTKxSJI6ZWKRJHXKxCJJ6pSJRZLUKROLJKlT\nJhZJUqdMLJKkTplYJEmdMrFIkjplYpEkdcrEIknqlIlFktQpE4skqVMmFklSp0aaWJJ8JMmWJJf2\nlJ2e5PokF7eP43rmnZJkQ5Irkhw7ytgkSaMx6iuWjwHPGFD+7qp6fPs4FyDJ4cALgMOB3wM+GH/S\nT5JWnJEmlqr6R+BnA2YNShjHA2dV1R1VtRHYABw9wvAkSSMwqTaW1yW5JMlfJdmvLTsYuK5nmRva\nMknSCjKJxPJB4KFVdSSwGfizCcQgSRqRPca9waq6qefpXwLntNM3AA/qmXdIWzbQunXr7pqenZ1l\ndna2sxglaTWYm5tjbm5u7NtNVY12A8la4JyqOqJ9PlNVm9vpNwFHVdWLkzwK+CTwRJoqsPXAYTUg\nwCSDiqVFNX1BljpuulvGY1TTJglVNfJOUSO9YknyKWAWODDJtcDpwFOTHAlsBTYCrwGoqsuTnA1c\nDtwOnGT2kKSVZ+RXLKPgFYt2hFcs2tWN64rFO+8lSZ0ysUiSOmVikSR1ysQiSeqUiUWS1CkTiySp\nU0MlliTHDFMmSdKwVyzvH7JMkrSLW/TO+yRPBv4VcP8kf9Aza19g91EGJklamZYa0mUvYJ92ufv0\nlN8KPH9UQUmSVq6hhnRJcmhVbRpDPENxSBftCId00a5u2gah3DvJh4G1va+pqqeNIihJ0so17BXL\n94D/BXwHuHO+vKq+M7rQFo3HKxYtm1cs2tVN2xXLHVX1oZFGIklaFYbtbnxOkpOSPCDJfecfI41M\nkrQiDVsVds2A4qqqh3Yf0tKsCtOOGG9V2D2A25aMac2aQ9m8eeOSy0ldGFdVmD/0pV3GuNtYll6m\nWc5jWeMyVW0sSU4cVF5Vn+g2HEnSSjds4/1RPdP3AH4XuBgwsUiStrNDVWFJ9gfOqqrjug9pqO1b\nFaZlsypMu7pp/837XwEP6TIQSdLqMGwbyzls+/q1O3A4cPaogpIkrVzDdjd+Ss/TO4BNVXX9yKJa\nOh6rwrRsVoVpVzdVVWFV9TXgSpoRjg8A/mWUQUmSVq5hf0HyBcBFwO8DLwAuTOKw+ZKku1nOIJRP\nr6qftM/vD5xfVY8dcXwLxWNVmJbNqjDt6qaqKgzYbT6ptG5exmulkZqZWUuSJR+SxmPYGyTPTfJl\n4NPt8xcCfzeakKTl2bJlE8NeHUgavUWrwpI8HFhTVRckeR7wO+2snwOfrKofjSHGQXFZFaa7DFfF\nBeOt5rIqTNNnKgahTPK3wClVdVlf+RHAf6+qZ404voXiMrHoLiYWaTjT0saypj+pALRla0cSkSRp\nRVsqsey/yLx7dhmIJGl1WCqxfDvJq/sLk7wKmMjv3UuSpttSbSxrgM/T3Gk/n0h+G9gLeG5VbR55\nhIPjso1Fd7GNRRrOVDTe9wTzVOC32qc/qKqvjDSqpeMxseguJhZpOFOVWKaNiUW9TCzScKalV9hO\nSfKRJFuSXNpTdkCS85JcleTLSfbrmXdKkg1Jrkhy7ChjkySNxqiHZfkY8Iy+spNpxhl7BPAV4BSA\nJI+iGeDycOD3gA/GcTgkacUZaWKpqn8EftZXfDzw8Xb648Bz2uln0/zc8R1VtRHYABw9yvgkSd2b\nxECSB1XVFoC2V9lBbfnBwHU9y93QlkmSVpBpGKHYlktJWkWGHd24S1uSrKmqLUlmgPnh+G8AHtSz\n3CFt2UDr1q27a3p2dpbZ2dnuI5WkFWxubo65ubmxb3fk3Y2TrAXOqaoj2udnAD+tqjOSvBU4oKpO\nbhvvPwk8kaYKbD1w2KB+xXY3Vi+7G0vDGVd345FesST5FDALHJjkWuB04J3AZ5O8AthE0xOMqro8\nydnA5cDtwElmD0laebxBUiueVyzScFbFDZKSpF2PiUWS1CkTiySpUyYWSVKnTCySpE6ZWCRJnTKx\nSJI6ZWKRJHXKxCJJ6pSJRZLUKROLJKlTJhZJUqdMLJKkTplYJEmdMrFIkjplYpEkdcrEIknqlIlF\nktQpE4skqVMmFklSp0wskqROmVgkSZ0ysUiSOmVi0VSbmVlLkkUfkqbLHpMOQLuumZm1bNmyaYgl\na4n5JhdpmqRqqQ/t9ElSKzFuba+52hgmaXSxTJfr6jYmj2WNSxKqauTfxKwKkyR1ysQiSeqUiUWS\n1CkTiySpUyYWaaL2XrI79czM2kkHKS2LvcI0MfYKG35dHu/qgr3CJEkrkolFktQpE4skqVMmFklS\npyY2VliSjcAtwFbg9qo6OskBwGeAQ4GNwAuq6pZJxShJWr5JXrFsBWar6nFVdXRbdjJwflU9AvgK\ncMrEopMk7ZBJJpYM2P7xwMfb6Y8DzxlrRJKknTbJxFLA+iTfSvKqtmxNVW0BqKrNwEETi06StEMm\n+Xssx1TVjUnuD5yX5CrufqfYgneFrVu37q7p2dlZZmdnRxGjJK1Yc3NzzM3NjX27U3HnfZLTgV8C\nr6Jpd9mSZAb4alUdPmB577xfBbzzfvh1ebyrC6v6zvsk90qyTzt9b+BY4DLgi8DL2sVeCnxhEvFJ\nknbcpKrC1gCfT1JtDJ+sqvOSfBs4O8krgE3ACyYUnyRpB01FVdhyWRW2OlgVNvy6PN7VhVVdFSZJ\nWr1MLJKkTplYJEmdMrFIkjplYpEkdcrEIknqlIlFktQpE4skqVMmFklSp0ws6tzMzFqSLPmQtDo5\npIs6N9xQLTCtw6dMY0we7+qCQ7pIklYkE4skqVMmFklSp0wskqROmVgkSZ0ysUiSOmVikSR1ysSi\nZRnm5kdJuzZvkNSydPc79cMut1K35w2Smj7eIClJWpFMLJKkTplYpKm395LtWjMzaycdpHQX21i0\nLLaxTG9Mfia0FNtYJEkrkolFktQpE4skqVMmFklSp0wsuot31Uvqgr3CdJfuenxNYy+tcW/PXmGa\nPvYKkyStSCYWSVKnTCySpE6ZWHYBwzTK2zAvqSs23u8ChmuUh5XccD1927PxXtNnl268T3JckiuT\nXJ3krZOOZ5rZRVhdG+aYctBLLWbqEkuS3YA/B54BPBp4UZJHTjaq4czNzY19m1u2bKL5NrvQ46tj\nj2lpc5MOYIC5SQcwwNwyll16BOQk7L77vZdcZphjqllmekzis7eUaYxpXPaYdAADHA1sqKpNAEnO\nAo4HrhzVBtevX8+FF1645HInnngiD37wgxecPzc3xwknvGyoD91uu92LrVt/vdPLLG1uJ18/CnPA\n7IRj6DfHyo7pNoapetu6ddiqt8XMDRfSGM3NzTE7OzvpMLYzjTGNyzQmloOB63qeX0+TbEbmNa95\nCxs3PgGYWWSpr7F161ZOO+20Rde17dve4ob5gHdzEpCk8ZrGxDJ2e++9J/e+99XsttuWBZe57bbN\n7LnnnmOMSppmew/VfrdmzaFs3rxx0WVmZtYueZU/zHqm1Wp/f4NMXa+wJE8C1lXVce3zk4GqqjN6\nlpmuoCVphRhHr7BpTCy7A1cBvwvcCFwEvKiqrphoYJKkoUxdVVhV3ZnkdcB5NL3WPmJSkaSVY+qu\nWCRJK1xVTfQB/CGwFbhvT9kpwAbgCuDYnvLHA5cCVwPv7SnfCzirfc03gAf3zHtpu/xVwIk95WuB\nb7bzPk1z9fbHwPeA7wLnAjOTjqktf1e73UuA/wPsO+m4gOcD3wfuBB7f9z+d2L7awWPwOJru7FcD\nb+3gmP4IsAW4tKfsAJqr8KuALwP7jXl/HQJ8BfgBcBnwhknHBewNXEjzebsMOH3SMfXM2w24GPji\nFMW0kW3np4umJa6Bn4Gd/RDt5AfwEJoT+DW0iQU4vN1xe9CcPH7ItiurC4Gj2um/A57RTr8W+GA7\n/ULgrJ6d/iNgP2D/+el23meA32+nPwS8BtinJ7bXAx9qpx81qZja6X8D7NZOvxN4x6TjAh4BHEZz\nsnp8z36b2P9vB4/B3doYDwX2pEnej9zJ4/p3gCPZPrGcAbylnX4r8M4x/w9ngCPb6X1oThCPnIK4\n7tX+3Z3mi8LRk46pnf8m4K/ZllimIaZ/Bg7oO9YmHtfAz8DOfIB29gF8FjiC7RPLyfR8awS+BDyR\n5oNxeU/5CWw78Z8LPLHnAP1J/zI9J6AXttM3se1k/STg3L7YTgY+ME0xteXPAc6clrhobu1/fN9+\nm4p9NeQx+CTgSwvFvxPH9qFsn1iuBNa00zPAlePcXwPi+xuaLyxTERdwL+DbwFGTjonmC+96mrtT\n5xPLxPcTzXnywL79NvG4Bj0mNqRLkmcD11XVZX2z+m+QvKEtO5jmZsl517dl272mqu4Ebkly34XW\nleRA4GdVtbVnXQ9s4/qTJNcCLwZO61//JGLq8wqabxnTFte8aYxpMYNuyD14gWV3xkFVtQWgqjYD\nBy2w/c73V38gSdbSXFF9k+akNLG4kuyW5LvAZmB9VX1r0jEB7wH+K9vfnTzpmGjjWZ/kW0leNUVx\n3c1Ie4UlWQ+s6S2i2TmnAm8Dnj6qTS8y7zXAK4FDk1zaLrsHcA+AqjoVOLUd/PL1wLoxxARN28l8\nTNBUy6xN8qyqOgcgyduB26vq0x3FtFRci+6rERqmn/1KH3Kgll5kaEPviyT7AJ8D3lhVvxxwT9hY\n42q/HDwuyb7A55M8ekAMY4spyTOBLVV1SZLZRRadxP/vmKq6Mcn9gfOSXDUgjokcV/1GesVSVU+v\nqsf0PI6oqsfQ1BWuBb6X5BqaS8+LkxxEkw17B+Q6pC27AXjQgHJ657X3wexbVT9dYF2nVdWjgVto\n6pyPAF7O3cci+xTwvP71jyimG6pqtiemx7QxzfUklZcB/5bmSmreqOMadl/1Gse+uhnYrx2wtH9d\ny7XQ8da1LUnWACSZAX7Ss/2R7q/5J0n2oEkqZ1bVF6YlLoCqupVmELLjJhzTMcCzk/wzTaeQpyU5\nE9g86f1UVTe2f2+iqco8esL7amGL1ZON60FTd3hAOz3f6LQX8BC2b3Sab9wLTXXQcW35SWxrdDqB\nwY1O89P7t/M+w7Y61Q8B/xl4eE9MrwfOnnRM7fRxNL15+utXJxpX+/yrwBOmKaZlHnu7s63xfi+a\nxvvDOzim1wKX9Tw/g7bOm8GNrCPdX+38TwDv7otzYnEB92NbI/49ga/TfHma+L5ql3kK29pY3jXJ\nmGjaoPZpp+8NXAAcOy376m7H/85+gLp40FzB9Hc3/iF37yb3BJpuiRuA9/WU7w2c3ZZ/E1jbM+9l\nbfnVbN/98iE0vSOupjlJ7Unzbe5SmpPLF4AHTDqmtnwDsImm++PF8//8Ce+r59DUu/6GZoSEL006\npp04/o6j6SW1ATi5g+P5U8CPaYYcvpbmKu8A4Px2O+ex/Ql/HPvrGJqu4ZfQnHAubt/3fScVF03H\nnYvbmC4F3t6WTyymvv9jb2KZaEw0x/v8/+4y2uN00nEt9PAGSUlSp6buh74kSSubiUWS1CkTiySp\nUyYWSVKnTCySpE6ZWCRJnZq6H/qSpk2SO2mGK58fkuisqnrXZKOSppf3sUhLSHJrVe27xDK71bZB\nMUmyezUD+S217qGWk1YSq8KkpQ0cjC/JNUnemeTbwPOTfDXJe5JcBLwhyaFJ/j7JJUnWJzmkfd3H\nknwoyTdphuSQVhWrwqSl3TPJxWyrCntHVX22nfd/q+q3AZK8lmZomaPb518EPlZVf53k5cD7gee2\nrzu4qp401nchjYmJRVrar6vq8QvM+8wiz5/MtkRyJttfnXwWaZWyKkzaOb9a5PliDZj9r5NWDROL\ntLQd/cGjfwJe1E7/B+AfuglHmm5WhUlLu0dfG8u5VfU2lv71vjcAH0vyZuAmmuHzBy0nrSp2N5Yk\ndcqqMElSp0wskqROmVgkSZ0ysUiSOmVikSR1ysQiSeqUiUWS1CkTiySpU/8flbRSI16jrO0AAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10989c860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(error, bins=np.arange(min(error), max(error) + 25000, 25000))\n",
    "plt.xlabel('Error')\n",
    "plt.ylabel('Count')\n",
    "plt.title('Histogram of Errors')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The errors do appear normally distributed, but with a slight left tail. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimating Standard Deviation\n",
    "Once we have the errors, it is possible to construct an unbiased estimator for the standard deviation $\\sigma$ of the residuals. The formula for the unbiased estimator is as follows:\n",
    "\n",
    "$$\\hat{\\sigma}^{2} = \\frac{1}{n-p}\\sum_{i=1}^{n}e_{i}^{2}$$\n",
    "\n",
    "Where $e$ is the error, $p$ is the number of feastures, and $n$ is the number of samples. \n",
    "\n",
    "#### _Proof_\n",
    "(1) $H = X(X^{T}X)^{-1}X^{T}$  \n",
    "(2) $H^{2} = H$   \n",
    "(3) $\\tilde{H} = I - H$    \n",
    "(4) $e = y - \\hat{y} = y - Hy = y(I - H) = (I - H)(X\\beta + \\epsilon) =\n",
    "X\\beta + \\epsilon - X(X^{T}X)^{-1}(X^{T}X)\\beta - H\\epsilon =\\\\ \n",
    "(I - H)\\epsilon = \\tilde{H}\\epsilon\\\\$    \n",
    "(5) tr(AB) = tr(BA)   \n",
    "\n",
    "$\\sum_{i=1}^{n}e_{i}^{2} = e^{T}e \\stackrel{by\\:(4)}{=} \\epsilon^{T}\\tilde{H}^{T}\\tilde{H}\\epsilon \\stackrel{by\\:(3)} = \\epsilon^{T}(I - H)(I - H)\\epsilon = \\epsilon^{T}(I^{2} - 2HI + H^{2})\\epsilon\\\\ \\stackrel{by\\:(2)}{=} \\epsilon^{T}(I^{2} - 2H + H)\\epsilon = \\epsilon^{T}(I - H)\\epsilon \\stackrel{by\\:(3)}{=} \\epsilon^{T}\\tilde{H}\\epsilon$   \n",
    "\n",
    "$$E(\\sum_{i=1}^{n}e_{i}^{2}) = E(\\epsilon^{T}\\tilde{H}\\epsilon|X) =$$ $$E(\\sum_{i=1}^{n}\\sum_{j=1}^{n}\\epsilon_{i}\\tilde{H}_{i, j}\\epsilon_{j}|X) =$$ $$\\sum_{i=1}^{n}\\sum_{j=1}^{n}\\tilde{H}_{i, j} E(\\epsilon_{i}\\epsilon_{j}|X) = $$\n",
    "$$\\sigma^{2} tr(\\tilde{H}) = \\sigma^{2} tr(I - H) = \\sigma^{2} (tr(I) - tr(H)) =$$\n",
    "$$\\sigma^{2} (n - tr(X(X^{T}X)^{-1}X^{T})) = \\sigma^{2} (n - tr((X^{T}X)^{-1}(X^{T}X))) =$$ $$\\sigma^{2} (n - p)$$\n",
    "\n",
    "Hence, $E(\\frac{1}{n-p}\\sum_{i=1}^{n}e_{i}^{2}) = \\sigma^{2}$\n",
    "\n",
    "Let's estimate what $\\sigma$ is in our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  4.86562032e+09]\n"
     ]
    }
   ],
   "source": [
    "variance = 1/(m2 - 9)*sum(error**2)\n",
    "print(variance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Logistic regression is best used in classification settings where the outcome $y$ is binary $\\in \\{0, 1\\}$. Probabilistically, we assume that the outcome is $1$ with probability $p$ and $0$ with probability $q = 1 - p$. Thus, the outcome is a Bernoulli random variable with parameter $p$. The goal of logistic regression is to find $p$ via a function of the data $X$. A good function is one that is bounded between $[0, 1]$, and one typically chosen in logistic regression is:\n",
    "\n",
    "$$ f(z) = \\frac{1}{e^{-z} + 1} $$\n",
    "\n",
    "Graphically this function appears as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEKCAYAAAACS67iAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHqRJREFUeJzt3XmUVOW19/HvRoGrF3HAAURBDQhiRMQEcaSUKOjNK16H\nVyQrilOMkXjVeyMao7RefTUSoxGNBiJEExWHaMQ4ABFalIi2IoMCAqLIJAbEAQHphv3+8VTbTdtD\nNVT1U3Xq91mrlzWcrvoVDdvd+5zzHHN3RESk8DWLHUBERLJDBV1EJCFU0EVEEkIFXUQkIVTQRUQS\nQgVdRCQhVNAlL5nZIDN7cSu/9x0zOy7bmZpCIWeX+EzHocu2MrMPgAvdfVKE9x4DLHH3G7bxdToC\nHwBrKx8CFrr7YdsYsb73zEp2kUrbxw4gkkcc2NnV5UiB0shFcsrMLjazBWa2ysz+Zmbtqj13kpnN\nM7M1ZnavmZWa2QXp584zs1eqbXunma00s8/NbKaZdTOzi4EfAVeb2Rdm9kx62w/M7IT07WZm9ksz\nW5j+3jIza19f5Fo+wzAz+3O1+x3NbLOZNUvfn2xmN5nZq+kcL5rZbtW2P8bMpqY/52IzOzfD7C3M\n7C4zW2ZmS9N/Bs3Tz/UxsyVmdlX6z2WZmQ1u7M9HkkUFXXImXZj+H3Am0A74CBibfm534AlgKNAG\neA84ssZLeHrbk4BjgE7uvjPwf4HV7j4KeBi43d1bu/uAWmL8N3A20D/9vRcA6+qLXcfjNbv2mvfP\nAc4D9gBaAv+Tzt4ReB74HbA70AOYkWH2XwG9gO7Aoenbv6r2fFtgJ2Bv4CLgXjPbuZ7PJgmngi65\nNAh4wN1nuns5cC3Q28w6ACcD77j7M+6+2d3vBlbW8TrlhMLVzczM3d9z97q2relC4Dp3Xwjg7rPd\nfU0d2xrwr3Qn/amZXZXhewCMcff33f1r4HFC4YZQ6Ce6++Puvsnd17j7rAxfcxBwo7uvdvfVwI3A\nj6s9vxH43/TrvkCY/3dpRGZJGM3QJZf2Bt6qvOPuX5nZp0D79HNLamy/tLYXcffJZnYPcC/Qwcye\nAv7H3dfWtn0N+wKLMszrQJutnKF/XO32OqBVtfd/fyteD8Kf0UfV7i9OP1ZptbtvruN9pQipQ5dc\nWg50rLxjZv9OGK8sA1YQil11+9T1Qu5+j7t/D+hG6EJ/UflUAxmWAN9pRObaRi5fATtWu9+ulm3q\ne/9OdTzXUPYt/vzSt5c34r2lyKigS7a0MLOW1b62Ax4Fzjez7mbWkjBPn+buHwHPAd81s1PNbDsz\nGwLsVdsLm9n3zKyXmW0PrAc2AJWd6UrggHpy/RH4XzPrlH6tQ8xs1zq2rWt+PgM4zsz2Tc+or6nn\n/Wp6GOhrZmemP+duZnZohtkfBX5lZrun9zlcD/y5nu2lyKmgS7Y8R/iVf336v8Pc/SVCEXqK0JXv\nDwwESM+EzwKGA6uArsCbwNe1vHZrYBTwKeFY8VXp7wN4ADg4PfN+Kv1Y9c73t4SZ9gQz+5xQ4Heo\n4zPU2jG7+z+Ax4BZQBnwbCbfl/7eJcAphJ2knwJvE3ZyZpL9ZsKfySxgZvr2LXW9V305pDg0eGKR\nmT0A/BBY6e7d69jmbsJOrq+Awe4+I9tBJdnMzAgz9EHu/nLsPCKFKJMOfQzQr64nzexk4Dvu3hm4\nBLg/S9kk4dLHoe+cHsdcl354WsxMIoWswYLu7q8CdR3mBTAAeCi97evAzmZW6yxUpIYjCUeAfAL8\nBzAgfdifiGyFbBy22J4tDz9bln4s0+OEpUi5+42EY6tFJAu0U1REJCGy0aEvY8vjifdJP/YtZqa9\n8CIiW8Hd6zqs9huZduhG3cfojgPOBTCz3sBn9Z2W7e6J/Ro2bFj0DPp8+nzF9tm29vOVlztLlzpv\nvuk8+6wzcqRz003OpZc6p53m9O7tdOzotGzptG7tdOni9OnjDBzo/OUvTfv5MtVgh25mjwApoI2Z\nfQQMA1qE2uwj3f15MzvFzBYSDls8P+N3FxHJsrVr4eOPYcWK8FXb7Y8/hk8/hd13h7ZtoV278NW2\nLXTrBn37Vj3eti3suGPD75sPGizo7j4og22GZCeOiEj91q2DKVNgxoyq4jxtGjz8cLhdUVFVoCsL\ncrt2cNxxWxbpPfaA7RO2mlXCPk5cqVQqdoSc0ucrXIX82dzh3Xdh/Pjw9dpr0KMHHHEEdOwIvXvD\nkUem6N8/FOvWrcEanDYnU5Negi6sfKr9oiJSv9WrYeLEUMAnTIAWLaBfv/B1wgmwc5Gt+m5meAY7\nRVXQRSS6ioowNqnswufNgz594KSTQhHv3Ll4u25QQReRPPfhh1UFfPJk2G+/qi78qKOgZcvYCfOH\nCrqI5JWvvoLS0qoivmZNVQd+4olhR6XUTgVdRKJyh5kzwwx8/Hh44w04/PCqLrxHD2imc9UzooIu\nIk3uk0+23JnZqlVVAT/+eNhpp9gJC5MKuojk3MaN4TDCyjHKwoWhcFeOUr7TmIv/SZ1U0EUkZ8rL\n4cYbYcSIcARKZRd+5JHQvHnsdMmTaUHXiUUi0iiLFsGgQbDrrjBnDrRvHzuRVNIuCRHJ2MMPhzM0\nBw6E555TMc836tBFpEFffglDhsDrr4ednYcdFjuR1EYduojUq6wsFPAWLeCtt1TM85k6dBGp1ebN\nMHw43HEH/P73cOaZsRNJQ1TQReRbli+Hc8+Fr7+GN9+EDh1iJ5JMaOQiIlt49lno2TOsHz55sop5\nIVGHLiIArF8PV18dCvqTT8Ixx8ROJI2lDl1EePdd6NULVq6Et99WMS9UKugiRcwd7rsPUim48kp4\n7LFwwpAUJo1cRIrU6tVw0UVhXfJXX4UuXWInkm2lDl2kCJWWhuVrDzggXClIxTwZ1KGLFJHycigp\ngTFjYPRo6N8/diLJJhV0kSJRfVGtt9+GvfaKnUiyTSMXkSLwyCNhUa2zzw6LaqmYJ5M6dJEE06Ja\nxUUdukhCVS6q1by5FtUqFurQRRKm+qJa994LZ50VO5E0FRV0kQSpXFRrw4bQoXfsGDuRNCWNXEQS\nonJRrWOPDceZq5gXH3XoIgVuwwb4xS+0qJaoQxcpaIsWaVEtqWLu3nRvZuZN+X4iSVZRAUcdBaef\nDkOHglnsRJIrZoa7N/gTVocuUqBuvx122UXFXKqoQxcpQLNmQd++MH067Ltv7DSSa+rQRRJq40Y4\n7zz49a9VzGVLKugiBeaWW2DvveH882MnkXyTUUE3s/5mNs/M5pvZ0Fqeb21m48xshpnNNrPBWU8q\nIkyfHq4wNGqU5ubybQ0WdDNrBtwD9AMOBs4xs641NrsMeNfdewDHA3eYmY5xF8mir78OZ4HeeWfo\n0EVqyqRD7wUscPfF7l4OjAUG1NjGgZ3St3cCVrt7RfZiikhJCRx4YFjTXKQ2mXTR7YEl1e4vJRT5\n6u4BxpnZcqAVcHZ24okIhMvEjRkTjm7RqEXqkq2xSD/gbXc/wcy+A0w0s+7uvrbmhiUlJd/cTqVS\npFKpLEUQSab162HwYBgxAvbcM3YaaQqlpaWUlpY2+vsaPA7dzHoDJe7eP33/GsDd/dfVtvk7cKu7\nT03ffwkY6u5v1ngtHYcu0khXXRVWURw7NnYSiSXT49Az6dDLgE5m1hFYAQwEzqmxzWLgB8BUM9sL\nOBBY1LjIIlLTK6+EQj57duwkUggaLOjuvsnMhgATCDtRH3D3uWZ2SXjaRwI3A38ys1npb7va3T/N\nWWqRIvDVV+FY8/vvhzZtYqeRQqBT/0Xy1JAh4ZqgDz4YO4nEls2Ri4g0sZdegmee0ahFGken/ovk\nmS++gAsvDGeD7rJL7DRSSDRyEckzF18cjjUfOTJ2EskXGrmIFKAXXoCJE8MJRCKNpYIukifWrIGf\n/CTsBG3dOnYaKUQauYjkifPOg512gnvuiZ1E8o1GLiIFZNw4mDoVZs6MnUQKmTp0kchWr4ZDDoHH\nH4djjomdRvJRph26CrpIZAMHQvv2cMcdsZNIvtLIRaQAPPEEzJgRlsYV2Vbq0EUiWbkSDj00nBF6\nxBGx00g+08hFJI+5wxlnQJcucOutsdNIvtPIRSSPPfIILFgAjz4aO4kkiTp0kSa2fDkcdlg4K7Rn\nz9hppBBk2qFrcS6RJuQe1mq59FIVc8k+FXSRJjRmDKxYAdddFzuJJJFGLiJN5KOP4PDDYdKkcCKR\nSKY0chHJI+5w0UVw5ZUq5pI7KugiTeAPf4DPP4err46dRJJMIxeRHFu0KJw4NGUKHHRQ7DRSiDRy\nEckDmzfDBRfANdeomEvuqaCL5NCIEVBRAVdcETuJFAONXERyZP58OPpoeO016NQpdhopZBq5iES0\naRMMHgzDhqmYS9NRQRfJgd/+Fv7t3+BnP4udRIqJRi4iWTZnDvTpA2VlsN9+sdNIEmjkIhJBRUW4\n2PMtt6iYS9NTQRfJottugzZtwgJcIk1NIxeRLJk5E048EaZPh332iZ1GkkQjF5EmtHFjGLUMH65i\nLvGooItkwc03Q4cOcO65sZNIMdMl6ES20fTpYfGtGTPAGvylWCR3NEMX2UY/+AGcfbZ2hEruaIYu\n0gRefhk+/DCcFSoSmwq6yFZyh+uvD6f3N28eO41IhgXdzPqb2Twzm29mQ+vYJmVmb5vZO2Y2Obsx\nRfLPxInwr3/BoEGxk4gEDc7QzawZMB/oCywHyoCB7j6v2jY7A/8ETnL3ZWa2u7uvquW1NEOXRHCH\n3r3hqqvC/Fwkl7I5Q+8FLHD3xe5eDowFBtTYZhDwV3dfBlBbMRdJkueeg/Xr4ayzYicRqZJJQW8P\nLKl2f2n6seoOBHYzs8lmVmZmP85WQJF8s3lzmJ3fdBM0014oySPZOg59e6AncALw78BrZvaauy/M\n0uuL5I2nn4bttoMBNX9PFYksk4K+DOhQ7f4+6ceqWwqscvcNwAYzmwIcCnyroJeUlHxzO5VKkUql\nGpdYJKJNm8JRLbffrpOIJHdKS0spLS1t9PdlslN0O+A9wk7RFcAbwDnuPrfaNl2BEUB/oCXwOnC2\nu8+p8VraKSoF7dFHw3VCp05VQZemk+lO0QY7dHffZGZDgAmEmfsD7j7XzC4JT/tId59nZuOBWcAm\nYGTNYi5S6CoqQnd+330q5pKfdOq/SIb+9KfwNXmyCro0rUw7dBV0kQxs3Ahdu8KDD8Kxx8ZOI8VG\na7mIZNGYMdC5s4q55Dd16CIN2LAhFPO//hV69YqdRoqROnSRLBk1Cnr0UDGX/KcOXaQe69ZBp07h\nVP/DDoudRoqVOnSRLPj97+Goo1TMpTCoQxepw5dfhu580iQ4+ODYaaSYqUMX2UYjRkDfvirmUjjU\noYvU4rPPwpEtU6fCgQfGTiPFTh26yDa480744Q9VzKWwqEMXqWH1aujSBcrKYP/9Y6cRUYcustV+\n8xs44wwVcyk86tBFqlm5Erp1gxkzYN99Y6cRCbQ4l8hWuOqqsEzu3XfHTiJSRQVdpJGWLYPu3eGd\nd6Bdu9hpRKqooIs00mWXwY47wvDhsZOIbEkFXaQRFi+Gnj1h3jzYY4/YaUS2pKNcRBrh5pvhpz9V\nMZfC1uA1RUWSbuFCePppmD8/dhKRbaMOXYreTTfB5ZfDbrvFTiKybdShS1GbNw9efDF06SKFTh26\nFLWSknDseevWsZOIbDsd5SJFa9YsOOmk0J23ahU7jUjddJSLSAOGDYOhQ1XMJTnUoUtReustGDAA\nFiyAHXaInUakfurQRepxww1w7bUq5pIsOspFis5rr4X1Wp56KnYSkexShy5F5/rrw1fLlrGTiGSX\nCroUldJS+PBDOO+82ElEsk8FXYqGe+jMb7gBmjePnUYk+1TQpWhMnAirVsGPfhQ7iUhuqKBLUajs\nzktKYLvtYqcRyQ0VdCkKzz0H69fDWWfFTiKSOyroknibN4fu/MYboZn+xkuC6a+3JN7TT4dCftpp\nsZOI5JZOLJJE27QprNly++1gDZ44LVLYMurQzay/mc0zs/lmNrSe7b5vZuVmdnr2Iopsvcceg512\ngpNPjp1EJPcaXJzLzJoB84G+wHKgDBjo7vNq2W4isB4Y7e7fOrFai3NJU6qogG7d4L77oG/f2GlE\ntl42F+fqBSxw98XuXg6MBQbUst3PgSeBTxqVVCRH/vIX2HtvOOGE2ElEmkYmM/T2wJJq95cSivw3\nzGxv4DR3P97MtnhOJIaNG8O1Qh98ULNzKR7ZOsrlLqD6bF3/hCSqMWOgUyc49tjYSUSaTiYd+jKg\nQ7X7+6Qfq+57wFgzM2B34GQzK3f3cTVfrKSk5JvbqVSKVCrVyMgi9duwAW6+GZ58MnYSka1TWlpK\naWlpo78vk52i2wHvEXaKrgDeAM5x97l1bD8GeFY7RSWWESNgwgR49tnYSUSyI9Odog126O6+ycyG\nABMII5oH3H2umV0SnvaRNb9lqxKLZMG6dXDrreFUf5Fio2uKSqL85jcwbZrGLZIsmXboKuiSGF9+\nGXaETpoEBx8cO41I9ugi0VJ0RowIJxCpmEuxUocuifDZZ9C5M7z6KnTpEjuNSHZp5CJF5cor4fPP\nYfTo2ElEsi9rR7mI5LspU8IiXLNnx04iEpdm6FLQ1q6F88+H+++HNm1ipxGJSyMXKWiXXRaK+oMP\nxk4ikjsauUjivfQSjBunUYtIJY1cpCB98QVccAGMGgW77BI7jUh+0MhFCtJFF4VlcUeNip1EJPc0\ncpHEev55+Mc/YNas2ElE8osKuhSUNWvgJz+Bhx6C1q1jpxHJLxq5SEE599xQyO+5J3YSkaajkYsk\nzjPPwD//CTNnxk4ikp/UoUtBWLUKuncPZ4TqsnJSbLSWiyTK2WdD+/bw29/GTiLS9DRykcR4/PEw\nZvnTn2InEclv6tAlr61cCYceGubnRxwRO41IHBq5SMFzh9NPh65dw3VCRYqVRi5S8B5+GBYuhLFj\nYycRKQzq0CUvLVsGhx0GL74IPXvGTiMSl64pKgXLHS6+GC69VMVcpDFU0CXvjB4NK1bAddfFTiJS\nWDRykbzy0Udw+OEwaRIcckjsNCL5QSMXKTjucOGF4YLPKuYijaeCLnnj/vvDhSuuvjp2EpHCpJGL\n5IVFi6BXL3jlFTjooNhpRPKLRi5SMDZvhvPPh2uvVTEX2RYq6BLd3XfDpk1wxRWxk4gUNo1cJKr3\n3oOjj4bXXoPOnWOnEclPGrlI3tu0CQYPhmHDVMxFskEFXaK54w7YYQe47LLYSUSSQSMXieLddyGV\ngrIy2G+/2GlE8ptGLpK3ysvhvPPglltUzEWySQVdmtxtt0GbNmEBLhHJHo1cpEnNmAEnngjTp8O+\n+8ZOI1IYsjpyMbP+ZjbPzOab2dBanh9kZjPTX6+amVbikG/ZuDGMWoYPVzEXyYUGO3QzawbMB/oC\ny4EyYKC7z6u2TW9grrt/bmb9gRJ3713La6lDL2LXXx869HHjwBrsNUSkUjYvQdcLWODui9MvPBYY\nAHxT0N19WrXtpwHtGxdXkq6sDEaODAVdxVwkNzIZubQHllS7v5T6C/ZFwAvbEkqSZcOGMGq56y5o\n1y52GpHkyupFos3seOB84Ji6tikpKfnmdiqVIpVKZTOC5KEbboBu3WDgwNhJRApDaWkppaWljf6+\nTGbovQkz8f7p+9cA7u6/rrFdd+CvQH93f7+O19IMvcj8859w+ukwaxbsuWfsNCKFKZtHuZQBncys\no5m1AAYC42q8WQdCMf9xXcVcis+6dWGtlnvvVTEXaQoZHYeePnLld4T/ATzg7reZ2SWETn2kmY0C\nTgcWAwaUu3uvWl5HHXoRueIK+OQTeOSR2ElEClumHbpOLJKcePllGDQIZs+G3XaLnUaksGktF4lm\n7dpwBaI//EHFXKQpqUOXrLv00nCo4pgxsZOIJEM2TywSydjEifD3v4dRi4g0LY1cJGs+/xwuvBD+\n+EfYZZfYaUSKjwq6ZMWcOXDcceGY8379YqcRKU4q6LJN3MPOzz594Oc/hzvvjJ1IpHhphi5b7dNP\nw0UqFi2CV16Brl1jJxIpburQZau8/DL06AEdOsC0aSrmIvlAHbo0Snk53HRT2PE5ejScfHLsRCJS\nSQVdMvbBB+Hsz9at4e23oW3b2IlEpDqNXCQjjz4KRxwBZ50FL7ygYi6Sj9ShS72+/BIuvzwsg/vi\ni9CzZ+xEIlIXdehSpzffDAXcDN56S8VcJN+poMu3bN4Mw4fDKafAzTeHnZ+tWsVOJSIN0chFtrBi\nRbj+51dfwRtvwH77xU4kIplShy7feO65MFY58shwnLmKuUhhUYcubNgAQ4fC3/4Gjz8Oxx4bO5GI\nbA0V9CI3dy4MHAidO8OMGbDrrrETicjW0silSLnDyJFhhcQhQ+CJJ1TMRQqdOvQiVLmo1vvvw5Qp\ncNBBsROJSDaoQy8ylYtq7btvWFRLxVwkOdShF4mKCrjxxrCo1gMPhGPMRSRZVNCLwIcfhkW1WrXS\noloiSaaRS8KNHQu9esEZZ4S1WFTMRZJLHXpCrV0bLgk3dWpYHfHww2MnEpFcU4eeQJULaZnB9Okq\n5iLFQh16QqxeDRMnhrHK88/D3XeHE4ZEpHiYuzfdm5l5U75fkpWXh8MOx4+HCRPgvfegTx/o1w/+\n8z9h771jJxSRbDEz3N0a3E4FvXB88EEo4OPHw+TJcMABoYD36wdHHQUtWsROKCK5oIKeAGvXQmlp\nVRH/4gs46aRQwE88EfbcM3ZCEWkKKugFaPNmmDmzaoxSVgbf/35VF969OzTTbmyRoqOCXiA++SQU\n7/Hjw07N1q2rCngqpSsFiYgKet7auDFccLlyjLJoERx/fFUR33//2AlFJN+ooOcJd1i4sKqAT5kC\nXbpUzcJ794bmzWOnFJF8poIeyYYN8PHH4WIRlUV8w4aqDvwHP4Ddd4+dUkQKSaYFPaMTi8ysP3AX\n4czSB9z917VsczdwMvAVMNjdZzQucv5yh88+CxdQXrEiFOyatyv/u24d7LUXdO0auvBnnoHvfjec\ntSkikksNFnQzawbcA/QFlgNlZvaMu8+rts3JwHfcvbOZHQHcD/TOUeasqaiAlStrL8w1H2vZEtq1\nC19t21b9t0ePqtvvv1/KqaemElu8S0tLSaVSsWPkTJI/X5I/GyT/82Uqkw69F7DA3RcDmNlYYAAw\nr9o2A4CHANz9dTPb2cz2cveV2Q6cqdWrYc6cuov0ihWwZk0Yf9Qs0t26Qd++Wz62444Nv+eTT5Yy\nYEAq558tlqT/o0ny50vyZ4Pkf75MZVLQ2wNLqt1fSijy9W2zLP1YtII+eTLceeeWxfq446put2sH\ne+wB220XK6GISHYldnGuM88MXyIixaLBo1zMrDdQ4u790/evAbz6jlEzux+Y7O6Ppe/PA/rUHLmY\nWbIPcRERyZFsHeVSBnQys47ACmAgcE6NbcYBlwGPpf8H8Flt8/NMAomIyNZpsKC7+yYzGwJMoOqw\nxblmdkl42ke6+/NmdoqZLSQctnh+bmOLiEhNTXpikYiI5E6UtfvM7OdmNtfMZpvZbTEy5JqZ/beZ\nbTaz3WJnySYzuz39s5thZn81s9axM20rM+tvZvPMbL6ZDY2dJ5vMbB8zm2Rm76b/vV0eO1O2mVkz\nM5tuZuNiZ8mF9GHgT6T/3b2bPtenVk1e0M0sBfwf4BB3PwT4TVNnyDUz2wc4EVgcO0sOTAAOdvce\nwALg2sh5tkm1E+f6AQcD55hZ17ipsqoCuMrdDwaOBC5L2OcD+C9gTuwQOfQ74Hl3Pwg4FJhb14Yx\nOvRLgdvcvQLA3VdFyJBrdwK/iB0iF9z9H+6+OX13GrBPzDxZ8M2Jc+5eDlSeOJcI7v5x5TIc7r6W\nUAzax02VPenm6RTgj7Gz5EL6N+Bj3X0MgLtXuPsXdW0fo6AfCBxnZtPMbLKZfS9Chpwxs1OBJe4+\nO3aWJnAB8ELsENuothPnElPwqjOz/YAewOtxk2RVZfOU1J2B+wOrzGxMeqw00sx2qGvjnJxYZGYT\ngb2qP0T4A/9V+j13dffeZvZ94HHggFzkyJUGPt8vCeOW6s8VlHo+33Xu/mx6m+uAcnd/JEJEaSQz\nawU8CfxXulMveGb2H8BKd5+RHuUW3L+1DGwP9AQuc/c3zewu4BpgWF0bZ527n1jXc2b2U+Cp9HZl\n6R2Hbdx9dS6y5EJdn8/MvgvsB8w0MyOMI94ys17u/kkTRtwm9f38AMxsMOHX3BOaJFBuLQM6VLu/\nT/qxxDCz7QnF/M/u/kzsPFl0NHCqmZ0C7ADsZGYPufu5kXNl01LCb/xvpu8/CdS54z7GyOVvpAuB\nmR0INC+kYl4fd3/H3du6+wHuvj/hh3FYIRXzhqSXUv4FcKq7fx07TxZ8c+KcmbUgnDiXtKMlRgNz\n3P13sYNkk7v/0t07uPsBhJ/bpIQVc9InaC5J10oIq97WuQM4xlouY4DRZjYb+BpI1A+gBid5vwaO\nAFoAE8MvIUxz95/FjbT16jpxLnKsrDGzo4EfAbPN7G3C38lfuvuLcZNJI1wOPGxmzYFF1HPipk4s\nEhFJiCgnFomISPapoIuIJIQKuohIQqigi4gkhAq6iEhCqKCLiCSECrqISEKooIuIJMT/By0YfsQm\nUYaTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10615da20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import math\n",
    "x = np.linspace(-5, 5, 10, endpoint=True)\n",
    "\n",
    "def logistic(x):\n",
    "    return 1 / (np.exp(-x) + 1)\n",
    "\n",
    "y = logistic(x)\n",
    "plt.plot(x, y)\n",
    "plt.title(\"Logistic Function\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which we can verify that the function indeed is bounded at $[0, 1]$. In logistic regression, the model we are trying to train using the data is $\\beta$ defined in:\n",
    "\n",
    "$$P(y = 1 | X) = \\frac{1}{e^{-X\\beta} + 1}$$\n",
    "\n",
    "Where $X \\in \\mathbb{R}^{m x n}$. Thus, once $\\beta$ is trained and the logistic function constructed, the output of this function has a probabilisitic interpretation. How one classifies a data point will then depend on the threshold. For example, one can classify any sample that outputs a probability greater than 50% as one class and any sample with probability below 50% as another class. Typically, one evaluates probabilistic models by using the area under the ROC curve. The closer the area is to 1, the better the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization\n",
    "One way to find $\\beta$ is by minimzing a function that quantifies the error of the model. In this tutorial, we are going to use cross entropy as our loss function: \n",
    "\n",
    "$$ L = \\sum_{i = 1}^{n}log(1 + e^{-z^{(i)}}) $$\n",
    "\n",
    "Where $n$ is the number of training samples. To understand why the $\\beta$ that minimizes cross entropy is the $\\beta$ closest to true $\\beta$, we must understand cross entropy.\n",
    "\n",
    "### Cross Entropy\n",
    "In information theory, cross entropy is used to quantify how similar to two probability distributions $Q$ and $P$ are. Cross entropy is defined as: \n",
    "\n",
    "$$ R = \\sum_{i}p_{i}log_{2}(\\frac{1}{q_{i}}) = -\\sum_{i}p_{i}log_{2}(q_{i})$$\n",
    "\n",
    "$R$ is minimized when $Q$ and $P$ is the same. To see how this is so, assume without loss of generality that $Q$ and $P$ are 2-valued vectors. Thus, the objective is to minimize:\n",
    "\n",
    "$$R = p_{1}ln q_{1} + p_{2}ln q_{2} $$\n",
    "\n",
    "With respect to $q_{1}$ and $q_{2}$ under the constraint that:\n",
    "\n",
    "$$q_{1} + q_{2} = 1$$\n",
    "\n",
    "To solve this we utilize the method of lagrange multipliers:\n",
    "\n",
    "$$\\frac{\\partial R}{\\partial q_{1}} = \\frac{p_{1}}{q_{1}ln 2}\\\\$$\n",
    "$$\\frac{\\partial R}{\\partial q_{2}} = \\frac{p_{2}}{q_{2}ln 2}\\\\$$\n",
    "$$\\frac{p_{1}}{q_{1}ln 2} = \\lambda\\\\$$\n",
    "$$\\frac{p_{2}}{q_{2}ln 2} = \\lambda\\\\$$\n",
    "$$\\frac{p_{1}}{q_{1}ln 2} = \\frac{p_{2}}{q_{2}ln 2}\\\\$$\n",
    "$$p_{1}q_{2}ln 2 = p_{2}q_{1}ln 2\\\\$$\n",
    "$$q_{2} = \\frac{p_{2}q_{1}}{p_{1}}\\\\$$\n",
    "\n",
    "Given the constraint that $q_{1} = 1 - q_{2}$:\n",
    "\n",
    "$$q_{2} = \\frac{p_{2}}{p_{1}} - \\frac{p_{2}q_{2}}{p_{1}}\\\\$$\n",
    "$$p_{1}q_{2} = p_{2} - p_{2}q_{2}\\\\$$\n",
    "$$q_{2} = \\frac{p_{2}}{p_{1} + p_{2}} = p_{2}$$\n",
    "$$q_{1} = 1 - q_{2} = 1 - p_{2} = p_{1}$$\n",
    "\n",
    "We can ensure that $R$ is minimized when $P = Q$ through numerical examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEPCAYAAACgFqixAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGU5JREFUeJzt3XuUpHV95/H3B0ZJI4s3No7CClGjHjdcxAsaiLaJ04OX\nRUfXqChRzxrdbBSMfdwEdl3mrOesZOMkgu6qExEFxBukVyCaggPTCCpXGRgu4yUqKi4Yjq4yOCjg\nd/+op2eKprunaqYu3T3v1zl9puqpXz31fbqhPvX7/ep5fqkqJEm7tz1GXYAkafQMA0mSYSBJMgwk\nSRgGkiQMA0kSAw6DJAckuTTJzUk2JTl+nnbjSa5PclOSDYOsSZL0UBnkeQZJVgIrq2pjkn2A64BX\nVNXmjjaPBL4GTFTV7Un2q6q7BlaUJOkhBtozqKo7qmpjc3sLcCuw/6xmxwLnVdXtTTuDQJKGbGhz\nBkkOAg4Drpr10FOBxyTZkOSaJMcNqyZJUtuKYbxIM0R0LnBC00OYXcPhwB8CjwC+nuTrVfWdYdQm\nSRpCGCRZQTsIzqqqL87R5EfAXVV1L3Bvkq8AhwIPCoMkXkRJknZCVWVHbYYxTPQJ4JaqOnWex78I\nHJVkzyR7A0fQnlt4iKpatj8nn3zyyGvw+Dy+3e3Ydofj69ZAewZJjgTeAGxKcj1QwEnAgUBV1fqq\n2pykBdwIPACsr6pbBlmXJOnBBhoGVfVVYM8u2n0A+MAga5Ekzc8zkBeJ8fHxUZcwUB7f0rWcjw2W\n//F1a6AnnfVTkloqtUrSYpGEWiQTyJKkRc4wkCQZBpIkw0CShGEgScIwkCRhGEiSMAwkSRgGkiQM\nA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJ\nGAYj12q1mJiYYGJiglarNepyJO2mUlWjrqErSWqp1NqtVqvFmjVr2Lp1KwBjY2NMTU2xevXqEVcm\nablIQlVlR+0G2jNIckCSS5PcnGRTkuMXaPucJPcledUga1pM1q1bty0IALZu3cq6detGWJGk3dWK\nAe//fuDdVbUxyT7AdUkuqqrNnY2S7AGcAjhOIkkjMNCeQVXdUVUbm9tbgFuB/edo+k7gXOAng6xn\nsZmcnGRsbGzb/bGxMSYnJ0dYkaTd1dAmkJMcBBwGXDVr+xOAV1bVR4AdjmstJ6tXr2ZqaopVq1ax\natUq5wskjcxQJpCbIaJp4H1V9cVZj30e+EBVXZ3kDODCqjpvjn0suwlkSRq0bieQBz1nQJIVtIeA\nzpodBI1nA59NEmA/4CVJ7quq82c3XLt27bbb4+PjjI+PD6RmSVqqpqenmZ6e7vl5A+8ZJDkTuKuq\n3t1F2zOAC6rqH+Z4zJ6BJPVoUfQMkhwJvAHYlOR6oICTgAOBqqr1s57iu70kjYAnnUnSMrYoTjqT\nJC0NhoEkyTCQJBkGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKE\nYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kSAw6D\nJAckuTTJzUk2JTl+jjbHJrmh+bkiycGDrEmS9FCpqsHtPFkJrKyqjUn2Aa4DXlFVmzvaPA+4tap+\nnuRoYG1VPW+OfdUga5Wk5SgJVZUdtVsxyCKq6g7gjub2liS3AvsDmzvaXNnxlCubxyVJQzS0OYMk\nBwGHAVct0OytwJeHUY8kabuB9gxmNENE5wInVNWWedq8CHgLcNR8+1m7du222+Pj44yPj/e1Tkla\n6qanp5menu75eQOdMwBIsgK4EPhyVZ06T5tDgPOAo6vqn+dp45yBJPWo2zmDYQwTfQK4ZYEgeCLt\nIDhuviCQJA3WDnsGSfasqgd2aufJkcBXgE1ANT8nAQcCVVXrk/w98CrgNiDAfVX13Dn2Zc9AknrU\nbc+gmzD4Lu1P7mdU1S19qq9nhoEk9a6fw0SHAt8CPp7kyiRvS7LvLlcoSVo0eppATvJC4BzgUbS/\nHfS+qvrOgGqb/dr2DCSpR33rGSTZM8kxSaaADwLrgCcBFwBf2uVKJUkj1815Bt8GNgB/U1Vf69h+\nbpIXDKYsSdIwdTOBvM98J4oNk8NEktS7fk4g/3aSC5LcleQnSb6Y5El9qFGStEh0EwbnAJ8HVgJP\nAL4AfGaQRUmShqubYaIbq+qQWdtuqKpDB1rZQ+twmEiSetTPk87+GvgZ8FnaZxC/Fng08DcAVfXT\nXa62C4aBJPWun2HwvQUerqoayvyBYSBJvetbGCwWhoEk9a5vK50leRjwZ8DMOQXTwMeq6r5dqlCS\ntGh0M0z0ceBhwKeaTccBD1TVWwdc2+w67BlIUo/6OWfwkG8O+W0iSVoa+nnS2QNJntyx4ycBO7W+\ngSRpcerm2kTvATY06xqE9sI0bxloVZKkoVowDJLsAWwFfhd4WrP5m1X1q0EXJkkanm7mDK6vqmcO\nqZ6F6nDOQJJ61M85g0uSvDrJDncmSVqauukZ3A08ArgfuJf2vEFV1VCXvrRnIEm969tJZ1X1r/pT\nkiRpsepm2ctLutkmSVq65u0ZJPktYG9gvySPpj08BLAvsP8QapMkDclCw0RvB95Fe0Gb69geBr8A\nPjzguiRJQ9TNBPI7q+pDQ6pnoTqcQNai02q1WLduHQCTk5OsXr16xBVJD9bXS1gn+X3gIDp6ElV1\n5q4U2CvDQItNq9VizZo1bN26FYCxsTGmpqYMBC0q/bxQ3VnAk4GNbL8mUVXV8btcZQ8MAy02ExMT\nXHzxxQ/atmrVKi666KIRVSQ9VN++Wgo8G3iG78SStHx1cwbyTcDKndl5kgOSXJrk5iSbkszZm0hy\nWpJvJ9mY5LCdeS1p2CYnJxkbG9t2f2xsjMnJyRFWJO28boaJNgCHAVcD2y5QV1XH7HDnyUpgZVVt\nTLIP7W8lvaKqNne0eQnwjqp6WZIjgFOr6nlz7MvOiRYdJ5C12PVzzuCFc22vqst2oqj/A3yoqi7p\n2PZRYENVfa65fyswXlV3znquYSBJPdrlOYMkT6+qzVV1WZK9Oi9bneQhn9y7KOgg2j2Mq2Y9tD/w\nw477tzfb7kSSNBQLTSCfAxze3P56x22A/z3r/oKaIaJzgROqakuvRc5Yu3btttvj4+OMj4/v7K4k\naVmanp5menq65+fNO0zUuY7B7DUNelnjIMkK4ELgy1V16hyPzx4m2gy80GEiSdp1/VjPoOa5Pdf9\nhXwCuGWuIGicD/wJbBt++n+zg0CSNFgLDRMdkOQ02tckmrlNc7+rC9UlORJ4A7ApyfW0Q+Qk2uso\nV1Wtr6ovJXlpku8A9+D6ypI0dAsNE71poSdW1acGUtE8HCaSpN719dpEi4FhIEm96+cayJKkZc4w\nkCQZBpKk7tZA/p9J9k3ysCSXJPmXJG8cRnGSpOHopmcwUVW/AF4OfB94CvCeQRYlSRqubsJg5lyE\nlwFfqKqfD7AeSdIIdLO4zYXNJSK2An+W5F8D9w62LEnSMHW7BvJjgJ9X1QNJ9gb2rao7Bl7dg2vw\nPANJ6lHfzjNI8hrgviYI/itwNvCEPtQoSVokupkzeG9V3Z3kKODFwOnARwZbliRpmLoJgweaf18G\nrK+qfwQePriSJEnD1k0Y3J7kY8BrgS8l2avL50mSlohu1kDeGzga2FRV307yeODgqrpoGAV21OEE\nsiT1qK9XLU1yKPAHzd3Lq+qGXayvZ4aBJPWun98mOgH4NPDbzc/ZSd656yVKkhaLboaJbgSeX1X3\nNPcfAXy9qg4ZQn2dddgzkKQe9XM9g7D9G0U0t3e4Y0nS0tHN5SjOAK5KMtXcfyXtcw0kSctEtxPI\nhwNHNXcvr6rrB1rV3DU4TCRJPerLt4mS7AncXFVP72dxO8MwkKTe9WXOoKoeAL6Z5Il9q0yStOh0\nM2fwaODmJFcD98xsrKpjBlaVJGmougmD9w68CknSSM07TJTkKUmOrKrLOn9of7X0R8MrUdIotFot\nJiYmmJiYoNVqjbocDdi8E8hJLgROrKpNs7YfDPyPqvp3Q6iv83WdQJaGpNVqsWbNGrZu3QrA2NgY\nU1NTrF69esSVqVf9mEB+3OwgAGi2HbQLtUla5NatW7ctCAC2bt3KunXrRliRBm2hMHjUAo+N9bsQ\nSdLoLBQG1yb509kbk7wVuK6bnSc5PcmdzfWN5np83yTnJ9mYZFOSN3dVtaSBmpycZGxs+2e+sbEx\nJicnR1iRBm2hOYPHAVPAr9n+5v9s2qucramqO3a48/ZSmVuAM+e6sF2SE4F9q+rEJPsB36Q9PHX/\nHG2dM5CGqNVqbRsampycdL5gierbegZJXgT8XnP35qq6tMdCDgQumCcM/go4oKrekeR3gFZVPXWe\n/RgGktSjbsNgh+cZVNUGYENfqnqoDwPnJ/kxsA/tpTUlSUPWzUlng7QauL6q/jDJk4GLkxxSVVvm\narx27dptt8fHxxkfHx9KkZK0VExPTzM9Pd3z87q6aumu2MEw0YXA+6vqq839S4C/rKpr52jrMJEk\n9aifi9vsci3MvxjObcCLYduE9VOB7w6hJklSh4H2DJKcA4wDjwXuBE6m/W2kqqr1SR4PfBJ4fPOU\n91fVZ+bZlz0DSepR375NtFgYBpLUu8U0TCRJWuQMA0mSYSBJMgwkSRgGkiQMA0kShoEkCcNAkoRh\nIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJ\nw0CShGEgScIwkCRhGEiSMAwkSQw4DJKcnuTOJDcu0GY8yfVJbkqyYZD1SJLmlqoa3M6To4AtwJlV\ndcgcjz8S+BowUVW3J9mvqu6aZ181yFolaTlKQlVlR+0G2jOoqiuAny3Q5FjgvKq6vWk/ZxBIkgZr\n1HMGTwUek2RDkmuSHDfieiRptzTqMFgBHA68BDgaeG+Sp4y2JPVTq9ViYmKCiYkJWq3WqMuRNI8V\nI379HwF3VdW9wL1JvgIcCnxnrsZr167ddnt8fJzx8fEhlKid1Wq1WLNmDVu3bgXgiiuuYGpqitWr\nV4+4Mmn5mp6eZnp6uufnDXQCGSDJQcAFVXXwHI89HfgQ7V7BXsBVwGur6pY52jqBvMRMTExw8cUX\nP2jbqlWruOiii0ZUkbT76XYCeaA9gyTnAOPAY5P8ADgZeDhQVbW+qjYnaQE3Ag8A6+cKAknSYA28\nZ9Av9gyWntnDRGNjYw4TSUPWbc/AMNBAtVot1q1bB8Dk5KRBIA2ZYSBJWhwnnUmSlgbDQJJkGEiS\nDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCTtrlotmJho/7jWhpejkLQbarVgzRpoLqLI2BhMTcEy\nvHaWl6OQpPmsW7c9CKB9u7mg4u7KMJAkGQaSdkOTk+2hoRljY+1tuzHnDCTtnlqt7UNDk5PLcr4A\nXM9AkoQTyJKkHhgGkiTDQJJkGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEliwGGQ5PQkdya5cQft\nnpPkviSvGmQ9kqS5DbpncAaw4NWfkuwBnALs1ksNTU9Pj7qEgfL4lq7lfGyw/I+vWwMNg6q6AvjZ\nDpq9EzgX+Mkga1nslvt/kB7f0rWcjw2W//F1a6RzBkmeALyyqj4C7PCqepKkwRj1BPIHgb/suG8g\nSNIIDHw9gyQHAhdU1SFzPPbdmZvAfsA9wNuq6vw52rqYgSTthG7WM1gxhDrCPJ/4q+pJ2xolZ9AO\njYcEQdPWXoMkDchAwyDJOcA48NgkPwBOBh4OVFWtn9XcT/6SNCJLZtlLSdLgjHoCuStJjk6yOcm3\nkvzljp+xdHR7Yt5SlOSAJJcmuTnJpiTHj7qmfkqyV5KrklzfHN/Jo65pEJLskeQbSeYcwl3Kknw/\nyQ3N3/DqUdfTb0kemeQLSW5t/j88Yt62i71n0JyU9i3gj4AfA9cAr6uqzSMtrE+SHAVsAc6ca5J9\nKUuyElhZVRuT7ANcB7xiufztAJLsXVW/TLIn8FXg+KpaVm8qSf4CeBawb1UdM+p6+qn5EsuzqmpH\n50MtSUk+CVxWVWckWQHsXVW/mKvtUugZPBf4dlXdVlX3AZ8FXjHimvqmyxPzlqSquqOqNja3twC3\nAvuPtqr+qqpfNjf3oj0Ht7g/XfUoyQHAS4GPj7qWAQlL432wZ0n2Bf6gqs4AqKr75wsCWBq/hP2B\nH3bc/xHL7A1ld5DkIOAw4KrRVtJfzRDK9cAdwMVVdc2oa+qzvwPewzILuQ4FXJzkmiR/Oupi+ux3\ngLuSnNEM861PMjZf46UQBlrimiGic4ETmh7CslFVv6mqZwIHAEckecaoa+qXJC8D7mx6d/N+RXyJ\nO7KqDqfd+/nzZth2uVgBHA78r+YYfwn81XyNl0IY3A48seP+Ac02LQHNOOW5wFlV9cVR1zMoTfd7\nA3D0qGvpoyOBY5px9c8AL0py5ohr6quq+r/Nv/8CTNEell4ufgT8sKqube6fSzsc5rQUwuAa4ClJ\nDkzycOB1wHL7VsNy/dQF8Anglqo6ddSF9FuS/ZI8srk9BqwCls3keFWdVFVPbE4OfR1waVX9yajr\n6pckeze9VpI8ApgAbhptVf1TVXcCP0zy1GbTHwG3zNd+GGcg75KqeiDJO4CLaIfX6VV164jL6pu5\nTsybmfBZ6pIcCbwB2NSMqxdwUlX902gr65vHA59qvvG2B/C5qvrSiGtS9x4HTDWXulkBfLqqLhpx\nTf12PPDpJA8Dvgu8Zb6Gi/6rpZKkwVsKw0SSpAEzDCRJhoEkyTCQJGEYSJIwDCRJGAbqUpIHmuub\nbEryuSS/1ePz7+6x/RlJXjXH9mcl+WBz+01JTmtuvz3JGzu2r+zx9TY0l0nfmOSWJKfNnFDWPH7F\nDp5/4g4evzDJvs3Jk5t6rO2FSZ7fcX/bse6KnazlTUk+tKuvrcXHMFC37qmqw6vqYOA+4D/ObpBk\nobOo+3JCS1VdV1XvmmP7x6rq7Obum9m5ixm+vqoOAw4Bfg1su3xGVe3omjUnzfdAklTVyzuuGNnr\n72Ic+P2OWjqPdVftzN/Fk5OWIcNAO+Nytl8iZHOSTzWfMA9I8vokNzY/p3Q8J0n+NslNSS5O8thm\n41uTXN0sLvKFWT2OVc3VJDc3F02b+ZR8weyCkpycZDLJq4FnA2c3PZmXJpnqaPfiJP8wz3EF2pf6\nBf4z8MQkBzfPu7v5d2WSy5p935jkyCTvB8aabWfN8Xv5N0m+l+Qxzes8LMnZTQ/k8zPH3Nmm6QFt\nSHIg7eB9V7P/I5tjfXfT7rAkX296NOd1XB5jQ5JT0l58Z3NzNvi8mk/85yX5cpJvJvnrjsfe0my7\nkvb1ima275fk3OY1rprpvST5YJL3NrdXJ5le6LW1OBgG6lZg24XnXgLMDC/8LvDhpsdwP3AK7U+y\nhwHPSTKzGMojgKur6veArwBrm+3nVdVzmyt/bgb+Q8drHlhVzwFeDnw07WtTwfyfTKuqzgOuBY5t\nejJfAp42Ez60T8c/fUcHW1W/AW4Anj7rNY8F/qm5CuShwMaqOhH4ZfN6xzXtnjLze6mqH8yq+WnN\nY88A7gb+0zzHVVV1G/BR4O+a/X91VptPAe9pejQ30V5nfMaeVXUE8Bds/30v5FDgNbR7Rq9Nsn8z\n3LYWeD5wFNB5VdZTgb9tXuPfs/33eiLwx0nGmzZv7uK1NWKGgbo1luQbwNXAbWz/H//7Hdfwfw6w\noap+2ryZfhp4QfPYb4DPN7fPZvsnzEOSfCXtZT+PBf5tx2t+HqCqvgP8M9vfmLvROWR1FvDG5lPz\n84Av78Q+ZlwDvCXJfwMOqap75nnubbPWNujc1w+q6srm9tm032Tne735i2svXvLIZoEkaAfDCzqa\nzPSArgMO7GKXl1TVlqr6FXBz85wj2P43vR/4XEf7FwMfTvu6U+cD+6S98ttW4G3AxcBpVfX9Xo5L\no7HoL1SnReOXzafhbZopgtlvht2+oc18Cj4DOKaqbkryJuCFc7SZ2e/OjlV/ErgA+BXwhSaoFpT2\nxecOZtZVHqvq8iQvAF4GfDLJumb8fvZxzxcSMEcPoPn3frZ/QOt2gn6h3/evmn8foLv/13/Vcfs3\nHc+Z7zUCHNGsQDjbIcBduBDVkmHPQN1a6A1hxtXAC5I8Ju01gV8PTDeP7UF7KAHaVzK9vLm9D3BH\n2ldVfMOsfb8mbU+mvWrTN7us9W5g35k7zTXrfwz8F9rhM5/OobBTaH+Cv3nWY08EflJVp9NeCnIm\nIH/dHPOD9jWPA7N9YfJj2f67+B7ttYYBXj3f8XQc1y+An3bMBxwHXLbQse2Eq2j/TR/d/I1e0/HY\nRcAJ214gObT590DaQ1PPBF6SZDmtEbBsGQbq1rzj9NtuVN1BeyWlaeB64NqqurB5eAvw3GZCdRx4\nX7P9vbRD5HLaayR3+kHz2D8Cb6+qX3dZ6ydpzzF8I8lezbZP017oY6FAOTvJRtrzIWM8eK3tmeMc\nB25ohsz+mPaYOMB62pfqPmtW+9nPh/bcyJ8nuQV4FO05AYD/DpyW5GravYQZFwBrZiaQZ+3rzcAH\nmroPbfaxo9fvRsG2v+la4Eraf6POntIJwLOT3JDkJuDtzfaPA5PNc98K/H3HfI8WKS9hrd1C2t+N\n/8ZyWStC6jfDQMtekmtp90xWzTO+Le32DANJknMGkiTDQJKEYSBJwjCQJGEYSJIwDCRJwP8HFyQH\nxa89BJcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10a9dd080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q1 = np.array([0.01, 0.49, 0.5])\n",
    "q2 = np.array([1/3, 1/3, 1/3])\n",
    "q3 = np.array([0.05, 0.1, 0.85])\n",
    "q4 = np.array([0.58, 0.12, 0.3])\n",
    "p = np.array([0.25, 0.25, 0.5])\n",
    "qs = [q1, q2, q3, q4]\n",
    "\n",
    "def crossEntropy(prob):\n",
    "    p = np.array([0.25, 0.25, 0.5])\n",
    "    R = -sum(p*np.log2(prob))\n",
    "    return R\n",
    "\n",
    "rs = []\n",
    "for q in qs:\n",
    "    rs.append(crossEntropy(q))\n",
    "\n",
    "plt.scatter(np.linspace(1, 4, 4, endpoint=True), rs, color=\"black\")\n",
    "plt.scatter(5, crossEntropy(p), color=\"red\")\n",
    "plt.xlabel(\"Probability Distribution Index\")\n",
    "plt.ylabel(\"Cross Entropy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The red dot in the graph above corresonds to the cross entropy when $P = Q$. In other words, the cross entropy was calculated: \n",
    "\n",
    "$$ R = -\\sum_{i}p_{i}log_{2}p_{i} $$\n",
    "\n",
    "We can visually see that this cross entropy is the lowest value compared to other cross entropies calculated when $P \\neq Q$. Thus, cross entropy can be interpreted as a measure of how close $P$ is to $Q$ - the closer they are, the smaller the cross entropy value. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Minimizing Cross Entropy\n",
    "Now returning to the topic of logistic regression, our probablility model is:\n",
    "\n",
    "$$ q = \\frac{1}{1 + e^{-z}} $$\n",
    "\n",
    "The true $P$ is just our labels $\\in [0, 1]$, so our cross entropy expression reduces to:\n",
    "\n",
    "$$ R = -log_{2}(1 + e^{-z})^{-1} = log_{2}(1 + e^{-z})$$\n",
    "\n",
    "Where $z^{(i)} = y^{(i)}x^{(i)}\\beta$ and is sometimes referred to as the functional margin. $(i)$ denotes the $i^{th}$ sample, and $y \\in \\{-1, +1\\}$. We now call this logistic loss, and this is the function that we want to minimize. Let us plot this loss function that we have chosen:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEPCAYAAACneLThAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHXhJREFUeJzt3Xu8XOPZ//HPlewkEhKCOAYRSotGkDpFdB4V50NfKElE\nHKuIpkqlDvXYyvPrQ59H41S00YhIUOrQoCIio0XjEAkhNNEoSaMtTbDDgyT7+v1xTw62fZi996y5\n16z5vl+vee3ZM2tmfceWa9/7Wve6l7k7IiKSTR1iBxARkeSoyIuIZJiKvIhIhqnIi4hkmIq8iEiG\nqciLiGRY4kXezH5oZq+a2StmNtHMOie9TxERCRIt8ma2BfB9YHd37wfUAEOS3KeIiKxRU4Z9dATW\nNbN6oBuwuAz7FBEREh7Ju/ti4H+Bd4C/Ax+4+xNJ7lNERNZIul2zAXA0sA2wBbCemQ1Lcp8iIrJG\n0u2aA4EF7r4EwMzuB/YFJq29kZlpAR0RkVZyd2tpm6Rn17wD7G1m65iZAd8CXm9sQ3fP5O3yyy+P\nnkGfT59Pny97t2Il3ZN/HrgPmAW8DBjwqyT3KSIiayQ+u8bdrwCuSHo/IiLyZTrjNWG5XC52hETp\n81U2fb7ss9b0dhILYeZpyCEiUinMDE/BgVcREYlIRV5EJMNU5EVEMkxFXkQkw1TkRUQyTEVeRCTD\nVORFRDJMRV5EJMNU5EVEMkxFXkQkw1TkRUQyTEVeRCTDVORFRDJMRV5EJMOqqsi7w6xZsVOIiJRP\nVRX5ujo44giYPj12EhGR8qiqIt+jB4wdCyefDEuXxk4jIpK8RIu8me1gZrPM7KXC1w/NbFSS+2zJ\noYfCUUfByJExU4iIlEfZLv9nZh2ARcBe7r6wwXNlvfzfJ5/AHnvAZZfBsGFl262ISMmk8fJ/BwJ/\nbVjgY+jWDSZOhPPOg3feiZ1GRCQ55SzyJwB3lXF/zdp9d/jhD0N/vr4+dhoRkWSUpV1jZp2AxcBO\n7v5eI8+XtV2zysqVkMvB0UfDj35U9t2LiLRZse2amnKEAQ4FZjZW4Fepra1dfT+Xy5HL5RIP1bEj\nTJgA3/gGDB4Mu+6a+C5FRNokn8+Tz+db/bpyjeTvAh5z9/FNPB9lJL/KHXfANdfAiy/COutEiyEi\nUrRiR/KJF3kz6wa8DfR197omtola5N3hhBNgyy3hF7+IFkNEpGipKfLFiF3kAZYsgX79YNy40LoR\nEUmzNE6hTLUNNwwF/rTTQsEXEckCjeQbOO88WLwY7rkHrMXfkSIicWgk30Y/+xnMnQt33hk7iYhI\n+2kk34iXX4YDD4QXXoA+fWKnERH5Mo3k22HXXWH0aDjppHDClIhIpVKRb8L550NNDfz857GTiIi0\nndo1zXjnHRgwAB57LKx1IyKSFmrXlMDWW8OYMXDiiWF5YhGRSqORfBGGDYONNoIbboidREQk0Bmv\nJbR0KfTvD7fcEq4sJSISm4p8iU2fDsOHh+mVG28cO42IVDsV+QRceCG8+Sbcf7/OhhWRuHTgNQFX\nXQULFoQ1bkREKoFG8q00Zw4ccADMmAHbbRc7jYhUK43kE/L1r8Oll4azYVesiJ1GRKR5KvJtMGoU\ndOsWFjMTEUkztWvaaNEi2GMPmDwZ9twzdhoRqTZq1ySsd2+48cYwrfLjj2OnERFpnEby7TRiRGjd\n3HJL7CQiUk1SM5I3s/XN7F4ze93MXjOzvZLeZzndcANMmQIPPxw7iYjIlyU+kjez24Gn3H2cmdUA\n3dz9owbbVOxIHuBPf4Ljjw9nw26ySew0IlINUnHGq5n1AGa5e7Mzyiu9yANcfDG89ho89JDOhhWR\n5KWlXbMt8L6ZjTOzl8zsV2bWNeF9RnHFFWHGza9/HTuJiMgaNWV4/92Bke7+opmNAS4CLm+4YW1t\n7er7uVyOXC6XcLTS6twZJk6EQYMgl4MddoidSESyJJ/Pk8/nW/26pNs1mwJ/dve+he/3A37s7kc2\n2K7i2zWr3HgjTJgATz8NnTrFTiMiWZWKdo27/xNYaGarxrXfAuYmuc/YRo6EDTcMi5mJiMRWjtk1\nuwJjgU7AAuBUd/+wwTaZGckDvPsu7LYbPPAA7LNP7DQikkWpmF1TrKwVeQhrzo8eDbNmQffusdOI\nSNaoyKfA6aeH6ZRjx8ZOIiJZk4qefLUbMyZcNvDBB2MnEZFqpZF8wp59Fo45JrRtNt88dhoRyQqN\n5FNi333hzDND6yajv8dEJMVU5Mvgssvg/ffh5ptjJxGRaqN2TZnMmwcDB8JTT8FOO8VOIyKVTu2a\nlNlhh3C5wGHD4NNPY6cRkWqhkXwZucN3vhOuKjVmTOw0IlLJNE8+pZYsgf79w5WkDjssdhoRqVQq\n8in21FMwZAjMng2bbho7jYhUIvXkU+yb3wxTKk85BerrY6cRkSxTkY/k8sth6VK4/vrYSUQky9Su\niWjBAthrL5g6NfTpRUSKpXZNBejbN8yyGToUPvkkdhoRySKN5FPgpJNg3XXDjBsRkWJoJF9Bbrop\ntGweeCB2EhHJGo3kU2LGDDj6aJg5M5wsJSLSHI3kK8zee8OoUTBiBKxcGTuNiGSFinyKXHRRKPA/\n/3nsJCKSFeW4kPffgA+BemC5u+/ZyDZV365ZZeFCGDAAJk+GPb/0X0pEJEhTu6YeyLn7bo0VePmi\nrbYKB2KHDYO6uthpRKTSlaPIW5n2kxnHHQf/8R9w7rmxk4hIpWtV8TWznmbWr5X7cGCqmb1gZt9t\n5Wur1pgxYcbNpEmxk4hIJatpaQMzywNHFbadCfzLzJ5x9/OL3MdAd3/XzHoRiv3r7v50w41qa2tX\n38/lcuRyuSLfPpvWXRfuugsOOQT22Qe23TZ2IhGJKZ/Pk8/nW/26Fg+8mtksd9/NzM4AtnL3y83s\nFXdv7YgeM7scqHP3axs8rgOvTbj2WrjvPvjjH6GmxV/JIlItSnngtcbMNgeOBx5uZYhuZrZe4f66\nwEHAq615j2p33nnQvTtceWXsJCJSiYoZG/4UmAI87e4vmFlfYH6R778p8ICZeWFfE9398bZFrU4d\nOsD48bDbbnDggTBoUOxEIlJJtKxBhXjkETjnnHA1qZ49Y6cRkdhK1q4xs2vMrIeZdTKzaWb2npkN\nL01MKdbhh4e1bc46K1wQXESkGMX05A9y94+AI4C/AdsDFyYZShp3zTUwdy7cfnvsJCJSKYo68Fr4\nejhwr7t/mGAeacY664RplaNHw7x5sdOISCUopsg/bGZvAHsA0wrz3T9NNpY0ZZdd4IorwrIHn38e\nO42IpF1RB17NbEPgQ3dfaWbdgB7u/o+ShdCB11ZxD/35r30Nrr46dhoRiaHYA6/FnAzVCTgb2L/w\n0FPALe6+vN0p1+xDRb6V3n8/XPz79tvD1EoRqS6lLPJjgU7A+MJDJwEr3f2Mdqdcsw8V+TZ44gk4\n+eQwrbJXr9hpRKScSlnkX3b3XVt6rD1U5Ntu9Gh44w146CGwFn/cIpIVpVzWYKWZbbfWG/cFdIG6\nlLjqKli8GG6+OXYSEUmjYkby3wLGAQsIa8NvA5zq7tNLFkIj+XaZNw8GDoTp08PsGxHJvpK1awpv\n1gXYsfDtX9z9s3bma/j+KvLtNG5cWLHy+eeha9fYaUQkae0u8mZ2THMvdPf725itsX2pyLeTOwwZ\nAptuCtdfHzuNiCStFEV+XDOvc3c/ra3hGtmXinwJLF0aVqu86aaw1o2IZFdJ2zVJU5EvnT/9CY4/\nHl56CTbfPHYaEUlKKWfXSAUZNAjOPBNOOQXq62OnEZHYVOQz6LLLoK4uXAxcRKqb2jUZ9dZbsNde\nMGVK6NOLSLaU8qIhI81sg7W+72lm57Q3oCRr223huutg6FD4+OPYaUQklmJOhprt7v0bPDbL3Us2\nPtRIPjknnwydO8Ovfx07iYiUUikPvHY0W7Mqipl1BDq3MkwHM3vJzH7fmtdJ+914YzgT9r77YicR\nkRiKKfKPAfeY2bcKSxzcVXisNX4AzG1tOGm/7t1h0qRwEfC33oqdRkTKrZgi/2NgOmFN+bOBacDo\nYndgZr2Bw4CxbQko7bfnnnDJJXDccfCpruklUlUSn11jZvcC/wWsD1zg7kc1so168glzhxNOgJ49\n4dZbY6cRkfYqtidf09QTZvZbdz/ezOYAX6rA7t6viBCHA/9099lmliOsYtmo2tra1fdzuRy5XK6l\nt5dWMIOxY8Oofvz4cEBWRCpHPp8nn8+3+nXNrV2zubu/a2bbNPa8u7/d4pub/T9gOLAC6Ap0B+53\n9xENttNIvkxeew1yOZg2Dfq1+GtaRNKq3bNr3P3dwt1z3P3ttW9AUfPk3f0Sd9/a3fsCQ4AnGxZ4\nKa+ddw5nwh57LHz4Yew0IpK0Yg68Dm7ksUNLHUTK58QT4aCDwvo2+gNKJNuaa9ecTRixbwe8udZT\n3YFn3H14yUKoXVN2n30G++8P3/kO/OhHsdOISGuVYj359YGewM+Ai9Z6qs7dl5Qk5Zp9qchH8M47\n4UDsb38bCr6IVI6SrSdfuIj3Inf/rDBDph9wh7t/UJKkqMjHNGUKnHYavPii1p8XqSSlXNbgd8BK\nM9se+BWwFTCpnfkkJQ4+OKw/P2QIrFgRO42IlFoxRb7e3VcAxwA3uPuFgMZ8GXLZZdCtWzgrVkSy\npZgiv9zMhgIjgIcLj3VKLpKUW4cOcOedoTf/wAOx04hIKRVT5E8F9gH+y93fMrNtgQnJxpJy22gj\nuPde+N73YP782GlEpFR0ZSj5gptvDrcZM0ILR0TSqRRTKNu9dk2xVOTTwx1GjAgtnNtvD2veiEj6\nlKLIt3vtmmKpyKfLxx+H68OOGhVm3ohI+pRsnnw5qMinz1/+AoMGwaOPwoABsdOISEOlvJB3nZl9\n1OC20MweMLO+pYkrabPjjvDLX4ZlD5aU9PxmESmnYs54vRJYRDgBygirSW4HvASc7e65dofQSD61\nLrgA3ngDJk8OfXoRSYdSLmvwsrvv2uCx2e7ev7Hn2kJFPr2WL4cDDghnxv7kJ7HTiMgqpVzW4BMz\nO97MOhRuxwOrrhSqypxxnTrBPfeE1s3UqbHTiEhrFTOS7wtcRzghCuDPwA+BvwN7uPvT7Q6hkXzq\n5fMwdCg8/zxstVXsNCKi2TVScldfDQ8+CE89BZ07x04jUt1KObumd2Emzb8Kt9+ZWe/SxJRKMno0\nbLKJLjIiUkmK6cmPA34PbFG4TS48JlXGDMaPh0cegbvvjp1GRIpRTE9+trv3b+mxdoVQu6aizJ4N\ngweHts1OO8VOI1KdSjm75t9mNtzMOhZuw4F/Fxmii5k9Z2azzGyOmV1ezOsk3fr3D/35Y4+FurrY\naUSkOcWM5LcBbiDMrnHgWeD77r6wqB2YdXP3T8ysI/AMMMrdn2+wjUbyFeiMM2DZMrjrLi1kJlJu\nJRvJu/vb7n6Uu/dy903c/dvAscUGcfdPCne7ADVobn1m3HBDWOPmxhtjJxGRprRpCqWZvePuWxe5\nbQdgJmEphJvc/eJGttFIvkItWAB77w0PPQT77NPy9iJSGsWO5Gva+v7Fbuju9cBuZtYDeNDMdnL3\nuQ23q62tXX0/l8uRy+XaGE3KqW9fuO02OOEEmDkTevWKnUgkm/L5PPl8vtWvS3wk3+B1lwEfu/u1\nDR7XSL7CXXJJOBt2yhTo2DF2GpHsa3dPvoklhj8yszrCfPliQmxsZusX7ncFBgNvFPkZpIL89KdQ\nXw9r/UEmIinQZLvG3buX4P03B8YX+vIdgHvc/dESvK+kTE1NmGUzYEDo0R9+eOxEIgJau0ZK7Jln\n4Jhj4LnnoE+f2GlEsquUJ0OJFG3gQLj4YjjuOPj005a3F5FkaSQvJeceZtv07Am33ho7jUg2aSQv\n0ZjB2LFhbZvx42OnEaluGslLYl57DXI5mDYN+vWLnUYkWzSSl+h23hnGjAkHYt97L3YakeqkIi+J\nOvHEcNnAQw6Bjz6KnUak+qhdI4lzh3PPDe2bP/wBunaNnUik8ukar5Iq9fVhVP/JJ/C734WTp0Sk\n7dSTl1Tp0CHMtFm+HE4/PRR9EUmeiryUTefOcN998Ne/wvnnhzaOiCRLRV7Kqls3ePhhmD4drroq\ndhqR7FNnVMpugw3CksT77QcbbQTnnBM7kUh2qchLFJttBlOnwv77h6I/bFjsRCLZpCIv0Wy7bZhS\neeCBodAfdljsRCLZo568RLXLLvDgg3DKKfD007HTiGSPirxEt/feMHEiHHsszJ4dO41ItqjISyoM\nHgw33RRaNvPnx04jkh3qyUtqHHccfPABHHRQaN1suWXsRCKVT0VeUuWMM2DJklDo//jHMMVSRNou\n0XaNmfU2syfN7DUzm2Nmo5Lcn2TD6NFwxBGhdVNXFzuNSGVLdIEyM9sM2MzdZ5vZesBM4Gh3f6PB\ndlqgTL7AHc48E956Cx55BLp0iZ1IJF1SsUCZu//D3WcX7i8DXgfUaZUWmcEtt4TrxA4bBitWxE4k\nUpnKNrvGzPoA/YHnyrVPqWwdO8Kdd4aWzVlnaUEzkbYoy4HXQqvmPuAHhRH9l9TW1q6+n8vlyOVy\n5YgmKdelC9x/fzgrdvRouOaaMMoXqTb5fJ58Pt/q1yV+0RAzqwEeBv7g7tc1sY168tKsJUvCOjfD\nh8NFF8VOIxJfsT35cozkfwPMbarAixRjww3h8cfDypUbbhgOyopIyxIt8mY2EDgRmGNmswAHLnH3\nx5Lcr2TTFluEQv/Nb4YFzY4/PnYikfTTNV6l4rzySlgG4Y474OCDY6cRiSMVUyhFktCvXzgYO3w4\n/PnPsdOIpJuKvFSkgQPDSP7b34Y5c2KnEUkvFXmpWIceCtddF74uWBA7jUg6aYEyqWhDhsDSpaFH\n//TTsPnmsROJpIuKvFS8s8/+4sqVPXvGTiSSHppdI5ngDhdcADNmhAuEr7tu7EQiySp2do2KvGRG\nfT2cdhr84x/w+99D586xE4kkR0VeqtKKFeEKU126wKRJYZEzkSzSPHmpSjU1cPfd8N57MHKkVq4U\nUZGXzFlnHXjwQXjxRbj4YhV6qW5q10hmvfdemEO/5ZZw222w8caxE4mUjto1UvV69YJnn4Udd4T+\n/eHJJ2MnEik/jeSlKkydCqecAiedBFdeCZ06xU4k0j4ayYusZfBgmD0bXn01rHvz5puxE4mUh4q8\nVI1evWDy5DCa32efsMCZ/oCUrFO7RqrSK6+EdW922w1++UtYf/3YiURaR+0akWb06xemWPboEQr9\njBmxE4kkQyN5qXoPPABnnQWjRoWLhOssWakEWtZApBUWLQq9eoAJE6B377h5RFqSinaNmd1mZv80\ns1eS3I9Ie/XuDU88EWbh7LFHGN2LZEGiI3kz2w9YBtzh7v2a2U4jeUmN556DYcNCwb/2WujWLXYi\nkS9LxUje3Z8Glia5D5FS22svmDULli2DAQPg5ZdjJxJpO82uEWlEjx5w551hgbMDD4Trr9eceqlM\nqbn8X21t7er7uVyOXC4XLYvIKiedBPvuG9o3U6bAuHGwySaxU0k1yufz5PP5Vr8u8dk1ZrYNMFk9\nealky5fDf/5nOEt23LhwPVmRmFLRk1+VpXATqVidOsHPfhamV55+Olx4IXz+eexUIi1LegrlJOBZ\nYAcze8fMTk1yfyJJO+CAcFB23ryw/s28ebETiTRPJ0OJtIE73HJLaOFcc01Yxtj096qUkc54FSmD\nV1+FoUNhp53g1lthgw1iJ5JqkaaevEhm7bILPP98mHHTvz8880zsRCJfpJG8SIlMngzf/S6cfTZc\neinUpGaCsmSR2jUiESxeDCNGwGefwcSJsPXWsRNJVqldIxLBFlvA44/DkUeGdepPPx3yeaivj51M\nqpVG8iIJWbwY7rornED1wQcwfHg4g/arX42dTLJA7RqRFHnllXAi1cSJsOWWodgPGaIlEqTtVORF\nUmjlSpg2LRT8yZNh0KBQ8I88Erp2jZ1OKomKvEjKLVsWLk5yxx0wcyYcc0wo+IMGQQcdLZMWqMiL\nVJC//x0mTQoFv65uTf9+xx1jJ5O0UpEXqUDu4SIlEyaEor/VVmv69716xU4naaIiL1LhVqxY079/\n+GHYf/81/ft11omdTmJTkRfJkLo6uP/+UPBfegmOPTacdDVwoPr31UpFXiSjFi0KUzEnTICPP17T\nv99hh9jJpJxU5EUyzh1mz17Tv+/TJxT7E06AjTeOnU6SpiIvUkVWrICpU0PBf/TR0MbZddcwuv/K\nV8LXjTfWmvdZoiIvUqU++ihcdPz112H+/HD1qnnzwsh/VdFfVfhX3dc6+JVHRV5EVnOHf/97TdFf\n++v8+dCt2xdH/au+br99eE7SJzVF3swOAcYQVry8zd2vbmQbFXmRSNzh3Xcb/wWwYEFo8zQs/l/5\nCvTtC507x05fvVJR5M2sAzAP+BawGHgBGOLubzTYLrNFPp/Pk8vlYsdIjD5fZWvp861cCQsXfnHU\nv+r+woVhsbWGLaDttgvtn+7doUuX8n2WxmT551dskU/62jV7AvPd/e1CqLuBo4E3mn1VhmT5fzLQ\n56t0LX2+jh3DrJ0+feCgg7743Oefw9/+tqboz50LDz0URv8ffhjm9puFYt+9O/To0fL95p7r2rX1\nB46z/vMrRtJFfktg4VrfLyIUfhGpcJ07h5F7c/PzP/ssHAiuqwu3pu6/+274ZdHcNsuXt/zLYL31\nQq6aGujUCZ59Fq6/fs337f3a1HNpPiFNV6EUkcR06RLW3CnFujvLlzde/Nf+ftmysN3//d+a5958\nMzy2YkUyX5cvD0W+W7ewv7RJuie/N1Dr7ocUvr8I8IYHX80smw15EZEEpeHAa0fgL4QDr+8CzwND\n3f31xHYqIiKrJdqucfeVZnYu8DhrplCqwIuIlEkqToYSEZFkpOaYsJl938xeN7M5ZvbfsfMkwcwu\nMLN6M9swdpZSMrNrCj+72Wb2OzPrETtTe5nZIWb2hpnNM7Mfx85TSmbW28yeNLPXCv/eRsXOlAQz\n62BmL5nZ72NnKTUzW9/M7i38u3vNzPZqattUFHkzywFHAl93968D/xM3UemZWW9gMPB27CwJeBzY\n2d37A/OBiyPnaZfCSXw3AgcDOwNDzeyrcVOV1ArgfHffGdgHGJmxz7fKD4C5sUMk5DrgUXf/GrAr\n0GQbPBVFHjgb+G93XwHg7u9HzpOEXwAXxg6RBHd/wt3rC9/OAHrHzFMCq0/ic/flwKqT+DLB3f/h\n7rML95cRCsSWcVOVVmFQdRgwNnaWUiv8pTzI3ccBuPsKd/+oqe3TUuR3APY3sxlmNt3MBsQOVEpm\ndhSw0N3nxM5SBqcBf4gdop0aO4kvU0VwFTPrA/QHnoubpORWDaqyeNBxW+B9MxtXaEf9ysy6NrVx\n2U6GMrOpwKZrP0T4AfykkKOnu+9tZt8Afgv0LVe2Umjh811CaNWs/VxFaebzXerukwvbXAosd/dJ\nESJKK5nZesB9wA8KI/pMMLPDgX+6++xCK7ji/r21oAbYHRjp7i+a2RjgIuDypjYuC3cf3NRzZnYW\ncH9huxcKByc3cvd/lytfezX1+cxsF6AP8LKZGaGVMdPM9nT3f5UxYrs09/MDMLNTCH8eH1CWQMn6\nO7D1Wt/3LjyWGWZWQyjwE9z9odh5SmwgcJSZHQZ0Bbqb2R3uPiJyrlJZROgMvFj4/j6gyckBaWnX\nPEihOJjZDkCnSirwzXH3V919M3fv6+7bEn5Au1VSgW9JYTnpC4Gj3P2z2HlK4AVgezPbxsw6A0OA\nrM3Q+A0w192vix2k1Nz9Enff2t37En52T2aowOPu/wQWFmolhJNNmzzAnJa1a8YBvzGzOcBnQGZ+\nII1wsvfn4w1AZ2Bq+GOFGe5+TtxIbZf1k/jMbCBwIjDHzGYR/p+8xN0fi5tMWmEUMNHMOgELgFOb\n2lAnQ4mIZFha2jUiIpIAFXkRkQxTkRcRyTAVeRGRDFORFxHJMBV5EZEMU5EXEckwFXkRkQxTkRdp\nhJl9z8xmFVb5W2Bm02JnEmkLnfEq0ozCQl7TgKvd/dHYeURaSyN5keZdT1jgSgVeKlJaFigTSZ3C\n8slbVfJiayIq8iKNMLM9gAuA/WJnEWkPtWtEGjcS6AlMX3WJtdiBRNpCB15FRDJMI3kRkQxTkRcR\nyTAVeRGRDFORFxHJMBV5EZEMU5EXEckwFXkRkQxTkRcRybD/DxuS6CJLsKa9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x108917eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import math\n",
    "\n",
    "z = np.linspace(-5, 5, 10)\n",
    "\n",
    "def crossEntropy(z):\n",
    "    return np.log2(1 + np.exp(-z))\n",
    "\n",
    "y = crossEntropy(z)\n",
    "plt.plot(z, y)\n",
    "plt.xlabel(\"z\")\n",
    "plt.ylabel(\"Logistic loss\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that when $z$ is positive, which occurs when $y^{(i)}$ and $x^{(i)}\\beta$ have the same sign, the logistic loss is small. However, when $z$ is negative, which occurs when $y^{(i)}$ and $x^{(i)}\\beta$ have different signs, then logistic loss is great. Hence, the more the predicted class and actual class disagree, the greater the penalty. \n",
    "\n",
    "One should note that the logistic loss is also called the negative log likelihood from a probabilisitic standpoint. Minimizing the negative log likelihood is equivalent to maximizing the log likelihood, so what we are doing is equivalent to finding $\\beta$ that maximizes the likelihood of the data being seen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we are going minimize our logistic loss using gradient descent. We will be using a modified logistic loss with natural log instead of log base 2 to make the derivation easier, but the transformation will not alter the end result. \n",
    "\n",
    "$$ R = ln(1 + e^{-y^{(1)}x^{(1)}\\beta}) + ... + ln(1 + e^{-y^{(m)}x^{(m)}\\beta})$$\n",
    "\n",
    "Let us derive our update equation by first finding the gradient. Note that $X \\in \\mathbb{R}^{m x n}$, where $m$ is number of samples and $n$ is number of features.\n",
    "\n",
    "$$ X = \n",
    "\\begin{bmatrix}\n",
    "x_{10} & \\dots & x_{1n}\\\\\n",
    "\\vdots & \\ddots & \\vdots \\\\\n",
    "x_{m0} & \\dots & x_{mn} \\\\\n",
    "\\end{bmatrix}$$\n",
    "\n",
    "$$ \\frac{\\partial R}{\\partial \\beta_{0}} = -\\Big[\\frac{x_{10}y^{(1)}e^{-y^{(1)}x^{(1)}\\beta}}{1 + e^{-y^{(1)}x^{(1)}\\beta}} + ... + \\frac{x_{m0}y^{(m)}e^{-y^{(m)}x^{(m)}\\beta}}{1 + e^{-y^{(m)}x^{(m)}\\beta}}\\Big]$$\n",
    "$$\\vdots$$\n",
    "$$ \\frac{\\partial R}{\\partial \\beta_{n}} = -\\Big[\\frac{x_{1n}y^{(1)}e^{-y^{(1)}x^{(1)}\\beta}}{1 + e^{-y^{(1)}x^{(1)}\\beta}} + ... + \\frac{x_{mn}y^{(m)}e^{-y^{(m)}x^{(m)}\\beta}}{1 + e^{-y^{(m)}x^{(m)}\\beta}}\\Big]$$\n",
    "\n",
    "Given that:\n",
    "\n",
    "$$\\frac{e^{-y^{(i)}x^{(i)}\\beta}}{1 + e^{-y^{(i)}x^{(i)}\\beta}} = \\frac{1}{e^{y^{(i)}x^{(i)}\\beta} + 1}$$\n",
    "\n",
    "and that $diag(y)$ means to turn a vector into a square matrix where each element in the vector becomes a diagonal element in the matrix, with zero all non-diagonal elements, we can write out the gradient in matrix form:\n",
    "\n",
    "$$\\frac{\\partial \\textbf{R}}{\\partial \\beta} = -(diag(y)X)^{T} \\frac{1}{e^{diag(y)X\\beta} + 1}$$\n",
    "\n",
    "Now with the gradient $G \\in \\mathbb{R}^{n x 1}$, we can write our update equation:\n",
    "\n",
    "$$ \\beta := \\beta - \\alpha G$$\n",
    "\n",
    "Where $\\alpha$ is our learning rate or step size. We are now ready to implement our logistic regression algorithm!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to use our logistic regression implementation to predict whether an email is spam or not spam (ham). Our dataset consists of $3450$ emails, each with $57$ features. Most features are proportions of words that match a certain word. Some features however, are created, such as the sum of lengths of uninterrupted sequences of capital letters. Let us load our dataset and randomly partition $90\\%$ of it as the validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from scipy import io\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "data = io.loadmat('datasets/spam.mat')\n",
    "Xtrain = data['Xtrain']\n",
    "Ytrain = data['Ytrain']\n",
    "m = Xtrain.shape[0]\n",
    "idx = np.random.permutation(m)\n",
    "Xtrain = data['Xtrain'][idx, :]\n",
    "Ytrain = data['Ytrain'][idx, :]\n",
    "Xvali = Xtrain[round(0.9*m):m, :]\n",
    "Yvali = Ytrain[round(0.9*m):m, :]\n",
    "Xtrain = Xtrain[:round(0.9*m), :]\n",
    "Ytrain = Ytrain[:round(0.9*m), :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often times, it is a good idea to preprocess our data in order to get better performance. A common preprocessing step is to standardize each feature so that it has mean $0$ and unit variance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def standardize(a):\n",
    "    return (a - np.mean(a)) / np.std(a)\n",
    "\n",
    "XtrainP = np.apply_along_axis(standardize, 0, Xtrain)\n",
    "XvaliP = np.apply_along_axis(standardize, 0, Xvali)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is time to write our gradient descent methods. There are two types of gradient descent methods - batch gradient descent and stochastic gradient descent. In batch gradient descent, the gradient is calculated using the whole dataset at each update step, and is computationally intensive. In stochastic gradient descent, the gradient is calculated using a single sample, and is computationally less intensive. While stochastic gradient descent is not guaranteed to converge to the minimum cost function, it is faster than batch gradient descent and may be a good strategy to use with large datasets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class logisticRegression():\n",
    "    def __init__(self, gradient=\"batch\", rate=0.001, stop=2):\n",
    "        if gradient != \"batch\" and gradient != \"stochastic\":\n",
    "            raise ValueError(\"Enter either 'batch' or 'stochastic' for gradient\")\n",
    "        self.gradient = gradient\n",
    "        self.rate = rate\n",
    "        self.stop = stop\n",
    "    def train(self, Xtrain, Ytrain):\n",
    "        def batch(self, Xtrain, Ytrain):\n",
    "            self.errors = []\n",
    "            i = 0\n",
    "            self.iterations = [i]\n",
    "            errBefore = float('inf')\n",
    "            self.w = np.zeros((Xtrain.shape[1], 1))\n",
    "            q = np.dot(np.diag(Ytrain[:, 0]), Xtrain)\n",
    "            errAfter = loss(q, self.w)\n",
    "            self.errors.append(errAfter)\n",
    "            while abs(errBefore - errAfter) > 2:\n",
    "                errBefore = errAfter\n",
    "                g = gradient(Xtrain, Ytrain, self.w, q)\n",
    "                if callable(self.rate):\n",
    "                    step = self.rate(i)\n",
    "                else:\n",
    "                    step = self.rate\n",
    "                self.w = self.w - step * g \n",
    "                errAfter = loss(q, self.w)\n",
    "                self.errors.append(errAfter)\n",
    "                i += 1\n",
    "                self.iterations.append(i)\n",
    "        def stochastic(self, Xtrain, Ytrain, repeat=3):\n",
    "            k = 0\n",
    "            self.errors = []\n",
    "            self.iterations = [k]\n",
    "            self.w = np.zeros((Xtrain.shape[1], 1))\n",
    "            errBefore = float('inf')\n",
    "            q = np.dot(np.diag(Ytrain[:, 0]), Xtrain)\n",
    "            errAfter = loss(q, self.w)\n",
    "            self.errors.append(errAfter)\n",
    "            for r in range(repeat):\n",
    "                for i in range(len(Xtrain)):\n",
    "                    errBefore = errAfter\n",
    "                    x = Xtrain[[i], :]\n",
    "                    y = Ytrain[[i]]\n",
    "                    g = gradient(x, y, self.w)\n",
    "                    if callable(self.rate):\n",
    "                        step = self.rate(i)\n",
    "                    else:\n",
    "                        step = self.rate\n",
    "                    self.w = self.w - step * g\n",
    "                    errAfter = loss(q, self.w)\n",
    "                    self.errors.append(errAfter)\n",
    "                    k += 1\n",
    "                    self.iterations.append(k)\n",
    "        def gradient(x, y, w, q=None):\n",
    "            if q is None:\n",
    "                q = np.dot(y, x)\n",
    "            g = np.dot(-np.transpose(q), (1 / (np.exp(np.dot(q, w)) + 1)))\n",
    "            return g \n",
    "        def loss(q, w):\n",
    "            return np.sum(np.log(1 + np.exp(-np.dot(q, w))))\n",
    "        if self.gradient == \"batch\":\n",
    "            batch(self, Xtrain, Ytrain)\n",
    "        else:\n",
    "            stochastic(self, Xtrain, Ytrain)\n",
    "    def plot(self):\n",
    "        plt.plot(self.iterations, self.errors)\n",
    "        plt.xlabel(\"Iterations\")\n",
    "        plt.ylabel(\"Cross Entropy\")\n",
    "        plt.title(\"Cross Entropy vs Iterations, gradient = %s\" % (self.gradient))\n",
    "        plt.show()   \n",
    "    def predict(self, x):\n",
    "        z = np.dot(x, self.w)\n",
    "        p = 1 / (np.exp(-z) + 1)\n",
    "        return p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us first perform batch gradient descent with a learning rate of $\\alpha = 0.001$ and monitor the cross entropy with the number of iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEZCAYAAAC5AHPcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcXGWd7/HPl6UJCQmBQBJJIIQlkqCAoFHRkVbvgCgi\nXodlxhlW0RFGGPUiRMebMK+5I+DVwQ2uOgiEYRF0EByRTWiUMQgiECQsEWRJIM2ShYRA1t/943mK\nPim6uqurq7q6q7/v1+u86tRztuecqjq/epZzjiICMzOzvtqs2RkwM7OhyQHEzMxq4gBiZmY1cQAx\nM7OaOICYmVlNHEDMzKwmDiBmTSDpQklfaXY+GknSRkm75fFBv7+Sbpd04gBs5zhJv2n0dgaCA0gD\nSPobSfdIWilpsaRfSHpPE/NzsaQ1kl7Ow0pJ91W57GxJcxudx4GU9+mywvvXT3QN2t4bThgR8dmI\n+D+N2uYg8fpFZvXaX0kHSXqmv+upt/wb++c+LNISF+A5gNSZpC8A3wT+BRgP7AJ8D/hohfk3H6Cs\nnRsRY/IwOiLeVq8VS1K91jWAosJ4n1T5+ak/2xgMavyMG/G9GPLHsqVEhIc6DcAYYCXwP3uYZzZw\nDXAZsBw4EWgDzgcWA4uAfwO2zPOPA34OLANeAu4orOvMPP/LwMPA+yts82LgnytMmwJsBI4FngKe\nB76cpx0CrMnDSuC+nH47KUDeCbwC7Aa8Cbgu5/Ex4FPd7PNVOa+/B96ap/0v4Cdlefo28G/d5PVL\nwDVlad8Czs/jxwOP5208Dvx1D5/B3Dx+R97/VXm5I3P6YcB9+bjfWcpvnvbnnJcHgFdJf8TOBP6U\n1/FH4Ig87155nnX5GC7t7jMBTgYWAi8CPwPeVJi2EfhMPq5Lge8Wpu0OdOTv0vPAlVV+VzcDvgG8\nkI/VqXk7m/XwGR8PLMj7+Cfg02XrPAN4lvSdPAHYAOxWYX97O75fzMd3Wf7etAEjgdXA+nwsXwYm\n1vH3ezvwr8DvgBXAtcDYwvSrgedynjqA6YXPbi3wWs7TdTl9MvDT/Lm8AHw7px8H/Ab4ev48Hwc+\n1OzzV03HrNkZaKWBdMJdW/oRVphnNumE/NH8fgTwz8BvScFiHPDfwNl5+r8CF+Qf/ObAe3L6NOBp\nYEJ+vwswtcI2qwkg388/0n3yD+HNhfzOLVvmduBJ0slxM2AL0on4O8CWwL75R9Nets8fz/vwReCJ\nPD4xnwzG5Hk3BzqB/brJ6y6kE/2o/H4z0gnrHfnksgLYI0+bUPqBV/gM5hbebyweO+BtOQ9vJ/3j\n/TvSSa0U1P8M/AHYCdgqp32i8FkcmfNZen8c8OtKnwnwgXyC2Tcfv2+z6R+FjcD1wGhg53xsD87T\nrgBm5fE24MAqv6t/Twp0bwK2BW4hnfCLAaT8Mz4U2DVP/wtSYNkvv/8Q6eQ6HdgauJwKAaTK43tX\n/gzHkoLWp/O0g4Cne9m3M0kn+aX5tTi+tIflbgeeKezDT4DLCtOPz9+zLUm1DPdV+o3lY3Y/8H9J\nv/HXP5v8fVhL+vOo/Fksbvb5q5bBVVj1NQ54MSI29jLfvIj4OUBEvAb8DSlgvBQRLwFnk35UkP65\nvol0gtsQEf+d0zeQvpRvkbRFRDwdEX/uYZtnSFoqaVl+vbgwLYA5EbE2IuaT/vnt28s+XBIRj+R9\nnQgcCJwZEesi4gHg30mlmpJ7I+LaiNhA+vGNAN4VEUuAX5NOupBOUi9ExP3lG4yIp0kn7o/npA8C\nr0TEPYVj8lZJIyKiMyIe7mUfiorVLScD/y8ifh/JZaQA+K7CPN+KiGcjYk3O208jojOPX0MqTcys\nctt/A1wUEQ9ExDpgFvBuSbsU5vlaRKyMiGdIJ7r9cvo6YIqkSfnz+22V2zwy78NzEbECOKebeV7/\njCNifUT8MiKezPv4G+BmUiApre/iiHg4Il4F5vSw7WqPb2dELCeVwPfrbkXdiYhzI2K7iNg+vxbH\nt+9l8csK+/BV4MhS9V1EXBIRq/Nn9M/AvpJGV1jPTNLv9ksR8Vo3n82TEfGjSBHlUmCipPHV7uNg\n4QBSXy8BO0jq7biWNwLuRCpNlDyV0yAVcx8Hbpb0J0lnAkTE48A/kn6onZKukPSmHrb59eKPKCJO\nKJveWRhfDWzTh33YifTPbnXZPkzqbv78o1lU2Me5wN/m8U+SqvcquRL46zz+16R/4ORtHw18FnhO\n0s8lvbmXfahkCvDFHGiXSlpGqo7YqTDPouICko6VdF8O0MuAvYEdqtzeTqTjRd6XV0jfpeLxq/T5\nnEH6Hd8t6UFJ5Z9rT9ssfobdNUxvkibpUEnzJL2U9/FQuvaxfH1PUbkNpJrj29fvY72U70Mb+Tct\n6Zz8G1xOKiUFlT/jnYGnevgzuaQ0koOVGLh9rBsHkPqaR/ondUQv85U3Ai4m/ahKppCqZoiIVRHx\nvyJid+Bw4AuS3p+nXRURf1FYtrt/kf1VqcGymP4ssL2kUYW0XUj7VbJzaST/o5ucl4NU57+PpL1J\ndeOX95Cfa4B2SZNIJZErXs9QxC0RcTCpRPQo8MMe1tOTZ4D/kwNtKehuExE/Lszz+v7nksIPgFNK\n/3iBh+g6gfbW6Psshc8/H8dxlAWp7kTE8xHx6YiYRKoKuaDKHmXPkT6Dkl26mae4j22kKp3zgB3z\nPv6Srn18jsJnnPen0n5Xc3wr6e1YImlW7mn4ctmwUtLLvSxevg9rSe1SnyR1hPlARIwFdiXte6XP\n+Blglyr+TA5pLb1zAy0iXibVr39P0sckbS1pi/zPraeT+1XAP0naQdIOpKLzZQCSPiJp9zzfSlID\n4kZJ0yS9P/+w15IaanurOqukp94yncCuPfXCiYhFpDacr0naStI+wElsWpI4QNIRudfS50ntLHfl\n5deQGhuvAH6X11dpWy+S2lsuBp6IiEcBJI2XdLikkaRqnVWkKq1qLCE1Epf8EPh7STPzukdJ+nBZ\ngCwaRTr2L+Z/qicAbylM7wQmS9qywvJXAidI2kfSVqR2r7tydVWPJP1VDqaQGtI35qF0XcP/rrDo\n1cDpknaSNJbUKaAnbXl4MSI2SjoUOLhsfcdLmp4/g0rbhb4f36JOYJykMZVmiIivReppOKZsGB0R\nFZfL/lbSXnkfziZ12ghS6WANsCzn82tsGjQ62fQ7dDcpqJ4jaWT+XRxYxf4NKQ4gdRYR3wS+APwT\nqbHzaeAU0r/sSv6F1DOp1P7we6DUZ35P4FZJK0mN69+LiDuArUgljhdI/2B3JNWdV/Klsn9izxez\nXb4bhfFrSAHmJUm/rzA/pOqkqTkvPwW+GhG3F6ZfR6piWkb6N/fx3B5ScinwVlJ1Vm+uILV/FEsq\nm5GO+2LSP8b3kaqzqjEHmJurU/4qIu4l1dN/V9JSUu+n4wrzb7L/ua3lG6SAuIRUfXVnYZbbSCWS\nJWXHvbT8r0h/Gv4z538qcEyl7ZV5B/C7/M/6Z8BppXYK0r/pOyss90NSG8Z84F7gF8D6QpVL+T6u\nAk4DrsnH5BjSZ1qafiOpJ+FtpOP1q0oZ7uvxLVv2UVLAfSJ/XhMrzVuDIP3puZT0PW4DTs/T5pJ+\ny4tJnQ/K25ouAvbOefrPfBw/Svr9Pk0qkRzVy7aHHKXg2qCVS5NJB34C6V/RDyLiO5LOIx3cNaT6\n/RPyv3ckzSL1TlgPnB4RN+f0/YFLSI2vN0TEPzYs41ZXkmYDu0fEsT3MszOpK/LEfLKyfsilkh9H\nxHurnP9DwIURMbWxObNW0ugSyHrgCxGxN/Bu4B8k7UX657N3ROxH6q0yC0DSDFKUnk5qoLugUHVy\nIXBSREwDpkk6pMF5twGS64m/CFzl4FEfEbG4p+AhaUSuWt08B5vZpBKQWdUaGkAiYknk7pj5xPAw\nMCkibi0Ule+iqzHvcNJJZH0uhi8EZuZi6ujo6q45l94bqm0IyHXNK0jXQsxucnaGE5Hq+JeSqrAe\nwsff+miLgdqQpF1Jfbl/VzbpRFKdJqRui/MK0xbntPVs2iNlEZt2cbRBLCLO7mHaatIFcjaActfR\naq9TMevWgDSiS9qG1AXw9GIVhdLdOddFxJUVFzYzs0Gp4SUQSVvQdUuA6wrpxwMfJlVdlCxm037Y\nk3NapfTutjckezOYmTVbRPTpBpgDUQL5EbAgIr5VSsg9Ps4ADs/XAJRcDxwjqU3SVGAP4O5It7tY\nIWlmblQ/lkIXwnIxCO4RMxiG2bNnNz0Pg2XwsfCx8LHoeahFQ0sgSs/A+CTwoNLzJwL4CulmcW3A\nLbmT1V0RcUpELJB0NenmaetIV/aW9uxUNu3Ge2Mj825mZj1raACJdOO/7p6XsGcPy3yNdJVnefq9\npAvNzMxsEPCV6C2svb292VkYNHwsuvhYdPGx6J+GXoneDJKi1fbJzKzRJBGDsBHdzMxakAOImZnV\nxAHEzMxq4gBiZmY1cQAxM7OaOICYmVlNHEDMzKwmDiBmZlYTBxAzM6tJSwaQjRt7n8fMzPqnJQPI\n8uXNzoGZWetryQDywgvNzoGZWetryQDy/PPNzoGZWetzADEzs5q0ZABxFZaZWeO1ZABxCcTMrPEc\nQMzMrCYNDSCSJku6TdJDkh6UdFpO307SzZIelXSTpG0Ly8yStFDSw5IOLqTvL2m+pMcknd/Tdl2F\nZWbWeI0ugawHvhARewPvBk6VtBdwFnBrRLwZuA2YBSBpBnAUMB04FLhAUukRixcCJ0XENGCapEMq\nbdQlEDOzxmtoAImIJRFxfx5fBTwMTAY+BlyaZ7sUOCKPHw5cFRHrI+JJYCEwU9JEYHRE3JPnm1tY\n5g0cQMzMGm/A2kAk7QrsB9wFTIiITkhBBhifZ5sEPFNYbHFOmwQsKqQvymndcgAxM2u8AQkgkrYB\nfgKcnksiUTZL+ft+Wb4cNmyo5xrNzKzcFo3egKQtSMHjsoi4Lid3SpoQEZ25eqpUZlgM7FxYfHJO\nq5TerS23nMNZZ8GoUdDe3k57e3ud9sbMrDV0dHTQ0dHRr3Uooq5//t+4AWku8GJEfKGQdi6wNCLO\nlXQmsF1EnJUb0S8H3kmqoroF2DMiQtJdwGnAPcAvgG9HxI3dbC9mzAh+/GN4y1saumtmZi1DEhGh\n3ufs0tASiKT3AJ8EHpR0H6mq6svAucDVkk4EniL1vCIiFki6GlgArANOia4IdypwCTACuKG74FGy\n447uymtm1mgNL4EMNElx5JHBJz4BRx/d7NyYmQ0NtZRAWvJK9PHj3RPLzKzRWjKA7LijA4iZWaO1\nZAAZP95tIGZmjdayAcQlEDOzxmrJAOIqLDOzxmvJAOIqLDOzxmvZAOISiJlZY7VkABk7FlatgrVr\nm50TM7PW1ZIBZLPNYIcd4MUXm50TM7PW1ZIBBFyNZWbWaA4gZmZWk5YNIO7Ka2bWWC0bQNyV18ys\nsVo6gLgEYmbWOA4gZmZWk5YNIH6olJlZY7VsAHEJxMyssRxAzMysJi0bQNyN18yssRoaQCRdJKlT\n0vxC2r6S5km6T9Ldkt5emDZL0kJJD0s6uJC+v6T5kh6TdH412x49Gtavh9Wr67tPZmaWNLoEcjFw\nSFnaecDsiHgbMBv4OoCkGcBRwHTgUOACSaUHvF8InBQR04BpksrX+QaSrwUxM2ukhgaQiLgTWFaW\nvBHYNo+PBRbn8cOBqyJifUQ8CSwEZkqaCIyOiHvyfHOBI6rZvquxzMwaZ4smbPPzwE2SvgEIODCn\nTwLmFeZbnNPWA4sK6Ytyeq9cAjEza5xmBJDPAqdHxM8k/RXwI+Av67mBOXPmAPDss/DrX7fz4Q+3\n13P1ZmZDXkdHBx0dHf1ahyKiPrmptAFpCvDziNgnv18eEWML05dHxFhJZwEREefm9BtJbSRPAbdH\nxPScfgxwUER8tsL2orRPZ5yRqrG+9KUG7qCZWQuQRESo9zm7DEQ3XuWhZLGkgwAkfZDU1gFwPXCM\npDZJU4E9gLsjYgmwQtLM3Kh+LHBdNRv21ehmZo3T0CosSVcA7cA4SU+TShQnA9+WtDnwGvBpgIhY\nIOlqYAGwDjgluopHpwKXACOAGyLixmq2P348PPRQ/fbHzMy6NLwKa6AVq7BuuAG+8x345S+bnCkz\ns0FusFZhNY278ZqZNU5LBxB34zUza5yWDiClEkiL1dKZmQ0KLR1ARo6ELbeElSubnRMzs9bT0gEE\n3JXXzKxRWj6A+LkgZmaN4QBiZmY1afkA4iosM7PGaPkA4hKImVljOICYmVlNHEDMzKwmLR9A3AZi\nZtYYLR9AXAIxM2sMBxAzM6tJS9/OHWDtWhg1Ctasgc1aPlyamdXGt3PvRlsbbLMNLF/e7JyYmbWW\nlg8g4GosM7NG6DWA5EfPDml+sJSZWf1VUwJZKOnrkmY0PDcN4gdLmZnVXzUBZF/gMeDfJd0l6dOS\nxlSzckkXSeqUNL8s/XOSHpb0oKRzCumzJC3M0w4upO8vab6kxySdX+W+vc5VWGZm9ddrAImIlRHx\nw4g4EDgTmA08J+lSSXv0svjFwCHFBEntwEeBt0bEW4H/m9OnA0cB04FDgQsklXoEXAicFBHTgGmS\nNllnbxxAzMzqr6o2EEmHS7oWOB/4BrAb8HPghp6WjYg7gWVlyZ8FzomI9XmeF3P6x4CrImJ9RDwJ\nLARmSpoIjI6Ie/J8c4Ejqtm5El+NbmZWf1W1gZBO7l+PiLdFxDcjojMifgLcWMM2pwHvy9Vht0s6\nIKdPAp4pzLc4p00CFhXSF+W0qrkEYmZWf1tUMc8+EbGquwkRcVqN29wuIt4l6R3ANaQSTd3MmTPn\n9fH29nbGj293ADEzK+jo6KCjo6Nf66gmgIyXdCXwbmAjMA/4fEQ8UeM2nwH+EyAi7pG0QdI4Uolj\nl8J8k3PaYmDnbtIrKgYQgD/+0VVYZmZF7e3ttLe3v/7+7LPP7vM6qqnCugK4GpgI7EQqMVzZh20o\nDyU/Az4AIGka0BYRLwHXA0dLapM0FdgDuDsilgArJM3MjerHAtf1YfuuwjIza4BqAsjIiLgsN26v\nj4j/AEZUs3JJVwC/JfWcelrSCcCPgN0kPUgKTscCRMQCUqBaQGqcP6VwU6tTgYtI3YkXRkSf2l7G\njUu3Mlm/vi9LmZlZT3q9maKkc0k9qa4CAjga2A74OkBELG1wHvuk/GaKJePHw4MPwoQJTciUmdkg\nV8vNFKsJIH/uYXJERF0bwPurUgDZe2/48Y/hLW9pQqbMzAa5WgJIr43oETG19iwNHm4HMTOrr14D\niKQtSRf/vS8ndQDfj4h1DcxX3TmAmJnVVzXdeC8EtgQuyO//Lqd9qlGZagRfjW5mVl/VBJB3RMS+\nhfe3SXqgURlqFJdAzMzqq5puvBsk7V56I2k3YEPjstQYDiBmZvVVTQnkDOB2SU+QLgicApzQ0Fw1\ngAOImVl99RhAJG0GvArsCbw5Jz8aEWsanbF6cxuImVl99RhAImKjpO9FxNuA+T3NO9i5BGJmVl/V\ntIH8StInCg93GpIcQMzM6quaK9FXAqOA9cBrpHaQiIiqHms70CpdiR4BW20Fq1ZBW1sTMmZmNog1\n6kr00bVnafCQYIcdUjvIpD49jsrMzLpTzSNtf1VN2lDgaiwzs/qpWAKRNAIYCewgaTu6nukxhj4+\nUnawcE8sM7P66akK6zPAP5IeInUvXQHkZeC7Dc5XQ7gEYmZWPxUDSER8C/iWpM9FxHcGME8N4wBi\nZlY/1TSif0fSgcCuxfkjYm4D89UQDiBmZvVTze3cLwN2B+6n6x5YAQy5ALLjjvCnPzU7F2ZmraGa\ne2G9HZjR7cUVQ4xLIGZm9VPNleh/BCbWsnJJF0nqlPSG26BI+qKkjZK2L6TNkrRQ0sOSDi6k7y9p\nvqTHJJ1fS17AAcTMrJ6qCSA7AAsk3STp+tJQ5fovBg4pT5Q0GfhL4KlC2nTgKGA6cChwQeH2KRcC\nJ0XENGCapDessxruxmtmVj/VVGHNqXXlEXGnpCndTPo30m3ii4HoY8BVEbEeeFLSQmCmpKeA0RFx\nT55vLnAEcFNf8+MSiJlZ/fR0IeFeEfFIRNwhaaviLdwlvavWDUo6HHgmIh4suz/jJGBe4f3inLYe\nWFRIX0SNFzJusw1s2ACrV8PIkbWswczMSnoqgVwB7J/H5xXGIT0fff83LNELSVsDXyZVXzXMnDlz\nXh9vb2+nvb09bz+VQl54AaZ0Vy4yMxsmOjo66Ojo6Nc6egogqjDe3ftq7U66nuSB3L4xGfiDpJmk\nEscuhXkn57TFwM7dpFdUDCDldtwxVWM5gJjZcFb8cw1w9tln93kdPTWiR4Xx7t73RHkgIv4YERMj\nYreImEqqjnpbRDxPag85WlKbpKnAHsDdEbEEWCFpZg46xwLX9WH7m3A7iJlZffRUApks6dukk39p\nnPy+qjYISVcA7cA4SU8DsyPi4sIsQVdwWSDpamABsA44pXDtyanAJcAI4IaIuLGa7XfHAcTMrD4q\nPlBK0nE9LRgRlzYkR/1U6YFSJWeckaqxvvSlAcyUmdkgV9cHSg3WANFf48dDZ2ezc2FmNvRVcyFh\nS3EVlplZfQy7AOKr0c3M6mPYBRCXQMzM6qOaZ6KfJ2mMpC0l/UrSC5L+diAy1wgOIGZm9VFNCeTg\niHgZOAx4knR9xhmNzFQjlS4kHPo3pzcza65qAkipp9ZHgGsiYkUD89NwW28NbW2wcmWzc2JmNrRV\nE0D+S9IjwAHAryTtCLzW2Gw1lquxzMz6r9cAEhFnAQcCb4+IdcArpFuvD1kOIGZm/VdNI/qRwLqI\n2CDpn4D/AHZqeM4ayF15zcz6r5oqrK9GxEpJ7wX+B3AR6QmBQ5ZLIGZm/VdNANmQXz8C/CAifgG0\nNS5LjecAYmbWf9UEkMWSvg8cDdwgaasqlxu0Sg+VMjOz2lUTCI4iPX/8kIhYDmzPEL4OBLquBTEz\ns9pV0wtrNfA4cIikfwDGR8TNDc9ZA7kKy8ys/6rphXU6cDkwPg//Ielzjc5YIzmAmJn1X8UHSr0+\ngzQfeHdEvJLfjwLmRcQ+A5C/PuvtgVIAzz4LBxwAzz03QJkyMxvkanmgVDVtIKKrJxZ5vE8bGWx2\n2AFefBE2bmx2TszMhq6enolecjHwO0nX5vdHkK4FGbLa2mD0aFi2DMaNa3ZuzMyGpmoa0b8JnAAs\nzcMJEXF+NSuXdJGkzlwNVko7T9LDku6X9FNJYwrTZklamKcfXEjfX9J8SY9JqmrbvXFXXjOz/ukx\ngEjaXNIjEfGHiPh2Hu7rw/ovBg4pS7sZ2Dsi9gMWArPytmaQugxPBw4FLpBUqiq7EDgpIqYB0ySV\nr7PP3JXXzKx/egwgEbEBeFTSLrWsPCLuBJaVpd0aEaXWh7uAyXn8cOCqiFgfEU+SgstMSROB0RFx\nT55vLqkarV/cE8vMrH+qaQPZDnhI0t2kO/ECEBGH12H7JwJX5vFJwLzCtMU5bT2wqJC+KKf3iwOI\nmVn/VBNAvtqIDUv6Cukuv1f2OnMfzZkz5/Xx9vZ22tvb3zCP78hrZsNZR0cHHR0d/VpHxQAiaQ9g\nQkTcUZb+XqBfV1BIOh74MPCBQvJiYOfC+8k5rVJ6RcUAUsn48fDww1Vl18ys5ZT/uT777LP7vI6e\n2kDOB17uJn1FnlYtUbhuRNKHSPfSOjwi1hTmux44RlKbpKmkZ6/fHRFLgBWSZuZG9WOB6/qw/W65\nCsvMrH96qsKaEBEPlidGxIOSdq1m5ZKuANqBcZKeBmYDXybdDv6W3Mnqrog4JSIWSLoaWACsA04p\nXFJ+KnAJMAK4ISJurGb7PXEVlplZ/1S8lYmkhRGxZ4Vpf4qIPRqasxpVcysTgIcegiOPhAULBiBT\nZmaDXL1vZfJ7SSd3s5FPAff2NXODjauwzMz6p6cSyATgWmAtXQHj7aTqp4/ntolBp9oSyIYNMGIE\nvPoqbFFNXzQzsxZWSwmkmrvxvh94S377UETcVmP+BkS1AQRSKeTBB2HChAZnysxskKslgPT63zsi\nbgdurzlXg1ipGssBxMys74b0s837y+0gZma1G9YBxF15zcxqN6wDiEsgZma1cwBxADEzq8mwDyCu\nwjIzq82wDiB+qJSZWe2GdQBxFZaZWe2GfQBxFZaZWW2GdQBxFZaZWe2GdQAZOxZWr4Y1a3qf18zM\nNjWsA4jkiwnNzGo1rAMIOICYmdVq2AcQ98QyM6uNA4gDiJlZTRxA3JXXzKwmDQ0gki6S1ClpfiFt\nO0k3S3pU0k2Sti1MmyVpoaSHJR1cSN9f0nxJj0k6v5553HFH6Oys5xrNzIaHRpdALgYOKUs7C7g1\nIt4M3AbMApA0AzgKmA4cClwgqfR0rAuBkyJiGjBNUvk6a3bggXDZZbBwYb3WaGY2PDQ0gETEncCy\nsuSPAZfm8UuBI/L44cBVEbE+Ip4EFgIzJU0ERkfEPXm+uYVl+u2gg+Dss+GQQ+DZZ+u1VjOz1tfr\nI20bYHxEdAJExBJJ43P6JGBeYb7FOW09sKiQviin183JJ8NLL6UgcscdsP329Vy7mVlrakYAKRf1\nXuGcOXNeH29vb6e9vb3XZc48E158EQ47DG65BUaNqneuzMwGj46ODjo6Ovq1DkXU/fy96QakKcDP\nI2Kf/P5hoD0iOnP11O0RMV3SWUBExLl5vhuB2cBTpXly+jHAQRHx2Qrbi1r3KQJOPBGWLIHrroO2\ntppWY2Y25EgiItT7nF0Gohuv8lByPXB8Hj8OuK6QfoykNklTgT2AuyNiCbBC0szcqH5sYZn6ZlTw\nwx+mwHHccbBxYyO2YmbWGhpaApF0BdAOjAM6SSWKnwHXADuTShdHRcTyPP8s4CRgHXB6RNyc0w8A\nLgFGADdExOk9bLPmEkjJq6/CoYfC3nvDd7+bAouZWSurpQTS8CqsgVaPAAKwYgW8//3w0Y+mXlpm\nZq2slgAyGBrRB6Vtt4Ubb4T3vhfGjYPTTmt2jszMBhcHkB6MH596ZJWCyCc/2ewcmZkNHg4gvZgy\nBW66CT7wgfQAqo98pNk5MjMbHIb9zRSrMWNG6tZ7/PFw553Nzo2Z2eDgAFKld74TLr8cPvEJmD+/\n9/nNzFruc6chAAAL80lEQVSdA0gfHHwwfOc7qYvv4483OzdmZs3lANJHRx0FX/0qfPCDqVqrxXpB\nm5lVzdeB1OiGG+Css9I9s845J93V18xsqPKFhAxcAAHYsAGuuiqVSKZNg3/9V9h//wHZtJlZXQ3W\ne2G1rM03T9eGPPJIumL9sMPgmGP8cCozGx4cQOqgrQ1OPTUFjn32gXe/Gz7zGVi8uNk5MzNrHAeQ\nOho1Cr78ZXjssXTR4T77pOeMLF3a7JyZmdWfA0gDbL89nHtuul5k+fKu9pFXXml2zszM6scBpIEm\nTYLvfx9++9sUTPbcE77yFZg3LzXAm5kNZe6FNYAeeCD12vrFL+C559IFiYcdli5QHDu22bkzs+HM\n3XgZ3AGk6KmnUiD5xS/gN7+BAw5IweSww1KVlx9iZWYDyQGEoRNAil55BW67LQWT//ovGDGiK5i8\n731+NruZNZ4DCEMzgBRFpKquUjBZsCBd5b7//rDffmmYMsUlFDOrryEVQCR9nvT8843Ag8AJwCjg\nx8AU4EnS89JX5PlnAScC6yk8L72b9Q7pAFLuhRfgjjvg/vu7hlWruoJJaZgxwyUVM6vdkAkgknYC\n7gT2ioi1kn4M3ADMAF6KiPMknQlsFxFnSZoBXA68A5gM3Ars2V2kaLUA0p0XXkillGJQefzx1HZS\nCij77pve77QTbOa+dmbWi6EWQOYB+wErgf8Evg18FzgoIjolTQQ6ImIvSWcBERHn5uV/CcyJiN91\ns+6WDyDdefVVeOihroDywAPwpz+l61CmTIHddoOpU9NraZg6FcaMaXbOzWwwqCWANOWRthHxrKRv\nAE8Dq4GbI+JWSRMiojPPs0TS+LzIJFLAKVmc0yzbemt4+9vTULR6NTz5JDzxRNfw6193jW+99aZB\nZcqUVGopDePHp3t+mZmVa0oAkTQW+BiprWMFcI2kTwLlRYfhV5Sos5EjU/vIjBlvnBYBzz8Pf/5z\nCiaPPw733gs//zk8+2y6l9eyZSmIlALKpEmbBpjSsN12riozG26aEkCA/wE8ERFLASRdCxwIdJZK\nIbkK6/k8/2Jg58Lyk3Nat+bMmfP6eHt7O+3t7XXNfKuQYMKENLzrXd3Ps3YtdHamYPLss11DR8em\n71etgnHjUrDZccc3vpanjR3rnmRmzdTR0UFHR0e/1tGsNpCZwEWkRvE1wMXAPcAuwNKIOLdCI/o7\nSVVXtzCMG9EHo7Vr4cUXUwP/889v+tpd2urV6Z5hfR3GjHFJx6wRhkwjOoCk2cAxwDrgPuBTwGjg\nalJp4ylSN97lef5ZpG6/6xhG3Xhb1Zo1qXps6dK+DatWwTbbpBLMttumoTTeU9qYMTB6dHrdZhsH\nIbNyQyqANIoDSGvbsAFefhlWrEg9zHp6LY0vXw4rV6bh5ZdT6WfrrTcNKt29jh6dgs2oUem1fLz4\n3tfg2FDnAIIDiPVu48Z0+5iXX940sJS/rlqV5lu1atOhuzSpK6CMGpU6L4wc2TXe2+vIkSmobb11\n5fG2NrcbWeM4gOAAYs2xdm1XMFm9OgWZnl67S3v11TRUGt+w4Y2BZcSIN752l1acVhy22qr78fL3\nW27p4NXqHEBwALHWtX79GwPLa6+loTRe/lqe9uqrqf3ptde6XnsbLwWvtrYUVEqv1QzlyxRfe5vW\n1pYCV/F9pWHzzR3g+ssBBAcQs0bYuDEFk+Kwdu0b0yrNs3btpuOV0tasgXXruqYVh0rpa9emAFcK\nNsXXSuPFtL4Of/mX6fELrWbIXIluZkPLZpt1VZ8NRhs3dgWYdesqj1eaXs2wZk2qonz11Wbv7eDh\nEoiZmdVUAnFveDMzq4kDiJmZ1cQBxMzMauIAYmZmNXEAMTOzmjiAmJlZTRxAzMysJg4gZmZWEwcQ\nMzOriQOImZnVxAHEzMxq4gBiZmY1cQAxM7OaNC2ASNpW0jWSHpb0kKR3StpO0s2SHpV0k6RtC/PP\nkrQwz39ws/JtZmZJM0sg3wJuiIjpwL7AI8BZwK0R8WbgNmAWgKQZwFHAdOBQ4ALJzx/rTUdHR7Oz\nMGj4WHTxsejiY9E/TQkgksYAfxERFwNExPqIWAF8DLg0z3YpcEQePxy4Ks/3JLAQmDmwuR56/OPo\n4mPRxceii49F/zSrBDIVeFHSxZL+IOkHkkYCEyKiEyAilgDj8/yTgGcKyy/OaWZm1iTNCiBbAPsD\n34uI/YFXSNVX5Y8S9KMFzcwGqaY80lbSBGBeROyW37+XFEB2B9ojolPSROD2iJgu6SwgIuLcPP+N\nwOyI+F0363bQMTOrQV8fadu0Z6JLugM4OSIekzQbGJknLY2IcyWdCWwXEWflRvTLgXeSqq5uAfb0\nw8/NzJpniyZu+zTgcklbAk8AJwCbA1dLOhF4itTziohYIOlqYAGwDjjFwcPMrLmaVgIxM7OhrWWu\nRJf0IUmPSHosV38NW5KelPSApPsk3d3s/AwkSRdJ6pQ0v5BW8QLVVlbhWMyWtCj3fvyDpA81M48D\nRdJkSbfli5YflHRaTh92341ujsXncnqfvxstUQKRtBnwGPBB4FngHuCYiHikqRlrEklPAAdExLJm\n52Wg5Q4Zq4C5EbFPTjsXeCkiziu2rTUznwOhwrGYDayMiG82NXMDLHfKmRgR90vaBriXdN3ZCQyz\n70YPx+Jo+vjdaJUSyExgYUQ8FRHrgKtIB2S4Eq3z2fZJRNwJlAfOSheotrQKxwLS92NYiYglEXF/\nHl8FPAxMZhh+Nyoci9J1dX36brTKSab8QsNFDO8LDQO4RdI9kk5udmYGgfEVLlAdrv5B0v2S/n04\nVNmUk7QrsB9wF5UvXh4WCseidElEn74brRJAbFPvyRdofhg4NVdlWJehX29buwuA3SJiP2AJMNyq\nsrYBfgKcnv99D9uLl7s5Fn3+brRKAFkM7FJ4PzmnDUsR8Vx+fQG4Ft83rDNfvFqq/32+yflpmoh4\nodAF/ofAO5qZn4EkaQvSCfOyiLguJw/L70Z3x6KW70arBJB7gD0kTZHUBhwDXN/kPDWFpJH5nwWS\nRgEHA39sbq4GnNi0Lvd64Pg8fhxwXfkCLWyTY5FPkiX/k+H13fgRsCAivlVIG67fjTcci1q+Gy3R\nCwtSN17SLeI3Ay6KiHOanKWmkDSVVOoI0oWilw+nYyHpCqAdGAd0ArOBnwHXADuTL1CNiOXNyuNA\nqXAs3k+q894IPAl8ptQG0MokvQf4NfAg6bcRwJeBu4GrGUbfjR6Oxd/Qx+9GywQQMzMbWK1ShWVm\nZgPMAcTMzGriAGJmZjVxADEzs5o4gJiZWU0cQMzMrCYOIGYFklbm1ymS/rrO655V9v7Oeq7fbKA5\ngJhtqnRh1FTShVVVk7R5L7N8eZMNRfgeZTakOYCYde9rwHvzg3VOl7SZpPMk/S7frfRkAEkHSfq1\npOuAh3LatflOyA9K+lRO+xqwdV7fZTltZWljkr6e539A0lGFdd8u6RpJD5eWy9POkfTHnJfzBuyo\nmBU085noZoPZWcAXI+JwgBwwlkfEO/P91v5b0s153rcBe0fE0/n9CRGxXNII4B5JP42IWZJOzXdJ\nLom87k8A+0TEWyWNz8vckefZD5hBujvqf0s6EHgEOCIi9srLj2nUQTDriUsgZtU5GDhW0n2kZyds\nD+yZp91dCB4A/yjpftLzJiYX5qvkPcCVABHxPNBB151Q746I5/JdUu8HdgVWAK/mZzZ8HHi1n/tm\nVhMHELPqCPhcRLwtD7tHxK152iuvzyQdBHwAeGd+rsL9wIjCOqrdVsmawvgGYIuI2EC6Rf9PgMOA\nG/u8N2Z14ABitqnSyXslMLqQfhNwSn6OApL2lDSym+W3BZZFxBpJewHvKkxbW1q+bFu/AY7O7Sw7\nAn9Bukts9xlM2x0bETcCXwD2qX73zOrHbSBmmyr1wpoPbMxVVpdExLfy4z//IEmkBw919/zsG4G/\nl/QQ8CgwrzDtB8B8SfdGxN+VthUR10p6F/AA6VbaZ0TE85KmV8jbGOC63MYC8Pnad9esdr6du5mZ\n1cRVWGZmVhMHEDMzq4kDiJmZ1cQBxMzMauIAYmZmNXEAMTOzmjiAmJlZTRxAzMysJv8f7u1KTHej\nmZYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10f8dba58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "logistic1 = logisticRegression(gradient = \"batch\")\n",
    "logistic1.train(XtrainP, Ytrain)\n",
    "logistic1.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the cross entropy decreases with the number of iterations. Let us now evaluate the performance of the $\\beta$ we trained. We can visually inspect the predictions to see if samples that are marked as spam $1$ do indeed have a probability close to $1$, while those marked as ham $-1$ have a probability close to $0$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.  ]\n",
      " [ 0.13]\n",
      " [ 0.  ]\n",
      " [ 1.  ]\n",
      " [ 0.01]\n",
      " [ 0.  ]\n",
      " [ 0.76]]\n",
      "[[-1]\n",
      " [-1]\n",
      " [-1]\n",
      " [ 1]\n",
      " [-1]\n",
      " [-1]\n",
      " [ 1]]\n"
     ]
    }
   ],
   "source": [
    "predictions = logistic1.predict(XvaliP)\n",
    "print(np.round(predictions[1:8], 2))\n",
    "print(Yvali[1:8])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, we can see that samples that are spam are predicted to be spam with high probability. Formally, let us evaluate our model by calculating the area under the ROC curve. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.977175663376\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "fpr , tpr, thresholds = metrics.roc_curve(Yvali, predictions, pos_label=1)\n",
    "auc = metrics.auc(fpr, tpr)\n",
    "print(auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The area under the ROC curve is close to $1$, which means our model performed well and is robust. Now let us run logistic regression again but with a learning rate defined by:\n",
    "\n",
    "$$rate(i) = \\frac{3}{1000 + 5i}$$\n",
    "\n",
    "Where $i$ is the iteration. In some situations it may be preferable to decrease the learning rate with iterations, because models that are closer to the minimum loss function only require a smaller step size. In contrast, if the step size is kept constant, the model may constantly overshoot the model of minimum loss because the step size is too large at the end. From the model we trained above with $\\alpha = 0.001$, we don't seem to have this problem, but let us train another model with decreasing step size anyways to see how the cross entropy behaves with iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEZCAYAAAC5AHPcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecXHW9//HXmxRqgFBCJCEhlEBAWoCAFFlQKSqhSLNc\nqqDAlShcL8T78ybehwWwXBEBpRgIUiSoFEU6ywUEEhAIAkKkhBBJEBJCKEJCPr8/vt8hk2XL7GRn\nz8zu+/l4nMfOfOfMOZ8zszOf+bZzFBGYmZl11gpFB2BmZo3JCcTMzKriBGJmZlVxAjEzs6o4gZiZ\nWVWcQMzMrCpOIGYFkHSBpP8qOo5akrRE0kb5dt0fr6S7JB3bDfs5StI9td5Pd3ACqQFJX5A0TdJC\nSbMl/VHSrgXGM0nSu5LeyMtCSY9U+NwJkibXOsbulI/p8rL7H3zR1Wh/H/rCiIgTI+J7tdpnnfhg\nkllXHa+kPSTNWt7tdLX8GfufTjylR0zAcwLpYpJOBX4CfBcYBAwDzgP2b2P9Pt0U2lkRsXpeBkTE\ndl21YUnqqm11o2jjdqdU+P5pefZRD6p8j2vxf9Hwr2WPEhFeumgBVgcWAge3s84EYApwOfA6cCzQ\nH/gpMBt4CfhfoF9ef23gRmA+8Bpwd9m2Ts/rvwE8BezZxj4nAf/TxmPDgSXAkcBM4BXgW/mxfYB3\n87IQeCSX30VKkPcCbwEbAR8Brs8xPgN8uZVjvjrH+hCwVX7sP4BrW8T0M+B/W4n1P4EpLcrOAX6a\nbx8NPJv38Szw+Xbeg8n59t35+N/Mzzs0l38WeCS/7veW4s2PPZ9jeQx4h/RD7HTg73kbfwUOzOtu\nntdZlF/Dea29J8DxwAzgVeA64CNljy0BvpJf13nAz8se2xhozv9LrwBXVfi/ugLwY+Cf+bU6Oe9n\nhXbe46OBJ/Mx/h04ocU2vwn8g/Q/eQzwPrBRG8fb0et7Wn595+f/m/7AKsDbwOL8Wr4BDO7Cz+9d\nwPeBB4EFwO+BNcsevwZ4OcfUDIwqe+/eA/6VY7o+lw8Ffpvfl38CP8vlRwH3AD/M7+ezwL5Ff39V\n9ZoVHUBPWkhfuO+VPoRtrDOB9IW8f76/EvA/wJ9JyWJt4D7gO/nx7wPn5w98H2DXXD4SeBFYL98f\nBoxoY5+VJJBf5g/p1vmDsFlZvJNbPOcu4AXSl+MKQF/SF/G5QD9gm/yhaWpxzAflYzgNeC7fHpy/\nDFbP6/YB5gLbthLrMNIX/ar5/gqkL6wd85fLAmCT/Nh6pQ94G+/B5LL7S8pfO2C7HMMOpF+8/0b6\nUisl9eeBvwDrAyvmss+VvReH5jhL948C/q+t9wTYK3/BbJNfv5+x7A+FJcANwABgg/za7p0fuxIY\nn2/3B3ap8H/1q6RE9xFgDeA20hd+eQJp+R7vB2yYH9+dlFi2zff3JX25jgJWBq6gjQRS4ev7QH4P\n1yQlrRPyY3sAL3ZwbKeTvuTn5b/lt+e187y7gFllx3AtcHnZ40fn/7N+pFaGR9r6jOXX7FHgR6TP\n+AfvTf5/eI/041H5vZhd9PdXNYubsLrW2sCrEbGkg/Xuj4gbASLiX8AXSAnjtYh4DfgO6UMF6Zfr\nR0hfcO9HxH25/H3SP+VHJfWNiBcj4vl29vlNSfMkzc9/J5U9FsDEiHgvIqaTfvlt08ExXBoRf8vH\nOhjYBTg9IhZFxGPAxaRaTcnDEfH7iHif9OFbCdg5IuYA/0f60oX0JfXPiHi05Q4j4kXSF/dBuegT\nwFsRMa3sNdlK0koRMTcinurgGMqVN7ccD/wiIh6K5HJSAty5bJ1zIuIfEfFuju23ETE3355Cqk2M\nqXDfXwAuiYjHImIRMB74mKRhZev8ICIWRsQs0hfdtrl8ETBc0pD8/v25wn0emo/h5YhYAJzZyjof\nvMcRsTgi/hQRL+RjvAe4lZRIStubFBFPRcQ7wMR29l3p6zs3Il4n1cC3bW1DrYmIsyJiYESslf+W\n316rg6dfXnYM3wYOLTXfRcSlEfF2fo/+B9hG0oA2tjOG9Ln9z4j4VyvvzQsR8atIGeUyYLCkQZUe\nY71wAularwHrSOrodW3ZCbg+qTZRMjOXQarmPgvcKunvkk4HiIhnga+TPqhzJV0p6SPt7POH5R+i\niDimxeNzy26/DazWiWNYn/TL7u0WxzCktfXzh+alsmOcDHwp3/4iqXmvLVcBn8+3P0/6BU7e9+HA\nicDLkm6UtFkHx9CW4cBpOdHOkzSf1Byxftk6L5U/QdKRkh7JCXo+sCWwToX7W5/0epGP5S3S/1L5\n69fW+/NN0ud4qqTHJbV8X9vbZ/l72FrH9DJlkvaTdL+k1/Ix7sfSY2y5vZm03QdSyevb2f/HrtLy\nGPqTP9OSzsyfwddJtaSg7fd4A2BmOz8m55Ru5GQluu8Yu4wTSNe6n/RL6sAO1mvZCTib9KEqGU5q\nmiEi3oyI/4iIjYGxwKmS9syPXR0Ru5c9t7VfkcurrQ7L8vJ/AGtJWrWsbBjpuEo2KN3Iv+iG5udB\navPfWtKWpLbxK9qJZwrQJGkIqSZy5QcBRdwWEXuTakRPAxe1s532zAK+lxNtKemuFhG/KVvng+PP\nNYULgZNKv3iBJ1j6BdpRp+8/KHv/8+u4Ni2SVGsi4pWIOCEihpCaQs6vcETZy6T3oGRYK+uUH2N/\nUpPO2cC6+Rj/xNJjfJmy9zgfT1vHXcnr25aOXkskjc8jDd9osSyU9EYHT295DO+R+qW+SBoIs1dE\nrAlsSDr2tt7jWcCwCn5MNrQefXDdLSLeILWvnyfpAEkrS+qbf7m19+V+NfD/JK0jaR1S1flyAEmf\nkbRxXm8hqQNxiaSRkvbMH+z3SB21HTWdtaW90TJzgQ3bG4UTES+R+nB+IGlFSVsDx7FsTWJ7SQfm\nUUvfIPWzPJCf/y6ps/FK4MG8vbb29Sqpv2US8FxEPA0gaZCksZJWITXrvElq0qrEHFIncclFwFcl\njcnbXlXSp1skyHKrkl77V/Mv1WOAj5Y9PhcYKqlfG8+/CjhG0taSViT1ez2Qm6vaJemQnEwhdaQv\nyUtpXsN/t/HUa4BxktaXtCZpUEB7+ufl1YhYImk/YO8W2zta0qj8HrS1X+j861tuLrC2pNXbWiEi\nfhBppOHqLZYBEdHm87IvSdo8H8N3SIM2glQ7eBeYn+P8Acsmjbks+z80lZRUz5S0Sv5c7FLB8TUU\nJ5AuFhE/AU4F/h+ps/NF4CTSr+y2fJc0MqnU//AQUBozvylwu6SFpM718yLibmBFUo3jn6RfsOuS\n2s7b8p8tfom9Uh52y8Mouz2FlGBek/RQG+tDak4akWP5LfDtiLir7PHrSU1M80m/5g7K/SEllwFb\nkZqzOnIlqf+jvKayAul1n036xfhxUnNWJSYCk3NzyiER8TCpnf7nkuaRRj8dVbb+Msef+1p+TEqI\nc0jNV/eWrXInqUYyp8XrXnr+HaQfDb/L8Y8Ajmhrfy3sCDyYf1lfB5xS6qcg/Zq+t43nXUTqw5gO\nPAz8EVhc1uTS8hjfBE4BpuTX5AjSe1p6/GbSSMI7Sa/XHW0F3NnXt8VznyYl3Ofy+zW4rXWrEKQf\nPZeR/o/7A+PyY5NJn+XZpMEHLfuaLgG2zDH9Lr+O+5M+vy+SaiSHdbDvhqOUXGu0cWko6YVfj/Sr\n6MKIOFfS2aQX911S+/4x+dc7ksaTRicsBsZFxK25fDRwKanz9aaI+HrNArcuJWkCsHFEHNnOOhuQ\nhiIPzl9WthxyreQ3EbFbhevvC1wQESNqG5n1JLWugSwGTo2ILYGPAf8uaXPSL58tI2Jb0miV8QCS\ntiBl6VGkDrrzy5pOLgCOi4iRwEhJ+9Q4dusmuZ34NOBqJ4+uERGz20seklbKTat9crKZQKoBmVWs\npgkkIuZEHo6ZvxieAoZExO1lVeUHWNqZN5b0JbI4V8NnAGNyNXVALB2uOZmOO6qtAeS25gWkuRAT\nCg6nNxGpjX8eqQnrCfz6Wyf17a4dSdqQNJb7wRYPHUtq04Q0bPH+ssdm57LFLDsi5SWWHeJodSwi\nvtPOY2+TJshZN8pDRyudp2LWqm7pRJe0GmkI4LjyJgqls3Muioir2nyymZnVpZrXQCT1ZekpAa4v\nKz8a+DSp6aJkNsuOwx6ay9oqb21/DTmawcysaBHRqRNgdkcN5FfAkxFxTqkgj/j4JjA2zwEouQE4\nQlJ/SSOATYCpkU53sUDSmNypfiRlQwhbijo4R0y1y4QJEwqPoTfG7viLXxx/sUs1aloDUboGxheB\nx5WuPxHAf5FOFtcfuC0PsnogIk6KiCclXUM6edoi0sze0pGdzLLDeG+uZexmZta+miaQSCf+a+16\nCZu285wfkGZ5tix/mDTRzMzM6oBnoteZpqamokOoWiPHDo6/aI6/8dR0JnoRJEVPOyYzs1qTRNRh\nJ7qZmfVATiBmZlaVHplAHmw5193MzLpcj0wg55zT8TpmZrZ8emQn+sCBweOPwxCfLcvMrCLuRM++\n8AW44IKiozAz69l6ZA3k6aeD3XaDmTNh5ZWLjsjMrP65BpKNHAk77ghXXll0JGZmPVePTCAA48al\nzvQeVsEyM6sbPTaBfOpTsHgxNDcXHYmZWc/UYxOIBKec4iG9Zma10iM70UvH9NZbMHw4TJ0KG21U\ncGBmZnXMnegtrLoqHHss/PznRUdiZtbz9OgaCKShvKNHwwsvwIABxcVlZlbPXANpxfDhsOeecNll\nRUdiZtaz9PgaCMA998Bxx8Hf/gYr9PiUaWbWea6BtGG33WC11eBmX0XdzKzL1DSBSBoq6U5JT0h6\nXNIpuXygpFslPS3pFklrlD1nvKQZkp6StHdZ+WhJ0yU9I+mnnYsDvv51D+k1M+tKta6BLAZOjYgt\ngY8BJ0vaHDgDuD0iNgPuBMYDSNoCOAwYBewHnC+pVKW6ADguIkYCIyXt05lADj8cpk+HJ5/sisMy\nM7OaJpCImBMRj+bbbwJPAUOBA4BSt/ZlwIH59ljg6ohYHBEvADOAMZIGAwMiYlpeb3LZcyqy4orw\nla/Az362PEdkZmYl3dYHImlDYFvgAWC9iJgLKckAg/JqQ4BZZU+bncuGAC+Vlb+Uyzrlq1+F3/wG\n5s3r7DPNzKylbkkgklYDrgXG5ZpIy6Ff3TIUbPBg2H9/uPji7tibmVnP1rfWO5DUl5Q8Lo+I63Px\nXEnrRcTc3Dz1Si6fDWxQ9vShuayt8lZNnDjxg9tNTU00NTV9cH/cODj4YDj1VOhb86M3M6tPzc3N\nNC/n2WZrPg9E0mTg1Yg4tazsLGBeRJwl6XRgYESckTvRrwB2IjVR3QZsGhEh6QHgFGAa8EfgZxHx\noYG5rc0DaWm33dKorEMO6aKDNDNrcNXMA6lpApG0K/B/wOOkZqoAvgVMBa4h1SpmAodFxOv5OeOB\n44BFpCavW3P59sClwErATRExro19dphApkxJnen33LO8R2hm1jPUXQIpQiUJZPHidHbe665L58ky\nM+vtPBO9Qn37wskne2Khmdny6JU1EEhDeTfeOJ0fa731uiEwM7M65hpIJ6y1Vpqd/otfFB2JmVlj\n6rU1EEinNfnEJ9K1QlZcsbZxmZnVM9dAOmmLLWCrrdLsdDMz65xenUAgTSw85xzoYRUxM7Oa6/UJ\nZL/9YOFCuO++oiMxM2ssvT6BrLACfO1rHtJrZtZZvboTvWThQthwQ3jkERg2rDZxmZnVM3eiV2nA\nADjySDjvvKIjMTNrHK6BZM89BzvtlIb0rrpq18dlZlbPXANZDhttBLvsAr/+ddGRmJk1BieQMuPG\npbP09rBKmZlZTTiBlNlzT+jTB26/vehIzMzqnxNIGWnpxEIzM2ufO9FbeOcdGD4c7r0XRo7swsDM\nzOqYO9G7wMorw/HHw7nnFh2JmVl9cw2kFbNnp5MsPv88rLFGFwVmZlbHXAPpIkOGwD77wK9+VXQk\nZmb1q6YJRNIlkuZKml5Wto2k+yU9ImmqpB3KHhsvaYakpyTtXVY+WtJ0Sc9I+mktYy4ZNy41Y73/\nfnfszcys8dS6BjIJ2KdF2dnAhIjYDpgA/BBA0hbAYcAoYD/gfEml6tQFwHERMRIYKanlNrvczjvD\nuuvCH/5Q6z2ZmTWmmiaQiLgXmN+ieAlQ6llYE5idb48Fro6IxRHxAjADGCNpMDAgIqbl9SYDB9Yy\n7hIP6TUza1sRfSDfAH4k6UVSbWR8Lh8CzCpbb3YuGwK8VFb+Ui6ruUMOgaefhunTO17XzKy36VvA\nPk8ExkXEdZIOAX4FfKordzBx4sQPbjc1NdHU1FTVdvr3hxNPTKc3ufjironNzKweNDc309zcvFzb\nqPkwXknDgRsjYut8//WIWLPs8dcjYk1JZwAREWfl8ptJfSQzgbsiYlQuPwLYIyJObGN/yz2Mt9w/\n/5kmFM6YAeus02WbNTOrK/U6jFd5KZktaQ8ASZ8g9XUA3AAcIam/pBHAJsDUiJgDLJA0JneqHwlc\n3w1xA6kj/eCD4cILu2uPZmaNoaY1EElXAk3A2sBcUo3iaeBnQB/gX8BJEfFIXn88cBywiNTMdWsu\n3x64FFgJuCkixrWzzy6tgUDqA9lvv3StkH79unTTZmZ1oZoaiGeiV2jPPeGEE+Dzn+/yTZuZFa5e\nm7B6BA/pNTNblhNIhfbfH15+GR57rOhIzMzqgxNIhfr0gSOOgGuuKToSM7P64ATSCYcdlhJID+s2\nMjOrihNIJ4weDUuWwKOPFh2JmVnxnEA6QVpaCzEz6+2cQDrJzVhmZokTSCdtuy2ssAL85S9FR2Jm\nViwnkE5yM5aZWeKZ6FV47DE48EB47rmUUMzMGp1noneTrbdOp3p/6KGiIzEzK44TSBXcjGVm5ias\nqj3+OHz2s+kMvW7GMrNG5yasbvTRj8Iqq8DUqUVHYmZWDCeQKrkZy8x6OzdhLYcnnoB994WZM9Pc\nEDOzRuUmrG625ZawxhrwwANFR2Jm1v06TCCS+nRHII3KzVhm1ltVUgOZIemHkraoeTQN6NBDYcqU\ndJZeM7PepJIEsg3wDHCxpAcknSBp9Uo2LukSSXMlTW9R/jVJT0l6XNKZZeXjJc3Ij+1dVj5a0nRJ\nz0j6aYXH1i1GjYK114Y//7noSMzMuleHCSQiFkbERRGxC3A6MAF4WdJlkjbp4OmTgH3KCyQ1AfsD\nW0XEVsCPcvko4DBgFLAfcL70wQyLC4DjImIkMFLSMtssmpuxzKw3qqgPRNJYSb8Hfgr8GNgIuBG4\nqb3nRsS9wPwWxScCZ0bE4rzOq7n8AODqiFgcES8AM4AxkgYDAyJiWl5vMnBgJQfXXQ49FK69Ft5/\nv+hIzMy6T0V9IKQv9x9GxHYR8ZOImBsR1wI3V7HPkcDHc3PYXZK2z+VDgFll683OZUOAl8rKX8pl\ndWOzzWDQILjvvqIjMTPrPn0rWGfriHiztQci4pQq9zkwInaWtCMwhVSj6TITJ0784HZTUxNNTU1d\nuflWlZqxPv7xmu/KzGy5NTc309zcvFzb6HAioaSNgHOAjwFLgPuBb0TEcxXtQBoO3BgRW+f7NwFn\nRcTd+f4MYGfgeICIODOX30zqb5kJ3BURo3L5EcAeEXFiG/vrtomE5WbMgN13h9mzoY8HPptZg6nV\nRMIrgWuAwcD6pBrDVZ2JKy8l1wF7AUgaCfSPiNeAG4DDJfWXNALYBJgaEXOABZLG5E71I4HrO7H/\nbrHpprD++nDPPUVHYmbWPSpJIKtExOW5c3txRPwaWKmSjUu6EvgzaeTUi5KOAX4FbCTpcVJyOhIg\nIp4kJaonSZ3zJ5VVJU4GLiENJ54REdX0vdScR2OZWW9SSRPWWaSRVFcDARwODAR+CBAR82ocY6cU\n1YQF8OyzsMsuqRmrbyW9S2ZmdaKaJqxKEsjz7TwcEdGlHeDLq8gEArDDDnD22bDXXoWFYGbWadUk\nkA5/J0fEiOpD6n1KzVhOIGbW01VSA+lHmvxXGqDaDPwyIhbVNrTqFF0Def552Gkn+Mc/3IxlZo2j\nVqOwLgC2B87Py/a5zFoxYgRsuCEs5/BqM7O6V8lv5B0jYpuy+3dKeqxWAfUEpWasT36y6EjMzGqn\nkhrI+5I2Lt3JEwt91qd2HHoo/P73sKguG/nMzLpGJTWQbwJ3SXqONCFwOHBMTaNqcMOHwyabwJ13\nwj51dd5gM7Ou024CkbQC8A6wKbBZLn46It6tdWCNrtSM5QRiZj1VJaOwHomI7bopnuVW9Cisklmz\nYNtt4eWXoX//oqMxM2tfrUZh3SHpc2UXd7IKbLABbL453HFH0ZGYmdVGJTWQhcCqwGLgX6R+kIiI\nii5r293qpQYCcM458OijMGlS0ZGYmbWvJqcyaTT1lEBmz4attoI5c9yMZWb1rSZNWJI+1AjTWpl9\n2JAhsOWWcNttRUdiZtb12kwgklaStBawjqSBktbKy4bU2SVl65lP8W5mPVWbTViSxgFfJ11EajZL\nLwr1BnBRRPy8WyLspHpqwoJ0Tqwtt0zNWCuuWHQ0Zmatq9Xp3L8WEecuV2TdqN4SCMAee8B//Afs\nv3/RkZiZta5mneiSdgE2pGziYURM7myA3aEeE8h558EDD8DllxcdiZlZ62pVA7kc2Bh4lKXnwIqI\nOKWqKGusHhPInDkwalSaVLhSRRcDNjPrXjW5oBSwA7BF3X0rN5DBg9Os9FtugQMOKDoaM7OuUclM\n9L8Cg6vZuKRLJM2VNL2Vx06TtCSP9CqVjZc0Q9JTkvYuKx8tabqkZyT9tJpYiubRWGbW01SSQNYB\nnpR0i6QbSkuF258EfOh0gpKGAp8CZpaVjQIOA0YB+wHnl50+5QLguIgYCYyU1HCnKDz4YPjjH+Gd\nd4qOxMysa1TShDWx2o1HxL2Shrfy0P+SThNfnogOAK6OiMXAC5JmAGMkzQQGRMS0vN5k4EDglmrj\nKsJ668H228PNN8NBBxUdjZnZ8mtvIuHmABFxN/BARNxdWoCqT+cuaSwwKyIeb/HQEGBW2f3ZuWwI\n8FJZ+Us06ERGN2OZWU/SXg3kSmB0vn1/2W1I10Yf/aFndEDSysC3SM1XNTNx4sQPbjc1NdHU1FTL\n3VXs4IPh9NPh7bdhlVWKjsbMerPm5maam5uXaxvtzUT/4DogLa8J0plrhOQmrBsjYmtJHwVuB94m\nzWwfSqppjAGOBYiIM/PzbgYmkPpJ7oqIUbn8CGCPiDixjf3V9YCxvfeGE06AQw4pOhIzs6W6+mSK\n0cbt1u63R3khIv4aEYMjYqOIGEFqjtouIl4h9YccLqm/pBHAJsDUiJgDLJA0JneqHwlc34n91xU3\nY5lZT9FeE9ZQST8j1xTybfL9ivogJF0JNAFrS3oRmBAR5VfHCJYmlyclXQM8CSwCTiqrSpwMXAqs\nBNwUETdXsv96dNBBcNpp8NZbsOqqRUdjZla99pqwjmrviRFxWU0iWk713oQFsO++cOyxqTZiZlYP\nfEEpGiOB/OpXcNNNcO21RUdiZpY4gdAYCWTePBgxIl2xcLXVio7GzKxGVyS0rrfWWrDrrvCHPxQd\niZlZ9ZxACuLRWGbW6Cq5JvrZklaX1E/SHZL+KelL3RFcT3bAAXDHHbBwYdGRmJlVp5IayN4R8Qbw\nWeAF0vyMb9YyqN5g4EDYfXe48caiIzEzq04lCaQ0V+QzwJSIWFDDeHoVN2OZWSOrJIH8QdLfgO2B\nOyStC/yrtmH1DmPHwl13wRtvFB2JmVnnVXpN9LWABRHxvqRVgNXzKUbqTiMM4y03dmyqiXzJvUpm\nVqCaDOOVdCiwKCeP/wf8Gli/yhitBTdjmVmj6rAGIml6PpPubsB3gR8C/x0RO3VHgJ3VaDWQN96A\nDTaAmTNhzTWLjsbMeqtaTSR8P//9DHBhRPwR6N/Z4Kx1q68Oe+4JN1R6kWAzszpRSQKZLemXwOHA\nTZJWrPB5ViE3Y5lZI6qkCWsVYF/g8YiYIekjwFYRcWt3BNhZjdaEBWky4dCh8MILaX6ImVl3q0kT\nVkS8DTwL7CPp34FB9Zo8GtWAAfDJT8J11xUdiZlZ5SoZhTUOuAIYlJdfS/parQPrbdyMZWaNpqJR\nWMDHIuKtfH9V4P6I2Lob4uu0RmzCAnjzTRg2DP78Z9h886KjMbPeplajsMTSkVjk253aiXVstdXg\ne9+Do46CxYuLjsbMrGOVJJBJwIOSJkqaCDwAXFLTqHqpr3wl9Yf8+MdFR2Jm1rFKT2UyGtgt370n\nIh6paOPSJaSz+M4tNXlJOhvYH3iX1Dl/TD7bL5LGA8cCi4Fxpc76vP9LgZWAmyLi6+3ssyGbsEpm\nzoQddoDmZthyy6KjMbPeossvaSupD/BERFTVKp9nr78JTC5LIJ8E7oyIJZLOBCIixkvagtRZvyMw\nFLgd2DQiQtKDwL9HxDRJNwHnRMQtbeyzoRMIwEUXwS9/CfffD/36FR2NmfUGXd4HEhHvA09LGlZN\nQBFxLzC/RdntEbEk332AlCwAxgJXR8TiiHgBmAGMkTQYGBAR0/J6k4EDq4mnUXz5y7D22nD22UVH\nYmbWtr4dr8JA4AlJU4G3SoURMbYL9n8scFW+PQS4v+yx2blsMfBSWflLubzHkuDii2H0aNh/f9i6\nLse7mVlvV0kC+XYtdizpv0hn+b2qw5U7aeLEiR/cbmpqoqmpqat3UXMbbJBqIEcdBVOnuinLzLpW\nc3Mzzc3Ny7WNNvtAJG0CrBcR97Uo3w14OSKerWgH0nDgxvJ5I5KOBo4H9oqId3PZGaT+kLPy/ZuB\nCcBM4K6IGJXLjwD2iIgT29hfw/eBlETAZz8LY8bAhAlFR2NmPVlX94H8FGjtWnkL8mMVx0XZvBFJ\n+5KuqT62lDyyG4AjJPWXNIJ07fWp+cJVCySNkSTgSOD6Tuy/YUlw4YVw3nnwl78UHY2Z2bLaSyDr\nRcTjLQtz2YaVbFzSlcCfgZGSXpR0DHAusBpwm6S/SDo/b/dJ4BrgSeAm4KSyqsTJpLknzwAzIuLm\nSvbfEwwZkuaFHH00vPtuh6ubmXWb9pqwZkTEpm089veI2KSmkVWpJzVhlUTAgQfCVlvBd79bdDRm\n1hN1dRPgwHE7AAAQ8UlEQVTWQ5KOb2UnXwYe7mxwVj0pzQu56CKYNq3j9c3MukN7NZD1gN8D77E0\nYexAuhrhQblvou70xBpIyVVXpRrIww/DSisVHY2Z9SRdPhM9b3RP4KP57hMRcWeV8XWLnpxAIuCQ\nQ2DTTeHMM4uOxsx6kpokkEbTkxMIwCuvpImF110HO+9cdDRm1lPU6nTuVkcGDYJzz02jst55p+ho\nzKw3cw2kQR1xRLqO+o9+VHQkZtYTuAmL3pNAXn01NWVNmQK77lp0NGbW6NyE1Yuss06aoX700fD2\n20VHY2a9kWsgDe5LX0rJ5KedObmMmVkLbsKi9yWQefPSDPUrr4Q99ig6GjNrVG7C6oXWWgt+8Qs4\n9lh4882iozGz3sQ1kB7i6KNh1VVTv4iZWWe5CYvem0Befz01ZV16KXziE0VHY2aNxk1Yvdiaa6Zr\nhxx3HLzR2lVczMy6mGsgPcyXvwx9+qSz95qZVcpNWDiBLFiQJhhedBHsvXfR0ZhZo3ATlrHGGnDx\nxakmsmBB0dGYWU/mGkgP9dWvwqJFcMklRUdiZo3ATVg4gZQsXJiass47Dz796aKjMbN6V3dNWJIu\nkTRX0vSysoGSbpX0tKRbJK1R9th4STMkPSVp77Ly0ZKmS3pGkk/aUYEBA1Lt44QTYP78oqMxs56o\n1n0gk4B9WpSdAdweEZsBdwLjASRtARwGjAL2A86XVMqGFwDHRcRIYKSkltu0Vuy1Fxx4IHz960VH\nYmY9UU0TSETcC7T8/XsAcFm+fRlwYL49Frg6IhZHxAvADGCMpMHAgIiYltebXPYc68CZZ8J998EN\nNxQdiZn1NEWMwhoUEXMBImIOMCiXDwFmla03O5cNAV4qK38pl1kFVlsNJk2CE0+E114rOhoz60n6\nFh0A0OU93hMnTvzgdlNTE01NTV29i4ay++5w2GEpiVx1VZpoaGa9W3NzM83Nzcu1jZqPwpI0HLgx\nIrbO958CmiJibm6euisiRkk6A4iIOCuvdzMwAZhZWieXHwHsEREntrE/j8Jqxdtvw/77w3vvwa9/\nDcOHFx2RmdWTuhuFlSkvJTcAR+fbRwHXl5UfIam/pBHAJsDU3My1QNKY3Kl+ZNlzrEKrrAK33pqS\nyA47pJqImdnyqGkNRNKVQBOwNjCXVKO4DpgCbECqXRwWEa/n9ccDxwGLgHERcWsu3x64FFgJuCki\nxrWzT9dAOvDww/DFL8KOO8LPf55mr5tZ7+aJhDiBVOqtt+C00+CWW1KT1q67Fh2RmRXJCQQnkM66\n/nr4ylfS8u1vQ996GFZhZt3OCQQnkGq8/HK6ouEbb8AVV8BGGxUdkZl1t3rtRLc695GPwJ/+BIcf\nDjvtBJMng3OwmXXENRBbxvTp8IUvwEc/ChdcAAMHFh2RmXUH10BsuW29NUybBoMGwbbbwt13Fx2R\nmdUr10CsTTfdlC5MdfTR8J3vQL9+RUdkZrXiGoh1qU9/Gh59NDVr7bILzJhRdERmVk+cQKxdgwbB\njTfCMcekJHLJJe5gN7PETVhWsSefTB3sG28MF14Ia69ddERm1lXchGU1tcUW8OCDMGJE6mC/446i\nIzKzIrkGYlW57bbUrPX5z8N3vwsrrlh0RGa2PFwDsW7zqU+lDva//x123hmeeqroiMysuzmBWNXW\nWQd+9zs46ST4+MfhJz+BefOKjsrMuosTiC0XCY4/Pl13/b77Uv/Ivvum0Vq+hK5Zz+Y+EOtSb76Z\nJiBOmZIuYLXTTnDIIXDQQbDuukVHZ2Zt8dl4cQKpJ2+9lU7SOGUK3HxzuhLioYemZLLeekVHZ2bl\nnEBwAqlXb7+dksi116YaynbbpZrJwQenswGbWbGcQHACaQTvvJOat6ZMgT/+EbbaKtVMDj4Yhgwp\nOjqz3qmhEoikb5Cuf74EeBw4BlgV+A0wHHiBdL30BXn98cCxwGLKrpfeynadQBrIv/6V5pRMmZJO\nmbLllimZfO5zMHRo0dGZ9R4Nk0AkrQ/cC2weEe9J+g1wE7AF8FpEnC3pdGBgRJwhaQvgCmBHYChw\nO7Bpa5nCCaRxvfsu3H57aua64QbYbLPUzHXIITBsWNHRmfVsjTaRsA+wqqS+wMrAbOAA4LL8+GXA\ngfn2WODqiFgcES8AM4Ax3Ruu1dqKK8JnPgOTJqXL7P73f8MTT8Do0Wk01/e/D3feCQsWFB2pmUGx\nTVinAN8D3gZujYh/kzQ/IgaWrTMvItaSdC5wf0RcmcsvBm6KiN+1sl3XQHqYRYvgrrtS5/u0aWkG\n/LBhsOOOaRkzBrbZBlZaqehIzRpXNTWQvrUKpj2S1iTVNoYDC4Apkr4ItPzmdyYw+vWDvfdOC6SE\n8sQTKZlMm5YmLT7zTDrZY3lSGTUK+vQpNnaznqyQBAJ8EnguIuYBSPo9sAswV9J6ETFX0mDglbz+\nbGCDsucPzWWtmjhx4ge3m5qaaGpq6tLgrVj9+qWzAW+7bZoFD2mY8KOPwtSp6SzBP/gBzJmThguP\nGbM0qWy4YZo9b9bbNTc309zcvFzbKKoTfQxwCalT/F1gEjANGAbMi4iz2uhE3wkYAtyGO9GtA/Pn\nw0MPpaQybVr6+957S5NJqbbiSY1mDTQKC0DSBOAIYBHwCPBlYABwDam2MZM0jPf1vP540rDfRXgY\nr1Vp9uylTV9Tp6YEs/rqKZFsu226WNbGG8NGG6ULZrm2Yr1FQyWQWnECsc5YsiSdkn7atHTt9+ef\nh2efTcuSJSmRlBJK+TJ8OPTvX3T0Zl3HCQQnEOs68+fDc88tXZ59dunt2bPTKVjKk0p5ollrLdde\nrLE4geAEYt1j0SKYNav15PLssxDx4aQybBgMGrR08bBjqydOIDiBWH0o1V7KE8usWfDPf8Irr6Sl\nf/9lE0p7y9pre0iy1ZYTCE4g1hgiYOHCpcmko2X+fBg4cNmksu66H04yq6++7OJ+GquUEwhOINYz\nvf9+usJjWwlm7tyUZN54Y+myYEGqtbRMKuXLgAHtP15aVlnFfTo9nRMITiBmJRHpBJXlSaXa5b33\nUhJZeeWlS8v7rS0drVP++IorpqV//7T06+ek1Z2cQHACMauFRYvSbP933ln6t72lo3Vae/zdd1Oi\nKi2LFqUkUp5UypfWytsr69s3ba9v3w/f7uz9lo+Vlj59Ol769q3PxNgw58Iys8bSrx+ssUZausuS\nJSmJlCeV9977cKKppOzdd2Hx4rS891663HLp/uLFaT9t3W/vsdL9RYtSM2Oli1R5sindXmGFpX9r\ncbsaroGYmXWjiJQcK0k0ixcvvb1kydKl/H5nbrf3+DHHuAnLCcTMrAqNdkEpMzNrYE4gZmZWFScQ\nMzOrihOImZlVxQnEzMyq4gRiZmZVcQIxM7OqOIGYmVlVnEDMzKwqhSUQSWtImiLpKUlPSNpJ0kBJ\nt0p6WtItktYoW3+8pBl5/b2LitvMzJIiayDnADdFxChgG+BvwBnA7RGxGXAnMB5A0hbAYcAoYD/g\nfKkez2e5/Jqbm4sOoWqNHDs4/qI5/sZTSAKRtDqwe0RMAoiIxRGxADgAuCyvdhlwYL49Frg6r/cC\nMAMY071Rd49G/ids5NjB8RfN8TeeomogI4BXJU2S9BdJF0paBVgvIuYCRMQcYFBefwgwq+z5s3OZ\nmZkVpKgE0hcYDZwXEaOBt0jNVy1Po+vT6pqZ1alCTucuaT3g/ojYKN/fjZRANgaaImKupMHAXREx\nStIZQETEWXn9m4EJEfFgK9t20jEzq0LDXA9E0t3A8RHxjKQJwCr5oXkRcZak04GBEXFG7kS/AtiJ\n1HR1G7CpL/xhZlacIi9pewpwhaR+wHPAMUAf4BpJxwIzSSOviIgnJV0DPAksAk5y8jAzK1aPuyKh\nmZl1jx4zE13SvpL+JumZ3PzVMCQNlXRnnlD5uKRTio6pGpJWyKPqbig6ls5qbWJr0TFVStI3JP1V\n0nRJV0jqX3RMHZF0iaS5kqaXlbU5kbietBH72fl/51FJv81TFepSa/GXPXaapCWS1qpkWz0igUha\nAfg5sA+wJfB5SZsXG1WnLAZOjYgtgY8BJzdY/CXjSM2MjajlxNanCo6nIpLWB74GjI6IrUnN0kcU\nG1VFJpE+r+VanUhch1qL/VZgy4jYljRPrV5jh9bjR9JQ4FOk7oOK9IgEQppUOCMiZkbEIuBq0qTE\nhhARcyLi0Xz7TdKXV0PNc8n/fJ8GLi46ls5qY2LrGwWH1Rl9gFUl9SUNRvlHwfF0KCLuBea3KG5r\nInFdaS32iLg9Ipbkuw8AQ7s9sAq18doD/C/wzc5sq6ckkJYTDV+iwb6ASyRtCGwLfGiIcp0r/fM1\nYqdaaxNbVy46qEpExD+AHwMvkibYvh4RtxcbVdUGtTGRuNEcC/yp6CA6Q9JYYFZEPN6Z5/WUBNIj\nSFoNuBYYl2siDUHSZ4C5uRalvDSSlhNb3yY1p9Q9SWuSfrkPB9YHVpP0hWKj6jIN92NE0n8BiyLi\nyqJjqVT+sfQtYEJ5cSXP7SkJZDYwrOz+0FzWMHLzw7XA5RFxfdHxdNKuwFhJzwFXAXtKmlxwTJ3x\nEunX10P5/rWkhNIIPgk8FxHzIuJ94HfALgXHVK25eZIxeSLxKwXH0ymSjiY14zZaAt8Y2BB4TNLz\npO/PhyV1WAPsKQlkGrCJpOF5BMoRQKONBPoV8GREnFN0IJ0VEd+KiGH5zAJHAHdGxJFFx1Wp3Gwy\nS9LIXPQJGmcwwIvAzpJWymeo/gQNMgCAD9dWbwCOzrePAur5h9QysUval9SEOzYi3i0sqsp9EH9E\n/DUiBkfERhExgvSDaruI6DCB94gEkn95/TtpJMQTpDP3NsqHCEm7Al8E9pL0SG6H37fouHqZ0sTW\nR0mjsL5fcDwViYippBrTI8BjpC+FCwsNqgKSrgT+DIyU9KKkY4AzgU9JepqUCM8sMsa2tBH7ucBq\nwG3583t+oUG2o434ywUVNmF5IqGZmVWlR9RAzMys+zmBmJlZVZxAzMysKk4gZmZWFScQMzOrihOI\nmZlVxQnErIykhfnvcEmf7+Jtj29x/96u3L5Zd3MCMVtWaWLUCDp5SgpJfTpY5VvL7Chit85s36ze\nOIGYte4HwG55VvG4fLGssyU9mC8adDyApD0k/Z+k60lnQUDS7yVNyxcH+3Iu+wGwct7e5blsYWln\nkn6Y139M0mFl276r7EJXl5etf2a+iNSjks7utlfFrEyR10Q3q2dnAKdFxFiAnDBej4id8vnW7pN0\na153O9LFhF7M94+JiNclrQRMk/TbiBgv6eR8tt+SyNv+HLB1RGyVT2A3TdLdeZ1tgS2AOXmfuwB/\nAw6MiM3z8+v26nfWs7kGYlaZvYEjJT1CulbLWsCm+bGpZckD4Ov5nFqlCwttSvt2JZ3FmHwCu2Zg\nx7JtvxzpnEOPks6augB4R9LFkg4C3lnOYzOrihOIWWUEfC0itsvLxmUXbnrrg5WkPYC9gJ3y5U0f\nBVYq20al+yopP7Pr+0DffPLQMaSTKH4WuLnTR2PWBZxAzJZV+vJeCAwoK78FOClftwVJm0papZXn\nrwHMj4h383Xtdy577L3S81vs6x7g8NzPsi6wOzC1zQDTfteMiJuBU4GtKz88s67jPhCzZZVGYU0H\nluQmq0sj4px8ueG/5OtuvELr1+y+GfiqpCeAp4H7yx67EJgu6eGI+LfSviLi95J2Jp2OfQnwzYh4\nRdKoNmJbHbg+97EAfKP6wzWrnk/nbmZmVXETlpmZVcUJxMzMquIEYmZmVXECMTOzqjiBmJlZVZxA\nzMysKk4gZmZWFScQMzOryv8HRRsJB+36bHQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10fa26208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "def f(i):\n",
    "    return (3 / (1000 + 5*i))\n",
    "logistic2 = logisticRegression(gradient=\"batch\", rate=f)\n",
    "logistic2.train(XtrainP, Ytrain)\n",
    "logistic2.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that performance is not affected, and the final cross entropy when the convergence criteria that the difference in cross entropy between iterations is less than or equal to 2 may actually be less than that with constant learning rate. Although our data is relatively small and performing batch gradient descent did not take a lot of time, let us perform stochastic gradient descent to see how the cross entropy behaves with the number of iterations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAEZCAYAAABFFVgWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8VWXd///XGxFxQsFUEBQHQHEWlZySo5lTRVZmZvft\nkFFppqW3Kd31k3voNuvbnVrqfVemYqGJ3qaWGTkch0TBEQdUnFBQcELECRk+vz+ua8vmeM5hH9j7\nrHP2fj8fj/3Ya1977bU+a+3hs69rXetaigjMzMxqoUfRAZiZWf1ykjEzs5pxkjEzs5pxkjEzs5px\nkjEzs5pxkjEzs5pxkjFbBZIukvSvRcdRS5KWStoyT9f99lZC0nOS9u+E9XT7/e0k0w5JR0maKmmB\npNmS/iJp7wLjuUTSQklv5dsCSQ9W+NqzJI2vdYydKW/T5WWPP/wxrNH6jpF0Z3lZRJwQET+u1Tq7\niA9PpqvW9koaJenFVV1OB9f5kfevK6nXz5eTTBsknQr8N/CfwEbAZsAFwGfbmH+1TgrtnIjok2/r\nRsQu1VqwJFVrWZ0o2pjukArfP63KOrqClXyPa/G5KGJfdvX3r6vHt3IiwrcWN6APsAD4QjvznAVM\nBC4H3gS+BvQCzgVmA7OAXwCr5/k3AG4A5gGvA7eXLeuMPP9bwHRgvzbWeQnw7208NxhYChwNzARe\nAX6QnzsIWJhvC4AHc/ltpCR6F/AOsCUwALgux/gU8PVWtvnKHOt9wA75uX8Brm4R0/nAL1qJ9fvA\nxBZl5wHn5uljgWfyOp4BvtLOezA+T9+et//t/Lov5fLPAA/m/X5XKd783HM5loeB90h/us4Ans7L\neBQ4LM+7TZ5nUd6Hb7T2ngBjgBnAa8CfgAFlzy0Fvpn36xvAr8qe2wpozp+lV4ArKvys9gB+Drya\n99W383p6tPMeHws8nrfxaeAbLZZ5OvAS6TN5HLAE2LKN7V3R/j0t7995+XPTC1gLeBdYnPflW0D/\nKn5/P/L5aef96wOMz/v8OeBfWyxrTNm+ehTYuY1tuwLolZ9bn/Rdf4X0PboBGLiS8bXc35/L+3t+\n/pwdWMRvZIfej6ID6Io30o/yB6UvahvznEX60f5sftwb+HfgblJC2QD4B/Bv+fn/Ai7MPwqrAXvn\n8mHAC8DG+fFmwBZtrLOSJPO/+Yu8I/A+sHVZvONbvOY24Pn8Ae8B9CT9WP8SWB3YKX9Rmlps8+fz\nNpwGPJun++cvR58872rA3NKXssV6NyMlg7Xz4x6kH7XdST9A84Eh+bmNgeHtvAfjyx4vLd93wC45\nht1I/xL/mfTjUEr8zwEPAJsAa+SyL5a9F1/KcZYeHwPc0dZ7AuxP+rHfKe+/81n+z8RS4HpgXWDT\nvG8PzM9NAMbm6V7AXhV+Vr9F+vEbAKwH/J2UFMqTTMv3+BBg8/z8J0jJp/TjeTDwMjAcWBP4A20k\nmQr37z35PVyf9GP9jfzcKOCFFWzbGaQf8Dfyffn0G228ps3PTxvv33jg2vy6wcCTwHFl7/+LwIj8\neEtg0wq2rR/pO7IGsDbwR+DalYyvfH+PJP0J2T8/HgAMK+p3stKbm8tatwHwWkQsXcF8kyPiBoCI\neB84ipRUXo+I14F/I33xIP1DGUD6EVwSEf/I5UtIPyrbS+oZES9ExHPtrPN0SW9ImpfvLyl7LoBx\nEfFBREwj/cvaaQXbcGlEPJG3tT+wF3BGRCyKiIeB35JqRyX3R8S1EbGE1JzYG9gjIuYAd5C+mJB+\nyF6NiIdarjAiXiD9uH8+F30SeCcippbtkx0k9Y6IuRExfQXbUK68aWcM8D8RcV8kl5OS5B5l85wX\nES9FxMIc2zURMTdPTyT9WxxZ4bqPAi6OiIcjYhEwFthT0mZl85wdEQsi4kVSAtg5ly8CBksamN+/\nuytc55fyNrwcEfOBn7Qyz4fvcUQsjoi/RsTzeRvvBCaRkk1peZdExPSIeA8Y1866K92/cyPiTdI/\n+p1bW1BrIuKciOgbEf3yffl0v3ZeWtHnR1IP4MvAmRHxbkTMJNUKS9/Z44GfRsQDOZ5n8/vW7rZF\nxBv5O7IwIt4Bzgb27Wh8rfga6fN1a17PyxHxVIWvLYyTTOteBz6WP4TtaXngchNSraRkZi4D+Bmp\najxJ0tOSzgCIiGeA75K+zHMlTZA0oJ11/qz8ixYRx7V4fm7Z9LvAOh3Yhk1I/xDfbbENA1ubPyKC\n1KRS2sbxwD/l6a+SmhLbcgWpmYB8PyEv813SF/8E4GVJN0jaegXb0JbBwGk5Gb8haR4wqCxecvwf\nknS0pAdzEp8HbAd8rML1bULaX+RteYf0WSrff229P6eTvo9TJD0iqeX72t46y9/D1g6mL1cm6RBJ\nkyW9nrfxEJZtY8vlzaTtYzKV7N+Ofh5XSQc/Px8j1exafmdL79empO9sW1rdNklrSvpfSc9LepPU\nOrC+JK3i53tF8XRJTjKtm0z6R3bYCuaLFo9nk754JYNJzUBExNsR8S8RsRUwGjhV0n75uSsj4hNl\nr23t3+iqahlra+UvAf0krV1Wthlpu0o2LU3kg8iD8usgHYPYUdJ2pLb6P7QTz0SgSdJAUo1mwocB\nRfw9Ig4k1ayeBH7TznLa8yLw45yMS4l5nYj4Y9k8H25/rnH8Gjix9M8ZeIxlP7Jt7cOSlyh7//N+\n3IAWiaw1EfFKRHwjIgaSmsAurLCn3Muk96Bks1bmKd/GXsDVwE+BDfM2/pVl2/gyZe9x3p62truS\n/duWFe1LJI3NPSjfanFbIOmtNhf80c/Pr9tY52vkGmRZ2WCWfd5fJB0r66h/AYYCu0fE+iyrxaiD\n8bW0svEUykmmFRHxFqm9/wJJn8v/THrmf4DtJYArgR9K+pikjwE/Iv+bl/RpSaUPyALSQc+lkoZJ\n2i9/+T8gHfxbUTNdW9rrBTQX2Ly93kURMYt0TOlsSWtI2pHUZFBeI9lV0mG5N9b3SMd97smvXwhc\nQ0oY9+bltbWu10j/8C4Bno2IJwEkbSRptKS1SD8Ab5OaFyoxh9RuXvIb4FuSRuZlry3p0BZJtNza\npH3/mqQeuTaxfdnzc4FBklZv4/VXAMdJ2lHSGqTjcPe0aGJplaTDc8KF1O6+NN+QdJuk/6+Nl14F\nnCJpE0nrkzoytKdXvr0WEUslHQIc2GJ5x0oant+DttYLHd+/5eYCG0jq09YMEXF2pB6UfVrc1o2I\nVl/Xxuen9H1a7v2L1ER8FfBjSetIGkz6TJc+778F/kXSiLzsrSSVJ+C2rEP6Hr8lqR9lTY4dia8V\nF5M+X/sp2WQVavmdxkmmDRHx38CpwA9JB2hfAE4k/Vtvy3+SelyVjofcB5T6uA8Fbpa0gNQh4IKI\nuJ10cPAnpAPGLwEbktry2/L9Fv/oXikPu+VmlE1PJCWh1yXd18b8kJqutsixXAP8KCJuK3v+OlJ1\nfx6pSezzkY7PlFwG7EBqOluRCaTjMeU1nh6k/T6b9E9zX1LTQiXGAeNz083hEXE/6bjBryS9QerV\ndUzZ/Mttf24b/zkpac4hNZXdVTbLraSazZwW+730+ltIfyz+L8e/BXBkW+trYXfg3vwP/U/AyaXj\nJqSaxV1tvO43pGMq04D7gb8Ai2PZ8cSW2/g2cDIwMe+TI0nvaen5m0g9JG8l7a9b2gq4o/u3xWuf\nJCXlZ/P71b+teTuovc9Pa+/fyaSmrmdJxxR/HxGX5BivJn1/J+T35VrSQf12t420/9bK678buHEV\n4vtQpGOWx+Xlzyf1Rmyt5tqlKDWr12jh0iDSj83GpGz964j4paSfks43WUhqYzwu1x6QNJZ0gGsx\ncEpETMrlI4BLSQeab4yI79YscGuVpLOArSLi6Hbm2ZTUDbt//kGzVZBrN3+MiH0qnP9g4KKI2KK2\nkZlVptY1mcXAqRGxHbAncJKkbUj/vLaLiJ1JvXfGAkjaFjiC1H3yEFK7dKl55yLg+IgYBgyTdFCN\nY7cOUuoocRpwpRNMdUTE7PYSjKTeuRl3tZyQziLVpMy6hJommYiYE7kLa/7RmU46Kenmsur8PSw7\ncDma9AO1ODcVzABG5qr0urGsi+t4VnxQ3jpRbmOeTzpX5KyCw2kkInWVf4PUXPYY3v/WhfTsrBVJ\n2pzUj/zeFk99jdQ2C6nr4OSy52bnssUs30NnFst3C7VOEBH/1s5z75JOMrROFOlclkrP4zHrdJ1y\n4F/SOqRuk6eUN6MojS66KCKuaPPFZmbWbdW8JiOpJynBXB4R15WVHwscSmpeKZnN8n30B+Wytspb\nW1/tejKYmdWxiKj6YKidUZP5HfB4RJxXKsg9YE4HRudzK0quB46U1EvSFsAQYEqkIUvmSxqZOwIc\nTVm3y5aiC4zX0xVuZ511VuExdJWb94X3hfdF+7daqWlNRunaK18FHlG67kkA/0oaOLAX8Pfceeye\niDgxIh6XdBVpsLlFpDOvS1v/bZbvwnxTLWM3M7NVV9MkE2kQyNau0zG0ndecTRpQrmX5/aST/MzM\nrJvwGf91rKmpqegQugzvi2W8L5bxvqi9mp7xXwRJUW/bZGZWa5KIbnrg38zMGpSTjJmZ1YyTjJmZ\n1YyTjJmZ1YyTjJmZ1UxdJhl3LjMz6xrqMsm8+mrREZiZGdRpknn22aIjMDMzcJIxM7Maqsskc801\nRUdgZmZQp8PKQPjgv5lZB3hYmQ7o2RPefbfoKMzMrC6TzJAhPi5jZtYV1GWSGToUnnyy6CjMzKwu\nk8zgwfCznxUdhZmZ1WWS2W8/mDKl6CjMzKwue5e9/nqwwQawcCH06lV0RGZmXZ97l3VAv37p/uc/\nLzYOM7NGV5dJBmDMGFi6tOgozMwaW90mmVGj4OGHi47CzKyx1eUxmYhgxgwYNgzeeQfWWqvoqMzM\nurZueUxG0iBJt0p6TNIjkk7O5X0lTZL0pKS/SVqv7DVjJc2QNF3SgWXlIyRNk/SUpHNXtO4hQ9L9\nPfdUf7vMzKwytW4uWwycGhHbAXsC35a0DXAmcHNEbA3cCowFkLQtcAQwHDgEuFBSKbNeBBwfEcOA\nYZIOam/FEhx7rM/8NzMrUk2TTETMiYiH8vTbwHRgEPA54LI822XAYXl6NHBlRCyOiOeBGcBISf2B\ndSNiap5vfNlr2jR8ODz6aLW2xszMOqrTDvxL2hzYGbgH2Dgi5kJKRMBGebaBwItlL5udywYCs8rK\nZ+Wydu26KzzwwKpGbmZmK6tnZ6xE0jrA1cApEfF2Go5/OVXtfTBu3DgA3nsP7ruviYgmVPXDWWZm\n3VdzczPNzc01X0/Ne5dJ6gn8GfhrRJyXy6YDTRExNzeF3RYRwyWdCUREnJPnuwk4C5hZmieXHwmM\niogTWllflG/T+uun4zKlEzTNzOyjumXvsux3wOOlBJNdDxybp48BrisrP1JSL0lbAEOAKblJbb6k\nkbkjwNFlr2lX795w991V2AozM+uwmtZkJO0N3AE8QmoSC+AHwBTgKmBTUi3liIh4M79mLHA8sIjU\nvDYpl+8KXAr0Bm6MiFPaWOdyNZlSM1mdnQ5kZlZVtarJ1O3JmCXnnQd33QUTJxYYlJlZF9edm8sK\nNWAAXH110VGYmTWmuk8yTU3Qp0/RUZiZNaa6TzIbbgirrQazZq14XjMzq666TzIS7LtvOi5jZmad\nq+6TDMDee8M//lF0FGZmjcdJxszMaqbuuzADLFwIG2wAL78M665bUGBmZl2YuzCvgjXWSBcv++1v\ni47EzKyxNESSAfje99zDzMysszVMkjn0ULjvvqKjMDNrLA1xTAZg5kzYfHNYsgR6NExqNTOrjI/J\nrKJBg9L9Y48VG4eZWSNpmJoMwMiRaSyz6yq6SICZWeNwTaYKvvAFuP76oqMwM2scDVWTee89WGut\ndN+7dycHZmbWhbkmUwVrrpnuJ00qNg4zs0bRUEkG4NRT4dFHi47CzKwxNFyS2WsvuOOOoqMwM2sM\nDXVMBmD+fNh0U5g7d1nzmZlZo/MxmSpZbz3YeWfXZszMOkPDJRmAgw+Gm24qOgozs/rXcM1lAPff\nD7vtBkuXpitnmpk1OjeXVdGIEen+6aeLjcPMrN7VNMlIuljSXEnTysp2kjRZ0oOSpkjarey5sZJm\nSJou6cCy8hGSpkl6StK5qx4XHHUU3H77qi7JzMzaU+uazCXAQS3KfgqcFRG7AGcBPwOQtC1wBDAc\nOAS4UPqwMesi4PiIGAYMk9RymR02apQP/puZ1VpNk0xE3AXMa1G8FFgvT68PzM7To4ErI2JxRDwP\nzABGSuoPrBsRU/N844HDVjW2UaNckzEzq7WeBazze8DfJP0cELBXLh8ITC6bb3YuWwyUX9NyVi5f\nJcOGwcKF8Pzz6TozZmZWfUUkmROAUyLiT5IOB34HfKqaKxg3btyH001NTTQ1NX1kHgn23TfVZpxk\nzKzRNDc309zcXPP11LwLs6TBwA0RsWN+/GZErF/2/JsRsb6kM4GIiHNy+U2kYzYzgdsiYnguPxIY\nFREntLG+FXZhLrngAnjwQfjtb1dhA83M6kB37sKsfCuZLWkUgKRPko69AFwPHCmpl6QtgCHAlIiY\nA8yXNDJ3BDgaqMplx3bZBR5+uBpLMjOz1tS0uUzSBKAJ2EDSC6SayRjgfEmrAe8D3wCIiMclXQU8\nDiwCTiyrknwbuBToDdwYEVU5X3/nneHJJ2HePOjbtxpLNDOzcg15xn+5z34WvvKVdN6MmVmj6s7N\nZV3a6NFwww1FR2FmVp8avibz8suw7bbwyiuw+uo1DMzMrAtzTaZGBgxIXZinTCk6EjOz+tPwSQZg\n7709xIyZWS04yQCHHAJ/+lPRUZiZ1Z+GPyYD8O67sPbaqTvzsGE1CszMrAvzMZkaWmstGDMGrr++\n6EjMzOqLk0x2xBFw1VVFR2FmVl/cXJa9/z5ssAG8+mqq2ZiZNZLCmsvy8C91r3fvNMzM5MkrntfM\nzCpTSXPZDEk/y1eurGsHHADXXlt0FGZm9WOFzWWS1gWOBI4jJaXfka5g+Vbtw+u4lW0uA5g6FUaO\nhAULYJ11qhyYmVkXVqvmsg4dk8lD9E8gXTb5auA/IuLpage1KlYlyQB84Quw//5w0klVDMrMrIsr\n9JiMpNGSrgXOBX4ObAncANxY7YCKdtJJcMklRUdhZlYfKmkuexa4Dbg4Iu5u8dz5EXFyDePrsFWt\nySxaBBtumE7M3HjjKgZmZtaFFdZcJmmdiHi72iuulVVNMgBHHw1Dh8KPflSloMzMurgik8yWwHnA\nnsBSYDLwvYh4ttrBVEM1ksyMGWl4mcceS5cBMDOrd0UOKzMBuAroD2wCTASuqHYgXcnQoXD88XD5\n5UVHYmbWvVVSk5kWETu2KHs4InaqaWQrqRo1GYAHHoBdd4WlS0FVz+1mZl1LkTWZv0o6U9LmkgZL\n+j5wo6R+kvpVO6CuYqecQn/5y2LjMDPrziqpyTzXztMREVtWN6RVU62aDKTmsquughtuqMrizMy6\nrC5xMmZ3UM0k8/bby7ozb7ZZVRZpZtYlFdm7bHXgBGDfXNQM/G9ELKp2MNVQzSSTlpfu6ywXm5kt\np8hjMhcBuwIX5tuuuWyFJF0saa6kaS3KvyNpuqRHJP2krHyspBn5uQPLykdImibpKUnnVrLuanno\noXT/3nuduVYzs/pQSU3mIz3JKu1dJmkf4G1gfKmHmqQm4AfAoRGxWNLHIuI1ScNJ3aV3BwYBNwND\nIyIk3QucFBFTJd0InBcRf2tjnVWtyQDsvTeMGgX/9V9VXayZWZdRZE1miaStygLZElhSycIj4i5g\nXoviE4CfRMTiPM9rufxzpNGdF0fE88AMYKSk/sC6ETE1zzceOKyS9VfLySfD2WfDkoq22szMSipJ\nMqcDt0lqlnQ7cCtw2iqscxiwr6R7JN0maddcPhB4sWy+2blsIDCrrHxWLus0X/4yDB7syzObmXVU\nz/aelNQDeA8YCmydi5+MiIWruM6+EbGHpN1JIwhUtRv0uHHjPpxuamqiqalplZd5xhlw1FHwuc/5\n8sxm1v01NzfT3Nxc8/VUckzmwYjYZaVXIA0Gbig7JnMjcE5E3J4fzwD2AMYARMRPcvlNwFnATOC2\niBiey48ERkXECW2sr+rHZFJc0Lcv/Pd/w9e+VvXFm5kVqshjMrdI+qK00oOrKN9K/gTsDyBpGNAr\nIl4Hrge+LKmXpC2AIcCUiJgDzJc0MsdwNHDdSsay0iS44gq44AJ3ZzYzq1QlNZkFwNrAYuB9UsKI\niOizwoVLE4AmYANgLqlmcjlwCbAzsBA4raxWMxY4HlgEnBIRk3L5rsClQG/gxog4pZ111qQmA2kc\ns3XXTSMBfOELNVmFmVkhfMZ/hWqZZAB+/GO47LI0CoAHzjSzelHk5ZdvqaSsUXz/++l6Mz/8YdGR\nmJl1fW32LpPUG1gL+Jikviw7rtKHTu5C3JWsvnoamfm6Tj8qZGbW/bTZXCbpFOC7pAuVzWZZknkL\n+E1E/KpTIuygWjeXASxYAAMHwosvwnrr1XRVZmadosgBMr8TEd3mqiqdkWQAjjgC9toLvvvdmq/K\nzKzmCj3wL2kvYHPKmtciYny1g6mGzkoyDz0Ehx4KzzwDa65Z89WZmdVUkTWZy4GtgIdYNmZZRMTJ\n1Q6mGjoryQBssw0MGQJ//nOnrM7MrGaKTDLTgW077Zd7FXVmknn6adhuu9SdefPNO2WVZmY1UeQZ\n/48C/au94nowZAiMHQs/+lHRkZiZdU2V1GRuI52dP4V0hj4AETG6tqGtnM6syQC8+WYa0ww83IyZ\ndV9FNpeNaq28NBRMV9PZSQbSeTMnn5zGNjvyyE5dtZlZVXR6kpG0TUQ8kafXKB/eX9IeEXFPtYOp\nhiKSDKSRmSdMSJdp9nAzZtbdFHFMZkLZ9OQWz11Y7UC6u4svTs1mV15ZdCRmZl1He0lGbUy39rjh\nSXDiiXDuuT42Y2ZW0l6SiTamW3tswKmnwpQpMGJE0ZGYmXUN7V1+eZCk80m1ltI0+XHDDpDZnrXX\nhsceg/32g8mTYc89i47IzKxY7R34P6a9F0bEZTWJaBUVdeC/3O9+B//zP3D33dCzvTRuZtZF+KJl\nFeoKSWbpUlhtNdhoI5g7t9BQzMwqUuQZ/9ZBPXrArFnwwQfwt78VHY2ZWXGcZGpk4ED42c/g8MPh\n5ZeLjsbMrBg+YlBDX/96SjD77AOPPAJrrVV0RGZmnWuFNRlJP5XUR9Lqkm6R9Kqkf+qM4OrBj34E\nO+wAhxxSdCRmZp2vkuayAyPiLeAzwPPAEOD0WgZVb37zG7jzTvi//ys6EjOzzlVJkik1qX0amBgR\n82sYT13acEP4xz/gi1+Ee7rkiG9mZrVRSZL5s6QngF2BWyRtCLxfycIlXSxprqRprTx3mqSlkvqV\nlY2VNEPSdEkHlpWPkDRN0lOSzq1k3V3NnntCv37pfunSoqMxM+scK0wyEXEmsBewW0QsAt4BPlfh\n8i8BDmpZKGkQ8ClgZlnZcOAIYDhwCHCh9OF4xhcBx0fEMGCYpI8sszt45RUYOTKNb2Zm1ggqOfD/\nJWBRRCyR9EPg98AmlSw8Iu4C5rXy1C/46HGdzwFXRsTiiHgemAGMlNQfWDcipub5xgOHVbL+rma1\n1eCii+C002DGjKKjMTOrvUqay34UEQsk7QMcAFxMqlmsFEmjgRcj4pEWTw0EXix7PDuXDQRmlZXP\nohuPnTZiBJxzDnzjG7BkSdHRmJnVViXnyZR+Cj8N/Doi/iLpP1dmZZLWBH5AaiqrmXHjxn043dTU\nRFNTUy1X12GnnZauonnyyXDBBUVHY2aNqLm5mebm5pqvp5LLL/+ZVKv4FDACeA+YEhE7VbQCaTBw\nQ0TsKGl74GbgXfLoznnZI4GvAUTET/LrbgLOIh23uS0ihufyI4FREXFCG+srfOyySkyfDttvD9dc\nA4d1y8Y/M6snRY5ddgTwN+CgiHgT6EfHzpNRvhERj0ZE/4jYMiK2IDV97RIRrwDXA1+W1EvSFqTz\ncaZExBxgvqSRuSPA0cB1HVh/lzR8eLocwJgx8PDDRUdjZlYblfQuexd4BjhI0knARhExqZKFS5oA\n3E3qEfaCpONaLp5lCehx4CrgceBG4MSyKsm3SceCngJmRMRNlay/qxs5Es47D770JXjuuaKjMTOr\nvkqay04BxgCl89U/Tzo288sax7ZSuktzWbnPfAb+8hd44gnYeuuiozGzRlTY9WTyiZR7RsQ7+fHa\nwOSI2LHawVRDd0wy778P3/8+NDfDXXdBnz5FR2RmjabIYzJiWQ8z8nTVA2lkvXunZrO994b11oO3\n3io6IjOz6qikJnMqcAxwbS46DLg0IrrkeevdsSZTsmRJGnZmq61SF2czs85S6OWXJY0A9skP74yI\nB6sdSLV05yQD8O67sOuu6RIBRx1VdDRm1ihqlWTaPRlT0mrAYxGxDfBAtVduH7XWWjBhAjQ1waJF\ncMwxRUdkZrby2j0mExFLgCclbdZJ8Riwyy5w2WVw7LFw/vlFR2NmtvIqOSZzB7ALMIU0AjMAETG6\ntqGtnO7eXFbuvvtg993hhhtSN2czs1opsgvzqNbKI+L2agdTDfWUZADuvTcNO/PVr8L/+39FR2Nm\n9arTj8lIGgJs3DKZ5NGYX652INa6j38c/v532GEHWHttOOss6FFJx3Mzsy6gvZ+rc4HWztiYn5+z\nTrL99vDSSynZHHIIvPpq0RGZmVWmvSSzcSvXfCGXbV6ziKxVAwbAHXek69FstBGcfTYsXlx0VGZm\n7WuvC/P67Ty3ZrUDsRXr2TMll3ffhR/8AKZOhT/8Adb0u2FmXVR7NZn7JI1pWSjp68D9tQvJVuS8\n89J4Z716pfNp5swpOiIzs9a12btM0sakoWQ+YFlS2Q3oBXw+X+ely6m33mXtiYD/+I/UGWDMGPjW\nt1JzmplZRxXZhXk/YPv88LGIuLXaQVRTIyWZkl//Gr75zTR9wAHwq1/5kgFm1jGFjl3WnTRikgH4\n4IN0Sed/+id49NF0WecvfKHoqMysu3CSqVCjJply990HX/xiSjj//u+w2mpFR2RmXV2R15Oxbma3\n3VLPs38O0/4YAAASLUlEQVT8I/VIe/rpoiMys0blJFOnNtoIbr4ZRo9O45/dcUfREZlZI3KSqWM9\ne8J118HEiTBqFJx+uk/gNLPO5STTAA44IA1L88gj8MlPwgO+MpCZdRInmQYxYAD85S+pRrPrrvCl\nL8Ef/5jOtTEzqxUnmQay2mqpt9nMmTB3Lhx5JBx6KLzwQtGRmVm9qmmSkXSxpLmSppWV/VTSdEkP\nSbpGUp+y58ZKmpGfP7CsfISkaZKekuQRoFfRZpuljgAffAB7751GCTj3XFiypOjIzKze1Lomcwlw\nUIuyScB2EbEzMAMYCyBpW+AIYDhwCHChpFKf7YuA4yNiGDBMUstl2kpYfXX44Q+huTl1DujZE845\nx8nGzKqnpkkmIu4C5rUouzkiluaH9wCD8vRo4MqIWBwRz5MS0EhJ/YF1I2Jqnm88cFgt4240228P\nd94Jv/89/PWvMHJkGprm/feLjszMuruij8l8DbgxTw8EXix7bnYuGwjMKiuflcusinr0SJd4vvXW\nVLu5/vp0CYHvfAcef7zo6Mysu2rvejI1JelfgUURcUW1lz1u3LgPp5uammhqaqr2KupWjx7w+c/D\nYYfBjBnw85/Ddtul69icfrqHqDGrF83NzTQ3N9d8PTUfu0zSYOCGiNixrOxYYAywf0QszGVnAhER\n5+THNwFnATOB2yJieC4/EhgVESe0sb6GH7us2mbOhOOOg3nz4BvfSCM+9yi6DmxmVdWdxy5TvqUH\n0sHA6cDoUoLJrgeOlNRL0hbAEGBKvm7NfEkjc0eAo4HrOiFuywYPTkPUHHMMnHhiqs2ccgq8/nrR\nkZlZV1frLswTgLtJPcJekHQc8EtgHeDvkh6QdCFARDwOXAU8TjpOc2JZleTbwMXAU8CMiLiplnHb\nR/XoAd/9bup5dsstaXia4cPh/PNh0aKiozOzrspD/dtKe/RROPVUePHF1Bvtk58sOiIzW1m+nkyF\nnGQ6VwT8+c9w0knQ1ATjxsHmm4Oq/lE1s1rqzsdkrI5J8NnPwmOPQf/+sOWW0K8ffP3rqaZjZo3N\nScaqYp11lo0WcM01KfkccAAccQRcfTUsWFB0hGZWBDeXWc288w785jcp6TzxRGpS+853Uk3HzLoW\nN5dZt7P22qlH2p13pktBz5wJQ4fC0UfDpEmwdOmKl2Fm3ZtrMtapXngBrr0Wxo9PF0/75jfhK1+B\nT3zCJ3iaFcm9yyrkJNN9PP54Ol4zcSK89VYazmb06HRhNQ9fY9a5nGQq5CTT/UTA/fenJrSJE1Ov\ntFNOgTFjYOuti47OrDE4yVTISaZ7i4C774YbboBLL03n3Hzzm+ly0eusU3R0ZvXLSaZCTjL1Y8mS\ndH2biy5KF1bbZRc4/PB02ej+/YuOzqy+OMlUyEmmPr33XrrWzcSJcN11sOee8KlPwQ47pNrPfvul\nK3ua2cpxkqmQk0z9W7AA/vKXVLu5+OI0WGffvmnstAMPTD3Vtt7aQ9uYdYSTTIWcZBrTyy+njgO3\n3AK33ZZ6px18MOy/fxpTbaONio7QrGtzkqmQk4xFwPTpcNNNKeHceWcafeDjH0/XxPn4x1Mzm2s6\nZss4yVTIScZaWrwY7rknJZ5Jk2Dq1FR++OFw0EHp+I57rlmjc5KpkJOMrUgEPPJIGlPtllvgwQdh\nxx1Ts9rBB8Pee7sTgTUeJ5kKOclYR733Xjo359ZbU4eCp59OCefAA9Ntm22KjtCs9pxkKuQkY6vq\n9ddTDWfSpHRcZ/XVYb31UieCffeFffaBj32s6CjNqstJpkJOMlZNS5emYzgPPpiSz513wuTJMHBg\n6ipdug0eXHSkZqvGSaZCTjJWa4sXw7RpKeHceSfccQf07p1qODvtBHvsAZtskobEWX31oqM1q4yT\nTIWcZKyzRcBTT8Fdd6XkM3lyOq6zeHFKOKNGwSuvpKFwtt8+9WZzc5t1NU4yFXKSsa5i3rxU07n9\ndvj739MF2xYsgHvvTSeH7r57uu22G4wYkS7yZlYUJ5kKOclYV7dkSboc9dSp6XbffenyBltuuSzx\n7Lpr6mwwZIivrWOdo1smGUkXA58B5kbEjrmsL/BHYDDwPHBERMzPz40FvgYsBk6JiEm5fARwKdAb\nuDEivtvOOp1krNv54IN07k4p6dxzTxoqZ9GilHD22CONxzZ0aHrcu3fREVu96a5JZh/gbWB8WZI5\nB3g9In4q6Qygb0ScKWlb4A/A7sAg4GZgaESEpHuBkyJiqqQbgfMi4m9trNNJxurGG2/AlCkp6Tz8\ncLp89eOPp1rOdtvB8OGpqW333WHbbV3rsZXXLZMMgKTBwA1lSeYJYFREzJXUH2iOiG0knQlERJyT\n5/srMA6YCdwaEdvm8iPz609oY31OMlbXFi2CuXPhscfS7f77Uw3opZdg3XVh4cJ0AmnpeM9uu6Va\nUI8eRUduXVmtkkwRg2dsFBFzASJijqTS+LgDgcll883OZYuBWWXls3K5WUNafXUYNCjdDjpoWfm8\neelcntdfT6MYTJ2arjA6bhy8+mqq8eywQ+p0MGxYmt58c1hrraK2xBpBVxihqerVjnHjxn043dTU\nRFNTU7VXYdbl9O2bbkOGpMflH/s33kg1nocfTtNXXgk//CG8+CKssUZKNjvskG5bbgmbbppqP337\nFrEl1hmam5tpbm6u+XqKaC6bDjSVNZfdFhHDW2kuuwk4i9RcdltEDM/lbi4zq5KlS1MN6LnnUseD\nRx6B559Px36efBLWXDMlm222Sfdbbw2bbZaSUL9+RUdv1dSdm8uUbyXXA8cC5wDHANeVlf9B0i9I\nzWFDgCn5wP98SSOBqcDRwPmdELdZ3evRAzbYIN1222355yJSD7cnn1x2u/XWlIRmz04jVQ8btuw2\ndOiye5/zYyW17l02AWgCNgDmkmomfwImApuSailHRMSbef6xwPHAIpbvwrwry3dhPqWddbomY1Zj\nEek4z1NPLbvNmJHun346Ja2hQ9O8Cxem5LP55um20UYwYEAa761vX188rqvotr3LOpuTjFmxli5N\nx3pmzIBnnkkdCxYtSk1yzz0Hr72WesK98EI6MXXw4JR8Nt44NcENHpyG39lll9RV2zqHk0yFnGTM\nuo/581Pz28yZaXy3119PiWjatNRJoU+fNNjo4MGpQ8IWWyy7798f1l+/6C2oH04yFXKSMasPS5em\nGs/s2anW8+yzKQGV7ufMSceFttoqJaOBA1NCKt1KjwcMSD3orH1OMhVykjFrDBGp5vPMM+n+1VdT\nUirdZs9O93PmpGa38sRTnpAGDkw1Kin1mttkk9SrrtE4yVTIScbMyi1dunwCKiWf8umHHkrdtN9/\nP/WoW3vtlHzaum28cao91VMvOieZCjnJmNmqiEidE2bPbvv20kvp/KJSM92gQel+wIBlCWjDDVOt\nqG/f9LhPn649tpyTTIWcZMysM0Sk0RNKiWfWrFQLeuUVeOutdP/SS6kpbv78dC2htdZKI2ivv/6y\nq6dusklqphswIHX97tcv3W+4YTrm1FmcZCrkJGNmXdHSpfD22/DOOykJzZ6deta9/HLq8j1nTjq2\n9MYbqSY1b15KOOuss6yX3YQJaboWnGQq5CRjZvVg0aKUbN55B958M9WKPv3p2jW5OclUyEnGzKzj\napVkfIUJMzOrGScZMzOrGScZMzOrGScZMzOrGScZMzOrGScZMzOrGScZMzOrGScZMzOrGScZMzOr\nGScZMzOrGScZMzOrGScZMzOrGScZMzOrmcKSjKTvSXpU0jRJf5DUS1JfSZMkPSnpb5LWK5t/rKQZ\nkqZLOrCouM3MrHKFJBlJmwDfAUZExI5AT+ArwJnAzRGxNXArMDbPvy1wBDAcOAS4UFLVh6SuN83N\nzUWH0GV4XyzjfbGM90XtFdlcthqwtqSewJrAbOBzwGX5+cuAw/L0aODKiFgcEc8DM4CRnRtu9+Mv\n0DLeF8t4XyzjfVF7hSSZiHgJ+DnwAim5zI+Im4GNI2JunmcOsFF+yUDgxbJFzM5lZmbWhRXVXLY+\nqdYyGNiEVKP5KtDykpa+xKWZWTdWyOWXJR0OHBQRY/Ljfwb2APYHmiJirqT+wG0RMVzSmUBExDl5\n/puAsyLi3laW7cRkZrYSanH55Z7VXmCFXgD2kNQbWAh8EpgKvA0cC5wDHANcl+e/HviDpF+QmsmG\nAFNaW3AtdpKZma2cQpJMREyRdDXwILAo3/8aWBe4StLXgJmkHmVExOOSrgIez/OfGEVUwczMrEMK\naS4zM7PGUDdn/Es6WNITkp6SdEbR8dSCpEGSbpX0mKRHJJ2cyzt8EqukEflE2KcknVvE9qwqST0k\nPSDp+vy4IfcDgKT1JE3M2/eYpI834v6o1kne3XU/SLpY0lxJ08rKqrb9eX9emV8zWdJmKwwqIrr9\njZQsnyb1VlsdeAjYpui4arCd/YGd8/Q6wJPANqRjWN/P5WcAP8nT25KaInsCm+d9VKq93gvsnqdv\nJHXEKHwbO7g/vgf8Hrg+P27I/ZBjvxQ4Lk/3BNZrtP1B6qn6LNArP/4j6dhuw+wHYB9gZ2BaWVnV\nth84AbgwT3+ZdP5iuzHVS01mJDAjImZGxCLgSlIX6boSEXMi4qE8/TYwHRhEB09izT331o2IqXm+\n8WWv6RYkDQIOBX5bVtxw+wFAUh/gExFxCUDezvk05v5Y5ZO8u/N+iIi7gHktiqu5/eXLuprUaatd\n9ZJkWp6sOYs6P1lT0uakfyz30PGTWAeS9lFJd9xfvwBOZ/lzqRpxPwBsAbwm6ZLcfPhrSWvRYPsj\nqneSd7feD63YqIrb/+FrImIJ8Kakfu2tvF6STEORtA7pX8QpuUbTUCexSvo0MDfX6trrsl7X+6FM\nT2AEcEFEjADeIY0D2GifC5/kXZlqbv8KTxmplyQzGyg/ADUol9Wd3AxwNXB5RJTOI5oraeP8fH/g\nlVw+G9i07OWl/dJWeXexNzBa0rPAFcD+ki4H5jTYfiiZBbwYEfflx9eQkk6jfS4OAJ6NiDfyv+xr\ngb1ovP3QUjW3/8PnJK0G9ImIN9pbeb0kmanAEEmDJfUCjiSdwFmPfgc8HhHnlZVdTzqJFT56EuuR\nuUfIFuSTWHOVeb6kkZIEHF32mi4vIn4QEZtFxJak9/rWiPhn4AYaaD+U5KaQFyUNy0WfBB6jwT4X\nlJ3kneP/JOncukbbD2L5GkY1t//6vAyAL5FGy29f0b0hqtir4mBSb6sZwJlFx1OjbdwbWELqPfcg\n8EDe7n7AzXn7JwHrl71mLKnXyHTgwLLyXYFH8v46r+htW4V9MoplvcsaeT/sRPqz9RDwf6TeZQ23\nP4Cz8jZNIx2gXr2R9gMwAXiJNJLKC8BxQN9qbT+wBnBVLr8H2HxFMflkTDMzq5l6aS4zM7MuyEnG\nzMxqxknGzMxqxknGzMxqxknGzMxqxknGzMxqxknGrIykBfl+sKSvVHnZY1s8vquayzfripxkzJZX\nOnFsC+CojrwwD7PRnh8st6KIfTqyfLPuyEnGrHVnA/vkUY1PUbpA2k8l3SvpIUljACSNknSHpOtI\nQ7kg6VpJU5UuLPf1XHY2sGZe3uW5bEFpZZJ+lud/WNIRZcu+TcsuRnZ52fw/Ubo410OSftppe8Ws\ng3oWHYBZF3UmcFpEjAbISeXNiPh4Hh/vH5Im5Xl3AbaLiBfy4+Mi4k1JvYGpkq6JiLGSvh1plOSS\nyMv+IrBjROwgaaP8mtvzPDuTLi41J69zL+AJ4LCI2Ca/vk+tdoLZqnJNxqwyBwJHS3qQdNXAfsDQ\n/NyUsgQD8F1JD5HGdhpUNl9b9iaNJk1EvAI0A7uXLfvlSOM/PUS6guF84D1Jv5X0eeC9Vdw2s5px\nkjGrjIDvRMQu+bZVpAtiQbp+S5pJGgXsD3w8InYmJYbeZcuodF0lC8umlwA9Iw1jP5J0yYfPADd1\neGvMOomTjNnySj/wC4B1y8r/BpyYr+eDpKH56pMtrQfMi4iFkrYB9ih77oPS61us607gy/m4z4bA\nJ4ApbQaY1rt+RNwEnArsWPnmmXUuH5MxW16pd9k0YGluHrs0Is7Ll7x+IF9j4xVav+77TcC3JD1G\nGlp9ctlzvwamSbo/0vVvAiAirpW0B/AwsBQ4PSJekTS8jdj6ANflYz4A31v5zTWrLQ/1b2ZmNePm\nMjMzqxknGTMzqxknGTMzqxknGTMzqxknGTMzqxknGTMzqxknGTMzqxknGTMzq5n/H0U1vjBDsfNR\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1105abf98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "logistic3 = logisticRegression(gradient = \"stochastic\")\n",
    "logistic3.train(XtrainP, Ytrain)\n",
    "logistic3.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can see that with stochastic gradient descent, it may not be possible to reach the model with minimum loss because we are not leveraging all of our data to calculate the gradient. Nevertheless, stochastic gradient descent may be useful when our data is big and it is sufficient to have a model that roughly approximates that of minimum loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Kernel Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we shall implement kernel logistic regression. The purpose of applying a kernel is to project your data into higher dimensional space to improve performance. As a motivating example, consider trying to fit a linear decision boundary between class 1 and class 2 in the below plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEACAYAAABGYoqtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFuRJREFUeJzt3X+U1XW97/HnewQEjjmNP0CXAqJmJJ1ucQK8nuxOeQjS\nUlPzoklpt5P245Z2uompxenHWlKufqp5vFlhy1Kju46YBlYyV1pWeioqCRA7N34FZIpaYSM/3veP\nvYFhnA8zw97DzGyfj7W+a/b3+/189vf9Zdj7Nd/Pd3/3NzITSZK60tTfBUiSBi5DQpJUZEhIkooM\nCUlSkSEhSSoyJCRJRXUJiYiYERErIuLRiLii0OZLEbEqIpZGxKs6LL8lIjZFxK87tW+JiPsiYmVE\nLIqI5nrUKknquZpDIiKagOuB6cBE4PyImNCpzRuB4zLzJcAlwFc6rP56tW9ns4EfZuZLgfuBK2ut\nVZLUO/U4kpgCrMrM1Zm5FbgdOLNTmzOBWwEy82dAc0SMrs7/GNjcxfOeCcyrPp4HnFWHWiVJvVCP\nkDgKWNthfl112d7arO+iTWejMnMTQGZuBEbVWKckqZcG04lrvz9EkvazIXV4jvXA2A7zR1eXdW4z\npps2nW2KiNGZuSkijgD+2FWjiDA8JGkfZGZ016YeRxIPA8dHxLiIGAbMBBZ0arMAeDtARJwEPLVz\nKKkqqlPnPhdVH78DuKtUQGY27PTxj3+832tw/9y/F+L+NfK+Zfb8b+uaQyIztwPvB+4DlgG3Z+by\niLgkIt5dbXMv8P8i4jHg34D37uwfEd8CHgROiIg1EXFxddVcYFpErAROBa6ttVZJUu/UY7iJzFwI\nvLTTsn/rNP/+Qt8LCsufBP6pHvVJkvbNYDpx/YLU2tra3yX0KfdvcGvk/WvkfeuN6M3Y1EAUETnY\n90GS9reIIHtw4rouw02SVA/HHHMMq1ev7u8yGsq4ceP4/e9/v8/9PZKQNGBU/7rt7zIaSunftKdH\nEp6TkCQVGRKSpCJDQpJUZEhIUo3mzZvHKaec0t9l9AlDQpLqIKLbc8B1c8MNNzB58mSGDx/OO9/5\nzj7dlh+BldQQNm6Eb34Tnn0Wzj4bXv7y/q6o7xx11FFcc801LFq0iGeffbZPt+WRhKRB4Wc/g+uu\nqwTBc8/tuW7dukooXHUV/Ou/wtSpsGRJ/WtYt24d55xzDqNGjeLwww/nAx/4QJftLrvsMsaOHUtz\nczOTJ0/mxz/+8a51Dz/8MJMnT6a5uZkjjzySD3/4wwC0t7cza9YsDjvsMFpaWpg6dSqPP/54l89/\n1llnccYZZ3DIIYfUfyc7MSQkDXjf+Aa8/vXw0Y/Ce94Dr30tbN26e/1118FTT1WW7dgBW7bAhz60\n53OsWAGveAWMHFn5uWJF72rYsWMHb3rTmxg/fjxr1qxh/fr1zJw5s8u2U6ZM4de//jWbN2/mggsu\n4K1vfSvPVZPtgx/8IJdddhlPP/00v/vd7zjvvPOAynmNZ555hvXr1/Pkk09y0003MWLEiN4V2QcM\nCUkDWia8732VN/6tW+Gvf4Vly2BBhxsSPPkkbN++Z7+nntr9eMuWSrA88khlOOqRRyrzW7b0vI6H\nHnqIDRs28JnPfIbhw4czbNgwTj755C7bXnDBBbz4xS+mqamJyy+/nPb2dlauXAnAsGHDeOyxx3ji\niScYOXIkU6ZMAWDo0KE88cQTPProo0QEr3rVqzjooIN6XmAfMSQkDWg7dsDf/vb8ZU88sXv+rW+t\nHCHsNHIkVP9AB2D5cmhvrwQOVH62t/fuaGLt2rWMGzeOpqbu3zavu+46TjzxRFpaWmhpaeGZZ57h\nT3/6EwC33HILK1euZMKECUydOpV77rkHgFmzZjF9+nRmzpzJ0UcfzezZs9neOfn6gSEhaUA74IDK\nOYahQ/dc/trX7n785jfD5z8PRxwBhxwC//zPlXMTO7W07Dk8BZX5F7+453WMGTOGNWvWsGPHjr22\nW7JkCZ/97GeZP38+mzdvZvPmzRx88MG7vhrjuOOO41vf+haPP/44H/nIRzj33HN59tlnGTJkCNdc\ncw3Lli3jwQcf5O677+bWW2/teYF9xJCQNODddRecfDIMGwajR8N3vgMTJuzZ5t3vhg0bKkcYX/gC\nDOnw2c1jj4WZM+Hv/g6amio/zz+/srynpkyZwpFHHsns2bPZsmUL7e3tPPjgg89r95e//IWhQ4dy\n6KGH8txzz/GJT3yCP//5z7vW33bbbbuOKpqbm4kImpqaaGtr45FHHmHHjh0cdNBBDB06tHjUsn37\ndv72t7+xfft2tm3bRnt7e58ddRgSkga8ww+HtrbKENHGjXDaab1/jltugXnzKkcYt94KX/1q7/o3\nNTVx9913s2rVKsaOHcuYMWO48847n9du+vTpTJ8+nRNOOIHx48czcuRIxowZs2v9woULmThxIgcf\nfDCXX345d9xxBwceeCAbN27k3HPPpbm5mYkTJ/K6172OWbNmdVnLpz71KUaOHMncuXO57bbbGDly\nJJ/+9Kd7t0M95LfAShow/BbY+vNbYCVJfcaQkCQVGRKSpCJDQpJUZEhIkooMCUlSkSEhSSoyJCRJ\nRYaEJNXI25dKkvZqf92+9LnnnuNd73oXxxxzDM3NzUyaNImFCxf22fYMCUmD37Zt8Nhju+f/+MfK\nTSYa0LZt2xg7dixLlizh6aef5pOf/CTnnXcea9as6ZPtGRKSBr716ytBsFPnN8SHHoLXvAaWLq0E\nxKmnwh131L2MgXD70pEjR/Kxj31s15cGnn766YwfP56f//zndd9fMCQkDQZXXQXveEclKO69FyZP\nroTBTiefDDfcAP/wD5XvEn/LW+DSS3ev3769cj/TDRsq8xs2VOZ78fXaA/X2pZs2bWLVqlVMnDix\nx/vSG4aEpIHvK1+Bxx+vfGf4rFmVG0yMGrVnm1NOqdyyDuDss6HjOYIDDqjcjej1r4df/rLy85BD\nKst7aCDevnTbtm1ceOGFXHTRRZxwwgk93pfeMCQkDXwjRsAll1RuXN3SAq9+9Z7rdw4xXXMNzJ8P\nM2ZUhp46uvrqSptJkyo/r766VyUMtNuXZiYXXnghBx54IF/+8pd7tS+9kpmDeqrsgqRGUHw933NP\n5uGHZy5enDltWuYFF2Ru3bp7/fLlmXPnZu7YUZmfPz/ze9/b8zn+8IfMCRMyhw+v/PzDH3pV209+\n8pMcPXp0bt++/XnrvvGNb+Qpp5ySmZkPPPBAjho1KpctW7ZrfUtLS/7oRz96Xr/58+fn8OHDc8uW\nLXssX716dZ544on5ta99rVjPxRdfnKeeemq2t7fvte7Sv2l1ebfvsR5JSBr4fvUrWLAAWlsrQ03N\nzfD007vXT5gAH/nI7iGmc86B00/fvX779srt7N72Nnj22crP007r1TmJgXT70ksvvZQVK1awYMEC\nhg0b1uN92BeGhKSB78or4aSTKo9HjIAbb4RDD+15/wMOgO9+d/cQ09VXV+Z7cU5ioNy+dM2aNdx8\n880sXbqU0aNH86IXvYiDDz6Yb3/72z3/9+gFb18qacDw9qX15+1LJUl9xpCQJBUZEpKkIkNCklRk\nSEiSigwJSVLRkP4uQJJ2Gjdu3H67L8MLxbhx42rq73USkvQC5HUSkqSaGRKSpKK6hEREzIiIFRHx\naERcUWjzpYhYFRFLI+KV3fWNiI9HxLqI+EV1mlGPWiVJPVfzieuIaAKuB04F/gA8HBF3ZeaKDm3e\nCByXmS+JiKnATcBJPej7ucz8XK01SpL2TT2OJKYAqzJzdWZuBW4HzuzU5kzgVoDM/BnQHBGje9DX\njzlIUj+qR0gcBaztML+uuqwnbbrr+/7q8NRXI6K5DrVKknqhv66T6MkRwo3AJzIzI+JTwOeA/9FV\nwzlz5ux63NraSmtrax1KlKTG0dbWRltbW6/71XydREScBMzJzBnV+dlUbos3t0Obm4DFmXlHdX4F\n8N+A8d31rS4fB9ydma/oYvteJyFJvbQ/r5N4GDg+IsZFxDBgJrCgU5sFwNurhZ0EPJWZm/bWNyKO\n6ND/bOCROtQqSeqFmoebMnN7RLwfuI9K6NySmcsj4pLK6rw5M++NiNMi4jHgr8DFe+tbferPVD8q\nuwP4PXBJrbVKknrHr+WQpBcgv5ZDklQzQ0KSVGRISJKKDAlJUpEhIUkqMiQkSUWGhCSpyJCQJBUZ\nEpKkIkNCklRkSEiSigwJSVKRISFJKjIkJElFhoQkqciQkCQVGRKSpCJDQpJUZEhIkooMCUlSkSEh\nSSoyJCRJRYaEJKnIkJAkFRkSkqQiQ0KSVGRISJKKDAlJUpEhIUkqMiQkSUWGhCSpyJCQJBUZEpKk\nIkNCklRkSEiSigwJSVKRISFJKjIkJElFhoQkqciQkCQVGRKSpCJDQpJUZEhIkooMCUlSkSEhSSoy\nJCRJRYaEJKmoLiERETMiYkVEPBoRVxTafCkiVkXE0oh4ZXd9I6IlIu6LiJURsSgimutRqySp52oO\niYhoAq4HpgMTgfMjYkKnNm8EjsvMlwCXADf1oO9s4IeZ+VLgfuDKWmutxaJF8IY3VKZFi9xeTdau\nhSVLds8vWVJZ5vbcXhca+rXQD9vrtcysaQJOAr7fYX42cEWnNjcB/73D/HJg9N76AiuA0dXHRwAr\nCtvPvrZwYeaIEZlQmUaMqCxze/vogQcyDzssc/HiynTYYZVlbs/tddLor4X9/trroPre2f17fE8a\n7fUJ4Bzg5g7zFwJf6tTmbuDkDvM/ACbtrS+wudNzPFnYft/8C3YwbdruX+LOado0t1eTxYt3b2zx\n4j7emNsbrNtr9NdCv7z2qnoaEkPqfWTSQ7EPfbK0Ys6cObset7a20traug9PL0mNq62tjba2tt53\n7EmS7G2iMmS0sMN8T4abVrB7uKnLvlSHpKqPjwCWF7bfV0G7S6Mfgjrc5PYG6vYa/bXwQhluOgB4\nDBgHDAOWAi/r1OY04J7cHSo/7a4vMLdDYFwBXFvYfl/+O+6ycGHlMHDatP3zS2zo7a1Zs+ebygMP\nVJa5PbfXhYZ+LfTD9nbqaUhEpW1tImIG8EUqn5a6JTOvjYhLqkXcXG1zPTAD+CtwcWb+otS3uvwQ\n4E5gDLAaOC8zn+pi21mPfZCkF5KIIDO7HfqvS0j0J0NCknqvpyHhFdeSpCJDQpJUZEhIkooMCUlS\nkSEhSSoyJCRJRYaEJKnIkJAkFRkSkqQiQ0KSVGRISJKKDAlJUpEhIUkqMiQkSUWGhCSpyJCQJBUZ\nEpKkIkNCklRkSEiSigwJSVKRISFJKjIkJElFhoQkqciQkCQVGRKSpCJDQpJUZEhIkooMCUlSkSEh\nSSoyJCRJRYaEJKnIkJAkFRkSkqQiQ0KSVGRISJKKDAlJUpEhIUkqMiQkSUWGhCSpyJCQJBUZEpKk\nIkNCklRkSEiSigwJSVKRISFJKjIkJElFhoQkqaimkIiIloi4LyJWRsSiiGgutJsRESsi4tGIuKK7\n/hExLiK2RMQvqtONtdQpSdo3tR5JzAZ+mJkvBe4HruzcICKagOuB6cBE4PyImNCD/o9l5qTq9N4a\n65Qk7YNaQ+JMYF718TzgrC7aTAFWZebqzNwK3F7t113/qLE2SVKNag2JUZm5CSAzNwKjumhzFLC2\nw/y66jKA0Xvpf0x1qGlxRLymxjolSftgSHcNIuIHwOiOi4AEru6iedZYz87+G4Cxmbk5IiYB/x4R\nJ2bmX7rqNGfOnF2PW1tbaW1trbEMSWosbW1ttLW19bpfZO77+3pELAdaM3NTRBwBLM7Ml3VqcxIw\nJzNnVOdnA5mZc3vSv9pnMfAvmfmLLtZlLfsgSS9EEUFmdjusX+tw0wLgourjdwB3ddHmYeD46ieW\nhgEzq/2K/SPisOoJbyLiWOB44D9rrFWS1Eu1HkkcAtwJjAFWA+dl5lMRcSTwvzPzTdV2M4AvUgml\nWzLz2m76nw18AngO2AF8LDPvLdTgkYQk9VJPjyRqComBwJCQpN7bX8NNkqQGZkhIkooMCUlSkSEh\nSSoyJCRJRYaEJKnIkJAkFRkSkqQiQ0KSVGRISJKKDAlJUpEhIUkqMiQkSUWGhCSpyJCQJBUZEpKk\nIkNCklRkSEiSigwJSVKRISFJKjIkJElFhoQkqciQkCQVGRKSpCJDQpJUZEhIkooMCUlSkSEhSSoy\nJCRJRYaEJKnIkJAkFRkSkqQiQ0KSVGRISJKKDAlJUpEhIUkqMiQkSUWGhCSpyJCQJBUZEpKkIkNC\nklRkSEiSigwJSVKRISFJKjIkJElFhoQkqaimkIiIloi4LyJWRsSiiGgutJsRESsi4tGIuKLD8nMj\n4pGI2B4Rkzr1uTIiVkXE8oh4Qy11SpL2Ta1HErOBH2bmS4H7gSs7N4iIJuB6YDowETg/IiZUV/8G\neAvwfzv1eRlwHvAy4I3AjRERNdY6KLW1tfV3CX3K/RvcGnn/GnnfeqPWkDgTmFd9PA84q4s2U4BV\nmbk6M7cCt1f7kZkrM3MV0DkAzgRuz8xtmfl7YFX1eV5wGv0/qvs3uDXy/jXyvvVGrSExKjM3AWTm\nRmBUF22OAtZ2mF9XXbY3nfus70EfSVKdDemuQUT8ABjdcRGQwNVdNM861SVJGggyc58nYDkwuvr4\nCGB5F21OAhZ2mJ8NXNGpzWJgUqkNsBCYWqghnZycnJx6P/Xkfb7bI4luLAAuAuYC7wDu6qLNw8Dx\nETEO2ADMBM7vol3H8xILgNsi4vNUhpmOBx7qqoDMfEGe0Jak/aHWcxJzgWkRsRI4FbgWICKOjIjv\nAWTmduD9wH3AMionpJdX250VEWupHG18LyK+X+3zW+BO4LfAvcB7s3rYIEnaf8L3XklSScNccR0R\n/7N64d1vIuLa/q6nL0TEv0TEjog4pL9rqaeI+Ez1d7c0Ir4bEQf3d021Kl1A2ggi4uiIuD8illVf\nbx/o75r6QkQ0RcQvImJBf9dSbxHRHBHfqb7ulkXE1FLbhgiJiGgF3gz8fWb+PXBd/1ZUfxFxNDAN\nWN3ftfSB+4CJmflKKtfEPO+izMGkmwtIG8E24EOZORH4r8D7Gmz/dvoglSHvRvRF4N7MfBnwX6h8\nCKlLDRESwHuAazNzG0Bm/qmf6+kLnwf+V38X0Rcy84eZuaM6+1Pg6P6spw6KF5A2gszcmJlLq4//\nQuUNpqGuY6r+UXYa8NX+rqXeqkfqp2Tm1wGqFy0/U2rfKCFxAvDaiPhpRCyOiFf3d0H1FBFnAGsz\n8zf9Xct+8E7g+/1dRI325QLSQSkijgFeCfysfyupu51/lDXiSdvxwJ8i4uvV4bSbI2JEqXGtH4Hd\nb7q5qG8I0JKZJ0XEZCqfjDp2/1e577rZv49SGWrquG5Q2cv+XZWZd1fbXAVszcxv9UOJ6qWIOAiY\nD3ywekTRECLidGBTZi6tDmUPutdbN4YAk4D3ZeZ/RMQXqFyb9vFS40EhM6eV1kXEpcD/qbZ7uHpy\n99DMfGK/FVij0v5FxMuBY4BfVb/k8Gjg5xExJTP/uB9LrMnefn8AEXERlcP71++XgvrWemBsh/mj\nq8saRkQMoRIQ38zMrq6PGsz+ETgjIk4DRgAviohbM/Pt/VxXvayjMjLxH9X5+UDxwxWNMtz071Tf\nXCLiBGDoYAqIvcnMRzLziMw8NjPHU/kFv2owBUR3ImIGlUP7MzKzvb/rqYNdF5BGxDAqF5A22idk\nvgb8NjO/2N+F1FtmfjQzx2bmsVR+d/c3UEBQ/b69tdX3Sqhc41Y8QT9ojiS68XXgaxHxG6AdaJhf\naBeSxjv8/TIwDPhB9Rvhf5qZ7+3fkvZdZm6PiJ0XkDYBt+y8gLQRRMQ/Am8DfhMRv6Tyf/Kjmbmw\nfytTL3yAyrdaDAX+E7i41NCL6SRJRY0y3CRJ6gOGhCSpyJCQJBUZEpKkIkNCklRkSEiSigwJSVKR\nISFJKvr/rLc5ihPLYvQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105ca8160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "x1 = [-5, -4, -3, 3, 4, 5]\n",
    "y1 = np.linspace(0, 0, 6, endpoint=True)\n",
    "x2 = [-2, -1, 0, 1, 2]\n",
    "y2 = np.linspace(0, 0, 5, endpoint=True)\n",
    "plt.scatter(x1, y1, marker=\"o\", color=\"blue\", label = \"class 1\")\n",
    "plt.scatter(x2, y2, marker=\"x\", color=\"red\", label = \"class 2\")\n",
    "plt.legend(loc=\"upper right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "It cannot be done without having a high training error. Instead, let us tranform our data using the following mapping:\n",
    "\n",
    "$$f: (x, y) \\rightarrow (x, x^{2})$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAEACAYAAABBDJb9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF7VJREFUeJzt3X10FfWdx/H3N/IQKJJGFLAGQjw9SMG6yimoddlzqVvj\nUSv4AFIKq4Bu7Wp5KN2KWiWof3StaF1Xj7U+lG7VClYUWku0yi2wLfWhsoJFRLtAoBILoojQ8PTd\nP2YCBPJwL5mbm/zyeZ0zh3n4zcx3cnM/zP3dyYy5OyIi0rYV5LsAERFpPoW5iEgAFOYiIgFQmIuI\nBEBhLiISAIW5iEgAMg5zM+tsZn80szfMbKWZzYznF5vZC2a2xswqzawod+WKiEh9LJvrzM2sq7vv\nNLNjgP8BJgOXAVvd/U4zuwEodvcZuSlXRETqk1U3i7vvjEc7Ax0AB0YAc+L5c4CRiVUnIiIZySrM\nzazAzN4ANgMvuvurQC93rwZw981Az+TLFBGRxmR7Zr7f3c8ASoChZjaI6Oy8TrOkihMRkcx0OJqV\n3H27maWB84FqM+vl7tVm1hv4oL51zEwhLyJyFNzdmmqTzdUsx9deqWJmXYCvAquBBcBVcbMrgeca\nKSjYYebMmXmvQcenY9PxhTdkKpsz8xOBOWZWQPSfwFPu/ryZLQfmmtlEYD0wOottiohIAjIOc3df\nCQyuZ/6HwD8nWZSIiGRHfwGakFQqle8Scirk4wv52EDH115k9UdDzdqRmbfUvkREQmFmeJJfgIqI\nSOulMBcRCYDCXEQkAApzEZEAKMxFRAKgMBcRCYDCXEQkAApzEZEAKMxFRAKgMBcRCYDCXEQkAApz\nEZEAKMxFRAKgMBcRCYDCXEQkAApzEZEAKMxFRAKgMBcRCYDCXEQkAApzEZEAKMxFRAKQcZibWYmZ\nvWxmb5nZSjP7djx/ppltNLM/xcP5uStXRETqk82Z+V7gO+4+CDgbuN7MBsTL7nb3wfGwKPEqs1BZ\nCeedFw2VlfmsRERC0tqzpUOmDd19M7A5Ht9hZquBk+LFloPaslZZCZdcArt2RdPLlsH8+VBent+6\nRKRtawvZclR95mbWDzgd+GM863ozW2FmD5tZUUK1ZW327IM/bIjGZ8/OVzUiEoq2kC0Zn5nXMrNu\nwNPAlPgM/QHgNnd3M7sDuBuYVN+6FRUVB8ZTqRSpVOpoahYRCVY6nSadTme9nrl75o3NOgC/An7j\n7vfWs7wUWOjup9WzzLPZ19E4/KNQly6t76OQiLQ9+cwWM8Pdm+zKzjbMfwZscffvHDKvd9yfjplN\nA4a4+9h61s15mEP0Q6/9+DN9uoJcRJKRr2xJPMzN7BxgCbAS8Hi4CRhL1H++H1gHfNPdq+tZv0XC\nXEQkJDk5M28OhbmISPYyDXP9BaiISAAU5iIiAVCYi4gEQGEuIhIAhbmISAAU5iIiAVCYi4gEQGEu\nIhIAhbmISAAU5iIiAVCYi4gEQGEuIhIAhbmISAAU5iIiAVCYi4gEQGEuIhIAhbmISAAU5iIiAVCY\ni4gEQGEuIhIAhbmISAAU5iIiAVCYi4gEIOMwN7MSM3vZzN4ys5VmNjmeX2xmL5jZGjOrNLOi3JUr\nIiL1MXfPrKFZb6C3u68ws27A68AIYAKw1d3vNLMbgGJ3n1HP+p7pvkREJGJmuLs11S7jM3N33+zu\nK+LxHcBqoIQo0OfEzeYAI7MvV0REmuOo+szNrB9wOrAc6OXu1RAFPtAzqeJERCQzHbJdIe5ieRqY\n4u47zOzwvpMG+1IqKioOjKdSKVKpVLa7FxEJWjqdJp1OZ71exn3mAGbWAfgV8Bt3vzeetxpIuXt1\n3K++2N2/UM+66jMXEclS4n3msUeBP9cGeWwBcFU8fiXwXJbbFBGRZsrmapZzgCXASqKuFAduAl4B\n5gJ9gPXAaHf/qJ71dWYuIpKlTM/Ms+pmaQ6FuYhI9nLVzSIiIq2QwlxEJAAKcxGRACjMRUQCoDAX\nEQmAwlxEJAAKcxGRACjMRUQCoDAXEQmAwlxEJAAKcxGRACjMRUQCoDAXEQmAwlxEJAAKcxGRACjM\nRUQCoDAXEQmAwlxEJAAK82aqrITzzouGysp8VyPSfui9V5eeAdoMlZVwySWwa1c03aULzJ8P5eX5\nrUskdO3pvadngLaA2bMP/jJBND57dv7qEWkv9N47ksJcRCQACvNmmD49+nhXq0uXaJ6I5Jbee0fK\nuM/czB4BLgKq3f20eN5M4Brgg7jZTe6+qIH1g+szh6jvrvbj3fTpYfbZibRG7eW9l2mfeTZh/o/A\nDuBnh4X5J+5+dwbrBxnmIiK5lPgXoO6+DNhW376yKUxERJKXRJ/59Wa2wsweNrOiBLYnIiJZ6tDM\n9R8AbnN3N7M7gLuBSQ01rqioODCeSqVIpVLN3L2ISFjS6TTpdDrr9bL6oyEzKwUW1vaZZ7osXq4+\ncxGRLOXqj4aMQ/rIzaz3IcsuBVZluT0REUlAxt0sZvYEkAJ6mNkGYCYw3MxOB/YD64Bv5qBGERFp\ngu7NIiLSiuneLCIi7YjCXEQkAApzEZEAKMxFRAKgMBcRCYDCXEQkAApzEZEAKMxFRAKgMBcRCYDC\nXEQkAApzEZEAKMxFRAKgMBcRCYDCXEQkAApzEZEAKMxFRAKgMBcRCYDCXEQkAApzEZEAKMxFRAKg\nMBcRCYDCXEQkAApzEZEAZBzmZvaImVWb2ZuHzCs2sxfMbI2ZVZpZUW7KFBGRxmRzZv4YUH7YvBnA\nb939FOBl4MakChMRkcxlHObuvgzYdtjsEcCceHwOMDKhukREJAvN7TPv6e7VAO6+GejZ/JJERCRb\nHRLenje2sKKi4sB4KpUilUolvHsRkbYtnU6TTqezXs/cG83fuo3NSoGF7n5aPL0aSLl7tZn1Bha7\n+xcaWNez2ZeIiICZ4e7WVLtsu1ksHmotAK6Kx68EnstyeyIikoCMz8zN7AkgBfQAqoGZwLPAPKAP\nsB4Y7e4fNbC+zsxFRLKU6Zl5Vt0szaEwT0ZlJcyeHY1Pnw7lh18sKpIn+t3MDYV5gCor4ZJLYNeu\naLpLF5g/X28ayT/9buZOrvrMJY9mzz74ZoFovPZMSCSf9LuZfwpzEZEAKMzbkOnTo4+vtbp0ieaJ\n5Jt+N/NPfeZtjL5kktZKv5u5oS9ARUQCoC9ARUTaEYW5iEgAFOYiIgFQmIuIBEBhLiISAIW5iEgA\nFOYiIgFQmIuIBEBhLiISAIW5iEgAFOYiIgFQmIuIBEBhLiISAIW5iEgAFOYiIgFQmIuIBEBhLiIS\ngA5JbMTM1gEfA/uBPe4+NIntiohIZhIJc6IQT7n7toS2JyIiWUiqm8US3JaIiGQpqQB24EUze9XM\nrklomyIikqGkulnOcff3zewEolBf7e7LDm9UUVFxYDyVSpFKpRLavYhIGNLpNOl0Ouv1zN0TLcTM\nZgKfuPvdh833pPclIhI6M8Pdral2ze5mMbOuZtYtHv8McB6wqrnbFRGRzCXRZ94LWGZmbwDLgYXu\n/kIC25V8q6qCpUsPTi9dGs2T1k+vXbvT7D5zd/8/4PQEapHWZt06uPRSmDcvmh41Cp55Bvr0yWtZ\nkgG9du1O4n3mDe5IfeZtUzoNw4dH44sXg760bjv02gWhxfrMRUQk/xTm0rClS6OP54sXR8OoUXX7\nYaX10mvX7qibRRpWVRX1vQ4bFk0vXQr9+qnftS3QaxeMTLtZFOYiIq2Y+sxFRNoRhbmISAAU5iIi\nAVCYi4gEQGEuIhIAhbmISACSup+5iLQj/fr1Y/369fkuIyilpaWsW7fuqNfXdeYikrX42ud8lxGU\nhn6mus5cRKQdUZiLiARAYS4iEgCFuYhIABTmItIuzJkzh2G1d5EMkMJcWofQn1kZ+vG1EWZNXhSS\nmPvvv58hQ4ZQWFjIxIkTc74/hbm0DrXPrEyno+HSS6N5oQj9+DK0fXt06EVFUFYGL76Y74py56ST\nTuKWW25h0qRJLbI/hbm0DsOGRQ8fHj48GubNO/hghRCEfnyx11+HM86A3r1hzJgovA91xRXw/PPR\n/HXrYORIWL26bpvdu6GyEubPhy1bsq9h48aNXHbZZfTs2ZMTTjiByZMn19tu6tSp9O3bl6KiIoYM\nGcKyZcsOLHv11VcZMmQIRUVFnHjiiXz3u98FoKamhvHjx3P88cdTXFzMmWeeyd/+9rd6tz9y5Egu\nvvhijjvuuOwP4igozEUkERs3Rv9PrVgB1dXw7LPRWXgt9+hMvKbm4Lz9++Gllw5O79wJQ4bA5ZfD\nlVdC//5Hhn1j9u/fz0UXXURZWRkbNmxg06ZNjBkzpt62Q4cO5c0332Tbtm2MHTuWUaNGsXv3bgCm\nTJnC1KlT+fjjj3nvvfcYPXo0EPW7b9++nU2bNvHhhx/y4IMP0qVLl8wLzCGFubQOoT+zMvTjA15+\nOQrsWjU10aHG+YgZHJ57xxwTdbnU+tGPYM0a2LEDPvkEPvoIrrkm8xpeeeUV3n//fe68804KCwvp\n1KkTX/7yl+ttO3bsWD772c9SUFDAtGnTqKmpYc2aNQB06tSJd999l61bt9K1a1eGDh0KQMeOHdm6\ndSvvvPMOZsYZZ5xBt27dMi8whxIJczM738zeNrN3zOyGJLYp7Uy/fvDMM5BKRcMzz0TzQhH68QGf\n+cyR8445Bjoccgeoe+6Brl0PBnvfvtFZeK333qt75u4OGzZkXkNVVRWlpaUUFDQdbXfddRcDBw6k\nuLiY4uJitm/fzpa4X+eRRx5hzZo1DBgwgDPPPJNf//rXAIwfP57y8nLGjBlDSUkJM2bMYN++fZkX\nmEvu3qyB6D+Ed4FSoCOwAhhQTzsXaTU2bHBfsuTg9JIl0TzJSH3v51273AcOdC8sdAf3rl3db7vt\nyHUXL3a/9Vb3++5z//TTusvmzInWi2LcvVMn9yuuyLyuP/zhD96rVy/ft2/fEct++tOf+rBhw9zd\nfcmSJd6zZ09/6623DiwvLi72l1566Yj1nn76aS8sLPSdO3fWmb9+/XofOHCgP/roo43W9P3vf98n\nTJjQZO0NZWQ8v8ksTuLMfCiw1t3Xu/se4BfAiAS2K5I7urokcYWF8MorMGsWXHcd/PzncMstR7ZL\npaI2118fnaUfavx4mDQpOpvv2BG+9CX48Y8zr2Ho0KGceOKJzJgxg507d1JTU8Pvf//7I9rt2LGD\njh070qNHD3bv3s1tt93GJ598cmD5448/fuAsvaioCDOjoKCAdDrNqlWr2L9/P926daNjx44NfgrY\nt28ff//739m3bx979+6lpqYmt2fxmSR+YwNwGfDQIdPjgP+sp12T/zOJtKjFiw+eAi5enO9q2pRc\nv58//dR927ajW7eqqspHjhzpPXr08BNOOMGnTJni7nXPzPft2+cTJ0707t27++c+9zn/4Q9/6GVl\nZQfOzMeNG+c9e/b0Y4891k899VRfsGCBu7s/+eSTfsopp3i3bt28d+/ePnXq1Ho/Bbi7V1RUuJl5\nQUHBgWHWrFkN1t3Qz5QMz8ybfQtcM7sMKHf3f42nxwFD3X3yYe185syZB6ZTqRSpVKpZ+xZplnQ6\nuvwCom/q9PuYMd0CN3m1P9N0Ok06nT4wf9asWRndAjeJMD8LqHD38+PpGUT/k/zHYe1cL760GkuX\nRl0r8+ZF06NGRV9KBnjtdy4ozJPX3PuZJxHmxwBrgHOB94FXgK+7++rD2inMpfWoqor6yGvDe+nS\n6OqSPn3yWVWboTBPXt7DPN7Z+cC9RFe2POLuP6injcJcJBAK8+S1ijDPhMJcJBwK8+TpsXEiIqIw\nFxEJgcJcRCQACnMRkQAozEWkXdBj40REAtFSj43bvXs3V199Nf369aOoqIjBgwezaNGinO5TYS4i\nLef+++G116LxPXvgxhth69b81pQDe/fupW/fvixdupSPP/6Y22+/ndGjR7Mhm/v5ZklhLiLJ2LED\nbr89CmmAtWvhwQfrtikpgQsvhOXL4RvfgDffPPJG6Dt2HBzfvbvuDc4z0BoeG9e1a1duvfVW+sR/\nUXzhhRdSVlbG66+/ntWxZENhLiLJ6NDhYEivXg1f+UrdJ1MAjBgBDzwAZ58Nv/sd/PKX0b1zay1f\nDoMHw6ZNUZCPHh09fihDrfWxcdXV1axdu5ZBgwZlfCzZUpiLSDIKC6NwXrECBg6Em2+Gq6+u22bP\nHnjqqWi8pgZWraq7/KyzonXOOScaAKZNy7iE1vjYuL179zJu3Diuuuoq+vfvn/GxZEthLiLJqaqK\nHtwJ0UNBa7tcak2dCp9+Crt2wZw5UZfLX/96ZJv166O+9XvugU6dsth963psnLszbtw4OnfuzH33\n3ZfxcRyVTG56nsSAHk4hEox6389btriXlLj/5CfRM+QuuMB90qS6bf7yl2hZrdWr6y6vqXEfMSIa\n7rjD/fOfd9+4MeO6Wttj4yZMmODnnnuu19TUNFl7QxlJCz42TkQEevSA556Luklqu1wO7yIpK6vb\nRz5gQN3lq1ZFz5KbOzfqprnmGliwIOMSWtNj46699lrefvttFixYQKcsPl0cLYW5iCRn8OCD44WF\nkO0XfoMHwxNPHOxa+d734Fvfynj1goICFi5cyNq1a+nbty99+vRh7ty5R7QrLy+nvLyc/v37U1ZW\nRteuXQ9ceQKwaNEiBg0aRPfu3Zk2bRpPPfUUnTt3ZvPmzVx++eUUFRUxaNAghg8fzvjx44/Y/oYN\nG3jooYdYsWIFvXr14thjj6V79+48+eST2f08sqBb4IpI1nQL3OTpFrgiIqIwFxEJgcJcRCQACnMR\nkQAozEVEAqAwFxEJQIemm4iI1FVaWtpi9wZvL0pLS5u1frOuMzezmcA1wAfxrJvcvd47sOs6cxGR\n7LXkdeZ3u/vgeMjtozRasXQ6ne8Scirk4wv52EDH114kEeb6rEX4v1AhH1/IxwY6vvYiiTC/3sxW\nmNnDZlaUwPZERCRLTYa5mb1oZm8eMqyM//0a8ABwsrufDmwG7s51wSIicqTEbrRlZqXAQnc/rYHl\n+vZTROQoZPIFaLMuTTSz3u6+OZ68FFjVUNtMihERkaPT3OvM7zSz04H9wDrgm82uSEREstZi9zMX\nEZHcafE/5zezb5vZ6viL1B+09P5zzcymm9l+Mzsu37UkyczujF+3FWb2SzPrnu+akmBm55vZ22b2\njpndkO96kmRmJWb2spm9Fb/fJue7pqSZWYGZ/cnMMn+2XBthZkVmNi9+371lZmc21r5Fw9zMUsDX\ngC+6+xeBu1py/7lmZiXAV4H1+a4lB14ABsVXLq0FbsxzPc1mZgXAfwHlwCDg62Y2oPG12pS9wHfc\nfRBwNnBdYMcHMAX4c76LyJF7gefd/QvAPwCrG2vc0mfm3wJ+4O57Adx9SwvvP9fuAf4930Xkgrv/\n1t33x5PLgZJ81pOQocBad1/v7nuAXwAj8lxTYtx9s7uviMd3EIXBSfmtKjnxydMFwMP5riVp8Sff\nYe7+GIC773X37Y2t09Jh3h/4JzNbbmaLzexLLbz/nDGzi4Eqd1+Z71pawETgN/kuIgEnAVWHTG8k\noLA7lJn1A04H/pjfShJVe/IU4hd/ZcAWM3ss7kZ6yMy6NLZC4ndNNLMXgV6HziL6YX8/3l+xu59l\nZkOAucDJSdeQK00c201EXSyHLmtTGjm+m919YdzmZmCPuz+RhxLlKJhZN+BpYEp8ht7mmdmFQLW7\nr4i7b9vc+60JHYDBwHXu/pqZ/QiYAcxsbIVEuftXG1pmZtcCz8TtXo2/KOzh7luTriMXGjo2MzsV\n6Af8r0X3BS0BXjezoe7+QX3rtEaNvXYAZnYV0cfar7RIQbm3Ceh7yHRJPC8YZtaBKMj/292fy3c9\nCToHuNjMLgC6AMea2c/c/V/yXFdSNhJ90n8tnn4aaPQL+pbuZnmWOAjMrD/Qsa0EeWPcfZW793b3\nk929jOiFOKMtBXlTzOx8oo+0F7t7Tb7rScirwOfNrNTMOgFjgNCuingU+LO735vvQpLk7je5e193\nP5nodXs5oCDH3auBqjgnAc6liS96W/rhFI8Bj5rZSqAGCOaHfxgnvI999wGdgBfjhxIsd/d/y29J\nzePu+8zseqIrdQqAR9y90SsG2hIzOwf4BrDSzN4g+r1s8JkD0upMBh43s47AX4AJjTXWHw2JiARA\nzwAVEQmAwlxEJAAKcxGRACjMRUQCoDAXEQmAwlxEJAAKcxGRACjMRUQC8P/yyCiNAGg2xwAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1061367f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x1 = np.array([-5, -4, -3, 3, 4, 5])\n",
    "y1 = np.square(x1)\n",
    "x2 = np.array([-2, -1, 0, 1, 2])\n",
    "y2 = np.square(x2)\n",
    "plt.scatter(x1, y1, marker=\"o\", color=\"blue\", label = \"class 1\")\n",
    "plt.scatter(x2, y2, marker=\"x\", color=\"red\", label = \"class 2\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now our data is linearly separable! The purpose of a kernel is to transform the data into linearly separable dimensions. The kernel we are going to use in this tutorial will be the radial basis function, which projects the features into infinite dimensional space. \n",
    "\n",
    "$$K(x^{(i)}, x) = e^{-\\frac{||x^{(i)} - x ||^{2}}{2\\sigma^{2}}}$$\n",
    "\n",
    "Where $x^{(i)} \\in \\mathbb{R}^{n x 1}$ is a vector representing training sample $i$ in the dataset with $n$ features and $x \\in \\mathbb{R}^{n x 1}$ could be a vector representing a test/prediction sample. \n",
    "\n",
    "A kernel function $K$ can also be interpreted as a similarity measure. For example, vectors that are similar will produce a large $K(x^{(i)}, x)$, while those that are dissimilar will produce a small $K(x^{(i)}, x)$. In regular logistic regression, our functional margin is:\n",
    "\n",
    "$$z^{(i)} = y^{(i)}f(x^{(i)}) = y^{(i)}x^{(i)}\\beta$$\n",
    "\n",
    "With kernel logistic regression with a quadratic kernel, the $f(x^{(i)})$ is now replaced by:\n",
    "\n",
    "$$f(x^{(i)}) = \\sum_{j=1}^{m}\\alpha_{i}K(x^{(i)}, x^{(j)})$$\n",
    "\n",
    "Where $m$ is the number of training samples and $\\alpha$ is the new weight we will be training. Notice that $\\alpha \\in \\mathbb{R}^{m x 1}$ instead of $\\beta \\in \\mathbb{R}^{n x 1}$ in regular logistic regression. Returning to the concept of viewing the kernel as a similarity measure, we can observe that what the kernel is doing is boosting the weight $\\alpha_{j}$ if $x^{(i)}$ and $x^{(j)}$ are similar.\n",
    "\n",
    "Let us now derive the stochastic gradient descent update equation for $\\alpha_{i}$ using the same logistic loss for one sample $j$:\n",
    "\n",
    "$$ L^{(j)} = ln(1 + e^{-y^{(j)}\\sum_{i = 1}^{n}\\alpha_{j}K(x^{(i)}, x^{(j)})})$$\n",
    "\n",
    "With $z^{(j)} = y^{(j)}\\sum_{i = 1}^{n}\\alpha_{j}K(x^{(i)}, x^{(j)})$\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial \\alpha_{i}} = \\frac{e^{-z^{(i)}}}{1 + e^{-z^{(i)}}} -y^{(i)}K(x^{(i)}, x^{(i)}) = -\\frac{1}{1 + e^{z{(i)}}}y^{(i)}(x^{(i)T}x^{(i)} + 1)^{2}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can further improve our regression by incorporating ridge regression. The purpose of ridge regression is to prevent overfitting the model to the training data. We do so by incrementally decreasing the size of the weights $\\alpha$ by a factor $\\gamma$ during stochastic gradient descent. Given the learning rate $\\eta$ our update equation is thus:\n",
    "\n",
    "$$\\alpha_{i} := \\alpha_{i} - \\gamma \\alpha_{i} - \\eta \\frac{\\partial L}{\\partial \\alpha_{i}}$$ \n",
    "$$\\alpha_{h} := \\alpha_{h} - \\gamma \\alpha_{h} \\text{ for } h \\ne i$$\n",
    "\n",
    "We will be implementing kernel logistic ridge regression using the same email dataset as before. As always, let us load up the dataset and preprocess it via standardization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scipy import io\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "data = io.loadmat('datasets/spam.mat')\n",
    "Xtrain = data['Xtrain']\n",
    "Ytrain = data['Ytrain']\n",
    "m = Xtrain.shape[0]\n",
    "idx = np.random.permutation(m)\n",
    "Xtrain = data['Xtrain'][idx, :]\n",
    "Ytrain = data['Ytrain'][idx, :]\n",
    "Xvali = Xtrain[round(0.9*m):m, :]\n",
    "Yvali = Ytrain[round(0.9*m):m, :]\n",
    "Xtrain = Xtrain[:round(0.9*m), :]\n",
    "Ytrain = Ytrain[:round(0.9*m), :]\n",
    "\n",
    "def standardize(a):\n",
    "    return (a - np.mean(a)) / np.std(a)\n",
    "\n",
    "XtrainP = np.apply_along_axis(standardize, 0, Xtrain)\n",
    "XvaliP = np.apply_along_axis(standardize, 0, Xvali)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now implement kernel logistic ridge regression (KLRR):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "class KLRR():\n",
    "    def __init__(self, rate=2.5, iterations=2, kernel=\"quadratic\", gamma=1e-5, sigma=1):\n",
    "        self.rate = rate\n",
    "        self.iterations = iterations\n",
    "        self.kernel = kernel\n",
    "        self.gamma = gamma\n",
    "        self.sigma = sigma\n",
    "    def train(self, Xtrain, Ytrain, Xvalidate=None, Yvalidate=None):\n",
    "        def kernel(self, xData, xsample):\n",
    "            diff = np.sum(np.square(xData - xsample))\n",
    "            return np.exp(-diff / (2*(self.sigma)**2))\n",
    "        def clamp(x):\n",
    "            if x <= 0:\n",
    "                x = max(x, -10000)\n",
    "            else:\n",
    "                x = min(x, 10000)\n",
    "            return x\n",
    "        def gradient(y, x, Xtrain, w):\n",
    "            kernels = list(map(lambda u: kernel(self, u, x), Xtrain))\n",
    "            kernels = np.array(kernels).reshape(1, len(kernels))\n",
    "            z = y*np.dot(kernels, w)\n",
    "            z = clamp(z)\n",
    "            S = 1 / (1 + np.exp(z))\n",
    "            gradient = -S*y\n",
    "            return gradient\n",
    "        counter = 0\n",
    "        self.Xtrain = Xtrain\n",
    "        self.trainIterations = []\n",
    "        self.trainErrors = []\n",
    "        self.validateErrors = []\n",
    "        self.validateIterations = []\n",
    "        self.w = np.array([1e-10]*Xtrain.shape[0])\n",
    "        kernelMatrix = np.empty([Xtrain.shape[0], Xtrain.shape[0]])\n",
    "        for i in range(len(Xtrain)):\n",
    "            rowKernel = []\n",
    "            for j in range(len(Xtrain)):\n",
    "                k = kernel(self, Xtrain[i], Xtrain[j])\n",
    "                rowKernel.append(k)\n",
    "            kernelMatrix[i] = rowKernel\n",
    "        z0 = np.dot(np.diag(Ytrain[:, 0]), kernelMatrix)\n",
    "        if Xvalidate is not None and Yvalidate is not None:\n",
    "            kernelMatrix2 = np.empty([Xvalidate.shape[0], Xtrain.shape[0]])\n",
    "            for i in range(len(Xvalidate)):\n",
    "                row = []\n",
    "                for j in range(len(Xtrain)):\n",
    "                    k = kernel(self, Xtrain[j], Xvalidate[i])\n",
    "                    row.append(k)\n",
    "                kernelMatrix2[i] = np.array(row)        \n",
    "        for r in range(self.iterations):\n",
    "            for i in range(len(Xtrain)):\n",
    "                y = Ytrain[[i]]\n",
    "                x = Xtrain[[i], :]\n",
    "                g = gradient(y, Xtrain[i], Xtrain, self.w)\n",
    "                self.w[i] = self.w[i] - self.gamma * self.w[i] - self.rate * g\n",
    "                self.w[:i] = self.w[:i] - self.gamma * self.w[i]\n",
    "                self.w[i+1:len(self.w)] = self.w[i+1:len(self.w)]\n",
    "                - self.gamma*self.w[i+1:len(self.w)]\n",
    "                z = np.dot(z0, self.w)\n",
    "                z = np.array(list(map(clamp, z)))\n",
    "                error = np.sum(np.log(1 + np.exp(-z)))\n",
    "                self.trainErrors.append(error)\n",
    "                counter += 1\n",
    "                self.trainIterations.append(counter)\n",
    "                if Xvalidate is not None and Yvalidate is not None and counter % 100 == 0:\n",
    "                    z = np.dot(np.diag(Yvalidate[:, 0]), np.dot(kernelMatrix2, self.w))\n",
    "                    self.validateErrors.append(np.sum(np.log(1 + np.exp(-z))))\n",
    "                    self.validateIterations.append(counter)\n",
    "    def plot(self):\n",
    "        if len(self.validateIterations) == 0:\n",
    "            plt.plot(self.trainIterations, self.trainErrors)\n",
    "            plt.xlabel(\"Iterations\")\n",
    "            plt.ylabel(\"Cross Entropy\")\n",
    "            plt.title(\"Cross Entropy vs Iterations\")\n",
    "            plt.show()  \n",
    "        else: \n",
    "            fig = plt.figure()\n",
    "            ax = fig.add_subplot(121)\n",
    "            ax.plot(self.trainIterations, self.trainErrors, color=\"blue\", \n",
    "                    label=\"Training\")\n",
    "            plt.title(\"Cross Entropy vs Iterations\")\n",
    "            plt.legend(loc=\"upper right\")\n",
    "            ax = fig.add_subplot(122)\n",
    "            ax.plot(self.validateIterations, self.validateErrors, color=\"red\",\n",
    "                    label=\"Validate\")\n",
    "            plt.title(\"Cross Entropy vs Iterations\")\n",
    "            plt.legend(loc=\"upper right\")\n",
    "            plt.show()\n",
    "    def predict(self, x):\n",
    "        predictions = []\n",
    "        for sample in x:\n",
    "            diff = np.sum(np.square(self.Xtrain - sample), axis=1)\n",
    "            kernel = np.exp(-diff / (2*(self.sigma)**2))\n",
    "            f = np.dot(kernel, self.w)\n",
    "            predictions.append(1 / (np.exp(-f) + 1))\n",
    "        return predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now let us train and plot the training and validation cross entropy with the number of iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEKCAYAAAD5MJl4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecVOXVwPHfoYsUAeldilIVRCyorBpBiIomqKhBjFgx\nr9gSRV9lTTRKfDWAhqixggKKFQOoELPGhiCCDUQiIE2QXkQFds/7x7nDzja2zcydmT3fz4cPM8/c\nufPM7DNz7tNFVXHOOeeiVQo7A84555KPBwfnnHMFeHBwzjlXgAcH55xzBXhwcM45V4AHB+eccwV4\ncHApQUT+LiK3h50P52ItWct2UgQHEblIROaLyE4RWSsiM0SkT4j5eUpEfhaRHcG/nSKysITPHS0i\nE+Odx0QK3tOkqPs5InJYHF9vmIi8G52mqteo6j3xes148bKd3LxsFy304CAiNwIPAncDjYBWwN+A\ns4o4vnKCsjZGVesE/2qrao9YnVhEJFbnSiAt4naplPDvJ+V5jWThZTtleNkujKqG9g+oA+wEfnWA\nY0YD04BJwDbgMqAaMBZYC6wB/gpUDY5vALwObAU2A+9EneuW4PgdwBLglCJe8yngj0U81hrIAS4B\nvgW+B24LHusP/Bz82wksDNL/jf1AvAf8ABwGNAVeC/L4NXB5Ie95apDXj4FuwWM3Ay/my9N44K+F\n5PUPwLR8aeOAscHtS4Fvgtf4BrjwAH+DicHtd4L3vyt43nlB+pnAwuBzfy+S3+CxFUFePgV+xC5K\nbgH+G5zjC+Cc4NgjgmP2Bp/hlsL+JsAVwDJgE/Aq0DTqsRzgquBz3QI8HPVYOyArKEvfA1O8bHvZ\n9rJdyGcTjy9GKb5A/YE9QKVivkA/A2cF92sAfwQ+wL4sDYD3gbuCx/8MTAj+SJWBPkF6R2AV0Di4\n3wpoW44v0KPYF7k78BNweP7CFvWcfwMrg8JRCagSFMSHgKrAkcEfMyPfez43eA83AcuD202CglUn\nOLYysAE4qpC8tgoK+sHB/UrAOuAYoCawHWgfPNYY6FTcFyiqgLaNut8jyEMv7MpoKPalifyorQA+\nAZoB1YO0X0f9Lc4L8hm5Pwz4T1F/E+BUYGPwuVXFfkDeyZe/6UBtoGXw2fYLHpsMjApuVwNO8LLt\nZdvLdsF/YTcrNQA2qWpOMcd9qKqvA6jqT8BF2Bdms6puBu7C/mhgUbkp9gfOVtX3g/Rs7APrKiJV\nVHWVqq44wGv+XkS2iMjW4P+noh5TIFNV96jqZ9hVw5HFvIenVfWr4L02AU4AblHVvar6KfA4dsUW\nsUBVX1HVbKxpogZwnKquB/6DFTqAAcBGVV2U/wVVdRVWcM8Nkk4DflDV+VGfSTcRqaGqG1R1STHv\nIVp088EVwCOq+rGaSdgPwHFRx4xT1XWq+nOQt5dUdUNwexp2pdS7hK99EfCEqn6qqnuBUcDxItIq\n6ph7VXWnqq7GfsCOCtL3Aq1FpHnw9/ugFO+5NLxse9lO6bIddnDYDBwqIsXlY3W++82wK6WIb4M0\ngPuxauRbIvJfEbkFQFW/Aa4HMoENIjJZRJoe4DXvV9X6qlov+P+3+R7fEHV7N1CrFO+hGVal3J3v\nPTQv7Hi1S4E1Ue9xIvCb4PbFWLNEUaYAFwa3L8SuLghe+wLgGuA7EXldRA4v5j0UpTVwU/BDs0VE\ntgItovJLkP/9ROQSEVkY/EBtBboAh5bw9ZphnxfBe/kBK0vRn19Rf5/fY+V+noh8LiL5/66x4mU7\n73vwsl0ySVO2ww4OH2JR+JxijtN899dif7SI1liVElXdpao3q2o74GzgRhE5JXhsqqqeFPXc+8qZ\n/5LktbD0dUB9ETk4Kq0V9r4iWkZuBJ18LYLngbVDdheRLlh76HMHyM80IENEmmNXWZP3Z0h1tqr2\nw672lgL/OMB5DmQ1cE/wQxP50amlqs9HHbP//QdXQY8BI4Jj6wFfknvFVtRnGLGOqL9/8Dk2IN+X\ntDCq+r2qXqmqzYGrgQlxGp3iZTuXl+0ULNuhBgdV3YG1+f1NRAaJyEEiUkVEBojIgQr3VOB/ReRQ\nETkUuIPgCkNEfiki7YLjdgL7gBwR6Sgip4hINawt+Ees/a4sDjQiYwPQ5kCjNlR1DdaufK+IVBeR\n7sBw8l4lHS0i5wQjIG7A2n7nBs//GXgJ+zJ8FJyvqNfahLUBPwUsV9WlACLSSETOFpGaWHV0F1YV\nL4n1WMdjxD+Aq0Wkd3Dug0VkYL4fiGgHY5/9JhGpFFzhdI16fAPQQkSqFvH8KcBvRaS7iFTH2uLn\nBtXsAxKRwcGPCVjHXQ5lLwdF8rLtZTvVy3bYNQdU9UHgRuB/sc6VVcAI7AqiKHdjoxwibaIfA5Fx\nwh2AOSKyE+vM+5uqvgNUx66mNmLRuSHWnleUP0jeseDfR2c7/9uIuj0N+4JtFpGPizgerBrcNsjL\nS8AdqvrvqMdfw6rGW7Hq9blBG23EM0A3rBpenMlYm2z0VVgl7HNfi42KOBmrhpdEJjAxqGYPVtUF\nWNvswyKyBRtJMSzq+DzvP2j/fQD7QViPVbvfizrkbexqa32+zz3y/H9hP5ovB/lvCwwp6vXyOQb4\nSER2YGXsOlVdWdwbLgsv2162SeGyLdbkd4ADRFpgf6TGWBR6TFUfEpHR2IcWeYO3qeobwXNGYcPy\n9gEjVfWtIL0n8DTWATVTVa8va8bTWfDZtlPVSw5wTEtsyGITVd2VsMw5Vw5etlNHlRIcsw+4UVUX\niUgtYIGIzA4eezC4OtpPRDoB5wOdsLbEOSLSIeh4+jswXFXni8hMEemvqm/G7u1UDGKdnDcBU/3L\n49KJl+3kUWxwCIaXrQ9u7xKRJeT2nBfW9jgI+8PuA1aKyDKgt4h8C9SOGmo2Eeus8+BQCkE76gZs\nfPWAkLPjXMx42U4uJak57CcibbAxtR8BJwK/E5GhWLvoTaq6HQscH0Y9bW2Qto+8Pe5ryDs8ywVU\n9a4DPLYbmwDjXMrxsp06StwhHTQpvYj1IezCZmoepqpHYTWLB+KTReecc4lWopqDiFTBAsMkVX0N\nQFU3Rh3yD2zNF7CaQsuox1oEaUWlF/Z6xY0Fdq5cVDXhC8R5uXaJEKuyXdKaw5PAYlUdF0kQkSZR\nj/8KW2AKbN2PISJSTUTaAu2BeUHfxXYR6R2Mk74EG9JWKI3xWjejR4+O+Tn9vKmXV9Vwf59T6XNK\npfOmUl7jdd5Akc12pVVszUFs7fmLgc/F1n1X4DbgIhE5ChveuhJbKRBVXSwiLwCLsQkoIzQ359eS\ndyjrG7F6I84552KnJKOV3sdWR8yvyB92Vb0XuLeQ9AXY5BbnnHNJLPQZ0omSkZHh543TeVMpr+ko\n1T5/L4NxLdtZsTpRsTOkwyAimoz5culBRNCQOqS9XLt4iXW5LtU8B1d2bdq04dtvvy3+QBczrVu3\nZuXKlWFnI+152U6sRJVrrzkkSBDVw85GhVLUZ+41h9jysp1YiSrXFabPwTnnXMl5cHDOOVeABwfn\nnHMFeHBwMZeTk0Pt2rVZs6bYnQ1LdaxzifLtt99SqVIlcnJsI7WBAwcyaVLh21nnPzZdeHBw1K5d\nmzp16lCnTh0qV65MzZo196dNmTKl1OerVKkSO3fupEWLFjE91rnSGDBgAJmZmQXSX3vtNZo2bVrs\nj3n0bqgzZ85k6NChJTr2QN555x1atmxZ/IFJwIODY+fOnezYsYMdO3bQunVrZsyYsT/twgsvLHB8\ndnZJt+N1LjzDhg3j2WefLZD+7LPPMnToUCpVSvzPn6qWOJCEzYODyyPfIl4A3HHHHQwZMoSLLrqI\nunXr8txzzzF37lyOP/546tWrR/PmzRk5cuT+oJGdnU2lSpVYtWoVAEOHDmXkyJEMHDiQOnXq0KdP\nn/3j4ktzLMCsWbM4/PDDqVevHtdddx0nnngiEyeWZKthV9Gcc845bN68mffey93Cedu2bfzzn/9k\n6NChzJw5k549e1K3bl1at27NXXcVvWbdKaecwpNPPglYU+jNN99Mw4YNad++PTNmzMhz7NNPP03n\nzp2pU6cO7du357HHHgNg9+7dDBw4kHXr1u2vma9fvx5V5b777qN9+/Y0bNiQIUOGsG3btjh8IqXj\nwcGVyKuvvspvfvMbtm/fzgUXXEDVqlUZP348W7Zs4f333+fNN9/k0Ucf3X98/qujKVOmcM8997B1\n61ZatmzJHXfcUepjv//+ey644AIeeOABNm3aRNu2bZk/fz7OFaZGjRqcd955eS4enn/+eTp16kS3\nbt04+OCDmTRpEtu3b2fGjBk88sgjTJ8+vdjzPvbYY8ycOZNPP/2Ujz/+mBdffDHP440bN2bmzJns\n2LGDp556ihtuuIFFixZRs2ZNZs2aRbNmzfbXzJs0acL48eOZPn067777LuvWraNevXqMGDEi5p9H\naXlwSCIi5f8XLyeeeCIDBw4EoHr16hx99NEcc8wxiAht2rThiiuu4J133tl/fP7ax+DBg+nRoweV\nK1fm4osvZtGiRaU+dsaMGfTo0YMzzzyTypUrc8MNN9CgQYN4vWUXK7Eo2GUs3MOGDWPatGns2bMH\ngEmTJjFs2DAA+vbtS5cuXQDo2rUrQ4YMyVOGizJt2jSuv/56mjVrxiGHHMKoUaPyPD5gwADatGkD\nwEknnUS/fv149913izzfo48+yj333EPTpk2pWrUqd955Jy+++GLoHdy+fEYSSeZJpvk70ZYuXcpN\nN93EggUL2L17N9nZ2Rx77LFFPr9Jk9ztP2rWrMmuXUXvHV/UsevWrSuQD+/ITgEhFuw+ffrQsGFD\nXn31VXr16sX8+fN55ZVXAPjoo48YNWoUX3zxBXv27GHPnj2cd955xZ4zfzls3bp1nsdnzZrFH//4\nR77++mtycnL48ccf6d69e5Hn+/bbbzn33HP394GoKlWrVmXDhg00bdq0LG87Jrzm4Eokf9PPVVdd\nRbdu3Vi+fDnbt2/nrrvuivsSCk2bNmX16tV50tauLXQzwXIRkRYi8raIfCkin4vI/wTpfxGRJSKy\nSEReEpE6Uc8ZJSLLgsf7xTxTrsyGDh3KM888w7PPPkv//v1p2LAhABdffDHnnHMOa9euZdu2bVx1\n1VUlKsP5y2F0n9iePXsYPHgwf/jDH9i4cSNbt25lwIAB+89bWGd0q1atmDVrFlu2bGHLli1s3bqV\nH374IdTAAB4cXBnt3LmTunXrctBBB7FkyZI8/Q3xcuaZZ7Jw4UJmzJhBdnY2Y8eOZdOmTfF4qX3A\njaraBTge+J2IHAG8BXRR2zd9GTAKQEQ6A+cDnYABwARJlSEpFcAll1zCnDlzePzxx/c3KQHs2rWL\nevXqUbVqVebNm8fkyZPzPK+oQHH++eczfvx41q5dy9atWxkzZsz+xyI1kEMPPZRKlSoxa9Ys3nrr\nrf2PN27cmM2bN7Njx479aVdddRW33Xbb/kEZGzduLFHfR7x5cHB5lPQ37YEHHuDpp5+mTp06XHPN\nNQwZMqTI8xR3zpIe26hRI55//nluuOEGDj30UFasWEGPHj2oXr16ifJcUqq6XlUXBbd3AUuA5qo6\nR1UjDcFzsX3QAc4GpqrqPlVdiQWO3jHNlCuz1q1bc8IJJ7B7927OPvvs/ekTJkzgjjvuoG7dutx9\n991ccMEFeZ5XVLm84oor6N+/P0ceeSS9evXi17/+9f7HatWqxfjx4znvvPOoX78+U6dOZdCgQfsf\nP/zww7nwwgs57LDDqF+/PuvXr2fkyJEMGjSIfv36UbduXU444QTmzZsXj4+iVHxV1gTxlStjLycn\nh2bNmvHSSy/Rp0+fAo/HYvVKEWmDbaDSNQgUkfTpwBRVnSIiDwEfqurk4LHHsW1wX853LtWff4aT\nT4YPPoAQxtnHg5ftxErUqqzeIe1Syptvvslxxx1HjRo1uPfee6lWrRq9e8fnIl1EagEvAiPzBYbb\ngb2qWurp45l//jN8+SXcdBMZgwb5bneuzLKyssjKyorb+b3mkCB+dRUbd9xxBxMmTCA7O5suXbrw\n0EMP0bNnz0KPLc8VlohUAf4JzFLVcVHplwJXAKeq6s9B2q2AquqY4P4bwGhV/SjfOa1c9+0Lo0fD\nqaeW4p0nLy/biZWomoMHhwTxL1DilTM4TAQ2qeqNUWlnAA8AJ6vq5qj0zsBzwLFAc2A20CF/Id5f\nri+/HHr1gquvLse7Sx5ethPLm5WcC4mI9AEuBj4XkYWAArcD44FqwOygg3Kuqo5Q1cUi8gKwGNgL\njDjg1U3HjrBsWZzfhXPl48HBuXxU9X2gciEPdTjAc+4F7i3RC3ToAFHr/TiXjNJjuIRzqaRDB/j6\n67Bz4dwBec0hQVq3bp0yS/Wmi/zLGiSNdu1g5UrYtw+qpP5X0Mt2YiWqXHuHtKtwYt1xV4rXzS3X\nrVvDv/8Nhx2W6Gy4NBXrcu3NSs6FwZuWXJLz4OBcGDp08BFLLql5cHAuDB4cXJLz4OBcGDp29GYl\nl9Q8ODgXBq85uCTno5VchZMUo5X27IHatWHnTqhWLdFZcWnIRys5lw6qVYMWLWDFirBz4lyhPDg4\nF5bmzWHdurBz4VyhPDg4F5amTeG778LOhXOF8uDgXFiaNoX168POhXOFKjY4iEgLEXlbRL4Ukc9F\n5LogvZ6IvCUiS0XkTRGpG/WcUSKyTESWiEi/qPSeIvKZiHwtImPj85acSxFNmnjNwSWtktQc9gE3\nqmoX4HjgWhE5ArgVmKOqhwNvA6Ng/8Yn5wOdgAHABMldlevvwHBV7Qh0FJH+MX03zqUSrzm4JFZs\ncFDV9aq6KLi9C1gCtAAGAc8Ehz0DnBPcPhuYqqr7VHUlsAzoLSJNgNqqOj84bmLUc5yreLzPwSWx\nUvU5iEgb4ChgLtBYVTeABRCgUXBYc2B11NPWBmnNgTVR6WuCNOcqJm9WckmsxIvJi0gt4EVgpKru\nEpH8s9RiOmstMzNz/+2MjAwyMjJieXpXgWRlZZGVlRV2NgryZiWXxEo0Q1pEqgD/BGap6rggbQmQ\noaobgiajf6tqJxG5FVBVHRMc9wYwGvg2ckyQPgToq6rXFPJ6PkPaxU1SzJAGyMmBGjVslnT16onO\njkszYc2QfhJYHAkMgenApcHtYcBrUelDRKSaiLQF2gPzgqan7SLSO+igviTqOc5VPJUqQaNGsGFD\n2DlxroBim5VEpA9wMfC5iCzEmo9uA8YAL4jIZVit4HwAVV0sIi8Ai4G9wIioy6VrgaeBGsBMVX0j\ntm/HuRQT6ZRu1SrsnDiXhy+85yqcpGlWAjjrLBg+HM7xgXuufHzhPefSiXdKuySVtMHhxx/DzoFz\nCeBzHVySStrg4N8XVyE0aeI1B5eUkjY4vPRS2DlwLgG85uCSVNIGh4MPDjsHziWABweXpJI2OKxe\nXfwxzqU8b1ZyScqDg3NhatLEJsHl5ISdE+fySNrgsGZN8cc4l/KqV4datWDLlrBz4lweSRscvObg\nKgzvd3BJKGmDw/LlXtN2FYQHB5eEkjY4AKxYEXYOnEsA75R2SShpg0NGhgcHV0E0buwrs7qkk7TB\noX17+O9/w86FcwnQqBF8/33YuXAuDw8OzuUjIi1E5G0R+VJEPheR64L0eiLylogsFZE3RaRu1HNG\nicgyEVkiIv1K9YJec3BJKGmDQ8OG8MADYefCVVD7gBtVtQtwPHCtiBwB3ArMUdXDgbeBUQAi0hnb\nz6QTMACYEGxoVTIeHFwSStrg8Mtf2v9794abD1fxqOp6VV0U3N4FLAFaAIOAZ4LDngEimzCcDUxV\n1X2quhJYBvQu8Qs2buzNSi7pJG1waNzYBnH4d8aFSUTaAEcBc4HGqroBLIAAjYLDmgPRM3PWBmkl\n41uFuiSUtMEBfPi3C5eI1AJeBEYGNYj82xPGZrvCRo1g40af2OOSSrF7SIepYUNYuxZ69Qo7J66i\nEZEqWGCYpKqvBckbRKSxqm4QkSZApF67FmgZ9fQWQVoBmZmZ+29nZGSQkZEB1apB7dq2hMahh8b4\nnbh0lZWVRVZWVtzOn9R7SIvAkUfCokVh58ilk5LstSsiE4FNqnpjVNoYYIuqjhGRW4B6qnpr0CH9\nHHAs1pw0G+iQf8PoA+6N3qmTbWLSuXN53pqrwCrUHtIPPgiffhp2LlxFIyJ9gIuBU0VkoYh8IiJn\nAGOA00VkKXAacB+Aqi4GXgAWAzOBEUVHgSJ4v4NLMkldc1iyxC6k9uyBqlXDzpVLF7G+wirF6xYd\nM84/H371KxgyJLGZcmmjQtUcOnWy/wcODDcfzsWdz3VwSSapgwNA3bowZ07YuXAuznyug0sySR8c\nHnkk7Bw4lwDe5+CSTNIHh7597f8k7BpxLna8WcklmaQPDk2bQuXKvny3S3MeHFySSfrgAJCdDY89\nFnYunIsj73NwSSapZ0hHjBoFBx0Udi6ci6NIn4MqlGJBV+fiJSVqDg0bwp13wr59YefEuTg5+GBr\nP925M+ycOAekSHA47TT7/z//CTcfzsWV9zu4JJISwaF7d+jZE3bvDjsnzsWR9zu4JJISwQGgTRv4\n+eewc+FcHPlcB5dEUiY41KgBP/0Udi6ciyNvVnJJJKWCw48/hp0L5+KoRQt47TXYvDnsnDhXfHAQ\nkSdEZIOIfBaVNlpE1gRLGUeWM448NkpElonIEhHpF5XeU0Q+E5GvRWRsaTMqAlOmlPZZzqWQ66+H\nI46ALl3g5ZfDzo2r4IpdsltETgR2ARNVtXuQNhrYqaoP5ju2EzAZOAbbDWsOwaYnIvIR8DtVnS8i\nM4FxqvpmEa9ZYGnjqVPhwgutU9rnPLjySMolu6PNm2dLES9YAK1bxz9jLi0kfMluVX0P2FpYXgpJ\nGwRMVdV9qroSWAb0DrZUrK2q84PjJgLnlCajF1xg/z/zTGme5VwK6t3bahHXXRd2TlwFVp4+h9+J\nyCIReVxE6gZpzYHVUcesDdKaA2ui0tcEaSUmAg8/DK+84ovwuQrg97+HpUutD8K5EJQ1OEwADlPV\no4D1wAOxy1LRfvELeOst6NUrEa/mXIiqV4fx4+H228POiaugyrS2kqpujLr7D+D14PZaoGXUYy2C\ntKLSi5SZmbn/dkZGBhkZGRx+uN33yXCuNLKyssjKygo7G6XXty8sWwZ79/o+uS7hSrSHtIi0AV5X\n1W7B/Saquj64fQNwjKpeJCKdgeeAY7Fmo9nkdkjPBa4D5gMzgPGq+kYRr1dkx90zz8C//gUTJ5bq\nfTq3X9J3SEdr08YKfLt2ccmTSx+xLtfF1hxEZDKQATQQkVXAaOAUETkKyAFWAlcBqOpiEXkBWAzs\nBUZEfRuuBZ4GagAziwoMxTnoIJg7tyzPdC4FtWsH33zjwcElXIlqDol2oCusLVtsrtCmTVCzZoIz\n5tJCStUcrrwSevSAa66JT6Zc2kj4UNZkU7++zZS+6aawc+JcAkRqDs4lWMoFB4Brr4U6dcLOhXMJ\n4MHBhSQlg0PPnvDSSz7fwVUAHhxcSFKuzwFss6w6dWxlgZUrE5cvlx5Sqs9hxw5o1swKvW8f6g6g\nwvc5ANSubRsA1asXdk6ci7M6dWyIni/l7RIsJYMDwNNPw6JFdjGVhJUf52InumnJN1J3CZKywaFH\nD/gsWET8l78MNy/OxVUkOGRnQ7duMH9+8c9xrpxSNjiAfU9uuw2aNg07J87FUbt28N//wvTp8NVX\n8OGHYefIVQApHRzA1id78smwc+FcHEVqDmPHwimnwMKFYefIVQApHxwuuwwaNAg7F87FUbt2MHu2\nBYg//cmDg0uIlA8OzZvblrubNoWdE+fipF072LgRRoywST5ffw179oSdK5fmUj44iNj+Dv37w5df\nhp0b5+KgSRM4+WRbZ+mgg+Cww7ywu7hL+eAAtif7J59A164+rNWlIRF45x049FC7f9RR3rTk4i4t\ngsP998PZZ9vtFSvCzYtzcdejhwcHF3dpERyaNLGtdvv2tRF/zpWHiDwhIhtE5LOotCNF5EMRWSgi\n80SkV9Rjo0RkmYgsEZF+cc9gjx42A9S5OEqL4BDRty+88krYuXBp4Cmgf760vwCjVbUHtuHV/QDB\n7ofnA52AAcAEkTgvgnTUUfDpp5CTE9eXcRVbWgWHyy6zbUTXrw87Jy6Vqep7wNZ8yTlA3eD2IeTu\ngX42MFVV96nqSmAZ0DuuGaxf3/75aq0ujordJjSVtG5tGwE1bQq7dsHBB4edI5dGbgDeFJEHAAFO\nCNKbA9FTltcGafHVowfMmwcdOsT9pVzFlFbBAWy+w6GHwlVXwbPPhp0bl0auAUaq6qsiMhh4Eji9\ntCfJzMzcfzsjI4OMjIyy5WbIEJsxfdFFvpR3BZWVlUVWVlbczp+S+zkUZ8UKGwq+ZYsv6+0KKsm6\n9yLSGnhdVbsH97ep6iFRj29T1UNE5FZAVXVMkP4G1jfxUSHnLFe5ziM7G7p0gQkT4NRTY3NOl9J8\nP4cSaNvWhra+9lrYOXEpTIJ/EWtFpC+AiJyG9S0ATAeGiEg1EWkLtAfmxT13lSvbqpN33x33l3IV\nU1rWHACmTYORI20RS99v2kUr7gpLRCYDGUADYAM2OmkpMB6oDPwEjFDVhcHxo4DhwF6s6emtIs4b\nu5oDwN69cPjhMGkS9OkTu/O6lBTrmkPaBgewZTW6dYOnnopBplzaSKltQoszdix88QU8/nhsz+tS\njjcrlcKIET4U3KW5Pn3g44/DzoVLQ2kdHI48EiZO9JqDS2Pdu9sqrT/+GHZOXJpJ62YlsH67nBzb\nerdy5Zic0qW4tGpWAjj6aPjb3+C442J/bpcyvFmplLKzoXdvmDEj7Jw4Fye9ennTkou5tA8OAJde\nagM6pkyBYcPCzo1zMebBwcVB2jcrge0U16sXrFxp95PwLbsESrtmpYULYehQG7XkKixvViqDBg18\nKQ2Xxrp0sWUBdu2CH37wYa0uJipEcAAb8XfzzXbbL7BcWqlWzbZBXLQIrr8errgC3n8/7Fy5FFch\nmpWiDR4ML73kTUsVWdo1KwFcey0sXWptp9dcA//6F8ycGZ/XcknJm5XK6bHHbDG+JUvCzolzMdSr\nF2RlWfsp8Nj4AAAZfklEQVTp734Hn38OCxbYOO6XXoKt+bencO7AKlzNASxAXHWVXVgNGBC3l3FJ\nKi1rDtu3w4cfwhln2P1x42D6dAsOCxbYcgH33Zd7/JIl8Oijdnvs2PjkySWUr60UA6pQqVLubVex\npGVwyG/3bjjpJBvFNGiQ1SyWLbMd5O67z4LHWWfBnDmwfHli8uTiyoNDjPzwA9SqZRdbPrG0YqkQ\nwSG/yy+HFi3g9NPh17+22kS9ehYsfvzRNwxKAwnvcxCRJ0Rkg4h8FpVWT0TeEpGlIvKmiNSNemyU\niCwTkSUi0i8qvaeIfCYiX4tI6PXYyBaixx/vy9K4CuDWW+Hhh+Hii22oa/PmULOm/du0KezcuSRU\nkg7pp4D++dJuBeao6uHA28AoABHpDJwPdAIGABNE9l+S/B0YrqodgY4ikv+cCacKp5xiM6j37g07\nN87FUfv2cM45cN55cOaZuenNm8PateHlyyWtYoODqr4H5B/qMAh4Jrj9DHBOcPtsYKqq7lPVldhu\nWb1FpAlQW1XnB8dNjHpOqK66Cl54AW6/PeycOBdn//gH3H9/3rQWLTw4uEKVdShrI1XdAKCq64FG\nQXpzYHXUcWuDtObAmqj0NUFa6M47z/6//374+edw8+JcXBXWr+A1B1eEWM1zSL5e7RKqVMmal846\nCx58MOzcOJdgzZvDmjXFH+cqnCplfN4GEWmsqhuCJqPvg/S1QMuo41oEaUWlFykzM3P/7YyMDDIy\nMsqY1ZIZNw4OOww6dLBZ1C59ZGVlkZWVFXY2klOLFjB3bti5cEmoRENZRaQN8LqqdgvujwG2qOoY\nEbkFqKeqtwYd0s8Bx2LNRrOBDqqqIjIXuA6YD8wAxqvqG0W8XihD/u66CzIzYds2qFu32MNdiqqQ\nQ1mLMnMmjB8PbxT6VXQpJIyhrJOBD7ARRqtE5LfAfcDpIrIUOC24j6ouBl4AFgMzgRFR34ZrgSeA\nr4FlRQWGMI0eDR07wiGHwAknhJ0b5xLAm5VcESrsJLiizJ9vO8eBz55OV15ziLJ5sw1z9bWXUp4v\nvBdnxxyTe3vBgvDy4VxC1K8PP/1kSwY4F8WDQyH27oV77rFhrg8+6P11Lo2JQLNmPpzVFeDBoRBV\nqsCoUdCmDdx0ky2x4Vza8olwrhAeHIogAm++mbuhlu//4NKWT4RzhfDgcABVq9qopSZNYNKksHPj\nXJz4iCVXCA8OJZCZ6f0OLo15s5IrhAeHEhgyBN55B667LuycOBcH3qzkCuHBoQTq1oWnn4aHHoIv\nvww7N87FWP7gsGsXvPeeb3RSwfkkuFL4zW+sBrF6dfHHuuTlk+DyWb0auna11Sc/+wy++cY63CZM\ngIsuCjt3roR8m9AQrVwJbdvCJ59Ajx55H9u+3VZ4rV07lKy5UvDgkE9ODvzlL9C0qQWJrl3hz3+2\n9D/9KezcuRKKdbku66qsFVKbNvDEE3D22RYgGjbMfWzwYFi6FFatCi17zpVNpUq2jWi0I46Al14K\nJz8uKXifQylddhmcf75tLZqTk5u+cKE3N7k00qkTfPVV2LlwIfJmpTLYu9cW51u0CD7/3GrhkU22\ndu2CatWsydYlJ29WKoHdu6FBA9i505YMcEnPF95LAlWrwowZdvv22/PWINq0seDgUpeIPCEiG0Tk\ns3zp/yMiS0TkcxG5Lyp9lIgsCx7rl/gcx0HNmjb7c8WKsHPiQuLBoYyaNbPVjr/5BipXzk3fsye8\nPLmYeQroH50gIhnAWUC3YNOr/wvSOwHnA52AAcAEkcI2a05BnTr5ujEVmAeHcqhfH15+OW/ajh3h\n5MXFjqq+B+Tf4OAa4D5V3RccsylIHwRMVdV9qroSWAb0TlRe48qDQ4XmwaGcOnYsvCN6377E58XF\nVUfgZBGZKyL/FpGjg/TmQHQJWBukpb4jjvBO6QrMg0MMtGhRcNc4r0GknSrYXunHAX8ApoWcn/gr\nquawe3fi8+ISzochxFBOjg0ZB7j/frj33nDz42JqNfAygKrOF5FsEWmA1RRaRR3XIkgrVGZm5v7b\nGRkZZGRkxCOvsREJDqq5w/EeeQTuvNMm9dSrF27+KrisrCyysrLidn4fyhoHtWvDsGHw8MNh58QV\npiRD/kSkDfB60PmMiFwJNFfV0SLSEZitqq1FpDPwHHAs1pw0G+hQWAFOyXLdsKEtqdG0qU3m6dfP\ndr9q2xbGjQs7dy6KD2VNAS++CI89Bl98UbC5KTMz9yLMJScRmQx8AHQUkVUi8lvgSeAwEfkcmAxc\nAqCqi4EXgMXATGBE6kWAA4jUHr77zvbNffhhePJJmDLFCrhLW15ziANV6NYtdwXX6Ldy8snw7rvW\nBOVBIhw+Ca4UrrrKJvXs2mVr1v/xj5b+8MPwyiswZ44X5CThNYcUIJI7cxpg06bcxyIL9vnS3y4l\n/P73tl7999/nBgaAq6+G5cttkTGXljw4xEkkQFxxhTXbRvai3rs393Hnkl779vCLXxSc9l+lCgwf\nDv/4Rzj5cnHnwSHOxo2DRo3gxBNtRnVk/sPPP4ebL+fK7dJL4YUX4Icfws6JiwMPDnF20EG5k+Q6\ndsy90PLg4FJeixbQp48FCJd2PDgkQLVq1sS0ZUtu2s6d4eXHuZi5/HJ4/PGwc+HiwINDgnTtmtsJ\nfeyxMH9+uPlxLiYGDrQtEt98M+ycuBjzoawhmDXLLrhWrPDlvcPgQ1lj7N134Ve/siGvvdNjzcFU\n5HtIp4F9+3I3A0rjt5m0PDjEwT//aVc8H3wAhx2W97GcHLsSatcunLxVED7PIQ1UqWJL0xx0kO3r\n7iu4upR35pk2eumRRwo+Nn489OqVd1esiK1brb112za/UkoyHhxC0rGjzS265RabeOpcyrvwQls7\nJvpH/ptv4O677Ypo2bK8x+fkwKmnwtln28in3/8+sfl1B+TBIUTnn29rmj3/PNx1l6Vt3AiTJoWb\nL+fKpHt32xZx4UK7n5NjTU2jRlkQmDcv7/HPPgs1asB//wuLF9uaTT/+mPh8u0J5cAhZt27w4Ycw\nYQLcc481M11ySdi5cq4MRGDwYKs9APz977b3w/XXW0d1dHD48Uf43/+FBx6w57VqBUcfDa+9Fk7e\nXQEeHJJAx452sfXii/B//2dp3vzqUtLgwTBtmtUEMjOtGly5so3f/uij3OPGjrWAccIJuWmXXgrP\nPJPoHLsi+GilJBI9ium992zyqYs9H60UR6o2Kik7G26/Ha680tJ/+MEWGdu61Y5p2dIK+eGH5z53\n925o3tyWAm+eHjutJlJSjVYSkZUi8qmILBSReUFaPRF5S0SWisibIlI36vhRIrJMRJaISL/yZj7d\nVKkC27fbhdeAAdYU61xKEYGLL7bRSVdckZt+8MFWRf70U1tuo1evvIEBoGZNq3k8+2xi8+wKVa6a\ng4gsB45W1a1RaWOAzar6FxG5Bdt399aoHbOOwbZSnEM67ZgVY488YhdeN94IdevaxVjLlvb92rbN\nAkn9+mHnMjV5zSHOsrNtv9z8Sw9feaV1sj33nBXus84q+Ny337YO7OgmKFciSTUJTkRWAL1UdXNU\n2ldAX1XdICJNgCxVPUJEbgVUVccEx80CMlW1QCmoMF+iYsyda6MAP/zQAsS+fbmL+LVqZasWLF0K\nRxxR8LmbN1vw8KXBC/LgEJInnrA5D9u22V4QlSsXPOaHH2wZ4y1boHr1xOcxhSVVsxKgwGwRmS8i\nlwdpjVV1A4CqrgcaBenNsU3aI9YGaa4Ixx1nE083b7bv0qpVtprrtm12u1Il28Vx40Y7ft8+a9Jd\nsAAOPRRuuinc/DuXR+/eNnb76qsLDwxgzU8dOljzU36ffhrf5Yy3b4/fuVNQlXI+v4+qficiDYG3\nRGQpFjCilelSKTMzc//tjIwMMjIyyprHtFKtmv37+Wf46SfrtB461L5r//mP7eYYsXixbf+7erUd\nd/DB4eU7TFlZWWRlZYWdDde5s21sMnz4gY+LjGyKXqdp1SorxI88Ar/5jaWtXg1r19pVVHmo2lr6\n111nw227dy/f+dJEzEYrichoYBdwOZAR1az0b1XtVEiz0hvAaG9WKp/Fi+1i7Lvv4PTTrVZx0km2\nmsE//2nHtG1rS9u0aWPBokYNS3/0UZtTcdBBoWU/FN6slOSeeAKysnJng6rCoEHWhpqRYQUX4Oab\nrfN62TKoXbtsr6VqE/XmzYNateCGG2x2agpKmmYlEakpIrWC2wcD/YDPgenApcFhw4DIrJbpwBAR\nqSYibYH2QL4pk660OneGIUOsTHftahdmn3wC06fDokW2Leny5RYUVq60QPDKK/bcq6+2kVHxsHp1\n8cc4V6j8cyJefdUCwMSJVj2OmDMHmjSBMWPK/lqffGKBaO5cu6ryIYL7lbnmEPzAv4I1G1UBnlPV\n+0SkPvAC0BL4FjhfVbcFzxkFDAf2AiNV9a0izu1XWHEwYQK8/LJ9D1q2hK++svRXX4Uzzoht/58I\nrF8PjRvH7pyx4jWHJJedDfXq2dVM9eo24uLZZ+3Kp0ED+PprK2Dt21s/xNFH2yzSVq1K/1q33GJD\n/+65x2ok8+ZZzSUFJdVopXjxL1F8rVoFr79uP97Z2fCvf9n3bds2ezyyb3xkUp6IPadly5KdX9U6\ny1evtvXUko0HhxRwyilw6632Y/3llzB1qqUPHGjNQHv32pDY6dNh9Gg7Ztq00g3PU7XlxV99FY48\n0r4If/qT1SRSUNI0K7nU1aoVXHutfQ/+/Gerwf/nPzBypD3+xBP2416tWu7F2Ikn2vBzEXjqqQOf\nPzKgJCfHvuObNx/4eOcK6N3b1lkaOxbuvTc3/aSTbHOhOXPgF7+wtFtusVrGPfcUf94vv4T337fb\nH39shTzSAd2una0i6wAPDi7QrZt9D/fssR/3hQttfkWXLlazz8zMbe697DJb6aAokbXTHn7YLsIe\neqjoY4cNs6XLncvj2GNt4b7f/tZGVEScfLIVxNmzc4NDzZpWFX78catNFGbtWlu76ZRT4JxzrGA+\n/7x1PkdqGy1b2rhwXxkW8GYlV0qqNqJw3TobHXXccdYRfuyxucf86U9w552593/5y9yRU9H27Mnt\n50jkn9ublVLA+vW2zPf771v/Q8TPP9v9Qw6xH/zoZqQvvrAq7urVeUcvLVpkw/eGDrWmqgULbBSH\niDUlde2ae+zhh9uIjc6dc9O+/trmQBxzTPzebwx4s5ILlYitlzZrFjRrZrX7006z2yJWA4kODFD0\n3vOR5qdzz41vnl0KatLEmoCiAwPY1cQxx1itIX//QteudrXyxhu5aVlZ0K8f/PWv1jxVt64FnQkT\nrDmpS5e854huWlq/3qrJPXrktrlWIOWdBOcqoMjM7MgGRV98YTWDWrXsYq52bZtwN2eOPb5vn03O\n27EDmjbN/U5HgsNPPyX+PbgUUFTn8v/8jy2xUZhf/cqG5J13nhW8Sy+1+RL9++c9bvBg+5dfdHC4\n804bsfHZZ9Czp1VvK9B6NB4cXLl17Zq3Zh5t506rGTRrZrevvNK+16efbiMRwZbRKUoF+z66kijs\nRz1i0CD4wx/simPmTFv6O39gOJBIcMjOts6zDz6wtJo1Yc2akg/ZSwPerOTiqnZt+47Onm37uHz3\nnS0m2Ldv7pL98+bZAJScnIJ9D5Uq2VptzpVI48bWXPSvf8G4caVvDmrf3oLD3Ll2rnbtLL1LF2vm\nqkA8OLi4q1bNOqwvucSGpatak9PAgTaze+xYG4TSqpUFg7lzLVDk5NjzX3893Py7FHPuuXYFsnx5\n6Tu02rWzWdKvvGKjmiK6drX20wrEg4MLxWmnwYwZ8OCDtt7Z/PnW9ATW5NS8ee4ci2++se+5cyVy\n7rl2hTFiRO7WiiXVtq3N+Hz55byBpajgkJNjVzpDh1pHeceOcN995ct/kvChrC4pffMNTJ5sE2OP\nPdaW1enVy+ZEFLZ/RWn4UNYK4O67LTiUZUesli2tCrtyZW6H14cf5l7FRPzwg7WPZmfbfIzjjrMO\ntCuugG+/tXMkkC+f4SqkLVts8cDu3cu+AGeEBwd3QKecYgVt3LjctB07rGq7Y4f96KtaO2nlyrZk\nQPSoia5dbZ2mBG8C7/McXIVUv75918obGJwr1rXXwjXX5E2rU8cK4cqVdv/vf7chrhMmFBxOd8EF\nNvu6pCZNKjg5KAl4cHDOuWiDBxfedhnpd/jqK/sxf/FFG+Ka3wUX2CKA2dkFH9uwwZZAjqzvBPDY\nYzbRLsl4cHAuHxF5QkQ2iMhnhTx2k4jkBEvTR9JGicgyEVkiIv0Sm1uXMJHgcPXVFhw6dCj8uI4d\nbYZ39N4TYJsVHX+8rSg7erSlLVlio6POPDO+eS8DDw7OFfQUUGDmlIi0AE7H9imJpHUCzgc6AQOA\nCSI+bS8tdeli465377ampwMZMgSmTMm9v22bLdtx++22vMeyZTbB5/HHbRZ3aUdVJYAHB+fyUdX3\ngK2FPPRX4Pf50gYBU1V1n6quBJYBvfM/0aWBrl1tZMRjj1lH9IEMG2ZzJT4LKp93320Te4YPt0Bw\n8822/sykSbY/RRLy4OBcCYjI2cBqVf0830PNgehNUdcGaS7d9OxpK7wedVTxxzZpYgv9DR9uzUlP\nP23LFUcMH27DYrt1y52FnWR8bSXniiEiBwG3YU1K5ZKZmbn/dkZGBhkZGeU9pUsUkaIXESvM8OHW\ntHTyyXDTTRYwImrWtJFOTZuWOTtZWVlkxXHXOp/n4CqckowHF5HWwOuq2l1EugJzgN2AAC2wGkJv\n4DIAVb0veN4bwGhV/aiQc3q5rmiWL4frr4cXXoAaNeL6Uj4JzrlyKmFwaIMFh26FPLYC6KmqW0Wk\nM/AccCzWnDQb6FBYAfZy7eLJJ8E5F2ciMhn4AOgoIqtE5Lf5DlGsBoGqLgZeABYDM4ERHgFcOvCa\ng6twfPkMl4685uCccy7uPDg455wrwIODc865Ajw4OOecK8CDg3POuQI8ODjnnCvAg4NzzrkCPDg4\n55wrwIODc865Ajw4OOecK8CDg3POuQI8ODjnnCvAg4NzzrkCEh4cROQMEflKRL4WkVsS/frOOeeK\nl9DgICKVgIeB/kAX4EIROSIRrx2v7fT8vKmV13SUap+/l8H4nVdEMmJ1rkTXHHoDy1T1W1XdC0wF\nBiXihVPtj5xK502lvKajVPv8vQzGtWxnxOpEiQ4OzYHVUffXBGnOOeeSiHdIO+ecKyCh24SKyHFA\npqqeEdy/FVBVHZPvON9L0cVVWNuEJvo1XcUTq7Kd6OBQGVgKnAZ8B8wDLlTVJQnLhHPOuWJVSeSL\nqWq2iPwOeAtr0nrCA4NzziWfhNYcnHPOpQhVTZp/wBnAV8DXwC0lOP4JYAPwWVRaPaxmshR4E6gb\n9dgoYBmwBOgXld4T+Cx43ceBt4Evgc+B62J03oeAj4CFwXlHx+i8Y7Fa2CfA9BiecyXwaZDfeTE8\nb11gWnDcl8CxMTjv00E+Pwn+3w5cF4v8hlG2iU+5Hgu0wMu2l+0Slu3QA0JUxisB/wVaA1WBRcAR\nxTznROAo8n6JxgB/CG7fAtwX3O4cfLhVgDbBa0VqTh8BxwS35wDXBLdrBR/+ETE470zg7OB2ZWAu\nNu8jFud9BHiW3C9QLM75HVAv3+cdi/O+Bfw2uF8F+0LF4rz9o8rROqBlLM+byLJNfMr1TGAIcJSX\nbS/blKBshx4Uov44xwGzou7fSslqD63J+yX6Cmgc3G4CfFXY+YBZWGRvAiyOSh8C/D3q/qvAL2J5\nXqAm8DFwTAzOOwKbO5JB7hcoFnndATTI91mX97y/BbYX8jeM5WfbD3g3HmUhkWWbOJdrL9tetvOX\nh/z/kmmeQ6wmyDVS1Q0AqroeaFTE+dcGac2D1yrwuiLSBruCm4v9Icp9XhFZCKwHZqvq/BicdzDw\nDaBRabHIa2VgtojMF5HLY3TeHCBHRJ4SkU9E5DERqRmj/EbKygXA5Bh+DrGYpBmLsh2zcg1etvGy\nXWwZTKbgEC9a/CEFiUgt4EVgpKruKuQ8ZTqvqvbA2n57i0iX8pxXRH4JbMWuhA40trkseX1PVXsC\nA4FrReSkQs5T2vNGqtp/C879A3alE5PPVkSqAmdj7b6FnadM501SZX4vXra9bJdEMgWHtUCrqPst\ngrTS2iAijQFEpAnwfdT5WxZy/sLS12Ffnkmq+loMz7sWQFV3AFlYJ2V5ztsH6ItVu6cAp4rIJGB9\nDPK6PMjrRqz5oXcMPoMqwC5V/Ti4/xLWSRarz3YAsEBVNwXpMfublVMsynZM3ouIVMHLtpftkpTB\nA7U5JfIfVtWLdNpVwzrtOpXgeW2Az6PujyFoc6PwjppqQFvydtREOtAE66iZAzyY73XKe97ZwK+C\n9IOA/2BXLrHI7xnYFynSLvuXcp7zDeCcIP1g4H2svTMWef0M6Bg8Njo4Z6w+gynAsBiXhTPCKNvE\np1yfAUzEy7aX7RKU7dCDQr5CegY2gmIZcGsJjp+MXeX/DKzCOoXqYT/uS7HRA4dEHT8q+LDyD/E6\nGhuCtwx4AcjGvsCRIWRnAPXLed5ng3MtCgrR7cEx5T3vuCAt+gtU3nM+FfX+P4/8LWKRV+BIYH5w\n/pexqngszlsT2AjUjjomJp9toss28SnX47CrcS/bXrZLVLZ9EpxzzrkCkqnPwTnnXJLw4OCcc64A\nDw7OOecK8ODgnHOuAA8OzjnnCvDg4JxzrgAPDs455wrw4OCcc66A/wc8JgexZ7oBFwAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14e85cf28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "klrr = KLRR(iterations=2, sigma=1.5)\n",
    "klrr.train(XtrainP, Ytrain, XvaliP, Yvali)\n",
    "klrr.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We see that as training error decreases, validation error does not increase, indicating that we are not over-fitting. Again let us predict on our validation dataset and evaluate how the model performed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.    0.92  0.39  0.29  0.52  0.5   0.01  0.8   0.76  0.5   0.34  0.5\n",
      "  0.02  0.5   0.86]\n",
      "[[-1  1 -1 -1  1 -1 -1  1  1 -1 -1  1 -1  1  1]]\n"
     ]
    }
   ],
   "source": [
    "predictions = klrr.predict(XvaliP)\n",
    "print(np.round(predictions[100:115], 2))\n",
    "print(np.transpose(np.array(Yvali[100:115])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, samples that are marked 1 (spam) are predicted to be spam with higher probability. Those marked -1 (ham) are predicted to be spam with lower probability. Let us formally evaluate the model with AUC. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.936121781192\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "fpr , tpr, thresholds = metrics.roc_curve(Yvali, predictions, pos_label=1)\n",
    "auc = metrics.auc(fpr, tpr)\n",
    "print(auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, the AUC supports the usage of a kernel logistic ridge regression in this dataset! This concludes the regression tutorial!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}