yhilpisch/01_bc_day_04.md

## 01_bc_day_04.md

      
    Raw
  

              01_bc_day_04.md
            
          
    For Python Quants Bootcamp

This is the Gist the For Python Quants Bootcamp in London 24. November 2017 (http://fpq.io)
This fourth boocamp day is about Python for Algorithmic Trading (II).

Topics


prediction with deep learning
streaming data & visualization
Oanda trading platform
deployment and automation

Beware the Robots

https://www.youtube.com/watch?v=fRj34o4hN4I
The Python Quants

http://twitter.com/dyjh
team@tpq.io | http://fpq.io
http://pyalgo.tpq.io | http://certificate.tpq.io


## 02_neural_network_python.ipynb
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://hilpisch.com/tpq_logo.png\" width=\"350px\" align=\"right\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple Neural Networks in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&copy; Dr. Yves J. Hilpisch\n",
    "\n",
    "The Python Quants GmbH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from pylab import plt\n",
    "plt.style.use('seaborn')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network for Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Regression & Neural Networks](https://www.cs.cmu.edu/afs/cs.cmu.edu/academic/class/15381-s06/www/nn.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear OLS Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(0, 10, 5)\n",
    "y = 3 * x + 2.5 + np.random.standard_normal(len(x)) * 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAD3CAYAAADSftWOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAD71JREFUeJzt3X+MZWddx/H3dG47dcwsTuwVwTQ2\nEffbNJUaitAtlNnIDy0Vy5JViSlKV3Qx/WOXXUJpaTUhNYhh1y2QYrOwthYJpINjgKS0/0gtIDGW\nCm1397stkShRzFin7MDo4Eyvf9y7ZSgzO3fO3B/z3Pt+JZvc85z7nPN9sjuffeY5594z0mg0kCSV\n5Zx+FyBJ2jjDW5IKZHhLUoEMb0kqkOEtSQWq9epEs7PzlW9rmZwcZ25uoZPlbHmOefAN23jBMW9U\nvT4xsta+Imbetdpov0voOcc8+IZtvOCYO6mI8JYk/TDDW5IKZHhLUoEMb0kqkOEtSQUyvCWpC8Zm\nppmc2gG1GpNTOxibme7o8Xt2n7ckDYuxmWm27d3z7HbtxONs27uH08Dirt0dOYczb0nqsPEjh1Zv\nv/1wx85heEtSh42eOrmh9ioMb0nqsOXtF2+ovQrDW5I6bGH/wdXb9x3o2DkMb0nqsMVduzl95zGW\nLrkUajWWLrmU03ce69jFSvBuE0nqisVdu1nctZt6fYK52fmOH9+ZtyQVaN2Zd0SMAkeBAJaB64ER\n4C6gATwG3JCZz3SvTEnSSu3MvN8AkJmvAP4IONz6c0tmXkUzyK/tWoWSpB+x7sw7M/82Ij7X2vxZ\n4D+Ba4AHW233Aa8DZs52nMnJ8U19KXm9PlG5b6kc8+AbtvGCY+6Uti5YZuZSRNwN7AJ2A7+WmWce\nazYPPG+9Y2zm0Uf1+gSzXVjw38oc8+AbtvGCY67Sdy1tX7DMzN8FttNc//6xFbsmgKcrVSZJqmTd\n8I6It0TETa3NBeAZ4J8iYmer7Wrgoe6UJ0laTTvLJn8D/GVE/D1wLrAfOAEcjYjzWq87+12HkqSz\naueC5feA31xl11Tny5EktcMP6UhSgQxvSSqQ4S1JBTK8JalAhrckFcjwlqQCGd6SVCDDW5IKZHhL\nUoEMb0kqkOEtSQUyvCWpQIa3JBXI8JakAhneklQgw1uSCmR4S1KBDG9JKpDhLUkFWvcZlhFxLnAM\nuAgYA24DvgV8Fnii9baPZOanulSjJOk52nl6/HXAU5n5loj4SeAR4L3A4cw81NXqJEmraie87wWm\nV2wvAZcDERHX0px978/M+S7UJ0laxUij0WjrjRExAXwGOEpz+eTrmflwRLwHmMzMd56t/9LScqNW\nG91svZI0TEbW2tHOzJuIuBCYAe7IzE9ExE9k5tOt3TPAh9Y7xtzcQjunWlW9PsHs7HBN7B3z4Bu2\n8YJjrtJ3LevebRIRzwceAG7MzGOt5vsj4mWt168GHq5UmSSpknZm3jcDk8CtEXFrq+0AcCQivg98\nG/iDLtUnSVrFuuGdmfuAfavsurLz5UiS2uGHdCSpQIa3JBXI8JakAhneklQgw1uSCmR4S1KBDG9J\nKpDhLUkFMrwlqUCGtyQVyPCWpAIZ3pJUIMNbkgpkeEtSgQxvSSqQ4S1JBTK8JalAhrektozNTDM5\ntYMLXjDJ5NQOxmam+13SUGvr6fGShtvYzDTb9u55drt24nG27d3DaWBx1+7+FTbEnHlLWtf4kUOr\nt99+uMeV6Ix1Z94RcS5wDLgIGANuA44DdwEN4DHghsx8pmtVSuqr0VMnN9Su7mtn5n0d8FRmXgVc\nDXwYOAzc0mobAa7tXomS+m15+8Ubalf3tRPe9wK3rtheAi4HHmxt3we8psN1SdpCFvYfXL1934Ee\nV6IzRhqNRltvjIgJ4DPAUeADmfnCVvsvA3sy87qz9V9aWm7UaqObLFdS33zyk/C+98Hx43DJJXDT\nTfDmN/e7qkE3staOtu42iYgLgRngjsz8RET82YrdE8DT6x1jbm6hnVOtql6fYHZ2vnL/EjnmwVfc\neF99TfPPShusv7gxd8BmxlyvT6y5b91lk4h4PvAAcGNmHms1PxIRO1uvrwYeqlSZJKmSdmbeNwOT\nwK0RcWbtex/wwYg4DzgBeLe+JPXQuuGdmftohvVzTXW+HElSO/yQjiQVyPCWpAIZ3pJUIMNbkgpk\neEtSgQxvSSqQ4S1JBTK8JalAhrckFcjwlqQCGd6SVCDDW5IKZHhLUoEMb0kqkOEtSQUyvCWpQIa3\nJBXI8JakAhneklQgw1uSCtTO0+MBiIiXA+/PzJ0R8RLgs8ATrd0fycxPdaNASdKPaiu8I+JdwFuA\n77WaXgIczsxD3SpM6raxmWnGjxxi9NRJlrdfzML+gyzu2t3vsqS2tDvz/gbwJuCe1vblQETEtTRn\n3/szc74L9UldMTYzzba9e57drp14nG1793AaDHAVYaTRaLT1xoi4CPhkZl4REdcDX8/MhyPiPcBk\nZr7zbP2XlpYbtdropguWOuLFL4ZHH129/Wtf63090upG1trR9pr3c8xk5tNnXgMfWq/D3NxCxVNB\nvT7B7OxwTewdc3ddcPz4qj8VjePH+a8e1eDf8XDYzJjr9Yk191W92+T+iHhZ6/WrgYcrHkfqi+Xt\nF2+oXdpqqob3HwJHIuILwCuA2zpWkdQDC/sPrt6+70CPK5GqaXvZJDO/CVzRev1V4Mou1SR13eKu\n3ZwGxm8//IO7TfYd8GKlilF1zVsq3uKu3Ya1iuUnLCWpQIa3JBXI8JakAhneklQgw1uSCmR4S1KB\nDG9JKpDhLUkFMrwlqUCGtyQVyPAeIGMz00xO7eCCF0wyObWDsZnpfpckqUv8bpMB4ZNhpOHizHtA\njB9Z/XGi47cf7nElknrB8B4Qo6dObqhdUtkM7wHhk2Gk4WJ4DwifDCMNF8N7QCzu2s3pO4+xdMml\nNGo1li65lNN3HvNipTSgvNtkgPhkGGl4OPOWpAK1PfOOiJcD78/MnRHxIuAuoAE8BtyQmc90p0RJ\n0nO1NfOOiHcBHwXObzUdBm7JzKuAEeDa7pQnSVpNuzPvbwBvAu5pbV8OPNh6fR/wOmDmbAeYnByn\nVhutUiMA9fpE5b6lcsyDb9jGC465U9oK78z8dERctKJpJDMbrdfzwPPWO8bc3MLGq2up1yeYnZ2v\n3L9EjnnwDdt4wTFX6buWqhcsV65vTwBPVzyOJKmCquH9SETsbL2+GnioM+VIktpR9T7vg8DRiDgP\nOAH43aOS1ENth3dmfhO4ovX6FDDVpZokSevwQzqSVCDDW5IKZHhLUoEMb0kqkOEtSQUyvCWpQIa3\nJBXI8JakAhneklQgw1uSCmR4S1KBDG9JKpDhLUkFMrwlqUCGtyQVyPCWpAIZ3pJUIMNbkgpkeEtS\ngao+gBiAiHgE+E5r818y8/rNlyRJWk/l8I6I8wEyc2fHqpEktWUzM+/LgPGIeKB1nJsz8yudKUuS\ndDYjjUajUseI+AXgCuCjwM8D9wGRmUurvX9pablRq41WrVOShtHIWjs2M/M+BTyZmQ3gVEQ8BbwA\n+LfV3jw3t1D5RPX6BLOz85X7l8gxD75hGy845ip917KZu032AIcAIuKFwDbgPzZxPElSmzYz8/4Y\ncFdEfBFoAHvWWjKRJHVW5fDOzO8Dv93BWiRJbfJDOpJUIMNbkgpkeEtSgQxvSSqQ4S1JBTK8JalA\nhrckFcjwlqQCGd6SVCDDW5IKZHhLUoEMb0kqkOEtSQUyvCWpQIa3JBXI8JakAhneklQgw1uSCmR4\nS1KBDG9JKlClBxBHxDnAHcBlwCLwtsx8spOFSZLWVnXm/Ubg/MzcAbwbONS5kiRJ66ka3q8EPg+Q\nmV8BXtqxiiRJ66q0bAJsA76zYns5ImqZubRWh8nJcWq10Yqng3p9onLfUjnmwTds4wXH3ClVw/s0\nsLKac84W3ABzcwsVT9Uc+OzsfOX+JXLMg2/YxguOuUrftVRdNvkS8HqAiLgCeLTicSRJFVSdec8A\nr42ILwMjwPWdK0mStJ5K4Z2ZzwBv73AtkqQ2+SEdSSqQ4S1JBTK8JalAhrckFcjwlqQCGd6SVCDD\nW5IKZHhLUoEMb0kqkOEtSQUyvCWpQIa3JBXI8JakAhneklQgw1uSCrSlw3tsZprJqR1QqzE5tYOx\nmel+lyRJW0LVJ+l03djMNNv27nl2u3bicbbt3cNpYHHX7v4VJklbwJadeY8fObR6++2He1yJJG09\nWza8R0+d3FC7JA2TLRvey9sv3lC7JA2TSmveETECfAt4otX0D5l5U8eqAhb2H/yhNe9n2/cd6ORp\nJKlIVS9Y/hzw1cx8QyeLWWlx125O01zjrp06ydL2i1nYd8CLlZIEjDQajQ13iojfAm4EvgP8D/CO\nzMyz9VlaWm7UaqOVipSkITWy1o51Z94R8XvAO57TfAPwvsy8NyJeCXwc+KWzHWdubqGNOldXr08w\nOztfuX+JHPPgG7bxgmOu0nct64Z3Zn4M+NjKtogYB5Za+78YET8TESOZufFpvCRpw6rebfLHwH6A\niLgM+FeDW5J6p+oFyz8FPh4R19Ccgb+1YxVJktZVKbwzcw64psO1SJLaVOluE0lSf23ZT1hKktZm\neEtSgQxvSSqQ4S1JBTK8JalAhrckFcjwlqQCbdlnWEbEOcAdwGXAIvC2zHyyv1V1V0ScCxwDLgLG\ngNsy8zN9LapHIuKngIeB12bmwD8uKSJuAn4dOA+4o/UdQgOr9W/7bpr/tpeB3x/kv+eIeDnw/szc\nGREvAu4CGsBjwA2Z+cxmz7GVZ95vBM7PzB3Au4HVH2o5WK4DnsrMq4CrgQ/3uZ6eaP1g30nz64UH\nXkTsBK4EXgFMARf2taDeeD1Qy8wrgfcCf9LneromIt4FfBQ4v9V0GLil9XM9AlzbifNs5fB+JfB5\ngMz8CvDS/pbTE/cCt67YXupXIT32AeAvgH/vdyE98ivAo8AM8Fngc/0tpydOAbXWb9TbgP/rcz3d\n9A3gTSu2LwcebL2+D3hNJ06ylcN7G82HPZyxHBFbdpmnEzLzu5k5HxETwDRwS79r6raIeCswm5n3\n97uWHrqA5mTkN4C3A3/derTgIPsuzSWTk8BR4IN9raaLMvPT/PB/Tiu/LnseeF4nzrOVw/s0sPKb\nyM/JzIGfiUbEhcDfAfdk5if6XU8P7AFeGxFfAH4R+KuI+On+ltR1TwH3Z+b3W0+g+l+g3ueauu0d\nNMe8neZ1rLsj4vx1+gyKlevbE8DTnTjoVg7vL9FcJyMirqD5a+ZAi4jnAw8AN2bmsX7X0wuZ+arM\nnMrMncA/A7+Tmd/uc1nd9kXgVyNiJCJeCPw4zUAfZHP84Dfp/wbOBYbluYiPtK5zQPNa1kOdOOhW\nXoaYoTkj+zLNRf7r+1xPL9wMTAK3RsSZte+rM3MoLuQNi8z8XES8CvhHmhOoGzJzuc9lddufA8ci\n4iGad9jcnJnf63NNvXIQOBoR5wEnaC6JbppfCStJBdrKyyaSpDUY3pJUIMNbkgpkeEtSgQxvSSqQ\n4S1JBTK8JalA/w8GSwG+2pvctgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1051912e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, y, 'ro');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg = np.polyfit(x, y, deg=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.20791083, -1.23705534])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAD3CAYAAADSftWOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4VdW9//H3PtMOYZBQIoq1qFWX\nqC2tVoU64FCtqC2Vi1p7cUJbqvwsCFYlJBAgCSAIKEpFxbm2FjTV2jpcq3VGb6ltUWDh7HXCiJEp\nnHn//jjRYksgOTknZ/q8nsfnydkn++zvMvBhZe3v3tvxPA8RESksvlwXICIiHafwFhEpQApvEZEC\npPAWESlACm8RkQIU6KoDNTVtSrutpaKinObmlkyWk/c05uJXauMFjbmjKit7Om29VxAz70DAn+sS\nupzGXPxKbbygMWdSQYS3iIh8mcJbRKQAKbxFRAqQwltEpAApvEVECpDCW0QkC9zGZVQMHQKBABVD\nh+A2Lsvo53dZn7eISKlwG5fRa8zoL14HVr9KrzGj2QhETh+ZkWNo5i0ikmHlC67Z/vZr52XsGApv\nEZEM869d06Ht6VB4i4hkWGL/Azq0PR0KbxGRDGsZPxGAzXRnHbv+a/u4CRk7hsJbRCTDWn44kutG\nPcve/nc5iFeJDTyYjYtvzdjJSlC3iYhIRr34op+qKpeVK4+kvNxjzhyHz854PuPH0cxbRCQDPvzQ\n4eKLy/jBD8pZudLPGWfEWL58C5dckp3j7XTmbYzxAzcDBkgAFwAOcDvgAa8AY621yeyUKCKSv8Jh\nWLw4xPz5IVpaHAYNStDQEOaww7Ibie2Zef8AwFp7JDAFmNf6X7W19mhSQT48axWKiOQhz4NHHvFz\nzDHdqa93KS/3mD8/zKOPtmQ9uKEdM29r7e+NMQ+1vhwArANOBZ5q3fYwcBLQuKPPqago79RNySsr\ne6a9b6HSmItfqY0XimPMa9bA+PHw6KMQCMBll8GUKT569y4Dyv7j+7Mx5nadsLTWxo0xdwCnAyOB\n06y1nz/WbBOwy84+ozOPPqqs7ElT06a09y9EGnPxK7XxQuGPeeNGmDvX5ZZbgsTjDkOHxqmri2BM\nklgMmpr+c5/OjHlHod/uE5bW2vOA/Umtf3fb5q2ewGdpVSYiUgCSSbjnngCDB3fnxhtD9O/vcccd\nW/nd77ZiTG5O9+00vI0x5xhjJrW+bAGSwF+NMce2bhsGPJOd8kREcuuvf/UxbFg548d3o6XFoaoq\nwrPPbmHYsDhOm48Hzr72LJvcD9xmjHkaCALjgdXAzcaYUOvXmb3XoYhIjq1b5zBjhsvvfhcEYMSI\nGFOmROjf39vJnl2jPScstwBnbuetoZkvR0QktyIRuOmmEPPmhdiyxeHggxM0NEQYPDiR69K+RFdY\nioi0evxxP9XVZbz5po8+fZLU1kYYNSqGP/1GuaxReItIyXvjDYeamjIefzyA3+9x0UVRrrgiQu/e\nua6sbQpvESlZmzfDvHkhFi8OEYs5HH10qvVv4MD8v2Bc4S0iJSeZhKVLA8yY4fLxxz723DNJbW2Y\n007LbQdJRyi8RaSkvPyyj6qqMlas8NOtm8cVV0QYOzZKt2473zefKLxFpCR8/LFDQ0OIe+4JATB8\neIypUyN89av50frXUQpvESlq0SgsWRJk7lyXTZscBg5Mtf4deWR+tf51lMJbRIrWE0/4qalxee01\nP717e8yaFebcc2MEiiD5imAIIiJf9tZbDlOnujzySBCfz+OCC6JceWWEPn1yXVnmKLxFpGhs3gzX\nXhviV78KEY06DBkSp74+wsEH53/rX0cpvEWk4Hke3HdfgOnTXT76yMcee6Ra/374w8Jp/esohbeI\nFLR//tNHVZXLSy8FcF2PiRMjXHpplPLyXFeWXQpvESlIn3ziMHNmiLvvDuJ5DqeeGqO2NsKAAYXZ\n+tdRCm8RKSixGNx2W5Crr3bZuNHhgAMS1NVFOOaYwm796yiFt4gUjKee8lNd7WKtn1128aivD3P+\n+TGCwVxX1vUU3iKS9955J9X696c/BXEcj3POiTJpUpS+fUtjiWR7FN4ikre2bIGFC0PccEOISMTh\n8MPjNDRE+OY3i6/1r6MU3iKSdzwPHnggQG2tywcf+Nh99yRTpoQZMaJ4W/86SuEtInnllVd8TJ7s\n8sILAUIhj/HjI/ziF1F69Mh1Zfllp+FtjAkCtwJ7AS5QB7wH/AF4rfXbfmWtvTdLNYpICfj0U5g1\ny+XOO4Mkkw4nnxxj2rQIe+9duuvaO9KemfcoYL219hxjzFeAl4HpwDxr7TVZrU5Eil48DnfcEWT2\nbJfPPnPYb79U699xx5VW619HtSe8lwLLtnkdBw4FjDFmOKnZ93hr7aYs1CciRey55/xUVbmsXu2n\nZ0+P6dPDXHhhabb+dZTjee37lcQY0xN4ELiZ1PLJP621K4wxk4EKa+3lO9o/Hk94gUAePoJZRLrc\nO+/AL38JS5eC48Do0VBfD/365bqyvNPm6dl2nbA0xuwJNAKLrLX3GGN6W2s/a327EVi4s89obm5p\nz6G2q7KyJ01NpTWx15iLX6mNF6BHj55MnRph4cIQ4bDDoYcmaGgI8+1vp1r/mppyXGAWdObnXFnZ\ns833fDvb2RjTD3gMuNJae2vr5keNMYe3fn0CsCKtykSkJHge/OEPAQYOhDlzXHr18rj++q388Y8t\nXwS3dEx7Zt5VQAVQY4ypad02AVhgjIkCHwE/y1J9IlLgVq3yUV3t8uyzAYJBuPTSCJddpta/ztpp\neFtrxwHjtvPWdzNfjogUi+ZmuPpql9tvD5JIOJx4YpwbbgjQu3c016UVBV2kIyIZlUjAXXcFmTUr\nxKef+thnnyR1dVv53vcSreu/ua6wOCi8RSRjli9Ptf698oqf7t09pkwJ87OfxQiFcl1Z8VF4i0in\nvf++w/TpLo2NqQbts86KUV0doV8/XR2ZLQpvEUlbOAy/+lWIa68N0dLi8O1vp1r/Dj1UHSTZpvAW\nkQ7zPHj44QBTpri8+66Pvn2TzJwZ5qyz4vh22oAsmaDwFpEOsTZ117+nnw4QCHhcfHGUiRMj9OqV\n68pKi8JbRNplwwaYO9fllltSrX/HHRenri7CfvtpiSQXFN4iskOJBPzmN0EaGkJ88omPvfZKMmPG\nVk46KaEHI+SQwltE2vTSSz4mTy7jH//wU17uUV0dYcyYKK6b68pE4S0i/+Gjj1Ktf8uWpVr/Ro6M\nUVMTYffd1fqXLxTeIvKFSAQWLw4xb16q9e+b30xQXx/hiCP0YIR8o/AWETwPHnvMT01NGW+/7eMr\nX0lSVxfh7LNj+HUb/ryk8BYpca+/7lBdXcYTTwTw+z3GjIly+eURdtkl15XJjii8RUrUpk2p1r+b\nbw4Sjzscc0yc+voIxqj1rxAovEVKTDIJ994boK7OpanJx9e+lmT69DDDhsXV+ldAFN4iJWTFilTr\n39/+lmr9u+qqCBdfHKVbt1xXJh2l8BYpAevWOdTVudx7b6r17/TTY0yZEmGPPdT6V6h0CxmRIhaN\nwg03BBkypDv33hvkoIMSPPBAC4sXhzsc3G7jMiqGDqHv7hVUDB2C27gsS1VLe2jmLVKk/vxnP9XV\nZbzxho+KCo+rrw5zzjnptf65jcvoNWb0F68Dq1+l15jRbAQip4/MXNHSbpp5ixSZN990GDWqG2ef\nXc5bbzmMHh1l+fLNnH9++j3b5Quu2f72a+d1olLpjJ3OvI0xQeBWYC/ABeqAVcDtgAe8Aoy11qq/\nSCSHNm+G+fNDLF4cIhp1OPLI1F3/Djqo8381/WvXdGi7ZF97Zt6jgPXW2qOBYcD1wDygunWbAwzP\nXokisiOeB0uXBhgypDsLF7rsuqvHkiVbuf/+rRkJboDE/gd0aLtkX3vCeylQs83rOHAo8FTr64eB\n72W4LhFph7//3cepp5Yzdmw3NmxwuPzyCM8+u4Uf/CCzPdst4yduf/u4CZk7iHSI43ntO+NsjOkJ\nPAjcDMy11vZv3X48MNpaO2pH+8fjCS8Q0E0SRDLh44+hqgpuvTU18x45EubOhQEDsnjQ3/4WZs6E\nVavgwANh0iT48Y+zeEAhtbKxXe3qNjHG7Ak0AoustfcYY67e5u2ewGc7+4zm5pb2HGq7Kit70tS0\nKe39C5HGXPzSGW8sBrfeGmTOHJeNGx0GDkzd9e+oo1J3/WtqykalrU44NfXftjpYf6n9jKFzY66s\n7NnmeztdNjHG9AMeA6601t7auvllY8yxrV8PA55JqzIRabe//MXPcceVU1NThs8HM2eG+fOfW74I\nbikt7Zl5VwEVQI0x5vO173HAdcaYELAaULe+SJa8/bbD1KkuDz8cxHE8zjsvylVXRfnKV3R1ZCnb\naXhba8eRCut/NzTz5YjI57ZsgeuuC7FoUYhIxGHw4NRd/77xDXXliq6wFMk7ngeNjQGmTXP58EMf\nu++epLY2zI9+pLv+yb8ovEXyyMqVPqqqXF58MYDrekyYEOHSS6N0757ryiTfKLxF8sD69Q4zZ4a4\n664gnudwyikxamsj7LWX1rVl+xTeIjkUj8PttweZPdtlwwaH/fdPUFcX4dhj1UEiO6bwFsmRJ56A\n//f/ylm92k+vXh51dWEuuCBGMJjryqQQKLxFuti77zrU1ro89BA4jo9Ro6JMmhSlslJLJNJ+Cm+R\nLtLSAgsXhrjhhhDhsMOQITB9eguDBqn1TzpO9/MWyTLPgwcfDHDkkd255hqXXXbxWLRoK889h4Jb\n0qaZt0gWvfqqj8mTXZ5/PkAo5PGLX0QYPz5Kjx6oZ1s6ReEtkgWffgqzZ7vccUeQZNLh+9+PM21a\nmH320bq2ZIbCWySDEgm4884gs2a5NDc77LtvqvXv+OPV+ieZpfAWyZDnn/dTVeWyapWfHj08amvD\nXHRRjFAo15VJMVJ4i3TS++87TJvm8vvfpxq0zz47RlVVhH79tEQi2aPwFknT1q2waFGI664LsXWr\nwyGHJGhoCHPIIeogkexTeIt0kOfBH/8YoLbW5d13fVRWJpk9O8yZZ8bxqflWuojCW6QD1qxJtf49\n80yAYNDjkkuiTJwYoWfbT6sSyQqFt0g7bNgAc+a4LFkSJJFwOP74OHV1YfbdV+vakhsKb5EdSCTg\nnnuCNDSEWL/ex957J5kxYysnnpjQRTaSUwpvkTa8+GKq9W/lSj/l5R7V1RHGjIniurmuTEThLfIf\nPvzQYfp0l/vuS7X+nXFGjJqaCLvtpiUSyR/tDm9jzBHAbGvtscaYQ4A/AK+1vv0ra+292ShQpKuE\nw7B4cYj580O0tDgMGpRq/TvsMLX+Sf5pV3gbY64AzgG2tG46BJhnrb0mW4WJZJvbuIzyBdfgs2t4\noP/PmRhp4K0ml759k9TXRzj77Jha/yRvtXfm/QYwArir9fWhgDHGDCc1+x5vrd2UhfpEssJtXEav\nMaNZg2E8D/HoeycTIMbYE15h/I0D2GWXXFcosmOO57VvHc8YsxfwW2vtYGPMBcA/rbUrjDGTgQpr\n7eU72j8eT3iBgL/TBYtkwoaDvsv0Vf/FdfyCOEFO5DEWMJ4DvxmEf/wj1+WJfK7NnqZ0T1g2Wms/\n+/xrYOHOdmhubknzUFBZ2ZOmptKa2GvM2ZFMwm9/G6BhVSMf04+9eZN5TGA4D+AA3qoAn3TR/3f9\njEtDZ8ZcWdn21V/prug9aow5vPXrE4AVaX6OSJf56199DBtWzvjx3djs9KSeKlZxID9qDW6AxP4H\n5LRGkfZKd+Z9MXC9MSYKfAT8LHMliWTWunUOM2a4/O53qda/ESNi1B3+BwZeNfM/vrdl3ISuLk8k\nLe0Ob2vt28Dg1q//Bnw3SzWJZEQkAjfdFGLevBBbtjgcfHCChoYIgwcngFPYWHEr5dfOw792DYn9\nD6Bl3AQip4/Mddki7aKLdKQoPf64n+rqMt5800efPklqayOMGhXDv80588jpIxXWUrAU3lJU3njD\noaamjMcfD+D3e1x0UZQrrojQu3euKxPJLIW3FIXNm2HevBCLF4eIxRyOPjpOXV2EgQN1daQUJ4W3\nFLRkEpYuDTBjhsvHH/vYc88ktbVhTjstrrv+SVFTeEvBevllH1VVZaxY4adbN48rrogwdmyUbt1y\nXZlI9im8peB8/LFDQ0OIe+5JPZZ9+PAYU6dG+OpXddc/KR0KbykY0SgsWRJk7lyXTZscBg5Mtf4d\neWQi16WJdDmFtxSEJ57wU13t8vrrfnr39pg1K8y558YI6E+wlCj90Ze89uabDlOnlvHoowF8Po8L\nLohy5ZUR+vTJdWUiuaXwlry0eTMsWBDixhtDRKMOQ4bEqa+PcPDBav0TAYW35BnPg/vuCzB9ustH\nH/nYY49U698Pf6jWP5Ft6TkhRcRtXEbF0CH03b2CiqFDcBuX5bqkDlmxAk47rZxLLulGc7PDxIkR\nnntuC8OHK7hF/p1m3kXi8yfDfC6w+lV6jRnNRsj7+3d88onDzJkh7r4bPM/PqafGqK2NMGCAWv9E\n2qKZd5EoX7D9x4mWXzuviytpv1gMbropyODB3bnrrhAHHgjLlrVw221hBbfITii8i4R/7ZoObc+1\np57yc/zx5VRXl+E40NAQ5u9/h2OOUc+2SHsovItEW0+Aybcnw7zzjsP555dxxhnlrF3r49xzo7zw\nwhYuukg92yIdofAuEi3jJ25/e548GWbLFpg1K8RRR3XnT38KcsQRcR5/vIW5cyP07aslEpGO0lyn\nSEROH8lGyLsnw3ge/P73AaZNc/ngAx+7755k6tQwp5+uDhKRzlB4F5F8ezLMypU+Jk92Wb48QCjk\ncdllES69NEqPHrmuTKTwKbwl49avd5g1K8RddwVJJh1OPjnGtGkR9t5byyMimdLu8DbGHAHMttYe\na4zZF7gd8IBXgLHWWl23XOLicbjjjiCzZ7t89pnDfvslqKuLcNxx6iARybR2nbA0xlwB3AKUtW6a\nB1Rba48GHGB4dsqTQvHss35OOKGcSZPKSCRgxowwf/lLi4JbJEvaO/N+AxgB3NX6+lDgqdavHwZO\nAhp39AEVFeUEAv4dfcsOVVb2THvfQlUIY37nHbj8cli2DBwHLrwQGhocdt21jH/9W99+hTDmTCq1\n8YLGnCntCm9r7X3GmL222eRYaz9fwNwE7LKzz2hubul4da0qK3vS1LQp7f0LUb6PuaUFrr8+xPXX\nhwiHHb7znQQNDWG+9a3U6llTU8c/M9/HnGmlNl7QmNPZty3pnrDcdn27J/BZmp8jBcbz4KGHAkyd\n6vLeez769Usyd26YkSPj+HTVgEiXSfev28vGmGNbvx4GPJOZciSfrVrlY8SIblx4YTfWrXO49NII\nL7ywhTPPVHCLdLV0Z94TgZuNMSFgNVBY9x6VDmluhquvdrnttlTr30knxZk+Pcw++6j1TyRX2h3e\n1tq3gcGtX68FhmapJskTiQTcdVeQWbNCfPqpj69/PUld3VZOOEEdJCK5pot0ZLteeMFPVZXLq6/6\n6dHDY+rUMD/9aYxQKNeViQgovOXfvP++w/TpLo2NQQB+/OMYkydH6NdPSyQi+UThLQCEw7BoUYjr\nrgvR0uLw7W+nWv8OPVQXzorkI4V3ifM8+NOfUq1/777ro7IyyaxZYXWQiOQ5hXcJszZ117+nnw4Q\nCHhcckmUiRMj9Cy9C+BECo7CuwRt2ABz5rgsWRIkkXA4/vg4M2ZE2G8/LZGIFAqFdwlJJOCee4I0\nNIRYv97HXnulWv9OPDGhByOIFBiFd4l48UU/kye7/POffsrLPaqrI4wZE8V1c12ZiKRD4V3kPvww\n1fp3332p1r+RI2NMmRJht93U+idSyBTeRSoSgRtvDDF/fqr1b9CgBPX1YQ4/XOvaIsVA4V1kPA8e\ne8xPTU0Zb7/to2/fJHV1Ec4+O4Y//dupi0ieUXgXkdde81Fd7fLkkwH8fo8xY6JcfnmEXXZ6t3UR\nKTQK7yKwcSNcc43LzTcHiccdjjkmTn19BGO0RCJSrBTeBSyZhHvvDTBjhssnn/j42teSTJ8eZtiw\nuFr/RIqcwrtArVjho6qqjJdfTrX+TZoU4eKLo5R1/LGRIlKAFN4FZt06h7o6l3vvTbX+jRiRav3r\n31+tfyKlROFdIKJRuOmmINdc47Jli8NBByWYOTPC4MF6MIJIKVJ4F4DHH0+1/r3xho8+fZLU1kYY\nNUqtfyKlTOGdx95806Gmpoz/+Z9U699FF0X55S8jVFTkujIRybVOhbcx5mVgQ+vLt6y1F3S+JNm8\nGebOhfnzuxOLORx1VJy6uggHHqjWPxFJSTu8jTFlANbaYzNWTYlLJmHp0lTr38cfw1e/6jFtWpjT\nTlPrn4h8WWdm3oOAcmPMY62fU2WtXZ6ZskrPyy+nWv9WrPBTVuZRWwvnn7+F8vJcVyYi+cjxvPRa\nzIwx3wAGA7cA+wEPA8ZaG9/e98fjCS8Q0Bm2f7duHVRVwW23pe5LcsYZMGcODBiQ68pEJA+0+Tt3\nZ2bea4HXrbUesNYYsx7YHfi/7X1zc3NL2geqrOxJU9OmtPfPR7EYLFkSZM4cl02bHAYOTNDQEOHI\nIz9v/Su+Me9MMf6cd6TUxgsaczr7tqUz4T0a+AZwiTGmP9AL+LATn1cynnzST3W1y2uv+end22Pm\nzDDnnRcjoN4fEWmnzsTFEuB2Y8yzgAeMbmvJRFLeesth6lSXRx4J4vN5nH9+lKuuitCnT64rE5FC\nk3Z4W2ujwE8yWEvR2rwZrrsuxKJFIaJRhyFDUnf9O/hgtf6JSHr0i3oWeR7cf3+AadNcPvrIR//+\nSWprwwwfrtY/EekchXeWrFzpY9Ikl5deCuC6HhMmRLj00ijdu+e6MhEpBgrvDPvkE4eZM0PcfXcQ\nz3M45ZQY06ZFGDBAd/0TkcxReGdIPA633x5k9myXDRscjElQVxdh6FDd9U9EMk/hnQFPP51q/Vuz\nxk+vXh51dWEuuCBGMJjrykSkWCm8O+Hdd1Otf3/8YxDH8TjnnCiTJkXp21dLJCKSXQrvNLS0wMKF\nIW64IUQ47HDYYQkaGsIMGqTWPxHpGgrvDvA8ePDBALW1Lu+/72O33ZJMmRLmv/5LrX8i0rUU3u30\n6qs+Jk92ef75AKGQx7hxEcaNi9KjR64rE5FSpPDeiU8/hdmzXe64I0gy6XDyyTFqayPss4/WtUUk\ndxTebYjH4c47U61/zc0O++6bav07/ni1/olI7im8t+P55/1UVbmsWuWnR4/U02wuvDBGKJTrykRE\nUhTe23jvPYdp01weeCDVoP2Tn0Spqoqy665aIhGR/KLwBrZuhRtuCLFwYYitWx0OPTRBfX2YQw5R\n65+I5KeSDm/Pg4ceSrX+/d//+dh11yRXXx3mjDPi+Hy5rk5EpG0lG96rV/uornZ55pkAwaDH2LFR\nJkyI0LPtpw6JiOSNkgvvzz6Dq692ue22IImEw/e+F2fGjDBf/7rWtUWkcJRMeCcS8OtfB2loCPHp\npz722SfJjBlbOfFEtf6JSOEpifBevtzP5MkuK1f66d7do6Ymws9+FsV1c12ZiEh6ijq8P/jAYfp0\nl/vvT7X+nXlmjJqaCP36aYlERApbWuFtjPEBi4BBQAS4yFr7eiYL64xwGG68McSCBSFaWhy+9a3U\nXf++8x21/olIcUi3Ie5HQJm1dghwFXBN5kpKn+fBww8HOPro7jQ0uJSXeyxYsJVHHmlRcItIUUk3\nvI8CHgGw1i4HvpOxitK0dq2Ps87qxnnndeP99x1+/vMoy5dv4Sc/Uc+2iBSfdNe8ewEbtnmdMMYE\nrLXxtnaoqCgnEPCneTiorNx+A/aGDTBtGixcmLqZ1EknwYIFDgMHhoDCvhlJW2MuZqU25lIbL2jM\nmZJueG8Etq3Gt6PgBmhubknzUKmBNzVt+tK2ZBJ+85sg9fUhPvnEx4ABSWbMCPP97ydwHGhqSvtw\neWF7Yy52pTbmUhsvaMzp7NuWdBcUngNOATDGDAZWpvk5afnf//Vx8snlXHZZGS0tDpMnR3jmmS2c\nfHJCT7QRkZKQ7sy7ETjRGPM84AAXZK6ktn30kcOMGS5Ll6Za/0aMiDFlSoT+/dX6JyKlJa3wttYm\ngZ9nuJY2RSKpB/7OmxdiyxaHb3wjQX19hMGDdXWkiJSmvL9I58kn/UyeDK+/7tKnT5Jp0yL893/H\n8Kd/7lNEpODldXjH43DOOd1IJOCnP43yy19G6N0711WJiOReXod3IAC//vVWDjqonL59I7kuR0Qk\nb+T95StDhyYYODDXVYiI5Je8D28REflPCm8RkQKk8BYRKUAKbxGRAqTwFhEpQApvEZECpPAWESlA\nCm8RkQKk8BYRKUAKbxGRAqTwFhEpQHkd3m7jMiqGDoFAgIqhQ3Abl+W6JBGRvJC3dxV0G5fRa8zo\nL14HVr9KrzGj2QhETh+Zu8JERPJA3s68yxdcs/3t187r4kpERPJP3oa3f+2aDm0XESkleRveif0P\n6NB2EZFSktaatzHGAd4DXmvd9IK1dlLGqgJaxk/80pr3F9vHTcjkYUREClK6Jyy/DvzNWvuDTBaz\nrcjpI9lIao07sHYN8f0PoGXcBJ2sFBEBHM/zOryTMeYs4EpgA7AVuMxaa3e0Tzye8AIBPfJdRKQD\nnLbe2OnM2xhzIXDZv20eC8y01i41xhwF3A0ctqPPaW5uaUed21dZ2ZOmpk1p71+INObiV2rjBY05\nnX3bstPwttYuAZZsu80YUw7EW99/1hizhzHGsdZ2fBovIiIdlm63yVRgPIAxZhDwroJbRKTrpHvC\nchZwtzHmVFIz8PMzVpGIiOxUWuFtrW0GTs1wLSIi0k5pdZuIiEhu5e0VliIi0jaFt4hIAVJ4i4gU\nIIW3iEgBUniLiBQghbeISAFSeIuIFKC8fYalMcYHLAIGARHgImvt67mtKruMMUHgVmAvwAXqrLUP\n5rSoLmKM2RVYAZxorS36xyUZYyYBPwRCwKLWewgVrdY/23eQ+rOdAH5azD9nY8wRwGxr7bHGmH2B\n2wEPeAUYa61NdvYY+Tzz/hFQZq0dAlwFbP+hlsVlFLDeWns0MAy4Psf1dInWv9iLSd1euOgZY44F\nvgscCQwF9sxpQV3jFCBgrf0LmVbDAAACA0lEQVQuMB2oz3E9WWOMuQK4BShr3TQPqG79e+0AwzNx\nnHwO76OARwCstcuB7+S2nC6xFKjZ5nU8V4V0sbnAjcAHuS6ki3wfWAk0An8AHsptOV1iLRBo/Y26\nFxDLcT3Z9AYwYpvXhwJPtX79MPC9TBwkn8O7F6mHPXwuYYzJ22WeTLDWbrbWbjLG9ASWAdW5rinb\njDHnA03W2kdzXUsX6ktqMnIG8HPg162PFixmm0ktmawBbgauy2k1WWStvY8v/+O07e2yNwG7ZOI4\n+RzeG4Ft70Tus9YW/UzUGLMn8CRwl7X2nlzX0wVGAycaY/4CfAu40xizW25Lyrr1wKPW2mjrE6jC\nQGWOa8q2y0iNeX9S57HuMMaU7WSfYrHt+nZP4LNMfGg+h/dzpNbJMMYMJvVrZlEzxvQDHgOutNbe\nmut6uoK19hhr7VBr7bHA34FzrbUf5bisbHsWONkY4xhj+gPdSQV6MWvmX79JfwoEgVJ5LuLLrec5\nIHUu65lMfGg+L0M0kpqRPU9qkf+CHNfTFaqACqDGGPP52vcwa21JnMgrFdbah4wxxwAvkZpAjbXW\nJnJcVrbNB241xjxDqsOmylq7Jcc1dZWJwM3GmBCwmtSSaKfplrAiIgUon5dNRESkDQpvEZECpPAW\nESlACm8RkQKk8BYRKUAKbxGRAqTwFhEpQP8fJ1/5Gp6w0GcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1082ae128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "yr = np.polyval(reg, x)\n",
    "plt.plot(x, y, 'ro')\n",
    "plt.plot(x, yr, 'b');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.4876045521534866"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "((y - yr) ** 2).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network Training &mdash; Single Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "l0 = np.array((x, len(x) * [1])).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0. ,   1. ],\n",
       "       [  2.5,   1. ],\n",
       "       [  5. ,   1. ],\n",
       "       [  7.5,   1. ],\n",
       "       [ 10. ,   1. ]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.array(((2., 2.)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  2.,   7.,  12.,  17.,  22.])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l1 = np.dot(l0, weights)\n",
    "l1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ -4.18804341,  11.55584317,  14.7240732 ,  20.46371507,  31.45690599])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-6.18804341,  4.55584317,  2.7240732 ,  3.46371507,  9.45690599])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = y - l1\n",
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33.579711209646916"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(d ** 2).mean()  # MSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = 0.01  # learning rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.45556897,  0.14012494])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "update = alpha * np.dot(d, l0)\n",
    "update"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights += update  # updating weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.45556897,  2.14012494])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.556858845812496"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l1 = np.dot(l0, weights)\n",
    "d = y - l1\n",
    "(d ** 2).mean()  # new MSE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network Training &mdash; Multi Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.array(((1., 100.)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE after    0 iterations: 8204.01\n",
      "MSE after    5 iterations: 4690.87\n",
      "MSE after   10 iterations: 3109.04\n",
      "MSE after   15 iterations: 2303.67\n",
      "MSE after   20 iterations: 1826.36\n",
      "MSE after   25 iterations: 1500.52\n",
      "MSE after   30 iterations: 1254.50\n",
      "MSE after   35 iterations: 1057.51\n",
      "MSE after   40 iterations: 894.95\n",
      "MSE after   45 iterations: 758.84\n",
      "MSE after   50 iterations: 644.09\n"
     ]
    }
   ],
   "source": [
    "# repeat a few times\n",
    "for _ in range(51):\n",
    "    # layer 1\n",
    "    l1 = np.dot(l0, weights)\n",
    "\n",
    "    # deltas of layer 1\n",
    "    d = y - l1\n",
    "    \n",
    "    # print MSE\n",
    "    if _ % 5 == 0:\n",
    "        print('MSE after %4d iterations: %6.2f' % (_, (d ** 2).mean()))\n",
    "\n",
    "    # update weights based on deltas\n",
    "    weights += alpha * np.dot(d, l0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAD3CAYAAADSftWOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XlgVfWZ//H3uVv23CxkYREEhQO4\nVlrrWiwVlzJKkZRaRSgUTdBxUKy2dux0prW/6XTq1mklkUUUtWpDkVIX1FEZ6m6VWhAPKpZ9Sci+\n3fX8/jghAdlDkntv7uf1F/ckufc54ebJN89ZPoZt24iISGJxxboAERE5dmreIiIJSM1bRCQBqXmL\niCQgNW8RkQTk6Y0XqapqPK5TWnJz06mtbemucuJesu0vaJ+Thfb52BQUZBmH+lhCrLw9HnesS+hV\nyba/oH1OFtrn7pMQzVtERPan5i0ikoDUvEVEEpCat4hIAlLzFhFJQGreIiIJSM1bRCQBxX3zDnza\nxq4ndxFtica6FBGRuNErV1gej9pHq9hTvhtXhousCTnklOSTcWEWhvuQFx6JiPR5cd+8c6cVkJGf\nyo4lO6l/uob6p2vwFHnJva4fhXcMiHV5IiIxEfdjk5STUxl29zCGv3MqJ/7JJHd6P+xAlND2YMfn\ntP6tmeDmQAyrFBHpXXG/8t7LcBlknJNJxjmZFN99AtGmzhn4jju30PpeM+nnZOKfnEf2lbl4chNm\n10REjllCdjhXigtXivNHg23b5E7thyvVRfPrjbS81cTOH28h82I/+aWFZJyXFeNqRUS6X9yPTY7E\nMAxyr+nHiX8cwYj3T6PoJwPxDU+l8fk6ghs7RymBT9qwowpbFpG+ISFX3ofiHeij383F9Lu5mLZ1\nLXgHpwAQbY6wcfx6XDlucibn4S/JJ3VUWoyrFRHpuoRfeR9K6inpuLOc++hGmqJkT8ol2hih+n92\n8dnYj/j0oo+o/u1OwnvCMa5UROTYHdXK2zTNQuCvwHggDCwGbGAtcJNlWXF9BY23yMvA+06k/38O\npvGleuora2h6uZ5dP9tG5ng/nnwPtm0TbYzizk6+m8WLSOI54srbNE0vUAG0tm+6F7jLsqwLAQOY\n2HPldS9Xqgv/FbkMfuQkRqw9nUEVQ0k1nfFJ61+bsU75G1tmbaThhTqiwbj+fSQiSe5oxia/BsqB\n7e2PxwCr2v/9PHBxD9TV4zy5HvyT8joeRxsieAen0PCnWrZM+4wNp33I9ts30fJ2E7atA50iEl+M\nwzUm0zS/BwyyLOtu0zRfA8qAVyzLGtD+8XHATMuyph7uRcLhiJ0I2XW2bdP0QRO7HtvF7t/vJrgz\niCfPw3k7zsPlc2FHbQyXLssXkV5zyIZzpJn3TMA2TfNi4EzgUaBwn49nAXVHevXjTYsuKMiiqqrx\nuJ7jqJ0A/juLyL69kObVjYR3h9hT3wxAdfku6pfWkFOSR/a38vAWeXukhF7d3zihfU4O2udj/9pD\nOWzztizra3v/vc/K+79N07zIsqzXgMuBV7tUVZwzPAaZX8/eb1t4e5C2tS3s/FsLO3+6lYyvZZNT\nkkfWN3NwZ8b/XxYi0nd05Tzv24D5pmn6gPVAZfeWFL+Kf3YC/W4upv5PtdRX1tD8WgPNrzWQdamf\nwUtOjnV5IpJEjrp5W5Z10T4Px3Z/KYnBU+Al//uF5H+/kMDGNuqX1pB2enrHx7ffsRnDA/6SfNK+\nlI5haEYuIt2vT11h2dtShqVSeHvnbWntsE3TS3WEtoWoWVCFb1gK/pI8cibn4xuaEsNKRaSv6bNX\nWMaC4TEY/s5pDH78ZPxX5RLaEaTqVzv45KtrqXm0KtbliUgfopV3NzO8Blnj/WSN9xNpitD4XB11\nf9hDxgXOUWM7arN9ziYyx2WTdWkOrnT9/hSRY6fm3YPcmW5ypuSTMyW/Y1vrmhbqntpD3VN7FO0m\nIl2mZV8vSz8rg5P/Mpp+txbjzvNQ/3QNm6Z8woYz/07gs7ZYlyciCULNOwZSRqRRdOdAhr/bHu02\nrR9uvxvfEOegZtuWNqoe2EFwi6LdROTgNDaJoX2j3Wzb7jitcNfju9j9i+3s/sV2J9qtJI/sKxTt\nJiKd1A3ixL7ngw8oG0AgNUpd5R5aXm9yot3u3ELW5TkMemio7q8iImre8cib4yX32n7kXtuP0LYg\n9X+soa5yD3Yg2tG4Wz9oJtoaJf2cTDVzkSSk5h3n9o12izZHOrZX3beDxhfq8Q704r9K0W4iyUYH\nLBOIK6Pz5lf5ZUXkXJNPpKEz2u2zr39E3dN7YlihiPQWNe8ElXFeFgPvPxFz7RkMWjiMrMv8BDa0\nEdoR7PiclvebiTREDvMsIpKoNDZJcK40J9rNf0Uu4dowe4972mGbLdM+JVIfIevSHPwleWSOy8bl\n0+9rkb5AzbsP2fdUQjtok/f9Quoqa2j4Uy0Nf6rFnesme2Ie+TcUknJyagwrFZHjpWVYH+VKd1Fw\na39O/stohr00kvzSQgyPQe3iKsI7Qx2fF9ysC4FEEpFW3n2cYRiknZFB2hkZFP10EM2vN5J+XiYA\ngY1tfHrOOlLPSO/xaDcR6V5aeScRw2OQOTa747xwu80mc7zfiXb7yVY2nPEhm77zCXVP7yHaGo1x\ntSJyOFp5J7HU0WkMefxkwlWhjmi3plcbaH6jkaxL/ZDmwg7ZYDiNX0Tih5q3HBDt1vZhC26/89ao\nX17Dzn/bin9SHv7JeYp2E4kTat6yn5RhqaQM6zwTJdocBRtq5u+mZv5uRbuJxAnNvOWw8qYXYH54\n+gHRbltmfdbxObZtx7BCkeSklbcc0QHRbs/W4cro/L2/89+2EtwYIKckT9FuIr1EzVuOiTvTTc53\n8vfbFrDaaH6tgaaX6nFluMj+p1z8JXlkXKBoN5GeoiWSHLcTnx7OSatH0+8WJ9qt7qk9bPr2J+y6\ne1usSxPps7Tylm6RaqaR+uOBFP5oAC3vNFFfWUP2FbkdH99SupHU0Wn4r8rDd4IOdIocLzVv6VZO\ntFsWGedkdWwL7QzS+FwdDctq94t281+ZiztHb0GRrtDYRHqct9iHufZ0Btw3hPTzM2l5u4kdP9iM\ndeqHNL/eGOvyRBKSlj3SK9x+zwHRbg3P1pJ6RjoA4fow22/bhH9ynqLdRI6CVt7S6/ZGuw17YRTu\nTCcdqPpP1dQuqeYf39rAJ2P+zq6fb6VtfWuMKxXpupRlleSOPRc8HnLHnkvKsspufX6tvCUuFF1T\nRFtmlLrKPTSsqKX6f3ZR/T+7SD0ljaF/NveLgBOJdynLKskundnx2LN+HdmlM2kAApNKuuU1tPKW\nuGC4DTIu2CfabYET7ebKdnc07ta/t1D7+2pFu0ncS7//noNvf+DebnsNrbwl7rjSXPivzMV/ZS52\npPPS+5qHq6h7rJodP9xM1qU55JTkkfF1RbtJ/HFv+PiYtneF3vUS1/a9QrNgTjGFdw7AO8hHw/Ja\nNl/3GRtO+5Ddv9wewwpFDhQZMZIoBq9yERsZut/27qLmLQnDNyTFiXZ7/RSGvTSSvPZot2hz5xil\n+a1GAp+2xbBKSXYtLfDbsx5iNB8xjleZS+eopGXO3G57nSOOTUzTdAPzAROIADMAA1gM2MBa4CbL\nshS9Ir1i32i34p8O6kj9sW2bHbdvJmC1kXqmE+3m/1YenkJFu0nP27XL4OGHvSxe7KWm5jx8ngjT\nM5fx08Y7CJun0jJnbrcdrISjm3lfAWBZ1vmmaV4E3IvTvO+yLOs10zTLgYnAsm6rSuQoGR4Dd1b7\nmSg2FNzan7rKPTS92sDONS3s/LetZI7Npt/NxWRckHX4JxPpgo8+clFR4WPpUg/BoEFeXpS5cwPM\nmBGiqOhiCgo+oaqq+y9GO2LztizrGdM0/9z+cAiwC5gArGrf9jxwCWreEmOGy8B/VR7+q/KcaLfl\ntdS3N/KcazrvhNj6t2ZST0lXtJt0mW3Dq6+6mTfPx6pVThs96aQopaUBpkwJkZ7e8zUYR3sjfdM0\nHwEmASXAYsuyBrRvHwfMtCxr6qG+NhyO2B6PztOV2Gj5pIWUE1Jwp7oJVgd5s/+bePI8FF5dSNHU\nIrK+nKVoNzkqbW3w+ONw332wbp2z7aKLYO5cmDABXN1/FPGQb8yjbt4ApmkWA28D2ZZl5bZvmwiM\ntyzrnw/1dVVVjccVtVJQkNUjf3bEq2TbX+i9fQ7tDFJ9/07ql9cS2RMGwHdSCv7JeeRNL8BT0Hvz\ncf0/J47qaoPFi70sWuSlutqFx2MzcWKY2bODnH764Q/3Hc8+FxRkHbJ5H/H3hGma15mmeWf7wxYg\nCrzXPv8GuBxY3aXKRHqZt9hH/18O7oh2y/5WLqFtTrRbtK39wGfYJlwdinGlEg82bHBx220pnHVW\nBr/6VQqhkMHNNwd4771m5s1rO2Lj7klHc8Dyj8DDpmn+H+AFbgHWA/NN0/S1/7t7L9oX6WFfjHZr\neaup4z7jzasb2XTtJ2SO8zvRbpco2i2Z2Db85S/OPPvll50WOWSIM8+++uoQmZkxLrDd0RywbAam\nHORDY7u/HJHe5850k3Wxv+Oxbdukjkqj6cV6ml6sx5XZHu02OY+MC7N0x8M+KhiEZcs8lJf7WLfO\nOUZ39tlhZs8OcdllYdxxdthOl8eLfEHWOD9Z4/y0Wa3UL62hfmkNdU/uoen/Ghjx/mkA2CEbPOhA\nZx9QWwuPPupjwQIvu3a5cLlsJk4MUVYWZMyY+L18Rc1b5BC+GO0WqQ53rLqr7t9Bw4pa/JPzFO2W\noDZuNHjoIR9PPumlpcUgM9OmrCzIrFlBBg8+rnMseoWat8gR7I1221e0MULw8wC7f7HdiXY7N9Np\n5Ip2i2u2DW+/7ebBB72sXOnBtg0GDYrywx8GmDo1RFYCXceld5lIFxT/7AQKbutPw5/rqKvcQ8vr\nTbS82UTz6kZOmD8s1uXJF4RCsGKFM89es8YZXp91VoTZs4NMmBDGk4CdMAFLFokP+0a7Bbc60W7p\nZ2V0fHxL6UZcGW5yShTtFisNDbBkiZcFC3xs2+bCMGwmTAhRVhbi7LMjJPIhCzVvkW7gG+Sj4F+K\nOx5H26K0vttMaGuQuseq8Q70Opful+STOiothpUmh02bDBYs8PHYY16amw3S021mzQpy/fVBhg6N\n/3n20VDzFukBrlQXw989leY3GqmvrKHhz53RbgMeGELud/vFusQ+6b33XMyb5+PZZz1EowbFxVFu\nvTXItGlBcnJiXV33UvMW6SGG2yDzwmwyL8ym/y8H0/hSPfWVe8j8ejYA0WCUzdM/I+syP9n/lNt5\nd0Q5JpEIPPech3nzfLz3nvM9PO00Z5595ZVhfL4YF9hD1LxFesG+0W57NbzVQOPzdTQ+X6doty5o\naoInnvDy0EM+Nm92vl+XXhqmrCzIeecl9jz7aKh5i8RIztdyGP7OqdT/sYa6yj00LK+lYXkt7jw3\nw14chW+wzh0/mG3bnHn2kiVeGhoM0tJspk8PUloa5OST+8Y8+2ioeYvEkO/EFArm9qffrcW0fdhC\nXWUNbR804z3B+Vs/8Gkb9ZU1+EvySDk5NcbVxtaaNS7Ky30sX+4hEjEoKIhy551Bpk0LkZ+fPE17\nLzVvkTiwb7Tbvuoq91B9706q7t2RlNFu0SisXOmhvNzLm2867WrUKGeePWlSmJQk/uNEzVskjhXc\nXEzqiLT9o91+upXsy3MYtHBYn723SnMzPPWUl4oKH59/7syzx41z5tljx/b9efbRUPMWiWOuDPdB\no92MFFdH425+o5FoW5TMr2UnfLTbzp0GixZ5eeQRH7W1BikpNtdeG6S0NMTIkfF7k6hYUPMWSRCe\nAi/5swrJn1VINNjZyKp+vYPmvzTi7ufBPymPnJI8Us9MT6hV+dq1zjx72TIPoZBBfn6UH/wgyPe+\nF6KwMPnm2UdDzVskAe17KmHhjwdQX1lD/TM11MzfTc383fiGpVBwS39yrs4/zLPEVjQKr7zihB6s\nXu20ouHDI5SVhSgpCZGmC1EPS81bJMGlfzmT9C9nUvzzE2h6tZ66pTU0Pl9HpDnS8TnNf2kkZVQa\nnvzY/8i3tkJlpZeKCi8bNjgX1Vx4oZMHOW5cpCdCfPuk2P9Piki3MLwGWZfkkHVJDpHGSEfueLQt\nyubvfUa0JULWN/z4J+eRdWkOrrTe7ZJVVQa/+x387ncZVFe78HptpkxxQg9OPVXz7GOl5i3SB+13\nqX3EpuC2/tRX7qFxZT2NKzuj3frdXEzK8J49f/zjj11UVHiprPQSCEBOjsEttwSYOTNEcbHm2V2l\n5i3Sx7ky3PSbXUS/2UUHRLvlzSoEnNzOgNVGipnaLQc6bRtWrXJTXu7jlVecNjN0aJQf/MDgm99s\nIiPjCE8gR6TmLZJE9o12a32/mdTTnKOCbWtb2fiN9aSMTMVfku9Euw069js6BQJOiO+8eT7Wr3dW\n/+eeG6asLMQll4QpLs6iqqpbdylpqXmLJCHDZZD+5cx9HkP2P+XQ+GI9u+/exu67tznRbiV55JTk\nH3E+XlMDixf7WLjQS1WVC7fb5qqrnHn2mWdqnt0T1LxFhNRT0jlh0UlE6sM0rGiPdnujiba1LeSU\nOKcbRluj4AJXSmcj//RTg4oKH08/7aW11SA72+amm5wQ34EDNc/uSWreItLB7feQO7UfuVOdaLfA\nx60dq+66J/ew6/9tI/uKXLaM7MdvV+Xz4kvOPVYGD45yww0BrrkmRGbm4V5Buouat4gclG+Qb7+5\ndyQKAcNF3WPVZFHNDFL4anEBw2/M5eJZvoQM8U1k+naLyGHV1cGjj/pYuPBEdtUN5UyjlhmDdnJK\ndTXFO7eS/kIdnjITADtqK2i5l6h5i8hBff65wfz5Pp54wktLi0FGhs31pSFmzfIxZMgJRFsH0vhS\nPe7sznPKt8/ZRGh7EP+388ieoGi3nqTmLSIdbBveecdNebmX557zYNsGAwZEuf32ANddFyI7u/Nz\n90a7dX6tTXh3iObVjTSvbmTHHZvJuiyHnMmKdusJat4iQjgMzz7rnJ/9/vvOavmMM5zQgyuuCOM9\niuwHwzAY8tRwgv8IdEa7PVNLwzO1FMztT+GPBvTwXiQXNW+RJNbYCI895mX+fB9bt7owDJvLLgtx\n440hvvrVroUeHCzaLXuSs0K3bZt/TPmE9LMyFO12nNS8RZLQli3OPPuxx7w0NRmkp9vMnBnkhhuC\nDBvWPednHyzare3zNlrfaaL5tQaq7t1B2pfS8U9Ormi37qLmLZJE3n/fCT1YscIJ8S0qijJnTpBp\n04Lk5h75649X2rA0zHWn0/BCPfWVe2h6rYHWD5xot6HLTdLP1kniR0vNW6SPi0TghRecEN+333Z+\n5E85JUJZmRPi6zvKW5ikLKsk/f57cG/4mMiIkbTcchuBSSXHXI8rw03O5DxyJucR3u1EuzWurCP1\nzHQAQrtC7Pr3rfi/ndcnot16ipq3SB/V1ARPPumE+G7a5JzpcfHFTujBBRcc2zw7ZVkl2aUzOx57\n1q8ju3QmDdClBt7xPIVe8q8vJP/6wo5tjSvrOu58mMjRbj1NzVukj9m+3WDhQi+PPuqjvt4gNdXm\nuuucEN8RI7p2k6j0++85+PYH7j2u5n0wudf1I3VUGnWVNTQs74x2SxmRyrD/HbXfvVWS2WGbt2ma\nXmARcCKQAtwNfAQsBmxgLXCTZVm6bZhIjP397y7mzfPxzDMewmGDfv2i3HGHE+Lbr9/xHYR0b/j4\nmLYfD8MwSP9KJulfyaT/3e3RbpU12G3Rjsbd/FYTgY9ayJ6YFxfRbrFwpL2eCuyxLOs60zTzgQ+A\nNcBdlmW9ZppmOTARWNbDdYrIQUSj8PLLTojv6687P86m6YT4Tp4cIrWbzsSLjBiJZ/26g27vSftG\nu9l25y+gmoW7aVhey467tjjRbiV5ZF3S+9FusWTs+w35ItM0MwHDsqzG9ub9Ls4KfJBlWbZpmhOB\nSyzLuulwLxIOR2yPR5fJinSXlhZ49FG47z7YsMHZNn483HYbXHIJXTo/+7CefBK++90Dt//+93D1\n1d38YkcW2BZg95O72fX4Lpo+aAKc6LdBcwYx9OdDe72eHnTI/8nDrrwty2oCME0zC6gE7gJ+bVnW\n3o7fCPiP9Oq1tS1HXenBFBRkUVXVeFzPkUiSbX9B+3y0du0yePhhL4sXe6mpceHz2Xz3u2FKS4OM\nHu1ML6ure6DYb0wgpWIR6Q/c23m2yZy5BL4xAY5hH7rt/9kHqdNyGDIth7aP26Pd/lhDS2uw4/kb\n/7ceb5GXlFPSYnqg83j2uaAg65AfO+KwyDTNE3DGIg9alvWEaZq/2ufDWUBdl6oSkaP20UcuKip8\nLF3qIRg0yMuLMndugBkzQhQV9U7oQWBSSbcfnOwOqSPTSP3XgRTeOQA74Hwv7KjNjts3E9oaPO5o\nt3h1pAOWRcCLwD9blvW/7Zs/ME3zIsuyXgMuB17t2RJFkpNtw6uvOvPsVaucH9WTTopSWhpgypQQ\n6ekxLjDOGC4DI619hW1D8d0nUF+554Bot8LbB5BxwaFXtIniSCvvHwO5wE9M0/xJ+7Y5wG9M0/QB\n63HGKSLSTdraYOlSLxUVXj7+2DlWdP75YcrKgowfH8GVPMfkusxwG2R/M4fsb+YQqQvT8OfOaLdo\nsPPkuOY3Gkkbk5GQpx8eaeY9B6dZf9HYnilHJHlVVxssXuxl0SIv1dUuPB6byZNDzJ4d5PTTdTZu\nV7lz9o928xY791AJbQvyj0kbcGW78V+Zi78kj/SvZiZMmERyniApEkc2bHBRUeHlD3/w0tZm4Pfb\n3HxzgO9/P8SAAQrx7U77zbzdkD+7iPo/1lC7pJraJdV4B/nwX5VHfmkhnoL4vlGWmrdIDNg2vPIK\n/Od/pvHyy86P4ZAhzjz76qsV4tsbvMU+iv99EEU/GUjz643UL62hYUUt1Q/uJH92EQDRQJRITRhv\n//g70KnmLdKLgkFYtsxDebmPdesAPJx9dpjZs0NcdlkYty6H6HWG2yDza9lkfi2b/r8cTOua5o6r\nNhtfrGfrrI1kXJAVd9Fuat4ivaC21gnxXbDAy65dLlwumylTYMaMZsaM0Tw7XrjSXGSc23kmiivD\nRdqXMw6MdivJI3O8P6bnj6t5i/SgjRsNKip8PPWUE+KbmWlTVhZk1qwgY8ZkUlWlxh3Pssb5yRrn\nPyDaLfBRK5njnesTo61RjFSj1xu5mrdIN7NtePttNw8+6GXlSifEd9CgKD/8YYBrr90/xFcSw37R\nbn9rIVIX6WjWu3+5ncbn65xEoJI8Uk7qnWg3NW+RbhIKwYoVzjx7zRpnLnrWWU6I74QJYTz6aUt4\nhmGQdmbGAdvDu0NU3bODqnvao91K8vFPzO3RaDe9nUSOU0MDLFniZcECH9u2OSG+EyaEKCsLcfbZ\nXQvxlcRR/B+DKLyj/wHRbm3rWhh4/4k99rpq3iJdtGmTwYIFTohvc7MT4jtrVpDrrw8ydKjOz04m\nB4t2S//ygSv07qTmLXKM3n3XCfF99lkP0ahBcXGUW291QnxzcmJdncTa3mi3Hn+dHn8FkT4gEoHn\nnvMwb56P995z5tmnnurMsydOPPoQX5HuouYtchhNTfDEE14eesjH5s3OzYsuvdS5SdR552meLbGj\n5i1yENu2OfPsJUu8NDQ4Ib7TpwcpLQ1y8smaZ0vsqXmL7GPNGmeevXy5h0jEoKAgyo9+FGT69BD5\n+WraEj/UvCVppSyrJP3+ezAsi+UDyrg37V95/ZNiAEaNilBWFuSqq8KkpMS4UJGDUPOWpJSyrBJ3\n6T/zENO5j2V8unU4AN8YvZXSf89j7FjNsyW+qXlL0tm50+CJH4d4iM3UkI+PADNZyK3cx0gMai96\nI9YlihyRmrckjbVrnXn2smUeQqEb6EcV/8Z/cCMPUsRuAOwN+pGQxKB3qvRp0Si88ooT4rt6tfN2\nHz48wq2NP+d7O/+LNNr2+/zIiJGxKFPkmKl5S5/U2gqVlU6I74YNzkU1F14YZvbsIOPGRUhbPoS0\n0rYDvq5lztzeLlWkS9S8pU+pqjJ4+GEvixc7Ib5er82UKSFKS4OcdlrnvbMDk0poANIfuBf3ho+J\njBhJy5y5BCaVxK54kWOg5i19gmW5KC/3UlnpJRAwyMmxmTMnwMyZIfr3P/j52YFJJWrWkrDUvCVh\n2TasWuWmvNzHK684b+WhQ50Q3+98J0RGz97UTSSm1Lwl4QQCTojvvHk+1q935tnnnOOE+F5yiUJ8\nJTmoeUvCqKmBxYt9LFzoparKhdttc9VVzjz7S19SFqQkFzVviXuffWZQXu7j6ae9tLYaZGXZ3Hij\nE+I7aJDuNyLJSc1b4pJtwxtvOPPslSudt+ngwVFuuCHANdeEyMyMcYEiMabm3YfsvdFSx6lvt9yW\ncGdThEKwfLkT4vvhh87wesyYCDfeGOTyyxXiK7KXfhT6iJRllWSXzux47Fm/juzSmTRAQjTw2lr4\nzW+cefaOHS5cLpsrrghRVhbkK1/RPFvki9S8+4j0++85+PYH7o3r5v355wbz5/v4/e+huTmFjAyb\nG25wQnyHDNE8W+RQ1Lz7CPeGj49peyzZNrzzjpvyci/PPefBtg0GDYIf/KCNqVND+P2xrlAk/ql5\n9xGRESPxrF930O3xIhyGZ591zs9+/31nnn3GGU6I78yZadTVhWJcoUjiUPPuI1puuW2/mXfH9ji4\n0VJjIzz2mJf5831s3erCMGwuuyzE7NkhzjnHCT3wemNdpUhiUfPuI+LxRktbtjjz7Mce89LUZJCW\nZjNjhhPiO2yY5tkix+Oomrdpml8F/suyrItM0zwZWAzYwFrgJsuydDpAHIiXGy29/74TerBihRPi\nW1QUZc6cINOmBcnNjXV1In3DEZu3aZp3ANcBze2b7gXusizrNdM0y4GJwLKeK1ESQSQCzz/vobzc\nyzvvOG+r0aOdefakSWF8vhgXKNLHHM3K+zPgKmBJ++MxwKr2fz8PXIKad9JqaoInn/RSUeFj0yYX\nABdfHKasLMiFFyrEV6SnHLF5W5a11DTNE/fZZFiWtXdg2Qgc8cSu3Nx0PJ7ju9VbQUHWcX19oon3\n/d26FX77W6iogLo6SEmB669K3HkpAAAK/UlEQVSHW2+FUaM8dOVwSrzvc0/QPieHntjnrhyw3He+\nnQXUHekLamtbuvAynQoKsqiqajyu50gk8by/H37ozLOfecZDOGzQr1+UO+4IMX16iIIC53d6VdWx\nP28873NP0T4nh+PZ58M1/a407w9M07zIsqzXgMuBV7tUlSSMaBReesm5SdTrrztvGdOMUFYWYvLk\nEKmpMS5QJAl1pXnfBsw3TdMHrAcqu7ckiRctLfD00848+7PPnHn22LFOiO/Xv655tkgsHVXztizr\nH8A57f/eAIztwZokxnbt6gzxralxQnyvvtq5SdTo0TorVCQe6CId6fDRRy4qKnwsXeohGDTIy4sy\nd26AGTNCFBXpohqReKLmneRsG1591c28eT5WrXLeDied5IT4TpkSIj09xgWKyEGpeSeptjZYutRL\nebkXy3JO4zz/fOf87PHjI7hcMS5QRA5LzTvJVFcbLF7sZdEiL9XVLjwem8mTQ8yeHeT00zXPFkkU\nat5JYsMGFxUVXv7wBy9tbQZ+v83NNwf4/vdDDBigebZIolHz7sNsG1avds7Pfvll5796yBBnnn31\n1QrxFUlkat59UDAIy5Y5Ib7r1jnz7LPPDjN7dojLLgvjPr47FYhIHFDz7kNqa+HRR30sWOBl1y4n\nxHfiROf87DFjNM8W6UvUvPuAjRsNKip8PPWUl5YWg8xMm7KyILNmBRk8WPNskb5IzTtB2Ta89Zab\nefO8rFy5N8Q3yh13BJg6NUR2dqwrFJGepOadYEIhWLHCmWevWeMMr886ywk9mDAhjEf/oyJJQT/q\nCaK+3gnxXbDAx7ZtTojvhAkhyspCnH22bhIlkmzUvOPcpk1OiO/jj3tpbjZIT7eZNSvI9dcHGTpU\n82yRZKXmHafefdcJPXj2WQ/RqEFxcZRbb3VCfHNyYl2diMSamncciUTguec8LFgAb76ZAcBppznz\n7CuvVIiviHRS844DTU3wxBNeHnrIx+bNzh2hLr3UuUnUeedpni0iB1LzjqFt25x59pIlXhobDVJT\nbaZPD3LnnT7y8lpjXZ6IxDE17xhYs8aZZy9f7iESMSgoiHLTTUGmTw+Rn29TUODrUoiviCQPNe9e\nEo3CypUeysu9vPmm820fNcqZZ0+aFCYlJcYFikhCUfPuYc3N8NRTTojv55878+xx45x59tixmmeL\nSNeoefeQnTsNFi3y8sgjPmprDXw+m2uvDVJaGmLkSN0kSkSOj5p3N1u71plnL1vmIRQyyM+Pcttt\nQWbMCFFYqItqRKR7qHl3g2gUXnnFCfFdvdr5lg4fHqG0NMS3vx0iLS3GBYpIn6PmfRxaW6Gy0ktF\nhZcNG5ybRF14YZjZs4OMG6cQXxHpOWreXVBVZfDww14WL3ZCfL1emylTQpSWBjntNM2zRaTnqXkf\nA8tyUV7upbLSSyBgkJNjM2eOE+JbXKx5toj0HjXvI7BtWLXKCfF95RXn2zV0qBPi+53vhMjIiHGB\nIpKU1LwPIRBwQnznzfOxfr0zzz733DBlZSEuuUQhviISW2reX1BTA4sX+1i40EtVlQu32+aqq5wQ\n3zPP1DxbROKDmne7zz4zKC/38fTTXlpbDbKybG680Qk9GDhQ82wRiS9J3bxtG954w5lnr1zpfCsG\nD45yww0BrrkmRGZmjAsUETmEpGzeoRAsX+6E+H74oTO8HjMmwo03Brn8coX4ikj8S6o2VVcHS5b4\nWLDAy44dLlwumyuucObZX/mK5tkikjiSonl//rkTevDEE15aWgwyMmxuuMGZZw8Zonm2iCSePtu8\nbRveecdNebmX557zYNsGAwZEuf32AFOnhvD7Y12hiEjXdal5m6bpAh4EzgACwCzLsj7tzsK6KhyG\nZ5915tl//aszzz7jDCf04Iorwni9MS5QRKQbdHXl/S0g1bKsc03TPAe4B5jYfWUdu8ZGePxxL/Pn\n+9iyxYVh2Fx2WYjZs0Occ45CD0SkbzFs+9hnvqZp3gu8Y1nWk+2Pt1mWNfBQnx8OR2yPp2cuSdy8\nGR54AObPdxp4WhrMmAG33ALDh/fIS4qI9JZDLju7uvLOBur3eRwxTdNjWVb4YJ9cW9vSxZdxFBRk\nUVXVuN+29993Qg9WrHBCfIuKovzLv4SYNi1Ibq7zOYka4nuw/e3rtM/JQft87F97KF1t3g3Avs/q\nOlTj7k6RCLzwghPi+/bbTumjR3eG+Pp8PV2BiEh86Grzfh24Ani6feb99+4r6UBNTbBwoRPi+49/\nOAkHF1/shPheeKHm2SKSfLravJcB403TfANnJjOj+0rqFArBf/+3j8WLoa4ulZQUm+uuc0J8R4zQ\nRTUikry61Lwty4oCZd1cywHef9/N/fenUFgId9wRYPr0EAUFuqhGRCSuL9I5++wIL7/czPnnZ9DY\nGIx1OSIicSOuI3INA04/PUpqaqwrERGJL3HdvEVE5ODUvEVEEpCat4hIAlLzFhFJQGreIiIJSM1b\nRCQBqXmLiCSguG7eKcsqyR17Lng85I49l5RllbEuSUQkLsTtFZYpyyrJLp3Z8dizfh3ZpTNpAAKT\nSmJXmIhIHIjblXf6/fccfPsD9/ZyJSIi8Sdum7d7w8fHtF1EJJnEbfOOjBh5TNtFRJJJ3Dbvlltu\nO/j2OXN7uRIRkfgTt807MKmEhopFhEefCh4P4dGn0lCxSAcrRUSI47NNwGnggUklFBRkUZtkoaUi\nIocTtytvERE5NDVvEZEEpOYtIpKA1LxFRBKQmreISAIybNuOdQ0iInKMtPIWEUlAat4iIglIzVtE\nJAGpeYuIJCA1bxGRBKTmLSKSgNS8RUQSUNzeVdA0TRfwIHAGEABmWZb1aWyr6lmmaXqBRcCJQApw\nt2VZf4ppUb3ENM1C4K/AeMuy+nxckmmadwJXAj7gQcuyFsa4pB7V/t5+BOe9HQGu76v/z6ZpfhX4\nL8uyLjJN82RgMWADa4GbLMuKdsfrxPPK+1tAqmVZ5wI/Ag4eatm3TAX2WJZ1IXA58NsY19Mr2n+w\nK4DWWNfSG0zTvAg4DzgfGAucENOCesc3AY9lWecBPwN+EeN6eoRpmncAC4DU9k33Ane1/0wbwMTu\neq14bt4XAC8AWJb1FvDl2JbTK/4A/GSfx+FYFdLLfg2UA9tjXUgvuRT4O7AMWAH8Obbl9IoNgKf9\nL+psIBTjenrKZ8BV+zweA6xq//fzwMXd9ULx3Lyzgfp9HkdM04zbMU93sCyrybKsRtM0s4BK4K5Y\n19TTTNP8HlBlWdbKWNfSi/rhLEa+DZQBj5umacS2pB7XhDMy+RiYD/wmptX0EMuylrL/LybDsqy9\n9yBpBPzd9Vrx3LwbgKx9Hrssy+rzK1HTNE8AXgWWWJb1RKzr6QUzgfGmab4GnAk8appmcWxL6nF7\ngJWWZQUty7KANqAgxjX1tFtx9nkEznGsR0zTTD3C1/QF+863s4C67nrieG7er+PMyTBN8xycPzP7\nNNM0i4AXgR9alrUo1vX0BsuyvmZZ1ljLsi4C1gDTLMvaGeOyetpfgMtM0zRM0xwAZOA09L6sls6/\npGsAL+COXTm95oP2YxzgHMda3V1PHM9jiGU4K7I3cAb9M2JcT2/4MZAL/MQ0zb2z78sty0qKA3nJ\nwrKsP5um+TXgHZwF1E2WZUViXFZPuw9YZJrmapwzbH5sWVZzjGvqDbcB803T9AHrccah3UK3hBUR\nSUDxPDYREZFDUPMWEUlAat4iIglIzVtEJAGpeYuIJCA1bxGRBKTmLSKSgP4/dwH4AcLBLzUAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1083b6358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "yr = np.polyval(reg, x)\n",
    "plt.plot(x, y, 'ro')\n",
    "plt.plot(x, yr, 'b')\n",
    "plt.plot(x, l1, 'm--');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network for Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://iamtrask.github.io/2015/07/12/basic-python-network/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sigmoid Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://en.wikipedia.org/wiki/Sigmoid_function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sigmoid function\n",
    "def sigmoid(x, deriv=False):\n",
    "    if deriv == True:\n",
    "        return sigmoid(x) * (1 - sigmoid(x))\n",
    "    return 1 / (1 + np.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-10, 10, 250)\n",
    "y = sigmoid(x)\n",
    "d = sigmoid(x, deriv=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = np.where(x > 0, 1, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAHRCAYAAACo+PfiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xl0VPX9//HnnT0bEEjYoqzCZZFF\nBURFqxb3qtQNd1yoWq0L2K91RynWtrj9ulBrK26ICCp1x1rrCqi4VJHlIrLvAQIJWSaZmfv7YxIM\nJIGQzMydzLwe5+QkM/fOfN4fBoZXPvfO+xq2bSMiIiIizedyugARERGRVKFgJSIiIhIjClYiIiIi\nMaJgJSIiIhIjClYiIiIiMaJgJSIiIhIjHqcLqFFYWBL3vg+5uZkUFZXFe5iklM5zh/SefzrPHdJ7\n/pp7es4d0nv+iZp7fn6OUd/9abVi5fG4nS7BMek8d0jv+afz3CG956+5p690nr/Tc0+rYCUiIiIS\nTwpWIiIiIjGiYCUiIiISIwpWIiIiIjHSrGBlmuaRpml+UM/9Z5qmucA0zfmmaf6iOWOIiIiItBRN\nDlamad4G/BMI7HW/F3gUOBn4CXCNaZodm1OkiIiISEvQnD5WPwDnAM/tdX9fYLllWUUApml+AhwL\nzGrGWCIiB6xyTZB5gxaSd3snci/KA2DFKUuoWFxeZ99WP8vloL91B2DzA+vZ9vjmOvu4ctz0WTwI\ngNJPSlh90ff1jtv9NZOMw7IAWNz1K4jU3Sf/1k7k39IJgDWXLWfXB8V19skclk23l3sDsO2fW9h8\n/7p6x+uzbDCuDBfBZeX88NMl9e5z0D960OrUNgB8P+w7qjZW1tkn97I8Ov2uCwAbbl3Njpnb6uzj\n7eKn19z+AOx8rYj1N6ysd7xDPu6Pr5uf8M4Q1qHf1rtPxwcOpu3l+QCs/NlSyr+p23so5+TWHPxk\nTwC2TN7A1j9tqrOPEXDR9/vBAJR+uovV5y+rd7yus3qTeWQ2FRWw8tD/YQejL4xdq4ti2ajO7BxV\nQFWVQbs/f0/mN0V1nqe4UzafXXIoVVVQ8NVmBn5U/WdgQ+2GjM+cNYxyt4fskgou/c9X9db0Sh+T\nRXl5RCJw45dfkltRUWefz9p04IVOvbBtuGDTco7ZUffPoMjj584uwwAYuGsbv9zy1R7F1Px4T8cj\n2ODJxBcJ87cN8+qt6fnsHrydUYBtw707v8Ws2llnn/952vK7rEOxbTgvuJrRwdV19rExOD/nWAB6\nhYt5sOx/9Y43ITCAhe5cAJ4rnUcWoTr7zPYczLO+6L/PXweXcEy4sM4+K41sxgUOp08fePNN8DjU\nqbPJw1qW9bJpmt3q2dQKqP0qlACt9/d8ubmZCek9kZ+fE/cxklU6zx3Se/7pOvdtn1dSubGS4Hu7\nyL8p+qa8tX8OHl/d95rWZvbuP6fgIdkEB5fW2ced7d69j/8gm22Ds+sdt11BDln50WCVMzgHO1K3\n/3Fuz5zdz7Wzbw7sqJu+svv+WFOkexmlDYyX1z4bd8BN+S4PmxuqqUsObaqfa9OAbCo71A1WbXr9\nWFOZmUNocLDOPv4C/+593F2r2NnQeJ2yCeQHCAVCrG9gn7bdfhxvW/8c3Ebdgyit+0T3KSuDSJsc\njF67CIcgHI5+hcIQwsUf/5jDzp3gXQ3HBLIJhSESqf6q/vmG87NZFIyO9xjZeOtJvHNmZfH6rEwA\nriWLgVTV2Wf12iz++Hs/AMeRgZ/65/fyv7yU4yGPMMMa2OezxT6+qv6v+CSyyMODAWCw+/uqsgw2\nbIj+nV0TzKCzK7t644+KDR+7drkwDNhR5WWVJ/vHXWrtG3K5cbtduFw2a317Pk/Nj6FsH7mtqp8r\nlMH6inCd5ynNzKBLZwPDAM92Pxu2152fjUHPntEHtS/3sGFt9t5lA9DhYA+VWdEthd9nURwO15mf\nr62Pfh2id0bWBdhYUne8XYFMDu1h0KcPdOyYg1Fv+874M2y76Q3Pq4PVDMuyhte6byDwe8uyTq++\n/Sgw17Ksl/b1XInovJ6fn0NhYUm8h0lK6Tx3SO/5p/Pci+fsYO3lP9DhvoPIu76D0+UkXLK/9lVV\nsHmzwYYNBhs2uNiwwWDjRhdbtxps326wbduP38vLD+x/SbcbsrJsMjJsAgHIzIx+r7kdCNhkZERv\nZ2REb/v94PWC12tXf9//bY/HxueLro54veB227hc4HJFa6j5bhjR7zXb97yP3Y+JVRhI9tc+nhI1\n94Y6r8djoWwJ0Ms0zbbALuA44KE4jCMism/h6O9rRvo2oXZUJAKbNhmsWOFi5UoXK1a4WLUqGp7W\nrzfYssXAthtOEpmZNm3b2vTuHaFtW5t27aJfrVvb5OTYtGplk50NrVpFf87JscnJgZwcmy5dcti6\ndVcCZysSFbNgZZrmxUC2ZVlPmKY5HniH6MnxUy3LWh+rcUREGsuuOdLjduiYQJqw7WiAWrTIxeLF\nbhYtcrF0qYtVq1z1rjT5fDadOtkMHx6mUyebgoIInTtH7+vUKUJ+fjRQZWY2vSanDgOJNCtYWZa1\nChhe/fP0Wve/DrzerMpERJqrZsVKHftiqrDQ4Isv3CxY4OJ//3OzaJGboqI9k0xmpk3PnhF69Pjx\nq3t3m+7do8FJwUdSlUPnzIuIxJ+vm5/Ov+yMr2+G06W0WLYNy5e7mD/fzeefu1mwwM3KlXsm1e7d\nIxx9dIj+/SP06xehf/8wBx8cPZdIJN0oWIlIysoYnEWXkzqm7Um8TVVUBB995OGDD9x88IGH9et/\nTEitW9v89Kchhg4NM3RomMMOC5Nd/wfeRNKSgpWIiLBxo8Fbb3l46y0P8+a5CYejx+pyc21Gjapi\nxIgww4aF6d07opUokX1QsBKRlFU6v4Rt/1pPxrmtyRymZZW9FRXBv/7lZdYsL1988eNHJw8/PMwp\np4Q44YQQAwZEcOtTlSKNpmAlIikruLyCTU9tomBIQMGqWjgM777rZsYML+++66GqysDlshkxIsQZ\nZ4Q47bQQnTvHva2gSMpSsBKR1FXTMFrtFti2zeD55708/bSXdeuix/L69g1zwQVVnHtuiI4dFaZE\nYkHBSkRSll3dboE0PpT1/fcu/vIXH6+84iEYNMjMtBkzppLLL6/i0EMjansgEmMKViKSuqpzleFK\nv/SwaJGLv/0NZs3KxLYNevSIcNVVQUaPrqL1fq/eKiJNpWAlIqmrZsUqjT7FZlkuHnjAx5w5XgAG\nDowwblwlp50W0qf5RBJAwUpEUpYrx02gRwBXduofC9y82WDyZB/TpnmJRAyGDAkzcaKbI44o0+E+\nkQRSsBKRlJV7cR69b+6e0g1CKypgyhQff/6zj9JSg169wkyYEOSkk8K0b59DYaHTFYqkFwUrEZEW\nau5cN7feGmDFChd5eREmTAhy6aVVePTOLuIY/fMTkZRVsaScLf8tJ3yoB097r9PlxExREdx/v5/p\n0324XDbXXFPJb34TJCfH6cpERMFKRFJW8RtFFE7eSNeXe5GdIsHqnXfcjBsXYOtWF/37h3nkkQoO\nOyzidFkiUk3BSkRSVwo1CC0rg/vu8/P00z78fpt77gly3XWVeFMjL4qkDAUrEUlZdqSm3ULLDlaL\nFrm47roAluWmb98wjz9eQd++WqUSSUbqaiIiqau6j5XRgrstTJvm5ZRTMrEsN2PHVjJnTplClUgS\n04qViKQsuyZ/tMBDgVVVcPfdfp56ykdurs3UqWWcfHLY6bJEZD8UrEQkddWsWLWwtfnCQoOxYwPM\nn++hb98wzzxTTrduukiySEvQwt5uREQaL++WThy5/Ej8fTKcLqXRFi1yccopmcyf7+FnP6vizTfL\nFKpEWhAFKxFJWZ62HjJ6ZuAKtIy3uvnz3Zx9dibr1rm4/fYgTz5ZQXa201WJyIHQoUARSVmR0jBV\nnirssJ30LRfeesvDtdcGiETg8cfLOeeckNMliUgTtIxf40REmmDTveuY23YulSuCTpeyT8895+Wq\nqwK43TBtmkKVSEumYCUiKWt3H6skbrfwl794ufXWALm5NrNnl3HCCfrkn0hLpkOBIpK6ajqvJ2mD\n0ClTvEycGKBz5wgvv1xGz546SV2kpVOwEpGU9WPndWfrqM/jj3u5774AnTpFmD27jO7dFapEUkES\nvt2IiMRIkl4r8B//8HLvvQE6dlSoEkk1ClYikrKS8RyrZ57xctddAdq3j/DKK2X06KFQJZJKmnQo\n0DRNFzAFGAQEgbGWZS2vtf3XwEVABPidZVmzY1CriMgByb04jw4n5eFulRxnPbz5pofbbvOTlxdh\n9uxyDjlEoUok1TR1xWoUELAs6yjgduDhmg2mabYBbgKOAk4GHmtukSIiTZH9k1YU/LIAV6bzi/Of\nfebml78MkJEB06eX06uXLqQskoqa+m4zApgDYFnWp8CQWttKgdVAVvWX3j1EJK0tW+bisssyqKqC\nqVPLGTxYb4siqaqp6+OtgJ21bodN0/RYllXT1W4tsJjomQ0PNuYJc3Mz8XjifyJEfn5O3MdIVuk8\nd0jv+afr3JePW86aT5Yx+IPBuLOcOdFqwwa4+GLYsQOeegpGj85M6Pjp+tpDes8d0nv+Ts69qcGq\nGKhdtatWqDoN6AR0r779jmmacy3L+nxfT1hUVNbEUhovPz+HwsKSuI+TjNJ57pDe80/nue9YWMyu\nL0rYWliCqyzxwaqsDM46K5M1a9zceWeQM86opLAwceOn82ufznOH9J5/oubeUHhr6qHAucDpAKZp\nDgcW1tpWBJQDQcuyKoAdQJsmjiMi0mR2TRNzB9ot2DaMHx/g22/dXHxxJTffXJnwGkQk8Zq6YjUb\nOMk0zXmAAVxpmuZ4YLllWa+ZpjkS+NQ0zQjwCfBubMoVETkA4ein7pzoY/WXv/h45RUvQ4aE+cMf\nghjJ1UpLROKkScHKsqwIcN1edy+ttX0CMKEZdYmINJtTfaz+8x83kyb56NQpwlNPleP3J3Z8EXGO\n859BFhGJFweuFbh8ucF112Xg98Mzz5TToYN6VYmkk+TomiciEgeZw7PJ7BhI2HilpXDllRkUFxv8\n9a9qqyCSjhSsRCRldbizIKGfjrrjjgCW5eYXv6jk/PND+3+AiKQcHQoUEYmBF1/0MGOGl8GDw9x7\nb9DpckTEIVqxEpGUte3JLZRWbSPrunZxHWfZMhe/+U2AnBybJ57Qyeoi6UwrViKSsnZM28rah9bG\ndYyyMhg7NkBZmcFjj1XQrZtOVhdJZwpWIpKy7Igd9x5Wd9/tZ+lSN1dfXcmZZ+q8KpF0p2AlIqkr\nHN/moG+95WHaNB8DBoS57z6dVyUiClYiksLssB235qBbthjceqsfv99mypQKnVclIoBOXheRVBan\nFSvbhltvDbBtm4tJkyqIXr1LRETBSkRSmBEwcHljvzD//PNe3nnHw7HHhhg7tirmzy8iLZeClYik\nrEM+6h/zBqErVxrcfbefVq1s/vSnClw6oUJEalGwEhFppHAYbrwx2lrhb38rp6BArRVEZE8KViKS\nssoW7KK4vQ1dY3Oe1dSpXj7/3MNZZ1VxzjlqrSAidWkRW0RS1poxP7D0sqUxea7Vqw0eeMBPbq7N\ngw8GMeLbHktEWiitWIlI6opRu4WaTwGWlRlMnlxOfr4OAYpI/bRiJSIpy45Ru4UXXvDw0UceRo4M\ncd55OgQoIg1TsBKR1BVu/iVtNm0yuPfeANnZNpMnV+gQoIjskw4FikjKisW1An/zGz/FxQaTJ1fo\nU4Aisl9asRKR1BWmWedYvf22h7ff9nL00SEuu0yNQEVk/7RiJSIpq9vs3rQryKGcA19p2rUL7rzT\nj9drM3lyUI1ARaRR9FYhIikrc2g22YOym/TYyZP9rF/v4sYbK+nVS9cCFJHGUbASEdnLd9+5eOIJ\nL127Rrj55kqnyxGRFkTBSkRSkh2yWXzQV3x7+rcH9LhIBP7v/wKEwwZ/+EMFGRlxKlBEUpKClYik\nJDtsY1fa2KEDO79q2jQvX37p5uyzqzjxxHCcqhORVKVgJSKpqToTHUi7hW3bDCZN8pOdbfPb3wbj\nVJiIpDIFKxFJSXakeqXqANot/OEPPnbsMLjttiAdO6pnlYgcOAUrEUlN4WgwauyK1eLFLp591ssh\nh4S5+mr1rBKRpmlSHyvTNF3AFGAQEATGWpa1vNb204AJ1Te/Am6wLEu//olIwtgHcCjQtuGee/xE\nIgYTJwbxeuNcnIikrKauWI0CApZlHQXcDjxcs8E0zRxgMvAzy7KGA6uAvGbWKSJyQFx+g/xbO5F/\nQf5+950zx8PHH3s48cQQI0fqhHURabqmBqsRwBwAy7I+BYbU2nY0sBB42DTNj4HNlmUVNqtKEZED\n5Mpy0/43nelwYYd97hcMwoQJftxum4kTdcK6iDRPUy9p0wrYWet22DRNj2VZIaKrUycAg4FdwMem\nac63LGvZvp4wNzcTj6cZF/VqpPz8nLiPkazSee6Q3vNP57nDvuc/eTKsWgU33QTHHJOVuKISJJ1f\n+3SeO6T3/J2ce1ODVTFQu2pXdagC2AYssCxrE4Bpmh8RDVn7DFZFRWVNLKXx8vNzKCwsifs4ySid\n5w7pPf90nXuosIr141bT4bQ8Ape0qXefLVsMJk7MIjcXbrhhF4Uptraerq89pPfcIb3nn6i5NxTe\nmnoocC5wOoBpmsOJHvqr8SVwqGmaeaZpeoDhwOImjiMi0iSRsgi7/r2T0m9KG9zn97/3sWtXtL1C\nbm4CixORlNXUFavZwEmmac4DDOBK0zTHA8sty3rNNM07gHeq951pWdZ3MahVRKTR7Op2Cw39+rhw\noYvnn/fSp0+YMWPUXkFEYqNJwcqyrAhw3V53L621fQYwoxl1iYg0TyT6rb52C7YNd9/tx7aj7RU8\nTf0VU0RkL2oQKiIpyd5Hg9A33vAwf76HU04Jcfzxaq8gIrGjYCUiqakmL+31YeOKCrj/fj9er839\n91ckvCwRSW0KViKSkoyAQcYRWWR0z9jj/scf97FmjYuxY6vo0UMXhBCR2NKZBSKSkvw9AvR4u88e\nH73etMngscd85OVFuPVWNQMVkdjTipWIpI0HHvBTVmZw++2VtGrldDUikoq0YiUiKalqcxXFrxfh\nOyECPV18/bWLF1/00r9/mEsuUXsFEYkPrViJSEqqXBlk051r2fb6tur2CgEAJk0K4o7/1bNEJE0p\nWIlIaor82G5h9mwPCxa4OeOMKo45Ru0VRCR+FKxEJCXV9LEKRQwmTvTj89lMmKAT1kUkvhSsRCQ1\nVXde/2QebNjg4rrrKunWTe0VRCS+FKxEJDVVr1h9+LFB+/YRbrml0uGCRCQdKFiJSEqyq1esgiGD\nu+4Kkp3tbD0ikh4UrEQkJS3NbsMZjGDlYQWMHh1yuhwRSRPqYyUiKScSgbsnZFCGm4f+BC79Ciki\nCaK3GxFJObNmeVj6tc0VI4sZPkCrVSKSOApWIpJSdu2CSZP8jPBuY8x/vmLztM1OlyQiaUTBSkRS\nyp//7GPzZhennFS9UqUu6yKSQApWIpIy1qwxmDLFR6dOEU4ZGb0eoOE2HK5KRNKJgpWIpIyJE/0E\ngwZ33x3EX/3RHAUrEUkkBSsRSQnz57t57TUvhx8e5txzQ9i1rhUoIpIoClYi0uKFw3D33X4AJk2q\nwOX68VqBOsdKRBJJwUpEWrwZM7wsXOjm3HOrGDIk2nI9+7hWHPRkD9oc28bh6kQknahBqIi0aCUl\n8MADPjIzbe65J7j7fl9XP76ufgL5AUoKqxysUETSiVasRKRFe/RRH1u3uvjVryrp3Nl2uhwRSXMK\nViLSYq1cafDEEz4KCiJcf33lHtuKpm9l2ZCFbP/3doeqE5F0pGAlIi3WhAl+KisN7r03SGbmntvC\nO8JUrakkUhFxpjgRSUsKViLSIv33v27mzPFy1FEhRo2q53qAYbVbEJHEU7ASkRanshLuuiuAy2Xz\nwANBjHqyU00fK7VbEJFEatKnAk3TdAFTgEFAEBhrWdbyevZ5E3jVsqzHm1uoiEiNf/zDyw8/uLjq\nqkoOPbSBQ33h6DetWIlIIjV1xWoUELAs6yjgduDhevaZBLRtamEiIvXZvNngoYf8tG0b4Te/CTa4\nn61DgSLigKYGqxHAHADLsj4FhtTeaJrmeUAEeLtZ1YmI7GXiRD+lpQZ33FFJbm7D+2UMzKTNpXn4\nC/yJK05E0p5h2wfe98U0zX8CL1uW9Xb17TVAD8uyQqZpHgpMBM4D7gU2NeZQYCgUtj0enQwhIg2b\nNw+OOQYOOwwWLAC33jJExDn1Loc3tfN6MZBT67bLsqyaj+VcDhQA/wW6AZWmaa6yLGvOvp6wqKis\niaU0Xn5+DoWFJXEfJxml89whveefKnMPh+GXv8wE3EycWMb27eFGPS5V5t8Umnt6zh3Se/6Jmnt+\nfk699zc1WM0FzgRmmqY5HFhYs8GyrNtqfjZN8z6iK1b7DFUiIvszfbqXb791c955VRx55P5D1c7X\niyj9oJjsiT0hKwEFiojQ9HOsZgMVpmnOAx4FxpmmOd40zbNiV5qISNSOHfC73/nIyrK5996GT1iv\nrXzBLoqe20rVVl0nUEQSp0krVpZlRYDr9rp7aT373deU5xcRqe3BB/1s2+binnuCdOzYuPNC7eou\nDPpUoIgkkhqEikhS+/prF08/7aVXrzDXXlu5/wfUULsFEXGAgpWIJK1QCH796wC2bTB5chCfr/GP\ntdUgVEQcoGAlIklr6lQvCxe6GT26iqOPbtynAHcL65I2IpJ4ClYikpQ2bjR48EE/bdrYTJjQuBPW\na3O3ceMt8OLy621ORBJH7zgikpTuvjvaYf3ee4Pk5R14I+MO9xxE768HktE9Iw7ViYjUT8FKRJLO\nf/7j5vXXvQwbFuLii9UuQURaDgUrEUkqu3bBbbcF8Hhs/vjHIK4mvkuVfVlK8etFhMsP8NwsEZFm\nULASkaQyaZKfdetc3HRTJf36RZr8PNue2Mzaq1cQKgrtf2cRkRhRsBKRpPHpp26mTvXRu3eYceMO\noGdVfdRuQUQcoGAlIkmhvBxuuSWAYdg8+mgFfn/zns9WuwURcYCClYgkhYce8rFihYtrrqli6NCm\nHwLcTZ3XRcQBClYi4rhvvnExZYqPLl0i3H77gfesqlf1gpXhUrASkcRRsBIRR1VUwE03BQiHDR55\npIKsrNg8rw4FiogTPE4XICLp7cEH/SxZ4mbMmEqOOy52rRE6P9SVSGkEd6YbymP2tCIi+6RgJSKO\nmTvXzeOPe+nRI8J998XoEGA1b+foFZt1KFBEEkmHAkXEEcXFcOONAVwu+Otfy2N2CLBGpCxCpDSM\nbR/45XBERJpKwUpEHHHXXQHWrXNxyy2VHHFEDD4FuJfVF33Pku7/wzC0YiUiiaNgJSIJ98YbHl58\n0cvgwWHGj29mI9CGhG29w4lIwultR0QSau1ag/HjAwQCNn/9awVeb3zGscPqYSUiiaeT10UkYaqq\n4JprMtixw+Dhhyvo1Sv2hwB3i9hqtSAiCacVKxFJmAce8PPll27OOaeKSy+tiutYdlifCBSRxFOw\nEpGEePddN1Om+OjRI8JDD1UQ93PKtWIlIg7QoUARibv16w1+9asM/H6bf/6znOzs+I+Zd2NHwjtj\n13BURKQxFKxEJK6CQfjFLzIoKjL44x8rOPTQOJ5XVUvrn7dNyDgiIrXpUKCIxI1tw+23+/nii+h5\nVWPGxPe8KhERp2nFSkTiZupUL88/72PgwDCPPJKA86pqWTt2BXZlhPy3D0vcoCKS9hSsRCQuPvnE\nzd13+8nLi/D00+VkZiZ2/PL/lWKHdDkbEUmsJgUr0zRdwBRgEBAExlqWtbzW9nHAhdU337Is6/7m\nFioiLcfq1QZjx0avAzh1agUHHeRAwImoQaiIJF5Tz7EaBQQsyzoKuB14uGaDaZo9gEuAo4GjgJNN\n0xzY3EJFpGXYuRMuvzyD7dtd/P73QYYPd+aTebYuaSMiDmjq284IYA6AZVmfAkNqbVsLnGpZVtiy\nrAjgBSqaVaWItAjBIFxxRQZLlri5+upKLrvMwZPVw7ZWrEQk4Zp6jlUrYGet22HTND2WZYUsy6oC\ntpqmaQCTga8ty1rW3EJFJLlFInDjjQHmzvVwxhlVTJoUdLQeO4JWrEQk4ZoarIqBnFq3XZZlhWpu\nmKYZAKYCJcD1jXnC3NxMPJ74t0nOz8/Z/04pKp3nDuk9/0TMffx4+Ne/YMQImDXLS0ZGnK6u3EiF\np7TFkxN9i9Nrn57See6Q3vN3cu5NDVZzgTOBmaZpDgcW1myoXql6FfivZVl/aOwTFhWVNbGUxsvP\nz6GwsCTu4ySjdJ47pPf8EzH3KVO8PPpogN69wzz5ZBm7dsGuXXEdcr/yHz1498967dNPOs8d0nv+\niZp7Q+GtqcFqNnCSaZrzAAO40jTN8cByolfn+gngN03ztOr977Asa34TxxKRJPb0017uuy9Ax44R\nZswoJzfX6YpERJzTpGBVfVL6dXvdvbTWz4EmVyQiLcZzz3m57bYAeXkRZs0qd6atQgMKH9mIO99D\n/vj0PRwiIomnUztFpEmmT/dw660B2rWL8PLL5ZhmYq4B2FiFj25kx/NbnS5DRNKMgpWIHLAZMzyM\nGxegbdsIL71UTt++yRWqoKaPldotiEhiKViJyAF58kkvN98coE0bmDWrnP79ky9UARBW53URSTxd\nK1BEGsW24Q9/8PHII37y86Mnqg8YkJyhyrZtsIl+lEZEJIEUrERkv0IhuO02P9Om+ejWLcLMmWV0\n65Y8J6rXUX0VHa1YiUiiKViJyD6VlsL11wd4+20vAwaEeeGFctq3T+JQBdgRG8NnYHgVrEQksRSs\nRKRBa9YYjBmTwaJFbo49NsTTT5eT0wK6F7h8LvqtO9zpMkQkDenkdRGp1yefuDn55EwWLXIzZkwl\nL7zQMkKViIiTtGIlInuw7egn/+65x4/LBQ89VMHll1c5XdYBsUM25V+W4m7ngTS+XpqIJJ6ClYjs\nVlQE48cHePNNL3l5EaZOrWD48LDTZR2wcEmYlWda5JzWhoPeyne6HBFJIwpWIgLA3Llurr8+wMaN\nLoYPD/G3v1VQUJDcJ6k3KBxHd/+XAAAgAElEQVSt21C7BRFJMJ1jJZLmKivhgQd8nHNOBlu2GNxx\nR5DZs8tbbqgC7JpFNrVbEJEE04qVSBr77DM3v/61H8ty06VLhMcfL2fIkORs+nlAIlqxEhFnaMVK\nJA3t3Am//rWfM8/MxLLcXHFFJe+/X5oaoYpaK1a6VqCIJJhWrETSSDgMs2Z5mDTJz5YtLvr0CfPQ\nQxUMG5YagWq33edYKViJSGIpWImkiQ8/dHP//X6++85NIGBzxx1BbrihEp/P6cpiz9PeS7fZvfHk\n6y1ORBJL7zoiKe6rr+C22zJ4773oP/cLLqjijjuCLfrk9P1xZbjIOkb9q0Qk8RSsRFLUggUuHn3U\nz3/+A+BhxIgQ990XZODAFDvsJyKSRBSsRFJIJALvv+9myhQfH38c/ed93HFwww1lHH98GCNNTjmq\nsMpZecpScsfkkz+lr9PliEgaUbASSQE7dsCMGV6eesrHypXRD/sef3yIceMqOeusTAoLW1739Gap\nsomURbCrtDonIomlYCXSQkUi0W7pM2d6ef11D2VlBn6/zUUXVXH11ZVpfcjPrpm6PhUoIgmmYCXS\ngtg2LFrk4l//8vDyy17Wr4+uTnXpEmHMmEouuaSStm0dLjIZ1LRbUB8rEUkwBSuRJBcOwxdfuHnr\nLQ9vveVh9epomMrJsbn00kouuCDEsGFhXGr3u5tdHaxQ53URSTAFK5EktH69wYcfuvngAw8ffeRm\n+/ZoasrKshk1qoozzghx8skhMjIcLjRZVZ9SpgahIpJoClYiDotEYNkyFwsWuPn8czcLFrhZseLH\n5afOnSNcckklp58e4thjwwQCDhbbQng6ecm7uSNZR6uXlYgkloKVSAJFIrB6tcHixW4WLXLx1Vdu\nvvjCTXHxjysrOTk2I0eGOP74EMcfH6ZXr0jatEmIFV8XPx3uKnC6DBFJQwpWInEQDsPatQYrVrhY\nudLFkiUuFi92s2SJi9LSPVNS9+4RTj01ep7U0KFhTDOi86VERFooBSuRJqiqgs2bDdavd7Fxo8GG\nDQYbNrhYtcrFihUGq1e7qKraM0B5PDa9ekXo2zdC//4R+vcPM2BAhPz81L20jFPKF5ZR+PBG2pzX\nlvwrdThQRBKnScHKNE0XMAUYBASBsZZlLa+1/RfAtUAImGRZ1hsxqFUkrsrLYft2g23b9vzavt1g\n69bo16ZNLjZsMNiyxcC26z8+16aNzYABEbp3j9CjR/Srd+/ol9+f4EmlqdCWKkre2kHG4VlOlyIi\naaapK1ajgIBlWUeZpjkceBg4G8A0zY7ATcAQIAB8Yprmu5ZlBWNRsEhVFVRUQHm5QXk5VFTs+b28\n3Ni9vaICSksNIhHYtMlPSYlBSQmUlBgUFxt73K6s3P+JTD6fTadONsOHh+nUyaZz5wgFBdH7OnWK\n0LVrRH2kkkFNHyu1WxCRBGtqsBoBzAGwLOtT0zSH1No2DJhbHaSCpmkuBwYCC5pVaTOtX23z9T+3\nU1oaxK4+8lLzvapTgEg7P7YN3h924SoP19knnOOlqksWtg2eLRW4t/yYE3fva0BF39YAuEpD+FaV\n1t3HhmDXLMKZ0T/6jCU7Ya+xbKCqnZ/KvOjHv3xrS/HsCu3eXrNPJOCmvGs2AJ6iIP7NFXvuU72i\nUtIzh5w2ULLVRfaqXXvtE/1e1jGDYKYP24bMFSUYlRFsO3quEEA4AhWZPnbmZkb32VqOb2cQ246G\nFtuGiA0hXGxq34pwGPzllbTZVv7j9urvoRCszcmhNOIhVGlz8LadhEPRsapCEAlDVZXBeiODTXaA\nqiqDrhUlBEIhQqHoODWK8bKc6J9BJ8rpRHm9r/9X+AEfWYQwKQYgJwAdMmwyM20y28KO9llkdvCQ\nl2fTs2QHrbNtWrWq/QVt+3kJ9Iy+LhWLywkVVtUuBhZCWZaLzCHRmqo2VhJcVlFvTZnDs3H5XUQq\nIpR9tqveffx9MvB28EZfowW7iJTV7abuae8l0DfadyG4vIKq9ZV7bHe3qaK0tGL3J+RC20JUfFdW\n73gZR2ThznZj2zalH5XUu4+vux9fl+jSW/k3pYR31L1cjruNm4xB0dWiyjVBKlfW/3tV1nE5GIZB\neFeY8i9L690ncGgmnnbRfy+l80uwK+seOvUW+PAfUv26LCmn/Nvq+andgogkWFODVStgZ63bYdM0\nPZZlherZVgK0buI4MXPlRX4eXf4Vvnq2PUYvXiUXgCksoS91/0P5hHbcw4Doc7GJy1ldZ58gLk7l\nOAAGsIM/sbTeWm5iMAuJ/if3NgsIUPc/y2fpylN0B+C3LGcE2+rss4QcbuIIAM5iO+P4vt7xfsYx\nlAKdcfE8S+rdZyL9eJ9WAEzjfxRQNwy8RUcepg8At7Cas9lQZ58t+BnNUQAcw04msaje8a5gKKsJ\n4CPM/XxX7z7Pt+nB++0OwuuN8Ks1y+lZVVxnnzUd2/DfU/qRkQGDvtlAn/lr6+xjG+D7+HhCoVIy\n15RQenn161JR/VUUvdn1wV5kHxf9M1hyiEWk+MfAEK7ezXVDBzpOOAiAwkc3UvxqUZ3xAv0z6Pl+\nPwB2/beYDePq/l0BML8biKu9i/C2EKvPr/+1K/hLN9pc0A6ADeNW1xvSWp/XloOmRP+ubH+6kO1P\nbKmzjzvPQ5/FgwAo/18pay5aXmcfgB7v9SVjQCZEaLCm9ncXkH9TRwA2T1xP6cd1/71kHZtDt5d7\nA1D8ahGbf7u+3ufqt/FwcEPlymCD43WZfgg5I6NvIWuvXkF4a6jOPm2vaU+nSQcDsPXPm9j50nYA\nXBn6FICIJFZTg1UxUPuMUFd1qKpvWw6wY39PmJubiccTv3X7R/4SZu3D3ercbxhwkpnDMdH3ZAIL\nOrO+uPq3a+PHb53zM3nksOjtnB/asHGFsXt7zT62y+BPJ0dv+4oCbP68W519AG4cHiDUNjp28Ttd\nKam1BFOzz4m9WnOUGd0ne0EHtm3J2WM7QF6un6nHRvfxrWrFjoX1z+9vp7kw/GCUeij+T/37/HJo\nFtd1BbcbMuYcREV5CMOIbnNVfz/1kGxOHwEuFxift4NlvujPBhjV37tle1h0eXSfyOosKt7ptsdz\nGEZ0jC/GesnsBG4MNj68559TjYdPakOrI6P/MW56pjMVa+seY+vbM4PLL4rG5R0ft2PHh3X/DhmG\nQddjALIIrvew8bd1/wwAOhzeloz86MpP+d1diQTrBt7WR7UmN7/6r/flnSkdUvd3Bl8HH/nV+wSO\nB99v6181ye/aCk+2h5AvRGUDNeUdm0d2fnT1q3LcwVRtraqzT/aAbPKqx3Of25FWB2fW2ced6d5d\nU9YQN66G/gz6tcGf78eO2HRrYJ/cE3NpXf1c4WsKqDi5btgLdAvsHs93aoQMn7fe58pvn4PhMgj2\n8xFpYLz2Q9uSmR+dU8WdXQmX1l0hazWsFW2rxzMu7UzuoFa4M910urpTdJz89D2BXXNPX+k8fyfn\nbtj2gX8iyTTNc4EzLcu6ovocqwmWZZ1Wva0j8C4wFPADnwGDLcuq/3hItcLCkrh/NCo/P4fCwvoP\nb6S6dJ47pPf803nukN7z19zTc+6Q3vNP1Nzz83Pq/a25qStWs4GTTNOcR3St4UrTNMcDyy3Les00\nzT8BHwMu4K79hSoRERGRVNCkYGVZVgS4bq+7l9ba/g/gH82oS0RERKTF0ZmdIiIiIjGiYCUiIiIS\nIwpWIiIiIjGiYCUiIiISIwpWIiIiIjHSpD5WIiIiIlKXVqxEREREYkTBSkRERCRGFKxEREREYkTB\nSkRERCRGFKxEREREYkTBSkRERCRGFKxEREREYkTBSkRERCRGFKxEREREYkTBSkRERCRGFKxERERE\nYkTBSkRERCRGFKxEREREYkTBSkRERCRGFKxEREREYkTBSkRERCRGFKxEREREYkTBSkRERCRGFKxE\nREREYkTBSkRERCRGFKxEREREYkTBSkRERCRGFKxEREREYkTBSkRERCRGFKxEREREYkTBSkRERCRG\nFKxEREREYkTBSkRERCRGFKxEREREYkTBSkRERCRGFKxEREREYkTBSkRERCRGFKxEREREYkTBSkRE\nRCRGFKxEREREYkTBSkRERCRGFKxEREREYkTBSkRERCRGPE4XUKOwsMSO9xi5uZkUFZXFe5iklM5z\nh/SefzrPHdJ7/pp7es4d0nv+iZp7fn6OUd/9abVi5fG4nS7BMek8d0jv+afz3CG956+5p690nr/T\nc0+rYCUiIiISTwpWIiIiIjGiYCUiIiISIwpWIiIiIjGy308FmqbpAqYAg4AgMNayrOW1to8DLqy+\n+ZZlWfebpmkA64Dvq++fb1nWHTGtXERERCTJNKbdwiggYFnWUaZpDgceBs4GME2zB3AJcCRgAx+b\npjkbKAO+sizrzPiULSIiIpJ8GhOsRgBzACzL+tQ0zSG1tq0FTrUsKwxgmqYXqACOAApM03wfKAfG\nWZZlxbRyEZG92LbN11u+5D+r/81nmz6lMhwk4PPRP3cQP+1yEkd3HoHblb4fQxeR+DNse999OU3T\n/CfwsmVZb1ffXgP0sCwrVGsfA5gM5FiWda1pmscBHSzLmmWa5gjgUcuyhu5rnFAobDvde0JEWq7/\nbfof498Zz/ur3t99n8twEbEju28P7DCQR05+hJ/2+KkTJYpIaqm3QWhjVqyKgZxat117haoAMBUo\nAa6vvvsLIARgWdYnpmkWmKZpWJbVYIpLUJdUCgtL4j5OMkrnuUN6zz/V527bNo99+RC//3wSNjY/\n7XISl/Qdw3EH/YRW/tZkt/Hw2rdzeHnZTF5a9iIjnxvJZf2u4MFjH8Ln9jldflyl+mu/L+k8d0jv\n+Sdq7vn5OfXe35hgNRc4E5hZfY7VwpoN1StVrwL/tSzrD7UeMwHYBvzRNM1BwJp9hSoRkaaoCFVw\ny/s38Mr3szgo+2AeOeHPHH/wiXvsk+HN4MQuIzmxy0iuHXQ9496/kecWP82KHT/w5KnP0jbQzqHq\nRSQVNSZYzQZOMk1zHtFlrytN0xwPLAfcwE8Av2map1Xvfwfwe2CaaZpnEF25uiLWhYtIegtFQlzz\n7pXMWfkmQzoM4+nTptM+s/0+HzMwfzCv/XwOv3rvWt5c8RqjXz+H2We/Qbav/t88RUQO1H6DlWVZ\nEeC6ve5eWuvnQAMPPaOpRYmI7Itt29z24TjmrHyTYw86nudPn0nA09Bb0Z6yvFk8ecqzjH//RqYv\nfY4r5lzK82fMxO/2x7lqEUkHahAqIi3On79+lGlLnmFg/mCeOfX5RoeqGi7DxUPH/z9O7XY6H617\nn9s+HBenSkUk3ShYiUiL8tnGT3nws9/SMasT0894qcmH8TwuD38/+SkG5R/GC0unMdN6IcaVikg6\nUrASkRZje8U2rnv3Kmxs/n7S1P2eU7U/GZ4Mnjj5KbK9Odz24XiWF32//weJiOyDgpWItBh3fnwb\n63et47ahd3JU52Ni8pzdW/fg0RP+TFmolBve+wXhSDgmzysi6UnBSkRahP+ueZdXvp/FYe0P5+bD\nb43pc599yDmc0+s8vt7yFU9994+YPreIpBcFKxFJeqVVpdz24XjchpuHj/9zXC5LM/GY39PG34YH\nPpvI+pJ1MX9+EUkPClYikvQe+/Ih1pSs5obBN3No3oC4jNE+sz33H/07Sqt2cffc2+MyhoikPgUr\nEUlqa0vW8Pg3f6FzVgHjh9wW17Eu7HMJQzseyZsrXuPTDfPiOpaIpCYFKxFJar/7dCLBcJA7h99L\npjczrmMZhsH9Rz8AwIR5d+5xAWcRkcZQsBKRpPX15i95+fuZDMwfzHm9RydkzCEdhzHqkHP4estX\n/Gv5ywkZU0RSh4KViCStBz6bCMB9R0/CZSTu7equ4ffhc/l48LPfUhWuSti4ItLyKViJSFL6dON8\nPlr3PscddAIjCo5L6NhdW3Xjkn6Xs7p4FS8tezGhY4tIy6ZgJSJJafLnvwPgtqF3OjL+zYffis/l\n4+Ev/6hVKxFpNAUrEUk68zfM5eP1H3L8wScyrNORjtTQObuAS/uNYU3xKmYtm+FIDSLS8ihYiUjS\neeSLPwLwf0PvcLSOmlWrR7+crEvdiEijKFiJSFJZWPgNH657nxEFxzG0ozOrVTU6ZXdmdJ9LWF28\nijdXvOZoLSLSMihYiUhS+cvXjwHwq8NudriSqOsH/woDgz9//Ri2bTtdjogkOQUrEUkaq4tX8eoP\ns+nX7lBOOHik0+UA0LNNL87ocRbfFH7NJ+s/crocEUlyClYikjQe/+YvROwIvzrsZgzDcLqc3WpW\nz2pW00REGqJgJSJJYWdwBy8seZ7OWQWc3fMcp8vZw+EdhjC809G8v/Y9lhd973Q5IpLEFKxEJClM\nXzKNslApVw34BV631+ly6rh6wDUAPPXdPxyuRESSmYKViDguHAnz5MK/k+HJ4NJ+Y5wup16ndz+T\nDpkdmWFNZ1fVLqfLEZEkpWAlIo779+o5rClZzXm9R9M20M7pcurldXsZ0/8qSiqLecnSZW5EpH4K\nViLiuH8u/DsAYwdc53Al+3ZZ/yvxurxM/e4JtV4QkXopWImIo1bs/IGP133AUZ2PoW+7fk6Xs08d\nMjtwZs+zWbp9CfM2fOJ0OSKShBSsRMRRzy16GoDL+13pbCGNdNWh1wLw5MInHK5ERJKRgpWIOCYY\nDjJj6TTaBtrys55nO11OowztOIwBeYN4e+UbrC9Z53Q5IpJkFKxExDFvrXidbRXbGG1egt/td7qc\nRjEMg6sHXEPYDvPs4qlOlyMiScazvx1M03QBU4BBQBAYa1nW8lrbxwEXVt98y7Ks+03TzACmAe2B\nEmCMZVmFsS5eRFq2Zxc9BcDl/a9wtpAD9PNe53HfvLt4bvEz/HrIHUnZd0tEnNGYFatRQMCyrKOA\n24GHazaYptkDuAQ4GjgKONk0zYHAL4GFlmUdCzwL3B3rwkWkZVte9D1zN3zMiILj6Nmml9PlHJAM\nTwYXmBextbyQf6+e43Q5IpJEGhOsRgBzACzL+hQYUmvbWuBUy7LClmVFAC9QUfsxwNtAclxNVUSS\nxrOLq1erWshJ63u7uO/lAExf8qzDlYhIMtnvoUCgFbCz1u2waZoey7JClmVVAVtN0zSAycDXlmUt\nM02z9mNKgNb7GyQ3NxOPx32A5R+4/PycuI+RrNJ57pDe80+2uVeEKpi5bDr5mfmMOfJifG5fXMeL\nx/x/kj+coZ2H8t6ad6n0F1PQqiDmY8RCsr32iZTOc4f0nr+Tc29MsCoGalfosiwrVHPDNM0AMJVo\ngLq+nsfkADv2N0hRUVlj6m2W/PwcCgtL4j5OMkrnuUN6zz8Z5/7SshfZXr6dXx12Czu3B4mevhkf\n8Zz/6F6XsmDDAv469++MG/J/cRmjOZLxtU+UdJ47pPf8EzX3hsJbYw4FzgVOBzBNcziwsGZD9UrV\nq8A3lmVda1lWeO/HAKcBHzetbBFJRc8tfhogaa8L2Fg/73UumZ5Mpi99jogdcbocEUkCjVmxmg2c\nZJrmPMAArjRNczywHHADPwH8pmmeVr3/HcDfgGdM0/wEqAQujnnlItIirdy5gvkb5nJM52Pp0bqn\n0+U0S46vFWf2HMWL1nTmbfiEEQXHOV2SiDhsv8Gq+qT0vS/gtbTWz4EGHnp+U4sSkdQ103oBgAv7\nXOJwJbFxSb8xvGhNZ9riZxSsREQNQkUkcSJ2hJnWC2R5s1tMp/X9ObLjcA5p04s3V7zGjooip8sR\nEYcpWIlIwszb8AlrS9ZwVs9RZHmznC4nJgzD4KK+lxEMB3n5+1lOlyMiDlOwEpGEmbH0eSB1DgPW\nuMC8CI/Lw/PqaSWS9hSsRCQhdlWW8MYPr9K1VTeO7HSU0+XEVIfMDozsegrfbf2W77Yu3P8DRCRl\nKViJSEK8/sOrlIXKGG1ejMtIvbee0Wb0w881J+eLSHpKvXc3EUlKM6zoYcALzIscriQ+RnY9mVx/\nLi8vm0koEtr/A0QkJSlYiUjc1fSuGlFwHF1adXW6nLjwu/38vNd5FJZv4YO17zldjog4RMFKROKu\n5vBYzeGyVFWzGqfDgSLpS8FKROIqFXtXNeSw9kfQq01v3l75JjuD+71EqoikIAUrEYmrVOxd1RDD\nMLjAvIhgOMhrP/zL6XJExAEKViISV6nau6oh55sXYmDw4tLpTpciIg5QsBKRuEnl3lUN6ZxdwLEH\nHc/nmz5l5c4VTpcjIgmmYCUicZPqvasacoF5IaCT2EXSUfq804lIwqV676qGnNHjLDI9WcyyZhCx\nI06XIyIJpGAlInGxungV8zfM5ZjOx6Zs76qGZHmzOLPn2awpWc1nG+c7XY6IJJCClYjExSxrBgCj\n+6R276qG1MxbJ7GLpBcFKxGJOdu2mWm9QKYnk5/1OMvpchxxdOcRHJR9MK/98C/KqsqcLkdEEkTB\nSkRi7vNNn7GqeCWn9ziTbF+O0+U4wmW4ON8cza6qEt5e+YbT5YhIgihYiUjMzbSih79S/RI2+3N+\nb13iRiTdKFiJSEyVh8p5dflsOmV1ZkTBcU6X46hDcntxRIehfLjufTaVbnS6HBFJAAUrEYmpd1a+\nRXHlTs7vfSFul9vpchx3gXkRETvCS8tmOl2KiCSAgpWIxFTNYa90613VkFGHnIPP5WOmNR3btp0u\nR0TiTMFKRGJmc9lm3l/7Hoe1P5zebU2ny0kKuYG2nNztNJZuX8LCrd84XY6IxJmClYjEzMvLZhK2\nw1qt2ot6WomkDwUrEYmZmdYLeF1eRh1yntOlJJUTDx5JXkYer3w/i8pwpdPliEgcKViJSEws3Pot\ni7d9x8iup9Auo53T5SQVr9vLOb3OZ1vFNt5b867T5YhIHClYiUhM1Jy0nu69qxoyus8lAMxY+rzD\nlYhIPClYiUizVYWreHnZTNoG2jKy68lOl5OUBuQNpH+7Aby7eg5by7c6XY6IxIlnfzuYpukCpgCD\ngCAw1rKs5Xvtkw/MAwZYllVhmqYBrAO+r95lvmVZd8S0chFJGh+sfY+t5YVcPeAafG6f0+UkrQv7\nXMw9c+9g9vez+MXAXzpdjojEQWNWrEYBAcuyjgJuBx6uvdE0zVOAfwMdat3dE/jKsqzjq78UqkRS\n2Is1vat669OA+3JOrwvwuDzM0KcDRVJWY4LVCGAOgGVZnwJD9toeAUYC22vddwRQYJrm+6ZpvmWa\nphraiKSoHRVFvLPqLXrnmgxuf7jT5SS1/Mx8RnY9hYVbv+G7rQudLkdE4mC/hwKBVsDOWrfDpml6\nLMsKAViW9S7AXtlpI/CgZVmzTNMcAUwDhu5rkNzcTDye+F/+Ij8/J+5jJKt0njuk9/zjOfdXvphO\nMBzkysOvoH37VnEbpzmS6bW/dthY5qx8k9fXvMQJfY+O+3jJNPdES+e5Q3rP38m5NyZYFQO1K3TV\nhKp9+AKoCV6fmKZZYJqmYVlWg9dzKCoqa0QpzZOfn0NhYUncx0lG6Tx3SO/5x3vuT375FAYGpxWM\nSso/42R77Ye2OZZ2gXY8981z/HrQ3Xjd3riNlWxzT6R0njuk9/wTNfeGwltjDgXOBU4HME1zONCY\n9esJwC3VjxkErNlXqBKRlmnFjuUs2PQZxx10PJ2zC5wup0XwuX2c2/sCtpZvVU8rkRTUmGA1G6gw\nTXMe8CgwzjTN8aZpnrWPx/we+Ilpmh8CjwBXNLtSEUk6M5fNAHTB5QOlnlYiqWu/hwIty4oA1+11\n99J69utW6+ci4IzmFiciyStiR3jJepEsbzan9zjT6XJalJqeVv9e/TZby7eSl5HndEkiEiNqECoi\nTTJ3/cesKVnNmT3PJsub5XQ5Lc6FfS4mFAkx+/tZTpciIjGkYCUiTfL8kmcBuLjv5Q5X0jKpp5VI\nalKwEpEDtqOiiDdXvEbPNodwZMfhTpfTIqmnlUhqUrASkQP2yvKXCIaDXNTnMgzDcLqcFqvmgtUv\nWlq1EkkVClYicsBeWDINt+FmtD4N2CwndT2FtoG2vLzsRarCVU6XIyIxoGAlIgfku60L+abwa0Z2\nPZkOWR2dLqdF87l9nNsr2tPq3dXvOF2OiMSAgpWIHJDpOmk9pi7pNwaAaYufdrYQEYkJBSsRabSK\nUAUvLXuR/Iz2jOxystPlpIR+7fpzRIch/Hftf1hfss7pckSkmRSsRKTR5qx8kx3BHVxgXhTXa9yl\nm0v6jiFiR5i+9DmnSxGRZlKwEpFGq/mP/+K+lzlcSWoZ1etcsrzZTF/yHOFI2OlyRKQZFKxEpFHW\nlqzhw7XvM6TDMHrl9na6nJSS7c3mnF7nsX7XOj5Y+57T5YhIMyhYiUijTFv8NDY2l/e/0ulSUtKl\nfaMnsT+3+BmHKxGR5lCwEpH9qgpXMW3xs7T2t+Gsnj93upyUNLj94bsvzLy5bLPT5YhIEylYich+\nzVn1JoXlWxhtXkSmN9PpclKSYRhc2m8MoUiIF3X9QJEWS8FKRPbr6UVTAbi831UOV5Lazut9ARme\nDJ5f8gy2bTtdjog0gYKViOzTih3L+XjdBxzdeQS925pOl5PSWvvbcGbPUazcuYKP13/odDki0gQK\nViKyT89WdwQf01+rVYlQsyr41Hf/dLgSEWkKBSsRaVBFqIIZS6eRl5HH6T3OdLqctDC04zAG5A1i\nzso31YldpAVSsBKRBr2x4lW2V2znwj6X4nf7nS4nLRiGwVWH/oKwHebZxVOdLkdEDpCClYg06Jnq\nk9Yv63eFs4WkmZ/3Oo82/jY8t/gZguGg0+WIyAFQsBKRei3etojPNs7n+INPpHvrHk6Xk1YyvZlc\n2OdStpYX8sYPrzpdjogcAAUrEanXP799nP/f3n3HR1Vn/x9/TUslDRIIRVqEQxGiCFgAQRSDiKLs\n6gp2RRFdFVlXRVdd/dp/VlxBQcXFLk1ZFRUVlSogIiD4gYBSpIUSSC8z8/tjJnGICQmYyU1mzpMH\nj1s+907en9zcmTP33ghmwt4AAB+oSURBVLkDcO0JN1icJDxdfcJ12LDx2topVkdRSh0FLayUUn+w\nr2AfMza8R9v4dgxqk2F1nLDUPiGNga3PZvmu71iT9aPVcZRSNaSFlVLqD95YN5VCdyGjuo3GYXdY\nHSdsXXvC9QB61EqpBkQLK6XUYUrcJUxd+wqNXHGM6Hy51XHC2sDWg2gT35ZZG6dzoHC/1XGUUjWg\nhZVS6jAfbf6QnXk7GNHpMuIi4q2OE9YcdgdXdx1FQWkBb66fZnUcpVQNaGGllDrM5NUTsWHjuu6j\nrY6igMu7XEmsqxGvrH6JYnex1XGUUtXQwkopVe773cv5fvcKzmk7mPYJaVbHUfi+P/DyzleyM28H\nH2TOtDqOUqoazuoWEBE7MBFIB4qAUcaYzArLpACLgW7GmEIRiQbeBJoCOcBVxpis2g6vlKpdU1ZP\nAuD67mMsTqICXd99DFPWvMSkVf/h4o6XYrPZrI6klKpCTY5YXQhEGWNOA+4Gng5sFJEM4HOgWcDs\nMcAaY0w/YBrwr9qJq5QKlp25O5iz6QM6N+5Cv5b9rY6jArSOb8MFaRfy0741fLv9a6vjKKWOoCaF\nVV/gUwBjzFKgZ4V2D3A2sL+ydYC5/nalVD320o8vUuop5YbuN+kRkXrophNvBWDiqgkWJ1FKHUm1\npwKBeOBgwLRbRJzGmFIAY8w8ABGpap0cIKG6H5KUFIPTGfz75aSkxAX9Z9RX4dx3CO/+V9f3/QX7\nmbbuNVrEtWBMn1FEOkPrC5dDYdsPSunPGcvPYP6WL9nt3cIJTU+o0Xqh0PdjFc59h/Duv5V9r0lh\ndQgITGgvK6pquE4ckF3dDzlwIL8GUf6clJQ4srJygv5z6qNw7juEd/9r0venVzxDXkke/+x1D4cO\nFAOh8+mzUNr213e9mW+3fMsj8x9nwsBJ1S4fSn0/WuHcdwjv/tdV36sq3mpyKnARMARARE4F1hzN\nOsC5wIIarKOUskBeSR5TVk8iMTKRK7tcbXUcdQSD2mRwfGIHZm54n115O62Oo5SqRE0Kq9lAoYgs\nBp4FbheRcSJywRHWmQR0FZGFwA3Ag38+qlIqGN5a91/2F+7num6jaRQRvqcOGgK7zc6N6X+nxFPC\nSz++aHUcpVQlbF6v1+oMAGRl5QQ9iB4aDc++Q3j3/0h9L3YX0/vNdLKLDvD9FT/RJLpJHacLvlDb\n9kXuInq/mc7BomxWXLGW5OjkKpcNtb4fjXDuO4R3/+vwVGCln/LRG4QqFcZmbZzOjrzfuLzLVSFZ\nVIWiSEckt5w0lvzSfF5a9R+r4yilKtDCSqkw5fF6eGHlszjtTsak32J1HHUULutyFU1jmvHq2sns\nL9xndRylVAAtrJQKUx9vnsPG7A38tePfaBnXyuo46ihEO6O55aSx5JXk8rJea6VUvaKFlVJhyO1x\n88SyR3DYHNzWY5zVcdQxuKLLNSRHpzBl9ctkFx6wOo5Syk8LK6XC0KyN09lwwPA3GUlaYger46hj\nEOOK4eYTbyO3JIfJq6u/p5VSqm5oYaVUmClxl/Dk8kdx2V2M63mn1XHUn3D1CdfRJKoJk1dP4mBR\ntfdhVkrVAS2slAoz75q32HLoV67ocjWt49tYHUf9CbGuWMaceCuHig/y8o8TrY6jlEILK6XCSmFp\nIU8vf4IoRxRjT77D6jiqFlzb7XpSopsycdUL7MnfY3UcpcKeFlZKhZE31k1lR95vXNvtBlJjm1sd\nR9WCRq5G3NHrbvJL83h6xeNWx1Eq7GlhpVSYyCvJ47nvnybW1YhbTrrd6jiqFl3e+SrSEo9n2k9T\n2ZS90eo4SoU1LayUChOTf5xIVsEeRqffpHdZDzEuh4t7TnkAt9fNI0sfsjqOUmFNCyulwsCuvJ08\nv/IZkqNTuEnvsh6Shra/gJOb9eKjzR+yYtcyq+MoFba0sFIqDDy89N/kl+Zxzyn3Ex+ZYHUcFQQ2\nm40HTvs/AB5acj9eb9C/114pVQktrJQKcct+W8b75h26NunGiE6XWx1HBdGpLU5ncNshLN25mM9+\nnWt1HKXCkhZWSoUwr9fL2E/HAvBI3ydw2B0WJ1LB9q9TH8Rhc3D/ovEUlhZaHUepsKOFlVIhbHbm\nDJZsX8LQ9sM4vWVfq+OoOtCxsTCq+438eugXnlz0pNVxlAo7WlgpFaLyS/J5aPH9RDoieeD0/7M6\njqpDd/YaT7OYVB5d8Ci/HNxsdRylwooWVkqFqAkrn2ZH3m+MO20cbeLbWh1H1aG4iHge6vMoRe4i\n7l1wp17IrlQd0sJKqRC0bt9PTPjhWVo2asX4vuOtjqMscOHxf2Fgu4F8sfVz5v7ysdVxlAobWlgp\nFWLcHjf/+PoWSj2lPNX/OeIi46yOpCxgs9l4cciLuOwu/rXwLvJK8qyOpFRY0MJKqRDz2trJfL97\nBcM7XMxZbc6xOo6yUKfkToxJv4Xtudt4ZoVeyK5UXdDCSqkQsi1nK48sfYjGUY15uO8TVsdR9cDt\nPf9J67g2vLjqeb7fvdzqOEqFPC2slAoRXq+XO7+5nfzSPB7q8xjJ0clWR1L1QKwrlgkDJ+Hxevj7\nl6PJL8m3OpJSIU0LK6VCxIwN7/Hl1nkMOG4gF3e81Oo4qh45vWVfbug+hk3ZmTz2nX5Js1LBpIWV\nUiFgy6FfuXvBHcS6GvH/+j+HzWazOpKqZ+455QHSEo9n8upJLP5todVxlApZWlgp1cCVekoZM28U\nOcWHeLzfU3rPKlWpGFcMLwx8CZvNxq3zbyK3OMfqSEqFJC2slGrgnlrxOCt2L2N4h79yiYywOo6q\nx3qm9uaWk25n66FfuW+R3t9MqWDQwkqpBmzJjkU89/1TtI5rw5NnPKunAFW17uh1Nyckd+et9dN4\n7+e3rY6jVMhxVreAiNiBiUA6UASMMsZkBrRfD4wGSoGHjTEfiUhjYAOw1r/YbGPM87UdXqlwdqBw\nP2PmjcKGjUmDXiE+MsHqSKoBiHRE8krGfxk0vT93fns73VLS6dKkq9WxlAoZNTlidSEQZYw5Dbgb\neLqsQURSgVuBPkAG8JiIRAI9gHeMMQP8/7WoUqoWebwebvnyRnbk/cYdve6mV+opVkdSDUj7hDQm\nDJxEQWkB1312hV5vpVQtqklh1Rf4FMAYsxToGdDWG1hkjCkyxhwEMoHuwMlADxH5RkSmi0jzWs6t\nVFh7/LuH+XzLp5zR6kzG9rjD6jiqATqv/fncdOKtbMrOZOz8v+sXNStVS6o9FQjEAwcDpt0i4jTG\nlFbSlgMkAD8D3xtjvhCRy4AXgL8e6YckJcXgdDqOKvyxSEkJ3+9NC+e+Q+j0/7217/HcyqdIS0pj\n9sgZNI5OrHadUOn7sQrn/h+p788NfYrV+1cyZ9Nsztx8BmNPHVuHyYIvnLc7hHf/rex7TQqrQ0Bg\nQru/qKqsLQ7IBr4Dym7vOxuo9o50Bw4E/27AKSlxZGWF5yHvcO47hE7/V2et4poPr6GRK46pGW/j\nznWRlXvkfoVK349VOPe/Jn2feOarnPV+P/7x+T9o6mjJoLaD6yhdcIXzdofw7n9d9b2q4q0mpwIX\nAUMARORUYE1A2zKgn4hEiUgC0BnfBeuvAH/xL3MW8P2xxVZKldmTv4er5o6ksLSQSYNeoVPjzlZH\nUiEgNbY504a8Q6Qjkus/v4bVWausjqRUg1aTwmo2UCgii4FngdtFZJyIXGCM2QVMABYAXwH3GmMK\n8V3kPkZEvgZuBG4LSnqlwkRuSS5Xzb2U33K3M/6U+8hoe67VkVQIOblZLyae/QoFpfmM/Phituds\nszqSUg2Wrb5csJiVlRP0IHpoNDz7Dg27/0XuIi77+BK+3T6fS2RE+d2za6oh9702hHP/j7bvL//4\nIvctGk+nxp3530WfkRBZ/fV79VU4b3cI7/7X4anASp+I9QahStVjbo+bMfNG8e32+QxuO4RnB/xH\nbwKqgmZ0+s1c3+1Gft6/nqvnXkZ+SfCvfVUq1GhhpVQ95fV6ueOb2/ho84f0adGPyee8jsvhsjqW\nCnEP9XmM89pfwKIdC7hy7ggKSgusjqRUg6KFlVL1kNfr5f7F9/DW+mmkp5zEtCHvEOWMsjqWCgMO\nu4OXB73G4LZD+Hb7fK785FItrpQ6ClpYKVXPuD1u7vhmLC//+CIdk4R3hs4kLiLe6lgqjEQ4Ingl\nYxoZbc/lm+3zuWruCApLC62OpVSDoIWVUvVIibuEm74YxRvrptItOZ0PLpxLcnSy1bFUGCorrs5p\nM5ivt33FlXMvJbck1+pYStV7WlgpVU8UlBZw9acjmZ05k96ppzJr2P+0qFKWinRE8urgN8hoey5f\nb/uKCz8Ywu783VbHUqpe08JKqXpgb8Fe/va/i5i35TPOPO4s3j//gwb9UXcVOiIdkUwd/BaXd76K\n1VmrGDLzLDbsN1bHUqre0sJKKYut3buGjBkDWLpzMcPShjNtyLvEuGKsjqVUOafdydMDJnB373+x\nLWcrQ2cPYsmORVbHUqpe0sJKKQv9b9OHDJ01iG05W7mr971MPmcqkY5Iq2Mp9Qc2m41xPe/khYEv\nkVuSy1/nXMCra16mvtxkWqn6QgsrpSxQ6inl8WUPc91nVwA2pg5+i3/0vEtv/qnqvb91Gsl7Q2cT\nHxHP+AX/ZPS8a8gtDs87fCtVGS2slKpjWw9t4cIPhvDMiidpHdeGj4fP47z251sdS6ka69eqP19e\nspBeqafwQeYszpkxgPX71lkdS6l6QQsrperQzA3vc+b7fVi2aykXpF3EFxd/S9fkE6yOpdRRa9Go\nJR8M+4Qx6beQmb2RjBkDeOnH/+D2uK2OppSltLBSqg5k5Wdx47zrGPPFKDxeDxMGTmLKOa+TGJVk\ndTSljpnL4eLBPo/w+uC3iXXFcv+iezh/dgaZBzZaHU0py2hhpVQQebwepv00lT7vnMysjdM5uVlP\nvrxkAZd2ukyvp1IhY0j7oSwYsZxhacNZsXsZA9/vw39+eJ4Sd4nV0ZSqc1pYKRUka/euYeisc7jj\nm9so9bh5tO+TfHTRPNonpFkdTalalxydzJSM13kt400aRcTx0JL76P/eqXyx5TOroylVp7SwUqqW\nbcvZyq1fjeHs6f1YsXsZw9KGs3jkCkZ1vxGH3WF1PKWCamjaBSwcsYyru17H5oObGPnxxVz60XC9\nqagKG1pYKVVL9hXs476Fd3PaWz149+e3kKROvDt0FlMyXic1trnV8ZSqM42jmvBk/2f56pJF9Gs1\ngK+2fkH/907lli9vZHN2ptXxlAoqp9UBlGrotuds4+XVE3lz3X/JK8mldVwb7up9L8M7XKxHqFRY\n69KkKzPO/5BPf/2ER5c+yHvmbaZveJfhHS5m3Ml3cnxSB6sjKlXrtLBS6hit3buGiasm8EHmTEo9\npTSPbcE9p9zHlV2v1bunK+Vns9k4t915ZLQ9l483z+Gp5U8wY8N7zNzwPue0Hcx13UbTv9WZ+mEO\nFTK0sFLqKOSV5DEnczZvrHudFbuXAdCpcWduOvFWhne4mAhHhMUJlaqf7DY756ddyHntL2DuLx/z\nwspn+OzXuXz261w6JHbk2m438NeOl+iXj6sGTwsrparh8XpYtnMpMzdOZ9bG6eQUH8KGjYGtz2ZU\nt9Gc1focfbetVA3ZbXbOa38+57U/n5W7V/DKmpf5MHMW4xfcwb8X30tG2yFcIpdy5nFn43K4rI6r\n1FHTwkqpSni8Hr7fvZwPM2cxZ9MH7MrbCUCL2Jbc0H0MIztfwXFxrS1OqVTD1qNZTyY268m/T3+E\nd39+k/fNO8zZNJs5m2aTHJ3Mue3OZ0i78+jbqr+eXlcNhhZWSvkdLMrm621fMW/LZ3y1dR57C/YC\nkBCZyMhOVzDs+OH0a9Ufp113G6VqU9OYptzaYxy3nHQ7q7NW8b55h9mZM3lj3VTeWDeVRq44zmo9\niLPaDGLAcQP1U7aqXtNXCBW2cotz+G7nEhbtWMiSHQtZtecH3F7f95ylRDdlRKfLOT9tGGe0OlOv\nnVKqDthsNtKbnkR605N4qM9jLN/1HZ/88hGf/PIRH26axYebZgEgSZ3of9yZnNaiLz1Te9MsppnF\nyZX6nc3r9VqdAYCsrJygB0lJiSMrKyfYP6ZeCue+AyQ0juTbn5fyw56V/Jj1Az/sWYnZv768kHLa\nnZyY0oOz2gzi7Nbn0C0lHbstNG7zFu7bPpz7Hyp993q9rN+/jm+2zeeb7V+xZMciCkoLytuPi2tN\nz2a96Jnam57NetMl+QRapSaHRN+PVahs+2NRV31PSYmr9OJaPWKlQorX6+W33O1sOPAzZr9h4wHD\nun1rWbtvDcXu4vLlop3RnNysF6e36MvpLfvSK/UUYl2xFiZXSlXFZrPRpUlXujTpypgT/06Ru4gV\nu5axfNd3rNi1jBW7lzE7cyazM2cCvjdKHZt0pGNCJzo37krnJl3p3KQLx8W1Dpk3TKr+0sJKNTjF\n7mJ+y93O1kNb2JqzhW2HtrI151c2Z29iw4EN5JfmHba8y+4iPTWdrknpnNS0B+kpJyGNO+m1Uko1\nUJGOSPq07Eeflv0A3xuqXw5uYvmuZazYvZyf9q7BHFjPuqx1fMCs8vWindG0jW9Hm4R2tItvT9uE\ndrRLaE/ruNakxrYgxhVjVZdUCNFXFlUveL1ecktyyC7KJit/D7vzd7M7bxe783exp3x8d/m0x+v5\nw2NEOiJJS+yAJAkdkgRp3ImOSZ1ol9CelqlNwvawuFKhzmaz0T7xeNonHs/fOo0EIDm5ESs3/8T6\n/etYv+8n1u/7iU0HN/HLwc2s37+u0sdJiEykeWxzUmOb0zy2hX+8BamxzWkc1YTGUY1pHN2YxMgk\nPfKlqlRtYSUidmAikA4UAaOMMZkB7dcDo4FS4GFjzEcikgy8DUQDO4BrjDH5QcivLOb1eskvzSe/\nJJ/80jzySvLIL/EPS/PLx/NK8jhYnM3BomwOFB7gYFE22UUHyC7KJrvQNyy73qkqUY4omsam0iv1\nFFrHtaF1fBvaxLctH28e20K/QkYpBfiKrdbxvueGjLbnls/3er3sK9zHrwc384v///bcbezM3cGu\nvJ3syN3Bz/vXH/Gx7TY7iZGJ/mKrCUlRScRFxBMXEUecyzdsFBFHI1ej3+dHxNHIFUe0M5ooZzRR\nziiiHFH6nBWCanLE6kIgyhhzmoicCjwNDAMQkVTgVqAnEAUsFJF5wP3A28aY10XkbnyF17PB6EBN\nuT1usguzyS48hLfsnxf/sHwOlI37L+qv2O71eqtcB6j0MStdx79wVe1l4x6vF4/XjdvrptRT6hv3\nuHF7Pbi97oBpt3/ac9i021OK2+smOsbFwdx8PJ7fH6vYXUyxp5hidxElnhKK3EWUuIspchdT4vl9\nWOwuothd4ht6iil2F1PkLiK/JJ+C0vzf+3OUXHYXCf4np3YJaSRFJZEQmUhydArNYlJpFtvMN/SP\nx0ck6I04lVJ/is1mIzk6meToZHqm9q50mbySPHbl7WBn3k525u5gd/5uDhTuZ3/hPvb7h2XTmw9u\nqvQIek257C5foeWI8hddUUQ5o4l0RPqG9gicDhcuuwuX3YnTXjbu+1827XQ4A+ZHkBTfiMJ8N067\nE4fNgc1mw2FzYLfZy4e2gHGHzY7dZsdeNm13YMeOvWwYsH7ZPJvNhg1b+e+1/J//edo3TvkyVLlM\nwONUnC4bP+IyHPa4ie6oY94etaEmhVVf4FMAY8xSEekZ0NYbWGSMKQKKRCQT6O5f51H/MnP945YW\nVhd9eB5Ldy62MkKDFWGPIMIRSaQjApfDN57ojKZFo5bEumKJccYQ44o9bDzGGUOsqxExrpjyeYmR\niSREJpIUmURCVCKxzlgtlJRS9U6sK5a0xA6kJVb/JdEer4ec4kPkFOeQU5xDbol/6J/OKTlEbnFu\neVt+ST5F7iIKSwsodBdSUFpAYWkhhW7fcF/BXgr803+mYAtnreJb8d2IHy27c39NCqt44GDAtFtE\nnMaY0kracoCECvPL5h1RUlIMTmfwDole2v0SmsWnAAGVdQ2GwJGXOdrlj3LZsncXDruj/J2Hw+6o\n8bCqdZx2J5HOSCIcEUQ6Ig8bj3BElE+77K6QKX5SUuKsjmCZcO47hHf/te/B16z6l7ij5vV6KfGU\nUOwupsRdQomn5LBhsbv4D/Mqa/Od6fAcdkbD4/VUOq+q6aqW+f1MzeFnWirOg0rODlWzzDE9rn9e\np+RONG+WZNlrV00Kq0NA4F+n3V9UVdYWB2QHzC8ImHdEBw4E9xKskWnXctupt4XtBczV3tfD4/9f\n4hsU4KWAInyX1TV8ek+X8Ow7hHf/te+h0ncnNpxEEE35rYptgMP/vxKh1f+jU4f3sap0fk0+1rAI\nGALgv8ZqTUDbMqCfiESJSALQGVgbuA5wLrDg2GIrpZRSSjUcNTliNRsYJCKL8dXI14jIOCDTGDNH\nRCbgK5zswL3GmEIReRj4r/8Tg3uBkUHKr5RSSilVb1RbWBljPMCNFWb/HNA+BZhSYZ3dwODaCKiU\nUkop1VDoHc6UUkoppWqJFlZKKaWUUrVECyullFJKqVqihZVSSimlVC2xld1QSymllFJK/Tl6xEop\npZRSqpZoYaWUUkopVUu0sFJKKaWUqiVaWCmllFJK1RItrJRSSimlaokWVkoppZRStaQmX8LcIInI\nRcDFxpiR/ulTgeeBUuBzY8yDFZZPBt4GooEdwDXGmPy6TV17RORufv++xkQg1RiTWmGZOUAToAQo\nMMacW7cpg0NEbMB2YKN/1hJjzPgKyzwAnIfv72GsMWZZ3aYMHhFJAN4E4oEIYJwxZkmFZSYAfYAc\n/6xhxpiDdRq0FomIHZgIpANFwChjTGZA+/XAaHzb+2FjzEeWBA0CEXEBrwFtgUh8/ZsT0D4OuA7I\n8s8abYwxdZ0zmETkB6Ds7/cXY8w1AW2hvO2vBq72T0YBJ+J7rs/2t4fUfh5IRE4BnjDGDBCR44HX\nAS+wFrjZ/z3HZctG43tObIrvd3GVMSbrj49aO0KysBKR54EMYFXA7JeAvwCbgY9FpIcxZmVA+/3A\n28aY1/1FyWjg2brKXNuMMY8DjwOIyEfAXZUsdjzQ1RgTajczSwNWGmPOr6xRRHoA/YFTgOOAmUCv\nuosXdOOAL40xz4mIAO8APSos0wPIMMbsrfN0wXEhEGWMOc3/JuppYBiAiKQCtwI98b34LBSRecaY\nIsvS1q7LgX3GmCtEpAnwAzAnoL0HcKUx5ntL0gWZiEQBGGMGVNIW0tveGPM6voICEXkReK2sqPIL\ntf0cABG5E7gCyPPPegb4lzHmaxF5Cd++PztglTHAGmPMv0XkUuBfwG3ByheqpwIX4/tFAiAi8UCk\nMWaTv4j4DDirwjp9gU/943OBs+siaLCJyHDggDHmswrzm+E7kvU/EVkoIkMtCRgcJwMtRWS+iHzi\nLy4C9cV31NJrjNkKOEUkpe5jBs2zwMv+cSdQGNjoP7rTAZgsIotE5No6zhcM5fuvMWYpvhfSMr2B\nRcaYIv+79Uyge91HDJrpwH0B06UV2k8Gxvv38/GEnnQgRkQ+F5Gv/IV1mVDf9gCISE98b5InB8wL\nxf28zCZgeMD0ycA3/vHKXr/r9PW9QR+xEpHrgNsrzL7GGPOeiAwImBcPHAqYzgHaV1gvnt8PJecA\nCbUYNaiO8HtYDowHRlSyWgS+d/XPA42BRSKyzBizJ6hha1kVfb8ZeMwYM11E+uI7BBx4RCoe2Bcw\nXba9g3ZoOFiOtO3979bfBMZWaI8FXsD3Ls8BzBeRFcaY1UEPHDyB+y+AW0ScxpjSStoa1P5dHWNM\nLoCIxAEz8L0bD/Qu8CK+58DZIjI0lE6HAfnAU8Ar+AqJuSIi4bDtA9wDPFhhXiju5wAYY2aKSNuA\nWbaAMy+VbeM6fX1v0IWVMeZV4NUaLHoIiAuYjgOyq1imoIr2equq34OIdAGyA681CbALeMn/5LPH\nf42CAA2qsKqs7yISg/9duzFmoYi0FJHAHa8mfw8NwhG2fTd8L6h3GGO+qdCcDzxfdg2hiHyF711/\nQ37CrbhN7f6/7craGuz2roqIHIfv1MdEY8zbAfNtwHNl19WIyMfASUAoFVYbgEz//r1BRPYBzYFt\nhMe2TwQ6GWPmV2gKxf28Kp6A8SO9vlfVXqtC9VTgYYwxh4BiEUnzP9FkAAsqLLYIGOIfP7eS9obo\nbHyHPatqex9ARBoBJwDr6yhXsD2A/yiNiKQDWytcR7YIyBARu4i0xvciHDLXIPgL6unASGNMZdu/\nI75rTRz+C5/7AisrWa4hKd9//aeC1gS0LQP6iUiU/8L+zvgucA0J/tP6nwN3GWNeq9AcD6wVkUb+\n576BQKhda3UtvqPviEgLfH3e6W8L6W3vdwbwRSXzQ3E/r8oPAWepKnv9rtPX9wZ9xOoo3Qi8he+Q\n6OfGmO9EpDHwijFmOPAw8F//J0j2AiOti1prBJh32AyRJ4EZxpi5IpIhIkvxVfv3hFBx8TjwpoiU\nfervajis78tEZAGwBN+bi5utChokj+G7UPd5/+VlB40xw/yfDss0xswRkbeApfg+ETrNGPOTdXFr\nxWxgkIgsBmzANRX6OwHfk6kduNcYU3iEx2po7gGSgPtEpOxaqylArDFmsojcA8zH92nJL40xn1iU\nM1heBV4XkYX4PhV2LXCriITDtgff8/zm8onQ3s+r8g9giohE4DtAMANARD4HhgKT8L2+LwSKCfLr\nu83rDbUPhCmllFJKWSMsTgUqpZRSStUFLayUUkoppWqJFlZKKaWUUrVECyullFJKqVqihZVSSiml\nVC3RwkoppZRSqpZoYaWUUkopVUu0sFJKKaWUqiX/H6Zv8tNO8h1EAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1083b6438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(2, sharex=True, figsize=(10, 8))\n",
    "ax[0].plot(x, y, 'b')\n",
    "ax[0].plot(x, s, 'm--')\n",
    "\n",
    "ax[1].plot(x, d, 'g');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# input dataset (features)\n",
    "# layer 0\n",
    "l0 = np.array([[0, 0, 1],\n",
    "               [0, 1, 1],\n",
    "               [1, 0, 1],\n",
    "               [1, 1, 1] ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# output dataset (labels)          \n",
    "y = np.array([[0,\n",
    "               0,\n",
    "               1,\n",
    "               1]]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Single Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.16595599],\n",
       "       [ 0.44064899],\n",
       "       [-0.99977125]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# initialize weights randomly with mean 0\n",
    "np.random.seed(1)\n",
    "weights = 2 * np.random.random((3, 1)) - 1\n",
    "weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.99977125],\n",
       "       [-0.55912226],\n",
       "       [-1.16572724],\n",
       "       [-0.72507825]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(l0, weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.2689864 ],\n",
       "       [ 0.36375058],\n",
       "       [ 0.23762817],\n",
       "       [ 0.3262757 ]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l1 = sigmoid(np.dot(l0, weights))\n",
    "l1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.2689864 ],\n",
       "       [-0.36375058],\n",
       "       [ 0.76237183],\n",
       "       [ 0.6737243 ]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e = y - l1\n",
    "e"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.30994584990928159"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(e ** 2).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.24553187],\n",
       "       [ 0.24190935],\n",
       "       [ 0.24650375],\n",
       "       [ 0.24346281]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(l1, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.06604473],\n",
       "       [-0.08799467],\n",
       "       [ 0.18792752],\n",
       "       [ 0.16402681]])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = e * sigmoid(l1, True)\n",
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.35195432],\n",
       "       [ 0.07603214],\n",
       "       [ 0.19791493]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u = np.dot(l0.T, d)\n",
    "u"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.18599833],\n",
       "       [ 0.51668113],\n",
       "       [-0.80185633]])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights += u\n",
    "weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.24425422705654065"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l1 = sigmoid(np.dot(l0, weights))\n",
    "e = y - l1\n",
    "(e ** 2).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.30962857],\n",
       "       [ 0.42918547],\n",
       "       [ 0.35072407],\n",
       "       [ 0.47522609]])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple Steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.16595599],\n",
       "       [ 0.44064899],\n",
       "       [-0.99977125]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# initialize weights randomly with mean 0\n",
    "np.random.seed(1)\n",
    "weights = 2 * np.random.random((3, 1)) - 1\n",
    "weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "after 0 iterations\n",
      "layer 1: [[ 0.2689864   0.36375058  0.23762817  0.3262757 ]]\n",
      "errors:  [[-0.2689864  -0.36375058  0.76237183  0.6737243 ]]\n",
      "MSE:     0.309945849909\n",
      "\n",
      "after 200 iterations\n",
      "layer 1: [[ 0.03581881  0.02486184  0.97910131  0.96983694]]\n",
      "errors:  [[-0.03581881 -0.02486184  0.02089869  0.03016306]]\n",
      "MSE:     0.000811915861218\n"
     ]
    }
   ],
   "source": [
    "for _ in range(201):\n",
    "    # forward propagation\n",
    "    # layer 1\n",
    "    l1 = sigmoid(np.dot(l0, weights))\n",
    "\n",
    "    # errors of layer 1\n",
    "    e = y - l1\n",
    "    if _ % 200 == 0:\n",
    "        print('\\nafter %d iterations' % _)\n",
    "        print('layer 1:', l1.T)\n",
    "        print('errors: ', e.T)\n",
    "        print('MSE:    ', (e ** 2).mean())\n",
    "\n",
    "    # multiply errors by the slope of the \n",
    "    # sigmoid at the values in l1\n",
    "    d = e * sigmoid(l1, True)\n",
    "\n",
    "    # update weights\n",
    "    weights += np.dot(l0.T, d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://hilpisch.com/tpq_logo.png\" width=\"350px\" align=\"right\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

## 03_deep_learning.ipynb

      
Display the source blob

    
Display the rendered blob

    
    Raw
  

              03_deep_learning.ipynb
            
          
      Sorry, something went wrong. Reload?
      Sorry, we cannot display this file.
      Sorry, this file is invalid so it cannot be displayed.
      
          Viewer requires iframe.
      
    
## 04_eikon_cufflinks.ipynb

      
Display the source blob

    
Display the rendered blob

    
    Raw
  

              04_eikon_cufflinks.ipynb
            
          
      Sorry, something went wrong. Reload?
      Sorry, we cannot display this file.
      Sorry, this file is invalid so it cannot be displayed.
      
          Viewer requires iframe.
      
    
## 05_oanda_basics.ipynb

      
Display the source blob

    
Display the rendered blob

    
    Raw
  

              05_oanda_basics.ipynb
            
          
      Sorry, something went wrong. Reload?
      Sorry, we cannot display this file.
      Sorry, this file is invalid so it cannot be displayed.
      
          Viewer requires iframe.
      
    
## 06_oanda_streaming_plot.ipynb

      
Display the source blob

    
Display the rendered blob

    
    Raw
  

              06_oanda_streaming_plot.ipynb
            
          
      Sorry, something went wrong. Reload?
      Sorry, we cannot display this file.
      Sorry, this file is invalid so it cannot be displayed.
      
          Viewer requires iframe.
      
    
## plyst.py
#
# plyst -- A Simple Plotly Streaming Wrapper Class
#
# For illustration purposes only
#
# The codes contained herein come without warranties or representations,
# to the extent permitted by applicable law.
#
# Read the RISK DISCLAIMER carefully.
#
# (c) The Python Quants GmbH
#
import configparser
import plotly.plotly as ply
from plotly.graph_objs import *

c = configparser.ConfigParser()
c.read('../pyalgo.cfg')
stream_ids = c['plotly']['api_tokens'].split(',')


class plyst(object):
    def __init__(self, streams=1, title=''):
        self.streams = {}
        self.scatters = []
        self.sockets = {}
        for stream in range(streams):
            self.streams[stream] = Stream(maxpoints=100,
                                          token=stream_ids[stream])
            self.scatters.append(Scatter(x=[], y=[],
                                         name='stream_%d' % stream,
                                         mode='lines+markers',
                                         stream=self.streams[stream]))
            self.sockets[stream] = ply.Stream(stream_ids[stream])
            self.sockets[stream].open()
        self.data = Data(self.scatters)
        self.layout = Layout(title=title)
        self.figure = Figure(data=self.data, layout=self.layout)
        self.fig_object = ply.plot(self.figure,
                                   filename='stream_plot_%s' % title,
                                   auto_open=False)

    def get_figure(self):
        return self.fig_object

## tick_client.py
#
# Simple Tick Data Client with ZeroMQ
#
import zmq
import datetime

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')

socket.setsockopt_string(zmq.SUBSCRIBE, 'AAPL')

while True:
    msg = socket.recv_string()
    t = datetime.datetime.now()
    print('%s | ' % str(t), msg)

## tick_collector.py
#
# Simple Tick Data Collector with ZeroMQ & pandas
#
import zmq
import datetime
import pandas as pd

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')

socket.setsockopt_string(zmq.SUBSCRIBE, 'AAPL')

raw = pd.DataFrame()

i = 0
while True:
    msg = socket.recv_string()
    i += 1
    t = datetime.datetime.now()
    print('%s | ' % str(t), msg)
    sym, value = msg.split()
    raw = raw.append(pd.DataFrame({sym: float(value)}, index=[t]))
    resam = raw.resample('5s', label='right').last()
    if i % 10 == 0:
        print('\n', resam.tail(3), '\n')

## tick_server.py
#
# Simple Tick Data Server with ZeroMQ
#
import zmq
import time
import random

context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind('tcp://127.0.0.1:5555')

AAPL = 100.

while True:
    AAPL += random.gauss(0, 1) * 0.5
    msg = 'AAPL %.4f' % AAPL
    socket.send_string(msg)
    print(msg)
    time.sleep(random.random() * 2)

## tick_sma_trader.py
#
# Simple Tick Data Collector with ZeroMQ & pandas
#
import zmq
import datetime
import pandas as pd

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')

socket.setsockopt_string(zmq.SUBSCRIBE, 'AAPL')

raw = pd.DataFrame()

i = 0
position = 0
while True:
    msg = socket.recv_string()
    i += 1
    t = datetime.datetime.now()
    print('%s | ' % str(t), msg)
    sym, value = msg.split()
    raw = raw.append(pd.DataFrame({sym: float(value)}, index=[t]))
    resam = raw.resample('5s', label='right').last()
    resam['SMA1'] = resam[sym].rolling(3).mean()
    resam['SMA2'] = resam[sym].rolling(6).mean()
    if i % 10 == 0:
        print('\n', resam.tail(3), '\n')
    if len(resam) > 6:
        if resam['SMA1'].iloc[-2] > resam['SMA2'].iloc[-2] and position in [0, -1]:
            print('\ngoing long the market\n')
            position = 1
            # place trading logic here
        if resam['SMA1'].iloc[-2] < resam['SMA2'].iloc[-2] and position in [0, 1]:
            print('\ngoing short the market\n')
            position = -1
            # place trading logic here


## tpqoa.py
#
# tpqoa is a wrapper class for the
# Oanda v20 API (RESTful & streaming)
# (c) Dr. Yves J. Hilpisch
# The Python Quants GmbH
#
# dependencies:
# pip install v20
# conda install pyyaml
#
import v20
import pandas as pd
import datetime as dt
import configparser

class tpqoa(object):
    ''' tpqoa is a Python wrapper class for the Oanda v20 API. '''

    def __init__(self, conf_file):
        ''' Init function expecting a configuration file with
        the following content:

        [oanda_v20]
        account_id = XYZ-ABC-...
        access_token = ZYXCAB...

        Parameters
        ==========
        conf_file: string
            path to and filename of the configuration file, e.g. '/home/me/oanda.cfg'
        '''
        self.config = configparser.ConfigParser()
        self.config.read(conf_file)
        self.access_token = self.config['oanda_v20']['access_token']
        self.account_id = self.config['oanda_v20']['account_id']
        self.ctx = v20.Context(
            hostname='api-fxpractice.oanda.com',
            port=443,
            ssl=True,
            application='sample_code',
            token=self.access_token,
            datetime_format='RFC3339')
        self.ctx_stream = v20.Context(
            hostname='stream-fxpractice.oanda.com',
            port=443,
            ssl=True,
            application='sample_code',
            token=self.access_token,
            datetime_format='RFC3339'
        )
        self.suffix = '.000000000Z'

    def get_instruments(self):
        ''' Retrieves and returns all instruments for the given account. '''
        resp = self.ctx.account.instruments(self.account_id)
        instruments = resp.get('instruments')
        instruments = [ins.dict() for ins in instruments]
        instruments = [(ins['displayName'], ins['name'])
                      for ins in instruments]
        return instruments

    def transform_datetime(self, dt):
        ''' Transforms Python datetime object to string. '''
        if isinstance(dt, str):
            dt = pd.Timestamp(dt).to_pydatetime()
        return dt.isoformat('T') + self.suffix

    def retrieve_data(self, instrument, start, end, granularity, price):
        raw = self.ctx.instrument.candles(
            instrument=instrument,
            fromTime=start, toTime=end,
            granularity=granularity, price=price)
        raw = raw.get('candles')
        raw = [cs.dict() for cs in raw]
        for cs in raw:
            cs.update(cs['ask'])
            del cs['ask']
        if len(raw) == 0:
            return pd.DataFrame() # return empty DataFrame is no data
        data = pd.DataFrame(raw)
        data['time'] = pd.to_datetime(data['time'])
        data = data.set_index('time')
        data.index = pd.DatetimeIndex(data.index)
        for col in list('ohlc'):
            data[col] = data[col].astype(float)
        return data

    def get_history(self, instrument, start, end,
                    granularity, price):
        ''' Retrieves historical data for instrument.

        Parameters
        ==========
        instrument: string
            valid instrument name
        start, end: datetime, str
            Python datetime or string objects for start and end
        granularity: string
            a string like 'S5', 'M1' or 'D'
        price: string
            one of 'A' (ask) or 'B' (bid)

        Returns
        =======
        data: pd.DataFrame
            pandas DataFrame object with data
        '''
        if granularity.startswith('S'):
            data = pd.DataFrame()
            dr = pd.date_range(start, end, freq='4h')
            for t in range(len(dr) - 1):
                start = self.transform_datetime(dr[t])
                end = self.transform_datetime(dr[t + 1])
                batch = self.retrieve_data(instrument, start, end,
                                          granularity, price)
                data = data.append(batch)
        else:
            start = self.transform_datetime(start)
            end = self.transform_datetime(end)
            data = self.retrieve_data(instrument, start, end,
                                      granularity, price)

        return data[['o', 'h', 'l', 'c', 'complete', 'volume']]

    def create_order(self, instrument, units):
        ''' Places order with Oanda.

        Parameters
        ==========
        instrument: string
            valid instrument name
        units: int
            number of units of instrument to be bought (positive int, eg 'units=50')
            or to be sold (negative int, eg 'units=-100')
        '''
        request = self.ctx.order.market(
            self.account_id,
            instrument=instrument,
            units=units,
        )
        order = request.get('orderFillTransaction')
        print('\n\n', order.dict(), '\n')

    def stream_data(self, instrument, stop=None):
        ''' Starts a real-time data stream.

        Parameters
        ==========
        instrument: string
            valid instrument name
        '''
        self.stream_instrument = instrument
        self.ticks = 0
        response = self.ctx_stream.pricing.stream(
            self.account_id, snapshot=True,
            instruments=instrument)
        for msg_type, msg in response.parts():
            # print(msg_type, msg)
            if msg_type == 'pricing.Price':
                self.ticks +=1
                self.on_success(msg.time,
                                float(msg.bids[0].price),
                                float(msg.asks[0].price))
                if stop is not None:
                    if self.ticks >= stop:
                        break

    def on_success(self, time, bid, ask):
        ''' Method called when new data is retrieved. '''
        print(time, bid, ask)

    def get_account_summary(self, detailed=False):
        ''' Returns summary data for Oanda account.'''
        if detailed is True:
            response = self.ctx.account.get(self.account_id)
        else:
            response = self.ctx.account.summary(self.account_id)
        raw = response.get('account')
        return raw.dict()

    def get_transactions(self, tid=0):
        ''' Retrieves and returns transactions data. '''
        response = self.ctx.transaction.since(self.account_id, id=tid)
        transactions = response.get('transactions')
        transactions = [t.dict() for t in transactions]
        return transactions

    def print_transactions(self, tid=0):
        ''' Prints basic transactions data. '''
        transactions = self.get_transactions(tid)
        for trans in transactions:
             templ = '%5s | %s | %9s | %12s'
             print(templ % (trans['id'],
                            trans['time'],
                            trans['instrument'],
                            trans['units']))
	#
	# plyst -- A Simple Plotly Streaming Wrapper Class
	#
	# For illustration purposes only
	#
	# The codes contained herein come without warranties or representations,
	# to the extent permitted by applicable law.
	#
	# Read the RISK DISCLAIMER carefully.
	#
	# (c) The Python Quants GmbH
	#
	import configparser
	import plotly.plotly as ply
	from plotly.graph_objs import *

	c = configparser.ConfigParser()
	c.read('../pyalgo.cfg')
	stream_ids = c['plotly']['api_tokens'].split(',')


	class plyst(object):
	def __init__(self, streams=1, title=''):
	self.streams = {}
	self.scatters = []
	self.sockets = {}
	for stream in range(streams):
	self.streams[stream] = Stream(maxpoints=100,
	token=stream_ids[stream])
	self.scatters.append(Scatter(x=[], y=[],
	name='stream_%d' % stream,
	mode='lines+markers',
	stream=self.streams[stream]))
	self.sockets[stream] = ply.Stream(stream_ids[stream])
	self.sockets[stream].open()
	self.data = Data(self.scatters)
	self.layout = Layout(title=title)
	self.figure = Figure(data=self.data, layout=self.layout)
	self.fig_object = ply.plot(self.figure,
	filename='stream_plot_%s' % title,
	auto_open=False)

	def get_figure(self):
	return self.fig_object
	#
	# Simple Tick Data Client with ZeroMQ
	#
	import zmq
	import datetime

	context = zmq.Context()
	socket = context.socket(zmq.SUB)
	socket.connect('tcp://127.0.0.1:5555')

	socket.setsockopt_string(zmq.SUBSCRIBE, 'AAPL')

	while True:
	msg = socket.recv_string()
	t = datetime.datetime.now()
	print('%s \| ' % str(t), msg)
	#
	# Simple Tick Data Collector with ZeroMQ & pandas
	#
	import zmq
	import datetime
	import pandas as pd

	context = zmq.Context()
	socket = context.socket(zmq.SUB)
	socket.connect('tcp://127.0.0.1:5555')

	socket.setsockopt_string(zmq.SUBSCRIBE, 'AAPL')

	raw = pd.DataFrame()

	i = 0
	while True:
	msg = socket.recv_string()
	i += 1
	t = datetime.datetime.now()
	print('%s \| ' % str(t), msg)
	sym, value = msg.split()
	raw = raw.append(pd.DataFrame({sym: float(value)}, index=[t]))
	resam = raw.resample('5s', label='right').last()
	if i % 10 == 0:
	print('\n', resam.tail(3), '\n')
	#
	# Simple Tick Data Server with ZeroMQ
	#
	import zmq
	import time
	import random

	context = zmq.Context()
	socket = context.socket(zmq.PUB)
	socket.bind('tcp://127.0.0.1:5555')

	AAPL = 100.

	while True:
	AAPL += random.gauss(0, 1) * 0.5
	msg = 'AAPL %.4f' % AAPL
	socket.send_string(msg)
	print(msg)
	time.sleep(random.random() * 2)
	#
	# tpqoa is a wrapper class for the
	# Oanda v20 API (RESTful & streaming)
	# (c) Dr. Yves J. Hilpisch
	# The Python Quants GmbH
	#
	# dependencies:
	# pip install v20
	# conda install pyyaml
	#
	import v20
	import pandas as pd
	import datetime as dt
	import configparser

	class tpqoa(object):
	''' tpqoa is a Python wrapper class for the Oanda v20 API. '''

	def __init__(self, conf_file):
	''' Init function expecting a configuration file with
	the following content:

	[oanda_v20]
	account_id = XYZ-ABC-...
	access_token = ZYXCAB...

	Parameters
	==========
	conf_file: string
	path to and filename of the configuration file, e.g. '/home/me/oanda.cfg'
	'''
	self.config = configparser.ConfigParser()
	self.config.read(conf_file)
	self.access_token = self.config['oanda_v20']['access_token']
	self.account_id = self.config['oanda_v20']['account_id']
	self.ctx = v20.Context(
	hostname='api-fxpractice.oanda.com',
	port=443,
	ssl=True,
	application='sample_code',
	token=self.access_token,
	datetime_format='RFC3339')
	self.ctx_stream = v20.Context(
	hostname='stream-fxpractice.oanda.com',
	port=443,
	ssl=True,
	application='sample_code',
	token=self.access_token,
	datetime_format='RFC3339'
	)
	self.suffix = '.000000000Z'

	def get_instruments(self):
	''' Retrieves and returns all instruments for the given account. '''
	resp = self.ctx.account.instruments(self.account_id)
	instruments = resp.get('instruments')
	instruments = [ins.dict() for ins in instruments]
	instruments = [(ins['displayName'], ins['name'])
	for ins in instruments]
	return instruments

	def transform_datetime(self, dt):
	''' Transforms Python datetime object to string. '''
	if isinstance(dt, str):
	dt = pd.Timestamp(dt).to_pydatetime()
	return dt.isoformat('T') + self.suffix

	def retrieve_data(self, instrument, start, end, granularity, price):
	raw = self.ctx.instrument.candles(
	instrument=instrument,
	fromTime=start, toTime=end,
	granularity=granularity, price=price)
	raw = raw.get('candles')
	raw = [cs.dict() for cs in raw]
	for cs in raw:
	cs.update(cs['ask'])
	del cs['ask']
	if len(raw) == 0:
	return pd.DataFrame() # return empty DataFrame is no data
	data = pd.DataFrame(raw)
	data['time'] = pd.to_datetime(data['time'])
	data = data.set_index('time')
	data.index = pd.DatetimeIndex(data.index)
	for col in list('ohlc'):
	data[col] = data[col].astype(float)
	return data

	def get_history(self, instrument, start, end,
	granularity, price):
	''' Retrieves historical data for instrument.

	Parameters
	==========
	instrument: string
	valid instrument name
	start, end: datetime, str
	Python datetime or string objects for start and end
	granularity: string
	a string like 'S5', 'M1' or 'D'
	price: string
	one of 'A' (ask) or 'B' (bid)

	Returns
	=======
	data: pd.DataFrame
	pandas DataFrame object with data
	'''
	if granularity.startswith('S'):
	data = pd.DataFrame()
	dr = pd.date_range(start, end, freq='4h')
	for t in range(len(dr) - 1):
	start = self.transform_datetime(dr[t])
	end = self.transform_datetime(dr[t + 1])
	batch = self.retrieve_data(instrument, start, end,
	granularity, price)
	data = data.append(batch)
	else:
	start = self.transform_datetime(start)
	end = self.transform_datetime(end)
	data = self.retrieve_data(instrument, start, end,
	granularity, price)

	return data[['o', 'h', 'l', 'c', 'complete', 'volume']]

	def create_order(self, instrument, units):
	''' Places order with Oanda.

	Parameters
	==========
	instrument: string
	valid instrument name
	units: int
	number of units of instrument to be bought (positive int, eg 'units=50')
	or to be sold (negative int, eg 'units=-100')
	'''
	request = self.ctx.order.market(
	self.account_id,
	instrument=instrument,
	units=units,
	)
	order = request.get('orderFillTransaction')
	print('\n\n', order.dict(), '\n')

	def stream_data(self, instrument, stop=None):
	''' Starts a real-time data stream.

	Parameters
	==========
	instrument: string
	valid instrument name
	'''
	self.stream_instrument = instrument
	self.ticks = 0
	response = self.ctx_stream.pricing.stream(
	self.account_id, snapshot=True,
	instruments=instrument)
	for msg_type, msg in response.parts():
	# print(msg_type, msg)
	if msg_type == 'pricing.Price':
	self.ticks +=1
	self.on_success(msg.time,
	float(msg.bids[0].price),
	float(msg.asks[0].price))
	if stop is not None:
	if self.ticks >= stop:
	break

	def on_success(self, time, bid, ask):
	''' Method called when new data is retrieved. '''
	print(time, bid, ask)

	def get_account_summary(self, detailed=False):
	''' Returns summary data for Oanda account.'''
	if detailed is True:
	response = self.ctx.account.get(self.account_id)
	else:
	response = self.ctx.account.summary(self.account_id)
	raw = response.get('account')
	return raw.dict()

	def get_transactions(self, tid=0):
	''' Retrieves and returns transactions data. '''
	response = self.ctx.transaction.since(self.account_id, id=tid)
	transactions = response.get('transactions')
	transactions = [t.dict() for t in transactions]
	return transactions

	def print_transactions(self, tid=0):
	''' Prints basic transactions data. '''
	transactions = self.get_transactions(tid)
	for trans in transactions:
	templ = '%5s \| %s \| %9s \| %12s'
	print(templ % (trans['id'],
	trans['time'],
	trans['instrument'],
	trans['units']))