Instantly share code, notes, and snippets.

wohlert/Backpropagation.ipynb

Last active October 30, 2018 15:58
Show Gist options
• Save wohlert/5bc9567d2b2c7b44bf41a5b56ea1755c to your computer and use it in GitHub Desktop.
Tutorial describing backpropagation in a MLP
Display the source blob
Display the rendered blob
Raw
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
 { "cells": [ { "cell_type": "code", "execution_count": 54, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Imports\n", "%matplotlib inline\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "sns.set_style(\"whitegrid\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Neural networks - Backpropagation\n", "\n", "In this notebook we will go through the backpropagation algorithm to give a better intuition for how it works in practice. This will be a regression problem with a single input and one output. In this example we will be working with a toy data problem in which the function to approximate contains heteroscedastic noise.\n", "\n", "The data is defined according to the second degree polynomial $f(x) = 2x^2 - 0.7x$. which is the true function to be approximated. We add noise to the data such that the oberved labels have the following form:\n", "\n", "$$y = f(x) + \\epsilon \\cdot f'(x)$$\n", "\n", "Where $\\epsilon \\sim \\mathcal{N}(0, 0.3)$ is noise. The function can be seen in the plot below." ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "code_folding": [ 0 ], "collapsed": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAEuCAYAAAB4RZ0yAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xlc1NX+P/DXzAAKAqIiOwNumFrupm1qm61m9svSyq/l\nbcUUSu6t67W9LJMSs+vSXnrLtExsua1umVqKuSFYqMgiICgiMCLMzPn9YcNlmeUzM5+Z+czM6/l4\n+FCZ4fM5Zz7LvD/nvM85KiGEABERERHZpPZ0AYiIiIi8BQMnIiIiIokYOBERERFJxMCJiIiISCIG\nTkREREQSMXAiIiIikijAXTvKyclx166IiIiInDZs2LB2P3Nb4GSpAK6Uk5Pj9n0qgb/WG/Dfuvtr\nvQH/rbu/1hvw37r7a70Bz9TdUoMPu+qIiIiIJGLgRERERCQRAyciIiIiiRg4EREREUnEwImIiIhI\nIgZORERERBIxcCIiIiKSiIETERERSVJcXIyZM2di6tSpmDx5Mp599lnU1dVZ/Z1PP/0UTU1NyMvL\nw5tvvunU/i+77LJ2P1u5ciVuuOEGfPHFF3j22WcBADt37kR+fr5T+7KEgZMVOp0Ohw8fhk6n83RR\niIiIPKqhoQGpqam4//77sWLFCqxatQqDBg3C7Nmzrf7e8uXLYTQa0a9fPzz66KOyl+v7779HVlYW\nJk6c2Bw4ff755zhx4oTs+wLcPHO4t9Dr9cjIyEB2djaKioqg1WoxYcIEZGZmIiCAHxkREfmfTZs2\nYcSIERg0aFDzzyZOnIhPPvkExcXF+Pe//w0hBMrKyqDT6TB//nzs3r0blZWVeOyxxzBt2jSsWrUK\nCxcuxLXXXoshQ4agsLAQl1xyCWpra7Fv3z706NEDCxYswB9//IFXXnkFBoMB1dXVmDJlitmZwz/9\n9FMcPHgQ//rXv7Bw4ULMnj0bTz/9NH7++Wfk5uaid+/eiIuLk/VzYBRgRkZGBhYtWtT8/8LCwub/\nZ2VleapYREREAID3vszFL3tLZd3mZYPiMX38AIuvFxcXQ6vVtvt5QkICjh8/DgBITEzE/PnzsXnz\nZixYsADLli3D0qVLsXDhQuzZs6f5d0pLS/Hhhx+ie/fuuPjii7FmzRo89dRTuPrqq3HmzBkUFBTg\niSeeQN++ffHll1/i66+/xpQpU9rt+84778RXX32FZ599FiqVCgBw4YUX4oorrsCNN94oe9AEsKuu\nHZ1Oh3Xr1pl9LTs7m912RETkl6Kjo1FSUtLu58eOHWsOUEaNGgUAGDJkCI4ePWpxWxEREYiLi0Ng\nYCBCQkLQu3dvqFQqhIWF4dy5c4iKisKSJUvwxBNP4LvvvoPBYHBNpRzAFqc2ysrKUFxcbPa14uJi\nlJWVoVevXm4uFRER0f9MHz/AauuQK1x99dVYtmwZ9u3bh4EDBwIA1qxZgy5duiAxMREAkJubi+HD\nh2P37t3o06cPAEClUsFoNLbalql1yJKXXnoJmZmZ6NWrF9544w3s27fPrrKqVCoIIez6HakYOLUR\nGxuLhIQEFBUVtXstMTERsbGxHigVERGRZ3Xq1AnLli3DvHnzcPr0aRgMBvTt2xevv/5683u2bNmC\nn376CUajES+//DIAYPjw4XjwwQcxY8YMyfu65ZZbkJaWhvDwcMTExKC6utqusg4aNAiZmZlISEiQ\nvbGDgVMLer0ec+bMsXiAJkyYgJCQEDeXioiISBm0Wi2WLVtm8fVp06Zh9OjRrX42f/785n+buvJ+\n+eWX5p+1/Hd2djYA4L777sN9993X/POcnJx27zVZsWJF879Xr14NAJg8eTImT55su0IOYODUQtuk\ncJPw8HDcd999yMzM9ECpiIiISCkYOP3FWlJ4REQE5s2bx6kIiIiILHjllVc8XQS34Ki6v1hLCi8t\nLUVZWZmbS0RERERKw8DpL7GxsWbnpwCYFE5ERETnMXD6S0hICCZMmGD2NSaFExEREcAcp1ZMyd/Z\n2dkoLi5GYmJi81IrRERERGxxaiEgIABZWVnIzc3FoUOHkJubi6ysLCaFExGRV5Jzsfpff/0Vl1xy\nCaZOnYp77rkHkydPxjfffGP1d44fP44NGzY4ve+28vLy8Oabb1p8/dy5c1izZo3s+wUYOJkVEhKC\nXr16sXuOiIi8kl6vR3p6OgYMGICUlBQMGDAA6enp0Ov1Tm131KhRWLFiBVauXIl3330X77zzDvLy\n8iy+f8eOHdi9e7dT+zSnX79+ePTRRy2+XllZ6bLAiU0pREREPsYdi9V36tQJd955J7799lukpKTg\n6aefRnl5OU6cOIGrrroKs2bNwltvvYWGhgYMGTIEYWFhePPNNyGEQH19PV577TX06NGjeXtr167F\njz/+iPr6elRXV2PGjBm47rrr8Msvv+Cll15C165dm6cHysvLw6pVq7Bw4UKMGzcOQ4cOxdGjR9Gt\nWzcsXrwYy5YtQ0FBAd58801ccsklmD9/PgICAhAcHIxFixYhNDTU4XqzxYmIiMiHuHOx+m7duqG6\nuhplZWUYPHgw3n33XXz22WdYtWoVNBoNHnzwQdx88824+uqr8eeff2LBggVYsWIFxo0bh2+//bbd\n9s6ePYv3338f7733Hl555RU0NTXhqaeewmOPPYaVK1dixIgRWLp0aavfKS4uRlpaGj799FOcOnUK\n+/fvx8MPP4zevXvj0UcfxY8//ogbbrgBK1euxJQpU3DmzBmn6swWJyIiIh/izsXqjx8/jpiYGERE\nRGD//v3YsWMHQkND0djY2O690dHReOmllxASEoKKigoMHTq03XtGjBgBtVqNyMhIhIeHo6qqCqGh\noejatWvz66+//jrGjh3b/DtdunRpnjIoNjYW586da7XNhx9+GMuWLcO0adMQHR3dvECxo9jiRERE\n5EPcNS9hXV0d1qxZg+uvvx5r165FWFgYXnvtNUyfPh0NDQ0QQkCtVsNoNAIAnnrqKcybNw+vvPIK\noqKiIIRot83c3FwAQFVVFerq6hAVFYW6urrmNWR/++03JCcnt/odlUrVbjst97t+/XpMnDgRK1as\nQJ8+fZrXs3MUW5yIiIh8iGleQnNrrzo7L+GOHTswdepUqNVqGAwGzJw5Ez179oTBYMDs2bOxZ88e\nBAUFISkpCSdOnEBKSgqWLl2KAQMG4JZbbsHdd9+N4OBgREZG4sSJE+22X1VVhWnTpqG2thbPPPMM\nNBoNXnzxRbz00kt455130LlzZ7z88sv4888/rZazW7duaGpqwoIFCzBu3DjMnTsXwcHBUKvVeP75\n5x2uP8DASTKdToeysjLExsZytB0RESmaK+YlHDlyJLZv3272tT59+mD9+vXtfh4dHY3vvvsOAHDT\nTTfZ3MeIESOQkZHR6meXXnopnn/+eQwbNqxVWUaOHAkA+OWXX5p/vnDhwuZ/Z2dnN//b2VamlthV\nZ4OrhnQSERG5CucldB1+gja4Y0gnERGRK5jmJfQGt912m6eLIAlbnKxw55BOIiIiUj6nAqeTJ09i\nzJgxOHz4sFzlURQpQzqJiIjIfzgcODU1NeHpp59Gx44d5SyPorhrSCcRERF5B4cDp/nz52Py5MmI\nioqSszyKYhrSaY6zQzqJiIjI+6iEuRmobFi7di3Ky8uRmpqKqVOn4tlnn7WZfJaTk+NwIT1Jr9cj\nKysLW7ZsQXl5OWJiYjB69Gikp6dzdAIREZEPazkFgolDgdPdd98NlUoFlUqFvLw8JCcnY+nSpeje\nvbvF38nJyTFbAFeSc5/eNI+TJz5rpfDXuvtrvQH/rbu/1hvw37r7a70BZcUQDjWZ/Oc//2n+t6nF\nyVrQ5AuUPqSzZWBHRERErsHpCLycuQk6MzMzOUEnERGRCzidpLNixQo5ykEOMjdBZ2FhIaKjozlB\nJxERkczY4uTFOEEnERGRezFw8mKcoJOIiMi9GDh5MU7QSURE5F4+GzjpdDqUlJT4dHcVJ+gkIiJy\nL5+bwVGv1yMjIwPZ2dkoKiqCVqvFhAkTkJmZ6ZMTVmZmZgI4n9NUXFyMxMREjBo1qvnnREREJB+f\niyTMjTIz/V/uUWZKmBQzICAAWVlZmDdvXnNZ8vLyfDJIJCIi8jSf6qpz1ygzc3Mnpaene3TuJNME\nneyeIyIich2fapaQMspMjtm/3dmqRURERMrhUy1O7hhlxrmTiIiIHKPT6XD48GGv/q70qcDJHaPM\nOHcSERGRfZSY4uIon+qqA1qPMms7qk4OplatwsLCdq9x7iQiIqL2fCnFxadanID/jTLLzc3F2rVr\nkZubi6ysLNlGmXHuJCIiIul8LcXF51qcTEJCQpCQkOCSQMbc3ElytmoRERH5CncN3HIXnw2cXMnc\n3ElsaSIiImrP11JcfK6rzp04dxIREZF1vpbiwhYnIiIicilfSnFh4EREREQu5UspLgyciIiIyC1M\nKS7ejDlOZJYvzO5KREQkNwZO1Iovze5KREQkN3bVUSu+NLsrERGR3NjiRM18bXZXIiIiuTFwomZc\nwJiIiNzFW3NpGThRM9PsruZ44+yuRESkPN6eS8vAiZr52uyuRESkPKZc2sLCQhiNxuZc2oyMDE8X\nTRIGTtRKZmYm0tLSkJycDI1Gg+TkZKSlpXnl7K5ERKQsvpBLy1F11Iovze5KRETKIiWXVukTZLLF\n6S/emqTmKlzAmIiI5OYLubR+Hzh5e5IaERGRt/CFXFqHu+oMBgPmzp2Lo0ePQqVS4bnnnkNKSoqc\nZXMLTvhIRETkPqac2ezsbBQXFyMxMRETJkzwmlxah1ucNm7cCABYtWoV0tPTsXDhQtkK5S6+kKRG\nRETkTUy5tLm5uTh06BByc3ORlZWFgADvSLt2uJTXXHMNxo4dCwA4fvw4wsPD5SqT2/hCkhoREZE3\nMuXSehuncpwCAgLwxBNP4IUXXsD48ePlKpPb+EKSGhEREbmPSgghnN1IZWUl7rjjDnz99dcWE7ty\ncnKc3Y1LZGZmYtWqVe1+PnnyZK+ZjIuIiMiXNDQ0oKqqCpGRkejYsaPHyjFs2LB2P3O4q27dunWo\nqKjAQw89hODgYKhUKqjV1huwzBXAlXJycmzuc8WKFYiOjjabpCZXf6tOp3PrnEhS6u2r/LXu/lpv\nwH/r7q/1Bvy37v5Qb71ej4yMDGRnZ6OoqAharRYTJkzAlClTMHLkSLeWxVKDj8ORwbhx4/DPf/4T\nd999N/R6PebMmePRqNBRrpzw0dIJIGdQRkRE5CssjXSvqKjAJ5984sGS/Y/D394hISGtKuft5EpS\na9m6NGfOHE51QEREJIG1ke5btmyBTqdTxDxPfj8BplzaTqTZr18/vPfee2bfy6kOiIiIWrM20r28\nvBxlZWVuLpF5DJxk0na156KiItTW1pp9r2mqAyIiIjrP2kj3mJgYxYx0Z+AkA2vNi+ZwqgMiIqLW\nrC3HMnr0aEV00wFO5DjR/1hrXjTHW9bj8TY6nQ4lJSXo168fP18iIi9kaTmWKVOmeLhk/8MWJxlY\na14MCwtDUlISNBoNkpOTkZaW5jXr8XiLlvllt912GxdqJiLyUt6wHAsDJxlYa16cPn06Dh48qNgT\nwBe0zS8zjV7kBKZERN7JNNJdib0HPhk4VZzS4Zm3tuNgkftGrmVmZiItLQ3JycntWpeUfAJ4Oy7U\nTERE7uSTTR+NTQYcOHISuw8ZUNWwGw/eehE6BQe6dJ+unEiTLONCzURE5E4+2eKUGB2GRY+PQVzX\nQGzYVYxZr23EgcNVbtk3W5fciws1ExGRO/lk4AQACVFh+Nu4KNx5bQqqTp/FnKW/4IOvctGkN3i6\naCQja/llHL1IRERy88muOhONWoV7ru+H4RdE4/WPd+PzjQXYfegEZt81DEmx4Z4uHsmk5fDVtmsC\nEhERyclnW5xauiC5KxbNHovrRiXh6PEzeCxrM9ZtPgyjUXi6aG6h0+lw+PBhn02Ubjl8de3atRy9\nSERELuMXgRMABHcIwKOTBmPufRcjpGMA3l1/AE8t34bK6rOeLprLtF0/z9fnNwoJCUFCQgK754iI\nyGX8JnAyGXlhLBZnXIkR/aOxr6AKMzM3YMOuYgjhe61PnN+IiIhIXn4XOAFAl7COeGr6SDw6aRAM\nRoGFn+zGS+//huraBk8XTTac34iIiEh+fhk4AYBKpcJ1o5KxOONKXNirG37NLceMVzfi599LPV00\nWUiZ34iIiEgqX8+XlcpvAyeTmG6d8NLDl+GBWy/EuSYDXl25C698tBM1dec8XTSncH4jIiKSg7/l\ny9ri94ETAKjVKtxyRS8snj0W/ZK74pe9xzFjwQZs23fc00VzGOc3IiIiOTBftjUGTi3EdQ/FyzMu\nx/TxA6Br0OPlD3cic2UOanWNni6aQ6ytn0dERGQL82XbY+DUhkatwsSxvbHo8bFI0UZg8+8lmPHq\nBvx2sNzTRbNby/mNDh065NH5jdg3TkTkfZgv2x4DJwsSo8Pw6qNX4P9u7IdaXRNeePdXvP5xDs7U\ne1/rkyfXz2PfOBGR92K+bHsMnKzQaNSYdHUKsh4bg94JnbExpwSpr/6En/eU+uS8T67AvnEiIu/F\nfNn2GDhJkBQbjsxZo3Hfzf1xtkGPV1fswrwPfsPJGt+ddVwO7BsnIvJ+9ubL+npqBgMniTQaNW67\nsg8WZ1yJAT27YceBcsx4dQO+//UYW58sYN84EZH3k5ov6y+pGX4fONkbGcd1D8W8Ry5D6u2DYBTA\n4tV7MGfJVvy6O89no2tHsW+ciMh32MqX9ZfUDL8NnJyJjNVqFW64JBlvPD4aIaIKB46cwrMf7MPo\nW1KR5oPRtaPYN05E5B/8KTXDbwMnOSLjV158CqsX3o/d37wOo74RsYMmIudEHGZmPOPCknsXziVF\nROT7/Ck1wy8DJzki45bbOJ6/BZs+eBSl+VvQJbYvilXD8G72XpxrMshabm+kpLmkiIj8kTuStZ1J\nzfC2ZHK/DJzkiIzbbqPx7Bn8/s3r+O2LF9FQdwrrthTi0QUbsDv/hGzl9maenEuKiMgfmUtJmTFj\nBg4dOiR7kOJIaoa3JpP7ZeAkR9KypW2cOLoLRzYuwPjLknCi+iyeeXs7FqzcheraBqfLTUREJJW5\nlJQlS5bgggsucEmQYm9qhrcmkzsUODU1NeHvf/877rrrLtx+++346aef5C6XS8mRtGxtG7fcfAMe\nvG0wFqaPQYo2Alt+L8Uj8zfg2+2FMBrlm7rA25o3iYjIPaylpABwSZBiT2qGNyeTOxQ4rV+/HhER\nEfj444/xzjvv4IUXXpC7XC4nR9KyrW30jO+MV2eOxsMTL4LRKPDvz/biyX9vxbGyM06V3VPNmwzU\niIi8g7WUlJZcEaRISc3w5mRyhwKn66+/HmlpaQAAIQQ0Go2shXIHOZKWpWxDo1bhpst7YukTV+Gy\ngXHIKzyFtNc34aNvDqKh0bFAx93Nm97aD01E5K+spaS05KkgxZvn+XMocOrUqRNCQ0NRV1eHWbNm\nIT09Xe5yuY0cSctSttGtczCenDYCT/1tJLp27og1P/2JGQs24tcDZXbNPO6J5k1v7YcmIvJX1tJJ\nWvJUkOLN8/yphIPrhZSVlWHGjBnNeU625OTkOLIbn9SoN2LT/jPYkV8HowD6xHXEDcMi0DXMdmtX\nSUkJbrvtNhiNxnavqdVqrF27FgkJCbKVtaGhAZMmTTL7RBIXF4fVq1ejY8eOsu2PiIjkodfrkZWV\nhS1btuD48eNm3zN58mSPPQS3LF95eTliYmIwevRopKenK2bKmmHDhrX/oXBAZWWluP7668W2bdsk\n/86uXbsc2ZVTPLFPexSVnxFzlmwVNz++Tkz8x3qx8r95oqFRb/V36uvrRXJysgDQ7k9ycrKor6+X\ntd4FBQVCrVab3Z9GoxEFBQWy7UsOSj/mruKv9RbCf+vur/UWwn/rbm+96+vrRUFBgaisrBT5+fki\nNTVVJCcnC41GI5KTk0VaWppoampyUWntL2d9fb3F9ygphnCoq27ZsmU4c+YMlixZgqlTp2Lq1Klo\naOBwe3slRofhxYcvxT/uGY6wkCCs+uEQZry6Ab/lllv8HXc3b3pzPzQRkT9qm5c6YsQILF26FIsW\nLVLkZMTeNs+fQ5/Y3LlzMXfuXLnL4pdUKhWuGBKPYf2i8OkPfyB7y2G88N6vGN4vGg/eehFiIzu1\n+x3TqL3s7GwUFxcjMTEREyZMcMkyJqZAbdGiRe1ekxKo6XQ6lJWVITY21msuCiIib2bKSzUx5aUC\nQFZWFnr16uWpovkEv5wAU4lCOgbivvED8MbssRjYOxK78iowY8EGfPxdfrulW9y9jIkjUzdwJB4R\nkft58/xI3sLzbXTUijYmHC8+fCm27jmOd9YfwCffH8JPO4swffyFuHRgLFQqVfN7Tc2brmYK1ObN\nmye59cjWEw8REclPyvxISm5x8oZeCrY4KZCp+27pE1fhtrG9cepMA175aCf+ueQXHCmt8Vi5pPZD\n84mHiMgzvDUv1Zt6KRg4KZip++7ff78KIwfEIPfISaQv3IQ31+zB6dpzni6eRd48IywRkTfz1vmR\nvGm+QAZOXiCueyjmTh+J5x+8BAlRYfhuxzE89MqP+GJTAZr07edz8jRvfeIhIvIFciwp5k7e1kvB\nwMmLDOkbhcWzx+LhiRdBo1bhvS9z8eiCDfjtYLlds4+7miufeLheHhGRde4eQOQsb+ul8KvAyZ1f\nuq7al0ajxk2X98Tyf16Dmy/vgfJTOrzw7q945q3tKCp3bvFgOcn9xONN/d9ERErgLfMjeVsvhV8E\nTu780nXXvsJCgvDQxIF4Y/ZYDE7pjt//qMTMzI14c80e1J412N6Ai8nxxNMy+PSm/m8iIpLO2/Ky\nlNluJzN3Do13x75aDtdMignH8w9egp15FXj/y1x8t+MYNgSoUFafj4lje6NjB88eYkemTNDr9cjI\nyEB2djaKioqQkJCA6upqs+/Nzs7GvHnzFHdhERGRdO6c2NlZPt/i5M6kM2f3Zat7z1JrlsFgwMX9\nY/BmxpVIvX0QggJU+Pj7Q3jolR/x3Y5jMBiVk/8kRdvWpaKiItTW1pp9rxL7v4mIyD7elJfl84GT\nO5POHN2X1O49W91VGo0aN1ySjFnjY3DntSmoO6vHm2v2IO21jdiVV6GoBHJLrAWf5iix/5uIiBzj\nDXlZPh84uTPpzNF9Scnfsac1q0OgGvdc3w9v/fNqXHuxFkUVtXjunR14avk2HC457UQNXc9a8GmO\nEvu/iYjcgaOMPcPnAyd3Jp05si+pAZEjrVndOgdj1p1D8MbsKzH0gijs/bMKj2Vtxmsf56D8ZH2r\nMijl4rMWfIaFhSEpKckr5iUhInIVjjL2LOV1HrqAO5PO7N2X1HWFTAFFYWFhu/fZajlLjg3Hcw9c\ngj1/nMD7Xx7EppwSbN1Timsv1uLg1o/xdfYaFBUVQavVNpfVmX5lZ9YaMgWfLRPsTaZPn27XenlE\nRL6Ia4F6ls+3OAHuTTqzd19Su/fkaDkbnBKFhY+NQcbdw9A9IgT/3X4Mh43D0CH+cqgDg50e4i/X\nU5C1OaC8of+biMhVvG2WbV/kF4GTiTu/dKXuy1ZABKC5G02OSSXVahXGDE1A5sxRKN39KfTn6tFn\n5O246m/L0WvEbVAHBDl88ck115I3ja7wNFM3a0NDg6eLQkRu4G2zbPsivwqclMpcQDRz5kwYjcZW\nrTcZGRnIzMyUJaCoPFGBvVs+xYb3HsHBLR8AEOh3xf/hqunLoIroj+KSUru254qnILYuWda2dW/S\npEnMcSDyA942y7YvYuDkRpaSsM21sKjVaixevNhs640cAYXp4jPqG3Fk1zpseOch/LFjNQKCgjHg\nqgexYPVRbNhVLHkOKD4FuVfb1r2ysjLOpE7kB7xtlm1fxMDJDaTm/rScZdvVfdhtLz59ow5/bPsY\nG957GJ2NJTh5pgELP9mNGa9uwKbdJTYDKD4FuQ9zHIj8m9xrgTpDSaOy3YWBkxvYm/vjrtYbcxff\nIw/ciw8WPITlT16D60YlofxkPV77Tw4eXbABm60EUN76FOSNF/2RI0fYukfkx5SQB+rPUyIwcHIx\nR1oH3NV6Y+3ii+oagkcnDcayJ89Ponm8qh6Z/8nBzMyN+Pn3UhjNBFBKegqyxRsvelOZb7rpJhiN\nRrPvYesekf/wZB6oPy+87peBkztbGRxpPXJ36421iy+mWyfMunMIlj95Na4ZoUVpZR1eXbkLM1/b\niK17WwdQSngKksobL3pTmYuKiiy+R8mte0TkG9yVLmAwCvz8eyn+/sYWbMk9I8s25eBXgZMnWhkc\nbT1SWutNTLdOSJs8BEufuApXj0hEyYk6zP9oF2b9FUC17MJT+mg4b8gRahvc21rDLyYmRrGte0Tk\nW1ydTmIwGLExpxiPLtiAV1fuwh/FpxGoUTm1TTkprynAheSYbdXeWbGtzYRtrXXA1HqjtJmy4yJD\nkT55KO64JgWf/vAHNuUUY/5Hu5AQFYrbr+qDMUMTEKBRdjwudbZ2T9Dr9cjIyEB2dnar2dwfeeQR\ni2VWq9VYtGgR7rjjDjeXloj8kTMrWVijNxixcVcx1vz0J8pO1kOjVuHai7WYdHUKjh/Ld7LU8vGb\nwMlWK8O8efOsBiaWvtCkLE/izJIvLUfaKUlcZCgemzIU4y9LxCffHUTOHzXIWvU7Pv7+EP7flb1x\nzQgtggI1ni6mWa666OVgKbhvamqyWGatVov4+Hg3lpKI/JmjDQKWNOkN+HFnMT7b8CdOnNIhQKPG\nDZck4/9d1QfRXc9v6/gxWYouC78JnJxtZXCmtUqprUfOaBtI9uo7ECOuux+nz2ix9PN9WPX9Idw6\npjduuDQZwR2UdZrJfdG35Mw6fdaC+6+++go333wzlixZ0u61CRMmoGPHjg6Vl4jIEXKsAatraMK3\n248he0sBTp05h6AANcZf0RO3je2NyIhgVxXdacruU5GRMyPV5MqJkZr74w1D5NsmV/+ZtwcfZz2K\n2HNb8P+u7I2GRj3e/yoXf3vxe3zy/SHU6ho9XeRW5M4hkyN/zlpwX1RUhLq6OsycOVMxeW9E5L+c\nGQxUXdszSGuvAAAgAElEQVSAj745iOkv/oD3v8rF2XN6TBzbG+/861o8eOtFig6aAD9qcXKmlcFd\nOTHOdAe6k7VA8uvsz/DqvOdw+1V98OXWo/jy58P4+Lt8fLHpT9xwSQ/cMronunX2/EUhdyugHPlz\n1roQAeCjjz5CWloacnNzfablkoi8mz3pJOUn6/HFpgL8+FsRGvVGdA4NwtQb+uHGS5MRGhLk4pLK\nx29anADHWxncNa+StwyRlxJIhoYEYcq4vnh37jhMHz8AHYMCsHZTAe5/6Qcs/GQ3CsuUMbRUjhGA\ntlokq6qqJLUgWpuGouX2ACh61CIReUfPgbscPV6DBSt34aGXf8Q32woREd4RD982EO/OHYc7rknx\nqqAJ8LPAydGmRXfMq+TuIfLOXNTWAsmYmBh07ty5+f/BHQKam2Bn3jEYMd06YcOuYszM3Ihn3t6O\nvX9UQghp6+EplbVAsrCwEIMHD5bcfZeZmYlp06ZZfJ0zgxMpmzdOrusKQgjsK6jEs29vx6zXNmHL\n76XQxoQj4+5heOvJq3HTZT3QQaEDiGxxqv9n7969yMzMxIoVK+Qqj1s4MlJNjkQ4a7ypO9Bat2dp\naSlGjBjRbptBgRqMG5mEa0ZosSu/Ams3FmB3/gnszj+BnvGdMXFsb1w+KE7xUxmYY6uLrbS0FIC0\n7ruAgAAsWbIEGzduNDvRpadH/RGRdXJ023uzJr0RP+8pQfbmIzhyvAYAMKBnN9x+VR8MuyAKKpVy\n5mNylMOB09tvv43169cjONjz+Sru4OqRce4aIm/toranbi0DybZltnajUKtVuLh/DC7uH4M/iqqx\ndlMBtu87jtf+k4OPvjmIW67ohXEjtQjpGOhUPZ0Z3WYva4GkObamvwgJCcHEiRNdMuqPiFzH2Wlv\nvNmZ+kb8d/tRfPPLUZw6cw5qFXDZoDjcOqYXLkjq6uniycrhx3utVovFixfLWRav4KpZsT3dHfje\ne++hX79+kpuWTYHkzp07Lc4hZKuLMUXbBU/+3wgs/+c1uPmyHjhT34h31x/AfS98j7ez96Osqt6+\nCsJzzeRt8+eszaskpbtNaTPHE5Ft7lqgXUlKTtRiyWd7cd8L32Plf/PR0GjArWN64a0512LW7QMQ\nqK/2uTwvlXAiwaSkpASPP/44Vq9ebfO9OTk5ju7Gb+j1emRlZWHLli0oLy9HTEwMRo8ejfT0dFlG\n1ZWUlOC2226zuEBsW5MnT8ajjz6KqqoqREZGmp0ryNo21Wo11q5di4SEBEn7050zYOef9dj5Rx3q\nGs5vLyW+I0b2DUXP6A6SmngzMzOxatUqs3VxR5J9Q0MDqqqqEBoaiqlTp5q9UcbFxWH16tWS5l4y\nbc/S509E1rnzGmpoaMCkSZOcvu6VwNrnJoTA0Ypz2J5fhz+PNwAAIjppMLJvKIb06oQAlRFZWVnY\nvHkzKioqEB0djTFjxsj2XeZOw4YNa/9D4YTi4mIxadIkSe/dtWuXM7tyiCf2KYf6+npRUFAg6uvr\nHfp9S/Wur68XycnJAoCkP2FhYUKr1Qq1Wi2Sk5NFWlqaaGpqkrzN5ORkh+rQ2GQQm3KKxeyszeLm\nx9eJmx9fJx6Z/5P4ZttRcbahyeLv1dfXi9jYWFnL4oy0tDSzZUlLS5N9X956rsvBX+vur/UWQlrd\nm5qaRFpamkhOTrZ6D5ObK697dxxza5+brqFJfPPLETHj1Z+a781/f2OL2Lq3VOgNxuZtuOIzUFIM\n4V2hn59w1TIr9ubi1NbWora2FoDlvCVXzMIdGKDGmKEJGDM0AYeOncJXW49i695SLPlsLz78+iDG\njUzCTZf1aJ6K36SsrAwVFRVmt+mJNehcPaCAiCzzVJK2t1/35j63dz5agypVbzSF9ICuQQ+NWoXR\ng+Nxy+ie6Nsmf8kf8rwYOPmZ559/HjU1Ndi4cSNKSkoQHx+P6urq5gDJFnMnvitvFH2TuqJvUlfc\nN34Avt1eiP9uK8QXmwqQvbkAFw+IwQ2X9sDgPt2hVqsQGxuL6Ohos83knhiN5otL7RB5A09+eXvz\ndd/qc1OpEd1zOJIH34juSYNRA6BLoAa3jumN60YloWu4+S5HJS+iLhenAqeEhARJ+U3keW2nIUhI\nSMA999yDN954A08//bTkVihzJ747bhRdwzvirusuwKSr++DnPcfx5c+HseNAOXYcKEdMtxBcPyoZ\n11ysxZgxY8zmOHlyNJpSF2om8lVK+PL2xuu+rKwMFVU16DXi/yFp0HUICY8CAJwsPoCifd9iw/r3\nkZLS2+o2lLyIulzY4uQn2ja/FhUV4cMPP0RERES7FiNrrVDWTnx33CgCAzS4angirhyWgD+LT+Pb\n7YXY/HspPvj6IFZ+m4e+o+7Fg9Ep+D77I5c0k5ub5sCdUx94gq/Xj3yPrS/vzp074/Dhw5LPaV+/\nBoQQOHDkJL7ZWoWr7n8Hak0A9I1nUbj3vzi257+oPVmE5ORkJCTE2dyWKxdRVwrvm22Q7Gar2bqx\nsbHVjOp5eXmYPn262fcr5cRXqVRI0XbBrDuH4MNnrsODt16E2MhQ5BY14LhmKK576G28tWYbNm3Z\njpkzZ6Kx0blFhs1NczBr1izMmjXLZ2cI5gzI5K2sTe8SERGBESNGSDqnvekacGQ1iJq6c1i7sQCP\nzP8Jc5b8gq37ytFBfQ4HNr6NH9/6Gw78tBy1J89PxGvPvd/np1PxdHa6r+1TCdrWu6CgQKjVarOj\nHDQajSgoKGi3jZYjKzQajdtGpDjDaDSKz7/5RSxYsUvc+vf14ubH14kbZ60Wg6+bJfoPu1LMcqL8\nqampkkcjumLUnC2uONfdOSrQGbzO/Y+9o+pM97DBgwfbdU4r7RowV297Rw8aDEaxO79CvPzhb+LW\nv2eLmx9fJyb+Y73I/M8usb+gUjQ2Nsp273d2hHhLSoohGDj5oLb1dmbKADlPfDlZKpep7o+mZYie\nw28VV05f2jxsdux9/xb/l/6aqDqtk7yfpqYmkZqaKjQajeTAyRNTH8h9rtfX14ukpCTF1M8aXuf+\nx566m+4VlZWVdp3TSrwGzNVbanB3suas+PSHQ+JvL37ffE+c8epPIntLgThTf67ddpV271dSDMEc\nJz/gTJ+z0hIcpay1p9Pp8OW6NTh27BiO7MpGt8QLkXjh1YjtcwlOqTvgvhe+x7ALonHNxVpc3D8G\ngQGWe6wzMjKwZMkSu8roCyNHlJBcSyQH0z3s8OHDdp3T3nAN2ErDeO75F3Gg8Aw27CzGrvwKGI0C\nHYI0uPZiLcaNSkJfbReLEwsr7d6vJAyc/IS3zy1iImVultY3PIGTxftxsng/Dmx4GwkXXIEbpzyG\nXXkV2JVXgfBOQRg7LAHXXpyE5NjwVvuydlOyxlICvTclmPrDyBjyL/ae095wDVgK7iJi+iCs11V4\n8JVN0J0zAAB6JXTGdSOTMGZogtNrgfo7Jof7CdOUAaYE8NzcXGRlZXnV9Pe2nq5MSZGmG15b+nP1\nQE0+nvvbUDxxZ2/cdKkWKhWwfssRzMzciMcWbkL2lsOoPnN+CQFrT5zWtG3F86YEUxN3rJ1I5E72\nntPecA20vNd1DO2GXiNuw5hpi3H5XQuQNOgGBAVqMHFsbyzOuBJZj43FDZf2YNAkA+/51iRZeHPz\nq9QFNK11TZpG1Ji6+W6ZcCseuW82Nuwqxa78ChSUHMB76w9gUJ/uuPSiKCT36I0jh/9otx2NRoMH\nHngAgYGB+PLLL6224kmdwVhpLVK+0kpJZGLvOa30a0CtCcJVtz6I3KJGRCYNhEqlhkHfiNL8nzGs\nTziWPT0XGg3bR+Tm1CK/9sjJyTG/WJ6P7VMJfLXeOp0OAwYMMNt0npycjNzcXOTl5WHYsGGtcqFM\nN7yIiAjs2bOn3e+mpaUhKysLp2vP4ec9pdi8uwSHiqoBACphQMmhX1CatxmVx/ZAGM83e6empuLf\n//53c7ksBTw6nQ79+/fHsWPH2u03Pj4ee/bsQUREhM28LVtcecyVFtC15avnuy3+Wm/A+brbe057\n8hpoue+8vDwMHDQEvx86gS2/l+LX3DI0NJ6/J9WfPIKje76HWncM428cZ9f9wxsoKYbwnU+VfJ49\nSe5tZzPv3Lkzhg8fbna7piUYIsJCMP6Knhh/RU8cr6zD5t0l2JhTDKEajfgLRqNRV4Pasn0YmByC\nrAVPtSqXpVY8a61kpaWlGDx4MLp3794qoHPXmlpSeXMrJZE59p7TnrgGWg2EKS5Bv6FXos/Q69Ex\nsgJ1Z5sAADHdQjBmSAKuGp6IiE7XoqzsFsU+4PgSBk7kMY48xdnbdO7oiJq47qGYcIUWI3oFQGfs\nhJ/3lmHrXg2CQq5AKYCH52/EpQPjcPmgOKRYGZliLcEUOB88lZaWmn3NVxbEJCL7zc7IwEer/4v4\nvlfjqnGXoWNoV+gBnNXVYcKY/hg9JB59EiNa3Xt69erVPBEmAyjXYecnuZ0zydKOJrlbShgH2o+Q\naVm+vn37YsJ1l+LPXz7C+0+NwzP3j8LYYQmoO9uEdZsPI+ONn3H/Sz/g3fUHcOjYKbTt+baWYGpL\ny7wtRzgykzAReY4QAn8UVeOddXuR33ARLp/yKnoMHQ+1JgDH9n2H7avnYu8XT+Kua3u2e2DzxkEo\n3ootTuR2UpOlrXGkqV1qN5+t8g3vF40mvQG/H6rEz3tL8VtuOdZtPox1mw8jqktwu5YoU2vYZ599\nZrF1yRxHhzxLmevKk5SeM0XexdvPJ4NRIO/oSWzfX4Zt+8tQdfosAEAd2AklBzfh+KGfW+VXntZo\nzM4hJcd9VSpv/8yd5ukZOH1tn0qg5Hq7ejZea3WXsoyMI+VrbNKLXw+Uicz/7BJ3zPmqeVbe+174\nTiz/Yp/Y88cJ0aQ3iMrKShEfH++SpVta1ltpy0SY2Ls0hFRKPt9dyV/rLcT5urvqfLJFjhm1m/QG\nkZNfId5cs0fc88x/m+8Zd875Srz2n11i086jokfPPoqb5dxTn7kQyoohfD5wUtq08e6g5BuqI+vm\n2UNK3a2dE86Wz1IQNflfX4vM/+wSf0t7UWgCO7bb9uDBg51aG2rr1q0OLSvhTq4K6JR8vruSu+qt\nxHvorl273P6A4GzQcLahSWzff1y8/nGOuPNfXzffG+5++huxePXvIievQjQ2GZrfb0/9XH1fdaRM\ncmPg5AZNTU1i8uTJHomMPU3JXyTW1s2Lj48XlZWVTm3f2bo7s65fW41NBrE7v0Is/XyvuPe5b5tv\nlDc/tlaMuWueSB50veh9wcDm89KRLyjTzTw2Nlao1WqrLVpy3kDt5conYiWf767k6np7snXBlq1b\nt7r9AcFS0JCammrxdypO1ouvth4Rz7y1TUz8x/rme8C9z30rln+xT+wrqBR6g9Hs75prIZ88ebLZ\nz1/O+5Ylnl67T0mBk+cTHlwkIyMDq1atav6/0oZ4+2sfsbVco9LSUowYMcKj+TjOrOvXVmCAGkP6\nRmFI3yg8NPEiHC6pwY7cMvx6oByFqv64MKY/AECjjcCaDYcx7IIo9O7RE2q1+RF65rTNa7CWQ+WK\nZSKknsfesO4XtebOnBl7VVVVufV8srZqwfLlywEAixYtgkqtwZ9F1fjtYDl2HqxAYdmZ5vclx4bj\n4gExuLh/NPokdrF5nbedUsU0j5O5+6Kc9y1LeA3/j08GTraW5vDkEG+lJ+66Q8spBdoO01fCzdkV\nswWrVCr0ToxA78QI3HN9P5SfrMdvueX4NbccB46cxB9Fp/Hxd/noHBqEIX2jMOyCaAxJ6Y7OoR0s\nbtPetfTkXCZC6nlsCqw6d+6s+HW/6H+UfA8FgMjISLeeT9aCBnVgML74YQ+q//4O1OFJqKlrBHD+\nwWl4v2iM6B+N4f2iEdXFsc9L6kAYV89y7g1r97mNp5u8XMFd/b2OcEcfsbd0XVhLlna06VfOursr\nt6NW1yh+3lMiFq3aLf7v2f8lio6fvU7MztosPv42Txw6dqpdk7618xyASEhIaG7iT01NFfn5+a3q\n4kz9bJ3H5rp5Bg8eLNu537Ls3nK+y82V9VbaPdR0vCsrK0VBQYHYunWr1XNQ7mu3ZVeYSq0RXeP7\ni5RL7xKXTXlV3PTY2uZrduoz5/OVduw/Ls42yN+l6WwOp7OY43SeTzZxKDUytvYU98UXX3j8Kc7d\nampqLM5TpISmX1tPeraWWpHaFRsaHIjLB8Xj8kHxEEKgsOwMduVVICf/BPIKT+FQUTU+/v4QwjsF\nYWDvSAzq0x0D+0QiJibG4nmenJyMnTt34uTJk3jjjTfwzTffYNmyZdBqtRg/fjwA4Msvv3So1VNK\na8ScOXPadfMAwODBg3H69GmHn4jNtXSNGjUKK1as8JsWW7mZO1eVcg81He9169bh2LFj0Gg0MBgM\niImJwaRJkzBz5sxWa0WOHz8eRqMRAwYMkK1FXwiB0/VGjL01FQeOVKNb4kUI7HD+czIa9DhVehCV\nhXtwsngvft3yDfr07i3nR2A3V85yrvS1+9zG05GbqyhxSLatFoJ7771XlsRLb3kCd0VCoxx1t/XE\nZi1pVu6E2jpdo9i6t1QsWrVbTGuZYP7XdAdT0peK+AtGiw6dupg9zy1dB85cG7ZaI/bv3281idTU\nauCKli5/4uy5butclfuzdqQlxNb527Z1yd4yWypT1Wmd2JRTLBav/l1Mf/H7VtfdlfctERde9YCI\n6jmi1QhZX02QNscTIy2V1OLks4FTy1F1jg7xlpu1QEHOLwClXFxSyH1zdqbuUoMea2V25Re70WgU\nJSdqxde/HBHzPvhVTJn7dasb+thpi8UlEx4X96e9IE6errc6CsaZYNVWwLt//36XdPN4elSP0jh7\nndvT3erMPdTRhwkp52/L427P+dG2TL0vGCSmp78kFq3KEQ/O+6HVdXXnnK/EvA9+Ff/ddlSUVdWJ\n1NRUjwXv3nRvlxsDJzfuU2lzkNh6gnL1JJBKI9fN2cTRutfX14t7773X5g3R2s1Zq9UKrVbrti92\ng8EoCoqrxecb/hSz5n8tbvtHdqsb/v0v/FcMvDZVxPcbI4LDo2wGTvYENda+dF2RuyaE8vJuPM2Z\n69yeIKNtfpG9x8/RViBrAbi5427P+TEj7R8i7oLR4qJrUsXY+/7d6rqZ9M8vxbNvbxefb/jjfG6h\n3tCqfHLfs+zh6nu70r4vW1JS4OTzSQFKW9k9MzMTp0+fxocffmj2dSXk9riTuSG3rsrzMpfLYcqh\n+OKLL1BUVGT291qOIrI2uqakpMTivl1xXNVqFXolRKBXQgSSwmswcNAQ/FFUjX0FVTh49CTyCk9B\ne9E4aC8aBwA4W1uFU6UHcao0D6dKD6K2qgjnv1fOsyd3xVyugym/ZMSIERanRbBnZF/b4xUbG4uE\nhASzx6lt2f11ug+p7BlaHhQUhMWLFzs0Etie0Xlt89cSEhLQqVMn1NbWWtx+y+Nu6fxQqTXoc9Gl\n2FPYhM+37cLBoydxUnMpht54KQCg6Vw9Ko7sxMniAzhZcgCdOxoQd+sE3HKv+fq5857lLhztbSdP\nR26+tk8p6uvrXdoyodR6u4O5ulvrKpCSA9TyadVaN5W7W5xs1VuvN4iH0uaKHkPGi2HjnxDXPvRB\nqyfrcakrxcW3PSNSLr1LRPUYLmakZdi9Xyn5JSqVyq6ncnPHa+bMmWLmzJkiLCxMcveS0iZtlJuz\nLU5S8wud6X62pxXInnw8059p06aJ+vr65uMeFhYmgoLDRXTPEeKCy+8Rl9zxorhh5qdtut7WixET\n5oiewyaIzlG9BFTmy2dtYktPcdW93R25g862ZikphmAo6QEhISGYOHGiSycro/+xNJFfU1MTvv76\na5u/3/Kp1tpEcxMnTgQAxRxXjUaNNzOf+etJ8lPs+aYYPfoOxtDLbgCCY3BO3wFRyUMQlTwEAHAM\nwN9e/B59tF2QktgFKdoI9E6IQMcOlm8TphZdnU6HL774wux74uLisHPnTkRGRkoqt7njtXjxYrPv\nDQ8Px4033tjcAqbkSRuVROqEiY7O52Tv/F3W9hMWFoYuXbqgqKioeVRdSEgINBoNPvn0c+zKLUWX\n2BTUGzpi+O3z0alLXPPvCmFEB9Sjg6ESebs3oeDANnQNC0B1dbXVliyg9cSWvtzq4uo5u3yxNcs7\nS+0DOKzTPWzdFCxNh9BS26BHyrFTynE1dSs8//zzSEtLw4YNG/DZu/Og1Wpx44034m//NwMNCMWx\nirP4o6gafxRV45e9x/HL3uMAALUK0MaEo1dCZ/SM74yecZ3RI64zOgUHNu9Dr9cjNTXVYldneXk5\nampqJAVO9k7qGRERgfvvvx/Hjh1D586dFT1po9JIOY/tnS3a3JdkRESE2d9veV1Z249Op8O2bdug\n0gSh+qwGi99ehT+LTqFzdC+Edolvfl8EgMaGOpwo3I3q4/moPp6P0IAG3Hj9NViyZEnz++qqpX0+\nBoMBS5YsQWBgoE8H3a6eEdwXH2ZUQghh+23Oy8nJwbBhw9yxK4/u016uyMXwhnq7Stu6Hz58GCkp\nKTAaje3eq1arERsbazEfJykpCbfeeqvFJyO55nGSg61jnp6ebrZ1IS0trdXNSwiBilM6/Fl0Gn8U\nnw+kCkpq0NhkaPV70V1D0DP+fBC14ZvVWPneG2iorTK77+TkZOTm5kr6HKwdL0uioqJQVVVl9Vhq\nNBocOnTI7i8AJedK2TrmUstu6zweMGCAxbnC2h5XS+eZpfm7TNdV6/2oEBIRjfDIJIRFJiFG2x8X\nDLoUZSd1rbbZ1FCPmhOHcbqiADUVh1FTUQBdTUWr91i7xsPCwhAREWExaLBWT09xxb1dp9OhX79+\nZh98nK27TqdD//79cezYMae3raQYgi1OHiY1eV3JN3AlszaRn6nVpeXTqMm0adOwZMkSq5+1tWOn\npEEJ9jTFq1QqxHTrhJhunXDFkPNP8waDEaWVdcg/WoncgnJU1RpRWF6H7fvLsH1/GaAZiGseeAeN\nZ8+g9mQRak8Wo+5kyfm/TxXjFju6Ka0dL0tOnDgBQN51+uztXvD09dly/0FBQXaV3dZ5LHUNNGvn\n2enTp7Fz507U1NS0+oxq6s7hWPkZFB4/g0tu/QfiK+oQ1i0RAUHBrX7/TH0jBvaORLdQ4PV5/0R1\n+Z/QnS63+bnExsZabFU2tWS99tpr+OCDDyxuwxMDdtx1Pun1esyZMwfV1eab4ZxNMfDV9e0YOCmc\nL/YPu5OtG39mZiYCAwPNdlf4yufr7M1LCCNef+XpVufgDTfeiLS/PYKiinrMee41hEUmIzwqGV3j\n+6NbwoWtfr+8YwD+/sYWJEaHQRsThsToMMR3D0X3iGBoNOpW77V2vJxh7xeA1O4FcyPBrrzySrzx\nxhsIDw93uLxSvzjPnDnT3AVbUlLS3DW2Z88em2WXun+paQVHjhwxe56pNQGorhf4+fdi6NWdULoz\nH6Un6lBaWYdaXVOLN8ahS4wRDWfKceLITqj1NRjcT4sX5j6G6G6hUKlU0Ol0WPx0saSgCTh/3L/5\n5huLOVY9e/bE22+/jZCQECxfvhwGg8Hs+9w9U7q77vdtz3OT8PBw3HfffU6nGChlBnrZOZptbjAY\nxFNPPSXuuOMOcc8994jCwkKHstNdyRdGlzky2sEX6u0oW6PqLM27ouT5S6SwdsydnaHd2mgnrVbb\naqSbOiBIhEUmi7i+V4gRNzwsXnhnm3hk/o9iQkbrOaZufnydmJCRLR6Y94N4+q1tYunne8W6zQXi\n1wNl4kjJKTEz7bFWx8s0qs70M0vzRJn+tFynz95RdfbMcWTpswkLC5NlskitViumTZsmampq2r1v\n8uTJFkcZmvsTHx8vKisr7dp/y8/P0jXS1NQkZqY9JvpceLGI6jFcJA++SfQfM12MuHWuuHL6UnFT\n+udmj/1DL/8gnn9nh/jw61yxKadYFB6vEY1NBpvXoqXPfPDgwWavcan3UE9ObGmrbqYyyL0Wp7U5\n6eS6F8o1Yk9JMYTDOU7ff/89NmzYgFdeeQV79uzB8uXLsXTpUovvV1L/pLdwtH9Y6fV2ZTO0tbp7\nujvFleTKcWrL2jloS8ttN+mNKKuqQ3FFHYoqanG8qg7lVfUoO1nfvJp8SyoV0DW8IyI6qREbGY6Y\nyDB07xKMsGA10FgHbVw3jB19idV1+tp2C0llLc+qZa6UlM+m7efbcrSZufJZOk5hYWGYPn16c8uD\npffZEh8fj9tvv91iC4al7T6aNhsZTz6NkzUNqDilQ8XJepyoPouKU/UoOHYCRlWQ2f2d051G3anj\nSI6LwO23XIP47qGIjwpFTLdOCGjT2iiVXq/H1KlTsWPHjnYtYI2NjRbnarPVqiz1fa4i5X6fl5cn\n271d6nnuLLk+VyXFEA4HTi+//DIGDhyIm266CQBwxRVX4Oeff7a7AK6k9ADCFkdPbKXW2x3N0Eqt\nu6vZqrfUm1fb4FJqsnZYWBi6du2KkpISu2+MuoYmlP0VRJVV1bf696kzDbB0h1IJPc6cKsPZ2io0\n1J1Eo+40ztXX4Moxo5A24wFEhHZARFhHhIUENnfzSAmcq6qqMHjwYLM5Uy0fWKR8NvHx8di2bRvO\nnj2LN954A19//XWrxWqTkpJw0003YdasWejWrRuGDx9uMxCbO3euxfJJlZaWhtdfXwhdQxNq6htR\nU3cOZSdq8I9/PYezTSp0DO2Gjp26omNoV3QI7YqAwI7mNySMqK+pgO50BXRnTkBXU4GzZyqgq6lA\nffVxxMVEOjzAwpqcnBz069fPrt+VI1neEVK3J+V+f/r0adnub/Ym/suxP2c+VyXFEA4HTv/6178w\nbtw4jBkzBgAwduxY/PjjjxZvlDk5OY7sxq81NDRg0qRJZpMb4+LisHr1anTs2LHd71RVVSEyMrLd\na3KXzd79ZGZmYtWqVe1+PnnyZGRkZMhdRDLD0nHT6/XIysrC5s2bUVFRgejoaIwZMwYPP/wwpkyZ\nYjpDCzEAABm0SURBVHPaBrVajU8++QQdOnSQ9dzTGwRqzxpQozPgTL0BNTp9q39XnKoH1B2sl00F\nCL0OdTVV0NWehAYGREV2xsXDBqNTxwB0DFIjOEiFQI3A2jWf4LcdP+N4yTEYmhpgNOhbbev222/H\nk08+CeB8wvNdd93VnJxucf9qtaRRgpGRkaiqMj8y0SQ4OBghISE4efLk/7YfEITADp0QEBSMgKAQ\nBASFILBD8F9/d0JgcDiCgsPQIbgzgoLDERgchuDQLggKDrcYlJqcqz+NhrqTOFdfjdGXDkdM93BE\ndNIgIjQAq1e+i9WffAAI83VTqVRYtWqV2Yc7S+dbenq6z+QW2ltHR+73zpJyT3bXd4pSyTqqLjQ0\nFPX19c3/NxqNNk94pUSL3uSOO+4w23Q+adIkXHbZZc3/d1dSoaP70el02L59u9nXduzYgX79+sny\ndOMLx9wRLevtyJNdenp6qxtoWVkZVq1aha5du1o8B1vSarW4+eabPdIFunHzVgSHdkVQSGecbVLh\ndO2583/qzuF0bQN+338Ip2rPokNYFDp11QIA9AC25de331jU9bjolutx0V//NRoNMDQ1wKBvhKGp\nAdUw4uWVB6CCESfKSxE/fCrihAHCYIAQBhjb/C0MeghhhDAaAZUKKpXq/N9QASo1VCr89bcKgApR\nag00mkCoNYFQBwRCExD017+Dzv/8r/9rAjog4K/gSK3W2PV5NZ49g3Nna5GSHIOobuEI7xSEzqEd\nEBqsxvNPPYGio/loqDuFc/WnIYznA8fk5GQ8959/tpoYc/YjP1kMmoDz03mYOyd0Oh1mzJhh9nyL\njo6WlMDuDde5pWvKWh1t3e/lrveKFSsQHR1tMfFfSQOTPNXiZI7DtR86dCg2btyIG2+8EXv27EFK\nSorDhSPLpI5ocdckY47ux1eHpTpLzm4BZ4JaS8PIly9fjgceeAAzZ87El19+aXGaAFfOjG7rMwoP\nDcawYf0t/m7/5+5s7v5SawIR2DEUgR06QZvcB8veeR9NBjWqa+oxPzML9Q368600HUKgCegITWAH\naAI7QhMYhIDAjtAEdMRZ0QFqTQDC47oiPM7sbl3CaGiCQd8Eo6ERRn0TGupOQX+uBPpGHfSNZ9F0\nTgc19Bh4YT9MvOUmCGMjnvrXEygvLUSjrgaNZ2vR1FALIYzQarX4+s28dp/nz+t7YNG29e323fb4\nWrueLf2OvetCejNHZ+N298TI1tbda5vv5gsTV8rG0Wxz06i6O++8U9xxxx02VyZXUka8N7I20sSe\nUUDOlsHR/Tg7sksqbznmcq+ptmvXLodHr1hbT6zlNurr60V+fr5ITU2VfWV40/ldWVnZfJ5L/Yys\nHXNrdVOr1eK3334TBQUFYv/+/TY/g5Z/VOoAEdChk+jQqYvoGBopgsOjRGRcL9Gj7xARFpksOkf1\nEhGxKaJLXD/RLfFCEakdKLolXii6JVwousb3F13i+okucReIiNi+IiKmj4iI6SM6R/cS4d17iNCu\nCSI4PEp06NRFBHboJNQBQRbXUzP9iYuLMzv6ztqov9TUVJGfn9/q2pMyAlUIISorKy2ObNRoNCI1\nNbXd70hZi06tVtv8LrF1zD2h7f3ZnjX6pGzPxF31dtd3ij2UFENwkV8f4OxF6q79uGMhSW855nJ/\nFlu3bnVJUGtpG3JN32D6ojaVXaPRCAAiKSlJDB48WNJnZGsqBksLL7fcX2JioggNDZUcOFna1m+/\n/WZzqgR7/kRFRdl8j7VpBloufmvt2EqdnqNlYGVpe+YWx7X2RWwp6LJ2jinlOrcU3NfU1LjkQdFd\n9XbXd4o9lBRDMHDyAe5qzXF2P1KfZp3hDcfcFU9z69atc0lQ6+qbpZRWCFufkaVjLiVokPOPqVxS\n6mQK2Gxt79ixYzYDsbaBZNuAw1bwaGk79hwvlUpl9XqW0qrZ8o9pTiZLLY1Kuc6tPQBJfTiy5yHE\nnS1O7vhOsYeSYggGTj7CHa05cu3HlZNNesMxd8XT3NatW50OalNTUy1+obviZim1FcLWZ2TpmDsS\nlLX8Ex4ebrHVy9o1YK4VTaVSiaSkpObuscrKyuZuT0uBnWl7lurRqVOnVgGFpdaP/Px8SUGLlJZJ\nS8fL1uSaUlo1pX6+1o65O9l6AKqpqbH6oOhId7076+2u7xSplBRDMHDyEXLnzEjZj6tajZzhDcfc\nFU9zzuQ4teSq2ZPNBcv2tkJY+ozMHXNHg7KWf7RabfOXn61Wn3vvvbfdNWAub8ucmpoace+994qk\npCSbX7AajaZ5NvFNmza12o6l42/KSbNVX41GI/bv32+xrK7qqnfkuCvhOreVP2f6PCw9KCp9VQgl\n3evr6+vFunXr3N7SxcDJT7jrBFPqEiXecszlfprbtWuXLDc6uW+WtpbwcKQVQkqOk6NBmaVgwFoy\ntFzLU9i6ptq+3rLetlo/LAXELf+EhYUJrVZr8cFL7q76tsv12HM8lHCdW/s8LCXItwymHemu90S9\nPXmvd1eDgCUMnPyIr9Xbni8Ub6m7IwGK1GRZOW50ct0sbQWItlohLK0/1pKp7i3LLEfXkNR16ZTQ\ndWGrNSg/P99mUreUusndVW9PK5TSWpyEsH3+tu2+NQUA1lowrbXeKaXe7uLpa46Bkx/xlXrbetow\n9/rkyZMV020ohZQARcpTlxKPuZQkeGv5QLYWlzXZsWOH2c9n5syZFoOxtqP4pNycldR1IUT7YFlK\na5C5KSWstfy0DR5d2SJp2p6U0ZRKOd+l5gY6GiC2pZR6u4MSpkRg4ORHfKXejrZWeKoFwFWk1NOZ\nY+6qpnh7cmKk5gOZM3nyZLP7mDlzpsUv+Zb7s3duKqV0U7c95vZeD6Z6WJvDylLrh9yfQcvtSQnO\nlHSPs3We79+/3658O6XkOHmaEqZEYODkR3yh3raeNhzNEfA2Up+6HDnmrs4fcMeQ5vr6ehEbG2t1\nH1K/5JUSEEnV9pg72hqkxKHnpnIpfR4nIWx/frYmVw0NDZV8vJRUb1dTwnlp6fNWg0iBbC3Rsm/f\nPptLuPgCKUvVOMq0fE5hYSGMRmPzkgpyLbgcEhKCCRMmmH1NriVaysrKUFFRYfY10+cTEhKCXr16\n2dyf1PcplWn5jNzcXBw6dAi5ubnIysqyua6YO46TI7zleNj6/Hr27AmtVmvx9+vq6nDPPfdIPl7+\nQqnnJQCwxckH+UK9bT1tVFZWevxpxB2kPnXZe8zdlT/g6rwgKS1OvkrO61xp+Vu2KO0eZ+vzs5Xj\nZLqn2WrxVFq9XY2j6hg4uY2v1Js5Tue5IsfJ3fkDruwGs5Tj5GvnQVuuuM69pbtSqfc4S59fU1OT\nmDZtmtXgKT4+3mZwoNR6u5rS5nFimyAplq2Vws29PmrUKJetJO4prlgxPTY2FlqtFoWFhe1eS0xM\nRGxsrMPbNsfU7eIK6enpiI6OdtuK8r7MlcfJH1j6/AICArBkyRJs3LgRRUVFZn+3tLQUAJq7zAEg\nKyur+XWdToeSkhL069dP8d2XcgsJCUFCQoJy6u3pyM3X9qkEvlZvX5zHyRFyJ8v6SouduXmc/IE3\nnOuuOibeUHdzHJmWwNPdVUqhpBiCLU6keLaegv3lKVnuerqiJcuT/OU88AZ6vR4ZGRnIzs5GUVER\ntFpt87nlz8nPba+5mJiY5pamtkyDGxYvXtzcAgVYbpEi9+GoOiI/5egoLCJbXD1i01u1veb27NmD\n5ORks+9NTExE586dsW7dOrOvZ2dnQ6fTubC0ZAkDJyI/5y3Dvsk76HQ6ftnbYLrmIiMjrQ65r6mp\n8YtpV7wNHy1JUXQ6HcrKyhAbG8svciIvJGXuMXap/o+1LvPGxka3DuIgadjiRIqg1+uRnp6OAQMG\nICUlBQMGDEB6ejr0er2ni0ZEdjCN2DSHX/btWesyV/QkkH6MLU6kCKacCBMmQBJ5J9OXfcvr2YRf\n9pZZGtzQskWqbaI9eQZbnHyYTqfD4cOHFZ9TwJwIIt+SmZmJtLQ0JCcnQ6PRIDk5GWlpafyyd0DL\nFqm1a9dyEIcCMHDyQd7W7eXK9diIyP04YlN+ipsE0o/xLPZBWVlZWLVqVfP/ld7t5e5ZrInIPTi3\nlmtwEI1nscXJx+h0OmzevNnsa0rt9mICpH/ylq5kIqVQcm+CP13PDJx8TFlZGSoqKsy+puRuL+ZE\n+A8l3/yJlCwrK0txE4v64/XMwMnHxMbGIjo62uxrSu72Yk6E/+Cs0uQtlNSKotTeBH+8nhk4+ZiQ\nkBCMGTPG7Gve0O3FWax9G0dQkjdQYiuKEnsT/PV6ZuDkg9LT09ntRYrEEZTkDZTYiqLE3gR/vZ4Z\nOPkgdnuRUnFWaVI6pbaiKLE3wV+vZwZOPozdXqQ0HEFJSqfkVhSl9Sb46/XMJggicitri5oSeZqS\n55Uz9SbMmzdPMfM4+eP17FSL0w8//IDZs2fLVRYi8gPsSiYl84ZWFCX1Jvjj9exwzV588UVs3boV\n/fr1k7M8ROQnOKs0KZU/tqI4y5+uZ4cDp6FDh+Kaa67Bp59+Kmd5iIiIPEqJXWKkHCohhLD2hjVr\n1uDDDz9s9bN58+Zh4MCB+PXXX7Fq1SosXLjQ5o5ycnKcKykRERGRGw0bNqzdz2y2OE2aNAmTJk1y\nWQFcKScnx+37VAJ/rTfgv3X313oD/lt3f6034L9199d6A56pu6UGH05HQERERCQRAyciIiIiiZwa\nLzhy5EiMHDlSrrIQERERKRpbnIiIiIgkYuBEREREJBEDJyIiIiKJGDgRERERScTAiYiIiEgiBk5E\nREREEjFwIiIiIpKIgRMRERGRRAyciIiIiCRi4EREREQkEQMnIiIiIokYOBERERFJxMCJiIiISCIG\nTkREREQSMXAiIiIikoiBExEREZFEDJyIiIiIJGLgRERERCQRAyciIiIiiRg4EREREUnEwImIiIhI\nIgZORERERBIxcCIiIiKSiIETERGRn9HpdDh8+DB0Op2ni+J1GDgRERH5Cb1ej/T0dAwYMAApKSkY\nMGAA0tPTodfrPV00rxHg6QIQERGRe2RkZGDRokXN/y8sLGz+f1ZWlqeK5VXY4kREROQHdDod1q1b\nZ/a17OxsdttJxMCJiIjID5SVlaG4uNjsa8XFxSgrK3NzibwTAyciIiI/EBsbC61Wa/a1xMRExMbG\nurlE3smhwKm2thYPP/ww7rnnHtx55534/fff5S4XERERySgkJAQTJkww+9qECRMQEhLi5hJ5J4eS\nw99//32MGjUK9957L44cOYLZs2fjiy++kLtsREREJKPMzEwA53OaiouLkZiYiAkTJjT/nGxzKHC6\n9957ERQUBAAwGAzo0KGDrIUiIiIi+QUEBCArKwvz5s1DWVkZYmP/f3v3H1NV/cdx/IncC+qFWmou\n/qGEZansDtA515auBUOdWyuGcKeR/bxqTOsW4Sh/zHul6bqrqWglZq5/DPWP0jJrK2NLcnlvyWCi\nZUr5RwEWyo+4XLif7x+Osy8K9x4ucK/n8n78xf2cc8b7tQ+fe9/nnsO9KfJO0zCFbJwOHz7MwYMH\nB4xVVFRgtVppaWmhtLSU8vLyMStQCCGEEKNr8uTJpKenR7sMQ4pTSqlwDrxw4QIOh4M33niDRYsW\nhdzf4/GE82uEEEIIIaJi7ty5t42Fdanut99+Y/369bz33ns8/PDDIypgLHk8noj/zjvBeM0N4zf7\neM0N4zf7eM0N4zf7eM0N0ck+1Bs+YTVObrebnp4etm3bBkBSUhJ79+4NvzohhBBCCAMIq3GSJkkI\nIYQQ45F8AKYQQgghhE7SOAkhhBBC6CSNkxBCCCGETtI4CSGEEELoFPbnOA2XfI6TEEIIIYxksI9A\niFjjJIQQQghhdHKpTgghhBBCJ2mchBBCCCF0ksZJCCGEEEInaZyEEEIIIXSSxkkIIYQQQqewvqvu\nTvPNN9/w1Vdf4Xa7b9tWXV3NoUOHMJlMrFmzhscee4x//vmH119/ne7ubqZPn87bb7/NpEmTolB5\n+Lq7uyktLeXatWtYLBa2b9/OlClTtO01NTXs27cPAKUUHo+H48eP4/P5sNvtPPDAAwDYbDaWLl0a\njQhhC5UdYM2aNfz777+YzWYSExOpqqqiqamJDRs2EBcXx4MPPsjmzZuZMME45w56cm/fvh2v10tv\nby+FhYUsX76ctrY28vLymDlzJgA5OTk888wz0YgwLIFAgC1btnDhwgUSEhJwuVzcf//92vZYXdsQ\nOvvHH3/MF198AcCiRYsoKSlBKcXChQu1tZ2Zmclrr70WjfJHJFR2l8uF1+vFYrEAsGfPHvx+v+Hn\nPVju8+fPU1FRoe37yy+/UFlZidVqNeTaHsy5c+d45513+OSTTwaMf/vtt1RWVmIymcjPz2f58uW6\nngvHlDI4p9Op8vLy1CuvvHLbtubmZrVs2TLl8/nUjRs3tJ+dTqc6evSoUkqpDz74QB04cCDCVY/c\nRx99pHbu3KmUUur48ePK6XQOue++ffuU2+1WSilVXV2t9u/fH5Eax4qe7EuWLFGBQGDAmN1uVz/+\n+KNSSqmNGzeqr7/+euyLHUWhctfW1qq1a9cqpZTy+XwqJydHtbW1qR9++EFt3bo14vWO1MmTJ1VZ\nWZlSSqmff/5ZrV69WtsWy2tbqeDZ//jjD/Xkk0+q3t5eFQgEVGFhoTp//ry6cuWKstvt0Sp51ATL\nrpRSRUVF6tq1awPGYmHeQ+Xu9+WXXyqHw6GUUoZd27f68MMP1bJly1RBQcGA8Z6eHu15zOfzqaee\nekq1tLQM6/VvLBjndHsI2dnZbNmyZdBtdXV1ZGVlkZCQQHJyMqmpqTQ2NuLxeHj00UcBWLhwIadP\nn45gxaPj1gy1tbWD7vfXX3/x2WefUVJSAkB9fT2nTp1ixYoVlJeX09HREbGaR0uo7K2trdy4cYPV\nq1djs9n47rvvAGhoaGD+/PnacUab91C5s7KyBpyV9vX1YTKZqK+vp6GhgZUrV7Ju3Tqam5sjWne4\n/j9vZmYm9fX12rZYXtsQPPt9991HVVUV8fHxxMXF0dvbS2JiIg0NDfz99988/fTTvPjii/z+++/R\nKn9EgmUPBAI0NTWxadMmioqKOHLkyG3HGHXeg+Xu19XVxa5du3jzzTcBDLu2b5WamsquXbtuG790\n6RKpqancfffdJCQkMHfuXH766Sfdr39jxTCX6g4fPszBgwcHjFVUVLB06VLOnDkz6DEdHR0kJydr\njy0WCx0dHQPGLRYL7e3tY1f4KBgs+9SpU3VlOHDgAKtWrSIhIQEAq9VKQUEBGRkZ7N27l8rKSsrK\nysY2wAiEk93v9/Pcc89RXFzM9evXsdlsWK1WlFLExcUNedydJJzciYmJJCYm4vf72bBhA4WFhVgs\nFtLS0sjIyOCRRx7h888/x+VysXPnzohlCVdHRwdJSUna4/j4eHp7ezGZTDGztocSLLvZbGbKlCko\npdixYwezZ89mxowZtLa28tJLL7FkyRLOnj1LaWkpR48ejWKK8ATL3tXVxcqVK3n22Wfp6+ujuLiY\njIyMmJj3YLn7HTlyhMWLF2uXpYy6tm+Vl5fH1atXbxu/U9e5YRqngoICCgoKhnVMUlISnZ2d2uPO\nzk6Sk5O18YkTJ9LZ2cldd9012uWOqsGyl5SUaNmGyhAIBDh16hSvvvqqNpabm6vtm5ubi9PpHMPK\nRy6c7NOmTaOoqAiTycTUqVOZNWsWly9fHnA/050+7+HO+fXr11m3bh3z58/HbrcDsGDBAu1+j9zc\nXMM8sd66fgOBgPYiEitreyjBsgP4fD7Ky8uxWCxs3rwZgIyMDOLj4wGYN28ezc3NA04WjCJY9kmT\nJlFcXKz9PS9YsIDGxsaYmPdQcw5w7NixAevXqGtbr1DrvH8s0vNt+Et1wVitVjweDz6fj/b2di5d\nusTMmTPJzs7m+++/B27eRD3Yd9Hc6fRkuHjxIjNmzGDixIna2PPPP09dXR0AtbW1zJkzJzIFj6JQ\n2U+fPs369euBm4vq119/JS0tjdmzZ2vvTtbU1DBv3rzIFj5CoXJ3d3ezatUq8vPzefnll7Xxt956\ni5MnTwLGmvPs7GxqamqAmzfD9t8AC7G9tiF4dqUUa9eu5aGHHmLr1q1as7R7927tXcrGxkZSUlIM\n1zRB8OxXrlzBZrPR19eH3+/H6/UyZ86cmJj3YLkB2tvb6enpISUlRRsz6trWKz09naamJtra2ujp\n6eHs2bNkZWVFfb5j4rvqzpw5w6FDh3j33XeBm5enUlNTefzxx6murubTTz9FKYXdbicvL4/W1lbK\nysro7Ozknnvuwe12M3ny5CinGJ7//vuPsrIyWlpaMJvNuN1u7r33Xnbs2MHixYuxWq2cOHECr9er\nXQ+Hm/f5OJ1OzGYz06ZNw+l0Dnh72Aj0ZN+2bRvnzp1jwoQJvPDCC+Tk5HD58mU2btyI3+8nLS0N\nl8ulvegYQajcXq+X3bt3M2vWLO2Y/nueysvLgZtn7C6Xi+nTp0clw3D0/5fRxYsXUUpRUVFBTU1N\nzK9tCJ49EAjgcDjIzMzU9nc4HKSlpVFaWkpXVxfx8fFs2rSJ9PT0KKYIT6h5r6qq4sSJE5jNZp54\n4glsNltMzHuo3HV1dbz//vvs2bNHO+bPP/805NoezNWrV3E4HFRXV3Ps2DG6urooLCzU/qtOKUV+\nfj4rVqwY8rkwUmKicRJCCCGEiISYvlQnhBBCCDGapHESQgghhNBJGichhBBCCJ2kcRJCCCGE0Eka\nJyGEEEIInaRxEkIIIYTQSRonIYQQQgidpHESQgghhNDpf1LX6hCKAgcwAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Define input output pairs\n", "n = 200\n", "\n", "## Input variable is just a grid\n", "x = np.expand_dims(np.linspace(-1, 1, n, dtype=np.float32), 1)\n", "\n", "## Output variable is a second-order polynomial with noise equal to gradient\n", "polynomial = 2*x**2 + -0.7*x\n", "d_polynomial = 4*x- 0.7\n", "\n", "noise = np.random.normal(0, .3, (n, 1)).astype(np.float32)\n", "y = polynomial + d_polynomial*noise\n", "\n", "plt.figure(figsize=(10, 5))\n", "\n", "plt.plot(x, polynomial, label=\"Optimal fit\")\n", "plt.scatter(x, y, c=\"k\", label=\"Data points\")\n", "\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Network\n", "\n", "The network is a perceptron with a single hidden layer containing $5$ neurons. The weights and biases have been initialised as using the following:\n", "\n", "$w \\sim \\mathcal{N}(0, 1)$\n", "\n", "$b = 1$.\n", "\n", "Here we choose a standard sigmoid activation function." ] }, { "cell_type": "code", "execution_count": 95, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Define network\n", "(n_input, n_hidden, n_output) = (1, 5, 1)\n", "\n", "## Weights\n", "w1 = np.random.randn(n_input, n_hidden)\n", "w2 = np.random.randn(n_hidden, n_output)\n", "\n", "## Biases\n", "b1 = np.ones(n_hidden)\n", "b2 = np.ones(n_output)\n", "\n", "# Define activation function\n", "def sigmoid(z): return 1/(1 + np.exp(-z))\n", "def d_sigmoid(z): return sigmoid(z) * (1 - sigmoid(z))\n", "\n", "activation = sigmoid\n", "d_activation = d_sigmoid" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Forward propagation\n", "\n", "The forward pass consists of sending a given $x$ through the network such that it returns the prediction for that specific input. \n", "\n", "1. $z_1 = x w_1 + b_1$\n", "\n", "2. $a_1 = \\sigma(z_1)$\n", "\n", "3. $\\widehat{y} = a_1w_2 + b_2$" ] }, { "cell_type": "code", "execution_count": 96, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def forward(x):\n", " # First hidden layer\n", " z1 = x @ w1 + b1\n", "\n", " # Sigmoid activation\n", " a1 = activation(z1)\n", "\n", " # Second hidden layer\n", " z2 = a1 @ w2 + b2\n", "\n", " # No activation\n", " y_pred = z2\n", " \n", " # Return [prediction, preactivations, postactivations]\n", " return y_pred, (z1, z2), (a1, z2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Backpropagation\n", "\n", "We are then interested in how much that prediction deviates from the target. For this we use the squared error function.\n", "\n", "$$E = \\frac{1}{2}\\sum_{i}^{N}(\\widehat{y}_i - y_i)^2$$\n", "\n", "In order to correct the network so that it outputs the correct value for all inputs we need to change the weights. As the whole network is differentiable we can formulate this as an optimization problem. In essence we use the gradient of the loss with respect to each of the weights to estimate how much each of the weights are to 'blame' for the wrong prediction.\n", "\n", "$$\\Delta w = \\frac{\\partial E}{\\partial w_{ij}}$$\n", "\n", "But from the above function it is not immediately clear how we differentiate through a network containing many complex operations. The chain rule however will help us split this partial derivative into a more manageable problem. The amount of applications of the chain rule that we need depends on the depth of which the weight is located at. For the output weights we have.\n", "\n", "\\begin{align}\n", "\\Delta b_2 &= \\frac{\\partial E}{\\partial b_{2}} = \\frac{\\partial E}{\\widehat{y}}\\frac{\\partial \\widehat{y}}{\\partial b_2}\\\\\n", "\\Delta w_2 &= \\frac{\\partial E}{\\partial w_{2}} = \\frac{\\partial E}{\\widehat{y}}\\frac{\\partial \\widehat{y}}{\\partial w_2}\n", "\\end{align}\n", "\n", "and for the input weights, we must go even deeper.\n", "\n", "\\begin{align}\n", "\\Delta b_1 &= \\frac{\\partial E}{\\partial b_{1}} = \\frac{\\partial E}{\\widehat{y}} \\frac{\\partial \\widehat{y}}{\\partial a_1} \\frac{\\partial a_1}{\\partial z_1} \\frac{\\partial z_1}{\\partial b_1}\\\\\n", "\\Delta w_1 &= \\frac{\\partial E}{\\partial w_{1}} = \\frac{\\partial E}{\\widehat{y}} \\frac{\\partial \\widehat{y}}{\\partial a_1} \\frac{\\partial a_1}{\\partial z_1} \\frac{\\partial z_1}{\\partial w_1}\\\\\n", "\\end{align}\n", "\n", "From the four equations it become immediately clear the some computation is shared. In fact for each layer we share unit of computation between the weights and biases. Additionally, deeper gradients are dependent on these previous computations as well. We refer to this shared information as the deltas of each layer and clever bookkeeping allow us to simplify a lot.\n", "\n", "\\begin{align}\n", "\\delta_1 &= \\frac{\\partial E}{\\partial \\widehat{y}} = (\\widehat{y} - y)\\\\\n", "\\delta_2 &= \\frac{\\partial E}{\\partial \\widehat{y}} \\frac{\\partial \\widehat{y}}{\\partial a_1} \\frac{\\partial a_1}{\\partial z_1} = \\frac{\\partial E}{\\partial \\widehat{y}} \\frac{\\partial (a_1w_2 + b_2)}{\\partial a_1} \\frac{\\partial \\sigma(z_1)}{\\partial z_1} = \\delta_1 w_2 \\sigma'(z_1)\n", "\\end{align}\n", "\n", "Now returning to the computation of the gradients, we find that they are simple to calculate.\n", "\n", "\n", "\\begin{align}\n", "\\Delta b_2 &= \\frac{\\partial E}{\\partial b_{2}} = \\delta_1 \\frac{\\partial (a_1w_2 + b_2)}{\\partial b_2} = \\delta_1\\\\\n", "\\Delta w_2 &= \\frac{\\partial E}{\\partial w_{2}} = \\delta_1 \\frac{\\partial (a_1w_2 + b_2)}{\\partial w_2} = \\delta_1 a_1\\\\\n", "\\end{align}\n", "\n", "Again for the input layer.\n", "\n", "\\begin{align}\n", "\\Delta b_1 &= \\frac{\\partial E}{\\partial b_{1}} = \\delta_2 \\frac{\\partial (x w_1 + b_1)}{\\partial b_1} = \\delta_2\\\\\n", "\\Delta w_1 &= \\frac{\\partial E}{\\partial w_{1}} = \\delta_2 \\frac{\\partial (x w_1 + b_1)}{\\partial w_1} = \\delta_2 x\\\\\n", "\\end{align}\n", "\n", "The code below is the entirety of the backpropagation pass for the network presented in this notebook. " ] }, { "cell_type": "code", "execution_count": 97, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def backward(y_pred, z1, a1):\n", " # Weights between hidden and output\n", " delta_1 = (y_pred - y)\n", " w2_delta = a1.T @ delta_1\n", " b2_delta = np.sum(delta_1, axis=0)\n", "\n", " # Weights between input and hidden\n", " delta_2 = delta_1 @ w2.T * d_activation(z1)\n", " w1_delta = x.T @ delta_2\n", " b1_delta = np.sum(delta_2, axis=0)\n", " \n", " # Return deltas for weights, biases\n", " return (w1_delta, w2_delta), (b1_delta, b2_delta)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Putting it all together\n", "\n", "Now that we have defined all the things we need, we can train the network for a couple of thousands epoch (runs through the whole dataset) and asses both the final loss and the goodness of fit.\n", "\n", "We simply run a forward pass to get $\\widehat{y}$ and then a backward pass to get the gradients for each of the weights. Finally we update the weights using basic gradient (steepest) descent." ] }, { "cell_type": "code", "execution_count": 98, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loss after 0 epochs:\t 287.5608797629085\n", "Loss after 1000 epochs:\t 69.75663827357482\n", "Loss after 2000 epochs:\t 67.23538645856755\n", "Loss after 3000 epochs:\t 66.07565455439564\n", "Loss after 4000 epochs:\t 65.3200997342123\n", "Loss after 5000 epochs:\t 64.810995975468\n" ] } ], "source": [ "## Train network\n", "learning_rate = 1e-3\n", "n_epochs = 5001\n", "\n", "for epoch in range(n_epochs):\n", " ### Forward pass of x ###\n", " y_pred, (z1, _), (a1, _) = forward(x)\n", " \n", " # Squared error loss term.\n", " loss = 1/2 * np.sum((y_pred - y)**2)\n", "\n", " if epoch % 1000 == 0:\n", " print(\"Loss after {} epochs:\\t {}\".format(epoch, loss))\n", "\n", " ### Backward pass of y_pred ###\n", " (w1_delta, w2_delta), (b1_delta, b2_delta) = backward(y_pred, z1, a1)\n", "\n", " # Gradient descent\n", " w1 -= learning_rate * w1_delta\n", " w2 -= learning_rate * w2_delta\n", "\n", " b1 -= learning_rate * b1_delta\n", " b2 -= learning_rate * b2_delta" ] }, { "cell_type": "code", "execution_count": 99, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAEuCAYAAAB4RZ0yAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XlcVOX+wPHPzIAigoAiiwKO4oamuOFWuWSrWWZl2WLd\nvLduuWFJ15tp1zbLpEQrtdvyy2wxK5O82era4oq5IWmg7Mgm+7ANc35/GMQywADDMDN83758vZRZ\nznNmzjl8z/N8n++jUhRFQQghhBBCNErd1g0QQgghhLAVEjgJIYQQQphIAichhBBCCBNJ4CSEEEII\nYSIJnIQQQgghTCSBkxBCCCGEiRwstaGoqChLbUoIIYQQosVGjhxZ52cWC5zqa0BrioqKsvg2rUF7\n3W9ov/veXvcb2u++t9f9hva77+11v6Ft9r2+Dh8ZqhNCCCGEMJEETkIIIYQQJpLASQghhBDCRBI4\nCSGEEEKYSAInIYQQQggTSeAkhBBCCGEiCZyEEEIIIUwkgZMQQgghqiQlJbFgwQJmz57NrFmzWLFi\nBYWFhQ2+5tNPP6W8vJyYmBjeeOONFm3/yiuvrPOz7777jptuuokvv/ySFStWAHDkyBF+//33Fm2r\nOSRwaoBOpyMuLg6dTtfWTRFCCCFaXUlJCXPnzuUf//gHmzdvZsuWLQQHB7N48eIGX/fWW29hMBgI\nCgpi/vz5Zm/XkSNHiIiIYMaMGVWB0xdffEFGRobZt9UYi1YOtxV6vZ6wsDAiIyNJTEwkICCA6dOn\nEx4ejoODfGRCCCHs0969ewkJCSE4OLjqZzNmzOCTTz4hKSmJN998E0VRSEtLQ6fTsWrVKo4dO0Zm\nZiaPP/44Dz74IFu2bGHNmjVcd911DB8+nPj4eMaNG0dBQQEnT56kd+/erF69mnPnzvHyyy9TUVFB\nTk4OK1asYMSIEXXa9OmnnxIfH8/TTz/NmjVrWLx4Mc888ww//fQT0dHR9O3blx49eljsM5IowIiw\nsDDWrl1b9f/4+Piq/0dERLRVs4QQQrQj7+2I5pcTKVX/Ly0ro+M337foPa8M7smcWwbX+3hSUhIB\nAQF1fu7n50dqaioA/v7+rFq1in379rF69Wo2btzIhg0bWLNmDcePH696TUpKCps2baJ79+6MHj2a\nzz77jOXLlzNlyhTy8/OJjY1lyZIlDBgwgB07drBt2zajgdPdd9/NJ598wqpVq1CpVABcccUVXH31\n1UydOtWiQRPIUF0dOp2O7du3G30sMjJShu2EEELYLW9vb5KTk+v8PCEhoSpAGTt2LADDhw/nwoUL\n9b6Xu7s7PXr0wNHREWdnZ/r27YtKpcLV1ZXS0lK8vLxYv349S5Ys4bvvvkOv17fOTpmZ9DjVkpaW\nRlJSktHHkpKSSEtLIzAw0MKtEkII0d7MuWVwjd4hSyx0O2XKFDZu3MjJkycZOnQoAJ999hkeHh74\n+/sDEB0dzahRozh27Bj9+vUDQKVSYTAYarxXZe9QfV588UXCw8MJDAxk3bp1pKSkNPj82lQqFYqi\nNOk15iCBUy2+vr74+fmRmJhY5zF/f398fX3boFVCCCFE6+vcuTMbN25k5cqV5ObmUlFRwYABA3jt\ntdeqnrN//3527dqFwWDgpZdeAmDUqFE88sgjzJs3z+Rt3XrrrYSGhtKlSxd8fHzIyclpUluDg4MJ\nDw/Hz8/Poh0aEjhVo9frWbp0ab1f3vTp03F2drZwq4QQQgjLCQgIYOPGjfU+/uCDDzJhwoQaP1u1\nalXVvyuH8n755Zeqn1X/d2RkJAAPPfQQDz30UJ33r/7cSsuXL68KjrZu3QrArFmzmDVrVqP7Y24S\nOFVTOym8UpcuXXjooYcIDw9vg1YJIYQQwlpI4PSnhpLC3d3dWblypZQiEEII0a69/PLLbd2ENiez\n6v7UUFJ4SkoKaWlpFm6REEIIIayNBE5/8vX1NVq7AiQpXAghhBCXSeD0J2dnZ6ZPn270MUkKF0II\nIQRIjlMNlcnfkZGRJCUl4e/vX7XUihBCCCGEBE7VODg4EBERwcqVK0lLS8PX11d6moQQQrQrSUlJ\nrF69mosXL+Lk5ISTkxNPPvlkVbFLS/jhhx8YOnQo3t7ejT53//797Ny5s07i+hNPPEFCQgJ33nkn\narWau+++m08//ZTbb78dR0fHZrdNAicjnJ2dpTq4EEKIdqe4uJjHHnuM559/nuHDhwNw8uRJnnvu\nOTZv3myxdnzwwQesWLHCpMCpPr/++isHDx6s8bO33nqL2267rUVtk8BJCCGEEADs2bOHsWPHVgVN\nAEOHDuWDDz4ALs9AX758OaWlpXTs2JHnn3+eiooKFi9ejI+PD0lJSQwZMoRnn32WgoICnn766aqi\n0suWLWPAgAFMnjyZPn36EBgYyJ133snLL79MRUUFOTk5rFixgvz8fGJiYliyZAkff/wxn376KVu2\nbMHV1ZWpU6fywAMPEBcXx9KlS+nUqROdOnXCzc2txn6sWLGCwsJCHnvsMa677jrOnz9Pr169yMzM\n5PHHH2f9+vXN/owkcBJCCCGsUNyTcWR8llH1/7KyMg50ONCi9/Sa6UXg6vpHVJKTk2vMMH/sscco\nLCwkIyODTZs2sWrVKmbPns3EiRM5cOAA4eHhPP7448THx/Puu+/SqVMnrr32WjIzM3n//fcZO3Ys\n9957L/Hx8Tz11FN88sknpKWlsW3bNjw8PNi5cydLlixhwIAB7Nixg23btvHCCy8QFBTEihUrSExM\nZOfOnaxYsYIRI0bw0EMPcdVVV/HKK6+wcOFCrrzySv773/9y/vz5GvuxYsUKfvjhBzZs2MC2bdsA\nmDlzJhs2bGDNmjUt+gwlcBJCCCEEAD4+Ppw+fbrq/xs2bADgrrvuQq/Xc+7cOd566y3eeecdFEWp\nKgwdEBCAi4sLAN27d6e0tJRz585x8OBBvvnmGwDy8vIA8PDwwMPDAwAvLy/Wr1+Pk5MTRUVFVe9R\n6dy5c6SmpvLiiy/i6upKXl4eCQkJxMfHVy1CPGLEiDqBU2uSwEkIIYSwQoGrA2v0DkVFRTFy5MhW\n3eaUKVN4++23OX78OMOGDQMgISGBixcvolKp6NOnD3PmzGHEiBHExcVx5MgRAFQqVZ336tOnD7fe\neiu33HIL2dnZfPbZZwCo1X9VQnrxxRcJDw8nMDCQdevWkZKSUvV+iqLQp08f+vbty2OPPcaoUaN4\n//33GTBgAIGBgfz2229MmDChRqDXGJVKhcFgaPbnAxI4CSGEEOJPnTt3ZsOGDbz66quEh4ej1+vR\naDQ89dRT9OzZkyVLlrBixQpKS0spKSnh6aefrve9Hn30UZ5++mm2bt1KYWEh8+fPr/OcW2+9ldDQ\nULp06YKPj09VPtTw4cP517/+xXvvvce4ceN49tlncXR0rJpp9+9//5slS5bw7rvv0rVrVzp27GjS\n/o0aNYpHHnmEDz74wGiwZwqVoihKs17ZRJaIlFtzmzqdzmZKFLTFZ20t2uu+t9f9hva77+11v6H9\n7nt73W+wrhhCKoc3Qq/Xs2jRIgYPHkz//v0ZPHgwixYtQq/Xt3XThBBCCGFhMlTXiLCwMNauXVv1\n//j4+Kr/R0REtFWzhBBCCNEGpMepATqdju3btxt9LDIyEp1OZ+EWCSGEEKIttShwys7OZuLEicTF\nxZmrPVYlLS2NpKQko48lJSWRlpZm4RYJIYQQoi01O3AqLy/nmWeewcnJyZztsSq+vr41CoFV5+/v\nj6+vr4VbJIQQQoi21OzAadWqVcyaNQsvLy9ztseqODs7M336dKOPTZ8+3epn1wkhhBDCvJpVjmDb\ntm1cvHiRuXPnMnv2bFasWNHoorhRUVHNbmRb0uv1REREsH//fi5evIiPjw8TJkxg0aJFVRVThRBC\niLZSUlJCVlYWnp6eLR4FOnPmDOvWraNnz54oikJFRQU33XQTY8eOrfc1WVlZJCQkmL1cQHx8PMeO\nHeP22283+nhZWRm//PILkydPNut2qzO6T0oz3Hvvvcp9992n3H///crIkSOVO+64Q8nIyGjwNUeP\nHm3OplrEnNssKipSYmNjlaKiIrO9Z2tpi8/aWrTXfW+v+60o7Xff2+t+K0r73ffa+11eXq6EhoYq\nWq1WUavVilarVUJDQ5Xy8vJmb+PgwYPKokWLqv5fWFiozJgxQzlz5ky9r/niiy+U1atXN3ubpjD2\nnSclJSkzZ8606DYVRVGa1WXy0UcfVf27ssepe/fuzY7obIGzs3OjvWptqXqBTiGEEPbPEuVyOnfu\nzN133823335L//79eeaZZ7h48SIZGRlcc801LFy4kP/+97+UlJQwfPhwXF1deeONN1AUhaKiIl59\n9VV69+5d9X7btm3jxx9/pKioiJycHObNm8cNN9zAL7/8QkREBB07dsTd3Z2VK1cSExPDli1bWLNm\nDY8//jjjx4/nwoULdOvWjddff52NGzcSGxvLG2+8wbhx41i1ahUODg506tSJtWvX1ln3zlykHIGN\nM1ags7JMvhBCCPtkyXI53bp1Iycnh7S0NIYNG8a7777L559/zpYtW9BoNDzyyCNMmzaNKVOm8Mcf\nf7B69Wo2b97M9ddfz7ffflvn/YqLi/m///s/3nvvPV5++WXKy8tZvnw5b7zxBh9++CEhISFViwtX\nysjIIDQ0lE8//ZRLly5x6tQpHn30Ufr27cv8+fP58ccfuemmm/jwww+55557yM/PN9v+19biJJ3N\nmzebox2imYzdccTHx+Pt7S0FOoUQwk6ZUi7HXKMkqamp+Pj44O7uzqlTpzh48CAuLi6UlZXVea63\ntzcvvvgizs7OpKenM2LEiDrPCQkJQa1W4+npSZcuXcjKysLFxQVvb++qx1977TUmTZpU9RpXV9eq\nERVfX19KS0trvOejjz7Kxo0befDBB/H29mbo0KFm2XdjpMfJhkmBTiGEaJ8sVS6nsLCQzz77jBtv\nvJFt27bh6urKq6++ypw5cygpKUFRFNRqNQaDAYDly5ezcuVKXn75Zby8vFCMzD+Ljo4GLieVFxYW\n4uXlRWFhIRkZGQAcPnwYrVZb4zXGFuStvt2vvvqKGTNmsHnzZvr168fWrVvNsv/GyLQwG2bJOw4h\nhBDWo7JcTvURh0otLZdz8OBBZs+ejVqtpqKiggULFtCnTx8qKipYvHgxx48fp0OHDvTq1YuMjAz6\n9+/Phg0bGDx4MLfeeiv33XcfnTp1wtPTsyoYqi4rK4sHH3yQgoIC/vOf/6DRaHjhhRdYsGABKpUK\nNzc3XnrpJf74448G29mtWzfKy8tZvXo1119/PcuWLaNTp06o1Wqee+65Zu9/YyRwsmGVdxzx8fF1\nHpMCnUIIYd/Cw8OByyMMSUlJ+Pv7M3369KqfN8eYMWM4cOCA0cf69evHV199Vefn3t7efPfddwDc\nfPPNjW4jJCSEsLCwGj8bP34848ePr9OWMWPGANTIeVqzZk3VvyMjI6v+3Zq9TNXZ7VCdTqcjOTnZ\nroerpECnEEK0Xw4ODkRERBAdHc3Zs2eJjo4mIiJCagy2Mrv7dPV6PWFhYURGRpKYmEhAQEBVBG6P\nB5OxO46xY8e26I5DCCGE7bD2cjnV1VfM0pbYXSRhiboWlarXTmqr3p3KO46VK1dWtSUmJsYug0Qh\nhBCirdnVUJ2lZpkZq520aNGiNq2dVHnHIcNzQgghROuxq24JS80ys2SvlhBCCCGsh131OFmiroXU\nThJCCCGaR6fTERcXZ9O/K+0qcLLELDNTerWEEEII8RdrTHFpLrsaqoOas8xqz6ozB6mdJIQQQjSN\nPaW42FWPE9Ssa7Ft2zaz17WQ2klCCCGE6ewtxcXuepwqOTs74+fn1yqBTGtUaxVCCCHskb0tD2a3\ngVNrMlY7SXqahBBCiLrsLcXF7obqLElqJwkhhBANs7cUF+lxEkIIIUSrsqcUFwmchBBCCNGq7CnF\nRQInIYQQQliELS1IXB/JcRJG2UN1VyGEEMLcJHASNdhTdVchhBDC3GSoTtRgT9VdhRBCCHOTHidR\nxd6quwohhBDmJoGTqCILGAshhLAUW82llcBJVKms7mqMLVZ3FUIIYX1sPZdWAidRxd6quwohhLA+\nlbm08fHxGAyGqlzasLCwtm6aSSRwEjWEh4cTGhqKVqtFo9Gg1WoJDQ21yequQgghrIs95NLKrDpR\ngz1VdxVCCGFdTMmltfYCmdLj9CdbTVJrLbKAsRBCCHOzh1zadh842XqSmhBCCGEr7CGXttlDdRUV\nFSxbtowLFy6gUql49tln6d+/vznbZhFS8FEIIYSwnMqc2cjISJKSkvD392f69Ok2k0vb7B6nPXv2\nALBlyxYWLVrEmjVrzNYoS7GHJDUhhBDCllTm0kZHR3P27Fmio6OJiIjAwcE20q6b3cprr72WSZMm\nAZCamkqXLl3M1SaLsYckNSGEEMIWVebS2poW5Tg5ODiwZMkSnn/+eW655RZztcli7CFJTQghhBCW\no1IURWnpm2RmZnLXXXfx9ddf15vYFRUV1dLNtIrw8HC2bNlS5+ezZs2ymWJcQgghhD0pKSkhKysL\nT09PnJyc2qwdI0eOrPOzZg/Vbd++nfT0dP75z3/SqVMnVCoVanXDHVjGGtCaoqKiGt3m5s2b8fb2\nNpqkZq7xVp1OZ9GaSKbst71qr/veXvcb2u++t9f9hva77+1hv/V6PWFhYURGRpKYmEhAQADTp0/n\nnnvuYcyYMRZtS30dPs2ODK6//nqeeuop7rvvPvR6PUuXLm3TqLC5WrPgY30HgDmDMiGEEMJe1DfT\nPT09nU8++aQNW/aXZv/2dnZ2rrFzts5cSWrVe5eWLl0qpQ6EEEIIEzQ0033//v3odDqrqPPU7gtg\nmkvtQppBQUG89957Rp8rpQ6EEEKImhqa6X7x4kXS0tIs3CLjJHAyk9qrPScmJlJQUGD0uZWlDoQQ\nQghxWUMz3X18fKxmprsETmbQUPeiMVLqQAghhKipoeVYJkyYYBXDdNCCHCfxl4a6F42xlfV4bI1O\npyM5OZmgoCD5fIUQwgbVtxzLPffc08Yt+4v0OJlBQ92Lrq6u9OrVC41Gg1arJTQ01GbW47EV1fPL\nbr/9dlmoWQghbJQtLMcigZMZNNS9OGfOHM6cOWO1B4A9qJ1fVjl7UQqYCiGEbaqc6W6Nowd2GTil\nX9Lxn7cPcCap2GLbDA8PJzQ0FK1WW6d3yZoPAFsnCzULIYSwJLvs+igt03M6NotjvxvILjnGI7cN\nwdnJsVW32ZqFNEX9ZKFmIYQQlmSXPU4BPl2IeGISvl0d2XUkiQXhe4g+n22RbUvvkmXJQs1CCCEs\nyS4DJwB/b1f+fp0Xd13bn6zcYp5a/zPv/y+acn1FWzdNmFFD+WUye1EIIYS52eVQXSUHjYrZNwUx\naqA3r30SxRd7Yjl2NoPF942kl0+Xtm6eMJPq01drrwkohBBCmJPd9jhVF9S7K2ufmMT1Y3pxITWf\nx9fsI3J/HAaD0tZNswidTkdcXJzdJkpXn766bds2mb0ohBCi1bSLwAnA2cmRBXcN4+mHRuPs5MA7\nkad55r+/kpVruZl3llZ7/Tx7r2/k7OyMn5+fDM8JIYRoNe0mcKo09gpfXg+bTMggb078kcX88D3s\niUpCUeyv90nqGwkhhBDm1e4CJwAPVyeWzxnDvDuDqagw8NrHx3hp0xFyCkraumlmI/WNhBBCCPOz\n28CpJLmkwV4klUrFjeO0vB42mcF9unHgVBrzXtnDT8dTLNjK1mNKfSMhhBDCVPaeL2squwyccvfn\nctD/ILoHdVz67lKDAZRPt86sfOxKHr7tCkrLK3hl81FWfXCEvMJSC7bY/KS+kRBCCHNob/myjbHL\nwMl1pCtes7wwnDFw8saTHJ90nNyfc+t9vlqt4tarA1m3eBJB2q78fCKV+av3cOCU7fbKSH0jIYQQ\n5iD5sjXZZeCk6axh0CeDcP7ImW7TupG3P4/jVx/n5NSTFBwrqPd1Pbu78NK8q3ho2mCKSspZ+f5h\nXv0oigJdmQVbbz4NrZ8nhBBCNEbyZeuyy8CpkmaAhiE7hjD81+G4T3Ln0jeXiBoZRfTMaIpiioy/\nRq3i9sl9WfvEJPr5u7P3WDLzV+/myJmLFm59y1Wvb3T27Nk2rW8kY+NCCGF7JF+2LrsOnCq5jXMj\neHcwQ38YiutoVzI/z+TIFUeI+VsMxReM13Hy93Zl9YKreWBqEPlFZTz37iHWfHKM/CLb631qy/Xz\nZGxcCCFsl+TL1tUuAie4PIuu67VdGXFwBFdEXkHnQZ1J35TO4QGHOTfvHKVpdZPBNRo1M6f0Z83j\nk+jr58buo0nMe2U3P59Iscu6T61BxsaFEMJ2Sb5sXe0mcKqkUqnwvNWTUSdGEfRxEE69nEhdn8qh\nPoeI+1cc5ZfK67xG69uF8IUTeGjaIHQl5az64Cgr3z/MpXz7qfvUGmRsXAghbF9T82XtPTWj3QVO\nlVRqFd73eBNyJoT+b/fH0dORpNVJHAo8ROKqRCp0FTWer9GouX1yP9b9Wffp4OmLzF21ix8OJUjv\nUz1kbFwIIWyfqfmy7SU1o90GTpVKyksonlzMkBNDCAwPBDWc//d5DvU7ROrbqRj0hhrP79ndhZWP\nXcncO4ZiUGDd1uMsXf8zh47F2G103VwyNi6EEPajsXzZ9pKa0W4Dp9qR8ZCRQ3g16VVGnR1FwNIA\n9Dl6zj1yjiNXHCFzW2aNXiW1WsVN43uz7okJOCvZnD5/iRXvn2TCrXMJtcPourlkbFwIIdqH9pSa\n0W4Dp/oi43+/8G/6vNiHMbFj8P2nL8WxxUTfEc2xccfI3VeziObLLyxn65q/c2znaxj0ZfgGzyAq\nowcLwv7TRntlfaSWlBBC2L/2lJrRLgMnUyLjjj06MmDjAEZHj6b7nd0pOFTA8UmXi2gWniis8R6p\nv+9n7/vzSfl9Px6+A0hSjeTdyBOUllcY3UZ7Yk21pIQQoj2yRLJ2S1IzbC2ZvF0GTk2JjJ0HODP4\ns8GMODQC98mXi2geHX6U0/eepjzxrxl4ZcX5/LbzNQ5/+QIlhZfYvj+e+at3c+z3jFbfH1vQlrWk\nhBCiPTKWrD1v3jzOnj1r9iClOakZtppM3i4Dp+ZExl1GdyF4VzBDvx1K56Gd0UXq2KRsYj7zccOt\n6nkZF45yfs9qbrmyFxk5xfzn7QOs/vAoOQVSukAIIYTlGEtJWb9+PQMHDmyVIKWpqRm2mkzerMCp\nvLycJ598knvvvZc777yTXbt2mbtdraq5ScsqlYquN3Rl1LFRBH0URGmXUu7gDj7iIx7gAZxwAuDW\naTfxyO3DWLNoIv0D3Nn/WwqPrdrNtwfiMRjMV7rA1ro3hRBCWEZDKSlAqwQpTUnNsOVk8mYFTl99\n9RXu7u58/PHHvPPOOzz//PPmblera0nSskqtwvteb65Lu46TE09Soa7gIR5ii3oLEZMieGXlKwD0\n6enGKwsm8OiMIRgMCm9+foJ/v/kzCWn5LWp7W3VvSqAmhBC2oaGUlOpaI0gxJTXDlpPJmxU43Xjj\njYSGhgKgKAoajcasjbIEcyQtd3DuwMK9C5mcOpkuC7rg4exB8N5gjg09RvqWdBSDgkat4uar+rBh\nyTVcObQHMfGXCH1tLx/sPENJWfMCHUt3b9rqOLQQQrRXDaWkVNdWQYot1/lrVuDUuXNnXFxcKCws\nZOHChSxatMjc7bIYcyQtu3q7MmLdCMbGjaXn/J6UJpYSc08MUSFRXPrxEgDd3Drx7wdDWP73MXR1\nc+KzXX8wb/UeDp1Oa1Ll8bbo3rTVcWghhGivGkpJqa6tghRbrvOnUpq5XkhaWhrz5s2rynNqTFRU\nVHM2Y5MMyQZKN5ai//Zyj4xmtIaOCzqiCbrcM1emN7D3VD4Hfy/EoEC/Hk7cNNKdrq6N93YlJydz\n++23YzAY6jymVqvZtm0bfn5+ZtuXkpISZs6cafSOpEePHmzduhUnJyezbU8IIYR56PV6IiIi2L9/\nP6mpqUafM2vWrDa7Ca7evosXL+Lj48OECRNYtGiR1ZSsGTlyZN0fKs2QmZmp3Hjjjcqvv/5q8muO\nHj3anE21SFtss7r8Y/nK8euPK3vYo+xhjxI9K1rRxeqqHk+8mK8sXf+zMu2J7cqMf32lfPhNjFJS\npm/wPYuKihStVqsAdf5qtVqlqKjIrPsdGxurqNVqo9vTaDRKbGys2bZlDm39nbeV9rrfitJ+9729\n7reitN99b+p+FxUVKbGxsUpmZqby+++/K3PnzlW0Wq2i0WgUrVarhIaGKuXl5a3U2qa3s6ioqN7n\nWFMM0ayhuo0bN5Kfn8/69euZPXs2s2fPpqREptvX5jrcleDvggn+MRiXkS5kbMng8MDDnJt/jrL0\nMvy9XXnh0fH86/5RuDp3YMsPZ5n3ym4OR1+s9z0t3b1py+PQQgjRHtXOSw0JCWHDhg2sXbvWKosR\n21qdv2Z9YsuWLWPZsmXmbovd8pjiwcjDI8n8PJPzS8+T+mYqF9+/iH+YP/6L/bl6eE9GBnnx6Q/n\niNwfx/PvHWJUkDeP3DYEX8/Odd6vcuZfZGQkSUlJ+Pv7M3369FZZxqQyUFu7dm2dx0wJ1HQ6HWlp\nafj6+trMSSGEELasMi+1UmVeKkBERASBgYFt1TS70C4LYLYFlVqF111ejI4ZTb83+6Fx0ZDwbAKH\nAg+R/HoyTmoND90ymHWLJzG0rydHY9KZt3o3H3/3e52lWyy9jElzSjfITDwhhLA8W66PZCskcLIw\ntaOannN7MiZ2DNrntBiKDcQujOVw0GHSP0nH36vm8N0n359l7qpd/HIitc7sO0t1bzYnUJOZeEII\nYXm2XB8JbKNeoARObcTBxQHtci1jzo+hZ2hPSpNKibk3hqhRUeT8kMNVw3qwYck13D6pL5fyS3j5\ngyM8tf4XzqfktVmbTQ3U5I5HCCHahq3mpdrSKIUETm2sQ/cO9Ivox+izo/G6z4vC44WcvOEkJ647\ngf50MQ/dMpg3n7yGMYN9iD6fzaI1e3njs+PkFpS2ddPrZet3PEIIYatstT6SLY1SSOBkJTr17sSg\nDwcx8thDwHmdAAAgAElEQVRIut7YldxduRwLOUb03dG456pZNmcMzz0yDj8vV747mMA/X/6RL/fG\nUq6vW8+prdnqHY8QQtiDliwp1hZsbZRCAicr4zrMlaHfDCV4dzCuIa5kbs3kyKAjnJt7jkFubry+\neBKPzhiCRq3ivR3RzF+9m8NnLjap+nhra807HlsY/xZCiLZk6QlELWVroxTtKnCy5C/dlm7LY7IH\nIw6NYNBng3Dq7UTqhlQOBR4i8dkEbhgawFtPXcu0q3pz8ZKO5989xH/+e4DEiy1bPNiczH3HY0vj\n30IIYQ1spT6SrY1StIvAyZK/dM25LZVKhdedXoREh9B/Y38cujiQ8PzlEgZ572Tw8NQrWLd4EsP6\nd+e3c5ksCN/DG58dp6C4ovE3b2XmuOOpHnza0vi3EEII09laXpZ19tuZWWPFwKx9W2pHNT3+2QPv\n+71JjkgmYVUCsaGxJL6WSOCLgTz7j7EcPZvB/+2I5ruDCex2UJFW9DszJvXFqWPbfsWVdzxNodfr\nCQsLIzIyksTERPz8/MjJyTH63MjISFauXGl1J5YQQgjTWbKwc0vZfY+TJZPOWrqtxob3lI4KazLX\nMM99Hp/zOUUJRcTcH0PUiCgCEx15ffEk5t4ZTAcHFR9/f5Z/vvwj3x1MoMJgPflPpqjdu5SYmEhB\nQYHR51rj+LcQQoimsaW8LLsPnCyZdNbcbZk6vFcZUJxOOs2bvMlsZvM931N4spBTN53i9PWnuNKh\nKwtv8eHu6/pTWKznjc+OE/rqHo7GpFtVAnl9Ggo+jbHG8W8hhBDNYwt5WXYfOFky6ay52zIlf8dY\nQJFOOi/xEs/0eAa3G9zI3ZPLsdHHMCwrZYY2gP8+NYXrRgeQmF7As+8cZPlbvxKXnGu2/W0NDQWf\nxljj+LcQQliCzDJuG3YfOFky6aw52zJ1eK+hgOJA+gG6vNmFYXuH4TrGFf0uPUcGHyEjNJ6HQway\nbvFkRgz04sQfWTwesY9XP47iYnZRjTZYy8nXUPDp6upKr169bKIuiRBCtBaZZdy2rG/wsBVYMums\nqdsyZXgvMDCwKqCIj4+v87zK3iznQGdGHBjBodcOoXlfQ/qmdDI+ysD3H74sXTaCmIl5/N+OM+yN\nSubn4ylcNzqAMz9/zNeRn5GYmEhAQEBVW1syrqzT6UhLS7vcpiYGppXBZ/UE+0pz5sxh5cqVzX5v\nIYSwB5ac8CTqsvseJ7Bs0llTt2Xq8J6pvVkqlQrHSY6MOjGKoI+DLteA2pjKwcCDuGzMZ9V94wi7\nbyTd3Z355kACcYaRdOx5FWrHTi2e4m+uu6CGakDZwvi3EEK0Flursm2P2kXgVMmSv3RN3VZjARFQ\nNYzWlKKSKrUK73u8CTkTwoB3B9DBuwPJa5I53PcQ/l+U8fL9o0g59in60iL6jbmTa/7+FoEht6N2\n6NDsk89ctZZsaXZFW6scZi0pKWnrpgghLMDWqmzbo3YVOFkrYwHRggULMBgMNXpvwsLCCA8Pb1JA\noXZQ4zvHlzHnxtD39b44uDqQuDKRU1ec4Ip9Dhx49wnO7H8fUAi6+gGumbMRlfsgkpJTmrQPrXEX\nJL1L9avduzdz5kzJcRCiHbC1Ktv2SAInC6ovCdtYD4tareb111832nvTnIBC3VGN33w/xsSNoc8r\nfVA7qpmjzOHDivcZedSRn99ewLmDW3Ho0InB1zzC6q0X2H00yeQaUHIXZFm1e/fS0tKkkroQ7YCt\nVdm2RxI4WYCpuT/Vq2y31hi2xllDwJMBjLswjpixMTjgwFzm8kH5Owz5tZSf312ImyGZ7PwS1nxy\njHmv7GbvseRGAyi5C7IcyXEQon0z91qgLWFNs7ItRQInC2hq7o8lem8cujjw8E8Ps++RfXzt9jXO\nODOf+Xym+oAXe05jw8LJ3DC2Fxezi3j1oyjmr97NvgYCKFu9C7LFk/78+fPSuydEO2YNeaCWKolQ\nlllG/Avx6A9YTxqCBE6trDm9A5bqvXFwcGD1W6t5NvVZAg8H4hvmS2d1Zy48eYG4USeZntSN9Qsn\ncd3oAFKzigj/KIoF4Xv46bcUDEYCKGu6C2qMLdZBqWzzzTffjMFgMPoc6d0Tov1oyzzQ1l54vSyz\njLglcRzUHiR+eTz6H63n2twuAydL9jI0p/fI0r03zs7O9A/pz4DVAxgbP5aApQEYdAbiFsdxYdQp\nZiR7smHBJK4NCSAls5BXPjzKglf38POJmgGUNdwFmaq1T/rWUNnmxMTEep9jzb17Qgj70JrpAtUD\npqRXkjB0VnH6DhVHppU1+z3NrV0FTm3Ry9Dc3qO26r1x7OZInxf7MDZ+LL2W9cJQbCAu7M8AKqkb\nb86fyJQQf5IzCln1wVEW/hlAVR/Cs/bZcLaQI1Q7uG9sDT8fHx+r7d0TQtiX1kgnqR0wVXRWEXWr\ngYh7LvFD7zwcOqla2myzaVeBkzl6GZraW9Xc3qO27r1x7OpI7+d7Xw6glvfCUGrg/L/OkxBymtsT\nPXlz3kSuGeVPUnoBqz44yvzVu9l1JBF9hfEhJGtizTMA6wvuk5KS6m2zWq1m7dq1Vtu7J4SwL+ZM\nJzEWMB2dVsHaey7xU/9CrhkfwMYlUxg30NVczW+xdhM4tbSXoSW9VS3pPWrr3hvHro70fu7PAOqZ\nXhjKDJxf8mcAdb4bqx8cw+iB7qRlFRGx5Tf++fIudv56gbLyijZprymseQZgfcH9unXr6m1zQEAA\nPXv2tHBLhRDtlTnSSeoETM4qDt+sZ+09l/glSMf1V2t566lrWXj3cHw9O5t7F1qk3dyemromXH1a\nsjZQZe+RLa+z5ujhSO9ne+P3uB8pa1NIikjiwrILFC8vJlnZTnK/GAbcfDe5+QFs+OIkW74/y20T\n+3LTeC2dOlrXYdbQengtzRFqyTp9DQX3//vf/5g2bRrr16+v89j06dNxcnJqVnuFEKI5mrsGbFlm\nGUnhSaS8kYJBZ8DQTc3hieUc7JeHxknN1HF9uH1SXzzdO1liN5ql3fQ4taSXwVw5Mab2HlnzFHlH\nd0e0/9ESOSuSjWykWCnmHu5h5R/L8IqIJTDjCHdM7ktJmZ7/+180f3/hez75/iwFOutJ7APz55CZ\nI3+uoeA+MTGRwsJCFixYYBOzFoUQ9q2p6SRlmWXE/TuOg70v9zCVOcHP15Wy7r4cjg0r5dZr+/LO\n09fxyG1DrDpognbU49SSXoaW9laZSq/XExYWRmRkJImJiQQEBFRF8NaUu6LT6fj8m89JIIFtbONG\nbuQe7mE606l4vwKvEgM3L76KHy+ls+OnOD7+7ne+3PsHN43rza0T+tDNre1PCnP3AppjtfLK4D4+\nPt7o4x988AGhoaFER0fbbM+lEMK+VC/cbExZZhlJr/7Zw1RkoNxDxS9jSzg+uBQX9w7ce3UQU8dr\ncXHuYMFWt4z1/Da2gOZ2LTb0C82cOTHm+OVrCdUDyXLK2cEOvuZrruEa7uVeNFs0ZG/J5orp3bh2\n8Th+rsjmy72xbNsby1c/xTFhuB8zJvVF69uljfek8ZPeFI31SC5btoy8vLxGA52Ggvvq77dy5Uqz\nBOpCiNbTkmF7e1A7YCpzV/HL5GJODCnD08uZRyYPZUpIAB0dNW3d1CZrN0N10PyZapaoq2TpKfIt\nGQ40NuxpwMCP/MgzPZ4hYHMArmNcyY7MJnrCCbT/KeCV4BEsuDMYn26d2X00iQXhe/jP2wc4cS4T\nRTFtPTxr1VCPZHx8PMOGDTN5+C48PJwHH3yw3sfbetafEKJhtlhc15xqDMmtSqLYwcDuycWsfyCX\nS1OdePzBkbz17ylMHd/bJoMmaGHgdOLECWbPnm2utlhMc2aqtXZdJUtNkTfHSd1QIJmcmsyU5VPY\nPGYzV/xwBR7Xe5C3P4/fbz+D20PpLHMbyLL7QrgisBvHfs9g2Vu/smjNPvYeS7aJUgbGNJQ/B5CS\nkmJy+QsHBwfWr19vtbP+hBANs8XiuuZQO2Aq0lwOmDY8mEf5na4sf2ws6xZPYuIIPzQa2+6zaXbr\n3377bZYtW0Zpaak522O1WruukqWmyDd0UjelF6p6IFlbfHw8a9et5YX/vUDwd8GMOjEK7we9KTlf\nwh+P/YH+xgv8M9OPV+4fz1XBPYhPzePVj6J45KUf2b4vDl1JeYv305IJ9g0FksY01oPo7OzMjBkz\njD4mlcGFsF62UFzX3GoHTIWqCnZPLubtOfl0/psnq564mpfmXsXIgd6oVNZTxLIlmh04BQQE8Prr\nr5uzLTahteoqtfVw4HvvvUdQUJDJvVCVgeSRI0fqrSFUeaFwGepC0PtBl5dz+XcASrlC/Ip4MsfH\ncNuvXVh355VMu6o3+UVlvPvVaR56/nvejjxFWlZRk/exrbrJa/dINlRXyZQeRFta908IcZk1F9c1\nt9KLpcQ9GccB7eWAKR89uycX8+FcHb0W+bNx+bUsuHMwDvocuwsYVUoLEkySk5N54okn2Lp1a6PP\njYqKau5m2g29Xk9ERAT79+/n4sWL+Pj4MGHCBBYtWmSWnq3k5GRuv/32eheIrW3WrFnMnz+frKws\nPD09jdYKaug91Wo127Ztw8/Pr8bPFZ1C+VfllH1ShpKigAocrnbAcKcDUa7FHI4tpLD48vv17+nE\nmAEu9PHuaNLdSnh4OFu2bDG6L5boKi8pKSErKwsXFxdmz55t9ELZo0cPtm7dalLtpcr3q+/zF0I0\nzJLnUElJCTNnzmzxeW8N6vvcDBcNlG0uo+zLclRlUNDZwJGQUhLHVRAy2IXhgZ3RqAxERESwb98+\n0tPT8fb2ZuLEiWb7XWZJI0eOrPtDpQWSkpKUmTNnmvTco0ePtmRTzdIW2zSHoqIiJTY2VikqKmrW\n6+vb76KiIkWr1SqASX9dXV2VgIAARa1WK1qtVgkNDVXKy8tNfk+tVtvgPlSUVyjpW9OVo6OPKnvY\no+xhj3L4isNK4vpkZc/PCcritfuUaU9sV6Y9sV15bNUuZecv55XikvJ636+oqEjx9fVtVltaQ2ho\nqNG2hIaGmn1btnqsm0N73ff2ut+KYtq+l5eXK6GhoYpWq23wGmZurXneW+I7r+9zKzhboJyec0bZ\n7XD5Wr29yy5l6bU7lSWv7lN+OZGi6CsMVe/RGp+BNcUQthX6tRPmmCJf3/s2Nt29uoKCAgoKCoD6\nSyO0pD6W2kGN10wvut/Znfxf80l+PZmsL7KIm/sHDu4OPPr3HpTN6M83iSn8fDyF9V+cZNPOGK4b\nHcDNV/bGp1vNMvxpaWmkp6cb3ZY5622ZqrnlL4QQLddW5V1s/byv/blVxFfQZUM3Dq87glpRkeNe\nwdGxZXS9y5MHrhnGgF5da7y+sTyvlStX2nyepgRO7cxzzz1HXl4ee/bsITk5mZ49e5KTk1MVIDXG\n2IHf0guFSqXC7Uo33K50ozS1lNSNqaS+lUryq8nwGlw/rRt3zhnNL045fHMgge374ojcH8foQT5M\nHd+bYf27o1ar8PX1xdvb22g3eVvMRrOHpXaEsEVt+cvbls/76p9bIH35e+f5jCkairpMRVa3Ck5N\nqmDg3wMIu1JL1y7GhxwtVTC6LbVoTqCfn59J+U2i7VUmTQcHB/PBBx+gKAr3338/p06dYs6cOSa/\nj7EER3POOOzYoyO9n+vNuMRxDNw8ENcQV7J3ZHNhxu/0fbyAFz0H8/itwwjs6cah6Iv85+0DPPLS\nj3y++w/KKjRMnDjR6Pu25Wy0tl6oWYj2xhqStG3xvE9LS8MtvTuvub/LO7zNuKJgMr0MfDg+hse7\nPct9K/twz00D6w2awLoXUTcX6XFqJ2p3vyYmJrJp0ybc3d3r9Bg11AvV0IFvziFGdUc1Pvf74HO/\nD/mH80l5PYWMTzM4vzAOh85qHr3Hl/Lb+rG7MIN9v6Ww6eszfPRtDP3H/I1HvAfwfeSmVukmN1YN\n2N4rBNv7/gn709hqD25ubsTFxZl8TNv7OWAwGPjtw0SSX0lhbclqKIEUnzK+8j7AN0nvUfBrIlqt\nFj+/Ho2+V2suom4tbLsKlTBJY93WZWVlNXqMYmJi6u2FaosDv8voLgRtDmJc0ji0z2tx7OZI2jtp\nZE37g/GvVBDeM5hHrh+Mr6cLZ5JKSNUM54Z/vs1/P/uVvfsPsGDBAsrKWrbIsLEyBwsXLmThwoV2\nWyG4vVdAFrarofIu7u7uhISEmHRM29I50JzadTk5JfzvmRNs999HwYPxuEUbSPIv5ZW+n/K37Bls\nPbGCgkuJQNOu/XZfTqWts9PtbZvWoPZ+x8bGKmq12ugsB41Go8TGxtZ5j+ozKzQajcVmpJjCoDco\nWTuzlFO3nVL2aC7P8NjXeZ/y+z9+V7ZH/KSs3nxUue3Jr5RpT2xXpi7cqgy7YaEyaORkZWEL2j93\n7lyTZyO2xqy5xrTGsW7JWYEtIed5+9PUWXWV17Bhw4Y16Zi2tnPA2H43dfZgRYVBOXo8TXlv9i/K\nl267lD3sUXazW9k2+ifl2LYEpayszGzX/pbOEK/OmmKIFtVxaoqoqCjj9RDsbJvWoPZ+63Q6Bg8e\nbLTbWqvVEh0dXe+dhLV2UVe2q5vSjdwtuaS9nUZp4uUq9i4jXNjvfoB3c37Ge/hkOrtfHloszEkh\nwK2E11Y8Rje3TiZtR6/XExoayltvvUVFRYVJr2nsM20N5j7WdTodgwYNIiEhoc5jbbF/DZHzvP1p\nyr5XXivc3NwYNWqUyce0NZ4DxvZ70aJFRofFQkNDa8wezM4rZteueFLXp9L/VxXOxWoqHBQqbnZh\n+PP96T7Ercbrre3ab00xhAzVtQMtqUpubQmOtbvOh183nIisCEadG8WQb4bgMMmBwhOFjNg9hFd/\ne4jx7yVTtPlDUs7spZOrJ5fUgTz0/Pc8+85BfjmRSrm+4WAoLCyM9evXmxw0gX1UCLaG5FohzKHy\nGpaXl9ekY9oWzoHG0jDy8gv55UQqq145wKZr9uM+6yLDdmnooFbTeb4XVyWN5/rtIXWCJrC+a781\nkeTwdsLWa4tUaqw2S6funXAtdeX5q57neuV6ruVars28loxvM9j1w15O9s1jxEMPcjQmnaMx6bg6\nd2DySD+uHR1A7x5177jquyg1pL4Eemu7g2tIY8m19jAzRrQvTT2mbeEcqC+4c/fph2vgFJ6Zt59B\nhxwJiXVArXTE4KUhYLE/AY/54eAqv/6bS3qc2onWXqTYEkxdQNNvmB8/9/qZB3iA+cxnBztwxpl7\nKu7ipbMP88Dnbjzp2JNbgv1Rq+Grn86z8NW9LFqzl8j9ceTklwAN33E2pHYvni0lmFayxNqJQlhS\nU49pWzgHqk/9d3LpRmDIHVxz/xs8PCyCpafvYMaHzgz4wxGnwc4M3DSQSclX0udfWgmaWkg+vXam\ntaqSW4KpXefVp8NG//nnDd7gSq5kputMBhwdgPqomt6U49zvEr0WjGJfhxyOxGYQl3ya9746TXC/\n7owf4oW2Tz/Ox56tsz2NRsPDDz+Mo6MjO3bsaLAXz9QKxtbWI2UvvZRCVGrqMW3t54Ba04FrbnuE\n6KQy/D2DCT7lxLBtjrjoNBhUCt1meOK/yA+3q91MWutTmEaSw+2Qve63KUnuMTExjBw5Er1eT1hY\nWI0Lnru7O8ePH8cTT67lWm7gBrRoAXDwcKDLrV1JGQ67DJmcTc4FQKVUkHz2F1Ji9pGZcBzFcDnX\nae7cubz55ptV7aov4GkowbRnz54cP34cd3f3qrYmJiYSEBBQdXE2tUewNb9zawvoarPX470x7XW/\noeX73tRjui3PgerbjomJYWjwcH47m8H+31I4FJ2GS4rCiGMdCYpxwMGgRqcuJjU4hVmfzsKln4tF\n29qarCmGkB4nYTOaUlit9rIHlTNqALLIYsuffwYwgBldZjDNaRqXNmXQaRPc7tMB51v6ERuk53td\nOsrACfQcOIEyXR4FaScZqnUmYvXyGu2qrxevoV6ylJQUhg0bRvfu3Tl+/HjVzy21ppapbLmXUghj\nmnpMt8U5UP3mLzEpmaARk+k34kacPNPRFZbTN9aRO884433hcsZNx8COdLqvEyPnjsTV29WibW1v\nJHASbaY5d3FN7TqvvODFxcUZDWDOcpbVRav5+5G/45nmSfon6WR+nknu2xl4ArN7OuE41ZvT/UrZ\nlauhg/PVpACPrtrD+KE9uCq4B/0DPOrtBm8owRQuB08pKSlGH7OXBTGFEE23OCyMD7Z+Q88BU7jm\n+itxculKp3wVwT8YGHnOHYfcy4NF7lPc8VvkR7ep3VCpVVWFMK21h9geSOAkLK7GnVQTh6aau4Bm\nYzNkevj1wLm/M+4T3em9pjfh94Sj/KgwLGUYnd+uoC9wRT9PKiZ25lRAMXt0mWzfF8f2fXF4eXSq\nN4hqqJesMS1dENPah9iEEDUpisIfSbnsP5bI7yVDuOqeyagM4HeujEGHCwnK8kWDGo2HGt/Hfenx\nzx44D7h8buv1esKeaNmQvzCNfJrC4kxNlm5Ic7raTR3m+9dT/2Jt5OXndaADYxjDNVzDVReuwuGP\nEgYCQ3t6YJjozBltKbuLshoMoip7wz7//PN6e5eMae6U55YEppYgAZ0wJ1s/nioMCjEXsjlwKo1f\nT6WRlVsMgFuJK9rdKYy64I2XvjsAMcSwQ7WDtT+tpe/gvjXexxzXVVPZ+mfeUm1/FRXtSmMlBVpz\naMqUYb7a7SujjJ/+/NOvRz/2rN5D/s58sr/KRv9xLv2BQV3dUCZ05lxgGbsK/gqiunt0YuwVvowZ\n7EP4q6+xbNkyhg0bZnLw1Nwpz5a8gDaFtQd0wra01fFkjqChXG/gVGwWv55K5dDpi+QWXl71wNXR\nkdv1PvQ+CoZfi1DjRTHF/I//8RVf8Qd/oO2lpWfvnnXaZInrqpzDl9n9nrb3yNjamFJSoLWSME0Z\n5muofedTzlMysoSgu4IwlBvI259H5rZMsr7Momx7HoFAv86uqMY5c763nr36S+z46Tw7fjpP506O\nhAR5c+Odj/H++heoKC+p8d7Dhg0jNze32VOeS0pKiIuLw83Nrc0C08ZYa0AnGmat11BLH08tDRqK\nS/UcP5fJgVOpHD6TTlFxOQDuLh251dePwaccMXydjz77co9Tjm8O76W9xx72UExx1fsYu6Gy1HVV\nzuHL7DZw0uv1hIeHc/DgwXYdGVubhnKNfHx8cHOrW/rf3Boa5jO1WrDaUY3HFA88pnjQ7/V+5B/O\nJ2tbFllfZlH8YyG9gAdxRjPYifRBcMg9n726ZNBcwU3zP6Yg/SwJ0ftwKE3l5hsmEx4eTllZWZN/\nQVVezLdu3Up6ejq+vr719mi1dmDakLbsaRTNY829CyUlJRY/nuoLGsrLy6tKk9SWcUnHkTMXOXwm\nnZOxWegrDAB4unfi+j49GRLXEdXXhRRGFVAGOHo54h/mj89DPnTs35Gfwn7iTOSZqhuqsWPHGr2h\nskSVczmH/2K3EURYWBhbtmyp+r+1RcbWehfX2hrKNUpJSSEkJKRNL85NyYWqpFKrcBvrhttYN/qs\n6kPxuWKyv84m++ts8vbn4RmtcDMduKWbM4XDHDjlW8wR9yBcpwQBoAlw57NdsYwM8qZv7z6o1aYX\nqqt9MW9oGLA1lokw9Thuy55G0TzW3LuQlZVl0eOpoaDhrbfeAmDt2rWo1BrOJeRwJOYiR86kE5+W\nX/W83j26MFrrxRWJTqi+LSTnxxyKDIAGuk3rhs/ffeh2czfUjn8t6FG7hzwmJsbodbE5162mknP4\nL3YZOFlzZGzNd3GWUj3XqPYdkjVcnFtSLVilUuE8wBnnAc74P+GPPl9Pzg85ZO/M5tLOSzjvKmEM\nKsZo3NAP6kBCLz1H0gr4JD6Xj78/S5fOHRgxwIuRA70YPsALN5eO9W6rqWvpmXOZCFOP4+or01v7\nul/iL9Z8DQXw9PS06PHUUNCgdnTmyx9PcOnJd1C79iK/qAwARwc1o4K8CennxcCLTpRszyVreRa5\nxZd7nbqM7YL3/d50v6s7Hbp3qHfbpk6Eae0q57awdp+l2OVvamuOjK35Ls5SKnONGkqWbsuLc3NL\nHhh9ry4OdL+jO93v6I5iUCj8rbCqN6rgSAGBpyCQztBZTd5ANTFexZxMTGVvVDIqNfTzd2fEAG9G\nBnnRz98DTbXeqMbW0vPz8yMtLQ1/f3+mTp3KY489hk6nq9qXlvR6NnYcGwus3N3djb5XcwK66m0X\n5mdt19DqAXheXh5Agz0sgFlrGVUPGlRqBzx8+9O91zC6a4fh5t0XlUpNIeChhhvG9mJUYHd6JTqS\nuz2b7CdTScy9vC5lp/6d8L7PG697vXDua95rmzmvW8ZYolfLVthl4GStkXFDd3Fffvllm9/FWVpe\nXl7V+nK1tXWAC43f6TW21Ertx1RqFa4jXXEd6Yr2GS3ll8rJ3ZtLzo855PyQA1HFjMWRsThS4anm\nYqDCqW46dpw5x5YfzuLq3IGh/TwJ7ted4L6e+Pj41Huca7Vajhw5QnZ2NuvWrWPnzp1s3LiRgIAA\nbrnlFgB27NjRrF5PU3ojli5dWiewgpYnwRsLyMaOHcvmzZvbTY+tuRk7Vq3lGlr5fW/fvp2EhAQ0\nGg0VFRX4+Pgwc+ZMFixYUGOtyFtuuQWDwcDgwYPN1qOvKAo5RQYm3TaX0xdy8fQfgkOHTgAYKvRc\nSjlDZvxx8hOj+WHFpzjucCDrq/Oczbu8PFNHv474/M0Hr3u9cB3l2uprxrVmlXNrX7vPYhQLOXr0\nqKU2pSiKooSGhipAnb+hoaEWbUd1sbGxilqtNtouQPnb3/6mlJeXt3g7lv6sm6uoqEjRarVGPwut\nVqsUFRU1+T3Nse9FRUVKbGxsvdsvLy9XQkNDFa1Wq6jVakWr1SqhoaFKeXl5g481RndBp6S+k6pE\nz4pWfvb8WdnDnqq/3/rsU/47/EdlyQ1fK/fNiVSmPb5d+dtz3yn3LNqg9AyaqHTs7GH0OK/vPGjJ\nuXl8wx4AACAASURBVNHQcazRaJRTp04pvXr1qvd7zczMbPDzbYg1ntdtpaXHemPHqrk/68bOK2Ma\nO35DQ0NrvG9T21xfm7JydcqeqCRl3ae/KXOe/06Z9sT2qr+THnpTGTz5YcWrT4jSxaGbMpnJyjKW\nKTtVO6vO118DflX+eOIPJfdArmKoMDTr8zLGWq7tzfkuW6ot9r2+bdpt4FReXq7MmjVL0Wq1ikaj\nadIvsNbSUKBgzl8A1nJymcLcF+eW7LupQU9DbTbX/hgqDErB8QIlYXWCcnLaSWW/2/6agZTHXuXt\nwT8oS6fsVB548Ctl2uPblUl/e0MZN32x8vfQ55Xs3CKlqKio3gCmJcFqYwHvqVOnGgysYmNjm/RZ\nVN9uQwGZJS/i1qCl53ljx2r186El19Dm3kyYcvxW/96bcnzUblPfgcHKnEUrlYhPopSHV/5QI1C6\n++mvlZfeP6x8e+CCcjG7SHnyvieVO7hDeZVXlR/4oeqc/Nr1ayU2LFbJO5SnGAzmC5aqs6Vru7lZ\nU+CkUhTl8oI3raytVjYOCgqyqtlrixYtanD5Da1WS3R0dIvaakurplcfeqnd9ducrvXm7rtOp2Pe\nvHm8//77dR4LDQ2tyj/T6XQMGjSIhISEOs8LCAgAIDExsc5jLf1elQqFotNF5O7PJW9/Hrn7cynP\nKK96vNRZIaWHnhSfClJ99aR7V9DVuyNH9u8gO/kMl1LOUJyf0eA2NBoNZ8+eNambv77jODQ0tMHc\ntZZ8DnFxcfTv3x+DwdCittuLlpznDR3Htb+j2vlFTb2WNnSsGMvrrNxecXExwcHBRr/vStW/96Yc\nH/MXLWH7dwfp5n8F3fyuoLNHj6rndurowOA+3RgS6MmQvt3QerpSdLiQ7G+yyf4qG12Mruq5v/M7\nZ9zP4DHVg2XvL8PR0dHkz6U5Wvvabs2zvdsqhjC2TbtPCrC2ld3Dw8PJzc1l06ZNRh+3htweS2rt\nhMbqjF0UKgO3L7/80mjAAzUT1RtKmk1OTq532y39XlUaFS7BLrgEu+C3wA9FUSg+V1wVSGX8mEGf\nWEf6xF6+cCsqyO5ewRCfW0nzv5m00RWkOGZwKTWGSylnuJQSQ0F2Iih//ZJpSu6KsVyHyvySkJCQ\nessiNCWJtPb35evri5+fn9HvqXbbrfkXgDVoSvJ3hw4deP3115s1E7gps/Nq56/5+fnRuXNnCgoK\n6n3/6t97fceHSq2h/9Dx/BZfzue/HCEm/hLZmvEMnzoegPLSItLjjpCdfJrspNO4ddTTb/J9jCr6\nJ3mvJXNgTw6GosvnibqTmm63dKPbrd1wnuyMP/484PuAzR9jMtu7aeQTsTAHBwfWr1/Pnj17TPoF\n0F60ZoDb0EWh9uwwY6r/ImkoadbPzw8w3uNk7u+1etmDHg/3QBelY7D3YAoOFZB/MJ/8g/lojhbg\nmaEh+OTl15R06MxF3wDSfK4lY3wFKW6FJBb+Tm76OXIu/sHNt0w0+ReAsYC3dkJ49bb26tXL5CRS\nY99XZUJ7Tk6O0ddUBmTyC8A0TUn+bslM4KYEaLW3U9+NTHUTJ04ELh8zS5cuJScnhw6d3PDoMQAP\n3wF49BiIu09fNA4d2bTzLAAuThouxh0iOyma7OTT5GfG4650IZhgZnAboxiFzwc+nP/gPADOA53x\nuN6Drtd3xf0adzSdNFXbd8f4TFFbY4nZ3vZ0MyNXkjbg7OzMjBkzZFqnhTRU8ffrr79u9PXVf5E0\nNCV3xowZAG32vTr5OeHk50T3Oy4vCGooN5D/Wz6blmyi8EghvYp6oU3wQ5tQOZzQmWKn7qR7jyfD\nq4L0uAoWPvE9PYa607+XB/0DPAj0c6dTx/ovE5UBr06n48svvzT6nB49enDkyBE8PT1N2g9j39fr\nr79u9LldunRh6tSpVQGZlPswjalTy5tbz6mp9bsa2o6rqyseHh4kJiZWzapzdnZGo9HwyadfcDQ6\nBY8e/SnSd2LUzFfo7P5X0KcYDHRUFdGxIpMzx/YQd/pXuro6QjYEFgUynVsJJhgt2qrX5JPPXvYS\npYpiwP0DePm9l+066G7tml32eDNjm622AzKt0zIauyjUVw6hutpBjynfnTV8r2pHNe6j3QndE0p+\nfj6hoaEc/uEwLqkuhLiFMKbbGHqV9kKboK4WTEFJxyIyuufzrdd5srtX4DigE12Hd6FPH3d693Sj\ndw83XDr99Xy9Xs/cuXPr7SG4ePEieXl5JgVOTS3q6e7uzj/+8Q8SEhKsep0+a2TKcdzUek7Nrd/V\n0HZ0Oh2//vorKk0HcnQaXn9nC38kXsLdu2+N3CR3oKykgIwLUeSknuVS6u+4OhYz9Ybr2LFhB4MY\nxLXcS3BOMP74//X+6DjMYU5wgt/4jbOcxYABFPjf5v+h76q366C7tWt22ePNjN0nh1t7knRrdF/a\nwn63ltr73lDCqFqtbnBtt169enHbbbfVe2fU1DpOramx77yhBN3V/1lN4W+FFBwroPBYIblH8imL\nK7k8H+lPCgq5bgayPA1keVZQrnXAZUhnfIe7s/v7L/jwvbWUFGQZ3XZTEsIb+r7q4+XlRVZWVoPf\nZXOTx615eKGx79zUtjd2HA8ePLjeWmG1v9f6jrP66ndVnlc1tqNS09nNG1fPXrh69sKnVxADh44n\nLVtX4z3LSgrJS48lLz2OvPQ4ctNjKc7PwAUXgv78M5jBDFINwkVxqXpdIYWc5jQxHWP4//buPTqq\n6uwf+HeuSSYXSAjEEJlAKNgQ3sjth5S+Sq1QFFkvr8VAUApotWAKJg1QXFjUZTBWF3lFVCyFlrL4\no4iXgnipNxBULkoQgUi4BxKNIRASkplkMpf9+yOdMZeZOSeTmcmZme/HNUsyt7Ofs88585y999nn\nYp+L+OzyZ22Jkgf+uGDHXwJxbDebzcjMzAzIRS3duQhBipJyCLY49TK5Y3uUfABXMm9jOYxGI6ZN\nm4b169d3eW3+/PlYv36913Xtre6UdFGCnKb4xF8mIvGXia7nbY02mI6bYDphQuOxJlz7+jpQZkbi\nOQeGndMBhwC8ZoFN8wOmJP4M41LHo/amFnyn+QEXHRU4bzmJq/Xn0Xi1Ev/TjW5Kb/XlyeXLbVcL\n+vM+fd3tXujt/bP98vV6fbfKLrUdy50t2tt2Vl9fj6+++qrDVXlCCNQ3WnDxh+uoqL6On/3vH3Hj\n5SbE9TNCq4vu8PlGsxU3D0tGvzhgzerHUP/DGZgbahCPeAzDMGTjJxiOX2IYhsEIY4fPVokq7Md+\nnMRJlKEM53AODjigsWlw9JOjKCkpcXs1rVNvzpQe6O2p/dgwd3o6xEBpM9D7CxMnhQvH/uFgkjrw\nr1mzBjqdzm/TISiRLwcvbbwWfSb2QZ+JfX7cBr/biabrTRg/YDymDJ2CcckTYDtpQZ9zNgy4osdN\n0ANIADAcwK/QGOdAXT8HGj8TePnevTAMj0G/rAQM/K++uHFgPAYkxkCjUXdYrrf66onu/gDI7V5w\ndyXY7bffjnXr1iEhIcHn8sr94XR2we7evRtVVVWurrGjR49Kll3u8uUOKzh//rzb7Uyt0eKaCdh3\n5BLsmjhUfVmO72qb8N3lJjQ1W9u9cSD6pjjQ3FCNmuqz0NgaMHpEOopWFaJ/XwMsFRbUHanDhZrR\nSG6YgmEYhlR0TIab0IRSlOLb//w34YEJ2LFnh8cxVhkZGdi4cSMMBgM2bNgAu93u9n3Bnik9WMd7\nTxfHJCQk4IEHHujxEAOlzEDvd75ODGW328WqVavErFmzxNy5c0VFRYVPE0kFUjhMFubLhIrhELev\n3MUuZyK/3pgJ15+81XlPZ2j3Nnuz0WgUCXEJoj/6izEYI+5RzRRLox8X6wwbxVvR73SYtNP5+ES1\nW+xI+ERsMX4kNo77RGy6Z5/YWvileHvDt+LgF1XifFWdWJL/hw71tWTJErFkyRLXc2lpaV4nRrzx\nxht9nrSxOxMpelo38fHxfpks0mg0ivnz54uGhoYu78vNzRXx8fGyJzhNS0sTtbW13Vp++/XnaR+x\nWq1iSX6hGDbyFjEg4/+JwaPvFiN+8Vsx/p5V4vYH/yLuLnizw4SS0wt3iBnLdoqFz34siv52UGx5\nt0zsPVIpKqobRKvFJurK6sTxvx4XZ54+I779zbfiqzFfib0xe7tsR2/hLfEcnhMP4SExCZPEHZl3\niCHpQ7rUu9xjaF5eXsAmJZZLqqz+PLZ7286NRqPfjoX+mhRYSTmEz2OcPvzwQ+zevRt//vOfcfTo\nUWzYsAGvvvqqx/crqX8yVPjaP6z0uAPZDO0t9t7uTgmknoxx8tYK4W0blJKfn4+SZ0pgPm1G00kT\nLh9tQP0pEyznm4HvbNBfc3/oaYkSaExwoKUvYE0GtGlRiEmPQdwQA2LTdRCJzUg3JuP22yZ6vU+f\nL5M1AvIn2pSzbjqvX6nJJD3VU3x8PB588EFXy4PURLqepKWl4d577/XYguHpexfnL8WyFU/gakML\naupMqKkzux7nLl2GQ6V3uzyLqR5N177D4IGJuPd/puDGAXEYmByLvmYNrOctMJ8xo/lM84+P880Q\nrR23C1WUCrGZsTBkGRCbFYvozGis3LwSu7/Zjcqqji1gra2tHudqk2pV9vdkvN0l53h/8uRJvx3b\ngzWhrL/Wq5JyCJ8Tp2effRbZ2dm4++67AQC33norPvvss24XIJCUnkBI8XXDVmrcwWiGVmrsgSYV\nt9yDV+fkUu5g7fj4eCQlJaGqqkr2gdHebEfLhRbUlzfhyonraDhlQsv5FjiqWqGtdUBrcf85AYGm\nWIGmeAfqdWbUaxtxTd2AOlUdropa/GR8Bu7Ly0Hi4Fgk9o1GvEEPtVolO3G+cuWKrJnP5aybtLQ0\n7N+/H83NzVi3bh3efffdDjerTU9Px913341HH30U/fr1w7hx4yQTMW8zs8uVn5+P//u/F2BqsaKh\nyYKGplZU1zZgxeNPo9mqQnRc0n8e/RAdmwSNLsr9FwkHTPU/wNRQg+aGGpiv18B+7Rpi6+2IaxC4\nKT4DE38yEbcOvxWtVa1oudQCS6UFwtr1Z0ebqEXMsBjEDIvpkCjFZMRApel4Y1xf7grhj8HyvpD7\nfXKO9/X19X47vnV34L8/lteT9aqkHMLnxOnxxx/Hr371K9cEZL/4xS/w8ccfezxQlpaW+rKYiNbS\n0oKcnBy3l8wPHDgQ27dvR3R0dJfPXLlyBcnJyV1e83fZurucNWvWYNu2bV2ez83NxbJly/xdRHLD\nU73ZbG2XXO/duxc1NTVISUnBpEmTsGjRIsyZM0dy2ga1Wo1//vOfiIqK8su2J4QAmgBbtR2mS3a0\nVNph/d4BR40D6lpAfxWIagA0ds93mhcQMBsETLECzdE2NGqbcV1tQrPeAnWSFmlZadAkaaBJBHT9\n1ND0FXhr52v48sBn+L7qIuzWFjjs1g7fee+99+Kxxx4D0Dbg+b777nMNTve2buRcJZicnIwrV9xf\nmegUExMDg8GAq1evup7TaKOgjTJAqzdAF2WAVh8Drd75dyz0MfHQxyRAb+jT9u/oBETHJUIfHQ9v\nB38hHLCY6tFiuobWxjpMuXk80rTx6GtVI75ZjeOffIlLR88hEX2RiEQkIQnJSEYf9PH4napkFVQ3\ntD2+/P5LHPruEL69/i1aB7Ri7O1jUVBQEDZjCz3tU55i9OV431NyjsnB+k1RKr9eVRcXFweTyeT6\n2+FwSG7wSskWQ8msWbPcNp3n5OTg5z//uevvYA0q9HU5ZrMZBw4ccPvawYMHkZmZ6Zezm3Coc1+0\nj9uXM7uCgoIOB9Dq6mps27YNSUlJHrfB9oxGI6ZPnx7ULlDhELDWWfHFrkPQm+OgaYyB9QcbWr63\nwFZjhaPWBnWdHX2v2TGgNgpAFNB+pucv2n9bW2LzsDYH86LuhcUgYIkSsOgdaNHb0aK1waxtResF\nO95ZfhFWnR1XTVcxOX0pWgdb0aJuhUXViha1BRZYYVG1oBVWOIQNwmGHEA6oVGpApQKggkrV9mj7\n94/Pp6jVUGv1UGt0UGt00Gj1UGt10Gja/q/W6KHR6qDW6tsSI30MVGoNPFE5AK0N0FlV0NoAbauA\nytQC1DRj2MBYJOtjEGfTItamRlQz8PXegxANFsRYdTDYoxGPOMRhKOIxCuq3nQP5BQA7foax+Bl+\n3NdMMOEqruI0TqMGNWjt24qlzy1FwrAERKdHIyotCuoo9Y/3hfz4Hz8WtAY4s+0MUlJSZA1gD4X9\n3NM+5S1GqeO9v+PeunUrUlJSPA78V9KFSb3V4uSOz9GPGTMGe/bswbRp03D06FEMHz7c58KRZ3Kv\naAnWJGO+LidcL0vtKX92C/QkqfV0GfmGDRvw8MMPY8mSJdi1a5fHaQICOTO6p3WkUqugT9YjIduA\nsWNHe/zsiBEj8P3F75GABCSqk9FPNwCJ2gFIT8rAfdPug/2aA62XW3HpWAX0Vi0M9mjENOnQ55oO\nWuHtENnfa7kFBGwawKEB7BoBu/rHfzvUgF0D2NUCDk3bvQVdrT/Of6vafZOqrRVO5RBQ2QXUdgGV\n3dH2bwegtgNqh4BGqBGl0SFGHQVVi4DK6q5k7VuErP95tJmI/3L9uwUtaEITruEarAOtyJyYCV2K\nDvoUPRrUDfj9qt+jTtShDnW4hmtoRWuHpeTPz0f679Jdf9tsNhQWFMq+L2Qo83U27mBPjOztXqGd\nx7uFw8SV/uJzV53D4cBTTz2F06dPQwiB4uJirz98SuqfDEVSk9T5a5IxqTL4upxg9aeHSp37u4Ww\ntLQUW7du9WkAuJyxOvn5+SguLkZlZSXWrVuH9957z68DaN0NmpY7H5G3OpeaAPXgwYNISkpCc3Mz\nbr755i7v00OPOMQhFrGIa/dftMoAgzoOBnUsohGNaFUM4rRxiNPGQdWCtudENLTQQgsNtKLd/4Wm\n7QEtNEIDPXRdyibFBhvssLv+L9QC2igtEvsnQq1TQxOjgdqgxvnvz+Ns1VlYYEFLu/+EXiBrTBb+\ne+p/o/+Q/jCkGKDtq4UqXoVn1j6Dtz56CxeqLnisX2/jwDQaDRYuXIgXX3yxw2fkDGhXq9U4ffq0\n5EmU0vbz7owNlDPw2tPxPlhxB+s3pTsUlUME45I+IZR1KWG4OXv2rFCr1W4v+dRoNOLs2bOKWI6/\nLkv1JlTq3N/r4vPPP5d9CX1n3qYr8PQd/pq+wXm5uLPsGo1GABDp6eli1KhRstaR1FQMRqPRY1zO\n5Q0aNEjExcXJvrTf03d9+eWXklMleHuooBJqqF2P1P6pQgut0EEndNAJDTTdmmbAuX69TVvQnek5\n2k9X4On78vLy3NaDp+2z8zrMy8vzOv2BVJ0Hk6fpGxoaGno0BYgnwYo7WL8p3aGkHIKJUxjo6Tw9\nwVqOnPmUeioU6rw78wTJtWPHjoAktYE+WEotV8468lTncpIGfz6c5ZITkzNhk/q+ixcvSiZinRPJ\nzgmHVPLo6Xu6U18qlcrr/uzth9jdY9SoUW7nkpKq82DzdgIk9+SoOychwYo7WL8p3aGkHIKJU5gI\nRmuOv5YTyMkmQ6HOA3E29/nnn/c4qc3Ly/P4gx6Ig6XcVgipdeSpzn1Jyto/EhISPLZ6edsH3LWi\nqVQqkZ6eLvLy8kR5ebmora0V5eXlIi8vz2Ni5/w+T3HExsZ2SCg8tX6Ul5fLSlrktEx6qi+pyTXl\ntGrKXb/e6jyYpE6AGhoavJ4oepts1JNgxh2s3xS5lJRDMHEKE77shD1dTqBajXoiFOo8EGdzhw8f\n9suBLlCzJ7tLlrvbCuFpHbmrc1+TsvYPo9Ho+vGTavVZsGBBl33AGXNtba3XE4WGhgaxYMECkZ6e\nLvkDq9FoXLOJf/rppx2+x1P95+XlyUpaNBqNOH78uMeyBqqr3pd6V8J+7m19qNVq1/rwdKKo9LtC\nKOlYbzKZxI4dO4Le0sXEKUIEawNT6i1KQqXO/X02d/jwYb8c6Px9sJS6hYcvrRByxjj5mpR5SgZq\na2s9Jk/+uj2F1D7V+fX2cUu1fnhKiNs/4uPjhdFo9Hji5e+ueqPR2K1u1Pb1oYT93Nv6aD9Wq/Nn\nnMm0L931vRF3bx7rg9Ug4AkTpwgSbnF35wclVGL3JUGRO1jWHwc6fx0spRJEqVYI51gXb+vIGXv7\nMvuja0jufemU0HUh1RpUXl4uOahbTmz+7qrvTiuU0lqchJDefjt33zoTAG8tmN5a75QSd7D09j7H\nxCmChEvcUmcb7l7Pzc1VTLehHHISFDlnXUqsczmD4L2NB5K6uazTwYMH3a6fJUuWeEzGOl/FJ+fg\nrKSuCyG6JstyWoNMJpNrbJWclp/OyWMgWySd3yfnakqlbO9yxwb6miB2ppS4gyEQF9F0FxOnCBIu\ncfvaWtFbLQCBIifOntR5oJriuzMmRu54IHdyc3PdLmPJkiUef+TbL69zIiGVDCilm7pznXd3f3DG\ncfz48W6PXfL3Omj/fXKSMyUd46S28+PHj3drvJ1Sxjj1NiVMicDEKYKEQ9xSZxu+jhEINXLPunyp\n80CPHwjGJc0mk0mkpqZ6XYbcH3mlJERyda5zX1uDlHjpubNcSp/HSQjp9ectMQUg4uLiZNeXkuIO\nNCVsl57Wt/PmQ0SKInWLlmPHjknewiUcyLlVja+ct8+pqKiAw+Fw3VLBXzdcNhgMmDFjhtvX/HWL\nlurqatTU1Lh9zbl+DAYDhg4dKrk8ue9TKuftM8rKynDq1CmUlZVh7dq1kjO6B6OefBEq9SG1/jIy\nMmA0Gj1+vqmpCXPnzpVdX5FCqdslALDFKQyFQ9xSZxu1tbW9fjYSDHLPurpb58EaPxDocUFyWpzC\nlT/3c6WN35KitGOc1PqTGuPkPKZJtXgqLe5A41V1TJyCJlzi5hinNoEY4xTs8QOB7AbzNMYp3LaD\nzgKxn4dKd6VSj3Ge1p/VahXz58/3mjylpaVJJgdKjTvQlDaPE9sESbGk7hTu7vUJEyYE7E7ivSUQ\nd0xPTU2F0Wh0e9PlQYMGITU11efvdsfZ7RIIBQUFSElJCdod5cNZIOspEnhaf1qtFuvXr8eePXtw\n6dIlt5913jDZ2WUOoMPNuc1mM6qqqpCZman47kt/MxgMuPHGG5UTd29nbuG2TCUIt7jDcR4nX/h7\nsGy4tNi5m8cpEoTCth6oOgmF2N3xZVqC3u6uUgol5RBscSLFkzoLjpSzZH/HGYiWrN4UKdtBKLDZ\nbFi2bBl27tyJS5cuwWg0uratSB783Hmfu+GGG1wtTZ05L2546aWXXC1QgOcWKQoeXlVHFKF8vQqL\nSEqgr9gMVZ33uaNHj2Lw4MFu3zto0CD06dMHO3bscPv6zp07YTabA1ha8oSJE1GEC5XLvik0mM1m\n/thLcO5zycnJXi+5b2hoiIhpV0INTy1JUcxmM6qrq5GamsofcqIQJGfuMXap/shbl3lra2tQL+Ig\nedjiRIpgs9lQUFCArKwsDB8+HFlZWSgoKIDNZuvtohFRNziv2HSHP/ZdeesyV/QkkBGMLU6kCM4x\nEU4cAEkUmpw/9u33Zyf+2Hvm6eKG9i1SnQfaU+9gi1MYM5vNOHfunOLHFHBMBFF4WbNmDfLz8zF4\n8GBoNBoMHjwY+fn5/LH3QfsWqbfeeosXcSgAE6cwFGrdXoG8HxsRBR+v2PQ/xU0CGcG4FYehtWvX\nYtu2ba6/ld7tFexZrIkoODi3VmDwIprexRanMGM2m7F37163rym124sDICNTqHQlEymFknsTIml/\nZuIUZqqrq1FTU+P2NSV3e3FMRORQ8sGfSMnWrl2ruIlFI3F/ZuIUZlJTU5GSkuL2NSV3e3FMROTg\nrNIUKpTUiqLU3oRI3J+ZOIUZg8GASZMmuX0tFLq9OIt1eOMVlBQKlNiKosTehEjdn5k4haGCggJ2\ne5Ei8QpKCgVKbEVRYm9CpO7PTJzCELu9SKk4qzQpnVJbUZTYmxCp+zMTpzDGbi9SGl5BSUqn5FYU\npfUmROr+zCYIIgoqbzc1JeptSp5XztmbUFxcrJh5nCJxf+5Ri9NHH32EpUuX+qssRBQB2JVMShYK\nrShK6k2IxP3Z58hWr16Nzz//HJmZmf4sDxFFCM4qTUoVia0oPRVJ+7PPidOYMWMwefJkvPbaa/4s\nDxERUa9SYpcYKYdKCCG8veH111/Hli1bOjxXXFyM7OxsHDp0CNu2bcMLL7wguaDS0tKelZSIiIgo\niMaOHdvlOckWp5ycHOTk5ASsAIFUWloa9GUqQaTGDURu7JEaNxC5sUdq3EDkxh6pcQO9E7unBh9O\nR0BEREQkExMnIiIiIpl6dL3gLbfcgltuucVfZSEiIiJSNLY4EREREcnExImIiIhIJiZORERERDIx\ncSIiIiKSiYkTERERkUxMnIiIiIhkYuJEREREJBMTJyIiIiKZmDgRERERycTEiYiIiEgmJk5ERERE\nMjFxIiIiIpKJiRMRERGRTEyciIiIiGRi4kREREQkExMnIiIiIpmYOBERERHJxMSJiIiISCYmTkRE\nREQyMXEiIiIikomJExEREZFMTJyIiIiIZGLiRERERCQTEyciIqIIYzabce7cOZjN5t4uSshh4kRE\nRBQhbDYbCgoKkJWVheHDhyMrKwsFBQWw2Wy9XbSQoe3tAhAREVFwLFu2DC+++KLr74qKCtffa9eu\n7a1ihRS2OBEREUUAs9mMHTt2uH1t586d7LaTiYkTERFRBKiurkZlZaXb1yorK1FdXR3kEoUmJk5E\nREQRIDU1FUaj0e1rgwYNQmpqapBLFJp8SpwaGxuxaNEizJ07F7Nnz8bXX3/t73IRERGRHxkMBsyY\nMcPtazNmzIDBYAhyiUKTT4PDN2/ejAkTJmDBggU4f/48li5din/961/+LhsRERH50Zo1awC0jWmq\nrKzEoEGDMGPGDNfzJM2nxGnBggXQ6/UAALvdjqioKL8WioiIiPxPq9Vi7dq1KC4uRnV1NVJT6Ybn\nvwAACeNJREFUU9nS1E2SidPrr7+OLVu2dHiuuLgY2dnZqK2txfLly7Fy5cqAFZCIiIj8y2AwYOjQ\nob1djJCkEkIIXz546tQpFBYW4o9//CMmTZok+f7S0lJfFkNERETUK8aOHdvlOZ+66s6ePYv8/Hys\nXbsWP/3pT3tUgEAqLS0N+jKVIFLjBiI39kiNG4jc2CM1biByY4/UuIHeid1Tg49PiVNJSQlaW1vx\nzDPPAADi4uLw6quv+l46IiIiohDgU+LEJImIiIgiESfAJCIiIpKJiRMRERGRTEyciIiIiGRi4kRE\nREQkk8/zOHUX53EiIiKiUOJuCoSgJU5EREREoY5ddUREREQyMXEiIiIikomJExEREZFMTJyIiIiI\nZGLiRERERCSTT/eqU5qPPvoI//73v1FSUtLlte3bt2Pbtm3QarV45JFHcPvtt6Ourg7Lli1DS0sL\nBgwYgGeffRYxMTG9UHLftbS0YPny5bh69SpiY2Px3HPPISkpyfX6vn37sHHjRgCAEAKlpaV45513\nYLFYsHDhQgwePBgAMGfOHEybNq03QvCZVOwA8Mgjj+DatWvQ6XSIiorCpk2bcPHiRTz22GNQqVQY\nNmwYnnzySajVoXPuICfu5557DkeOHIHNZsPs2bMxa9Ys1NfXY+rUqRg+fDgAYPLkyZg/f35vhNAt\nDocDTz31FE6dOgW9Xo/Vq1cjPT3d9Xq47tuAdOz/+Mc/8O677wIAJk2ahMWLF0MIgdtuu821b48a\nNQpLly7tjeL3iFTsq1evxpEjRxAbGwsAWL9+PaxWa8jXu7e4T548ieLiYtd7jx49ildeeQXZ2dkh\nuW+7880332DNmjXYunVrh+d3796NV155BVqtFjNnzsSsWbNkHQsDSoS4oqIiMXXqVFFQUNDltcuX\nL4vp06cLi8Uirl+/7vp3UVGRePPNN4UQQmzYsEFs3rw5yKXuub///e9i3bp1Qggh3nnnHVFUVOTx\nvRs3bhQlJSVCCCG2b98u/va3vwWljIEiJ/a77rpLOByODs8tXLhQHDx4UAghxKpVq8SHH34Y+ML6\nkVTcBw4cEHl5eUIIISwWi5g8ebKor68XX3zxhXj66aeDXt6e+uCDD8SKFSuEEEJ8/fXXYtGiRa7X\nwnnfFsJ77JcuXRL33HOPsNlswuFwiNmzZ4uTJ0+KiooKsXDhwt4qst94i10IIXJzc8XVq1c7PBcO\n9S4Vt9N7770nCgsLhRAiZPftzv7617+K6dOni5ycnA7Pt7a2uo5jFotF/PrXvxa1tbXd+v0LhNA5\n3fZgzJgxeOqpp9y+duzYMYwePRp6vR7x8fEwGo0oLy9HaWkpbr31VgDAbbfdhv379wexxP7ROYYD\nBw64fd8PP/yAnTt3YvHixQCAEydO4NNPP8X999+PlStXoqmpKWhl9hep2K9cuYLr169j0aJFmDNn\nDvbs2QMAKCsrw/jx412fC7V6l4p79OjRHc5K7XY7tFotTpw4gbKyMsydOxePPvooLl++HNRy+6p9\nvKNGjcKJEydcr4Xzvg14j/2GG27Apk2boNFooFKpYLPZEBUVhbKyMtTU1OA3v/kNHn74YZw/f763\nit8j3mJ3OBy4ePEinnjiCeTm5uKNN97o8plQrXdvcTuZzWa89NJLePzxxwEgZPftzoxGI1566aUu\nz587dw5GoxF9+vSBXq/H2LFj8dVXX8n+/QuUkOmqe/3117Fly5YOzxUXF2PatGk4dOiQ2880NTUh\nPj7e9XdsbCyampo6PB8bG4vGxsbAFdwP3MXer18/WTFs3rwZCxYsgF6vBwBkZ2cjJycHI0eOxKuv\nvopXXnkFK1asCGwAPeBL7FarFQ8++CDmzZuHhoYGzJkzB9nZ2RBCQKVSefyckvgSd1RUFKKiomC1\nWvHYY49h9uzZiI2NRUZGBkaOHImJEyfi7bffxurVq7Fu3bqgxeKrpqYmxMXFuf7WaDSw2WzQarVh\ns2974i12nU6HpKQkCCHw/PPPY8SIERgyZAiuXLmC3/3ud7jrrrtw+PBhLF++HG+++WYvRuEbb7Gb\nzWbMnTsXDzzwAOx2O+bNm4eRI0eGRb17i9vpjTfewJ133unqlgrVfbuzqVOnoqqqqsvzSt3PQyZx\nysnJQU5OTrc+ExcXB5PJ5PrbZDIhPj7e9Xx0dDRMJhMSEhL8XVy/chf74sWLXbF5isHhcODTTz/F\nH/7wB9dzU6ZMcb13ypQpKCoqCmDJe86X2JOTk5GbmwutVot+/fohMzMTFy5c6DCeSen17mudNzQ0\n4NFHH8X48eOxcOFCAMCECRNc4z2mTJkSMgfWzvuvw+Fw/YiEy77tibfYAcBisWDlypWIjY3Fk08+\nCQAYOXIkNBoNAGDcuHG4fPlyh5OFUOEt9piYGMybN8+1PU+YMAHl5eVhUe9SdQ4Au3bt6rD/huq+\nLZfUfu58Ltj1HfJddd5kZ2ejtLQUFosFjY2NOHfuHIYPH44xY8Zg7969ANoGUbu7F43SyYnh9OnT\nGDJkCKKjo13P/fa3v8WxY8cAAAcOHEBWVlZwCuxHUrHv378f+fn5ANp2qjNnziAjIwMjRoxwtU7u\n27cP48aNC27Be0gq7paWFixYsAAzZ87E73//e9fzf/rTn/DBBx8ACK06HzNmDPbt2wegbTCscwAs\nEN77NuA9diEE8vLycNNNN+Hpp592JUsvv/yyq5WyvLwcqampIZc0Ad5jr6iowJw5c2C322G1WnHk\nyBFkZWWFRb17ixsAGhsb0draitTUVNdzobpvyzV06FBcvHgR9fX1aG1txeHDhzF69Oher++wuFfd\noUOHsG3bNrzwwgsA2rqnjEYj7rjjDmzfvh2vvfYahBBYuHAhpk6diitXrmDFihUwmUxITExESUkJ\nDAZDL0fRPc3NzVixYgVqa2uh0+lQUlKC/v374/nnn8edd96J7OxsvP/++zhy5IirPxxoG+dTVFQE\nnU6H5ORkFBUVdWgeDgVyYn/mmWfwzTffQK1W46GHHsLkyZNx4cIFrFq1ClarFRkZGVi9erXrRycU\nSMV95MgRvPzyy8jMzHR9xjnmaeXKlQDazthXr16NAQMG9EoM3eG8yuj06dMQQqC4uBj79u0L+30b\n8B67w+FAYWEhRo0a5Xp/YWEhMjIysHz5cpjNZmg0GjzxxBMYOnRoL0bhG6l637RpE95//33odDrM\nmDEDc+bMCYt6l4r72LFj+Mtf/oL169e7PlNZWRmS+7Y7VVVVKCwsxPbt27Fr1y6YzWbMnj3bdVWd\nEAIzZ87E/fff7/FYGCxhkTgRERERBUNYd9URERER+RMTJyIiIiKZmDgRERERycTEiYiIiEgmJk5E\nREREMjFxIiIiIpKJiRMRERGRTEyciIiIiGT6/0JTuxu5+at+AAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.figure(figsize=(10, 5))\n", "\n", "plt.plot(x, polynomial, label=\"Optimal fit\")\n", "plt.plot(x, y_pred, c=\"m\", label=\"Generated fit\")\n", "plt.scatter(x, y, c=\"k\", label=\"Data points\")\n", "\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Note on gradient computation\n", "\n", "Of course computing gradients by hand can be tiresome and even infeasible when working with very deep networks. Therefore most libraries for neural network like Tensorflow and Torch include some sort of **automatic differentiation** for computing derivatives over large graphs. Automatic differentiation is more similar to symbolic differentiation rather than numerical differentiation as the ladder can give numerical instabilities." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python [conda env:base]", "language": "python", "name": "conda-env-base-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 1 }

very nice