edwardeasling/lesson2SGD_withmomentum.ipynb

## lesson2SGD_withmomentum.ipynb
{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from fastai.basics import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part of the lecture we explain Stochastic Gradient Descent (SGD) which is an **optimization** method commonly used in neural networks. We will illustrate the concepts with concrete examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Linear Regression problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of linear regression is to fit a line to a set of points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "n=100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.5494,  1.0000],\n",
       "        [-0.9676,  1.0000],\n",
       "        [-0.6262,  1.0000],\n",
       "        [-0.1999,  1.0000],\n",
       "        [ 0.7904,  1.0000]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.ones(n,2) \n",
    "x[:,0].uniform_(-1.,1)\n",
    "x[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3., 2.])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = tensor(3.,2); a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x@a + torch.rand(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHQJJREFUeJzt3X+QXWV5B/Dvk80FbvzBJrJasrAkzNBQYoas3qG0O6OCDkErYYVYYrXFH52MtnW01rRLdUro1GFtZkrb0alNLf6oVBAC29CgEV0YZhyC7roJIUAkomJu0MSa6w9yTW42T/+45yxnz56f97zn3HPO/X5mGHbPPfc9L2eX5777nOd9X1FVEBFReSzqdgeIiMgsBnYiopJhYCciKhkGdiKikmFgJyIqGQZ2IqKSYWAnIioZBnYiopJhYCciKpnF3bjoOeecoytWrOjGpYmICmt6evpnqjoQdp6RwC4i/QA+C+DVABTAe1X1Ub/zV6xYgampKROXJiLqGSLyoyjnmRqx/wuAr6nqBhE5A8ASQ+0SEVFMiQO7iLwcwOsAvBsAVPUkgJNJ2yUios6YeHh6IYCjAD4nIjMi8lkReYn7JBHZJCJTIjJ19OhRA5clIiIvJgL7YgCvAfBvqjoM4AUAY+6TVHWbqtZUtTYwEJr7JyKiDpkI7IcAHFLVx6zv70E70BMRURckDuyq+hMAPxaRVdahNwJ4Mmm7RETUGVNVMR8EcIdVEfMsgPcYapeIqNAmZurYuusADjeaWN5fxeZ1qzA6PJjqNY0EdlXdA6Bmoi0iojKYmKnjlvv349jx1tyxeqOJm+7dBwCpBveuzDwlIiqysFH4xEwdN927D83W7IL3Nluz2LrrAAM7EVFeuIO21yh8664DnkHddrjRTLWPXASMiCgGr6Btj8JtYYF7eX81lb7ZGNiJiGLwC9rO40GBu1rpw+Z1q3xfN4GBnYgoBr+g7Ty+ed0qVCt9C87pr1Zw63VrilEVQ0RUNn4PSDevW7XgwaignWsfGZ+c9yA16zJHGwM7EZFL1Aek9UYTgvZa5V7nZRXI3ZiKISJyCXtAOjo8iG+NXYnB/upcUPc6r1sY2ImIXKI8IA06z07LTMzUjfctCgZ2IiKXKA9Ig84DXkzLdCO4M7ATEbn4VbW8cOLUvEDtd56tW2kZBnYiIpfR4UHcet0aLF1SmXe80WzNG4Xb5w2GjNyzHrUzsBNRaU3M1DEyPomVYztj57xHhwex5IyFhYPOUbizJLJPxLetrFMyDOxEVEp2yWK90YSis5x30ENUd/uz6q6PeVHWKRkGdiIqpShruoQJeogattCXW9oLfzkxsBNRKUUtWQzi9XDUXuslbqBOe+EvJwZ2IiqlqCWLQZwPRwXAYH91bq0Xv3b6qxXfD4OscEkBIiolrzVdOgmwfksD+LW/Zf1qAN1bJwZgYCeikkp7Ia6w9ru1TgwAiAY8yU1LrVbTqampzK9LRFRkIjKtqqH7SxsZsYvIDwH8CsAsgFNRLkxEROkwmYq5QlV/ZrA9IiLqAKtiiIhKxlRgVwBfF5FpEdlkqE0iIuqAqVTMiKoeFpFXAnhQRJ5W1UecJ1gBfxMADA0NGbosERG5GRmxq+ph699HANwH4DKPc7apak1VawMDAyYuS0REHhIHdhF5iYi8zP4awFUAnkjaLhERdcZEKuZVAO6T9pKViwH8t6p+zUC7RETUgcSBXVWfBXCpgb4QEZEBLHckIioZrhVDRKXi3NWoGwtw5QEDOxGVhr2rkb3ior1rEtDdRbmyxlQMEZWGiV2TyoAjdiIqjaS7JpUljcPATkSlsby/irpHEPfa7cgdxK+4eADbp+ulSOMwFUNEpRG0R6mTnYuvN5pQtIP4HbufK00ahyN2IkpN1qmNqLsmeeXi/bYcirtpdR4wsBNRKoIqVIB0t6wLaytOsI6z+XVeMLATUSr8KlRuuX8/ftM6nUouO+pfCH65eMH8kXsnm1/nAXPsRJQKv1HxseOtVHLZXnnzm+7dh4mZ+tzrI+OTWDm2Ey+cOIVKn8x7f7XSh3dePoTB/ioEwGB/Fbdet6ZwD04BjtiJKCV+o2I/9UYTI+OTHadlwmrYnWmhRrOFyiLB0iUVNI63Cl3a6IWBnYhSsXndqnnBFGiPis9cvAiNZsvzPUnSMkE17F5Bv3VaseSMxZj5u6tiXacImIoholSMDg/i1uvWLEhtbFm/ekFJolOnaRm/h5zL+6uJJy4VDUfsRJSaoAqVrbsO+KZqOgm4fn8hbF63yvdaRax4iYKBnYgS6aRW3Q74I+OTxgJuWA27X9AvI1H1K8tPT61W06mpqcyvS0TJOQP52dUKXjh5Cq3ZF+NItdKH6187iIeePhoa7N217vb706hGKcM6MCIyraq10PMY2IkoKq9A7MWrHtwZrN0fDiIoZXWKaVEDO1MxRBSZV3WJF/dw0X4gOjo8uODDodFsoVrpw203rGVAN4RVMUQUWZIqEvu9XDM9fQzsRBRZlIea4nP87GrF92EpUN7Sw24wlooRkT4AUwDqqvpWU+0SUfrsnHe90USfCGZVMeiR7/YqKawsErz0rMVzOXL3uub2OS+cPOU7MQkob+lhN5jMsX8IwFMAXm6wTSJKmTvnPWsVVHjNAo26LG7tgmXzzjl+8hSOHfcP6mUuPewGI1UxInIegC8A+ASAj4SN2FkVQ5QfQekRoD1j9FtjVya6xsqxnb7rnXv9ZUDesq6K+WcAfw3gZQEd2gRgEwAMDQ0ZuiwRJRWW246zkJcfvwXBTHxo0EKJH56KyFsBHFHV6aDzVHWbqtZUtTYwMJD0skRkSFhuW4C5pW87FXXLOjLDRFXMCID1IvJDAHcCuFJEvmSgXSLKQFhwVSBxKaLfgmBMv6TD6MxTEXkDgI8yx06UnjSmxq+95euBFSsA8MPxP0h0DUouao6ddexEBRK2S1CnwpbSNZGOoewYDeyq+jBr2InSk9asTTtV0l+teL5uIh1D2eGInahA0twwYnR4EHtu9t9NiDNDi4OBnahAgnYJMmUwg2tQuhjYiQokadngxEwdI+OTWDm2EyPjk/Py5vZr9UZzwXovLE0sFi7bS1QgUaf0e3EvHeBcMgCYv8OQ4sU11TkztHgY2IkKJmgf0SBhD17dr9lBnTNDi4eBnahHdPLglQ9Mi4mBnahH+K3XYj8U9Xqtf0l7DfUi7xPai/jwlKhHBD149Xqt0if49W9OGZ8MRenjiJ2oR0R58Op87YUTCzfGcO5dSvlldK2YqLhWDPWaNNZ3SZvfGuoC4AdcN6Yrsl6PnYh8BJUZjg4P5jboh+XkKb8Y2IlSFlZmGBT0TYvzIeK1vyknKhUDAztRCpwB1C/ZebjRDAz6pgN72F8ObkkmQ1F3MbATGeYOoH6W91dTXdTLrZMPkU4nQ1F3sdyRyDCvAOpmpzSyWNTLluWHCHUXAzuRYWGBsk9kblu4LPcCzfJDhLqLgZ3IsLBAeVp1Lr2R5V6g3FC6dzDHTmSYVzWJkzvwp5nHdlfBXP/aQTz09FE+DC05BnYiw+xAedO9j6PZOr3g9SsuHsikH15VMNun66n9RUD5wVQMUQpGhwex7CVner628/HnM+lDWvujUv4lDuwicpaIfFtE9orIfhG5xUTHiIrO7yHqseOtTBbSYhVM7zIxYj8B4EpVvRTAWgBXi8jlBtolKrSgh6hZjJpZBdO7Egd2bfu19W3F+if7lcWIciYol57FqJlVML3LSI5dRPpEZA+AIwAeVNXHTLRLVFQTM3Vsn/ZPt2Qxas6ylJLyxUhVjKrOAlgrIv0A7hORV6vqE85zRGQTgE0AMDQ0ZOKyRLkVNPs0y1EzlwToTUarYlS1AeBhAFd7vLZNVWuqWhsYyKbci6hbglItHDVT2kxUxQxYI3WISBXAmwA8nbRdoiLzS7UM9lcZ1Cl1JlIx5wL4goj0of1B8RVV/V8D7RIZFWUtclObXphYyzyvG3BQ/iUO7Kr6OIBhA30hSk2UtcjjrlcexLmWeb3RRJ/IvMlBYe2Z7Av1Hs48pVKbmKljZHwSH75rT+gsTNMzNZ2rN85aewvbATpsghJnjVISDOxUWvao12vfTpvzIWcaMzU7DdCcNUpJMLBTaUXZ8ML5kDONmZqdBmjOGqUkGNiptMKCp/thZhozNTsN0Jw1SkkwsFNpBQVPr1mYaczU7DRAc9YoJcH12Km0/EoOgwKks5rlcKMZuYrFj7u9OGWLnDVKnWJgp9Jx1n+fXa3grMoiNI63IgXVNMoMGaApawzsVCruwNxotlCt9OG2G9ZGCq5BVSwMzlQUDOxUKkkDc1gVi3s26BUXD3APUcodBnYqhKjT65PWfy/vr3rWvS/vr3qmab60+7m5czg7lPKCVTGUe86JRorg2ZtJ67+Dqlii1MVzdijlAQM75V6c2ZtegVkQvJuRU1CZYdRRP2eHUrcxFUO5Fye9Mjo8iKkf/Rx37H5ubn9GBbB9uo7aBcsSlRn6pWm8ziPqJo7YKffiplceevrogk13TaRIvP4acOPsUMoDBnbKvbizN9NaQMsrTfOuy4c4O5Ryh6kYyr24szeDKltM9IWBm/KOgZ0KIU5ANbF7EVGRMbBTKbjr3K9/7SAnDlHPYmCnwvOaOLR9ut5Rvpv7jFIZMLBTZtIKmqbWd+E+o1QWrIqhTMSZPRqXqSoY7jNKZZE4sIvI+SLykIg8JSL7ReRDJjpG5RIWNO1Np1eO7cTI+GSsgG9qGznuM0plYSIVcwrAX6nqd0XkZQCmReRBVX3SQNtUYM7Ui3vCkO1wo9lxCsRuv95oQoB51+ikCibNMkmiLCUesavq86r6XevrXwF4CgATkj3OnXrxs7y/2lEKxNk+0A7qYr3W6UQh7jNKZWE0xy4iKwAMA3jM47VNIjIlIlNHjx41eVnKoSgrIdpBs5MUiFf7inZQ/9bYlR097OQ+o1QWxqpiROSlALYD+LCq/tL9uqpuA7ANAGq1WtAgjkogKCgLMK8qxk6nuAWlQII+DJJU33BmKZWBkcAuIhW0g/odqnqviTap2Pzy1faI2qmTmaJ+7Z9drbBkkXqeiaoYAfCfAJ5S1X9K3iUqgzj56k5SIH7ti4Ali9TzTIzYRwD8MYB9IrLHOva3qvqAgbapoIIW7vJLlXgF8rBz3a/95V17FrQBsGSReouoZp/urtVqOjU1lfl1qfvcpY1Ae6TtNUKPc65tZHwycgqIqGhEZFpVa2HnceYpGRNlklGc0ka/c7fs2O97HZYsEnGtGDIk6iSjOKWNfuc2mi00mi3P68Rdu52ojBjYyYioC3HFmd0ZdY9R93VYski9jqkYMiLqSDxOqiTKHqNh1yfqRRyxUyRhk36ijsTjpEq8zj1+8hSOHW+FXoeolzGw94CJmTq27Ng/l5deuqSCm69ZHZqucC6y5VRvNLH5nr0AXgy+cSYZxUmVuM/1q5Thw1GiFzEVU3ITM3VsvnvvXFAHgGPHW9h8z97ApXHdi2y5tWYVt9y/f+77rNZZ4XouROE4Yi+5rbsOoHV64VyF1qwG7jAUZREvd0okq4eWfDhKFIwj9pILeqgYVHHCh5FExcXAXnJBDxUF8E3HRHkY2V+tdNotIkoRA3vJbV63CpVF4vmaAr6LY4WVGlYWCbasX22ii0RkGHPsJWfnoj8cc3Esd6lh/5IKVIFfNFuczUmUcwzsBdJp2WKnm1nwISVRMTGwF4RdtuiscLHLFgH/TSRMb/hMRPnHHHtBhJUtekljw2ciyj+O2AsiqPzQ77WwDZ+JqJw4Yi+IoFy432txlsglovJgYC8Iv7LFSp/45sn9Aj4XzCIqNwb2ghgdHsTWt186b1LQ0iUVbN1wqW+enLsJEfUm5tgLJG75IXcTIupNRgK7iNwO4K0Ajqjqq020SWawFp2o95gasX8ewKcAfNFQe5RjYZtuEFF3GQnsqvqIiKww0Ra15TV4Rt20moi6hzn2LvMK4AByGzyjblpNRN2TWWAXkU0ANgHA0NBQVpfNNb/Rr0DRbJ2ed25egidr44nyL7PArqrbAGwDgFqttnBufA/yG/36cQbPNFI1UdqMumk1EXUPUzEpCguUcUe5dvBMkuf261PUNuNsWk1E3WFkgpKIfBnAowBWicghEXmfiXaLzLkAl+LFQOncsSjuKNcOnn4jfefm0nH7FJQ7d+Jm0kT5Z6oq5h0m2imTKA8ZvUa/7qV1bUuXVOZG1n57lR473sLETD3WBtV2n+LkzlkbT5RvXFIgJVECpdfo952XD3kuA3DzNavnRtxB/JbwDesT15UhKg/m2FMS9SGj1+i3dsEyzzz4yPhk4MNVwDt426kWvyfW9jWYOycqBwb2lCQJlH6pjigPW90fHO6Hom52n7iuDFF5MLCnJI1A6fdXgM3rg8Mrr24bdPWJuXOicmBgT1GSQOlVlhj0sNUdpG1+o3wBuIsSUUkxsOeQe+PqeqOJzXfvxda3X4pbr1sT668ATigi6j0M7Dm0Zcf+BRtXt04rtuzYjz03XxXrrwA+FCXqPQzsOdRotmIdB/xnlPKhKFHvYWAvgbDlAPhQlKi3cIJSRiZm6hgZn8TKsZ0YGZ+ct7SA29IllVjHoy4HQES9gYE9A/bDUOcaLZvv3usb3G++ZjUqfTLvWKVPcPM1qz3P51K6ROTEwJ6BoIehXkaHB7F1w6XzlhrYuuFS33QKlwMgIifm2DPQycPQOHlxVr4QkRMDe450unkGK1+IyImBPQNLl1Rw7PjC0bnzYWjSTaJZ+UJENubYMxDlYSgrW4jIFI7YMxAlVcLKFiIypfCBPY1NndNoPyxVwjVdiMiUQqdiouwrmuf2nTavW+W5cxIrW4gorkIH9rTz0lnmvblJNBGZUuhUTNp56azz3qxsISITjIzYReRqETkgIgdFZMxEm1GkPeOSMzqJqIgSB3YR6QPwaQBvBnAJgHeIyCVJ240i7bw0895EVEQmUjGXATioqs8CgIjcCeBaAE8aaDtQ2jMuOaOTiIrIRGAfBPBjx/eHAPyu+yQR2QRgEwAMDQ0ZuGxb2nlp5r2JqGhM5NjF45guOKC6TVVrqlobGBgwcFkiIvJiIrAfAnC+4/vzABw20C4REXXARCrmOwAuEpGVAOoANgL4IwPtGpP27FQiojxJHNhV9ZSI/AWAXQD6ANyuqt47SHTBxyf24Y7dz83lhuKummjjhwMRFYWRCUqq+gCAB0y0ZdLETH1eULfZs0ejBuakS+oSEWWp0DNPw2zddWDhU1yLPXs0ykg8aGkBBnYiyptSB/agqf/L+6uRR+Jeqy4GHSci6qbCLgI2MVPHyPgkVo7txMj4pOeKi35T/wXtWaVRF/nqE6+KTv/jRETdVMjAHnU5Xa8lAQTAOy8fwujwYORFvmbVO6Hjd5yIqJsKGdijjrS9lsK97Ya1+IfRNQCiL/I16HOe33Eiom4qZI49znK6QUsCbF63al6OHfBe5CvqeUREeVDIwG5qG7moi3xxMTAiKpJCBvZOR9B+pY1RAjQXAyOioihkYO9kBM1JRkTUKwoZ2IH4I2hOMiKiXlHIqphOZL1/KRFRt/RMYOf+pUTUK3omsHP/UiLqFYXNscfFkkUi6hU9E9gBliwSUW/omVQMEVGvYGAnIiqZQqZiuE0dEZG/wgV2U3uYEhGVVaFSMWF7mBIRUcLALiJvF5H9InJaRGqmOuUnyh6mRES9LumI/QkA1wF4xEBfQoXtYUpERAkDu6o+paqZ5UDC9jAlIqKC5djD9jAlIqIIVTEi8g0Av+Xx0sdU9X+iXkhENgHYBABDQ0ORO+jEZQGIiMKJqt/jyBiNiDwM4KOqOhXl/FqtplNTkU4lIiKLiEyramihSqFSMUREFC5puePbROQQgN8DsFNEdpnpFhERdSrRzFNVvQ/AfYb6QkREBjAVQ0RUMgzsREQlY6QqJvZFRY4C+FGMt5wD4GcpdSeJvPYLyG/f8tovIL99y2u/APatE0n6dYGqDoSd1JXAHpeITEUp8claXvsF5Ldvee0XkN++5bVfAPvWiSz6xVQMEVHJMLATEZVMUQL7tm53wEde+wXkt2957ReQ377ltV8A+9aJ1PtViBw7ERFFV5QROxERRZSLwB51JyYRuVpEDojIQREZcxxfKSKPicgzInKXiJxhsG/LRORBq+0HRWSpxzlXiMgexz+/EZFR67XPi8gPHK+tzbJv1nmzjuvvcBxP5b5FvGdrReRR6+f+uIjc4HjN6D3z+71xvH6m9d9/0LofKxyv3WQdPyAi65L0o8O+fUREnrTu0TdF5ALHa54/1wz79m4ROerow586XrvR+vk/IyI3Ztyv2xx9+p6INByvpXbPROR2ETkiIk/4vC4i8q9Wvx8Xkdc4XjN7v1S16/8A+B0AqwA8DKDmc04fgO8DuBDAGQD2ArjEeu0rADZaX38GwAcM9u0fAYxZX48B+GTI+csA/BzAEuv7zwPYkNJ9i9Q3AL/2OZ7KfYvSLwC/DeAi6+vlAJ4H0G/6ngX93jjO+TMAn7G+3gjgLuvrS6zzzwSw0mqnz+DPL0rfrnD8Ln3A7lvQzzXDvr0bwKc83rsMwLPWv5daXy/Nql+u8z8I4PaM7tnrALwGwBM+r78FwFfR3kbicgCPpXW/cjFi12g7MV0G4KCqPquqJwHcCeBaEREAVwK4xzrvCwBGDXbvWqvNqG1vAPBVVT1usA9+4vZtTsr3LbRfqvo9VX3G+vowgCMAQidedMDz9yagv/cAeKN1f64FcKeqnlDVHwA4aLWXWd9U9SHH79JuAOcZvH6ivgVYB+BBVf25qh4D8CCAq7vUr3cA+LKhawdS1UfQHtT5uRbAF7VtN4B+ETkXKdyvXAT2iAYB/Njx/SHr2CsANFT1lOu4Ka9S1ecBwPr3K0PO34iFv0ifsP70uk1EzuxC384SkSkR2W2niJDufYt1z0TkMrRHX993HDZ1z/x+bzzPse7HL9C+P1Hem0Tc9t+H9ojP5vVzzbpv11s/p3tE5PyY702zX7DSVisBTDoOp3nPwvj13fj9SrS6YxySfCcm8TimAceN9C1mO+cCWAPAuXzxTQB+gnbg2gbgbwD8fcZ9G1LVwyJyIYBJEdkH4Jce50W+b4bv2X8BuFFVT1uHE90z9yU8jrn/O1P73QoRuX0ReReAGoDXOw4v+Lmq6ve93p9S3+4H8GVVPSEi70f7r54rI743zX7ZNgK4R1VnHcfSvGdhMvs9yyywq+qbEjZxCMD5ju/PA3AY7TUX+kVksTXaso8b6ZuI/FREzlXV560gdCSgqT8EcJ+qthxtP299eUJEPgfgo1n3zUp1QFWflfZuV8MAtiPBfTPRLxF5OYCdAD5u/Wlqt53onrn4/d54nXNIRBYDOBvtP6mjvDeJSO2LyJvQ/sB8vaqesI/7/FxNBanQvqnq/zm+/Q8An3S89w2u9z6cVb8cNgL4c+eBlO9ZGL++G79fRUrFfAfARdKu5DgD7R/aDm0/fXgI7dw2ANwIIPJerBHssNqM0vaCfJ4V2Oyc9igAzyfmafVNRJbaqQwROQfACIAnU75vUfp1Btpr+X9RVe92vWbynnn+3gT0dwOASev+7ACwUdpVMysBXATg2wn6ErtvIjIM4N8BrFfVI47jnj/XjPt2ruPb9QCesr7eBeAqq49LAVyF+X/Fptovq2+r0H4Q+ajjWNr3LMwOAH9iVcdcDuAX1iDG/P1K6wlxnH8AvA3tT60TAH4KYJd1fDmABxznvQXA99D+hP2Y4/iFaP8PdxDA3QDONNi3VwD4JoBnrH8vs47XAHzWcd4KAHUAi1zvnwSwD+3g9CUAL82ybwB+37r+Xuvf70v7vkXs17sAtADscfyzNo175vV7g3ZqZ7319VnWf/9B635c6Hjvx6z3HQDw5hR+98P69g3r/wn7Hu0I+7lm2LdbAey3+vAQgIsd732vdT8PAnhPlv2yvt8CYNz1vlTvGdqDuuet3+tDaD8TeT+A91uvC4BPW/3eB0cFoOn7xZmnREQlU6RUDBERRcDATkRUMgzsREQlw8BORFQyDOxERCXDwE5EVDIM7EREJcPATkRUMv8PynFmgCx5CLQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(x[:,0], y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to find **parameters** (weights) `a` such that you minimize the *error* between the points and the line `x@a`. Note that here `a` is unknown. For a regression problem the most common *error function* or *loss function* is the **mean squared error**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mse(y_hat, y): return ((y_hat-y)**2).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we believe `a = (-1.0,1.0)` then we can compute `y_hat` which is our *prediction* and then compute our error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = tensor(-1.,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(7.5648)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_hat = x@a\n",
    "mse(y_hat, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X90HeV5J/DvI3GBa0Iku6hZLCxszrJmwy8r6GTZuqcJOMuPBoxqCJBuWmhonDTpD8rGi2l6wLCbxazPKWlPcrbrpoR0kxIMGNXgUEOwKbucmFaqsMGAgwMBLNPgBOQGW9jX0rN/zIw0mjs/77wz987c7+ccjqV75868uVKe++qZ531eUVUQEVF5dDR7AEREZBYDOxFRyTCwExGVDAM7EVHJMLATEZUMAzsRUckwsBMRlQwDOxFRyTCwExGVzDHNuOhJJ52kCxcubMaliYgKa2Rk5Geq2hN1nJHALiLdAL4J4CwACuCzqvrDoOMXLlyI4eFhE5cmImobIvJ6nONMzdj/HMDfq+pVInIsgDmGzktERAmlDuwi8kEAvwbgegBQ1SMAjqQ9LxERNcbEzdPTAOwH8C0RGRWRb4rICd6DRGSliAyLyPD+/fsNXJaIiPyYCOzHAPgIgP+lqv0ADgJY7T1IVder6oCqDvT0ROb+iYioQSYC+14Ae1X1Wfv7B2EFeiIiaoLUgV1V/wXAmyKy2H5oGYAX056XiIgaY6oq5g8AfNeuiHkVwO8YOi8RUWENjY5h3Zbd2Dc+gfndVay6eDEG+3szv66RwK6qzwEYMHEuIqKiGxodw+2P7MK7h2rTj42NT+CWjc8DQObBvSkrT4mIiijODHxodAy3bHweE7XJutdP1CaxbstuBnYiolbgDdhBM/B1W3b7BnXHvvGJbAcKNgEjIorFL2A7M3C3qMA9v7tqfGxeDOxERDEEBWzv42GBu1rpxKqLFwc+bwoDOxFRDEEB2/v4qosXo1rprDuuu1rBnSvOLk5VDBFRmfjdJF118eK6m6KVDsGhI0exaPXmupupzShzdIiq5nYxx8DAgLJtLxG1Ir+qlmqlE3euOBvATMDuqlZw8MhR1Ca17risgriIjKhqZGk5UzFERC5hN0kH+3vxzOoL8draT+KE446ZFdTdxzUbAzsRkUvcm6RBx42NT2BodMz4uJJgYCcicol7kzSs+uWWjc83NbgzsBMRufhVtQiAC87oiTzO0eyUDAM7EZHLYH8vrjyvF+J6TAE8NDI2axY+2N87fUPVTzNTMix3JKJSStNZcdvL++GtF/T2eXHOHyavpl9enLETUek4JYtj4xNQzPR1iTuDjrqB6j5/mGalZBjYiah04vZ1CRJ1AzWq0ZdbHk2/vBjYiah04pYsBvG7Meru85IkWOfR9MuLgZ2ISiduyWIQ58Zob3cVAqC3uzprRWnQecTzfV5Nv7x485SISsevr0vSIDvY3xt40zPo/Fee14ttL+9vWo8YBwM7EZVO1o24WqHRVxg2ASMiKoi4TcCMzNhF5CcAfgFgEsDROBcmIqJsmEzFXKCqPzN4PiIiagCrYoiISsZUYFcAj4vIiIisNHROIiJqgKlUzFJV3ScivwzgCRF5WVWfdh9gB/yVANDX12foskRE5GVkxq6q++x/3wbwMICP+hyzXlUHVHWgp6fH+zQRERmSOrCLyAkicqLzNYCLALyQ9rxERNQYE6mYDwF4WESc8/2tqv69gfMSEVEDUgd2VX0VwLkGxkJERAaw3JGIqGTYK4aISiPNrkllwsBORKXg7GrkdFx0dk0C8t+artmYiiGiUki7a1KZcMZORKWQdtekMqVxGNiJqBTmd1d9N5f27nbkF8ABlCqNw1QMEZVC1D6lwEwefmx8AoqZAH77I7tKlcbhjJ2IMpF3aiPOrkZBeXjvY44km1a3EgZ2IjIurEIFyHbLurBzJQ3UcTe/bjUM7ERkXNDM+PZHduH92lQmuew4fyEE5eG7qxUcPjqVavPrVsIcOxEZFzQzfvdQLZNcdlDufGh0bPr5pWu3Ymx8AuJ5bbXSiTXLz8SdK85Gb3cVAqC3u4o7V5xdyBunAGfsRJSBoJlxkLHxCSxdu7XhtExUDbs7LaQAxP631zOzL2og9+KMnYiMC6pQ6a5WAl/jnWUnEVbD7hf0naD+zOoLSxPM3RjYici4wf5e39TGmuVn1gV8t0bTMkE3Oed3V1MvXCoipmKIKBNhFSrrtuwOTNU0EnBXXbx4VroFmLn5GXStola8xMHATkQNa6RW3Qn4zs1Mr0YCblQNe1DQLytR1dwvOjAwoMPDw7lfl4jScQfyrmoFB48cRW1yJoZUK5248rxebHt5f2Sw99a6O6/PohqlLH1gRGREVQcij2NgJ6I4/AKxH6fixOEN1t4PBxFg/FCt0AE3L3EDO1MxRBSLX3WJH+9U0bkhOtjfW/fhMD5RQ7XSibuvWcKAbhCrYogoljRVJM5r2TM9HwzsRBRLnJua3lWdjq5qJfBmKVDu0sNmMJaKEZFOAMMAxlT1MlPnJaJsuXPe3XMqUAUOTNTnvP1KCisdgg8cf8x0jvyCM3rw0MhY3TEHjxzF+EQtcAxlLj1sBpM59j8C8BKADxo8JxFlyJvzfvfQTPD1NuiK0xYXAAZOnTfrmENHjs46r1fZSw+bwUhVjIicAuDbAL4K4KaoGTurYohaQ1h6xOEsvW/UotWb626ous/NSpj48q6K+RqA/wrgxJABrQSwEgD6+voMXZaI0oiT206b/w5qCJb2A4OCpb55KiKXAXhbVUfCjlPV9ao6oKoDPT09aS9LRAbEyW13zwlu3BVHnC3ryCwTVTFLASwXkZ8A+B6AC0XkOwbOS0QZixNc02ZrgxqCMf2SHaMrT0Xk4wC+zBw7UTayWBq/5PbHQytWAOAnaz+Z6hpkRtwcO+vYiQoiapegRkW10hX72lQcRgO7qj7FGnaibGS1atNJlQRtgqH2tak4OGMnKogsN4wY7O/Fc7ddlPja1JoY2IkKImyXIFN6c7gGZY+Bnagg0pYNDo2OYenarVi0ejOWrt06K2/uPDc2PlHX74WlicXDtr1EBRF3Sb8fb+sAd7sAYPYOQ4qZnupcGVpMDOxEBRK2j2iYqBuv3uecoM6VocXEwE7UBhq58cobpsXFwE7UBoL6tTg3Rf2e655j9VAv+j6h7Yg3T4naQNiNV7/nKp2C994/anwxFOWDM3aiNhDnxqv7uYOH6zfGcO9dSq3NaK+YuNgrhtpNFj1eshTUQ10AvMa+MU2Tdz92IgoQVmo42N/bkkE/KidPrY2BnShjUaWGYUHfpCQfIH77m3KhUnEwsBNlwB1Eg5Kd+8YnQoO+ycAe9VeDV5rFUNR8DOxEhnmDaJD53dVMG3u5NfIB0uhiKGo+ljsSGeYXRL2ctEYejb2AbDtDUuthYCcyLCpYdopMbw2X136geX2AUGtgYCcyLCpYTqlOpzjy2g+UG0q3F+bYiQzzqyhx8wb+rHLZ3iqYK8/rxbaX9/NmaBtgYCcyzAmWt2zciYnaVN3zF5zRk/kY/KpgHhoZy+SvAWo9TMUQZWCwvxfzTjjO97nNO9/K/PpZ7Y9KxZA6sIvI8SLyjyKyQ0R2icjtJgZGVHRBN1HfPVTLvJkWq2Dam4kZ+2EAF6rquQCWALhERM43cF6iQgu7iZr1zJlVMO0tdWBXy3v2txX7v/w7ixG1kKHRMRw8fDTw+axnzqyCaW9Gbp6KSCeAEQD/FsA3VPVZE+clKqI4K0+znjmzJUB7MxLYVXUSwBIR6QbwsIicpaovuI8RkZUAVgJAX1+ficsStaSolad5zZzZEqB9Ga2KUdVxAE8BuMTnufWqOqCqAz092Zd7ETVLWJolqwVIRG6pZ+wi0gOgpqrjIlIF8AkAd6UeGVFBBfUy7+2u4pnVFzZhRNRuTKRiTgbwbTvP3gFgg6o+auC8REbF6UduYtMLE73MW3HzDSoObo1HbcHvhma10jkrLRLnmCTXcwJz95wKVIEDE7VYQdrkOKhc4m6Nx5WnVGpDo2NYunYrbrz/uciVmCZXaw729+KZ1Rfi7muW4P3aFMYnalDMbHARtkCJq0YpLQZ2Ki1n5uuX73a4b3RmsVqzkSDNVaOUFgM7lVacDS/c9eRZrNZsJEhz1SilxcBOpRU1w/Xe0MxitWYjQZqrRiktBnYqrbDg6VdPnsWmF40E6bw236DyYj92Kq2gssOwIOldiu/kwhsNqo0u7eeqUUqDgZ1Kx11q2FWt4PhKB8YPNVZq6FSxAOmCO4M05YmBnUrFG5jHJ2qoVjpx9zVLYgXXsCoWBmcqCgZ2KpW0gTmqisW7IvSCM3q4jyi1HAZ2KoS4S+zT1oAH9XmZ3131TdN8Z/sb08eYSNsQmcCqGGp57oVGUas309aAh1WxxKmL5wpRagUM7NTykqze9AvMAuCCM+K1ig4rNYw76+cKUWo2pmKo5SVJrwz292L49Xfw3e1vTO/PqAAeGhnDwKnzYqVIgqpYgtI0fscRNRNn7NTykqZXtr28v27TXRMpEr+/Bry4QpRaAQM7tbykqzezaqLll6b5zPl9XCFKLYepGGp5SVdvhlW2mBgLAze1OgZ2KoQkAdXEDkZERcbATqWRppUAUZkwsFMppG0l4D4P9xqlomNgp9xkGTRN9HjJogEYUTOwKoZykWT1aCNMVMJwr1Eqi9SBXUQWiMg2EXlJRHaJyB+ZGBiVS1TQdDadXrR6M5au3Zo44JvYTo57jVJZmEjFHAXwX1T1n0XkRAAjIvKEqr5o4Nz+dm4AHrsZmHjH+r46D7j0LuCcqzO7JCXnTr14Fww59o1PNJwC8d4srXQKapMzV0paCZNlmSRRnlLP2FX1LVX9Z/vrXwB4CUB2CcmdG4ChL84EdcD6euPngDXdwJou4O6zrOOoabyplyDzu6sNpUC85x+fqAEKzJ1TaXixEPcapbIwevNURBYC6AfwrM9zKwGsBIC+vr7GL/LkHcBULeBJO4QceBN45A+trzmLb4o4nRCdoPnH9z/n+3xYCsTv/LUpxZxjj8HorRclHzAa38aOqNUYC+wi8gEADwG4UVX/1fu8qq4HsB4ABgYGwiZx4Q7sjXdcbcL6EDjn6vrUjXQAOgV0LQCW3crgn4GwoCzArKC5bsvuxCmQsHx4muobriylMjAS2EWkAiuof1dVN5o4Z6CuU6wZeRwH9s6kbtyzfJ2yn3fN7AHrg+DAXusaDPipBOWre7ureGb1hbMea2SlaND5u6oVlixS2zNRFSMA/hrAS6r6Z+mHFGHZrUBHJd6xXadEpG5gzewfu9kK8AfeBKAzAZ95+oYlyVeH9UBPen4RsGSR2p6JGftSAL8F4HkRcZKlf6Kq3zdw7nrOLNqdWvFTqVofAhtXRp/T7zzeVM6Td8z+S0E6gfOuBy7L/rOsiMLy1UGpEr9AHnWs97lG8vVEZSOqjae7GzUwMKDDw8NmTuYOutIJ6OTs3PndZ8VP3dQRYMV6a/ZeCwgMAzcwuCfgLW0ErJm23ww9ybGOpWu3xk4BERWNiIyo6kDkcYUP7FH8cuxulSpwTNV/1t61wPo37INBOoHbQv5yaDNRNy6TBN6wY509SP3+Gkj6YUBUFHEDe/l7xfilbrxVMUD9rDxuKkfDS/oAtE0qJ85CoySrO4OOdc4bdh2WLFI7K39gB6zgHqfCxa8qxhuQvSR8qzTs3OCfytFJYPivra/7zi9FRU6cRlxJVncGHdspEnodlixSu2uPwB5HUPBfdmt4jv2868PP++Qdwa8FgJFvATv+duaYA29afyVs/Fz9sYs+Bly3Kfx6TRRnNp6ktDHo2KCFT7xBSmRhYI/iBPtGUylRC6p0yifwB9z3eO0fgP/WA0zWmjKzj8qfx5mNJ0mVBB3byIImonZS/punzZaqKieBiA+aodExrNm0y+qpAqunym2Xn5moydYvDh/F5NTM70ulU7DuqnOnz5HXjUveIKV2FffmKfuxZ23ZrdaN2CCVE8xcx8nZf3W+1QzN1QhtaHQMqx7YMR3UAeDdQzWsenBHYHtcvyZb7qAOALVJxe2P7Jr+vpGFRo3I6zpERcVUTNaiUjl954fn8JOqHbT+dbVLmNz0EF6ubEGHK8UzBcF3Jpdh3ZYv+QbEOE28AOsDwi2vG5e8QUoUjIE9D3GqcqYDvyAwx55UbQJ45EasmDoIkdlPdULx250/wIpD/xfY+fW68fFGJFFxMbC3Anfg96t5n9ZA0K/VB/Xpswlwohz2bYT2w+NPwv848ilsmvrV0NN3V2P27SGi3PDmaZHs3AA8cuNMusWk6jzg6MSslNAR7cR7qKIb72GfnoRX9UP4lY6X0IkpTKID900tw4kr/pwpEaKcsKVAmT16EzBy7+xVr9JhT+anZh6rVIGjh2faFKeksP5mmPX9sScAl32tkAuqiIqGLQVKyCk/HBtfhk75BCZVp/umDPb3utI4rhWsb2yfWeGakjejIwBw5ODs3ar8xsCgT5QrztgLwq922xFVw/3jb30eC1+/Hx2un/WsvHtYI7S4nL47dRU+9n2BrgXA6RcBrzzOoE/UIKZiSiao06EjqC2t3wfC8o7/h9XHbsDJ+DnECbBAyrJLSba7laM6D7j0LgZ4ohiYiimZqPLDoOf96tE3Tf0qRo7/T/79yZ00SnUucOQ9YPJIvAF2nRJ/P1q3iXesD5Q3tnM2T2QIA3tBBPVhcT/vJ0mb3Lp6e2++fN5pwGtPo67k0mlxHNUJM0htAhi+Z+a8vnvRBmykQkR1GNgLwq/ToSNs4+ckbXLrBC2sCrtB2nA6x/Nh4exF6y7BdKqADrwJ/N2XZpd+MqVDNI059gKZqYqZQKdIfVVMwGtybZg1a4GVwVW0sXQA1W4rvcPZPZUQb57StKh2u5nxzuxPvwjY9bBP9U1OHwAM8lRwuQZ2EbkHwGUA3lbVs6KOZ2Bvc34B373ZCGCmBNOXq/ySQZ4KJu+qmHsBfB3A3xg6H7Ww1H8B+OXu/bYHBMx2vgTge4PWPRb3qt4S7ktL7cFIYFfVp0VkoYlz0YympVAixhS1kXRDwjpgeqtiqvOA9w/E20g8TG3COrdz3Udvmr1K1+lx//M9Lb0lIZEXq2JagF8AB5BNAE0pzobVRoVV5jx280yqpnICMFWLX3fvcNfej9zrf8xr/2Bdjy0TqCByC+wishLASgDo6+vL67ItL2gGLFBM1GY378o0gMaUqC4+S34B3xvs4+g6ZebrsL8AnrzD+tedGjrwJjD0Rfua7zLQU8vILbCr6noA6wHr5mle1211QTPgIO4AajpVE+d8qeris+YEe2+Alw67w6Wn+sZZWOVwUj1+Duy1grs33z9Vm7mOO2//xvb6DpyOgRuYt6dMMRWTg7CAmXSm6wTQNLnuNKkfv4VSYQukmqKRhVWAdaM0qBNm3JYJtQng0RutrpdBmLenjJkqd7wPwMcBnATgpwBuU9XAXrHtVO4YtUAoqrmX19euWRL6uqBmYFHjOb7SUbd/adD5WvGmrjHfXm7l1N0qVeDyv2i8ZUKQFX/FvD0lkmu5o6p+2sR5yijqZqPfDDhouc7cORUM9vdiaHQs8MMg6kMiaDxB6R+/vyhKvZH0dZsyapngIyhvz6ZolBJTMRmLutnoBEj3DPiCM3rw0MhY3az6tsvPnJ5xB+kM2uA0YjxBWiJ3nregVI7zWFgHzEoVqL2PWCtpg/L2sZqiMeBTMAb2jMW52eg3Ax44dZ5vumPp2q2hN1cnfVJr7tRJh91jxqu7WsHho1OtnTtvBVEdMJPsWhWat4/RFC1okRW1PQb2jDV6szEo3RE14+71zLC9OXW/oF6tdGLN8jMBoLy586z4ze6d74OqYoDGWh37lXF6F1m5MXffthjYM+aXakkTMMP6svt9YPjl1AErZTOlWjceBnJDLvuz2SWNifL2CZui+c36d26oz91v/Dzw8Oet0k+2Syg1dndsYXHKEh3d1QrWLD+zLjAvWr3ZN0QIgNfWfjKjkVMiaZuidS0A/viF2Y/dfVa8vwQGbrD69Mwq0RRg4LMM+i2IbXsLbmh0DKse2IHa1MzPp9IhWPepcwHE/wug0bJIajK/GT5QP7t3SjG9KZY13Yg165cOa2fzqYi+O+yG2RIY2Atuye2PY3yivq68u1rBc7ddFPs8uW+0QdmKmzePO2NPolIFzv1NlmE2ETezLji/oB72OBC+cIg3RUsirAum27Jbzbc89ivDZK+clsTAXhJRLQYYyNuMt+a+Mmdmf1i3ygn+jwfy/IXv1ytnenEVNyBvFqZicpRkKX7/HY/7LvGfO6eC0VvrUzHMpVMkv01E+s4Hhr4QnWNPJKiqx368Os/6lrP8xJiKaTHem6Fj4xNY9cAOAP4lhrddfiZWPbgDtUnXzdNOwW2Xn+l7/pZpp0uty1uC6RbVuAxA/DLMoGPsx92VPQfeBDZ+zvqPs3pjOpo9gHaxZtOuWRUuAFCbUqzZtMv3+MH+Xqy76lz0dlchsGbe6646N3CGH7T0vy1bAlAy51wN/Mk+YM0Bq/xRfMJC1wKrBLKS4e+Tk8rZuSG7a7QJzthz0sjN0CS58UK006XWFzarB2bvTevXKyfp4iqvsJW0FBsDe4tptCUuq18oF1G9cvwWVyUVp+89hWJgz8ncOZXAm6GOtBtFs/qFcudXfjk9q3dVxSSZybu3K0zCu3NWdR5w6V1tOftnVUxOhkbHfG+GuvPmrGyh0pqe2UcsmgpaSRvn/ENftMov/ZSkEiduVQxvnuYkzs1QVrZQaZ1ztdXPZs0Ba+eorgUAxAq41XnW110LGgvqgPWhERTUAWsWP/EOALUrcVYCa7qsFbolvFlbmlRMltu1mTp3VKqkpTeKJjIl7urZJBLn5SM2ManOtb4v6Ay/FDN2Jzc9Nj4BxUxuemh0rKXP7bXq4sWoVjpnPcbKFqIYGs3LAzObmDzyh3aqSP1n+I/eZGq0mSvFjD1qX9FWPbcXK1uIGrTs1vAcexS/dsizqN0nBzNN0KpzgaOHZ1oytNDN2lIE9ixz03nnvVnZQtQAJ5i6q2KM09lN0LzXmXjHWkHrbGbSxJW0RlIxInKJiOwWkT0istrEOZPIctUlV3QSFcQ5VwM3v2bdoA28SQvre7dK1fVclBhVhDpl/eu0S1jTnfuN2tSBXUQ6AXwDwKUAPgzg0yLy4bTnTSLL3DTz3kQFNV2JM24FfCfor1g/E/CdSpxL78qwXYLnRm0Owd1EKuajAPao6qsAICLfA3AFgBcNnDuWLHPTzHsTlUxYVY5TFXPsHJ+maCnbJQC5tUxIvUBJRK4CcImq/q79/W8B+A+q+vue41YCWAkAfX19573++uuprktElKks2iUAAMT6K6KRV+bYtld8Hqv7tFDV9QDWA9bKUwPXJSLKTmi7BJ+qmLjSlGbGZCKw7wWwwPX9KQD2GTgvEVFrCUrjuFsmSMfMDVSvSnVmY/IMmQjs/wTgdBFZBGAMwLUAftPAeY3KcmUqEbW5wK6XzdkeMHVgV9WjIvL7ALYA6ARwj6r67x7RJH869Dy+u/2N6fxQ0q6JDn44EFEsWbRNSMDIAiVV/T6A75s4l2lDo2Ozgroj6erRtC11iYjyUoqVp2HWbdkdWKDkXj0aNRvPs7UAEVEapQ/sYUv/ndWjcWbjfl0Xwx4nImqWwnd3HBodw9K1W7Fo9WYsXbu1ruti0NJ/AaZXj4bNxh2d4lfVGfw4EVGzFDqwx2mp69cSQAD85/P7pmfjcRp9TQYs5Ap6nIioWQod2OPMtAf7e3HnirNn7Vx09zVL8N8Hz54+Jk6jr96AY4IeJyJqlkLn2OO21I1qhbvq4sWzcuxAfaOvOMcQEbWCQgd2U1vJxWn0xWZgRFQUqZuANWJgYECHh4dTn8dbzQJYs+g7V5wdGHC5yIiIiirPJmBNk3QWzUVGRNQOCh3YgWRbyXGRERG1g0JXxSSV9/6lRETN0FaBnfuXElE7aKvAzv1LiagdFD7HngRLFomoHbRVYAeS3WwlIiqitkrFEBG1AwZ2IqKSKWwqhitIiYj8FTKwm9rDlIiojAqXionaw5SIqN2lCuwi8ikR2SUiUyIS2ZjGhLh7mBIRtau0M/YXAKwA8LSBscQSZw9TIqJ2liqwq+pLqppr/iPOHqZERO2scDn2OHuYEhG1s8iqGBH5AYB/4/PUV1T17+JeSERWAlgJAH19fbEH6MW2AERE4YzsoCQiTwH4sqrG2hbJ1A5KRETtJO4OSoVLxRARUbi05Y6/ISJ7AfxHAJtFZIuZYRERUaNSrTxV1YcBPGxoLEREZABTMUREJcPATkRUMkaqYhJfVGQ/gNcTvuwkAD/LYDhpteq4gNYdG8eVDMeVTKuOC0g/tlNVtSfqoKYE9kaIyHCcMp+8teq4gNYdG8eVDMeVTKuOC8hvbEzFEBGVDAM7EVHJFCmwr2/2AAK06riA1h0bx5UMx5VMq44LyGlshcmxExFRPEWasRMRUQwtFdjj7sgkIpeIyG4R2SMiq12PLxKRZ0XkFRG5X0SONTSueSLyhH3eJ0Rkrs8xF4jIc67/3heRQfu5e0XkNddzS/Ial33cpOvam1yPN/P9WiIiP7R/3jtF5BrXc0bfr6DfF9fzx9n/+/fY78dC13O32I/vFpGL04yjwbHdJCIv2u/RkyJyqus5359rTuO6XkT2u67/u67nrrN/9q+IyHU5j+tu15h+JCLjrueyfL/uEZG3ReSFgOdFRP7CHvdOEfmI6znz75eqtsx/AP49gMUAngIwEHBMJ4AfAzgNwLEAdgD4sP3cBgDX2l//JYDfMzSu/wlgtf31agB3RRw/D8A7AObY398L4KoM3q9Y4wLwXsDjTXu/APw7AKfbX88H8BaAbtPvV9jvi+uYLwL4S/vrawHcb3/9Yfv44wAsss/TafDnF2dsF7h+j37PGVvYzzWncV0P4Os+r50H4FX737n213PzGpfn+D8AcE/W75d97l8D8BEALwQ8/+sAHoO1fcT5AJ7N8v1qqRm7xtuR6aMA9qjqq6p6BMD3AFwhIgLgQgAP2sd9G8CgoaFdYZ8v7nmvAvCYqh4ydP0gScc1rdnvl6r+SFVxfIevAAAD+0lEQVRfsb/eB+BtAJELLxrg+/sSMt4HASyz358rAHxPVQ+r6msA9tjny21sqrrN9Xu0HcApBq/f8LhCXAzgCVV9R1XfBfAEgEuaNK5PA7jP0LVDqerTsCZzQa4A8Ddq2Q6gW0RORkbvV0sF9ph6Abzp+n6v/dgvARhX1aOex034kKq+BQD2v78ccfy1qP+F+qr9J9jdInJczuM6XkSGRWS7kx5CC71fIvJRWDOwH7seNvV+Bf2++B5jvx8HYL0/cV6bRtLz3wBr1ufw+7nmOa4r7Z/RgyKyIOFrsxwX7JTVIgBbXQ9n9X7FETT2TN6vVN0dGyHpd2QSn8c05PHU44p7Dvs8JwM4G4C7hfEtAP4FVvBaD+BmAHfkOK4+Vd0nIqcB2CoizwP4V5/jmvV+/R8A16nqlP1ww++X3yV8HvP+78zkdyqG2OcXkc8AGADwMdfDdT9XVf2x3+szGNcjAO5T1cMi8gVYf/FcGPO1WY7LcS2AB1V10vVYVu9XHLn+juUe2FX1EylPsRfAAtf3pwDYB6v/QreIHGPPupzHU49LRH4qIier6lt2IHo75FRXA3hYVWuuc79lf3lYRL4F4Mt5jstOdUBVXxVrt6t+AA+hye+XiHwQwGYAf2r/eeqcu+H3y0fQ74vfMXtF5BgAXbD+rI7z2jRinV9EPgHrA/NjqnrYeTzg52oiUEWOS1V/7vr2rwDc5Xrtxz2vfcrAmGKNy+VaAF9yP5Dh+xVH0Ngzeb+KmIr5JwCni1XRcSysH+Amte5EbIOV3waA6wDE3pM1wib7fHHOW5fXs4Obk9ceBOB75zyLcYnIXCeVISInAVgK4MVmv1/2z+5hWHnHBzzPmXy/fH9fQsZ7FYCt9vuzCcC1YlXNLAJwOoB/TDGWxGMTkX4A/xvAclV92/W47881x3Gd7Pp2OYCX7K+3ALjIHt9cABdh9l+vmY7LHttiWDcif+h6LMv3K45NAH7bro45H8ABewKTzfuV1V3iRv4D8BuwPsEOA/gpgC324/MBfN913K8D+BGsT9uvuB4/Ddb/8fYAeADAcYbG9UsAngTwiv3vPPvxAQDfdB23EMAYgA7P67cCeB5WgPoOgA/kNS4Av2Jfe4f97w2t8H4B+AyAGoDnXP8tyeL98vt9gZXaWW5/fbz9v3+P/X6c5nrtV+zX7QZwaQa/81Fj+4H9/wXnPdoU9XPNaVx3AthlX38bgDNcr/2s/V7uAfA7eY7L/n4NgLWe12X9ft0Hq7KrBiuG3QDgCwC+YD8vAL5hj/t5uKr+sni/uPKUiKhkipiKISKiEAzsREQlw8BORFQyDOxERCXDwE5EVDIM7EREJcPATkRUMgzsREQl8/8BK7dA0lu16tYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(x[:,0],y)\n",
    "plt.scatter(x[:,0],y_hat);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far we have specified the *model* (linear regression) and the *evaluation criteria* (or *loss function*). Now we need to handle *optimization*; that is, how do we find the best values for `a`? How do we find the best *fitting* linear regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would like to find the values of `a` that minimize `mse_loss`.\n",
    "\n",
    "**Gradient descent** is an algorithm that minimizes functions. Given a function defined by a set of parameters, gradient descent starts with an initial set of parameter values and iteratively moves toward a set of parameter values that minimize the function. This iterative minimization is achieved by taking steps in the negative direction of the function gradient.\n",
    "\n",
    "Here is gradient descent implemented in [PyTorch](http://pytorch.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([-1.,  1.], requires_grad=True)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = nn.Parameter(a); a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'def update(): \\n    y_hat = x@a\\n    loss = mse(y, y_hat)\\n    if t % 10 == 0: print(loss)\\n    loss.backward()\\n    with torch.no_grad():\\n        a.sub_(lr * a.grad)\\n        a.grad.zero_()'"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#this was the old update function\n",
    "\"\"\"def update(): \n",
    "    y_hat = x@a\n",
    "    loss = mse(y, y_hat)\n",
    "    if t % 10 == 0: print(loss)\n",
    "    loss.backward()\n",
    "    with torch.no_grad():\n",
    "        a.sub_(lr * a.grad)\n",
    "        a.grad.zero_()\"\"\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_step = tensor(0.,0.) #creating new tensor to hold value of previous step\n",
    "set_moms = 0.8 #setting momentum value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update():\n",
    "    global last_step\n",
    "    y_hat = x@a\n",
    "    loss = mse(y, y_hat)\n",
    "    loss.backward()\n",
    "    with torch.no_grad(): \n",
    "        if t == 0: moms = 0 #for first epoch, set moms to zero (since there is no previous epoch)\n",
    "        else: moms = set_moms\n",
    "        step = (moms*last_step) + ((1-moms)*a.grad)\n",
    "        a.sub_(lr * step)\n",
    "        last_step = step\n",
    "        print(f'Epoch {t}', loss, a.grad, last_step) #printing the loss, a.grad, and last_step values to check they're calculating correctly\n",
    "        a.grad.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 tensor(7.5648, grad_fn=<MeanBackward1>) tensor([-2.5998, -2.8958]) tensor([-2.5998, -2.8958])\n",
      "Epoch 1 tensor(6.1540, grad_fn=<MeanBackward1>) tensor([-2.4394, -2.3239]) tensor([-2.5678, -2.7814])\n",
      "Epoch 2 tensor(4.9780, grad_fn=<MeanBackward1>) tensor([-2.2808, -1.7747]) tensor([-2.5104, -2.5801])\n",
      "Epoch 3 tensor(4.0327, grad_fn=<MeanBackward1>) tensor([-2.1253, -1.2657]) tensor([-2.4333, -2.3172])\n",
      "Epoch 4 tensor(3.2936, grad_fn=<MeanBackward1>) tensor([-1.9740, -0.8091]) tensor([-2.3415, -2.0156])\n",
      "Epoch 5 tensor(2.7254, grad_fn=<MeanBackward1>) tensor([-1.8279, -0.4125]) tensor([-2.2388, -1.6950])\n",
      "Epoch 6 tensor(2.2902, grad_fn=<MeanBackward1>) tensor([-1.6876, -0.0797]) tensor([-2.1285, -1.3719])\n",
      "Epoch 7 tensor(1.9527, grad_fn=<MeanBackward1>) tensor([-1.5535,  0.1888]) tensor([-2.0135, -1.0598])\n",
      "Epoch 8 tensor(1.6837, grad_fn=<MeanBackward1>) tensor([-1.4259,  0.3951]) tensor([-1.8960, -0.7688])\n",
      "Epoch 9 tensor(1.4609, grad_fn=<MeanBackward1>) tensor([-1.3052,  0.5436]) tensor([-1.7778, -0.5063])\n",
      "Epoch 10 tensor(1.2689, grad_fn=<MeanBackward1>) tensor([-1.1914,  0.6399]) tensor([-1.6605, -0.2771])\n",
      "Epoch 11 tensor(1.0984, grad_fn=<MeanBackward1>) tensor([-1.0846,  0.6907]) tensor([-1.5454, -0.0835])\n",
      "Epoch 12 tensor(0.9443, grad_fn=<MeanBackward1>) tensor([-0.9847,  0.7031]) tensor([-1.4332,  0.0738])\n",
      "Epoch 13 tensor(0.8047, grad_fn=<MeanBackward1>) tensor([-0.8916,  0.6843]) tensor([-1.3249,  0.1959])\n",
      "Epoch 14 tensor(0.6794, grad_fn=<MeanBackward1>) tensor([-0.8052,  0.6415]) tensor([-1.2209,  0.2850])\n",
      "Epoch 15 tensor(0.5685, grad_fn=<MeanBackward1>) tensor([-0.7253,  0.5811]) tensor([-1.1218,  0.3442])\n",
      "Epoch 16 tensor(0.4725, grad_fn=<MeanBackward1>) tensor([-0.6516,  0.5091]) tensor([-1.0278,  0.3772])\n",
      "Epoch 17 tensor(0.3913, grad_fn=<MeanBackward1>) tensor([-0.5840,  0.4308]) tensor([-0.9390,  0.3879])\n",
      "Epoch 18 tensor(0.3242, grad_fn=<MeanBackward1>) tensor([-0.5220,  0.3506]) tensor([-0.8556,  0.3805])\n",
      "Epoch 19 tensor(0.2701, grad_fn=<MeanBackward1>) tensor([-0.4655,  0.2721]) tensor([-0.7776,  0.3588])\n",
      "Epoch 20 tensor(0.2275, grad_fn=<MeanBackward1>) tensor([-0.4141,  0.1982]) tensor([-0.7049,  0.3267])\n",
      "Epoch 21 tensor(0.1946, grad_fn=<MeanBackward1>) tensor([-0.3675,  0.1309]) tensor([-0.6374,  0.2875])\n",
      "Epoch 22 tensor(0.1696, grad_fn=<MeanBackward1>) tensor([-0.3254,  0.0716]) tensor([-0.5750,  0.2443])\n",
      "Epoch 23 tensor(0.1508, grad_fn=<MeanBackward1>) tensor([-0.2875,  0.0212]) tensor([-0.5175,  0.1997])\n",
      "Epoch 24 tensor(0.1368, grad_fn=<MeanBackward1>) tensor([-0.2534, -0.0202]) tensor([-0.4647,  0.1557])\n",
      "Epoch 25 tensor(0.1263, grad_fn=<MeanBackward1>) tensor([-0.2229, -0.0527]) tensor([-0.4163,  0.1140])\n",
      "Epoch 26 tensor(0.1183, grad_fn=<MeanBackward1>) tensor([-0.1956, -0.0766]) tensor([-0.3722,  0.0759])\n",
      "Epoch 27 tensor(0.1121, grad_fn=<MeanBackward1>) tensor([-0.1712, -0.0928]) tensor([-0.3320,  0.0422])\n",
      "Epoch 28 tensor(0.1072, grad_fn=<MeanBackward1>) tensor([-0.1496, -0.1022]) tensor([-0.2955,  0.0133])\n",
      "Epoch 29 tensor(0.1032, grad_fn=<MeanBackward1>) tensor([-0.1304, -0.1057]) tensor([-0.2625, -0.0105])\n",
      "Epoch 30 tensor(0.0999, grad_fn=<MeanBackward1>) tensor([-0.1134, -0.1043]) tensor([-0.2327, -0.0293])\n",
      "Epoch 31 tensor(0.0972, grad_fn=<MeanBackward1>) tensor([-0.0984, -0.0991]) tensor([-0.2058, -0.0432])\n",
      "Epoch 32 tensor(0.0949, grad_fn=<MeanBackward1>) tensor([-0.0852, -0.0910]) tensor([-0.1817, -0.0528])\n",
      "Epoch 33 tensor(0.0930, grad_fn=<MeanBackward1>) tensor([-0.0736, -0.0810]) tensor([-0.1601, -0.0584])\n",
      "Epoch 34 tensor(0.0914, grad_fn=<MeanBackward1>) tensor([-0.0634, -0.0697]) tensor([-0.1407, -0.0607])\n",
      "Epoch 35 tensor(0.0902, grad_fn=<MeanBackward1>) tensor([-0.0544, -0.0580]) tensor([-0.1235, -0.0602])\n",
      "Epoch 36 tensor(0.0893, grad_fn=<MeanBackward1>) tensor([-0.0466, -0.0463]) tensor([-0.1081, -0.0574])\n",
      "Epoch 37 tensor(0.0886, grad_fn=<MeanBackward1>) tensor([-0.0397, -0.0351]) tensor([-0.0944, -0.0529])\n",
      "Epoch 38 tensor(0.0881, grad_fn=<MeanBackward1>) tensor([-0.0338, -0.0248]) tensor([-0.0823, -0.0473])\n",
      "Epoch 39 tensor(0.0877, grad_fn=<MeanBackward1>) tensor([-0.0286, -0.0156]) tensor([-0.0716, -0.0410])\n",
      "Epoch 40 tensor(0.0875, grad_fn=<MeanBackward1>) tensor([-0.0240, -0.0076]) tensor([-0.0621, -0.0343])\n",
      "Epoch 41 tensor(0.0873, grad_fn=<MeanBackward1>) tensor([-0.0201, -0.0009]) tensor([-0.0537, -0.0276])\n",
      "Epoch 42 tensor(0.0872, grad_fn=<MeanBackward1>) tensor([-0.0167,  0.0045]) tensor([-0.0463, -0.0212])\n",
      "Epoch 43 tensor(0.0872, grad_fn=<MeanBackward1>) tensor([-0.0138,  0.0086]) tensor([-0.0398, -0.0152])\n",
      "Epoch 44 tensor(0.0871, grad_fn=<MeanBackward1>) tensor([-0.0112,  0.0115]) tensor([-0.0341, -0.0099])\n",
      "Epoch 45 tensor(0.0871, grad_fn=<MeanBackward1>) tensor([-0.0091,  0.0134]) tensor([-0.0291, -0.0052])\n",
      "Epoch 46 tensor(0.0871, grad_fn=<MeanBackward1>) tensor([-0.0072,  0.0144]) tensor([-0.0247, -0.0013])\n",
      "Epoch 47 tensor(0.0871, grad_fn=<MeanBackward1>) tensor([-0.0056,  0.0146]) tensor([-0.0209,  0.0019])\n",
      "Epoch 48 tensor(0.0871, grad_fn=<MeanBackward1>) tensor([-0.0042,  0.0141]) tensor([-0.0176,  0.0043])\n",
      "Epoch 49 tensor(0.0871, grad_fn=<MeanBackward1>) tensor([-0.0031,  0.0132]) tensor([-0.0147,  0.0061])\n"
     ]
    }
   ],
   "source": [
    "lr = 1e-1\n",
    "for t in range(50): update()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XucHGWd7/HPr3s6SQdkJpB4yU0uy8ajgEQG1j3JWYWI8QJDYN2A6ALegqtHxJwNhN09IWTXFwHOirArRyMgmxWFEUwIQcwieNlwxGWG4HBRVi6LyQSFABPADMlcnvNHVU9ququ6q6ere7p7vu/Xa5yZ6uqnHmrGX5751e95HnPOISIizSM13h0QEZFkKbCLiDQZBXYRkSajwC4i0mQU2EVEmowCu4hIk1FgFxFpMgrsIiJNRoFdRKTJtIzHRadPn+4OPfTQ8bi0iEjD6u7u3uWcm1HqvEQCu5m1AdcDRwEO+KRz7udR5x966KF0dXUlcWkRkQnDzJ6Nc15SI/ZrgB865z5iZpOAqQm1KyIiZao4sJvZQcCfAecBOOf2AfsqbVdERMYmiYenhwMvAN8ys21mdr2ZHZB/kpktM7MuM+t64YUXErisiIiESSKwtwDvAv6vc24+8AdgZf5Jzrl1zrl251z7jBklc/8iIjJGSQT2HcAO59wv/O9vwwv0IiIyDioO7M653wHbzWyef2gR8Hil7YqIyNgkVRXzBeBmvyLmaeATCbUrItKwNm7r5aotT7Czr5+ZbVlWLJ7Hkvmzqn7dRAK7c+5hoD2JtkREGt3Gbb1cdudjvLxnYORYb18/l3z/EYCqB/dxmXkqItKI4ozAN27r5ZLvP0L/wFDB+/sHhrhqyxMK7CIi9SA/YEeNwK/a8kRoUM/Z2ddf3Y6iRcBERGIJC9i5EXhQqcA9sy2beN/yKbCLiMQQFbDzjxcL3NlMmhWL50W+nhQFdhGRGKICdv7xFYvnkc2kC85ry2a4/IyjG6cqRkSkmYQ9JF2xeF7BQ9FMytizb5DDVt5V8DD1qi1P0P7KPVwy6Xu8iV3YQbMhvQpYWvX+m3Ou6hfJ197e7rRsr4jUo7CqlmwmzeVnHA0wEvBbsxn+sG+QgSFXcN6S9P1w98XQ/9LoxjNZOPVaOGZswd3Mup1zJUvLNWIXEQko9pD0/pUnjYzIF6y9j77+gVHnnTz0U957x6eBV8MbH+iHe9eMObDHpcAuIhIQ9yFp8PuO1FYubVnPwfYaVuoCu3dU2MPSFNhFRAJmtmXpDQnu+Q9JZ7ZlOe6Ve1idWc80XsNKRnRf6+wEelmcqmJERALCqloMOPFto5cbX/+mW/lq5jpvlB43qGeysGhVMh0tQiN2EZGAJfNn0fXsS9z8wG/JPRZ1wO3dvZyW+n8c/9Q/we7tHAGUzrsEZA+GD15R9fw6KLCLSJOqZGXFH//6BfLrBU8e+ilHPXQDsDd2H5yDfZNamXzq/6lJQM9RYBeRphN3XZcoYQ9QL2rpJBszqDsHL3MgqwfOoTt7Mvcfc1IZva+cAruINJ1iJYtxAnvuwehFLZ3MtF3sdNOZabtKvs85eMkdyGWD57BpeCEAVoNFv/IpsItI04lbshhl/Ztu5bD+W0j5OfTZtovhEnM5hx3869D7uHTwk6OO12LRr3wK7CLSdOKWLIbq6eSIZ28teDDqBXmDQPZ92HlHet10rhxcyp3+KD2nVot+5VNgF5GmE7auS8kg29PpzQrdvb1Iyw5a5+B272CnO4QrBpaOpFyymTQfO2EWP/71CzXfCi+fAruINJ3gQlyxgmxPJ9x5gTflv5jWOfClRzHgwW29dG95AhvnIB5Gi4CJiFx9VImROoDBGetqWrZY0INaLgJmZv+Ft+rNEDAY58IiIuNiJOWyw5vev2hVjPVbDNo/Oa5BvRxJpmJOdM6VrgcSERkv+SmX3du977PTCpfYzWmd4wX/BgnqoBy7iEwk964pzKMP9ENL1lvHJfhahWunj6ekFgFzwL+ZWbeZLUuoTRGRZPR0Fs+j97/sBfHWOYB5nxs0qENyI/YFzrmdZvZG4B4z+7Vz7mfBE/yAvwxg7ty5CV1WRCTCqPLF0fXnBVpne0G8QQN5vkRG7M65nf7n54ENwAkh56xzzrU759pnzJiR/7KISHJyufSREXqRoF6jpXRrqeIRu5kdAKScc6/6X78fWFNxz0REyhVrklFAAz4YjSOJVMybgA3mrTTfAnzHOffDBNoVEYlv83LoupGio/Mgf7JRM6o4sDvnngbemUBfRETGpqezvKDehOmXIG2NJyKN7941lA7q/qpeDV7xEofq2EWk8ZWaOdqkufQoCuwi0vhaZ4c+MB120H3clRzfcf44dGr8KBUjIo1v0Sr6mTzqUG7jiwsfP3KcOjV+NGIXkcZ3zFJW3rKNFS2dzLQX2ekO4cpBb630uFvTVbL5db1RYBeRptB10Mks7FtYcDx/16SwAA5UtPl1vVEqRkSaworF88hm0qOO5e+atHFbL5d8/xF6+/px7A/gl935WOTm141II3YRqYpapzbi7Jp01ZYnQgN4/rGcuJtf1xsFdhFJXG5kHJbagDK2rCvTkvmzirZVbqCOtfl1HVJgF5HERY2ML7vzMV4fGB557bhX7uH4jZ/B3fEiltvNaIy15nH+QpjZlqU3JLi3ZTPsHRwub/PrOqbALiKJixoZv7xnYOTrjtRW1mauZ6rt8w7kdjOCsoN7sb8QlsyfNRL0e/v6CxbwzWbSrO54B1C9vyRqTYFdRBIXNTIGuKzlRj6Wvo80w5jlvTjQ7y0PUGZgj/oLIffwMxj0HftXZ5+VF8AbNZDnU2AXkcStWDxvVDDtSG3l4kwnM20XOAoDeoDbvYMiL4eK+gthZ19/aNDPBfX7V55U5pUagwK7iCQuWKHS/so9rJ10A1n2ei+WiNq/ZzpvLvN6UX8hzGzLFg36zUp17CJSFUvS93P/5Au4ZtJ1+4N6CXvcJC7f9xdlX6tYDXtUZUujVrzEoRG7iIxZaCVK+n64+2Lofyl2O85Br5vOlYNL6T7o5LL7UaqGPZgWgsaueIlDgV1EYgsG8tZshj/sG2RgyKsx6e3rZ+uG6/hQ+ptMcvFG6OAF9fVD7+PSwU+SzaS5fIwBN6qGPc7EpWajwC4iseSXFPb1DxSccyG3xArqzv+fIUtxm53M6sFzCypUklRq4lKzUWAXkVjCqks6Ulu5qMWrdtnppntVL0U4YKebzhUD3sqL4KVFrj7z6AkVeKtNgV1EYsmvIsmfYDTbdjFcbHe6TJbL3Pnc9NoJow7n6s0V2JOjwC4iscxsy3LcK/eMjNCHSdFiw6POSZm3wUUqUNLoHPTZG7jGfaogqOc0c+nheEgssJtZGugCep1zpyTVrohUV/CBaNvUDM7B7v6BgoeMX337bziq+3qy/gg9xXBoewbsGJ4+suHFPw6fyWa3kIHXo4fzzVx6OB6SHLF/EfgVcFCCbYpIFeU/EA2u5ZK/3srxT/0T5NZ1KaJ/6ls40103UoGyZ98gA3sKH7TmNHvp4XhIJLCb2Wzgw8CXgeVJtCki1Rf2QDRoVP57947SDWayTP3gGu4/Zv9U/cNW3hV5ejUrYSaypEbsXwUuAt4QdYKZLQOWAcydOzehy4pIJYrltkcqXvp3wdVzIDstfNKRpcENQ8Syu1HT/Zt5rZbxVnFgN7NTgOedc91m9t6o85xz64B1AO3t7cWenYtIjeQH3Y7UVlZn1jON14DAYl27t0N6EqQyMBxIq2SycOq1RVdjzF8QDJR+qbYk1opZAHSY2X8BtwAnmdm3E2hXRKosF1w7Ult5aPIyrslcx8H2GmYhKzAO7YPJb4DWOYB5n0sEdfDy85efcTSz2rIY3kj98jNUt15N5lxyg2d/xP7Xpapi2tvbXVdXV2LXFZkoqrGP6C2rl3Km21J0Kd39DFb3VXQ9GTsz63bOtZc6T3XsIg2i1C5BY9LTyZnEDep4eXSpe4ku2+uc+4lq2EWqo9QuQWNy75r4m1pkst7DUal7Wo9dpEFUZcOIGCWMzsFL7sBY+XSpD0rFiDSIYrsExbJ5OXTfBG7IK1E87jwvtbJ7e+Rbckvqrjvw86Nq06W+acQu0iCK7RJU0ubluK4bvKAO4Ia87w8+3CtjzJMbpX9x4HOstc+oNLHBaMQu0iDK3jCipxPuXQO7d+BwBbl0A4b/69/pnr+WI7r/nmm8CngBfc3gOdwxvNArTdTM0IaTaLljXCp3FKmynk648wIYKJ5/d8DCKRs0M7RBxC13VCpGpBndu6ZkUAcYcqnqPJSVcaVUjEgzilntckfq/ZEPZdumZliw9r4Js09oM9GIXaQZRUwkcs77GHQpvuNOJt3xldCHspm08drrg/T29ePYPxlq47beGnReKqXALtKMFq3yJhQFDKancFnmQg7f+x3ek72dA06/ZmST5/y1XA6Y1MJA3j53FU+GkppRKkakBqqxxktRuYlEflUMrbNpWbSK1ccsZXXI6bkAnxO1hrry7o1BgV2kykqt8VJW0A+UMEatfz7imKVjnila8WQoGVcK7CJVVmqNl9gLe+WXMO7e7n0PsQJ4Of+AaA31xqY6dpEqCAbRqP+HGWXuLnT1UeHT/1vnwJceLdmfsEBdbF30mqePpCQt2ysyTsKCaJiZbdnInHX7K/fA1ReMTrlElTDGKG0s9ldDVLDOz7tL41BVjEjCSm0QDfvTGmE5647UVtZOusEfnbv9KZfstPDGYqyRrklIE4sCu0jCSgXLtNlICiSshvziTCdZ9o5+Uy6vnlfCGHeN9KiHnnoY2pwU2EUSVipYDjs3kuJYMn8W649/lgemfJGnJ5/NA1O+yEzbFf7G/pe9NdHL3HMUKlwZUhqOcuwiCQurKAkaFfh7Ojn+kUuBfjB4My/gPVYNeeTaOrusEsb8h59/ftwsfvzrF/QwdAJQYBdJWC5YXvL9HvoHhgteP/FtM/Z/E7pYl6MguJe5LV1Y7fzt3b1Fq2CkeSgVI1IFS+bP4uADJoe+dlfPc/u/iaxocWNKueRUZX9UaRgVj9jNbArwM2Cy395tzrlLK21XpNEFH6J2pLZyUUsnM20XOwen8+Cmizi+4/zoreli1KbHvXac49Jckhix7wVOcs69EzgW+ICZvTuBdkUaWi6Xvj7zZa7JXMfs1C5SBrNTuzjqof/tzSQNWayr3LRLsWvHPS7NpeLA7jyv+d9m/I/aT2cVqSMPbvoG33v9Mzwz+Wz+R+oxLG9fuix7vfz6MUvHXOlSjKpgJrZEHp6aWRroBv4I+Jpz7hdJtCvSiB7c9A3e2X0Jk2yIgo1Gg3L59QoW64pS9v6o0lQSCezOuSHgWDNrAzaY2VHOuVEJQjNbBiwDmDt3bhKXFalLRz70915QLyXGjNFKaEmAiSvRqhjnXB/wE+ADIa+tc861O+faZ8yYUfBekWbR6l6Nd2KFeXSRKBUHdjOb4Y/UMbMs8D7g15W2K9KwiqVfcg57T+LpF5GcJFIxbwH+xc+zp4BO59zmBNoVSVScZWjLXqo2ZOOLfZk2Jg/0Rb9n+tvg3E0V91UkSsWB3TnXA8xPoC8iVVNqF6O454wSsfHF5HeezVD3etJuYORUBww7Y2NqMekFX2FJhX0VKUZLCkhTy418wzazyF+PPNaa5cERuqXA5T0kHeiH3/wb6dOvg3vX4HbvYKc7hCsGlrJpeCEA2RJBeixrp4sEKbBL04qz4UVwJmbJ2Zr5I/T8oJ6ze8dICePCtfcV/KNSKkhr1qhUSmvFSNOKs+FFcCZmydmaoQt2hQiUMY4lSGvWqFRKgV2aVqkRbv5MzJKzNWNsQZe/HMBYgrRmjUqlFNilaRULnrPasgVL2C6ZP4vNh2/gqSln88zks3lmytn0TPoES9L3eydETSiyNFHLAYwlSC+ZP4vLzziaWW1ZLKKvIsWYc7Vf1qW9vd11dXXV/LoysYTl2LOZdHSQ3Lwcum4oODxsaVKnf937JphjB2+EXmJtF5UuSlLMrNs5117qPD08laYTDKSt2QxTMin69gyUDqrdN4UeTrkh9ty9iqkX+/Pu8urWS0000tR+qTUFdmkq+aP0vv4Bspk0V595bOngGlXlAkzp/533RRUW7BJJmgK7NJWKasAtHRncdw4fwmwK0yonvm2G9hGVuqPALg0hbp66ohrw487Ddd1QsNTLoEtx/aSPc2zIjNBvP/DbkfM0Q1TqhapipO7l0iu9ff049gfQjdt6C84tWl7Y0wlXHwWr27zPPZ2jTzrlKzz91rMYcuD8j9fcFFa6z3Hsh5fFqovXvqJSDxTYpe6VszFzfnlhR2orWyddwNbXT4fvL/P3F3Uj67rkB/cjPvEN7lzyOAuzGzh873dYnP0uC0//HEvmz4o981MzRGW8KRUjda+c9MqS+bPoevYlbn7gt5ya2srazPVMtX3+q3mlvQP9+7eny2sjLJUysy0buuZM2Hki40kjdql75c7enP/Lf+DJyR/jmsx1gaAeIc5sUl/YZKN8miEq9UCBXepeWbM3Ny/njOEfkjZXsIF0qDK2pwubEfrxd8/VDFGpO0rFSN2LvTFzTyd03RAvoEPBui5x+6LALfVOgV0aQsmAmltStyQDnLeuS4xZoyKNSIFdGlepTS/yWRpO/7qCuTQ9BXZpTJuXQ9eNjFS6JBTUtWCXNAMFdqmZxIJmT+fooF6EAyxzAJz61VhBXXuNSjNQYJeaqDho5qddYgT1PW4Slwx8mmtWXx6rj9prVJpFxYHdzOYA64E3A8PAOufcNZW2K82lVNCMHM33dMLdF0P/S/vfWCTtMuhSpHDsdIdw5eBSug86OXYftdeoNIskRuyDwP9yzj1kZm8Aus3sHufc4wm0LQ0sGKyjxtc7+/ojR/Oztm/m+EcujbfPKDDsYPnAZ9k0vBDwN9UoY7JQ1MxSzSSVRlPxBCXn3HPOuYf8r18FfgXo79YJLn/hrigz27KRo/k5D11VVlC/efhk/n3KiWOeLKS9RqVZJJpjN7NDgfnAL0JeWwYsA5g7d26Sl5U6FGclxFzQ/NKtD9OR2spFLZ3MtF3sdNO5cnApb3QvULCGbkB+2mXT8EJmHdDCtlXvH1OfY0+EEqlzie15amYHAj8Fvuyc+36xc7XnafM7bOVdkSN1g1FBc/U/XMpFA6PXddnjJrHPJtPGq6Ft7HGTWDnw6ZG0S7Dtq888VsFZmlJN9zw1swxwO3BzqaAuE0NUvnpWW5b7V5406thFmVuZOjh6sa6pto90JgtkC9Mx2YO5cuAcNu09oaD91mxGJYsy4VWcYzczA24AfuWc+0rlXZJmUDRfnbfhxdT+50LbmDzwCpx6rTf9H/M+n/FNuPgZjv3wstD2zYi9drtIs0pidccFwF8CJ5nZw/7HhxJoVxpY2EqIl59xNEvS9zN4xxdGbXgRmQxsnc3GoQUs2Hsth71+Mwv2XsvGoQVF2+/bMxDalEoWZSJJLMdeDuXYJ6ieToY3nE/KDRe85Mh7TprJ8uDRl3HOg28dNQLPZtJFq10WrL0vdgpIpNHUNMcuklN0otGdF4QG9RGtc7yZpa2zYdEqLvzBdPrz8uvBtErYdVYsnjcqxw4qWZSJR4FdEpM/0ei4V+7h+I2fwd3xIlZi9cXe4enM/tKjo47t/M5d4ef6D0SLPSBVVYxMZArskphg7XpH/n6jRYL6HjeJ6yd9nNV5x6Mqa9JmRZcn0GYYMtFpazxJzM6+fjpSW9k66YJ4+43iTTJa5ZZx7IeXFbwWVVkzFPFcSA9IRTwasUtspZbdPffA/+CigetjBXTwRupXZj7Hwg8vCx1hR6VVrtryhNZ0ESlCgX2C2Litl9WbHqOv3ysHnDY1w6WnvqNoyiIYyFuzGV7dO8jQsDda7u3rZ8VtvwT2B+CwiUYFLA1uGFpnM3XRKlaXWCM9Kq2iB6Qi0RTYJ4CN23pZ8b1fMjC8P4Xx8p6BgsCc/55g8Mz9gxA0MOS47M7HRt4/tf93xTuSyXoTjircmk4PSEWKU2CfAK7a8sSooJ4zMOQiN5EotojXqAW7BqdDz+VesG6d7U88CpHw5tF6QCoSTYF9Aij2ULGczSUua7mRj6d/RAowfzbRbNsFd17gfbNolfd1sPY8oVG6iMSnqpgJoNhDxbapmVjvuazlRs5J/4i07Q/qIwb6vW3rjllauLaLgrpIzWnEPgGsWDyPC299OPS1qBUlViyex9YN13EhtzDTdmGEBPSg3Tu8z8csVSAXGWcasU8AxXLRu0MeigIsSd/P2sz1zE7tIhU2Ss/XOruCHopIkjRibyC58sPevn7SZgw5x6yYFSGzyt3P8941tAy9Hq9jmayXXxeRuqDA3iDyyw9zsy/jbCSxcVsve/YNFhwvqP3u6fRy5bt3QNGdSgMmHQCnfFXpF5E6olRMgyhWflhsI4ncPwgv561T3pbNjF7+1l99cWSd9FIsDe2fgr/ZqaAuUmc0Ym8QpdZBiXo96h+EAya3jB7h37umcAu6fCpdFGkIGrE3iFLroES9HhbwO1JbuXXPZ0a2pqOnc39VSyiVLoo0Eo3YG0TYBhI5xdZJCS5925HayurMeqbx2v4ql93bvRRMdhr0v1TYQOscyFsnXUTqm0bsDSK4xyd4a5JDYC/RiAenuaVvL2u5ka9mruNgey18ghF4qZYgVbuINCTteToB/P6fFvPGFx+geCm6wRnr9lfF+NvTKfUiUj9quuepmd0InAI875w7Kok2JSGbl/OmFx8ofV7rbM0aFWkSSeXYbwL+GVifUHuSlO6bSp9TZsql1IYbIjK+EgnszrmfmdmhSbQl+yUSQIvsNQqApcqqdsmfKBVngpSI1JaqYupAWAAHyg+gwZmjuRy5paODeyoNS75eVvolrC4+uJG0iIy/mgV2M1sGLAOYO3durS5b96JGwIajf2B41LlFA2hu5miuwiVXxnjoQnjmp4XnpyfBaV8rO6dezvrtIjI+ahbYnXPrgHXgVcXU6rr1LmoEHCUYQJ/61vm89dlO0m4YjMKql4F+eOlpb+p/903eyN3ScNx5cMpXCtqOk/qZWe5iYiJSc0rF1ECxgFnuSHdmWxY2L8d13cjhznk16aXWST/lK6MC+cZtvVy19r4xpX7CJkppI2mR+pJUueN3gfcC081sB3Cpc+6GJNpudKUeNkaNgMN0pLbyj/u+BV39XiwvtUY6FKyTHtWfKZlUrNy5NpIWqX9JVcV8NIl2mlGph41hI2CjcH3FjtRWrph0PZnhffEvHlLGGNWfqPRP2F8U2khapL4pFVNlpR42ho2AT3zbDG7v7uXkoZ9yUUsnM20Xw6RoYTi0rSDnwMwiZ46OKfUjIg1Fgb3K4jxsDBsBf+Llf+bwZ28ZybakYgb19UPv49x/uH3U8WCOP+XvvJSvLZth7+CwcuciTUCLgFVZbhGuoKIBs6cTrjiMIwJBvRTnYNgP6usO/Pyo13I59d6+fhyEBvVsJs3qjneMLDJmlF5cTETql0bsVVbWw8b8WvQSnINhjG8PLeLSwU+SzaS5PO8fjKiNNtJmDDtX0B8FcpHGp8BeAyUfNo7MGN1esi0HOGfsdIdw5eBSNg0vBLxUyuqOdxRcJyqnPuwcz6z9cOz/BhFpHArs463MUToYm5Y8NvIXwKwS5YaaUCQy8Siwj7c4e436nINb7f2cVUa5oSYUiUw8Cuy1FLZIV9G9Rr1gDsFc+rmcFXFusRmumlAkMnEosNdKTyfc8XkY8icY7d7ufR+x16hz0Oumj8qjF1NqhqsCucjEocBeCz2dsOF8cHm16EP7YGivN0M0mI7JZPmbgc/w3b3vLmhq2tRM6CW0nK6I5KiOvdpyD0fzg7rP7fuDt9FF6xzAvM+nXsufnPZZMunRleyZtHHpqe8IbUfL6YpIjkbs1Rbn4WjIXqNL/M9xc+OqfhGRHAX2aivxcPSl4QM5JOK1cnLjqn4RkRwF9mprnR058WivS3PZ4DlcGzg21n1OVf0iIjkK7NW2aFXBBCTn4GUOZPXAOfz7lBNHjle6UbSqX0QEFNirz8+d77l7FVP2/G7UUgCZtHFV4GGoKltEJAkK7LVwzFKmHrN0VJolbCkAVbaISBKaJrCPNTddTtvnv/Y1zm65jzTDWJFNoaOUSpWoskVEktAUdez5a47nctMbt/Um0vbWDdexpf+j/GX6R7Qw7K2T7oag6wbYvLzia+SUvXa7iEiIpgjsxXLTlXr4rnWssXUcaK9jYTtfdN9U8TVylsyfpc0uRKRiTZGKqWZu+tP7vs3UVJENpF34JtBjpcoWEalUIiN2M/uAmT1hZk+a2cok2ixHVA46idz0zNSLxU+wdPHXRURqrOLAbmZp4GvAB4G3Ax81s7dX2m45qpmbfj375uInHHdexdcQEUlSEiP2E4AnnXNPO+f2AbcApyXQbmyJ5KZ7OuHqo2B1m/e5pxOAqR9cw2B6yqhTvSXSDdo/VVZVjIhILSSRY58FBOfM7wD+JP8kM1sGLAOYO3duApcdraLcdP72dLu3e98DHLPUu0mBDTJs0aqCRbtEROpFEoE9rFbEFRxwbh2wDqC9vb3g9XEVtgLjQL93PLfyogK5iDSIJAL7DmBO4PvZwM4E2q2OcranK7Eyo4hIPUoisD8IHGlmhwG9wFnA2Qm0m6gHN32DIx/6e1rdq/vr0XMpl4jt6WidXdM+iogkoeLA7pwbNLP/CWwB0sCNzrnHKu5Zgn5+7Xn8yYsbSBmFiaOBfmjJhm5Px6JVo06t5rIFIiJJSaSO3Tn3A+fcHzvnjnDOfTmJNpPy4KZv7A/qUfpfDt2eLphXr+ayBSIiSWqKJQWKmfPQVcWDOkDrbDYOLWDB3ms57PWbWbD3WjYOLRh1SjWXLRARSVJTLClQzBvdC+F1OzmZLA8e8YWSG1yErbpY7LiIyHhp+BH7xm29LFh7H1/8m0v43eo/wuVNMHreZoS+zznYm2mFU6/lwsePLDkaT4euABZ9XERkvDT0iD23pO6ddhPTMq/tH5gHJhhtf9cKWrv/jqztX8hr2MEvDjmdP73gJgB2fueu0PaDi4gNufDS+6jjIiLjpXEDe08n771jOafZq+HL6foTjI4Rc2JXAAAIOElEQVT/0qM8iJdrf6PbxfM2ne3HreBPO84fOTXOBhezIs6ZpU0wRKTONGZg37wcum6kDVc8f+5PMDq+43zwA/mb/Y+gFYvnjcqxQ+EiYnHOERGpB40X2Hs6oetGQlYtKBRzglHuAWmxGvU454iI1ANz45Ajbm9vd11dXWN789VHeTn0UjLZglp00CQjEWlcZtbtnGsvdV7jjdhLrN/iAMseDB+8IjSolyprFBFpdI1X7lgsvZI9GDvjm3DxM6GrMWqSkYhMBI0X2Bet8tIso/ibXkQE9Jxq7o0qIlIvGi+wH7O0cF2XM9bF2smomnujiojUi8bLscOYN75QyaKITASNGdjHSCWLIjIRTKjADhXujSoi0gAaL8cuIiJFKbCLiDSZhk3FaAapiEi4hgzsf7fxEW5+4Lcjq8VoBqmIyH4Nl4rZuK13VFDP0QxSERFPRYHdzP7CzB4zs2EzK7kwTRKu2vJE5LqOmkEqIlL5iP1R4AzgZwn0JZZiwVszSEVEKgzszrlfOedqmv+ICt4GmkEqIkID5thXLJ5HNpMedcyAj717rh6ciogQoyrGzH5E4W5yAH/rnLsj7oXMbBmwDGDu3LmxO5hPywKIiBSXyA5KZvYT4K+dc7G2RapoByURkQkq7g5KDZeKERGR4iotdzzdzHYAfwrcZWZbkumWiIiMVUUzT51zG4ANCfVFREQSoFSMiEiTUWAXEWkyiVTFlH1RsxeAZ8t823RgVxW6U6l67RfUb9/Ur/KoX+Wp135B5X17q3NuRqmTxiWwj4WZdcUp86m1eu0X1G/f1K/yqF/lqdd+Qe36plSMiEiTUWAXEWkyjRTY1413ByLUa7+gfvumfpVH/SpPvfYLatS3hsmxi4hIPI00YhcRkRjqKrDH3ZHJzD5gZk+Y2ZNmtjJw/DAz+4WZ/cbMbjWzSQn162Azu8dv9x4zmxZyzolm9nDg43UzW+K/dpOZPRN47dha9cs/byhw7U2B4+N5v441s5/7P+8eMzsz8Fqi9yvq9yXw+mT/v/9J/34cGnjtEv/4E2a2uJJ+jLFvy83scf8e3Wtmbw28FvpzrVG/zjOzFwLX/3TgtXP9n/1vzOzcGvfr6kCf/tPM+gKvVfN+3Whmz5vZoxGvm5ld6/e7x8zeFXgt+fvlnKubD+C/AfOAnwDtEeekgaeAw4FJwC+Bt/uvdQJn+V9/HfirhPp1JbDS/3olcEWJ8w8GXgKm+t/fBHykCvcrVr+A1yKOj9v9Av4YONL/eibwHNCW9P0q9vsSOOdzwNf9r88CbvW/frt//mTgML+ddII/vzh9OzHwe/RXub4V+7nWqF/nAf8c8t6Dgaf9z9P8r6fVql95538BuLHa98tv+8+AdwGPRrz+IeBuvO0j3g38opr3q65G7C7ejkwnAE865552zu0DbgFOMzMDTgJu88/7F2BJQl07zW8vbrsfAe52zu1J6PpRyu3XiPG+X865/3TO/cb/eifwPFBy4sUYhP6+FOnvbcAi//6cBtzinNvrnHsGeNJvr2Z9c879OPB79AAwO8Hrj7lfRSwG7nHOveScexm4B/jAOPXro8B3E7p2Uc65n+EN5qKcBqx3ngeANjN7C1W6X3UV2GOaBWwPfL/DP3YI0OecG8w7noQ3OeeeA/A/v7HE+WdR+Av1Zf9PsKvNbHKN+zXFzLrM7IFceog6ul9mdgLeCOypwOGk7lfU70voOf792I13f+K8txLltv8pvFFfTtjPtZb9+nP/Z3Sbmc0p873V7Bd+yuow4L7A4Wrdrzii+l6V+1XR6o5jYZXvyGQhx1yR4xX3K24bfjtvAY4GgksYXwL8Di94rQMuBtbUsF9znXM7zexw4D4zewR4JeS88bpf/wqc65wb9g+P+X6FXSLkWP5/Z1V+p2KI3b6ZfRxoB94TOFzwc3XOPRX2/ir0607gu865vWb2Wby/eE6K+d5q9ivnLOA259xQ4Fi17lccNf0dq3lgd869r8ImdgBzAt/PBnbirb/QZmYt/qgrd7zifpnZ783sLc655/xA9HyRppYCG5xzA4G2n/O/3Gtm3wL+upb98lMdOOeeNm+3q/nA7Yzz/TKzg4C7gL/z/zzNtT3m+xUi6vcl7JwdZtYCtOL9WR3nvZWI1b6ZvQ/vH8z3OOf25o5H/FyTCFQl++WcezHw7TeBKwLvfW/ee3+SQJ9i9SvgLODzwQNVvF9xRPW9KverEVMxDwJHmlfRMQnvB7jJeU8ifoyX3wY4F4i9J2sJm/z24rRbkNfzg1sur70ECH1yXo1+mdm0XCrDzKYDC4DHx/t++T+7DXh5x+/lvZbk/Qr9fSnS348A9/n3ZxNwlnlVM4cBRwL/UUFfyu6bmc0HvgF0OOeeDxwP/bnWsF9vCXzbAfzK/3oL8H6/f9OA9zP6r9eq9svv2zy8B5E/Dxyr5v2KYxNwjl8d825gtz+Aqc79qtZT4rF8AKfj/Qu2F/g9sMU/PhP4QeC8DwH/ifev7d8Gjh+O93+8J4HvAZMT6tchwL3Ab/zPB/vH24HrA+cdCvQCqbz33wc8ghegvg0cWKt+Af/dv/Yv/c+fqof7BXwcGAAeDnwcW437Ffb7gpfa6fC/nuL/9z/p34/DA+/9W/99TwAfrMLvfKm+/cj//0LuHm0q9XOtUb8uBx7zr/9j4G2B937Sv5dPAp+oZb/871cDa/PeV+379V28yq4BvBj2KeCzwGf91w34mt/vRwhU/VXjfmnmqYhIk2nEVIyIiBShwC4i0mQU2EVEmowCu4hIk1FgFxFpMgrsIiJNRoFdRKTJKLCLiDSZ/w+pq2nF1FtfUgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(x[:,0],y)\n",
    "plt.scatter(x[:,0],x@a);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}