fastai_v1_ml_nn_vision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome\n",
    "As per the previous lessons, this is the `ml1/lesson4-mnist_sgd.ipynb` attempted in v1 of the fastai library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports and Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First Error\n",
    "Here we get the error \n",
    "\n",
    "    ModuleNotFoundError: No module named 'fastai.torch_imports'\n",
    "\n",
    "But actually both torch_imports & io are missing.  Based on the newer example at `/examples/dogs_cats.ipynb` I will just import that base fastia & fastai.vision instead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.imports import *\n",
    "#from fastai.torch_imports import *\n",
    "#from fastai.io import *\n",
    "\n",
    "from fastai import *\n",
    "from fastai.vision import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we are downloading the data and extracting it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#path = 'machine_learning_data/mnist_png/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Error\n",
    "It seems this has all fundamentally changed.  Once again basing things off the dogs_cats example / vision examples given at https://docs.fast.ai/vision.html\n",
    "\n",
    "The tutorial code downloaded the data here, we use the new `untar_data` to replace this code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = 'machine_learning_data/mnist_new/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import os\n",
    "#os.makedirs(path, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#URL='http://deeplearning.net/data/mnist/'\n",
    "#FILENAME='mnist.pkl.gz'\n",
    "\n",
    "#def load_mnist(filename):\n",
    "#    return pickle.load(gzip.open(filename, mode='rb'), encoding='latin-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get_data(URL + FILENAME, path + FILENAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://s3.amazonaws.com/fast-ai-imageclas/mnist_png'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "URLs.MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('machine_learning_data/mnist_new')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dl_path = untar_data(URLs.MNIST, dest=path); dl_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like `untar_data` unzips into the parent directory, so doesn't fully use `path`.  Which is perhaps why this ends up in mnist_png instead of minst_new?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls: cannot access 'machine_learning_data/mnist_new': No such file or directory\r\n"
     ]
    }
   ],
   "source": [
    "!ls {dl_path} #path is borked?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "models\ttesting  training\r\n"
     ]
    }
   ],
   "source": [
    "dl_path = 'machine_learning_data/mnist_png/'\n",
    "!ls {dl_path}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so we have the data now, but there are more fastai objects now so looking at the data is somewhat different"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#((x, y), (x_valid, y_valid), _) = load_mnist(path + FILENAME)\n",
    "data = ImageDataBunch.from_folder(dl_path, \n",
    "                                  train='training', \n",
    "                                  valid='testing').normalize(mnist_stats)\n",
    "                                  #ds_tfms=get_transforms, size=224)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(fastai.vision.data.ImageItemList,\n",
       " (60000,),\n",
       " fastai.data_block.CategoryList,\n",
       " (60000,))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#type(x), x.shape, type(y), y.shape\n",
    "type(data.train_ds.x), data.train_ds.x.items.shape, type(data.train_ds.y), data.train_ds.y.items.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([3, 28, 28]), 2352, 784)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "img,label = data.valid_ds[4000]\n",
    "img.data.shape, (3 * 28 * 28), (28 * 28)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMgAAADSCAYAAAAPFY9jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAABhlJREFUeJzt3TFsVdcdx/FzGiaT0Y4YsnWxlaESnjJgZWPoAEYgNUgwlSEbbFjIk5FQ1gwdnAnRscKeOlYVMIGyuYCiLBVDBpgyPIiicDO0WfLO/cWPZ/v5Pj4fiYG/jvCV4cvlHe47r3ZdV4C2P8z6AuA4EwgEAoFAIBAIBAKBQCAQCAQyILXWv9dav6+1/lBr/bbW+tdZX9O8q/6jcDhqrZ+UUr7ruu7HWutyKeXfpZQ/d133zWyvbH65gwxI13X/6brux19/+v8ff5zhJc09gQxMrfVvtdZRKeV5KeX7Uso/Z3xJc80/sQao1vpBKeXTUspnpZQvu677abZXNL/cQQao67qfu657VEr5uJTyxayvZ54JZNhOFK9BDpVABqLW+lGt9S+11g9rrR/UWs+WUj4vpfxr1tc2z7wGGYha61Ip5R+llD+V//3F9t9Syldd13090wubcwKBwD+xIBAIBAKBQCAQCASCE0f5xWqttsw4Nrquq7+3xh0EAoFAIBAIBAKBQCAQCAQCgUAgEAgEAoFAIBAIBAKBQCAQCAQCgUAgEAgEAoFAIBAIBAKBQCA40lNN3ierq6tjs/X19ebaxcXF5nxlZaU5P3PmTHN+//795nx3d3ff60ejUXPt+8odBAKBQCAQCAQCgUAgONJPmJrHs3n7dpr29vbGZn3f61rbR8Qe9vqdnZ2x2cWLF5tr55GzeWFKAoFAIBAIBAKBQOBZrCn1PRfVt3M07dqDXN96Nmxtba259sGDBxN9zXnhDgKBQCAQCAQCgUAgENjFOiSt56Imfe5tFuuXl5eba+1iAWMEAoFAIBAIBAKBwC7WlLa3t/e9tu/cqlevXh3Itdy6das539raOpBf/33kDgKBQCAQCAQCgUAgEDgXa4AuXLjQnN+9e7c5X1hYaM5bv/enTp1qrj2onbbjxLlYMCWBQCAQCAQCgUAg8CzWMXDjxo3m/Pz5881531lck57u/vDhw7HZPO5WTcMdBAKBQCAQCAQCgRfpR+jevXvN+eXLl5vzvhfdk855d+4gEAgEAoFAIBAIBAKBN0wdktXV1bHZ48ePm2v7HgWZ9NGRg1i/u7vbXHvlypXmfDQaNedD4A1TMCWBQCAQCAQCgUAg8CzWERrCR7D1vUlrY2OjOd/c3Jzoaw6NOwgEAoFAIBAIBAKBQGAX65C0nlF68eJFc+3S0lJz3veRbY8ePZroWq5du9acLy8vj81OnjzZXLuysjLR15wX7iAQCAQCgUAgEAgEAoF3FB6hxcXFiebPnz8/zMspT548GZudPn26ubbvz8mJE8PdCPWOQpiSQCAQCAQCgUAgEAx3C+KY6HtG6dmzZ2Ozvo83m9XHnrXOwGqd55X0PUf28uXLd7qm48YdBAKBQCAQCAQCgUAgsIu1T327VX0ntreeo+o7If2wn7nq0zoDa9LPP1xfX2/Ot7e33/3CjhF3EAgEAoFAIBAIBF6k79Pt27eb84WFhX3/GrN6Md53lE/r2vs+rq3PrB6TOSruIBAIBAKBQCAQCAQCgWN/fqPvkZK9vb3mvO/79/r167HZnTt3mmt3dnaa89abrkopZW1trTlvHUZdSilnz55tzs+dOzc269vF6jtI+9KlS835EDj2B6YkEAgEAoFAIBAIBHaxfqNvF+vp06fN+du3b5vz1m5Q3/e6b+doFutbu2+l9L/Zq28HbgjsYsGUBAKBQCAQCAQCgcAu1j5tbW015xsbG835UHex+p6tGvJuVR+7WDAlgUAgEAgEAoFAIHAu1j5tbm4252/evGnO+3a9WiY9i+qg1l+9enVsNo+7VdNwB4FAIBAIBAKBQCAQCDyLdUiuX78+Nrt582Zz7dLSUnM+6bNVfWdX9Z3H1TptfjQaNdfOI89iwZQEAoFAIBAIBAKBwC4W7y27WDAlgUAgEAgEAoFAIBAIBAKBQCAQCAQCgUAgEAgEAoFAIBAIBAKBQCAQCAQCgUAgEAgEAoFAIBAIBEd67A8MjTsIBAKBQCAQCAQCgUAgEAgEAoFAIBAIBAKBQCAQCAQCgUAgEAgEAoFAIBAIBAKBQCAQCAQCgUAgEAgEAsEvPxCY4kUrZBwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "img.show(title=label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalise\n",
    "Normalise the data to be able to feed into ML network\n",
    "\n",
    "I believe normalisation is now done when creating the data set with `.normalize(mnist_stats)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#mean = x.mean()\n",
    "#std = x.std()\n",
    "\n",
    "#x = (x-mean) / std\n",
    "#mean, std, x.mean(), x.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#x_valid = (x_valid - mean) / std\n",
    "#x_valid.mean(), x_valid.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Helper funcs\n",
    "\n",
    "I've slightly tweaked these to support the new image object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# def show(img, title=None):\n",
    "#     plt.imshow(img, cmap=\"gray\")\n",
    "#     if title is not None: plt.title(title)\n",
    "        \n",
    "# def plots(ims, figsize=(12,6), rows=2, titles=None):\n",
    "#     f = plt.figure(figsize=figsize)\n",
    "#     cols = len(ims) // rows # divides and rounds to int\n",
    "#     for i in range(len(ims)):\n",
    "#         sp = f.add_subplot(rows, cols, i+1)\n",
    "#         sp.axis('Off')\n",
    "#         if titles is not None:\n",
    "#             sp.set_title(titles[i], fontsize=16)\n",
    "#         plt.imshow(ims[i], cmap='gray')\n",
    "\n",
    "# Had to tweak these based on the new image objects\n",
    "\n",
    "def show(img, title=None):\n",
    "    if title is not None: img.show(title=title)\n",
    "    else: img.show()\n",
    "        \n",
    "def plots(imglist, figsize=(12,6), rows=2):\n",
    "    f = plt.figure(figsize=figsize)\n",
    "    cols = len(imglist) // rows # divides and rounds to int\n",
    "    for i in range(len(imglist)):\n",
    "        img,lbl = imglist[i]\n",
    "        sp = f.add_subplot(rows, cols, i+1)\n",
    "        sp.axis('Off')\n",
    "        if lbl is not None:\n",
    "            sp.set_title(lbl, fontsize=16)\n",
    "        img.show(ax=plt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show an img..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we know the images are 28x28 and the dataset flattened them\n",
    "# so we turn them back into 2d\n",
    "#x_imgs = np.reshape(x_valid, (-1, 28, 28)); x_imgs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMgAAADSCAYAAAAPFY9jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAABuxJREFUeJzt3T9oVWcYx/HnVVFi0wTqUIXgliytJigG7aI4qqDgIFa8U0EaBEERMmhwcvDPoFJREEGr9IpIUUzFRXBy8e/QRaeKRYhKiEkFA/E4xEK493l/5uTm3nNPzvczPhySV+Xra17PPSckSWIAfPOyXgDQzAgEEAgEEAgEEAgEEAgEEAgEEAgkJ0II34UQ/gwh/BdC+CeE8HPWayqCBVkvANP2m5mNm9n3ZtZjZoMhhGdJkvyd7bLmtsD/pDe/EMI3ZjZsZj8mSfL8y+x3M/s3SZL+TBc3x/FPrHzoMrOJ/+P44pmZ/ZDRegqDQPKh1cxGKmYjZvZtBmspFALJhzEza6uYtZnZaAZrKRQCyYfnZrYghNA5ZdZtZvyAXmf8kJ4TIYSymSVm9otNnmL9ZWY/cYpVX+wg+dFnZi1mNmRmf5jZr8RRf+wggMAOAggEAggEAggEAggEAggNvZs3hMCRGZpGkiTha9ewgwACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQACgQAC7yicQ1avXu3O9+7d685LpVLV7PLly+61Z86cceePHz+e5uryiR0EEAgEEAgEEAgEEAgEEBr6himezTs7enp63Pm9e/fceVtb5Qty0xsZqXwL9aQlS5bU/LWzwrN5gRoRCCAQCCAQCCAQCCBwL1YT6+3tdec3btxw5+3t7e48dlI5OjpaNRsfH3evjZ1WrVu3zp0/evTInce+frNiBwEEAgEEAgEEAgEEAgEE7sVqoMWLF7vzVatWufMrV664846ODncegn9rUezP2Ps04LFjx9xry+Vyqu95+PBhd3706FF3ngXuxQJqRCCAQCCAQCCAQCCAwL1YDXT+/Hl3vnPnzgavZJJ3etba2upee//+fXe+YcMGd75ixYoZr6uZsIMAAoEAAoEAAoEAAoEAAqdYdeI9aX3z5s3utbH7mWJiJ0q3b99258ePH3fnr1+/rpo9efLEvXZ4eNidb9y40Z2n/TU1K3YQQCAQQCAQQCAQQCAQQOAThTVK86T1tE9Zv3PnjjuP3bu1fv16d75y5Up3fuHCharZmzdvprm6SRMTE+78w4cP7jy2xizedcgnCoEaEQggEAggEAggcKvJNHV1dbnzgwcPunPvQdJv3751r/Vu+TAzu3TpkjsfGxtz54ODg6nm9dTS0uLODxw44M537dpVz+XMGDsIIBAIIBAIIBAIIBAIIHCKVWHRokXu/MSJE+5806ZN7tx7vVmpVHKvffjwoTuPnQTl2fLly7NeQirsIIBAIIBAIIBAIIBAIIDAKVaF2OvQYqdVMVu3bq2axR7Xg+bFDgIIBAIIBAIIBAIIBAIInGJVOHnypDuPPYw5djJVlBOrefP8v2M/ffrkzvP2UGt2EEAgEEAgEEAgEEAgEEAo7CnWli1b3HnsYdSxh3zfunVr1taUR7HTqtjv19OnT+u5nFnHDgIIBAIIBAIIBAIIBAIIhT3Fij1zauHChe58aGjInV+7dm3W1tQMYs8FO3LkSKqv472Czsysv78/7ZIyxQ4CCAQCCAQCCAQCCAQCCIU9xUrr48eP7jz2fsE88E6sDh065F4bexfjq1ev3Hnsk5mx9ys2K3YQQCAQQCAQQCAQQOCH9GnK8wejYh8C837w3rFjh3vtzZs33fn27dtnvrAcYAcBBAIBBAIBBAIBBAIBhMKeYsUeohybb9u2zZ3v27dv1tZUq/3797vz2O0j7e3tVbOrV6+615ZKpZkvLMfYQQCBQACBQACBQACBQAChsKdYsYcrx+ZLly5156dPn3bnFy9erJq9e/fOvXbt2rXufPfu3e68u7vbnXd0dLjzly9fuvO7d+9Wzc6ePeteW1TsIIBAIIBAIIBAIIBAIIBQ2FOstObPn+/O+/r63Ln3Sbv379+713Z2ds58YVM8ePDAncceJD0wMDAr33cuYwcBBAIBBAIBBAIBBAIBhBC796gu3yyExn2zr4jdt3T9+nV3vmbNmlRf3/tkYtrf69i9W+Vy2Z0306cb8yBJEv/jo1OwgwACgQACgQACgQACgQBCYU+xYpYtW+bO9+zZ485jz5xKc4p16tQpd37u3Dl3/uLFC3eOdDjFAmpEIIBAIIBAIIBAIIDAKRYKi1MsoEYEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggEAggNfewPkDfsIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIBAIIDwGREbZOIhcxH8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#show(x_imgs[0], y_valid[0])\n",
    "img,lbl = data.train_ds[0]\n",
    "show(img,lbl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Image (3, 28, 28), Category 0)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#show(x_imgs[0, 3:10, 10:15])\n",
    "data.train_ds[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAAF0CAYAAADGqzQSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XuUXlWZJ+B3JxggpOXaoCMQBMJVjIjaIDQ3MUha0IDcBFytYpCLiBLlJgsw0DQgMtraoiigA4gK2ItLFEENIQ3BBoxMzxhplKsIQpBgTEgkOfNHxRmHvYucSlV9X1Xt51mLlfDzfGe/rpyceutQ79mpaZoAAIAajep2AQAA0C2aYQAAqqUZBgCgWpphAACqpRkGAKBammEAAKqlGR4iUkqTU0qzUkoLU0ovpJTuTSnt3e26oDcppb1SSrNTSotTSs+llP5HSmmjbtcFrySltGdKqSn883y3a4NXklLaN6X0k5TSUymlJSmlJ1JK300pbdft2oa71bpdABEppWMi4ksr/pkePd+kvCkixnazLuhNSunvI+JHEXFrRBwUEetHxLkR8eOU0k5N0yzpZn3QwokR8R9/9e8vdasQaGm9iLgvIv41Ip6JiE0j4tSImJNS2qFpmke7Wdxwlmy60V0ppc0i4pcRcVrTNP+9u9VAOyml2yNis4jYpmmal1Zkb42In0XE8U3T/GsXy4NepZT2jIifRsQ7m6a5vcvlQL+klLaOiHkRMa1pmou7Xc9w5cckuu9DEbE8Ii7tdiHQBztHxG1/aYQjIpqm+Y+ImB8RU7pWFUBd5q/49c9drWKY0wx3327R813dYSmlX6eUXkopPZRSOr7bhcErWBYRSwv5koh4Q4drgVVxdUppWUppfkrpmpTSpt0uCNpIKY1OKY1JKU2IiK9GxFMRcW2XyxrW/Mxw9/23Ff9cFBGnR8SvI+LgiPhSSmm1pmm+0M3ioBe/ip6nw/9XSml8RLw2PKFgaFsQERdHxB0R8UJE7Bg99967U0o7Nk3z+24WBy3cExE7rfj9QxGxt+u2f/zMcJellB6MiAkRcVDTNDf8Vf6D6LlJv7bxh8QQk1I6IiKuiojzIuKL0TPY8bWIeHtE/LlpmjW7WB70SUrpzdHz8+7/3DTNZ7pdD7ySlNK2EfHqiNg8IqZFxEYRsVvTNI90s67hTDPcZSmlu6PnCdurm6b541/ln4iIz0fE65qmebJb9UFvUkrTo+dGvEZENBHxnYhYKyLe0DTN5t2sDfoqpfS/I+Lxpmn27XYt0FZKaZ2IeCQirm2a5qNdLmfY8jPD3fe/esnTil+Xd6oQ6Iumac6MiA0i4o3R818wDo+e/8oxu6uFwapJ0fNNHQwbTdM8Hz0/KrFlt2sZzjTD3ff9Fb++/GnEvhHxRNM0T3W4HmitaZo/NU3zP5umeTql9K6I2Ca8GYVhJqX0lojYKnp+FhOGjRUbHW0TPfNGrCIDdN03I3reefnVlNIGEfGbiHhfREyKiA92szDoTUppx4jYLyLuXxHtFhGfiogLm6a5q2uFwUqklK6OiIej59p9PnpmM06LiN9GxL90sTR4RSml70fPdftA9Ax/bhURn4ieDWO8Y7gf/MzwEJBSenVEnB89TfC60fOqtX9umuaarhYGvUgpbR89r/R5Q0SsHj0bx/xL0zRXdLUwWImU0mkRcXhEjI+eXT6fiogfRMRZTdP8rpu1wStJKZ0SEYdExBYRMSYiHo+ImRFxvuG5/tEMAwBQLT8zDABAtTTDAABUSzMMAEC1NMMAAFSro69WSymZ1qPfmqZJKz9qYLl2GQidvnZdtwwE91yGq7bXrifDAABUSzMMAEC1NMMAAFRLMwwAQLU0wwAAVEszDABAtTTDAABUSzMMAEC1NMMAAFRLMwwAQLU0wwAAVEszDABAtTTDAABUSzMMAEC1NMMAAFRLMwwAQLU0wwAAVEszDABAtTTDAABUSzMMAEC1Vut2AQysNdZYI8u23nrrLDvxxBOz7EMf+lCWzZ49u7jO/vvvn2XPP/98mxIBAIYMT4YBAKiWZhgAgGpphgEAqJZmGACAaqWmaTq3WEqdW6xSU6ZMybLrrrtuwNe55pprsuyoo44a8HVKmqZJHVnor7h2GQidvnZH2nW75557ZtlZZ51VPPaOO+7IsrPPPnuAK6qDe25nnXrqqVm25ZZbZtmmm26aZfvss0/rdVLK/1jb9oQ33XRTlr3nPe9pvXantL12PRkGAKBammEAAKqlGQYAoFqaYQAAqmUHumGgtKvcpZdeWjz28MMPX+V1nnzyySxbtGhR8dgHHnhglddhaFprrbWK+UMPPZRlG264YZaNGpV/b718+fIsu+WWW4rrlIab7r///uKx1OmnP/1p62NLw3YlhurohM033zzLSoNyERFHH310lpUG25YuXZplpfv19ddfX1znlFNOabVOyeLFi1sdN1x4MgwAQLU0wwAAVEszDABAtTTDAABUyw50Q0xpCOnyyy/Psv7u9nb11Vdn2Sc+8Yksmz9/fr/WGQx2Q+qb1VdfPcumTZuWZZ/85CeLn1977bWz7Omnn86yu+66K8t23XXXLCsN30VEzJ07N8ve8pa3FI8druxA115pWK7tUFxfzJw5M8v22muvAV9nOHPP7V3p/jp16tQsu/DCC7NszJgxxXOWdoZ78MEHs+wzn/lMlvVlx9nSgHPbnrA0/HfRRRe1XrtT7EAHAAAroRkGAKBammEAAKqlGQYAoFqaYQAAqmU75i7abLPNsuzWW2/Nsi233LL1OV944YUs++xnP5tlX/jCF7KsNFnK8Pe5z30uy4499tgse/TRR4uf/8EPfpBlH/zgB7Psz3/+c5aNHTs2y6688sriOvfee28xZ+QrvSViMN4c0Xbt0kR96a0TERF33HFHlu2xxx6rvHZvSuufc845rY6j/3bfffdiftppp2XZpEmTWp3z+eefL+al7ZPPOuusLPvd737Xap3jjjuu1XF9sWDBggE/Zzd5MgwAQLU0wwAAVEszDABAtTTDAABUy3bMHbLWWmtl2Q9/+MMse/vb397qfE888UQx33vvvbPs17/+datzDhe2Bu3duHHjsqw0ULN48eIs22+//YrnXLhwYb/rGkiloaHSfez8888vfn7JkiUDXlNbtW/H3NvAWGnrZVau9Hfh7LPPHvB1arvnvvvd786yG2+8sXhs2x7q9ttvz7Ljjz++eOxDDz3U6pwl48ePz7LSVvcREWuvvXaWlf7/lIar3/jGN2bZUPtaEWE7ZgAAWCnNMAAA1dIMAwBQLc0wAADVsgNdhxxzzDFZ1nZY7rHHHsuy3oadRtqwHGWlQbmIiEsvvTTLSgMVRxxxRJYNxeGH0u6Jpd3zli1blmWXXXZZ8Zy//e1v+18Yq6S0i1Zf9HdgrDTAV6qpU7vflYZbSzva9WYwhuWIuPnmm7Ost10FJ0yYkGWlYbnS1/HBULomSoNyEREp5bNlpZ1EP/CBD2TZUPx60R+eDAMAUC3NMAAA1dIMAwBQLc0wAADVsgPdAJs8eXIx/853vpNlY8eOzbLSn8cBBxyQZTNmzFiF6kaG2nZDKvnc5z5XzE866aQsK+10WNphaSi65ZZbsmzffffNsgsuuCDLzjjjjEGpqT9q2oGuNITW353mSgM/DD733O4bNSp/djlt2rQsO/fcc7Ns9OjRxXOW/j5NmjQpy0oDgcOFHegAAGAlNMMAAFRLMwwAQLU0wwAAVEszDABAtWzH3A+vec1rsuzb3/528djSmyNKTjvttCyr+c0RlG2yySatj7388ssHsZKB8973vjfLSluW//u//3uWlSaoGXlKb6PobftiWxUzkpTeHHH++ef365y33nprls2ZM6df5xyuPBkGAKBammEAAKqlGQYAoFqaYQAAqmU75n74yle+kmVTp05t/fnSIFBp6+Xnn3++b4WNcLVtDbreeutlWW/b2m6//fZZVhrqPOqoo/pfWAs777xzlu2www7FY0844YQsK/3/KQ3a3XzzzatQXefVtB1zSW/XbWnr5sEwc+bMLDvnnHNaHVez2u653XbFFVdk2RFHHJFlpW2Wly5dmmWlQbmIiMMOOyzLXnzxxTYlDhu2YwYAgJXQDAMAUC3NMAAA1dIMAwBQLQN0Lb3+9a/Psrlz52bZuHHjip9fuHBhq3M+99xzq1BdXQxzRHz6058u5v/0T/+UZUuWLMmy0i5DBx10UPGcpby0o2Jp98RRo/Lvt88444ziOl/4whey7Be/+EWW7bXXXllWGhoZimofoOuL0g5ye+yxR5YNxvBd20G73o4dadxz+680AHfggQcWj50yZUqWte3VFi9enGU///nPi8deeeWVWXbddddl2YIFC1qtPRQZoAMAgJXQDAMAUC3NMAAA1dIMAwBQLQN0LX3+85/Pso9//OOtP3/66adn2Ze+9KUsKw1G3XPPPVk2Y8aM1muPNIY5yjsPRUTccsstWbbPPvu0Omdp8CIiYo011siylPI/gunTp2fZRRddlGUHH3xwcZ1vfOMbWXbhhRdmWenv0nBhgG7g9TZAV8rPOuusAV+/NFhXGv4bztxzI3bddddi/tGPfjTLSsNyffHHP/4xyx544IEse/DBB7Ns4sSJWfbmN7+5uE7pPl4aWj7llFOyrLdd7YYaA3QAALASmmEAAKqlGQYAoFqaYQAAqmWArqXbbrsty/bee+8s+93vflf8/Jve9KYse8973pNlX/va17KstHvd+9///uI6pQGqkcYwR99sscUWWXbIIYf065wXX3xxlpV2gRs/fnyW/eY3vymec/ny5Vk2efLkLCv9XRwuDNB1V2mobjAG7Uq70pV2Thwu3HPLA+8REccee2yrz8+fPz/Lbr755uKxpQHMxx57rNU6Y8aMybLS7p4REcccc0yWlXrCUg+y//77Z9msWbPalNhRBugAAGAlNMMAAFRLMwwAQLU0wwAAVEszDABAtbxNoqW2b5P44Q9/WPz8gQcemGV33nlnlu20006t6rn88suL+Uc+8pFWnx/OTDYPH6Up5uOPP7547OzZs7Ost612hytvkxi++vu1svQ2idJbJ4Yi99yI9dZbr5jvttturT5f6g1Kb+DppNIbfNpe56eeemqWXXTRRf2uaaB5mwQAAKyEZhgAgGpphgEAqJZmGACAaq3W7QJqsWTJkiybOnVqlr3zne/MsuOOOy7LJk2aVFxn9OjRWbZs2bI2JUK/jBqVf2+99tprt/58b9udwlBwzjnnFPO2WzeXhkGHywAdEc8991wxv/HGGztcCYPBk2EAAKqlGQYAoFqaYQAAqqUZBgCgWgboWrrvvvuyrLQDXV/MnTs3y37xi19k2cSJE7Ps8MMPL56zNMRkgI5O2HzzzbPsyCOPzLKnnnqq+Pk5c+YMeE0Atdtqq60G/JwPPvjggJ+zmzwZBgCgWpphAACqpRkGAKBammEAAKplgK6lGTNmZNlJJ52UZeuvv37x82uuuWaWLV68OMs222yzLOttWA6GkjPPPLPVcQsXLizmTzzxxECWwzB39tlnZ1lpx7bB2MWttHbbneZ6Y7c5OmHs2LFZdt555xWPTSm1Ouett97aKhvOPBkGAKBammEAAKqlGQYAoFqaYQAAqmWArqVZs2Zl2aWXXpplH/vYx4qfv+qqq7LsrrvuyrKpU6e2qud73/teMX/ppZdafR4GWmm3uaZpsqz0d4G69WdgrS+DaXvuuWerdUrH9cVee+2VZQbo6IQddtghyw488MDisaX783PPPZdlJ598cpa9+OKLq1Dd0OXJMAAA1dIMAwBQLc0wAADV0gwDAFAtzTAAANVKpWnCQVsspc4t1gEf+chHsuz8888vHrvuuuuu8jp/+tOfsqw0uR8RceONN67yOsNF0zTt9pAcQCPt2h0My5Yty7Jnnnkmy7bbbrvi50tTzCNNp6/d4XzddvJr00AbaW+TcM/tvgkTJmRZ6e1Vpb5kzJgxxXOWtmOeNGlSlt1+++1tShyS2l67ngwDAFAtzTAAANXSDAMAUC3NMAAA1bIdcz9cdtllWfarX/2qeOxNN92UZePGjWu1zvTp07OshkE5hq5jjz221XFf/vKXs6yGQTn675xzzsmytls0D4beBuBKw3J0ztixY4t5aeDs8MMPz7J77703y2bPnp1lTz311CpU98rWWWedYv6+970vy0pblr/2ta/NsqVLl2bZvHnziuscffTRWfazn/2seOxI58kwAADV0gwDAFAtzTAAANXSDAMAUC0DdANs1qxZxXzttdfucCUweHob/Hi5P/zhD4NcCSNVaWCopL9DdaXBuNLw3nDeQW4k23HHHYt5b1+L2ygN0H3rW98qHlvaKfEd73hHlm266aZZtuGGGxbPWRr+K63z6KOPZlnp701vtfP/eDIMAEC1NMMAAFRLMwwAQLU0wwAAVCuVfih70BZLqXOLMWI1TZM6vaZr9/83ZsyYLHvhhReyrLQ719133z0oNQ0Hnb52XbcMBPdchqu2164nwwAAVEszDABAtTTDAABUSzMMAEC1NMMAAFTL2yQYdkw2M1x5mwTDkXsuw5W3SQAAwEpohgEAqJZmGACAammGAQColmYYAIBqaYYBAKiWZhgAgGpphgEAqJZmGACAanV0BzoAABhKPBkGAKBammEAAKqlGQYAoFqaYQAAqqUZBgCgWpphAACqpRkGAKBammEAAKqlGQYAoFqaYQAAqqUZBgCgWpphAACqpRkGAKBammEAAKqlGR5iUko/TCk1KaVzu10LvJKU0sYppX9JKd2dUlq04rrdrNt1wStJKe2bUvpJSumplNKSlNITKaXvppS263Zt8Epcu4NHMzyEpJQOj4iJ3a4DWtoyIg6JiD9ExJ1drgXaWi8i7ouIEyJiUkScFhHbR8SclNL4bhYGK+HaHSSpaZpu10BEpJTWiYh5EfGJiLgmIs5rmuYz3a0KepdSGtU0zfIVvz86Ii6LiNc3TfNIVwuDPkopbR09999pTdNc3O16oC3X7sDwZHjouDAi/lfTNN/udiHQxl8aYRgB5q/49c9drQL6zrU7AFbrdgFEpJR2i4gPhB+RAOiIlNLoiBgdEeMj4p8j4qmIuLarRUELrt2BpxnuspTSqyLiqxHxuaZpftXtegAqcU9E7LTi9w9FxN5N0/y+i/VAW67dAebHJLrvlIhYMyLO63YhABU5KiJ2joj3R8QLEXGbt6EwTLh2B5hmuItSSptGxBkRcWZErJ5SWmfFIF381b+P7l6FACNT0zS/bJrmnhVzGu+IiHERcWqXy4KVcu0OPM1wd20eEWtExFXR83qqv/wTETFtxe936E5pAHVomub56PnPzVt2uxboC9fuwPAzw901NyL2KuQ/jZ4G+RvRc5EDMEhSShtFxDYRcXW3a4G+cO0ODM1wF634jm7my/OUUkTEo03TZP8bDCUppfet+O1fhjn2Syk9ExHPNE1zR5fKgl6llL4fEfdHxAPR8/OWW0XP+91figjvaWXIcu0OHs0w0B/fe9m//+uKX++IiD07Wwq0Mid6dk48OSLGRMTj0fNQ4nwbxjDEuXYHiR3oAAColgE6AACqpRkGAKBammEAAKqlGQYAoFodfZtESsm0Hv3WNE3q9JquXQZCp69d1y0DwT2X4arttevJMAAA1dIMAwBQLc0wAADV0gwDAFAtzTAAANXSDAMAUC3NMAAA1dIMAwBQLc0wAADV0gwDAFAtzTAAANXSDAMAUC3NMAAA1dIMAwBQLc0wAADV0gwDAFAtzTAAANXSDAMAUC3NMAAA1dIMAwBQrdW6XQAwtB155JFZdvHFF2fZY489lmVvfetbB6UmAHL77LNPln3oQx8qHrvHHntk2aJFi7Ls6aefzrLS14VHHnmkRYVDkyfDAABUSzMMAEC1NMMAAFRLMwwAQLUM0AGvaLfddsuy9ddfP8vGjh2bZdtss02WzZs3b2AKA6jExhtvnGXTpk3LstKw3Lhx4/q19hZbbJFlhx56aJZdcMEF/VqnmzwZBgCgWpphAACqpRkGAKBammEAAKqVmqbp3GIpdW6xEa60c0xExMyZM7Ns+fLlWbbrrrtm2Zw5c/pdVyc0TZM6vWbN127p+indN1LK/1i++tWvZtmxxx47MIUNQ52+drt53R5xxBFZ1tuOhKVhnI022qjVOnPnzi3mb3rTm7Lsi1/8YqtzlsyePbuYX3fddat8zuHCPbezTjjhhCy78MILs2yNNdbo1zqPP/54lo0ZMybLSgPTu+++e5YNxR6i7bXryTAAANXSDAMAUC3NMAAA1dIMAwBQrap3oCv9UHhExFZbbZVlRx99dJZdcsklWfaf//mf/S+shTPPPLOYtx12mjx5cpYNxR9+p/tK108nB28Z+l73utdl2WWXXZZlq6++eutztr3GJk6c2PrzJ5544iqvc8wxxxTzo446Kst+/vOfZ1np68WCBQtarc3wt95662XZtddeWzx2n332WeV1HnrooSwrDd9FRNxzzz1Z9q1vfSvLrr766iwbaf2CJ8MAAFRLMwwAQLU0wwAAVEszDABAtTTDAABUq+q3Sey9997F/Nvf/naWlbaaPeigg7Jst912K55zoN8yMWHChAE9H/SmdO23Pe7ZZ58d6HIYgjbffPMsGz16dL/OWZpgf/HFF/t1zl122SXLtttuu1afLW1TGxHx7ne/u1W28cYbZ1npLUUML/vvv3+WHXLIIVm23377ZVnpDRO9+fGPf5xl11xzTatsyZIlxXN+85vfzLItttgiyz74wQ+2KXFY82QYAIBqaYYBAKiWZhgAgGpphgEAqFbVA3T9NW7cuCzbfffdi8d2apvmtmbMmNHtEhgmfvnLX2bZ1ltv3YVKGKruvPPOLFt33XX7dc7FixdnWX+3AS8Nwa22Wrsvg8cff3wxnzJlSpb93d/9XZYdeeSRWXbppZdm2b333tuqHjprp512Kub/9m//lmWlYeLS8OfXv/714jkvvvjiLCtts7xs2bLi51/utNNOK+YHH3xwlp188slZNnfu3FbrDGeeDAMAUC3NMAAA1dIMAwBQLc0wAADVqnqArredtUr5qFH59w3Lly8f8JpKxo4dm2W9DX2Uap8zZ06rDEpKw1HbbLNNlpWuvQ022GBQamLoW7RoUbdLyCxdurRVVnLRRRcV8yuuuCLLzjzzzCw74YQTsmzixIlZZoBuaOrta+4zzzyTZZdcckmWXXvttVn26KOP9r+wlyndcw877LDisTfccEOWfeUrXxnwmoYDT4YBAKiWZhgAgGpphgEAqJZmGACAalU9QNfbbkalvDQsN3PmzCz7zne+0++6Xu4f/uEfsmyjjTYqHluq/cknnxzwmqhbf3cCg5Hi2WefzbLSgHJpgI7ho7fBxvHjx2fZkiVLBnz90rDlcccdl2WHHnpo63Puu+++/appJPFkGACAammGAQColmYYAIBqaYYBAKiWZhgAgGpV/TaJvkxdlpS2LZw/f36/zjkY+vOGi9JW0BERO++8c5b95Cc/WeV1GLoOPPDALCttvVzKtt1220GpCTptzJgxxfyAAw7IsrPOOqvVOV966aV+1UTnLFu2rE95f5S+Zr/3ve/Nsle96lX9Wuf666/PssmTJ2fZggUL+rXOcODJMAAA1dIMAwBQLc0wAADV0gwDAFCtagbodtpppywr/aB4bxYuXJhl//Vf/9WvmjqltJ3zJptskmVvfvObs2zvvfcunnPdddfNshNPPDHLvv71r7cpkSHshhtuyLKjjz661We32WabgS4HBt2ECROy7Oqrry4eW/raUvLxj388y775zW/2rTBGlA9/+MPF/OCDD271+VtvvTXLSoP906dPL35+l112ybJTTjkly04//fRW9QxnngwDAFAtzTAAANXSDAMAUC3NMAAA1apmgO6d73xnlq2++uqtP18aoHvuueeyrDSY1puTTjqp1XGHHXZYlpV2++ot/8AHPtC6prbrLFmyJMvuv//+VV6H4aXtDnTf//73O1EOI9C73vWuLCvtArfjjju2Pufuu++eZU3TZNnb3va2LFtrrbWK5yzd9y655JIsu+6669qUSEWmTp1azP/0pz9l2Wc/+9ks+/KXv5xlixYtyrLSoF1ExJNPPplle+yxR/HYkc6TYQAAqqUZBgCgWpphAACqpRkGAKBa1QzQlYYsSoMTvXnNa16TZQ8//HC/aioNHLWtqS+19+XYl7vggguK+W233ZZlBujq0faa2nbbbQe5EkaCAw44IMuuv/76LBs1qn/Pb/pzz+3N/Pnzs6w06Ld06dJ+rcPIc9RRRxXz0nU+b968VV6nNPAeEfHQQw9l2XbbbZdlG220UZY9/fTTq1zPUOTJMAAA1dIMAwBQLc0wAADV0gwDAFCtagboHnzwwW6X0DVz5szJsieeeKLVZ08//fSBLocRoO0OdH//93/fiXIY5kr3mdGjR7f67FNPPVXMS0NsL7zwQpa1HaDbbLPNivm+++6bZZMmTcqy2bNnZ1lpgIl6dLsvueOOO7LsrW99a5aVdqX77ne/Oyg1dYsnwwAAVEszDABAtTTDAABUSzMMAEC1NMMAAFSrmrdJnHvuuVm24YYbFo/98Ic/PODrX3bZZVlW2s6wtKXxDTfc0HqdW265JcsOPfTQLFu8eHHrc8LLDca24dTrU5/6VJats846rT579913F/NXv/rVWfab3/ymb4X9lenTpxfz0psj3vKWt2RZaXvpiRMnrnI90F+zZs3KsmnTpmXZFlts0YlyusqTYQAAqqUZBgCgWpphAACqpRkGAKBa1QzQLVmyJMuOOeaY4rH33ntvlv3N3/xNlt18881Z1t/tFXfZZZcsK21z25ubbropywzLMdDabscMbdx5550Dfs5nn312QM935plnFvMrrrgiy+66664s23777bPswAMPzLK+DEzTf1OmTMmy3XbbLcv+9m//tvj5Ur9QGpgfil+HFy5c2Oq4LbfccpAr6T5PhgEAqJZmGACAammGAQColmYYAIBqVTNA1xelH37vlMmTJ2dZX3bx6mbt1MMOdNCjtKvdww8/nGUbbLBBlq255pqDUhPtfexjH8uyPffcM8t6GzYr5ePTV/0/AAAELklEQVTHj8+yefPm9b24Qfa6172u1XGlIcGRxpNhAACqpRkGAKBammEAAKqlGQYAoFoG6IaYrbbaqtslwEqVdpsbNSr/3nr58uWdKAe6ZsKECVlW2rFr0aJFWXb11VcPSk2098ILL7Q6buzYscV82bJlWdZ2WLJTu9L1NihX2gHxmWeeybIrr7xyoEsacjwZBgCgWpphAACqpRkGAKBammEAAKplgA7os9LOcqVhOTvQMdKtv/76Wbbeeutl2eWXX96Jcuijww8/PMt+/OMfZ9kuu+xS/Pzxxx+fZSeccEKWPfLII1l2yy23FM9ZGmKbNWtW8diXe8Mb3pBlH/3oR4vHloY/zzvvvCzr1KBfN3kyDABAtTTDAABUSzMMAEC1NMMAAFRLMwwAQLVSJ6e9U0pGy//KJptskmWlidO+GD16dL8+Pxw0TZPvBTzIar52zzjjjCw79dRTs2zcuHFZVnrDRA3XaG86fe3WfN0OtE9/+tPFfMqUKVn2xje+Mcu23XbbLHvsscf6X1gH1HbPXX311bPs0EMPLR573HHHZdnb3va2Aa+pPxYsWFDMp06dmmXf+973Brucjmp77XoyDABAtTTDAABUSzMMAEC1NMMAAFTLAF0XlQboHn744X6dc7XVRv4O27UNcwxFp59+epZNnz49y375y19mWWm70FrUNEBXuheVtr6NiJg5c2aWPf744wNdUtH48eOz7Nhjj82yT37yk8XPl4bg3v72t2fZ73//+1Wobmhwz+1d6Trfeeeds+wf//Efs6w0aBkRsd1222XZDTfckGULFy7MstK2zaXtpSPK2z6PNAboAABgJTTDAABUSzMMAEC1NMMAAFTLAF0Xrb/++ll23333ZdnGG2/c+pwG6AaHa5eBUNMA3ZgxY7JsxowZxWO33377LDvppJOybPHixa3X33HHHbNs9913z7Iddtghy0r35nnz5hXXOe+887LsmmuuaVPisOGey3BlgA4AAFZCMwwAQLU0wwAAVEszDABAtUb+tNUQNn/+/Cy75557sqw0QHf//fcPSk0AA2Hp0qVZ9qMf/ah47F577ZVlgzGEllI+S1MaIr/qqquy7OSTTy6e89lnn+1/YUBXeTIMAEC1NMMAAFRLMwwAQLU0wwAAVEszDABAtWzHzLBja1CGq5q2Y2bkcM9luLIdMwAArIRmGACAammGAQColmYYAIBqaYYBAKiWZhgAgGpphgEAqJZmGACAammGAQCoVkd3oAMAgKHEk2EAAKqlGQYAoFqaYQAAqqUZBgCgWpphAACqpRkGAKBammEAAKqlGQYAoFqaYQAAqqUZBgCgWpphAACqpRkGAKBammEAAKqlGQYAoFqaYQAAqqUZBgCgWpphAACqpRkGAKBammEAAKqlGQYAoFqaYQAAqqUZBgCgWpphAACq9X8ANmR5drU3duQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 8 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "inds = [random.randint(0, len(data.train_ds)) for _ in range(8)]\n",
    "plots(data.train_ds[inds])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PYTORCH Adventure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.metrics import *\n",
    "#from fastai.model import *\n",
    "#from fastai.dataset import *\n",
    "\n",
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we are building a 3 layer NN, the input is 28 * 28 (no colour so only 2D).  We then do an internal linear transformation layer (100 --> 100) then output finally to 10 categories (0 -> 9 = 10 categories)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can still use the normal sequential model, however there is a new `create_cnn` function.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = nn.Sequential(\n",
    "    Flatten(), # Add a flatten layer, as the new structure needs this\n",
    "    nn.Linear(28 * 28 * 3, 100), # added * 3 here as channels are now in\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(100, 100),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(100, 10),\n",
    "    nn.LogSoftmax()\n",
    "    )#.cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'machine_learning_data/mnist_new/'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Recall this set at the top of the file...\n",
    "path#, x.shape, y.shape, x_valid.shape, y_valid.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't need to load the data, we already have it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#md = ImageClassifierData.from_arrays(path, (x, y), (x_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#type(md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optimiser loading has slightly changed into a partial functions, presumably so more params can be injected later?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = nn.NLLLoss()\n",
    "metrics = [accuracy]\n",
    "# 1e-1 = learning rate\n",
    "#opt = optim.SGD(net.parameters(), 1e-1, momentum=0.9, weight_decay=1e-3)\n",
    "opt = partial(optim.SGD, lr=1e-1, momentum=0.9, weight_decay=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def binary_loss(y, p):\n",
    "    return np.mean(-(y * np.log(p) + (1-y)*np.log(1-p)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_log(idx, p, y): \n",
    "    return np.log(p[idx]) if y[idx] == 1 else np.log(1-p[idx])\n",
    "\n",
    "def binary_loss_if(y, p):\n",
    "    return np.mean(-(np.fromiter((get_log(idx, p, y) \n",
    "                                  for idx, _ in enumerate(p)), float)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.164252033486018, 0.164252033486018)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acts = np.array([1, 0, 0, 1])\n",
    "preds = np.array([0.9, 0.1, 0.2, 0.8])\n",
    "binary_loss(acts, preds), binary_loss_if(acts, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We make a learner class now to handle the model fitting.  The old fit function should still work I think... but anyway, we just pass params to the learner now and use learn.fit instead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, net, loss_func=loss, opt_func=opt, metrics=metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#fit(model=net, data=data, epochs=5, loss_func=loss, \n",
    "#    opt=opt, metrics=metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 01:57 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.155069</th>\n",
       "    <th>0.158309</th>\n",
       "    <th>0.951100</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>0.115493</th>\n",
       "    <th>0.107260</th>\n",
       "    <th>0.968100</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>0.099199</th>\n",
       "    <th>0.096462</th>\n",
       "    <th>0.969700</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n"
     ]
    }
   ],
   "source": [
    "learn.fit(epochs=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([235200, 100, 10000, 100, 1000, 10], 246410)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = [o.numel() for o in net.parameters()]\n",
    "t, sum(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAGQCAYAAABYn1CDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXe4FsXZ/z+3gCAgUsSKgKJYULFgmg0bgj1ijNFoNJZXjdGoiVHz8xVjjRrfRPNaEgt2TXwt0WCPYIsNFaWosSBoRKlSRcX5/TFzltnl2ec8pzznnD3n+7muc5179p6dmZ1yP7P3zu6Ycw4hhBDFZKXmLoAQQoj6IyMuhBAFRkZcCCEKjIy4EEIUGBlxIYQoMDLiQghRYGTEq4yZDTUzF/6GhmOjao41Uh5TQ3qjGyO91kildW5mo0O8qU1UtBZDY/YjMzsy6vf96xKvVBuoj+cjI948fAS8GP4qotSPQcRrIa33Gq+IJcuwdch/ZCOmWdFgryZmdqqZLTOzNZoj/yzVqOcWzEyWj4WlZeKt0Mdl2D3tm7sALR0zW9k592Vjpumcux64vhHT+35jpVULBwBfAI80UX5NxQHA8865z8ysucsCjVjP1ei/jYlz7h/APyqI11R9vHC0mZl49Kt9i5mdb2afmtkiM7vDzFbLxLnVzH5vZrMIs2UzW9nMzjGzt81sqZnNDuf2yeRzvJlNM7PFZvYgsG6JspS8tTezg83sWTNbEM6faGYHmNko4Kko6lPxDKTUjMTMeprZn0JZvjKzz8zsTjMbUKocZraLmb1qZkvC/++UqMYDgCecc4vMrJ+ZPWxm08M5S0J5f2GRJTTP8WY2PlzTwiB/L5T3pij9D0JZRoVzXRwOx8aGY2OjY5eZ2SQzmxeu9T9mdrOZrV3iGrJt0QvYHri/TJx2ZnZ6yGOpmc03s3+a2W6ZOBeY2buhLuaa2etmdkkUZ7iZPRd0S8zsAzO7z8zWL1PP8Z3K/uH8L8zs32b2/SjtON4PzOwVM/sS2CvodzCzR83s83ANb5vZb8ysQ+lLtnPMbIZlxkhQ/jJc25yob91rZgNzqnAzM3u6gnL3L9MGSR83s/7mx06/oP5JlMaekbxpdP5Pw7ElZtY9L59C4pxrE3/AVMDhZzjzgLdD2AF/zcRZGv7eBJ4JugeDbhnwBjAnhD8EeoQ4e0Vpzsbf+i2Mjg0N8UbVHIvKd3oUb37IY2GIewwwOdJPBl4AzsmUe3QIdwpld8DXwCRgSQjPBPpkyxHq5S3gqxCeCrSPytc/HD86hIeE8HTgVeDTKK2fReddFR2fA0wMeR0JnBPqqEb/WriuY8K5NcdHRemNDcfGRscmhjZ9E5gCfBPivBTFWaHOw/GjwvEBITy65vqjONdHZXkXmMXyvjAixPl5VN8T8P3rC+DdoF8d36ccMA14Hd9HHLBDmXo+skQbzY/y2qJEvKXAx8C/gf2BoVG7ziXd9+8sMUYWkjNGQryHQpzJoc6/jq6rU4nyLKyw3P3LtEFN2UYDa+P7SU19zgzhFwAD3gnHL4vOHxOO3dXctqjRbVtzF6DJLnR5J5gNrBGO/SEc+wbYgLQR3zLEaQfsFHW0YeF499B5HPCbcGxcTecDVgvHbo3OHRqOjao5FsKdWW7sXwS6R8c3CfLQbDqlOngIHxXFPSgc2zwabL/PlgP4eTh2cnRskyiPU/FGa43o+vtH+pWi66/54evPcoP6AMsHeA9g/SAfSWYQR2lWasS3BFaKwsdE5w4oVedR3AeAN6Pw6Jo2DOENomv4Uzi2KssNxfhwrObH6oYorVWA7wV5W5b/QHfOlL13mXqO6+eCcKwPy/vLzSXi3V5TH/j+W9Mu01g+4bgkil9jUKdSyxgJxwYBHaIy7x6ltVsDyt2/VBuU6uN5x6I6dPiJRXtgNZYb/OHNbYsa+6/NuFMixjrnPgvy3eG/4TtmDU85594AcM4tA74d6R4Nt3Jz8bMrgBrXwxY1cZxznwf5rxWUaRDQJchXO+fmhbwXO+fequD8LNuF/18C/xfSmoif3YOfRWe5NfyfHB1bM5ITv3EIfwWcYWYfmtlXeMOzU9CtE5WjxrVyhXPui1CWuc65D+p8VfkMBl4OrhoH/CXSrZNzDmbWGdiDMq4UfF3VXMMdAM65BfjZKMBWZtYuhB3wUzP7xMzGARfijTb4u6H38T8An5nZa2Z2G7AZfmZfQ7aeY+4O+X8EPBeObV4i3lXOuW9C3GUs7w+POOfmxtcSXWNMbWOkL96lN9/MvgEej84tVd+VlruxuAlYDKwB7APsB6wMfEK6rK2Ctv5gM+8p1owy8V7CD9aYafXIoynIljM/YvjhwM/WazBI+Y1/Hen+gJ/xgr9tnwMMwP+wtatnefOI01stVpjZDsDNoayz8T9CXYFNS5ybZU/8bLmcEY/JrU/n3KNmtg3wA/yPytb4H7VjzWwz59x0M9sWOBw/KdgMOBQ4DO8e+J+ces6jXL/K9t9ay19pPma2Ab6+VgYWAOPxdmSrEKW2tq/6eHDOzTOzO4Gjw9+yoLo1/Ki1KtriTHxnM+sd5IOi45PKnPNSJF/hnPuOc+47wHeBM4Drgm5i+D/MzLoFuZJlYpOARUE+vuZcM+tkZhuH44uj+F0oz8vhf8ea/M1sc/ytO8ArFZQpZj/84IyNXc3dx2POuYF4d8/HJcpRYzh+YWYdQ1lWix5ilbuumtnggHDehqw4g/s2yw3DFs65bwG31HpFngOA6c658WXijI+u4bBQjlXxMzyA151zy8xsS+Az59xvnHP7sLx+ugLfCm26Cd4l82Pn3DYsf1i9a/hfqp5jfhDyXwff92B5nytHTX8YYWY9gnxopM/2h3JjZGu8AQfY0zm3HfC7WvKvb7lro6bvlBoP/xv+jwCGB/nmRsiz5dHc/pym+qP8Q5u/ZeKMLnH+mCj+O/gHOjUPao4McfaJ4szCP7T7Ijo2NMQbVXMsSj/7YHNC+D8q6Hvh3SM1PssXWO7vTpWb0g82F7P8IdAKDzajcgwtUd6U3zgcuz2K93ZIt+ZB3dQoXvxgczbepbM4qrMtI/0n4bq2D7o7It24UKfLiHzieHdIXOdTonLk1jneWM7Cux7i6xpd4hoqebB5Ad5vPA1v+Guel3yNN94bhvCcUAdvRWleWKaej4ziLQzX93mU/5Yl4vXPpDGUhj/YrBkjm7D82crn+H42M4p3ZEPLndMGNWUbHR27N0pvPHBT5rqfj9J+Kda1pr+2OBP/P+D3+NvyxXh/3bEVnPd94Fz84OuHf0jzfkhrLIBz7iHgJPyMtAveiJ9QSaGcc78HfojveCsBA/ErPyYE/Wz8Q8fp+AeD3wbWyknrC2Bn/Gzkk5DWonCt33HeN1kRZfzGp+GNzkK8n/cy/AqeLCcDJ+JXnnTGPyicgnfB4Pyzh/PxD6HWCtdVM1s8Db+GeCGwPn7G92zmWh/Hux/+g3eNvEVldb4T/oexElfKfwG/wrtq1sPf4TyFf8j9cIgzDv9Db/i7hfb4thzp/HON2Xhf7Sf4B7798P3jd8B5FfrnD8YbzI7h3B+G+iuLc24ssAvwGL5vrY+fiJwDHFHilNwxEq7lp8AH+Bn5LOBHtRShXuWugP+H/9H/EtiG5c+kavjfSG6ds3DAwi9Wq8f8K7z98E/Fj2ze0hSHsKb3XmCIK+92KBRm9ke8f3oN59zXtcVvgvKUrGczO5Lla+nXd85NbfrSFZPwjGI8/m54XefcnGYuUlVoizNxUTcWAf+vNRnwwCT8sspmN+CB1lrPTY6ZbWpmd+DvFMEv+2yVBhy0OkXUgnPuMfxteKvCOffn5i5DTGut52ZiTbyLZxF+ie8ZzVuc6tJm3ClCCNEakTtFCCEKjIy4EEIUGBlxIYQoMDLiQghRYGTEhRCiwMiICyFEgZERF0KIAiMjLoQQBUZGXAghCoyMuBBCFBgZcSGEKDAy4kIIUWBkxIUQosDIiAshRIFpciNuZmPN7AszWxj+3i4R589mdlyQe5vZHWY2z8zmmtnttaS/jpmV3H7MzLYys2fM7HMz+8jM/rtMOj8xs/FmNj/EvdTMVvj+upltFK7ntgqu/TEzG1biuJnZ78xsdvi71MxydwU3s0PN7EMzW2Rm95tZz0jX08zuC7oPzezQvHQaEzO7zcw+CfX1jpkdUyLO2WZ2UZCPMbN3Qx94JGyiWy79lc1slpl1LaFbmPlbZmZX5aSzuZk9GtJa4TvMZtbfzMaEvjbDzP5Uqt3zrquELretSsTdKvS5xeH/VpGuTn2ksaikbjPterCZTTGzBWY22cwOqCCPd8xsYI5udzN7NdTfdDM7uIL0bjIzZ35j7Yqvo0Q6iR0qoTs19I/PzexGC5uA58TdzczeCu36lJn1i3Qdw/nzQ3qn1XZ9K9DUm3ri96M8ppY401i+me8zwBX4/f46AFvXcu4xwPU5usnAhfhNcgfg9zvcLyfuCcCO+H0E18Vv83RmiXiPhTLeVku5uuD3WexYQvdf+E1p+4S8JgPH56QzCFiA3yOyK34z4bsi/Z34PRG7AjvgN6cd1ATtOqjm2vCb6c4Ats3EeTaUaWf8TvaDQv1eA4yrJf3dgScqKEcX/J6cO+XoNwaOBvYn2iA60o/Bb9TbCb/n55vAybXk+SywQ13bKhN3ZeBD4FT8XpQnh/DKde0jVWzjknUbteu6+P0uR+D3Gt0bv0fnGmXSHAC8m6PbLPSTEfgNbHoBA2op4w7A0/jNkTesTx+J4iV2KHN8T/yesIPw+8GOBS7JSWP1MAZ/EPrUZcALkf5ivP3oAWwaxs3wOrVLU3aCUOixlDHi+N3P3wjyMPwu1+3qkP69wIE5usXAZlH4b8BZFaZ7GvBg5tgh+J1DRlG7Ed8P+HuO7nnguCh8dNzQmbgXAXdE4QFh4KwaOueXwMBIf2teB6tiG2+M/4E8ODrWIwzIdsDlwP9GunXCoMsdoPgf8tMqyPsn+A2srZZ4G1LaiE8B9orClwHXlUknua66tFWJuMPwG2xbdGxazYCuSx+pYruuULeZdv028FnmnJnAd8ukeTJwZY7uDuD8OpSvPX5D7i0pb8Rr7SNEdiinXBdF4d2AGTlxjwOej8JdgCXAJiH8MX7D7Rr9+eT80Of9NZdP/OJwO/ucmQ3N6PbC73AO8B387OPmcAv5spntnJeomXXAz3oez4nyB+AIM+tgZhsD3wWeqLDMO+H3ZazJqxvwW+D0Cs+PryvLIMKu9oEJ4VitcZ1z7xEMd/hb5px7p8K0GhUzu9rMFuN3nP8EP6utYU/gSefcMvwsLXYF1Mibl0m+XP3F/AS4xYURUQ/+CBxiZp3NbF38LPCRMvHj68pSrq1KxX0jU+43WN52dekj1aJU3cbX/wowxcz2M7N2wZWyFH8deZRr1+8AmNmbwVV3Wzl3FP4u5mnnXLn88q6jLuUq1RZrmlmv2uI65xYB7wGDzKwHfgLToHZtDiP+a2AD/K3Xn4EHzWxApN+b5YO/D36G8hT+1vb3wANmtnpO2jsBE5xzC3L0DwEH4X8J38JvoPpybQU2s6OAIfgZZA3nh/On13Z+YARpoxbTFX/LVcPnQNccn2c2bk38VWvRVR3n3Ikhrx3xd0RLI3XcrmOAg81sSzNbBfhv/Mypc6l0zWwDoINzboXnJ5l4ffGumpsbcBnj8INoPvAR3jDdXyZ+fF1Z6tIetcWtSx9pdMrUbXL9wZDfgp+pLg3//ysYrlJpdga2w9d5KfoAhwMjgY2AVYC8Zx3r4V1Ouc+5armOLHVp1xq5ru3aNQpndRXT5EbcOfeic26Bc26pc+5m4Dn8rx5m1h3vT30+RF8CTHXO3eCc+8o5dxcwHdg+J/m9yKn48Av+CH723AlYD9jTzE4sV94wm7gEGOGcmxWObYX30f5PJddsZlsA88sY/IVAtyjcDViYM1PIxq2Jv6AWXZPgnFvmnHsWPwBPADCzlYA9CDNa59yTwLnA/+H9vlNDGUs+kKb8gIo5AnjWOfdBfcoeyvko/geoC96f2QP4XZn4yXWVoC7tUVvcuvSRarBC3Wav38x2By4FhuJ9/DsD18cPaDPshnc1fJGjXwLc5Jx7xzm3EO+e2isn7h+A3zrnsgaz1uvIUsIOZSnVFlD3dl2YOT/WVUxLWGLoWH47nb01fSPoK6XcLdAGeFfDLc65r51zHwF3kd8pMLPhwF+AfZ1zb0aqoUB/YJqZzQB+CYw0s1frUS7wbprBUXgwkeumXNwwS+0IvBP+2pvZRhWmVU3a433A4GdbU51zM2uUzrn/dc5t5JxbA2/M2wMTc9Kq1JVyBA2bhffE/7j/KUwyZgM3kd9HVriuDOXaqlTcLTMz6y1Z3nZ16SPVoFTdZq9/K7w74xXn3DfhLvdF/ISnFLW1a13G/27AZWGFx4xw7F+24uqsSvpIORcZlG6LT0N/KRvXzLrgx8Uk59xcvNuxYe1aFwd6Q/+A7qGCOuEH7WHAImDjoL8FOCKK3xOYi/dhtcO7QuYAq5dIe33g/TJ5dwPmAYfif7zWAv4FXJgTf1f8apIVnmDjb/vXiv4uB+4Beuek9XSpdCL98fgHauvifWSTKL86ZT7eZdEFuI306pS78CtUuuDvWKq+OgVYA/+Qt2topz1Du+4f9L8F/juK3wnv/zagL/5h90U5aa8S2qFTLWX4XshzhYeGmXgW8t8MbyA6Ea0Ywj/wOjP0z+7AfcDtOWmlrquubZWJW7M65RS8oT+J9OqUivtIFdq3ZN2WaNedgVnAViG8dWi7YTnpTgX6lsn3p8AH+AlYZ/wiglvL9MF4TDq8T32VevSRlB0qoR+OX0WyGf5O7Z/kr07pHcbgyNDXfkd6dcoleHdSD/zs/xNa8uqUcEEv428X5gEvAHtEg+sTMsuRwgB4E3/r8QqwY07aJ+FnUOXy3zXk/3lohL8AnYOub8ijbwg/BXwdjtX8PZyT7ihyVqfgl0bOBNqXKZfhb0PnhL9LSa8AWBhfN/6HaFrokA8APSNdT7wPd1GIc2gTteu40KbzQ3sdG+lfAYZE4e74Wdai0A4Xk7MCCdgHeKiCMlxXaoCXaNf++AEe/02N4m+F/1GZizdIf8v2ybzryolTrq0eBs6Owlvjl7IuAV4lWk5bWx+pcvvm1e0K1x/G4bv4Mf4+cHpOmpsDEyvI+7wwfmbiV1r1yBsXmfNWWJ2Sdx2ZOCXtUIl4p+GXGc7H363FE4FJwGFReHf8M7gloW/1j3QdgRtDOp9SwQqs7J+FhJodM/sW3gh/q57njwnnV+I7bTLCywkHOedqfUmhNWJmawKvA+u4enQ2M7saP9ivbvTCNYCGXlfRaYR2PQN/R31GoxeuATTUDjUHZd9EawbObcC5Y/Gz55bGPCp8ANpKWQ0/u6ivoXsdeLARy9NYNPS6ik5Dr38qLbNdoWF2qMlpMTNxIYQQdaclrE4RQghRT2TEhRCiwDSpT9xKfDVONB3Ouaq83ad2bV7Urq2TSttVM3EhhCgwMuJCCFFgZMSFEKLAyIgLIUSBkREXQogCIyMuhBAFRkZcCCEKjIy4EEIUGBlxIYQoMDLiQghRYGTEhRCiwMiICyFEgZERF0KIAtPSdvYRot707t07FT7//PMT+fvf/35K99Zbb5VMo3Pnzqnwdttt10ilE6I6aCYuhBAFRkZcCCEKTJPusamPzDcvrXHzgE033TSRx4wZk9L17ds3kbP93MxK6pYsWZKKd/HFFyfyRRdd1LDCVonW2K5Cm0IIIUSbQEZcCCEKjIy4EEIUGPnE2xCtwXeaXUZ47bXXJvIBBxyQ0k2fPj2Rn3766ZRu2rRpiXzWWWcl8quvvpqKV4Qlhi25XYcOHZobPvfcc3PPGzt2bCKPGzeuocUoS5xXqXBzIZ+4EEK0AWTEhRCiwMid0gT06tUrkbNvBMastdZaibzTTjuldLNnz07k22+/PaX76quvKipHS77trpQrrrgiFT7llFMSOduX4/qcNWtWSnfBBRck8plnnpnIJ554Yipe/KbnEUcckat75plnUropU6aUvoAq0JLbddSoUalwORdKkYmXrDYWcqcIIUQbQEZcCCEKjD6AVQd22GGHRB40aFBKF7s/Bg8enNL16dMnkbt169bgcqy99tqpcPxWYWukX79+iXzYYYeldPFtbPaNyqwLJebAAw9M5Pvvvz+Rf/GLX6Tibbzxxon86aefpnSx++aEE05I6ZrSnSKah1122aW5iwBoJi6EEIVGRlwIIQqMjLgQQhSYNukT79SpUyq8/fbbJ3J284CRI0cmcteuXRN58uTJqXjxW1633HJLSjdhwoREnjFjRt0LDGy99daJnH2rsLX7xFdfffVEjpdrworLCisl9nUPHDgwkbNLxe69995Ejv3o2bzj5yXZ88r55ls75d5+LOJyw9gP3lLe7NRMXAghCoyMuBBCFJg288Zm//79E/nyyy9P6fbdd99EfvPNN1O6q666KpH/8Y9/JHJT3yJfeumlibzFFlukdCNGjKgojZb8Zl854o9cHXvssdm8Ezm7rC9+izLrJos/pDVz5sxEzn6wKU4zfssT0h/OyrphrrvuukTOLj9sbIrarjH1/VBWpWRdH7FbJPtWaUw5XbXRG5tCCNEGkBEXQogCIyMuhBAFplX5xFddddVEPuOMM1K60047LZGzXwGMfeTvvPNOlUpXN4488shU+Oc//3kiDx8+PKWLfbrlKKrvdM8990zk+LlEyDuRK90MGeDZZ59N5LhvZJdvxmS/QPnSSy8lcrxhM6T98Vl/b2NT1HatlGz9PfXUUw1Os9KvDmZ94jvvvHPJeNV4BV8+cSGEaAPIiAshRIEpvDsl/irg3//+90QeMGBAKl78sf8HH3ywsYvRKOy4446JHLtPIP31vko3gchS1NvueDngmDFjUrptt902kbN9OXaNZJcfxm+51veLgy+//HLJcmTL0q5du3qlXylFbdf6ErtXsq6WSpcjlnvzMk6zUtfNeeedlwo3xtJEuVOEEKINICMuhBAFRkZcCCEKTOF84vHmtwD33XdfIs+bNy+RszvAzJkzp6FZV53Yv9+xY8eUrtJlhOVoDb7T+IuGAH379s2NW265YGNw9tlnJ/L555+f0sXjqn376n4stDW0a2NRqT0r9wXCSpeExmloiaEQQoh6ISMuhBAFpnDulCOOOCIV/vWvf53I8eYOsWulNlZbbbVEzm468P7779e1iC0W3XY3LvFbmpMmTUrp4nF1xx13JPLhhx/e6OVQuy4nXtpXjU0n4qWE1f7CodwpQgjRBpARF0KIAlM4d8pDDz2UCo8fPz6R63v7FL/pmf3AzdNPP53I8b6JsQywYMGCRP7mm2/qVY5qo9vu6rFs2bJUOB5Xr732WiJvt912jZ632rU01bBtTbnHptwpQgjRBpARF0KIAiMjLoQQBaZwPvH4y3EhzUQeMmRIQ5NnvfXWS4X33nvvRD7qqKMSOevbjP3q2S8QTp8+vcHlagzkO60e2ecg8biK2z/bRxtjw221a2niLxDWZWOOar+JWSnyiQshRBtARlwIIQpM4dwpm222WSr8+uuvJ/L111+fyNm3qT777LOGZp36kNGIESNSumOPPTaRs8sUR44cmchPPPFEg8tRX3TbXT20xLBlUB97ll0q2JwulBi5U4QQog0gIy6EEAVGRlwIIQpMdb9WXwUmT56cCp9yyimJfOGFFybyD3/4w1S8e+65J5FvvfXWlO7ZZ5+tKO+vv/46kbObLcfhyy67LKW75ZZbEnmbbbZJ6WbMmFFR3qJl85e//CUVjp+RdOnSJZE7d+6cird48eLqFqyVU5elg3mMGzeu4QVpRjQTF0KIAiMjLoQQBaZw7pQs11xzTUn59NNPT8XbddddE/nhhx9O6Tp16pTITz75ZL3Kseaaayby4MGDU7rrrrsukeU+aRjxRgwABx54YCIfcMABKV01lvPlkS1XvNStKZfxtjXitzLbKpqJCyFEgZERF0KIAiMjLoQQBabwPvE8fv/73+eGe/fundL169cvkbP+7Jjvfe97ifz888/nxnvxxRdT4eyySFF/rrjiilR4+PDhiZz9kuCPf/zjRJ4yZUpKF+8IFRP3BYDVV189kY877riULvbBZ/tU7AefNm1aImtJYcOp9gbFRUMzcSGEKDAy4kIIUWBarTulHDNnzswNv/LKK7nn3XDDDVUrk6iM++67LxUeNmxYImeX8t188825uvjLgjF9+/ZNhXv16pXI8QYk2TSz6cfum8MPP7xkXkI0BpqJCyFEgZERF0KIAtMm3SmiuPz5z39OheNVIWeeeWZKF394Ksu2226byLErpJzLJKuL98c84YQTUrp77703N2/RMLKbrrR1NBMXQogCIyMuhBAFRkZcCCEKTOE2Shb1p7VvqLvJJpukwvHbnFl/eexLj8fA7NmzU/Fi33bsA4f0RhDxW5lNTWtv1yyNbbOyGyNnN05uLrRRshBCtAFkxIUQosDIndKGaGu33W2FttaujWGzzjvvvERuqR/UkjtFCCHaADLiQghRYGTEhRCiwOi1eyFEoYj92dlX8IcOHVrynJa6jLAx0ExcCCEKjIy4EEIUGC0xbEO0taVobQW1a+tESwyFEKINICMuhBAFRkZcCCEKjIy4EEIUGBlxIYQoMDLiQghRYJp0iaEQQojGRTNxIYQoMDLiQghRYGTEhRCiwMiICyFEgWlyI25mCzN/y8zsqkycs83soiDvZmZvmdliM3vKzPrVkv46ZvZRGf0pZvaBmS0ysylmNjAnnpnZ78xsdvi71Mws0u9rZhPDNTxvZptVcO3vlMrPzDqa2Y1mNt/MZpjZabWkc2qI93k4r2Ok6x/qaXGot91rK1djYGYnmdkrZrbUzEbnxInbtbOZXW1ms8J1PF1L+iuHuF1z9IeE9lxkZu+Z2Y5l0ipXf98zs5fMbIGZvWFmO1Rw7Y+Z2bASx8v2oRLxDzWzD8M13G9mPSNdTzO7L+g+NLNDaytXQwn98oaQ3wIze83MRpSIl7RrdOxcM3O19b9y49XMpprZkshWPFYmnXXN7AEzm2NmH5nZ8Rm9C3VXk9b1tZQrt7/VpS0qsCNbmdn4MF7evKvZAAAgAElEQVTHm9lW5cpVEudcs/0BXYCFwE6Z488COwCrA58DPwA6AZcBL9SS5jHA9WV0bwCbAQYMAHrmxP0v4G2gD7AuMBk4Pug2AuaHMrYHzgLeBdqXKdcA4N0c3cXAM0APYFNgBjA8J+6ewKfAoBB/LHBJpP8XcAWwCjASmAf0boK2PBA4ALgGGJ0T51lghyDfBtwF9AbaAdvWkv7uwBM5uj2AD4Hv4Ccm6wLr1rX+gJ7ArNDf2gE/BuYCPWrpw7OBjnXpQyXiDgIWADsBXYE7gLsi/Z3A3UG3QxgXg5pgfI4C+od63SeUsX9eu0Z9/U3gP8DuteRRbrxOre38KO5TwB+ADsBgYA6wS6R3wIZ1uPZy/a3itijXB4CVQ789FegInBzCK9epnarZCSqoqJ8A7xOWOoZjPYDPwiA6Dng+06mWAJuUSfNe4MASx1cCpgO7VVi254HjovDRhB8Q4CTgH5m0l5RLOzTQlTm6j4FhUfj8eABn4t4BXBSFdwNmBHkgsBRYNdI/Q47hqFKbXkAJI55p143xP4Ld6pDuFcBpZdrq6ArTKVd/+wCTMvHfKZc2sB/w97r2oRJxLwLuiMIDgC+BVUO//xIYGOlvJfrxbsL2fQMYWapdo2MPA3tRgRHOG69BV+v5IV5XvJHuHR37M3BrFK6rES/Z3+raFrXYkWFh7Mf2bxo5E7i8v+b2if8EuMWF0gf2BJ50zi3Dz04m1Cicc4uA98LxFTCzDviZzOMl1H3C3+ZmNt28S+U8M8urg1TeQa7J18IfmfDmOWmB79T/KFHmHsA6ZfKqpFxrmlmvoHvfObegwrSakrhdv42fcZwXblnfNLORtZyfV3/tgCFAbzN7N9xK/8nMVslJp1z9ZdsV6tmuZfKqqF2dc+8RjEX4W+ace6fCtKqCma0ZyjIpOhy3K2b2A+BL59yYCtIrN15ruN3MZgaX1eC8pDL/a+Rsuz0d3Gj3mln/WoqX1651bYtyfWAQ8EbG/r1RJq2SNJsRN7O+wM7AzRnV3kBNB+iKv1WJ+Rw/OynFTsCEjBGroU/4PwzYAtgF+BH+l7EU2bw/B7oGf9bjwM5mNtTMVgbOxt8adS6VkJl1BrYDxuXkU5N+nFfeNZYqFyF+XeurKYnbtQ9+gH2O/wE7CbjZzDYtdaKZbQB0cM69XUK9Jv4W+iBgR2ArYGvg/+WUo1z9PQ+sY2Y/MrMOZvYT/Iy4ZLsGRkTXVUleNX2otrg18VtEuwaDeztws3PurUiVtGvwH18E/KLCZMuNV4DD8K6cfnh3yaNm1j0bKZz/HHCOmXUys23wrsS43XYOaW2Cd/M8ZGYlt6espb/VtS3K9YFGadfmnIkfATzrnPug5kCYFe8BPBIOLQS6Zc7rhvfLlWIv8gfUkvD/UufcPOfcVOC6cE4psnl3AxY6z1v4u4g/AZ/gffeTgbwHqrvh3UJf5ORTk36cV941lioXIX5d66tJKNGuS4CvgAucc18658bhB+kKDwcD8Q9Alpp2vco594lzbhb+Vrgu7QqwwDk3G9gfOA3vNx8OPEFOu5rZFsB859z0OuS1MDPzyotbE7/Z2zW03634O4OTMsfjdj0P78L4YIVESlNuvOKce845t8Q5t9g5dzH++U7eA+vDgPXxLtNr8D84Sbs5554OfW0ecEqIW3LSQPn+Vte2KNcHGqVdm9uIZ2fh2wFTnXMzQ3gS/iEFAGbWBT8zmkRpyt3avo3vhJV+ZyCVd5CTfJ1z9zjnNnfO9QLOxc8WXq5ruZxzc/E/BLl5VVCuT4MBmgRsYGarZvR5aTUV2XZ9o47n11Z/H9Gwdq2pP5xz45xz2znnegKH4/33L9W1XGXyqqhdw2ywI94n/w7Q3sw2qjCtRiPMGG/A3/GMdM59Famz7bobcHJwWcwA1gP+ama/zkm+tvrL4ljR3eUVzn3onNvHOdfbOfdtoBf57VY2rVrKVde2KNcHJgFbZu7MtiyTVmnq4kBvrD/ge8Aiogdw4fhvgf+Owr3xtxcj8atTfkf+g6H18f7gcvneAjyEv13pA7xFzkMr4HhgCv6J8jqhYo+P9NviH9L1xj+pvqNMvlOBvmX0l+BdLT3wt3ufkL86ZTh+9cpmIf4/Sa9OeQG4PNTX92m61SntQ54X42dtnQirdUq0awf8ap5zwnnb42cfKzywxq+ymQ10KpP3b/E/oGuEOnkGOL+e9bd1KF83/GqH58rk+zSZlVV16UOZuIPwD3t3xD88u4306pS78KsiuoT6qvrqlJDvtaFPdc2p97hdewFrRX/T8St9Sp1bdrwCfcN1rhz60q+AmUCvnPib4sf1yvhVRbNq+n2o263w47VraNe38S6T+vS3ituiXB9g+eqUU/A/2CdRlNUpeDfGrSWOvwIMyRzbHW9sl+CXg/XPSfMk4E+15NstNMCC0MH+m+UfAdsRf5tTE9eAS/FLleYEOX6K/GxIZ064ni45eW4OTKylXB2BG8Mg/pToqXjozAuJfgRYfrs/H7iJaHkb3u83NtTX21S4RKsR2nQUfnYT/40q066D8MshF+FdUd/PSXcf4KFa8u4AXI3/wZoBXFkzCOtRf3eGQfk5/sd5jZw8V8MblXLLSmvrQwuBHaPwofjVCYuAB4iWv+KXP94fdNOAQ5ugTfuFdvwilLXm77C8ds2cPzWv/1HLeA39441wvbOBJ+O88O6TSVH4F6E9FuHHZhx31zAWFuFX0twPbNSA/pbbFtTdjmwNjMeP11eBrevaTi3mK4bhyffrwDquHoUyszH4TlHrU/GmxMzOAFZ3zp3R3GVpDhqhXa/G/whe3eiFawBmdjBwkHPu4OYuS3PQisdri+xv5Sj5dLaZWA0/A63vr8pY/MOxlsZU4MHmLkQz0tB2fZ2WWX/zgP9p7kI0I611vLbU/pZLi5mJCyGEqDvN/bKPEEKIBiAjLoQQBaZJfeJmJt9NM+Kcy/2CXkNQuzYvatfWSaXtqpm4EEIUGBlxIYQoMDLiQghRYGTEhRCiwMiICyFEgZERF0KIAtOSXrsXQoiq0Llzel+Pu+66K5Hff//9lO4Xv6h0T4uWgWbiQghRYGTEhRCiwDTpB7D0Bljzojf7Widq19oZOHBgKvzWW8u3CV2yZElK16dPn0SeO3dudQtWBr2xKYQQbQAZcSGEKDAy4kIIUWC0xFC0Cfr165cKf/DBBxWd97Of/SyRr7nmmkYtk2gZfPbZZ6nwl19+2UwlqR+aiQshRIGRERdCiAIjd4poE2SX0la6tFZ70LZ+Hn744VR40aJFzVSS+qGZuBBCFBgZcSGEKDAy4kIIUWDajE+8S5cuiRy/VgtwzDHHJPIJJ5yQ0r3yyiuJ3Lt370TedNNNU/EmT56cyFkf22WXXZbI2eVMQojqkx3X8TLCP/zhD01dnEZFM3EhhCgwMuJCCFFgWu1XDDt06JAK33DDDYm89tprp3S77rprbjpmyz8kVt+6Wrp0aSLvscceKd1zzz1XrzTrQ1v+2l3fvn1T4fq8sXnttdc2apkai7bcruWI2/z1119P6eKx3KtXryYrU13QVwyFEKINICMuhBAFptWuTsneIu2www6J3LVr15Ru4sSJiTx79uyU7oknnkjk1VdfPZE322yzVLzYRbP55pundJ06dUrkX/3qVyldU7pT2jIXXnhhcxdBNDG77757Infv3j2lO+uss5q6OFVDM3EhhCgwMuJCCFFgZMSFEKLAtFqfeNa3/de//jWRs/7sjz/+OJFHjRqV0n366acV5dejR49Evv/++1O6HXfcMZGzSwy/+93vJvK//vWvivISdSfrE81j6tSpqXD27VvRslljjTUS+Ywzzkjk7DgePXp0UxWp6mgmLoQQBUZGXAghCkyrdaf85je/SYUHDBiQyFOmTEnpxo8fn8j1/SD83LlzE/mTTz7JjRcvN4T0ksMDDzywXnmL0myyySYl5XIsWLAgFf7www8btUyiuowYMSKRBw4cmMj33HNPKl7sXllllVVSuvbtl5vFbH9oiWgmLoQQBUZGXAghCoyMuBBCFJjC+8S33XbbRD7kkEMSeeTIkal4++yzTyLHGzhUg6effjoVXmuttRJ5p512SunizSpE4xI/B9lggw0qOuf000+vVnFEFciOn8MPP7xkvEsvvTQVjv3ed911V0q35pprJvJee+2V0s2ZM6de5awmmokLIUSBkREXQogCU3h3Sry0L94IotLb52qQXc60/vrrJ3L89ibUf6MJUTuVukbmzZuXyC3xdlnkc+qpp6bC8QYvTz31VCLHe+UCDBs2LJH33Xff3PTXW2+9VLgl9g/NxIUQosDIiAshRIGRERdCiAJTeJ947PvedNNNm7EkyzniiCNS4dhHftpppzV1cdoslT4XGTNmTCK/9tpr1SqOaCTinbOOO+643Hg33nhjIse7cgFcddVVuefFn82YMWNGfYrYpGgmLoQQBUZGXAghCkzh3SlLlixJ5JZ663PCCSfk6iZNmtSEJWlbrLTS8jmKmeXGK6cTzUO8XHj48OEp3TXXXJPI66yzTm4a9913XyLvueeeKd1GG22Ue96yZcsSOX6zE6Bjx46JvHTp0tw0mhLNxIUQosDIiAshRIEpvDvlueeeKym3JMptNPHII480YUnaFt98800il3szVm/NNj+rrbZaKhy7QoYOHVqvNBcuXFiv8/r06ZPI06dPT+ni8DHHHJPIjz/+eL3yagw0ExdCiAIjIy6EEAVGRlwIIQpM4X3i8duR8ZtcZ5xxRpOWI34jbNVVV03p4o2Y33///ZTu3//+d3ULJmolXs6WXVL29ddfN3Vx2gyxH/zyyy9P6cr5weNnTNnz5s+fn8g/+tGPEnnIkCH1LWaKr776KpG32WabRJZPXAghRL2QERdCiAJjTbm8yswaPbOTTjopkZ9//vlEfvXVVxs7q7K8/vrribzFFlukdAsWLEjk+BYMVnSvVBPnXFVeTaxGuzYGU6dOTeTsx/3zyO6B2lKXrcYUqV1j19XVV1+dyEcffXTFafzyl79M5CuuuCKl69SpUyJ//PHHidyjR49UvNjuvfHGGyndk08+mcgPPfRQShfbldh1Uw0qbVfNxIUQosDIiAshRIGRERdCiAJT+CWGn376aSK3a9euyfLddtttU+F4M+QsF1xwQSI3pQ+8rTNx4sRErtQnHrcVwC677NKoZWrrxF8PrNQPftttt6XCf/zjH3Pj/vCHP0zknj17JnL22V/8uYu99967onK0VDQTF0KIAiMjLoQQBaZw7pT+/funwvHbWy+//HJV8x40aFAiZz9UP23atEQeN25cSpd9q0w0DT/96U8T+eabb07phg0bVvKcrbfeOhXea6+9EjneizNLvAEFpL+EF785eOWVV6bixZuatAUqfZP6gw8+SORzzjknpYs3bcjSu3fvRI5dKLfeemsq3lFHHVVROYqAZuJCCFFgZMSFEKLAyIgLIUSBKZxPPH6VulS4sYmXH91xxx2JnP263RNPPJHIZ555ZlXLJCrjs88+S+RDDjkkpYt3jtl5550TOfsFyp///OeJvOOOO6Z0DzzwQCIffPDBKd0pp5ySyG+99VYi33PPPal47733Xv4FtAJ69eqVCuct2fzyyy9T4UMPPTSRP/zww4rzW3fddRM53sj47rvvTsWLd30qOpqJCyFEgZERF0KIAlP4rxhWm3jZ4uDBgxN59uzZqXixCyW7nK2lUKSv3VWb+Gt3Dz74YCLvuuuuuedklxEuXry4ZHpZzjvvvET+7W9/W6dyVkJLbte11lorFZ4wYUIix8sB99hjj1S8+EuCdWHttdcumfdrr71Wr/SaE33FUAgh2gAy4kIIUWAK507ZYIMNUuFPPvkkkRvj7bf4rUyAN998M5HjW8Hsm31FoCXfdjcn8W19dvXE2WefnchbbrllSldu7MQfbTruuOMSOV4x0VioXVsncqcIIUQbQEZcCCEKjIy4EEIUmML5xC+66KJU+C9/+Usix9eSfZMz/vph9i2y+E28iy++OKWL/ezxF+1eeOGFygvdQpDvtGGccMIJqXD8fKZ79+4p3fXXX5/IL774YlXLpXZtncgnLoQQbQAZcSGEKDCFc6dk2XfffRP52muvTeRZs2al4sXXmX3zLl5WOHny5JQu3tChpb6JWSm67W6dqF1bJ3KnCCFEG0BGXAghCoyMuBBCFJjCbQqRJV7aFX+1LP6aGaQ3CMj6vePXqbMfoF+4cGGjlFMIIaqBZuJCCFFgZMSFEKLAFH6JoagcLUVrnahdWydaYiiEEG0AGXEhhCgwMuJCCFFgZMSFEKLAyIgLIUSBkREXQogC06RLDIUQQjQumokLIUSBkREXQogCIyMuhBAFRkZcCCEKTJMbcTPrb2ZjzGyumc0wsz+ZWftMnEPN7I4gb2Vm481scfi/VS3pr2xms8ysawndWDP7wswWhr+3y6TT3cxuNrPPwt+ojP4pM5tpZvPNbIKZ7V/BtT9mZsNKHDcz+52ZzQ5/l5pZ7ncTQv18aGaLzOx+M+sZ6Xqa2X1B96GZHVpbuRoTM9so1PFtJXR/NrPjgtzbzO4ws3mhL9xeS7rrmNlHObqpZrYkatfHyqTT08zuDn1klpndbmbdIv33zOwlM1tgZm+Y2Q4VXHOrbdd6jNc/m9nbZvaNmR1ZYR7vmNnAEsdHm9mXUbsuNLN2OWlcm4m31MwWRPqKx350TjHa1TnXpH/AGGA00AlYC3gTODkT5zbgx8DKwIfAqUBH4OQQXrlM+rsDT+ToxgLHVFjOm4C/AZ2B/sB7wFGRfkugfZC/DSwA1i6TXhdgNtCxhO6/gLeBPsC6wGTg+Jx0BoW8dgK6AncAd0X6O4G7g24H4HNgUBO272PAM8BtJXTTgD5Bfga4AlgN6ABsXUu6xwDX5+imArtXWL6rQxm7hbyfAK4Iup7ALOAHQLvQB+cCPdpqu9ZlvAb5Z8BuwCvAkRWkPwB4N0c3GrignuUeDdwYhcdS4dgvWrtWtQPkXNQUYK8ofBlwXRReCfgUWB0YBnxMWAoZ9NOA4WXSvwI4LUdXcUOGwbxdFD4beCYn7reAL4BvlUlvP+DvObrngeOi8NHACzlxLwLuiMIDgC+BVUPH+xIYGOlvBS5porY9BPgrMIqMEcf/6L0R5GF4w9uuDmnfCxyYo5tK5Ub8YeDEKPwz4NEg7wNMysR/Bzi6rbZrXcZr5rxnqcyInwxcmaMbTT2MeKivBcDO0bGKx37R2rU5fOJ/BA4xs85mti4wAngk0n8LeN85Nwv/K/aGC1cXeCMcz2Mv4B9l9BeH2+jnzGxoLWW1jLx5Smn2kJl9AbyI7ySv1LNcg4AJUXgC+deYiuuce4/QEcLfMufcOxWm1WgEl8RvgdNzosTX/x38TObmcDv6spntXCbtDviZzONlinC7effWY2Y2uEy8/wX2MbMeZtYDGIk37ODbOHtbvEK7Z2jV7Urdxmt9qG28nmhmc8y7UkdWmOZIYCbwdOZ4XcZ+Ydq1OYz4OHwh5wMf4Q3f/ZF+b/wtHPhbjM8z53+O/xVbATPbAOjgnMvzd/0a2AB/C/Rn4EEzG5AT9xHgTDNb1cw2BH6Kd60kOOf2CWXZCz+b+yYnLfCdf0yOLnudnwNdc/xs5eqkTvXVyJwP3OCcm56jj9u1D342/hT+Fv33wANmtnrOuTsBE5xzC3L0h+FdXv1Cmo+aWfecuK/i3XSzw98yvIsF/AxrHTP7kZl1MLOf4GdOnUum5Gnt7VqX8VonzKwzsF3IoxRXAhsBawDnAKPNbPsKkv4JcEtm8leXsQ8FatcmNeJmthLwKP7WuAveZdID+F0UbS+WV95CvO8yphv+VqkUZTuUc+5F59wC59xS59zNwHMhv1KcDCwB/g08gPddrfBgzTn3lXPuYWBPM9uvVEJmtgUwv4yBy15nN2BhphPmxa2Jv6AWXdUw/7B5d+B/cvTdgU3wRhJ8vU51zt0Q6u8uYDqQN0DjPrECzrnnnHNLnHOLnXMXA/OAHXOi/w3vIlkVXzfv4X26OOdmA/sDp+FdBMPxPvO8B6qtvV3rOl7rym7A8865L0opnXOvOudmO+e+ds6NAW4HDqylzOsBOwO3ZNKqeOwXrV2beibeE1gP+FOozNn4B4h7AZjZWsDa+NkSwCRgy8wv3JbheClquzXL4ljx9tkrnJvjnDvMObeWc24Qvq5eKpNWe/ysrT7lmgTELoDB5F9jKm64++iIN0zvAO3NbKMK02oshuJnwtPMbAbwS2CkmdW0457Ak865ZSH8Br7uK6XR2hVfH9c55xY55xYC1xINZufcOOfcds65nsDhwMbkt3trb9e6jte60pjtWsMR+B+G9xuQVrHata4PDRr6B7wPnIk3et2B+4Dbg+6npJ8o16xOOSVc+EnkrE4BVsHfHnfKybc73ph0CnkfBiwCNs6JPwDohV+lMAL/oHNQ0G0Sjq2CX1nxY7yfa5uctJ4GdipTJ8fjHyCtC6wTGrHc0+75+JlmF/wsMn7afRf+rqELfmZb9VUMeHfDWtHf5cA9QO+gvwU4IorfE7/q4yehfg8C5pB5OBbiro/3uebl3Tdc58qhbX+F94f2yon/FHBVaLtV8K6U5yL91qFNuwF/iHVtrV3rOl6jMdsJP9M9Nsgr5aQ9FehbJu+D8C6HlfDutwXA0FrK+zbw08yxuo79QrVrVTtAzkVthX8IOBdvGP8GrBF09wAHZeJvDYzH34K/Ss5SNPzKgofK5NsbeDl0hHnAC8AekX5H/C1RTfhg4D/AYuB1YM9Ityn+YWZNWi8D38/JdzW8UWlfpmwGXIo3ZHOCHK/IWQjsGIUPxa/SWYR39fSMdD3xPstFIc6hzdDGowirU8K1fVLTxpn6fjNc2yvx9WXinYSfCeblNQg/s1+E/xF/EhgS6Q8jWnGC/1F4MMSdg3/2sVGkvzMMpM/xS7/WyMm3TbQrdR+vY/Gz3PhvaIl0Nwcm1pL3M6Ed5uMf+B0S6fqG+usbHftuqJ9VM+mUHftFb9cW8xXD8ALBDGCAcy7r7K/k/KvxneLqWiM3IWZ2ML6jH9zcZWkOzOxbeCP8rXqePyacX1+/a1VQuzZ4vJ6Bv/M6o9EL1wCK2K7ta4/SZPQEzqlPhwi8jp9htTTmkfPArw1xbgPOHYt3gbQ02nq7NnS8TkXjtVFoMTNxIYQQdUcfwBJCiAIjIy6EEAWmSX3iZibfTTPinKttjW29ULs2L2rX1kml7aqZuBBCFBgZcSGEKDAy4kIIUWBkxIUQosDIiAshRIGRERdCiAIjIy6EEAVGRlwIIQqMjLgQQhQYGXEhhCgwMuJCCFFgZMSFEKLAyIgLIUSBkREXQogCIyMuhBAFRkZcCCEKTEvaKLlF0L59ukp22WWXRN5///1zz4t1H3/8cUr3yiuvJPLVV1+d0k2ePLle5RRCCNBMXAghCo2MuBBCFBgZcSGEKDDmXNPthdpSN15db731EnnYsGEp3XXXXVfyHLP0HqaV1uN//vOfVPjKK69M5Msvv7yiNOqLNtRtnahdl/Pwww8n8pAhQxJ5+PDhqXjjx49vsjLVF22ULIQQbQAZcSGEKDBtconh3nvvnQofccQRiTxy5Mh6pbl06dJEPvzww1O69957L5EfffTRlO6iiy5K5IEDBybycccdV69yCNGW+PGPf5wKDx06NJE7duyYyAcddFAqXhHcKZWimbgQQhQYGXEhhCgwbWZ1yrrrrpvITz31VEq3wQYb1Dm9d999NxU+++yzE/nee+/NPS+7+uWGG25I5GXLliXyiBEjUvGmTJlS5zJmaQ2rGOLbZYBzzz03Vxe/bTt27Ngqlqp5aQ3tWl/++c9/psJxH/jss88S+bvf/W4q3gcffNDgvGO36RprrJEbb8KECanwE088UVH6Wp0ihBBtABlxIYQoMDLiQghRYNrMEsM777wzkQcMGJDSlXsu8NhjjyXyhRdemMhZH/WcOXMqKkecHsADDzyQyMcff3win3feeal4Bx98cEXpt0ZiP2f2eYYQecRfE20MH3iW0047LZEHDx6cGy/71nelPvFK0UxcCCEKjIy4EEIUmFblTok3dHjwwQdTuu233z6Rsx+vivniiy9S4ffff7+kXKn7pDauueaaRD7hhBMSeaeddmqU9IvKqFGjEjleRliO7DLC1ryssC3TrVu3RO7du3czlqRloJm4EEIUGBlxIYQoMDLiQghRYFqVT3y77bZL5D322COlK7eMMNbFXz6D9LK/Tp06JfIxxxxT73LGzJ8/P5Hj14TbGrEPHCr3g8dLMbNpNCXZV/6z4Txiv718+JURLxEeNGhQbrzRo0dXtRx77rlnInfo0CE33qJFi6paDs3EhRCiwMiICyFEgWlVXzF86KGHEjm7p16mHKlwfeogXs7YWMRfZNt0001TurXXXrvB6bfkr91V2gbxlwmhaV0QWRdJY789mr2W7LXm0ZLbtRpce+21iVxu85Sjjz46kW+66aaqlqka6CuGQgjRBpARF0KIAiMjLoQQBaZVLTEcMmRIk+UV7xQE6S+mierR1MvwmvKZUaXLEkU+8e5Y1fhyYUtEM3EhhCgwMuJCCFFgWpU7ZeLEiYlc7VvT/fffPxW++uqr65XOhhtumMixO6jab3m1NLJukrz2yy7rq3QZXqU051ufojTZJcHZt6pjli5dmsht5Q1YzcSFEKLAyIgLIUSBaVVvbO66666J/Pjjj5crRyr86quvJvKNN96Ye95vfvObRF5nnXVSurvvvjuRDznkkNoLW4LYVTBz5syUrjH22GzJb/Y1xtuQ2X1J68POO++cClfbLRff8tfXNdSS27UxWHnllVPh7MYtMffff38iH3jggVUrU1OgNzaFEKINICMuhBAFRkZcCCEKTKtaYvjss88m8iWXXJLSnXjiiYkcb7QK8O677yZyuaWC8UbGa621Vkr3wKoKmv0AAAOlSURBVAMP1K2wgf79+yfylltumch//OMf65VeUSn3Bb94g4hyPupKN5JoalrKxhVtgbfffru5i9DkaCYuhBAFRkZcCCEKTKtyp3z55ZeJHC8HBNhvv/0SebPNNkvplixZUlH611xzTSK3a9cupbvnnnsqLmfMz372s0SO9/C84YYb6pVeayFv78m67GUZLxes71LBSpctyk0imgvNxIUQosDIiAshRIGRERdCiALTqnzi9WX06NEVxavvlwrLEX8N8cEHH0xkbTJRmuxSxLbypToh8tBMXAghCoyMuBBCFBi5U4DNN988kceNG1fVvC644IJUeMCAAYl8yy23VDVvIUTrQzNxIYQoMDLiQghRYGTEhRCiwLQZn/jcuXMTObuzzz777JPI1113XUr39ddfNzjvb3/724l8/PHHp3RTpkxJ5JtuuqnBeQkh2haaiQshRIGRERdCiALTZtwpP/rRjxI5uwnvHnvskciXXnppSnfFFVck8kcffVRRXkOGDEmF4w0junfvntIddthhiay3NIUQdUUzcSGEKDAy4kIIUWDajDsldlVk98M89dRTE/nkk09O6Y466qhEHj9+fCLH+3lCetOJDTfcMKXr0KFDIl911VUp3aOPPlpr2YVoy3z11Vep8N/+9rdE/sEPftDUxWlxaCYuhBAFRkZcCCEKjIy4EEIUmDbjE485//zzU+HZs2cn8rHHHpvS9evXL5HjzXZ32WWXVDznXCJnfXjxZsg33nhj3QssRBsmHluQfvtaaCYuhBCFRkZcCCEKTJt0p8yfPz8VvuSSSxL5hhtuSOkGDx6cyPF+mPFGEgATJ05M5Ouvvz6lmzBhQv0LK4RIMX369OYuQotCM3EhhCgwMuJCCFFgZMSFEKLAWHb5TlUzM2u6zMQKOOes9lh1R+3avLS1dl177bUT+aWXXkrpbrvttkQ+66yzmqxM1aDSdtVMXAghCoyMuBBCFBi5U9oQbe22u62gdm2dyJ0ihBBtABlxIYQoMDLiQghRYGTEhRCiwMiICyFEgZERF0KIAtOkSwyFEEI0LpqJCyFEgZERF0KIAiMjLoQQBUZGXAghCoyMuBBCFBgZcSGEKDAy4kIIUWBkxIUQosDIiAshRIGRERdCiAIjIy6EEAVGRlwIIQqMjLgQQhQYGXEhhCgwMuJCCFFgZMSFEKLAyIgLIUSBkREXQogCIyMuhBAFRkZcCCEKjIy4EEIUGBlxIYQoMDLiQghRYP4/9T083bs9LyYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x432 with 9 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "interp = ClassificationInterpretation.from_learner(learn)\n",
    "interp.plot_top_losses(9, figsize=(6,6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMgAAADSCAYAAAAPFY9jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAABqVJREFUeJzt3Utolekdx/HnNaNQvCDNopCdSiui2HoBpSBBlxqErJQiLlTwspQsigQR3OhGcDEoCCoUaRdqQbCYjRfUiLiQUaoyKFhQkCpqjFUoNm8XTqGTPOc/50xybjmfz/LH4eRh4Dvv5JmTpCjLMgF5M5p9AGhlAoGAQCAgEAgIBAICgYBAICCQNlEUxS+LovhrURT/KoriH0VR/KHZZ+oE3zT7AFTt25TSv1NKv0op/S6ldLkoiu/Ksvx7c481vRX+T3rrK4pidkrpXUppWVmW3/+w/Sml9LIsyz829XDTnP/Eag+/SSn9539x/OC7lNLSJp2nYwikPcxJKY2M20ZSSnObcJaOIpD28DGlNG/cNi+lNNqEs3QUgbSH71NK3xRF8ev/236bUvINep35Jr1NFEXxl5RSmVLalb7eYv0tpfR7t1j15QnSPvallH6RUvpnSunPKaW94qg/TxAIeIJAQCAQEAgEBAIBgUCgoZ/mLYrClRktoyzL4qde4wkCAYFAQCAQEAgEBAIBgUBAIBAQCAQEAgGBQEAgEBAIBAQCAYFAQCAQEAgEBAIBgUBAIBAQCAQEAgF/o7CB1q1bl92Hh4ez++LFi7N7X19fdt+0aVN2v3z5chWn++rOnTvZ/ebNm1W/x3TiCQIBgUBAIBAQCAQEAoGG/oWp6fi7eefNG//HZ786d+7chG3Dhg3Z137+/Dm7z5o1K7vPmTOnytPVrtJZPn36lN337t2b3c+fPz9lZ6oXv5sXJkkgEBAIBAQCAYFAwC3WJJ04cSK77969e9Lv/fjx4+z++vXr7P7hw4ea3n/GjIn/fty4cWNN7zE6OprdK33u7MGDBzW9fz25xYJJEggEBAIBgUBAIBBwi1WlpUuXZvfr169n9+7u7gnbixcvsq/dvn17dn/69Gl2f//+fXb/+PFjdq8kd4t18ODB7GsHBweze1dXV3a/ePFidt+1a1d2f/fuXXavJ7dYMEkCgYBAICAQCAgEAn4vVpXmzp2b3XO3VSmllLsdPHr0aPa1lW7C6m1sbGzCdujQoexrK/1048DAQHbv7+/P7qdPn87utfzurkbyBIGAQCAgEAgIBAICgYDPYlWpt7c3u1+7di27nz17dsK2Y8eOqTxSS3j27Fl2X7BgQXY/c+ZMdt+5c+eUnalaPosFkyQQCAgEAgKBgEAg4LNYVTp8+HBNr797926dTtJahoaGsvuePXuy+9q1a+t5nCnnCQIBgUBAIBAQCAR8kz7OwoULs3tPT092HxkZye4PHz6csjO1sqtXr2b3St+ktxtPEAgIBAICgYBAICAQCLjFGmfbtm3ZvdLt1oULF7L78PDwlJ2J5vEEgYBAICAQCAgEAgKBgFuscbZu3ZrdK33m6vjx4/U8Dk3mCQIBgUBAIBAQCAQEAgG3WFV68uRJdr9161aDT0IjeYJAQCAQEAgEBAIBgUCgY2+xZs+end1nzpzZ4JPQyjxBICAQCAgEAgKBgEAg0LG3WFu2bMnuixYtyu5v3ryp53Ha1ubNm2t6/ZcvX+p0kvrwBIGAQCAgEAgIBAICgUDH3mJRm1WrVmX3vr6+mt7nwIEDU3GchvEEgYBAICAQCAgEAr5J50cqfTO+f//+7D5//vzsfvv27ew+NDT08w7WJJ4gEBAIBAQCAYFAQCAQ6NhbrOfPn2f30dHRxh6kibq6uiZsAwMD2ddW+gGzly9fZvdK7+MHpmAaEQgEBAIBgUBAIBAoyrJs3BcrisZ9sZ/p0aNH2b3SP6fe3t7s3oxfE7R8+fLsvm/fvuy+cuXKCdvq1atr+prr16/P7jdu3KjpfZqhLMvip17jCQIBgUBAIBAQCAQEAoGO/SxWrZYsWZLdr1y5kt1fvXpVz+NkrVmzJrt3d3dX/R6Vbt8uXbqU3e/du1f1e7cjTxAICAQCAoGAQCAgEAj4LNY4/f392X1wcDC7r1ixop7HmRJjY2PZ/e3btxO2Y8eOZV975MiRKT1TK/BZLJgkgUBAIBAQCAQEAgG3WFXq6enJ7pU+i7Vs2bJ6Hifr1KlT2f3+/fvZ/eTJk/U8TstziwWTJBAICAQCAoGAQCDgFouO5RYLJkkgEBAIBAQCAYFAQCAQEAgEBAIBgUBAIBAQCAQEAgGBQEAgEBAIBAQCAYFAQCAQEAgEBAIBgUBAIBBo6K/9gXbjCQIBgUBAIBAQCAQEAgGBQEAgEBAIBAQCAYFAQCAQEAgEBAIBgUBAIBAQCAQEAgGBQEAgEBAIBAQCAYFAQCAQ+C/HAVDIfA2fYwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "img,lbl = learn.data.valid_ds[0]\n",
    "img.show(title=lbl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Category 0,\n",
       " tensor(0),\n",
       " tensor([9.9896e-01, 1.6453e-06, 8.0349e-04, 1.5958e-06, 4.6403e-07, 2.7865e-05,\n",
       "         6.0855e-05, 3.9501e-06, 1.4178e-05, 1.2925e-04]))"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can see above it is a zero, and below shows it predicts a zero\n",
    "learn.predict(img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new predict function doesn't seem to support batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(learn.data.valid_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7795, 4909, 7184, 9227, 5776, 2321, 1284, 2773, 1529, 3749]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inds = [random.randint(0, len(learn.data.valid_ds)) for _ in range(10)]\n",
    "inds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n",
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n",
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n",
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n",
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n",
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n",
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n",
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n",
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n",
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py:91: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  input = module(input)\n"
     ]
    }
   ],
   "source": [
    "preds = [(learn.predict(i), i, l) for (i,l) in learn.data.valid_ds[inds]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "imgs_p = [(img, pred_cat) for ((pred_cat,tn,cats),img,act_cat) in preds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAAFWCAYAAACMz5AvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XncjnX+9/HP11LZyRohZkbL3CNuY0iZDBW60mab5FZpmtEiTTUpSQtah+Y3tEqSoVIiZEnrkMqW7hlpRH6WkLUSisv1vf+49Lv7fI/LuVznchzn+X09H48e/d6HY/n8xum4Ph0+x/c01loBAAAAfFQm7AIAAACAsNAMAwAAwFs0wwAAAPAWzTAAAAC8RTMMAAAAb9EMAwAAwFs0w2lgjHnXGGOP8s+8sOtDdBlj5h35nIwIuxZEB/cUJMoY8ztjzCJjzAFjzG5jzCRjTN2w60K0GGM6HOV+8nXYtUVBubALyBPXiUhVZ9sZIjJaRGZmvxzkAmPMZSJyeth1IJK4pyAuY0x7EXlDROaLSHcRqSkiI0TkLWNMK2vtD2HWh0i6UUSW/iQXhlVIlNAMp4G19lN3mzHmGhE5KCIvZr8iRJ0xprqIPCoifxaRKSGXg4jhnoIE3S0iG0TkYmttoYiIMeYzEVkiIleLyOMh1oZoWm2t/TDsIqKGMYkMMMZUEJGeIjLLWrs77HoQSQ+LyCpr7QthF4Lo456Co2grIgt+bIRFRKy1S0Vkl4hcElpVQI6hGc6MS0WkiohMDLsQRI8x5iwR6SfFfxUOJIJ7CkpyWIr/tsD1g4j8ryzXgtww2Rhz2BizyxgzxRjTKOyCooAxiczoJyLbRWRu2IUgWowx5UXkKRH5q7X2P2HXg5zBPQUl+Y8UPx3+H8aYxiJygogcCqUiRNU3IjJKRN4TkW9FpKWIDBGRD4wxLa2128MsLmw8GU4zY0x9ETlHRCb/9K+ugCMGi0gFERkZdiHIDdxTEMN/ichvjDEjjDF1jDGniMgkESk68g8gIiLW2o+ttbdaa2dZa9+z1v5NRLqISF0pfqnOazTD6ddXiv935a8zoRz566g7ReQuETnWGFP9yIt08pNcNrwKEVHcU1Aia+1kKV494hYR+UpEPhWRL0VkjohsDbE05ABr7QoRWSMircOuJWzGWht2DXnFGPNvESm01rYIuxZEizGmg4i8E2e3ltbalVkoBzmCewriMcZUEpGmIrLdWvuVMWa1iCy11vYLuTRE3JHPygZrbZewawkTM8NpZIz5tYj8UkRuDrsWRNJKEfldCdvfEZF/iMh4EVmb1YoQadxTkAhr7T4R+ZeIiDGmi4icIsVLqwFHdeT+0kxEpoZdS9hohtOrnxQvYM26sQiw1n4tIu+6240xIsX/ZR74NXiPewqOyhjTUkS6isiKI5vOEpG/iMjD1trFoRWGyDHGTBaR9VL8Wflail+gu0OKx2rGhFhaJNAMp8mRVQIuE5F51tqvwq4HQG7jnoIEHBSR80XkNhE5VkRWi8gAa+2EUKtCFP1biu8nA0WkoohsE5FXReRua+3OMAuLAmaGAQAA4C1WkwAAAIC3aIYBAADgLZphAAAAeItmGAAAAN7K6moSxhje1ssj1lqTifPyOckvmfqciPBZyTfcU5AI7ilIVKKfFZ4MAwAAwFs0wwAAAPAWzTAAAAC8RTMMAAAAb9EMAwAAwFs0wwAAAPAWzTAAAAC8RTMMAAAAb9EMAwAAwFs0wwAAAPAWzTAAAAC8RTMMAAAAb9EMAwAAwFs0wwAAAPAWzTAAAAC8RTMMAAAAb9EMAwAAwFs0wwAAAPAWzTAAAAC8RTMMAAAAb5ULuwAApdezZ0+Vp06dqvKHH36o8hlnnJHxmgAAua1169Yqt2jRIrDPlVdeqbL782X48OEq33fffSofPnw4hQrTiyfDAAAA8BbNMAAAALxFMwwAAABvMTMM5Ihy5YJ/XAcNGqSytTZmBgDkn169eql8+eWXJ3W8MUblX//61yrXrVs37jncnzdDhw5VeePGjSqPHz8+mRIziifDAAAA8BbNMAAAALxFMwwAAABv5f3MsDsHU6ZMmZi/XrZs2cA53LVcmzVrllQN/fv3V7lBgwZJHV+SRx99VOU777xT5QMHDqR8DRzd4MGDA9sqVKig8rhx41T+8ssvU7rmiSeeGNjGusGAvzp06KDyJZdconKPHj1Url+/vsq7d+8OnHP58uUqX3bZZSrv2rUr2TKRBRdddJHKF1xwQVLHu71QJt43Oemkk9J+znThyTAAAAC8RTMMAAAAb9EMAwAAwFsmm+uQGmMyfrEaNWqo7H43dtOmTVVu2LChyqeddlpmCvuJ/fv3q1xUVBTYp2LFiiq7s86u2rVrq1zSLFi6WWtN/L2Sl43PSbLc2bz58+cH9nHXAXbneZcsWZJSDQMHDgxs+9vf/qayO/flfhf8Pffck1INpZGpz4lI/M/Kcccdp3L58uUD+/Tu3VvlJk2apKGy/8/9PRER+e6771QeM2ZMzHMkW+O1114b2Fa9enWV77//fpXdNUHD4NM9JRHuzyv3z7P7uYj3c6I0c6ErVqxQecCAASq7M8bZEOY9JQrcd4ZERG688caUzpmOmeFvv/1W5U8++URld675m2++SfoayUr0s8KTYQAAAHiLZhgAAADeohkGAACAt/JuneFhw4apXNLsXKr27t2r8qFDh1SeOHGiyps2bVL5hRdeUHn79u2Ba7gzhNddd13SdSJ9rr76apXd+eBMqFq1qsrumqGJePnll9NVTiTVrFlTZffPf6dOnVQ+9dRTM16Tq6SZYXcez323IROy+X4I4qtTp47KDzzwQGCf7t27q+zeE+L9nrrrzc+dO1flL774InCMuz7t6tWrVXbn1cOYGc537rsN7vcI9O3bN+M1uPO/Jf0sefvtt1X+4IMPVN6wYUP6C8sQngwDAADAWzTDAAAA8BbNMAAAALyVdzPDr7zyispr1qxRuVq1air36dNH5SeeeCLuNebMmaNyJuZi3DkthOvMM89UuaQ50HRr2bKlyu3bt497zLx581RetWpVWmuKmmbNmqncpUsXld31tt3Z79Jo3ry5yuvWrVN53759Kpf0WalUqZLKJa0hnYzKlSurXK9evZTOh/SrX7++ypMmTVLZXcs8Ee687rRp01R+/fXXVf73v/8d95yDBw9Oug6k1/HHH6/yXXfdlfQ53HnwZOeM3ZnhfO9JeDIMAAAAb9EMAwAAwFs0wwAAAPAWzTAAAAC8lXcv0L3//vsxs+vBBx/MZDmlNmjQoLBL8Frr1q1VPumkk1QuabH7JUuWqLxixYqUavjd734X95qupUuXpnTNXOMu8n7yySeHVElqxo4dm9LxvXr1Utn9Yp+SzJgxI6VrIjb3hbmFCxeq7H55RUm2bNmi8rhx41R++OGHVXa/ZMP1/PPPq1xQUBDYZ+TIkSqPHj06bp2IHvclXfcLiJ577jmVd+zYobL7ZWL5jifDAAAA8BbNMAAAALxFMwwAAABv5d3MMFAaFStWVDmRL19xufN7hYWFKdXUrVu3pI9xF0pv2LChyps2bUqpJuSPfF9EP2y33nqryu6MsPsOgPsFUSIibdu2Vfmbb75JqSb3mtWrVw/sc88996g8e/ZslUuqE9FTt25dlYcPHx4zL1iwQOVrr71W5fXr16exuujhyTAAAAC8RTMMAAAAb9EMAwAAwFvMDIegatWqKrvrOoqI1KtXL1vlQER69+6tcsuWLWPuv3PnzsC2Tz/9VOU+ffqoXLNmTZXbtWunsruW8SmnnBKzhpI88sgjKnfv3j3mNZEfLr/88rBL8J67rmvXrl1j7u+u6+r+WRVJfka4bNmyKv/+979X+YILLoh7Dvf/D3ftbmaGM2/Pnj0qjx8/XuWrr7467dc877zzVHbXrP/HP/4ROMadi0/1PZkw8WQYAAAA3qIZBgAAgLdohgEAAOAtZoazoFq1air/4Q9/UPm6666Le45du3apfPPNN6v83XfflbI6PzVo0EDlUaNGJXV8rVq1AtvcmeF4jDEqu2uApoNbZ5UqVVTeu3dv2q+J7HPXky7JCy+8oPL333+fqXK8dPHFF6vcrFmzmPtPmjRJ5WTvHyIirVq1UvnOO+9U+aKLLlK5NPccd5Z51qxZyZSIUjh48KDKN954o8rTpk0LHNOlSxeV3XdUkn2voEaNGioPHDgwsI87X37NNdckdY0o4ckwAAAAvEUzDAAAAG/RDAMAAMBbzAxnwdixY1V2158tibuOrTu3tWjRotQL85g721S9evWY+7uzdumQjnMuW7ZMZfdz465livxw4oknqnz88cfHPeadd95R+fDhw2mtyXcnnHCCyvH+fJc09xnPFVdcofLjjz+ucoUKFWIev3v3bpUrV64c2OeYY45RORP3PiTHne+fP39+YJ+Stv1Uv379Yv76hRdeqPKUKVNUrlixYuAY9/0n9+fRU089FfOaUcKTYQAAAHiLZhgAAADeohkGAACAt5gZTgN33tRdbzLe98GXNNfZs2dPlZkRTi/3f0931sldv/Prr79W+bPPPguc012zN956nO3bt1e5R48eMfdfsGBBYFu3bt1UPnToUMxzID/86le/UjmRdYZff/31TJUDEVm+fLnK7hq+7uztsGHDVF63bl3gnO67IvXq1Yt5DXd9Wnc9enet6XvvvTdwzeuvvz7mNZCfZs6cqfJNN92kcknzv+5nw51pZ2YYAAAAyAE0wwAAAPAWzTAAAAC8xcxwGrgzwuPHj4+5//bt21Xu1atXYJ+FCxemXhiO6s0331TZXYO1SZMmKu/fv1/lLVu2pFxDmzZtktp/6dKlgW3MCAPR4L6H4P4ccNdk7dy5c8rXnDdvnsojR45UefHixTGPd9evBX60fv36pI9xZ9rdvG3btpRqyiSeDAMAAMBbNMMAAADwFs0wAAAAvEUzDAAAAG/xAl0pjBo1SuX+/fvH3N99Ya53794q87Jc+A4fPqzy2rVr036NY489VuV4L9C4n5tnnnkm7TUhN3Xo0EFl9wsdkH3uy6x33323ykVFRSpfc801cc/pvuj76quvqjxhwgSV3S/dSIemTZum/ZyIHvfLwR566KGkz+G+IBflF+ZcPBkGAACAt2iGAQAA4C2aYQAAAHiLmWFH1apVA9v69u2rsjsj7B4Tb0b4n//8ZyolIkeNHTtW5Vq1aqlsrVX5ySefVHnDhg2ZKQw554wzzlDZ/ewgfFu3blV5wIABMXMYSprpdOfPzzrrrGyVgyMaNWqksvulTzt37kz6nDVq1FD5gQceUNl9D+EXv/hF0tfIpRlhF0+GAQAA4C2aYQAAAHiLZhgAAADeYmbY0bVr18C2MWPGxDxmx44dKvfq1Utl1hGGiEiDBg1Ujrc27JYtWzJZDgDPzZ49O7DtvvvuU7lChQoq9+zZU+WXX345/YV57k9/+pPK7du3V/mSSy4JHOO+g/Kzn/1M5dGjR6vszgS7P49K8x7C1KlTkz4mKngyDAAAAG/RDAMAAMBbNMMAAADwlvczwxdeeKHKTz/9dNLneOmll1RmRhgiwXUdO3bsqLI7k7V3716V586dm5nCkPd2794d2Hbo0KEQKkGUrVmzJrDt888/V7l58+Yqd+/eXWVmhjPvzDPPVLmk37fy5curXLZsWZWPO+64lGooLCwMbHPfj5o/f35K1wgTT4YBAADgLZphAAAAeItmGAAAAN7ybma4SpUqKg8ePFjlypUrxz1H//79VXZnhgERkaFDh6rsznS5XnzxRZU3b96c9prgh9deey2wbdeuXSFUgnxTvXr1sEvIe2+99ZbKt99+u8qZ+D1w3ylYsGCByg8++GDgmEWLFqW9jrDwZBgAAADeohkGAACAt2iGAQAA4K28nxmuVKmSyoMGDVK5bdu2cc8xYcIEld3Zzh9++KGU1SGftWvXLqn9ly9fnqFKACAxixcvVvn0009X2RiTzXK8tGLFCpW3bNmicv369VO+xvr161UePXq0yo8//njK18glPBkGAACAt2iGAQAA4C2aYQAAAHgr72eGzz33XJXvvffemPt/9tlngW3XXXedygcPHky9MOS9O+64Q+X77rtP5aZNm6o8ZcqUjNeE3FS7dm2Va9WqFVIlyHfuvKq1NmZG+n399dcqFxQUqDxv3rzAMVu3blW5YcOGKo8YMULlyZMnq+z7OuQ8GQYAAIC3aIYBAADgLZphAAAAeItmGAAAAN4y2RyGN8ZkffL+scceU3nAgAEq79mzR+UuXboEzrFs2bL0F5YHrLUZWX09jM8JMidTnxMRfz4r7du3V/ndd9+NuX9JXya0dOnSdJaUEdxTwlelShWVV65cqbL78qb7InA2XsTinoJEJfpZ4ckwAAAAvEUzDAAAAG/RDAMAAMBbef+lG/HccsstKjMfDCBqWrdundT+7rsQQKL27t2r8lNPPaXyjTfeqHJhYWHGawIyjSfDAAAA8BbNMAAAALxFMwwAAABv5f06w8gc1gRFIlgTNHVz5sxRuXPnzipPnz5d5V69egXOUVRUlP7C0ox7ChLBPQWJYp1hAAAAIA6aYQAAAHiLZhgAAADeyurMMAAAABAlPBkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt2iGAQAA4C2aYQAAAHiLZhgAAADeohlOA2NMD2PMNGPMBmPMAWPMf4wxDxhjqoRdG6LFGHOiMWaMMeYDY8x+Y4w1xpwUdl2IFu4pSIYx5nxjzD+NMd8ZY741xiwzxnQMuy5EhzGmszHmbWPMNmPMD8aYzcaYqcaY08KuLQqMtTbsGnKeMeZDEdkoIq+JyGYRaSki94jIZyLSzlpbFF51iBJjTAcReUlElotIWRE5T0SaWGv/O8SyEDHcU5AoY8yfRGTskX/mSPFDrhYisspaOzvM2hAdxpjLROR/i8hHIrJDRBqJyO0i0lBEfmWt3RBieaGjGU4DY0xta+0OZ1s/EZkoIp2stW+HUxmixhhT5sdGxhjzBxEZJzTDcHBPQSKO/K3SahG5w1r7t3CrQa4xxpwsxf+Bfau1dlTY9YSJMYk0cH9oHbH0yL8bZLMWRBtP9JAI7ilIUH8RKRKRJ8MuBDlp15F/Hwq1igigGc6cs4/8e3WoVQDIF9xT4DpLip/s/d4Ys84YU2iMWWuMuT7swhBNxpiyxphjjDG/EJGnRGSbiLwYclmhKxd2AfnIGNNARO4TkTettcvCrgdAbuOegqOof+SfR0RkiIisE5GeIjLWGFPOWvtfYRaHSPpIRFod+b/XikhHa+32EOuJBJ4Mp5kxprIUv/RSKCJXhVwOgBzHPQUxlBGRKiLyJ2vtOGvt29baa0VknojcYYwx4ZaHCPo/ItJWRPqIyLcisoAVjWiG08oYc5yIzBSRpiLS2Vq7OeSSAOQw7imI48eZzwXO9jdEpK6InJDdchB11trV1tqPrLUviEgnEaksxatKeI1mOE2MMeVFZJqI/EZEzrfW/ivkkgDkMO4pSMCqo2z/8YkwL+ziqKy1X0vxqMTPw64lbDTDaWCMKSMik6X4v7IustZ+GHJJAHIY9xQkaPqRf3d2tncWkc3W2m1Zrgc5xBhTV0ROkeJZc6/xAl16PCbFLy2MFJF9xpi2P/m1zfzVJn7KGNPjyP/540sMXY0xO0Rkh7X2vZDKQrRwT0Ei5ojIOyLylDGmloh8ISI9pPjLfJgvx/8wxkwXkRUi8n+leFa4mYj8WYrfRfB6jWERvnQjLYwx/y0ijY/yy/daa+/JXjWIOmPM0f7QvWet7ZDNWhBN3FOQKGNMVRF5QIqb4BpSvNTag9baKaEWhkgxxgwWkV4i8jMROUZENonIuyLyAF/6RDMMAAAAjzEzDAAAAG/RDAMAAMBbNMMAAADwFs0wAAAAvJXVpdVivEWPHGStzchXffI5yS+Z+pyI8FnJN9xTkAjuKUhUop8VngwDAADAWzTDAAAA8BbNMAAAALxFMwwAAABv0QwDAADAWzTDAAAA8BbNMAAAALxFMwwAAABv0QwDAADAWzTDAAAA8BbNMAAAALxVLuwCAABA6ipVqqTylClTVL7wwgtV/utf/xo4x1/+8pf0FwZEHE+GAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt3iBLgOOOeYYlW+66SaVhw0bpnLFihUD5zDGqDxz5kyV3Zcc1qxZk3SdyK5zzjlH5QULFqh8yy23qDx69OiM14Rw1KpVS+Wbb75Z5VNPPVXlJUuWqHz//ferXFRUpPLHH38cuOa0adNUfu6551TeunXr0QtGTujUqZPKBQUFKrsv0O3cuTPjNSGaatasqbLblwwcOFBltycREbHWxjzHiBEjUikxq3gyDAAAAG/RDAMAAMBbNMMAAADwFjPDadC6dWuVhw8frvK5554b83h37qakbRdccIHK69atU9mdOUT0nHbaaSq7c57unCjyw8UXXxzY1rJlS5Vvu+22mOfo1q2byu5nx71ftGjRInAOd1u7du1iXgO5p0+fPjF/vVq1airPmjUrk+Ugi9x3j9zfa/fP+8svv6yyew+Jl0vSv39/lb/88kuVJ0yYEPccYeHJMAAAALxFMwwAAABv0QwDAADAW8wMJ6B8+fIqu+vFPvPMMyrXq1cvqfMvW7YssK1Ro0Yq16lTR2V3Nmz69OkqL1y4MKkakHlt27YNuwRkwQknnKCy+w6BSDTmwzt27Kiyu67omDFjslkOSsF9X+XSSy+NuX+FChUyWQ6yxO0PRIJ/Xt33jJJ16NAhlX/44YfAPpUrV1a5cePGKnfu3FllZoYBAACACKIZBgAAgLdohgEAAOAtZoYT4H6H9+zZs5M6fsuWLSoPGDBA5blz5waOcdcEfO+991SuXbu2yu68HzPD0XP66afH/PWuXbuqXK6c/uNZWFiY9pqQuvr166s8depUlTMxHzxjxgyVmzRponLz5s3jnuPYY49V+bLLLlOZmeHoKVu2rMpDhgxRuUyZ2M+3xo0bl/aakHnuuuRvvfVWYB93XeF4Vq5cqfL8+fNVdnuOq666KnCOHj16xLzGV199lVRNYeLJMAAAALxFMwwAAABv0QwDAADAW8wMp8HOnTtVfuKJJ1QeP368yps2bYp7zo0bN6ZeGHKKuz6tMSakSpCMNm3aqFya9aRXrVqlsjvb+dFHH6m8dOlSlevWravyJ598ErhGrVq1kq4L0eK+d9CtW7eY+69ZsyaT5SBL3Pn+rVu3Bvb5/PPPVXbfKxo0aJDKs2bNinlN9z7Wvn37uHXu2bNH5bFjx8Y9Jip4MgwAAABv0QwDAADAWzTDAAAA8BYzw6XgzsU8+OCDKj/66KMpX8NdizieJ598MuVrAkjeokWLVHbXAO3UqVPgmMWLF6vcq1cvlbdt25ZUDe56ngcPHkzqeOSGeDPCrhEjRmSoEmTThx9+qPLZZ58d2Gffvn0qu+8IxHtX6Te/+Y3K06dPV7lOnTpx65wyZYrK7hxzlPFkGAAAAN6iGQYAAIC3aIYBAADgLWaGE+DO4zVs2FDlAwcOpHT+rl27BrbdcsstMY9x55Td7xEHkB07duxQuW/fviq3a9cucMySJUtUTnZG2HX++eerfPzxx6d0PkRTsu+SfPDBBxmqBGFyv9ugJPFmhN33FK644gqVSzMjPGTIkLjHRBVPhgEAAOAtmmEAAAB4i2YYAAAA3qIZBgAAgLd4gS4B1lqVU31hzh1MnzFjRmCfcuX0b427ePXTTz+t8uHDh1OqCeFzX6JyP3fIDe4Lda+99lrGr9mkSROVjzvuuIxfE5nXsWNHlatXrx5zf/dl7/3796e9JmRfQUGByr/97W8D+1xyySVJnfPnP/+5yqX5eTN58mSVv/vuu6TPERU8GQYAAIC3aIYBAADgLZphAAAAeIuZ4Sw466yzVJ4/f77K7nywiMju3btVdr+YY8OGDWmqDlExZ84clQsLC0OqBD6aPXt22CXAcdttt6lcvnz5mPu7X+bizhAjN7izuN27d1c53ucgW9w6hw0bpvJjjz2WzXJSwpNhAAAAeItmGAAAAN6iGQYAAIC3mBnOgDZt2qj897//XWV3DdCioqLAOR5++GGV169fn6bqAOSb8847L+lj1q1bp7I7/4fwVatWTWVjTMz9Fy5cmMlyRCT488td8/aNN97IeA35rkaNGipnY0Z4wYIFKrv3hwEDBgSOcde9vuGGG1R+5ZVXVI7yDDtPhgEAAOAtmmEAAAB4i2YYAAAA3mJmOA3cGeHhw4erfPrpp8c8/vbbbw9sGzVqVOqFAchL8+bNU/ncc89N+hzuWuWsXR4udz5YRKRKlSoqW2tVdn/Pnn/++fQX5mjRooXK/fr1U5mZ4dR9//33Ku/bty/pcyxbtkzlN998U+X7778/qfNdf/31gW1z585VuXPnzio3btxYZWaGAQAAgAiiGQYAAIC3aIYBAADgLWaGS+Hiiy9WeeLEiSpXrlw55vHu982PHj06PYUhp7lrNrprirrzgvDHGWecobI7I5zIZ+OLL75QuaQZQITn1FNPDWw75ZRTYh6zd+9elXfu3JnWmkryy1/+MuPX8N2ll14adgkJce87bh4yZIjKPXr0ULmwsDAzhZUCT4YBAADgLZphAAAAeItmGAAAAN5iZjgBBQUFKk+ePFll97vaXZ9++qnKTz/9tMrMgkIkOCdWrpz+43no0KFsloM0Oe+88wLb6tWrp3LVqlVVdt8rcH+9NOrXr6/y4MGDVd68ebPKzz77rMobN25MuQYcXbt27cJaYI4yAAAF/UlEQVQuISEdO3ZUmZ9fOJpu3bqpfNJJJ6m8du3aLFYTG0+GAQAA4C2aYQAAAHiLZhgAAADe8n5muFKlSiq/8sorgX3OOecclcuU0f8NUVRUpPKrr76qcp8+fVQ+fPhw3Lrca5QtW1Zl5keBaHDvD+68b9u2bQPHVKxYMaM1lcR9t+HKK6+Muf9VV12l8sCBA1WeOXNmWupCsVatWoVdAnBUTZo0CWxr0aJFCJVkBk+GAQAA4C2aYQAAAHiLZhgAAADeohkGAACAt/L+BboKFSqofPbZZ6v80ksvqVy5cuW453RfXrvrrrtUfv7551Xu3LmzyjVr1lS5TZs2gWu4dezZs0flP//5z3HrRG5zFyj//PPPwykEyuzZs1Xu0KGDyvG+hCdXNGjQQOVq1aqFVEl+cj8njRs3Tvoca9asSVc5yCL3z9a4ceOSOn716tWBbUOHDlX5wIEDyRf2E27vNGTIkMA+devWjXmOhQsXqrxly5aUasokngwDAADAWzTDAAAA8BbNMAAAALyV9zPDPXv2VHnChAkpn9OdxWnZsqXKt956q8q1atVK+hruTFBBQUHS50C0TJo0SeWRI0fG3L9Lly4qMzMcDc2bN1c5CjPCy5cvV7mkL/ZJtm7387Zz585SVoeSuO+elOZ/3/nz56erHGTRH//4R5Xd94riKWn/9u3bq+zO67755psqG2NUttaqXL9+fZX79++fVI0iIu+//77K+/fvT/oc2cKTYQAAAHiLZhgAAADeohkGAACAt/JuZvjkk09WefTo0Wm/RtWqVVXu3bt3zP0/+ugjld25mUWLFgWOefbZZ1XeuHFjMiUigqI8L4Xc0r17d5XdtY8LCwsDx3Tq1EnleGuqL1myROWtW7cmUyLiqF69usruzGdJioqKVHbXn0duOPHEE9N+zlatWsXMN910k8rxZoZLw50RHjFiRMrnzBaeDAMAAMBbNMMAAADwFs0wAAAAvJV3M8M33HCDyjVq1Mj4NRcvXqyyu77f8OHDVU71O8Phh3Ll8u6PJxL01VdfqXz77berPGPGjKTP+dZbb6VUE9Jr165dKs+bNy+wj/s+ivuz49VXX01/YUmaPn162CXknU2bNqn8xRdfqOz2HCIi7dq1U/nss89Of2EO992mu+++W+Vc6nV4MgwAAABv0QwDAADAWzTDAAAA8FbeDSXWq1cvpeM//vjjwLb33ntP5ccff1zlbdu2qbxv376UagBERAoKClR+9NFHQ6oE6eauF/vcc8+pPGbMGJX/9a9/ZbokhOyhhx4KbOvQoYPKa9euzVI1Rzdx4kSV3Z+PiM/9833llVeq3LBhw5i5pHngTPcdO3bsCGxzv8dhy5YtGa0hk3gyDAAAAG/RDAMAAMBbNMMAAADwlknH91EnfDFjMn6xvn37quyu8duoUSOVn3jiCZWHDBkSOOe3336bpuryi7XWxN8redn4nIShTp06Kr/zzjsqr1q1SmV3zezt27dnprAMy9TnRCScz8r111+v8tChQ1WuXbu2yps3bw6cY8SIESo/88wzaaout3FPQSJy/Z5Spox+DtmkSROV3TWnmzZtmvYaCgsLVX7kkUdUfv3111VeuXJl4By5sI5wop8VngwDAADAWzTDAAAA8BbNMAAAALyVdzPDyB7m+5CIXJ/vQ/ZwT0EiuKcgUcwMAwAAAHHQDAMAAMBbNMMAAADwFs0wAAAAvEUzDAAAAG/RDAMAAMBbNMMAAADwFs0wAAAAvEUzDAAAAG/RDAMAAMBbNMMAAADwFs0wAAAAvEUzDAAAAG/RDAMAAMBbNMMAAADwFs0wAAAAvGWstWHXAAAAAISCJ8MAAADwFs0wAAAAvEUzDAAAAG/RDAMAAMBbNMMAAADwFs0wAAAAvEUzDAAAAG/RDAMAAMBbNMMAAADwFs0wAAAAvEUzDAAAAG/RDAMAAMBbNMMAAADwFs0wAAAAvEUzDAAAAG/RDAMAAMBbNMMAAADwFs0wAAAAvEUzDAAAAG/RDAMAAMBbNMMAAADwFs0wAAAAvEUzDAAAAG/9P6UsuwG+ZDuUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x432 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plots(imgs_p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Broadcasting / Matrix Multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([10,  6, -4]), array([2, 8, 7]))"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.array([10, 6, -4])\n",
    "b = np.array([2, 8, 7])\n",
    "a,b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([12, 14,  3])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([False,  True,  True]), 0.6666666666666666)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(a < b), (a < b).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Broadcasting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True, False])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([11,  7, -3])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3],\n",
       "       [4, 5, 6],\n",
       "       [7, 8, 9]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = np.array([[1,2,3], [4,5,6], [7,8,9]]); m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2,  4,  6],\n",
       "       [ 8, 10, 12],\n",
       "       [14, 16, 18]])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2 * m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10, 20, 30])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = np.array([10,20,30]); c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[1, 2, 3],\n",
       "        [4, 5, 6],\n",
       "        [7, 8, 9]]), array([10, 20, 30]))"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m, c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([140, 320, 500])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m @ c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([300, 360, 420])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c @ m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "module 'numpy' has no attribute 'broadbcast_to'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-57-dd2b7fa43478>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbroadbcast_to\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mm\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m: module 'numpy' has no attribute 'broadbcast_to'"
     ]
    }
   ],
   "source": [
    "np.broadbcast_to(c, m.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[10, 10, 10],\n",
       "       [20, 20, 20],\n",
       "       [30, 30, 30]])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.broadcast_to(c[:,None], m.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[10, 20, 30]])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.expand_dims(c, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[10],\n",
       "       [20],\n",
       "       [30]])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.expand_dims(c, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[10, 10, 10],\n",
       "       [20, 20, 20],\n",
       "       [30, 30, 30]])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.broadcast_to(np.expand_dims(c, 1), (3,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[0],\n",
       "        [1],\n",
       "        [2],\n",
       "        [3],\n",
       "        [4]]), array([[0, 1, 2, 3, 4]]))"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xg, yg = np.ogrid[0:5, 0:5]\n",
    "xg, yg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 2, 3, 4],\n",
       "       [1, 2, 3, 4, 5],\n",
       "       [2, 3, 4, 5, 6],\n",
       "       [3, 4, 5, 6, 7],\n",
       "       [4, 5, 6, 7, 8]])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xg + yg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix Mulutiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[1, 2, 3],\n",
       "        [4, 5, 6],\n",
       "        [7, 8, 9]]), array([10, 20, 30]))"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m, c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([140, 320, 500])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m @ c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([140., 320., 500.])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Tensor(m) @ Tensor(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 10,  40,  90],\n",
       "       [ 40, 100, 180],\n",
       "       [ 70, 160, 270]])"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m * c # element-wise multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([140, 320, 500])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(m * c).sum(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([120, 300, 540])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(m * c).sum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing a training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 01:43 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>nan</th>\n",
       "    <th>nan</th>\n",
       "    <th>0.098000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>nan</th>\n",
       "    <th>nan</th>\n",
       "    <th>0.098000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>nan</th>\n",
       "    <th>nan</th>\n",
       "    <th>0.098000</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def get_weights(*dims): return nn.Parameter(torch.randn(dims)/dims[0])\n",
    "def softmax(x): return torch.exp(x)/(torch.exp(x).sum(dim=1)[:,None])\n",
    "\n",
    "class LogReg(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.l1_w = get_weights(28*28*3, 10)\n",
    "        self.l1_b = get_weights(10)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = x.view(x.size(0), -1)\n",
    "        x = x @ self.l1_w + self.l1_b\n",
    "        return torch.log(softmax(x))\n",
    "\n",
    "net2 = LogReg()#.cuda()\n",
    "#opt = optim.Adam(net2.parameters())\n",
    "opt = partial(optim.Adam)\n",
    "\n",
    "learn2 = Learner(data, net2, loss_func=loss, opt_func=opt, metrics=metrics)\n",
    "learn2.fit(epochs=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "net2 = LogReg()#.cuda()\n",
    "loss = nn.NLLLoss()\n",
    "learning_rate = 1e-3\n",
    "#optimizer = partial(optim.Adam, lr=learning_rate)\n",
    "optimizer=optim.Adam(net2.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "#dl = iter(md.trn_dl) # Data loader\n",
    "dl = iter(data.train_dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "xt, yt = next(dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = net2(Variable(xt))#.cuda())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(2.3013, grad_fn=<NllLossBackward>)\n"
     ]
    }
   ],
   "source": [
    "# Loss\n",
    "l = loss(y_pred, Variable(yt))#.cuda())\n",
    "print(l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0625"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Accuracy\n",
    "np.mean(to_np(y_pred).argmax(axis=1) == to_np(yt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer.zero_grad()\n",
    "l.backward()\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "xt, yt = next(dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = net2(Variable(xt))#.cuda())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(2.6404, grad_fn=<NllLossBackward>)\n"
     ]
    }
   ],
   "source": [
    "l = loss(y_pred, Variable(yt))#.cuda())\n",
    "print(l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.265625"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Accuracy\n",
    "np.mean(to_np(y_pred).argmax(axis=1) == to_np(yt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/michael/anaconda3/envs/fastai/lib/python3.6/site-packages/ipykernel_launcher.py:8: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 2.673752546310425 \t accuracy: 0.34375\n",
      "loss: 0.5768854022026062 \t accuracy: 0.8125\n",
      "loss: 0.5686171054840088 \t accuracy: 0.828125\n",
      "loss: 0.386569619178772 \t accuracy: 0.859375\n",
      "loss: 0.3499939739704132 \t accuracy: 0.890625\n",
      "loss: 0.820191502571106 \t accuracy: 0.765625\n",
      "loss: 0.4876093864440918 \t accuracy: 0.890625\n",
      "loss: 0.26617351174354553 \t accuracy: 0.875\n",
      "loss: 0.3273327350616455 \t accuracy: 0.9375\n",
      "loss: 0.6281328201293945 \t accuracy: 0.828125\n"
     ]
    }
   ],
   "source": [
    "for t in range(100):\n",
    "    xt, yt = next(dl)\n",
    "    y_pred = net2(Variable(xt))#.cuda())\n",
    "    l = loss(y_pred, Variable(yt))#.cuda())\n",
    "    \n",
    "    if t % 10 == 0:\n",
    "        accuracy = np.mean(to_np(y_pred).argmax(axis=1) == to_np(yt))\n",
    "        print(f\"loss: {l.data[0]} \\t accuracy: {accuracy}\")\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    l.backward()\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it together in a training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score(x, y):\n",
    "    y_pred = to_np(net2(Variable(x)))\n",
    "    return np.sum(y_pred.argmax(axis=1) == to_np(y)) / len(y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "937"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.train_dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9137141719745223\n"
     ]
    }
   ],
   "source": [
    "net2 = LogReg()#.cuda()\n",
    "loss=nn.NLLLoss()\n",
    "learning_rate = 1e-2\n",
    "optimizer=optim.SGD(net2.parameters(), lr=learning_rate)\n",
    "\n",
    "for epoch in range(1):\n",
    "    losses=[]\n",
    "    dl = iter(data.train_dl)\n",
    "    for t in range(len(data.train_dl)):\n",
    "        # Forward pass: compute predicted y and loss by passing x to the model.\n",
    "        xt, yt = next(dl)\n",
    "        y_pred = net2(Variable(xt))\n",
    "        l = loss(y_pred, Variable(yt))\n",
    "        losses.append(l)\n",
    "\n",
    "        # Before the backward pass, use the optimizer object to zero all of the\n",
    "        # gradients for the variables it will update (which are the learnable weights of the model)\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # Backward pass: compute gradient of the loss with respect to model parameters\n",
    "        l.backward()\n",
    "\n",
    "        # Calling the step function on an Optimizer makes an update to its parameters\n",
    "        optimizer.step()\n",
    "    \n",
    "    val_dl = iter(data.valid_dl)\n",
    "    val_scores = [score(*next(val_dl)) for i in range(len(data.valid_dl))]\n",
    "    print(np.mean(val_scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stochastic Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9066480891719745\n"
     ]
    }
   ],
   "source": [
    "net2 = LogReg()#.cuda()\n",
    "loss=nn.NLLLoss()\n",
    "lr = 1e-2\n",
    "w,b = net2.l1_w,net2.l1_b\n",
    "\n",
    "for epoch in range(1):\n",
    "    losses=[]\n",
    "    dl = iter(data.train_dl)\n",
    "    for t in range(len(data.train_dl)):\n",
    "        # Forward pass: compute predicted y and loss by passing x to the model.\n",
    "        xt, yt = next(dl)\n",
    "        y_pred = net2(Variable(xt))\n",
    "        l = loss(y_pred, Variable(yt))\n",
    "        losses.append(l)\n",
    "\n",
    "        # Backward pass: compute gradient of the loss with respect to model parameters\n",
    "        l.backward()\n",
    "        w.data -= w.grad.data * lr\n",
    "        b.data -= b.grad.data * lr\n",
    "        \n",
    "        w.grad.data.zero_()\n",
    "        b.grad.data.zero_() \n",
    "    \n",
    "    val_dl = iter(data.valid_dl)\n",
    "    val_scores = [score(*next(val_dl)) for i in range(len(data.valid_dl))]\n",
    "    print(np.mean(val_scores))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}