Attempt to reproduce baseline using memory-cached dataset

how_to_train_your_resnet_fastaiv1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "how_to_train_your_resnet_fastaiv1.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "pDnf_yQxleHx"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/davidpfahler/8c397cffe57734a03183003129a108dd/how_to_train_your_resnet_fastaiv1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OKqJD7RwMOKQ",
        "colab_type": "text"
      },
      "source": [
        "# \"How to train your ResNet: Part 1\" in fastai v1\n",
        "\n",
        "The goal of this notebook is to reproduce the results of the great article series called \"How to train your ResNet\" [Part 1](https://myrtle.ai/how-to-train-your-resnet-1-baseline/) by myrtle.ai. If all goes well, I might implement the entire article series, but we are starting with Part 1 for now.\n",
        "\n",
        "You can find the code for the experiments corresponding to the diffferent parts of the article series here: https://github.com/davidcpage/cifar10-fast/blob/master/experiments.ipynb."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HBNI1f-vCG_P",
        "colab_type": "text"
      },
      "source": [
        "## Setup\n",
        "\n",
        "Install & import fastai."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "31Pw8JjsJP43",
        "colab_type": "code",
        "outputId": "08410658-0036-4598-c8f7-8a6f99cda733",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 714
        }
      },
      "source": [
        "!pip install fastai"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: fastai in /usr/local/lib/python3.6/dist-packages (1.0.57)\n",
            "Requirement already satisfied: numpy>=1.15 in /usr/local/lib/python3.6/dist-packages (from fastai) (1.16.5)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.6/dist-packages (from fastai) (4.6.3)\n",
            "Requirement already satisfied: numexpr in /usr/local/lib/python3.6/dist-packages (from fastai) (2.7.0)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (from fastai) (0.3.0)\n",
            "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from fastai) (0.6)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from fastai) (3.13)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.6/dist-packages (from fastai) (4.3.0)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from fastai) (1.3.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from fastai) (2.21.0)\n",
            "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from fastai) (1.1.0)\n",
            "Requirement already satisfied: spacy>=2.0.18 in /usr/local/lib/python3.6/dist-packages (from fastai) (2.1.8)\n",
            "Requirement already satisfied: typing; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from fastai) (3.7.4.1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from fastai) (19.1)\n",
            "Requirement already satisfied: fastprogress>=0.1.19 in /usr/local/lib/python3.6/dist-packages (from fastai) (0.1.21)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from fastai) (0.24.2)\n",
            "Requirement already satisfied: bottleneck in /usr/local/lib/python3.6/dist-packages (from fastai) (1.2.1)\n",
            "Requirement already satisfied: nvidia-ml-py3 in /usr/local/lib/python3.6/dist-packages (from fastai) (7.352.0)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from fastai) (3.0.3)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision->fastai) (1.12.0)\n",
            "Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from Pillow->fastai) (0.46)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->fastai) (1.24.3)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->fastai) (3.0.4)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->fastai) (2.8)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->fastai) (2019.6.16)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (2.0.2)\n",
            "Requirement already satisfied: srsly<1.1.0,>=0.0.6 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (0.1.0)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (0.2.2)\n",
            "Requirement already satisfied: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (2.0.1)\n",
            "Requirement already satisfied: blis<0.3.0,>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (0.2.4)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (1.0.2)\n",
            "Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (0.9.6)\n",
            "Requirement already satisfied: thinc<7.1.0,>=7.0.8 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (7.0.8)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->fastai) (2.4.2)\n",
            "Requirement already satisfied: attrs in /usr/local/lib/python3.6/dist-packages (from packaging->fastai) (19.1.0)\n",
            "Requirement already satisfied: python-dateutil>=2.5.0 in /usr/local/lib/python3.6/dist-packages (from pandas->fastai) (2.5.3)\n",
            "Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas->fastai) (2018.9)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->fastai) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->fastai) (0.10.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /usr/local/lib/python3.6/dist-packages (from thinc<7.1.0,>=7.0.8->spacy>=2.0.18->fastai) (4.28.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib->fastai) (41.2.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_NP5L7hrCF1D",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%reload_ext autoreload\n",
        "%autoreload 2\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fTRD0vpIDQF-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from fastai.vision import *"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zjx0ruKgj4HM",
        "colab_type": "text"
      },
      "source": [
        "### Transforms\n",
        "\n",
        "The below is supposed to give us the same transforms as in the Part 1: Baseline  experiment, namely `[Crop(32, 32), FlipLR()]`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AHUznZRXiNxm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "crop = rand_resize_crop(32, ratios=(1.,1.))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sY1zckY9kZnU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "54010207-2c86-4beb-d9ab-7fa51cd25add"
      },
      "source": [
        "crop"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[RandTransform(tfm=TfmCoord (zoom_squish), kwargs={'scale': (1.0, 2.0, 8), 'squish': (1.0, 1.0, 8), 'invert': (0.5, 8), 'row_pct': (0.0, 1.0), 'col_pct': (0.0, 1.0)}, p=1.0, resolved={}, do_run=True, is_random=True, use_on_y=True),\n",
              " RandTransform(tfm=TfmPixel (crop), kwargs={'size': 32}, p=1.0, resolved={}, do_run=True, is_random=True, use_on_y=True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PJSHFbUUu_8j",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tfms = get_transforms(max_rotate=0., max_zoom=0., max_lighting=0., max_warp=0., xtra_tfms=[*crop])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E8VGEK7BkIXO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "aeec7045-8f39-40f1-bf1c-3709b615e39e"
      },
      "source": [
        "tfms[0] # train set transforms"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[RandTransform(tfm=TfmCrop (crop_pad), kwargs={'row_pct': (0, 1), 'col_pct': (0, 1), 'padding_mode': 'reflection'}, p=1.0, resolved={}, do_run=True, is_random=True, use_on_y=True),\n",
              " RandTransform(tfm=TfmPixel (flip_lr), kwargs={}, p=0.5, resolved={}, do_run=True, is_random=True, use_on_y=True),\n",
              " RandTransform(tfm=TfmCoord (zoom_squish), kwargs={'scale': (1.0, 2.0, 8), 'squish': (1.0, 1.0, 8), 'invert': (0.5, 8), 'row_pct': (0.0, 1.0), 'col_pct': (0.0, 1.0)}, p=1.0, resolved={}, do_run=True, is_random=True, use_on_y=True),\n",
              " RandTransform(tfm=TfmPixel (crop), kwargs={'size': 32}, p=1.0, resolved={}, do_run=True, is_random=True, use_on_y=True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bdVDoIOYlV4Q",
        "colab_type": "text"
      },
      "source": [
        "TODO: check if the transforms are actually identical"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Em3uv7BzlYgp",
        "colab_type": "text"
      },
      "source": [
        "## create databunch"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uxPKhALbChdZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "path = untar_data(URLs.CIFAR)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6QIGq9O93XYo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "batch_size = 128"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ZGZf2KEix88",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data = ImageDataBunch.from_folder(path, valid='test', bs=batch_size, ds_tfms=tfms)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v_Lbl0l2z5qn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cifar_stats = ([0.491, 0.482, 0.447], [0.247, 0.243, 0.261])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "odwwR8Rj0ECj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        },
        "outputId": "c6b64b94-6e72-4e45-a25b-63f323038746"
      },
      "source": [
        "data.normalize(cifar_stats)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ImageDataBunch;\n",
              "\n",
              "Train: LabelList (50000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "horse,horse,horse,horse,horse\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Valid: LabelList (10000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "horse,horse,horse,horse,horse\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Test: None"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TGnWaj3JKr9p",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 369
        },
        "outputId": "ab327538-4af9-4a0e-a0ca-efc0b79b9a82"
      },
      "source": [
        "data.show_batch(rows=3, figsize=(5,5))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVIAAAFgCAYAAADpZ/FJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvWm0JdlVHvjtuPP05pdj5VBVUkkq\nSiAmy8OS29B0M8nG0AbsNg3CQMOyQeqWGdfCIGNkuqG1MFjd4CUMtBDYEgIsMKNlUBsNDBIIIZVQ\nVamyqrKqsnJ8+YY73xunf+y94+y49+bLrLpVL9+t3N9amTdexImIE3EiTuzh23tTCAEOh8PheO5I\nbncHHA6HY9HhE6nD4XDMCZ9IHQ6HY074ROpwOBxzwidSh8PhmBM+kTocDsecWNiJlIj+MRH93hz7\nv46I3v989snx3OHjeeeBiH6eiH74dvfj+cDCTqQhhF8MIfyPt7sfjucHPp6ORcbCTqT7gYiKt7sP\njucPPp6OW8XtelYO/URKRN9LRJ8mol0iepCIvlLW51Q5IgpE9M+I6GEAD5t1ryeiR4noChH9GBHN\nvGYi+gkiOk9EO0T0ESJ6jdn2JiJ6FxG9XfrxCSL6PLP9BBH9ChFdJqJzRPT6F+yGLDh8PO9cENFn\nE9GfyT1/J4Cq2fZaIvooEV0nog8S0WeabTccDxnLdxPRO4hoB8DrDvSiBId+IgXwaQCvAbAM4F8C\neAcRHb9B278P4NUA7jfrvhLA5wH4HABfAeCf3GDfPwXwKgBrAH4JwC8TUdVs/3sA/iOAFQC/DuCt\nACAv8m8A+AsAJwH89wD+NyL64md1lXcOfDzvQBBRGcB/AvAL4DH5ZQD/k2z7bAA/C+BbAawD+HcA\nfp2IKrc4Hl8B4N3gsfzFA7mgSYQQFuofgI/KjXsdgPeb9QHAF060DQC+xPz9TwH8V1nO7T/jPFsA\nPkuW3wTgvWbb/QC6svxqAE9M7Pt9AH7udt+rRfjn43ln/APwtwE8DYDMug8C+GEAPwXgX020/xSA\n/+5m4yFj+d9u9/UdetsTEX09gDcCOCurmgA2AIxnND9/k3WPAzhxg/N8J4Bvku0BwJKcR/GMWe4A\nqIo95gyAE0R03WwvAPjD2Vd0Z8PH847FCQBPBZn9BI/L7xkA30BE32G2lWWfMW4+HrOekwPFoZ5I\niegMgLeBxfkPhRDGRPRRAHSDXWalsjoF4BOyfBr8VZw8z2sAfLec5xMhhJSItvY5j8V5AOdCCC+9\nhbZ3NHw872hcAHCSiMhMpqfBpp7zAN4cQnjz5E5E9Ddw8/G47SnsDruNtAG+SZcBgIi+EcADz/IY\n30VEq0R0CsAbALxzRpsWgJGcp0hEPwCWYG4FfwJgl4i+h4hqRFQgogeI6POfZT/vBPh43rn4EHhM\nXk9EJSL6KgB/Tba9DcC3EdGridEgoi8nohYWZDwO9UQaQngQwFvAg3ARwCsBfOBZHuY9AD4CtsX9\nJoB/P6PN7wL4HQAPgdWNHm5RXQghjAG8FuzYOAfgCoCfATtTHAY+nncuQggDAF8FtmVfA/C1AH5V\ntn0YwLeAHX5bAB6RdgszHpQ3Wby4QEQBwEtDCI/c7r445oePp+Ow4lBLpA6Hw7EI8InU4XA45sSL\nWrV3OByOg4BLpA6HwzEnDpRH+sAPvj8AABk6H5H+Uu43DzL/I8cGVIF6cre8oD0hdYfptSGkssAH\nSs03Js3ajwAAhXGbf3vXsjalPeZ3V3bYOUx7l2Pvhx0AQLHIt7uwxLzwvaWzWZud6lEAwIP/5utv\nhet4KPC2X34v3xkT7l5IeDmRAUiI72vB7FeUK9R1ibni2eMPhNxopbm2s3chOXaS+zt3TOlj1MrM\nOYiXCwXev1yIr0ohKeT6PZZQgnGIMQW6/OVf+gULM56v/R8+JwDxegC900ClXgYAbB5fAwAcOxFj\nG0juX6/TAwCM+vEAK2urAIBCgdclRXkeivGJKJQ4crderwEAyhX+u98fZW12d3a5TZX7UTQzV7lS\n4n7IO9uWfuzu7WZtmi0+ZqPJv6WCeb9Hsjzkvi0t1XmfpRhRPBhzX/7ea39w5ni6ROpwOBxz4kAl\n0rp8Rojil/9GEml+2r+5RDolcAS7OGkHnhZJ05DMbMIbRboZiwA2ZEk07EQWTnGbo92Welf571En\n2zaQT3woMfWtXOUv3qjaytr0yk0sGkqJSHSUZusSUkmQ1yVyIxNzQ5NsjG8urGXSYu6ZkXMk++2/\nn7SaawISLST/lOSl1GCkzTDma0v1HEF/7TUunu+BwNJewfS9VOL3olpjqS+RGaNkZo6QqpTJfydJ\nOdu2tMJxEIUCH3MwYmmxNxxkbQZDlvaSYR8AkIocPB6Y50p+KyU+dppGaXUk7QoF7ZvMM0WjKZV4\nY1EkUTtWBdFGg4zjaMDbOp1h1maYxv7OgkukDofDMSd8InU4HI45caCqfbPM4nWw6lamCvKfJXVE\npFY1Ksp+ifxGkT+IGpCSOolEvjcnIUkslGS/eVEeAMaq58m6glEdigNxLvXZgdS+/CAAoHf107GL\n4ngqVriv1dZqtikprwAAuvWTAIDdlXsAAP3yWmyDqA4tCmQ4YYdKzTYkKjEFdTYZ1VzGyI6QIjqA\n8uea6aBUOWBf9f3GKn52zFnOx8ycJP1J4zOnKn2SWR2SXFsg539bGIzFdFU2npx6kx1AtRV2vCSs\n4aPba2dtymVxopb5oofDeB+2d3d4P3lW9L6Mzc3uDXtyflbtlxoNAECjGs1dSw02EdRr3I9ev5dt\nG41ZBVcTnvb/7iN3mz7ymA2l33tbMZnUuCdzyJg7N5KxLqWlrM0Qrto7HA7HC4oDlUiXRVqDkSgR\nxNAsUmJRLMZElawJZd1Up4CRSMUJMG3at2t0P5Fss78NBUKWE/kqhvZOtm20xY6k/qWHeIVQnCr9\n+FVT43u5xl/R8kpM+p40OWXmWCTSfoWpTkV7jYvnm4D4D2Z/jVVayyQ7mtoWKC8R7gerPehymk44\nKHNSZ8gd2m6LzirRcMK01BnS/aTVyV9eSs1zeWsZ+w4XRuL0UccSACQVkbZFoiyK02ZsLk+vmhId\nh3i3VFpU+lixyMcuJvGpGYtEPxx0+e+aOCgNRapU5f1GMl9UalGDK465Xb/P7+5wKBJqmDGeel2G\nzjYMvN9grFKnaM5mDPvDqKHOgkukDofDMScO1kZa4a9IknazdYO9LQBA2mfybE3sI4XGZtYmQL4w\nqVJpYrcpo1qobY4lVMrZUfkLM5T9Uv28WrubfI1Cn/vT330i2zbcepj7eoXpTjRkabVejV/uxhLb\nRCutIwCA4vppc+F3AQBK5XUAQJX4GktGykpTK80sCNJpaTMTMrNrm6YWqeQyOzp5tiSXk2jT/HFm\nSrTZ6WeQ7Sf2o4ld+I8bH1tXRUKU2NxzB1g8GUUl61I1vl+1Br9fZZFSVQMcGcN4TaTMgtzBkaE2\njUb87kazq9rH4/3RIRoOWOrr9tj+Wa3GeaIkknEqVMKmoQ6WKto3PtDu7h7/fX0ra7O8Iu+cdKRU\nihJte8znS4TqVVPSvgQIAEDasSEl01i80XY4HI5DhgOVSBsaytWPX5r+da4U0b16DgBQXmXSeuXY\nS7I2Y7F1pPIVLFfWs23l8jEAQFCbC/EXMIG1afBl9knIvOp1HUfPX1E88v0d9sSHnXPZtlHvAu+X\nSHuhFrSWYtL15voxWRDb6PqZuL/aRBP+wjU0GtV6gsPi2dSwT5/3I9vHIIoZ0uKMI8X/GcmER123\n5STcoNGrco/T6XNk939mEEh2smlkdlNlGOhxotSSjhdvPDUctliJ11Gu8XJZpLV2V6Q3E2Kprvio\naZhQWQleGMn9UHtmYscjHUtbfmf74pEfpTFEMylxEEtQ778REKtVbqeBEZcv8bvc7UTJeG2VJdJq\nmeeAfqGfbRsMRcpt8Dlay/xe11rx/S5W4lwxCy6ROhwOx5zwidThcDjmxIGq9pWSZmmJhl4ac0x6\n99KjAIDRM2woHm8/GXdUA3GNRe3y0Vdkm+rLrDYPy6w2j1GTc0TVXlXBotJc+tsAgHQ3lvEZXuDC\nlMXORQDAshH9N47xeYsn2aQQRK/oGo5ul1gtoCqT7CvNI9m2cUlUhFQoVkL2tyywdIHzws7M1zXB\ngLd5b1NVhSOj/YbHTjSO3ZxFMzoZQtR0R7JVqocb56PGymeZw1Q1zV2Anmzq/IHyWaNC1jQeYDBa\nvPHM1HUybrSRUIoCm8zGfX7oR4Y+1Nd1cl+GxtmUaHC+6OQaI09JvD/FJB+UUxbTQmslOns2jnNQ\ni5pMhr3Yx5IEBBSJf+s1fhfL1ThmSy12TulzOTR0pr7QpZYkaqBS4fM2m1G1L98kF4ZLpA6HwzEn\nDjhEVL0sUdpblo/OqCEUpV026hb7MddnMWXiek0I/Q3ai9tGlwAABXEEhYJQrAzhV7MNjTpMh+he\nZWpT/0p0KFGHnV7lhL9OlUqUmksVNlQnJf5NRaLud+NXsZBIWNsmU50S+SoCQEm+dBqCp1KWSqhA\nTmBaPOybYmk/3Fxqm0zKldttIh2YDdHcjw+vPqbo9NIV5hSTxzbXOMmMMnmEsqVef38C92FGvRYD\nRVp1duQU5KKLInuVao2sTSGTxPmaNT8oABTkfSxIJqZCgY9TKZl8pEIjLFUkDHSZ96FCvIdXr1+S\nJe5bMNpgKkmaSsTHqdVqck4jdXbZwZ1mqoqRVpdZ2q1W+ZqE84+xyataM+/zLLhE6nA4HHPiYPOR\nitS4t/N07MCIpctWU2Z8oTYVi5ZKIjaYNkupncuPZtuSNhP5CxW2gRRK/DUKxShRjkQi7O0KxWmb\ns9mPO1ezNtUif6mqkp27XIlfoGJVcirW2P7ZliQHBfNVa9T4q7a8xoEEweRk1AQNqV6SSqImbG0R\nbaTPqt7XdISoCcEz9s8kLwGqbdKeSuWMhPL758JAY7JR2d8mHdGkNRMZSK2tNslLLrMk0thtHs+x\n0TDaJpflomCpye9QqxHtgc2GUJDk/g0GfF2lQrwfSnIvyDtn7ckq/acxikK22JQ1fK9rIgmXJRtO\nasas0+ZkIyMJIx22o7TYL7GGq0lO1NRbLFmNj5+aoTg2RoYa1ZLrXltdlfNzP9JBHMNx4vQnh8Ph\neEHhE6nD4XDMiQNV7WvE4vHO7qVsXVEsxavrTBeqVzkyqN2OpTp2t64AADq7nG2pvRtjaAtFzsxU\nKUssbZlV8qQcoyLGYiYYdNmMQHLOWjWaD+pCdahIGqdiKVIvClWOtio1ueDXdaFPlRrxHM1VVumb\nkitgZBIuakSWZqDRKKZgojsWUbVXfXvfuKSZiUCVZkP5X0R1XyNiUvm1UWCZnj+Zs9QcpyBZxAqF\niXMh5gGImqPyl0w+UY2MmkG/iraJvNkgHZlibXv7q4KHEbWKOHKMn2wwyOeuUBV9NIgOY80IVSjx\n/kkhvldaRqRQkKxPWv7DPO+9NqvrFSlQl1GcTF7TsbyzYwmK7GxF1ZxEhS8va17UbEvWJkhhu5GY\nXEadaBpoLWnxPTXniRljGMcwNZkVZsElUofD4ZgTBxxrz1+qzc1IVscaS3tVyS9Yk9/Ll6PUiiLH\nv4/Hkhd0J1KjRh2hQhWEZC8UhlLNEGhFIq2IdFKVrC7NVmzTqAv1Qb58SSlSQArigNLqXvoxrRmK\n1JJkjClLreG0ME1tipKp5K+0JPEFFEjTWWWMBYl8ozNBMBeXr1KJOhXilpHEW3eFrjIUg/9obHMt\n8Q5jLSqopZ8N5a0swR/qOEgKViJFbj/dYtsUC5qzFHId05m6RkLk1mxHlMZn5vr1mEF+UZCO+H5e\n34pljDsDvgGVmgSTSBLaXi/my6BEMzrJdFKIz/VA7lFD3rWSvFfBkPa7IpEKewkkbazvT+85jUSb\nMIR8rcKQDDU3sWSLMw5nUWwwFidV2jPPw1idhaL9iPSpuVQBoJDs/4K6ROpwOBxz4mAlUq3Fcvre\nbJ2W6Y02Lf6tLt+VtTlylHN77l3jsNGtCzFX6MWnHgMADCWjVFFsMRVDCk6EnlER6aQu/ahVox10\nNOAv5NIyb1s/cizbVhZq05OX+Eu9LAJqkhg7jZRfrha5LSXx1kaJVP/WDO/WRoqFQ0+kDUuE1+oF\nRaXEZOl64jd7OOL71pHQwk4v2ts6Hb6Pe7u70pYlVGtDVuF2kr5kpUa10+nzQOb8up/ef5VkS6Z8\nr+bPpInnEwB6ki9ze5tt9t09fvZWlmJVhO5ufP4WBbtCMYIpg9yUjGUVIeCrpD8yNsPhQKlREt5t\n7NlDoR3peFRl/1LJ5PJtaaCLaghilzZazFCkzWGXn7mesdFWJGpj1BGNsyp5RcsxaKDX5fZdkUj7\ncXds7fAzt3yMteNiWUOQ4ztcrERtYxZcInU4HI45cbAhouIlLyXRNkmZLU1/+SvUrMcM2Fji5Y58\nuQpGurguWbA1N6QmQLBk3LJkw65L6FhJvo69TrRjXb7MZP0Q2H575FiUSNdESu31WILaucaJTcaj\naCdKJJdhUz5c1lk96SXWD/Z4PG23WyQ8dZHvQ8nmhhQbt5Kb6y2558UogfREEiU1qRXjxavZuV7j\njZndykqkaiOVPJZp1ib2Q6UaTZphQ3An2+tYFXIkc7UJ8t9DQ87ekUqUO9fZVp9qzaCezcO5eFVh\nC+LDGNl7nYn/fD8GEvo6NqwUtVWr1Dgw9k+VMuvig+hXhbRfjlNPkon/qrKpDT32Tati6Htdicok\nSH0Pct6S2kZ7sR9V8ZOMhGkzTCI1oVDm/fuSnzgEraQRn4eOkYBnwSVSh8PhmBM+kTocDsecOODi\nd5oX1BS+miilm2S5CU0pVJHQ26katU3qF0FRjNdKW7HGbI3hLYkO2uuxcfnK5StZmyefYkfWMOVj\nt1aWs21lIfcn4iQZdtjJYEn/1aKUKyhPx2ZPq/ZK34n9TxfQ29STkhDNtWiGWV/n+3Zkk4MXlsQs\nUy7H8Rj0WU1Waoyq6EC8D5n6rRtsPlPZNp4oUTHrHqqKn3PsTRQajPQpS38S56eohOoEA4BlIY63\nGhVpy3pmrbwRr7G3eM6m+hKr3wOjxupt74qzplTSAIVozxkJab8vqnTf5iMVOqA6c7tSqmRkUhGM\nh31pK+Y9/TWBAd0e75CKSaFgqG76Xmke1CDveWLKY9cbkqe4wvt3KJr1RsQmuu1dNtVUhmqfi9do\nr2kWXCJ1OByOOXHA+UjlK28lUv3VjC1Zj+LX5PIFzmT/6Qf/EgDw2CN/lW1LJLdpVehOWgiraqhN\nLcmOvSuUmouXOPvTU089lbXRbeGiGKVNv8+fl3ZCWK6I0+rosVgyenNVJC/5jJaMBKZiVaQ/6d+W\n5L14xdI2Njgb1itfEQsVnjnFFKCGON+UBpV3voXcNkstUqfjlNRoPQ+pZrhPc22t1BkTMylp34j/\nk1mfsmNbTUn6Js6qvtGCNFigJ9IVieQSUkOR6ixegtmSOAoTQwMbibSolC9A6EumTeZkEkdUahxR\nmsVt0FeJlCX7YdE8+6LpJdK2JAXqrINx5zpTqwYS6tkoGzoSac+4T30JnCnV45gHLcwnjs2RcTZ1\nRQIfXOe+qQZKBaNFDffP5uUSqcPhcMyJA5VIq5ntIn65k4kSrr0ef3keffShrM2nH/okAODyBZYM\nS8Z4Uq/x16MmCQeUbF+tRzKuSiolkVZX1jjn6WAUv1g9Sa6wKUT8u++JUtbqEtv9nj7PkrF+XS0l\nRgMLqnJHKzmBNG+nU0J+MPkrxwsokd59+gQA4OTRWB67KVUMyhpaqZqFlQKVdiS2TRgbaUw/L+Rs\nytfz4WNOLBTz1Dl7mHTiOLkOZIfRZ9DKFfm8mdVitIcvic1daT/9rLy4Of+a4YQtCDpyHVUTWlkS\nLXIsxHp9TzWJCAD0RaLT2k22dlVBpMOOJCEiCTFdXokUyLpojF2RejW8eGApZxJy2+lyP3q1aMcd\nysuWiCRbFg1haBLH7EmX2hIrurUXbd5pia+pKElSBmKPtUE1uQdwBlwidTgcjjlxwBIpz9tFU4Fw\nLPaRq1c5ScnDIn0+Yuygu9tMuieRXBoNY/8UCTSTRJV0bzzqJDa4itRd0TR27U78Yh07fhIAcP8r\nHgAAvPzl92fblpc4xd66pMrb3d0BADSXomd/SbKKt0RaKRgbkso/agcNIf83kPfgLwqefoJDdUMv\nJrlo1fn6jx7l6q6rQszXOjoAMMo0gWkvu9rFelnY6LRHXe9tTK0nUmdiJSG1W2qqP0syzyePmZXq\nL7PPZZkzptuoRNpusxZVqcRnrlbdv8bPYURHQl17IUprtWo+mZBWAQ0mRFQDGYqSgnI0ivbhkYT4\n1qqSPEYiHGybYZLmtqlk2zeEepVyNaimYAJuVBLVpEI9kSiHiFJrS0JcUznOcDTDhq3PngTaWFtv\nxTy/s+ASqcPhcMwJn0gdDodjThysai8eiOEwxqhfu8Lx2o8+ws6lT/zlnwEAtq/HnKNVCcButVjF\n1ixSANAUlbos4n0xy0MZDeYNUcGVWqNxwlevxkz7J06wav/y+14OALjvJS/Ntmmc8NoqO1Xae2z4\nHpsA7rL0sVGfzhKTqZJBnU5K37HOpsUj5O/tcGDCBTOe19XwL2paSfIcFIzhvi+pdyaJ9QAwEqff\n7h6ry0G2WVOJOkAmaUyWRqXnVYeebZqR/ifykQZDuQsh71yxZ1KTgpootJpDo26KxrV4/xN3YWGQ\nWTGs41PuX0J8P6tiHhuMotpcrgmdTbMmGRNLXZJP1BpiGpB3ZjiMDmMtblkUsn9bCtzZlGga2KGV\n7fbakVCvpoCO5CHVAndLxaiOq4O7QGP524y1zAvFopoveP3I9HE48HykDofD8YLiQCXSkpBwnxQa\nEQD85cf+AgDw8EPsXFLH0pLJXt+UcqkNLRcrkikQpdOhEqbFcaBfTgBYWVnNtVVCtQ0jXRKH0pEj\nm3IOk6FKvrDrQkBfXePjRZIyMJKYt+yYMwjo6UQmozQnJS0egfuVD7BjrphYpxl/8atSoUAJ1Du7\n0YGh5PaxSHQ2JLEtoZhaflfvY9EQuLMMQKIFaHavctncdM1onmXDt84iyv1kIbupzd6k4adDexhe\nVklaqDQjyeLV6RkpC4uXIb8oWoPN5VvKImQ0GxZvG9swUrmRlXo+3y8Q8/uWZMy6klk/J1GKw7kg\nWqRqelaLqTf5OLrKZujX0sptCQwoSDjnsnFIUaK1wHiMRrkwVs1Qlq8Z1Tca53Cw//vpEqnD4XDM\niQOVSD/54McBAB/+8B9l657WZCHyhVteZqlzqWntTZx1XiXSmiHbqyS5sc4JI1bX1nLrAaAudtSh\nhHldlDyaNgRNJUmVcpTKYaHVIpUuo5QQAEhTlUSF4mT3U0lUJFu1EwVbsyldvG/amtxrTUICxKqR\nPfmC97ekphb2sjZqewoTIZ5ArM00knBL5XbnKq6KlKv5TItiMw0mpE/tXppP1JhYc9ny7fmDyUsh\neTgwnpBe+aDaKQkWkH6M0iiR7rS7WDQU5GbVapHGpcpGrKXFfw+MhKa25YIQ20smQa0eq5BoBV2x\nkRpCP4Yaes3jl+WFTXODJsfmm103Y90bi81dnp2CvJfFenw/JcVoFkTTMyG/lbIkNJGiUWofH40M\nZe4mGuPivb0Oh8NxyOATqcPhcMyJA1XtP/ShPwQAXL50MVuXSuRDU9T1JYm7XTGquUYQtcRptLK6\nlm1bkm3Ly8vyN+9fN+q/dSoBwMoKmwpOnDgx1cdYIjmK9UqrmZTubbRNpDjlf4GoZmYlZSfKN/DZ\nFo/+dOEZzqKl9BMAGIoDKWQUL15vMzMNs/bTheWmI5LknplyuEVRHZtDPbZmmDJOEhlzjfUfWzUt\nKzEiMdmiElqH2JaUsOn1dV1U28fyzGoezgAtR2yucRQdkYsCpXP1uvE+kJT/yKh+Q77X9h3sCV2p\nq3l+r0a1vVzS0kGSDUzemVYzOoM3m/yuF8XZ197WAojRPNK9LlneZL+GMS2U5T0qyZyxeTe/1wXz\n3hekzFEZEqlVj+fX97Pb00xX6gy1L/z+uRNcInU4HI45caAS6XmJza6YXJ0tcSCpJLok1KZlE8e+\nIk6NtU2mJm1sHsm21ev5vJcqiVgpR6VDXXfkCO//ile8Imuzs8Px8xUpu2rDr6PkmY+3ziU0mpC8\n8hnydUmyi8/I9BSmVx16nHv8EQDAyORqjEXR9II0Lj7erIEQ8rWF1RhsIAUApCKlkJH+SzJGkioT\nfc0IZMo6D4aSU1KcJCGnTUgWsgIfp9fnc1y8EoNAnn6GnaDbu+qYjFIaxFGiWbxqVS1VbHJIpPvn\nrzyMGEgW/CSY7Gwa+6A+tzDtfCsoNU2cRn2TtakvNKUkC4IQQrwpeNiq8DtcVvqZtCkVcx4+Pu3W\nNv/ZtoXt+L43JMO/VjDojKIW0R9KEI2MS70ex0q6jVSobjF3gHFodT1DvsPhcLygOFCJtCE0pGYz\n2i9VIl0WSXRlie2Xy8ZGunmMMwlpiWSlOAFR8htPpE+aldFHpR3NTKSZ8wGgLQRhtbVaifZGxwxh\n2q6pbey2TDbTbVnOTZMlHItHyE/HkvUpl31e81UqhWX6WguS7SezKaZRkhwNhLYkFCO1HWuJXCCG\n620NWIK8cuWKtIkSCCVDOR4fu1GLz1OrxVS5UoWfte1tljYuXjI1vJ5+Wn6ZslcqR/rW0hJLKpvr\n/BxWK6wpJabkdHEBRZRBT/OJmusoT9DQxFYcbcdAQyTASp3fnaEh65eVVqjSnYxjydi8lyTrfkMp\na3X5bZj8rvJ6XH+Gy6Z36nH/RDJLlRp87IFIn8VSDNceqrSZqE/GVGUQIv71Hb6mmJ3MvJO0vw9j\nAYfb4XA4DhcOVCI9ITk/bdKRRlOlVMnnKVLiiePHszYbGxu5tkkyPf8Xi7d+KWqTs6R9Pe8siXKW\n5HkzEKyNVOymWYpLlUyN1LqANtJWfdpbngmZU8wEI13I71jC9FJDZFcpQEP51OY6tvlERVDoi7e4\nLXk01c4NADu77HXf22OpudFYzbatr58GACwvnQEAXLsmiVJ2o6d9Z0cyu8vv+mb08i41y7K/MAOI\npdVeL0owaVg8DWOstZcK5mFG/j4tAAAgAElEQVTUEF2RTLNYBKOFNFt8bypSHqKzF/PTZlWBNYxU\niPQ1c46msC5aab4ekx6Xj81S41Xxr1wxVV33RMPpyRTQHvB4NIyNU/tdlHeuaIIGGnIerUKqCVUK\nZpqxwTez4BKpw+FwzAmfSB0Oh2NOHKhqr2UnrLOoOaHaa5v19VhQTSlOsxxAiltxBE223bfExAxM\nbpu1f9YWYbqdkvVvuNeCQQjLA0Mz0dISiQSgJ6Tqf/xmp5I1aSjF1sZjm0lI6DHSvt8XtW0YVclh\nNg78PLTbPfmNKmW7zWr+5SvsnLhwMeaefeYStz9+jFVI4d5jYDj0WmYiCfzsNQ2Be3WF1byVZVX3\nJhxsmCyktxjQ59vmTtB8CElR1G3JubqyGh21qoKrKtxtx2u/vsU5a3tChF8SqpMtl46BZmSS8RPn\nVW055tvYFPPesePscH5yJ47nuWtMUevsMTUqlWeua/KJavawgjgEiyYzVK3K65aFPjUciTkiic9c\nrR7zD8zC4o22w+FwHDIcqES6KYR6K5Gqk0d/lX5ki6VNOpf2kwRvpc2tSKtzI1d9eOJ8Gbl5atVC\nYSAk5SvXonRw+epVAJGkXqnwONoS3Hu77AwYDJimkpjy2tU6Swery/yMdDq8bWsn0o92pWT36irT\nl/Tm1Wo21JSlma3r3J+rW7GPW1tSbhcs5Wxfl5LDw/g6lKVssGYDazUilWalxctLLW6fZffKhREu\n4ogybPCEZuoaj6XMthTDK1Wi80WdMx0h4u9uR0fQtYtMUSuN+B3uCNVoYPYfyeKGvPKVCt/HrmHV\n7Ym03yjx/qvGUbzd5+ehLefvSzan7Z0YYloVKlRFgiZS805ez5yMUt1BJNqCeUGbTZdIHQ6H4wXF\ngUqkd93FBWyqRtpUCpLaSDVEcxaeL2lxXmn1VuhQuVNknHSxzWrWdrrRDosBEqPYpStXs3Uf+OCH\nAABHjjJ9bW2VpT7Lbf7kg1wNYSRlbzc2o3TxspefBQCcufseAICY1LDXjXa7nWcuAADuuovtZWqv\nKxajRKi5JPc6bHez1Ka9Xanx02H76c4uS1T9ThxXLQu1tqbBA5txW0ny4lbEhqd0ttSGJS/eeGZX\nb0s3JNnDCyAm8lC7NAB0pKz5UMogt22yEclePxiqf0DCgw01qdLkaagptKeRvAvDJN7PnoZXZ4lE\nogzYrLD2syIk/S3JcJ+OomQ8GGjtL62mYGqIZUlo+FdfxXoj9rHRcPqTw+FwvKDwidThcDjmxIGq\n9i+VEsfW+ZIUNGLixnP68+YAOmSw0U+LeI3Xdjkf6YhiIbNyM5F1Em3UZbV/OIgOpc6IqUktobks\nbURDfnVJyu0OeL+9XVblLl29lLV58jzHwTdbqnpx7gRLyVFq1Lqs6xyJDs5rRe5bgVjtT6Svo7Ep\nqCb9XRqzGapcjnS8ijgukiD9nqXGL95wRuRKsfCFjKS8y+4236PeYDjVRtX+saEdlatimwn5ciDj\nJNp6SOhHI4lOHMic0Df3sKslsCXTWCG1eRXEIVjjsR4JZe56EulwXclspQFnweTZgGQBG6ZshihV\nNKtXVOdrtRubHAGXSB0Oh2NuHKhEmmU/t6G8NyDH27+fb2ntucTOzwe9Rjn/rEwyC8jSv7zFkuEw\njY6H1Q3+0g/7WgSPqSl2DDePcdDF2hq33TgSnU1lyWS+12Wp9doOH/uKkUgvPsME7KVlfnyXl4Us\nbihKSYG3rS7zscPJKDO0mixVqQOkWNKcpZEitS0s/XFgOt7YXONwpBmQxIEh8kjBSlkLmM1LYYMn\n9FUZSKnp8TUez0IlOoI0q1pB7rneewBYX+P7N9D4+z7fx4rZv77ONLYgOUr74jRsm5IUqWSUKmve\nilGUiBO5/+UyO53WJI5+z8Tj969xZq9ul89PZaMFtVjrKAg1qipk/WYjtilX9p8qXSJ1OByOOXGg\nEmmWXHsfCfO5brs1StKth5HO2m/f9vuEgWYS+MTu82aYut3Y2eHwv7Gh/axJPawLF1iC1CxBS8Ym\ndULyyjbki1+u2PBR/u0LlaYrUkWvF+2wWg+pJ9LFUCTD8dCEaIowo6GIm5vR3rW0zNKVEslX1qqy\nT8we1ZdyvUlRztWPAQHbu9xOqyFURIIqmZDChBZPIs2eQdt3fXhllWaIGo9NzSShFFUkQ1K1boIX\n1vh5aEv+0UFbSjcbibS4xHbUoG0kw9POONpB2yLRVkSSLJl3uZxw+7rkJV2WGlBHzHWkMka7IhFT\nKT4PKlE3qrxftcT7FU2G/ptl83KJ1OFwOObEgUqks+yhN5Iyn62NdD+p8dnYWGf18UbJStIZEmWW\n/d7agdVTiGlp9bn08bBgdYkly14v2qu6Y5bkKol6tPlqq4UYhBGELN/dZbvXsDdtb1O+dq3M0srZ\n03dlbVYaEla8JEEcJbaN7W3bmk0i0YptbTiyOU/FNipSCkkfj2xGqblSuUv6obJG3H97mxkFHUmM\nUpIBLpgM/bDLCwK995ZAEysU5H8T87yqRF6rSfb7atzW6bEk2ZawXr0vYxM80ZUQ4UKNx7ErHWj3\nol06yHiurkioajFKvVpQVJOt9IWFEUyO4qIE+pREQyKTD7Uktu2i2IaVTVMqmKABk7RmFlwidTgc\njjnhE6nD4XDMidtCf3quavutICvrcQv73KqDZ1Ktyfpl/k5keTzjtIE0O9BE8TtL70gXzzlxcpNL\ndgxNOWZdPrK2nmtrcygUJEafxLmQmFLLGpiRFb8Th85wLarKg6NcTltpOpozc2QoMSQmhkLgdSls\nyWiJrdbqv6K+lgvxHKtK9pexGw+is2lvu5c7f6I5O20RwAWkPyUyLhVD9UkK+eKSWvK6aHIDaxmO\nppRgSYzzcCDlO4rliZLJy1FVTsvcfijn19j3fj+Oh5aaGUue22ol7l8gLajHY9wQR1jJZAMbBi0V\nrcEX5thSfHFdciG3pBRSpWqe2dKNcyEDLpE6HA7H3KBFpN04HA7HYYJLpA6HwzEnfCJ1OByOOeET\nqcPhcMwJn0gdDodjTvhE6nA4HHPCJ1KHw+GYEz6ROhwOx5zwidThcDjmhE+kDofDMSd8InU4HI45\n4ROpw+FwzAmfSB0Oh2NO+ETqcDgcc8InUofD4ZgTPpE6HA7HnPCJ1OFwOOaET6QOh8MxJ3widTgc\njjnhE6nD4XDMCZ9IHQ6HY074ROpwOBxzwidSh8PhmBM+kTocDsec8InU4XA45oRPpA6HwzEnfCJ1\nOByOOeETqcPhcMwJn0gdDodjTrzoJlIiehkRfZSIdono9be7P45bAxH9YyL6vTn2fx0Rvf/57JNj\nfhDRY0T0Rbe7Hy80ire7Ay8AvhvAH4QQXnW7O+K4dYQQfhHAL97ufjgczwUvOokUwBkAn5i1gYgK\nB9wXx/MAInoxfvAdt4hFGP8X1URKRL8P4AsAvJWI9ojol4jop4jot4ioDeALiGiZiN5ORJeJ6HEi\n+n4iSmT/AhG9hYiuENE5Ivp2IgqLMJCLAiL6XiL6tJheHiSir5T1OdVc7vs/I6KHATxs1r2eiB6V\nMfoxHbsZ5/kJIjpPRDtE9BEieo3Z9iYiepc8B7tE9Aki+jyz/QQR/Yo8I+fcRDQ3XkVEHyOibSJ6\nJxFVAYCIvoWIHiGia0T060R0QneYHH9i/DgRXZIx/UsiekDaVojo/yKiJ4joIhH9NBHVDvQKQwgv\nqn8A3gfgm2X55wFsA/hb4I9GFcDbAbwHQAvAWQAPAfgmaf9tAB4EcBeAVQDvBRAAFG/3db1Y/gH4\nagAnZDy+FkAbwHEArwPwftMuAPgvANYA1My6P5B1p2XsdKwn9/86AOtg89U/B/AMgKpsexOAHoAv\nA1AA8CMA/ki2JQA+AuAHAJQB3APgUQBffLvv3SL+A/AYgD+RMV8D8El5z74QwBUAnwOgAuDfAvhv\nNxp/AF8s47ICgAC8AsBxafvjAH5d2rYA/AaAHznQ67zdN/oFGLjJifTtZlsBwADA/WbdtwJ4nyz/\nPoBvNdu+yCfSF3y8PgrgK24wkX7hRNsA4EvM3/8UwH+V5dz+M86zBeCzZPlNAN5rtt0PoCvLrwbw\nxMS+3wfg5273vVrEfzKRfp35+0cB/DSAfw/gR836JoAhgLOzxl8m3ocA/HUAiVlP4I/xvWbd3wBw\n7iCv805QWc+b5Q0AJQCPm3WPAzgpyycm2ttlx/MAIvp6AG8EawMAv0AbAMYzms+6/3bd4+Axm3We\n7wTwTbI9AFiS8yieMcsdAFUx4ZwBcIKIrpvtBQB/OPuKHLeAyXt9Aqwt/JmuDCHsEdFV8Lv4mKw+\nb7b/PhG9FcD/DeAMEf0qgO8Ea5l1AB8hIm1O4DE7MLyobKQ3QDDLV8BfvTNm3WkAT8nyBbBarzj1\nwnbtzgIRnQHwNgDfDmA9hLAC4OPgB38Wwox1dkxOA3h6xnleA2ZvfA2AVTnP9j7nsTgPlmZWzL9W\nCOHLbmFfx63jaZj3kIga4Mn1KdMmN/4hhJ8MIXwuWIO4D8B3gd/pLoDPMOO1HEJovtAXYHEnTKQZ\nQghjAO8C8GYiasmL/UYA75Am7wLwBiI6SUQrAL7nNnX1xYoG+OW4DABE9I0AHniWx/guIlololMA\n3gDgnTPatACM5DxFIvoBsER6K/gTALtE9D1EVBMH5ANE9PnPsp+O/fEfAHwjEb2KiCoA/jWAPw4h\nPDarMRF9PhG9mohKYFW+ByANIaTgj/OPE9ERaXuSiL74QK5CcEdNpILvAA/EowDeD+CXAPysbHsb\ngN8D8DEAfw7gt8Av5Cy10/EsEUJ4EMBbAHwIwEUArwTwgWd5mPeAnQ4fBfCbYFvbJH4XwO+AbWqP\ng1+6WzLTyMf2tQBeBeAcWOL5GQDLz7Kfjn0QQngvgH8B4FfAmuC9AP7hPrssgd/PLfCYXgXwY7Lt\newA8AuCPiGgH7CR+2QvT89kgMc46ZoCIvhTAT4cQzty0seMFBxEFAC8NITxyu/vicFjciRLpDSGq\n3JcRUZGITgL4QQC/drv75XA4Djd8Is2DAPxLsPrw52DO2w/c1h45HI5DD1ftHQ6HY064ROpwOBxz\n4kAJ+d/1/d8SAKDWbGTrNlorAIDhdgcAkKYp/ybRUd5aZYdprcbhs91ON9s2HvQBAAUK8svrDTkX\nSYEvs1qtAgDK5RIAoD8YZG16fT5OOhzxuUqVuL/89lM+x/buLgBgfX09a7O5ucnbdnjblSvXsm0D\nOU+xKLc74b4NQ5q1Gcl1f+93/eitcB0PBf7V//NrWWjJJMIkBdRoPtm2W1CGst3M/qnurmMN/Z1x\nQHkuyGpeE81S2Wb7HEL+qqzmFmnflOsbzbgTP/TGf7Qw41mrfxbHyCZRvioUeFmf3ZDys1xI4rvT\nqPI7s7nO71di3t3+mLeNC/zOpSnz5IeDctam2+V3bZjyulRyCwWK78d4wPNDUe55sRSnrlKV9y/I\ne63ZF8i8X4nuVyjLdZXNNp0feH4pybELhcjpT2T5fb/9b2eO54FOpKUyd75oBmrU4wmsu7MNACiX\n+aZU6tWsTX+XJ84w4AEajUbZtnQ8BAAkOoPqwFfiRFiu8LF0ItYJtTeIk11RJ86EB6Pb7cV+y0Tc\n7nE/9tptAECzGTm/4/FYTs83PJs0AQyH3Eed3Cty/qIZ6MEwXtNiI/+cpdkEZNZn8+j0xEdTv9wm\nnWoJM7mSPSyvmZhkc0vZ/CcTIE2/Gzop6qRtm0y1zw4YVyULM31GJAk/s0nRTKRFeZ5L/Dsa8nvR\nqMULvPsEC0P3v4xjWfbae9m2xy9cBgC0R7z/WCbSQhLfjwCZ1Ib87gVN0paYj6dMzkW5sfb9SiYE\nlJHMCcN+J2szlgm9WmEhrlqOwlyplOTa6HCqUAcASTrzCYzb993qcDgcjpvCJ1KHw+GYEweq2pdF\n3SZrf8pEZv7lwBIgpFGs39llVSEVVbhRj6kGWw1erqg6IupaUWwyAFAStT0RtV3tX/o3ABQrcivk\nvJ12VO3VbKAq3fraGoCozgPArthNZ+WOLpX4PGpzqYmpYWj2T0eLGzyVU6lnrJu9wtgUc+bLkFs1\nU0OeXJmp6DfraX5/yuSICaOrQTLdxexE+59u8XR7Kuh7YfwLam9UWzNYba5VYpsTR9nEdf99nAbh\nytWY72V7j81hgx1WmwvyniaJNX3xb2EsY6/nLMZ3KZUcJCV5hwpGtddBCpmJReeSqI4HmWe014mx\nfxaL+XlhPJ7ePx0NsR9cInU4HI45caASaVuM0EXz5a/XWwCAVMLZx9mvkQHE2QNxMtlvfU2k3IoY\nw9Wj3+tEiRIkHkP54m1t8bae8dpnHj+RbgrF6NUbDPi8zSb3dWNjXY4TnVV63kS+bupgAqKUVZCv\nqm6znuDCAkowk15zwEhuYXrbdCP9c1pczZz1mfg47XXPJU2bQHQITXdEj6ltMg2JpvtBE/vMwgxB\ndt/2hxXqoS8UilPrEnH8qCBYLFjnC78fPWHQWDZKQbSxSlWcVbJpZDSwoeyXjuQdLNXknNFhrHNA\nEBE5mMIIJOtUkFZtNBjPPI0nxtpovCqtpnLMRLTjomEN1JL9NUaXSB0Oh2NOHKhE2hWOZcWUQBrI\ncl+kzUS4XCohAsCaSIIj4XoWjeQwlq/JYChSn3DAcrRBsbsOpc3O9g7vY+wexYrYXqQ/ZfM16/dZ\ncl1dYVtQrcRftY6hZ/RG3KYv9pWxkUiKhTwlSik9wVAqZghDCwClHdnO5y9k5mXl6Zcz20xG3OXs\nnzc0xO5zslnrJoyquTGYZDjZvk0cLeiO5ngLKZGK/dL6F/TZJWItrlhUO2q8C50ua2NPXLgEAOgN\nDf9T6IClimiFot2laT9rM+jxvDAO7DtQiVhKO/Gy8lCV82sVVlkuF/Udptx6ABipjVXmh5KheOly\nQfpYKfGOrWKcH9aLLpE6HA7HC4oDlUhbYj8sGxuMeu31i1MSQr61MWrU0EiiIUbDaNtUIaAj0mpB\nDCWVivHaF+UrJjYQJcQnw/gdKSZ5r14w0mpJbDCdPfZGltdZam7EU2A75S/tSL6CFRMQUNIgATmd\n2kOHxk4zMPbaxYHYq4xNbFJYzLzv1kaZOclniaSzxcz9UkI8W+/5ZATSVKQSpiViuy1e4z4i9ezi\npocaidpIjbSWiJ2w32f/Rr0iXvtq1NjqtToAQJSxSKhHZOq0+yzRZgEa5n4qIT5JRdqV3QtG6i3I\nMVP1qBvGC1Ih65fUoy/PZRL7Ear8si4vs3a7urqWbVteYZ9HqcLX1BT77wqi1LyaxOVZWLzRdjgc\njkMGn0gdDodjTtyWKqIlQ6ZNRURXGlNT1ISrhlp09TIbsVdWuOwOGcLw3h6rHEqIb9Sq8ruatVHj\ns6oXSrFSVQYASqI6ZI4gQ3cYDjm2vt8TFURMEwWT2ATEcb2J0qdMdZLxgM8bgsTTC7UqDdY5sYDe\nJmVOw16H/uaV+7yqvM+1TtgEZlGLaFKlnp015cZNJnhTk32d1WbWOQqivtdFbaxU4nO9117cAAsE\nEygiz+xwxM93gV9PmJwhCBLb3t5jJ265FnNQtCRnxvXttjSeQXWTe5sUNSGJxtUbc5c4ihMZI0pM\nHLyYAI6ucXKjlTV+96uN6KwaC0WrLOp/s1nPtq2tcXHZ7S023YXtLQBAr72dtdlL21P9tnCJ1OFw\nOObEgUqkYyHhUjV+3tUwnAoJttPmrxoFm+FJvkzyNUuMoVrDNDUjVJpqOj1j8C6xBLgnEunOHku7\nlBoxo1iV/dmoPBpHQv/Vqxd53TBPW6o1Y2FKJfOqkEtp7L8uF+XLmQpVqjuIX9XeYPEkmHQqV5Ml\n5Ies1TREEgz7bJvwWgXDTcqcRXTjcNIpalUy3cdJITMvtdIN2yqRv9lgjeT0EXZglMZxzB9eQOdh\nkmllEapFhVSDYfgBtynqxkN+Z/oddSRF+Swk/F6p87YktMK6ye6W3TbZL2TPTMwiVZbw1WUJCa9X\nTZpLmcXWjvC6xopkV6vFDE8dJf3L+01JdCZXq9zvSx2eF9qXucJ3aetq1qYPDxF1OByOFxQHKpEm\nM5I0qlQShMKws8W2TmtJKcl+bQk365vcnb2e2h/z0kmpFr94hTJf5nBLvkYi4e5sbWVtLu9KYmmx\n91Sq8dZonsIrl9mG0h1yfzaO3ZW1abXYLqRfpqJJuFAVSlezLlQQCScd9k0Y6wImsEz1avctV6N3\nxFKk8pJsmGGbnBJWbYBFZrYMud9ZcsFMZtKktDkj20mUiKel7mqZx/aIaFabV1hj2b0UbWr92rEZ\nZz7cqIhWliC+X+OR2vwlP+tQNMBBlNDUz6E5O6+b96rdl2CYEY/N0iprcZVaK2uzucHH7ku+3+0d\nlgStjXJ9ifv2snuOAABOHNvIthWEPnVdcghf3rkCAHjiwoWszaWr23I9fA4NrgEAkvf7mae4Ynfn\nmWcAANW9KBEXjU11FlwidTgcjjnhE6nD4XDMiQNV7TXax5YJ0HAIkgiKltRzyqX5FyXs6aefBJCn\nHS0vr8j+oq6Jirxnop8GoiJcvshG5D0xInd2o+qgeReHEif8zIWYU7Eq+UNXJQ/pybP3AADaw6ju\njcb5nJgD4zzq9FkN2m1L6QPpY2rURZsfcVGQzlTbJ6Hqt40aotzv/okGZirnAGLUDc1omk7tN4tq\nNhlQPyOgX37s6KyJ4+XIRS6jsfdXjwAAtptR3eycOjmj34cbpcBqr83sVBAL2bKo4tUyP9eNegzr\nayzxO3j6nrMAgKvbUSV+9DF+ZwdizmrU2Nm0th4dtUMxHzz9FDt7agU2wW0eje/5S88eBQCcPcnn\nWl+NqrZmajv3DPft4jV+z3a3DWVJrBUnjhwHALzk7hPZplfc9xLu/wbTpgZCvUx2drI2lZuwE10i\ndTgcjjlxoBLpQKTPookxL2mWJ/kIqrTaMIXllNqkmZWKpfg1rDfky6Tx8H3+8l29dilrkw55XXuX\nvzSDHkuijZqJKZa43J0hf01N8imUy5JNhrgf3V2WaIcwlQira9KW1w2MRJxRtESu0ez/fZNPoLCQ\nEqlmBorrMllvJrVJMeFsyrHu0/w2zR1qHFqjHksaqYxHWWgu+YxLk1Kn9VZlad9zvconf6Jck1aI\n47l+nbWVRJwj9LmfCQDolWP89mgvPhuLgntOsdRZqxrqoDz7ZclXURAifDB0oIE4aPsiWQ7HZqyE\nYtgX59SukN0TQ/ofCi1y1OX7utbkfU5JUT0AOHWc+7a+wi9ms27e3azyBN/zlswdRzZiJY2EeNux\nTZaETx6JFYBXmrytBt6vB8nqP46S9bizP53NJVKHw+GYEwcqkZJKkkbcC5rvUDJYp5n90GRmqvCX\npdaUTNy9SBvqSDnnolCMGrK/5gwFgOGYbSajAUsy7Q5LpCMT51YRu6t2cXV1mu5Akg1m69LjvE/r\nSNx/k5c1cU6pbDLPyPUqab9UFpKyrdU+XjxCflBy9iz740RkZcjZUce6Un5vbIBSelx3N0oHn/qL\nDwMAykIne+BzXg0g2rn5JNq3GchqLqlkGntvGgEAWkKdO9WK0s3SJkvAz3RYyroUWKLpBkMyv+UC\nUocHr3wpS2n1RpSmy/IcJzIOWgHi0vXoQ/jkI2wj7j7K70XX+Ad2pVJFr8sS3cWnngIAXBd/BQAE\nCVRZbvGxj61ztrd7TkUJv9GQsGySmm6mJDsJ6b8m88TmuuQNrsfrKAprvy423gqZfKiqYcq80Ntm\nOtve5adim64T8h0Oh+MFxYFKpJpHdGiShYTMviEVRmX90HxxSvJ1L5eEMGyqf2bSkNgdhz0+x/bl\nGN7VFoLu9pZIomKbJGOngYTCVSVXacnUbFLhYtiX2i4z9q+U1V7IjRsmPI3E/qoRoeplLJh8iaXS\nbckfMxdSmrZfxhHUcMNp0v2k9dRKtFl78eSPJdTyqcfOZW3OP/YoAOAlD9zP+2iM4H5CoJUQs+XJ\nwIAIVShOrbFmcvJE9DI/fpW9uR9/mPt0VUIjm0dOZW2S8v4E7sOIZo99CL2dTrZuWxLu7HV43dqx\n0wCAHZMF/4KQ3XefZhaDZawEeeiLYgctNfndXV2P78fR42wLPXqE79mxTZYo60arLAV+5wY9kZDN\neDYlmdExseMOxqx5Xr50JWvzzFNMzq8nfD0bS5ER0DvCbIsw4m39Ld6vvx3nkEYz2mtnwSVSh8Ph\nmBM+kTocDsecOFB98spFjmG1+UgrksVlRYj1WrbAEvKVPjQOquJHtbsi3qGCKIyDlFWAa5cj/Wnr\nGhuPE7CaWJO8kRvry1mbUjlfNG9vbzfbRuK4aG8zLaJSY1Vi2I8q0KDHzpBmKx5T0RXnmDrNVIns\nCn0GADqd/ehChxRqVTFKcRBqkWbf0kRABfPNTjInlVBKxtbpxr8a4NCTgonnPvXJuL84FNfWmaSt\nTKswo2Rz3MmYDzSefyIPqWWgHTvO5Oy7anzw0YPx/H0x8dx3HzsYxzV20nzqiegEDTOzXh1uXHni\nMQDAyMbay7DpELV32ayx14/X15UY97Y4ZKxqn8iOTbnpVXlfV4zz7ugGq/mNKh9TaYYINuaf99+R\nvMO0Z8azyu9cReYOzQtbNiVTChK8UReK1GormmrWJR9pRaiLdFxyaBgK49JqDLaYBZdIHQ6HY04c\nqERalezUNguUllDVNUq+Hxs6kDpwCiXNLB+pCHvyhRz32cC8e50N5qOBCe8qS/5Q+RotNdmh1DQZ\ntDMJqK9fofiNUQk4Fft0ImJWZzee48LTTwAAVuTLVTNZwrt9vqZKXWkZEgY7jl/1dnv/DNyHEwXz\nP+P0cZYKjjb4HpXlq140OSqVQF9Z5rZJPRr+94T4/Nij7Bz42FXOyDM2BHCVgNMhS0KbayzdJJb/\nNJnYnqaDL4qSNags2sjqSsxIdM8ZHsfCJdZmfvt3PpVt+60/+igA4N6X3gcA+DuveQ0AoE5x/y4t\nYICFiPZFUziyWpOxERibBzsAACAASURBVHphT57ZbjvS0UjI9qT5hnNlxiWcV4tSilO1YcZ8uSW5\ngMe78svHS03gjmqFO/J89E3G+qTB+62sSoE8Wd9qRYfWPfdwWPexFX7mThyN1KrNoxwuWpey7xpU\nUzIZ3GozNE0Ll0gdDodjThyoRLqxwbYktUUAQEln/7JmqJ9OKKFhoyQ1XcbjaLvY22OqQuc6Uy86\nuxyCNhpF+2NTvngbqywRigCSkwKV418SGpaGlAHRPtMUibLTZTKvZvMHgP7TEgInOQzPnr0vXqOQ\n/Yf65SaxmZpQV2sTXhzwWL3kbLQ33bMqpafFllVMRBo3+V1JwvsKUs96ZJhJj5xj2/bv/uc/BQBs\n74qt24SclsSYee8Zfp7+7pd8DgAgNXQ0MqEA8X/ZJpKr2kpVkiVDR+v1eKz++BxLxh8Qag8AXJeA\ngmcefRgA8LjQf/qf/ZqszfDIaSwaXv7qzwUAFCrGtigS5Egk7IcfYtL9+GKkFtUkiUtbKj+kwexf\n1Gdcx0GI/UZgr0rNq243T0ML1q5eYulyLDu2TSrfi1tC0epL6K4Q9FfXo13zlfe/EgBw/Cg/M1qW\nGQBqDX5+y1U+hwb32LDtcJPy2i6ROhwOx5w4UIm0LrbBqsleX5SKnEX5cqkndTQyVTjV9iJOvLHJ\nzp1kFhH2oI9TqWRovqqNplymhKL1BlIZ0dgok4J8OeVzaO0zutgUe9FwVysixv2Hkhhld08ycRvp\nSEn+fTlvX+rHDI0deBHl0c+4l+1G952K5HMl0HfGEtggdvFxZfpRK47E3taJ7IfHn2Bb80MP/SUA\nIB3KNpM0pFpjm+hL7mEC/J6kQ0wScw7S/vCzMjZBA0kmDkmAhdz84SiOwtXr/Bw9LP3ZNfWYWkeY\nLUB1lmQebx2TvprnaTeyPhYFuwPWplaXYwXehpDdtb5YRWybLeNf2GiJrVu0y7a5j+rA16KflRq/\nZw2zf13qMKVptLsCkQECAK01vuefeYLtmaFkCPKisV67Klppm5+Vej3aNc+IjXRdtGLVPIGoiWQa\niUiftqTbvsEecInU4XA45oZPpA6HwzEnDlS1LwshvWji2NXQH7Pmsw4wNrH2Y6FDKK1i0I6qoOYa\nTRIW5xstFs8r1ahuar7CbSlwNxYVrGBi9mtVpTzwuuFgNLW/phdS+k3DFMTScq9qGximJo5fs6zL\ncYaac2AcM9CEKQb54UezwEb+Uddk+hcSdGNJ1TXJNWnMGGMZx8fPs3PgoYefybY9+SQ7MZotPs7e\nnhDiB/H+DEXNfvJpbnv2pS8DAFy/Hh1CAykRPBY1czCM59cAiZ3rbBK4epUpc9e2Ykajdof37wrx\nfP3I2Xjd65xlvbnCzoxiRZ4Diq/TzZwThxHqZNKxA4CmUNQg6nJTiPRra9HBWBGa0IY4pK7sRkfv\n5S02gw2lSsTyMu+3uh5V86a8R+mYj62F9Yx1Dc0lVsmPn/0M7uNqLDw5lOfpwgXO1rS3w+esliP9\naXWDx0opTmQcSVO5H4JWdYi42du5eKPtcDgchwwHS8iv8hfH0gqUbF8UatNopA4AS8iHrGMpYdCP\nkkO/J8tC0q+LZAlDiB705Jsj8W56aDLfkVKR+9YQw7cNCFCaVC9VeoZm6o8G66J8FUsSptZoREJ+\nvc7rNHROnSJWah0aZ8ai4I8+IiT1bqx9FUaSLV2CEDQP5XgUHYSaEX97h7dduhLL944DOxzGA9E0\nZmRm0gfiD/6AKVKPPsaaxtUrkZKzK5l7MonYOC/V2TeScOKC0NCWViJJe+M4OzU2TnIYaKMZHReN\nugQACI9ut6dEcEOX2bdE9eHE2ppkoV+P1KCKOFi1qsOSSKvjI1Gi3BDpsiTP/OMXYtakwZjzjrb3\nunJsdmQd2YwZ6ltLdTkH77+3w8+AdfjW6tynI0KeXz92d7ZtKPd6RcavLzmKE6MhaPioVlEIVtxN\nyP6Y8twmIOAm3iaXSB0Oh2NOHKhEWpIvv60iSlkon9SCka+L/aKrBDuWTPfDYZSAxiOWFpVkT1oj\nZhglvNFIQgKF4lSTnIjNZrTzNISUG4RsbfODjoVoPJCkGEWxCdkvptpbWxJKtrxsJBipjFrq87Gr\nVaF4GRJ+f7h/TZjDiJ0h37Nzn/x0tu7aM0zYhtC/dBxTU58KE2M8HFtpVSqDyv5RoozjmUo1hSee\nYGnn/FM7co5oc241JbAi0equhkojdJ2VdZY2VzakukE1ahGpSDNKz7MhpgN5xmioNaey3mdtkgW0\neZ84wfdhfcXcB7mMnmSmr0iOUA1yAYCihFBXRWq8vBNpTJp6WLWy1lIz9wtEKlQClmyVOTiwVXpF\nU1XGodUmC9JsWaqZBhGoEzNmUQsm87+uybLeyI9qQYlpM203tXCJ1OFwOOaET6QOh8MxJw5UtS+I\nmlQyMeaq2qv6rHSoctmW+uBGe102YmvuTwAYieOgklGZZP9cGQ+J7RaH1voGy/4bpiSrlqu4ekWy\nRxlqkpaMLopgr9SqTi86MAbiLGo2WnKt044HXUeJlugwxItkf2P2YURjmY37R0+/JFtXkiiuukTA\nVCWTUDowVK8s6kvKiYxt3kleVmdAR7IMPS5x7QDQ64gzQhxIVSnWVqrEe3jqDFOUWhKZc+VSpEYV\nRRdcWuX+rx45yecsRHW1P9CYcIm4K0TH4iDIayPOyzDDOYF9y1EfTiyJs6hhnKjqmKOU388lKVC3\nvBzv1ZKYsToDoTI+cTHbNh6rI4jHYUl/jelrVTIyNfSdH0qmp71o8qEg4xCm3xM12+i4ZgVkcgUI\nJ2VG60jKL8wyygR3NjkcDscLiwOVSNVJo5IpgOzLPc7a6DZT4lfa9HrsbOqbzPLjvkg6QvZXadd+\nVSpCiVIDeUsKXxWLMYVMmiiRnqWdXj+eQ50JZZGuNKPNOBinWZG/qmuScaZYLJn98w6P7EtpOfsL\nWL5Xx3NpNVJhqg2+t80G34+qaAMh51BS/pk6nYz0NiHIdYR61u7G8Xjsrz4ubbnx8VOcaWltI2b7\nWRFKzSWRRJ98MkpJRYn/b22e5cMUuC0Vo5RVKupYiaxhcp0q2T5KKdMS6QzB6dCjUpSS5IY2NAry\nfkm2+kZdYuVb0Vm0KZrdU5d4jOy7N5Z8Fi2hTdWFfF+tx2CWZoOdsUM5b6/O5xqZLG+VCp+voDkx\nZhRMpElZ0vy5f3zExDhmRR332ycPl0gdDodjThywjVQl0jh/qw1l+pNhMunY5IUAymVjf0yUpsLH\nqUoWfBiKlYYtFoXSpPbYresxQ09B9iORlsuGClMQUnIiBG6lReg+AHB8QwjcQhgumxBVpTmpfYdm\nUDimvqaLAJG0y/UYilcWsnpZLo1E7E7M5z2jlcwIxcOEZN4UCeTel92frRPlAYM+axRn73s5AGB5\nddW04Zu8db0j245k21orLEG31nhdIiHLIbGBIpPPo9GQsiCB2WWdZ1zGgsGWrtaqFvo+sbRob08i\nvoeB2KyHJhxXj6TvbFKYDrDI7mfQNkKnqsX3S+tzVaW6gtXgYgZjeb+ikfTWrnHC/vlchs4lUofD\n4ZgTh4CQr18jIW5nYZPJ1H5Hj7MtrF41xsUx2zRVAikkEr5XilKSyrY9CR0bdOWrWorJGUKxKefi\nr+JZk/V9V5IwXJa6TMoyri5F2+CRu7hvNVmXk2i0DlWqCTgkKYMlcBcPdCieVxQr0baobGoS6V3j\ncUPO5p3f325TaT2TMkRKPHLiVNamucSSp9rMVzdZWikYCURJEKfuvlv2P5FtK0u4oGoNeo7Zds1p\naROaJ3NK4rFtFk/D6EsIdi+NknlXpEvNR9rua80kM55FHocr1/i324v28IJIoBVhvug7sGuqU1y6\nIjljJZS70+NzUCG+n1pNIksglNMY8lUQZoxYxLNQFZ5NIiGXSB0Oh2NO+ETqcDgcc+K2OJusoVgJ\n+LpNVXwtkQsANSkt0axzuYBaPardw96O/LL6XRJDdcmo7Ury7guRe2+HY/Ub1Zjl5szpl3I/RMXe\n2ooZiS5efoTPIbpfKeFznDx1JmuztrHJ1yOOixydSZelH6mUzbDan43tXxSETKUyjsFsm8Yt58tt\n2+UQA6entkaStJTiNjlslzZYJV8Ss4GWEQ4mm9ZY1lWFWlNtmvjx7Oh5JyblHEr5/oSczJFMtJm+\njEX0NfXFCVscxXvdGfFVdqVC4UDyHBRCbBMKfG9H8lxrETkAWF3joIf1NaamVcWcYtL94rrmL5Wc\nGKORlnCOba5c40CZyjJne6NyHM9KVd/12Sr+QcAlUofD4ZgTt0UitUgzalA+jNKGiOqyMqWWNqLj\noFTg8D6VNq9dYuL17l6kNrVEKlmWkEAIxanejHSZWpUdFp0e73d9OxrDR9K3iuSkPHP2XgDAXaei\nA6QkWdKHEtqYzshepVKqMIQwMtmjKJm+N4cd0ag/yyivUlvIN4YlUI/zbYDIgs6cCXocQ4jX+zbp\ntMqdQ9vK82U7kBU5m2xtoRK1VkUw55/4nS35LJ5M2hfyfCk19MSEqX+plDhWKmGtEd+dtU3WzLb2\n+B2467QpFCjZ0O6+l51+dSH0FwrRITWQ0unlAr+nOjxDkx3t0XOPcR9T7s9pEwxz8sxZADdxMj0H\n3CwHqYVLpA6HwzEnDlQi1ZySiQ23C5rjU8jyJjO9QqVWKgqB2whvWn9ne5cTmly4zPkwK0kUV1pL\nUrNJ6vAMRepZ3oj0pb6QibVWz4mTUdq86zRTm7akxs+Ro5wQo1a3daGk/33NCG+kTbXzZdI2/10k\nex+mLntxMIM3FBN56DVOWxCjBDGe2qaDHGZ+6/NE/iwEN3cPk3xbmxSGJqXcSYJ93JZmEvG0/TQ7\nXRZSaIMOFk8iRcLPc33laLZqs8W2/9oyVx+4dvUCAGCpFZOOnDghFLPjXE/pFQ/sZNvabdbw1lZY\ngr106UkAwPVrsU5XkLprZ05z7S2S0ONrWzHT/mPnzgMAPvzhjwAAnr4Uq2R8ieSVbdQ1B20+4IMP\n+sKOh0ukDofDMSd8InU4HI45caCqfcyCFMVsVel1m0YxKeXJth+BDdRlkxuyLO1qsq4lERS9vSj6\nq+q5tsHOpuVVUVcakUaViidrrcIqSKUaszepSl5bXpe+6mHjdyiLyFIzhLnuZMLJpvunOVVwEZHP\nlgPAqFOT0T82rdON8itYlVhU6jC9/4zsnxP7RgdYLBtht2ke0bzanlPfZb80i7yb6upUxq58k8VT\n7etNpiitrkez1qo4dgtCN+q0VW2PY1cWquHqyjEAwNqRY9k2LYtdlsxau5Jf9ropfa3FB9eOsAmt\nUuO2tVZ0aJUqTFX82Me54OL2diw3tLvL5oOqUCZLFXnfbtFe9mwKFd7IAeUSqcPhcMyJA5VIVaYY\nmIzokZA/kRnfZIgibSPSSSGJ1KhSmSXRgmSFWVkWB1Iaz6FfESXLQzLZpCYPZip0jCTN024AYKzF\nzmQ/Lao1NgRwpdlo9ijT/akvHunGkc2Ss3gyqd5Xm7U8owRlkuCs68pnw8oT4yekQ1JHUk7G5zbq\nXZLY9yTnbVJy9oSEjChlgvL9z1GsZlzbFPaRZBZvNIHmEmtqK2uRXri8xo7VkbwDSpbvm/ywfYmN\nX860yeiE1cAIyHs1kt/eIN6humiVdZFA61IsslCOmueaZOq6ssXx/LvtmEu4J1nAxhn1ULdYLUTw\nHL26s7RpC5dIHQ6HY04cqETaHfCXo2hqNlUlNHIg1KiyEICtIKBShdoabdakseRH1HiyVKSbkslQ\nX9DsPsjTkIrmHOOxSKlCSk4NGVjtnyo1j8ZaathcnNB1CkLdSMxGPabSv/S0RZuXdQFr/ISZ9sNJ\ne+ONLJp2m12Vz1c5yyYV85mqZJmvAcX7KRF/hj12glo1y46qMsZszn4+Y1m8QiMBLZ6JFNU6S4S1\n5lq2riCVH6oV9idcu8bv8M5ODKG++yrbO1ePs1YX37w4Ivo+be+KjbTdydrUJHyXZC7I7NPmfmrG\nr+OSZa2xFwNmypIXWPcb66tkHsxMW6HJ5zKu2m/MblbBwiVSh8PhmBO3x2tv1qmUlkmJ4nWn3BdD\nCOzi4bdfB032kVUm1Q9PXlzMn1e2jYyNctKOOSswYD/zSra//CTWoz9ZiGhG2OQiGtWyq5oV/hnC\n9EZB2EdKzeSF/SSAaIiVX32M432etG0GE8WRZjbZvCRqWRgzzGzTmOzjtIl2oVBfYkm0YpKOJKRB\nJFI5V2ybz1yOpPtz55lkf+olTKivtyIbJlFbtTwPgyG/78OhrYDBWqjm8FUNMjFy3tYWe+l3dvk3\nNf6JplaRlfDVgtYCg8mMIhqfaofjcTy/vuuj0XDmLy/zsU6euhez4BKpw+FwzAmfSB0Oh2NO3Jbs\nT1aNHgyYOrG0xOpALGBlCL/lUm7/Wchi9rWNyayky7q/Uq6s+j6pyue2iaF8LOpISSha1iE1uZ/t\nq6qpk5muFtIjcYuI9CVdMSPWfsb1hwlnwGxHjqqL8WxA3jlBU3H009QmzHQyTV1I/jpexGi0mDpY\nKkXakTrptCzLisS1X7h0JWty/gLHzbcl78VqmC5+R1LOuahmukJ0SWnp8tGQVemB7LSzHUn7jz56\nDgDw5JOcS6PRjOVtOrtccnvYkYxv8mBokUsAGA0kz4acQ38BYCgqfE9yGvelxPtA9uFlPpar9g6H\nw/ECgZ5NeJTD4XA4puESqcPhcMwJn0gdDodjTvhE6nA4HHPCJ1KHw+GYEz6ROhwOx5zwidThcDjm\nhE+kDofDMSd8InU4HI454ROpw+FwzAmfSB0Oh2NO+ETqcDgcc8InUofD4ZgTPpE6HA7HnPCJ1OFw\nOOaET6QOh8MxJ3widTgcjjnhE6nD4XDMCZ9IHQ6HY074ROpwOBxzwidSh8PhmBM+kTocDsec8InU\n4XA45oRPpA6HwzEnfCJ1OByOOeETqcPhcMwJn0gdDodjTvhE6nA4HHPCJ1KHw+GYEy+6iZSIXkZE\nHyWiXSJ6/e3uj+P5AxH9PBH98O3uh+PwgojeR0TffINtp4loj4gKN2v7bPGim0gBfDeAPwghtEII\nP3m7O+Nw3Gl4Pieo5xMhhCdCCM0Qwvj5PvaLcSI9A+ATszbol8jhUBBR8Xb3wbH4eFFNpET0+wC+\nAMBbRYT/JSL6KSL6LSJqA/gCIlomorcT0WUiepyIvp+IEtm/QERvIaIrRHSOiL6diIK/bLcHRPTZ\nRPRnYqZ5J4Cq2fZaMeFcJ6IPEtFnmm0niOhXZIzPWRMPEb2JiN5NRO8goh0ArzvQi1ogENH3EtGn\n5f4/SERfKevfRETvMO3O6ntCRG8G8BrEd/Ct0uZvEtGfEtG2/P5Ns//7iOiHZRz3iOg3iGidiH6R\niHak/VnT/obHEtxLRH8i+76HiNYm+3mD6/0nRPRJItoiot8lojO3fLNCCC+qfwDeB+CbZfnnAWwD\n+Fvgj0YVwNsBvAdAC8BZAA8B+CZp/20AHgRwF4BVAO8FEAAUb/d13Wn/AJQBPA7gfwdQAvAPAAwB\n/DCAzwZwCcCrARQAfAOAxwBUZJw/AuAH5Bj3AHgUwBfLcd8kx/n70rZ2u6/1sP4D8NUATsh9+loA\nbQDH5R6+w7Q7a98T+w7K32sAtgD8LwCKAP6R/L1u2j8C4F4Ay/IOPgTgi6T92wH83LM41lMAHgDQ\nAPAr2tf9+gngK6QPr5Djfj+AD97qvXpRSaQ3wHtCCB8IIaTgF+gfAvi+EMJuCOExAG8BDwoAfA2A\nnwghPBlC2ALwf9yWHjsA4K+DJ9B/E0IYhhDeDeBPZdv/CuDfhRD+OIQwDiH8vwD6ss/nA9gMIfxQ\nCGEQQngUwNvA4674UAjhP4UQ0hBC9+AuabEQQvjlEMLTcp/eCeBhAH/tORzqywE8HEL4hRDCKITw\nHwD8FYC/a9r8XAjh0yGEbQC/DeDTIYT3hhBGAH4Z/PG81WP9Qgjh4yGENoB/AeBrbsGs920AfiSE\n8Ek5578G8KpblUrvBJX1vFneAL+cj5t1jwM4KcsnJtrbZcfB4gSAp4KICwIdtzMAvoGIvsNsK8s+\nYwAniOi62VYA8Ifmbx/XWwARfT2AN4IlOQBogt+hZ4sTyL9zQP69A4CLZrk74+/mszjW+YltJdy8\n32cA/AQRvcWsIznu5PmmcCdIpPZFvAKWSu1X5jRYFQCAC2C1XnHqhe2aYx9cAHCSiMisOy2/5wG8\nOYSwYv7VRTo5D+DcxLZWCOHLzHHsM+GYAZHE3gbg28Fq8wqAj4MnlzaAuml+bGL3yfv7NPLvHJB/\n754NbuVYpya2DcHv/n44D+BbJ56bWgjhg7fSqTthIs0QmPbwLgBvJqKWPCxvBKCG83cBeAMRnSSi\nFQDfc5u66gA+BGAE4PVEVCKir0JUK98G4NuI6NXEaBDRlxNRC8CfANglou8hopo4EB8gos+/Tdex\nqGiAJ8TLAEBE3wi2OwLARwH8beFlLgP4vol9L4Jt04rfAnAfEf3P4pD6WgD3A/jPz6Fft3KsryOi\n+4moDuCHALw73Jzy9NMAvo+IPgMAxCn91bfaqTtqIhV8B/iL+iiA9wP4JQA/K9veBuD3AHwMwJ+D\nB20EVhcdB4gQwgDAV4G96tfAzo5flW0fBvAtAN4KdjQ8Iu30Y/laAK8CcA4sifwM2InhuEWEEB4E\n+w8+BJ4YXwngA7LtvwB4J/g9+QimJ8SfAPAPxPv9kyGEq+Ax+ecAroK53q8NIdxMSpzVr1s51i+A\nHc3PgB3MNw3MCSH8GoD/E8B/FDbHxwF86a32i/ImKIcFEX0pgJ8OIdw6DcLhcNxxuBMl0htCVMEv\nE5XhJIAfBPBrt7tfDofjcMMlUgOxqfx/AF4O9hT+JoA3hBB2bmvHHA7HoYZPpA6HwzEnXLV3OByO\nOXGghPx3/+gbAgB0R5EaeO3yVQDA3qOfAgB0H2c62EqpnLVpNni5M+4BALbTUbatU+Lw64t7vO7p\nrTYAoNyM+69stHhB9mtUedvdp09nbVZXjgAAag127jaWWtm2UoVvU6lUAQAQSgCAkMZr6w64b0O7\nUjH+/9v7kl1JsiM789nDY44Xb8qhMmsgi6JIEYIaaC20knYC9APaaqM/EaAf6B/QN+gTBEhoqQsU\n0ByripXDy5dvijl8dtfCznWzZCWLgKL5wADuWWS8DJ/9evg9ZnbMrCAiIr/hZV6L+auVc7y55wSb\n//xf/qtDR4K/+2//qSUiqipJGtnc8d+7RUlERFmWExFR5YvwIRzxd3XO96Wu1Hjiu9vlnj9vt0RE\n9OlLGY8kxv2v+H6GPbaqYqVsnI0mREQ0GvSIiKgXdWn65DX83WrD53F9f8PbTHrdOqNJn4iIFnse\n11evRd9f7nDeOW/f4rlKa7HufvuGn8N/+O+/Pprx/Lf/4T+2RETr+9vuu6riceuNp0RE9OLHXxIR\nUb5cdevs766JiChEktgvfv7jbtmn/+pviIjoyUtWQ52OeVyGsbx68HOkJORxDTx+hmp1P/OS/359\nzef2Bu8NIqJVyff/Fued5/zsNY3ceqNGDnz+I1EPS9zjsfbx+y4rHtfpUJ65UcLr//t/868/Op6W\nkVpYWFgciEdlpI3L7+2qyrrv2pTjOGOHZ74TwyAceceHIU8CLk53nQnre7vg7X+35H06fc4ke3F5\n0q3TG/AMV+6ZJZQ1zzh3C5l59ylvPxrxdud0LifugPlECT4xU7lyjpVhVS6fa6vmrarmazNfBSBw\nbVV068wmwk6PBcOY79Gkd9p9F02YOXhgi63PLCOPZHYvGmYzxY7Hg9q8W5ZVzESvblgW+PU3zHbO\nToJuHd/lG7jd8Xa7nFlrE8m55Snf2wbs1QmEETcFbxeApox7/MwljrDWds/LtmveT5HKgNY1H9/3\n+HnsRTx2Q1VU6G5Q0rGhe4Yb+X05JoaCT5Oxnqg3Rw+/zz5+n/NEBiIOXOynwa6xP5X6XoPRV+YY\n4HdFI2NW4n0QgSVOSLZvYDXsCt5PTfxcpak8VyXGvKx5P40jVlDe8DLX4+9cF9ffbrt19L4+BstI\nLSwsLA6EfZFaWFhYHIhHNe29mE0ntxbTIQzZLKrxXUFMr8taqHeE4JSfsFN44EpQoLnj9bKCzYL5\nJS/rq2CT4/I6wxkHklo4p6tGzK8sYxpfbpnCh7mY3QQT3A953+NTrtEQxmIKlhXvy/X5lma5mAJ5\nzk74QcLrez02fdxQztFPBnRs8Asej0lPxmPs832MezxHB1MuyuM/7eouU5Oz6dWkvG7bqHtV8bLF\nBQeAvpxw4Z3Qk3UcPLaLPX/37fKKiIiyDwo+wbTHGNe1POplVmGf/DyMI76OKhOX0wIuo0XKYxc6\nEpzY4XybmvcdI+A5HPS7dU4nezo2NJ35LmazYxxScKe0Po/rQB596sPOH8L87ocqsGe2g0Vfd6a9\ncLgGwdeq9fF//txXKtiE90Pt87595VsIWnbfeAGeK4fHpajlN5zj/eDiXZCXUj3R80p8uvjk7zfO\nTtZxfjhmaBmphYWFxYF4VEZaYRaqXDmsE/BM3zjMzirMTv2+zO5PP2UG2PZ43W9+LRWzbvc86wzA\nVj+/vCQiopOJzIq391zaMMW+W1T5GgQy8yY+s8Sr31xho0W3rIf1jRrD+ZSZS3886tbJ4MwOwTbf\nXb3rlu3vWKoxGTHrnD/lc+ydS5Cm6h9fYsT//h+/JCKi1Vjm4y8vwFKnLJcJXQ4OjC9lRvdcZhA+\nmCC5EogKie/JMObP533eT5VKcpkDlmOkZi82LF373fK33TrLkhltBWZ6t1eBB+zqGZ6xISrCvdvL\nMR72HBALI2bbJxMJdr1/gNW05k/ESSkgWed0OKNjQwCrSCfpVGCQHu6VA7YZ1/LbmY/4u5lnLC0Z\nzxpsEXEgKkoes1JJmxwEJMuGP5uS3w+7UoJNFYJLNQK8jgosRi3/Z4xb3iJo6PpKorVj66fImGW2\nKqBWITBqpF6m8KDcKQAAIABJREFURpFe589VXrSM1MLCwuJAPK6PNOCZqi7Fd5GBUVYFzwLGp5L0\nZVbrj7m49S1mjm9XIktYg5V8+oxZyY+e8qcXymySFfB/QprUEC/TouB5wv6y/IZnMXcr/rJ0zz6U\nFBIpP+AC3MVKyWV8nrFSzK6rV1fdsnDH2+3vH4iIaAfhbzQWiVZByid7JPjqV98QEVF7Kffh6fRz\nIiIaGF+Y8YPupMpZC9+ig1neDZWSHvemxrLdjn2NTi2M1odv2XyeT2EFrG+6dd7v+O8d/NsfdNXB\nM7YDS9kWkFPl4hML8Pz0wbbrRsYH7nDabsFkiNmu58o6855YK8eC6Scsmq8LefbNMLpgouGILYQg\nFIbmgck2sDQWrfJfwhIo8RsII1534gqlNCyzaXi7HL7R9V5iGCXYoR+zheCrBIseYg9nYMQTJGPk\nKk6Rwde9322IiKjI5RrLEn50DGwFQX6l3lNF/sO/T8tILSwsLA7EozLSJ5fMVoqlRDRXS/ZFZvBP\n9QKeeXaOzEa/ec3+xrdrninuH4Q5TOGnfPGUHSS9mGezvJWo//yU2abf5xlrv+XtXSWybVs+3uiE\nZ7xc+U+ziM8pGfKxQjAqb/XQreNigtzBzxLuJSoYYfKuMG2ZGbxVTKyLjh4R7jfM6N7Hwv63OV+s\nET4TWEG1VnV3YUV4YO+GIRJ1+QxUQmR/d8vMMvTlXg3G/Lffx3gEvJ9Z/0m3zvstC/nLFoJ6pQIp\nTQQYDGoP1lqU8sz5Pq8TgC29u5dn9u6e11stEP2HhdM64pMb947P5z16wozUdeTcXQyIg+QDDwJ7\nfyAWYwufYoGod6HGs0LsI8TzkDa8LGvFn9w08HsihpIVfH8Xa/mdLzd8b2dz9p2fqBTPHiyTyDxP\nUJFoH2eNpJii+D7bNEy0rooP/l8V8n4wltGfgmWkFhYWFgfCvkgtLCwsDsSjmvbPIU2qbiUQ8zpj\nyoyiLOQjNzpVebYpTPE7uAQcJc+Yj9nEmI3YzAtQQeZWVYdZpuxgnp1z0OruhpcVG3E4jz9jt4Nn\nzIJCBTd8NkdcOM6HOYJVezlHH7KOwFyHL6bLFiZjNGYXQ9jnYzgqwNT7s223/wqBGgTbVMzmFRIa\nNht2bUQN3/t4KXUNApiAAQIHji+PoYceZdWWxfXvr/hZSfoSmPNjVOjysD0ChSdTMe2nOxby7yt2\nEez2m25ZusfzM4X8Cr4XJ1fVgjD+FXLsN0t55u4XfI5b7CeBOyreyfOUb4/PtHdiSNZ6kigSxewO\nC+DqMuauEa8TEVUhL8vxI/bUc+3ATdBA6lYgILRrZPsSfM7DbyCDib1W9/Pqml08bsjbGxOfiMg3\n52LcEJA5uspd5nn422G3mqPkTC5cM6YeAOHdUytXz9aa9hYWFhZ/WTyu/AlVn5qdOOV9pHG5YH0l\nZAmtLwytjXgWylpmpklfUhIvTlB3Et/5yO9KdxLs2ayY3YwgiG874a3MWAFm3grBES+RGevFM2ar\nuzsOLjWv32BdOQYy57pahrluPNrjSFQAB/0DAmxOLjKu2UgSEI4FgwgspZJrvUMiQ4hZvTfATK4C\na4M+pDQlvnPVOCDdcrfi/bwDIx2OhPUmIw4sjhFMiMF2xqO5HCPg56LOOHlj9SDbZyke+xmPeZwg\nPTmV56rY8XcpghRNJhYGVHBUt3z9Hp45R/GS7PjUbPT2LVtqw7GkK09O+NrGCPB5Ad+7WgV7Cjzf\nhLTgnicXH5o6oEjCKX1mu2tVHCsEowzBGisw0zYQiVSFe5tDpJ8XsoMISRMu0jjNp6/SOg1r9fFD\nVYSaAhMExqexeBuVyj7o//Dv0zJSCwsLiwPxqIzUB1u7f/tt912+ZpaXYBaosU6tKuQbH4xhdkko\nM1UyMOJs3t4smk6VtMhHDcMBL0xiZjSe8ksmmE3TNdLDlBopTHjGq07Zh3S3YilP40+7ddwhH2/z\nwL7A22sRh88GfLxkyiypN2fmdL+VlMTbO/HpHgvOZzxL53ciU/nuDcuOFvd8j0YT9l1XqkjF2Rnf\nhxhCalIdDyIItk3nhHfvOL13uRZ/WWJqUl5wQZRw/pyIiFxfRNqxw8doMt7fZi0pvxl83B6IdAK2\nkfvCOjaQr+3gw18qP3AGq6mreQNpUKQKaTjD46svu3jDz2yqEl4K+CnrU2aZUcLXFagUzxDSsrDk\n31Oh0zfhW42RsFLCN1mq+xni0Qg84+NEooYq5DNBoaAg5u+ySh+f//Y7JooFipEaRVeA4/uubG+M\n347Bgojua2G9+07cr6q1KFhGamFhYXEg7IvUwsLC4kA8qmm/fWBzL1av7zkyk1pInDyCXRCJmZW8\n+JS3e8qc+9XbN92y1thncHAHEe/8/EJyneM+b+ciF7p1mab3VT1R32VTrjS1MlVDtz1aGWzQviKD\nbMcbjrt1Lr7gcxy84+DIuwcx202ecANXwmDGUp7Rk6fdOqsHlflzJBjO0EbkQUyoFRrDmawSP+R7\nvXoQ10W/j8BDxfe8VSZUi4aAf/iW7+Nuy+ukmazz9i1LmyYXz4iIaPz8J3ws1X5ijpYx4x6b+IEr\nkrvKQd71gt1K6QJtRbYicUkhw0oRiQhUjc5Rys/oCg0LMzwX9Ug9T/7xyZ+qDV9/qgI5ZPLQUas1\nQCHSUAV8Y+ThR5AeusrFEUPOGCP7qIcAkm5GGMGNF/jmXkNGpVxv4YDH0Yl4f5lyLbioL9vJ6rB9\n6Mk6jZFYoWZDoEz71jE1ExBkQhAxLcX9sN6bZ0PceRqWkVpYWFgciEdlpL//6u+JiGh9LezgZMbV\nmu4rZmTp9sMqUEREk5BPM3Z5dhzq6thQwGeozhKZSuhK314h39qobEyXVl3QpY+Z07DGRkWbQniv\nR5BIeROelcpaGEy+5/Ofh3z+k5++kOMjJ32/Q43SFIz05PNuneHwCzo2DMZ8rx8CmY/3KV9/gPa5\nJSp2pakEpHZ705yMly0XUtk+xfbX1xwcMnnPhcqNvr3hANT7K67C9XKJwF4h++kjR36MrgZ9XxhU\n3qLV8w1vZ7or6JbRe7CUBYIMigBRS6bBIaqJtSZIop5LVZv0WOBirBrVjDBFIKmAVeYg2OTHusFd\niE9894GgH4E4/L56kDL2emJxRh2jxboILseR3MMY33UkNRO2WKBjgQlsmZbLoXouIwQEjVg/cFSD\nP1hElblWjF2qZH37P6Nns4zUwsLC4kA8KiNNIOZ9vRUh+82K/84hN1lidnBVvUB6YElNAinNl589\n6xZtt8z20pxZxgBVtrveMERUodJMiFlptWBGtF1I2mD4Y2aJs3NuMXx3LT7OuMcz3RDVo2q0c3b2\nMkv590hl3PP5BCR+puEcKaYxy3ZKTMatK7Oqm0zo2OBDbd24MrvnoG5bpMz2QfvN+BARrdfMHBuI\n3XWrW9NiedvVg+V71Cpx9GbJCR23b18REdH9d7/i8yllPOOIx3oMtnSSiM98sebtV3gOVwWYWCxW\nSG4YKdoxq6xkSiHWb2HamBRiVbCM2r2M7dGgMe2Q9VdI2TU1OuGrdlz5fW7xuzLtqcnXFiM+wSgD\nsMwwFgshhL80wrIeGG2ieoElkC4O8dlXPtoE1ccMow0xjmEoZmlsdJHwf8bKR0pgtDu8g/bw03tK\nZqm7BnwMlpFaWFhYHIhHZaQZfElNJOzAjzFrFMxY+jGie5dSpGIEIXx/xIwuilTqmIn4IkLnoyjC\n+xsRYL+5YnY5HHGUfbngbfK9mlWhGpggVXOzEX/b/R0z4hrn//U3zD5HobCOpycsFB5gHU/5WKuM\nZ7yghxqOcNaaAiv8n0cdin8SZKb/DilGimh9mUH0juId477iOaaQBRhQlqturvi7RMcEU0DCVcco\nIZK/uWL1xq9/+b94QSF+2JfPPyEiogF8pJdDiba+vmYfK8qRUo19r5UVtEMk3hQ4aZSFk2dIf8UO\nIqQ/ZqqwxW6j6OmxAJfYqmfXdPg0TK4LhKt2Rk4X7TatQsW3aKrdFxhXB4V/HE+sUhdM1qSJB/gt\nhKrLbgzf7MAw06H4WIdIvU7QLy3p4x2SyPY9JPOU+K6nrnFoTAkoFByMtY76h42c78dgGamFhYXF\ngbAvUgsLC4sD8aj25O9+9zsikpasRETjCZvbjcdmUYsWD60S7H77hmVDDcGkHkoObmNqfcJBXZZG\nbiNm9wJtIjZrXpbDYd6LhN6naNM6n/G+40TM7qxgWu+hMtGzlyxtqlbvu3VyBJ5cD4LlQHL9U8hJ\n7iDtCVGLMVGC47Y5vuDE7drkT6valPg0mu4Gpv0okIBSDrM/CLtyO92yFjUha7gIjFjfVbn6DZbd\n3LLk7BrBp7yQ4N+/Q4uLz19yosS8L+Nxhr+bgMd6heSJTLXA2W4gsm+/zzXGMZuVJ0jocFDB7M1C\nkiruN0dY/gnPYPtBbVxz/ais1AVdVDvlPzb3lRfHfGVKfZIJGupgDwJ7DiRrmcdj7qsqUrsdL9us\n+ZW16ElgMenx+MWxqeYF872nanIk/Peiz79r70zeL8/RartnapYiUSRdS8B5t/zhWhiWkVpYWFgc\niEdlpLMBqs9H4oyOUFmpmXF1816P2UIWiryBUqSuIRiQ7iSoYGRTfdT/7KPmqZY+RD2ecYrCsEae\neSdTSfEMUT2qxqzsqFtTQB4zhuQiDpmRtJ4w6wazVwGhcRmLo3sD6c89pD1nqEQeqWt0HS06OQ58\n94qlXtVK1V6FqJs8vv7M5c/rrQRf1hDnh6GpEfn9ay+LD8ej54mFUIHVrGFZLHF/ayVRubrjYOHF\nKTOXfiDHuBgyOzEyPAcV2T11Gj7E3T6Y8EDJ0744ZYncbsNj/vqag5F3uZLc+cdX/YlaiO5bfe7m\npuD3BGrpKAuh04YZaqqe5S55gYy0qv1gVSz8YN9G6lYrPVmTwfrZIwFnK7/PbcTj70Hq5kNzFYay\nTg+1c++RzBNlEvDutfz36Zifi82Wn5n9ThjpZinB64/BMlILCwuLA/GojDRImcE0lfjLihBi2jOW\nO51fcm1JJ5Z2rzkkMLdozXuvan3mMc8eQYQWybii/kAu7QQtljOk+zlOgHXk3IqSz22z4WVGbE1E\nlMO3SRMjBeHZ8eTsolsn7DPLdFEvsUrEJ7e7Q/3SgJlPjPqZraqfqeU9x4K65Huka1wYAtPCB2YK\nQazV5e3A8L3cFJP5IP8S4Hs9hJRlqtr/FhUfJIfc6QEMQteQfbfk8VwhYWM+VrVKwYDvYdksUzBT\nRZOGsCgC+Lp/+pOfd8tcWD0PGy56smtg6fTkmTs7wnqkTmssve8/iyJI53H5UJ7e4t8/LVp32j/9\nfwfj7+B33hWxUe+JDige0yixfAUmW0AyVyMlXGdR+LAet/CfniireNpDYZaSr+093jN71TOqym2K\nqIWFhcVfFI/KSG9/8w0REcUnQgV7c64eP5gwIx0M+f+O8h/24P/MIJrPeorJIeK3yzly225Np1GZ\nVUdYfzpGb5cAM5/qVGoyOk1VvtOpHD9FlNnFNGr6vviRXEeLaP0dZi5d0q3G+p3PB+emiZjn60jp\ncWCUMEtcqHRIQ05rlDRrcc8yHXWv0OXRBHD1TsEqXURwhxEzwuFEBPUNHtsVUn89pHFmlZzHNRjp\nAr275qpjQoLnwUR5Z7BYSkWtQ/h4T+fsu//yRz/ulv39V/+TiIiWGSrJB3yNA8VIzy9UmfgjgQdG\nquoFkYNLqsgUNDFFfeRa2z+mmwriNjWmyh/9n4hc4xuFhdGk/Fuu9tLbrWvrBV+5E8nv04e1Eg6Q\nxINzVFnFXam8CHEOV/lPjdWyQ6nHFeIdXix+1OHg4+XzuvP7waUWFhYWFn8W9kVqYWFhcSAe1bSv\njLmrKvGcf8o1OcfnXC3eQ7Cm1hIKOJEb1AssK3H8Ggd3VvCyumXzLPZljmhhSkYw6Ux1mO1W9rNZ\nIJc25v2dnYrZXkCQfw/hdjRAPnkpt2+N9sFvUJEoCiU4cn7CZqULufp2ycGnE5iNfOHHl2s/G7O5\ns11Ls7TaDJu5/zDpS+VGaSo2z0xFdD+UsXL8D03AFvnWoaoE1E9YYlZA5vKAIOJyJ8GJBQJJ95A4\nfabu7wyyt595PC5VwP9fbGT7AG6bTz/7EV+Xeubeo5vBFlXTQ9PgTbwHNBgen6umLjho57gyVq6D\nxnZwXRECpK2q51nj7xouOGP+ExG1hqt135nnQt8fJALAtG/3HMRzM5EcBbjHbYN20LXI4cqKz7vb\noz/DNuJe6aOC2+Wcz3/Yl2euhgB/t4KEEe+bfl+2H0+k9sfHYBmphYWFxYF4VBp09i9YQjJ6KrKh\n8ROu0hP2mcEFqJZdqwoyRQmmALISKLG7aYmbtMxSAp+/yDbCkm6uWZw9TJllXD7nqvzZToILr/7A\nLOPihGez+fy8W3aL1L8Skqgg5xl0uRSWYqqKb/Y8O3733btuWZae8r7PmMEZ9joZiwO7bo6PwUzQ\nTvlbFaRxIqaSpn9O3RiJlAr+DZldhqiaHgSq/46J9iHYtEt537tC2OIZ0oqfoSrYbcmM4kLRglv0\nfLp94LFfbSWJ45OnbBGdTPmZWTdMJZdzea5ayO8iJIj8w//5v7Jz0/4XwvUWVfx9Jf9x6+PjKBWq\najWNjGeA2rsegr9Bi7RYX9iaj8Bc4wQffBIRNeCJNYJTDZ5zXd+zhVypQQU4Ql3Z2JWKSwNYJDUC\nU2kpz0OGGsClj+AjUkMjVV1tjtTQF1N+HmexXGOJSl8Vqj756Bc3GkkSxnxig00WFhYWf1E8KiN9\n8bd/Q0REjfJXuZC3NBDamj44Rv5CROSj4rYPJuqomWa7Zb/GesOz2BRpXr4v6+SQJDUrXufsKXwo\nobAko6aI+mCGgerpAqISQHrx/oZlErutsM4J5DUJ2NZOVX2/vmefT4Tz91F8pa+carESnB8L3t/y\nfcgLlRoJf1kAAXQMv/R8Jtd3AnF8iS6NeSXMw2SCmnqTHpisq+qBmuIWxjfpxjy+F2dyjGzN9/YV\nztH7SiycLGN//Odf/IyIpJ+Pq1jvP/6WO5V+9RsU2lEV/nsRRN0nzGxNxf31Rq5jPDi+LqINWF5V\nKiE64gN+gGQa1AoN1LPrI64RopiLq/pjVeh/ZAySEoyyVBZnC9bbwrLwiMdzlIiVdjqHPxubrVSX\njSW6n2Z7tvQcly2WaCxxjhMULXkS8Xax6uqwwbMa9Pn8kyFbkJfnp3L8icre+QgsI7WwsLA4EPZF\namFhYXEgHtW0d2D2RoFkJpnqOkUGqo6WuJ4y/31UdQlR/7FWmRQZHP2mfqXJr41jMe1nMMFMDq7j\n8LGSvuznk5dzIiIaD9gscFUXkCmCGjnMigTVi+4eJPNigFs5hmthl8n5v33D7odz+Ld3yA3frR5k\n++EPmw5/jXgLF4ceD8+0iUCWz3jKY/f0iTjuh1i2XrAJuVW1Y8Me7+scppxRQ9X3EtgrUKbJh4nd\ngzsmUBW/+shie7/ge/yP30gL8AytdXcN3/MJWtBsVPvdG+RbXxnXQCDn6IR8brXP31UwDfdbMRcf\nFsdXj9QxdUF1zrxpVYwqXDXcOLlqy+KjVkEQoaqXMvsDBG4SBKta/C6LSjhchu1bVJ8ybqGxykY7\nOR1jf7yfkXKj1N9x9a0Kz5HbmKpeEmCsUT1qu0KFq5G8g0K4Fy9GbMrP5xwMfz4XyVPiqCzIj8Ay\nUgsLC4sD8aiM1FRVOTsV+dMQzHO359ljkZkAhjCADZbdr5ldLLbCBDPMjA2c2BVE+4lqnHV6zmxo\nn22wLipFqYZzMeqINhAKZ4Wwi9sH5OJCijPoo1L+81m3zmQWY58Q8w7k+PMzXq+PAMoATKrflwZe\nsWpPeyxYoZFZoGp9RmhmmCTm08H3sp0LR7+phv+xTreII1GEPxaKESwLHnPT3juGDKtVj3PYNxFC\n3vniToJFv391y9v1uI3z6ZNLIiLaK5F4WiKANWdLpSKpTdmVWDBN0pCF4BeyfaUE68eC8RTJMIqZ\nVwgKme8qWH4mOYZIrMkawcNKsVVhqWypBZAx9UJ59gn1gRvUogjQdG4wkHWSAbPG0Yx/S8lIjr9a\n8/H2RjYVod6wOkQfdUhb1In1eyqPfsL7HELuNIO8zq9FIrVZG+vxS/oYLCO1sLCwOBCPykhLtD9e\n3UqvozbhWfDtDUuJbuE3XK6Fdd6hKssO1fAT5U+MMcO5aIlbokZmqfoxFRBQrzFzuRGv2+uLI/R2\nwcLt0GUmch7JsnfvmEnfvOfZ9fkzltk8/0RmNQ8SrSv4Q31VIf3shPflmpn2hH2uyVR8MFrgfCxw\nILp3VNtafEUe/KYVxPoPt8LoTLXyPeq8pqrWo6lfutrwWPUhdVsrOVm64echQWpoPeD9pYVK+wM5\ndAOco6pVmkPmlC3Yb7qGv2yrKgoRjvsUvrmsUgLyhM93W/L5F0hb1MeIvOOTP11c8PNYqN5XOdoo\n5/Ar52CdRrpGRFTh99Wid1VZy/Z1yZZACbYelnyve6rjQIvfrNnehWM8VlXeTPJGH9WbdBLHaMhs\n9Q4p3CcjPtbLp/KeeP4JJ+HMJ6gENxNrcjzmcxmgl5cDOdaDek/t1hLP+BgsI7WwsLA4EI/KSCP4\nLReq2+K798wKvv6OBdCblGeV1pEZb7lgdpplxicns9EA/rHFlte/R8fQ6kz6MeUpr391zfuu0Bny\nySfiRDG1KI2bLAhkjpmNeb0qw0yJYhlxJOusFnzcq1csCjaCciKioMfrPdzz8T97yYVanFBm3LI6\nPgbjo8+Vyn2gbAdWAgazBfsMffE3nVxg5jcReVWRvULV/MWa109bjOtafJwZeiV5PlsR0SnvL1TP\nTA01R1mZHkNyf3s43hxR/gjV1v2RWBi3Oz7ed2+5hm6gCqtUuMYChViMtjxWqgHXO76o/dOn7A8u\nVMpvBktgD2vSJJqY3yKRJLyY7cpSfKy1ifqnpoo939c8VYWHkGpbwS8dozavTqoJkLBjapfqesMh\nGGzg8rFmQx6rl08kQePHX7AffIi2GP2+jHWI91KDwjRvr97y55tX3TqejdpbWFhY/GVhX6QWFhYW\nB+JRTXsT5NkqKcpixWb79T1LUrY7/v90Ko7iEOahaVqnlE2UZRwAur1jcy83rZOV9qEHmVEPzuQK\nLSmURUpnc3YFOA2CE0q98gTyqcszFuw6Rr7TiMyjgQRmNmUndtvKHBWaWgG42+OxyfUX0749QtN+\nMOILGg3lTu5RUctcTYAxN9WgiIjmc1TqgrmcZjKgOQIdxpQrEZCKEjlGiAQNB1kTrd/i2PJcOTWP\njcntrlSrkwxW2vUGZ4nWzU0ug76C/O5hy/upVBuTHCaokeiZXJBQtYuZTI6vvfbJCZvCuvJaiWvM\nYL6nnakvgnjztzH/s/QjwSqY/QVkU7p2QYtqUy3kZA0qTHVJNiRNoVNUetqqXHvjLmjRLG/1wEGi\nq1dfd+sMUY+ULp7xZyXjs8b13t7zdr///a9wHuLiuDy39UgtLCws/qJ4VEbqoSK6rnBfYhbpQepQ\nI7ig5UNJz0geePs4kWW3YBNlxTPVDNKi0UjmiAhSqJcvmFFWkL/o5nd9iOyN8kNXz/e7ABTPmK/e\nMHvOMmEp4zE7r19+ymzz7Wup7m0m1p/9/J/xOU7ZqW+qSenrPiZECK6cqqpLOzSi62HMTs+YzRth\nPRHRySnfa1PDNU1lWZbxPamgX6pjvnm+pyRvHu/bQZe2HfHYZ3tVIQoBjHuYFgvFNns5r9ef8rNS\njHjdZSMsZwOWVYKJpiqNNUOucGkaJiLYMVBV10/Hx9eOuQ/W3ypbrUEHubLigFxR8NjlmW4OacYR\nwnjVqSBNiw+XYd1cJbxURrIIZtii0v5+LymebcMWYwGx/z6VZYblmnqid8bCqL/p1lkiGDybvSEi\nop7qxW4SQ3YpC/q3qCKlreK0UOr+j8AyUgsLC4sD8aiMtDbpnKruY5Xxd2OkgyWoSJ6rdQr4pEL4\n2zxXTruHGp8vnjETPL84xX5knQz9eyIUu3CwfZYJIzVtZx9WfFzfkxn3HH1eHpZcHOG7P7Bkq2lU\nS1is3zs1kiDVMwoz/CfPPyUi1UumPr40Qo0T+KH7iZJxoXX1DMu+eMmdBoaJJByYnlkFevTs9zJW\n+z18aV1fJ0hhlNo99lCrFE7n3IXvW93OENbD7YhZxWu66ZZl18w8Tk7ZinBegDWrZ45CI5viZ2Sp\nGhgYy6iAX9tYKlNlBc36x8dRwtDEIlS/NOTvRqhsXyNRpe7JeBYlP/tG/pQqH2mamu8gnwLTT/eK\ntcJaMH5Y08N5t5UuF8Y3G/XA9FU6rxn3CvENY03udyKiX6BATm/Az4FpxU1E5GE8TcJMr8/P1XAo\n6+giLR/D8Y22hYWFxV8Z7IvUwsLC4kA8qmn/7vfctqHJxKnvIj4QQ9ISw1Tf78Q8MNKm/Y5NyP5A\nTMkAWUbzKZtpI7Q9cFVNxQp1SI1kYr9n57QTiwnjxjynLNYIfqkslWdPeN+m/cn5GVevqms5jy7j\n44rN/9P5ZbdsNmbztm9MepM5opOzj7Ba0I8+5/tQkWpUCNO+j+pLI+TBT1Q1LAfSphyyJZ2lYupf\nei6vPxyhVcVegj1tyePgIfNlDjlZ7YhcZYTjnfRRyaeU43+9/ZaIiIKE93N6yevM1K9hNuWxvZny\ns3d9o1pOo5ZmgWhT0yAjx5Pj16qB3LHAdb/f5sfUJm1bPKtdu2zVgBJBngo2dtVXUrE/lk9lJiAl\npnKa8b3a7k0giX+nmXpPrJY8DhOPx9pRDfbKEseHBLGojOtInsscOf8BXAxxrCu/oRrbkMd8MMTv\nXb1DmuqHx9MyUgsLC4sD8aiM9NUvvyIiokhVXqEEb38Ips0c0CqW4kImVICJrN+L9CGBFMqI4+/e\no7lVJJd2imDRZMrMo0Ll7M1eWK9nakpi5nUaWYaJmoZ9lsukAwSkYpFHLFYsubhDYsHZXLFdNBW7\n+ZrlGCPJDRqDAAAFm0lEQVRIpYYzafHqRI86FP8k+NtffEFERPd7qdS1fcoMxNQhCCK0zy2k+lPr\nQGqGWT5VUpg9GMwOLKUybZmXwmD2S9R8RWXzFwE/T4vdvRwDwa7JkJddXkojs9ffcqWxBQIIUwRU\nRn0JHg5QbehsytKu6WjZLXMQmTRVjhZLXnZzI8e/3R8fI/UdE8yVZ9cYTS1+ma7hXirBgdByu0HQ\nrYmU1MxIkyqMOQJT+VB+Xxk0hzsEqdYb1Ca+l2DR3R1LkjaQVtUq4WUNtmrqoSKmTI6S3JnEEKOq\nDFXthAQJPyO8SwawimNVrzhwf7hdumWkFhYWFgfiUWnQFjNM6chhB31OvXIww6yQMuqqWfH0lNcx\nfsh9LrO9hynGzFTv37K8IVI+zjjhtLCnYCUoY9n5YoiIAsg7hpjFHF/8IyZjrkC9y1v08Zmeyjma\nKlT5mmfcm2/fdMseyj8QEdEVfKRnF+wz/em//EW3Tv9UsfQjwQhJDHFfS1HAXCAlCeBTc0hVnzfK\nF4iqC1eNA3p2GXG4yVmolR82Qf3JASyC5YLZyjoT1lubSmEt3/PpSNjmT376ks/J5X0mqInbU7nH\nramODonadCTWh4NnMyscXA/OtSd+u3H/+OrLBr6pm6usqe5P48/GeGpGivWNEdlq32LL4256PkUV\n/19bjAn810nC6/QhlRsmMmbLJVJD8fDsVDpwBSYbIQ3ZxDf8UDFKfOfh9z1V1dnO52wZnqE3G0Hy\nNlASL91x42OwjNTCwsLiQDxuPVJUgR/3pVboyYTTJR82zET3W55pIjVjmPmtbng2ms5kWX/As1aF\nCOr8gmcV1UaIavg/d0jzMwLc+Vj2M0j47zTndTZKgLvbIcrvMxM6PeeI/GAgXQ4LzJAx6OvyWgTg\nDnJEN+hPVeOKPs9/ItdR/3C9w79KIDIdqGIdPgq6GEbqIy24VVaIqV/pgPX1tL8qxLPhY2xQxzRR\nPa1Cl33MvQBC/JKfnYkjqao1fK0RWGKiBNjPP2O1QV4iQYSQBqrSSE36sEl/LEoRkLueEeQj1Rj1\nTfuqovsk+eGUwr9GeFBTfOAjNS5R8wV+jFq03/GxjsmqbgLY0jd+VDwrofI/VugHFUX82YsNI1U9\nm5ByvFwzM12FEieJ8Yw18HWbVHRP+UjNc+hD5fP04qxbdnnOluoY75zdnhM2eqpLRhKrpmMfgWWk\nFhYWFgfCvkgtLCwsDsSjmvY9mA4nqg3xENGEt/cLrIMalyMx/9/dcmsS04BrNhQTzpiJxhp59oRd\nBeVWqH8K6Uwaspk4hewons27dUIEQ3YQCrtbcWZvUS3o4imb9D/64p/zuttNt87Vtyxt8rFvby/b\nPzyw6Vkjh3gI57YTqupP1fGZ9g2CLb6jxfZ8HR4SDDzYhrWas2FlURiidUsoLpIoREIF2vXWLt/H\n8VBMSR+59qHLJqCpedCQyvHG/e9cPK6Ymx6E+A5ax6x3G5y7vjj+2K45ELZUz0OIamJVW+N8TNKB\ntK8YR0do2mNc3A/kTya4hC9gPn9g2pt1yEgYv19b18POTe6+EfETEfkIMAYI7EUR1umplkJw4c1m\n/NvfqvHY4LdeQf5kju+oMTdJBiH6gr/4RBJmTowM0bQHh6vH1PEgkup0fwqWkVpYWFgcCMfMEBYW\nFhYW/3+wjNTCwsLiQNgXqYWFhcWBsC9SCwsLiwNhX6QWFhYWB8K+SC0sLCwOhH2RWlhYWBwI+yK1\nsLCwOBD2RWphYWFxIOyL1MLCwuJA2BephYWFxYGwL1ILCwuLA2FfpBYWFhYHwr5ILSwsLA6EfZFa\nWFhYHAj7IrWwsLA4EPZFamFhYXEg7IvUwsLC4kDYF6mFhYXFgbAvUgsLC4sDYV+kFhYWFgfCvkgt\nLCwsDoR9kVpYWFgcCPsitbCwsDgQ/w+sh7uiPgujIQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 360x360 with 9 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pDnf_yQxleHx",
        "colab_type": "text"
      },
      "source": [
        "## reproduce the architecture"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9NXBdCgzJyIB",
        "colab_type": "text"
      },
      "source": [
        "Custom resnet variant: ![ResNet18 variant graph](https://296830-909578-raikfcquaxqncofqfm.stackpathdns.com/wp-content/uploads/2019/06/Artboard-1-5.svg)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YS-_x9MUI-N6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def conv(ni, nf, ks=3, stride=1, padding=None, bias=False):\n",
        "    if padding is None: padding = ks//2\n",
        "    return nn.Conv2d(ni, nf, kernel_size=ks, stride=stride, padding=padding, bias=bias)\n",
        "\n",
        "def noop(x): return x\n",
        "  \n",
        "class ResBlock(Module):\n",
        "    def __init__(self, c_in, c_out, stride=1):\n",
        "        self.bn1   = nn.BatchNorm2d(c_in)\n",
        "        self.relu1 = nn.ReLU(True)\n",
        "        self.branch = nn.Sequential(\n",
        "            conv(c_in, c_out, ks=3, stride=stride, padding=1),\n",
        "            nn.BatchNorm2d(c_out),\n",
        "            nn.ReLU(True),\n",
        "            conv(c_out, c_out, ks=3, stride=1, padding=1)\n",
        "        )\n",
        "        projection = (stride != 1) or (c_in != c_out)\n",
        "        self.idconv = noop if not projection else conv(c_in, c_out, ks=1, stride=stride, padding=0)\n",
        "\n",
        "    def forward(self, x):\n",
        "      _out = self.relu1(self.bn1(x))\n",
        "      return self.branch(_out) + self.idconv(_out)\n",
        "\n",
        "class Pool(Module):\n",
        "  def __init__(self, concat=True):\n",
        "    self.concat = concat\n",
        "    self.maxpool = nn.MaxPool2d(4)\n",
        "    if concat:\n",
        "      self.avgpool = nn.AvgPool2d(4)\n",
        "    \n",
        "  def forward(self, x):\n",
        "    if self.concat:\n",
        "      return torch.cat([self.maxpool(x), self.avgpool(x)], 1)\n",
        "    return self.maxpool(x)\n",
        "\n",
        "def make_DAWN_Net(c=64, Block=ResBlock, prep_bn_relu=False, concat_pool=True):\n",
        "  if isinstance(c, int):\n",
        "    c = [c, 2*c, 4*c, 4*c]\n",
        "  layers = [conv(3, c[0], ks=3, stride=1, padding=1)]\n",
        "  if prep_bn_relu:\n",
        "    layers.append(nn.BatchNorm2d(c[0]))\n",
        "    layers.append(nn.ReLU(True))\n",
        "  layers.append(nn.Sequential(\n",
        "    Block(c[0], c[0], 1),\n",
        "    Block(c[0], c[0], 1)\n",
        "  ))\n",
        "  layers.append(nn.Sequential(\n",
        "    Block(c[0], c[1], 2),\n",
        "    Block(c[1], c[1], 1)\n",
        "  ))\n",
        "  layers.append(nn.Sequential(\n",
        "    Block(c[1], c[2], 2),\n",
        "    Block(c[2], c[2], 1)\n",
        "  ))\n",
        "  layers.append(nn.Sequential(\n",
        "    Block(c[2], c[3], 2),\n",
        "    Block(c[3], c[3], 1)\n",
        "  ))\n",
        "  layers.append(nn.Sequential(\n",
        "    Pool(concat=concat_pool),\n",
        "    Flatten(),\n",
        "    nn.Linear(2*c[3] if concat_pool else c[3], 10, bias=True),\n",
        "  ))\n",
        "  return nn.Sequential(*layers)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bDlv591gV7lG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "54b3c785-ebc8-4382-eb50-5c13e12aa57c"
      },
      "source": [
        "model = make_DAWN_Net(); model"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Sequential(\n",
              "  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (1): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (2): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (3): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (4): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(256, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (5): Sequential(\n",
              "    (0): Pool(\n",
              "      (maxpool): MaxPool2d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False)\n",
              "      (avgpool): AvgPool2d(kernel_size=4, stride=4, padding=0)\n",
              "    )\n",
              "    (1): Flatten()\n",
              "    (2): Linear(in_features=512, out_features=10, bias=True)\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YDh6LqLIl4EC",
        "colab_type": "text"
      },
      "source": [
        "TODO: find a way to test if the model is actually idendical at the PyTorch level"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b4zXSIhTl9Ro",
        "colab_type": "text"
      },
      "source": [
        "## reproduce optimizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9D-0FWxc7VM2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "opt = partial(optim.SGD, lr=1e-3, momentum=0.9, weight_decay=5e-4*batch_size, nesterov=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ixxilqyRmGIn",
        "colab_type": "text"
      },
      "source": [
        "TODO: make sure the optimizer is actually identical"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tl2MbkD1mJ5d",
        "colab_type": "text"
      },
      "source": [
        "## create learner"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9dm1VX4lmMsQ",
        "colab_type": "text"
      },
      "source": [
        "TODO: find a way to use `nn.CrossEntropyLoss` as loss function like in Part 1"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hFPuzq329mqP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn = Learner(data, model, metrics=accuracy, loss_func=CrossEntropyFlat(), opt_func=opt)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u9JPIWkAICjc",
        "colab_type": "text"
      },
      "source": [
        "## learning rate schedule\n",
        "\n",
        "lr_schedule from the bag of tricks article:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-2EDNTj6Ejyv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class PiecewiseLinear(namedtuple('PiecewiseLinear', ('knots', 'vals'))):\n",
        "    def __call__(self, t):\n",
        "        return np.interp([t], self.knots, self.vals)[0]\n",
        "\n",
        "sched = PiecewiseLinear([0, 15, 30, 35], [0, 0.1, 0.005, 0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WhuxJLSNFIZc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jzrNsWmBFZgU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        },
        "outputId": "293cf445-ae56-4490-84da-bf4dcf91689b"
      },
      "source": [
        "plt.plot([sched(o) for o in range(35)])"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f22d9cab668>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XtcVXW+//HXhzuCgAiiIgoqiqCA\nSdrF7GpZjpF5SX8zc5qZzqNmppuppd3M7KppVjN1znSmqU7T5D3TtLSyycrJROUiXvEKiIqKqKhc\nv78/9nYOkcpWNqx9+Twfjx5u1l5783aFb/Ze67PXEmMMSimlvIOP1QGUUkq1HC19pZTyIlr6Sinl\nRbT0lVLKi2jpK6WUF9HSV0opL6Klr5RSXkRLXymlvIiWvlJKeRE/qwM0FBUVZeLj462OoZRSbmX9\n+vWHjTHRja3ncqUfHx9PVlaW1TGUUsqtiMheR9bT3TtKKeVFtPSVUsqLaOkrpZQX0dJXSikvoqWv\nlFJexKHSF5EhIrJNRApEZPI57h8kIhtEpEZERja4724R2WH/725nBVdKKXXxGi19EfEF3gRuBZKB\nsSKS3GC1fcBvgH80eGwk8AwwAOgPPCMibZoeWyml1KVw5JV+f6DAGLPLGFMFzAEy669gjNljjMkF\n6ho89hbgC2PMUWNMGfAFMMQJuZVy2Mr8A+w+XGF1DKVcgiOlHwsU1vu6yL7MEQ49VkTuFZEsEckq\nLS118KmVatzq7aXc+8F6fv3OWo6fqbY6jlKWc4kDucaYt40xGcaYjOjoRj9FrJRDyk9XM2lhLrER\nwZSUn2HK4k1WR1LKco6UfjEQV+/rTvZljmjKY5Vqkuc+3cyhE5W8+cvLeOiGRBZn72fxRv3xU97N\nkdJfBySKSIKIBABjgCUOPv8K4GYRaWM/gHuzfZlSzeqLzQdZsL6IP17XjfS4CO6/vhsZXdrw1OJN\nFB49ZXU8pSzTaOkbY2qAB7CV9RZgnjEmX0SmicjtACJyuYgUAaOAv4hIvv2xR4HnsP3iWAdMsy9T\nqtkcraji8UV59OoQxoM3JALg5+vD7LvSEeDhORupqW04c6CUdxBjjNUZfiIjI8PoWTZVU9z/jw2s\nzD/AkgcG0qtD2E/u+yS7mIfnZPPwjYk8MriHRQmVcj4RWW+MyWhsPZc4kKuUsyzN2c+y3BLG3dTj\nZ4UPkJkey519Y/nTqh1k7dE3ncr7aOkrj3HoxBme/mQTaXER3Deo63nXezYzhU5tWvHwnGwd41Re\nR0tfeQRjDI8vzON0VS2zRqXh53v+H+3WQf68PiadA8fP8LSOcSovo6WvPMKC9UV8tfUQjw1Jonu7\n0EbX79u5DeNuTOST7P18vLGoBRIq5Rq09JXbKz52mmlLNzMgIZLfXhXv8OP+eH13+sdH8vTifPYd\n0TFO5R209JVbq6szTFqQS60xzByVho+POPxYXx9h9ph0RGDcXB3jVN5BS1+5tQ/X7uW7gsM8NTSZ\nuMhWF/342IhgXhzehw37jvHGVzuaIaFSrkVLX7mtvUcqeHH5Vgb1iGZs/7jGH3Aew9I6MuKyTvz5\n6wJ+3K1jnMqzaekrt1RbZ5g4Pwd/X2HGiFREHN+tcy7PZqYQF9mKR+ZmU35axziV59LSV27pb9/t\nZt2eMqbenkL78KAmP19ooB+v3WUb43xq8SZc7ZPqSjmLlr5yOzsOnuCVldu4OTmG4X0dvbRD4/p2\nbsMjNyWyNGc/izbo2TiVZ9LSV26luraOCfNzCA3044XhfZq8W6ehP1zXnf4JkUz5ZBN7j+jVtpTn\n0dJXbuW//rmT3KJynr+jN9GtA53+/L4+wuy70vH1ER6ek021jnEqD6Olr9zGpuJy3vhqB5npHbmt\nT4dm+z6xEcG8eGcfsgt1jFN5Hi195RYqa2qZMC+HyJAAnr09pdm/3y9SOzKyXyfe1DFO5WG09JVb\neO3LHWw7eILpI1KJaBXQIt9z6u06xqk8j5a+cnnr95bxl292MubyOK5Patdi3zc00I/Xx/Tl4PEz\nPPlxno5xKo+gpa9c2umqWibOz6FDeDBPDu3V4t8/PS6CRwb34NPcEhbqGKfyAFr6yqXNWLGV3Ycr\neGVkKq2D/C3J8PtruzEgIZJnPtnEnsM6xqncm5a+cllrdh7m3e/38Jur4rmqe5RlOX4yxjlXxziV\ne9PSVy7pZGUNj87PJSEqhElDkqyOQ8eIYF66M5WcwmO8/qWOcSr3paWvXNILyzZTUn6amaNSCQ7w\ntToOAENTOzCqXyfe/GcBP+w6YnUcpS6Jlr5yOV9vO8RHPxZy76Bu9OsSaXWcn5h6ewpdzo5xntIx\nTuV+tPSVSzl2qopJC3LpERPKI4MTrY7zMyH2Mc7SE5U8oWOcyg1p6SuXMnVJPkcrqnh1dDqBfq6x\nW6ehNPsY57K8Euav14uqK/eipa9cxmd5JSzO3s+DNyTSOzbc6jgXdHaMc+qSfHbrGKdyI1r6yiUc\nPlnJk4s30Sc2nD9e383qOI06O8bp7+vDuDkbdYxTuQ0tfWU5YwxPLMrj5JkaZo1Ow9/XPX4sO0YE\n8/KdfcgpKmf2F9utjqOUQ9zjX5fyaIuzi1m5+SATbu5Bj5jWVse5KLf26cBdGXH81zc7+ddOHeNU\nrk9LX1mqpPw0Uz7Jp1+XNvznNV2tjnNJpgxLJr5tCOPnZXPsVJXVcZS6IIdKX0SGiMg2ESkQkcnn\nuD9QROba718rIvH25f4i8r6I5InIFhF53LnxlTszxjBpYR41tYZZo9Lw9XHupQ9bim2MM13HOJVb\naLT0RcQXeBO4FUgGxopIcoPV7gHKjDHdgdnAdPvyUUCgMaYP0A+47+wvBKU++rGQ1dtLefy2JOKj\nQqyO0ySpnSKYcHNPlucdYH6WjnEq1+XIK/3+QIExZpcxpgqYA2Q2WCcTeN9+ewFwo9iuWG2AEBHx\nA4KBKuC4U5Irt7bvyCmeX7aZq7u35VcDulgdxynuG9SVK7u2ZepSHeNUrsuR0o8FCut9XWRfds51\njDE1QDnQFtsvgAqgBNgHzDTG6LXnvFxdneHRBTn4iDBjZBo+brpbpyEfH+HVu2zTRw/P2UhVjY5x\nKtfT3Ady+wO1QEcgAZggIj87Wici94pIlohklZaWNnMkZbV31+xh7e6jTPlFMrERwVbHcaoO4cFM\nH9GH3KJyZn+pY5zK9ThS+sVAXL2vO9mXnXMd+66ccOAI8P+Az40x1caYQ8D3QEbDb2CMedsYk2GM\nyYiOjr74v4VyGztLTzLj863cmNSOURmdrI7TLIb07sCYy+P47292smbnYavjKPUTjpT+OiBRRBJE\nJAAYAyxpsM4S4G777ZHAKmMbYdgH3AAgIiHAFcBWZwRX7qemto4J83IIDvDlpTv7YDvs45mmDEsm\noW0I4+fm6BincimNlr59H/0DwApgCzDPGJMvItNE5Hb7au8AbUWkABgPnB3rfBMIFZF8bL883jXG\n5Dr7L6Hcw19W7yK78BjTMnvTLizI6jjNqlWA7WycRyoqmbxQxziV6/BzZCVjzHJgeYNlU+rdPoNt\nPLPh406ea7nyPltKjvPal9sZ2qcDw1I7WB2nRfTpFM6Em3vy8mdbmbuukDH9O1sdSSn9RK5qflU1\ndYyfl0N4sD/P3dHbo3frNHTvNV25qltbnl26mZ2lJ62Oo5SWvmp+f1q1gy0lx3npzlQiQwKsjtOi\nfHzEdm0Afx/GzcnWMU5lOS191axyCo/x1j93MuKyTgxOjrE6jiXahwcxfUQqecXlzPpim9VxlJfT\n0lfN5kx1LRPm59CudSBThjU8c4d3uSWlPWP7d+bt1btYU6BjnMo6Wvqq2cxauY2CQyeZPiKV8GB/\nq+NY7ulf9CIhKoRH5mVTVqFjnMoaWvqqWfy4+yh//W43v7qiM4N66AfuwDbG+caYvhytqGLyolwd\n41SW0NJXTldRWcPE+TnEtWnF47f2sjqOS+kdG86jt/RkRf5B5qwrbPwBSjmZlr5yupc/20ph2Slm\njkojJNChj4J4lf8c2JWB3aOYpmOcygJa+sqpvt1Rygc/7OWeqxPonxBpdRyX5OMjzBqdRpC/no1T\ntTwtfeU0x89U89iCXLpFhzDxlp5Wx3FpMWG2Mc5NxceZtVLHOFXL0dJXTjNt6WYOnahk1uh0gvx9\nrY7j8m5Oac//G9CZv6zexfc6xqlaiJa+coovNh9kwfoi/nhdN9LjIqyO4zaeHppMt2jbRdV1jFO1\nBC191WRHK6p4fFEevTqE8eANiVbHcSvBAb68bh/jnLRQxzhV89PSV0329CebKD9dxauj0wjw0x+p\ni9U7NpzHbkli5eaD/OPHfVbHUR5O/4WqJlmas59luSWMu6kHvTqEWR3Hbd0zMIFrEqN47tPNFBw6\nYXUc5cG09NUlO3T8DE9/son0uAjuG/SzSx+ri+DjI8wclUawvy8PfZRNZU2t1ZGUh9LSV5fEGMPj\ni/I4XVXLrNFp+Pnqj1JTxYQFMWNkGptLjjNzhY5xquah/1LVJVmwvoivth7isSFJdIsOtTqOxxic\nHMMvB3Tmf77dzbc7Sq2OozyQlr66aMXHTjNt6WYGJETy26virY7jcZ4amkz3dqFMmJfDUR3jVE6m\npa8uSl2dYdKCXGqNYeaoNHx8vOfShy3FNsaZzrFTtk846xinciYtfXVRPly7l+8KDvPU0GTiIltZ\nHcdjpXQM57EhPflyy0E+XKtjnMp5tPSVw/YcruDF5VsZ1COasf3jrI7j8X53tW2M8/llOsapnEdL\nXzmkts4wcX4O/r7CjBGpiOhunebm4yPMGpVGqwA/HtQxTuUkWvrKIe98t4usvWU8m5lC+/Agq+N4\njXb2s3FuKTnOK5/rGKdqOi191ajtB08wc8V2bkmJ4Y70WKvjeJ3ByTH86orO/PW73azermOcqmm0\n9NUFVdfWMWFeDqFBfrwwvI/u1rHIk7fZxzjn53DkZKXVcZQb09JXF/TW1zvJKy7nhTt6ExUaaHUc\nrxUc4MsbY/pSfqpaz8apmkRLX53XpuJy/rRqB5npHbm1Twer43i95I5h9jHOQ/xdxzjVJdLSV+dU\nWVPL+HnZRIYE8OztKVbHUXa/uzqBQT2ief7Tzew4qGOc6uJp6atzmv3FDrYfPMn0EalEtAqwOo6y\ns52NM5XQQD8emqNjnOriaemrn1m/9yhvr97JmMvjuD6pndVxVAPtWgcxY6RtjHOGjnGqi+RQ6YvI\nEBHZJiIFIjL5HPcHishc+/1rRSS+3n2pIvIvEckXkTwR0SFvF3aqqoYJ83LoEB7Mk0N7WR1HnceN\nvWL4jyu78I6OcaqL1Gjpi4gv8CZwK5AMjBWR5Aar3QOUGWO6A7OB6fbH+gF/B35vjEkBrgOqnZZe\nOd2Mz7ex58gpXhmVSusgf6vjqAt44rZe9IjRMU51cRx5pd8fKDDG7DLGVAFzgMwG62QC79tvLwBu\nFNtA981ArjEmB8AYc8QYozshXdSagsO8t2YPv7kqnqu6RVkdRzUiyN92UfXy03o2TuU4R0o/Fiis\n93WRfdk51zHG1ADlQFugB2BEZIWIbBCRx871DUTkXhHJEpGs0lJ9q2qFE2eqeXRBLglRIUwakmR1\nHOWgXh3CmDwkia+2HuLvP+y1Oo5yA819INcPGAj80v7ncBG5seFKxpi3jTEZxpiM6OjoZo6kzuWF\nZVsoKT9tu05rgK/VcdRF+O3V8VzbI5rnl21hu45xqkY4UvrFQP3z6HayLzvnOvb9+OHAEWzvClYb\nYw4bY04By4HLmhpaOdfXWw8xZ10h913bjX5d2lgdR10kEdtF1UMD/Xjoo42cqdY9qOr8HCn9dUCi\niCSISAAwBljSYJ0lwN322yOBVca2g3EF0EdEWtl/GVwLbHZOdOUMx05VMWlhLj1jWjPupkSr46hL\nFN06kFdGpbL1wAkd41QX1Gjp2/fRP4CtwLcA84wx+SIyTURut6/2DtBWRAqA8cBk+2PLgFex/eLI\nBjYYY5Y5/6+hLtUzS/I5WlHFrNFpBPrpbh13dkNSDHdf2YW/fb+bf247ZHUc5aLE1Y74Z2RkmKys\nLKtjeIXP8kr4w4cbeOSmHjysr/I9wpnqWjL//D1HKqr4fNw1epI8LyIi640xGY2tp5/I9VKHT1by\n5OJN9IkN54/Xd7M6jnKSIH9fXh+bzvEz1Tw6P0fHONXPaOl7IWMMTyzK42RlDbNGp+Hvqz8GniSp\nfRhP3JrE19tK+d9/6Rin+in91+6FFmcXs3LzQSbe3IMeMa2tjqOawd1XxXN9z2heWL6FbQd0jFP9\nHy19L1NSfpopn+ST0aUN9wzsanUc1UxEhBkj0wgL0jFO9VNa+l7EGMOkhXnU1BpmjkrD10cvfejJ\nbGOcaWw7eIKXP9tqdRzlIrT0vchHPxayenspT9yWRHxUiNVxVAu4vmc7fnNVPO+t2cPXOsap0NL3\nGoVHT/HCss0M7B7FLwd0sTqOakGTb00iqX1rHp2fQ+kJPRunt9PS9wJ1dYaJ83PwEWH6yFR8dLeO\nVzl7Ns7jZ2p4bIGOcXo7LX0v8N6aPazdfZSnhyUTGxFsdRxlgZ7tW/Pkbb34elsp76/ZY3UcZSEt\nfQ+3s/Qk0z/fyo1J7RjVr5PVcZSF/uPKLtyQ1I4XP9vK1gPHrY6jLKKl78FqauuYMC+H4ABfXrqz\nD7br2ihvZRvjTCUsyJ+HP8rWMU4vpaXvwf6yehfZhcd4LrM37cL00sQKokIDmTkqVcc4vZiWvofa\nUnKc177cztDUDgxL62h1HOVCruvZjt9ebRvjXLX1oNVxVAvT0vdAVTV1jJ+XQ3hwAM9l9rY6jnJB\nk4acHePM5dCJM1bHUS1IS98D/WnVDraUHOelO/sQGRJgdRzlgoL8fXljbF9OVtbw6Pxc6up0jNNb\naOl7mOzCY7z1z52MuKwTg5NjrI6jXFiPmNY8ObQX32wv5T0d4/QaWvoe5Ex1LRPmZdOudSBThiVb\nHUe5gV9f0YUbk9rx8mdb2VKiY5zeQEvfg8xcsY2dpRVMH5FKeLC/1XGUGxD7p7TDgv31bJxeQkvf\nQ6zddYR3vt/Nr67ozKAe0VbHUW4kKjSQWaPT2HHoJC8u32J1HNXMtPQ9QEVlDRMX5BDXphWP39rL\n6jjKDV3bI5rfXZ3A//5rL19t0TFOT6al7wFeXL6ForLTzByVRkign9VxlJt6bEhP2xjnAh3j9GRa\n+m7um+2lfLh2H/dcnUD/hEir4yg3FuTvy5/G9rW9c9QxTo+lpe/Gyk9XM2lBLt3bhTLxlp5Wx1Ee\nIDGmNU8N7cXq7aW8q2OcHklL341NW7qZ0pOVzBqVRpC/r9VxlIf41RVduKlXO6Z/tpXN+3WM09No\n6buplfkHWLihiD9e1420uAir4ygPIiK2sd9W/jw8R8c4PY2Wvhs6WlHFEx/n0atDGA/ekGh1HOWB\n2oYG8qp9jPOFZTrG6Um09N2MMYanFudRfrqaV0enEeCn/wtV87gmMZr/HJjABz/s5cvNOsbpKbQx\n3MzS3BKW5x1g3E096NUhzOo4ysM9OqQnyR3CeGxhLoeO6xinJ9DSdyOHjp/h6cWbSI+L4L5BXa2O\no7xAoJ8vb4xN51RVDRPm5+gYpwfQ0ncTxhgmL8rjTHUts0an4eer/+tUy+jerjVPDU3m2x2H+dv3\nu62Oo5rIoeYQkSEisk1ECkRk8jnuDxSRufb714pIfIP7O4vISRGZ6JzY3md+VhGrth5i0pAkukWH\nWh1HeZlfDujM4OQYZny+jfz95VbHUU3QaOmLiC/wJnArkAyMFZGG5+29BygzxnQHZgPTG9z/KvBZ\n0+N6p6KyU0z7dDMDEiL5zVXxVsdRXujsGGdEK9vZOE9X6Rinu3LklX5/oMAYs8sYUwXMATIbrJMJ\nvG+/vQC4UUQEQETuAHYD+c6J7F3q6gyPLcjFGMPMUWn4+IjVkZSXigwJ4NXR6ewsreD5ZZutjqMu\nkSOlHwsU1vu6yL7snOsYY2qAcqCtiIQCk4Bnmx7VO/197V7W7DzCU79IJi6yldVxlJcbmBjFvYO6\n8uHafazMP2B1HHUJmvto4FRgtjHm5IVWEpF7RSRLRLJKS0ubOZL72H24gpeWb+XaHtGMuTzO6jhK\nATDx5p6kdAxj0sJcDuoYp9txpPSLgfqN08m+7JzriIgfEA4cAQYAM0RkDzAOeEJEHmj4DYwxbxtj\nMowxGdHRegEQgNo6w8T5Ofj72val2veWKWW5AD8fXh/Tl9PVtUyYp2Oc7saR0l8HJIpIgogEAGOA\nJQ3WWQLcbb89ElhlbK4xxsQbY+KB14AXjTF/dlJ2j/bXb3exfm8Zz2am0D48yOo4Sv1E93ahTPlF\nCt8VHOad73SM0500Wvr2ffQPACuALcA8Y0y+iEwTkdvtq72DbR9+ATAe+NlYp3Lc9oMnmLVyO7ek\nxHBHesPDJ0q5hrH947glJYYZK7ayqVjHON2FGONab80yMjJMVlaW1TEsU11bx/C3vmf/sTOsfGQQ\nUaGBVkdS6rzKKqoY8vpqQgP9+PTBawgO0FN8W0VE1htjMhpbTz/W6WLe+nonm4qP8+Lw3lr4yuW1\nsY9x7jpcwXM6xukWtPRdyKbicv60agd3pHdkSO8OVsdRyiFXd4/i3mu68o+1+1ihY5wuT0vfRVTW\n1DJ+XjZtQwN49vbeVsdR6qJMuLknvWPDmKxjnC5PS99FzP5iB9sPnuRl+xWLlHInZ8c4z1TXMX5e\nto5xujAtfRewfu9R3l69k7H947i+Zzur4yh1SbpFhzJlWDLfFxzhr9/tsjqOOg8tfYudrqpl4vxc\nOkYE8+TQhuexU8q9jLncNsb5yoptOsbporT0LTb9863sPlzBKyPTCA30szqOUk0iIrx8ZyptQwJ5\naM5GTlXVWB1JNaClb6E1Ow/z3po9/PbqeK7s1tbqOEo5hW2MM43dhyt47lO9qLqr0dK3yIkz1Tw6\nP5euUSE8dkuS1XGUcqqrukdx36BufPTjPj7fpGOcrkRL3yIvLNtCSflpZo5O008xKo80fnAP+sSG\nM3lRLgfKdYzTVWjpW+DrrYeYs66Q31/bjcs6t7E6jlLNwjbGmU5ldR0T5usYp6vQ0m9hx05VMWlh\nLkntW/PwTYlWx1GqWXWNDmXq7bYxzv/5Vsc4XYGWfgt7Zkk+RyuqmDkqjUA/3a2jPN/ojDhu7d2e\nmSt1jNMVaOm3oOV5JXySvZ+Hbkykd2y41XGUahEiwkt39iEqNJCHPtIxTqtp6beQ0hOVPLV4E6md\nwvnDdd2sjqNUi4poZTsb5+4jFTz3qZ6N00pa+i3AGMOTH+dxsrKGWaPS8PfVza68z5Xd2vL7a7vx\n0Y+FfJZXYnUcr6Xt0wI+3ljMys0HefTmniTGtLY6jlKWeeSmHqR2CmfyojxKyk9bHccraek3s5Ly\n0zyzJJ+MLm343cAEq+MoZamzZ+Osrq1j/NwcanWMs8Vp6TcjYwyPLcilptYwc1Qavj5idSSlLJcQ\nFcLUYSn8a9cR3l6tY5wtTUu/Gf3jx318u+MwT9yWRHxUiNVxlHIZozI6cVuf9sxauY3comNWx/Eq\nWvrNZN+RU7ywbAsDu0fxywFdrI6jlEsREV4ankp060AenpNNRaWOcbYULf1mUFdnmLggB18Rpo9M\nxUd36yj1M+Gt/Jl9Vzp7jlQwbamOcbYULf1m8O6aPfy4+yhThiUTGxFsdRylXNYVXdvyh2u7MTdL\nxzhbipa+kxUcOsmMz7dyU692jOzXyeo4Srm8Rwb3IM0+xrn/mI5xNjctfSeqqa1jwvwcggN8efHO\nPojobh2lGuPvW2+Mc162jnE2My19J/rL6l3kFB7juczetGsdZHUcpdxGfFQIU29P4YddR/nL6p1W\nx/FoWvpOsnn/cV77cjtDUzswLK2j1XGUcjuj+nViaGoHXl25nZxCHeNsLlr6TlBVY3tbGh4cwHOZ\nva2Oo5RbEhFevKMP7VoHMm6ujnE2Fy19J3jjqx1sPXCCl+7sQ2RIgNVxlHJb4a38edU+xvns0nyr\n43gkLf0m2rivjLf+WcDIfp0YnBxjdRyl3N4VXdvyx+u6MS+riOU6xul0WvpNcKa6lgnzc2gfFsSU\nYclWx1HKY4y7qQdpcRFMXpirY5xO5lDpi8gQEdkmIgUiMvkc9weKyFz7/WtFJN6+fLCIrBeRPPuf\nNzg3vrVeWbGNXaUVzBiZRliQv9VxlPIY/r4+vH5XOrV1hkfm6hinMzVa+iLiC7wJ3AokA2NFpOHL\n2nuAMmNMd2A2MN2+/DAwzBjTB7gb+MBZwa22dtcR/vb9bn59RRcGJkZZHUcpj3N2jHPt7qP89zc6\nxuksjrzS7w8UGGN2GWOqgDlAZoN1MoH37bcXADeKiBhjNhpj9tuX5wPBIhLojOBWqqisYeKCHDpH\ntmLyrUlWx1HKY420j3HO/kLHOJ3FkdKPBQrrfV1kX3bOdYwxNUA50LbBOiOADcaYyobfQETuFZEs\nEckqLS11NLtlXly+haKy07wyMo2QQD+r4yjlsc6OccaEBfHwnI06xukELXIgV0RSsO3yue9c9xtj\n3jbGZBhjMqKjo1si0iX7ZnspH67dx38OTKB/QqTVcZTyeGfPxrnv6CmmLtExzqZypPSLgbh6X3ey\nLzvnOiLiB4QDR+xfdwI+Bv7DGOPWO+bKT1czaUEu3duFMuHmnlbHUcpr9E+I5P7ruzN/fRHLcnWM\nsykcKf11QKKIJIhIADAGWNJgnSXYDtQCjARWGWOMiEQAy4DJxpjvnRXaKs8uzaf0ZCWzRqUR5O9r\ndRylvMpDNyaSHhfB44tyKdYxzkvWaOnb99E/AKwAtgDzjDH5IjJNRG63r/YO0FZECoDxwNmxzgeA\n7sAUEcm2/9fO6X+LFrAi/wCLNhRz/3XdSIuLsDqOUl7HdjZOHeNsKjHGtTZcRkaGycrKsjrGTxw5\nWcktr62mXesgFt9/NQF++pk2payycH0RE+bn8OgtPbn/+u5Wx3EZIrLeGJPR2HraXo0wxvD0J5so\nP13Nq3elaeErZbE7L4tlWFpHZn+xnWwd47xo2mCNWJpbwvK8AzwyuAdJ7cOsjqOU1xMRnr+j97/H\nOE/qGOdF0dK/gEPHz/D04k2pGcTWAAALmklEQVT07RzBvdd0tTqOUsouPNif18akU3j0FM98omOc\nF0NL/zyMMUxelEdlTS2zRqXh56ubSilXcnl8JA9c352FG4pYmrO/8QcoQEv/vOZnFbFq6yEmDUmi\na3So1XGUUufw0I2J9O0cwRMf51FUdsrqOG5BS/8cispOMe3TzVzRNZK7r4y3Oo5S6jz8fH14/a6+\nGAPj5+boGKcDtPQbqKszPLYgF2MMr4xMw8dHrI6klLqAzm1bMS0zhR/3HOWtrwusjuPytPQb+OCH\nvazZeYSnfpFMXGQrq+MopRwwvG8smekdee2rHWzYV2Z1HJempV/P7sMVvPTZFq7rGc2Yy+Maf4BS\nyiWICM/d0ZsO4UGMm5PNiTPVVkdyWVr6drV1hgnzsgn082X6iFREdLeOUu4kLMif1+5Kp6jsFM/o\n2TjPS0vf7q/f7mLDvmM8e3sKMWFBVsdRSl2CjPhIHrwhkUUbilmiY5znpKUPbD94glkrtzMkpT2Z\n6R2tjqOUaoIHb+hOvy5teHKRjnGei9eXfnVtHePnZdM6yI/nh/fW3TpKuTk/Xx9euysdgEfmZlNT\nW2dxItfi9aX/5tcFbCo+zgvDexMV6vaX71VKAXGRrXjujt6s21PGW/9062s3OZ1Xl35eUTl/XlXA\n8L6xDOndweo4SiknuqNvLHekd+T1r3awfq+OcZ7ltaV/prqWCfOzaRsawNRhKVbHUUo1g2lnxzjn\nbtQxTjuvLf3ZX25n+8GTTB+RSngrf6vjKKWaQViQP6+PSWf/sTNM0bNxAl5a+uv3HuXt1bsY2z+O\n63q65dUblVIO6tclkgdv6M7HG4v5JLvY6jiW87rSP1VVw4R5OcRGBPPk0GSr4yilWsAD13cno0sb\nnvp4E4VHvXuM0+tKf/pnW9lz5BSvjEwjNNDP6jhKqRbg5+vDbPsY5zgvH+P0qtJfU3CY9/+1l99e\nHc+V3dpaHUcp1YLiIlvx/PDerN9bxp+9+GycXlP6J85U8+iCXLpGhfDYLUlWx1FKWSAzPZbhfWN5\n46sdrN971Oo4lvCa0n/+0y2UlJ9m5ug0ggN8rY6jlLLItMwUYtsE8/CcbI574RinV5T+qq0HmZtV\nyH3XduOyzm2sjqOUslDrIH9eu6svJeVn+N2763j3+93kFB6jqsY79vN7/JHMsooqJi3Mo2dMa8bd\nlGh1HKWUC+jXpQ3P3p7Cn1cV8OzSzQAE+vnQJzacvp0juKxzG/p2bkP7cM87464Y41rXlMzIyDBZ\nWVlOe76HPtrI8rwSFt9/Nb1jw532vEopz1BSfpoNe4+xcV8ZG/aVsan4OFX26Z6O4UH07dzG9oug\nSxtSOoYR6Oeau4dFZL0xJqOx9Tz6lf6y3BKW5Oxn/OAeWvhKqXPqEB7M0NRghqbazr9VWVPL5v3H\n2bjvGBv2lbFx3zGW5ZUAEODrQ0psGH3j2nBZlwj6dm5Dx/Agtzo7r8e+0i89UcnNs78hLrIVC/9w\nFf6+XnH4QinVDA4eP8PGff/3biC3qJxK+zGAmLDAn/wS6BMbTpB/y78b8OpX+sYYHl+UR0VVLbNG\npWnhK6WaJCYsiCG92zOkd3vAdh2OLSU/fTfwef4BAPx8hOSOYfbjArbjA53aBLvMuwGPLP1FG4r5\ncstBnrytF4kxra2Oo5TyMP6+PqR2iiC1UwR3XxUP2PYuZBee/SVQxtx1hby3Zg8AUaGB9O0c8e9f\nAqmdwmkVYE39OvRdRWQI8DrgC/zVGPNyg/sDgf8F+gFHgLuMMXvs9z0O3APUAg8ZY1Y4Lf05lJSf\nZurSfC6Pb8PvBiY057dSSql/i24dyODkGAYnxwBQU1vHtoMn2LDvGBv3lrGx8BhfbD4IgK+PkNS+\n9U/eDXRp26pF3g00Wvoi4gu8CQwGioB1IrLEGLO53mr3AGXGmO4iMgaYDtwlIsnAGCAF6Ah8KSI9\njDG1zv6LgG23zmMLcqmpNcwclYavj2u8nVJKeR8/Xx9SOoaT0jGcX1/RBYCjFVVkF5b9e7fQxxuL\n+eCHvQC0aeXP6MvjePzWXs2by4F1+gMFxphdACIyB8gE6pd+JjDVfnsB8Gex/crKBOYYYyqB3SJS\nYH++fzkn/k/948d9fLvjMM/d0ZsubUOa41sopdQliwwJ4IakGG5Isr0bqK0z7Dh0wvZLYG8ZUSHN\nf8lWR0o/Fiis93URMOB86xhjakSkHGhrX/5Dg8fGXnLaC9h35BQvLNvCNYlR/GpA5+b4Fkop5VS2\n3TxhJLUPY2z/luktlxhrEZF7RSRLRLJKS0sv6TnqjKFflzZMH5HqMkfJlVLK1ThS+sVAXL2vO9mX\nnXMdEfEDwrEd0HXksRhj3jbGZBhjMqKjox1PX098VAgf3DOAjhHBl/R4pZTyBo6U/jogUUQSRCQA\n24HZJQ3WWQLcbb89ElhlbJ/6WgKMEZFAEUkAEoEfnRNdKaXUxWp0n759H/0DwApsI5t/M8bki8g0\nIMsYswR4B/jAfqD2KLZfDNjXm4ftoG8NcH9zTe4opZRqnMeehkEppbyJo6dhcIkDuUoppVqGlr5S\nSnkRLX2llPIiWvpKKeVFtPSVUsqLuNz0joiUAnub8BRRwGEnxWkJ7pYXNHNLcbfM7pYXPCtzF2NM\no59udbnSbyoRyXJkbMlVuFte0Mwtxd0yu1te8M7MuntHKaW8iJa+Ukp5EU8s/betDnCR3C0vaOaW\n4m6Z3S0veGFmj9unr5RS6vw88ZW+Ukqp8/CY0heRISKyTUQKRGSy1XkcISJ7RCRPRLJFxCXPMici\nfxORQyKyqd6ySBH5QkR22P9sY2XGhs6TeaqIFNu3dbaI3GZlxvpEJE5EvhaRzSKSLyIP25e77Ha+\nQGZX3s5BIvKjiOTYMz9rX54gImvt3THXfgp5y10g73sisrveNk6/qCc2xrj9f9hO+bwT6AoEADlA\nstW5HMi9B4iyOkcjGQcBlwGb6i2bAUy2354MTLc6pwOZpwITrc52nrwdgMvst1sD24FkV97OF8js\nyttZgFD7bX9gLXAFMA8YY1/+38AfrM7aSN73gJGX+rye8kr/3xdvN8ZUAWcv3q6ayBizGts1EurL\nBN63334fuKNFQzXiPJldljGmxBizwX77BLAF27WkXXY7XyCzyzI2J+1f+tv/M8ANwAL7cpfZzhfI\n2ySeUvrnuni7S/8A2hlgpYisF5F7rQ5zEWKMMSX22weAGCvDXIQHRCTXvvvHZXaV1Cci8UBfbK/q\n3GI7N8gMLrydRcRXRLKBQ8AX2PYQHDPG1NhXcanuaJjXGHN2G79g38azRSTwYp7TU0rfXQ00xlwG\n3ArcLyKDrA50sYztvac7jID9F9ANSAdKgFnWxvk5EQkFFgLjjDHH69/nqtv5HJldejsbY2qNMenY\nrtfdH0iyONIFNcwrIr2Bx7HlvhyIBCZdzHN6Suk7dAF2V2OMKbb/eQj4GNsPoTs4KCIdAOx/HrI4\nT6OMMQft/4DqgP/Bxba1iPhjK88PjTGL7ItdejufK7Orb+ezjDHHgK+BK4EIETl76ViX7I56eYfY\nd60ZY0wl8C4XuY09pfQduXi7SxGREBFpffY2cDOw6cKPchlLgLvtt+8GPrEwi0POlqfdcFxoW4uI\nYLvO9BZjzKv17nLZ7Xy+zC6+naNFJMJ+OxgYjO1YxNfASPtqLrOdz5N3a70XAoLt+MNFbWOP+XCW\nfTTsNf7v4u0vWBzpgkSkK7ZX92C7QP0/XDGziHwEXIftzH4HgWeAxdgmHjpjOyPqaGOMyxw4PU/m\n67DtcjDYpqbuq7e/3FIiMhD4FsgD6uyLn8C2j9wlt/MFMo/FdbdzKrYDtb7YXvDOM8ZMs/9bnINt\nV8lG4Ff2V9GWukDeVUA0tumebOD39Q74Nv68nlL6SimlGucpu3eUUko5QEtfKaW8iJa+Ukp5ES19\npZTyIlr6SinlRbT0lVLKi2jpK6WUF9HSV0opL/L/AS+8x3TLH6X4AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zbLvQmb5H97t",
        "colab_type": "text"
      },
      "source": [
        "So this is basically the one cycle scheduler.\n",
        "\n",
        "TODO: make sure it is actually the learning rate scheduler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_6sOwvl1mdkV",
        "colab_type": "text"
      },
      "source": [
        "## half precision training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dhy7_bcu7JX6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "75863096-7730-400a-e8cc-639a0e882e52"
      },
      "source": [
        "learn.to_fp16()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Learner(data=ImageDataBunch;\n",
              "\n",
              "Train: LabelList (50000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "horse,horse,horse,horse,horse\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Valid: LabelList (10000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "horse,horse,horse,horse,horse\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Test: None, model=Sequential(\n",
              "  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (1): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (2): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (3): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (4): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(256, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (5): Sequential(\n",
              "    (0): Pool(\n",
              "      (maxpool): MaxPool2d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False)\n",
              "      (avgpool): AvgPool2d(kernel_size=4, stride=4, padding=0)\n",
              "    )\n",
              "    (1): Flatten()\n",
              "    (2): Linear(in_features=512, out_features=10, bias=True)\n",
              "  )\n",
              "), opt_func=functools.partial(<class 'torch.optim.sgd.SGD'>, lr=0.001, momentum=0.9, weight_decay=0.064, nesterov=True), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7f22dcee8e18>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/root/.fastai/data/cifar10'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[MixedPrecision\n",
              "learn: Learner(data=ImageDataBunch;\n",
              "\n",
              "Train: LabelList (50000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "horse,horse,horse,horse,horse\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Valid: LabelList (10000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "horse,horse,horse,horse,horse\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Test: None, model=Sequential(\n",
              "  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (1): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (2): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (3): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (4): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(256, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (5): Sequential(\n",
              "    (0): Pool(\n",
              "      (maxpool): MaxPool2d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False)\n",
              "      (avgpool): AvgPool2d(kernel_size=4, stride=4, padding=0)\n",
              "    )\n",
              "    (1): Flatten()\n",
              "    (2): Linear(in_features=512, out_features=10, bias=True)\n",
              "  )\n",
              "), opt_func=functools.partial(<class 'torch.optim.sgd.SGD'>, lr=0.001, momentum=0.9, weight_decay=0.064, nesterov=True), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7f22dcee8e18>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/root/.fastai/data/cifar10'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[...], layer_groups=[Sequential(\n",
              "  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (2): ReLU(inplace)\n",
              "  (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (5): ReLU(inplace)\n",
              "  (6): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (8): ReLU(inplace)\n",
              "  (9): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (10): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (11): ReLU(inplace)\n",
              "  (12): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (13): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (14): ReLU(inplace)\n",
              "  (15): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (16): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (17): ReLU(inplace)\n",
              "  (18): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (19): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (20): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (21): ReLU(inplace)\n",
              "  (22): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (23): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (24): ReLU(inplace)\n",
              "  (25): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (26): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (27): ReLU(inplace)\n",
              "  (28): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (29): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (30): ReLU(inplace)\n",
              "  (31): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (32): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (33): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (34): ReLU(inplace)\n",
              "  (35): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (36): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (37): ReLU(inplace)\n",
              "  (38): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (39): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (40): ReLU(inplace)\n",
              "  (41): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (42): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (43): ReLU(inplace)\n",
              "  (44): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (45): Conv2d(256, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (46): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (47): ReLU(inplace)\n",
              "  (48): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (49): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (50): ReLU(inplace)\n",
              "  (51): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (52): MaxPool2d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False)\n",
              "  (53): AvgPool2d(kernel_size=4, stride=4, padding=0)\n",
              "  (54): Flatten()\n",
              "  (55): Linear(in_features=512, out_features=10, bias=True)\n",
              ")], add_time=True, silent=False, cb_fns_registered=False)\n",
              "loss_scale: 65536\n",
              "max_noskip: 1000\n",
              "dynamic: True\n",
              "clip: None\n",
              "flat_master: False\n",
              "max_scale: 16777216], layer_groups=[Sequential(\n",
              "  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (2): ReLU(inplace)\n",
              "  (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (5): ReLU(inplace)\n",
              "  (6): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (8): ReLU(inplace)\n",
              "  (9): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (10): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (11): ReLU(inplace)\n",
              "  (12): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (13): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (14): ReLU(inplace)\n",
              "  (15): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (16): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (17): ReLU(inplace)\n",
              "  (18): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (19): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (20): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (21): ReLU(inplace)\n",
              "  (22): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (23): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (24): ReLU(inplace)\n",
              "  (25): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (26): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (27): ReLU(inplace)\n",
              "  (28): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (29): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (30): ReLU(inplace)\n",
              "  (31): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (32): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (33): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (34): ReLU(inplace)\n",
              "  (35): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (36): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (37): ReLU(inplace)\n",
              "  (38): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (39): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (40): ReLU(inplace)\n",
              "  (41): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (42): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (43): ReLU(inplace)\n",
              "  (44): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (45): Conv2d(256, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (46): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (47): ReLU(inplace)\n",
              "  (48): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (49): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (50): ReLU(inplace)\n",
              "  (51): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (52): MaxPool2d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False)\n",
              "  (53): AvgPool2d(kernel_size=4, stride=4, padding=0)\n",
              "  (54): Flatten()\n",
              "  (55): Linear(in_features=512, out_features=10, bias=True)\n",
              ")], add_time=True, silent=False, cb_fns_registered=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cvzaCwzmmgNy",
        "colab_type": "text"
      },
      "source": [
        "## learning rate"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JrQ2ctUqmbeq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# learn.lr_find()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tl8zAhGkmfUD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# learn.recorder.plot()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lh6GxXeBmkzn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "lr = 1e-2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yzu9P7NNmj-G",
        "colab_type": "text"
      },
      "source": [
        "## training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "7bed4bc9-cf56-4e3c-aa92-bbac51c505bd",
        "id": "oxnNAL8WeG6_",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 97
        }
      },
      "source": [
        "# %%prun -s cumulative -l 40\n",
        "# learn.fit_one_cycle(1, max_lr=lr)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.877775</td>\n",
              "      <td>0.862519</td>\n",
              "      <td>0.696200</td>\n",
              "      <td>02:04</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            " "
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rSG-8f1x4q10",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "3379b007-8570-4242-dcac-b08da6a48264"
      },
      "source": [
        "# with torch.autograd.profiler.profile() as prof:\n",
        "#   learn.fit_one_cycle(1, max_lr=lr)\n",
        "# print(prof.key_averages().table(sort_by=\"self_cpu_time_total\"))"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.177362</td>\n",
              "      <td>1.135423</td>\n",
              "      <td>0.589600</td>\n",
              "      <td>00:55</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------\n",
            "Name                                  Self CPU total %   Self CPU total      CPU total %        CPU total     CPU time avg     CUDA total %       CUDA total    CUDA time avg  Number of Calls\n",
            "------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------\n",
            "_local_scalar_dense                            28.68%           7.683s           28.68%           7.683s        358.128us              NaN          0.000us          0.000us            21454\n",
            "to                                             18.97%           5.083s           19.37%           5.188s        197.461us              NaN          0.000us          0.000us            26274\n",
            "cudnn_convolution_backward                      8.82%           2.364s            8.82%           2.364s        303.060us              NaN          0.000us          0.000us             7800\n",
            "pin_memory                                      8.41%           2.254s            8.45%           2.265s          2.414ms              NaN          0.000us          0.000us              938\n",
            "cudnn_convolution                               5.20%           1.393s            5.20%           1.393s        148.479us              NaN          0.000us          0.000us             9380\n",
            "mul_                                            4.44%           1.189s            4.44%           1.189s         28.264us              NaN          0.000us          0.000us            42066\n",
            "add_                                            3.99%           1.070s            3.99%           1.070s         22.112us              NaN          0.000us          0.000us            48384\n",
            "add                                             3.43%        919.341ms            3.43%        919.341ms         32.018us              NaN          0.000us          0.000us            28713\n",
            "sum                                             2.60%        697.788ms            2.60%        697.788ms         32.531us              NaN          0.000us          0.000us            21450\n",
            "zero_                                           2.40%        641.927ms            2.40%        641.927ms         15.240us              NaN          0.000us          0.000us            42120\n",
            "cudnn_batch_norm                                2.02%        542.037ms            2.02%        542.037ms         72.233us              NaN          0.000us          0.000us             7504\n",
            "torch::autograd::AccumulateGrad                 2.02%        540.504ms            2.02%        540.504ms         25.665us              NaN          0.000us          0.000us            21060\n",
            "div_                                            1.70%        455.219ms            1.70%        455.219ms         21.615us              NaN          0.000us          0.000us            21060\n",
            "cudnn_batch_norm_backward                       1.38%        369.229ms            1.38%        369.229ms         59.171us              NaN          0.000us          0.000us             6240\n",
            "relu_                                           0.79%        210.723ms            0.79%        210.723ms         28.081us              NaN          0.000us          0.000us             7504\n",
            "threshold_backward                              0.67%        180.782ms            0.67%        180.782ms         28.971us              NaN          0.000us          0.000us             6240\n",
            "empty                                           0.41%        110.881ms            0.41%        110.881ms          8.058us              NaN          0.000us          0.000us            13761\n",
            "addmm                                           0.30%         81.596ms            0.30%         81.596ms        173.979us              NaN          0.000us          0.000us              469\n",
            "CudnnBatchNormBackward                          0.30%         80.290ms            1.73%        463.435ms         74.268us              NaN          0.000us          0.000us             6240\n",
            "_convolution                                    0.27%         71.046ms            5.53%           1.482s        158.011us              NaN          0.000us          0.000us             9380\n",
            "_batch_norm_impl_index                          0.26%         70.605ms            2.42%        649.537ms         86.559us              NaN          0.000us          0.000us             7504\n",
            "contiguous                                      0.26%         69.170ms            0.26%         69.170ms          1.302us              NaN          0.000us          0.000us            53140\n",
            "item                                            0.24%         65.112ms           28.92%           7.748s        361.163us              NaN          0.000us          0.000us            21454\n",
            "CudnnConvolutionBackward                        0.22%         58.983ms            9.04%           2.423s        310.622us              NaN          0.000us          0.000us             7800\n",
            "mul                                             0.21%         56.248ms            0.21%         56.248ms         34.340us              NaN          0.000us          0.000us             1638\n",
            "nll_loss_forward                                0.15%         39.802ms            0.15%         39.802ms         84.866us              NaN          0.000us          0.000us              469\n",
            "ReluBackward1                                   0.14%         37.225ms            0.81%        218.006ms         34.937us              NaN          0.000us          0.000us             6240\n",
            "cat                                             0.13%         33.744ms            0.13%         33.744ms         71.948us              NaN          0.000us          0.000us              469\n",
            "sub                                             0.11%         28.348ms            0.11%         28.348ms         60.444us              NaN          0.000us          0.000us              469\n",
            "_log_softmax                                    0.10%         27.921ms            0.10%         27.921ms         59.533us              NaN          0.000us          0.000us              469\n",
            "batch_norm                                      0.10%         26.593ms            2.52%        676.129ms         90.103us              NaN          0.000us          0.000us             7504\n",
            "detach_                                         0.09%         24.399ms            0.09%         24.399ms          0.567us              NaN          0.000us          0.000us            43058\n",
            "view                                            0.09%         22.999ms            0.09%         22.999ms         11.764us              NaN          0.000us          0.000us             1955\n",
            "mm                                              0.08%         22.175ms            0.08%         22.175ms         28.430us              NaN          0.000us          0.000us              780\n",
            "conv2d                                          0.08%         21.722ms            5.67%           1.520s        162.072us              NaN          0.000us          0.000us             9380\n",
            "max_pool2d_with_indices                         0.08%         20.861ms            0.08%         20.861ms         44.479us              NaN          0.000us          0.000us              469\n",
            "avg_pool2d_backward                             0.08%         20.476ms            0.08%         20.476ms         52.502us              NaN          0.000us          0.000us              390\n",
            "div                                             0.07%         18.515ms            0.07%         18.515ms         21.529us              NaN          0.000us          0.000us              860\n",
            "nll_loss_backward                               0.07%         18.503ms            0.07%         18.503ms         47.442us              NaN          0.000us          0.000us              390\n",
            "max_pool2d_with_indices_backward                0.06%         16.836ms            0.06%         16.836ms         43.170us              NaN          0.000us          0.000us              390\n",
            "convolution                                     0.06%         16.377ms            5.59%           1.499s        159.757us              NaN          0.000us          0.000us             9380\n",
            "avg_pool2d                                      0.05%         14.647ms            0.05%         14.647ms         31.230us              NaN          0.000us          0.000us              469\n",
            "_th_eq                                          0.05%         12.851ms            0.05%         12.851ms        162.674us              NaN          0.000us          0.000us               79\n",
            "_log_softmax_backward_data                      0.05%         12.598ms            0.05%         12.598ms         32.303us              NaN          0.000us          0.000us              390\n",
            "max                                             0.04%         10.821ms            0.04%         10.821ms        136.972us              NaN          0.000us          0.000us               79\n",
            "unsigned short                                  0.03%          8.241ms            0.03%          8.241ms          4.062us              NaN          0.000us          0.000us             2029\n",
            "unsqueeze                                       0.03%          7.963ms            0.03%          7.963ms          4.245us              NaN          0.000us          0.000us             1876\n",
            "_th_set_                                        0.03%          7.309ms            0.03%          7.309ms          3.896us              NaN          0.000us          0.000us             1876\n",
            "detach                                          0.03%          6.951ms            0.03%          6.951ms          6.645us              NaN          0.000us          0.000us             1046\n",
            "max_pool2d                                      0.01%          3.985ms            0.09%         24.846ms         52.976us              NaN          0.000us          0.000us              469\n",
            "mean                                            0.01%          3.941ms            0.01%          3.941ms         49.888us              NaN          0.000us          0.000us               79\n",
            "AddmmBackward                                   0.01%          3.941ms            0.11%         29.718ms         76.200us              NaN          0.000us          0.000us              390\n",
            "transpose                                       0.01%          3.430ms            0.01%          3.430ms          2.583us              NaN          0.000us          0.000us             1328\n",
            "NllLossBackward                                 0.01%          3.368ms            0.08%         21.871ms         56.079us              NaN          0.000us          0.000us              390\n",
            "set_                                            0.01%          3.015ms            0.04%         10.324ms          5.503us              NaN          0.000us          0.000us             1876\n",
            "reshape                                         0.01%          2.911ms            0.02%          5.542ms          7.105us              NaN          0.000us          0.000us              780\n",
            "MulBackward0                                    0.01%          2.782ms            0.07%         19.370ms         49.667us              NaN          0.000us          0.000us              390\n",
            "clone                                           0.01%          2.673ms            0.01%          2.673ms         24.747us              NaN          0.000us          0.000us              108\n",
            "as_strided                                      0.01%          2.631ms            0.01%          2.631ms          3.372us              NaN          0.000us          0.000us              780\n",
            "AddBackward0                                    0.01%          2.251ms            0.01%          2.251ms          0.721us              NaN          0.000us          0.000us             3120\n",
            "MaxPool2DWithIndicesBackward                    0.01%          2.173ms            0.07%         19.010ms         48.743us              NaN          0.000us          0.000us              390\n",
            "LogSoftmaxBackward                              0.01%          1.911ms            0.05%         14.509ms         37.203us              NaN          0.000us          0.000us              390\n",
            "torch::autograd::CopyBackwards                  0.01%          1.875ms            0.05%         14.374ms         36.856us              NaN          0.000us          0.000us              390\n",
            "log_softmax                                     0.01%          1.839ms            0.11%         29.760ms         63.455us              NaN          0.000us          0.000us              469\n",
            "slice                                           0.01%          1.693ms            0.01%          1.693ms          2.170us              NaN          0.000us          0.000us              780\n",
            "nll_loss                                        0.01%          1.651ms            0.15%         41.453ms         88.386us              NaN          0.000us          0.000us              469\n",
            "AvgPool2DBackward                               0.01%          1.611ms            0.08%         22.087ms         56.633us              NaN          0.000us          0.000us              390\n",
            "CatBackward                                     0.01%          1.346ms            0.02%          4.319ms         11.075us              NaN          0.000us          0.000us              390\n",
            "ViewBackward                                    0.00%          1.298ms            0.03%          6.840ms          8.769us              NaN          0.000us          0.000us              780\n",
            "narrow                                          0.00%          1.281ms            0.01%          2.974ms          3.813us              NaN          0.000us          0.000us              780\n",
            "TBackward                                       0.00%        558.003us            0.01%          1.927ms          4.942us              NaN          0.000us          0.000us              390\n",
            "TransposeBackward0                              0.00%        548.340us            0.00%          1.326ms          3.400us              NaN          0.000us          0.000us              390\n",
            "argmax                                          0.00%        451.589us            0.04%         11.272ms        142.688us              NaN          0.000us          0.000us               79\n",
            "eq                                              0.00%        334.114us            0.05%         13.185ms        166.904us              NaN          0.000us          0.000us               79\n",
            "is_floating_point                               0.00%        251.189us            0.00%        251.189us          0.561us              NaN          0.000us          0.000us              448\n",
            "torch::autograd::GraphRoot                      0.00%        234.270us            0.00%        234.270us          0.601us              NaN          0.000us          0.000us              390\n",
            "stack                                           0.00%        124.513us            0.00%        124.513us        124.513us              NaN          0.000us          0.000us                1\n",
            "random_                                         0.00%         49.978us            0.00%         49.978us         24.989us              NaN          0.000us          0.000us                2\n",
            "is_complex                                      0.00%          1.509us            0.00%          1.509us          0.755us              NaN          0.000us          0.000us                2\n",
            "------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------\n",
            "Self CPU time total: 26.789s\n",
            "CUDA time total: 0.000us\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7xiLFBFm91KX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "dc126c73-996e-478f-93bc-1a9d7bfbc0ba"
      },
      "source": [
        "# learn.fit_one_cycle(35, max_lr=lr)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.755360</td>\n",
              "      <td>0.798539</td>\n",
              "      <td>0.721900</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.724884</td>\n",
              "      <td>0.740708</td>\n",
              "      <td>0.746300</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.689003</td>\n",
              "      <td>0.795453</td>\n",
              "      <td>0.729000</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.643591</td>\n",
              "      <td>0.749863</td>\n",
              "      <td>0.747000</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.619617</td>\n",
              "      <td>0.678545</td>\n",
              "      <td>0.772600</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.557650</td>\n",
              "      <td>0.712036</td>\n",
              "      <td>0.764300</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>6</td>\n",
              "      <td>0.525982</td>\n",
              "      <td>0.628469</td>\n",
              "      <td>0.789800</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>7</td>\n",
              "      <td>0.489476</td>\n",
              "      <td>0.601703</td>\n",
              "      <td>0.799400</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>8</td>\n",
              "      <td>0.446550</td>\n",
              "      <td>0.513119</td>\n",
              "      <td>0.827200</td>\n",
              "      <td>01:59</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>9</td>\n",
              "      <td>0.411602</td>\n",
              "      <td>0.566839</td>\n",
              "      <td>0.811200</td>\n",
              "      <td>01:59</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>10</td>\n",
              "      <td>0.406884</td>\n",
              "      <td>0.532525</td>\n",
              "      <td>0.819400</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>11</td>\n",
              "      <td>0.371704</td>\n",
              "      <td>0.536400</td>\n",
              "      <td>0.823000</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>12</td>\n",
              "      <td>0.368738</td>\n",
              "      <td>0.477462</td>\n",
              "      <td>0.836900</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>13</td>\n",
              "      <td>0.348643</td>\n",
              "      <td>0.509209</td>\n",
              "      <td>0.833700</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>14</td>\n",
              "      <td>0.324770</td>\n",
              "      <td>0.483217</td>\n",
              "      <td>0.840300</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>15</td>\n",
              "      <td>0.311770</td>\n",
              "      <td>0.444769</td>\n",
              "      <td>0.849200</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>16</td>\n",
              "      <td>0.297338</td>\n",
              "      <td>0.446841</td>\n",
              "      <td>0.854700</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>17</td>\n",
              "      <td>0.284720</td>\n",
              "      <td>0.420779</td>\n",
              "      <td>0.862800</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>18</td>\n",
              "      <td>0.267742</td>\n",
              "      <td>0.400152</td>\n",
              "      <td>0.866700</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>19</td>\n",
              "      <td>0.258534</td>\n",
              "      <td>0.410638</td>\n",
              "      <td>0.867700</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>20</td>\n",
              "      <td>0.246835</td>\n",
              "      <td>0.391739</td>\n",
              "      <td>0.871300</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>21</td>\n",
              "      <td>0.228389</td>\n",
              "      <td>0.374244</td>\n",
              "      <td>0.878100</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>22</td>\n",
              "      <td>0.214114</td>\n",
              "      <td>0.377271</td>\n",
              "      <td>0.877700</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>23</td>\n",
              "      <td>0.198912</td>\n",
              "      <td>0.329379</td>\n",
              "      <td>0.894000</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>24</td>\n",
              "      <td>0.185296</td>\n",
              "      <td>0.321134</td>\n",
              "      <td>0.896200</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>25</td>\n",
              "      <td>0.163980</td>\n",
              "      <td>0.353952</td>\n",
              "      <td>0.885100</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>26</td>\n",
              "      <td>0.158317</td>\n",
              "      <td>0.340016</td>\n",
              "      <td>0.891700</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>27</td>\n",
              "      <td>0.143901</td>\n",
              "      <td>0.331729</td>\n",
              "      <td>0.898400</td>\n",
              "      <td>01:56</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>28</td>\n",
              "      <td>0.119838</td>\n",
              "      <td>0.330769</td>\n",
              "      <td>0.900500</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>29</td>\n",
              "      <td>0.104753</td>\n",
              "      <td>0.307226</td>\n",
              "      <td>0.907100</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>30</td>\n",
              "      <td>0.082374</td>\n",
              "      <td>0.285755</td>\n",
              "      <td>0.914900</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>31</td>\n",
              "      <td>0.076399</td>\n",
              "      <td>0.285294</td>\n",
              "      <td>0.914100</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>32</td>\n",
              "      <td>0.072202</td>\n",
              "      <td>0.276401</td>\n",
              "      <td>0.918600</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>33</td>\n",
              "      <td>0.062335</td>\n",
              "      <td>0.275956</td>\n",
              "      <td>0.917400</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>34</td>\n",
              "      <td>0.058365</td>\n",
              "      <td>0.274776</td>\n",
              "      <td>0.917700</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ETZKIlyGhHVX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "79c9fdc8-689c-4ec1-b92d-1d9aafc80990"
      },
      "source": [
        "learn.fit_one_cycle(1, max_lr=lr)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.200495</td>\n",
              "      <td>1.173985</td>\n",
              "      <td>0.576900</td>\n",
              "      <td>02:14</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Toh6_-e8eQCu",
        "colab_type": "text"
      },
      "source": [
        "## use in-memory cached dataset\n",
        "\n",
        "Thanks to Tom B, who pointed out that myrtle.ai used an in-memory version of CIFAR10, and to baz, who wrote the `MemoryImageList` below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UmdbgvKueRqh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class MemoryImageList(ImageList):\n",
        "  _map = {}\n",
        "  def open(self, i):\n",
        "    item = self._map.get(str(i))\n",
        "    if isinstance(item, Image):\n",
        "      return item\n",
        "    item = super().open(i)\n",
        "    self._map[str(i)] = item\n",
        "    return item"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W-EIhIkkf81b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data = MemoryImageList.from_folder(path).split_by_folder(valid='test').label_from_folder().databunch(bs=batch_size).normalize(cifar_stats)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DehIGDILgj_I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn = Learner(data, model, metrics=accuracy, loss_func=CrossEntropyFlat(), opt_func=opt)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2V7-9BdUg0NW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "42e6626c-1aa2-4986-e84e-7acd4f6a5fbe"
      },
      "source": [
        "learn.to_fp16(); \"\""
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "''"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2eVBDFtSg2BW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "d7b73524-07f6-422e-dcde-c6065aae7bb7"
      },
      "source": [
        "learn.fit_one_cycle(1, max_lr=lr)"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.772558</td>\n",
              "      <td>0.732093</td>\n",
              "      <td>0.739800</td>\n",
              "      <td>02:09</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WUDPAIBliGT1",
        "colab_type": "text"
      },
      "source": [
        "## use in-memory loaded dataset\n",
        "\n",
        "Thanks to Karl, who wrote the below class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uNroHZXig6u1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class ImageListInMemory(ImageList):\n",
        "    _bunch,_square_show,_square_show_res = ImageDataBunch,True,True\n",
        "    def __init__(self, *args, convert_mode='RGB', after_open=None, **kwargs):\n",
        "        super().__init__(*args, **kwargs, convert_mode=convert_mode, after_open=after_open)\n",
        "        self.loaded = False\n",
        "\n",
        "    def load_data(self):\n",
        "        if not self.loaded:\n",
        "            self.images = [open_image(item, convert_mode=self.convert_mode, after_open=self.after_open)\n",
        "                           for item in self.items]\n",
        "            self.loaded = True\n",
        "\n",
        "    def purge(self):\n",
        "        if self.loaded:\n",
        "            self.images = None\n",
        "            gc.collect()\n",
        "            self.loaded = False\n",
        "\n",
        "    def get(self, i):\n",
        "        if self.loaded:\n",
        "            res = self.images[i]\n",
        "            self.sizes[i] = res.size\n",
        "            return res\n",
        "\n",
        "        else:\n",
        "            res = super().get(i)\n",
        "            self.sizes[i] = res.size\n",
        "            return res  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IanuG0eJiSWM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data = ImageListInMemory.from_folder(path).split_by_folder(valid='test').label_from_folder().databunch(bs=batch_size).normalize(cifar_stats)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rYNR7xCHjIWW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data.load_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nQUG0wQqiWk6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "63c08e9a-9b57-4586-f4c5-5c196c2df6fb"
      },
      "source": [
        "learn = Learner(data, model, metrics=accuracy, loss_func=CrossEntropyFlat(), opt_func=opt)\n",
        "learn.to_fp16(); \"\""
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "''"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oGvxK3wQiYjS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "85dfd1f7-a9af-4655-e06e-79e51dd95eb8"
      },
      "source": [
        "learn.fit_one_cycle(1, max_lr=lr)"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.459892</td>\n",
              "      <td>0.530229</td>\n",
              "      <td>0.815800</td>\n",
              "      <td>02:11</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c8yR54ZiidiK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}