Character Level Classification of Surname Ethnicity using Fastai

char_rnn_classification_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifying Name Ethnicity with a Character level RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is heavily modeled on the Pytorch tutorial:\n",
    "https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html\n",
    "\n",
    "We use fastai libraries extensively to make dataloading and training easier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the Pytorch tutorial data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a list of surnames and their ethnicities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://download.pytorch.org/tutorial/data.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!unzip -o data.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data\n",
    "fastai import pandas and all sorts of other goodies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unidecode import unidecode\n",
    "import string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reduce the ouput to 20 rows to prevent it from taking too much of the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.max_rows = 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in the data; names for each language is in a separate file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = Path('data/names')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Arabic.txt   English.txt  Irish.txt\tPolish.txt\tSpanish.txt\r\n",
      "Chinese.txt  French.txt   Italian.txt\tPortuguese.txt\tVietnamese.txt\r\n",
      "Czech.txt    German.txt   Japanese.txt\tRussian.txt\r\n",
      "Dutch.txt    Greek.txt\t  Korean.txt\tScottish.txt\r\n"
     ]
    }
   ],
   "source": [
    "!ls {path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Khoury\r\n",
      "Nahas\r\n",
      "Daher\r\n",
      "Gerges\r\n",
      "Nazari\r\n"
     ]
    }
   ],
   "source": [
    "!head -n5 {path}/Arabic.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = []\n",
    "for p in path.glob('*.txt'):\n",
    "    lang = p.name[:-4]\n",
    "    with open(p) as f:\n",
    "        names += [(lang, l.strip()) for l in f]\n",
    "df = pd.DataFrame(names, columns=['cl', 'name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check the Data\n",
    "It's always worth doing some sanity checks on your data (even supposedly clean tutorial data).\n",
    "\n",
    "No matter how good your model is: garbage in, garbage out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Ahn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Baik</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Bang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Byon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Cha</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       cl  name\n",
       "0  Korean   Ahn\n",
       "1  Korean  Baik\n",
       "2  Korean  Bang\n",
       "3  Korean  Byon\n",
       "4  Korean   Cha"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20074"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Character Set\n",
    "What letters outside of ASCII are in the names?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(' ', 115),\n",
       " (\"'\", 87),\n",
       " ('-', 25),\n",
       " ('ö', 24),\n",
       " ('é', 22),\n",
       " ('í', 14),\n",
       " ('ó', 13),\n",
       " ('ä', 13),\n",
       " ('á', 12),\n",
       " ('ü', 11),\n",
       " ('à', 10),\n",
       " ('ß', 9),\n",
       " ('ú', 7),\n",
       " ('ñ', 6),\n",
       " ('ò', 3),\n",
       " ('Ś', 3),\n",
       " ('1', 3),\n",
       " (',', 3),\n",
       " ('è', 2),\n",
       " ('ã', 2),\n",
       " ('ù', 1),\n",
       " ('ì', 1),\n",
       " ('ż', 1),\n",
       " ('ń', 1),\n",
       " ('ł', 1),\n",
       " ('ą', 1),\n",
       " ('Ż', 1),\n",
       " ('/', 1),\n",
       " (':', 1),\n",
       " ('Á', 1),\n",
       " ('\\xa0', 1),\n",
       " ('õ', 1),\n",
       " ('É', 1),\n",
       " ('ê', 1),\n",
       " ('ç', 1)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "foreign_chars = Counter(_ for _ in ''.join(list(df.name)) if _ not in string.ascii_letters)\n",
    "foreign_chars.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few of these look suspicious. (Note the use of a regular expression in `contains` to check each of the characters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2494</th>\n",
       "      <td>Czech</td>\n",
       "      <td>Maxa/B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2590</th>\n",
       "      <td>Czech</td>\n",
       "      <td>Rafaj1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2703</th>\n",
       "      <td>Czech</td>\n",
       "      <td>Urbanek1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2732</th>\n",
       "      <td>Czech</td>\n",
       "      <td>Whitmire1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3214</th>\n",
       "      <td>Chinese</td>\n",
       "      <td>Lu:</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14506</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Jevolojnov,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15347</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Lysansky,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15366</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Lytkin,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18052</th>\n",
       "      <td>Russian</td>\n",
       "      <td>To The First  Page</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            cl                name\n",
       "2494     Czech              Maxa/B\n",
       "2590     Czech              Rafaj1\n",
       "2703     Czech            Urbanek1\n",
       "2732     Czech           Whitmire1\n",
       "3214   Chinese                 Lu:\n",
       "14506  Russian         Jevolojnov,\n",
       "15347  Russian           Lysansky,\n",
       "15366  Russian             Lytkin,\n",
       "18052  Russian  To The First  Page"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "suss_chars = [':', '/', '\\xa0', ',', '1']\n",
    "df[df.name.str.contains('|'.join(suss_chars))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of these look like legitimate names with extra junk (except 'To The First Page').\n",
    "Since it's so few names it's easiest just to drop them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[~df.name.str.contains('|'.join(suss_chars))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Single quotes and spaces are common"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>369</th>\n",
       "      <td>Italian</td>\n",
       "      <td>D'ambrosio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>371</th>\n",
       "      <td>Italian</td>\n",
       "      <td>D'amore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>Italian</td>\n",
       "      <td>D'angelo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>373</th>\n",
       "      <td>Italian</td>\n",
       "      <td>D'antonio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>374</th>\n",
       "      <td>Italian</td>\n",
       "      <td>De angelis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>375</th>\n",
       "      <td>Italian</td>\n",
       "      <td>De campo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>Italian</td>\n",
       "      <td>De felice</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>377</th>\n",
       "      <td>Italian</td>\n",
       "      <td>De filippis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>378</th>\n",
       "      <td>Italian</td>\n",
       "      <td>De fiore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>379</th>\n",
       "      <td>Italian</td>\n",
       "      <td>De laurentis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18161</th>\n",
       "      <td>Russian</td>\n",
       "      <td>V'Yurkov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19061</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zasyad'Ko</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19740</th>\n",
       "      <td>Portuguese</td>\n",
       "      <td>D'cruz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19741</th>\n",
       "      <td>Portuguese</td>\n",
       "      <td>D'cruze</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19743</th>\n",
       "      <td>Portuguese</td>\n",
       "      <td>De santigo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19858</th>\n",
       "      <td>French</td>\n",
       "      <td>D'aramitz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19864</th>\n",
       "      <td>French</td>\n",
       "      <td>De la fontaine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19869</th>\n",
       "      <td>French</td>\n",
       "      <td>De sauveterre</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20051</th>\n",
       "      <td>French</td>\n",
       "      <td>St martin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20052</th>\n",
       "      <td>French</td>\n",
       "      <td>St pierre</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               cl            name\n",
       "369       Italian      D'ambrosio\n",
       "371       Italian         D'amore\n",
       "372       Italian        D'angelo\n",
       "373       Italian       D'antonio\n",
       "374       Italian      De angelis\n",
       "375       Italian        De campo\n",
       "376       Italian       De felice\n",
       "377       Italian     De filippis\n",
       "378       Italian        De fiore\n",
       "379       Italian    De laurentis\n",
       "...           ...             ...\n",
       "18161     Russian        V'Yurkov\n",
       "19061     Russian       Zasyad'Ko\n",
       "19740  Portuguese          D'cruz\n",
       "19741  Portuguese         D'cruze\n",
       "19743  Portuguese      De santigo\n",
       "19858      French       D'aramitz\n",
       "19864      French  De la fontaine\n",
       "19869      French   De sauveterre\n",
       "20051      French       St martin\n",
       "20052      French       St pierre\n",
       "\n",
       "[150 rows x 2 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.name.str.contains(\"'| \")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since hyphens mainly join multiple last names (and are pretty rare) we won't lose heaps by dropping them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2982</th>\n",
       "      <td>Chinese</td>\n",
       "      <td>Au-Yong</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3088</th>\n",
       "      <td>Chinese</td>\n",
       "      <td>Ou-Yang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3089</th>\n",
       "      <td>Chinese</td>\n",
       "      <td>Ow-Yang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10156</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abdank-Kossovsky</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10639</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Amet-Han</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11221</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Bagai-Ool</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11757</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Bei-Bienko</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11787</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Beknazar-Yuzbashev</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11790</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Bekovich-Cherkassky</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11904</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Bestujev-Lada</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11952</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Bim-Bad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12209</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Chyrgal-Ool</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13071</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Galkin-Vraskoi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13307</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Gorbunov-Posadov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16687</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Porai-Koshits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17222</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Shah-Nazaroff</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17430</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Shirinsky-Shikhmatov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17748</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Tsann-Kay-Si</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17999</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Tzann-Kay-Si</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18315</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Van-Puteren</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>23 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            cl                  name\n",
       "2982   Chinese               Au-Yong\n",
       "3088   Chinese               Ou-Yang\n",
       "3089   Chinese               Ow-Yang\n",
       "10156  Russian      Abdank-Kossovsky\n",
       "10639  Russian              Amet-Han\n",
       "11221  Russian             Bagai-Ool\n",
       "11757  Russian            Bei-Bienko\n",
       "11787  Russian    Beknazar-Yuzbashev\n",
       "11790  Russian   Bekovich-Cherkassky\n",
       "11904  Russian         Bestujev-Lada\n",
       "...        ...                   ...\n",
       "11952  Russian               Bim-Bad\n",
       "12209  Russian           Chyrgal-Ool\n",
       "13071  Russian        Galkin-Vraskoi\n",
       "13307  Russian      Gorbunov-Posadov\n",
       "16687  Russian         Porai-Koshits\n",
       "17222  Russian         Shah-Nazaroff\n",
       "17430  Russian  Shirinsky-Shikhmatov\n",
       "17748  Russian          Tsann-Kay-Si\n",
       "17999  Russian          Tzann-Kay-Si\n",
       "18315  Russian           Van-Puteren\n",
       "\n",
       "[23 rows x 2 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.name.str.contains('-')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[~df.name.str.contains('-')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalising non-ASCII Characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's normalise all non-ASCII characters to ASCII equivalents.\n",
    "\n",
    "This makes our classification problem harder in practice: any names containing a ß are almost surely German, wheras \"ss\" could occur in many language. It also reduces the set of characters we need to represent our language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>Italian</td>\n",
       "      <td>Abbà</td>\n",
       "      <td>Abba</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112</th>\n",
       "      <td>Italian</td>\n",
       "      <td>Abelló</td>\n",
       "      <td>Abello</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>Italian</td>\n",
       "      <td>Airò</td>\n",
       "      <td>Airo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>Italian</td>\n",
       "      <td>Alò</td>\n",
       "      <td>Alo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>238</th>\n",
       "      <td>Italian</td>\n",
       "      <td>Azzarà</td>\n",
       "      <td>Azzara</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300</th>\n",
       "      <td>Italian</td>\n",
       "      <td>Bovér</td>\n",
       "      <td>Bover</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>445</th>\n",
       "      <td>Italian</td>\n",
       "      <td>Giùgovaz</td>\n",
       "      <td>Giugovaz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>461</th>\n",
       "      <td>Italian</td>\n",
       "      <td>Làconi</td>\n",
       "      <td>Laconi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>462</th>\n",
       "      <td>Italian</td>\n",
       "      <td>Laganà</td>\n",
       "      <td>Lagana</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>463</th>\n",
       "      <td>Italian</td>\n",
       "      <td>Lagomarsìno</td>\n",
       "      <td>Lagomarsino</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19912</th>\n",
       "      <td>French</td>\n",
       "      <td>Géroux</td>\n",
       "      <td>Geroux</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19920</th>\n",
       "      <td>French</td>\n",
       "      <td>Guérin</td>\n",
       "      <td>Guerin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19924</th>\n",
       "      <td>French</td>\n",
       "      <td>Hébert</td>\n",
       "      <td>Hebert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19949</th>\n",
       "      <td>French</td>\n",
       "      <td>Lécuyer</td>\n",
       "      <td>Lecuyer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19951</th>\n",
       "      <td>French</td>\n",
       "      <td>Lefévre</td>\n",
       "      <td>Lefevre</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19955</th>\n",
       "      <td>French</td>\n",
       "      <td>Lémieux</td>\n",
       "      <td>Lemieux</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19960</th>\n",
       "      <td>French</td>\n",
       "      <td>Lévêque</td>\n",
       "      <td>Leveque</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19961</th>\n",
       "      <td>French</td>\n",
       "      <td>Lévesque</td>\n",
       "      <td>Levesque</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19965</th>\n",
       "      <td>French</td>\n",
       "      <td>Maçon</td>\n",
       "      <td>Macon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20047</th>\n",
       "      <td>French</td>\n",
       "      <td>Séverin</td>\n",
       "      <td>Severin</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>156 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            cl         name   ascii_name\n",
       "100    Italian         Abbà         Abba\n",
       "112    Italian       Abelló       Abello\n",
       "160    Italian         Airò         Airo\n",
       "195    Italian          Alò          Alo\n",
       "238    Italian       Azzarà       Azzara\n",
       "300    Italian        Bovér        Bover\n",
       "445    Italian     Giùgovaz     Giugovaz\n",
       "461    Italian       Làconi       Laconi\n",
       "462    Italian       Laganà       Lagana\n",
       "463    Italian  Lagomarsìno  Lagomarsino\n",
       "...        ...          ...          ...\n",
       "19912   French       Géroux       Geroux\n",
       "19920   French       Guérin       Guerin\n",
       "19924   French       Hébert       Hebert\n",
       "19949   French      Lécuyer      Lecuyer\n",
       "19951   French      Lefévre      Lefevre\n",
       "19955   French      Lémieux      Lemieux\n",
       "19960   French      Lévêque      Leveque\n",
       "19961   French     Lévesque     Levesque\n",
       "19965   French        Maçon        Macon\n",
       "20047   French      Séverin      Severin\n",
       "\n",
       "[156 rows x 3 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['ascii_name'] = df.name.apply(unidecode)\n",
    "df[df.name != df.ascii_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check case: I expect names to be in CamelCase.\n",
    "\n",
    "These seem to be mistakes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2508</th>\n",
       "      <td>Czech</td>\n",
       "      <td>MonkoAustria</td>\n",
       "      <td>MonkoAustria</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2677</th>\n",
       "      <td>Czech</td>\n",
       "      <td>StrakaO</td>\n",
       "      <td>StrakaO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3266</th>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>an</td>\n",
       "      <td>an</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              cl          name    ascii_name\n",
       "2508       Czech  MonkoAustria  MonkoAustria\n",
       "2677       Czech       StrakaO       StrakaO\n",
       "3266  Vietnamese            an            an"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[~df.ascii_name.str.contains(\"^[A-Z][^A-Z]*(?:[' -][A-Z][^A-Z]*)*$\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df.ascii_name.str.contains(\"^[A-Z][^A-Z]*(?:[' -][A-Z][^A-Z]*)*$\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's lowercase the ascii_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ascii_name'] = df.ascii_name.str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a check we've normalised correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('a', 16511),\n",
       " ('o', 11120),\n",
       " ('e', 10768),\n",
       " ('i', 10416),\n",
       " ('n', 9943),\n",
       " ('r', 8245),\n",
       " ('s', 7980),\n",
       " ('h', 7673),\n",
       " ('k', 6902),\n",
       " ('l', 6704),\n",
       " ('v', 6301),\n",
       " ('t', 5939),\n",
       " ('u', 4725),\n",
       " ('m', 4343),\n",
       " ('d', 3894),\n",
       " ('b', 3641),\n",
       " ('y', 3604),\n",
       " ('g', 3209),\n",
       " ('c', 3068),\n",
       " ('z', 1928),\n",
       " ('f', 1774),\n",
       " ('p', 1707),\n",
       " ('j', 1346),\n",
       " ('w', 1125),\n",
       " (' ', 112),\n",
       " ('q', 98),\n",
       " (\"'\", 87),\n",
       " ('x', 72)]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ascii_chars = Counter(''.join(list(df.ascii_name)))\n",
    "ascii_chars.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many classes does each name have?\n",
    "\n",
    "In practice a surname could have multiple ethnicities, but we'd have to be really careful of how we use this in training.\n",
    "\n",
    "If we end up with e.g. 'Michel' as French in the training dataset, but German in the validation set our model has no hope of getting it right (and we may discard an actually good model).\n",
    "\n",
    "We could handle this by:\n",
    "1) Allowing multiple class labels\n",
    "2) Picking the country that the name most commonly associates to\n",
    "3) Dropping ambiguous cases\n",
    "\n",
    "Without any information about frequency we can't do (2) and (1) is a harder problem, so we'll stick to (3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ascii_name\n",
       "michel     6\n",
       "adam       5\n",
       "albert     5\n",
       "abel       5\n",
       "martin     5\n",
       "simon      5\n",
       "ventura    4\n",
       "costa      4\n",
       "jordan     4\n",
       "han        4\n",
       "salomon    4\n",
       "samuel     4\n",
       "klein      4\n",
       "franco     4\n",
       "wang       4\n",
       "oliver     4\n",
       "garcia     3\n",
       "horn       3\n",
       "lim        3\n",
       "rose       3\n",
       "Name: cl, dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "name_classes = df.\\\n",
    "  groupby('ascii_name').\\\n",
    "  nunique().cl.sort_values(ascending=False)\n",
    "name_classes.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>872</th>\n",
       "      <td>Polish</td>\n",
       "      <td>Michel</td>\n",
       "      <td>michel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2077</th>\n",
       "      <td>Dutch</td>\n",
       "      <td>Michel</td>\n",
       "      <td>michel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3489</th>\n",
       "      <td>Spanish</td>\n",
       "      <td>Michel</td>\n",
       "      <td>michel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6163</th>\n",
       "      <td>German</td>\n",
       "      <td>Michel</td>\n",
       "      <td>michel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8709</th>\n",
       "      <td>English</td>\n",
       "      <td>Michel</td>\n",
       "      <td>michel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19978</th>\n",
       "      <td>French</td>\n",
       "      <td>Michel</td>\n",
       "      <td>michel</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            cl    name ascii_name\n",
       "872     Polish  Michel     michel\n",
       "2077     Dutch  Michel     michel\n",
       "3489   Spanish  Michel     michel\n",
       "6163    German  Michel     michel\n",
       "8709   English  Michel     michel\n",
       "19978   French  Michel     michel"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.name == 'Michel']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1 in 40 of our names have multiple classes (most of them do before normalisation too)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17380, 0.027445339470655927)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(name_classes), sum(name_classes > 1) / len(name_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While some names like [Abel](https://en.wikipedia.org/wiki/Abel_(surname)) do seem to occur commonly in multiple countries, for example:\n",
    "- [Adamson](https://en.wikipedia.org/wiki/Adamson_(surname)) is very unlikely to be Russian\n",
    "- [Wong](https://en.wikipedia.org/wiki/Wong_(surname)) is much more prevalant in Chinese than English\n",
    "- [Yang](https://en.wikipedia.org/wiki/Yang_(surname)) is very rare in English\n",
    "\n",
    "It seems like Korean and Chinese have a lot of overlap, as to English and Scottish.\n",
    "While this makes some linguistic sense it will make it hard to make a reliable classifier.\n",
    "\n",
    "Note that most names only occur once; so we can't pick a \"most common\" frequency class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                       name\n",
      "ascii_name cl              \n",
      "abel       English        1\n",
      "           French         1\n",
      "           German         1\n",
      "           Russian        1\n",
      "           Spanish        1\n",
      "abello     Italian        1\n",
      "           Spanish        1\n",
      "abraham    English        1\n",
      "           French         1\n",
      "abreu      Portuguese     1\n",
      "           Spanish        1\n",
      "adam       English        1\n",
      "           French         1\n",
      "           German         1\n",
      "           Irish          1\n",
      "           Russian        1\n",
      "adams      English        1\n",
      "           Russian        1\n",
      "adamson    English        1\n",
      "           Russian        1\n",
      "adler      English        1\n",
      "           German         1\n",
      "           Russian        1\n",
      "aitken     English        1\n",
      "           Scottish       1\n",
      "albert     English        1\n",
      "           French         1\n",
      "           German         1\n",
      "           Russian        1\n",
      "           Spanish        1\n",
      "...                     ...\n",
      "wilson     English        1\n",
      "           Scottish       1\n",
      "winter     English        1\n",
      "           German         1\n",
      "wolf       English        1\n",
      "           German         1\n",
      "wong       Chinese        1\n",
      "           English        1\n",
      "woo        Chinese        1\n",
      "           Korean         1\n",
      "wood       Czech          1\n",
      "           English        1\n",
      "           Scottish       1\n",
      "wright     English        1\n",
      "           Scottish       1\n",
      "yan        Chinese        2\n",
      "           Russian        1\n",
      "yang       Chinese        1\n",
      "           English        1\n",
      "           Korean         1\n",
      "yim        Chinese        1\n",
      "           Korean         1\n",
      "you        Chinese        1\n",
      "           Korean         1\n",
      "young      English        1\n",
      "           Scottish       1\n",
      "yun        Chinese        1\n",
      "           Korean         1\n",
      "zambrano   Italian        1\n",
      "           Spanish        1\n",
      "\n",
      "[1051 rows x 1 columns]\n"
     ]
    }
   ],
   "source": [
    "with pd.option_context('display.max_rows', 60):\n",
    "    print(df[df.ascii_name.isin(name_classes[name_classes > 1].index)].groupby(['ascii_name', 'cl']).count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than finding the \"right\" ethnicity the easy thing to do is to remove all ambiguous cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[~df.ascii_name.isin(name_classes[name_classes > 1].index)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How often do (class, name) pairs occur?\n",
    "\n",
    "We need exactly one row per pair; if separate copies appear in the training and validation set we'll get a higher validation accuracy than is reasonable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some names occur very frequently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>n</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ascii_name</th>\n",
       "      <th>cl</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tahan</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>28</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fakhoury</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>28</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>koury</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nader</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sarraf</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>26</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hadad</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>26</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>kassis</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>26</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>antar</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>26</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>shadid</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cham</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mifsud</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nahas</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gerges</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ganim</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>23</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tuma</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>23</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>to the first page</th>\n",
       "      <th>Russian</th>\n",
       "      <td>23</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>atiyeh</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>23</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>malouf</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>23</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sayegh</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>naifeh</th>\n",
       "      <th>Arabic</th>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           name   n\n",
       "ascii_name        cl               \n",
       "tahan             Arabic     28  28\n",
       "fakhoury          Arabic     28  28\n",
       "koury             Arabic     27  27\n",
       "nader             Arabic     27  27\n",
       "sarraf            Arabic     26  26\n",
       "hadad             Arabic     26  26\n",
       "kassis            Arabic     26  26\n",
       "antar             Arabic     26  26\n",
       "shadid            Arabic     25  25\n",
       "cham              Arabic     25  25\n",
       "mifsud            Arabic     25  25\n",
       "nahas             Arabic     24  24\n",
       "gerges            Arabic     24  24\n",
       "ganim             Arabic     23  23\n",
       "tuma              Arabic     23  23\n",
       "to the first page Russian    23  23\n",
       "atiyeh            Arabic     23  23\n",
       "malouf            Arabic     23  23\n",
       "sayegh            Arabic     22  22\n",
       "naifeh            Arabic     22  22"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts = df.assign(n=1).groupby(['ascii_name', 'cl']).count().sort_values('n', ascending=False)\n",
    "counts.head(n=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remove the \"To The First Page\" junk (probably some artifact of where the data was scraped from)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df.ascii_name != 'to the first page']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no multiples in English, and a lot in Arabic. It seems like a data entry error rather than meaningful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>n</th>\n",
       "      <th>multiple</th>\n",
       "      <th>rows</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cl</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Russian</th>\n",
       "      <td>9326</td>\n",
       "      <td>9326</td>\n",
       "      <td>35.0</td>\n",
       "      <td>9263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>English</th>\n",
       "      <td>3359</td>\n",
       "      <td>3359</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Arabic</th>\n",
       "      <td>1892</td>\n",
       "      <td>1892</td>\n",
       "      <td>103.0</td>\n",
       "      <td>103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Japanese</th>\n",
       "      <td>983</td>\n",
       "      <td>983</td>\n",
       "      <td>1.0</td>\n",
       "      <td>982</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Italian</th>\n",
       "      <td>665</td>\n",
       "      <td>665</td>\n",
       "      <td>5.0</td>\n",
       "      <td>660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>German</th>\n",
       "      <td>613</td>\n",
       "      <td>613</td>\n",
       "      <td>33.0</td>\n",
       "      <td>578</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Czech</th>\n",
       "      <td>480</td>\n",
       "      <td>480</td>\n",
       "      <td>16.0</td>\n",
       "      <td>464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dutch</th>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>10.0</td>\n",
       "      <td>244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chinese</th>\n",
       "      <td>219</td>\n",
       "      <td>219</td>\n",
       "      <td>19.0</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spanish</th>\n",
       "      <td>214</td>\n",
       "      <td>214</td>\n",
       "      <td>2.0</td>\n",
       "      <td>212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>French</th>\n",
       "      <td>213</td>\n",
       "      <td>213</td>\n",
       "      <td>3.0</td>\n",
       "      <td>210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Greek</th>\n",
       "      <td>195</td>\n",
       "      <td>195</td>\n",
       "      <td>2.0</td>\n",
       "      <td>192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Irish</th>\n",
       "      <td>170</td>\n",
       "      <td>170</td>\n",
       "      <td>6.0</td>\n",
       "      <td>164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Polish</th>\n",
       "      <td>124</td>\n",
       "      <td>124</td>\n",
       "      <td>1.0</td>\n",
       "      <td>123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Korean</th>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>0.0</td>\n",
       "      <td>61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Vietnamese</th>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>1.0</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Portuguese</th>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>0.0</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Scottish</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name     n  multiple  rows\n",
       "cl                                    \n",
       "Russian     9326  9326      35.0  9263\n",
       "English     3359  3359       0.0  3359\n",
       "Arabic      1892  1892     103.0   103\n",
       "Japanese     983   983       1.0   982\n",
       "Italian      665   665       5.0   660\n",
       "German       613   613      33.0   578\n",
       "Czech        480   480      16.0   464\n",
       "Dutch        255   255      10.0   244\n",
       "Chinese      219   219      19.0   200\n",
       "Spanish      214   214       2.0   212\n",
       "French       213   213       3.0   210\n",
       "Greek        195   195       2.0   192\n",
       "Irish        170   170       6.0   164\n",
       "Polish       124   124       1.0   123\n",
       "Korean        61    61       0.0    61\n",
       "Vietnamese    56    56       1.0    55\n",
       "Portuguese    32    32       0.0    32\n",
       "Scottish       1     1       0.0     1"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.assign(multiple=counts.n > 1, rows=1).groupby('cl').sum().sort_values('n', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It makes sense to drop the duplicates and only have a single row per `ascii_name` and `cl`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.drop_duplicates(['ascii_name', 'cl'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16902"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Length Check\n",
    "It's worth checking if the shortest and longest names make sense.\n",
    "\n",
    "They look reasonable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "      <th>len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3265</th>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>An</td>\n",
       "      <td>an</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Oh</td>\n",
       "      <td>oh</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1150</th>\n",
       "      <td>Japanese</td>\n",
       "      <td>Ii</td>\n",
       "      <td>ii</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Ra</td>\n",
       "      <td>ra</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3891</th>\n",
       "      <td>Arabic</td>\n",
       "      <td>Ba</td>\n",
       "      <td>ba</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Ri</td>\n",
       "      <td>ri</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Si</td>\n",
       "      <td>si</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>Korean</td>\n",
       "      <td>So</td>\n",
       "      <td>so</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3311</th>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>To</td>\n",
       "      <td>to</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Yi</td>\n",
       "      <td>yi</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11475</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Bakhtchivandzhi</td>\n",
       "      <td>bakhtchivandzhi</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10191</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abdulladzhanoff</td>\n",
       "      <td>abdulladzhanoff</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17299</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Shakhnazaryants</td>\n",
       "      <td>shakhnazaryants</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11393</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Baistryutchenko</td>\n",
       "      <td>baistryutchenko</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14965</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Katzenellenbogen</td>\n",
       "      <td>katzenellenbogen</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2228</th>\n",
       "      <td>Dutch</td>\n",
       "      <td>Vandroogenbroeck</td>\n",
       "      <td>vandroogenbroeck</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14947</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Katsenellenbogen</td>\n",
       "      <td>katsenellenbogen</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19552</th>\n",
       "      <td>Greek</td>\n",
       "      <td>Chrysanthopoulos</td>\n",
       "      <td>chrysanthopoulos</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2841</th>\n",
       "      <td>Irish</td>\n",
       "      <td>Maceachthighearna</td>\n",
       "      <td>maceachthighearna</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6380</th>\n",
       "      <td>German</td>\n",
       "      <td>Von grimmelshausen</td>\n",
       "      <td>von grimmelshausen</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>16902 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               cl                name          ascii_name  len\n",
       "3265   Vietnamese                  An                  an    2\n",
       "50         Korean                  Oh                  oh    2\n",
       "1150     Japanese                  Ii                  ii    2\n",
       "54         Korean                  Ra                  ra    2\n",
       "3891       Arabic                  Ba                  ba    2\n",
       "57         Korean                  Ri                  ri    2\n",
       "69         Korean                  Si                  si    2\n",
       "71         Korean                  So                  so    2\n",
       "3311   Vietnamese                  To                  to    2\n",
       "85         Korean                  Yi                  yi    2\n",
       "...           ...                 ...                 ...  ...\n",
       "11475     Russian     Bakhtchivandzhi     bakhtchivandzhi   15\n",
       "10191     Russian     Abdulladzhanoff     abdulladzhanoff   15\n",
       "17299     Russian     Shakhnazaryants     shakhnazaryants   15\n",
       "11393     Russian     Baistryutchenko     baistryutchenko   15\n",
       "14965     Russian    Katzenellenbogen    katzenellenbogen   16\n",
       "2228        Dutch    Vandroogenbroeck    vandroogenbroeck   16\n",
       "14947     Russian    Katsenellenbogen    katsenellenbogen   16\n",
       "19552       Greek    Chrysanthopoulos    chrysanthopoulos   16\n",
       "2841        Irish   Maceachthighearna   maceachthighearna   17\n",
       "6380       German  Von grimmelshausen  von grimmelshausen   18\n",
       "\n",
       "[16902 rows x 4 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.assign(len=df.name.str.len()).sort_values('len')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distribution by Language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The  dataset is very unbalanced.\n",
    "\n",
    "I doubt there's enough data to tacke Portuguese (which will be close to Spanish) and Scottish (which will be close to English)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "cl\n",
       "Russian       9262\n",
       "English       3359\n",
       "Japanese       982\n",
       "Italian        660\n",
       "German         578\n",
       "Czech          464\n",
       "Dutch          244\n",
       "Spanish        212\n",
       "French         210\n",
       "Chinese        200\n",
       "Greek          192\n",
       "Irish          164\n",
       "Polish         123\n",
       "Arabic         103\n",
       "Korean          61\n",
       "Vietnamese      55\n",
       "Portuguese      32\n",
       "Scottish         1\n",
       "Name: name, dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('cl').name.count().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3711</th>\n",
       "      <td>Scottish</td>\n",
       "      <td>Hay</td>\n",
       "      <td>hay</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            cl name ascii_name\n",
       "3711  Scottish  Hay        hay"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.cl.isin(['Scottish'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remove the rarest classes; we're not likely to have enough data to guess them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[~df.cl.isin(['Scottish', 'Portuguese'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note Russian contains variant transliterations to English like Abaimoff and Abaimov (which both correspond to Абаимов).\n",
    "\n",
    "But this doesn't quite explain it's high frequency: it seems a lot more Russian data was extracted.\n",
    "\n",
    "(Side note: [Chebyshev](https://en.wikipedia.org/wiki/Pafnuty_Chebyshev) can also be spelt e.g. Chebychev, Tchebycheff, Tschebyschef)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10112</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Ababko</td>\n",
       "      <td>ababko</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10113</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abaev</td>\n",
       "      <td>abaev</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10114</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abagyan</td>\n",
       "      <td>abagyan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10115</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abaidulin</td>\n",
       "      <td>abaidulin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10116</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abaidullin</td>\n",
       "      <td>abaidullin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10117</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abaimoff</td>\n",
       "      <td>abaimoff</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10118</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abaimov</td>\n",
       "      <td>abaimov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10119</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abakeliya</td>\n",
       "      <td>abakeliya</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10120</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abakovsky</td>\n",
       "      <td>abakovsky</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10121</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Abakshin</td>\n",
       "      <td>abakshin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19510</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zolotavin</td>\n",
       "      <td>zolotavin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19511</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zolotdinov</td>\n",
       "      <td>zolotdinov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19512</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zolotenkov</td>\n",
       "      <td>zolotenkov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19513</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zolotilin</td>\n",
       "      <td>zolotilin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19514</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zolotkov</td>\n",
       "      <td>zolotkov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19515</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zolotnitsky</td>\n",
       "      <td>zolotnitsky</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19516</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zolotnitzky</td>\n",
       "      <td>zolotnitzky</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19517</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zozrov</td>\n",
       "      <td>zozrov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19518</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zozulya</td>\n",
       "      <td>zozulya</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19519</th>\n",
       "      <td>Russian</td>\n",
       "      <td>Zukerman</td>\n",
       "      <td>zukerman</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9262 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            cl         name   ascii_name\n",
       "10112  Russian       Ababko       ababko\n",
       "10113  Russian        Abaev        abaev\n",
       "10114  Russian      Abagyan      abagyan\n",
       "10115  Russian    Abaidulin    abaidulin\n",
       "10116  Russian   Abaidullin   abaidullin\n",
       "10117  Russian     Abaimoff     abaimoff\n",
       "10118  Russian      Abaimov      abaimov\n",
       "10119  Russian    Abakeliya    abakeliya\n",
       "10120  Russian    Abakovsky    abakovsky\n",
       "10121  Russian     Abakshin     abakshin\n",
       "...        ...          ...          ...\n",
       "19510  Russian    Zolotavin    zolotavin\n",
       "19511  Russian   Zolotdinov   zolotdinov\n",
       "19512  Russian   Zolotenkov   zolotenkov\n",
       "19513  Russian    Zolotilin    zolotilin\n",
       "19514  Russian     Zolotkov     zolotkov\n",
       "19515  Russian  Zolotnitsky  zolotnitsky\n",
       "19516  Russian  Zolotnitzky  zolotnitzky\n",
       "19517  Russian       Zozrov       zozrov\n",
       "19518  Russian      Zozulya      zozulya\n",
       "19519  Russian     Zukerman     zukerman\n",
       "\n",
       "[9262 rows x 3 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.cl == 'Russian']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Validation and Training Sets\n",
    "\n",
    "We want our final model to work well on any language.\n",
    "\n",
    "But if we pick our validation set uniformly at random from the data we're likely to get many Russian names and not many Vietnamese names, which isn't a good test of this.\n",
    "\n",
    "So instead we'll take our validation set from an equal number from each subclass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Ahn</td>\n",
       "      <td>ahn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Baik</td>\n",
       "      <td>baik</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Bang</td>\n",
       "      <td>bang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Byon</td>\n",
       "      <td>byon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Cha</td>\n",
       "      <td>cha</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Cho</td>\n",
       "      <td>cho</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Choe</td>\n",
       "      <td>choe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Choi</td>\n",
       "      <td>choi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Chun</td>\n",
       "      <td>chun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Chweh</td>\n",
       "      <td>chweh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16859</th>\n",
       "      <td>French</td>\n",
       "      <td>Travere</td>\n",
       "      <td>travere</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16860</th>\n",
       "      <td>French</td>\n",
       "      <td>Traverse</td>\n",
       "      <td>traverse</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16861</th>\n",
       "      <td>French</td>\n",
       "      <td>Travert</td>\n",
       "      <td>travert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16862</th>\n",
       "      <td>French</td>\n",
       "      <td>Tremblay</td>\n",
       "      <td>tremblay</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16863</th>\n",
       "      <td>French</td>\n",
       "      <td>Tremble</td>\n",
       "      <td>tremble</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16864</th>\n",
       "      <td>French</td>\n",
       "      <td>Victors</td>\n",
       "      <td>victors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16865</th>\n",
       "      <td>French</td>\n",
       "      <td>Villeneuve</td>\n",
       "      <td>villeneuve</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16866</th>\n",
       "      <td>French</td>\n",
       "      <td>Vipond</td>\n",
       "      <td>vipond</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16867</th>\n",
       "      <td>French</td>\n",
       "      <td>Voclain</td>\n",
       "      <td>voclain</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16868</th>\n",
       "      <td>French</td>\n",
       "      <td>Yount</td>\n",
       "      <td>yount</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>16869 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           cl        name  ascii_name\n",
       "0      Korean         Ahn         ahn\n",
       "1      Korean        Baik        baik\n",
       "2      Korean        Bang        bang\n",
       "3      Korean        Byon        byon\n",
       "4      Korean         Cha         cha\n",
       "5      Korean         Cho         cho\n",
       "6      Korean        Choe        choe\n",
       "7      Korean        Choi        choi\n",
       "8      Korean        Chun        chun\n",
       "9      Korean       Chweh       chweh\n",
       "...       ...         ...         ...\n",
       "16859  French     Travere     travere\n",
       "16860  French    Traverse    traverse\n",
       "16861  French     Travert     travert\n",
       "16862  French    Tremblay    tremblay\n",
       "16863  French     Tremble     tremble\n",
       "16864  French     Victors     victors\n",
       "16865  French  Villeneuve  villeneuve\n",
       "16866  French      Vipond      vipond\n",
       "16867  French     Voclain     voclain\n",
       "16868  French       Yount       yount\n",
       "\n",
       "[16869 rows x 3 columns]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.reset_index().drop('index', 1)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "cl\n",
       "Russian       9262\n",
       "English       3359\n",
       "Japanese       982\n",
       "Italian        660\n",
       "German         578\n",
       "Czech          464\n",
       "Dutch          244\n",
       "Spanish        212\n",
       "French         210\n",
       "Chinese        200\n",
       "Greek          192\n",
       "Irish          164\n",
       "Polish         123\n",
       "Arabic         103\n",
       "Korean          61\n",
       "Vietnamese      55\n",
       "Name: name, dtype: int64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts = df.groupby('cl').name.count().sort_values(ascending=False)\n",
    "counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_size = 30 # We'll pick 30 at random from each subclass\n",
    "train_size = 500 # For a balanced training set we'll pick 500 at random with replacement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(6011)\n",
    "valid_idx = []\n",
    "for cl in counts.keys():\n",
    "    # Random sample of size \"valid_size\" for each class\n",
    "    valid_idx += list(df[df.cl == cl].sample(valid_size).index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['valid'] = False\n",
    "df.loc[valid_idx, 'valid'] = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also create a balanced training set as an alternative to using everything not in validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(7012)\n",
    "balanced_idx = []\n",
    "for cl in counts.keys():\n",
    "    # Random sample of size \"train_size\" for each class from the data outside of the validation set\n",
    "    balanced_idx += list(df[(df.cl == cl) & ~df.valid].sample(train_size, replace=True).index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the balanced index contains all 25 (= 55 - 30) Vietnamese names outside of the training set, but only contains 486 of the Russian names (because we sampled randomly with replacement there will be a couple of double ups)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "      <th>valid</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cl</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Russian</th>\n",
       "      <td>1</td>\n",
       "      <td>486</td>\n",
       "      <td>486</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>English</th>\n",
       "      <td>1</td>\n",
       "      <td>459</td>\n",
       "      <td>459</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Japanese</th>\n",
       "      <td>1</td>\n",
       "      <td>383</td>\n",
       "      <td>383</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Italian</th>\n",
       "      <td>1</td>\n",
       "      <td>357</td>\n",
       "      <td>357</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>German</th>\n",
       "      <td>1</td>\n",
       "      <td>330</td>\n",
       "      <td>330</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Czech</th>\n",
       "      <td>1</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dutch</th>\n",
       "      <td>1</td>\n",
       "      <td>195</td>\n",
       "      <td>195</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>French</th>\n",
       "      <td>1</td>\n",
       "      <td>172</td>\n",
       "      <td>172</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spanish</th>\n",
       "      <td>1</td>\n",
       "      <td>170</td>\n",
       "      <td>170</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chinese</th>\n",
       "      <td>1</td>\n",
       "      <td>158</td>\n",
       "      <td>158</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Greek</th>\n",
       "      <td>1</td>\n",
       "      <td>153</td>\n",
       "      <td>153</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Irish</th>\n",
       "      <td>1</td>\n",
       "      <td>129</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Polish</th>\n",
       "      <td>1</td>\n",
       "      <td>93</td>\n",
       "      <td>93</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Arabic</th>\n",
       "      <td>1</td>\n",
       "      <td>73</td>\n",
       "      <td>73</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Korean</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Vietnamese</th>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            cl  name  ascii_name  valid\n",
       "cl                                     \n",
       "Russian      1   486         486      1\n",
       "English      1   459         459      1\n",
       "Japanese     1   383         383      1\n",
       "Italian      1   357         357      1\n",
       "German       1   330         330      1\n",
       "Czech        1   295         295      1\n",
       "Dutch        1   195         195      1\n",
       "French       1   172         172      1\n",
       "Spanish      1   170         170      1\n",
       "Chinese      1   158         158      1\n",
       "Greek        1   153         153      1\n",
       "Irish        1   129         129      1\n",
       "Polish       1    93          93      1\n",
       "Arabic       1    73          73      1\n",
       "Korean       1    31          31      1\n",
       "Vietnamese   1    25          25      1"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[balanced_idx].groupby('cl').nunique().sort_values('ascii_name', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's record our balanced set in the dataframe: this will make it easy to reload at a later point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['bal'] = 0\n",
    "for k, v in Counter(balanced_idx).items():\n",
    "    df.loc[k, 'bal'] += v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "      <th>valid</th>\n",
       "      <th>bal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Ahn</td>\n",
       "      <td>ahn</td>\n",
       "      <td>False</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Baik</td>\n",
       "      <td>baik</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Bang</td>\n",
       "      <td>bang</td>\n",
       "      <td>False</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Byon</td>\n",
       "      <td>byon</td>\n",
       "      <td>False</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Cha</td>\n",
       "      <td>cha</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       cl  name ascii_name  valid  bal\n",
       "0  Korean   Ahn        ahn  False   13\n",
       "1  Korean  Baik       baik   True    0\n",
       "2  Korean  Bang       bang  False   13\n",
       "3  Korean  Byon       byon  False   15\n",
       "4  Korean   Cha        cha   True    0"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can always retrieve the indexes from the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx = []\n",
    "for k, v in zip(df.index, df.bal):\n",
    "    idx += [k]*v\n",
    "sorted(balanced_idx) == idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('names_clean.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benchmarks\n",
    "\n",
    "The first benchmark is random guessing/always guessing the same class.\n",
    "\n",
    "The expected return is 1/(number of classes) = 1/16 ~ 6.25%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('names_clean.csv')\n",
    "\n",
    "valid_idx = df[df.valid].index\n",
    "train_idx = df[~df.valid].index\n",
    "\n",
    "bal_idx = []\n",
    "for k, v in zip(df.index, df.bal):\n",
    "    bal_idx += [k]*v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sanity check out data\n",
    "Check training/balanced training data doesn't contain any names in validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 0)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_intersect_valid = sum(df.iloc[train_idx].ascii_name.isin(df.iloc[valid_idx].ascii_name)) \n",
    "bal_interset_valid = sum(df.iloc[bal_idx].ascii_name.isin(df.iloc[valid_idx].ascii_name))\n",
    "train_intersect_valid, bal_interset_valid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure the data looks right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "      <th>valid</th>\n",
       "      <th>bal</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cl</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Russian</th>\n",
       "      <td>1</td>\n",
       "      <td>9232</td>\n",
       "      <td>9232</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>English</th>\n",
       "      <td>1</td>\n",
       "      <td>3329</td>\n",
       "      <td>3329</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Japanese</th>\n",
       "      <td>1</td>\n",
       "      <td>952</td>\n",
       "      <td>952</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Italian</th>\n",
       "      <td>1</td>\n",
       "      <td>630</td>\n",
       "      <td>630</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>German</th>\n",
       "      <td>1</td>\n",
       "      <td>548</td>\n",
       "      <td>548</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Czech</th>\n",
       "      <td>1</td>\n",
       "      <td>434</td>\n",
       "      <td>434</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dutch</th>\n",
       "      <td>1</td>\n",
       "      <td>214</td>\n",
       "      <td>214</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spanish</th>\n",
       "      <td>1</td>\n",
       "      <td>182</td>\n",
       "      <td>182</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>French</th>\n",
       "      <td>1</td>\n",
       "      <td>180</td>\n",
       "      <td>180</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chinese</th>\n",
       "      <td>1</td>\n",
       "      <td>170</td>\n",
       "      <td>170</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Greek</th>\n",
       "      <td>1</td>\n",
       "      <td>162</td>\n",
       "      <td>162</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Irish</th>\n",
       "      <td>1</td>\n",
       "      <td>134</td>\n",
       "      <td>134</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Polish</th>\n",
       "      <td>1</td>\n",
       "      <td>93</td>\n",
       "      <td>93</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Arabic</th>\n",
       "      <td>1</td>\n",
       "      <td>73</td>\n",
       "      <td>73</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Korean</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Vietnamese</th>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            cl  name  ascii_name  valid  bal\n",
       "cl                                          \n",
       "Russian      1  9232        9232      1    3\n",
       "English      1  3329        3329      1    4\n",
       "Japanese     1   952         952      1    5\n",
       "Italian      1   630         630      1    5\n",
       "German       1   548         548      1    5\n",
       "Czech        1   434         434      1    6\n",
       "Dutch        1   214         214      1    9\n",
       "Spanish      1   182         182      1   10\n",
       "French       1   180         180      1    9\n",
       "Chinese      1   170         170      1    9\n",
       "Greek        1   162         162      1   10\n",
       "Irish        1   134         134      1   10\n",
       "Polish       1    93          93      1   11\n",
       "Arabic       1    73          73      1   13\n",
       "Korean       1    31          31      1   13\n",
       "Vietnamese   1    25          25      1   16"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[train_idx].groupby('cl').nunique().sort_values('ascii_name', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "      <th>valid</th>\n",
       "      <th>bal</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cl</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Russian</th>\n",
       "      <td>1</td>\n",
       "      <td>486</td>\n",
       "      <td>486</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>English</th>\n",
       "      <td>1</td>\n",
       "      <td>459</td>\n",
       "      <td>459</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Japanese</th>\n",
       "      <td>1</td>\n",
       "      <td>383</td>\n",
       "      <td>383</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Italian</th>\n",
       "      <td>1</td>\n",
       "      <td>357</td>\n",
       "      <td>357</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>German</th>\n",
       "      <td>1</td>\n",
       "      <td>330</td>\n",
       "      <td>330</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Czech</th>\n",
       "      <td>1</td>\n",
       "      <td>295</td>\n",
       "      <td>295</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dutch</th>\n",
       "      <td>1</td>\n",
       "      <td>195</td>\n",
       "      <td>195</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>French</th>\n",
       "      <td>1</td>\n",
       "      <td>172</td>\n",
       "      <td>172</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spanish</th>\n",
       "      <td>1</td>\n",
       "      <td>170</td>\n",
       "      <td>170</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chinese</th>\n",
       "      <td>1</td>\n",
       "      <td>158</td>\n",
       "      <td>158</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Greek</th>\n",
       "      <td>1</td>\n",
       "      <td>153</td>\n",
       "      <td>153</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Irish</th>\n",
       "      <td>1</td>\n",
       "      <td>129</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Polish</th>\n",
       "      <td>1</td>\n",
       "      <td>93</td>\n",
       "      <td>93</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Arabic</th>\n",
       "      <td>1</td>\n",
       "      <td>73</td>\n",
       "      <td>73</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Korean</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Vietnamese</th>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            cl  name  ascii_name  valid  bal\n",
       "cl                                          \n",
       "Russian      1   486         486      1    2\n",
       "English      1   459         459      1    3\n",
       "Japanese     1   383         383      1    4\n",
       "Italian      1   357         357      1    4\n",
       "German       1   330         330      1    4\n",
       "Czech        1   295         295      1    5\n",
       "Dutch        1   195         195      1    8\n",
       "French       1   172         172      1    8\n",
       "Spanish      1   170         170      1    9\n",
       "Chinese      1   158         158      1    8\n",
       "Greek        1   153         153      1    9\n",
       "Irish        1   129         129      1    9\n",
       "Polish       1    93          93      1   11\n",
       "Arabic       1    73          73      1   13\n",
       "Korean       1    31          31      1   13\n",
       "Vietnamese   1    25          25      1   16"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[bal_idx].groupby('cl').nunique().sort_values('ascii_name', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "      <th>valid</th>\n",
       "      <th>bal</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cl</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arabic</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chinese</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Czech</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dutch</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>English</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>French</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>German</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Greek</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Irish</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Italian</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Japanese</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Korean</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Polish</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Russian</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spanish</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Vietnamese</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            cl  name  ascii_name  valid  bal\n",
       "cl                                          \n",
       "Arabic       1    30          30      1    1\n",
       "Chinese      1    30          30      1    1\n",
       "Czech        1    30          30      1    1\n",
       "Dutch        1    30          30      1    1\n",
       "English      1    30          30      1    1\n",
       "French       1    30          30      1    1\n",
       "German       1    30          30      1    1\n",
       "Greek        1    30          30      1    1\n",
       "Irish        1    30          30      1    1\n",
       "Italian      1    30          30      1    1\n",
       "Japanese     1    30          30      1    1\n",
       "Korean       1    30          30      1    1\n",
       "Polish       1    30          30      1    1\n",
       "Russian      1    30          30      1    1\n",
       "Spanish      1    30          30      1    1\n",
       "Vietnamese   1    30          30      1    1"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[valid_idx].groupby('cl').nunique().sort_values('ascii_name', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Picking any one class in validation will give 1/16 = 6.25%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0625"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(df[df.valid] == 'Korean').cl.sum() / df.valid.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## n-grams and naive Bayes\n",
    "\n",
    "A reasonable way to guess a language is by the frequency of characters and pairs of characters.\n",
    "\n",
    "For example 'cz' is very rare in English, but quite common in the slavic languages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = 'zozrov'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A function to count the occurances of sequences of one, two or three letters (in general these sequences are called \"n-grams\" particularly when referring to sequences of words)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Counter({'z': 2, 'o': 2, 'r': 1, 'v': 1}),\n",
       " Counter({'zo': 1, 'oz': 1, 'zr': 1, 'ro': 1, 'ov': 1}),\n",
       " Counter({'zoz': 1, 'ozr': 1, 'zro': 1, 'rov': 1}))"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def ngrams(s,n=1):\n",
    "    parts = [s[i:] for i in range(n)] # e.g. ['zozrov', 'ozrov', 'zrov']\n",
    "    return Counter(''.join(_) for _ in zip(*parts))\n",
    "\n",
    "ngrams(name, 1), ngrams(name, 2), ngrams(name, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.assign(letters=df.ascii_name.apply(ngrams))\n",
    "df = df.assign(bigrams=df.ascii_name.apply(ngrams, n=2))\n",
    "df = df.assign(trigrams=df.ascii_name.apply(ngrams, n=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "      <th>valid</th>\n",
       "      <th>bal</th>\n",
       "      <th>letters</th>\n",
       "      <th>bigrams</th>\n",
       "      <th>trigrams</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Ahn</td>\n",
       "      <td>ahn</td>\n",
       "      <td>False</td>\n",
       "      <td>13</td>\n",
       "      <td>{'a': 1, 'h': 1, 'n': 1}</td>\n",
       "      <td>{'ah': 1, 'hn': 1}</td>\n",
       "      <td>{'ahn': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Baik</td>\n",
       "      <td>baik</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "      <td>{'b': 1, 'a': 1, 'i': 1, 'k': 1}</td>\n",
       "      <td>{'ba': 1, 'ai': 1, 'ik': 1}</td>\n",
       "      <td>{'bai': 1, 'aik': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Bang</td>\n",
       "      <td>bang</td>\n",
       "      <td>False</td>\n",
       "      <td>13</td>\n",
       "      <td>{'b': 1, 'a': 1, 'n': 1, 'g': 1}</td>\n",
       "      <td>{'ba': 1, 'an': 1, 'ng': 1}</td>\n",
       "      <td>{'ban': 1, 'ang': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Byon</td>\n",
       "      <td>byon</td>\n",
       "      <td>False</td>\n",
       "      <td>15</td>\n",
       "      <td>{'b': 1, 'y': 1, 'o': 1, 'n': 1}</td>\n",
       "      <td>{'by': 1, 'yo': 1, 'on': 1}</td>\n",
       "      <td>{'byo': 1, 'yon': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Cha</td>\n",
       "      <td>cha</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "      <td>{'c': 1, 'h': 1, 'a': 1}</td>\n",
       "      <td>{'ch': 1, 'ha': 1}</td>\n",
       "      <td>{'cha': 1}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       cl  name ascii_name  valid  bal                           letters  \\\n",
       "0  Korean   Ahn        ahn  False   13          {'a': 1, 'h': 1, 'n': 1}   \n",
       "1  Korean  Baik       baik   True    0  {'b': 1, 'a': 1, 'i': 1, 'k': 1}   \n",
       "2  Korean  Bang       bang  False   13  {'b': 1, 'a': 1, 'n': 1, 'g': 1}   \n",
       "3  Korean  Byon       byon  False   15  {'b': 1, 'y': 1, 'o': 1, 'n': 1}   \n",
       "4  Korean   Cha        cha   True    0          {'c': 1, 'h': 1, 'a': 1}   \n",
       "\n",
       "                       bigrams              trigrams  \n",
       "0           {'ah': 1, 'hn': 1}            {'ahn': 1}  \n",
       "1  {'ba': 1, 'ai': 1, 'ik': 1}  {'bai': 1, 'aik': 1}  \n",
       "2  {'ba': 1, 'an': 1, 'ng': 1}  {'ban': 1, 'ang': 1}  \n",
       "3  {'by': 1, 'yo': 1, 'on': 1}  {'byo': 1, 'yon': 1}  \n",
       "4           {'ch': 1, 'ha': 1}            {'cha': 1}  "
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to guess the name using [Naive Bayes](https://en.wikipedia.org/wiki/Naive_Bayes_classifier).\n",
    "\n",
    "TL;DR: This is a *really* simple model that works quite well and will give a good benchmark.\n",
    "\n",
    "This uses \"Bayes Rule\" which uses the data to answer questions like: \"given the name contains the bigram 'ah' what's the probability it's Korean?\".\n",
    "\n",
    "The \"Naive\" part means that that we assume all these probabilities are independent (knowing it contains 'ah' doesn't tell you anything about the fact it contains 'hn'). Even though this definitely isn't true, it's often a reasonable approximation.\n",
    "\n",
    "This makes it really fast and simple to fit a model and often works well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "vd1 = DictVectorizer(sparse=False)\n",
    "vd2 = DictVectorizer(sparse=False)\n",
    "vd3 = DictVectorizer(sparse=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df.cl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "letters = vd1.fit_transform(df.letters)\n",
    "bigrams = vd2.fit_transform(df.bigrams)\n",
    "trigrams = vd3.fit_transform(df.trigrams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The letters matrix contains the number of times each of the 28 letters occurs (e.g. number of spaces, number of apostrophes, number of 'a', ...)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' ', \"'\", 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h']"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vd1.get_feature_names()[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 1., 0., ..., 0., 0., 0., 0.],\n",
       "       [0., 0., 1., 1., ..., 0., 0., 0., 0.],\n",
       "       [0., 0., 1., 1., ..., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 1., ..., 0., 0., 1., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., 0., ..., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., ..., 0., 0., 0., 0.],\n",
       "       [0., 0., 1., 0., ..., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., ..., 0., 0., 1., 0.]])"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "letters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly bigrams and trigrams contains the number of times each sequence of 2 or 3 letters occurs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([' a', ' b', ' c', ' e', ' f'], ['zu', 'zv', 'zw', 'zy', 'zz'])"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vd2.get_feature_names()[:5], vd2.get_feature_names()[-5:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((16869, 28), (16869, 623), (16869, 5794), (16869,))"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "letters.shape, bigrams.shape, trigrams.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How good a model can we get looking at individual letters (e.g. saying 'z' occurs much more frequently in Chinese than in English names)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "letter_nb = MultinomialNB()\n",
    "letter_nb.fit(letters[train_idx],y[train_idx])\n",
    "\n",
    "bal_letter_nb = MultinomialNB()\n",
    "bal_letter_nb.fit(letters[bal_idx],y[bal_idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The balanced set does mut better than random; around 33%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.14791666666666667, 0.33541666666666664)"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "letter_pred = letter_nb.predict(letters[valid_idx])\n",
    "bal_letter_pred = bal_letter_nb.predict(letters[valid_idx])\n",
    "(letter_pred == y[valid_idx]).mean(), (bal_letter_pred == y[valid_idx]).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a function to test the Naive Bayes on any dataset; fitting on the whole dataset and the balanced dataset separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nb(x):\n",
    "    model = MultinomialNB()\n",
    "    model.fit(x[train_idx], y[train_idx])\n",
    "    preds = model.predict(x[valid_idx])\n",
    "    acc_train = (preds == y[valid_idx]).mean()\n",
    "    \n",
    "    model = MultinomialNB()\n",
    "    model.fit(x[bal_idx], y[bal_idx])\n",
    "    preds = model.predict(x[valid_idx])\n",
    "    acc_bal = (preds == y[valid_idx]).mean()\n",
    "    \n",
    "    return acc_train, acc_bal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.14791666666666667, 0.33541666666666664)"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb(letters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using bigrams and a balanced training set gives a much better prediction performance 53% (up from the baseline of 6.25%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.35833333333333334, 0.5291666666666667)"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb(bigrams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding letters doesn't make much difference (which isn't surprising "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.3854166666666667, 0.5166666666666667)"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb(np.concatenate((letters, bigrams), axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trigrams alone also performs worse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.33958333333333335, 0.4895833333333333)"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb(trigrams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try every combination with trigrams:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.24375, 0.5083333333333333)"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb(np.concatenate((letters, trigrams), axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.36875, 0.5416666666666666)"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb(np.concatenate((bigrams, trigrams), axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.32916666666666666, 0.55625)"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb(np.concatenate((letters, bigrams, trigrams), axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "None of them significantly outperform the simple bigram model (with 623 parameters; we could probably remove some of the uncommon ones without too many problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examining the Bigram Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remove the bigrams that only occur once as they have practically no value (and there's 100 of them)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "503"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "common_bigrams = (bigrams[bal_idx].sum(axis=0)) >= 2\n",
    "common_bigrams.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(16869, 503)"
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "common_bigram_index = [i for i, t in enumerate(common_bigrams) if t]\n",
    "bigrams_min = bigrams[:, common_bigram_index]\n",
    "bigrams_min.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_model = MultinomialNB()\n",
    "bigram_model.fit(bigrams_min[bal_idx], y[bal_idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get around 53% accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5291666666666667"
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_pred = bigram_model.predict(bigrams_min[valid_idx])\n",
    "(bigram_pred == y[valid_idx]).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([244,   9,  20,  18, ...,  86, 422, 143, 431])"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_prob = bigram_model.predict_proba(bigrams_min[valid_idx])\n",
    "bigram_prob.max(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>cl</th>\n",
       "      <th>pred</th>\n",
       "      <th>prob</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16557</th>\n",
       "      <td>Kotsiopoulos</td>\n",
       "      <td>Greek</td>\n",
       "      <td>Greek</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012</th>\n",
       "      <td>Rooijakker</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16470</th>\n",
       "      <td>Akrivopoulos</td>\n",
       "      <td>Greek</td>\n",
       "      <td>Greek</td>\n",
       "      <td>0.999999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>826</th>\n",
       "      <td>Warszawski</td>\n",
       "      <td>Polish</td>\n",
       "      <td>Polish</td>\n",
       "      <td>0.999998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>Romeijnders</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>0.999997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16478</th>\n",
       "      <td>Antonopoulos</td>\n",
       "      <td>Greek</td>\n",
       "      <td>Greek</td>\n",
       "      <td>0.999995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>813</th>\n",
       "      <td>Sokolowski</td>\n",
       "      <td>Polish</td>\n",
       "      <td>Polish</td>\n",
       "      <td>0.999994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>839</th>\n",
       "      <td>Zdunowski</td>\n",
       "      <td>Polish</td>\n",
       "      <td>Polish</td>\n",
       "      <td>0.999950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>Romeijn</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>0.999917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16497</th>\n",
       "      <td>Chrysanthopoulos</td>\n",
       "      <td>Greek</td>\n",
       "      <td>Greek</td>\n",
       "      <td>0.999895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2053</th>\n",
       "      <td>Sneijers</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>0.999792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2031</th>\n",
       "      <td>Schwarzenberg</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>German</td>\n",
       "      <td>0.999774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>795</th>\n",
       "      <td>Rudawski</td>\n",
       "      <td>Polish</td>\n",
       "      <td>Polish</td>\n",
       "      <td>0.999751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16715</th>\n",
       "      <td>De sauveterre</td>\n",
       "      <td>French</td>\n",
       "      <td>French</td>\n",
       "      <td>0.999604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1160</th>\n",
       "      <td>Kawagichi</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>0.999600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   name        cl      pred      prob\n",
       "16557      Kotsiopoulos     Greek     Greek  1.000000\n",
       "2012         Rooijakker     Dutch     Dutch  1.000000\n",
       "16470      Akrivopoulos     Greek     Greek  0.999999\n",
       "826          Warszawski    Polish    Polish  0.999998\n",
       "1997        Romeijnders     Dutch     Dutch  0.999997\n",
       "16478      Antonopoulos     Greek     Greek  0.999995\n",
       "813          Sokolowski    Polish    Polish  0.999994\n",
       "839           Zdunowski    Polish    Polish  0.999950\n",
       "1996            Romeijn     Dutch     Dutch  0.999917\n",
       "16497  Chrysanthopoulos     Greek     Greek  0.999895\n",
       "2053           Sneijers     Dutch     Dutch  0.999792\n",
       "2031      Schwarzenberg     Dutch    German  0.999774\n",
       "795            Rudawski    Polish    Polish  0.999751\n",
       "16715     De sauveterre    French    French  0.999604\n",
       "1160          Kawagichi  Japanese  Japanese  0.999600"
      ]
     },
     "execution_count": 217,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_preds = (df\n",
    "                .iloc[valid_idx]\n",
    "                .assign(pred = bigram_pred)[['name', 'cl', 'pred']]\n",
    "                .assign(prob = bigram_prob.max(axis=1)))\n",
    "bigram_preds.sort_values('prob', ascending=False).head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The names it's least confident with: they typically seem to be quite short"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>cl</th>\n",
       "      <th>pred</th>\n",
       "      <th>prob</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2907</th>\n",
       "      <td>Do</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Irish</td>\n",
       "      <td>0.176679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Mo</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>0.179534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>So</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Korean</td>\n",
       "      <td>0.188088</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Si</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Greek</td>\n",
       "      <td>0.190236</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13775</th>\n",
       "      <td>Prigojin</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Italian</td>\n",
       "      <td>0.191639</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Seok</td>\n",
       "      <td>Korean</td>\n",
       "      <td>French</td>\n",
       "      <td>0.197154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Cho</td>\n",
       "      <td>Korean</td>\n",
       "      <td>German</td>\n",
       "      <td>0.202991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1091</th>\n",
       "      <td>Isobe</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>English</td>\n",
       "      <td>0.206442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Sin</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Italian</td>\n",
       "      <td>0.218300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5332</th>\n",
       "      <td>Ingram</td>\n",
       "      <td>English</td>\n",
       "      <td>Spanish</td>\n",
       "      <td>0.220205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2935</th>\n",
       "      <td>Ta</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>0.226875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2700</th>\n",
       "      <td>Ban</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>0.228022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1697</th>\n",
       "      <td>Togo</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>0.236658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Cha</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Irish</td>\n",
       "      <td>0.239172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3445</th>\n",
       "      <td>Graner</td>\n",
       "      <td>German</td>\n",
       "      <td>Spanish</td>\n",
       "      <td>0.240844</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           name          cl        pred      prob\n",
       "2907         Do  Vietnamese       Irish  0.176679\n",
       "24           Mo      Korean    Japanese  0.179534\n",
       "47           So      Korean      Korean  0.188088\n",
       "45           Si      Korean       Greek  0.190236\n",
       "13775  Prigojin     Russian     Italian  0.191639\n",
       "41         Seok      Korean      French  0.197154\n",
       "5           Cho      Korean      German  0.202991\n",
       "1091      Isobe    Japanese     English  0.206442\n",
       "46          Sin      Korean     Italian  0.218300\n",
       "5332     Ingram     English     Spanish  0.220205\n",
       "2935         Ta  Vietnamese    Japanese  0.226875\n",
       "2700        Ban     Chinese  Vietnamese  0.228022\n",
       "1697       Togo    Japanese    Japanese  0.236658\n",
       "4           Cha      Korean       Irish  0.239172\n",
       "3445     Graner      German     Spanish  0.240844"
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_preds.sort_values('prob', ascending=True).head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The names it's most confidently wrong with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>cl</th>\n",
       "      <th>pred</th>\n",
       "      <th>prob</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2031</th>\n",
       "      <td>Schwarzenberg</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>German</td>\n",
       "      <td>0.999774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16578</th>\n",
       "      <td>Malihoudis</td>\n",
       "      <td>Greek</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>0.992311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16576</th>\n",
       "      <td>Louverdis</td>\n",
       "      <td>Greek</td>\n",
       "      <td>French</td>\n",
       "      <td>0.990256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4758</th>\n",
       "      <td>Fairbrace</td>\n",
       "      <td>English</td>\n",
       "      <td>Irish</td>\n",
       "      <td>0.987143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3743</th>\n",
       "      <td>Spellmeyer</td>\n",
       "      <td>German</td>\n",
       "      <td>English</td>\n",
       "      <td>0.976530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16468</th>\n",
       "      <td>Adamou</td>\n",
       "      <td>Greek</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>0.973496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3009</th>\n",
       "      <td>De la fuente</td>\n",
       "      <td>Spanish</td>\n",
       "      <td>French</td>\n",
       "      <td>0.969431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3011</th>\n",
       "      <td>De leon</td>\n",
       "      <td>Spanish</td>\n",
       "      <td>French</td>\n",
       "      <td>0.964697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3263</th>\n",
       "      <td>Boulos</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Greek</td>\n",
       "      <td>0.962321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16513</th>\n",
       "      <td>Egonidis</td>\n",
       "      <td>Greek</td>\n",
       "      <td>Italian</td>\n",
       "      <td>0.954264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2478</th>\n",
       "      <td>Suchanka</td>\n",
       "      <td>Czech</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>0.949000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2515</th>\n",
       "      <td>Weichert</td>\n",
       "      <td>Czech</td>\n",
       "      <td>German</td>\n",
       "      <td>0.946457</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5476</th>\n",
       "      <td>Keene</td>\n",
       "      <td>English</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>0.944270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3511</th>\n",
       "      <td>Jaeger</td>\n",
       "      <td>German</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>0.938891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3174</th>\n",
       "      <td>Attia</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Italian</td>\n",
       "      <td>0.935905</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name       cl      pred      prob\n",
       "2031   Schwarzenberg    Dutch    German  0.999774\n",
       "16578     Malihoudis    Greek    Arabic  0.992311\n",
       "16576      Louverdis    Greek    French  0.990256\n",
       "4758       Fairbrace  English     Irish  0.987143\n",
       "3743      Spellmeyer   German   English  0.976530\n",
       "16468         Adamou    Greek    Arabic  0.973496\n",
       "3009    De la fuente  Spanish    French  0.969431\n",
       "3011         De leon  Spanish    French  0.964697\n",
       "3263          Boulos   Arabic     Greek  0.962321\n",
       "16513       Egonidis    Greek   Italian  0.954264\n",
       "2478        Suchanka    Czech  Japanese  0.949000\n",
       "2515        Weichert    Czech    German  0.946457\n",
       "5476           Keene  English     Dutch  0.944270\n",
       "3511          Jaeger   German     Dutch  0.938891\n",
       "3174           Attia   Arabic   Italian  0.935905"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_preds[bigram_preds.cl != bigram_preds.pred].sort_values('prob', ascending=False).head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our very simple system does great on Japanese and Russian, but relatively poorly on Vietnamese where our data is most sparse (but still much better than random)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "cl\n",
       "Japanese      0.866667\n",
       "Russian       0.733333\n",
       "Polish        0.666667\n",
       "Irish         0.666667\n",
       "Dutch         0.633333\n",
       "Italian       0.600000\n",
       "Greek         0.533333\n",
       "German        0.500000\n",
       "English       0.500000\n",
       "Spanish       0.466667\n",
       "French        0.466667\n",
       "Arabic        0.433333\n",
       "Czech         0.400000\n",
       "Chinese       0.400000\n",
       "Korean        0.366667\n",
       "Vietnamese    0.233333\n",
       "Name: yes, dtype: float64"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(bigram_preds\n",
    " .assign(yes=bigram_preds.cl == bigram_preds.pred)\n",
    " .groupby('cl')\n",
    " .yes\n",
    " .mean()\n",
    " .sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Arabic', 'Irish', 'German', 'Dutch', ..., 'Russian', 'Polish', 'Irish', 'Italian'], dtype='<U10')"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[11,  1,  0,  6, ...,  0,  0,  2,  1],\n",
       "       [ 0, 18,  0,  1, ...,  2,  1,  2,  1],\n",
       "       [ 0,  1, 20,  1, ...,  0,  0,  0,  0],\n",
       "       [ 1,  0,  0, 26, ...,  1,  0,  0,  0],\n",
       "       ...,\n",
       "       [ 1,  2,  0,  1, ..., 15,  0,  1,  1],\n",
       "       [ 0,  2,  1,  1, ...,  1, 22,  0,  0],\n",
       "       [ 0,  2,  1,  1, ...,  0,  1, 16,  3],\n",
       "       [ 0,  3,  1,  0, ...,  1,  1,  4, 14]])"
      ]
     },
     "execution_count": 246,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm = confusion_matrix(y[valid_idx], bigram_pred, labels=y.unique())\n",
    "cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=90)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vietnamese is often confused for Chinese (which makes sense) and Irish (which doesn't).\n",
    "Korean is often confused for Japanese.\n",
    "Spanish is often confused for Italian."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyMAAANICAYAAADU4DBcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3Xl8VNXdx/HvDxJABBFEWWXfNAgoCVRBRCkuBbUuiJQHpYJohVqXWq1atS6tVat1abHSulR5xEqtPqgguICKyioiggIVFIIo2BZBBEL4PX/MJAwwLIFkzkn4vF+veWXmzp073zn3zoVfzrkn5u4CAAAAgEyrFDoAAAAAgP0TxQgAAACAIChGAAAAAARBMQIAAAAgCIoRAAAAAEFQjAAAAAAIgmIEAAAAQBAUIwAAAACCoBgBAAAAEERW6AAAAADA/qLyQU3dN38XOsYu+XerXnH3UzPxXhQjAAAAQIb45u9Ute15oWPs0oY5f6ybqfdimBYAAACAIChGAAAAAATBMC0AAAAgY0wy+gOK0BIAAAAAgqAYAQAAABAEw7QAAACATDFJZqFTRIOeEQAAAABBUIwAAAAACIJhWgAAAEAmMZtWMVoCAAAAQBAUIwAAAACCYJgWAAAAkEnMplWMnhEAAAAAQVCMAAAAAAiCYVoAAABAxhizaaWgJQAAAAAEQTECAAAAIAiKEQAAAABBcM0IAAAAkElM7VuMnhEAAAAAQVCMAAAAAAiCYVoAAABAppiY2jcFLQEAAAAgCIoRAAAAAEEwTAsAAADIGGM2rRT0jAAAAAAIgmIEAAAAwB4zs8PN7A0zW2BmH5nZz5LLbzGzfDObk7z9YHfbYpgWAAAAkEnlfzatzZKudvfZZlZT0iwzm5R87j53v2dPN0QxAgAAAGCPufsXkr5I3l9rZgskNdqbbZX7sgwAAABAqaprZjNTbsN2tqKZNZN0tKRpyUUjzGyumT1qZrV390b0jAAAAACZFP9sWqvdPXd3K5lZDUn/kHSFu39jZiMl3SbJkz9/L+miXW2DnhEAAAAAJWJm2UoUIqPd/TlJcvcv3b3Q3bdIGiWpy+62QzECAAAAYI+ZmUn6q6QF7n5vyvIGKaudJWne7rbFMC0AAAAAJdFN0iBJH5rZnOSy6yUNMLNOSgzTWirpkt1tiGIEAAAAyBgr91P7uvvbktJd+PJySbdVvlsCAAAAQLlFMQIAAAAgCIZpAQAAAJliKg9T+2YMPSMAAAAAgqAYAQAAABAEw7QAAACATCrns2mVJloCAAAAQBAUIwAAAACCYJgWAAAAkDHl/48eliZaAgAAAEAQFCMAAAAAgmCYFgAAAJBJlfijh0XoGQEAAAAQBMUIAAAAgCAoRgAAAAAEwTUjAAAAQKaYmNo3BS0BAAAAIAiKEQAAAABBMEwLAAAAyCRjat8i9IwAAAAACIJiBAAAAEAQDNMCAAAAMsaYTSsFLQEAAAAgCIoRAAAAAEEwTAsAAADIJGbTKkbPCAAAAIAgKEYAAAAABMEwLQAAACCTmE2rGC0BAAAAIAiKEQAAAABBMEwLAAAAyBQzZtNKQc8IAAAAgCAoRgAAAAAEQTECAAAAIAiuGQEAAAAyial9i9ESAAAAAIKgGAEAAAAQBMO0AAAAgExiat9i9IwAAAAACIJiBAAAAEAQDNMCAAAAMsaYTSsFLQEAAAAgCIoRAAAAAEEwTAsAAADIJGbTKkbPCAAAAIAgKEYAAAAABMEwLQAAACBTTMymlYKWAAAAABAExQgAAACAIChGAAAAAATBNSMAAABAxvAX2FPREgBQjpjZAWY2zszWmNmz+7CdgWY2sTSzhWJmx5vZJ6FzAABKjmIEAMqAmf3IzGaa2Toz+8LMxptZ91LY9LmS6kk6xN377e1G3H20u59cCnnKlJm5mbXa1Tru/pa7t81UJgBA6WGYFgCUMjO7StJ1ki6V9IqkTZJOlXSmpLf3cfNNJS109837uJ0KwcyyaAsA5Q5/gb0YPSMAUIrMrJakWyUNd/fn3P1bdy9w93Hufk1ynapm9gczW5G8/cHMqiaf62lmy83sajP7Ktmr8uPkc7+WdJOk/skelyFmdouZPZXy/s2SvQlZyceDzexTM1trZkvMbGDK8rdTXnecmc1IDv+aYWbHpTw32cxuM7Opye1MNLO6O/n8Rfl/kZL/h2b2AzNbaGb/NrPrU9bvYmbvmtl/k+s+ZGZVks+9mVztg+Tn7Z+y/WvNbKWkx4qWJV/TMvkexyQfNzSz1WbWc592LACgTFCMAEDpOlZSNUn/3MU6N0j6nqROkjpK6iLpxpTn60uqJamRpCGS/mhmtd39Zkm/kfSMu9dw97/uKoiZHSjpAUmnuXtNScdJmpNmvTqSXkque4ikeyW9ZGaHpKz2I0k/lnSYpCqSfr6Lt66vRBs0UqJ4GiXpfyR1lnS8pJvMrEVy3UJJV0qqq0Tb9ZJ0mSS5e4/kOh2Tn/eZlO3XUaKXaFjqG7v7vyRdK2m0mVWX9Jikx9198i7yAgACoRgBgNJ1iKTVuxk6NFDSre7+lbuvkvRrSYNSni9IPl/g7i9LWidpb6+J2CKpvZkd4O5fuPtHadbpI2mRuz/p7pvd/WlJH0s6PWWdx9x9obt/J+nvShRSO1Mg6Q53L5A0RolC4353X5t8/48kdZAkd5/l7u8l33eppD9LOmEPPtPN7r4xmWcb7j5K0iJJ0yQ1UKL4A4B4WKW4bxlEMQIApetrSXWLhkntRENJn6U8/iy5rHgb2xUz6yXVKGkQd/9WUn8lrl35wsxeMrN2e5CnKFOjlMcrS5Dna3cvTN4vKha+THn+u6LXm1kbM3vRzFaa2TdK9PykHQKWYpW7b9jNOqMktZf0oLtv3M26AIBAKEYAoHS9K2mDpB/uYp0VSgwxKtIkuWxvfCupesrj+qlPuvsr7t5biR6Cj5X4T/ru8hRlyt/LTCUxUolcrd39IEnXS9rdlZ2+qyfNrIakP0j6q6RbksPQAAARohgBgFLk7muUuE7ij8kLt6ubWbaZnWZmdyVXe1rSjWZ2aPJC8JskPbWzbe7GHEk9zKxJ8uL5XxY9YWb1zOyM5LUjG5UY7lWYZhsvS2qTnI44y8z6SzpS0ot7makkakr6RtK6ZK/NT7Z7/ktJLXZ41a7dL2mWuw9V4lqYh/c5JQCUJrO4bxlEMQIApczd75V0lRIXpa+StEzSCEnPJ1e5XdJMSXMlfShpdnLZ3rzXJEnPJLc1S9sWEJUkXa1Ez8e/lbgW47I02/haUt/kul9L+oWkvu6+em8yldDPlbg4fq0SvTbPbPf8LZKeSM62dd7uNmZmZyoxjfKlyUVXSTqmaBYxAEBczH2Xvd0AAAAASkmlg5t61Z5xz6ux4YVLZrl7bibeiz96CAAAAGSKWcZnrIoZLQEAAAAgCIoRAAAAAEFQjAAAAAAIgmtGAAAAgEzK8PS5MaMYKWMH1qrjtes32v2KGXbogVVDR0irUqTfzfWb0v1phvCqZMXbuZkV6c7cwgSCFUKkh5c2c4CVSKzniZjFeoxVivA/159/vlRfr14dXzBsg2KkjNWu30g/+/Pzu18xwy7u2ix0hLSqZVcOHSGtuZ+vCR0hraZ1q+9+pUBqVc8OHSGtDQVxFpYomVjPFWvWF4SOUK7Eep6IWazHWNXs+H45dmK3rqEjYA9QjAAAAAAZZBH2JIUSXxkLAAAAYL9AMQIAAAAgCIZpAQAAABliYphWKnpGAAAAAARBMQIAAAAgCIZpAQAAAJliyRsk0TMCAAAAIBCKEQAAAABBUIxE6O+/u06/PquLfv/j04qXzZ38sn4/+FRde1JrLfvkw4DpthpxyVC1btpAx+Z2DB1lGxNfmaAOOW2V066V7r7rztBxiq395r+6bvgFOq93nvqf3EUfzp4eOpIkKX/5Mp3dt7eOzztKPbp21KiRD4aOVCzGfRnrcU+ukovx+Ir1+xhrLinO/SjFmyvWfRnzuaL0mczivmUSxUiEck89W0N+9+g2y+o1b6NBt/5JzTvkBUq1owGDLtDY518KHWMbhYWFuuLy4Xph3Hi9P3e+nh3ztBbMnx86liTp3luv07E9vq+/T5qhp158W81atQkdSZKUlZWlW26/S2/N+FAvv/q2Hhs1Up98HL7NYt2XMR73ErlKKtbjK9bvY6y5Yt2PseaS4t2XsZ4rUPYoRiLUomMXVT/o4G2W1WvaSoc1aREoUXrduvdQ7Tp1QsfYxozp09WyZSs1b9FCVapUUb/+5+vFcS+EjqV1a7/R+zPe0RnnDZIkZVepoprb7eNQ6tVvoA6djpYk1ahZU63bttPKFSsCp4p3X8Z43EvkKqlYj69Yv4+x5op1P8aaS4p3X8Z6rkDZoxhBhbJiRb4aNz68+HGjRo2Vn58fMFHCimVLVbtOXd32i8s06PTjdccvf6rv1n8bOtYOPv9sqebN/UDH5HYJHSXafYmKoTwcXzF9H1PFlCvW/Rhrru3FtC+x/yq3xYiZrUu5/wMzW2RmTUJmQnjuvsOyGP7KaeHmQn3y0Qc6e+AQPTnuLVU7oLqeePi+0LG28e26dRo6qL9u/e09qnnQQaHjRLsvUTHEfnzF9n0sEluuWPdjrLlSxbYv9zehrwnhmpFSZGa9JD0o6VR3/3wPX8PfV6mgGjVqrOXLlxU/zs9froYNGwZMlHBYg4Y6rH5Dte+UK0k66bQz9clHcwOn2qqgoEBDBvXX2ecNUJ8zzgodR1K8+xIVQ8zHV4zfRynOXLHux1hzFYlxX2L/Va6LETM7XtIoSX3c/V/JZU3N7DUzm5v82SS5/HEzu9fM3pD0OzM70MweNbMZZva+mZ2ZXK+Zmb1lZrOTt+OSy3ua2WQzG2tmH5vZaIvt1xxQbl6eFi9epKVLlmjTpk169pkx6tP3jNCxdMih9XRYg8b67NNFkqSZ70xR81ZtA6dKcHddOWKYWrdtp0tHXBE6TrFY9yUqhliPr1i/j7HminU/xppLindfYv9VnouRqpJekPRDd/84ZflDkv7m7h0kjZb0QMpzbSR9392vlnSDpNfdPU/SiZLuNrMDJX0lqbe7HyOp/3avP1rSFZKOlNRCUrey+GCjb7tCfxzeT6uWLdEd/bpp+kt/17y3JuqOft302fz39dgvh+ov1wwui7cukSEXDtTJPbtr8cJPlNOqqZ58/NHdv6iMZWVl6b77H9LpfU5Rp6OO0Dn9ztOROTmhY0mSfn7z73TTlRdr4A+O08L5H2rwZVeHjiRJmv7eOxo7ZrTefvMN9eqeq17dc/XqxPGhY0W7L2M87iVylVSsx1es38dYc8W6H2PNJcW7L2M9V5SV0MOwYhqmZenGNZYHZrZe0uuS/uXuP0tZvlpSA3cvMLNsSV+4e10ze1zSG+7+RHK9mZKqSdqcfGkdSadIWqFEQdNJUqGkNu5e3cx6SrrB3XsnXz9S0lR3fypNtmGShknSwfUadr5+zJul/vn31cVdm4WOkFa17MqhI6Q19/M1oSOk1bRu9dARdqpW9ezQEdLaUFAYOgJKQaznijXrC0JHKFdiPU/ELNZjrGp2fL/fPrFbV70/e2Z0o1gq12nuB57869AxdmntMxfOcvfcTLxXfEfOntsi6TxJeWZ2/S7WS622UqcvMknnuHun5K2Juy+QdKWkLyV1lJQrqUrKazam3C+UlPbaE3d/xN1z3T33wFpMUwcAAACkU56LEbn7ekl9JQ00syHJxe9IOj95f6Ckt3fy8lck/bToug8zOzq5vJYSvSlbJA2SFOev3wAAAFAuhR6GFdMwrXI/q5S7/9vMTpX0ZnKI1uWSHjWzayStkvTjnbz0Nkl/kDQ3WZAsVaKw+ZOkf5hZP0lvaNveFAAAAAClpNwWI+5eI+X+MknNU54+Kc36g7d7/J2kS9Kst0hSh5RFv0wunyxpcsp6I/YqOAAAAABJ5bgYAQAAAModS94gqZxfMwIAAACg/KIYAQAAABAEw7QAAACADDFlfsaqmNEzAgAAACAIihEAAAAAQTBMCwAAAMgghmltRc8IAAAAgCAoRgAAAAAEQTECAAAAIAiuGQEAAAAyiGtGtqJnBAAAAEAQFCMAAAAAgmCYVhmrV6Oqftq9ZegYO/je7a+FjpDWezf2Ch0hrTYNaoSOkFa17MqhI5Q7tFnF8Nnq9aEjpFWvVtXQEdKK9bhfs74gdISdqprN72tRdhimtRXfNAAAAABBUIwAAAAACIJhWgAAAECmWPIGSfSMAAAAAAiEYgQAAABAEAzTAgAAADKI2bS2omcEAAAAQBAUIwAAAACCYJgWAAAAkCEmY5hWCnpGAAAAAARBMQIAAAAgCIqRcmDiKxPUIaetctq10t133Rk0yy1nHqHXrzleYy/rWrysbf0a+tvQXD1zaReNHpan9o0OCpgwrvZKNeKSoWrdtIGOze0YOsoOYm0zcpUMuUqusLBQZ/U+VpcMOid0lGKxniti3I/5y5fp7L69dXzeUerRtaNGjXwwdKRise7HWNss1vZC2aMYiVxhYaGuuHy4Xhg3Xu/Pna9nxzytBfPnB8vzf3O+0GVPzdlm2RW9W+nPk5eo/8PTNfKNT3VF71aB0sXXXqkGDLpAY59/KXSMHcTaZuQiVyb8bdQf1aJ129AxthHjuSLW/ZiVlaVbbr9Lb834UC+/+rYeGzVSn3wcPpcU536U4m2zWNurrJhZ1LdMohiJ3Izp09WyZSs1b9FCVapUUb/+5+vFcS8EyzP7s//qm+8KtlnmLh1YtbIkqUbVLK1auzFENEnxtVeqbt17qHadOqFj7CDWNiMXucrayhX5mvLaBPX70eDQUbYR47ki1v1Yr34Ddeh0tCSpRs2aat22nVauWBE4VUKM+1GKt81ibS+UPYqRyK1Yka/GjQ8vftyoUWPl5+cHTLSjuycs1JUnt9aEK7vpqpNb6YFX/xUsS3lor9jE2mbkKhlyldxvbvqFfn7jHbJK/FO4OzHvxyKff7ZU8+Z+oGNyu4SOUm7QZohBhTkDm9m65M9mZvajPVi/mZnNS97PNbMHyjrj3nD3HZbFNh1cv7zGumfCQp1631Td88oi3XzmEcGylIf2ik2sbUaukiFXybwxabwOqXuo2nc8OnSUciHW/Vjk23XrNHRQf93623tU86Cw1y2WF7RZYBb5LYMqTDGSopmk3RYjqdx9prtfXjZx9k2jRo21fPmy4sf5+cvVsGHDgIl2dHrHBnptwSpJ0sSPvgp6AXt5aK/YxNpm5CoZcpXM7Onv6vWJL+mkvCN09aUXatrbU3TN8ItCx4pWrPtRkgoKCjRkUH+dfd4A9TnjrNBxygXaDDGpiMXInZKON7M5ZnZlsgfkLTObnbwdt/0LzKynmb2YvN/FzN4xs/eTP9smlw82s+fMbIKZLTKzuzLxYXLz8rR48SItXbJEmzZt0rPPjFGfvmdk4q332Kq1G5Xb7GBJUpfmtfX51+uDZSkP7RWbWNuMXOQqS1ffcKumzF6k12cs0O8ffkJdu5+gu//4aOhY0Yp1P7q7rhwxTK3bttOlI64IHadcoM0Qm4r4F9ivk/Rzd+8rSWZWXVJvd99gZq0lPS0pdxev/1hSD3ffbGbfl/QbSUVzPnaSdLSkjZI+MbMH3X3ZTrZTKrKysnTf/Q/p9D6nqLCwUBcOvkhH5uSU5Vvu0m/PyVFus9o6uHq2Xrmqm0a+8aluHbdAvzi1jSpXMm3avEW3jfs4WL7Y2ivVkAsHauqbU/T116uV06qprrvxZg0aHP43sbG2GbnItb+K8VwR636c/t47GjtmtI7Iaa9e3RP/tP/yptv0/ZNPC5wszv0oxdtmsbZXmbC4hjmGZunGgZZHZrbO3WuYWU9tW4zUkvSQEoVEoaQ27l7dzJpJetHd26e+xswOl/SApNaSXFK2u7czs8GSurn7xcntjpd0h7u/nSbLMEnDJOnwJk06L/zXZ2X4yffO925/LXSEtN67sVfoCGltKCgMHSGtatmVQ0cAgvhsdbge2F2pV6tq6AhpxXquWLO+YPcrBVI1O87BIxsLtoSOkFaM7XVit656f/bM6P7Xn31oS69zZhx/q2dnvvrrebPcfVe/vC818R05pe9KSV9K6qhEj0iV3ax/m6Q33L29pNMlVUt5LnXO2kLtpGfJ3R9x91x3zz207qF7HRwAAACoyCriMK21kmqmPK4labm7bzGzCyXt7tdDtSQVzVc4uPTjAQAAYH/GMK2tKmLPyFxJm83sAzO7UtKfJF1oZu9JaiPp2928/i5JvzWzqdp94QIAAABgL1WYnhF3r5H8WSBp+wsPOqTc/2VyvaWS2ifvT5Y0OXn/XSWKliK/Si5/XNLjKe/Xt9TCAwAAAPuhClOMAAAAAOUBw7S2qojDtAAAAACUAxQjAAAAAIKgGAEAAAAQBNeMAAAAABliMq4ZSUHPCAAAAIAgKEYAAAAABMEwLQAAACCTGKVVjJ4RAAAAAEFQjAAAAAAIgmFaAAAAQKYYf4E9FT0jAAAAAIKgGAEAAAAQBMO09lOTr+0ZOkJaba78v9AR0pp7V5/QEdJas74gdISd+nLNhtAR0mrToGboCCgFTetWDx2hXNlQUBg6Qlq1qmeHjlDuVMuuHDpCuVEp4pFQDNPaip4RAAAAAEFQjAAAAAAIgmFaAAAAQAYxTGsrekYAAAAABEExAgAAACAIihEAAAAAe8zMDjezN8xsgZl9ZGY/Sy6vY2aTzGxR8mft3W2LYgQAAADIJIv8tnubJV3t7kdI+p6k4WZ2pKTrJL3m7q0lvZZ8vEsUIwAAAAD2mLt/4e6zk/fXSlogqZGkMyU9kVztCUk/3N22mE0LAAAAQKq6ZjYz5fEj7v5IuhXNrJmkoyVNk1TP3b+QEgWLmR22uzeiGAEAAAAyqBxM7bva3XN3t5KZ1ZD0D0lXuPs3e/O5GKYFAAAAoETMLFuJQmS0uz+XXPylmTVIPt9A0le72w7FCAAAAIA9ZokukL9KWuDu96Y89X+SLkzev1DSC7vbFsVIOTDxlQnqkNNWOe1a6e677gwdp9iIS4aqddMGOja3Y9AcDQ6upjE/PU6v3XCiXr2+py46obkkqVb1bI0efqym/OokjR5+rGodkB00Zyzttb385ct0dt/eOj7vKPXo2lGjRj4YOlKx045rr3N6f0/nndpNA/qcEDpOsVi/k+QqGXKVTKznsFjbK9ZcUrzZYs1V2sws+tse6CZpkKSTzGxO8vYDSXdK6m1miyT1Tj7eJYqRyBUWFuqKy4frhXHj9f7c+Xp2zNNaMH9+6FiSpAGDLtDY518KHUOFW1y3//Mj9brjDZ35+7d0QY/mal2/hob3bq2pC1fphNte19SFq3RZ71ZBc8bSXtvLysrSLbffpbdmfKiXX31bj40aqU8+juMYk6S/PPOS/j5hqp5+aUroKJLi/U6Si1xlLcZzWKztFWsuKd5sseZCeu7+trubu3dw907J28vu/rW793L31smf/97dtihGIjdj+nS1bNlKzVu0UJUqVdSv//l6cdxue7wyolv3Hqpdp07oGPrqm42at3yNJOnbjYVavHKt6tc6QL2Pqq+x05ZJksZOW6aTOzQIGTOa9tpevfoN1KHT0ZKkGjVrqnXbdlq5YkXgVPGK9TtJLnKVtRjPYbG2V6y5pHizxZoLZY9iJHIrVuSrcePDix83atRY+fn5ARPFrXGdA5TTuJbe/+w/qluzqr76ZqOkRMFSt2aVwOni9/lnSzVv7gc6JrdL6CgJZrr0f36o83/QQ2NHPxY6jaR4v5PkKhlyVQyxtlesuaR4s8Waq6yEHoZVCsO0Sk2Fm9rXzAolfajEZ1sg6UJ3X7+L9de5ew0zayjpAXc/dyfrNZP0oru3L/3UO+fu6bJkMkK5Ub1KZf15SJ5+/dxHWrdhc+g45c6369Zp6KD+uvW396jmQQeFjiNJeuIfE3VY/Qb6evUqXTrwTDVv1Uadu3YLminW7yS5SoZcFUOs7RVrLinebLHmQtmriD0j3yXHrbWXtEnSpXvyIndfsbNCJKRGjRpr+fJlxY/z85erYcOGARPFKauS6c9D8/TPmcs14YMvJEmr127UYQdVlSQddlBVrV67KWTEqBUUFGjIoP46+7wB6nPGWaHjFDusfmJo3SF1D9VJp/TVvDmzAieK9ztJrpIhV8UQa3vFmkuKN1usuVD2KmIxkuotSa0kycyuMrN5ydsV269oZs3MbF7yfo6ZTU/ODDDXzFonV6tsZqPM7CMzm2hmB5T1B8jNy9PixYu0dMkSbdq0Sc8+M0Z9+p5R1m9b7tw9sJMWr1yrv7zxafGySR+u1LldE12+53Y9XJM+XBkqXtTcXVeOGKbWbdvp0hE7fDWCWb/+W327bm3x/Xffel2t2h4ROFW830lykWt/FGt7xZpLijdbrLnKSuhhWAzTygAzy5J0mqQJZtZZ0o8ldZVkkqaZ2RR3f38nL79U0v3uPtrMqkiqLKmepNaSBrj7xWb2d0nnSHoqzXsPkzRMkg5v0mSfPkdWVpbuu/8hnd7nFBUWFurCwRfpyJycfdpmaRly4UBNfXOKvv56tXJaNdV1N96sQYMvyniOvBZ1dE6Xw7Ug/xuNvzYx/etd4xboT5MWaeRFuer/vSZa8Z/vdOmjMzOeLVUs7bW96e+9o7FjRuuInPbq1T3xx1Z/edNt+v7JpwXN9e9VX+nKYQMlSZs3b9YPfthP3Xr2DppJivc7SS5ylbUYz2GxtlesuaR4s8WaC2XP0o3RK89SrhmREj0jV0v6iaRD3P2m5Dq3SVrl7g+kXDPSTMlrQszsR5JukPQ3Sc+5+6Lk85PcvXVyG9dKynb323eVp3PnXJ86Lex/gtPZUFAYOkJaHX4R17SRRebe1Sd0hLQ2FmwJHWGnvlyzIXSEtNo0qBk6ApBxsZ7zq2VXDh0BFVi3rrmaNWtmdBeeVK3X2hsO+EPoGLu09P6+s9w9NxPvVRF7Rr5z906pC6yE/U3u/r9mNk1SH0mvmNlQSZ9K2piyWqGkMh+mBQAAgAomuhIpnIp+zUiRNyX90Myqm9mBks5SotckLTNrIelTd39AiT9r3yEzMQEAAID9R0XsGdmBu882s8clTU8u+ssurheRpP5pi/OCAAAgAElEQVSS/sfMCiStlHSrpDjmOgUAAAAqiApXjLh7jZ0sv1fSvTtb392XSmqfvP9bSb/dbtV/Fz2fXOee0kkMAAAA7J8qXDECAAAAxIw/6LjV/nLNCAAAAIDIUIwAAAAACIJhWgAAAECmGMO0UtEzAgAAACAIihEAAAAAQTBMCwAAAMgQk8Qora3oGQEAAAAQBMUIAAAAgCAYpgUAAABkjDGbVgp6RgAAAAAEQTECAAAAIAiGaZWxLS5tKCgMHWMH1bIrh46Q1sL7zggdIa3aeSNCR0jrPzMeCh1hp2pVzw4doVyJ8TwhxXuuoL1KJtZcse5HKd42Q8XAKK2t6BkBAAAAEATFCAAAAIAgKEYAAAAABME1IwAAAEAGMbXvVvSMAAAAAAiCYgQAAABAEAzTAgAAADLFmNo3FT0jAAAAAIKgGAEAAAAQBMO0AAAAgAwxSZUqMU6rCD0jAAAAAIKgGAEAAAAQBMVIOTDikqFq3bSBjs3tGDrKNia+MkEdctoqp10r3X3XnaHjFIslV+N6B2vCI5fr/X/cqFljb9DwAT2Ln/vJ+Sfog3/+SrPG3qA7fnZmsIxFYmmz7ZFrz8V6npBor5KKsb2keHPFui9jbS8p3myx5ioLZnHfMolipBwYMOgCjX3+pdAxtlFYWKgrLh+uF8aN1/tz5+vZMU9rwfz5oWNFlWtz4RZdd+9zOvqc23XCBffokv491K5FffXIba2+PY9S3nm/Vedz79Af/vZakHxFYmozcu29GM8TEu1VUrG2V6y5pDj3ZcztFWu2WHOh7FGMlAPduvdQ7Tp1QsfYxozp09WyZSs1b9FCVapUUb/+5+vFcS+EjhVVrpWrv9Gcj5dLktat36iPl6xUw0MP1rB+x+uexyZpU8FmSdKq/6wLkq9ITG1Grr0X43lCor1KKtb2ijWXFOe+jLm9Ys0Way6UPYoR7JUVK/LVuPHhxY8bNWqs/Pz8gIkSYs3VpEEddWrbWDPmLVWrpoep29Et9ebffq6Jf/mZOh/ZJGi2WNuMXBUD7VUysbZXrLliFXN7xZot1lxlxcyivmVSlMWImYX9VTF2y913WJbpgzedGHMdeEAVPX3PUF1zzz+09tsNyqpcSbUPqq4eF9yj6+97Xk/ddVHQfDG2mUSuioL2KplY2yvWXLGKub1izRZrLpS9KIsRxK9Ro8ZavnxZ8eP8/OVq2LBhwEQJseXKyqqkp++5WM+Mn6kXXv8gkenL/+r51xL3Z370mbZscdWtXSNYxtjarAi5Kgbaq2Riba9Yc8Uq5vaKNVusuVD2oi1GzKyGmb1mZrPN7EMzOzO5vJmZfWxmT5jZXDMba2bVk8/dZGYzzGyemT1iyZLazCab2e/MbLqZLTSz45PLK5vZ3cnXzDWzS5LLG5jZm2Y2J7mtovVPNrN3k5meNbNw/4MMLDcvT4sXL9LSJUu0adMmPfvMGPXpe0boWNHlevjmgfpkyUo98NTrxcvGTZ6rnl3aSJJaNTlMVbKztDrgdSOxtRm5Khbaq2Riba9Yc8Uq5vaKNVusuVD2oi1GJG2QdJa7HyPpREm/LyouJLWV9Ii7d5D0jaTLkssfcvc8d28v6QBJfVO2l+XuXSRdIenm5LIhkta4e56kPEkXm1lzST+S9Iq7d5LUUdIcM6sr6UZJ309mminpqnTBzWyYmc00s5mrV6/a54YYcuFAndyzuxYv/EQ5rZrqyccf3edt7qusrCzdd/9DOr3PKep01BE6p995OjInJ3SsqHId16mFBvbtqhPy2ui9MdfpvTHX6ZTuR+qJ599V80aHaOaz1+tvd/5YQ296Mki+IjG1Gbn2XoznCYn2KqlY2yvWXFKc+zLm9oo1W6y5ykQEU/fGNLWvpRujF1rympHaku6T1EPSFiUKkOaSqkl6092bJNc9SdLl7v5DMztH0i8kVZdUR9KD7n6nmU2WdIO7TzWzepKmunsrMxsrqYOk9cm3riXpEiUKoUclPSXpeXefY2Z9JT0uaXly3SqS3nX3Ibv6LEcfk+tvTJ22z21S2qplVw4doVypnTcidIS0/jPjodARUEo2FBSGjpBWrOcK2qtiiHU/SuzLiqBb11zNmjUzugtPDmjYxlsN+WPoGLs07/aTZ7l7bibeKysTb7KXBko6VFJndy8ws6VKFCKStH0F5WZWTdKfJOW6+zIzuyVlfUnamPxZqK2f2yT91N1f2f7NzayHpD6SnjSzuyX9R9Ikdx+wz58MAAAAQNTDtGpJ+ipZiJwoqWnKc03M7Njk/QGS3tbWwmN18lqOc/fgPV6R9BMzy5YkM2tjZgeaWdPke4+S9FdJx0h6T1I3M2uVXLe6mbXZx88IAACA/YiJqX1TRdczYmZZSvRijJY0zsxmSpoj6eOU1RZIutDM/ixpkaSR7r7ezEZJ+lDSUkkz9uDt/iKpmaTZyetRVkn6oaSekq4xswJJ6yRd4O6rzGywpKfNrGry9TdKWrj3nxYAAADYf0VXjEjKkfQvd18t6djtnzSzZpK2uPul2z/n7jcqUSBsv7xnyv3VShQgcvctkq5P3lI9kbxtv53XlbjQHQAAAMA+iqoYMbNLJV2uxIxXAAAAQAWT+aFQMYuqGHH3hyU9vJt1lkpqn5FAAAAAAMpMzBewAwAAAKjAouoZAQAAACo6RmltRc8IAAAAgCAoRgAAAAAEwTAtAAAAIIOYTWsrekYAAAAABEExAgAAACAIihEAAAAAQXDNCAAAAJApxtS+qegZAQAAABAEPSNlrJJJ1bIrh46BffSfGQ+FjpBWh+snhI6wU3N/c2roCGltKCgMHSEtzhMVQ6zH15drNoaOkFbTutVDRyh3Yj3G1qwvCB1hBwWFHjoC9gDFCAAAAJAhJqb2TcUwLQAAAABBUIwAAAAACIJhWgAAAEAGMUprK3pGAAAAAARBMQIAAAAgCIZpAQAAABnEbFpb0TMCAAAAIAiKEQAAAABBMEwLAAAAyCBGaW1FzwgAAACAIChGAAAAAARBMVIOTHxlgjrktFVOu1a6+647Q8cpRq6SiSnXb/q117s3nagXr+pWvKxdg5p6Zvj3NO7Kbnp48DE6sGrlgAkTYmqzIiMuGarWTRvo2NyOoaPsIMb2kuLMFfN+jDlbYWGhzup9rC4ZdE7oKMViPL6keHPFenxt2LBBZ/burlNP6KLe3Y7RvXfeFjpS2bHEbFox3zKJYiRyhYWFuuLy4Xph3Hi9P3e+nh3ztBbMnx86FrnKea7nZuZryF9nbbPsjnPb657xn+j0+6Zq0rwvNfSE5oHSJcTWZkUGDLpAY59/KXSMHcTaXrHminU/SnFn+9uoP6pF67ahYxSL9fiKNZcU7/FVtWpV/e8/J2jClOl6efI0TXl9ombPnBY6FjKAYiRyM6ZPV8uWrdS8RQtVqVJF/fqfrxfHvRA6FrnKea6ZS/6jNesLtlnW/NADNePT/0iSpi76WqccVT9EtGKxtVmRbt17qHadOqFj7CDW9oo1V6z7UYo328oV+Zry2gT1+9Hg0FGKxXp8xZpLivf4MjMdWKOGJGlzQYE2F2zmb3HsJyhGIrdiRb4aNz68+HGjRo2Vn58fMFECuUom1lypFq5cq15HHiZJOq1DfdU/uFrQPOWhzWISa3vFmgsl95ubfqGf33iHrFI8/3WI9fiKNVfsCgsLdVrPrup8RBN173mSju7cJXQkZEA8Z5RSYmaFZjbHzD4ysw/M7Coz2+3nNLPr92Cdx83s3NJJumfcPV2OTEZIi1wlE2uuVNc/O08Dj2ui5y4/VgdWrayCzVuC5ikPbRaTWNsr1lwomTcmjdchdQ9V+45Hh46yjViPr1hzxa5y5coaP3ma3p27WB/MnqlPFnwUOlKZMCWm9o35lkkV8e+MfOfunSTJzA6T9L+Sakm6eTevu17Sb8o4W4k1atRYy5cvK36cn79cDRs2DJgogVwlE2uuVJ+u+lYX/WWmJKlZ3erq2e7QoHnKQ5vFJNb2ijUXSmb29Hf1+sSXNOW1V7Rp4watW7tW1wy/SHf/8dGguWI9vmLNVV7UqnWwvteth6a8NlFtj8gJHQdlrML1jKRy968kDZM0whIGm9lDRc+b2Ytm1tPM7pR0QLJHZXTyuQvMbG6yd+XJlM32MLN3zOzTTPSS5OblafHiRVq6ZIk2bdqkZ58Zoz59zyjrtyXXfpIrVZ0Dq0hK/Ebksl4t9fR7y3bzirJVHtosJrG2V6y5UDJX33CrpsxepNdnLNDvH35CXbufELwQkeI9vmLNFbOvV6/SmjX/lSRt+O47TX3zdbWMaLIElJ2K2DOyDXf/NDlM67BdrHOdmY1I6VHJkXSDpG7uvtrMUq/0aiCpu6R2kv5P0tjtt2dmw5QognR4kyb7lD8rK0v33f+QTu9zigoLC3Xh4It0ZE743xKQq3znuvdHHdWlRW3VPrCK3ry+px6YtEjVq2Rp4HGJ43XSvC/1j5lhxzfH1mZFhlw4UFPfnKKvv16tnFZNdd2NN2vQ4ItCx4q2vWLNFet+lOLOFptYj69Yc0nxHl9ffblSV4+4WFsKC7Vlyxb1OfMc9TrlB6FjlZHMT58bM0s3rrE8M7N17l5ju2X/ldRW0mmSct19RHL5i5LucffJqa8zs59Kqu/uN2y3ncclTXL3ot6Tte5ec1d5OnfO9anTZpbSpwO21eH6CaEj7NTc35waOkJaGwoKQ0dIq1p2+L/rUp7Euh9j9eWajaEjpNW0bvXQEcqdWI/97WdojMHpvbpp7pxZ0f2vv0bjdn7UTx8JHWOX3rvuhFnunpuJ96rQw7QkycxaSCqU9JWkzdr2M+9suiCTtLMqbeN26wEAAADYCxW6GDGzQyU9LOkhT3QBLZXUycwqmdnhklLnjCsws+zk/dcknWdmhyS3E9+E3AAAACiXQs+WxWxaZesAM5sjKVuJnpAnJd2bfG6qpCWSPpQ0T9LslNc9Immumc1294FmdoekKWZWKOl9SYMzlB8AAADYL1S4YsTddzrwOtk7MnAnz10r6dqUx09IemK7dQZv93iba1MAAAAA7LkKV4wAAAAAMWM2ra0q9DUjAAAAAOJFMQIAAAAgCIZpAQAAAJkSYMaqmNEzAgAAACAIihEAAAAAQVCMAAAAAAiCa0YAAACADDExtW8qekYAAAAABEExAgAAACAIhmkBAAAAGcQwra3oGQEAAAAQBMUIAAAAgCAYprWfWrO+IHSEtKpmx1kfx9peb93YK3SEnbpt0sLQEdK6vFvz0BHSqpZdOXQElIJY92O1SM+tGwoKQ0dAKalVPTt0hB1UrhTvUChGaW0V59kJAAAAQIVHMQIAAAAgCIZpAQAAABnEbFpb0TMCAAAAIAiKEQAAAABBMEwLAAAAyBRjNq1U9IwAAAAACIJiBAAAAEAQFCMAAAAAguCaEQAAACBDTMbUvinoGQEAAAAQBMVIOTDxlQnqkNNWOe1a6e677gwdR5KUv3yZzu7bW8fnHaUeXTtq1MgHQ0cqNuKSoWrdtIGOze0YOso2NmzYoDN7d9epJ3RR727H6N47bwsdSVJc+/LF+36pPww4Vo/8pG/xstf++js9POxUjbrsdI29bbg2rPsmWD4prvbaXoznCinOXLGeJ6Q42yvW85cU774kV8nEmgtlj2IkcoWFhbri8uF6Ydx4vT93vp4d87QWzJ8fOpaysrJ0y+136a0ZH+rlV9/WY6NG6pOPw+eSpAGDLtDY518KHWMHVatW1f/+c4ImTJmulydP05TXJ2r2zGmhY0W1Lzt8/2ydf9tftlnW/OhuGjbyRV38p3Gq06iZ3vn7n4NkKxJTe6WK9VwRa65YzxOxtles5y8p3n1JrpKJNVdZMYv7lkkUI5GbMX26WrZspeYtWqhKlSrq1/98vTjuhdCxVK9+A3XodLQkqUbNmmrdtp1WrlgROFVCt+49VLtOndAxdmBmOrBGDUnS5oICbS7YHMWY0Zj2ZZOj8lStZq1tlrU4prsqVU5c3taoXSetXb0yRLRiMbVXqljPFbHmivU8EWt7xXr+kuLdl+QqmVhzoexRjERuxYp8NW58ePHjRo0aKz8/P2CiHX3+2VLNm/uBjsntEjpK9AoLC3Vaz67qfEQTde95ko7uHFebxb4vP5j4D7XM7RE6RrGY2ivWc0WsuWIVc3vFfv4CUD5VuGLEzOqb2Rgz+5eZzTezl82sTSls9xYz+3lpZCwJd0+XJdMxdurbdes0dFB/3frbe1TzoINCx4le5cqVNX7yNL07d7E+mD1Tnyz4KHSkYrHvy6ljRqpS5crKOfGM0FEkxddesZ4rYs0Vq5jbK+bzF1DeVDKL+pbRtsjou5UxS5yx/ylpsru3dPcjJV0vqV7YZHuvUaPGWr58WfHj/PzlatiwYcBEWxUUFGjIoP46+7wB6nPGWaHjlCu1ah2s73XroSmvTQwdRVL8+3Luq//U4umTdeY190TxH7MY2yvWc0WsuWJVHtortvMXgPKtQhUjkk6UVODuDxctcPc5knqb2ZzkLd/MHpMkM/sfM5ueXP5nM6ucXH6qmc02sw/M7LWU7R9pZpPN7FMzuzwTHyg3L0+LFy/S0iVLtGnTJj37zBj16Rv+N8PuritHDFPrtu106YgrQscpF75evUpr1vxXkrThu+809c3X1bJ128Cp4t+X/5r5pt59dpTOvXmksqsdEDpOtO0V67ki1lyxirW9Yj1/ASj/Klox0l7SrO0XuvtN7t5J0gmSvpb0kJkdIam/pG7J5wolDTSzQyWNknSOu3eU1C9lU+0knSKpi6SbzSw7XQgzG2ZmM81s5qrVq/bpA2VlZem++x/S6X1OUaejjtA5/c7TkTk5+7TN0jD9vXc0dsxovf3mG+rVPVe9uufq1YnjQ8eSJA25cKBO7tldixd+opxWTfXk44+GjiRJ+urLlRrww1N1ao88ndG7u7qf0Eu9TvlB6FhR7cvnf3eVnrjqfP17+RI9OKiH5rzyrCaOvE2bvvtWT9/wY/1lxJka/+BNQbIViam9UsV6rog1V6zniVjbK9bzlxTvviRXycSaq6yEni0rptm0LN341PIq2VvR3N2vTPOcSRon6R/u/piZjVBiCNdXyVUOkPS0EsXM+e4+cLvX36JEr8sdyccLJPV29+W7ytS5c65PnTZz3z5YGVizviB0hLSqZsdZH8faXtWyK4eOsFMPTF0SOkJal3drHjpCWrWqp/3dBnZiQ0Fh6Ahpxfqd/HLNhtAR0uK4R1k6sVtXvT97Zvixvds5qMkR/r1rHwsdY5cmjTh2lrvnZuK9sjLxJhn0kaRzd/LcLZKWu3vR3jdJT7j7L1NXMrMzJO2sQtuYcr9QFa/9AAAAgIyJ89fQe+91SVXN7OKiBWaWZ2Y3S+otKfU6j9cknWtmhyXXq2NmTSW9K+kEM2tetDxj6QEAAFChJYZCWdS3TKpQv9l3dzezsyT9wcyuk7RB0lJJ1SU1lDQ92cD/5+43mdmNkiaaWSVJBZKGu/t7ZjZM0nPJ5V8pUcgAAAAAKEUVqhiRJHdfIem8PVz3GUnPpFk+XtL47Zbdst3j9nufEgAAAEBFG6YFAAAAoJyocD0jAAAAQMwqRTfHVzj0jAAAAAAIgmIEAAAAQBAM0wIAAAAyKNPT58aMnhEAAAAAQVCMAAAAAAiCYVoAAABABjFKayt6RgAAAAAEQTECAAAAIAiGaQEAAAAZYpJMjNMqQs8IAAAAgCAoRgAAAAAEwTCt/VSt6tmhI5Qr1WpVDh0hrQ0FhaEj7NQ1PVuGjpBWh1+8FDpCWgvvOyN0hLRiPcY2FmwJHaFc4ZxfcVTL5t+jiqASo7SK0TMCAAAAIAiKEQAAAABBMEwLAAAAyBQzGX/1sBg9IwAAAACCoBgBAAAAEATFCAAAAIAguGYEAAAAyCAuGdmKnhEAAAAAQVCMAAAAAAiCYgQAAADIEJNUySzq224/g9mjZvaVmc1LWXaLmeWb2Zzk7Qd70h4UIwAAAABK4nFJp6ZZfp+7d0reXt6TDVGMlAMTX5mgDjltldOule6+687QcYqRq2RizTXikqFq3bSBjs3tGDrKNmLK1eDgahrz0+P02g0n6tXre+qiE5pLkmpVz9bo4cdqyq9O0ujhx6rWAdlBc8Z4jMW0H1PlL1+ms/v21vF5R6lH144aNfLB0JGKxdpm5CqZWHNJnCuw79z9TUn/Lo1tUYxErrCwUFdcPlwvjBuv9+fO17NjntaC+fNDxyJXBcklSQMGXaCxz78UOsYOYspVuMV1+z8/Uq873tCZv39LF/Rortb1a2h479aaunCVTrjtdU1duEqX9W4VLmOkx1hM+zFVVlaWbrn9Lr0140O9/OrbemzUSH3ycfj2kuJtM3KVTKy5OFfEwSzu2z4YYWZzk8O4au/JCyhGIjdj+nS1bNlKzVu0UJUqVdSv//l6cdwLoWORq4LkkqRu3Xuodp06oWPsIKZcX32zUfOWr5EkfbuxUItXrlX9Wgeo91H1NXbaMknS2GnLdHKHBsEyxnqMxbQfU9Wr30AdOh0tSapRs6Zat22nlStWBE6VEGubkatkYs3FuQJ7qK6ZzUy5DduD14yU1FJSJ0lfSPr9nrwRxUjkVqzIV+PGhxc/btSosfLz8wMmSiBXycSaCyXXuM4BymlcS+9/9h/VrVlVX32zUVKiYKlbs0qwXBxje+/zz5Zq3twPdExul9BRgDLHuQJ7aLW756bcHtndC9z9S3cvdPctkkZJ2qOT6n5XjJjZul08987evrasuHu6HJmOsQNylUysuVAy1atU1p+H5OnXz32kdRs2h46zDY6xvfPtunUaOqi/bv3tPap50EGh4wBljnNFHMws6ttefqbU4QFnSZq3s3VT8RfYJZlZ5WQld1zoLNtr1Kixli9fVvw4P3+5GjZsGDBRArlKJtZc2HNZlUx/Hpqnf85crgkffCFJWr12ow47KNE7cthBVbV67aZg+TjGSq6goEBDBvXX2ecNUJ8zzgodB8gIzhUoDWb2tKSeSgznWi7pZkk9zayTJJe0VNIle7Kt/a5npIiZ9TSzN8zsfyV9mFy2LvmzgZm9mZwjeZ6ZHZ/yujvM7AMze8/M6pV1zty8PC1evEhLlyzRpk2b9OwzY9Sn7xll/bbk2k9yYc/dPbCTFq9cq7+88WnxskkfrtS5XRPDHc7tergmfbgyVDyOsRJyd105Yphat22nS0dcEToOkDGcK1Aa3H2Auzdw92x3b+zuf3X3Qe5+lLt3cPcz3P2LPdnWfluMJHWRdIO7H7nd8h9JesXdO0nqKGlOcvmBkt5z946S3pR0cbqNmtmwogt+Vq1etU8Bs7KydN/9D+n0Pqeo01FH6Jx+5+nInJx92mZpIFfFyCVJQy4cqJN7dtfihZ8op1VTPfn4o6EjSYorV16LOjqny+E6rs2hGn/tCRp/7Qk68cjD9KdJi3R820M15Vcn6fi2h+qPkxYFyxjrMRbTfkw1/b13NHbMaL395hvq1T1Xvbrn6tWJ40PHkhRvm5GrZGLNxbkivNAzZZXxbFolb490YwcrMjNb5+41zKynpJvd/cQ0z/WQ9KikpyQ97+5zks9vlFTN3d3M+kvq7e5Dd/V+nTvn+tRpM8vs82D/tqGgMHSEcqfDL+KcOnLhfXH+ZjLWY2xjwZbQEdKqmr2//44PZa1aduXQEdKK8VxxYreuen/2zOguiKnT/EjvdfNToWPs0tgfd57l7rmZeK/9/az5bbqFyT/k0kNSvqQnzeyC5FMFvrV6KxTX3AAAAAB7bX8vRtIys6aSvnL3UZL+KumYwJEAAACACoff7KfXU9I1ZlYgaZ2kC3a9OgAAALBnKjGdcrH9rhhx9xrJn5MlTd7Jc09IemJnr03eHytpbBlGBQAAACo0hmkBAAAACGK/6xkBAAAAQmKQ1lb0jAAAAAAIgmIEAAAAQBAM0wIAAAAyyJhNqxg9IwAAAACCoBgBAAAAEATDtAAAAIAMMUmVGKVVjJ4RAAAAAEFQjAAAAAAIgmFaAAAAQKaYMZtWCnpGAAAAAARBMQIAAAAgCIZplbEtLm0oKAwdYwfVsiuHjpBWjG2FvRPrMTb3rj6hI6R126SFoSOkddzhtUJHSKtXu3qhI6S1Zn1B6AhpVc3md48lFes5LNZ/J2NsL2asKh8oRgAAAIAM4pKRrfhVCQAAAIAgKEYAAAAABMEwLQAAACCDmNp3K3pGAAAAAARBMQLg/9m78/ioq3v/468PYRPBKFURiIAssoSdAFoQUcSNRcUqoEW5omgr9afV22s3tVqvFq3Wai+2tC5tVRSqWFAWN0BQWZUoIIKCSnDDhYoQCMPn90eGEGAICZCcM+H95DGPzHznO995zzknE86c8z0jIiIiEoSmaYmIiIiIVBBDyw4Xp5EREREREREJQp0REREREREJQtO0REREREQqkFbT2kEjIyIiIiIiEoQ6I2lg1JWX06JxfU7M6RA6yk6mT5tK++yWZLdqzl2j7wwdp0is5aVcZRdjG4upvCbf+3P+MPRE/vKj/kXbXvrb73hw5JmM/fEAJtx2Nfkb/hMwYaGJf/8zPz63Fz8+72RG/+wqtmzODx0JiLN95a35mEH9+3JS13b06t6BsWPuDx2pSExtv7hYc8XYviDe8oJ4y0zKlzojaWDosEuYMPG50DF2kkgkuPaaq3l20hTezF3K+HFPsGzp0tCxgDjLC5SrrGJtYzGVV/vTBjHktr/utO24Tj0YOWYyV/zfJOo2bMJrT/05ULpC6z77hEmP/5V7x03j/56ZybZEgllTJgbNBPG2r6pVq3LLb0fz6vy3ef7F2Tw8dgzL3w2fC+Jq+8XFmCvW9gVxlhfEXWblwSK/VCR1RtJAj569OKJu3dAxdjJ/3jyaNWvOcU2bUr16dS4YPITJk54NHQuIs7xAucoq1jYWU3k1ateVmnUyd9rWtHNPqmQUng7YsFVHvl33acYhXdQAACAASURBVIhoO0lsTbBlcz6JrVvZnL+JukcfEzpStO2r3jH1ad+xEwC169ShRctWfLp2beBUhWJq+8XFmCvW9gVxlhfEXWZSvvbYGTGzw0q6VGRIic/atXlkZR1bdLthwyzy8vICJpLKRm1s/y2e/i+a5fQKmuHIevU5b/iP+K++XRh2antq1T6Mzt/vHTQTpEf7+ujD1byTu5jOOd1CR5EySof2FRuV2cGrpJGRJcA7yZ9Ldrn9TvlH25mZHWNm48zsfTNbambPm9lIM5u8h/3/amZtKjrnwcLdd9umlSHkQFIb2z9zxo2hSkYG2acMDJpjw/pvmPvKVP42dR5/f2kxmzdt5JVJE4Jmgvjb13cbNnD5sMHcesfd1DlMn/+lm9jbV4xUZgevPS7t6+7H7um+imaFrfEZ4FF3H5Lc1hEYsKfHuPvlFRTvoNSwYRZr1nxcdDsvbw0NGjQImEgqG7WxfZf74jOsnDeDi/73keB/zN96Yxb1GjYis+6RAJx42tksWzyfUwb8IGiumNtXQUEBI4YNZtCFQ+k38LzQcWQfxNy+YnUwlZkZVFFHq0ipzhkxsyFm9ovk9Swz61K+sXZzClDg7g9u3+DubwGvArXNbIKZvWtmjyU7LpjZDDPLSV7fYGa3m9liM3vDzOoltx9lZv8ys/nJS4/k9pPN7K3k5U0zq5Pc/t/J/XLN7DcVXAZRyenalZUrV7B61Sq2bNnC+CfH0a9/2E9gpXJRG9s37y+Yxevjx/KDm8dQreYhoeNwVP0slucuJH/TRtydxXNf5djjWoSOFW37cneuGzWSFi1bcdWoa0PHkX0Ua/uKmcrs4LXXzoiZPUBhZ2BYctNG4ME9P6JctAUW7uG+TsC1QBugKdAjxT6HAm+4ewdgFnBFcvt9wL3u3hU4H9i+LM0NwNXu3hE4CdhkZqcDLYBuQEegi5mlnIydnD62wMwWrFv3RdleaQojLr2Y03v3ZOV7y8lu3ph/PPLQfh9zf1WtWpV773uAAf3OoGO71px/wYW0yc4OHQuIs7xAucoq1jYWU3lN/N1PefSnQ/hqzSruH9aLt6aNZ/qY29iy6Tue+OV/8ddR5zDl/puC5QNo2b4zPfr259oLT+fqQb3xbc6ZFwzb+wPLWazta94brzFh3GPMnvUKfXrm0KdnDi9OnxI6FhBX2y8uxlyxti+Is7wg7jKT8mWp5ujttIPZInfvbGZvunun5LbFyf/YVwgzuwY4zt2v22V7b+CX7t43eXsMMMfd/2lmM4Ab3H2BmW0Garq7m9lgoK+7X25mnwPFlyk5CmgFXA2cBzwGPO3ua8zsbuAHwDfJfWsDd7j730rK3qlzjr8yZ+5+vf7yULNaRugIKeUXJEJHkANEbaxs7prxfugIKX3/2My97xRAn1b1QkdIaf3GgtARUqpRTYtnlpXew8omxvLq0T2HhQsXRDcf6qhm2X7uHU+GjlGivw5ut9DdcyriufZ4zkgxBWZWBXAAM/sesK1cU+1uCYUdgVQ2F7ueIPVrKvAdva7i+1QBTnT3Tbvsf6eZPQecDbxhZqdRuOzyHe4edtF+EREREZFKojQflfwJ+BdwVPI8idnA78o11e5eBmqY2fbpVZhZV+Dk/TzudGBUsWN2TP5s5u5vu/vvgAUUjpZMAy4zs9rJfRqa2dH7+fwiIiIiIgetvY6MuPvfzWwhcFpy0wXuXqFL+yanV50H/MHMbgTygdXA/n6N7zXAn8wsl8KymAVcBVxrZqdQOIqyFJji7pvNrDXwevIc+Q3AD4HP9zODiIiIiBxEQq90GJPSTNMCyAAKKJyqFWTiqbuvBS5McdfYYvuMKna9d7HrtYtdnwBMSF5fBwxO8Vw/2UOG+yg86V1ERERERPZTaVbT+iXwBNAAyAIeN7Ofl3cwERERERGp3EozMvJDoIu7bwQws9spXGb3jvIMJiIiIiJSGWmW1g6lmXL1ITt3WqoCH5RPHBEREREROVjscWTEzO6l8ByRjcASM5uWvH06hStqiYiIiIiI7LOSpmltXzFrCfBcse1vlF8cEREREZHKyzCqaJ5WkT12Rvb2zeIiIiIiIiL7Y68nsJtZM+B2oA1Qc/t2dz++HHOJiIiIiEglV5rVtB4BfgvcDZwF/BewrRwziYiIiIhUTqbVtIorzWpatdx9GoC7v+/uvwJOKd9YIiIiIiJS2ZVmZGSzFX5n/ftmdhWQBxxdvrFERERERKSyK01n5DqgNnANheeOZAKXlWcoERERERGp/PbaGXH3ucmr3wLDyjeOiIiIiEjlZjpppEhJX3r4DIVfcpiSuw8ql0QiIiIiInJQKGlk5IEKSyEVLr8gETqCSBDvfbIhdISUBretHzpCSl9vLAgdQQ6AmtUyQkdIKea/RbFmi7UuP1ufHzrCbgoSe/xMXSJS0pcevlSRQUREREREDgalWc72YKGyEBERERGRINQZERERERGRIEqztC8AZlbD3TeXZxgRERERkcrM0Gpaxe11ZMTMupnZ28CK5O0OZnZ/uScTEREREZFKrTTTtP4I9Ae+BHD3xcAp5RlKREREREQqv9JM06ri7h/uMpwU53p3IiIiIiKRq6JZWkVK0xn52My6AW5mGcBPgPfKN5aIiIiIiFR2pZmm9SPgp0Aj4DPghOQ2ERERERGRfbbXkRF3/xwYUgFZREREREQqPU3T2mGvnREzGwv4rtvdfWS5JBIRERERkYNCaaZpvQi8lLzMAY4G9H0jFWjUlZfTonF9TszpEDrKTpSrbJSr7KZPm0r77JZkt2rOXaPvDB2nyLf/+YYbr76EC/t2ZfDp3Xh70bzQkQA46/ttOb/vCVx4Zg+G9js5dBwAPvpgBSPOPbnocnaXxox/9MHQsYA421femo8Z1L8vJ3VtR6/uHRg7Jp6V9GMsL4j3PSzWXBBnXebn53NO356ceXI3+vbozD133hY6klSQvXZG3P3JYpdHgUFAm/KPJtsNHXYJEyY+FzrGbpSrbJSrbBKJBNdeczXPTprCm7lLGT/uCZYtXRo6FgD33HojJ/Y6jademM8/J8+mSfPjQ0cq8tcnn+OpqXN44rmZoaMA0KhpC/42cSZ/mziTv/zrZWocUouTTusXOla07atq1arc8tvRvDr/bZ5/cTYPjx3D8nfD54q1vCDe97BYc8ValzVq1ODxZ6YydeY8np8xl5kvT2fRgrmhY0kFKM3IyK6OAxof6CCyZz169uKIunVDx9iNcpWNcpXN/HnzaNasOcc1bUr16tW5YPAQJk96NnQsNnz7H96c/xoDLxwGQLXq1alz2OGBU6WHRa/PouGxTTim4bGho0TbvuodU5/2HTsBULtOHVq0bMWna9cGThVveUG872Gx5oq1Ls2MQ2vXBmBrQQFbC7ZW2m8pNyt8vTFfKlJpvoH9azP7Knn5BngB+EX5RxORg9natXlkZe34T2vDhlnk5eUFTFRo7cerOaLukdz2sx8zbMBJ3P7zn7Bp43ehYxUy46ofnsuQs3sx4bGHQ6fZzcvPP82p/QaFjgHE276K++jD1byTu5jOOd1CR0mL8pLSibkuE4kEZ/XuTpfWjejZ+1Q6dQnf9qX8ldgZscKuUQfgqOTlCHdv6u5P7eVxM8zsjF22XWtmD5nZhL08tomZXVS6+CJSWbnvtm5GFJ+SJbYmWL5kMYMuHsE/Jr1KzUNq8eiD94aOBcCj/5rOk8+/yp/+/i+e/PtYFs6dEzpSkYItW5jz8lR6n3lO6ChAvO1ru+82bODyYYO59Y67qXPYYaHjRF9eUnox12VGRgZTZszl9dyVLF60gOXLloSOJBWgxM6IF7bYZ9w9kbzs3oJTe4LdlwMeAjzs7j/Yy2ObAOqMiBzkGjbMYs2aj4tu5+WtoUGDBgETFTq6fgOOPqYBbTvmAHDqWeewfElu4FSFjj6mPgDfO/IoTj2jP++8tTBwoh3mvvoix7dpT90jjw4dBYi3fQEUFBQwYthgBl04lH4DzwsdB4i7vKRs0qEuMzMP54QevZj50vTQUcpNFYv7UqFlUYp95plZ5zIedwLQ38xqQOFoB9AAWGNm7yS3ZZjZXWY238xyzezK5GPvBE4ys7fM7DozG25mT5vZVDNbYWajtz+JmY0xswVmtsTMflNs+2oz+18zez15f2czm2Zm75vZVcX2++9iz/+b5LZDzew5M1tsZu+Y2eDk9i5mNtPMFiaPVb+MZSIiZZDTtSsrV65g9apVbNmyhfFPjqNf/4GhY/G9o+pxdP0sPvxgBQALXpvJcc1bBk4FGzd+x3cbvi26/vqrL9O8ZevAqXZ46bmn6RPJFC2It325O9eNGkmLlq24atS1oeMUibW8pOxircsv133B+vXfAJC/aRNzZr1Msxbh31ul/O2xM2Jm27+DpCeFHZLlZrbIzN40s0UlHdTdvwTmAWcmNw0BnmTn7ysZAax3965AV+AKMzsOuBF41d07uvv2uQ8dgcFAO2CwmW2f7PhLd88B2gMnm1n7Ysf/2N1PBF4FHgF+QOG3x9+afH2nAy2AbsnjdzGzXsnMa929g7u3BaaaWTXgfuAH7t4FeAi4vYSyG5nsBC1Yt+6LkoqqVEZcejGn9+7JyveWk928Mf945KH9PuaBoFxlo1xlU7VqVe697wEG9DuDju1ac/4FF9ImOzt0LABuuPl33HTdFVx89vd5b+nbDP/x9aEj8dUXnzP8/DO44Izvc/GAUzjp1DPo0btv6FgA5G/ayMI5Mzjp9AGhoxSJtX3Ne+M1Jox7jNmzXqFPzxz69MzhxelTQseKtrwg3vewWHPFWpeff/YpQ889kzN7dWVg3570PLkPfc44O3QsqQC2p5lXZrbI3TubWbNU97v7+yUe2OyHQD93H2pmbwGXAV8Bk929bfLckfbAxuRDMoErgS3ADe7eP3mc4UAPd78ieXsKcLu7z06Ocoyk8Msb6wM/cfdxZrY6+Zg8M7sMOLHY4z9KPu+vKOygfJN8/trAHRR2XqYBTyWzvmpmbYHXgA+S+2YAn7j76SWVAUCnzjn+yhwtTScHn5rVMkJHSCn3o/WhI6RUs9q+LG5Y/r7eWBA6Qkrdm8W3ShHA+kjLK7NWtdARUsovSISOkHZifW/9bH1+6Ai7GdCnB7lvLYzjhJhijmnR1i/5w79CxyjRXf1bLUx+4F/uSvoGdoO9dzpKMBG4JznF6xB3X5ScrlX8+D9x92k7PalZ7xTHKv4liwmganIU5Qagq7t/bWaPADVTPGbbLo/fRuHrNuAOd//zrk9mZl2As4E7zGw68AywJDnSIiIiIiIiB0BJnZGjzOyne7rT3e8p6cDuvsHMZlA4pemJFLtMA35kZi+7e4GZHQ/kAd8CdfaaHA4DvgPWm1k94CxgRikeV/z5bzOzx5JZGwIFFJbJV+7+TzPbAAyn8DyWo8zsRHd/PTlt63h31zIPIiIiIiL7qKTOSAaFU5f2Z3jrCeBpdl9ZC+CvFK6ctSi5hPAXwLlALrDVzBZTeK7H16kO7O6LzexNYAmF06fKtIalu083s9bA68kl7TYAPwSaA3eZ2TYKOyc/cvctZvYD4I9mlklhuf0h+dwiIiIiIqViQJVIllOOQUmdkU/c/db9Obi7P0Oxzoy7rwbaJq9vo/DLE1N9gWKfXW4/UuwY/YtdH76H521S7Pojuzy++H33Afft8vD3KRw12fWYbwG9Uj2fiIiIiIiUXUlnTKrLJiIiIiIi5aakkZFdRydERERERGQ/xbl+Yhh7LAt3/6oig4iIiIiIyMFFHTMREREREQlCnREREREREQmipHNGRERERETkANPKvjtoZERERERERIJQZ0RERERERILQNC0RERERkQpiZvoG9mI0MiIiIiIiIkGoMyIiIiIiIkFompaIiIiISAXSLK0dNDIiIiIiIiJBaGSknFUxqFktI3SM3eR+tD50hJTaN8oMHUEOkPyCROgIKTU+slboCCll1qoWOkJaufKp3NARUrrvvOzQEdJKjH8fY/fZ+vzQEUQOKHVGREREREQqUBVN0yqiaVoiIiIiIhKEOiMiIiIiIhKEpmmJiIiIiFQQA33pYTEaGRERERERkSDUGRERERERkSDUGRERERERkSB0zoiIiIiISAXSKSM7aGRERERERESCUGdERERERESC0DQtEREREZGKYvoG9uI0MpIGpk+bSvvslmS3as5do+8MHafIt//5hhuvvoQL+3Zl8OndeHvRvNCRgHjLK9ZcEGe2UVdeTovG9Tkxp0PoKDvJW/Mxg/r35aSu7ejVvQNjx9wfOlKRGOsR4sk1onsW9w9qw+1nH7/bfWe1OpJHL2pP7RoZAZLtLNa2H0s97kq5yiY/P59z+vbkzJO70bdHZ+6587bQkYB4c0n5U2ckcolEgmuvuZpnJ03hzdyljB/3BMuWLg0dC4B7br2RE3udxlMvzOefk2fTpPnuf+ArWqzlFWsuiDfb0GGXMGHic6Fj7KZq1arc8tvRvDr/bZ5/cTYPjx3D8nfDl1es9RhTrtkffM3dr6zabXvdWtXIrl+Hdd9tCZBqdzG2/ZjqUbn2T40aNXj8malMnTmP52fMZebL01m0YG7oWNHmkvKnzkjk5s+bR7NmzTmuaVOqV6/OBYOHMHnSs6FjseHb//Dm/NcYeOEwAKpVr06dww4PnCre8oo1F8SbrUfPXhxRt27oGLupd0x92nfsBEDtOnVo0bIVn65dGzhVvPUYU67lX3zHd1u27rb9os71efLNT3APECqFGNt+TPWoXPvHzDi0dm0AthYUsLVgKxbB0k6x5iovFvm/iqTOSOTWrs0jK+vYotsNG2aRl5cXMFGhtR+v5oi6R3Lbz37MsAEncfvPf8Kmjd+FjhVveUWaC+LOFruPPlzNO7mL6ZzTLXSUaOsx1lzbdWp4GF9v2srH3+SHjhK1WOtRufZNIpHgrN7d6dK6ET17n0qnLuHfwyDeXFK+0qozYma/NLMlZpZrZm+ZWfcDfPzX9nL/hgP5fKXhKT6qi+GTgsTWBMuXLGbQxSP4x6RXqXlILR598N7QsaItr1hzQdzZYvbdhg1cPmwwt95xN3UOOyx0nGjrMdZcANUzjAHZR/N07qeho0Qv1npUrn2TkZHBlBlzeT13JYsXLWD5siWhIwHx5pLylTadETM7EegPdHb39sBpwMcH8jnc/fsH8ngHQsOGWaxZs+Nl5uWtoUGDBgETFTq6fgOOPqYBbTvmAHDqWeewfElu4FTxllesuSDubLEqKChgxLDBDLpwKP0Gnhc6DhBvPcaaC+Do2jU4qnZ1bjvreO4e2Iq6tapx65ktyKyphSZ3FWs9Ktf+ycw8nBN69GLmS9NDR9lJrLkOFKNwNa2YLxUpbTojQH1gnbtvBnD3de6+1sxWm9nvzGxe8tIcwMwGmNlcM3vTzF40s3rJ7beY2UNmNsPMPjCza7Y/wfaRDzOrb2azkqMv75jZScX2ud3MFpvZG9uPWZ5yunZl5coVrF61ii1btjD+yXH06z+wvJ92r753VD2Orp/Fhx+sAGDBazM5rnnLwKniLa9Yc0Hc2WLk7lw3aiQtWrbiqlHXho5TJNZ6jDUXwJr1+fzk6aXc8O93ueHf7/LVxgJumrqC9fm7n1dysIu1HpWr7L5c9wXr138DQP6mTcyZ9TLNWoT/+x1rLil/6fTxz3TgJjN7D3gReNLdZybv+4+7dzOzS4A/UDiCMhs4wd3dzC4HfgZcn9y/FXAKUAdYbmZj3L2g2HNdBExz99vNLAOoldx+KPCGu//SzEYDVwC/LbdXTOHKPffe9wAD+p1BIpHg0uGX0SY7uzyfstRuuPl33HTdFWwt2EKDY5vw69H/FzpStOUVay6IN9uISy9mzqyZfPnlOrKbN+bGX93MsOGXhY7FvDdeY8K4x2id3ZY+PQtHBn9+022cdvpZQXPFWo8x5frR9xvRqt6h1K5RlXvPbcUzuZ8x64Ovg2QpSYxtP6Z6VK798/lnn3L9qCvYlkiwbds2+p1zPn3OODt0rGhzSfmzVPMaY5XsGJxEYUfiSuBG4BbgVHf/wMyqAZ+6+/fMrB3wewpHVKoDq9z9TDO7BShw99uTx1wG9HX3NWa2wd1rm1kv4CHgn8BEd38rue9moGaygzM4+bjLU+QcCYwEOLZRoy7vvf9huZXJvsr9aH3oCCm1b5QZOoIcIPkFidARUtpcsC10hJQya1ULHSGtXPlU+Gmhqdx3Xhz/4dxVzWrhvz9FDozP1muxhdIa0KcHuW8tjOdknaSslu38mgcnho5Rov85tflCd8+piOdKp2lauHvC3We4+83AKOD87XcV3y35837gAXdvR2HHpWaxfTYXu55glxEid58F9ALygH8kR1ygsBPje3pcscf/xd1z3D3nqCOPKtNrFBERERE5WKRNZ8TMWppZi2KbOgLbhxwGF/v5evJ6JoWdCYBLy/hcjYHP3X0s8Deg8z6FFhERERGRPUqnc0ZqA/eb2eHAVmAlhVOh+gM1zGwuhZ2rocn9bwHGm1ke8AZwXBmeqzfw32ZWAGwALil5dxERERERKau06Yy4+0Jgt6V3k+t2/8ndf7PL/s8Cu33dqbvfssvttsWu107+fBR4NMVjaxe7PgGYUMaXISIiIiIHuZi+dya0tJmmJSIiIiIilUvajIzsibs3CZ1BRERERETKLu07IyIiIiIi6WL7N7BLIU3TEhERERGRINQZERERERGRIDRNS0RERESkohhoMa0dNDIiIiIiIiJBqDMiIiIiIiJBaJqWiIiIiEgFqqJ5WkU0MiIiIiIiIkGoMyIiIiIiIkFompaIiIiISAXRlx7uTCMjIiIiIiIShEZGDlL1MmuEjpBWPlufHzpC2qmXWTN0hJTWbywIHSGlTKqFjpBWbj3j+NARUvrllOWhI6R0+1ktQ0dIqWa1jNAR0k5mrTjfK2Ksy2oZGn5IB+qMiIiIiIhUIC2mtYOmaYmIiIiISBDqjIiIiIiISBDqjIiIiIiISBA6Z0REREREpMIYVdBJI9tpZERERERERIJQZ0RERERERILQNC0RERERkQpiaGnf4jQyIiIiIiIiQagzIiIiIiIiQWialoiIiIhIRTGoomlaRTQykgamT5tK++yWZLdqzl2j7wwdB4D8/HzO6duTM0/uRt8enbnnzttCRyoSY3lBvGUWay6Isy5VXmUXY66Y6vHlP/2Kh//rJMZde07RtrlP/JFx153Hk9cP4t+3XsF3X30eLN92o668nBaN63NiTofQUXYSY/uCeHPFWo8Qb5lJ+VJnJHKJRIJrr7maZydN4c3cpYwf9wTLli4NHYsaNWrw+DNTmTpzHs/PmMvMl6ezaMHc0LGiLS+It8xizRVrXaq8KkeumOqxVe9z6f/rP++0rdM5lzHk3mcY/PunadLlZOaPHxMkW3FDh13ChInPhY6xk1jbV6y5IM56hLjLTMqXOiORmz9vHs2aNee4pk2pXr06FwwewuRJz4aOhZlxaO3aAGwtKGBrwVYsgqUhYi0viLfMYs0Va12qvCpHrpjqsUF2DjVqZ+60rXqt2kXXCzZvwiL4grQePXtxRN26oWPsJNb2FWsuiLMeIe4yKw9VzKK+VGhZVOizSZmtXZtHVtaxRbcbNswiLy8vYKIdEokEZ/XuTpfWjejZ+1Q6dekWOlLU5QVxlhnEmSvmulR5lV6suSDOeizujcfu49GRfVgxazLdhowKHSdKsbavWHPFTGWWXszsITP73MzeKbatrpm9YGYrkj+PKM2xKlVnxMzOMzM3s1b78NgNe9h+q5mdtv/p9o2777Ythk9hATIyMpgyYy6v565k8aIFLF+2JHSkqMsL4iwziDNXzHWp8iq9WHNBnPVY3AkX/z8u/ctLtOjVn7enPB46TpRibV+x5oqZyiztPAKcucu2G4GX3L0F8FLy9l5Vqs4IMBSYDQzZ9Q4zy9iXA7r7Te7+4v4G21cNG2axZs3HRbfz8tbQoEGDUHFSysw8nBN69GLmS9NDR0mL8oK4yqy4mHKlQ12qvPYu1lzFxVSPqRzfsx8fvPFC6BhRirV9xZorZgdTmW3/0sOYL3vj7rOAr3bZfA7waPL6o8C5pSmPStMZMbPaQA9gBMnOiJn1NrNXzOxx4O3ktolmttDMlpjZyF2O8XszW2RmL5nZUcltj5jZD5LXu5rZa2a22MzmmVmd8n5dOV27snLlClavWsWWLVsY/+Q4+vUfWN5Pu1dfrvuC9eu/ASB/0ybmzHqZZi1aBk4Vb3lBvGUWa65Y61LlVTlyxVqP232z9sOi66sWvMLhDY8LmCZesbavWHPFTGVWKdRz908Akj+PLs2DKtP3jJwLTHX398zsKzPrnNzeDWjr7quSty9z96/M7BBgvpn9y92/BA4FFrn79WZ2E3AzUDRJ18yqA08Cg919vpkdBmwq7xdVtWpV7r3vAQb0O4NEIsGlwy+jTXZ2eT/tXn3+2adcP+oKtiUSbNu2jX7nnE+fM84OHSva8oJ4yyzWXLHWpcqrcuSKqR6n33MDa5fMJ//bb3j0ilPpOvhqPlo0i2/WrgarQp2j6nPylTcHyVbciEsvZs6smXz55Tqymzfmxl/dzLDhlwXNFGv7ijUXxFmPEHeZHaSONLMFxW7/xd3/Uh5PZKnm6KUjM3sO+IO7v2Bm1wDHAs8BN7v7KcX2uwU4L3mzCXCGu79hZgmghrtvNbOmwNPu3tHMHgEmA8uBB929RymyjARGAhzbqFGX997/cC+PqHifrc8PHSGlepk1Q0dIKdbyipnqsmxiLa9YxVqPo2d+EDpCSrefFc+oT3E1q+3TDOqDWn5BInSElGKsyx7dc1i4cEF0J540ad3ef/nIpNAxSjTyhCYL3T2npH3MrAkw2d3bJm8vB3q7+ydmVh+Y4e57ffOpFCMjZvY94FSgrZk5kAE48DzwXbH9egOnASe6+0YzmwHs6X8Au/bSk2XX+QAAIABJREFULMW21A8s7Dn+BaBLl5zK0dsTERERkQOiopfPrSD/Bi4F7kz+LNXazJXlnJEfAH9398bu3sTdjwVWAT132S8T+DrZEWkFnFDsvirJ4wBcROGJ8MW9CzQws64AZlbHzCpFZ05EREREpLTM7AngdaClma0xsxEUdkL6mtkKoG/y9l5Vlv9MD2X3F/wv4EfA+8W2TQWuMrNcCqddvVHsvu+AbDNbCKwHBhc/mLtvMbPBwP3J8002UTjKknJJYBERERGRysjdh+7hrj5lPVal6Iy4e+8U2/4I/HGXbZuBs/ZwjO1fd/vrXbYPL3Z9PjuPpoiIiIiIlEnlnKW1byrLNC0REREREUkz6oyIiIiIiEgQlWKaloiIiIhIOjA0GlCcykJERERERIJQZ0RERERERILQNC0RERERkYpiYFpOq4hGRkREREREJAh1RkREREREJAhN0xIRERERqUCapLWDRkZERERERCQIdUZERERERCQIdUZERERERCQInTMiIiIiIlJBDKiipX2LqDNykMqsVS10hLRSL7Nm6Agp5X60PnSEPYq1zGLNJWVTs1pG6Agp/X5gm9ARUrro0YWhI6Q05oL2oSPskf5Ols1n6/NDR9hNQcJDR5BS0DQtEREREREJQiMjIiIiIiIVSJO0dtDIiIiIiIiIBKHOiIiIiIiIBKFpWiIiIiIiFUiLae2gkREREREREQlCnREREREREQlC07RERERERCqMYZqnVUQjIyIiIiIiEoQ6IyIiIiIiEoSmaYmIiIiIVBBDowHFqSzSwPRpU2mf3ZLsVs25a/SdoeMUGXXl5bRoXJ8TczqEjrKTWMsr1lwA3/7nG268+hIu7NuVwad34+1F80JHAuItM+Uqmxhz5a35mEH9+3JS13b06t6BsWPuDx2pSCzldfVJjXn4ovb8YVCbom2DO9Vn7JB2/P7c1vz+3NZ0zjosWD5QPe6LWP925+fnc07fnpx5cjf69ujMPXfeFjqSVBB1RiKXSCS49pqreXbSFN7MXcr4cU+wbOnS0LEAGDrsEiZMfC50jJ3EWl6x5trunltv5MRep/HUC/P55+TZNGl+fOhI0ZaZclWOXFWrVuWW347m1flv8/yLs3l47BiWvxs+V0zl9cqKL7lt2ordtk9+53Oun7iM6ycuY9Ga/wRItoPqsexi/NsNUKNGDR5/ZipTZ87j+RlzmfnydBYtmBs6llQAdUYiN3/ePJo1a85xTZtSvXp1Lhg8hMmTng0dC4AePXtxRN26oWPsJNbyijUXwIZv/8Ob819j4IXDAKhWvTp1Djs8cKp4y0y5KkeuesfUp33HTgDUrlOHFi1b8enatYFTxVVeSz/dwLebE0Geu7RUj2UX499uADPj0Nq1AdhaUMDWgq1aceogoc5I5NauzSMr69ii2w0bZpGXlxcwUdxiLa9YcwGs/Xg1R9Q9ktt+9mOGDTiJ23/+EzZt/C50rGjLTLnKJtZcxX304WreyV1M55xuoaOkRXmd1eYo7jmvNVef1JhDq2eEjlNE9Zj+EokEZ/XuTpfWjejZ+1Q6dQlfl+XFzKK+VKS06YyYWT0ze9zMPjCzhWb2upmdFzpXeXP33bbpk4I9i7W8Ys0FkNiaYPmSxQy6eAT/mPQqNQ+pxaMP3hs6VrRlplxlE2uu7b7bsIHLhw3m1jvups5hYc9/gPjLa+qyL/jx+He4/pllfL2xgOHds0JHAlSPlUVGRgZTZszl9dyVLF60gOXLloSOJBUgLTojVvgbPBGY5e5N3b0LMAQo1bugmcXz0U0ZNWyYxZo1HxfdzstbQ4MGDQImilus5RVrLoCj6zfg6GMa0LZjDgCnnnUOy5fkBk4Vb5kpV9nEmgugoKCAEcMGM+jCofQbGMdnWzGXF8D6/K1sc3DgheXraHHUoaEjqR4roczMwzmhRy9mvjQ9dBSpAGnRGQFOBba4+4PbN7j7h+5+v5llmNldZjbfzHLN7EoAM+ttZq+Y2ePA22bWxMzeNbO/mtk7ZvaYmZ1mZnPMbIWZdUs+rpuZvWZmbyZ/tkxuH25mT5vZ1OT+oyvihed07crKlStYvWoVW7ZsYfyT4+jXf2BFPHVairW8Ys0F8L2j6nF0/Sw+/KDwRNUFr83kuOYtA6eKt8yUq3LkcneuGzWSFi1bcdWoa0PHKRJreW13xCE7vhGge+PD+ejrTQHTqB4rky/XfcH69d8AkL9pE3NmvUyzFuH/FpUXi/xSkdLle0aygUV7uG8EsN7du5pZDWCOmW3vSncD2rr7KjNrAjQHLgBGAvOBi4CewEDgF8C5wLtAL3ffamanAf8LnJ88XkegE7AZWG5m97v7jo8+ykHVqlW5974HGNDvDBKJBJcOv4w22dnl+ZSlNuLSi5kzayZffrmO7OaNufFXNzNs+GVBM8VaXrHm2u6Gm3/HTdddwdaCLTQ4tgm/Hv1/oSNFW2bKVTlyzXvjNSaMe4zW2W3p07NwVPDnN93GaaefFTRXTOV1Xe/jaFu/DnVqVmXskHaMW7SW7Pp1OK5uLRzni2+38OCcD4Nk2071WHYx/u0G+PyzT7l+1BVsSyTYtm0b/c45nz5nnB06llQASzWvMTZmdg1wnLtfl7z9Jwo7EVuAD4H2wMbk7pnAlcn7bnb3U5KPaQK84O4tkrf/Dkxz98fMrCnwtLt3NLNjgT8CLSgcia7m7q3MbDjQw92vSD5+CnC7u89OkXckhR0ejm3UqMt774d9s04lvyDOFVJqVkvbGXVB5H60PnSEPWrfKDN0BKnE1m8sCB0hpcxa1UJHSOmiRxeGjpDSmAvah46wR7HWZax/v2P8nRzQpwe5by2M7mSdZm06+J2PTwkdo0QXdmq40N1zKuK50mVkZAk7Ridw96vN7EhgAfAR8BN3n1b8AWbWG9h1SaDNxa5vK3Z7GzvK4jbgFXc/L9mBmbGHxyfYQ/m5+1+AvwB06ZITf29PRERERCqGaUGD4tLlnJGXgZpm9qNi22olf04DfmRm1QDM7Hgz258z6jKB7evvDd+P44iIiIiISAnSojPihXPJzgVONrNVZjYPeBT4H+CvwFJgkZm9A/yZ/RvxGQ3cYWZzAM0ZEhEREREpJ+kyTQt3/4TC5XxT+UXyUtwMik2xcvfVQNtit4enus/dXweOL3acXye3PwI8Uuwx/cv0AkRERETkoGekyWhABVFZiIiIiIhIEOqMiIiIiIhIEGkzTUtEREREpDLQalo7aGRERERERESCUGdERERERESC0DQtEREREZEKpElaO2hkREREREREglBnREREREREglBnREREREREgtA5IyIiIiIiFUgr++6gkREREREREQlCnREREREREQlC07RERERERCqIAVW0uG8RdUbK2TaH/IJE6Bi72VywLXSElGpWywgdIaUY6xDg+Pq1Q0dIO5+tzw8dIaVY235mrWqhI8gB8NBFHUNHSOmyx98KHWGPHr+0S+gIaSXG94qMKvoPfzrQNC0REREREQlCIyMiIiIiIhVIq2ntoJEREREREREJQp0REREREREJQtO0REREREQqjGFaTauIRkZERERERCQIdUZERERERCQITdMSEREREalAWk1rB42MiIiIiIhIEOqMiIiIiIhIEOqMiIiIiIhIEDpnRERERESkghhQRUv7FtHISBoYdeXltGhcnxNzOoSOUiRvzccM6t+Xk7q2o1f3Dowdc3/oSEWmT5tK++yWZLdqzl2j7wwdp0iM9Qjx5oI46zI/P59z+vbkzJO70bdHZ+6587bQkQD9TpaVyqvsYnqvuPqkxjx8UXv+MKhN0bbBneozdkg7fn9ua35/bms6Zx0WMKHqsaxizSXlT52RNDB02CVMmPhc6Bg7qVq1Krf8djSvzn+b51+czcNjx7D83aWhY5FIJLj2mqt5dtIU3sxdyvhxT7BsafhcEGc9Qry5Yq3LGjVq8PgzU5k6cx7Pz5jLzJens2jB3NCx9DtZRiqvsovpveKVFV9y27QVu22f/M7nXD9xGddPXMaiNf8JkKyQ6rHsYs0l5U+dkTTQo2cvjqhbN3SMndQ7pj7tO3YCoHadOrRo2YpP164NnArmz5tHs2bNOa5pU6pXr84Fg4cwedKzoWMBcdYjxJsr1ro0Mw6tXRuArQUFbC3YikWwRqN+J8tG5VV2Mb1XLP10A99uToSOsUeqx7KLNVe5sMKlfWO+VCR1RmS/ffThat7JXUznnG6ho7B2bR5ZWccW3W7YMIu8vLyAiWRfxVyXiUSCs3p3p0vrRvTsfSqduoRv+8Xpd7JsVF6Vx1ltjuKe81pz9UmNObR6RrAcqkeR0kvLzoiZJczsrWKXG/fjWBuSPxuY2YQS9mtiZu/s6/NUVt9t2MDlwwZz6x13U+ewsPNzAdx9t20xfGotZRdzXWZkZDBlxlxez13J4kULWL5sSehIRfQ7WTYqr8pj6rIv+PH4d7j+mWV8vbGA4d2zgmVRPYqUXrquprXJ3TseyAO6+1rgBwfymJVdQUEBI4YNZtCFQ+k38LzQcYDCT5/WrPm46HZe3hoaNGgQMJHsq3Soy8zMwzmhRy9mvjSdlq2zQ8fR72QZqbwql/X5W4uuv7B8Hb88vXmwLKpH2Rv1TXdIy5GRPTGz1Wb2GzNbZGZvm1mr5PajzOyF5PY/m9mHZnbkLo8tGvkws2wzm5ccdck1sxbJ3TLMbKyZLTGz6WZ2SAW/xGi4O9eNGkmLlq24atS1oeMUyenalZUrV7B61Sq2bNnC+CfH0a//wNCxZB/EWpdfrvuC9eu/ASB/0ybmzHqZZi1aBk6l38myUnlVPkccsuPz1e6ND+ejrzcFy6J6FCm9dO2MHLLLNK3Bxe5b5+6dgTHADcltNwMvJ7c/AzTay/GvAu5Ljr7kAGuS21sAf3L3bOAb4PxUDzazkWa2wMwWrFv3xT69wOJGXHoxp/fuycr3lpPdvDH/eOSh/T7m/pr3xmtMGPcYs2e9Qp+eOfTpmcOL06eEjkXVqlW5974HGNDvDDq2a835F1xIm+zwn1hDnPUI8eaKtS4//+xThp57Jmf26srAvj3peXIf+pxxduhY+p0sI5VX2cX0XnFd7+O4c0ArGmTWZOyQdvQ5/nsM65bFvee14Z7zWtOufh0efuPjvR+onKgeyy7WXFL+LNW8xtiZ2QZ3r51i+2qgh7vnmVl34HZ3P83M3gLOc/dVyf2+Ao5393Xbj2VmTYDJ7t7WzC4Cfgn8HXja3Vck73/B3Vskj/E/QDV3/21JWTt1zvFX5oRf9nNXmwu2hY6QUmataqEjpJRfEO+qLbGqWS3cyaMl+Wx9fugIKcVaXrH+Tq7fWBA6Qkqxlles72GXPf5W6Ah79PilXUJHSCnWuozRKT268+aiBdFNiDq+bUf/0/gXQ8co0eltjlro7jkV8VzpOjJSks3Jnwl2nBNTpobo7o8DA4FNwDQzO3WXY+96fBERERERKaPK2BlJZTZwIYCZnQ4cUdLOZtYU+MDd/wj8G2hf7glFRERERA4y6frJ/iHJqVfbTXX3kpb3/Q3wRPLckpnAJ8C3Jew/GPihmRUAnwK3AuHXfBQRERGRtGZAlegmj4WTlp0Rd085udrdmxS7vgDonby5HjjD3bea2YnAKe6+Oblf7eTP1UDb5PU7gDt2OfxX2+9P7nP3AXgpIiIiIiIHrbTsjOyDRsBTZlYF2AJcETiPiIiIiMhB76DojLj7CqBT6BwiIiIiIrLDQdEZERERERGJhZVtoddK7WBZTUtERERERCKjzoiIiIiIiAShaVoiIiIiIhXINEuriEZGREREREQkCHVGREREREQkCE3TEhERERGpQFpNaweNjIiIiIiISBDqjIiIiIiISBCapiUiIiIiUkEMqKJZWkU0MiIiIiIiIkFoZKScVTGoWS0jdIzdbC7YFjpCSvkFidAR0kqMbWu7WOuyXmbN0BHkAMisVS10hJRibfexeuiijqEj7NHRw/4eOkJKn//jktAR0oZGH9KDOiMiIiIiIhXGtJpWMZqmJSIiIiIiQagzIiIiIiIiQagzIiIiIiIiQeicERERERGRimJgOmWkiEZGREREREQkCHVGREREREQkCE3TEhERERGpQJqltYNGRkREREREJAh1RkREREREJAhN0xIRERERqSAGVNFyWkU0MpIGpk+bSvvslmS3as5do+8MHQeAvDUfM6h/X07q2o5e3Tswdsz9oSMVGXXl5bRoXJ8TczqEjrKTWHNBnG1M5VV2ylU2seaKte0rV8ka1q3F5F+dzvy7BzL3roH86MxWANx2URcW3H0Or/1uAI/9tDeZtaoFzQnxtv1Yc0n5UmckcolEgmuvuZpnJ03hzdyljB/3BMuWLg0di6pVq3LLb0fz6vy3ef7F2Tw8dgzL3w2fC2DosEuYMPG50DF2E2uuWNuYyku5DsZcEG/bV66Sbd3m/PKfC+h6w7/p8+vnueL0VrRsmMkrb6+l+8/+zff/ZxIrP/kPPz2nXdCcsbb9WHNJ+VNnJHLz582jWbPmHNe0KdWrV+eCwUOYPOnZ0LGod0x92nfsBEDtOnVo0bIVn65dGzhVoR49e3FE3bqhY+wm1lyxtjGVl3IdjLkg3ravXCX77JtNLF79FQAb8reyPG89DerW4uW3PyGxzQGYv+ILGtatFTJmtG0/1lzlxSK/VCR1RiK3dm0eWVnHFt1u2DCLvLy8gIl299GHq3kndzGdc7qFjiL7IB3aWExiLS/lKptYc0nl0OjIQ2nfpC4LVq7bafuw3s15YXHYdhZr2481l5S/tOmMmFnCzN4ys3fMbJKZHX4Aj51jZn88UMc7kNx9t20W0UlP323YwOXDBnPrHXdT57DDQseRfRB7G4tNrOWlXGUTay5Jf4fWqMo/ruvNjX+fz7ebCoq233BuO7Zuc56cvSpgunjbfqy5pPylTWcE2OTuHd29LfAVcPWBOrC7L3D3aw7U8Q6khg2zWLPm46LbeXlraNCgQcBEOxQUFDBi2GAGXTiUfgPPCx1H9lHMbSxGsZaXcpVNrLkkvVXNMP55XW+emvMBk+Z/VLT9ol5NObNTFpc/8GrAdIVibfux5io3oedhRTRPK506I8W9DjQEMLPeZjZ5+x1m9oCZDU9ev9PMlppZrpndndx2QXJ0ZbGZzdr1GGbWzcxeM7M3kz9bJrcPN7OnzWyqma0ws9EV8UJzunZl5coVrF61ii1btjD+yXH06z+wIp66RO7OdaNG0qJlK64adW3oOLIfYm1jsYq1vJSrcuSS9Pankd9n+dpv+NPzy4q2ndahAdcOaMvgu19m05ZEwHSFYm37seaS8pd2nREzywD6AP/ey351gfOAbHdvD/w2eddNwBnu3gFI1crfBXq5e6fkvv9b7L6OwGCgHTDYzI5N8XjMbKSZLTCzBV+s+6L0Ly6FqlWrcu99DzCg3xl0bNea8y+4kDbZ2ft1zANh3huvMWHcY8ye9Qp9eubQp2cOL06fEjoWACMuvZjTe/dk5XvLyW7emH888lDoSEC8uWJtYyov5ToYc0G8bV+5SnZCy6MZ2qsZvbLrM/uO/sy+oz+nd2zI3cO7UfuQajz7i77MvqM/947oHiTfdrG2/VhzSfmzVHP0YmRmCeBtoAmwEDjd3RNm1hu4wd37J/d7AFgA/DO53wLgOWCyu28xsweBZsBTwNPu/mXxYyQ7GH8EWgAOVHP3VsnRlh7ufkXyeaYAt7v77JJyd+mS43PmLjiAJXFgrN9YsPedAqhRLe36x0HVrJYROsIe5ReE/wQwlZjLTNJfrO1eyq7RZY+FjpDS5/+4JHSEtNGjew4LFy6I7sST1u06+SMTZ4SOUaITmh++0N1zKuK50ul/fpvcvSPQGKjOjnNGtrLz66gJ4O5bgW7Av4BzganJ7VcBvwKOBd4ys+/t8jy3Aa8kz00ZsP14SZuLXU+gb7AXEREREdln6dQZAcDd1wPXADeYWTXgQ6CNmdUws0wKp3BhZrWBTHd/HriWwilWmFkzd5/r7jcB6yjslBSXCWxfS254eb8eEREREZGDVdp1RgDc/U1gMTDE3T+mcMpVLvAY8GZytzrAZDPLBWYC1yW332Vmb5vZO8Cs5HGKGw3cYWZzAM3nEBEREREpJ2kzzcjda+9ye0Cx6z8DfpbiYbt9C5+7D0qx34zkBXd/HTi+2H2/Tm5/BHik2HH6lzK6iIiIiEgRfYXKDmk5MiIiIiIiIulPnREREREREQkibaZpiYiIiIhUBpqltYNGRkREREREJAh1RkREREREJAhN0xIRERERqUiap1VEIyMiIiIiIhKEOiMiIiIiIhKEpmmJiIiIiFQQA0zztIpoZERERERERIJQZ0RERERERILQNC0RERERkYpiYJqlVUQjIyIiIiIiEoRGRkTkoJJfkAgdIaWa1TJCR0grsdbj+o0FoSOkVC+zZugIaeejhy4OHSGlgX9+I3SElMYO6Rg6wm4KEh46gpSCRkZERERERCQIjYyIiIiIiFQgnTKyg0ZGREREREQkCHVGREREREQkCE3TEhERERGpSJqnVUQjIyIiIiIiEoQ6IyIiIiIiEoSmaYmIiIiIVBjDNE+riEZGREREREQkCHVGREREREQkCE3TEhERERGpQKZZWkU0MpIGpk+bSvvslmS3as5do+8MHQeAvDUfM6h/X07q2o5e3Tswdsz9oSMVGXXl5bRoXJ8TczqEjrKTWHNBnG0s1vKKNRfEWY8QZ65Y6zE/P59z+vbkzJO70bdHZ+6587bQkYrEWI8Qb66Y2thPT23KU//Vhb8Mab/T9nPa1eNvF3XgL0Pbc/mJjQKlKxRz25fypc5I5BKJBNdeczXPTprCm7lLGT/uCZYtXRo6FlWrVuWW347m1flv8/yLs3l47BiWvxs+F8DQYZcwYeJzoWPsJtZcsbaxWMsr1lyx1mOsuWKtxxo1avD4M1OZOnMez8+Yy8yXp7NowdzQsaKtx1hzQVxt7IVlX/CLSct22tah4WGceFxdrhqXy8gncpnw1tpA6QrF2val/KkzErn58+bRrFlzjmvalOrVq3PB4CFMnvRs6FjUO6Y+7Tt2AqB2nTq0aNmKT9eGfSPbrkfPXhxRt27oGLuJNVesbSzW8oo1V6z1GGuuWOvRzDi0dm0AthYUsLVgKxbBfI5Y6zHWXBBXG3v7k2/5dnNip23929bjyUV5FGxzAL7ZtDVEtCKxtv3yYGlwqUjqjERu7do8srKOLbrdsGEWeXl5ARPt7qMPV/NO7mI653QLHUX2QTq0Mdm7WOsx1lwxSyQSnNW7O11aN6Jn71Pp1CX8e2us9RhrrnSQdXhN2jY4jD/+oC13n9uG448+NHSkKNu+lL9K1xkxs3pm9riZfWBmC83sdTM77wAde8OBOE5ZuHuqHBUdY4++27CBy4cN5tY77qbOYYeFjiP7IPY2JqUTaz3GmitmGRkZTJkxl9dzV7J40QKWL1sSOlK09RhrrnSQYUadGhlcM+Edxr72Ib86o0XoSFG2fSl/laozYoXvQBOBWe7e1N27AEOArF32S5tVxBo2zGLNmo+LbuflraFBgwYBE+1QUFDAiGGDGXThUPoNPCD9PQkg5jYmpRdrPcaaKx1kZh7OCT16MfOl6aGjRFuPseZKB19s2MLs978GYPnn37HNIbNmHP89iqntS/mrVJ0R4FRgi7s/uH2Du3/o7veb2XAzG29mk4DpAGb232Y238xyzew32x9jZj80s3lm9paZ/dnMMoo/iZkdmRxx6VfeLyina1dWrlzB6lWr2LJlC+OfHEe//gPL+2n3yt25btRIWrRsxVWjrg0dR/ZDrG1MyibWeow1V6y+XPcF69d/A0D+pk3MmfUyzVq0DJwq3nqMNVc6eG3VV3TMKpzR0DCzJtWqGOvzw503EmvbLzehTwqJ6KSRytYZyQYWlXD/icCl7n6qmZ0OtAC6AR2BLmbWy8xaA4OBHu7eEUgAF28/gJnVA54DbnL3lMtkmNlIM1tgZgu+WPfFfr2gqlWrcu99DzCg3xl0bNea8y+4kDbZ2ft1zANh3v9v777j5KrqN45/HkIaJASR3lukShISpEVAEAgqvYNIk6KAgD9RQBTEggIWRERABMUoTZESSiBAMHQSAkhHivSiECAhpPD9/XHu7M5udpNsyM45mzzvvPa1c+/Ozjy5M3vnnnvP+Z577uLKS4cz5o7b2GroELYaOoRbRt6QOxYAB++/L9tsMZRnnnqSdVZfiUsu/kPuSEC5uUp9j5W6vUrNVerrWGquUl/HN15/jb13GsawzTZgh62HMnTzrdhq2y/kjlXs61hqLijrPXbC1qvzq13XYflFezF8/0EMW2sJbnr8TZZZpCfn77UeJ27bnzNG/TtbPij3vW/tk/S8pEeqk/cPzPHjtNXfsquS9A1glYg4tlo+BxgKTAHOATaPiAOrn50J7Aa8U/16H+A0oDdwIvBGtb438NeIOEXSh8DTwBERMXp2Mg0ePCTuvHeOX59OM2HS1NwR2tSz+7zWPu5cvbp3m/WdMpk8dfqs72RNSn4tS1Tq+6vUfetS/XrljtDllPoe2+MP9+eO0KYL9hqYO8IMtt9qUx4eP7a4QUTrrLd+/GXEbB1GZjNwxUXGRsSQmd1H0vPAkIh46+M8VxmdA+eeR4FdawsRcYSkxYFaa2Bi3X0FnBYR59U/gKSjgD9GxAltPP40YCywLVD2u8jMzMzMiqSGF9At17x2GvpWoJekr9WtW6id+94EHCSpD4Ck5SQtCYwCdqtuI2kxSStVvxPAQcCako7vlP+BmZmZmVn5AhhZVa89dE4fZJ66MhIRIWkn4JeSvg28Sboa8h1Sd6v6+46sxofcXZUBfB/4ckQ8Jukk0sZdAJgKHAG8UP3edEl7AddKejciftuo/5+ZmZmZWQMs3mocyPkRcX6r+2waEa9UJ/BvlvRERNzR0Seapxoab+9qAAAgAElEQVQjABHxKqmcb1subnXfs4Cz2niMy4DL2ljfp/o+hdRVy8zMzMysQ7rAdDhvzWrMSES8Un1/Q9JVpKJQHW6MzGvdtMzMzMzMrBNJWlhS39ptYBvgX3PyWPPclREzMzMzM+tUSwFXVUMdFgT+EhE3zskDuTFiZmZmZtZA5ffSmrmIeBYYMDcey920zMzMzMwsCzdGzMzMzMwsC3fTMjMzMzNrFNH1+2nNRb4yYmZmZmZmWbgxYmZmZmZmWbgxYmZmZmZmWXjMiJmZmZlZA8mDRpr4yoiZmZmZmWXhxoiZmZmZmWXhblpmZmZmZg0iQO6l1cSNkfnU5KnTc0doU8/uZV6s69W9W+4IbZowaWruCO3qt1D33BHaVOp73zqm1Pf+Uv165Y7QpfjvsePO3nW93BHa9P2bnsodYQavvDs5dwSbDWUe+ZmZmZmZ2TzPV0bMzMzMzBrIvbSa+cqImZmZmZll4caImZmZmZll4W5aZmZmZmaN5H5aTXxlxMzMzMzMsnBjxMzMzMzMsnA3LTMzMzOzBpL7aTXxlREzMzMzM8vCjREzMzMzM8vC3bTMzMzMzBpI7qXVxFdGuoCRN93IeuuswTprrs4Zp/80dxwAJk+ezI5bD2XY5p9h603X5xc//WHuSE2OPOyr9F9pGTYeMiB3lBZKfB0BXn7pRXb50tZ8doNPs9mGA7jg3LNzR2pS4jYr9f0FZW4vKDNXyfuwErcXlJur1L/JUnMBTJ8+nZ233pjD9ts1a46DN1yes3dZmx9/4VMz/Gy7NRfnj/usR5+e3TIks0ZyY6Rw06dP55hvHMHV197Agw8/xhWX/pXHH3ssdyx69uzJX666kRtH38f1t9/L6FtHMu6Be3PHAmDv/b7Clf8YkTtGC6W+jgALLrggp/zodP55/yNcf8sYLrrgXJ58In+2UrdZie8vKHd7lZqr1H1Yqdur1FxQ7t9kqbkA/nTBOazaf43cMRjz7NucedtzM6xfbKHurLNMX96aOCVDKms0N0YKd/9997Haaquzyqqr0qNHD3bfcy+uu/bq3LGQxMJ9+gAwbepUpk2dhgq55rjp0M34xGKL5Y7RQqmvI8BSSy/DegMHAdCnb1/6r7Emr73ySuZU5W6zEt9fUO72KjVXqfuwUrdXqbmg3L/JUnO99srLjB51I7vvc0DuKDz55kQmTpk2w/p91l+Gyx58lYgMoazh3Bgp3CuvvMzyy6/QtLzccsvz8ssvZ0zUbPr06Wy3xYYMXmtFhm6xJYMGfyZ3pGKV/DrW+88Lz/Ovhx9i/SH5X8uuss1KUer2KjUXlLkPK3V7lZrLOu4n3/823zrpx2iBMg8BBy23CG9/MI0X35mcO0qnUuFfjVTmO3E2SZouaXzd18qd+FxbSLqusx6/PdHGaYESzt4BdOvWjRtuv5e7H36Gh8Y9wJOPP5o7UrFKfh1rJr7/Pl/db09OPe1M+i6ySO44XWKblaTU7VVqLihzH1bq9io1l3XMbTffwCcXX4J1BwzKHaVNPbqJ7ddZkr8//FruKNZAXboxAnwQEQPrvp6v/6GkLl8tbLnlluell15sWn755ZdYdtllMyaaUb9+i7LRppsxetTI3FGKVfrrOHXqVA7eb0922WNvvrjDzrnjAOVvs9KUur1KzVWvpH1Yqdur1FzWMePuu5tbR45gyw3W4v8O3597x4zmuCMOyh2ryZJ9erJEnx78cLtPceYOa7LYQt05dVh/+vXq8odzNhNdvTEyA0kHSLpC0rXAyGrdcZLul/SwpB9U61aW9LikCyQ9KmmkpN7Vz1aXdIukhySNk7Ra9fB9JF0p6QlJw9WA00JDNtiAZ555muefe44pU6ZwxWWX8sUv7dDZTztL/33rTSZMeAeAyR98wJ133MpqBQyGK1WpryOkM57HHnko/ddYk8OPPCZ3nCYlb7MSlbq9Ss1V6j6s1O1Vai7rmP/77qmMHvc0t97/OD//3R/ZcOjmnHHOH3LHavLShMkc9ffH+NY1T/Cta57gf5Om8v0bn2bC5BnHlXR5ufthFdRPq6s3RnrXddG6qm79xsD+EbGlpG2A/sBngIHAYEmbVffrD5wTEesA7wC1GnfDq/UDgE2AV6v1g4BjgLWBVYFN2wol6VBJD0h64M233vxY/8EFF1yQX571G7b/4rYM/PRa7Lr7Hqy9zjof6zHnhjdef429dxrGsM02YIethzJ0863Yatsv5I4FwMH778s2WwzlmaeeZJ3VV+KSi/PvaEt9HQHuu+currx0OGPuuI2thg5hq6FDuGXkDbljFbvNSnx/Qbnbq9Rcpe7DSt1epeaCcv8mS81Vkq9tsiLf22Z1ll6kJ7/caU02W/UTuSNZBmqrH2hXIen9iOjTat0BwOYRcWC1fCawG6mxAdAHOA0YBdwcEf2r+30H6A6cBTweEcu3etwtgO9GxNbV8rnAnRHx55llHDx4SNx57wMf57/ZKV6fUObAsH4Ldc8doU29updZ53zCpKm5I7Sr1Ndy8tTpuSO0qdT3WKlK3Yct1a9X7ghdSql/jyV7fcKHuSO06Se3PpM7wgxGnLQ3bz37aHGDm9YdsH78feSY3DFmao2lFx4bEUMa8Vzzaie8iXW3BZwWEefV36Ea7F7/Fz0d6M3ML061vv+8uv3MzMzMrBOknlDFtZGy6erdtGbHTcBBkvoASFpO0pLt3Tki3gVekrRTdf+ekhZqTFQzMzMzs/nHPN8YiYiRwF+AuyU9AlwJ9J3Fr+0HfEPSw8BdwNKdm9LMzMzMbP7TpbsZtR4vUq27GLi41bqzSGNBWlu37j5n1t1+Gtiy1X2fBW6vu8+RcxDZzMzMzOZnAk/T02yevzJiZmZmZmZlcmPEzMzMzMyy6NLdtMzMzMzMuhr30mrmKyNmZmZmZpaFGyNmZmZmZpaFGyNmZmZmZpaFx4yYmZmZmTWSB4008ZURMzMzMzPLwo0RMzMzMzPLwt20zMzMzMwaRsj9tJr4yoiZmZmZmWXhxoiZmZmZmWXhblpmZmZmZg0k99Jq4sZIJxs3buxbvbvrhbn0cIsDb82lx5qbnKtjnKtjnKtjnKvjSs3mXB3jXB0zP+RaaS49jnUiN0Y6WUQsMbceS9IDETFkbj3e3OJcHeNcHeNcHeNcHVdqNufqGOfqGOeyUrgxYmZmZmbWIMJzHtbzAHYzMzMzM8vCjZGu5fzcAdrhXB3jXB3jXB3jXB1Xajbn6hjn6hjnsiIoInJnMDMzMzObL6w3cHBcM+rO3DFmapXFe49t1NgdXxkxMzMzM7Ms3BgxMzMzM7Ms3BgxMzMzM7MsXNrXrBNJWgj4P2DFiDhEUn9gjYi4LnM0MzMzy0Qu7tvEjZEuQNImwMrUvV4R8adsgSqSliPNblqf6458iUBST2BXZtxep2aKdBEwFti4Wn4JuAJwY2QWJHUDlqLl6/iffIlA0hLAIcz4/jooV6YaSUOB/hFxUZWzT0Q8lznTUsBPgGUjYjtJawMbR8SFmXMV+TpKWhj4ICI+qpYXAHpFxKScuaosxe3voch9fpMSt5mk7SLihlbrDo+I3+XKVJejyGMd63xujBRO0iXAasB4YHq1OoCsf6CSfgbsCTxGy1y5P5yuBiaQGgAfZs4CsFpE7Clpb4CI+EBSMadDJO0C/AxYkuZ5mCIiFsmc6yjgZOB14KNqdQDrZQuVXA38E7iF5vd9dpJOBoYAa5AawN2BPwOb5swFXEzK891q+SngMiBrY4RCX0dgFPB54P1qeSFgJLBJtkQUvb+H8vb5QNHb7HuSPoyIWwEkfQfYAsjaGCn1WMcaw42R8g0B1o7yajDvROpuVMzOv7J8RAzLHaLOFEm9STtVJK1GQR+YwOnA9hHxeO4grRxNen/9N3eQVhaKiO/kDtGGnYFBwDiAiHhFUt+8kQBYPCIul3QCQERMk1TCwX+pr2OviKg1RIiI96uunrmVur+H8vb5NaVusx2A6yQdBwwD1qzW5VbqsU6nKee0ZH4ewF6+fwFL5w7RhmdJZ19Lc5ekT+cOUedk4EZgBUnDSWc+v503UguvF9gQAXiRdLazNNdJ+kLuEG2YUn2I1xq9C2fOUzNR0idpzrURZbyupb6OEyWtX1uQNBj4IGOemlL391DePr+myG0WEW+RGh/nAMsCu0XE1LypgHKPdawBfGWkfIsDj0m6j7oz6hGR+0zGJGC8pFG0zPWNHGEkPUI64FkQOFDSs1WuWrejLN17IuJmSeOAjaosR1cfBllV3bMAHpB0GfAPWr6Of8+U65vVzWeB2yWNaJXrF5lyvUd6fwk4UdKHwFQK6dYGXC7pPGBRSYcABwEXZM4E8E3gGmA1SXcCSwC75QrTBV7HY4ArJL1SLS9D6uqTW1H7eyh3n1+nqG1W996v6QGsCuwmKdt7X9K1Va6+lHmsYw3gxkj5TskdoB3XVF+l+FLuAG2RtCkwPiJGSPoy6QDorIh4IXO07etuTwK2qVsOIEtjhPSBBPCf6qtH9ZVVRJTQ5aldEXGmpK2Bd0njRr4fETdnjkVEjJO0eZVJwJM5z8J2gdfxfklr0ry9nijkrHVp+3sodJ9fp6htVvB7/8zcAXJxL61mmo+659l8oBqT8VJEfChpC9KA5z9FxDuZ8jwMDKjlAP4A7BIRm+fIYx9PXeNyYtW4XB/4VQFVvhYGJkfEdElrkA5mb8h9ICtpd+DGiHhP0kmk7fWjiBiXOVdRr6OkLSPi1rorli3kulLZFZS2zy9dVUBlX2CViPihpBWAZSLivsy5VgFejYjJ1XJvYKmIeD5nrs6y3sDBMeLWu3LHmKkVP9lrbEQMacRzecxI4SRtJOl+Se9LmiJpuqR3C8jVX9KVkh6T9GztK3cu4G/AdEmrkyr2rAL8JWOeaVVf/h2BX0fEWTSf/c9O0h8lLVq3/AlJf8iZqcpxcxu5bsqZqXIuMEnSANLYnxeAS/JGAlKFnp5VKdFbgANJlaxy+17VEBkKbAv8kbQNcyvtdaydnNi+ja/sVwAK3t9Deft8oOht9ltSqfl9quX3SeNHcruC5sqJkCpqXZEpizWYu2mV7zfAXqQ/yiHAV4D+WRMlF5EGZ/8S+Bzp4KeEq44fVRV7diGd6Txb0oMZ87xXVRLaD/is0twZJQ1qXK/+DGJEvC1pUM5AlSXayLVkzkCVaRERknYEzoqICyXtnzsU6Sr3JEkHA2dHxOmZ3/c1tcpZXwTOjYirJZ2SMU9NUa9jRJxcfT8wV4ZZKHV/D+Xt82tK3WYbRsT6tW1U7Vuzd4UFFoyIKbWFiJhSSK7OIVfTqucrI11ARDwDdIuI6RFxEakmeG69I2IU6SDohYg4BdgycyaAqUpzenyF5okFcx7870kajHdQRLwGLAeckTFPawtI+kRtQdJilHGSYrqkFWsLklai5eDLXGqNyy8DIwpqXErSxqTuFyOqdSW8ji9XA+v3AK5XmqCuhM+dIl9HSZ+U9GtJ4ySNlXSWUjWy3Erd30N5+/yaUrfZ1Or9XqtwtwQtr0jk8qakpsHq1YmC7MVerDFK+LCymZtUnR0YL+l04FWghLKdk5VmB35a0pHAy6SJ83I7EDgc+HFEPFf1Q/1zrjAR8Zqkv9F8Nest4Kpcedrwc1JpzCtJH057kGbMzu27wBhJo6vlzYBDM+ap2ZPUveHg6rVdkTIal8cAJwBXRcSjklYFbsucCdL7aRhwZkS8I2kZ4LjMmaDc1/FSUpe7XavlfUmTRH4+W6Kk1P09FLbPr1PqNvs16TNoKUk/JlW3OylvJCC9hsMl/YZ0BelFUgPT5gMewF646ozw66SKQscC/YDfVldLcubaAHgcWBT4YZXr9Ii4J2eu0iiVWT0UWCwiVpPUH/hdRGyVOVoTSWuTztgJGBURj2WOBICkxWkuiXx3CSWRSydp4YiYmDtHvWq8SP+IuKg6C9snIp7LnatEksZGxOBW6x5o1CDS9nh/33ElbzOlim1b0bzPL2auKUl9SMem7+XO0pnWGzQ4rr/17twxZmqFxXo2bAC7GyNdQFVVYsWIeDJ3llJJujwi9lBz7fkWctWclzQe+Axwb0QMqtY9EhFFTNIl6ZKI2G9W6xqtruLLqhFxanXmeulcFV8kjYmIoZqxVn8R81NUXbQuJB3or1gNzD4sIr6eOdfJpLFua0TEpyQtC1wREZtmylP663gm8ABwebVqN2Cd2pgSa1bqPr8rKPEEQdWFc1dgZep67UTEqbkydSY3RlpyN63CSdqeVIe7B7CKpIHAqbkmApL0q4g4Rs0TFbWQcYKio6vv2SvPtPJhNRAPAEkLUsbYh5p16heqvsSD27lvI/2W1I95S+BU4D1S1ZwNcoSJiKHV92IqobXyK1K1qmsAIuIhSZvljQTAzsAgYBxARLwiKds2LPV1VMvJGL9JczejBUjVjrI2RiQNIXWdXImWB4o5D/iL3OcX/BkJtDxBQBpk3530fstygqDO1cAEYCx1kx7a/MGNkfKdQjqzfjtARIyXtHK+OE3lL4uaqCgiXq2+555MsLXRkk4EeitNSvd14NrMmagG79ZyvUtzlZcpwPnZgjUrruJL1f/74YhYN2eO9kTEi2pZnmV6e/dtoClV1araYNkSxrsBTQ3vpWh5cJ1lnpHSGkdtGE4a6/MIZQx2LnmfX+RnZJ2iThDUWT4ihuUOYXm4MVK+aRExQYXUgIuIsdX30bO6byO11+2C/N0vjgcOJn2IHwZcD/w+U5YmEXEacJqk0yLihNx52lBcxZeI+EjSQ5JWzHXQOhMvStoEiKrR9g1Sf/XcLq+qaS1ajZ86CLggcyYkHUW62vA6ze+rIE2Yl1VV3a4/0Ku2LiLuyJcIgDcjopjZxKHNfX7Tj8i4z2/rM7J6TVeIiIdzZGql1BMEd0n6dEQ8kjtIIwiX9q3nxkj5/iVpH6BbNfj5G0C2aTvb659bk+uyfalnFiPiI9LBV/YDsHbc0FZ3ngIOfmoVX5YsrOLLMsCjku4DmgaK5+56QapEcxapdPRLwEjgiKyJgIg4s7oi+C6pW8j3I+LmzLEgdfFZIyL+mztIPUlfJWVbHhhPKuBwN/lLwp4s6ffAKOq60ETGmeFL3efXSLod2IF0nDWeVLp2dER8M2uwQk8QAEOBAyQ9R3qP1RqV2U8QWOdzY6R8R5H66n5ImlX2JuBHGfMU1T+3LdXg3c9Wi3fkPBslaVNSV7taX+vaDnbVXJlaqS+z2ovUJXAsmQ9+ImK4pLE0V3zZqZCKLz/IHaAtVaWxfXPnaEtE3CzpXqrPG0mLRcT/Msd6kdQ/vTRHk8ZF3RMRn6uqHpXwnjsQWJM0vqD+SlK2xkiN0txIrb0XEVMbHqalfhHxbtXAvCgiTpaU/cpIwScItssdwPJxY6RgVTeVH0TEcaQGSXYF9s9tQdLRwCE0f0gOl3R+RJydKdKFpJLMYymjD38LEbF9/bKkFYDTM8Wpz3EhaSbxc+rWnRJp4rBsSuueWFN1YzuEGSvRHJQrE4Ckw0gFCD4gHcTWuk/mbow/C9wuaQQtz/T/Il8kACZHxGRJSOoZEU9IWiNzJoABpVQAbMM4YAXgbdL7a1HgVUlvAIfUuk1lsKDSvDp7UMjnd3VMcVNEfB4ooQHSJCJeaKvKV+5cncm9tJq5MVKwiJguqYTKRjOQtBFwNrAWqdJXN2Bi7tKYpPEZG0Y114Kkn5G6OeRqjEyIiBsyPfeceAkoYYD2tsBgSb+IiD9V63YgXWVquFL7p9e5GvgncAtlNXq/RSpNW9ocMf+pvnpUX6V4SdKiwD+AmyW9DbySORPAPZLWjkLmIGrlRtJknzcBSNqGNNHm5aSqfBtmynUqqSfDmIi4X2ki0qczZQGajikmSeoXEUVdGSy4ypc1gBsj5XtQ0jXAFbTso5778vhvgL1IuYaQZkpdPWuiRLQ8GJtO3hMQt0k6g3Slpv4M7Lh8kZpJOpvmg+wFgIHAQ/kSNXkD2IJ0ZWtDUveVbK9j6f3TgYUi4ju5Q7Th38Ck3CFai4gSuj7NICJ2rm6eIuk20kR5N2aMVDMU2L/Q/vxDIuLw2kJEjJT0k4j4ZjV3RRYRcQXp87G2/CxpHo3cJgOPSLqZlscU38gXCSi3ypc1gBsj5VsM+C8t+/AX0Vc3Ip6R1C0ipgMXSco2sL7ORcC9kq6qlncidZXKpXZWrn7ioCD/gNSaB+puTwP+GhF35gpTRxHxLrC9pFNIpa37ZU1UtuskfSEirs8dpJUTSFVy7qVlYzzrgU/VBeTbpHl26qtWZfu7bF06urAugSWXXP2fpO8Al1bLewJvV12SslXgk3Q6aXznB6QG5QDgmIj480x/sfONqL6g+URUCT2GSq3y1WlcTauZGyOFi4gDc2dox6SqhOj4aqf7KpB95xERv6iqmAwl7WAPjIgHM+b5XK7nnh0R8cfqwIyIeDN3HkmrA0tTTd4HEBGnVB9QJUziV6qjgRMlfQhMpZzuY+cBt1LQ/BSV4cBlpIIchwP7A1nf/6WWjq4aSSNKnV8H2IdUpvkfpPf9mGpdN9J4jVy2iYhvS9qZ1P11d+A2mie0bChJO5Lm8jinWr4PWILUICnhqmqpVb6sAdwYKZyk5UnjHTYl7TTGAEdHxEtZg8F+pG49R5IGaK8A7JIrjKRepIOK1UkHPr+NiGm58tST9EVmPAN7ar5EoDRxzcmk10/AApKmkQaN58z2K+DEiGg94/QIWl5dsjoFdyObVkAp07Z8MiIulHR0dQVitKQSrkS0VTo6ImLHXIFKbSTVVOORjmrnx880Mksr3avvXyBdcf5f5vnCvk3qWl3TAxhMGiR+EXVdynIouMqXNYAbI+W7iFTSd/dq+cvVuq2zJUp2ioizSP1PfwBNlazOypTnj6Qzwv8klQhcCzgmU5Ymkn4HLAR8jjTZ4W7AfVlDJceQGrgbRMRzANUAy3MlHRsRv8yUa+W2SjFHxAOSVm58nLJJWrOquLR+Wz8vYGzSbZIOBa6lZTet3KV9a2VfX61OFrxCmtsjt/qxLCJd4d07U5Z6pc6vg6RPkQolrEzLSnK5u8JeK+kJUjetr1dXoCdnzNMjIl6sWx5T/R3+L3eXqJKrfHUmFdE7rgyKaHf+OiuApPERMXBW6xpN0riIWL/VugcjYlCmPI/USk9KWhC4r3W+HCQ9HBHr1X3vA/w9IrbJnOtBYOvWVY6qD8yRGV/HZyKizUIIM/vZ/KoqW31oNdi5tch9QFYNeG4tIvM8O5K+RDpxsQLpyvMipDLq2WcZlzSQ1M1oD+A50v4iVzXAWqbN21pfwrgWSQ8Bv6NV+fSMJX2bKM28/m5VxWohYJGIeC1TlpntW/8dEas1OlOrDNcA+5VW5auzDBg0OG66/Z7cMWZqmUV7jI2IhvRI8JWR8r0l6cvAX6vlvUkD2rOQtDfpg3KVaudR05eMuWg+00lETMt8ObzeB9X3SZKWJW2jVTLmqeneVrnViHhTUve2fqFB7pd0SES06Css6WDSwYbViYhDq+9Fjk2KiBLe6zOIiOuqmxNIVy2zqs7u70Xz/v0y0snC7NkgNTokrUSaA+KW6sC6W+5clWkRcW7uEK1J+krd7fof/WnGezfEve3sWw+jjKv1pVb5sgZwY6R8B5HK6P6SNGbkLtJsuLncRRqsvjjw87r17wE5Z5cdIOnd6raA3tVy7oG811XzBpxBKlkYlDEob8oc/qyzHQNcJWlfmhsfQ0j9m3du97cMSZswY1eVXAc+TSStC6xNyzFTWXNVXRLPAjYmDay/Gzi2Kr+awxOkKzXbR8QzVcZjM2WZQTWg+FBSdcfVgOVIVyO2ypmrcq2krwNXUVZXwA3qbvcibatx5GuMHAv8Q9I+VQ5IY0Z6kqpO5lZf5Wv+UMw50/zcTatQkpZvb5C6pO0j4tpGZ7KPp6p536uEy9CSplN39qn+R6SMOa+OIOlzNE+++GhE3JozT+kkXUI6SBxPc1eVyH1WUWkisy1IjZHrSeO5xkTEbplz3QOcQ/MV572AoyIiywR5VcWlvYBNSGVgLwV+X8qVJUnjgc8A99a6cNZ3jc2p1K6ArUnqB1ySe5yNpC1JBVWggH1rqYUROtuAQYPjptGFd9Pq525aBqMkbRsRz9evlHQgcBJpQGjDqfyZqItSVfn6OmkgagBjJJ0bETkHMhIRpXSxaFNE3EYqg2mzZwiwdpR3dmk30vwKD0bEgZKWIhVyyE0RcUnd8p8lHZkrTERcRboiuDDpLPWxwFKSziXNLj4yV7bKhxExpdbdqBqXV8R7rZQG22yYBPTPHaJqfJR0cucfwPoAkv4WESVMDGkNtkDuANauY4GbJTXtvCSdAHwTaHMwYSNERN+IWKSNr75uiLTpT6SzUGeTututBVwy098w67h/keZnKc0HEfERME3SIsAbQAlnrG+TdLyklSWtJOnbwAhJi0laLFeoiJgYEcMj4kuk6l7jgeNz5akzWtKJpO6vW5PKwGa9Ol+9ZrXbu7f62U8an6glSddKuqb6ug54krr5k6xJfWelEvYNDaPCvxrJV0YKFRHXK01gdoOknYCvkvqgbhYRb+dNZx2wRkQMqFu+rar+YvaxSbqWdIa6L/BYVXq1vt987tKrD1Rjpi4gjQF6nzIGy+5ZfT+0+l777D2ItD2zHxRVYx7Oq75yOx44mDSH06GkSRBzX+HaCzi9un0CLefJGAac2PBELZ1Zd3sa8EJ7Xa/nc9HObZuPuDFSsIgYJekA4HbSwPGtcnfvsQ57UNJGEXEPgKQNgTszZ7J5xzXAUqTBz/U2B15ufJyWIuLr1c3fSbqRVNo0W6ELSRsAL9a69kjaH9gVeB44pYBBz0VRy1m7L6gGsi8BDJb0TkRcmTNeO7fbWm641mWPJXWTtG9EDM+VqVC14jP1hWfAXb/nK26MFKpubIZI1S62At5Q6rTrP9CuY0PgK5JqA/RWBB6X9AjpdVwvXwFGRrkAAAmWSURBVDSbB+xImrG+xQG+pInAycCFWVK1zLIcsBLV542kzSLijkxxzgM+X8sBnEaavXsgcD5pjIs1m9Ws3TkbIzM7o57tDHvVHfEIUsWxa0iT+B0BHEfqdufGSJ3Sxy9aY7gxUqiI6Js7g80Vw3IHsHla0TPWS/oZqUvUY9RV+QJyNUa61V392BM4PyL+BvytqhhlLRU7azczP6Peq/1f63SXAG+TykV/ldQI6QHsGBF+jxkAUvqyxI0Rs04UES8ASFqSlvMszHelDK1TzOygq3fDUrRvJ9K4qQ9nec/G6CZpwYiYRrrafGjdz/x5OKNP1C9ERH3FsSUanKWFgs+or1oreSzp98BbwIoR8V7eWGblcjUts04kaQdJTwPPAaNJfdNvyBrK5iX3V/34Wyhoxvpngaxz1rTyV1JlqKuBD6jG2khanTQbu7V0bzvvr1Jm7S7R1NqNiJgOPOeGiNnM+UyQWef6IbARcEtEDKom89s7cyabd5Q+Y/0kYLykUbSs8pVlMsaI+HGVZRlgZN28LAuQxo5YS6XP2l2iAa26jPWu607m8Z7WRPnrLBTDjRGzzjU1Iv4raQFJC0TEbVU/erOPLSJeBzZpNWP9iNyzKte5hhnnVshavrNW2a7VuqdyZCldRLxBen/Vz9pd0vurOAV3HzMrlhsjZp3rHUl9SN1Bhkt6g1Rz3myuKXjG+kUj4qz6FZKOzhXG5kyBs3ab2TzEY0bMOtcOpK4qRwM3As8AX8qayKxx9m9j3QGNDmFmVpzcU6wXNAW7r4yYdYK6eWJarK6+f1/Sv4HvRsSoxiYz63yS9gb2AVaRVN9Nqy/w3zypzMysRG6MmHWCmc0TI6kbqX//cJr7+ZvNS+4CXgUWB35et/49INsM7GZmVh43RswarCr3+JCks3NnMesM1fw6LwAbS1oJ6B8Rt0jqTZr/xKVOzWy+5lpazTxmxCyTiDgvdwazzlTNUXElUHuvLw/8I18iMzMrjRsjZmbWWY4ANgXeBYiIp4ElsyYyM7OiuJuWmZl1lg8jYoqUOiRIWpDM84yYmZVA7qfVxFdGzMyss4yWdCJpFuqtgSuAazNnMjOzgrgxYmZmneV44E3gEeAw4HrgpKyJzMysKO6mZWZmnSIiPgIuqL7MzMxm4MaImZnNVZIuj4g9JD1CG2NEImK9DLHMzAoh5OK+TdwYMTOzue09SZsC2+MB62ZmNhNujJiZ2dz2MHAmsAxwGfDXiBifN5KZmZXIA9jNzGyuioizImJjYHPgf8BFkh6X9H1Jn8ocz8wsK5FK+5b81UhujJiZWaeIiBci4mcRMQjYB9gZeDxzLDMzK4gbI2Zm1ikkdZe0vaThwA3AU8CumWOZmVlBPGbEzMzmqmqCw72BLwL3AZcCh0bExKzBzMysOG6MmJnZ3HYi8BfgWxHxv9xhzMysXG6MmJnZXBURn8udwczMugY3RszMzMzMGqjRFatK5gHsZmZmZmaWhRsjZmaFkDRd0nhJ/5J0haSFPsZjbSHpuur2DpKOn8l9F5X09Tl4jlMkfWt217e6z8WSduvAc60s6V8dzWhmZmVzY8TMrBwfRMTAiFgXmAIcXv9DJR3eb0fENRHx05ncZVGgw40RMzObMyr8XyO5MWJmVqZ/AqtXVwQel/RbYBywgqRtJN0taVx1BaUPgKRhkp6QNAbYpfZAkg6Q9Jvq9lKSrpL0UPW1CfBTYLXqqswZ1f2Ok3S/pIcl/aDusb4r6UlJtwBrzOo/IemQ6nEekvS3Vld7Pi/pn5KekvSl6v7dJJ1R99yHfdwNaWZm5XJjxMysMJIWBLYDHqlWrQH8qZrJfCJwEvD5iFgfeAD4pqRewAXA9sBngaXbefhfA6MjYgCwPvAocDzw7+qqzHGStgH6A58BBgKDJW0maTCwFzCI1NjZYDb+O3+PiA2q53scOLjuZysDm5PmI/ld9X84GJgQERtUj3+IpFVm43nMzKwLcjUtM7Ny9JY0vrr9T+BCYFnghYi4p1q/EbA2cKdSOZYewN3AmsBzEfE0gKQ/A4e28RxbAl8BiIjpwARJn2h1n22qrwer5T6kxklf4KqImFQ9xzWz8X9aV9KPSF3B+gA31f3s8oj4CHha0rPV/2EbYL268ST9qud+ajaey8zMuhg3RszMyvFBRAysX1E1OOpnLhdwc0Ts3ep+A4GYSzkEnBYR57V6jmPm4DkuBnaKiIckHQBsUfez1o8V1XMfFRH1jRYkrdzB5zUzK5Nc2reeu2mZmXUt9wCbSlodQNJCkj4FPAGsImm16n57t/P7o4CvVb/bTdIiwHukqx41NwEH1Y1FWU7SksAdwM6SekvqS+oSNit9gVcldQf2bfWz3SUtUGVeFXiyeu6vVfdH0qckLTwbz2NmZl2Qr4yYmXUhEfFmdYXhr5J6VqtPioinJB0KjJD0FjAGWLeNhzgaOF/SwcB04GsRcbekO6vSuTdU40bWAu6ursy8D3w5IsZJugwYD7xA6ko2K98D7q3u/wgtGz1PAqOBpYDDI2KypN+TxpKMU3ryN4GdZm/rmJlZV6OIuXVV38zMzMzMZmb9wUNi9J335Y4xU4v07jY2IoY04rncTcvMzMzMzLJwY8TMzMzMzLLwmBEzMzMzs0ZyNa0mvjJiZmZmZmZZuDFiZmZmZmZZuJuWmZmZmVkDyf20mvjKiJmZmZmZZeHGiJmZmZmZZeFuWmZmZmZmDST30mriKyNmZmZmZpaFGyNmZmZmZpaFGyNmZmZmZpaFx4yYmZmZmTWQh4w085URMzMzMzPLwo0RMzMzMzPLwt20zMzMzMwayf20mvjKiJmZmZmZZeHGiJmZmZmZZeFuWmZmZmZmDST302riKyNmZmZmZpaFGyNmZmZmZpaFGyNmZmZmZg0iQCr7a7b+H9IwSU9KekbS8XO6PdwYMTMzMzOz2SapG3AOsB2wNrC3pLXn5LHcGDEzMzMzs474DPBMRDwbEVOAS4Ed5+SBXE3LzMzMzKxBxo0be1Pv7lo8d45Z6CXpgbrl8yPi/Lrl5YAX65ZfAjackydyY8TMzMzMrEEiYljuDHNBWyNLYk4eyN20zMzMzMysI14CVqhbXh54ZU4eyI0RMzMzMzPriPuB/pJWkdQD2Au4Zk4eyN20zMzMzMxstkXENElHAjcB3YA/RMSjc/JYipij7l1mZmZmZmYfi7tpmZmZmZlZFm6MmJmZmZlZFm6MmJmZmZlZFm6MmJmZmZlZFm6MmJmZmZlZFm6MmJmZmZlZFm6MmJmZmZlZFv8PLkGSB2slUsAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x864 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,12))\n",
    "plot_confusion_matrix(cm, y.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>cl</th>\n",
       "      <th>pred</th>\n",
       "      <th>prob</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2907</th>\n",
       "      <td>Do</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Irish</td>\n",
       "      <td>0.176679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2935</th>\n",
       "      <td>Ta</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>0.226875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2924</th>\n",
       "      <td>Luc</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>0.253900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2944</th>\n",
       "      <td>Ton</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>0.282859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2930</th>\n",
       "      <td>Pho</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Dutch</td>\n",
       "      <td>0.296166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2910</th>\n",
       "      <td>Ly</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Russian</td>\n",
       "      <td>0.298872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2915</th>\n",
       "      <td>Doan</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>0.307318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2916</th>\n",
       "      <td>Dam</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>0.325199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2906</th>\n",
       "      <td>Dang</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>0.388393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2900</th>\n",
       "      <td>Pham</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>0.396346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2928</th>\n",
       "      <td>Nghiem</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>English</td>\n",
       "      <td>0.428147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2932</th>\n",
       "      <td>Quach</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>0.450314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2922</th>\n",
       "      <td>Lac</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Irish</td>\n",
       "      <td>0.450851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2938</th>\n",
       "      <td>Thi</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>0.455845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2902</th>\n",
       "      <td>Hoang</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Korean</td>\n",
       "      <td>0.487704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2927</th>\n",
       "      <td>Mach</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Irish</td>\n",
       "      <td>0.505350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2917</th>\n",
       "      <td>Dao</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Irish</td>\n",
       "      <td>0.512267</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2923</th>\n",
       "      <td>Lieu</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>French</td>\n",
       "      <td>0.515336</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2939</th>\n",
       "      <td>Than</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>0.530805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2937</th>\n",
       "      <td>Thai</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>Irish</td>\n",
       "      <td>0.580569</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        name          cl        pred      prob\n",
       "2907      Do  Vietnamese       Irish  0.176679\n",
       "2935      Ta  Vietnamese    Japanese  0.226875\n",
       "2924     Luc  Vietnamese  Vietnamese  0.253900\n",
       "2944     Ton  Vietnamese  Vietnamese  0.282859\n",
       "2930     Pho  Vietnamese       Dutch  0.296166\n",
       "2910      Ly  Vietnamese     Russian  0.298872\n",
       "2915    Doan  Vietnamese     Chinese  0.307318\n",
       "2916     Dam  Vietnamese      Arabic  0.325199\n",
       "2906    Dang  Vietnamese     Chinese  0.388393\n",
       "2900    Pham  Vietnamese      Arabic  0.396346\n",
       "2928  Nghiem  Vietnamese     English  0.428147\n",
       "2932   Quach  Vietnamese  Vietnamese  0.450314\n",
       "2922     Lac  Vietnamese       Irish  0.450851\n",
       "2938     Thi  Vietnamese     Chinese  0.455845\n",
       "2902   Hoang  Vietnamese      Korean  0.487704\n",
       "2927    Mach  Vietnamese       Irish  0.505350\n",
       "2917     Dao  Vietnamese       Irish  0.512267\n",
       "2923    Lieu  Vietnamese      French  0.515336\n",
       "2939    Than  Vietnamese     Chinese  0.530805\n",
       "2937    Thai  Vietnamese       Irish  0.580569"
      ]
     },
     "execution_count": 256,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_preds[bigram_preds.cl == 'Vietnamese'].sort_values('prob').head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So our baseline is 53%. Let's see if we can do better with deep learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning\n",
    "## Build a Fastai Data Loader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load in the dataframe and extract indexes for training, validation and balanced trainings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('names_clean.csv')\n",
    "\n",
    "valid_idx = df[df.valid].index\n",
    "train_idx = df[~df.valid].index\n",
    "\n",
    "bal_idx = []\n",
    "for k, v in zip(df.index, df.bal):\n",
    "    bal_idx += [k]*v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As of December 2018 Fastai only has Word level tokenizers; we'll have to create our own letter tokenizer.\n",
    "\n",
    "The fastai library injects `BOS` markers (`xxbos`) at the start of every string; we'll have to parse them separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LetterTokenizer(BaseTokenizer):\n",
    "    \"Character level tokenizer function.\"\n",
    "    def __init__(self, lang): pass\n",
    "    def tokenizer(self, t:str) -> List[str]:\n",
    "        out = []\n",
    "        i = 0\n",
    "        while i < len(t):\n",
    "            if t[i:].startswith(BOS):\n",
    "                out.append(BOS)\n",
    "                i += len(BOS)\n",
    "            else:\n",
    "                out.append(t[i])\n",
    "                i += 1\n",
    "        return out\n",
    "            \n",
    "    def add_special_cases(self, toks:Collection[str]): pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a vocab of all ASCII letters, and a character tokenizer that doesn't do any specific processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "itos = [UNK, BOS] + list(string.ascii_lowercase + \" -'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab=Vocab(itos)\n",
    "tokenizer=Tokenizer(LetterTokenizer, pre_rules=[], post_rules=[])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a data pipeline using the `TextDataBunch.from_df` constructor.\n",
    "\n",
    "`mark_fields` puts and extra `xxfld` marker between each field of text. Since we only have 1 field this is unnecessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = df.iloc[train_idx, [0,2]]\n",
    "valid_df = df.iloc[valid_idx, [0,2]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>ascii_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Korean</td>\n",
       "      <td>ahn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Korean</td>\n",
       "      <td>bang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Korean</td>\n",
       "      <td>byon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Korean</td>\n",
       "      <td>gil</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Korean</td>\n",
       "      <td>gu</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        cl ascii_name\n",
       "0   Korean        ahn\n",
       "2   Korean       bang\n",
       "3   Korean       byon\n",
       "10  Korean        gil\n",
       "11  Korean         gu"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = TextClasDataBunch.from_df(path='.', train_df=train_df, valid_df=valid_df,\n",
    "                         tokenizer=tokenizer, vocab=vocab,\n",
    "                         mark_fields=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='90%'>  <col width='10%'>  <tr>\n",
       "    <th>text</th>\n",
       "    <th>target</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  v o n   g r i m m e l s h a u s e n</th>\n",
       "    <th>German</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  m a c e a c h t h i g h e a r n a</th>\n",
       "    <th>Irish</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  c h k h a r t i s h v i l i</th>\n",
       "    <th>Russian</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  t z e h m i s t r e n k o</th>\n",
       "    <th>Russian</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  c h e p t y g m a s h e v</th>\n",
       "    <th>Russian</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can create it using data block API.\n",
    "This uses the `processors` to tokenize and numericalize the input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "processors = [TokenizeProcessor(tokenizer=tokenizer, mark_fields=False),\n",
    "            NumericalizeProcessor(vocab=vocab)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = (TextList\n",
    "     .from_df(df, \n",
    "              cols=[2], \n",
    "              processor=processors)\n",
    "     .split_by_idxs(train_idx=train_idx, valid_idx=valid_idx)\n",
    "     .label_from_df(cols=0)\n",
    "     .databunch(bs=32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='90%'>  <col width='10%'>  <tr>\n",
       "    <th>text</th>\n",
       "    <th>target</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  v o n   g r i m m e l s h a u s e n</th>\n",
       "    <th>German</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  p a r a s k e v o p o u l o s</th>\n",
       "    <th>Greek</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  d z h a v a h i s h v i l i</th>\n",
       "    <th>Russian</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  s h a h n a z a r y a n t s</th>\n",
       "    <th>Russian</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  m o g i l n i c h e n k o</th>\n",
       "    <th>Russian</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sanity Checking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Korean': 30,\n",
       "         'Italian': 30,\n",
       "         'Polish': 30,\n",
       "         'Japanese': 30,\n",
       "         'Dutch': 30,\n",
       "         'Czech': 30,\n",
       "         'Irish': 30,\n",
       "         'Chinese': 30,\n",
       "         'Vietnamese': 30,\n",
       "         'Spanish': 30,\n",
       "         'Arabic': 30,\n",
       "         'German': 30,\n",
       "         'English': 30,\n",
       "         'Russian': 30,\n",
       "         'Greek': 30,\n",
       "         'French': 30})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(_.obj for _ in data.valid_ds.y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Russian', 9232),\n",
       " ('English', 3329),\n",
       " ('Japanese', 952),\n",
       " ('Italian', 630),\n",
       " ('German', 548),\n",
       " ('Czech', 434),\n",
       " ('Dutch', 214),\n",
       " ('Spanish', 182),\n",
       " ('French', 180),\n",
       " ('Chinese', 170),\n",
       " ('Greek', 162),\n",
       " ('Irish', 134),\n",
       " ('Polish', 93),\n",
       " ('Arabic', 73),\n",
       " ('Korean', 31),\n",
       " ('Vietnamese', 25)]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(_.obj for _ in data.train_ds.y).most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check no text is both in Validation and Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_set = set(_.text for _ in data.valid_ds.x)\n",
    "for _ in data.train_ds.x:\n",
    "    assert _.text not in valid_set, _.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examine a minibatch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainiter = iter(data.train_dl)\n",
    "batch, cl = next(trainiter)\n",
    "batch2, cl2 = next(trainiter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 6, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13],\n",
       "        device='cuda:0'), 16)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cl, len(cl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([20, 16])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first 22 letters run *down* the batch backpadded by `BOS`; we have 16 names across.\n",
    "\n",
    "Somehow it looks like we also have an extra space at the beginning of each name that wasn't in the input data.\n",
    "\n",
    "(Note this is different to what the fastai wrappers will give you; they concatenate the data and split it into 16 chunks)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>v</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>o</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>n</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>g</td>\n",
       "      <td>t</td>\n",
       "      <td>l</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>r</td>\n",
       "      <td>c</td>\n",
       "      <td>e</td>\n",
       "      <td>t</td>\n",
       "      <td>m</td>\n",
       "      <td>b</td>\n",
       "      <td>c</td>\n",
       "      <td>z</td>\n",
       "      <td>b</td>\n",
       "      <td>p</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>i</td>\n",
       "      <td>h</td>\n",
       "      <td>i</td>\n",
       "      <td>c</td>\n",
       "      <td>i</td>\n",
       "      <td>a</td>\n",
       "      <td>h</td>\n",
       "      <td>h</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>g</td>\n",
       "      <td>s</td>\n",
       "      <td>a</td>\n",
       "      <td>b</td>\n",
       "      <td>v</td>\n",
       "      <td>v</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>m</td>\n",
       "      <td>a</td>\n",
       "      <td>h</td>\n",
       "      <td>h</td>\n",
       "      <td>n</td>\n",
       "      <td>k</td>\n",
       "      <td>a</td>\n",
       "      <td>e</td>\n",
       "      <td>h</td>\n",
       "      <td>t</td>\n",
       "      <td>r</td>\n",
       "      <td>h</td>\n",
       "      <td>w</td>\n",
       "      <td>a</td>\n",
       "      <td>y</td>\n",
       "      <td>i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>e</td>\n",
       "      <td>t</td>\n",
       "      <td>e</td>\n",
       "      <td>h</td>\n",
       "      <td>a</td>\n",
       "      <td>h</td>\n",
       "      <td>t</td>\n",
       "      <td>o</td>\n",
       "      <td>h</td>\n",
       "      <td>i</td>\n",
       "      <td>s</td>\n",
       "      <td>n</td>\n",
       "      <td>o</td>\n",
       "      <td>h</td>\n",
       "      <td>c</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>l</td>\n",
       "      <td>o</td>\n",
       "      <td>n</td>\n",
       "      <td>l</td>\n",
       "      <td>z</td>\n",
       "      <td>t</td>\n",
       "      <td>o</td>\n",
       "      <td>k</td>\n",
       "      <td>i</td>\n",
       "      <td>o</td>\n",
       "      <td>h</td>\n",
       "      <td>d</td>\n",
       "      <td>r</td>\n",
       "      <td>t</td>\n",
       "      <td>h</td>\n",
       "      <td>h</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>s</td>\n",
       "      <td>r</td>\n",
       "      <td>b</td>\n",
       "      <td>a</td>\n",
       "      <td>e</td>\n",
       "      <td>a</td>\n",
       "      <td>r</td>\n",
       "      <td>h</td>\n",
       "      <td>v</td>\n",
       "      <td>r</td>\n",
       "      <td>e</td>\n",
       "      <td>e</td>\n",
       "      <td>k</td>\n",
       "      <td>i</td>\n",
       "      <td>e</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>h</td>\n",
       "      <td>i</td>\n",
       "      <td>e</td>\n",
       "      <td>k</td>\n",
       "      <td>t</td>\n",
       "      <td>n</td>\n",
       "      <td>i</td>\n",
       "      <td>o</td>\n",
       "      <td>a</td>\n",
       "      <td>k</td>\n",
       "      <td>l</td>\n",
       "      <td>r</td>\n",
       "      <td>h</td>\n",
       "      <td>g</td>\n",
       "      <td>s</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>a</td>\n",
       "      <td>z</td>\n",
       "      <td>r</td>\n",
       "      <td>o</td>\n",
       "      <td>d</td>\n",
       "      <td>o</td>\n",
       "      <td>z</td>\n",
       "      <td>v</td>\n",
       "      <td>n</td>\n",
       "      <td>o</td>\n",
       "      <td>e</td>\n",
       "      <td>o</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>l</td>\n",
       "      <td>o</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>u</td>\n",
       "      <td>h</td>\n",
       "      <td>g</td>\n",
       "      <td>v</td>\n",
       "      <td>i</td>\n",
       "      <td>w</td>\n",
       "      <td>h</td>\n",
       "      <td>t</td>\n",
       "      <td>d</td>\n",
       "      <td>v</td>\n",
       "      <td>v</td>\n",
       "      <td>v</td>\n",
       "      <td>n</td>\n",
       "      <td>r</td>\n",
       "      <td>a</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>s</td>\n",
       "      <td>s</td>\n",
       "      <td>s</td>\n",
       "      <td>s</td>\n",
       "      <td>n</td>\n",
       "      <td>s</td>\n",
       "      <td>s</td>\n",
       "      <td>s</td>\n",
       "      <td>z</td>\n",
       "      <td>s</td>\n",
       "      <td>s</td>\n",
       "      <td>i</td>\n",
       "      <td>o</td>\n",
       "      <td>e</td>\n",
       "      <td>v</td>\n",
       "      <td>s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>e</td>\n",
       "      <td>k</td>\n",
       "      <td>k</td>\n",
       "      <td>k</td>\n",
       "      <td>o</td>\n",
       "      <td>k</td>\n",
       "      <td>k</td>\n",
       "      <td>e</td>\n",
       "      <td>h</td>\n",
       "      <td>k</td>\n",
       "      <td>k</td>\n",
       "      <td>c</td>\n",
       "      <td>f</td>\n",
       "      <td>e</td>\n",
       "      <td>o</td>\n",
       "      <td>k</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>n</td>\n",
       "      <td>y</td>\n",
       "      <td>y</td>\n",
       "      <td>y</td>\n",
       "      <td>v</td>\n",
       "      <td>i</td>\n",
       "      <td>y</td>\n",
       "      <td>v</td>\n",
       "      <td>i</td>\n",
       "      <td>y</td>\n",
       "      <td>y</td>\n",
       "      <td>h</td>\n",
       "      <td>f</td>\n",
       "      <td>v</td>\n",
       "      <td>v</td>\n",
       "      <td>y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category</th>\n",
       "      <td>German</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Russian</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              0        1        2        3        4        5        6   \\\n",
       "0          xxbos    xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "1                   xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "2              v    xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "3              o    xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "4              n    xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "5                                     xxbos    xxbos    xxbos    xxbos   \n",
       "6              g        t        l                                       \n",
       "7              r        c        e        t        m        b        c   \n",
       "8              i        h        i        c        i        a        h   \n",
       "9              m        a        h        h        n        k        a   \n",
       "...          ...      ...      ...      ...      ...      ...      ...   \n",
       "11             e        t        e        h        a        h        t   \n",
       "12             l        o        n        l        z        t        o   \n",
       "13             s        r        b        a        e        a        r   \n",
       "14             h        i        e        k        t        n        i   \n",
       "15             a        z        r        o        d        o        z   \n",
       "16             u        h        g        v        i        w        h   \n",
       "17             s        s        s        s        n        s        s   \n",
       "18             e        k        k        k        o        k        k   \n",
       "19             n        y        y        y        v        i        y   \n",
       "category  German  Russian  Russian  Russian  Russian  Russian  Russian   \n",
       "\n",
       "               7        8        9        10       11       12       13  \\\n",
       "0           xxbos    xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "1           xxbos    xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "2           xxbos    xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "3           xxbos    xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "4           xxbos    xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "5           xxbos    xxbos    xxbos    xxbos    xxbos    xxbos    xxbos   \n",
       "6                                      xxbos    xxbos    xxbos    xxbos   \n",
       "7               z        b        p                                       \n",
       "8               h        a        a        g        s        a        b   \n",
       "9               e        h        t        r        h        w        a   \n",
       "...           ...      ...      ...      ...      ...      ...      ...   \n",
       "11              o        h        i        s        n        o        h   \n",
       "12              k        i        o        h        d        r        t   \n",
       "13              h        v        r        e        e        k        i   \n",
       "14              o        a        k        l        r        h        g   \n",
       "15              v        n        o        e        o        a        a   \n",
       "16              t        d        v        v        v        n        r   \n",
       "17              s        z        s        s        i        o        e   \n",
       "18              e        h        k        k        c        f        e   \n",
       "19              v        i        y        y        h        f        v   \n",
       "category  Russian  Russian  Russian  Russian  Russian  Russian  Russian   \n",
       "\n",
       "               14       15  \n",
       "0           xxbos    xxbos  \n",
       "1           xxbos    xxbos  \n",
       "2           xxbos    xxbos  \n",
       "3           xxbos    xxbos  \n",
       "4           xxbos    xxbos  \n",
       "5           xxbos    xxbos  \n",
       "6           xxbos    xxbos  \n",
       "7                           \n",
       "8               v        v  \n",
       "9               y        i  \n",
       "...           ...      ...  \n",
       "11              c        c  \n",
       "12              h        h  \n",
       "13              e        e  \n",
       "14              s        p  \n",
       "15              l        o  \n",
       "16              a        l  \n",
       "17              v        s  \n",
       "18              o        k  \n",
       "19              v        y  \n",
       "category  Russian  Russian  \n",
       "\n",
       "[21 rows x 16 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.options.display.max_columns = 100\n",
    "(pd\n",
    " .DataFrame([[vocab.itos[y] for y in x] for x in batch])\n",
    " .T\n",
    " .assign(category=[data.classes[_] for _ in cl])\n",
    " .T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxbos', ' ', 'a', 'h', 'n']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[vocab.itos[_] for _ in data.train_ds[0][0].data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A', 'h', 'n']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(df.iloc[0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the length of strings varies between batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxbos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b</td>\n",
       "      <td>m</td>\n",
       "      <td>r</td>\n",
       "      <td>f</td>\n",
       "      <td>m</td>\n",
       "      <td>b</td>\n",
       "      <td>d</td>\n",
       "      <td>j</td>\n",
       "      <td>m</td>\n",
       "      <td>b</td>\n",
       "      <td>k</td>\n",
       "      <td>u</td>\n",
       "      <td>m</td>\n",
       "      <td>m</td>\n",
       "      <td>a</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>e</td>\n",
       "      <td>c</td>\n",
       "      <td>i</td>\n",
       "      <td>e</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>e</td>\n",
       "      <td>e</td>\n",
       "      <td>o</td>\n",
       "      <td>a</td>\n",
       "      <td>i</td>\n",
       "      <td>f</td>\n",
       "      <td>a</td>\n",
       "      <td>e</td>\n",
       "      <td>n</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>n</td>\n",
       "      <td>g</td>\n",
       "      <td>d</td>\n",
       "      <td>r</td>\n",
       "      <td>s</td>\n",
       "      <td>j</td>\n",
       "      <td>m</td>\n",
       "      <td>f</td>\n",
       "      <td>r</td>\n",
       "      <td>l</td>\n",
       "      <td>n</td>\n",
       "      <td>i</td>\n",
       "      <td>k</td>\n",
       "      <td>a</td>\n",
       "      <td>s</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>i</td>\n",
       "      <td>r</td>\n",
       "      <td>g</td>\n",
       "      <td>r</td>\n",
       "      <td>a</td>\n",
       "      <td>e</td>\n",
       "      <td>a</td>\n",
       "      <td>f</td>\n",
       "      <td>a</td>\n",
       "      <td>b</td>\n",
       "      <td>c</td>\n",
       "      <td>m</td>\n",
       "      <td>i</td>\n",
       "      <td>d</td>\n",
       "      <td>e</td>\n",
       "      <td>u</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>t</td>\n",
       "      <td>o</td>\n",
       "      <td>w</td>\n",
       "      <td>a</td>\n",
       "      <td>o</td>\n",
       "      <td>n</td>\n",
       "      <td>k</td>\n",
       "      <td>e</td>\n",
       "      <td>n</td>\n",
       "      <td>o</td>\n",
       "      <td>h</td>\n",
       "      <td>k</td>\n",
       "      <td>o</td>\n",
       "      <td>h</td>\n",
       "      <td>l</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>e</td>\n",
       "      <td>r</td>\n",
       "      <td>a</td>\n",
       "      <td>r</td>\n",
       "      <td>k</td>\n",
       "      <td>o</td>\n",
       "      <td>i</td>\n",
       "      <td>r</td>\n",
       "      <td>d</td>\n",
       "      <td>n</td>\n",
       "      <td>i</td>\n",
       "      <td>i</td>\n",
       "      <td>k</td>\n",
       "      <td>r</td>\n",
       "      <td>m</td>\n",
       "      <td>i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>z</td>\n",
       "      <td>y</td>\n",
       "      <td>y</td>\n",
       "      <td>o</td>\n",
       "      <td>a</td>\n",
       "      <td>v</td>\n",
       "      <td>s</td>\n",
       "      <td>s</td>\n",
       "      <td>i</td>\n",
       "      <td>i</td>\n",
       "      <td>n</td>\n",
       "      <td>n</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>i</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category</th>\n",
       "      <td>Spanish</td>\n",
       "      <td>English</td>\n",
       "      <td>English</td>\n",
       "      <td>Italian</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Greek</td>\n",
       "      <td>English</td>\n",
       "      <td>Italian</td>\n",
       "      <td>Italian</td>\n",
       "      <td>English</td>\n",
       "      <td>Russian</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>Irish</td>\n",
       "      <td>Italian</td>\n",
       "      <td>Russian</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               0        1        2        3         4        5      6   \\\n",
       "0           xxbos    xxbos    xxbos    xxbos     xxbos    xxbos  xxbos   \n",
       "1                                                                        \n",
       "2               b        m        r        f         m        b      d   \n",
       "3               e        c        i        e         a        a      e   \n",
       "4               n        g        d        r         s        j      m   \n",
       "5               i        r        g        r         a        e      a   \n",
       "6               t        o        w        a         o        n      k   \n",
       "7               e        r        a        r         k        o      i   \n",
       "8               z        y        y        o         a        v      s   \n",
       "category  Spanish  English  English  Italian  Japanese  Russian  Greek   \n",
       "\n",
       "               7        8        9        10       11        12     13  \\\n",
       "0           xxbos    xxbos    xxbos    xxbos    xxbos     xxbos  xxbos   \n",
       "1                                                                        \n",
       "2               j        m        b        k        u         m      m   \n",
       "3               e        o        a        i        f         a      e   \n",
       "4               f        r        l        n        i         k      a   \n",
       "5               f        a        b        c        m         i      d   \n",
       "6               e        n        o        h        k         o      h   \n",
       "7               r        d        n        i        i         k      r   \n",
       "8               s        i        i        n        n         a      a   \n",
       "category  English  Italian  Italian  English  Russian  Japanese  Irish   \n",
       "\n",
       "               14       15  \n",
       "0           xxbos    xxbos  \n",
       "1                           \n",
       "2               a        b  \n",
       "3               n        a  \n",
       "4               s        b  \n",
       "5               e        u  \n",
       "6               l        r  \n",
       "7               m        i  \n",
       "8               i        n  \n",
       "category  Italian  Russian  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pd\n",
    " .DataFrame([[vocab.itos[y] for y in x] for x in batch2])\n",
    " .T\n",
    " .assign(category=[data.classes[_] for _ in cl2])\n",
    " .T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxbos   b e n i t e z'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.textify(batch2[:,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='90%'>  <col width='10%'>  <tr>\n",
       "    <th>text</th>\n",
       "    <th>target</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  c h r y s a n t h o p o u l o s</th>\n",
       "    <th>Greek</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  v o n   i n g e r s l e b e n</th>\n",
       "    <th>German</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  s c h w a r z e n b e r g</th>\n",
       "    <th>Dutch</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  d e   s a u v e t e r r e</th>\n",
       "    <th>French</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>  a r e c h a v a l e t a</th>\n",
       "    <th>Spanish</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch(ds_type=DatasetType.Valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One Hot Encoding\n",
    "\n",
    "The torch nn.RNN expects the data to be one hot encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_hot = torch.eye(len(vocab.itos))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_hot[batch][:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([20, 16, 31])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_hot[batch].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how we could do it without storing the one_hot matrix in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def one_hot_fly(y, length=len(vocab.itos)):\n",
    "    length = len(vocab.itos)\n",
    "    shape = list(y.shape)\n",
    "    assert len(shape) == 2\n",
    "    tensor = torch.zeros(shape + [length])\n",
    "    for i,row in enumerate(y):\n",
    "        for j, val in enumerate(row):\n",
    "            tensor[i][j][val] = 1.\n",
    "    return tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1, dtype=torch.uint8)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(one_hot[batch] == one_hot_fly(batch)).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using matrix operations is ~250 times faster at this size than the double for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "36.1 µs ± 2.61 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n",
      "8.91 ms ± 210 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit one_hot[batch]\n",
    "%timeit one_hot_fly(batch)\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fitting a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(31, 16)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_letters = len(vocab.itos)\n",
    "n_hidden = 128\n",
    "n_output = df.cl.nunique()\n",
    "n_letters, n_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use an RNN to take our sequence of letters in and calculate the hidden state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn = nn.RNN(input_size=n_letters,\n",
    "             hidden_size=n_hidden,\n",
    "             num_layers=1,\n",
    "             nonlinearity='relu',\n",
    "             dropout=0.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([20, 16, 128]), torch.Size([1, 16, 128]))"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output, hidden = rnn(one_hot[batch])\n",
    "output.shape, hidden.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "lo = nn.Linear(n_hidden, n_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = lo(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([20, 16, 16])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 6, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nn.functional.softmax(preds[-1], dim=1).argmax(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_hot = torch.eye(len(vocab.itos))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyLetterRNN(nn.Module):\n",
    "    def __init__(self, dropout=0., n_layers=1, n_input=n_letters, n_hidden=n_hidden, n_output=n_output):\n",
    "        super().__init__()\n",
    "        self.one_hot = torch.eye(n_letters).cuda()\n",
    "        self.rnn = nn.RNN(input_size=n_letters,\n",
    "                         hidden_size=n_hidden,\n",
    "                         num_layers=n_layers,\n",
    "                         nonlinearity='relu',\n",
    "                         dropout=dropout)\n",
    "        self.lo = nn.Linear(n_hidden, n_output)\n",
    "        \n",
    "    def forward(self, input):\n",
    "        rnn, _ = self.rnn(self.one_hot[input])\n",
    "        out = self.lo(rnn)\n",
    "        return out[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn = MyLetterRNN().cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([6, 6, 6, 6, 1, 6, 6, 1, 6, 6, 6, 1, 6, 1, 1, 6], device='cuda:0'),\n",
       " tensor([ 6, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13],\n",
       "        device='cuda:0'))"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = rnn(batch)\n",
    "out.argmax(dim=1), cl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.7832, device='cuda:0', grad_fn=<NllLossBackward>)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "F.cross_entropy(out, cl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, rnn, loss_func=F.cross_entropy, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XecVPW9//HXZ2ZnewF2lyLSOxZAVwW7WGI0saSaxBJLDImNxOTml+T+cnPjL7maoineFCJ21CSWJOqN5VqCaAQBAVGKCAgKUgUWlmV3dj6/P2ZY13UbsGfO7sz7+Xicx86e+Z45ny+zzHu+p5q7IyIiAhAJuwAREek6FAoiItJIoSAiIo0UCiIi0kihICIijRQKIiLSSKEgIiKNFAoiItJIoSAiIo1ywi5gX1VUVPjgwYPDLkNEpFuZN2/eZnevbK9dtwuFwYMHM3fu3LDLEBHpVszs7Y600+YjERFppFAQEZFGCgUREWmkUBARkUaBhYKZDTCz58xsiZm9bmbXtdCmzMweNbOFqTaXBlWPiIi0L8ijj+LA9e4+38xKgHlm9rS7v9GkzVXAG+7+STOrBJaZ2Qx3rwuwLhERaUVgIwV3X+/u81OPq4ElQP/mzYASMzOgGNhKMkxERCQEaTlPwcwGAxOA2c2euhX4O7AOKAE+7+6JdNTUUYmEUxtvYHddA7XxBLvr4lTXxtm5J/WzNk594sMluyfTDncciJjRuySPfmUF9C3Lp7wol0jEWlxfbX0D22rqqamLEzEjGklOZrCzNs623fVsr6ln++56auobqIsnGqeGRIJoJEI0AtFIhFjU6FWUS3lxHuVFuZQX55KXEyViEIkYETMMMAMjuY5oxIhFtatJJFsFHgpmVgw8BEx19x3Nnv4YsACYDAwDnjazF5q3M7MrgSsBBg4cuF91vLRiM7f873L2pD5AP/yzgbqG5O+JNNyyOhY18mNRYtEIOakP4bqGBNt311MXDz8TywpilBfnUlGcR2VxHr2Kcj80FcSiRCI0hpY7yWBqSLAn3kBDAkrzc+hZlEvPwhhlBbnkRIyEOwkHdyc3J0JJfoxoK+G4VyLhbN9dz/s1dThQmBulIBalIDdKbjRCcpApIp0l0FAwsxjJQJjh7g+30ORS4EZ3d2CFma0CRgNzmjZy92nANICqqqr9+tiOpD58i/NyyM2JkJcTJTcnkpyiH/xs/g3eSH0Q5UbJz4mSnxulJC+H4vwcSvJzKMrNIS+nhW/WTb59xxucjdW1rN9ey3vba3lvRy276xqob0gQb3DqEwlikQg9CmOUFcYoK4hRlJtDwp2GhDd+mBbn5VBWEGucCvOi5EU/6EfEIOHQkEguVxdPsLWmjq279rB5Zx1bd9Wlgi/5vDskUqMZT/2r1sUTbNm1h80797C5uo4l63ewtaaObTX1+/PP3iYzKMnLoaww2V+gsa8NqTDYVlPXalBHI0ZhKiAKc6Pkx6LkRC05WjLIiUTIi0UoiEUb38PKknxG9C5mRJ9ihlQUkZcT7fR+iXRngYVCaj/BdGCJu9/cSrM1wKnAC2bWBxgFrAyinolDy5k4tDyIl+6QvmX5HH5w8OuJpjYBARTkRikrjDGkouiAXzfekGDb7nq27qpjT32ChsZgSX5i7w2mvZunduyO835NXWpzVx0NCScSMSy1yWpPPDky2rE7uSls5554crOWJTdrRSJGWUEOPQtzk1NRjIgZNXUN1NQ1UFvfQE1dnJq65Ka9XamfCXfiCSeRcOKJBDv3xNlUvYfd9cnltuzc0xgy0YjRryyfWDTSuO69m89yosmfudEIBblRSvJzKM2PUZqfQ2lBjJ6FyVFTj8IY5UV59C3LJ7elLwci3UyQI4XjgIuA18xsQWre94CBAO7+e+AG4E4ze43kl/LvuPvmAGuS/ZQTjVBRnEdFcV7YpRyQ2voGVm3exfIN1by5YSfvbtudDBFPBklDIhkq9anNibvq4mzeuYfq2jg7apPh5S2MXCIG/coKGNirkAG9CigriJEfi5KXEyE/FqW8OJeRfUoYVllMfkyjE+m6AgsFd59F8oO+rTbrgDOCqkGkufxYlDH9ShnTr3S/lk8knOra5Cho77S5uo533q9hzdbk9NyyTeysjVMbb/hIgEQMBpcXMaZfKROH9uLY4RUMrSjSvhHpMrrdVVJFwhSJWHK/T2GMwbS9Wc7dqW9IHr323vZalm+oZvl71SzfsJNX17zP46+tB6BvaT7HDivn+BEVHD+8gt6l+enoikiLFAoiATEzcnOM3JwIpfkxRvYpgcOTz7k7b2+p4aW3tvDiW5t5fvkmHn71XQBG9Snh+BEVnHloX6oG9dQoQtLKvKUNpF1YVVWV634KkmkSCeeN9TuYtWIzs97czJzVW6mLJxjVp4QLJw7kvAn9KcmPhV2mdGNmNs/dq9ptp1AQ6Xpq6uI8unAd97z8Novf3UFRbpQzD+3H0UN6cuSgngytKG71BEiRligURDKAu7Pwne3c+/LbPP3GBrbvTp4vUlYQY8LAHowf0INxA3ow7uAe9CrKDbla6co6GgrapyDShZkZ4wckP/wTCWfl5l3MX/M+r655n3lvv88/l29qPMJpYK9CrjxxKF86ZqD2Q8h+UyiIdBORiDG8dzHDexfzuaoBAOzcE2fxu9tZuHYbzyzZyL//dTGvrN7KT84/jKI8/feWfafNRyIZIpFwfvv8Cm5+ejnDKov53YVHMLx3SdhlSRfR0c1HOi9fJENEIsbVk0dwz+XHsHVXHefc+iJPLH4v7LKkm1EoiGSY44ZX8Ni1xzOiTwnXPvAqS99rfnFikdYpFEQyUL+yAqZfUkVpfoxr73+V3XUNYZck3YRCQSRDVRTnccvnx7F8w07+3+NvtL+ACAoFkYx2wohKvnrSUGbMXsMTi9eHXY50AwoFkQx3/emjOPzgMr7z0Gus27Y77HKki1MoiGS43JwIv75gAvGGBFMfWEB9Q/i3fJWuS6EgkgUGVxTxk08dxpzVW/nOg4vobucnSfrolEeRLHHu+P68vaWGm59eTmVpHt/9+JiwS5IuSKEgkkWumTycDTtq+cM/V9K7JJ/Ljx8SdknSxSgURLKImfGjcw9ly846bnjsDSpL8jhn3EFhlyVdiPYpiGSZaMT45QXjOXpwL67/8wIWrt0WdknShSgURLJQfizKHy+uoiQ/xq+feTPscqQLUSiIZKmywhiXTBrMM0s3snxDddjlSBehUBDJYhdPGkRBLMq0mSvDLkW6CIWCSBbrWZTL548awN8WvMv67TrbWQIMBTMbYGbPmdkSM3vdzK5roc23zWxBalpsZg1m1iuomkTkoy4/fggJhzteXB12KdIFBDlSiAPXu/sYYCJwlZmNbdrA3X/m7uPdfTzwXeCf7r41wJpEpJkBvQo5+7B+3Dd7Ddt314ddjoQssFBw9/XuPj/1uBpYAvRvY5EvAPcHVY+ItO7KE4eyc0+c+2avCbsUCVla9imY2WBgAjC7lecLgTOBh1p5/kozm2tmczdt2hRUmSJZ69D+ZZwwooLbX1zFnrhuyJPNAg8FMysm+WE/1d1buy/gJ4EXW9t05O7T3L3K3asqKyuDKlUkq331xGFsqt7DI/PfDbsUCVGgoWBmMZKBMMPdH26j6QVo05FIqI4bXs7oviXc/8rasEuREAV59JEB04El7n5zG+3KgJOAvwVVi4i0z8z41BH9Wbh2G6s37wq7HAlJkCOF44CLgMlNDjs9y8ymmNmUJu3OB55yd/0VioTsk+MOwgweXbgu7FIkJIFdJdXdZwHWgXZ3AncGVYeIdFy/sgKOGtyLvy54l6snDyc54JdsojOaReRDzh1/EG9t2sUb61s7LkQymUJBRD7krEP7kRMx/q5NSFlJoSAiH9KzKJcTR1by6IJ1JBK6l3O2USiIyEecO/4g1m2vZd6a98MuRdJMoSAiH3HamD7kxyL8bYFOZMs2CgUR+YiivBxOH9uXxxetp74hEXY5kkYKBRFp0bnjDuL9mnpmrdgcdimSRgoFEWnRiSMrKSuI8fcFOgopmygURKRFuTkRzjqsL0+9/h619bpyarZQKIhIq84+7CB21TXw/DJdsj5bKBREpFUTh/aiV1Euj7+2PuxSJE0UCiLSqpxohDMP7cszSzZoE1KWUCiISJvOPqwfNXUNPL9sY9ilSBooFESkTccM6UV5US6PLdImpGygUBCRNu3dhPTs0o3srtMmpEynUBCRdmkTUvZQKIhIu44e0ouK4lwe01FIGU+hICLtatyEtESbkDKdQkFEOuSsw/qxu76B57QJKaMpFESkQ44ZUk5FcS6P6yikjKZQEJEOiUaMjx/aj2eWbmDXnnjY5UhAFAoi0mHnTehPbX2CGx57A3fdqjMTKRREpMOOHNSTq04ZxgOvrGXG7DVhlyMBCCwUzGyAmT1nZkvM7HUzu66Vdieb2YJUm38GVY+IdI5vnj6Kk0dV8p+Pvs7c1VvDLkc6WZAjhThwvbuPASYCV5nZ2KYNzKwH8FvgHHc/BPhsgPWISCeIRoxfXTCB/j0KmHLvfN7bXht2SdKJAgsFd1/v7vNTj6uBJUD/Zs2+CDzs7mtS7XSsm0g3UFYQY9rFVeyuizPl3nnsievchUyRln0KZjYYmADMbvbUSKCnmT1vZvPM7OJ01CMiB25knxJ+8bnxLFi7jV88tTzscqSTBB4KZlYMPARMdfcdzZ7OAY4EzgY+BvxfMxvZwmtcaWZzzWzupk26A5RIV3HmoX254KgB3D5rFSs27gy7HOkEgYaCmcVIBsIMd3+4hSbvAE+4+y533wzMBMY1b+Tu09y9yt2rKisrgyxZRPbRtz42ioLcKD/SYaoZIcijjwyYDixx95tbafY34AQzyzGzQuAYkvseRKSbqCjOY+ppI5m5fBPPLNFuwe4uyJHCccBFwOTUIacLzOwsM5tiZlMA3H0J8ASwCJgD3ObuiwOsSUQCcPGkQQzvXcyPHntDt+3s5nKCemF3nwVYB9r9DPhZUHWISPBi0Qj/8cmxXDR9DtNnreKqU4aHXZLsJ53RLCKd4oQRlXzskD7c+uwK1m/fHXY5sp8UCiLSaf797LE0uHPTP5aGXYrsJ4WCiHSaAb0KueL4Ifx1wToWv7s97HJkPygURKRTTTl5GD0KY9z0hEYL3ZFCQUQ6VWl+jKtPGc4Lb25m1pubwy5H9pFCQUQ63UWTBtG/RwE3PbGUREIntHUnCgUR6XR5OVG+efpIXnt3O4+/ptt3dicKBREJxHkT+jO6bwk/f2oZ9Q2JsMuRDlIoiEggohHjO2eO5u0tNTwwR3dp6y4UCiISmJNHVXLMkF786pk3qamLh12OdIBCQUQCY2b825mj2byzjrteejvscqQDFAoiEqgjB/Xk5FGV/GHmW1TX1oddjrRDoSAigfvGaSPZVlPPXS+tDrsUaYdCQUQCN25AD04b05tpM1eyQ6OFLk2hICJpMfW0keyojXP7rFVhlyJtUCiISFoc2r+MMw/py/QXVrG9RqOFrkqhICJpM/X0EVTviXPbrJVhlyKtUCiISNqM7lvK2Yf34/ZZq3h/V13Y5UgLFAoiklZTTx1BTX0Df3xBo4WuSKEgImk1ok8JZx3Wj7v/9TbbajRa6Go6FApmNszM8lKPTzaza82sR7CliUimumbycHbuiXPHi6vDLkWa6ehI4SGgwcyGA9OBIcB9gVUlIhltdN9SzhjbhzteXKXzFrqYjoZCwt3jwPnAL939G0C/4MoSkUx3zeQR7KiNc7fOcu5SOhoK9Wb2BeAS4LHUvFgwJYlINjjs4DImj+7N9Fmr2LVHV1DtKjoaCpcCk4Afu/sqMxsC3NvWAmY2wMyeM7MlZva6mV3XQpuTzWy7mS1ITT/Y9y6ISHd1zeThvF9Tz70v6wqqXUVORxq5+xvAtQBm1hMocfcb21ksDlzv7vPNrASYZ2ZPp16rqRfc/RP7WriIdH8TBvbkhBEV/PGFlVw8aTAFudGwS8p6HT366HkzKzWzXsBC4A4zu7mtZdx9vbvPTz2uBpYA/Q+0YBHJLNdMHsHmnXXMmK3RQlfQ0c1HZe6+A/gUcIe7Hwmc1tGVmNlgYAIwu4WnJ5nZQjP7h5kd0tHXFJHMcPSQXpwwooJbnl7Omi01YZeT9ToaCjlm1g/4HB/saO4QMysmeUjr1FSwNDUfGOTu44DfAH9t5TWuNLO5ZjZ306ZN+7J6EekGbvz04UTM+OafF9CQ8LDLyWodDYUfAU8Cb7n7K2Y2FHizvYXMLEYyEGa4+8PNn3f3He6+M/X4f4CYmVW00G6au1e5e1VlZWUHSxaR7qJ/jwJ+dN4hzH37ff4w862wy8lqHQoFd/+Lux/u7l9L/b7S3T/d1jJmZiRPdFvi7i3ufzCzvql2mNnRqXq27EsHRCQznDe+P2cd1pdbnl7O6+u2h11O1urojuaDzewRM9toZhvM7CEzO7idxY4DLgImNznk9Cwzm2JmU1JtPgMsNrOFwK+BC9xdY0eRLGRm/Pi8w+hRmMs3/7SQ2vqGsEvKStaRz2Aze5rkZS3uSc26EPiSu58eYG0tqqqq8rlz56Z7tSKSJs8t28ild7zCV04YwvfPHht2ORnDzOa5e1V77Tq6T6HS3e9w93hquhPQxn0R6XSnjOrNBUcN4PYXV7NxR23Y5WSdjobCZjO70MyiqelCtO1fRALy1ZOG0ZBw/jx3bdilZJ2OhsJlJA9HfQ9YT3JfwKVBFSUi2W1IRRHHDivn/jlrdYhqmnX06KM17n6Ou1e6e293P4/kiWwiIoH44jEDeXfbbma+qXOT0ulA7rz2zU6rQkSkmTPG9qW8KJf7Z68Ju5SsciChYJ1WhYhIM7k5ET5TdTDPLN3IBu1wTpsDCQVt6BORQH3hqIE0JJw/vaIdzunSZiiYWbWZ7WhhqgYOSlONIpKlBlcUcfzwCv70inY4p0uboeDuJe5e2sJU4u4duheDiMiBaNzhvFw7nNPhQDYfiYgE7vSxfagozmOGdjinhUJBRLq0WDTCZ6sO5tmlG1i7VfdbCJpCQUS6vIsmDqIgFuXbDy4koX0LgVIoiEiXd1CPAv7jk4fw8sqt3DZrZdjlZDSFgoh0C5+tOpgzxvbh508uZ8n65jdxlM6iUBCRbsHMuPHTh1NWGGPqAwt0v4WAKBREpNvoVZTLTz9zOMs2VPPzJ5eFXU5GUiiISLdyyqjeXDRxELfNWsUTi98Lu5yMo1AQkW7ne2eN4dD+pUy5dx4/f3IZ8YZE2CVlDIWCiHQ7BblR/vLVY/l81QBufW4FF06frbu0dRKFgoh0SwW5UW76zOH84rPjWLh2O2f9ehb/eks3hDxQCgUR6dY+feTB/P3q4+hRGOOSO+Yw683NYZfUrSkURKTbG9GnhAenTGJoRRFfuXsur6zeGnZJ3ZZCQUQyQo/CXO65/Bj69cjn0jteYeHabWGX1C0pFEQkY1SW5HHfFRPpVZTLxbfP4Y11OvN5XwUWCmY2wMyeM7MlZva6mV3XRtujzKzBzD4TVD0ikh36luUz44pjKMyNctH02bqy6j4KcqQQB6539zHAROAqMxvbvJGZRYGbgCcDrEVEssiAXoXMuOIY4gnn8rteobq2PuySuo3AQsHd17v7/NTjamAJ0L+FptcADwEbg6pFRLLP0MpifvelI1i5aRfXPbBAt/PsoLTsUzCzwcAEYHaz+f2B84Hft7P8lWY218zmbtqkW/KJSMccO7yCH55zCM8u3ciN/1gSdjndQuChYGbFJEcCU929+V6fXwLfcfc2L3fo7tPcvcrdqyorK4MqVUQy0IUTB3HJpEH88YVV/Hnu2rDL6fJygnxxM4uRDIQZ7v5wC02qgAfMDKACOMvM4u7+1yDrEpHs8n8/MZaVm3fx/UdeY0DPQiYNKw+7pC4ryKOPDJgOLHH3m1tq4+5D3H2wuw8GHgS+rkAQkc6WE41w6xePYEhFEVfcpXMY2hLk5qPjgIuAyWa2IDWdZWZTzGxKgOsVEfmIsoIY91x+DL2Kc7nkjjks31Addkldkrl3rz3yVVVVPnfu3LDLEJFuas2WGj7z+5cAeHDKsQwsLwy5ovQws3nuXtVeO53RLCJZZWB5Ifdcfgx1DQkunD6bDbrk9ocoFEQk64zqW8Kdlx7N5p17+M9HXw+7nC5FoSAiWWn8gB586oj+PL9sE3vibR4Vn1UUCiKStU4d3YeaugZmr9SltvdSKIhI1po0rJz8WIRnl+oqO3spFEQka+XHohw/vIJnlm6gux2JGRSFgohktVNG92bt1t2s2Lgz7FK6BIWCiGS1yaN7A/CMNiEBCgURyXL9ygoY26+UZ5d8NBTcneUbqrNq05JCQUSy3qljejP37a1sq6n70Py7XlrNGbfM5Kr75rNrTzyk6tJLoSAiWW/y6N4kHP65/IP7tazZUsNNTyxjaGURTyx+j/N/+yKrN+8Kscr0UCiISNYbd3APyotyeSa1Ccnd+c5Di8iJGDOuOIa7LzuGjdV7OOfWWTy3LLP3PSgURCTrRSLGKaN78/yyjcQbEtw3Zw3/WrmF7509hn5lBRw/ooJHrz6e/j0LuezOV7j7X6vDLjkwCgUREeDU0b3ZURvn0UXr+K//Wcpxw8u54KgBjc8P6FXIw187llNH9+EHf3ud++esCbHa4CgURESA40dUEIsa//bgIhLu3Pipw0ndFbJRQW6U//7SBE4aWcn3HnmNR159J6Rqg6NQEBEBSvJjHDOknPoG5ztnjmZAr5bvs5CXE+UPFx3JxCHlXP/nhfzPa+vTXGmwFAoiIilfPWkoXz52MBdNHNRmu/xYlNsuqWLCwJ5ce/+rPLt0Q5oqDJ5CQUQk5YQRlfzwnEOIRKzdtkV5Odxx6VGM6lvCt/+yiNr6zLj8tkJBRGQ/lebH+P7ZY9iyq46/vvpu2OV0CoWCiMgBmDS0nLH9Srlt1qqMuByGQkFE5ACYGV85cQgrNu7k+SZnRHdXCgURkQN09mEH0ac0j9teWBl2KQdMoSAicoBycyJ8+dghvLhiC2+s2xF2OQdEoSAi0gm+ePRACnOj3Dare48WAgsFMxtgZs+Z2RIze93MrmuhzblmtsjMFpjZXDM7Pqh6RESCVFYY43NVA3h04To27KgNu5z9FuRIIQ5c7+5jgInAVWY2tlmbZ4Bx7j4euAy4LcB6REQCdelxg4knnLteWh12KfstsFBw9/XuPj/1uBpYAvRv1manf3AMVxHQ/Y/nEpGsNai8iI+N7cuM2WvY2U1vypOWfQpmNhiYAMxu4bnzzWwp8DjJ0UJLy1+Z2rw0d9Om7n/Il4hkriknD2P77nrumLUq7FL2S+ChYGbFwEPAVHf/yG55d3/E3UcD5wE3tPQa7j7N3avcvaqysjLYgkVEDsD4AT04fWwfps1cyfu76tpfoIsJNBTMLEYyEGa4+8NttXX3mcAwM6sIsiYRkaB964xR7KyL8/uZb4Vdyj4L8ugjA6YDS9z95lbaDE+1w8yOAHKBLUHVJCKSDqP6lnDe+P7c9dLqbnckUpAjheOAi4DJqUNOF5jZWWY2xcympNp8GlhsZguA/wY+75lw8RARyXpTTxtBvMH5zbNvhl3KPskJ6oXdfRbQ5vVn3f0m4KagahARCcug8iIuOHoAD8xZy5UnDGNgecs37elqdEaziEhArpk8gpyo8cv/XR52KR2mUBARCUif0nwuOXYwjyx4lwVrt4VdTocoFEREAvS1k4bRtzSfC2+bzYsrNoddTrsUCiIiAepRmMvDXz+W/j0KuOT2OTzy6jthl9QmhYKISMD6lRXwl69N4qjBvfjGnxby2+dXdNm7tCkURETSoDQ/xp2XHcW54w/ip08s4zfPrgi7pBYpFERE0iQvJ8otnxvPxw/ty+//+Rbba+rDLukjFAoiImkUiRjXTB5BTV0D985+O+xyPkKhICKSZmMPKuWEERXc+dJq9sQbwi7nQxQKIiIhuPLEoWyq3sPfXl0XdikfolAQEQnB8cMrGNOvlGkvrCSR6DpHIikURERCYGZceeIQVmzcyfPLN4ZdTiOFgohISD5x+EH0K8tn2syVYZfSSKEgIhKSWDTCZccN4eWVW1n0Tte4NpJCQUQkRBccPYCSvBz+0EVGCwoFEZEQleTHuPjYQTy+aD1/nrs27HIUCiIiYbvu1JGcMKKC7z78Gs8u3RBqLQoFEZGQ5eZE+N2FRzK2XylfnzGfV9e8H1otCgURkS6gOC+H2798FH1K87nszld4a9POUOpQKIiIdBGVJXncfdnRRCPGxdPnsHVXXdprUCiIiHQhg8qLmH7JUazbvps/vpD+I5IUCiIiXcy4AT04+7B+3P3SarbVpHe0oFAQEemCrp48nF11Ddzx4uq0rjewUDCzAWb2nJktMbPXzey6Ftp8ycwWpaaXzGxcUPWIiHQno/uWcsbYPtzx4iqqa9N3M54gRwpx4Hp3HwNMBK4ys7HN2qwCTnL3w4EbgGkB1iMi0q1cPXk4O2rj3PNy+m7GE1gouPt6d5+felwNLAH6N2vzkrvvPSD3ZeDgoOoREeluDj+4ByeNrOS2F1ZRUxdPyzrTsk/BzAYDE4DZbTS7HPhHOuoREekurj11OFt31XHf7DVpWV/goWBmxcBDwFR339FKm1NIhsJ3Wnn+SjOba2ZzN23aFFyxIiJdzJGDejFpaDnTZq6ktj74W3cGGgpmFiMZCDPc/eFW2hwO3Aac6+5bWmrj7tPcvcrdqyorK4MrWESkC7pm8nA2Vu/hL2m4YF6QRx8ZMB1Y4u43t9JmIPAwcJG7Lw+qFhGR7mzSsHLOGXcQPQpzA19XToCvfRxwEfCamS1IzfseMBDA3X8P/AAoB36bzBDi7l4VYE0iIt2OmfHrL0xIy7oCCwV3nwVYO22uAK4IqgYREdk3OqNZREQaKRRERKSRQkFERBopFEREpJFCQUREGikURESkkUJBREQambuHXcM+MbNNQPPryJYB29uZ19bvLT2uADYfQKkt1bQv7To6v7V+NP296fx09KutNpn4XrX23P70q7u9V83nBf1etVbDvrTJxL/Bjswf5O7tXyfI3bv9BExrb15bv7f0GJjb2TXtS7uOzm9wLH+dAAAHK0lEQVStH8360rRN4P1qq00mvled2a/u9l515P3pzPcqXf3qbn+D+zq/rSlTNh892oF5bf3e2uMD0dHXaa1dR+e3Vfujrcw/EB15rbbaZOJ71dpz+9Ov7vZeNZ8X9HvV0dfKtr/BfZ3fqm63+ShdzGyuZ+B1mDKxX5nYJ8jMfmVinyCz+pUpI4UgZOqtQTOxX5nYJ8jMfmVinyCD+qWRgoiINNJIQUREGmVFKJjZ7Wa20cwW78eyR5rZa2a2wsx+nbp50N7nrjGzZWb2upn9tHOrbreuTu+Tmf3QzN41swWp6azOr7zd2gJ5r1LPf8vM3MwqOq/iDtcWxPt1g5ktSr1XT5nZQZ1feZt1BdGnn5nZ0lS/HjGzHp1febu1BdGvz6Y+JxJm1rX3PRzo4WHdYQJOBI4AFu/HsnOASSTvDfEP4OOp+acA/wvkpX7vnQF9+iHwrUx7r1LPDQCeJHmOS0Um9AsobdLmWuD3GdCnM4Cc1OObgJsy5L0aA4wCngeq0t2nfZmyYqTg7jOBrU3nmdkwM3vCzOaZ2QtmNrr5cmbWj+R/vH958p29Gzgv9fTXgBvdfU9qHRuD7cWHBdSn0AXYr1uAfwNC2YkWRL/cfUeTpkWkuW8B9ekpd4+nmr4MHBxsLz4qoH4tcfdl6aj/QGVFKLRiGnCNux8JfAv4bQtt+gPvNPn9ndQ8gJHACWY228z+aWZHBVptxxxonwCuTg3dbzeznsGVuk8OqF9mdg7wrrsvDLrQfXTA75eZ/djM1gJfInl727B1xt/gXpeR/LbdFXRmv7q0IO/R3GWZWTFwLPCXJpud81pq2sK8vd/GcoCewETgKODPZjY09Q0h7TqpT78Dbkj9fgPwC5L/MUNzoP0ys0Lg+yQ3S3QZnfR+4e7fB75vZt8Frgb+o5NL7bDO6lPqtb4PxIEZnVnj/ujMfnUHWRkKJEdI29x9fNOZZhYF5qV+/TvJD8mmw9eDgXWpx+8AD6dCYI6ZJUhe/2RTkIW34YD75O4bmiz3R+CxIAvuoAPt1zBgCLAw9R/6YGC+mR3t7u8FXHtbOuNvsKn7gMcJMRTopD6Z2SXAJ4BTw/qS1Uxnv1ddW9g7NdI1AYNpsuMIeAn4bOqxAeNaWe4VkqOBvTuOzkrNnwL8KPV4JLCW1Hkf3bhP/Zq0+QbwQCa8V83arCaEHc0BvV8jmrS5BngwA/p0JvAGUBnGexT03yDdYEdz6AWk6Q2+H1gP1JP8hn85yW+PTwALU3+EP2hl2SpgMfAWcOveD34gF7g39dx8YHIG9Oke4DVgEclvPv3S1Z8g+9WsTSihEND79VBq/iKS17jpnwF9WkHyC9aC1JTWI6oC7Nf5qdfaA2wAnkx3vzo66YxmERFplM1HH4mISDMKBRERaaRQEBGRRgoFERFppFAQEZFGCgXJCGa2M83ru83MxnbSazWkrnS62Mwebe/KoGbWw8y+3hnrFmlOh6RKRjCzne5e3Imvl+MfXJgtUE1rN7O7gOXu/uM22g8GHnP3Q9NRn2QXjRQkY5lZpZk9ZGavpKbjUvOPNrOXzOzV1M9RqflfNrO/mNmjwFNmdrKZPW9mD6au8T+jyfXxn997XXwz25m6MN1CM3vZzPqk5g9L/f6Kmf2og6OZf/HBhfyKzewZM5tvyWv0n5tqcyMwLDW6+Fmq7bdT61lkZv/Zif+MkmUUCpLJfgXc4u5HAZ8GbkvNXwqc6O4TSF5Z9CdNlpkEXOLuk1O/TwCmAmOBocBxLaynCHjZ3ccBM4GvNFn/r1Lrb/caOKlr6ZxK8mxygFrgfHc/guT9O36RCqX/A7zl7uPd/dtmdgYwAjgaGA8caWYntrc+kZZk6wXxJDucBoxtcmXLUjMrAcqAu8xsBMmrWMaaLPO0uze9lv4cd38HwMwWkLwmzqxm66njg4sHzgNOTz2exAf3dLgP+HkrdRY0ee15wNOp+Qb8JPUBnyA5gujTwvJnpKZXU78XkwyJma2sT6RVCgXJZBFgkrvvbjrTzH4DPOfu56e2zz/f5OldzV5jT5PHDbT8f6beP9g511qbtux29/FmVkYyXK4Cfk3yHgmVwJHuXm9mq4H8FpY34L/c/Q/7uF6Rj9DmI8lkT5G8xwAAZrb30sdlwLupx18OcP0vk9xsBXBBe43dfTvJ22p+y8xiJOvcmAqEU4BBqabVQEmTRZ8ELktd9x8z629mvTupD5JlFAqSKQrN7J0m0zdJfsBWpXa+vkHycucAPwX+y8xeBKIB1jQV+KaZzQH6AdvbW8DdXyV5Jc4LSN5gpsrM5pIcNSxNtdkCvJg6hPVn7v4Uyc1T/zKz14AH+XBoiHSYDkkVCUjqrm+73d3N7ALgC+5+bnvLiYRJ+xREgnMkcGvqiKFthHxrU5GO0EhBREQaaZ+CiIg0UiiIiEgjhYKIiDRSKIiISCOFgoiINFIoiIhIo/8PC5Yslgh79boAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:35 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.811252</th>\n",
       "    <th>2.653636</th>\n",
       "    <th>0.260417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>0.928508</th>\n",
       "    <th>3.329767</th>\n",
       "    <th>0.216667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>0.830531</th>\n",
       "    <th>3.436638</th>\n",
       "    <th>0.218750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>0.947136</th>\n",
       "    <th>3.056552</th>\n",
       "    <th>0.202083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>0.878935</th>\n",
       "    <th>3.361734</th>\n",
       "    <th>0.210417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>0.818984</th>\n",
       "    <th>3.208372</th>\n",
       "    <th>0.214583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>0.811538</th>\n",
       "    <th>2.896590</th>\n",
       "    <th>0.252083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>0.745542</th>\n",
       "    <th>3.237130</th>\n",
       "    <th>0.283333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>0.753505</th>\n",
       "    <th>2.819807</th>\n",
       "    <th>0.302083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>0.763112</th>\n",
       "    <th>2.878011</th>\n",
       "    <th>0.297917</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(10, max_lr=3e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8VOXZ+P/PNdkTQkIWtuyEsMuObOK+oFVxL1qt1lZrW7vXPu3PPurXPj52sbW2WqttXapVq2jdHhQ3qCAgixAUCBACWYFsZN+T+/fHnIQhZJkkczKZyfV+veblzJn7nLmOE3Ll3sUYg1JKKdUTh7cDUEopNfRpslBKKdUrTRZKKaV6pclCKaVUrzRZKKWU6pUmC6WUUr3SZKGUUqpXmiyUUkr1SpOFUkqpXgV6OwBPiYuLM6mpqd4OQymlfMr27dtLjTHxvZXzm2SRmprKtm3bvB2GUkr5FBHJdaecNkMppZTqlSYLpZRSvdJkoZRSqleaLJRSSvVKk4VSSqleabJQSinVK00WSimleqXJwkM+yjpGXlmdt8NQSilbaLLwgIbmVr753Hb+vC7b26EopZQtNFl4wJ4jVTS3Gg6X1Xo7FKWUsoUmCw/YlV8BQK42Qyml/JQmCw/ILKgE4EhlAw3NrV6ORimlPE+ThQdk5lcQHOD8X5lXrrULpZT/0WQxQJV1zeSU1nL2ZOcKv9oUpZTyR7YmCxFZLiL7RCRbRH7WxfspIvKhiOwSkXUikujy3s0icsB63GxnnAOxq9DZX7FidgIAudrJrZTyQ7YlCxEJAB4DLgamAdeLyLROxR4C/mGMmQncDzxonRsD3AssBE4H7hWRUXbFOhCZVuf2GRlxRIUF6YgopZRfsrNmcTqQbYzJMcY0AS8BKzqVmQZ8aD1f6/L+RcD7xphyY8xx4H1guY2x9tvO/EomxEcQFRZEamy4NkMppfySnckiAch3eV1gHXOVCVxtPb8SiBSRWDfP9TpjDDvzK5idGA1ASmyE1iyUUn7JzmQhXRwznV7/BDhLRHYAZwGFQIub5yIit4vINhHZVlJSMtB4++xIZQOlNY3MSnImi9TYcAqP19PU0jbosSillJ3sTBYFQJLL60SgyLWAMabIGHOVMWYOcLd1rNKdc62yTxpj5htj5sfH97rfuMe191e0J4uU2AjaDBRW1A96LEopZSc7k8VWIENE0kQkGFgJvOlaQETiRKQ9hp8DT1nP1wAXisgoq2P7QuvYkJJZUElQgDB1XCQAKbHhANoUpZTyO7YlC2NMC3Anzl/ye4GXjTG7ReR+EbncKnY2sE9E9gNjgAesc8uBX+JMOFuB+61jQ0pmfgXTxo0kJDAAcNYsAHJLNVkopfxLoJ0XN8asBlZ3OnaPy/NVwKpuzn2KEzWNIae1zfB5YSVXzT3R7x43IpiI4AAO64gopZSf0Rnc/ZRTUkNNYwszrZFQACJCSmyELvmhlPI7miz6aafVuT07Keqk4ymx4dpnoZTyO5os+imzoIIRIYFMiBtx0vGU2Ajyy+tobTtlpK9SSvksTRb9lJlfyczEKByOk6eEpMaG09xqKNLhs0opP6LJoh8amlvZe6SqY36Fq/YRUdpvoZTyJ5os+mHvkSpa2gyzEqNOeS81TudaKKX8jyaLfsg6Wg3A9PGnJosxkaEEBzp0QUGllF/RZNEPWUeqiAgOICE67JT3HA4hJSacwzoxTynlRzRZ9EPW0Womj408pXO7XUpshNYslFJ+RZNFHxljyDpazZRxI7stkxobTm55Lcbo8FmllH/QZNFHR6saqKxvZsrYyG7LpMRF0NDcRnF14yBGppRS9tFk0UdZR5yd21PGdl+zSImxRkRpv4VSyk9osuij9pFQk3uoWaS2rz6r/RZKKT+hyaKPso5WMT4qlKiwoG7LjI8OJdAhOtdCKeU3NFn0UdaRnju3AQIDHCTFhJOrs7iVUn5Ck0UfNLW0cbCkpsfO7XYpsTrXQinlPzRZ9MHBkhpa2kyP/RXtUmLCySur0+GzSim/oMmiD7KOVgEwtZdmKHBOzKtubKG8tsnusJRSynaaLPog62g1wQEO0uIiei2bEuscPqv9Fkopf6DJog+yjlSTPnoEQQG9/29L6Rg+q/0WSinfp8miD7KOVjHVjf4KgKSYMETgcKnWLJRSvk+ThZuO1zZxrKqRKePcSxYhgQGMjwrTTZCUUn5Bk4WbTszc7r1zu11KbLhOzFNK+QVNFm7qGAnlZjMU6FLlSin/YWuyEJHlIrJPRLJF5GddvJ8sImtFZIeI7BKRS6zjqSJSLyI7rcdf7IzTHfuOVhMTEUx8ZIjb56TEhlNe20RVQ7ONkSmllP0C7bqwiAQAjwEXAAXAVhF50xizx6XYL4CXjTGPi8g0YDWQar130Bgz2674+mrv0Womj4lEpOsNj7qSag2fzSurY0bCqVuwKqWUr7CzZnE6kG2MyTHGNAEvASs6lTFAeydAFFBkYzz91tZm2H+02u3O7Xbtw2e130Ip5evsTBYJQL7L6wLrmKv7gBtFpABnreK7Lu+lWc1T/xGRZTbG2au88jrqm1uZ2ofObYBka18L7bdQSvk6O5NFV+01nRdKuh54xhiTCFwCPCciDuAIkGyMmQP8CHhBRE75TS0it4vINhHZVlJS4uHwT2jv3HZnTShXESGBxEeG6MQ8pZTPszNZFABJLq8TObWZ6evAywDGmE1AKBBnjGk0xpRZx7cDB4FJnT/AGPOkMWa+MWZ+fHy8DbfgtDmnHIfApDF9Sxbg7Lc4rDULpZSPszNZbAUyRCRNRIKBlcCbncrkAecBiMhUnMmiRETirQ5yRGQCkAHk2BhrtwqO1/HCljyunJNIWHBAn89PjokgT5OFUsrH2ZYsjDEtwJ3AGmAvzlFPu0XkfhG53Cr2Y+A2EckEXgRuMc41vc8EdlnHVwF3GGPK7Yq1J79/b78z0AtPqdi4JTU2nKNVDdQ3tXoyLKWUGlS2DZ0FMMasxtlx7XrsHpfne4ClXZz3KvCqnbG5Y3dRJf/eWcg3z0xnfHRYv66RYq1Qm1de1+c+D6WUGip0BncPfvVOFlFhQXzr7PR+XyOlY0SUdnIrpXyXJotufLy/hPUHSrnznIlEhQX1+zqpHUuVa7+FUsp3abLoQlub4VfvZJE4KoybFqcM6FpR4UFEhwfpxDyllE/TZNGFNzIL2XOkirsumkxIYN9HQHWWEhOuS5UrpXyaJosuPLMxlyljI7ls5niPXC8lNkJrFkopn6bJoguHS2tZkBqDw+H+ooE9SY0Np/B4PU0tbR65nlJKDTZNFp1UNTRTWd9MUkz/hsp2JTk2gjYDhRX1HrumUkoNJk0WnRSUO3+hJ44K99g125cq16YopZSv0mTRScFxZ0d04ijP1SzalyrPLdVkoZTyTZosOik47vmaRdyIYMKDA8jVEVFKKR+lyaKT/ON1RAQHMCq8/xPxOhMR3Y9bKeXTNFl0UnC8nsRR4X3aPtUdzqXKtRlKKeWbNFl04kwWnuuvaJccG05BeT2tbZ33f1JKqaFPk4ULYwwF5XUkxXiuv6LdhLgImlrb2JF33OPXVkopu2mycFFV30J1Y4stNYuLTxvH+KhQfrpql+5toZTyOZosXOTbMGy23cjQIB66dhY5pbU8+M5ej19fKaXspMnChR3DZl0tmRjHrUvT+MemXD7eX2LLZyillB00Wbhon5CXZFOyAPjp8slkjB7BXasyqahrsu1zlFLKkzRZuCg4Xk9kSCAjw+zbbTY0KICHvzybspom/vuN3bZ9jlJKeZImCxcFx+tIGBXm8TkWnc1IiOIH52fwVmYR735xxNbPUkopT9Bk4SK/vN6WYbNdueOsdDJGj+Ch9/br3Aul1JCnycJijKHgeJ0tI6G6Ehjg4PvnZ5BdXMPbu4oG5TOVUqq/NFlYKuqaqW1qtW0kVFcumTGOyWMi+eOHB7R2oZTql10FFazNKrb9czRZWE4Mmx2cmgWAwyF8//wMDpbU8lam1i6UUu4zxvD3DYe4+vGN/PrdLNps/oPT1mQhIstFZJ+IZIvIz7p4P1lE1orIDhHZJSKXuLz3c+u8fSJykZ1xwokJeXYOm+3K8uljmTLWWbtoadVtV5VSvauoa+K2f2znl2/v4axJo3np9kUe2wa6O7YlCxEJAB4DLgamAdeLyLROxX4BvGyMmQOsBP5snTvNej0dWA782bqebdrnWCQMYs0CrNrFeRnklNbylvZdKKV6se1wOZc8sp7/7C/m3sum8devziM6PNj2z7WzZnE6kG2MyTHGNAEvASs6lTHASOt5FND+23IF8JIxptEYcwjItq5nm4Lj9YwMDSQqzHP7WLjroo7aRbbWLpRS3couruaGv31KUKCDV7+1hK8tTbN9qH87O5NFApDv8rrAOubqPuBGESkAVgPf7cO5iMjtIrJNRLaVlAxs+Yz88rpB7dx25XAIPzh/EodKa3ljp9YulFKnam0z/OSVXUQEB7DqjiXMTIwe1M+3M1l0le4698BcDzxjjEkELgGeExGHm+dijHnSGDPfGDM/Pj5+QMEWHK8nKWZwm6BcXTR9DBNHj+C1HQVei0EpNXT9fUMOO/MruO/y6cRHhgz659uZLAqAJJfXiZxoZmr3deBlAGPMJiAUiHPzXI9xzrGo91rNApxbr85MiOJQie6mp5Q6WU5JDb97bz8XTBvD5bPGeyUGO5PFViBDRNJEJBhnh/WbncrkAecBiMhUnMmixCq3UkRCRCQNyAC22BVoWW0T9c2tgzpstitpcREUVTbofhdKqQ6tbYafrtpFaFAAD1wxY9D6KDqzLVkYY1qAO4E1wF6co552i8j9InK5VezHwG0ikgm8CNxinHbjrHHsAd4FvmOMse03aPsci8EeNttZWnwEgO7VrZTq8MzGw2zLPc69l01j9MhQr8Vh3/KqgDFmNc6Oa9dj97g83wMs7ebcB4AH7IyvXfuw2UQv9lkApMZayaK0lqnjRvZSWinl7wqO1/HbNVmcO2U0V845ZYzPoNIZ3JyoWSREe78ZCiCnVGsWSil4fUchDc1t3L9iutean9ppssA5bDY6PIjI0MGfY+EqIiSQMSNDOKTJQikFrNl9jDnJ0V4dfNNOkwXWsNkh8GWAsynqsCYLpYa9oop6Pi+s5MJpY70dCqDJAmBQlybvzYT4CK1ZKKV4b/dRwDkHaygY9snixByLoZEsUmMjKKttorK+uV/nH69t6uiwV0r5rjW7j5ExegQT4kd4OxRAkwUlNY00trQN2g55vWnv5O5vU9R9b+3my09sxhjdH0MpX3W8tokth8u5cIjUKkCTBdFhwbz93TNYPn1otAtOsOZa9Lcpasuhcgor6jlYUuPJsJRSg+jDrGJa2wwXDZHfS6DJguBABzMSorw62cVVUkw4Dunf8NkjlfUcqWwAYMOBUk+HppQaJGt2H2VcVCinJUR5O5QOwz5ZDDUhgQEkjArrVzPUzrwKAIIDHHxysMzToSmlBkF9UyvrD5Rw4bQxXp9b4cqtZCEi6SISYj0/W0S+JyKDuz7uMJIWN6JfzVA78isIDnBw2azxbM4p070xlPJB/9lfQkNz25BqggL3axavAq0iMhH4O5AGvGBbVMNcWmw4h0pr+9xJvTOvgukJIzl7cjzVDS18XlhpU4RKKU8wxpzy7/y9PUeJCgtiQVqMl6LqmrvJos1aGPBK4A/GmB8C4+wLa3hLi4ugprGF0pomt89pbm1jV2EFc5JGsSQ9FoCN2hSl1JD25Sc3c9Zv1/HcpsPUN7XS3NrGh3uLOW/qaIIChlYvgbvRNIvI9cDNwNvWMe+ujeHHUuP6PiJq39FqGprbmJMcTeyIEKaOG8kn2e51ct/7xhc88sGBfsWqlOofYwyfF1RSWtPIf7+xm6W//oj/WrWLyvrmITNr25W7yeJrwGLgAWPMIWuPieftC2t4mxDnnIRzqPTk4a9FFfVc/fjGLofF7sg7DsCcZGdX0tL0WLblHqehueeV3TPzK3h2Uy7PbT6sczOUGkQ1jS3UN7fy/fMyeOWOxcxNjua1HYWEBQVw1qSB7fxpB7eWKLeWEv8egIiMAiKNMb+yM7DhLGFUGEEBwqHSk2diP7c5l+25x3l242HuXzHjpPd25FUQHxnSsXLu0olx/G3DIbYdPs4ZGXHdftbDH+wHoLSmiQPFNUwaE+nhu1FKdaW4uhGA0SNDWJAaw4LUGLKLq6lraiUsOMDL0Z3K3dFQ60RkpIjEAJnA0yLye3tDG74CHEJyTPhJNYumljZe2ZYPwBs7i06pMezIr2B2UnTHULvT02IIdAifHOy+KWp77nHW7SvhxkXJAG43WymlBq64ykoWkSfmeE0cHcnMxKE50NTdZqgoY0wVcBXwtDFmHnC+fWGptLgRHHapWby/5xilNU3cujSNyvpm3t9zrOO947VNHCqt7WiCAudy53OSo9nYQwL4wwf7iY0I5ucXTyU5Jlw7xJUaRMXVzgm0oyNDvByJe9xNFoEiMg64jhMd3MpGE+IjOFRWS1ubsx/hxS15JESH8fNLpjA+KpRXthd0lN1Z4JyMNydp1EnXWJIex67CSirrTl2UcMuhctYfKOWOs9KJCAlk6cRYNueU0dqm/RZKDYaS6lNrFkOZu8nifpx7aR80xmwVkQmADp+xUWpsBE0tbRRV1nO4tJYN2aWsXJBEUICDa+Ylsv5ACUUVzh3+duRV4BCYmXjy0gBLJ8ZhDGzKObXG8PD7+4kbEcKNi1IAWJweR3VDC1/o3AylBkVxdSPBgQ5Ghtm6u7XHuJUsjDGvGGNmGmO+Zb3OMcZcbW9ow9uJ1WfreHFrHgEO4boFSQBcMy8JY+C1z5y1ix15x5k8diQRISf/0M1OiiY8OICNnfotNh4sZVNOGd8+O72jI23xBJ2bodRgKq5qYHRkyJBa0qMn7nZwJ4rIv0WkWESOicirIpJod3DDWXuy2HesmlXbCjh/6mjGWIsdJseGs2hCDK9sL6C1zbDT6tzuLDjQwelpMWzILiW/vI5th8t5K7OIB1dnMWZkCDcsTO4oGx8ZwuQxkackFqWUPYqrG32mvwLcHDoLPI1zeY9rrdc3WscusCMoBWNGhhAWFMBTGw5RVtvEDQtTTnr/2nlJ/PiVTP61NZ/qhpaTOrddLU2PY92+vSz7zdqOYw6Bh66dRWjQycPzFqfH8tLWPBpbWgkJHHpD95TyJ8XVjUwcIhsbucPdZBFvjHna5fUzIvIDOwJSTiJCWlwEe45UkRQTxrKJJ8+VuPi0sdz75m5+syYLgLndJIvrFiTRZgzR4UGMjQpj7MhQxkaFEhV26gT8JemxPLPxMDvzKlhoNUsppexRXNXQsTSPL3C3g7tURG4UkQDrcSPQa+O2iCwXkX0iki0iP+vi/YdFZKf12C8iFS7vtbq896b7t+Q/2puiVi5IxuE4uV0zPDiQS2eOo6KumcjQwI5Z351FhQXxzbPS+fKCZM6aFM/ksZFdJgqAhRNicQi6vLlSNmtobqWqocWnmqHcTRa34hw2exQ4AlyDcwmQbolIAPAYcDEwDbheRKa5ljHG/NAYM9sYMxv4E/Cay9v17e8ZYy53M06/MmVsJMGBDq6d33X30LXznR3es5OiT0km/REVFsRpCVFs0n4LpWzV1YS8oc7d0VB5xpjLjTHxxpjRxpgrcE7Q68npQLY1cqoJeAlY0UP564EX3Yp6mPjGsgm894Mzu/2BmpsczaUzx3HNPM+NNVicHseOvArqmlo8dk2l1Mk6JuSN9L+aRVd+1Mv7CUC+y+sC69gpRCQF5x4ZH7kcDhWRbSKyWUSuGECcPissOKBjBdquiAiP3jCXFbO7/N/aL0vSY2lpM2w5VO6xayqlTlbsYxPyYGDJord2j67e72568EpglTHGdcGjZGPMfOAG4A8ikn7KB4jcbiWUbSUlJW4FrXq2IDWGoABhk/ZbKGWb4irfq1kMZOpgb+tCFABJLq8TgaJuyq4EvnPSxY0psv6bIyLrgDnAwU5lngSeBJg/f76uU+EBYcEBzEkexbu7jwJwuKyWw6V1lNU28eytC5g+fuhsIK+UryqubiTQIcSEB3s7FLf1WLMQkWoRqeriUQ2M7+XaW4EMEUkTkWCcCeGUUU0iMhkYBWxyOTbKZc/vOGApsKdPd6b67bwpo8ktq+PpTw5zsKSWpJgwquqb+fdnhd4OTSm/UFzdSNyIEI8MTBksPdYsjDH93tzAGNMiInfiXFMqAHjKGLNbRO4Hthlj2hPH9cBL5uSdd6YCT4hIG86E9itrTw01CG5bNoEr5iQQNyKEAOuH+eantvDenmPc/aWpPrM8gVJDVXF1o081QcHAmqF6ZYxZDazudOyeTq/v6+K8jcBpdsamuudwSMfSIu0unD6Gu//9BfuP1TB5rG6QpNRAFFc1kDgqzNth9MnQ2hFcDVkXTB0DwHtWX4ZSqv9KqhuJ96GRUKDJQrlp9MhQZidF8/7eY70XVkp1q7m1jbLaJp+avQ2aLFQfXDh9DLsKKjlSWe/tUJTyWaU1J/be9iWaLJTbLpzmbIr6YI/WLpTqL19c6gM0Wag+SI8fwYS4CN7TZKFUv52Yva01C+WnRIQLpo9h08EyKutP3te7trGlY70bpVT3jvng7G3QZKH66MJpY2hpM6zbV9xxLL+8josfWc+lf9xAQ3NrD2crpYqrGxGBuBGaLJQfm500irgRIR1NUTklNVz3xCaKqxsorm7kzZ3dreiilAIoqW4gJjyYoADf+vXrW9EqrwtwCOdPHc1/9pWwq6CC657YTFNLG69+awlTxkby1CeHOHkyvlLKVXFVI/E+1l8BmixUP1w4fQw1jS1c/fhGAh3Cv765mOnjo7j1jDSyjlbzSbauWKtUd4qrG09ZIcEXaLJQfbYkPY7I0EDGRoXyyh2LmTjauaXr5bPGEzcimKc+OeTlCJUauoqrG3xuJBTYvDaU8k+hQQG8decZjAoPJio86KTjNy5K4Q8fHOBgSQ3p8V3vC67UcNXaZiitafK5kVCgNQvVT6lxESclinY3LkohOMDB01q7UOoU5bVNtLYZn5uQB5oslIfFjQhhxezxvLq9kIq6Jm+Ho9SQ0rH3tg82Q2myUB739WVp1De38uKW/N4LKzWMdMze1mYopWDK2JEsnRjLsxsP09TS5u1wlBoySnx0XSjQZKFscsdZ6RytauDRtdneDkWpIaO9GUrnWShlWZYRz1VzE3hsbTa7Ciq8HY5SQ8KxqkZGhgYSGhTg7VD6TJOFss29l00nfkQIP3o5U9eMUgprjoUPTsgDTRbKRlFhQfzmmplkF9fw+/f3ezscpbyuuLrRJ0dCgSYLZbMzJ8XzlYXJ/HV9DlsPl3s7HKW8qrhKk4VS3fr/LplK4qgwfvxyJrWNLd4ORymvMMZQUt2ozVBKdSciJJCHrplFXnkdf12f4+1wlPKKyvpmmlrbtGahVE8WTojlnMnx/PPTPJ17oYalI5XOYbO+uOIs2JwsRGS5iOwTkWwR+VkX7z8sIjutx34RqXB572YROWA9brYzTjU4bl6SSkl1I+98ccTboSg16HJKagFIi4vwciT9Y9uqsyISADwGXAAUAFtF5E1jzJ72MsaYH7qU/y4wx3oeA9wLzAcMsN0697hd8Sr7nZkRT1pcBE9/cpgVsxO8HY5Sg+pgSQ2Az67GbGfN4nQg2xiTY4xpAl4CVvRQ/nrgRev5RcD7xphyK0G8Dyy3MVY1CBwO4ebFKezMryAzXyfqqeHlYEkNCdFhhAX73oQ8sDdZJACuK8kVWMdOISIpQBrwUV/PVb7l6nmJRAQH8OzGw94ORalBdbCkhvTRvlmrAHuThXRxrLvNmVcCq4wx7dN83TpXRG4XkW0isq2kpKSfYarBFBkaxDXzEnlrVxEl1gqc7Zpb2yiv1WXNlf9pazMcLK4lPd43+yvA3mRRACS5vE4Eiropu5ITTVBun2uMedIYM98YMz8+Pn6A4arB8tUlqTS3Gl7cktdxbN/Rai770wbO+s1aqhuavRidUp53tKqB+uZWn+2vAHuTxVYgQ0TSRCQYZ0J4s3MhEZkMjAI2uRxeA1woIqNEZBRwoXVM+YH0+BGcOSme5zfn0tjSylMbDnHZoxvILaujurGFzTk601v5F1/v3AYbk4UxpgW4E+cv+b3Ay8aY3SJyv4hc7lL0euAlY4xxObcc+CXOhLMVuN86pvzELUtSKK5u5OI/rOf+t/ewbGIcH/74LMKDA/h4vzYpKv9ysNhKFqN9txnKtqGzAMaY1cDqTsfu6fT6vm7OfQp4yrbglFedPWk0E+IiKKqs54ErZ3DD6cmICIsmxLL+gCYL5V8OltQSGRpI/AjfnL0NNicLpbrjcAgv3LYIgLFRJ2a0npkRx0dZxeSV1ZEcG+6t8JTyqIMlNaTHj0Ckq7E7vkGX+1BeMzYq9KREAbBsknOgwsdau1B+pD1Z+DJNFmpImRAXQUJ0mPZbKL9R3dDMsapGn+6vAE0WaogREc6cFMemg2U0t+qCg8r3ta8JpTULpTxsWUY81Y0tuiSI6lJrW3dze4cmfxg2C5os1BC0ND0Oh6BNUeoUGw6UMv3ed8k6WuXtUNx2sKSGQIeQ4uMDNjRZqCEnKjyIWUnRfHyg1NuhqCHmiY8P0tDcxt/XH/J2KG47WFxLcmw4QQG+/evWt6NXfuvMjHh2FVRQUadrRSmn7OIa1h8oZVR4EG9kFlFW09j7SUOAP4yEAk0Waog6c1IcbQY+yS7zdihqiHhu02GCAxz85cZ5NLW08cKneb2e420trW0cLqtlog+vNttOk4UakmYlRhMZGqj9FgpwDj9dtb2AS2eOY+GEWM6cFM8/NucO+S1684/X09xqtGahlF0CAxwsTY9j/YESXJYNU8PUq9sLqG1q5eYlqQDcutS5Re/qz4f2Fr0da0L58NLk7TRZqCFr2aQ4iiobOoYequGprc3wj025zE6KZlZSNODs05oQH8HTnxwa0n9MtP/sTtCahVL2OXfKaETgjZ3dbYOihoP12aXklNZy85KUjmMOh/C1JalkFlTyWd7QnY9zsKSG+MgQosKCvB3KgGmyUEPWuKgwzp4Uz79LYHeDAAAaG0lEQVS25tOis7mHhaKKeo5WNpx07B8bDxM3IphLTht30vGr5iYSGRrI058M3WG0B0t8e3c8V5os1JB2/enJFFc38lFWsbdDUTbLL6/joj98zKIHP+RLf1zP79/bx5rdR/loXzE3nJ5MSGDASeUjQgK5/vRk3vniKEcq670UdfeMMWQX+8ewWdBkoYa4c6eMZnRkyElbsCr/09zaxvde2gEGfnTBJCKCA3l0bTbffG47ASLcsDCly/O+ujgFYwxPfpwzyBF3zbX/pLy2icr6Zr9JFrqfhRrSAgMcfHlBEo+uzaawop6E6DBvh6Rs8MgHB9iRV8Gfrp/DZbPG873zMqioa2LdvhJCgwJOWcq+XeKocL68IInnN+dy8+JUUuO80+TT1ma49dmt7MirYPGEWJZmxBEe5KwJpfvBHAvQZKF8wHXzncniX1vz+dEFk7wdjvKwTQfLeGxdNtfNT+SyWeM7jkeHB3PFnIRez//hBZN4Y2cRv1mTxZ+/Mq/X8rsKKvjW859RUddEYICDoAAh0OEgPCSA6LAgosKCiA4PZnF6LNfNT3LrHv6+4RDr9pVw9uR4Pi+s5N3dRzvem+ClBOZpmizUkJcUE86ZGfG8vDWf7507kUAfX2NHnXC8tokf/msnabER3Hf59H5dY3RkKHeclc7v39/PtsPlzE+N6bZsfnkdtz6zjZBAB19ekExLWxstbYbmljbqmlqpqG+ipKaR3UVV/HtHIenxI5iXMqrHz99dVMlv1mSxfPpYHr9xLgC5ZXV8crCUusZWEkf5R21Yk4XyCTcsTOabz21n3b4Szp82xtvhKA8wxvDTV3dRXtvE325eQnhw/38dfWNZGv/8NJf/+b+9/PvbS7rcvrSyrplbnt5CU0srL92+kImjI7u9Xm1jC+c8tI5fvr2H1761BIej6+1Q65ta+f5LO4mJCObBq07r+NzUuAivNYnZRf9EUz5BO7r9z/Obc3l/zzF+unwyMxKiBnSt8OBAfnzhZHbmV/B/Xczqbmxp5bbntpFfXs9fvzq/x0QBzpFWP7nIeb23dnU/z+fBd/aSXVzD766dzaiI4AHdw1CnyUL5hKAAB9fNT2LtvmIOltSwI+84//w0l1+8/jmPrzvo7fBUH+07Ws3//N9ezpoUz61L0zxyzavnJjJlbCS/fjeLxpbWjuNtbYa7XtnFlkPl/PbamSycEOvW9a6Zm8j08SP59TtZ1De1nvL+h3uP8Y9Nudy2LI0zMuI8cg9DmQzlqfJ9MX/+fLNt2zZvh6FslF9ex5m/XYvrj2xwgIOm1jbe+f4ypo4b6b3glNsamltZ8egnlNU28c73lxEfGeKxa68/UMJNf9/CuVNG4xDIK68jr7yOhuY2/mv5FL51dnqfrrc5p4yVT27mxxdM4rvnZXQc/2DPMX6yKpNxUWG8/p0lp8wB8SUist0YM7+3ctpnoXxGUkw496+YQWl1I9PGj2TauJFEhgZyxq/X8uhH2Tz2lbneDlG5qGpopqXVENOpeebB1XvZd6yaZ762wKOJApxb8l48Yyz/2V9Cckw4KbERLMuIZ1ZSNJfNHNf7BTpZNCGWi6aP4fH/HOS6BUlEhATyP2/v4aWt+UwdN5K/3DjXpxNFX9iaLERkOfAIEAD8zRjzqy7KXAfcBxgg0xhzg3W8FfjcKpZnjLnczliVb7hp0amTs25Zkspj67I5cKyajDE9t0UPZVUNzTy0Zh8rFyQzbbzv15K+8ew2Pss9zsWnjeOWJSnMTR7Fh3uLeXZTLl8/I42zJ4+25XMfv3EexpguO7n74+cXT+WjrP/wk1cyOVxWS+Hxer51djo/OD9j2CQKsDFZiEgA8BhwAVAAbBWRN40xe1zKZAA/B5YaY46LiOtPT70xZrZd8Sn/cesZaTz1ySEeXZvNIyvneDucfvvrxzn8Y1Mur31WyOM3zmVZRry3Q+q33LJathwqZ17KKNbtK+atzCKmjx9JUUU908eP5KfLJ9v6+Z5KFOAc2XTLklT+uv4QyTHhvPzNxT0Oz/VXdnZwnw5kG2NyjDFNwEvAik5lbgMeM8YcBzDG6AJAqs9iIoK5aXEKb2UWkeOjy5mX1TTy1IZDnDkpnsRRYXzt6a28si3f22H12xs7ixCBP10/h80/P48HrpxBS6uhpdXwyMo5PvcX+Y8umMxvr5nJ6u8vG5aJAuxNFgmA6097gXXM1SRgkoh8IiKbrWardqEiss06fkVXHyAit1tltpWU6I5qw9ltyyYQHOjgsbW+OTLqiY9zqG9u5Z5Lp/LKHYtZNCGWu1bt4pEPDgzp/Rq6Yozh9Z2FnJ4aw/joMCJCAvnKwhTe/cEytv7ifJ/cYjQsOIBr5ycxImT4dvPaeedd1QM7/9QHAhnA2UAisF5EZhhjKoBkY0yRiEwAPhKRz40xJ/0mMMY8CTwJztFQnr4B5TviRoTwlYUpPLPxMN8/L4Pk2HBvh+S2Y1UNPLvxMFfMSegY///ULQv42Wu7ePiD/az6LJ/kmHASo8NJGBXGGRlxzE3ueVaxN31RWEVOSS3fOGPCScdFhNAg36pRqBPsrFkUAK4LqyQCnWe3FABvGGOajTGHgH04kwfGmCLrvznAOsB3G6PVoPjmmRMIcAh/Xpft7VD65LG12bS2GX5w3ol1r4IDHfzu2ln88ooZzE4aRV1TK2v3FfP79/ez8onNHDhW7bHPb20zbD1c7rE9Q17fWUhQgHDJaWM9cj01NNiZLLYCGSKSJiLBwErgzU5lXgfOARCROJzNUjkiMkpEQlyOLwX2oFQPRo8M5foFSazaXkBuWa23w3FLwfE6XtySx3ULkk6pDYkINy1K4U/Xz+Hf317KlrvPZ8vd5xEREsBdq3bR2jbwynTW0Squ+ctGrv3LJn7x+hcDbvJqbTO8lVnE2ZNHEx3u3zOahxvbkoUxpgW4E1gD7AVeNsbsFpH7RaR9GOwaoExE9gBrgbuMMWXAVGCbiGRax3/lOopKqe5855yJhAQ6+OXbnv1x+f17+3h/zzGPXhPgTx9mIyJ899yJbpUfHRnKfZdPZ2d+xYB2iGtobuWhNfu49I8byC2rY/n0sby0NZ9/bMrt9zXBuYJscXUjV8zufbVY5Vts7a0xxqwGVnc6do/LcwP8yHq4ltkInGZnbMo/jR4ZyvfPz+B/V2fxUdYxzp0y8EUHDxyr5o8fZZMQHcY5k+M9turtodJaVn1WwFcXpzAuyv2VSS+fNZ63Mo/w2zX7OG/qGNL6uGDdnqIqvvPCZxwqreWquQn84kvTiA4L4vbntnH/23uYOHoESyf2b/mK13cWMiIkkPOm2jOHQnmPrg2l/M4tS9JIj4/g/721h4bmU9f06asXrMULCyvqeeeLo72Udt9TGw4R6BC+fbZ7tYp2IsIDV84gONDBf726i7Y+NEfll9dx89NbqG9q5fmvL+T3180mJiIYh0N4+MuzSY+P4Nv//KxfzXgNza28+8VRls8Yqx3ZfkiThfI7wYEO/t/lM8gtq+Nv6we23WZDcyuvbi/gSzPHMSEugic/zvHIUNaG5lbezCxi+Yyx/VryYszIUP770mlsOVTOPz91r+mosq6Zrz2zlYbmVp77+umnLH4XGRrEX786HxHn7OvqhuY+xfRRVjE1jS2smD2+98LK52iyUH7pjIw4LjltbMd2rP319q4jVDW0cNOiFL6+LI3PCyv59FD5gOP7cG8xlfXNXDMvsd/XuHZeIssy4vif/9vLLU9v4f639vD85lw2HSw7pUbV2NLK7c9tI7eslidvmt/tsigpsRH8+Ya55JTWsuTBj/jGs9t4+pNDHDhW3WuSfH1HIfGRISxJ9/8VWIej4TvDRPm9u780jY+yivmft/fw+I29b7fZlX9+mkt6fAQL02KYnRTN797bz9/W57DIzWWuu/PK9nzGRYUO6BeriPDQtbP49TtZZB2tZnNOGQ3NzuGvI0ICuWj6WFbMHs/i9Fh+umoXnx4q55GVs1mc3nPsSybG8fzXF/JmZiGfZJfxwV5nx/6c5Gheun1Rl7OvD5fWsnZfMTctSiWgm42ClG/TZKH8VkJ0GHeeM5GH3tvPxuxSlvSx03ZPURU78ir470undUwo++riFP7wwQGyi2v6PRP5WFUDH+8v4dtnTxzwL9YxI0P5/ZedS6i1tRmOVDWw72gV735xlHe+OMqrnxUwIiSQmsYW7rpoMivcHKW0OD22I6nkl9ex+vMjPPhOFn/88AB3XTTlpLLGGH722i5CAwO4/cwJXV1O+QFthlJ+7RvLJhA3IrhfQ0Jf2JJLcKCDq+ee+AV706IUQgId/H1D//tCXvuskDYDVw+gCaorDoeQEB3GuVPG8JtrZrHtF+fzxE3zOHtyPN85J51v93Evh3ZJMeF886x0rpmXyF/+k8PnBZUnvf+vrflszinn55dMZWxUqCduRQ1BmiyUXwsNCmDF7AQ+zDpGRV2T2+fVNrbw+o4iLp057qTJZbEjQrh6XiKvflZIaU1jn+MxxrBqez4LUkf1echrX4UEBnDR9LE8esNc7rpoyoBXYv3vL00jNiKYu1Zl0tTibO46VtXAA6v3sjAthpULknq5gvJlmiyU37tqbgLNrYa3dp26N3N33swsoqaxha8sTD7lva+fkUZza1uPtRVjnDOZOw9B3ZFfwcGS2gF1bHtLVHgQ/3vlaWQdrebRtc4lVe554wuaWtr41dUzcWhfhV/TZKH83rRxI5kyNpLXPitw+5wXPs1jytjILhfsS48fwXlTxvD85txu53FkFlTy3Rd3cOmfNrBu34mV91dtLyAsKIAvzfTN4aXnTxvDlXMS+PPabB5+fz9rdh/jB+dPsr2WpLxPk4XyeyLCVXMT2JFX4dZ+FzvyjvN5YSU3LEzutunmliWplNc28c4XXddWXvg0l/DgABKiw7j1ma088Z+DNDS38lZmERfPGOvTS13fe9k0osODeeTDA0wfP5LblqV5OyQ1CDRZqGHhitkJOMTZudybJz/OYWRoIFfN7b6paOnEWCbER/BcF01RlfXNvJlZxIrZCbz27SUsnzGWB9/J4qo/b6S6ocUnm6BcRYcH8+urTyMhOoxfXz3TY8ufqKFNv2U1LIweGcqyjHj+vaOwx+UxDpXW8u7uo9y0OKXHv/5FhBsXpvBZXgVfFJ48OujfnxXQ0NzGVxYmEx4cyGM3zOUnF05iz5EqEqLDBjxHYyg4b+oYNvzXOcxIiPJ2KGqQaLJQw8ZVcxMorKjvcQb2X9fnEBTg4JYlvTetXD0vkdAgx0nLbRhjeGFLHrMSozp+kYoId56bwSt3LOaJm+b5TUewJ/e5VkOfJgs1bFw4zdlX0F1Hd0l1I6u2F3DNvES31muKCgviitkJvL6jiMp65zpK23KPs/9YDTd0MYpqQWqM/iWufJYmCzVshAUHcMlpY1n9+RHqm04dxfTMxkM0t7Zx2zL3ZyHfuCiFemuxQYB/bs4lMiSQy2b55mgnpbqjyUINK1fNTaS2qZX39py81HhNYwvPbcpl+fSxfRoGOiMhijnJ0Ty/OZeymkZWf3GUq+YmEB7su6OdlOqKJgs1rJyeGkPiqDD+d/VeXt6a37Hv9Etb8qhqaOnX2kY3LUohp7SWu1btoqmljRsWpng6bKW8TpOFGlYcDuGxG+YyNiqMn766iwsf/pg3M4t4asMhFqbFMKeLSXi9ueS0ccREBPNRVjHzU0YxeWzXy38r5cs0WahhZ1ZSNK9/ewlP3DSPwADhey/uoKiygTvO6t9Ce6FBAVw337kuUlcd20r5A21YVcOSiHDR9LGcP3UMb2YWcqiklrMnx/f7et88cwKRoYFc6qPLeCjVG00WalgLcAhXzhn4jOpREcF855y+7aWtlC/RZiillFK90mShlFKqV7YmCxFZLiL7RCRbRH7WTZnrRGSPiOwWkRdcjt8sIgesx812xqmUUqpntvVZiEgA8BhwAVAAbBWRN40xe1zKZAA/B5YaY46LyGjreAxwLzAfMMB269zjdsWrlFKqe3bWLE4Hso0xOcaYJuAlYEWnMrcBj7UnAWNM+y4xFwHvG2PKrffeB5bbGKtSSqke2JksEoB8l9cF1jFXk4BJIvKJiGwWkeV9OFcppdQgsXPobFfrF3feSCAQyADOBhKB9SIyw81zEZHbgdsBkpN1MpRSStnFzppFAZDk8joRKOqizBvGmGZjzCFgH87k4c65GGOeNMbMN8bMj4/v/4QqpZRSPRNjut81bEAXFgkE9gPnAYXAVuAGY8xulzLLgeuNMTeLSBywA5iN1akNzLWKfgbMM8Z0u2uNiJQAnfe4jAIqeznW0+uunscBpd3F4aau4upLmaF4X+7cU0/l3LmnzsfceT4Y96XfVdfHu7sP19f6XfU/XnfL9XZfKcaY3v/aNsbY9gAuwZkwDgJ3W8fuBy63ngvwe2AP8Dmw0uXcW4Fs6/G1fn7+k70d6+l1V8+BbR74/3JKXH0pMxTvy5176qmcO/fkzn108dz2+9Lvqm/30ele9Luy8bty977cedi63IcxZjWwutOxe1yeG+BH1qPzuU8BTw0whLfcONbT6+6eD5Q71+qpzFC8L3ev0105d+6p8zH9rvrHju+qq+M9xf5WN8cHQr8r99/rc3y2NUP5KxHZZoyZ7+04PE3vy3f44z2Bf96XP92TLvfRd096OwCb6H35Dn+8J/DP+/Kbe9KahVJKqV5pzUIppVSvhnWyEJGnRKRYRL7ox7nzRORza5HEP4qIuLz3XWsBxd0i8hvPRu1WbB6/LxG5T0QKRWSn9bjE85H3GJct35X1/k9ExFjDtweVTd/VL0Vkl/U9vScig7ojk0339FsRybLu698iEu35yHuNzY77utb6PdEmIkO7b2Mgw7p8/QGciXMuxxf9OHcLsBjn8N93gIut4+cAHwAh1uvRfnJf9wE/8afvynovCViDc45OnD/cFzDSpcz3gL/4wT1dCARaz38N/NpPvqupwGRgHTB/sO+pL49hXbMwxnwMnDTRT0TSReRdEdkuIutFZErn80RkHM5/kJuM8xv/B3CF9fa3gF8ZYxqtzyjufL7dbLovr7Lxnh4GfkoXy8kMBjvuyxhT5VI0gkG+N5vu6T1jTItVdDPOVR0GlU33tdcYs28w4h+oYZ0suvEk8F1jzDzgJ8CfuyiTgHNJknauCx1OApaJyKci8h8RWWBrtO4b6H0B3Gk1AzwlIqPsC9VtA7onEbkcKDTGZNodaB8N+LsSkQdEJB/4CnAP3ueJn792t+L863wo8OR9DWm6B7cLERkBLAFecWnWDumqaBfH2v96CwRGAYuABcDLIjLB+ovCKzx0X48Dv7Re/xL4Hc5/tF4x0HsSkXDgbpzNG0OGh74rjDF3A3eLyM+BO3HuD+MVnron61p3Ay3APz0ZY3948r58gSaLkzmACmPMbNeD4tzIabv18k2cvzhdq8GuCx0WAK9ZyWGLiLThXB+mxM7AezHg+zLGHHM576/A23YG7IaB3lM6kAZkWv/QE4HPROR0Y8xRm2PviSd+Bl29APwfXkwWeOiexLlj5qXAed7848uFp7+roc3bnSbefgCpuHRYARuBa63nAszq5rytOGsP7R1Wl1jH7wDut55Pwrkvh/jBfY1zKfND4CVfv6dOZQ7jhQ5um76rDJcy3wVW+cE9Lce5hly8N74ju38G8YEObq8H4OUv/kXgCNCMs0bwdZx/bb4LZFo/nPd0c+584AuciyQ+2p4QgGDgeeu9z4Bz/eS+nsO52OMunH8tjRus+7HrnjqV8UqysOm7etU6vgvnGkAJfnBP2Tj/8NppPQZ1hJeN93Wlda1G4BiwZrDvy92HzuBWSinVKx0NpZRSqleaLJRSSvVKk4VSSqleabJQSinVK00WSimleqXJQvk1EakZ5M/7m4hM89C1Wq2VY78Qkbd6W2lVRKJF5Nue+GylOtOhs8qviUiNMWaEB68XaE4saGcr19hF5FlgvzHmgR7KpwJvG2NmDEZ8anjRmoUadkQkXkReFZGt1mOpdfx0EdkoIjus/062jt8iIq+IyFvAeyJytoisE5FV1h4L/3TZn2Bd+74EIlJjLeiXKSKbRWSMdTzder1VRO53s/aziRMLII4QkQ9F5DNx7pGwwirzKyDdqo381ip7l/U5u0Tk/3nwf6MaZjRZqOHoEeBhY8wC4Grgb9bxLOBMY8wcnCu1/q/LOYuBm40x51qv5wA/AKYBE4ClXXxOBLDZGDML+Bi4zeXzH7E+v9c1gqy1hs7DOXMeoAG40hgzF+f+Kb+zktXPgIPGmNnGmLtE5EIgAzgdmA3ME5Eze/s8pbqiCwmq4eh8YJrLSqEjRSQSiAKeFZEMnKuCBrmc874xxnUvgy3GmAIAEdmJc82gDZ0+p4kTCy5uBy6wni/mxJ4aLwAPdRNnmMu1twPvW8cF+F/rF38bzhrHmC7Ov9B67LBej8CZPD7u5vOU6pYmCzUcOYDFxph614Mi8idgrTHmSqv9f53L27WdrtHo8ryVrv8tNZsTnYLdlelJvTFmtohE4Uw63wH+iHOPinhgnjGmWUQOA6FdnC/Ag8aYJ/r4uUqdQpuh1HD0Hs49HgAQkfYlpqOAQuv5LTZ+/maczV8AK3srbIypxLk96k9EJAhnnMVWojgHSLGKVgORLqeuAW619l1ARBJEZLSH7kENM5oslL8LF5ECl8ePcP7inW91+u7Buaw8wG+AB0XkEyDAxph+APxIRLYA44DK3k4wxuzAubLpSpwb/8wXkW04axlZVpky4BNrqO1vjTHv4Wzm2iQinwOrODmZKOU2HTqr1CCzdumrN8YYEVkJXG+MWdHbeUp5k/ZZKDX45gGPWiOYKvDi9rRKuUtrFkoppXqlfRZKKaV6pclCKaVUrzRZKKWU6pUmC6WUUr3SZKGUUqpXmiyUUkr16v8HjXw9futU+ewAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('char_rnn_1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:17 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.696038</th>\n",
       "    <th>2.910773</th>\n",
       "    <th>0.304167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>0.734545</th>\n",
       "    <th>2.814250</th>\n",
       "    <th>0.306250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>0.634951</th>\n",
       "    <th>2.827829</th>\n",
       "    <th>0.295833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>0.636780</th>\n",
       "    <th>2.758662</th>\n",
       "    <th>0.312500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>0.696148</th>\n",
       "    <th>2.838843</th>\n",
       "    <th>0.312500</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(5, 3e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('char_rnn_1_final')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is abysmal; 31% is much worse than 52% from the simple Naive Bayes bigram model.\n",
    "\n",
    "Does it improve if we add another layer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, MyLetterRNN(n_layers=2), loss_func=F.cross_entropy, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8XHW9//HXZ7KvTdNm6ZaGrrSFLjSUQkU2BQUuiwoigih4kesGCPd33a7X5SeiXvFef6jIFRG5FReKICACyi5LaUtXUqCF7kvSNVuzzuf3x0xDGpImJTk5ycz7+XjMo5Mz33PO55tJ5z3nnO85x9wdERERgEjYBYiIyOChUBARkXYKBRERaadQEBGRdgoFERFpp1AQEZF2CgUREWmnUBARkXYKBRERaZcadgFHauTIkV5eXh52GSIiQ8rSpUt3uXtRT+2GXCiUl5ezZMmSsMsQERlSzGxjb9pp95GIiLRTKIiISDuFgoiItFMoiIhIO4WCiIi0UyiIiEg7hYKIiLQbcucp9FVb1Nm8p4HXd9ayvrqe4dlpTC3NY2ppHtnpb/86olFnT0Mze+qbiRikRCKkRoyUiNHa5hxoaaOxpY0DLW2kpRjjCrMpys3AzELsnQC4u94HkXcpaULh2TequfmRtayrqqOpNfqO182grDCbgux0qmsaqaptojV6ZPevzkpLoawwm7IR2YwdnsXY4bF/xw3PpiQ/g4LsdFIiwX9YtbRF2VvfzK66ZvY1NFPb1Ep9Uyt1Ta3UN7UR7eK+3CkRIzViRMyIGLRGnZY2p7UtSkvUwR0HDs4aiRhZaSlkpkXITEvBgB01jWzbd4Bt+xrZtv8AzZ1+zxEz0lKM9NQU0lOM1JQIHl9ubBWxD/PUiJFi1v67ao1GaY06rW1OW9RJiYfzwZBubGmjtrGVmsYWahpbiUad4TnpFGanMzwnjcKcdAqyYz8XZMd+LsrLYNSwLEYNyyQnI2n+G4j0KGn+N2Snp1KYk85l88czpSSXKSV5TCzOZV99C5U7anhtRy1rd9RQ29jKpKKRlORnUJKfSWFOOhDbwoh9MEVJTYkc8oHY1NrG5j0H2Li7gU17Gti4u55/rNtFQ3PbITVEDApzMhiZm87I3Ni/RXkZFOVlUJiTQUoE2qIQdcc9tr6W1igtbU5LNEpbmxOJfxCmxL8J765vpqqmkZ21jeysaWJXXRP7Glr6/fd38Iu3AWZGWzeBWZyXwZjhWUwrzScj7dC9k9F40DS3RWlujdIajWIYZrHAMIuFTux3HaUt6jhOaiRCZtrBEIgQjf9u2qKx301BdjrjCrPJz0ojLzOViBn74lt5e+tbeG1HLfsaWth3oKXLuvMyUynOy2BETkZ7iAzLSic9Jba+lEhsSzE3M5Wi9vcu9veRlZ7S379qkVCZd/GtcTCrqKjwoXCZC3dnb0MLW/Y2sGXvAapqGtkd//a+u66J6rrYB3hVTVOXWy69lZZiFOdltofYwQ+swtx0Rsa/IedlppKbkUpuZio56anv2FqJuhP12LfwtqgTdUhNMdIiEVJTYh/GnXfHRKNOU2u0fRdaW9Qpzs8gI3XwfkhGo05tYyt7Gpqprm1i+/4DbN/fyI79jeysaWRvQzP7GlrYUx/7t7nt8O+LGUwsyuWY0fkcM2YY00fnM3pYFiNy08nNSNUuLBlUzGypu1f02E6hEC53p66plT31zbjHvjFHIrFv42kRIy0l9sGclhIhJWLxD22Pf4uGPH34BCoaPbhV4tQ0trCrrolddc3sqm1i054G1myrYc22/Wzf33jIfBmpEUbmZjCuMIsZo4cxfVQ+00fnM6k4l7QUje+QgdfbUEia3UeDlZmRl5lGXmZar9qnDd4v4gkpEjHS41tWWekplORndtmuuraJtTtqqKppYnf928Hx5q56Fr60kcaW2FZHdnoK759ewnmzRnPy5CLSUxUQMrgoFET6QezYUNdXJW5ti7Jhdz1rttXwwvrdPLJ6Bw8s38awrDQ+MKOUKaV5jMxNZ0ROBiNyY8dHcnXwW0Ki3UciA6y5Ncqzb1Tz4IptPP7qTuq7GJAwtTSfueMLmDt+OMeXFzJ2eHZI1Uqi0DEFkSEg2n6sIjYAYVddM6/trGXZxr0s37yPuqZWABZMGsFlJ4znfdNLdExC3hWFgsgQ1xZ1Xt9Zy98rd3LP4s1s3XeA4rwMLjl+HB84ZhRHl+YRGYDzXiQxKBREEkhb1HlybRULX9rIU69X4w7Ds9M44agRnDhxBO+fXsLogqywy5RBTKEgkqB27G/k+fW7eGH9bp5fv5ut+w6Ql5nKTy89jvdO6fEWvJKkFAoiSWJdVS2f/+0rvFFVxzfOnc4VJ5WHXZIMQr0NBR2xEhniJhXnce+/nMRpU4v4jz+v4d/vX01rD2dji3RHoSCSAHIzUvnF5RVc/d4J3P3iRj7165ffcUFCkd4ILBTMbJyZPWlmlWa2xsyu7aLNv5rZ8vhjtZm1mVlhUDWJJLKUiPHVs6fxnfNn8Owbu7h36ZawS5IhKMgthVbgBnefBswHPmdm0zs2cPcfuvtsd58NfAV42t33BFiTSMK7bP54Zo0r4KdPrqNFu5HkCAUWCu6+3d2XxZ/XApXAmMPM8jHgnqDqEUkWZsa1Z0xi674D3LdMWwtyZAbkmIKZlQNzgJe6eT0b+ACwaCDqEUl0p00t5tgxw7hVWwtyhAIPBTPLJfZhf52713TT7J+Af3S368jMrjazJWa2pLq6OqhSRRKGmfHFMyazec8B7n9la9jlyBASaCiYWRqxQFjo7vcdpuklHGbXkbvf7u4V7l5RVKSTc0R6433Tipk+Kp+fPrlOQ1Sl14IcfWTAHUClu99ymHbDgFOAB4KqRSQZHdxa2LC7gT+v2BZ2OTJEBLmlsAC4HDi9w7DTs83sGjO7pkO7C4HH3L0+wFpEktKZ00s4ujSPW59Y1+19tUU6CuxOHu7+HLH7vPfU7tfAr4OqQySZRSKxrYXPLlzGw6u2c96s0WGXJIOczmgWSXAfmFHKUSNzuPuFDWGXIkOAQkEkwUUixsfmjePlDXt5fWdt2OXIIKdQEEkCHz5uLOkpEX770qawS5FBTqEgkgRG5GZw1jGl3LdsC40tbT3PIElLoSCSJC6dV0ZNYysPr9wedikyiCkURJLE/AmFTBiZw28XaxeSdE+hIJIkzIyPzStj6ca9vLZDB5ylawoFkSTy4bmxA873aGtBuqFQEEkihTnpfOCYUhYt28KBZh1wlndSKIgkmUtPKKO2sZWHV+mAs7yTQkEkyZxwVCETi3L4w5LNYZcig5BCQSTJmBmnTClm5ZZ9RHWRPOlEoSCShKaW5tLYEmXz3oawS5FBRqEgkoSmlOQBaGiqvINCQSQJTY6Hgi6QJ50pFESSUG5GKuMKs3htZ13Ypcggo1AQSVJTS/J4XbuPpBOFgkiSmlKSx/rqOppbo2GXIoOIQkEkSU0tzaM16ry1S7dHl7cpFESSVPsIJB1slg4UCiJJakJRDikR6/a4wrNvVFNV0zjAVUnYFAoiSSojNYWjRuZ0uaWwp76ZK361mJ888UYIlUmYFAoiSWxqaV6X5yo89VoVUYelG/eFUJWESaEgksSmluSxaU8DDc2th0x/Ym0VAK/tqKGuqbWrWSVBKRREktiUkjzcYV3V2yextbRFefr1asYOzyLqsHKzthaSiUJBJIlNLY2NQFrb4WDzkg17qW1s5YtnTAZg2aa9odQm4VAoiCSxssJsMlIjh4xAemLtTtJTIpx97CgmF+eydKNCIZkoFESSWErEmFySe8gIpCfWVnHChEJyM1I5rmw4r2zeh7vuu5AsAgsFMxtnZk+aWaWZrTGza7tpd6qZLY+3eTqoekSka1NL8ttHIG3YVc/66npOP7oYgOPGF7CvoYU3ddZz0ghyS6EVuMHdpwHzgc+Z2fSODcysAPgZcJ67zwAuCrAeEenC1NJcdtY0sa+huX3UUXsolA0HYJl2ISWNwELB3be7+7L481qgEhjTqdmlwH3uvineriqoekSka1Pa761QxxNrq5hUnMv4ETkATCzKJT8zlWWbNAIpWQzIMQUzKwfmAC91emkKMNzMnjKzpWb2iW7mv9rMlpjZkurq6mCLFUkyB0cgLdu0l5fe2s0Z8a0EgEjEmF02XFsKSSTwUDCzXGARcJ2713R6ORWYC5wDnAX8u5lN6bwMd7/d3SvcvaKoqCjokkWSSml+JnmZqdz1/AZa2pzTOoQCwNyy4bxeVUtNY0tIFcpACjQUzCyNWCAsdPf7umiyBfiru9e7+y7gGWBWkDWJyKHMjKkleWzf30h+Zipzxw8/5PXjxhfgDit0EltSCHL0kQF3AJXufks3zR4ATjazVDPLBk4gduxBRAbQwV1Ip0wtJi3l0I+F2eMKMINlug5SUkgNcNkLgMuBVWa2PD7tq0AZgLvf5u6VZvZXYCUQBX7p7qsDrElEunAwFM7otOsIIC8zjSnFeTqzOUkEFgru/hxgvWj3Q+CHQdUhIj17//QSlm/ex/uml3T5+nHjC3h45XaiUScS6fG/tQxhOqNZRBg1LItbLp5NbkbX3xPnlA2nprGV9dV1Xb4uiUOhICI9OnjwWbuQEp9CQUR6NGFkDgXZaTrYnAQUCiLSIzNjzrgC/r52p66amuAUCiLSK9e/fwrpKREuuu15bvpLJY0tbWGXJAFQKIhIr8wcW8Cj17+Xjx5fxu3PvMnZP3lWxxgSkEJBRHotLzON733oWO6+ah6NzW185OfPs2TDnrDLkn6kUBCRI3by5CL+ev17Kc7L5DsPvUo0qpvwJAqFgoi8K/mZadx41lRWbNnPgyu3hV2O9BOFgoi8ax+aM4ZjxuTz/UfW6sBzglAoiMi7FokYXzt7Otv2N3LHc2+FXY70A4WCiPTJiRNH8L5pJfz8qfVU1zaFXY70kUJBRPrsK2cfTWNLGz/+2+thlyJ9pFAQkT6bWJTLZfPH87vFm3h9Z23Y5UgfKBREpF9ce8ZkcjNS+fr9qzVEdQhTKIhIvxiek87Xz5nO4rf2sPCljWGXI++SQkFE+s1FFWM5efJIvvfIWjbvaQi7HHkXFAoi0m/MjJs/PBMDvnLfKty1G2moUSiISL8aU5DFV86exnPrdvH7lzeHXY4cIYWCiPS7S+eVceKEEXz34Uq27z8QdjlyBBQKItLvIhHj+x+eSWvU+bdFq2jTaKQhQ6EgIoEoG5HN18+dxjOvV/PtB9fo+MIQkRp2ASKSuD5+wng27Krnf559i1EFWVxzysSwS5IeKBREJFBf+eA0tu9v5OZH1jJqWCbnzx4TdklyGAoFEQlUJGL86OJZVNc2ceMfV1CUm8FJk0aGXZZ0Q8cURCRwGakp3P6JCo4amcNn7l7K+uq6sEuSbigURGRADMtK49efmkdqinH975fT0hYNuyTpQmChYGbjzOxJM6s0szVmdm0XbU41s/1mtjz++EZQ9YhI+EYXZHHThceycst+bn1iXdjlSBeCPKbQCtzg7svMLA9YamaPu/urndo96+7nBliHiAwiHzx2FB86bgy3PrmOU6cWMadseNglSQeBbSm4+3Z3XxZ/XgtUAhp2ICJ887wZlOZn8qU/rKChuTXscqSDXoWCmU00s4z481PN7ItmVtDblZhZOTAHeKmLl080sxVm9oiZzejtMkVk6MrPTONHF89iw+56bvpLZdjlSAe93VJYBLSZ2STgDuAo4Le9mdHMcuPzX+fuNZ1eXgaMd/dZwP8D7u9mGVeb2RIzW1JdXd3LkkVkMJs/YQT/fPIE/vfFTTz5WlXY5Uhcb0Mh6u6twIXAf7n79cConmYyszRigbDQ3e/r/Lq717h7Xfz5X4A0M3vHAGZ3v93dK9y9oqioqJcli8hgd8OZU5hSksvX/7Rau5EGid6GQouZfQy4AngoPi3tcDOYmRHbqqh091u6aVMab4eZzYvXs7uXNYnIEJeRmsJNFx7L1n0H+O+/vxF2OULvQ+FTwInAd939LTM7CvjfHuZZAFwOnN5hyOnZZnaNmV0Tb/MRYLWZrQB+AlziumqWSFKpKC/k4oqx3PHsW7y2ozbscpKeHelnsJkNB8a5+8pgSjq8iooKX7JkSRirFpGA7Klv5vQfPcWU4jx+/5n5xHcgSD8ys6XuXtFTu96OPnrKzPLNrBBYAdxpZl3uEhIROVKFOel85YNHs3jDHu5duiXscpJab3cfDYuPHPoQcKe7zwXeF1xZIpJsLpo7jrnjh/O9R9ayt7457HKSVm9DIdXMRgEX8/aBZhGRfhOJGP/3gmPYf6CFm/5SqZvyhKS3ofBt4FFgvbu/bGYTAA0VEJF+NW1UPv988gT+uHQLV921hF11TWGXlHSO+EBz2HSgWSSxRaPOXS9s4HuPrCU/M43/vGgmp04tDrusIa+/DzSPNbM/mVmVme00s0VmNrbvZYqIHCoSMT614Cj+/PkFjMhJ55N3vsw3/7xGl9oeIL3dfXQn8GdgNLGL2j0YnyYiEoijS/N54PML+ORJ5fz6+Q38bvGmsEtKCr0NhSJ3v9PdW+OPXwO63oSIBCozLYVvnjeDmWOH8ZsXNurg8wDobSjsMrPLzCwl/rgMXY5CRAbIZfPH80ZVHS++uSfsUhJeb0PhSmLDUXcA24ldnuJTQRUlItLRebNGU5Cdxt0vbgi7lITXq1Bw903ufp67F7l7sbtfQOxENhGRwGWmpXBxxTgeXbOTHfsbwy4nofXlzmtf6rcqRER68PETyoi6c48OOAeqL6GgK1aJyIAZPyKHU6YUcc/iTRqeGqC+hIKGAYjIgPrEieOpqm3i0TU7wi4lYR02FMys1sxqunjUEjtnQURkwJwypZhxhVn85oWNYZeSsA4bCu6e5+75XTzy3D11oIoUEQFIiRiXnTCexW/tYe2Ozrd8l/7Ql91HIiID7qKKcaSnRrhbWwuBUCiIyJBSmJPOebNGc9+yrew/0BJ2OQlHoSAiQ84nTyrnQEsbf1yyOexSEo5CQUSGnGPGDOP48uH85oWNtEU1ELI/KRREZEi64qRyNu1p4Mm1VWGXklAUCiIyJJ01o5TS/EzuemFD2KUkFIWCiAxJaSkRLptfxrNv7GJdVW3Y5SQMhYKIDFkfm1dGemqEu57X8NT+olAQkSFrRG4G580azaJlWzQ8tZ8oFERkSPvkSeU0NGt4an9RKIjIkHbMmGFUjNfw1P6iUBCRIe+q9xzFpj0NPLxqe9ilDHmBhYKZjTOzJ82s0szWmNm1h2l7vJm1mdlHgqpHRBLXWTNKmVScy0+fWEdUWwt9EuSWQitwg7tPA+YDnzOz6Z0bmVkK8H3g0QBrEZEEFokYnzttIq/trOVvlTvDLmdICywU3H27uy+LP68FKoExXTT9ArAI0GmJIvKu/dPM0ZQVZvPTJ9fhrq2Fd2tAjimYWTkwB3ip0/QxwIXAbT3Mf7WZLTGzJdXV1UGVKSJDWGpKhH85dSIrtuzn2Td2hV3OkBV4KJhZLrEtgevcvfNdMf4L+Dd3bzvcMtz9dnevcPeKoqKioEoVkSHuQ8eNoTQ/k1ufWBd2KUNWoKFgZmnEAmGhu9/XRZMK4HdmtgH4CPAzM7sgyJpEJHFlpKbwmVMmsHjDHl56c3fY5QxJQY4+MuAOoNLdb+mqjbsf5e7l7l4O3At81t3vD6omEUl8lxxfxsjcdG59UlsL70aQWwoLgMuB081sefxxtpldY2bXBLheEUliWekpXPWeCTz7xi5e3rAn7HKGHBtqR+krKip8yZIlYZchIoNYXVMrZ/34GQD+8sWTGZadFnJF4TOzpe5e0VM7ndEsIgknNyOVWy+dQ1VtIzfeu0JDVI+AQkFEEtKcsuF8+YPTePzVndzx3FthlzNkKBREJGFduaCcs2aUcPMja1m2aW/Y5QwJCgURSVhmxg8+MotRBZl8fuEy9tY3h13SoKdQEJGENiwrjZ9eehy76pr5zkOvhl3OoKdQEJGEN3NsAZfMG8fDq7ZT06g7tB2OQkFEksIFc8bQ1Brlr6t3hF3KoKZQEJGkMGdcAeNHZPPA8q1hlzKoKRREJCmYGefPHsPz63ezY39j2OUMWgoFEUkaF8wejTs8uGJb2KUMWgoFEUkaE4pymTWugD+9ol1I3VEoiEhSuWD2aF7dXsPrO2vDLmVQUiiISFI5d+ZoUiLG/dpa6JJCQUSSSlFeBidPHskDy7cRjepCeZ0pFEQk6Vwwewxb9x3Q/Ra6oFAQkaRz5owSstNTuH+5RiF1plAQkaSTnZ7KWTNKeXjlNppbo2GXM6goFEQkKZ07cxQ1ja28+ObusEsZVBQKIpKUFkwaSXZ6Co+9qmshdaRQEJGklJmWwilTinhszU6NQupAoSAiSevMGSVU1TaxYsu+sEsZNBQKIpK0Tp9aQkrEeOzVnWGXMmgoFEQkaQ3LTmP+hEIeW6PjCgcpFEQkqZ01o5T11fWsq6oLu5RBQaEgIkntfdNKAHhcu5AAhYKIJLnRBVnMHDtMQ1PjFAoikvTOnF7CK5v2UVWjO7IFFgpmNs7MnjSzSjNbY2bXdtHmfDNbaWbLzWyJmb0nqHpERLpz5oxSAB6vfHsX0vb9B/jM3Uv4xgOrwyorFKkBLrsVuMHdl5lZHrDUzB5391c7tPk78Gd3dzObCfwBODrAmkRE3mFycS7lI7J5bM1OLp1XxqJlW/nWg2uoa2rFPXb281nx4Eh0gW0puPt2d18Wf14LVAJjOrWpc/eDpxLmADqtUEQGnJlx5oxSnl+/i0/ftYQb/7iCaaX5PH79KRxdmse/37+a/Qdawi5zQAzIMQUzKwfmAC918dqFZrYWeBi4ciDqERHp7MzpJbS0Oc+u28XXz5nGPVfPZ1JxLj/4yEx21TVx8yNrwy5xQAQeCmaWCywCrnP3ms6vu/uf3P1o4ALgO90s4+r4MYcl1dXVwRYsIklp7vjh3HThsfzliyfz6ZMnkBIxAGaOLeCq9xzFPYs3hXpF1dueXj8g95UONBTMLI1YICx09/sO19bdnwEmmtnILl673d0r3L2iqKgooGpFJJmZGZeeUMak4tx3vPal90+lrDCbLy9aSWNL24DX9sDyrdz8yFoWLdsS+LqCHH1kwB1Apbvf0k2bSfF2mNlxQDqgi5uLyKCSlZ7C9z50LBt2N/Bff3tjQNe9dkcNX160innlhdx45tTA1xfk6KMFwOXAKjNbHp/2VaAMwN1vAz4MfMLMWoADwEc7HHgWERk0FkwayUcrxvE/z77JCRMKOW1qceDr3H+ghWvuXkpeZiq3fnwOaSnBHwYOLBTc/TnAemjzfeD7QdUgItKf/v2fprN6234+t3AZv7t6PjPHFgS2rmjUueEPy9my9wC/u3o+xXmZga2rI53RLCLSS7kZqdz5yeMZnp3Olb9+mU27GwJb18+fXs/fKqv4+jnTqCgvDGw9nSkURESOQHF+JnddOY/WqHPFnYvZXdfU7+tYvXU///nYa5w/ezRXnFTe78s/HIWCiMgRmlScyx1XVLBt3wGuvGsJDc2t/br8l97agzt87ZxpxMfiDBiFgojIuzB3fCE/+dgcVmzex6+ee6tfl72uqo6C7DSKcjP6dbm9oVAQEXmXzppRyrzyQh5csb1fl7u+qo5JRbkDvpUACgURkT45Z+YoXttZyxv9eLbxG1W1XZ5ENxAUCiIiffDBY0oxg4dW9s/Wwu66JvY2tCgURESGouL8TE44qpCHV22nP869PXiv6IkKBRGRoemcmaNZV1XH6zvr+rysddWxZUwqUiiIiAxJH5hRSsTgoZXb+rysdVV1ZKWlMKYgqx8qO3IKBRGRPirKy2D+hBE8vLLvu5DWVdUxoSiHSGTgRx6BQkFEpF+cO3M0b+6qp3L7oaOQ9tY385dV24lGexcW66vqmBzS8QRQKIiI9IuzZpSQEjEeXvX2LqQd+xu5+Bcv8NmFy3i8cmePy6hvamXb/sbQRh6BQkFEpF+MyM3gpIkjeCi+C2nT7gYu+sXzbN/fSHFeBrc9vb7HXUvrDx5kViiIiAx9584cxcbdDdy/fCsX/eJ5ahtbWfjpE/jC6ZN4ZdM+Xt6w97DzHxyOqlAQEUkAZ04vJTViXP/7FUQdfn/1icwaV8BH5o6jMCedXzy9/rDzv1FVR2rEGD8iZ4AqfieFgohIPxmek85ZM0oZOzyLP37mRKaW5gGx23lecWI5f19bxeuHuRzGuqo6xo/IHpA7rHVHoSAi0o9+/NHZPHXjqZSPPPTb/idOHE9WWgq3P/Nmt/Our6oLddcRKBRERPpVemqE1C6+6Q/PSeejx4/jgeVb2b7/wDteb26NsnFPg0JBRCRZXPWeo4g63PmPDe94bcPuetqirlAQEUkW4wqzOefYUfz2pU3sP9ByyGvtI4+K8sIorZ1CQURkAH3mlAnUNbW+425tb18dNbyRR6BQEBEZUDNGD+OcY0dx29Pr2bynoX36uqo6xhRkkZ2eGmJ1CgURkQH39XOnkRIxvvXgmvZp6wbByCNQKIiIDLhRw7K49ozJ/K2yir+9upO2qLO+WqEgIpK0PrXgKCYV5/Kth9bwZnUdTa1RhYKISLJKT43w7fNnsHnPAf5t0Uog3GseHaRQEBEJyUkTR3LerNEs27QPCO8WnB0FFgpmNs7MnjSzSjNbY2bXdtHm42a2Mv543sxmBVWPiMhg9LVzppGbkcqInHSG56SHXQ5Bjn1qBW5w92VmlgcsNbPH3f3VDm3eAk5x971m9kHgduCEAGsSERlUSvIzueXiWeypbw67FCDAUHD37cD2+PNaM6sExgCvdmjzfIdZXgTGBlWPiMhgdeaM0rBLaDcgxxTMrByYA7x0mGZXAY8MRD0iItK1wE+dM7NcYBFwnbvXdNPmNGKh8J5uXr8auBqgrKwsoEpFRCTQLQUzSyMWCAvd/b5u2swEfgmc7+67u2rj7re7e4W7VxQVFQVXsIhIkgty9JEBdwCV7n5LN23KgPuAy9399aBqERGR3gly99EC4HJglZktj0/7KlAG4O63Ad8ARgA/i2UIre5eEWBNIiJyGEGOPnoOsB7afBr4dFA1iIjIkdEZzSIi0k6hICIi7czdw67hiJhZNbCx0+RF7gMvAAAHeElEQVRhwP4eph3u566ejwR29bHcruo6kja96Vfnab153te+9aZfh2vX2+kD/Z71tV/dvRZ2v7qr60ja6G9x6P8tjnf3nodvuvuQfwC39zTtcD939RxYEkRdR9KmN/3qTV+6eN6nvvWmX4dr19vpA/2e9bVfvX3P9Leov8Wg+3Uky+j8SJTdRw/2Ytrhfu7ueV/1ZlmHa9ObfnWeNlj6dbh2vZ0+0O9ZX/vV3Wth96u3y9LfYvfTE+VvsUdDbvfRQDGzJZ6gw2MTtW/q19CTqH0byv1KlC2FINwedgEBStS+qV9DT6L2bcj2S1sKIiLSTlsKIiLSLilCwcx+ZWZVZrb6Xcw718xWmdk6M/tJ/JpOB1/7gpm9Fr+z3A/6t+pe1dbv/TKzb5rZVjNbHn+c3f+V96q+QN6z+Os3mpmb2cj+q7jXtQXxnn0nfvfC5Wb2mJmN7v/Ke6wtiH790MzWxvv2JzMr6P/Ke1VfEH27KP65ETWzwXXsoS/DpobKA3gvcByw+l3Muxg4kdglOx4BPhiffhrwNyAj/nNxgvTrm8CNifiexV8bBzxK7FyXkYnQLyC/Q5svArclSL/OBFLjz78PfD9R/haBacBU4CmgIox+dfdIii0Fd38G2NNxmplNNLO/mtlSM3vWzI7uPJ+ZjSL2H+4Fj72TvwEuiL/8L8DN7t4UX0dVsL14p4D6NSgE2LcfA/8HCOVgWhD98kPvU5JDCH0LqF+PuXtrvGlod2YMqG+V7v7aQNR/pJIiFLpxO/AFd58L3Aj8rIs2Y4AtHX7eEp8GMAU42cxeMrOnzez4QKvtvb72C+Dz8U32X5nZ8OBKPWJ96puZnQdsdfcVQRd6hPr8npnZd81sM/BxYlcfHgz642/xoCsZXHdm7M++DSqB33ltMLLY3eBOAv7YYXdzRldNu5h28FtYKjAcmA8cD/zBzCbEvxGEop/69XPgO/GfvwP8iNh/yFD1tW9mlg18jdguiUGjn94z3P1rwNfM7CvA54H/6OdSj0h/9Su+rK8BrcDC/qzx3erPvg1GSRkKxLaQ9rn77I4TzSwFWBr/8c/EPiA7brKOBbbFn28B7ouHwGIzixK73kl1kIX3oM/9cvedHeb7H+ChIAs+An3t20TgKGBF/D/yWGCZmc1z9x0B1344/fG32NFvgYcJORTop36Z2RXAucAZYX7h6qS/37PBJeyDGgP1AMrpcKAIeB64KP7cgFndzPcysa2BgweKzo5Pvwb4dvz5FGAz8fM+hni/RnVocz3wu0R5zzq12UAIB5oDes8md2jzBeDeBOnXB4BXgaKw/gaD/ltkEB5oDr2AAXpD7wG2Ay3EvuFfRexb41+BFfE/vG90M28FsBpYD9x68IMfSAf+N/7aMuD0BOnX3cAqYCWxbzujBqo/QfetU5tQQiGg92xRfPpKYte7GZMg/VpH7MvW8vhjwEdVBdi3C+PLagJ2Ao+G0beuHjqjWURE2iXz6CMREelEoSAiIu0UCiIi0k6hICIi7RQKIiLSTqEgCcHM6gZ4fb80s+n9tKy2+BVOV5vZgz1dDdTMCszss/2xbpHONCRVEoKZ1bl7bj8uL9XfvhhboDrWbmZ3Aa+7+3cP074ceMjdjxmI+iS5aEtBEpaZFZnZIjN7Of5YEJ8+z8yeN7NX4v9OjU//pJn90cweBB4zs1PN7Ckzuzd+Xf+FHa6H/9TB6+CbWV38gnQrzOxFMyuJT58Y//llM/t2L7dmXuDtC/jlmtnfzWyZxa7Jf368zc3AxPjWxQ/jbf81vp6VZvatfvw1SpJRKEgi+2/gx+5+PPBh4Jfx6WuB97r7HGJXFL2pwzwnAle4++nxn+cA1wHTgQnAgi7WkwO86O6zgGeAf+6w/v+Or7/Ha97Er51zBrEzyQEagQvd/Thi9+/4UTyUvgysd/fZ7v6vZnYmMBmYB8wG5prZe3tan0hXkvWCeJIc3gdM73Aly3wzywOGAXeZ2WRiV61M6zDP4+7e8dr5i919C4CZLSd2DZznOq2nmbcvHLgUeH/8+Ym8fS+H3wL/2U2dWR2WvRR4PD7dgJviH/BRYlsQJV3Mf2b88Ur851xiIfFMN+sT6ZZCQRJZBDjR3Q90nGhm/w940t0vjO+ff6rDy/WdltHU4XkbXf+fafG3D8511+ZwDrj7bDMbRixcPgf8hNi9EYqAue7eYmYbgMwu5jfge+7+iyNcr8g7aPeRJLLHiN1bAAAzO3ip42HA1vjzTwa4/heJ7bYCuKSnxu6+n9jtNG80szRidVbFA+E0YHy8aS2Q12HWR4Er49f5x8zGmFlxP/VBkoxCQRJFtplt6fD4ErEP2Ir4wddXiV3uHOAHwPfM7B9ASoA1XQd8ycwWA6OA/T3N4O6vELvy5iXEbipTYWZLiG01rI232Q38Iz6E9Yfu/hix3VMvmNkq4F4ODQ2RXtOQVJGAxO/2dsDd3cwuAT7m7uf3NJ9ImHRMQSQ4c4Fb4yOG9jEIbmsq0hNtKYiISDsdUxARkXYKBRERaadQEBGRdgoFERFpp1AQEZF2CgUREWn3/wHgiqYENFYvqgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 01:34 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.929248</th>\n",
       "    <th>3.101529</th>\n",
       "    <th>0.189583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>0.695901</th>\n",
       "    <th>2.869615</th>\n",
       "    <th>0.250000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>0.745567</th>\n",
       "    <th>2.520683</th>\n",
       "    <th>0.316667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>0.620927</th>\n",
       "    <th>3.530135</th>\n",
       "    <th>0.262500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>0.742575</th>\n",
       "    <th>2.512531</th>\n",
       "    <th>0.318750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>0.723677</th>\n",
       "    <th>2.616584</th>\n",
       "    <th>0.343750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>0.839355</th>\n",
       "    <th>2.454891</th>\n",
       "    <th>0.335417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>0.817186</th>\n",
       "    <th>2.794391</th>\n",
       "    <th>0.291667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>0.653000</th>\n",
       "    <th>2.695168</th>\n",
       "    <th>0.302083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>0.683367</th>\n",
       "    <th>2.637764</th>\n",
       "    <th>0.358333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>0.611877</th>\n",
       "    <th>2.308675</th>\n",
       "    <th>0.333333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>0.586979</th>\n",
       "    <th>2.296229</th>\n",
       "    <th>0.352083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>0.611386</th>\n",
       "    <th>2.224956</th>\n",
       "    <th>0.381250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>0.580687</th>\n",
       "    <th>2.247524</th>\n",
       "    <th>0.383333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>0.512482</th>\n",
       "    <th>2.244857</th>\n",
       "    <th>0.387500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>0.516693</th>\n",
       "    <th>2.303736</th>\n",
       "    <th>0.412500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>0.409016</th>\n",
       "    <th>2.413911</th>\n",
       "    <th>0.412500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>0.435291</th>\n",
       "    <th>2.442951</th>\n",
       "    <th>0.422917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>0.386392</th>\n",
       "    <th>2.507006</th>\n",
       "    <th>0.425000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>0.352908</th>\n",
       "    <th>2.518786</th>\n",
       "    <th>0.433333</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(20, max_lr=1e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the fit has converged, again at a much worse result than our Naive Bayes bigrams.\n",
    "\n",
    "But that was trained using a balanced dataset; maybe that will help with RNNs too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJztnXd4VGX2xz8nHUjoVUoCiKIiJUY6CIoFUFFExV1dERXbrrqWXSw/e8GOvYu6q7KKXUBFARFEICAdKUKAUANICSWQ5P398d6ZzCQzySRkMjOZ83meeXLn3vfee+Zm5n7ve857zivGGBRFURQFICbUBiiKoijhg4qCoiiK4kZFQVEURXGjoqAoiqK4UVFQFEVR3KgoKIqiKG5UFBRFURQ3KgqKoiiKGxUFRVEUxU1cqA0oLw0bNjRpaWmhNkNRFCWimD9//g5jTKOy2kWcKKSlpZGZmRlqMxRFUSIKEVkfSDt1HymKoihuVBQURVEUNyoKiqIoipuIiykoilJ9OHLkCNnZ2Rw6dCjUplQbkpKSaNGiBfHx8RXaX0VBUZSQkZ2dTUpKCmlpaYhIqM2JeIwx7Ny5k+zsbFq3bl2hY6j7SFGUkHHo0CEaNGigglBJiAgNGjQ4qp5X0ERBRJJEZK6ILBKRZSLyoI82I0QkR0QWOq9rgmWPoijhiQpC5XK01zOY7qM84HRjTK6IxAMzRWSyMebXYu3+Z4z5exDtiF72bYOsn+HkYaG2RFGUCCFoPQVjyXXexjsvnRC6KvnlBfj0avgzoJwVRYk6du7cSefOnencuTNNmzalefPm7veHDx8O6BhXXXUVK1euDLKlVUdQA80iEgvMB44FXjbGzPHR7CIR6QusAv5pjNkYTJuiivWznL+/QL3U0NqiKGFIgwYNWLhwIQAPPPAAycnJ3HHHHV5tjDEYY4iJ8f0MPW7cuKDbWZUENdBsjCkwxnQGWgBdRaRDsSZfA2nGmI7AD8B7vo4jIqNEJFNEMnNycoJpcvXh0F7Yssguu8RBUZSAWLNmDR06dOD6668nPT2dLVu2MGrUKDIyMjjppJN46KGH3G179+7NwoULyc/Pp27duowePZpOnTrRo0cPtm/fHsJPUTGqZEiqMWa3iEwHzgGWeqzf6dHsTeAJP/u/AbwBkJGRoS6oQNg4B0wh1GyooqBEBA9+vYzlm/dW6jFPPKY29593UoX2Xb58OePGjeO1114DYMyYMdSvX5/8/Hz69+/PsGHDOPHEE7322bNnD6eddhpjxozhtttu45133mH06NFH/TmqkmCOPmokInWd5RrAAOD3Ym2aebw9H1gRLHuijvWzICYOul0Hu9bC3i2htkhRIoq2bdty6qmnut9/9NFHpKenk56ezooVK1i+fHmJfWrUqMHAgQMBOOWUU8jKyqoqcyuNYPYUmgHvOXGFGOBjY8w3IvIQkGmM+Qq4WUTOB/KBXcCIINoTXWTNgmPS4dgBMO1R2PALdLgo1FYpil8q+kQfLGrVquVeXr16Nc8//zxz586lbt26XH755T5zARISEtzLsbGx5OfnV4mtlUkwRx8tNsZ0McZ0NMZ0MMY85Ky/zxEEjDF3GWNOMsZ0Msb0N8b8XvpRlYA4vB82L4C0XtC0IySkWJFQFKVC7N27l5SUFGrXrs2WLVv47rvvQm1S0NAyF9WRjXOhMB9Se0FsHLTqZkcgKYpSIdLT0znxxBPp0KEDbdq0oVevXqE2KWiIMZEVt83IyDA6yU4ZTH0Efn4G/r0ekmrb5R8fgjvXQq0GobZOUdysWLGCE044IdRmVDt8XVcRmW+MyShrX619VB1Z/ws062QFAWyPAWDD7NDZpChKRKCiUN04cgiyM4uEAGzAOS5Jh6YqilImKgrVjU2ZUJAHab2L1sUlQItTVRQURSkTFYXqRtYsQKBVd+/1qb1g6xI4tCckZimKEhmoKFQ31s+EJh2gRj3v9ak9bYbzBl/lpxRFUSwqCtWJ/MOwcZ7NTyhOi1MhJl5dSIqilIqKQnVi82+Qf9A7yOwioSY0T9d8BUXxoF+/fiUS0caOHcuNN97od5/k5GQANm/ezLBhvucq6devH2UNnR87diwHDhxwvx80aBC7d+8O1PSgoaJQnVg/0/71JQpgXUibF9iMZ0VRuOyyyxg/frzXuvHjx3PZZZeVue8xxxzDhAkTKnzu4qIwadIk6tatW+HjVRYqCtWJrFnQ6AT/CWqpvW2mc/a8qrVLUcKUYcOG8c0335CXlwdAVlYWmzdvpnPnzpxxxhmkp6dz8skn8+WXX5bYNysriw4d7GwABw8eZPjw4XTs2JFLL72UgwcPutvdcMMN7pLb999/PwAvvPACmzdvpn///vTv3x+AtLQ0duzYAcCzzz5Lhw4d6NChA2PHjnWf74QTTuDaa6/lpJNO4qyzzvI6T2WhZS6qCwX5tlx2x0v9t2nZFSTGupDa9KsqyxQlMCaPtiPkKpOmJ8PAMX43N2jQgK5du/Ltt98yZMgQxo8fz6WXXkqNGjX4/PPPqV27Njt27KB79+6cf/75fuc/fvXVV6lZsyaLFy9m8eLFpKenu7c9+uij1K9fn4KCAs444wwWL17MzTffzLPPPsu0adNo2LCh17Hmz5/PuHHjmDNnDsYYunXrxmmnnUa9evVYvXo1H330EW+++SaXXHIJn376KZdffnnlXCsH7SlUF7YsgsO5voPMLpJq2wJ5WhxPUdx4upBcriNjDHfffTcdO3ZkwIABbNq0iW3btvk9xowZM9w3544dO9KxY0f3to8//pj09HS6dOnCsmXLfJbc9mTmzJlceOGF1KpVi+TkZIYOHcrPP/8MQOvWrencuTMQvNLc2lOoLrhGFaX2Lr1dWm+Y+ybk50FcYvDtUpRAKeWJPphccMEF3HbbbSxYsICDBw+Snp7Ou+++S05ODvPnzyc+Pp60tDSfpbI98dWLWLduHU8//TTz5s2jXr16jBgxoszjlFaPLjGx6DcbGxsbFPeR9hSqC+tnQYNjIaVJ6e1Se9qM500LqsYuRQlzkpOT6devHyNHjnQHmPfs2UPjxo2Jj49n2rRprF+/vtRj9O3blw8++ACApUuXsnjxYsCW3K5VqxZ16tRh27ZtTJ482b1PSkoK+/bt83msL774ggMHDrB//34+//xz+vTpU1kft0xUFKoDhQWwfrb/UUeetOph/2q+gqK4ueyyy1i0aBHDhw8H4K9//SuZmZlkZGTwwQcf0L59+1L3v+GGG8jNzaVjx448+eSTdO3aFYBOnTrRpUsXTjrpJEaOHOlVcnvUqFEMHDjQHWh2kZ6ezogRI+jatSvdunXjmmuuoUuXLpX8if2jpbOrA1sWwet9Yeib0PGSstu/0tP2KK74PPi2KUopaOns4KCls6MdV+A4tWdg7VN72nIXBZE3VaCiKMFFRSFQNsyBwsJQW+Gb9bOgbirUaRFY+9SecGQ/bF0UXLsURYk4VBQCYesSGHcOfHF9+D1dFxbavIO0MkYdeeKKPejQVCUMiDQXdrhztNdTRSEQmnSA/nfD4v/Bx3+zE9mECzm/w8FdgQWZXaQ0sSOVtA6SEmKSkpLYuXOnCkMlYYxh586dJCUlVfgYmqcQCCLQ905IrAOT74QPL4bhH0JiSqgtKxpFVFrSmi9Se8LyL21PI0afDZTQ0KJFC7Kzs8nJyQm1KdWGpKQkWrQI0JXsg6CJgogkATOAROc8E4wx9xdrkwi8D5wC7AQuNcZkBcumo6bbKJsV/MWN8P4F8NdPoGb90NqUNRNqN7cxhfKQ2gsWvA/bl9lSAIoSAuLj42ndunWozVA8COYjYh5wujGmE9AZOEdEik0HxtXAn8aYY4HngCeCaE/l0Gk4XPI+bF0M7w6GfVtDZ4sxtqeQ2sv2ZsqDy92kLiRFUTwImigYS67zNt55FXccDgHec5YnAGeIv4pT4cQJ59pewp/r4Z2z4c+s0Nixcw3szym/6wigbkuo00qT2BRF8SKozmQRiRWRhcB2YIoxpvhckM2BjQDGmHxgD+Cn7nOY0aYfXPkVHNwN75wD23+vehuyXPMnlGPkkSepPW1PQYN8iqI4BFUUjDEFxpjOQAugq4h0KNbEV6+gxB1KREaJSKaIZIZVQKpFBlw1yc59PG5g1dcTWj8LkptAg7YV2z+tl+1p7FhduXYpihKxVMmwE2PMbmA6cE6xTdlASwARiQPqALt87P+GMSbDGJPRqFGjIFtbTpqcBFdNhsRkeO/8oqf3YGOMzTNI7Vn+eIILd1xBXUiKoliCJgoi0khE6jrLNYABQHEfy1fAlc7yMGCqicQByw3awsjvoPYx8N+LYNV3Ze9ztPy5DvZtLl9+QnHqt7E9DRUFRVEcgtlTaAZME5HFwDxsTOEbEXlIRM532rwNNBCRNcBtwOgg2hNcah9jewyN2sP4v8CSis/dGhCuUUPlyWQujojtaWTN0riCEjQKCg0v/Lia3Lwwqwag+CRoeQrGmMVAiXqvxpj7PJYPARcHy4Yqp1YDuPJr+Gg4fHoNHNoDp14dnHNlzYKaDawIHQ2pvWDZ57B7PdRLqxTTFMWTbxZv5tkpq8jZl8fDFxQPKyrhhqayVjZJteHyT6HdWTDxNpj5XHDOs37m0cUTXGi+ghJkDufbQpL7D2tPIRJQUQgG8TVg+AfQ4SL44QH7qkz3zO6NsHtDxYeietKoPdSop8XxlKAR43pwUQ9lRKC1j4JFbLyd9Caxtu0tSAyccV/Z+wXC+nLOn1AaMTG2t6DBZiVIuDShUONWEYH2FIJJTCyc+xx0uRx+fhbW/lQ5x10/C5Lq2OGwlUFqTzuaae/myjmeonjg6imoJEQGKgrBRgQGPmmHrX5+PRwokYZRfrJmQaueVnQqA1ePQ+MKShAo6imE1g4lMFQUqoKEWnDRW7B/O3xz69HFF/ZthV1/VKzekT+adoSEFHUhKUElElOQohEVharimC5w+r12DoOFH1b8OO56R5UoCjGx0Kp79egpHNwNWxZr3kUYoe6jyEJFoSrpebMdMTT5X7BrbcWOsX6Wfapv2rFybUvtaWdx27+jco9blRQcsRnlr/eBV3vB3DetSCghxT34SIU6IlBRqEpiYmHo6/bvp9fam1h5yZoFrbpBbCUPHKsO+QpTH4FNmdD9Rjv6a9Id8Ex7+OIm2DhPew8hwt1T0MsfEagoVDV1WsC5Y+3Na8ZT5ds3Nwd2rKxc15GLY7pAXI3IFYU/psKssXDKCDjncbjuJxg1HTpdCsu/gLcHwGu9be/h0J4QGxtduNIrdUhqZBA1orBt7yFuHf8bBw8XhNoU6DAUOv3FisKGXwPfb0Ml1DvyR1wCtDzVZkpHGrk58Nl1NhHv7MeL1h/TBc57Hm7/3QpxTKztPTx9vO09ZGfq42sVINpTiCiiRhSe/m4lXyzczLfLtoTaFMvAJ6BOS/js2sCfXLNmQXxNe7MLBqm9YOvSyPLDFxbCF9dD3l4Y9g4k1CzZJjEFMq6C62bY3kPHS2y9p7fO0N5DFaBDUiOLqBGFoektAGiSkhRiSxySatthqns2waR/BbbP+lnQsqv1lweD1F6AgY3FJ8gLY359Gdb8AGc/Glgy3zFd4PwXnN6Dk2nuij18eRNsmAMFWqOnMnGXudDxRxFB1JS5SE60H3V/OLiPXLTsCqf9C6Y/Du3OhJOH+W97YBdsWwb97w6ePS0yICbeis9xZwfvPJXFpgXww4PQ/lzIKGc12qTakDESTrkKNv8G88fBkk/ht/9CQrK9Fi2726B+i1Ntb0OpEEUxhZCaoQRI1IhCzUSb/bs/3Gq697kD1vwI39xmRaJuK9/tNvwKmOAEmV3E14Dmp0RGcbxDe2HCSDtJ0PkvVrxarAg0T7evsx6F1d/ba73hV/jpCcDY3kSTDtCqhxWJlt2hTvNK/TjVGR2SGllEjSjUiLeikJcfRj0FsENLh74Br/WxwdIR3/guX7F+FsQm2pt2MEnrBbOeh7xcO8VouDLpDjsHxIiJULN+5Rwzqbbtrbl6bIf2QvY8KxAbf4Xf/gNzX7fb6rRyBKKbFYvGJ1Re2ZFqhiavRRZRIwrxsTZ8crggDL+a9VvDoKdswHTWWOhze8k2WTOtGyM+yDGR1J7w8zP2Zti2f3DPVVEWfgSL/wf97q6cSrH+SKoNx55hX2DzSrYusTGXDb/Cup9hySd2W2JtmxV+zhhb50opQgPNEUXUBJoTHFE44kz4EXZ0Gg4nDYVpj8Gm+d7bDu2BrYsrt96RP1p2A4k9ujpI+Ydh7XQ7VLSy2bEGJt5u3Wh976j845dGbLx1M3W/AS55zwarb1kEF75h587YOAf+dzkcPlC1doU5RclrqgqRQNSIQnyc/WIeLghTURCBc5+F5KY22zkvt2jbhjlgCoP7VOwiMQWadapYEtveLTDtcRjbAd4fAi+fCovGV94A9fw8mHCVzakY+mbo3TUidgrTTpfCeWPhondg+3JbxkRxE+OOKYTWDiUwokcUwr2nAHYGtAtfs3WRvvMYZbR+lh0V1KJr1diR2tMmdh05VHZbY2xg+pMRVgx+esLWZRr6JjRoB59fBx9cDHuyj96uHx6wPaYhL4dnoLfdAOv6++0/VgwVoKinoBnNkUHUiEKc87hyJFx7Ci5a94Het8KC92DF13bd+lnWbeErMSsYpPaCgrySbixPDu+HzHG28Ny7g2yZiW7Xwz/mw+UTbILYyG+tj339LHi5O2S+Y5PNKsKq7+DXV6DrKGg/uGLHqAr63W2v3zf/hJyVobYmLNCZ1yKLqBEFESEhLiY8A83F6Xc3NOsMX/0Ddv5hx9EHcyhqcVp1B8S3C2nnH/DtXfDMCXZuiJgYOO8FuO13m0DmGWSNibX+9xt+geZd7I3y/fPLXyF27xb44gY7LPTMh4/qowWd2Di46G2bef7xlVY8o5yinkKIDVECImiiICItRWSaiKwQkWUicouPNv1EZI+ILHRelTSJsW8SYmPCv6cA1md+0VvWh/7e+VCYXzVBZhc169vsYFewubAAVn4L/xkKL6bD3Ddsst3I7+C6n+GUK0vvxdRvDX/7yorHlkXwSk/45SV73LIoLIDPR8GRg7aMRbBHX1UGtZvBRW/aUuST7gy1NSFHA82RRTCHpOYDtxtjFohICjBfRKYYY5YXa/ezMebcINrhJj5WIkMUABq2g7Mfs0/jEmtHBVUlqT1tdu/MsZD5NuzeACnNoP89kH4lpDQp3/FErHgcOwAm3gbf32PrDw15yY7x98fM52DdDJug1uj4o/tMVUnb06HvnTDjSdvL6/LXUFsUMmJ0SGrgFByBw7m2h3l4f7Hl/dDgWDimc1BNCJooGGO2AFuc5X0isgJoDhQXhSojPjaGw+EcaC7OKSMg62c7xLGqyyyk9rQ9gh/utxMDnfmw9eUfbd2lOs3hsvGw9FP7FP16X+j7LxtHKX7sjXPtEN2ThkKXK47uvKGg32jYMNsOoW2eXrr4VWNEA82wa519uNk4Fw7thrx93jd7182/IK/04/S6JXJFwRMRSQO6AL4qrfUQkUXAZuAOY8yyYNkRFyMURNLjioj1T1e0hMPRcPxgOOcJaN0XmpxYuccWsVnDrU+zwzenPWKnKR3yUtEX/uBumHC1FZHzxobmGhwtMbHWDfhaHxtfuHZqeGeJB4mo7Cns3WIf6Nb9ZMVg9wa7vmZDW5olMdkmR9ZuZmttJSTbudzdf2vZNp7vE5KhVsOgmx50URCRZOBT4FZjzN5imxcAqcaYXBEZBHwBtPNxjFHAKIBWrfzUBgrMlsj7YobqZhiXAN2vD+45khvBxeOsQHxzG7x5OvS6GU4bDV/fAns32bhFUp3g2hFMUpra+ML7F9gew4WvRabAHQVREVM4sMtWHXCJwI5Vdn1SXTv/Sc+b7QNWw+PC/v8fVFEQkXisIHxgjPms+HZPkTDGTBKRV0SkoTFmR7F2bwBvAGRkZFT4myVSzb+YkUr7wdZd9f29Noaw8EPI3QZn3G8n/ol02vSD0/4NP42xN4j0CHSFHQXVMk8hbx+sn10kAluXAAbia9nvcpcrrAg0PTn0SZblJGiiINaR+DawwhjzrJ82TYFtxhgjIl2xo6F2BsumGJHq9cWsTtSoZ5PSOlwEX98K7c6CXreG2qrK47R/2fjCpDtsfCGQuR+qCe48hQgK5wE2eXPvJpt46X5ttPknmxfYUYGxCXYQSP+7rQgck2572RFMMHsKvYArgCUistBZdzfQCsAY8xowDLhBRPKBg8BwE8RH+RjRSo1hT9vTbT0hY2wORHXBHV/obeMLo6ZHXXwhrB7IjIH9O+xNvvhN37W8f3vJ/ZKb2NImPW+GNqdZQYivUeXmB5Ngjj6aSdH8Gv7avAS8FCwbihMTiTGFaEQk7P2uFSK5sRWG94fYRL6hb1TPz+mHkGuCMbZQ46+vWJdPfrEyLvE1oU4L+2rawU6X63pfpwXUbg5xiSExvSqJmtLZYH9/YfW0okQfrftCv7tg2qM2IfGUEaG2qMoI2W8vPw+WTIDZL8P2ZVCrkb3u9dt43PRbWhdmFIm0P6JKFGJENNCshJ4+t9sSIpP+ZSdNanpyqC2qEqpcFPbvsPW25r5pXUGNT3LiVsMiIzM+RESdKERcsEupfsTE2iqyr/W21WVHTY+KOaCrTBNyVloX0aLx1kV07JnQ4yY7Ckx7AmUSVaKg7iMlbEhuBMPehvfOszkZoUpSrEKC+tszxg4Pnf2ynWc7NtFOXNX9RmjcPnjnrYZElShooFkJK9J621pSUx+2yxkjQ21RUAnKby8/z5ZMmf0ybFtq4wX97oZTr66S7N/qSHSJQowmrylhRu/bbHxh8mhb+6ZmA1vOwF3iwGM5MeXoa0+FkErtKeRut3OOzH3TJjo2PlHjBZVEdImCJq8p4UZMjB2a+vaZNqO7LGITvUXCJRr10qzvvHUfWycnDKnwT88YWztow6+w4RebSbzDmcDo2AHQ4zVo07/au9+qiqgShYisfaRUf2o1hJvmeVTPzLVzdB/OLXp/eL+zbp/HNuf9gV32RjnvLSsaab3tfBftzvKe9CjEBNxLLyy0c1G4BGDDbJtZDJBYB1p1s/GC4wdpvCAIRJUoxGigWQlXYuOsOFTUD56fZ91Qa36wgdZvR9tXvdZWHNqdZfMiQph96/eBLP8wbFlo7d8w2/YIDu2221KaQase9pXaw7qJIqyWUKQRVaIghEFWpaIEg7hEaNvfvs5+FP7MgtVT7GvB+zD3dYirYd1L7c6ybpf6rcs+rjH2Bp2bY8f6527zXj7o3Lwlxr5iYu2kUO7lGI45WMAjcdtIyI+Hyd8XtTUGti6G7EzIP2iP0+BYOOE8W1SuVQ/rFlO3UJUSVaKwbsd+Dh3RRAUlCqiXBl2vta8jB+3Uqqun2F7E6u9tmwbtrJvpmC5w8E/nhr8d9ud43/wLDpc8vsTakT416wMCphBMgZ0+1RTY94V2XUp+PgNjDxFbaGDh7KJ2GGjUHjKucnoD3W0pECWkRJUo/HngSKhNUJSqJ76G7RkcOwAGPgE7/ygSiHlvF832JbHWfZXcGGo1tjds13JyYysCyU3sco36ARcsXLlpD+e+OJP6tRJYcNeZQfygSmUQVaKgKAo2+NygrZ1E6fAB2L3e3vDLcaOvCBrPiwxUFBQlmkmoWWVzRxfq0L+IoBoVrFcUJZzRjkJkoKKgKEqVoO6jyCCqRKF7m/qhNkFRohb1HkUGUSUK7ZvWpnaShlEUJRRoTyEyiCpRiIsRCvRxRVFCgmpCZBBVohAbK+SrKChKSNCeQmQQVaKgPQVFCR0qCpFBQKIgIm1FJNFZ7iciN4tI3eCaVvnExsSQX2h0TgVFCQH6PBYZBNpT+BQoEJFjgbeB1sCHpe0gIi1FZJqIrBCRZSJyi482IiIviMgaEVksIunl/gTlIC7GFtbSL6eiKIpvAhWFQmNMPnAhMNYY80+gWRn75AO3G2NOALoDN4nIicXaDATaOa9RwKsBW14BYh1RyC/UoniKoii+CFQUjojIZcCVwDfOulLnBTTGbDHGLHCW9wErgObFmg0B3jeWX4G6IlKW2FQYV09B4wqKoii+CVQUrgJ6AI8aY9aJSGvgv4GeRETSgC7AnGKbmgMbPd5nU1I4EJFRIpIpIpk5OTmBnrYEew7aKqlHClQUFEVRfBGQKBhjlhtjbjbGfCQi9YAUY8yYQPYVkWRsTOJWY8ze4pt9nc7H+d8wxmQYYzIaNWoUyGl98sr0PwCY/ceOCh9DURSlOhPo6KPpIlJbROoDi4BxIvJsAPvFYwXhA2PMZz6aZAMtPd63ADYHYtPRsHXPoWCfQlEUJSIJ1H1Ux3nKHwqMM8acAgwobQcREexIpRXGGH8C8hXwN2cUUndgjzFmS4A2lRtXTCEnNy9Yp1AURYloAi0EFOcEgC8B7glwn17AFcASEVnorLsbaAVgjHkNmAQMAtYAB7Cxi6DRKCWRLXsOkbNPRUFRFMUXgYrCQ8B3wCxjzDwRaQOsLm0HY8xMfMcMPNsY4KYAbThqXKKwXUVBURTFJwGJgjHmE+ATj/drgYuCZVSwaJScCKA9BUVRFD8EGmhuISKfi8h2EdkmIp+KSItgG1fZNEqxorBDYwqKoig+CTTQPA4bFD4Gm0fwtbMuonCJgvYUlGCyYsteTnl4in7PlIgkUFFoZIwZZ4zJd17vAhVPGAgRDR33kSY0K8HkrZ/XsXP/Yaat3B5qUxSl3AQqCjtE5HIRiXVelwM7g2lYMHD1FBQlmMQ6v6pCffpQIpBARWEkdjjqVmALMIwgDx8NBo1VFJQqwFV4sUBLtCsRSKBlLjYYY843xjQyxjQ2xlyATWSLKFzuI0UJJjHilGjXnoISgRzNzGu3VZoVVUTdmqUWdlWUSiFWq/EqEczRiEKpiWnhSEqSioISfFw9BS3Gq0QiRyMKEfeVdz3BKUowcX3P1H2kRCKlZjSLyD583/wFqBEUixQlwtFAsxLJlCoKxpiUqjKkqoiNEVrUUz1TgofbfaQ9BSUCORr3UURSUGhYv/MARp/ilCARp+4jJYKJOlFw8csfEZd7p0QILvdRvoqCEoFErSjk5uWH2gSlmhKnQ1JvKclWAAAgAElEQVSVCCbqRMGV1VyvZkKILVGqK7Gx2lNQIpeoE4UruqcC8NyUVSG2RKmuFPUUCkNsiaKUn6gThZ37DwOwbPOeEFuiVFdiY+zPSnsKSiQSdaKQkmRH4WoimxIsNKagRDJRJwrX9m0DwF+7pYbYEqW6oqOPlEgm6kShtlP/aPu+QyG2RKmuuDOatfiREoEETRRE5B1nTuelfrb3E5E9IrLQed0XLFt88XFmdlWeTokitMyFEsmUWubiKHkXeAl4v5Q2Pxtjzg2iDYpS5WhMQYlkgtZTMMbMAHYF6/iVwbyssDZPiVA0pqBEMqGOKfQQkUUiMllETqrqk2dm/VnVp1SigDhnSKrmKSiRSChFYQGQaozpBLwIfOGvoYiMEpFMEcnMycmpNAMmLtmsXXyl0nGNdj6igWYlAgmZKBhj9hpjcp3lSUC8iDT00/YNY0yGMSajUaNGR33uhy/oAMDSTXs578WZR308RfHEqZxNfoH2FJTII2SiICJNRezPR0S6OrZUSenSvCMF7uXlW/ZWxSmVKERjCiXRcuLhT9BGH4nIR0A/oKGIZAP3A/EAxpjXgGHADSKSDxwEhpsqmuQgKT62Kk6jRDlHtKdQgvxCQ4JWEwhrgiYKxpjLytj+EnbIapVzYZfm3PuFz/QJRak08jWmUAKN4YU/oR59FBJqJQYzPUNRLEf0BliCfB2RFfZEpSgoSlWggeaSaO8p/FFRABZn7w61CUo1RG+AJdHge/gTtaIwZujJ7uXzX5pFzr68EFqjVEeOqKukBOo+Cn+iVhSGd23l9f7UR38IkSVKdUVHH5VEe0/hT9SKAkBqg5pe73UMtVKZ6A2wJOo+Cn+iWhQm3tzH6/2/P10cIkuU6oiWubB4Zh9pPajwJ6pFIbnY0NRP5uscC0rlof7zkqhQhj9RLQqKEgxcT8Y681pJNHkt/Il6UZhwfY9Qm6BUU3T0UUk0+B7+RL0odGxR1718zklNQ2iJUt3QQHNJtKcQ/kS9KCTExZA1ZjANkxP5dtnWUJujVCN0pE1J9JqEP1EvCi525NrktbTRE0NsiaJUX7T3FP6oKPhA8xUUJTjoiKzwR0XBBwc8JuFRFKXy0J5C+KOi4HB6+8bu5e17D4XQEkWpvmhMIfxRUXAY2KFo5NG/Jixmw84DIbRGUaon6j4Kf1QUHIZ0bu5ezlz/J32fmsaijbv5aO6GEFqlKNULHZIa/qgoOCTExfDCZV281g15eRZ3fbYkRBYp1YE/9x8OtQlhhZa5CH9UFDw4rkmyz/U6GkmpKFs1PuWFFsQLf1QUPGjT0Lco5B7Or2JLlOrC4Xy9CXqigebwR0XBg4Q435djxea9VWyJUl3IU1HwQoekhj9BEwUReUdEtovIUj/bRUReEJE1IrJYRNKDZcvRMmfdLjbvPsiijTqXs1I+asTHhtqEsEIL4oU/wewpvAucU8r2gUA75zUKeDWIthwVz05ZRc8xUxny8qxQm6JEGAfU9eiF9pzCn6CJgjFmBrCrlCZDgPeN5Vegrog0C5Y9ihIK3p65LtQmhBUqCuFPKGMKzYGNHu+znXUhpX3TlFK3z8vahTGG8XM3YIz6R5XSmb/+z1CbEFbkaQmZsCeUoiA+1vm8y4rIKBHJFJHMnJycoBr16Q09OaFZba91nkNVN+8+SPfHf2T0Z0u4ZfzCoNqiRCaeX+JRfduEzI5w5JCKQtgTSlHIBlp6vG8BbPbV0BjzhjEmwxiT0ahRo6AaVSsxjsm39KFRSqJ7nWeXNzcvn217bZntrxb5NFdR3OgQTG/UfRT+hFIUvgL+5oxC6g7sMcZsCaE9Xnx2Q09Ob9+Yi9JbsN6jDtI9n/scTKUoPlF3iTfaUwh/gjkk9SNgNnC8iGSLyNUicr2IXO80mQSsBdYAbwI3BsuWitCyfk3eGXEqny7ILrVd2uiJmqCk+EWfjL05dESvR7gTF6wDG2MuK2O7AW4K1vmrkuw/D9Cmke9s6OrMT6ty+G7ZVh678GQArho3l2krc8gaMzjEloUPKgre5OVrTyHc0YzmMnDd8EpjyaY9rNy6L+rGpF/5zlw+nLOB3Qds0bdpK4M7CCASUXeJN9pTCH9UFMpg2Cktymxzy/iFnD12Btf/d0EVWBR+vPWz91j8fM1adXNQRcGLQ9pTCHtUFMrAXz0kX8xYVfSkvGXPwYgco15QaEgbPZEJ80uPpew9dMS9vHN/nte20m6EuXn5bNlzsMT69Tv388uaHeW0tiRTf9/G8jCpVRUjcPCw3gRdxAjkaU8h7FFRqCCDTm7KVb3SSqzfsucgx90zmR6PT+WiV3+p9PP+sHwbl74+u9KP62JRtq3vdMcni0pt1/GB793LW/Z4l4cu7UbY4f7v6PH4VPf7w/mFdHvsB057ajp/eWsOO3PzOHSkoMLB+5HvZjLohZ/9bjfGsKuK5jiomRAXdT2FIwWFZP/pe9bCxLhY7SlEACoKATDln32ZcWd/r3U5+/K4/7yTSrTt8fhUDnu4T/zd3N6fncX5L80ssb6g0JTqh77m/UzmrNvFxl3BmS40RnzlFHqzbPMer/fTV+bw3i9Z7vcbymHbd8u2uvM+AE555Afa/9+3HHfv5ICPUR5e+2kt6Q9P4ZyxM4JyfE9y8/L5efUO5q8vrdpL9eLhb5bT+4lpPoU3KT5GewoRgIpCALRrkkKrBjX56Nru7nXzsgJzDfm7Qd735TIWZ+9hz8EjXuv//uEC2v/ft2Ue92iqTU5fud1LVAoLDXd9tphtew/xcWZR5RFf81Tvz8tn8Aslxez+r5a5l4e9NputxXoPXy7cxKvT/3C//32rdfH846PfArL54OECvly4CWMMz05ZxfA3Su8t5ebl892yraSNnkja6Inu9U98+7tz/n2ALViXNnoiq7btC8iOivDpgk1BO3a4MdNxAe4q5lIE21PQ0Ufhj4pCOejRtoF7ys7T2zcGYOLNvUvdJzevaETSf39dz4s/rvba/uKPq90/FGMMk5duBWDfIW+xKE5sjP8n+gnzs0kbPZGduSV/mAAjxs2jz5PTmLtuF/+asIjnf1zNR3M30u2xH/lwTtGc1AuzS5YKP+n+70q1y0X3x3/0en/L+IXuGzJA1o79ZR4jbfREtjszlw1/Yza3jF/I3HW7eOHH1fy6dhc5+3x/PrBuquv+M9/9fvs+3zOgnf+SrXx71nMV6znMXbeLUe9nljo7XzTVyEqKs6XCfY0ySoqP0dFHEYCKQjk5r2Mznr64E69ebqd/OOmYOqW2d93cjTHc+8VSnpmyihVbigKhb81cx/H32p7Bha8UxSB+WpVDbl4+I9+d557s3HPS8wOO337X/sMcOlLAgg1/um9MrnjAC8UEqDiXvD6bjzOzed5Pu5s9nuLzCwq9gsuB8u8Ji33GJxLjAptn4HZn30XZ1mXlEk2wQWWwLrr3Z2eVemPu+qgVqSeHdfRa74p/NK9bw+d+hU7g3d+1vPq9eXy/fBu7D/q/Njn7omee5sR4e0vx5QLNzSvwekhSwhMVhXIiIgw7pYXXTW3spZ05ublvcbji7bkADHy+KPjpuezJQo9JfP7+4W90uP87pv6+nbZ3T2L7vkN863FDfGnqGgDSH55C+//7lqGv/MIzU1Z6He+92etLBH13+Ok9+MMVK7j3i6VeweVA+HDOBv6XudHnSKb//Lqez8rIFoeSo1Xe9YhduK7XcfdO5r4vlwVUptrVv4qPtUsXOUOON+32HhGVtWM/Bw7nu6difXGqb1Fw3fxKiwNFWkwh+88DFR7BlZxo82F93fxd3z0VhvBGRaESuKBLc764qVepbVw+bH8Ud/WM7NXa6/38rD9Z6eH3nrikZJmorxeVXDfy3XkAfLvU+tczHvmhVDuKc/9Xy8gvKGT8vI0ltl2SUXoOx92fL/G7berv27nt49JHOAGkNazp19X00Vxvm+ZmlX3zdfUljhQYduTmcdAj4dB1Yy8sNPR7ejo3frCAPQdsD8AVgJ+yfBsv/Lja7Y464kwv6Stx8b5zTwRgSOeQV4QvF72fmFbqCK7ScGVwf/5byTiKy+V6QEUhrFFRqCRiY4QhnY/xuc1XwLY4pxS7WScnertXCk1Jd5BnABV8B7Vnr91Jbl4+1/93foltgXLsPb5HAj05rBML7zvTa939551Y4fP44uPMbPo9Pd3v9gOH82nTqBZgb9ilcc/nS7xGg2U88gNveiTeXft+JgD7nJvW9JU5bpdZXn4hxhiufT+TZ6esouujP/p053nS89gGJMTGuF0qFcEYwx85uV7rXp3+B98v2+pnj9AS58S6Zq4umXNSq5RehBI+qChUIs8P70LWmMGsfWwQKx8pmom071PTyn2sFxz3kIubPgwsW/q2j0vO8dAhwOCwi04t65bZZtDJTQGoWzOB6Xf0AyC1QU2uKtbDCTYrtuxjbU7ZQWuAD+ZsKDUp7+fVOzDG0OnBIjeZ50ir1ndN8mq/1uNmvTh7j9vl4sr1EITDBYW8/tPaEueal7XLXR6kND6au5EznvmJOWt3utc98e3vjPpPxUU+mHRtXR+Ak3y4U12xtO/LEG8ltKgoBIGYGCExLpbzO/nuOQSTzyph+OO/zzm+zDYvXpbuXk5rWIvlD53ND7edBsAZjpsgUL76e+mut9IoK0HwumKT3HjGbXzx4NfLAz73mR4jlu79YimDXviZh79Z7lMEPMkvKOTi12Zz5TtzOVJQ6NXjKM63To/A9VDg2Tuc+vs2floVvHpTFRk1lRRve7gzfNjV7zg7F0qbhrWOzjAlqKgoBJG4UoaNHg2VOZvXj7ef5vX+8aEn06NNgzL3Kz4ktmZCHPGx9ut0Zc+0ctnQsYV3z6RV/Zrl2r807hp0Qrnar95+dPkKnsHu3zYU5bK4SntMXrKF12dY0ViUvYd290ym7d3ePRBPfnNKpezIPVxidNXIdzO58p257MzN49P52Vzz3ryjsr04ZZU68UVsKcmP9ZMTgKI4jBKeqCgEkbNOaur13vUk7eLJYR25tk9rviwjSF2cnm3LvmmXxcMXdCBrzGDaNkrm4+t6cG7HZtx/3olc1rUVIkLSUfjB+x7XiG9v7eN3+6pHBpZY9+VNvejQ3E6DenXv1gw/tSVD04sCtMsePJvfHz6Hm/q35ed/9ef1K07xe/xv/uGdO3L7mceVaq9nj67XsQ1LbVseaiTEuuMdpz01HYAbPljAU9+tLNHWX2/hxv7Hupfb+BGPy9+ey+2fLOKHFdvd6w4dKSgxosoXhYWGWWt2+OwV/OfX9WXuXx5qJdiYwv4oqyYcaagoBJFzOjRlwvU93O8bJRdN8Zk1ZjCXZLTknsEnurvcvuhd7CY1/94B9G3nPSXp7w+fQ3HuHVz6E7KrKw/WD/zSX9K94gEDTmhS6v5l0b5pbZ4f3rnEDbpTizo+iwx2almXD6/tzoieaVx6akvGXNSRZy7uREZqPT6/sSe1EuNIio/lzrPb07J+Tc460b99Jzarzc//6s/axwYB0N2HiH52Y0/38tW9W7uz1Z/8tuQNG2Bol/KPIBrYoZl76PLh/MJS80Zu/GA+785ax3H3TvbKVq9fK77M83jmvbjE5V8TFtNrzNRSM98fm7SCjg9+z1/fmsOkJSUD14uz9/jYqyRnPDOduz6zI82M72nWAaiZYK+Fjj4Kb1QUgkxGWn33DbpGgu+bf2Kxm6Sr53DacY1468oM9/o6NeJpkJxIjIfrJmvMYJLiYzm+SYrXMU5oVrvEeXof25AlD5zFuBGn0rIMF80zl3Tyu+25S/1v82RI5+Z0aF6HZQ+ezXsju7L60YF8dqP9bBmp9QD7GV3UTorngfNPcoukiDDhhp50aVWvxLGlFDdFTIzQsn5N93Wq4UN00z2OmRgfQ50a3jff4iXTH76gg89z+SqK6CIhLsbrhv3slFV+2363bBsPfL2cw/mFtLtnMmmjJ7Jo425Gf+Z/WK8viua2sL2G7X6yvl+etoY3Zqx1jwTyl4cRCH/k7OejuRtKrC/u7kpJsj2Fp7/3fx3ClbU5uaQ/PIXNAfS+Ih0VhSrgmj5tyBoz2G8Z7niP9Vf1SqNTy7pkjRnMeyO7evUiZo0+3b387CWdvJ7CX7k8nX7H2xts87o1fLqYGqckkpIUT/8AAsGJcbGsfWwQf+nWigEnNOGOs4pcMN0DiDl4UisxjtOOa0R8bIw7FvHMJZ24sEtz3vxbRhl7B87Mf/dn5r/7l1hfliss70ih+4bl4tyOzVj96EA6NK/NpJv7eP0fPEeW3VNGzGJwx2aBmO6TIS/Poryx3vucGlQuV03x/BdXLajiLizPPBqXywug75PTfIrZgcP5fJK5kf/NKxKD4i6oAc/95PXeFXMqbUjqnLU7g1bs8Wj437yN7Np/mC8Xbg61KUEnaNNxKr7xNVVls9pJ/K1HKpdktKSDj6F8H13bnY8zN1LLo6cxNN37SbZto2Tevaqr17qUpDj2Hcrn6t6teXvmOlIblG/UR0yMeM08d2O/Y9l98Aj1ayWU6zi+SG1Qi+cu7XxUx2hRrwbZfx7kiu6p3DKgHQ093HOexMV4i0LxIbftmiSXqMnTrXUD4mNj+OYfJWMjntnscbG+Bcf1f375L+kczs8sM4eisvhjey6PT17BVqdmVO4hewPee+gIh8qY26Gw0BATI25BAZv78sKPq7mtWFzmHx/+xo+/b/dad/BIgZeIlTZUeO+hI9ROKukau/SNXwHfv5NQkug8FETD7IoqCmFATIzw0BDf7gmwhfh6VCC4PO+eASzZtIeM1Hr0aNOAvsc1KnunUoiJkUoRhMrii5t68dPKHHepCn/ULOa2G35qSwAWP3CWsz2Omgm29IVrZIw/V5+L54d3pl5Ney3e/FsGK7fu5c8DR3yW2njxsi4BVb4tjW6t6zNnnc3YvqJ7qt8g8O9b93k99f/lrTmM6tuGP7bnlriJ+9r3xamrWbKpZCzhoa+Xc2GX5pzcog778/J9HmtnbuA1npZu2kPPthUL6m/be4gYERql+H4ICAZbndFjr0z/g9vPKnvIdiSj7qNqTFJ8LKem1UdEGHBik3LNIhcJNExOLFMQABrXTuK5S2329QfXdHOLQu2keK+n1dWPDgr43EM6N3eL7JknNuHvp7fj3sEncO/gE/j4uh5ebZPiY73cb2OGes/7HciUr/+9ppt7+ZwOTd3DnS/r2qrMfd+YsbZMQQAY9MLPXgUHPXln1jrOe2kmew4e8Vspt8+T00qEmd+Y8YfP4PP/fJRNCTTTudtjP3Lqo+Ur13K0nJpmk/L6tqu80WnhSvW6SyiKHy7s0oK6NRPodWzDUoPUgN88jX8OOI5revvP2BYRrunTxp3V68nfT2/nXh7etZVXAcWnhnUsNYHvsxt7uv3xAKek1nPXYvIUm4rQs20Dbj7D2tayvu9KsZ54Znv7YsxkWx7d1aN8bNLv7vLkAO+NtC5OX775v3tk7V/zXunlyKsaV69wfxRMrxpUURCRc0RkpYisEZHRPraPEJEcEVnovK4Jpj2KUhZZYwbz0ajuPrfdMqAd955bObWdXnTm5QArJh1b1OXpizsx4foe7kqjYHMu0ouNvkqKj2XAiXawQEpSPFf2SK2wHR9e251/DrCisHFX5Y2sqVvT91DaU9OKPkva6Imc/dwMvly4iR25eUxfWZQF/cOKbbS5exL5zpDawkLD7D92st+jN7GylCKT+QWF3P/l0oByNQIh1qmqO3ddZFW8rQhBiymISCzwMnAmkA3ME5GvjDHF6wj8zxjz92DZoSjhwv+deyJpDexQ4LSGtXjpL11o17hoKLHLjbT0wbM5nF9IXn4BKR7uLVcMBODZSzrz73MOkRAXw4NDOvDgkA70GjPVfRP88Npu/OXNOaXac/eg9oDv4b2vX3GK1yRF5eXYRsklAs3rduznjGL5Lyu37eOW8Qvp1MJ36fnXZ6zlpv7HcsMH8/lumXew/uyxM1jwf2f6jHP9unYX781ez3uz1/PiZV04LwQlZyKVYPYUugJrjDFrjTGHgfHAkCCeT1HCmqt7t/a6KZ7b8RiOb5ris21CXIyXIIB3DCQpPrbEaDLPwGvPtg1pX+zYxTO7L+/uu4dx3WltODOA5MVTUkvmj5S2bfkW/3M0rNme63O9qzdQXBBcpD88xev93kNHKCw0xMUWCV2gU776Ysueg3b2Pw9PVlmzIkY6wRSF5oBnNCnbWVeci0RksYhMEJGWvg4kIqNEJFNEMnNyglcATFEimSuK3eS/vbUvIzzqUA3p3Jznh3dmyQNnkTVmMDU9hp62qFcUT7i6d2uvBEkXxTsU89f7n6e8Rb2SyZGuYK0v/Pnqv1q0mfU7S6+C+9yUVQx9ZRZ/5OTS8YHvaXP3pKMaAjx//S53ifUej0+l62PeU8ve8/nSCh87EgimKPiK5hWPHH0NpBljOgI/AO/5OpAx5g1jTIYxJqNRo6MbVqko1ZULfZTiuNsjua5Vg5oM6dy8RA8EYOa/T2fZg2fz4bXdaJyS5PP4c+4+g+//2ded6OdyhQH8dGc/93LWmMEMOrkpH3iMmBo34lQuzfD5zFeCJR5uMiiqG+WP539czYINuznjmaJkOc+hwf5mRdy8+2CJxLy/f7iAi16dzWOTVnitL/RIwPhqUfAT2A4dKeCOTxaxba/vucWDSTBFIRvw/Ba0ALyupjFmpzHGlXL5JuC/ypmiKKUSEyNk3jvAqxZWeYYh10qM88odyLx3ADPu7O8uRZIUH8txTVKYf++ZXJTegv9c3c29LbVBLdo0rOWeaEpEvIoL9m/f2N37cMUy/JGSFM94P8H+irBk0x5u/3gRaaMncsN/51NQaNh94DA9x0zlhR9X88r0orlLvllsZy9cummP13wX/noyk5dsIW30xEq/eX+/fBsT5mfz6MQVZTeuZIKZvDYPaCcirYFNwHDgL54NRKSZMcY1h+T5QNVfAUWpRvjK6p5+R78KVSZtmJwIyfYpf/fBogzkhLgYd22st6/MIN8ZOjrVmWzJk9PbNy5RWG9U37aM6tuWvYeOlJj32xUHKauUyoTrezDstdkBf5ZPnfnAJy/dWqJU+ZPfrmTvwXxuHVA0bDhz/Z8MebloKK0reQ1swcU123MpKDTc+4V1JT3z/UoevfBkCgoNSfGxpI2eyJU9UnmwWFLq5t0HqV8rgdl/7GT19n2M6tvWp727nPIkociglopMpBHwwUUGAWOBWOAdY8yjIvIQkGmM+UpEHseKQT6wC7jBGPN7acfMyMgwmZmZQbNZUZSqobDQlCgHfnXv1vyfM+z30YnLvaZLbZicyA7nZjnh+h6s27GfOycsrjqDA2TciFO5ypkb3bNcx5sz1vLopBXUrRnPbmfu73WPD0JE+HXtTm7/eBHX9GlNWsNaXDWuaG6Myir5ISLzjTFlFhsLapkLY8wkYFKxdfd5LN8F3BVMGxRFCU98BbMvzijK7r5n8IleovDOiAx3IlxyUhxD01u4ReGqXmk0TE7kr91asXzLXvdw3HM7NnO7hKoKlyAALMnew5Y9B72mT3UJAsCeg0dYumkvl79t7fU1858xpsyEy8pEax8pihIyxo/qTot6NXyOVgIYfHIzJi7ZwjsjMrxm6Gvf1Ls0/MZdB7j/vJMAaOlxrCt7pnFtnzZerqCq5LyXZpa6vfNDU+jmIwPek28Wb6nSPIuguo+CgbqPFCV6MMZgTFGvYuLiLdRIiOH09jaP4oGvlvHuL1lM+Wdf2nnMKfL71r20ql/TPex2yvJtdGpZh68WbuaFH1ez95B/X/2tA9ox9oeKzy8RDL67ta/fnJZACdR9pKKgKErUMW3ldi+/vSd/PDaIbxZvpmvr+tSpEc+J9xUVAPz21j6cM/Znr/b/OP1YXpy6pvhhgsIN/dry73NKH73lj7CIKSiKooQj/Y9vTNaYweTlFyAIx907GbBVZ2NjhCGdi3I+nr64E3VrxNPv+EZe82cse/BsYmOEXfsPBywKqx4ZyPM/ruLlaX+U2DbslBZMmG9HSb0/siv1ayVw7ove7qdXp/9RYVEIFBUFRVGiFteESd/e2ocpy7Zx3Wklh4gWL22+5tGBQNEES8fUrcEvo0+nSe0kYmOEW8b/5rMK7JlO+fo7zjqeK3ukeWVKz7izP41SEt2isGrbPq7p06bEMYqXLgkGKgqKokQ97ZvWLhG89oev2faOqVtUJuThCzrQuWVdruyRxsLs3dROiqdJ7USvuccb104qtn+S13GHO/NkZI0ZTNaO/fR7ejqA1xS8wUJjCoqiKCFg1/7D7Nqfx7GNy376f+Sb5azcto//XN2tzLb+0JiCoihKGFO/VkLA09tW1jwegaAzrymKoihuVBQURVEUNyoKiqIoihsVBUVRFMWNioKiKIriRkVBURRFcaOioCiKorhRUVAURVHcRFxGs4jkAOsruHtDYEclmlOd0WsVGHqdAkOvU+AE61qlGmMaldUo4kThaBCRzEDSvBW9VoGi1ykw9DoFTqivlbqPFEVRFDcqCoqiKIqbaBOFN0JtQASh1yow9DoFhl6nwAnptYqqmIKiKIpSOtHWU1AURVFKIWpEQUTOEZGVIrJGREaH2p5QICJZIrJERBaKSKazrr6ITBGR1c7fes56EZEXnOu1WETSPY5zpdN+tYhcGarPU1mIyDsisl1Elnqsq7TrIiKnONd9jbOvVO0nrDz8XKsHRGST871aKCKDPLbd5XzulSJytsd6n79HEWktInOca/g/EQlswoEwQ0Raisg0EVkhIstE5BZnffh/r4wx1f4FxAJ/AG2ABGARcGKo7QrBdcgCGhZb9yQw2lkeDTzhLA8CJgMCdAfmOOvrA2udv/Wc5Xqh/mxHeV36AunA0mBcF2Au0MPZZzIwMNSfuZKv1QPAHT7anuj81hKB1s5vMLa03yPwMTDcWX4NuCHUn7mC16kZkO4sp1kKjcYAAAa7SURBVACrnOsR9t+raOkpdAXWGGPWGmMOA+OBISG2KVwYArznLL8HXOCx/n1j+RWoKyLNgLOBKcaYXcaYP4EpwDlVbXRlYoyZAewqtrpSrouzrbYxZraxv+T3PY4Vcfi5Vv4YAow3xuQZY9YBa7C/RZ+/R+dJ93RggrO/53WPKIwxW4wxC5zlfcAKoDkR8L2KFlFoDmz0eJ/trIs2DPC9iMwXkVHOuibGmC1gv8hAY2e9v2sWLdeysq5Lc2e5+Prqxt8dt8c7LpcI5b9WDYDdxpj8YusjGhFJA7oAc4iA71W0iIIvX1s0DrvqZYxJBwYCN4lI31La+rtm0X4ty3tdouF6vQq0BToDW4BnnPVRf61EJBn4FLjVGLO3tKY+1oXkWkWLKGQDLT3etwA2h8iWkGGM2ez83Q58ju3Gb3O6ojh/tzvN/V2zaLmWlXVdsp3l4uurDcaYbcaYAmNMIfAm9nsF5b9WO7Buk7hi6yMSEYnHCsIHxpjPnNVh/72KFlGYB7RzRjYkAMOBr0JsU5UiIrVEJMW1DJwFLMVeB9eIhiuBL53lr4C/OaMiugN7nO7ud8BZIlLPcROc5ayrblTKdXG27ROR7o7P/G8ex6oWuG5yDhdiv1dgr9VwEUkUkdZAO2xw1Ofv0fGNTwOGOft7XveIwvlfvw2sMMY867Ep/L9XoY7SV9ULG91fhR31cE+o7QnB52+DHeWxCFjmugZYP+6PwGrnb31nvQAvO9drCZDhcayR2KDhGuCqUH+2Srg2H2HdHkewT2BXV+Z1ATKwN8o/gJdwkkYj8eXnWv3HuRaLsTe3Zh7t73E+90o8Rsf4+z0639O5zjX8BEgM9Weu4HXqjXXnLAYWOq9BkfC90oxmRVEUxU20uI8URVGUAFBRUBRFUdyoKCiKoihuVBQURVEUNyoKiqIoihsVBSXsEJECp9rmIhFZICI9y2hfV0RuDOC400VE5wn2QETeFZFhZbdUogUVBSUcOWiM6WyM6QTcBTxeRvu6QJmiECo8MnQVJexRUVDCndrAn2DryIjIj07vYYmIuCrdjgHaOr2Lp5y2/3LaLBKRMR7Hu1hE5orIKhHp47SNFZGnRGSeU9TtOmd9MxGZ4Rx3qau9J2LnqHjCOeZcETnWWf+uiDwrItOAJ8TW0f/COf6vItLR4zONc2xdLCIXOevPEpHZzmf9xKmhg4iMEZHlTtunnXUXO/YtEpEZZXwmEZGXnGNMpKggm6JYQp35py99FX8BBdgM0N+BPcApzvo4bLlggIbYDE8B0vCu7z8Q+AWo6bx3ZY1OB55xlgcBPzjLo4B7neVEIBNb//92ijK/Y4EUH7ZmebT5G/CNs/wu8A0Q67x/EbjfWT4dWOgsPwGM9ThePeezzQBqOev+DdyHram/kqJpdOs6f5cAzYut8/eZhmLLL8cCxwC7gWGh/p/rK3xe2q1VwpGDxpjOACLSA3hfRDpgBeAxsdVdC7Glgpv42H8AMM4YcwDAGONZ/99VmGw+VkzA1pPp6OFbr4Ot0zMPeMcpbPaFMWahH3s/8vj7nMf6T4wxBc5yb+Aix56pItJAROo4tg537WCM+VNEzsVOyDLLlrUhAZgN7AUOAW85T/nfOLvNAt4VkY89Pp+/z9QX+Mixa7OITPXzmZQoRUVBCWuMMbNFpCHQCPt03wjbczgiIllAko/dBP9lhPOcvwUUff8F+IcxpkRhP0eABgP/EZGnjDHv+zLTz/L+Yjb52s+XrYKdWOUyH/Z0Bc7ACsnfgdONMdeLSDfHzoUi0tnfZxI7VabWtlH8ojEFJawRkfZYV8dO7NPudkcQ+gOpTrN92CkPXXwPjBSRms4x6pdxmu+AG5weASJynNiqsqnO+d7EVrxM97P/pR5/Z/tpMwP4q3P8fsAOY+vrf4+9ubs+bz3gV6CXR3yipmNTMlDHGDMJuBU7fwEi0tYYM8cYcx+2/HRLf5/JsWO4E3NoBvQv49ooUYb2FJRwpIaIuFw1AlxpjCkQkQ+Ar0Ukk6KYA8aYnSIyS+xk8pONMXc6T8uZInIYmATcXcr53sK6khaI9dfkYKc27AfcKSJHgFxszMAXiSIyB/uQVeLp3uEBYJyILAYOUFQ++RHgZcf2AuBBY8xnIjIC+EhEEp1292LF70sRSXKuyz+dbU+JSDtn3Y/YSriL/Xymz7ExjSXYKqU/lXJdlChEq6QqylHguLAyjDE7Qm2LolQG6j5SFEVR3GhPQVEURXGjPQVFURTFjYqCoiiK4kZFQVEURXGjoqAoiqK4UVFQFEVR3KgoKIqiKG7+H0vQwL40RizhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_losses()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('char_rnn_2_p0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('English', 141),\n",
       " ('Russian', 97),\n",
       " ('Chinese', 54),\n",
       " ('Italian', 43),\n",
       " ('Japanese', 38),\n",
       " ('Greek', 25),\n",
       " ('German', 22),\n",
       " ('Czech', 12),\n",
       " ('French', 11),\n",
       " ('Dutch', 11),\n",
       " ('Spanish', 10),\n",
       " ('Polish', 10),\n",
       " ('Korean', 4),\n",
       " ('Vietnamese', 2)]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob, targ = learn.get_preds()\n",
    "Counter(data.classes[_.item()] for _ in prob.argmax(dim=1)).most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rebalancing\n",
    "### Less is more\n",
    "\n",
    "Even though the balanced set is a subset of the training set (and throws away a lot of data), the model performs much better on the balanced validation set with it.\n",
    "\n",
    "This is because on the whole training set heuristics like \"when in doubt, guess Russian/English\" and \"it's almost never Vietnamese\" are good, but are terrible on our validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = (TextList\n",
    "     .from_df(df, \n",
    "              cols=[2], \n",
    "              processor=processors)\n",
    "     .split_by_idxs(train_idx=bal_idx, valid_idx=valid_idx)\n",
    "     .label_from_df(cols=0)\n",
    "     .databunch(bs=1024))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sanity Checking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Korean': 30,\n",
       "         'Italian': 30,\n",
       "         'Polish': 30,\n",
       "         'Japanese': 30,\n",
       "         'Dutch': 30,\n",
       "         'Czech': 30,\n",
       "         'Irish': 30,\n",
       "         'Chinese': 30,\n",
       "         'Vietnamese': 30,\n",
       "         'Spanish': 30,\n",
       "         'Arabic': 30,\n",
       "         'German': 30,\n",
       "         'English': 30,\n",
       "         'Russian': 30,\n",
       "         'Greek': 30,\n",
       "         'French': 30})"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(_.obj for _ in data.valid_ds.y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Korean', 500),\n",
       " ('Italian', 500),\n",
       " ('Polish', 500),\n",
       " ('Japanese', 500),\n",
       " ('Dutch', 500),\n",
       " ('Czech', 500),\n",
       " ('Irish', 500),\n",
       " ('Chinese', 500),\n",
       " ('Vietnamese', 500),\n",
       " ('Spanish', 500),\n",
       " ('Arabic', 500),\n",
       " ('German', 500),\n",
       " ('English', 500),\n",
       " ('Russian', 500),\n",
       " ('Greek', 500),\n",
       " ('French', 500)]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(_.obj for _ in data.train_ds.y).most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>y</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Russian</th>\n",
       "      <td>486</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>English</th>\n",
       "      <td>459</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Japanese</th>\n",
       "      <td>383</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Italian</th>\n",
       "      <td>357</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>German</th>\n",
       "      <td>330</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Czech</th>\n",
       "      <td>295</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dutch</th>\n",
       "      <td>195</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>French</th>\n",
       "      <td>172</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spanish</th>\n",
       "      <td>170</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chinese</th>\n",
       "      <td>158</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Greek</th>\n",
       "      <td>153</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Irish</th>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Polish</th>\n",
       "      <td>93</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Arabic</th>\n",
       "      <td>73</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Korean</th>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Vietnamese</th>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              x  y\n",
       "y                 \n",
       "Russian     486  1\n",
       "English     459  1\n",
       "Japanese    383  1\n",
       "Italian     357  1\n",
       "German      330  1\n",
       "Czech       295  1\n",
       "Dutch       195  1\n",
       "French      172  1\n",
       "Spanish     170  1\n",
       "Chinese     158  1\n",
       "Greek       153  1\n",
       "Irish       129  1\n",
       "Polish       93  1\n",
       "Arabic       73  1\n",
       "Korean       31  1\n",
       "Vietnamese   25  1"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pd.DataFrame({'x': [_.text for _ in data.train_ds.x], 'y': [_.obj for _ in data.train_ds.y]})\n",
    "   .groupby('y')\n",
    "   .nunique()\n",
    "   .sort_values('x', ascending=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_set = set(_.text for _ in data.valid_ds.x)\n",
    "for _ in data.train_ds.x:\n",
    "    assert _.text not in valid_set, _.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, MyLetterRNN(n_layers=2), loss_func=F.cross_entropy, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmcXFWZ//HP03t6SSe9ZF86CVlZDBDCEkUWWVxQcBlgFBF1GBUVHFRGnN/IiIyKI7jgyERRFEEdAUdQFFCDgKydkJCd7PvSSaeX9FJd1fX8/qjbSdF0dypdVV3dle/79apXqs49995zqKafPss9x9wdERGR/srJdAFERGRoUyAREZGkKJCIiEhSFEhERCQpCiQiIpIUBRIREUmKAomIiCRFgURERJKiQCIiIknJy3QBBkJVVZXX1NRkuhgiIkPK4sWL97l79ZHyHROBpKamhtra2kwXQ0RkSDGzLYnkU9eWiIgkRYFERESSkrZAYmYTzWyRma02s5Vmdn0PeUaa2W/N7FUze8nMTjjSuWZ2i5ntMLOlwesd6aqDiIgcWTrHSCLAje6+xMzKgMVm9qS7r4rLczOw1N0vM7NZwA+A8xM49053/680ll1ERBKUthaJu+9y9yXB+2ZgNTC+W7Y5wF+CPGuAGjMbneC5IiIyCAzIGImZ1QAnAy92O7QMeG+QZz4wGZiQwLmfDrrDfmJmI9NSaBERSUjaA4mZlQIPATe4e1O3w98ARprZUuAzwCvEurX6OveHwDRgLrAL+HYv973WzGrNrLauri6VVRIRkThpDSRmlk8sENzv7g93P+7uTe5+jbvPBT4MVAOb+jrX3fe4e6e7R4EfAfN7ure7L3T3ee4+r7r6iM/TiIhklV2NbXz7ibVs2teS9nulc9aWAfcAq939jl7yjDCzguDjx4Gn3b2pr3PNbGzcx8uAFakvvYjI0LbjQBvf/+t6ttW3pv1e6Zy1tQC4ClgedF1BbJbWJAB3vxuYDfzczDqBVcDH+jrX3R8DbjezuYADm4F/TmMdRESGpPZwFICi/Ny03yttgcTdnwXsCHmeB6YfzbnuflVKCigiksVCkU4ACvPSP6dKT7aLiGShUCTWIinMVyAREZF+aA/HWiRFeenv2lIgERHJQmqRiIhIUkLhrjEStUhERKQf2iNds7bUIhERkX4IBdN/C3IVSEREpB9CkU7ycow8BRIREemPUCQ6IA8jggKJiEhWag93DsjDiKBAIiKSlUKRqAKJiIj0n7q2REQkKe3hTgrUIhERkf4KRaIUqkUiIiL9FQp3UqQWiYiI9Fe7WiQiIpKMUDZM/zWziWa2yMxWm9lKM7u+hzwjzey3Zvaqmb1kZifEHbvYzNaa2Xoz+9e49Clm9qKZrTOzX8dt1SsiIoGOLJm1FQFudPfZwBnAdWY2p1uem4Gl7n4S8GHguwBmlgv8AHg7MAe4Mu7cbwJ3uvt04ACHt+cVEZFAVjyQ6O673H1J8L4ZWA2M75ZtDvCXIM8aoMbMRgPzgfXuvtHdO4BfAe8xMwPOAx4Mzv8ZcGm66iAiMlRl3QOJZlYDnAy82O3QMuC9QZ75wGRgArGAsy0u3/YgrRJocPdIt/Se7nmtmdWaWW1dXV1qKiIiMkRk1QOJZlYKPATc4O5N3Q5/AxhpZkuBzwCvEOsSsx4u5X2kvzHRfaG7z3P3edXV1f0uv4jIUDSQXVt56by4meUTCyL3u/vD3Y8HgeWaIK8Bm4JXMTAxLusEYCewDxhhZnlBq6QrXUREApHOKJGoD8juiJDeWVsG3AOsdvc7eskzIm7W1ceBp4Pg8jIwPZihVQBcATzi7g4sAt4fnHM18Lt01UFEZCjq6By43REhvS2SBcBVwPKg6wpis7QmAbj73cBs4Odm1gmsIpiB5e4RM/s08DiQC/zE3VcG17gJ+JWZfY1YV9g9aayDiMiQ0x7sjjjku7bc/Vl6HtOIz/M8ML2XY48Bj/WQvpHYrC4REelBKNIJoCfbRUSkf7r2ax+ori0FEhGRLNPe1SIZ6oPtIiKSGaEBHiNRIBERyTKhSFfXllokIiLSD+3hrq4ttUhERKQfulokGiMREZF+OTz9Vy0SERHph64HEovUIhERkf5Qi0RERJKi6b8iIpKUrgcSNf1XRET6patFUpCrFomIiPRDKBKlIDeHnJw+181NGQUSEZEs0x7uHLCBdlAgERHJOqFIdMAeRoT07pA40cwWmdlqM1tpZtf3kKfczB41s2VBnq5td881s6Vxr3YzuzQ4dq+ZbYo7NjdddRARGYpCkYHbrx3Su0NiBLjR3ZeYWRmw2MyedPdVcXmuA1a5+yVmVg2sNbP73X0RMBfAzCqA9cATced9wd0fTGPZRUSGrFA4OmB7kUAaWyTuvsvdlwTvm4HVwPju2YCyYH/3UqCeWACK937gj+7emq6yiohkk1iLJAu6tuKZWQ1wMvBit0N3Edu3fSewHLje3aPd8lwB/LJb2m1m9qqZ3WlmhakvsYjI0BWKRLNrsN3MSoGHgBvcvanb4YuApcA4Yl1Zd5nZ8LhzxwInAo/HnfMlYBZwGlAB3NTLfa81s1ozq62rq0tVdUREBr32cOeArbMFaQ4kZpZPLIjc7+4P95DlGuBhj1kPbCIWJLr8A/Bbdw93JQRdZu7uIeCnwPye7u3uC919nrvPq66uTlWVREQGvaxpkQTjHvcAq939jl6ybQXOD/KPBmYCG+OOX0m3bq2gldJ1/UuBFaktuYjI0BYKR7Nm1tYC4CpguZktDdJuBiYBuPvdwK3AvWa2HDDgJnffB4fGVSYCf+t23fuDGV5GrFvsE2msg4jIkBOKdA7YOluQxkDi7s8S+2XfV56dwIW9HNvMG2d54e7npaJ8IiLZqn2AWyR6sl1EJMtk5fRfEREZOKFIljyQKCIiA8/dY4s2qkUiIiL9EYk6UR+43RFBgUREJKuEIrHFQQZy1pYCiYhIFmkPx7bZzYoHEkVEZOB1tUjUtSUiIv0SClok6toSEZF+aQ+rRSIiIkkIRYIxEk3/FRGR/jg0RqLBdhER6Y9Ds7bUIhERkf7QrC0REUmKHkgUEZGkHO7ayoIWiZlNNLNFZrbazFaa2fU95Ck3s0fNbFmQ55q4Y51mtjR4PRKXPsXMXjSzdWb2azMrSFcdRESGmmwbbI8AN7r7bOAM4Dozm9Mtz3XAKnd/E3AO8O24wNDm7nOD17vjzvkmcKe7TwcOAB9LYx1ERIaUUDYNtrv7LndfErxvBlbzxh0PHSgL9l8vBeqJBaAeBfnOAx4Mkn5GbN92EREhfowkO1okhwT7r58MvNjt0F3AbGAnsBy43t2jwbEiM6s1sxfMrCtYVAIN7t4VbLbTw3a8IiLHqq4WSUHuwAWStO3Z3sXMSoGHgBvcvanb4YuApcRaGdOAJ83smSDfJHffaWZTgb+a2XKg+/kQa9X0dN9rgWsBJk2alJrKiIgMcqFIbL/2WAfOwEhryDKzfGJB5H53f7iHLNcAD3vMemATMAvA3XcG/24EniLWotkHjDCzrgA4gVhr5g3cfaG7z3P3edXV1SmslYjI4NUe7hzQqb+Q3llbBtwDrHb3O3rJthU4P8g/GpgJbDSzkWZWGKRXAQuIDco7sAh4f3D+1cDv0lUHEZGhpqtFMpDS2bW1ALgKWG5mS4O0m4FJAO5+N3ArcG/QbWXATe6+z8zOAv7HzKLEgt033H1VcI2bgF+Z2deAV4gFKxERIQgkAzjQDmkMJO7+LLHg0FeencCFPaQ/B5zYyzkbgfmpKKOISLZpD3dSNIBTf0FPtouIZJVMtEgUSEREskgo0jmgDyOCAomISFZpD0cH9GFEUCAREckqapGIiEhSQuGBn/6rQCIikkXaI1n0QKKIiAw8tUhERCQpmXiyXYFERCSLZNVaWyIiMrDcXS0SERHpv47Orm121SIREZF+OLRfu1okIiLSH+1d+7WrRSIiIv0RCg/iFomZTYvbaOocM/usmY1Ib9FERORodHVtDdZZWw8BnWZ2HLGNpKYAD/R1gplNNLNFZrbazFaa2fU95Ck3s0fNbFmQ55ogfa6ZPR+kvWpml8edc6+ZbTKzpcFrbsK1FRHJYoe6tgbpDolRd4+Y2WXAd9z9+2b2yhHOiQA3uvsSMysDFpvZk3E7HQJcR2wL3UvMrBpYa2b3A63Ah919nZmNC8593N0bgvO+4O4PJl5NEZHsl6nB9kQDSdjMriS2R/olQVp+Xye4+y5gV/C+2cxWA+OB+EDiQFmwv3spUA9E3P21uOvsNLO9QDXQgIiI9CgUibVIBmvX1jXAmcBt7r7JzKYAv0j0JmZWA5wMvNjt0F3AbGAnsBy43t2j3c6dDxQAG+KSbwu6vO7sGrsRETnWDerBdndf5e6fdfdfmtlIoMzdv5HIuWZWSmyM5QZ3b+p2+CJgKTAOmAvcZWbD484dC9wHXBMXYL4EzAJOAyqAm3q577VmVmtmtXV1dYkUVURkSOtqkQzK/UjM7CkzG25mFcAy4KdmdkcC5+UTCyL3u/vDPWS5BnjYY9YDm4gFCYKA8gfg39z9ha4T3H1XkD8E/BSY39O93X2hu89z93nV1dWJVFNEZEg7PGtrELZIgPKgNfFe4Kfufirwtr5OCMY97gFWu3tvQWcrcH6QfzQwE9hoZgXAb4Gfu/tvul13bNz1LwVWJFgHEZGslqkHEhMdbM8LfoH/A/DlBM9ZAFwFLDezpUHazcAkAHe/G7gVuNfMlgMG3OTu+8zsQ8DZQKWZfSQ49yPuvhS4P5jhZcS6xT6RYHlERLLaYJ+19VXgceDv7v6ymU0F1vV1grs/S+yXfV95dgIX9pD+C3oZzHf38xIss4jIMSVTg+0JBZKge+k3cZ83Au9LV6FEROTodXVtDcrpv2Y2wcx+a2Z7zWyPmT1kZhPSXTgREUlcKBIlxyAvp8/OoJRLtP3zU+ARYtN0xwOPBmkiIjJIhCKdFOblEpuLNHASDSTV7v5Td48Er3uJPWkuIiKDRHs4OuBTfyHxQLLPzD5kZrnB60PA/nQWTEREjk5Xi2SgJRpIPkps6u9uYutnvZ/Yw4QiIjJIhCJRCgdri8Tdt7r7u9292t1HufulxB5OFBGRQaI93EnRIG6R9ORfUlYKERFJ2qBukfRiYKcFiIhIn0Lh6IA/jAjJBRJPWSlERCRp7ZHOAX8YEY7wZLuZNdNzwDBgWFpKJCIi/RIKR6ksGfgWSZ+BxN3LBqogIiKSnME+/VdERAa59vDQG2wXEZFBJBSJqkUiIiL9F+vaUotERET6KRSOZmTWVtoCiZlNNLNFZrbazFaa2fU95Ck3s0fNbFmQ55q4Y1eb2brgdXVc+qlmttzM1pvZ92ygl7kUERmEolGno3PoPUdyJBHgRnefDZwBXGdmc7rluQ5Y5e5vAs4Bvm1mBWZWAXwFOB2YD3zFzEYG5/wQuBaYHrwuTmMdRESGhI7OYHfEbBpsd/dd7r4keN8MrCa2l8nrsgFlQauiFKgnFoAuAp5093p3PwA8CVwc7Bs/3N2fd3cHfg5cmq46iIgMFYd2R8zWwXYzqwFOBl7sduguYDawE1gOXO/uUWIBZ1tcvu1B2vjgfff0nu55rZnVmlltXV1dCmohIjJ4hSJZ2CLpYmalwEPADe7e1O3wRcBSYjsvzgXuMrPh9LyOl/eR/sZE94XuPs/d51VXaw8uEcluoXAQSLKtRWJm+cSCyP3u/nAPWa4BHvaY9cAmYBaxlsbEuHwTiLVatgfvu6eLiBzT2iNB11Y2tUiCcY97gNXufkcv2bYC5wf5RwMzgY3A48CFZjYyGGS/EHjc3XcBzWZ2RnD9DwO/S1cdRESGiky2SPpcaytJC4CrgOVmtjRIuxmYBODudwO3Avea2XJi3VY3ufs+ADO7FXg5OO+r7l4fvP8kcC+xRSP/GLxERI5poaBFkonpv2kLJO7+LEfYs8TddxJrbfR07CfAT3pIrwVOSEUZRUSyRXvQIsmqBxJFRGTgZLJFokAiIpIFsnr6r4iIpF9XiyRrH0gUEZH06hojUYtERET6JRTuGiNRi0RERPqhpSMWSIoLFEhERKQfGtvCFOblaPqviIj0T2NrmBHF+Rm5twKJiEgWaGjrYMSwgozcW4FERCQLNLSGKVeLRERE+quxLcyIYZkJJOlctFFSbP/BEK/uaGT59kY6o85VZ06mqrQw08USkUGgoTXMSRMUSI450ajzyrYDvLCxnlljyjhjaiUlhYe/kvZwJ0+trePxlbt5aVM9OxraADCLrYb542c28rG3TOWf3jKFsqKef4DcnbrmEKt2NdHUHiG2Q3HM6OFFzK+pICenz7U1RWQIaGjrYERxZsZIFEjSLNIZpbEtTNRjv9SjDjsaWnls+W4eW76LXY3th/Lm5xonTxrJGVMr2bD3IH9ds5e2cCcjivNZcFwVHzmrhhMnlHP8uOHsbQ5xxxOv8b2/rOO+5zfzkbOmMHxYHm3hTto6Omluj7Ch7iCrdjaxv6Wj1/JNrBjG5fMm8v5TJzKmvIhIZ5Q1u5tZuq2B1/Y0M6mimLkTR3D8uHKGZWB+eqLCnVHW7z3Iyp1NrNjRSI4Z58ys5vSpFRl5QEtkILWHO2kPRynPUNeWxf+Fmq3mzZvntbW1A3a/bfWtPL2ujmde28ffN+yjuT3yhjwFuTmcPaOad500lrNnVLNmVxPPrN/HM+vqWLGjiarSQi46fjRvP2Esp0+tID+35+Gs5dsbuf3xNTyzbt+hNDMoKcijpqqY2WOGM2fccGaPHX6oG8yCBsiKHY38+uVtPLdhPzkGs8YMZ9O+FtrChx9sag0ecsrNMWaNKeOyk8fz4TNrKEjRCqPRqHOgtYP6lg72t8T+PdDaQXN7hOb2ME1tEToiUcaUFzFh5DAmVhQzengROw60sWZ3E2t3N7N2TzNrdjfTESxaV1yQS2fUCUWilBTk8pbp1Zw3exTnzRrVZ1dgpDPKzoZ2ttS3sLW+lfqDHbR0dNLaEaEl1EmOweTKYiZXljClqoTJlcW9tgRTobUjwrb6NqrLCqkoycxfmjI07Glq5/T//Au3XXYCHzx9csqua2aL3X3eEfOlK5CY2UTg58AYIAosdPfvdsvzBeCDwcc8YDZQHbx+HZd1KvDv7v4dM7sF+CegLjh2s7s/1ldZUh1IQpFO9jaF2NPUzq7GdrbWt7KtvpUt+1vZsr+FnUErY2x5EWdPr2b22DJyc3PIMcgxo6woj7NnVDO8l19Cze1higvyyD2KLqe65hC5OUZxQS6FeTmYJX7ulv0t/G/tNhZvOcCsMcM5edIITp44kokVw6g7GOLVbY0s3dbA8xv3s3jLAaZUlXDzO2bzttmjer1Pe7iTtbub2bSvhUg09jNmQNSd7Qfa2FB3kA11LWysO3ho1dLucnNi/63ycnLY3xKipx/VqtJCZo0pY8644Rw/bjjHjytnSlUJHZEoz2/cx59X7+Wvq/eyu6kdMzhl0kjeNns0C46rZE9TiNW7mli9q4k1u5vZWt9KZ/T1NynIy6G4IJeSgjzCnVH2NocOHcsxWHBcFZfOHc9FJ4yhtPDIDXx3p6k9wr6DIfYf7KC+JURDa5iGtjAHWjs40NLBlv2tbN7fwp6mw/eaObqM06dWcPqUSkYPL2Rvc4i9Te3sbQ7R0Bams9OJRJ2oO+5OWVE+I0sKGFmcT0VJATWVJcwcU5aRh9Uk/dbubuai7zzND/7xFN550tiUXXcwBJKxwFh3X2JmZcBi4FJ3X9VL/kuAz7n7ed3Sc4EdwOnuviUIJAfd/b8SLUsqAsm2+lZ+/MxG/rB8F/sOvrGrqKq0gEkVxUyqKObECSN464wqplWXHtUv9KFg0dq93PaH1azfe5CzplXyvlMm0NIRobE1TGNbmN1N7aze1cSmfS1Ee/nRMoOJI4uZVl3CcaNKGT9iGBWlhVSWFFBRUsDI4gKGD8tjWH7uof9+oUgnuxra2X6gjd1N7YwrL2LmmDIqE5hs4O6s3NnEn1fv4c+r97BiR9PryjK5opjZY4czrbo09h1Wxr7H6rLCN7QEWzsibK1vZfO+Fl7d3sijr+5kW30bhXk5nDdrFEX5uewJfsHXNYdoD3eSY0aOgZnREYnS0dlz4CzIzWFEcT4TK4qpqSxhSlUxEyuK2X6gjRc27qd284FDrcUuuTnGiGH55OfmkJtjh/74aGwL09Qefl3wzTGoqSph9tjhzBpdxvTRpRw3qozJlcW9tnhlaHhx434uX/gC93/8dBYcV5Wy62Y8kLzhRma/A+5y9yd7Of4AsMjdf9Qt/ULgK+6+IPh8CwMYSJZvb+R/nt7AY8t3kZtjXHT8GGaMLmPM8CJGlxcxZnisy6Ukgb9Gs0W4M8oDL27lzj+/RkNr+FD6sPxcKksLmDVmOHPGljF77HCmjy6lMC/3db/QRg0vzOhfxjsb2li85QDjRgxj1piypL47d2fJ1gP83ys7eXLVHnJzjNHDCxlVVsSo4YUMy8/FiXXhRR3y84zq0kKqSgupLC2gsqSQkSX5jBhWQFF+3y3JcGeU5TsaaWoLH7p+RXFBr5MlOqNOY1uY+pYQ6/ceZNWu5kMtsO0H2g7ly881po8q48xplSw4rpL5UyoTal3J4PH4yt38832L+f1n3swJ48tTdt1BFUjMrAZ4GjjB3Zt6OF4MbAeOi9ubvevYT4Al7n5X8PkW4CNAE1AL3OjuB3q45rXAtQCTJk06dcuWLUdd7lseWcm9z22mrDCPfzx9EtcsmMKY8qKjvk62aglF2NPUTvmwfMqK8lM2biLp1xKKsLGuhXV7m1m39yBLtzaweOsBOiJR8nKMWWPLKC7IIy9o5eTlGFWlhYwbMYxxI4oYN2LY68bdJLP+9+VtfPGhV3n2pnOZMLI4ZddNNJCk/c8OMysFHgJu6CmIBC4B/t5DECkA3g18KS75h8CtgAf/fhv4aPcLuvtCYCHEWiT9KftbZ1QzbkQRV8yf1Ot4xrGspDCPqdWlmS6G9ENJYR4nTijnxAmH/3ptD3dSu/kAf9+wjxU7GumIRAl3RmkLO+HOKCt3NlF38PVjVTNGl3LWtCrOmFrJaTUjE+pqlNRraIt1t2fl9F8zyycWRO5394f7yHoF8Mse0t9OrDWypysh/r2Z/Qj4fYqK+wbnzhrFubNGpevyIoNKUX4ub55exZun997H3hGJsqcpNla1dFsDz23Yx69f3sa9z20GoLosNvlh5uhY1+abJpYztapUzyqlWUNrmLwcoyRDU/TTFkgs1tl7D7Da3e/oI1858FbgQz0cvpJuAcbMxrr7ruDjZcCK1JRYRI6kIC+HiRWxSQBnTqvkk+dMoyMSZdn2BpZta2DN7mbW7m7mvhe2HJqNVxa0fuZOHMFFx4/hpAnlWTcJJdMa2mIr/2bqv2s6WyQLgKuA5Wa2NEi7GZgE4O53B2mXAU+4e0v8ycG4yQXAP3e77u1mNpdY19bmHo6LyAAqyMvhtJoKTqupOJTWGXU21B1k2baGIMg0svDpjfz3UxuYWlXCu+eO4z1zxzOlqiSDJc8eja3hjD2MCHogUUQGSGNrmD+t3MX/vbKTFzbtxx1KCnKpKC2goiQ2/futM6r58JmTe/zLetXOJn750lZOm1LBhXNG65mYOB/88Qu0h6M89MmzUnrdQTPYLiICUF6cz+WnTeLy0yaxq7GNP63Yzdb6Vg4EqxpsrW/lK4+s5KXN9Xzr/SdRXHD419Ojy3byhQeX0RGJct8LWygrzOMdJ47lvaeMZ/6UimO+q6yhNcyY4ZmbUapAIiIDbmz5MK5ZMOV1ae7O3X/byO2Pr2HD3oP86MPzGDdiGP/1xFp++NQGTp08kv/+4ClsqDvIw0t28PtXd/Lr2m2cPGkE//bOOZw6eWSGapN5Da1hZo4py9j9FUhEZFAwMz55zjRmjy3js798hUvuepbZY4bz/Mb9XDl/Ire8+3gK83IZPbyIs6ZV8dX3HM/vlu7kzidf430/fI53njiWmy6exaTK1D1HMVTE9iLJ3HpseoJMRAaVc2aO4pFPv5lRZYW8vLmer116Al9/70lvWMW5uCCPK+dPYtHnz+H686fz1zV7edsdf+Puv23gWBj77RLujHIwFMnYfu2gFomIDEI1VSU88uk3s7+lg/EjhvWZt6Qwj89dMIMr50/iPx5dyTf+uIZ9zSG+/M7Zx8TYSWNbbJmiTAYStUhEZFAqys89YhCJN6a8iB/84yl85KwafvzsJr744KtEelkgM5t0rXeXyem/apGISNbIyTG+cskcyofl892/rKO5PcJ3r5yb1ZubNWZ4eRRQi0REsoyZ8bkLZvDv75rDn4JVcbvvM5NNulokIzLYIlEgEZGs9NE3T+HWS0/gqbV1/GDR+kwXJ20OBRKNkYiIpN6HTp/EpXPH8Z0/v8aLG/dnujhpcWiwXdN/RURSz8z42mUnMrmyhOt/tZT6ljfubjrUNbSFMYOyoswNeSuQiEhWKy3M4/tXnkx9Swef/82yrHvGpLG1g+FF+Rldql+BRESy3gnjy7n5HbP465q93PPspkwXJ6W6lpDPJE3/FZFjwtVn1fDchv187Q+reWLlHt5z8jjeeeLYjE6bTYWG1nBGZ2yBWiQicowwM+68fC43XjCD/S0hvvzbFZx225/5+M9q2Vbfmuni9VtDW5jyDAfDtAUSM5toZovMbLWZrTSz63vI8wUzWxq8VphZp5lVBMc2m9ny4Fht3DkVZvakma0L/j12l/wUkaNSUpjHZ86fzp//5a38/jNv5uoza3hx036uWPjCkA0mja0dWd0iiQA3uvts4AzgOjObE5/B3b/l7nPdfS7wJeBv7l4fl+Xc4Hj8xir/CvzF3acDfwk+i4gkzMw4YXw5//auOTzw8TNobg9z5Y9eYPuBoRdMBsMYSdoCibvvcvclwftmYDUwvo9T3rA/ey/eA/wseP8z4NJkyikix7YTJ5Tzi4+fTlNbLJjsaGjLdJESFo16sIR8lgaSeGZWA5wMvNjL8WLgYuChuGQHnjCzxWZ2bVz6aHffBbFgBYxKR5lF5Nhx0oQR3Pex02loDXPlwhfYOUSCSXN7BHeyd4yki5mVEgsQN7h7Uy/ZLgES5K5/AAAP2ElEQVT+3q1ba4G7nwK8nVi32NlHed9rzazWzGrr6ur6VXYROXa8aWIsmBxo6eCT9y8hPARWDm7oWrAxm1skZpZPLIjc7+4P95H1Crp1a7n7zuDfvcBvgfnBoT1mNja4/lhgb08XdPeF7j7P3edVV1cnVxEROSbMnTiCr7/vRJZta+D7f03d+lwvb65n7e7mlF2vy2BYZwvSO2vLgHuA1e5+Rx/5yoG3Ar+LSysxs7Ku98CFwIrg8CPA1cH7q+PPExFJ1rtOGsd7TxnPXX9dx+ItB5K61vq9B/nYvS/zgbuf553fe4YfPrUhpSsRNwyCTa0gvS2SBcBVwHlxU3zfYWafMLNPxOW7DHjC3Vvi0kYDz5rZMuAl4A/u/qfg2DeAC8xsHXBB8FlEJGX+493HM27EMD7366UcDEWO+vwDLR3c8shKLv7O07y0qZ4vXjyTC+aM5pt/WsMHf5y6MZiG1ljXVnkGF2yEND7Z7u7PAkdc/MXd7wXu7Za2EXhTL/n3A+cnX0IRkZ6VFeVz5+Vzufx/nuerj67k9vf3+OuoR6FIJ+/83jPsbmrnyvmT+NwFM6gqLcTd+c3i7YcCzO3vP4mLTxibVDkHwza7oCfbRUR6dFpNBZ865zj+t3Y7f1qxK+HzNta1sLOxna+/90Ruu+xEqkoLgdizK/8wbyKPffYtTKku5boHXqF2c/0Rrta3wbDNLiiQiIj06vq3TefE8eXc8sgqQpHOhM55bU9sUP2kCSN6PF5TVcIvPjafCSOH8ZlfvsKBJJa2b2gNU1qYR35uZn+VK5CIiPQiPzeHL148k91N7fzfKzsSOmfdnoPk5hhTq0t6zVNWlM9dV57C/oPJLW3f0NaR8dYIKJCIiPTpzcdVcdKE8oRnXL22p5nJlcUU5uX2me/ECbGl7f+SxNL2ja2ZXx4FFEhERPpkZnzqnGls3t/KY8uPPFaybu9BZowqS+jaV59Vw0XHj+Ybf1zD0m0NR122wbDOFiiQiIgc0YVzxjCtuoQfLFrfZzdUe7iTLftbmDG6NKHrmhm3v+9NjB5exKcfWEJrx9FNNW5o7cjoXu1dFEhERI4gJ8f41DnHsWZ3M39d0+NiGgBsqDtI1GH66MRaJADlxfl86wMnsf1AG79NcBymS2NbmHK1SEREhoZ3zx3H+BHDuKuPVsm6PQcBmHEUgQTgzKmVHD9uOD9/bkvCA+/uPih2RwQFEhGRhOTn5vCJt07lla0NvLCx5+c/XtvTTF6OMaWq9xlbPTEzrj6zhrV7mnlxU8/XfmXrAfY2tR/63NLRSSTqGiMRERlKPjBvIlWlhfxgUc8LOr625yA1VSUU5B39r9ZL3jSO8mH53Pf8lh6u28wH7n6ef/p57aEWS9fyKBojEREZQoryc7n6zMk8u35fj+tlrdvbnPBAe3fDCnK5/LSJ/GnlbnY3Hm55RKPOzQ8vx4Fl2xt5ctUeIO6pdrVIRESGlotOGAPA3157/T5HbR2dbK1vZXqCU3978qHTJxN154EXD7dKfl27jdotB7jt0hOYWlXCt594jc6o09S1zpbGSEREhpbpo0oZP2IYi7rN3tpQdxD3ox9ojzepsphzZ47igZe20RGJUtcc4uuPreaMqRVcftpEPnfBDNbuaeb3r+6MW0JeXVsiIkOKmXHOzGr+vn7f69bf6lpjq79dW10+fOZk9h0M8ccVu/jaH1bRHo5y22UnYma888SxzB47nDuefI19B0NA5lf+BQUSEZGjdu7MUbR0dFK7+fDGV6/tOUh+rlFzlDO2ujt7ejU1lcV8/bE1/G7pTj517jSmVceCU06OceMFM9iyv5V7n9sMZH7lX0jvDokTzWyRma02s5Vmdn0Peb4Qt+nVCjPrNLOKvs41s1vMbEf8ZlnpqoOISE/OOq6Sgtyc13VvrdvTzJSqkqRX4s3JMT50xmR2N7UztaqET54z7XXHz589ipMnjWBjXQtF+TkU5fe9ptdASGeLJALc6O6zgTOA68xsTnwGd/+Wu89197nAl4C/uXt9Aufe2XWeuz+WxjqIiLxBcUEep0+t4Km4Afd1ew8e1RPtffnAvIm8ZXoV3/rASW9Y/NHM+MKFM4HBMfUX0hhI3H2Xuy8J3jcDq4HxfZxyJfDLfp4rIjKgzpk5ivV7D7KtvpW2jk62HWhNeLHGIykfls99HzudUydX9Hj8rOOqOHtGNRMrhqXkfskakDESM6sBTgZe7OV4MXAx8FCC537azF41s5+Y2cgUF1dE5IjOnVkNwFNr97J+b9eMreQG2o/GwqtO5afXzB+w+/Ul7YHEzEqJBYgb3L2pl2yXAH8PurWOdO4PgWnAXGAX8O1e7nutmdWaWW1dXV1PWURE+m1KVQmTK4tZtLbu0IytVHVtJaIoP5fSwrwBu19f0hpIzCyfWCC4390f7iPrFQTdWkc61933uHunu0eBHwE9hmR3X+ju89x9XnV1dbJVERF5HTPj3JmjeG7DPpbvaKQgN4eayuJMFysj0jlry4B7gNXufkcf+cqBtwK/S+RcMxsb9/EyYEUqyy0ikqhzZlbTHo7y0JLtTK0uIS/De6dnSjrbRQuAq4DlZrY0SLsZmATg7ncHaZcBT7h7y5HODWZo3W5mcwEHNgP/nMY6iIj06oyplRTl59DcHhnQbq3BJm2BxN2fBSyBfPcC9yZ6rrtflYLiiYgkrSg/lzOnVrJobR0zRg3cQPtgc2y2w0REUuTcWaOAgR1oH2wGx5C/iMgQ9Z6549myv5W3TK/KdFEyRoFERCQJ5cPy+X/vmnPkjFlMXVsiIpIUBRIREUmKAomIiCRFgURERJKiQCIiIklRIBERkaQokIiISFIUSEREJCnm7pkuQ9qZWR2wpVtyOdB4hLS+Pne9j0+rAvYlUdSeynQ0eY62Tom8T6ZOidSnr3yJph9NnQbiO+orXyLfUfe0wVCnVP7cdf+sOiUmE3Wa7u7lRyyZux+TL2DhkdL6+tz1vltabarLdDR5jrZOCb7vd50SqU9f+RJNP5o6DcR3dLR1OlLaYKhTKn/uVKehWae+Xsdy19ajCaT19fnRXvIkI5Fr9ZXnaOuUyPtkJHqd3vIlmj7U63SktMFQp1T+3HX/rDolJtN16tUx0bU1UMys1t3nZbocqZRtdcq2+oDqNFRkY526HMstknRYmOkCpEG21Snb6gOq01CRjXUC1CIREZEkqUUiIiJJUSDpgZn9xMz2mtmKfpx7qpktN7P1ZvY9M7O4Y58xs7VmttLMbk9tqY9YrpTXycxuMbMdZrY0eL0j9SXvs1xp+Z6C4583MzezAd2tKE3f061m9mrwHT1hZuNSX/I+y5WOOn3LzNYE9fqtmY1Ifcn7LFc66vSB4HdD1MyG1lhKMtPRsvUFnA2cAqzox7kvAWcS23P+j8Dbg/RzgT8DhcHnUVlQp1uAz2fT9xQcmwg8TuzZo6qhXidgeFyezwJ3Z0GdLgTygvffBL6ZBXWaDcwEngLmDWR9kn2pRdIDd38aqI9PM7NpZvYnM1tsZs+Y2azu55nZWGL/0z7vsZ+MnwOXBoc/CXzD3UPBPfamtxavl6Y6ZVQa63Qn8EVgwAcQ01End2+Ky1rCANcrTXV6wt0jQdYXgAnprcXrpalOq9197UCUP9UUSBK3EPiMu58KfB747x7yjAe2x33eHqQBzADeYmYvmtnfzOy0tJY2McnWCeDTQffCT8xsZPqKmrCk6mRm7wZ2uPuydBf0KCT9PZnZbWa2Dfgg8O9pLGuiUvGz1+WjxP6yz7RU1mlI0Z7tCTCzUuAs4DdxXemFPWXtIa3rr788YCRwBnAa8L9mNjX4q2TApahOPwRuDT7fCnyb2P/UGZFsncysGPgysW6TQSFF3xPu/mXgy2b2JeDTwFdSXNSEpapOwbW+DESA+1NZxqOVyjoNRQokickBGtx9bnyimeUCi4OPjxD7xRrfxJ4A7AzebwceDgLHS2YWJbb2Tl06C96HpOvk7nvizvsR8Pt0FjgBydZpGjAFWBb8MpgALDGz+e6+O81l700qfvbiPQD8gQwGElJUJzO7GngXcH6m/iCLk+rvaWjJ9CDNYH0BNcQNpAHPAR8I3hvwpl7Oe5lYq6NrIO0dQfongK8G72cA2wie4xnCdRobl+dzwK+G+vfULc9mBniwPU3f0/S4PJ8BHsyCOl0MrAKqB7ou6f7ZYwgOtme8AIPxBfwS2AWEibUkPkbsL9U/AcuCH+B/7+XcecAKYANwV1ewAAqAXwTHlgDnZUGd7gOWA68S+2tr7EDVJ1116pZnwANJmr6nh4L0V4mtnTQ+C+q0ntgfY0uD10DPREtHnS4LrhUC9gCPD2SdknnpyXYREUmKZm2JiEhSFEhERCQpCiQiIpIUBRIREUmKAomIiCRFgUSOSWZ2cIDv92Mzm5Oia3UGK/muMLNHj7TyrZmNMLNPpeLeIj3R9F85JpnZQXcvTeH18vzwIoJpFV92M/sZ8Jq739ZH/hrg9+5+wkCUT449apGIBMys2sweMrOXg9eCIH2+mT1nZq8E/84M0j9iZr8xs0eBJ8zsHDN7ysweDPbKuD9ur4mnuvaYMLODwSKKy8zsBTMbHaRPCz6/bGZfTbDV9DyHF5wsNbO/mNkSi+138Z4gzzeAaUEr5ltB3i8E93nVzP4jhf8Z5RikQCJy2HeBO939NOB9wI+D9DXA2e5+MrGVc/8z7pwzgavd/bzg88nADcAcYCqwoIf7lAAvuPubgKeBf4q7/3eD+x9x/aVgHafzia0qANAOXObupxDb/+bbQSD7V2CDu8919y+Y2YXAdGA+MBc41czOPtL9RHqjRRtFDnsbMCdu9dbhZlYGlAM/M7PpxFZqzY8750l3j9+X4iV33w5gZkuJrcf0bLf7dHB4gcvFwAXB+zM5vC/KA8B/9VLOYXHXXgw8GaQb8J9BUIgSa6mM7uH8C4PXK8HnUmKB5ele7ifSJwUSkcNygDPdvS0+0cy+Dyxy98uC8Yan4g63dLtGKO59Jz3/Pxb2w4OTveXpS5u7zzWzcmIB6Trge8T2GqkGTnX3sJltBop6ON+Ar7v7/xzlfUV6pK4tkcOeILZXBwBm1rUkeDmwI3j/kTTe/wViXWoAVxwps7s3Ets69/Nmlk+snHuDIHIuMDnI2gyUxZ36OPDRYA8NzGy8mY1KUR3kGKRAIseqYjPbHvf6F2K/lOcFA9CriC39D3A78HUz+zuQm8Yy3QD8i5m9BIwFGo90gru/Qmy12SuIbe40z8xqibVO1gR59gN/D6YLf8vdnyDWdfa8mS0HHuT1gUbkqGj6r8ggEezQ2ObubmZXAFe6+3uOdJ5IpmmMRGTwOBW4K5hp1UAGty0WORpqkYiISFI0RiIiIklRIBERkaQokIiISFIUSEREJCkKJCIikhQFEhERScr/B3QpmHhuW0+jAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that our balanced dataset is about half the size of our training dataset. Useful to keep in mind when comparing number of epochs and runtime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.048625"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_idx) / len(bal_idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only get around ~51% accuracy on a balanced test set (similar to the Naive Bayes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:12 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.773524</th>\n",
       "    <th>2.755105</th>\n",
       "    <th>0.256250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.756307</th>\n",
       "    <th>2.651738</th>\n",
       "    <th>0.252083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.653947</th>\n",
       "    <th>2.520690</th>\n",
       "    <th>0.222917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.586744</th>\n",
       "    <th>2.220565</th>\n",
       "    <th>0.277083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>2.418116</th>\n",
       "    <th>2.046903</th>\n",
       "    <th>0.358333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>2.238451</th>\n",
       "    <th>2.456228</th>\n",
       "    <th>0.300000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>2.139630</th>\n",
       "    <th>1.903009</th>\n",
       "    <th>0.404167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.966441</th>\n",
       "    <th>1.928596</th>\n",
       "    <th>0.441667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.810611</th>\n",
       "    <th>1.840151</th>\n",
       "    <th>0.439583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.651867</th>\n",
       "    <th>1.802120</th>\n",
       "    <th>0.445833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.560050</th>\n",
       "    <th>1.871820</th>\n",
       "    <th>0.450000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.477005</th>\n",
       "    <th>1.901750</th>\n",
       "    <th>0.479167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>1.461073</th>\n",
       "    <th>2.092149</th>\n",
       "    <th>0.406250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>1.403493</th>\n",
       "    <th>2.019508</th>\n",
       "    <th>0.462500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>1.279949</th>\n",
       "    <th>1.945947</th>\n",
       "    <th>0.470833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>1.140172</th>\n",
       "    <th>2.048897</th>\n",
       "    <th>0.504167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>0.996752</th>\n",
       "    <th>2.194067</th>\n",
       "    <th>0.472917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>0.863288</th>\n",
       "    <th>2.319874</th>\n",
       "    <th>0.500000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>0.765463</th>\n",
       "    <th>2.229700</th>\n",
       "    <th>0.491667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>0.672631</th>\n",
       "    <th>2.348388</th>\n",
       "    <th>0.516667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>21</th>\n",
       "    <th>0.577968</th>\n",
       "    <th>2.426682</th>\n",
       "    <th>0.506250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>22</th>\n",
       "    <th>0.488059</th>\n",
       "    <th>2.629519</th>\n",
       "    <th>0.508333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>23</th>\n",
       "    <th>0.406027</th>\n",
       "    <th>2.711037</th>\n",
       "    <th>0.520833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>24</th>\n",
       "    <th>0.335120</th>\n",
       "    <th>2.838989</th>\n",
       "    <th>0.512500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>25</th>\n",
       "    <th>0.274594</th>\n",
       "    <th>2.929183</th>\n",
       "    <th>0.510417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>26</th>\n",
       "    <th>0.225410</th>\n",
       "    <th>3.002161</th>\n",
       "    <th>0.506250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>27</th>\n",
       "    <th>0.185401</th>\n",
       "    <th>3.071880</th>\n",
       "    <th>0.506250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>28</th>\n",
       "    <th>0.154014</th>\n",
       "    <th>3.097990</th>\n",
       "    <th>0.506250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>29</th>\n",
       "    <th>0.130493</th>\n",
       "    <th>3.111022</th>\n",
       "    <th>0.510417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>30</th>\n",
       "    <th>0.113012</th>\n",
       "    <th>3.112732</th>\n",
       "    <th>0.510417</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(30, max_lr=3e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's starting to overfit and so could perhaps do with some regularization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4VNX28PHvSu8JqQQIEAhIJ4TQpEgTBRUsIEVUmtixXL0X9f5s7y1WRMWGXoqIIBYEEStFpBMghN5baCnUBBJS9vvHGUOAACFkMklmfZ4nT2bO7DOz5hBmzTl777XFGINSSikF4OLoAJRSSpUfmhSUUkoV0KSglFKqgCYFpZRSBTQpKKWUKqBJQSmlVAFNCkoppQpoUlBKKVVAk4JSSqkCbo4O4GqFhoaa2rVrOzoMpZSqUFavXp1mjAm7UrsKlxRq165NQkKCo8NQSqkKRUT2FqedXj5SSilVQJOCUkqpApoUlFJKFdCkoJRSqoAmBaWUUgU0KSillCqgSUEppVSBCjdPQSmlKixj4GwGZKZCZrr1+3Q65J0Fkw/5eWDyzv02+ZCff25bzbYQ082uIWpSUEqp0pJ9ClK3QspmSN8OGSmQmWZLAmlwOg1ys0r+/B2e0qSglFLlTs4Z68M/dQukbIKULVYiOLHvXBtXD/ANB99Q8A2D8IbnbvvYfvuGgk8IuHmCuIKLK4iL7Xeh++IKLmVztV+TglJKFUdGCqz8FDZ8C0d3Acba7uIOofUgqhXE3Wd9+Ic3hCq1rQ/1CkaTglJKXU7qNlg2DtZNt6791+0KTftBeAMIbwTBdcDV3dFRlhq7JQUR8QIWAZ621/nGGPPSBW08gc+BlkA60N8Ys8deMSmlVLEYA3uXwNJxsO0ncPOC2EHQ7lHrrKASs+eZQjbQ1RiTISLuwGIR+ckYs7xQm+HAMWNMjIgMAF4H+tsxJqWUurS8XNg8C5a+DwfXWtf7bxgNrUaA3xWrTlcKdksKxhgDZNjuutt+zAXN+gAv225/A4wTEbHtq5RSZSM7A9ZOgWUfWp3FwXXhljHQfCB4+Dg6ujJl1z4FEXEFVgMxwAfGmBUXNKkO7AcwxuSKyAkgBEizZ1xKKQXA6aOw4hNY8TFkHYea7aDna1C/Z5mN9ilv7JoUjDF5QKyIBAEzRaSJMWZDoSZS1G4XbhCRkcBIgJo1a9olVqWUEzl5EJZ9AAkTIScTrutlzQGIau3oyByuTEYfGWOOi8hC4GagcFJIBqKAZBFxAwKBo0XsPx4YDxAfH6+XlpRSJZO+E5a8C+umWTOEm/aF9k9CRCNHR1Zu2HP0URiQY0sI3kB3rI7kwmYD9wPLgL7AfO1PUEqVusPr4c8xsOl7a15Bi3uh/ShrLoE6jz3PFCKBybZ+BRdghjFmjoi8CiQYY2YD/wOmiMgOrDOEAXaMRynlbJJXwx+vwfZfwcMPrn8c2j4C/lUdHVm5Zc/RR0lAiyK2v1jodhbQz14xKKWc1KkjMO8VSJxqDSvt8k9oPQK8qzg6snJPZzQrpSqP3LOw8hNY+LpVeK79k9DpGfD0d3RkFYYmBaVU5bD9d/h5tFWdtN5NcPN/IaSuo6OqcDQpKKUqtqO74JcXYOtca9LZoBlQ/yZHR1VhaVJQSlVM2RmweIxVksLVA7q/Am0ftspQqxLTpKCUqng2zYKfRsOpg1Ypiu4v64iiUqJJQSlVceTnw/xXYfE7ENkc7p6ss5BLmSYFpVTFkJ0B342ErT9Cy6HQ681KtY5BeeFUSSErJ4/9R08TE+6HSFFll5RS5dLxfTBtoLX0Zc83oPVI0P/DduE0SSHlVBYDxi9nV2omtzaLpHmNIM7k5OHh5oKvhys+Hm74erpSxceDOmF+hPp5aOJQqjzYtxym3wN5OXDPN3ZfuN7ZOU1SWLP3GCkns+kQE8qcpEPMSTp0ybb+nEa8Aqgb7kfDyADa1gmhcbUAoqr44OHmQnZuHikns6ke5I2LiyYOpexm7VSY8yQERsGgryr9qmflgVS0+nPx8fEmISGhRPsezTxLFR93Nh48SfUgb/y93Dibl8/ps3lkZueSmZ3H6QMbaPZLPxaEDWaSuY0NhzM5lZULQKC3O02qB7B811Hy8g2NqwXwwi0NaV07GDdX56y9rpRd5OfB7y9Zw02jb4B+k8An2NFRVWgistoYE3/Fds6UFIolIxV+fBo2z4bq8eT1+ZB1WeHsSctk0bZUvk88SBUfd0Z0rMMnf+zkZFYuUcHejOxYh1uaVSPY18N+sSnlDLJOwncPwLafrWUwb35NO5RLgSaFa2EMbPgW5j4DZ09Dt/+zKiu6uLI3PRMXEaKCfcjMzmXh1lQ+WLCDTYdO4uvhyqt9mnBnXHXtj1CqJI7utjqU07ZBz9eh9QOOjqjS0KRQGk4dgTlPWUPgotpAnw8hNOaiZsYYNh86xcs/bGTl7qM0qOrP+wNbUC9Ci3ApVSxnT1tLYi4ea40qunsy1Ons6KgqFU0KpcUYWP81zH3WqrrY7SVo81CR67dm5eQxI2E/783bAcA3D7Wjdqhv2cWqVEWTexbWTIZFb0LGES1kZ0eaFErbyUPWKIhtP0PN66HPuEv+4e5IOcXdnyzHz9ONHx7vQKC3Xg9V6jz5+dYl2gX/gmN7oGY76wtXrXaOjqzS0qRgD8ZYa7v+NBryc6x6K60eKPKsYfXeo9z9yXLqhfvhIkJMuB+PdKlLg6oBZR62UuWGMdYqaPNehSMbIKKJlQzq3aiT0exMk4I9nTwIs0fBjt+gwa1w53jwuPgy0YItKTw/cz2HTmQVbGtZqwov3NKQuJq6ApRyMnuXwu+vwP7l1trIXf8PGt9Z5JcqVfo0KdibMbD8I/j1BajaFAZ+BQGRFzU7fTaXU1m5nDiTw5/b0/h00S4On8wiOtSXEF8PHu0aQ5frwh3wBpQqI9kZ8O0I2PYT+EXADX+HFveBmw7fLkuaFMrK1p/hm2HgFWjNuIxsdtnmJ87k8N2aZP7Ylsr2IxkcOH6GQW1q8myP66iicxxUZWOMNedg/TfQ9Z/W0G4PH0dH5ZQ0KZSlw+vhy/5w5jj0nQDX3Vys3VJOZfHk9ERW7D5K3TBfvn7oeu2UVpVLwkRrgEaXF6wzBOUwxU0KejGvNFRtCg/Mt+qyTB9oXVYqRrIN9/fiywfa8vmw1uxKzeTOD5dw+mxuGQSsVBk4mAg//QPqdoOOzzg6GlVMmhRKi39VGDoXrutlLR4+9xnIK94HfPuYUD65tyU7UzN54+et5Obl2zlYpews6wR8fT/4hFgDMbQzucLQf6nS5OELd0+B9k/Aqs9gWn+rjksxdG0QTpvoYCYt3cNjX64lP79iXdZTqoAxMOtROL4f+k0E31BHR6Sugt2SgohEicgCEdksIhtF5Iki2nQWkRMikmj7edFe8ZQZFxe48VW47T3YtRAm3GQtEHIFIsLkYa15vGsMP288zC8bD9s/VqXsYcXHsPkHuPEVqNnW0dGoq2TPM4Vc4G/GmIZAW+BREWlURLs/jTGxtp9X7RhP2Wp5Pwz+Fk4cgE+7WtdXr8DL3ZUnu9enTpgvb/6ylaycvDIIVKlStH8V/PpPuO4WaPeYo6NRJWC3pGCMOWSMWWO7fQrYDFS31+uVS3U6w4jfwdUDvn/Emtp/Ba4uwiu9G7MrLZORU1ZrYlAVx+mj8PUQCKgGt3+gM5QrqDLpUxCR2kALYEURD7cTkXUi8pOINC6LeMpUWH2rHEbKRtj0fbF26VgvjH/d3oRF21L514+bqGjDhpUTys+HmQ9CZgr0mwzeOmO/orJ7UhARP+Bb4EljzIW9rmuAWsaY5sD7QJGfmiIyUkQSRCQhNTXVvgHbQ5O7IKwBLPyvtaJUMQxuW4sHOkbzxfJ9fPTHTjsHqNQ1WjLWqml003+gepyjo1HXwK5JQUTcsRLCVGPMdxc+bow5aYzJsN2eC7iLyEVDFYwx440x8caY+LCwMHuGbB8urtD5OWvhkPVfF3u353s1pHvDcMb+tp27P17GrMQDV/e6eblwbO9VBqvUVdqzGOb/P+vLT6sRjo5GXSN7jj4S4H/AZmPMmEu0qWprh4i0tsWTbq+YHKphb4hoCgtfg7ycYu0iIvzj5gaczctn5Z6jPDE9kQ0HThT/Nf98G8a1gszKeUhVOZCRYpV5Ca4Dt72r/QiVgD3PFNoD9wJdCw057SUiD4nIQ7Y2fYENIrIOeA8YYCrrBXQXF+jyPBzbbZXfLqZ6Ef5MGtqK53o2oGqAF30/XsrsdQevvGN+PqydAnnZsGvBNQSu1CXknoVvh1sT1e7+HDx1pcHKQGsflSVjrOGpmanw+Gpw87yq3XekZDBq2lr2HzvN/L91Jsz/MvvvWgif97Fux94Dt39Y8riVKiwvx/pi88ebcGIf9PkAWgx2dFTqCrT2UXkkAl1fgBP7Yc3nV717TLgf7w1sQWZ2LuMXXaHzOfFLq3LrdbfAjnnFqsWk1GXl5Vp/V+PiYfbj4BdmzcXRhFCpaFIoa3W7QVRb63p/zpmr3j0m3I/bW1Rn0tI9rN13rOhGWSdg02xo0hca9IKMw3Bk4zUGrpxWfh4kzYAPWsP3D9vKxM+AEfMgprujo1OlTJNCWROx6sqfOmSVFS6BF29tRESAF0MmrmL/0dMXN9j4PeSesS4b1e1qbds57xqCVk7pr3WUP2xrrYng7g39p8LIP6D+TdqpXElpUnCE6I4Q3QkWj4GzmVe9e5CPB1NHtCErJ4+3f9168eS2xC8h9DprvHhANQhvZF1CUqo4jLHOND9ub40sEhfoNwke/BMa3qrJoJLTpOAoXf5pdTivHF+i3WuF+DKsQzTfJx7k5dkbyfmr3Hb6TmsN3Bb3nPvPG9MN9i0rUQJSTubMMZg2EGbcC3ln4a7/wcNLofEdWv7aSei/sqPUbAMxN8KSd4tdXvtCz/a4jvva1WLysr30eGcRGdm5kDjV+mbXrP+5hnW7Wf/B9ywupeBVuZKRahVevFaH1sEnN8CO3+Gm/8IjK6BpX2vypXIamhQcqcvz1jez5R+VaHcXF+Gl2xrz4q2N2J2WyfiF22DddKvzz7/quYY124Gbt15Cqoz2LIYP28B7sTDvVcjOKNnzrPkcPrsR8nNh6E/Q7hFwdSvdWFWFoEnBkarHWUNGl31gJYcScHURhnWIpmeTqmxd9iOcPACxg85v5O4FtTtoZ3Nls+ozay6KTwg06mObwR4PSV8Xfwhyzhn4/lFriGmtdvDgIohqZd+4VbmmScHRujwP2Sdg6bhrepoRHevQK28+GeLHBr/2FzeI6Q7pO+DYnmt6HVUO5J6FH56EH/9mXRoc8Tvc9RkM+xX8IuC7ETCxp3U56HLSd1pnB4lfQKe/w+DvdJU0pUnB4ao2sTrxln8EmWklfpqW4cIt7qv5Nqcdw6cmkZl9wfrQMd2s33oJqWLLTLPODlZPhPZPwsBp1rwBsPqpHphvrfqXtt3qH/jhyaJrX235EcZ3sSZSDvramlSpfQcKTQrlQ+fnrHkFS8aW/Dk2zsQtP5vWdzxGyqlsHp665vwFekJiILAm7Jx/7fEqxzi83vogP7gG7vzMWu7ywg9yF1dr1b/HV0Pbh62+gvdbwIrx1ozkvFz47UWYPgiCo63LRfV7OOb9qHJJax+VF9+NtMaGP7EO/COufv/PboTsU/DIMmYkJPOP75Lw83Aj0MeddnVCeKNvM2TOk7D+W/jHbnB1L/33oOxn4/e22cRBMGBq8dcsSNkCP//DqoUV3sjaf99SaDkUbn7N6m9STkFrH1U0N/zDGja68D9Xv2/qNkheWTA34e5WUUy4vxWd6oeRfOwMX69O5t7/rSSrVhc4ewr2ryz9+JV95OfD/H/D1/dDRBMYufDqFrEJbwD3fm/NRD6bAQfXwu0fw21jNSGoIumYs/IipK51ur9snPWfv/UDxd933ZcgrtD07oJNXRqE06VBOO/lG96bt5335m+nd6orP4srbP8dl9pFdEar8iX7FMx8CLbMsYrO3TLmqivrAtYkxoa3Qr0brSGrviGlH6uqNDQplCfdX7FGhMx9Fvwjrf/IV5KfZ81NqHdjkZedXF2Ep26sT/Uq3ny5Yh8JR2IIXDaLFX73M7B1Tdxd9WSxXDp1BL64C1I2wc2vQ5sHr728hJtnyZKKcir6iVCeuLpB3wlQvaW1eMm+FVfeZ+cCq7he7D2XbXZ3fBTfP9qewCY3Uy9/F2NnLeOOD5dwNPNsKQWvSs3x/daQ0qM74Z4Z0PYhrTekyowmhfLGwwcGfWUVspvW3xpaeDmJU8E7GOrfXKynv6797bhgmNDxFNuPZDBs0irOnM278o6qbKTtgAk3W0NP7/1eS1OrMqdJoTzyDbUWL3Fxgy/utC4lFOXMMWu8edN+4OZRvOeOjAWfEGKzV/PewBYkJR9n+ORVpGVkl178qmQOr4eJN0NuFgyZY807UKqMaVIor4LrWGcMmWnwZT+r0/FCG76z1mC+sKzF5bi4QJ0usHM+NzUM542+zUnYe4xXfthUerGrq7d/FUy6BVw9rNpDkc0cHZFyUpoUyrPqLaHfZDi8AWbcb62NW1jiVGukUmTzq3vemG6QmQJHNtC3ZQ2GXl+bH5MOsmJX+sVrMyj7+2s9be9gGPYzhNV3dETKiWlSKO/q97DGlO+cBz88ca7QWcoWOLDaOku42k7IC1Zje/CGugT7etJ//HL6j19OyqmsUnwD5VRejnUMHZ0Et8yFqXdDlVpWQgiq6dh4lNPTpFARxN0HN4y2zgwW2Ca3rfvS6nMoNDeh2PyrQkTTgjpIwb4eTBraiuEdoklKPs4dHywl9VQl7mMwxppB/mEbmHATbP3JmiRW1pK+hq8GQ0RjGPLj+eXOlXIQTQoVRefR0OJeWPSGVTJ53XSo1wP8wkr2fDFdYd/ygvr7TaoH8n+3NuKrke1Iz8zmwSkJ59dOqkwWvwMbv4Mmfa3hvNMGWEtPJs2wagOVhYQJ1rrHNdvB/bPBJ7hsXlepK9CkUFGIwK3vWKu1/fg3yDhyxbkJlxXTHfJzYM+f521uHhXEmLtjWbPvOH/7el3lG6667VdrMZomd1nlph9fA3eMt509PGAVj1v5qbXOgD0YA3+OgTlPWUl98Dfg6W+f11KqBOyWFEQkSkQWiMhmEdkoIk8U0UZE5D0R2SEiSSJyFUVdnJCru7WAerUW4F/N+lApqai24O5bZCntXk0jGd2zAT8mHWLYpFXn1n+u6NJ2wLcjrHLlvcdZidbVHZr3t9YhHjDNWo9g7jMwtqm1aE3WidJ7/eP7rSHG816xyqX3/wLcvUvv+ZUqBXarkioikUCkMWaNiPgDq4HbjTGbCrXpBTwO9ALaAO8aYy47OLvSVkm9GrlnreJm13rJ4cv+kLoVnkgs8uFvVifzzNfrGNi6Jv+5owlSkWfVZp2Ez7rB6XSrqNylOnSNgb1LrG/zO+eBZwC0Gg5tHwG/8JK9tjFWCetfXgCTb5W8jh9uDQ9WqowUt0qq3WofGWMOAYdst0+JyGagOlB4QHwf4HNjZablIhIkIpG2fdWluHmAWylcg67bDbb9DEd3WfMiLtC3ZQ12pWbw4cKd1Av3Y1iH6Gt/TUfIz7c6lo/ugvtmXX6Ej4i1dGntDnAw0VrjYvFYaxGkuPvg+lEQFFX81z6RDLNHWQmmdkfoMw6q1L7mt6SUvZTJVxURqQ20AC4s5lMd2F/ofrJtmyoLxViN7Zke19GjUQT/+nETf2xLLaPAStnC/8C2n6z1A2p3KP5+1WKty3WPJUDTvlbn8Hux1prGVyo/8tfZwYftYN8y6PUW3DdbE4Iq9+yeFETED/gWeNIYc/LCh4vY5aLrWSIyUkQSRCQhNbWCfjCVRyF1rQ+py6zG5uIijB0QS/0If56Yvpb0ilYOY9MsWPSmNXKr1YiSPUdoDPT5AEYlWpd9NnwD41rB10PgUNLF7U8cgKl9YfbjULWZ1V/R+gG9XKQqBLv+lYqIO1ZCmGqM+a6IJslA4XPxGsDBCxsZY8YbY+KNMfFhYSUcgqmKVrcb7F5k9VNcgo+HG+8PbEFGVi7v/L6tDIO7Rkc2wsyHoUZruOXta680GhQFvd6AJzdAh6esM6xPOsLUflZFW2Ng7RfW2cHepdDzTbj/B2vZS6UqCHuOPhLgf8BmY8yYSzSbDdxnG4XUFjih/QllLKab1Wm9//JluutF+NMvvgZfrtjHsp1FLARf3pw+CtMGglcA9J9SuusI+IVB95fgyfXQ9Z+QnAATesC7zWDWo9bopoeXQJuRenagKhx7/sW2B+4FuopIou2nl4g8JCIP2drMBXYBO4BPgUfsGI8qSnQna2b0zkv3K5CfB+k7GVV9G8Pcf+Ppz37kl42Hyy7Gq5WXa13aOXXIGvZpr5nC3kHQ6Vl4agPc9F/wq2otiHP/nCI77pWqCOw2JNVedEiqHUy8BbJPWkM1j+6G1C2QutkarpqyBdK2WdVYbbLwZHxeb9oMfpk219VwWNiX9PPzsPwD6POhtW61UsrxQ1JVBRLT1Zrl++/I8z78CYyCsAZQ5wYIb2jd9vBDfv8Xo7Z9zcEv53O4x6tUvf4ex64MlpNlLVt5aB3sX2nVhWrzkCYEpUpAk4KC5gOtTtmAahBm+/APq3/J8gueg77g2Mb5nPz6KRr89ihZGybhdesbUL0MJqRnZ8CRDVYC+OsndQvk22oWeQZC80HQ41/2j0WpSkgvH6kSW7c3nZkTX2cU06nCSST2Huj2IvhHlO4LHd0Fyz+2hs6m76Bg1LJvmLWSXGQza02JyOYQVEvXM1aqCHr5SNld81oh7L/zSW7+pi3D8r/lgXXTcd30PXT8m1UWwt3r2l5g/ypY+h5s/sHqDI/pDs3utsb+Rza3OpA1AShVqvRMQV2znLx8np6xjqSkNXxV+0eqHppnfWNvca81g7h6XPGHhObnWesbLH0f9i8Hr0BrwlibB3W9AaWugZ4pqDLj7urCm32bMfj4GTrtr8aMG+8jdtt7sMB2Xd/NC2q0smr/1G4P1eMvPovIOQOJX8KyD+DoTqs+0c2vQ4vB4OlX9m9KKSelZwqq1BzNPMvA8cvZeuQUT99Yn1HtQqyZvXsWw97F1lrTGHD1tCWJ9lCzrTUbeNWnVgXTanFw/ePQsDe46ncWpUpLcc8UNCmoUpWVk8czX6/jx/WH+OGxDjSpHnjuwTPHrNXe9iy2fg4nWaWkAer3tJJBreu1n0ApO9CkoBzmxOkcuo35Az9PV75/tD1BPh5FN8w6YZWICKoJofXKNkilnExxk4IWZlGlLtDHnU/ubcnB41k8/MWaS6/c5hVo1V7ShKBUuaFJQdlFy1pVeO2upizblc7/fb+B/PyKdUaqlLMqVlIQkboi4mm73VlERolIkH1DUxXdnXE1eLhzXaav2k+/T5aRlZPn6JCUUldQ3DOFb4E8EYnBKocdDXxpt6hUpfH3m67jP3c0Zc2+Y4z+NomK1oellLMp7pi/fGNMrojcAYw1xrwvImvtGZiqHESEQW1qcjQzm7d+3Ya3hxv/ur0Jri46wkip8qi4SSFHRAYC9wO32ba52yckVRk92iWGMzl5fLBgJ0czs3l3QAu83F0dHZZS6gLFvXw0FGgH/NsYs1tEooEv7BeWqmxEhGdvasBLtzXi101H6PvxUg4eP+PosJRSFyhWUjDGbDLGjDLGTBORKoC/MeY1O8emKqGh7aP59N549qSdpve4xazac9TRISmlCinu6KOFIhIgIsHAOmCiiFxq3WWlLqt7owi+f/R6/L3cuX/CSpKPnXZ0SEopm+JePgo0xpwE7gQmGmNaAt3tF5aq7GLC/ZkyvDXGwEuzNuqoJKXKieImBTcRiQTuBubYMR7lRGpU8eHpG+szb0sKv2w87OhwlFIUPym8CvwC7DTGrBKROsB2+4WlnMXQ9rVpFBnAU1+t481ftrBgS4pOclPKgbQgnnK43WmZPP/depbtSgcgIsCTIddH0yEmlKY1Aq+wt1KqOEq1SqqI1ADeB9pjLZC7GHjCGJN8rYFeLU0KlVNOXj6zEg/i4ebC//7cxbrkEwBEBnrRLz6KvnE1qBni4+Aolaq4Sjsp/IZV1mKKbdNg4B5jzI3XFGUJaFKo/IwxJB87w2+bjvDn9lQWbE0F4Pq6ITzQqQ6d64chuuaCUleltJNCojEm9krbLnh8AnArkGKMaVLE452BWcBu26bvjDGvXikWTQrOZ9Weo6zcfZQpy/Zy+GQWrWpX4anu9WkQGUCw7yXWalBKnae012hOE5HBwDTb/YFA+hX2mQSMAz6/TJs/jTG3FjMG5aRa1Q6mVe1gHuhYh2/XJPPSrI0M+mwFPh6uvHZXM3o3r+boEJWqNIo7+mgY1nDUw8AhoC9W6YtLMsYsAnS6qio1Hm4uDGxdk8X/6MLUEW1oXC2AUdPW0vrfv/OfuZs5c1ZHLSl1rYp1pmCM2Qf0LrxNRJ4Exl7j67cTkXXAQeAZY8zGa3w+5QTCA7wID/AiNiqI6av2s3rvUT79cxfLd6UzZXgbAr21VqNSJVXiIakiss8YU/MKbWoDcy7RpxCAVZI7Q0R6Ae8aY4pcl1FERgIjAWrWrNly7969JYpZVV6/bjzMo1+uIaqKD1MfaENkoLejQ1KqXCmLNZqvafiHMeakMSbDdnsu4C4ioZdoO94YE2+MiQ8LC7uWl1WVVI/GVZkyvA1HTmbx4JTVHMs8W2S7HSmnmLx0D899l8TSHWllHKVS5V9xO5qLck2z3kSkKnDEGGNEpDVWgrpS57VSl9S2TgjvDmjBI1+u4a6PlvL58NZUC/TmTE4evp5u7EnL5MZ3FvHXyfH0Vfu5sWEE+4+dITrUh5a1gomq4k2n+mG61oNyWpdNCiJyiqI//AW47Pm5iEwDOgOhIpIMvIRtYR5jzMdYndUPi0gucAYYYCra9GpV7nRvFMHUEW0YPmkVXd/+g7O5+Xi4ufBczwa4uQjGwKxH21M/wp/nZ67n981HaBQZwIpdR5m73qq/1KxGIBOGtCLUz9PB70apsqdlLlSltO3IKf7fnE38uf38S0TVg7xZ/I8uF01+M8Zw/HQOi3ek8ew364gI8GLCkFbUDfMry7DVinEoAAAcEElEQVSVsptSnbxWnmhSUMWVk5fPH1tTuT4mhHHzdzBz7QEGtKrJE92LHM9QYO2+YwyfnMCprBxqVPHhwU51GND6smMqlCr3NCkodQ2OnMziH98msXBrKq4uwowH29KyVrCjw1KqxMpi9JFSlVZEgBeThrZm3Us9qB7kzaNT15KWke3osJSyO00KSl1GoLc7Hw2O49jps4yYnEBGdq6jQ1LKrjQpKHUFjasF8v7AFqw/cIJ7/7eCbUdOOTokpexGk4JSxdCjcVXGDWzBrtRMer77Jy/P3siJ0zmODkupUqdJQali6tk0koXPdGZAqyg+X7aHLm8vZNrKfeTnV6zBGkpdjiYFpa5CFV8P/n1HU354vAMxYX4899167vxoKRsOnHB0aEqVCk0KSpVA42qBfPVgW97p35zkY6fp88ESJi3ZfeUdlSrnNCkoVUIiwh0tajDv6c50uS6cl3/YxNz1hxwdllLXRJOCUtco0MedcYNa0LJWFZ6cnsiMhP2ODkmpEtOkoFQp8HJ35X/3x9Mqugp//yaJF2dtICcv39FhKXXVNCkoVUqCfDyYPLQ1D3SM5vNlexkycSWnz+pkN1WxaFJQqhS5ubrwwi2NeKtfc5btTGfIxFVk6ixoVYFoUlDKDvq2rMHYAS1YvfcYQyetIisnz9EhKVUsmhSUspPezavxTv9YVu4+ym3vLyYp+bijQ1LqijQpKGVHvZtXY9LQVmRk5zJg/HJW7z3m6JCUuixNCkrZWefrwpn1aHvC/T0ZMmElP6w76OiQlLokTQpKlYHwAC++fKAtMRF+PPVVop4xqHJLk4JSZaRakDcTh7QiMsiLez5bzpwkPWNQ5Y8mBaXKUJCPB9893J4m1QJ5fNpanf2syh1NCkqVsTB/T74Y0YYOMaE89916Fm5NcXRIShXQpKCUA3i5u/LhPXHUj/Bn5Oermbx0D8bougzK8TQpKOUg/l7ufDmiDR3qhfLS7I28/vNWTQzK4eyWFERkgoikiMiGSzwuIvKeiOwQkSQRibNXLEqVV1V8PfjsvngGt63Jx3/s5MVZGzmbq4X0lOPY80xhEnDzZR7vCdSz/YwEPrJjLEqVWy4uwqu9mzCyUx2mLN/LgPHLOHwiy9FhKSdlt6RgjFkEHL1Mkz7A58ayHAgSkUh7xaNUeebiIjzfqyHjBrVgy+FTDBi/jORjpx0dlnJCjuxTqA4UHo+XbNt2EREZKSIJIpKQmppaJsEp5Qi3NqvGlOFtSM84y+0fLNHEoMqcI5OCFLGtyF42Y8x4Y0y8MSY+LCzMzmEp5Vgta1Xhu0euJzsnnwenrObMWa2wqsqOI5NCMhBV6H4NQKd4KgXUi/Dnnf6xbDp0kuGTV+liParMODIpzAbus41CagucMMboqudK2XRvFMHb/ZqzfFc6d364lN1pmY4OSTkBew5JnQYsA64TkWQRGS4iD4nIQ7Ymc4FdwA7gU+ARe8WiVEV1Z1wNPrs/npRT2QwYv4xdqRmODklVclLRJsvEx8ebhIQER4ehVJnaevgUgz5dTm6+YczdzenWMMLRIakKRkRWG2Pir9ROZzQrVQFcV9WfmY+0JyrYm+GTE3jtpy3k5ukkN1X6NCkoVUHUDPHhm4euZ1Aba/bzoM9WkHJSJ7mp0qVJQakKxMvdlf/c0ZR3+jdnffIJer33Jwu0yqoqRZoUlKqA7mhRg9mPtSfY14OhE1fxwsz1ejlJlQpNCkpVUPUi/Pnh8Q6M7FSHqSv2cfcny9iRcsrRYakKTpOCUhWYp5srz/dqyNj+sexKy6TXu4t5f952snN1FrQqGU0KSlUCt7eozu9P38CNjSN4+7dtdHv7D35Yd1DXZ1BXTZOCUpVEqJ8nHwyK44vhbfDzdOPxaWu548OlJOy5XLFipc6nSUGpSqZDvVB+HNWRN/o249CJM/T9eBkvz95IVo5eUlJXpklBqUrI1UW4Oz6KBc90Zmj72kxauofOby5kyY40R4emyjlNCkpVYj4ebrx0W2OmPdAWfy83hkxcydQVe3XJT3VJmhSUcgLt6obwzcPXExsVxAszN9DpjQVMXrqHHJ3boC6gSUEpJxHo7c6MB9sxeVhraoX48NLsjdz0ziJ+3XhYRympApoUlHIiIsIN9cOYPrItE4bE4+IijJyymgHjl5OUfNzR4alyQJOCUk5IROjaIIKfn+jIv25vwo6UDHqPW8Lbv27V/gYnp0lBKSfm5urC4La1WPhsZ+6Kq8H783fQ9W0dpeTMNCkopfD3cuetfs2YPKw1Xu6u3Pu/Fbw/bzt5+drX4Gw0KSilgHP9DbMebU/v5tV4+7dtDP5sBUd0zQanoklBKXUeX0833ukfyxt9m5G4/zhd31rI2N+3kZGd6+jQVBnQpKCUuoiINSN67hMd6VQ/jLG/b6fTGwuYsHi3dkRXcpoUlFKXFB3qy0eDW/L9o+25LsKfV+ds4uZ3F+lqb5WYJgWl1BXFRgXx5QNtmDAkHmNg6MRVDJ+0it1pmY4OTZUyTQpKqWIpmNvwZEdG92zA8l3p9HjnD/7702ZOZeU4OjxVSjQpKKWuiqebKw/dUJcFz3amT2x1PvljF13e+oMZq/aTr0NYKzy7JgURuVlEtorIDhEZXcTjQ0QkVUQSbT8j7BmPUqr0hPt78Va/5sx6tD01g735+7dJ9PlgiS7qU8GJvQphiYgrsA24EUgGVgEDjTGbCrUZAsQbYx4r7vPGx8ebhISE87bl5OSQnJxMVpaOpy5NXl5e1KhRA3d3d0eHoso5Ywyz1x3kv3O3cPhkFrc1r8bong2oHuTt6NCUjYisNsbEX6mdmx1jaA3sMMbssgU0HegDbLrsXiWQnJyMv78/tWvXRkRK++mdkjGG9PR0kpOTiY6OdnQ4qpwTEfrEVufGRhF8/McuPvljJ79tOsyDnery0A118fZwdXSIqpjsefmoOrC/0P1k27YL3SUiSSLyjYhEFfVEIjJSRBJEJCE1NfWix7OysggJCdGEUIpEhJCQED37UlfFx8ONp2+sz7y/3UC3hhG8O287Xd9eyKzEA1qeu4KwZ1Io6hP6wr+KH4DaxphmwO/A5KKeyBgz3hgTb4yJDwsLK/rFNCGUOj2mqqRqVPHhg0FxzHiwHcG+HjwxPZF+Hy/T8twVgD2TQjJQ+Jt/DeBg4QbGmHRjTLbt7qdASzvGYzfp6enExsYSGxtL1apVqV69esH9s2fPFus5hg4dytatW+0cqVJlq3V0MLMf68DrdzVlT3omfT5YwtNfJXLg+BlHh6YuwZ59CquAeiISDRwABgCDCjcQkUhjzCHb3d7AZjvGYzchISEkJiYC8PLLL+Pn58czzzxzXhtjDMYYXFyKzsMTJ060e5xKOYKri9C/VU16NY1k3IIdTFyyhznrDzGsfTQPd65LoLcOZChP7HamYIzJBR4DfsH6sJ9hjNkoIq+KSG9bs1EislFE1gGjgCH2iscRduzYQZMmTXjooYeIi4vj0KFDjBw5kvj4eBo3bsyrr75a0LZDhw4kJiaSm5tLUFAQo0ePpnnz5rRr146UFC0poCo+fy93nuvZkAXPdObWppF8smgnN7y5gM/+3EVWTp6jw1M29jxTwBgzF5h7wbYXC91+DniuNF/zlR82sungydJ8ShpVC+Cl2xqXaN9NmzYxceJEPv74YwBee+01goODyc3NpUuXLvTt25dGjRqdt8+JEye44YYbeO2113j66aeZMGECo0dfNM1DqQqpepA3Y/rHMrxjNK//vJV//biZj//YRf9WNRjWPpoQP09Hh+jUdEazndWtW5dWrVoV3J82bRpxcXHExcWxefNmNm26eISut7c3PXv2BKBly5bs2bOnrMJVqsw0rhbI58Na8+WINsRGBfHRwp10fGMBb/2ylROntWyGo9j1TMERSvqN3l58fX0Lbm/fvp13332XlStXEhQUxODBg4sc8unh4VFw29XVldxcrWOvKq/rY0K5PiaUHSkZjP19G+MW7GDysj2M7FiHoR2i8fOsdB9T5ZqeKZShkydP4u/vT0BAAIcOHeKXX35xdEhKlRsx4X6MGxTHT090pG2dEN7+bRsdX5/PJ3/s5KQW3CszmoLLUFxcHI0aNaJJkybUqVOH9u3bOzokpcqdhpEBfHpfPOv2H2fMb9v4709bGPv7dvrEVuOeNrVoWiPQ0SFWanarfWQvRdU+2rx5Mw0bNnRQRJWbHlvlaOuTTzB1xV5mJR7kTE4ezWsEck+bWtzWvJqWz7gKxa19pJePlFLlWtMagbx2VzNWvNCNV3o35vTZPP7+bRKt//M7/2/OJvaln3Z0iJWKXj5SSlUIAV7u3H99be5rV4tVe44xZfleJi/dw4Qlu+nWIILBbWvSISYUN1f9rnstNCkopSoUEaF1dDCto4M53KshU1fsZeqKffy++Qihfp7c1jyS22Or06xGoNbvKgFNCkqpCqtqoBd/63Edj3WNYcGWVGYlHmDq8n1MXLKHOqG+3Na8Grc1r0ZMuJ+jQ60wNCkopSo8TzdXbm5SlZubVOXEmRx+3nCImWsP8N787bw7bzuNIgO4rXk1bm0WSVSwj6PDLdc0KSilKpVAb3f6t6pJ/1Y1OXIyix+TDvFD0kFe/3kLr/+8heZRQXRvEE7XhuE0igzQS0wX0B6ZUtC5c+eLJqKNHTuWRx555JL7+PlZp7MHDx6kb9++l3zeC4ffXmjs2LGcPn1u9EWvXr04flxr1isFEBHgxbAO0cx8pD1//r0Lz950HQKM+X0bt7y3mHb/nc/zM9czb/MRzpzVonygZwqlYuDAgUyfPp2bbrqpYNv06dN58803r7hvtWrV+Oabb0r82mPHjmXw4MH4+FinxHPnzr3CHko5p6hgHx7tEsOjXWJIPZXNwq0pzN+Swqy1B/hyxT483Vy4vm4IXRtG0DEmlFohPk55FqFnCqWgb9++zJkzh+xsa72gPXv2cPDgQWJjY+nWrRtxcXE0bdqUWbNmXbTvnj17aNKkCQBnzpxhwIABNGvWjP79+3PmzLmFSB5++OGCktsvvfQSAO+99x4HDx6kS5cudOnSBYDatWuTlpYGwJgxY2jSpAlNmjRh7NixBa/XsGFDHnjgARo3bkyPHj3Oex2lnEGYvyf94qP4aHBL1rx4I18Mb8OgNjXZmZrJ/32/gc5vLaTNf+bx5PS1zFi1n/1HnWcuROU7U/hpNBxeX7rPWbUp9Hztkg+HhITQunVrfv75Z/r06cP06dPp378/3t7ezJw5k4CAANLS0mjbti29e/e+5LePjz76CB8fH5KSkkhKSiIuLq7gsX//+98EBweTl5dHt27dSEpKYtSoUYwZM4YFCxYQGhp63nOtXr2aiRMnsmLFCowxtGnThhtuuIEqVaqwfft2pk2bxqeffsrdd9/Nt99+y+DBg0vnWClVwXi6udKhXigd6oXy4q2N2JmayfJd6azYfZTFO9L5PtFaMDIq2JtWtYNpERVE86ggGlQNwMOt8n2vrnxJwUH+uoT0V1KYMGECxhief/55Fi1ahIuLCwcOHODIkSNUrVq1yOdYtGgRo0aNAqBZs2Y0a9as4LEZM2Ywfvx4cnNzOXToEJs2bTrv8QstXryYO+64o6BK65133smff/5J7969iY6OJjY2FtDS3EoVJiLEhPsRE+7H4La1MMawPSWDZTvTWbIjjUXbUvluzQEAPNxcaFwtgNioIBpXCyQ61JdGkQEVvvRG5UsKl/lGb0+33347Tz/9NGvWrOHMmTPExcUxadIkUlNTWb16Ne7u7tSuXbvIUtmFFXUWsXv3bt566y1WrVpFlSpVGDJkyBWf53I1rTw9zy1i4urqqpePlLoEEaF+hD/1I/y5//raGGNIPnaGdcnHSdx3nHXJx5m2ch9ZOfmAtfRog6r+NK0eSN0wP+qG+1In1I8aVbwrzEzrypcUHMTPz4/OnTszbNgwBg4cCFgrqIWHh+Pu7s6CBQvYu3fvZZ+jU6dOTJ06lS5durBhwwaSkpIAq+S2r68vgYGBHDlyhJ9++onOnTsD4O/vz6lTpy66fNSpUyeGDBnC6NGjMcYwc+ZMpkyZUvpvXCknIiJEBfsQFezDrc2qAZCTl8/+o6fZkZJhJYv9x/ll42GOFVooyMPVhVohPtQJ86VOmB91w/yoE+ZL3VA/An3K1xrVmhRK0cCBA7nzzjuZPn06APfccw+33XYb8fHxxMbG0qBBg8vu//DDDzN06FCaNWtGbGwsrVu3BqB58+a0aNGCxo0bX1Rye+TIkfTs2ZPIyEgWLFhQsD0uLo4hQ4YUPMeIESNo0aKFXipSqpS5u7pQJ8yPOmF+9Gh87tLwscyz7ErLYGdKJjvTMtiVmsmOlAzmbU4hN//cmXyonwd1Qq0kER3qS9VAL8L8PYkI8CLAy51Ab/cy7bvQ0tnqsvTYKlW6/jqz2JWayc5UK1nsSstgZ2omRzPPFrlPqJ8njasFMHlY6xK/bnFLZ+uZglJKlaHCZxbdiTjvsZNZOaSczCblZBYpp7I5lZVDeuZZjpzMwtOtbDqwNSkopVQ5EeDlToCXu0ML+FWM7nCllFJlotIkhYrWN1IR6DFVyvnYNSmIyM0islVEdojI6CIe9xSRr2yPrxCR2iV5HS8vL9LT0/VDrBQZY0hPT8fLy8vRoSilypDd+hRExBX4ALgRSAZWichsY8ymQs2GA8eMMTEiMgB4Heh/ta9Vo0YNkpOTSU1NLY3QlY2Xlxc1atRwdBhKqTJkz47m1sAOY8wuABGZDvQBCieFPsDLttvfAONERMxVfuV3d3cnOjr62iNWSiknZ8/LR9WB/YXuJ9u2FdnGGJMLnABCLnwiERkpIgkikqBnA0opZT/2TApFlQK98AygOG0wxow3xsQbY+LDwsJKJTillFIXs2dSSAaiCt2vARy8VBsRcQMCgaN2jEkppdRl2LNPYRVQT0SigQPAAGDQBW1mA/cDy4C+wPwr9SesXr06TUQuX1nu0kKBtBLuWxnp8TifHo9z9FicrzIcj1rFaWS3pGCMyRWRx4BfAFdggjFmo4i8CiQYY2YD/wOmiMgOrDOEAcV43hJfPxKRhOLU/nAWejzOp8fjHD0W53Om42HXMhfGmLnA3Au2vVjodhbQz54xKKWUKr5KM6NZKaXUtXO2pDDe0QGUM3o8zqfH4xw9FudzmuNR4dZTUEopZT/OdqaglFLqMpwmKVypOF9lJCITRCRFRDYU2hYsIr+JyHbb7yq27SIi79mOT5KIxDku8tInIlEiskBENovIRhF5wrbdWY+Hl4isFJF1tuPxim17tK045XZbsUoP2/ZSKV5ZnomIq4isFZE5tvtOeSycIikUKs7XE2gEDBSRRo6NqkxMAm6+YNtoYJ4xph4wz3YfrGNTz/YzEviojGIsK7nA34wxDYG2wKO2vwFnPR7ZQFdjTHMgFrhZRNpiFaV8x3Y8jmEVrYRCxSuBd2ztKpsngM2F7jvnsTDGVPofoB3wS6H7zwHPOTquMnrvtYENhe5vBSJttyOBrbbbnwADi2pXGX+AWVgVfJ3+eAA+wBqgDdYELTfb9oL/N1jzjdrZbrvZ2omjYy/FY1AD60tBV2AOVgkepzwWTnGmQPGK8zmLCGPMIQDb73Dbdqc5RrbT/RbACpz4eNgulyQCKcBvwE7guLGKU8L577lYxSsrsLHA34F82/0QnPRYOEtSKFbhPSfnFMdIRPyAb4EnjTEnL9e0iG2V6ngYY/KMMbFY35JbAw2Lamb7XWmPh4jcCqQYY1YX3lxE00p/LMB5kkJxivM5iyMiEglg+51i217pj5GIuGMlhKnGmO9sm532ePzFGHMcWIjV1xJkK04J57/nyly8sj3QW0T2ANOxLiGNxTmPhdMkhYLifLYRBAOwivE5o7+KEGL7PavQ9vtso27aAif+uqxSGYiIYNXa2myMGVPoIWc9HmEiEmS77Q10x+pkXYBVnBIuPh5/HadiFa+sKIwxzxljahhjamN9Nsw3xtyDEx4LwDk6mm3/Xr2AbVjXTV9wdDxl9J6nAYeAHKxvN8Oxrn3OA7bbfgfb2grWCK2dwHog3tHxl/Kx6IB1ip8EJNp+ejnx8WgGrLUdjw3Ai7btdYCVwA7ga8DTtt3Ldn+H7fE6jn4PdjounYE5znwsdEazUkqpAs5y+UgppVQxaFJQSilVQJOCUkqpApoUlFJKFdCkoJRSqoAmBVXuiEieiCTaKniuEZHrr9A+SEQeKcbzLhQRp1hnt7hEZJKI9L1yS+UsNCmo8uiMMSbWWBU8nwP+e4X2QcAVk4KjFJoVq1S5p0lBlXcBWGWLERE/EZlnO3tYLyJ9bG1eA+razi7etLX9u63NOhF5rdDz9bOtI7BNRDra2rqKyJsissq2dsKDtu2RIrLI9rwb/mpfmIjsEZHXbc+5UkRibNsnicgYEVkAvG5bt+F72/MvF5Fmhd7TRFusSSJyl217DxFZZnuvX9tqNiEir4nIJlvbt2zb+tniWycii67wnkRExtme40fOFQBUyuLo2XP6oz8X/gB5WDOOt2BVoGxp2+4GBNhuh2LNKBUuLg/eE1gK+Nju/zVLeSHwtu12L+B32+2RwD9ttz2BBCAa+Bu22e+AK+BfRKx7CrW5j3OzYSdhlWB2td1/H3jJdrsrkGi7/TowttDzVbG9t0WAr23bP4AXgWCsEt5/TToNsv1eD1S/YNul3tOdWBVRXYFqwHGgr6P/zfWn/Pzoaa0qj84Yq3onItIO+FxEmmAlgP+ISCesEsfVgYgi9u8OTDTGnAYwxhQuVvZXIbzVWMkEoAfQrNC19UCsxXVWARNshfS+N8YkXiLeaYV+v1No+9fGmDzb7Q7AXbZ45otIiIgE2mId8NcOxphjtqqdjYAlVskmPIBlwEkgC/jM9i1/jm23JcAkEZlR6P1d6j11AqbZ4jooIvMv8Z6Uk9KkoMo1Y8wyEQkFwrC+3YdhnTnk2KpaehWxm3DpUsbZtt95nPv7F+BxY8wvFz2RlYBuAaaIyJvGmM+LCvMStzMviKmo/YqKVYDfjDEDi4inNdANK5E8hrV62kMi0sYWZ6KIxF7qPYlIryJeT6kC2qegyjURaYB1qSMd69tuii0hdAFq2ZqdAvwL7fYrMExEfGzPEXyFl/kFeNh2RoCI1BcRXxGpZXu9T7EqrF5qneb+hX4vu0SbRcA9tufvDKQZaz2HX7E+3P96v1WA5UD7Qv0TPraY/IBAY8xc4EmsZTQRkbrGmBXGmBexVgGLutR7ssUxwNbnEAl0ucKxUU5GzxRUeeQt1opgYH3jvd8YkyciU4EfRCSBc30OGGPSRWSJiGwAfjLGPGv7tpwgImeBucDzl3m9z7AuJa0R63pNKnA7VsXMZ0UkB8jA6jMoiqeIrMD6knXRt3ubl4GJIpIEnOZc6eV/AR/YYs8DXjHGfCciQ4BpIuJpa/dPrOQ3S0S8bMflKdtjb4pIPdu2ecA6rOqnRb2nmVh9Guuxqgb/cZnjopyQVklV6hrYLmHFG2PSHB2LUqVBLx8ppZQqoGcKSimlCuiZglJKqQKaFJRSShXQpKCUUqqAJgWllFIFNCkopZQqoElBKaVUgf8PAHJrINA3MjgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_losses()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model is a little worse in accuracy than the Naive Bayes Bigram model.\n",
    "\n",
    "But our Neural Network is much more computationally intense and has about 4 times as many parameters!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1056"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(len(_) for _ in learn.model.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regularisation: Dropout\n",
    "\n",
    "Adding 50% dropout increases our accuracy a little above what we got with Naive Bayes; to 55%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, MyLetterRNN(n_layers=2, dropout=0.5), loss_func=F.cross_entropy, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:12 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.770683</th>\n",
       "    <th>2.746132</th>\n",
       "    <th>0.206250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.737838</th>\n",
       "    <th>2.597004</th>\n",
       "    <th>0.302083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.625530</th>\n",
       "    <th>2.253647</th>\n",
       "    <th>0.272917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.453825</th>\n",
       "    <th>2.077854</th>\n",
       "    <th>0.350000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>2.315888</th>\n",
       "    <th>1.959774</th>\n",
       "    <th>0.372917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>2.161286</th>\n",
       "    <th>3.241577</th>\n",
       "    <th>0.212500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>2.172942</th>\n",
       "    <th>1.954330</th>\n",
       "    <th>0.412500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>2.082301</th>\n",
       "    <th>1.910599</th>\n",
       "    <th>0.429167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.960596</th>\n",
       "    <th>1.690655</th>\n",
       "    <th>0.493750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.844915</th>\n",
       "    <th>1.704401</th>\n",
       "    <th>0.495833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.749084</th>\n",
       "    <th>1.702470</th>\n",
       "    <th>0.462500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.649408</th>\n",
       "    <th>1.623091</th>\n",
       "    <th>0.491667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>1.574916</th>\n",
       "    <th>1.608889</th>\n",
       "    <th>0.506250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>1.477580</th>\n",
       "    <th>1.634902</th>\n",
       "    <th>0.529167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>1.378102</th>\n",
       "    <th>1.681566</th>\n",
       "    <th>0.497917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>1.285141</th>\n",
       "    <th>1.700663</th>\n",
       "    <th>0.506250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>1.198917</th>\n",
       "    <th>1.739364</th>\n",
       "    <th>0.510417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>1.111936</th>\n",
       "    <th>1.807743</th>\n",
       "    <th>0.520833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>1.044053</th>\n",
       "    <th>1.796086</th>\n",
       "    <th>0.495833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>0.981998</th>\n",
       "    <th>1.776384</th>\n",
       "    <th>0.522917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>21</th>\n",
       "    <th>0.910434</th>\n",
       "    <th>1.867425</th>\n",
       "    <th>0.522917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>22</th>\n",
       "    <th>0.834056</th>\n",
       "    <th>1.867015</th>\n",
       "    <th>0.522917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>23</th>\n",
       "    <th>0.770921</th>\n",
       "    <th>1.860593</th>\n",
       "    <th>0.537500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>24</th>\n",
       "    <th>0.708352</th>\n",
       "    <th>1.918845</th>\n",
       "    <th>0.543750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>25</th>\n",
       "    <th>0.653929</th>\n",
       "    <th>1.950722</th>\n",
       "    <th>0.522917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>26</th>\n",
       "    <th>0.603225</th>\n",
       "    <th>1.974899</th>\n",
       "    <th>0.531250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>27</th>\n",
       "    <th>0.564611</th>\n",
       "    <th>1.989453</th>\n",
       "    <th>0.535417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>28</th>\n",
       "    <th>0.534487</th>\n",
       "    <th>2.007739</th>\n",
       "    <th>0.539583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>29</th>\n",
       "    <th>0.512126</th>\n",
       "    <th>2.006823</th>\n",
       "    <th>0.545833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>30</th>\n",
       "    <th>0.493982</th>\n",
       "    <th>2.006345</th>\n",
       "    <th>0.547917</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(30, max_lr=3e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4VFX6wPHvmx7SCYSSEEJvIYRQQhfEQlFRRAFFRUAE667uuu6666o/3cWO2FBXURRBRYqiiAUElBo6BJAWICSEEEiBQEg5vz/uEAIEEpKZTMr7eZ55ZubOnXPfe8W8c849RYwxKKWUUgAuzg5AKaVU5aFJQSmlVCFNCkoppQppUlBKKVVIk4JSSqlCmhSUUkoV0qSglFKqkCYFpZRShTQpKKWUKuTm7ACuVJ06dUxERISzw1BKqSpl3bp1R40xdUvar8olhYiICOLi4pwdhlJKVSkisr80+2nzkVJKqUKaFJRSShXSpKCUUqpQlbunoJSqPnJzc0lMTOT06dPODqXa8PLyIiwsDHd39zJ9X5OCUsppEhMT8fPzIyIiAhFxdjhVnjGGtLQ0EhMTadKkSZnK0OYjpZTTnD59muDgYE0IdiIiBAcHl6vmpUlBKeVUmhDsq7zXs8YkhZ2Hs5i0cAeZp3OdHYpSSlVaNSYpHDiWzdSle9h95ISzQ1FKVRJpaWlER0cTHR1N/fr1CQ0NLXx/5syZUpVx7733snPnTgdHWnFqzI3m5iG+AOw5coKY8CAnR6OUqgyCg4PZuHEjAM888wy+vr785S9/OW8fYwzGGFxciv8NPW3aNIfHWZFqTFJoFOSNh6sLC7ceZu/Rk5zMyaNHs2BAOHgsm9QTOeTlG1rV9yW6URDNQ3xxddG2TqVqot27d3PzzTfTq1cvVq9ezYIFC3j22WdZv349p06dYvjw4Tz99NMA9OrVi7feeovIyEjq1KnDhAkTWLhwIbVq1WL+/PmEhIQ4+WyuTI1JCm6uVpZfvOMIi3ccAWD6ynNTgXi6ueAiwqncfABqebjSPjSA6EaBtA8LoHfzugTUKlu/X6VUyZ79dhvxSZl2LbNtQ3/+fWO7Mn03Pj6eadOmMXXqVAAmTZpE7dq1ycvLo1+/fgwbNoy2bdue952MjAyuuuoqJk2axGOPPcZHH33Ek08+We7zqEg1JikA9GgezK87U1nyl774eLjy2+6jNA6uRYt6fvh5umEMJBw5zplf/ssc72GsTs5j2u8JnMkvwMPVhchQf9JP5dIhLJDbOoXRrWkwLlqbUKpaatasGV26dCl8P3PmTD788EPy8vJISkoiPj7+oqTg7e3NwIEDAejUqRPLly+v0JjtoUYlhTeGd+RUbj71A7wAGBoTdt7nItD0zC7Y8xH/aLAOxswhx92PbUmZfLspiY0H02lax4eft6cwd8Mh2ocG8OE9nQnx93LG6ShVrZT1F72j+Pj4FL7etWsXb7zxBmvWrCEwMJBRo0YVOxbAw8Oj8LWrqyt5eXkVEqs91aikEFDLnQBKaAIKj4Xbp8OX98D0IXjeNZeY8Nrn3Zw+nZvPN5uSeOabbdzyzgr+O7Q9fVqWOE25UqqKyszMxM/PD39/f5KTk1m0aBEDBgxwdlgOUWO6pF6R1oNhxAw4Eg/Tb4KTaed97OXuyu2dG/Hl/d3xcHPh7o/W8PyCeIwxTgpYKeVIMTExtG3blsjISO677z569uzp7JAcRqraH7LOnTubCltkZ/fPMOtOqN0M7p4PvhfXBnLy8nnhu+1MX7mfO2LDeX5IpN5nUKqUtm/fTps2bZwdRrVT3HUVkXXGmM4lfVdrCpfT/Bq44ws4thc+uQGyUi7axdPNlWdvasfEvs34fPUB/vLVJvLyC5wQrFJKlZ8mhZI07Qt3fgXpB+HjwZCZfNEuIsIT17fi8WtbMmfDIZ6YvbnCw1RKKXvQpFAaTXrDqNmQlWwlhoxDF+0iIjzcvwWPXN2cORsOsWTnEScEqpRS5aNJobQa94BRc+DEEfh4EKQfKHa3h65uQaPa3rz6406yz1S97mhKqZpNk8KVCI+1bjhnH4dpg+F4wkW7eLi5cE/3CLYeyqTfK79yJk/vLyilqg5NClcqrBPcMx9yMq3EUEyNYVS3xgxqX5+UzBzmrE90QpBKKVU2mhTKomFHuOdbyD4Kv0+56GMvd1feviOGmPBAnv02ns2J6U4IUilVkr59+7Jo0aLztk2ePJkHHnjgkt/x9bVmXE5KSmLYsGGXLLekrvOTJ08mOzu78P2gQYNIT3f+3wpNCmXVIMrqsrrjOyi4uIlIRJh6Vydq+3gw7pM4ftmewoo9R50QqFLqUkaOHMmsWbPO2zZr1ixGjhxZ4ncbNmzI7Nmzy3zsC5PC999/T2BgYJnLsxdNCuXR5kbISoKkDcV+HOLnxft3d+J49hnGfhLHHR+s5uGZG/gpPoVTZ/IrOFil1IWGDRvGggULyMnJASAhIYGkpCSio6Pp378/MTExtG/fnvnz51/03YSEBCIjIwE4deoUI0aMICoqiuHDh3Pq1KnC/SZOnEjnzp1p164d//73vwGYMmUKSUlJ9OvXj379+gEQERHB0aPWD8fXXnuNyMhIIiMjmTx5cuHx2rRpw3333Ue7du247rrrzjuOvdSouY/sruX14OIG27+x7jUUo13DAL64vzsnc/JYu+8Yby7Zzbebkgiq5U6nxkHU9fPEGKvJ6b4+TQkN9K7gk1Cqklj4JBzeYt8y67eHgZMu+XFwcDBdu3blhx9+YMiQIcyaNYvhw4fj7e3N3Llz8ff35+jRo3Tr1o2bbrrpkusfv/vuu9SqVYvNmzezefNmYmJiCj974YUXqF27Nvn5+fTv35/NmzfzyCOP8Nprr7FkyRLq1KlzXlnr1q1j2rRprF69GmMMsbGxXHXVVQQFBbFr1y5mzpzJBx98wO23387XX3/NqFGj7HOtbLSmUB7eQdCkD2z/Fi4zXUhMeBC9W9Tlsetasf6f1zJjXCyNg334efsRZq45yKy1B/l4RQLXvbaUwxkXz7yolHKcok1IZ5uOjDH84x//ICoqimuuuYZDhw6RknLxjAZnLVu2rPCPc1RUFFFRUYWfffnll8TExNCxY0e2bdtGfHz8ZeP57bffuOWWW/Dx8cHX15ehQ4cWTsHdpEkToqOjAWtq7oSEhPKcerG0plBerW+A7x6D1B0QUvIcLkE+HvRsXofYJrU5dvIMq/cd49OV+7mnRwQPfr6eT1clcHXresSEB17yV4lS1dJlftE70s0338xjjz1WuKpaTEwMH3/8Mampqaxbtw53d3ciIiKKnSq7qOL+f923bx+vvPIKa9euJSgoiNGjR5dYzuXmo/P09Cx87erq6pDmI60plFfrwYBYtYUr4ObqQoi/Fzd2aMiXE7ozOKoBXZvU5u0le7j13RU8PX+bY+JVSp3H19eXvn37MmbMmMIbzBkZGYSEhODu7s6SJUvYv3//Zcvo06cPM2bMAGDr1q1s3mxNdZOZmYmPjw8BAQGkpKSwcOHCwu/4+fmRlZVVbFnz5s0jOzubkydPMnfuXHr37m2v0y2RJoXy8qsPjbpa9xXK6b1RnfjHoNZ0bxrMp6v2s2Bzkh0CVEqVZOTIkWzatIkRI0YAcOeddxIXF0fnzp2ZMWMGrVu3vuz3J06cyIkTJ4iKiuKll16ia9euAHTo0IGOHTvSrl07xowZc96U2+PHj2fgwIGFN5rPiomJYfTo0XTt2pXY2FjGjRtHx44d7XzGl+awqbNFpBEwHagPFADvG2PeuGCfvsB8YJ9t0xxjzHOXK7dCp84urRVvwo//hEc3QVBEuYs7mZPHLe/8zh8pJ5h2bxf6tapaC38rVVo6dbZjVNaps/OAx40xbYBuwIMi0raY/ZYbY6Jtj8smhEqr9Q3W8/YFdinOx9ONbx7qRat6fvxt9mbSs8/YpVyllCqJw5KCMSbZGLPe9joL2A6EOup4TlW7CdRrf8X3FS7Hy92VV2/vwLGTZ5j42Xpy8nRcg1LK8SrknoKIRAAdgdXFfNxdRDaJyEIRKXblbhEZLyJxIhKXmprqwEjLoc2NcHB1sQvxlFVkaAAv3xbFyr1pvLhwp93KVaoyqWqrP1Z25b2eDk8KIuILfA38yRiTecHH64HGxpgOwJvAvOLKMMa8b4zpbIzpXLfuxUtiVgptbgQM7PzOrsXe0jGM0T0i+Oj3fQx/byV7U0/YtXylnMnLy4u0tDRNDHZijCEtLQ0vL68yl+HQNZpFxB1YACwyxrxWiv0TgM7GmEtOElQpbzSDNXjtzRjrRvNdc+1adE5ePk/N3cq3m5Lw9XTjs3GxtGngb9djKOUMubm5JCYmlth3X5Wel5cXYWFhuLu7n7e9tDeaHTZ4TayRHB8C2y+VEESkPpBijDEi0hWr5pLmqJgcSsSqLax8G06lg7f9JrbydHPllds68EDfZtz+3iqe/y6eGeO62a18pZzF3d2dJk2aODsMVYQjm496AncBV4vIRttjkIhMEJEJtn2GAVtFZBMwBRhhqnI9ss1NUJAHfywqed8yaFrXl7G9mvD77jT+SLl40ItSSpWXI3sf/WaMEWNMVJEup98bY6YaY6ba9nnLGNPOGNPBGNPNGLPCUfFUiIYx4NfALgPZLuW2zmG4CLy3dK/2SFJK2Z2OaLYnFxdrzMLuX+BMdsn7l0EdX0+6NQ3m6/WJ3DttrSYGpZRdaVKwtzY3Qt4p2POLww7xzE3tGBzVgBV70hj3SRx5+boOtFLKPjQp2FvjntaU2nYcyHahlvX8ePuOGP5zS3uW7zrKO7/ucdixlFI1iyYFe3N1g1aD4I8fIM+x01PcERvODVENeGuxtXCPUkqVlyYFR2hzI5zOgITlDj/U0ze0pWldHx6euYGpS7XGoJQqH00KjtC0H7j7OLQJ6awQfy8WPNyLa9qE8OYvuzh+UifPU0qVnSYFR3D3ghbXwo7voMDxvYPcXF346/Wtyckr4K+zN+uUAUqpMtOk4ChtboSTRyBxbYUcrlV9P/4+qA0/b0/hy7iDFXJMpVT1o0nBUVpcB64eFdKEdNa9PSLo3jSYp+ZuZePB9Ao7rlKq+tCk4Che/tC0r5UUStOck3EIFj0Faz8s8yFdXIT37u6Er5cbr/64U8cvKKWumCYFR2pzI6Tvh8NbLr1PRiIseAymRMPKt6zEkH2szIf093Ln7m6NWb7rKI/O2ljmcpRSNZMmBUdqNQjEpfgmpPSDsODP8EY0rJ8O0XfAyFnWaOj108t12EevacnoHhF8tyWZ9QeOl6sspVTNoknBkXzqQHgP2FFk7eb0A/Dtn2BKR1j/KcTcBY9sgBvfgFYDIaI3rP0f5OeV+bCuLsITA1rh5+nGZ6v22+FElFI1hSYFR2tzIxyJhz2L4ZtHYEoMbJwBne6BRzfCDa9DYKNz+8dOgIyD5V7BrZaHGzd0aMj3W5I5kqULmCilSkeTgqO1Hmw9f3oLbJoJnUbDIxth8KsQEHbx/q0GQmA4rJpa7kOP79OUggJ4/MtNnMgpe81DKVVzaFJwtMBG0O1BiJ0Ij26Cwa9AQOil93dxha7j4cAKSN5UrkM3qePDs0Pa8fvuowx7d4UmBqVUiTQpVIQB/4GBk8C/Yen273iXNU2GHWoLI7uG8+HoLvyRksWYaWs5eMwx6zwopaoHTQqVkXcgRI+ErbPhRGq5i+vXKoTnhkQSn5zJiPdXcSj9lB2CVEpVR5oUKqvYCZB/BtZNs0txo7o1Ztb4bmSezuU+XZhHKXUJmhQqqzotoPk1VvdUO63LEBkawEu3RhGfnMnEGes5qfcYlFIX0KRQmcVOgBMpED/PbkUOiKzPX69vxeIdR7jq5V+Z/PMf5GqtQSllo0mhMmvWH4Kbw6p3Szd/UimICA/2a857ozrRtqE/k3/exehpa0jO0PsMSilNCpWbi4tVW0haD4lxdi36mrb1mD6mK5OGticu4TiPfbFJ12FQSmlSqPQ6jATPAFj9rkOKH9E1nH/e0JaVe9OI/PciPvxtn0OOo5SqGjQpVHaevtb8SPHzITPJIYcYFRvO3we2BuCF7+L5ffdRhxxHKVX5aVKoCrqMs5b1XPs/hxQvItx/VTPWPHUNzer6cu/Ha3nmm206AlqpGkiTQlVQu4k1DXfcNMh13A1hH083ZoyLZWjHUKavTGDE+yt1Mj2lahhNClVFtwlw6hhsme3Qw4T4ezHp1ij+d09n9hw5yb3T1pJ9RmsMStUUmhSqiojeUC8SVk+1W/fUy7m6dT3evrMj25MzGT1tLTl5+Q4/plLK+TQpVBUiEHs/pGyFhN8q5JBXt67H68OjWbPvGOM+iSM92z4jq5VSlZcmhaqk/W3gXduqLVSQIdGh/Hdoe1bvPcbEz9br6GelqjlNClWJu7e1SM+O7+B4QoUddmTXcP4ztD0r96bx+k9/VNhxlVIVz2FJQUQaicgSEdkuIttE5NFi9hERmSIiu0Vks4jEOCqeaqPLOBAXu6y1cCWGdQrj5uiGTPs9gbQTORV6bKVUxXFkTSEPeNwY0wboBjwoIm0v2Gcg0ML2GA84ZthudRIQCu2HWSOcP+hv9UbKz62QQz/Yrzm5+QWM+SSOM3najKRUdeSwpGCMSTbGrLe9zgK2AxeuQzkEmG4sq4BAEWngqJiqjRsmw8CXrC6qX4+Fye1h6ctw0rEjkVvU8+ONER3ZdDCdt5bsduixlFLOUSH3FEQkAugIrL7go1DgYJH3iVycONSFPGpZPZEeWgd3fAkhbWDJ8/BaW5j3IBze4rBDD45qwNCOoUz5ZRfLd5V/VTilVOXi8KQgIr7A18CfjDGZF35czFcu6oQvIuNFJE5E4lJT9Q9RIRcXaHk93DUXHlgNHe+EbXNgai+YNhjiv7Gmx7Cz/wxtT1iQN6/8+Af5BTqzqlLViUOTgoi4YyWEGcaYOcXskgg0KvI+DLho1jdjzPvGmM7GmM5169Z1TLBVXUhruOF1eCwerv0/SD8AX94Fr0fCNw/Dtnlw6rhdDuXl7sqfr2nJpoPpPP9dvF3KVEpVDm6OKlhEBPgQ2G6Mee0Su30DPCQis4BYIMMYk+yomGoE7yDo+Qh0ewB2fg+bv7ASwvrpVq+l0E7W4j3N+0PDGHAt2z+BWzuFseVQBtN+T8DL3ZUbohrQrmGAnU9GKVXRxFELq4hIL2A5sAU421XlH0A4gDFmqi1xvAUMALKBe40xl11NpnPnziYuzr4LzlR7+blwaB3s/gX2/AKH1gMGvAKgyVVWgmh+DQSEXVGxp87kc+u7K4hPzsTD1YVnbmrHHbHhjjkHpVS5iMg6Y0znEveraqttaVKwg+xjsPdXK0HsXgxZSVYtovfjcNWTV1R7OH7yDAu3HmbehkOsSTjGJ2O6clVLbeJTqrLRpKBKxxhI3QkrpsDGGdAoFoZ+AEGNr6iYnLx8rnt9GSmZp4lsGMCfr21Jz+Z1HBS0UupKaVJQV27LbFjwZ0Dgxtch8tYr+vre1BO8vGgnGw6kczInj6VP9KO2j4djYlWqKjMG8nIg77T1XJAHJt/qLWgKrPcF+UW22Z796kNg2ZpoNSmosjmeAF+Pg8S1ED0KBr5oLQl6BXalZHH95GWM7tGEp2+8cBC7UtVEfh6cPAJZh+FEysXPJ1Mh9/S5P/xFn/PLOFVMzz/Btc+W6aulTQoO632kqqigCLh3Ifw6CZa/CgdXwa0fQsPoUhfRop4fw7s0YvrKBA4ez+bq1iHc1ikMN1edf1FVUaczYd8y6z5c4lrITIbsNIoZVmXNZOxXH3zqWg83T3DzuvSzqwe4uoO4govruefzXrtZr4MiHH6qWlNQl7ZvOcwZb/3iueYZq5urS+n+sB87eYY/f7GRNfuOcSo3H38vN8b0asIDfZvj4abJQVVyBQWQvPFcZ4zENVaTjrsPhHezeur51Qfferbn+uBXD3xCwK1yNplq85Gyj+xjMP8h2Pmd1W315nfBN6TUXzfG8FN8Cp+tPsCyP6zR6NGNAnljRDSNg30cFbVSVy7rMOxZbHXd3rvEVhMA6kdZ3bab9bc6YlTSP/ol0aSg7McYiPsQFj0Fnn5wz7fWfEtXVITh1z9SiUs4xmerDhDg7c7sCd0J8fdyUNCqWtv9M2SlQJM+ENio5P2LY4w1T9jOhdaPnuRN1nafutDsaisJNOt3RT+CKjNNCsr+UuLhkxutds2xP1ptnWWw8WA6d3ywiqiwAD4f1w0Xl+KmwFLqEtZ8AN//5dz72k2tQZhNr4KIPuATfOnv5ufC/hXWaP8d30PGAUAgrAu0GmDVhuu1L3UzaVWiSUE5xuYvYc59MPBliB1f5mJmrTnAk3O2MDQmlFdv64A1uF2pEqx+Hxb+FVoNhr5PWuuV71sKCb/DmSxrn/rtbUmiL4R3t7p47vnFSgK7FsHpDOsGb9O+0GoQtBpYbWoDl6NJQTmGMfDZUDi4Fh5cbS36U6ZiDC/+sJOpS/cwfUxX+ugoaFWSswmh9Q0wbNr5bfv5uZC0AfYutZLEwdWQf8bWa8fFeu1d20oArQZZzUIeNeueliYF5TjH9sE73a2bbyNmlLmYnLx8rnltKeknc3ljZDRXt65nxyBVtbL6PVj4RPEJoThnsuHASqsbqSmwkkGj2DI3eVYHpU0K1a/hTDle7SbQ7++wYwFs/7bMxXi6uTLzvm6EB9dizMdx/HPeFk7n2n/9B1XFrZp6ZQkBrIWomve3Bnpd93/QuEeNTghXQpOCKptuD1g35L7/qzWwp4zCgmrx9cQejO3VhM9WHeCF77ZToAv3qLNWTYUf/mYlhNs+rrLdQasSTQqqbFzd4aY3rL7dvzxXrqK83F351w1tGdm1EZ+u2s8DM9ZT1Zo1lQOsetdKCG1utBKCq7uzI6oRNCmosgvtBLETYO3/4OCachf33JBIJlzVjB+2Hebr9YfsEKCqsla+Az88aSWEYdM0IVQgTQqqfK5+CvxD4dtHrR4g5eDu6sIT17eia0Rtnp6/lQ+W7eVMXkHJX1SVW2Yy7PzB6hl0ZIc1Sv5yNcGV78Civ0ObmzQhOIH2PlLlt3MhzBwB/Z+2Fuopp8MZp/n7nM0s2ZlKy3q+zJ7YA38v/cNQJRTkQ8o2q0vowdVwYLVtgNgFXNytsQG+Idb8QWefc07A6neh7RBrIkZNCHZj1y6pItIMSDTG5IhIXyAKmG6MSS93pFdIk0Il9eXd1q/BB1ZCcDO7FPnD1sM8MGMd17atxz8Ht6VR7Vp2KdepTh6Fz26FBlFwzbNQq7azIyqfnCxr1tADtiSQGHduEJlvfQiPhUbdIDTGmlDuRAqcOHLBs+31yVSr+2i7oTD0fU0IdmbvpLAR6AxEAIuAb4BWxphB5YzzimlSqKSyDsNbXaFhB7j7G7DTCOX3l+1h0sIduLu68MOf+tCkThUecFRQAJ/fZjWjmALwDoLr/wNRt9vtelWY3NOw9EVrxb6CPGuAWEg7WxKwPQLDr+y8CvKt0cZVPVFWUvYep1BgjMkDbgEmG2P+DDQoT4CqmvGrD9c+Yw0W2jTLbsWO79OMBQ/3BuC5b7eRnn3GbmVXuN9ftyZyG/gi3L/MmkNq7niYPgTS9jg7utI7uBbe6wO/vQbtb4O75sLf9sPE32Dwq1aSC2p85YnOxVUTQiVQ2qSQKyIjgXuABbZtWrdT54sZbTUVLPqH1UxiJ20b+vO3Aa1ZsjOVHpMWszbhmN3KrjAJv8Pi560lTjuPgfqRMPYnGPwaJG20Rogvfclamauyyj1lzZT70XVw5iSM+hpumWrNKOrl7+zolJ2UNincC3QHXjDG7BORJsBnjgtLVUkuLnDjZKudedFTdi16TK8mzHmgB8G+Hgx/byVfr0u0a/kOdSIVvh4LQU3ghsnnfkG7uECXsfDQGmg9GJa8AFN7WQmkstm/At7tCSvfgph7rHtHza9xdlTKAa6495GIBAGNjDGbHRPS5ek9hSpg8Quw7CXrV3FA2LlVqXzrnXvt6VemolOzcrhvehx7U0/wxIDWXN+uPnX9PO18AnZUUAAzbrX+qI772ZrB81J2/QTfPQbpB6DjKLj2/5zfnHLmJPz8LKx531q34Ka3rCmqVZVj7xvNvwI3Ya3pvBFIBZYaYx4rZ5xXTJNCFZB72morP7Te6lmSX8x9AHcfqxuiX30I62wlkAbRpWqH3pN6gtumruTYyTP4ebnxUL/m3H+VfXo8XSQzyepV0+J6az6dK7X0ZVjyPNz4BnQaXfL+Z7KtG7gr3wKvALj2Oehwh3Pm99+7FL55GNL3Q9f7rS7Hnr4VH4eyC3snhQ3GmI4iMg6rlvBvEdlsjImyR7BXQpNCFWMMnDpuJYesw+e6IGalwInD1h/dxDgoyIXazazkEHkrhLS+bLGpWTmsTTjGzDUHWL7rKE8ObM39fZrab12GlHhY8SZs+cqKLbgF3PIehHUqfRn7lsP0m6zzGfrBld14TdkGC/5sJaSGMTDwJWjU5crPoyxyTsCP/4R106wFbIa8bU0op6o0eyeFLcB1wCfAU8aYtZoUlN1kH7NmXN0yGxKWW90160XaEsRQq5fOJeTlF/DQ5xv4YdthBrdvwFt3dCx7YjDG6j21YorVS8i9FnS8y+pm+eO/rKTW+3G46omS+9CfOGLdH/D0h/G/lu0XdkGBlZR+/jdkJUPUCLjmGfB3YMe/rMMw4zZI2WpNetjvqbLVkFSlY++kcBvwL+B3Y8xEEWkKvGyMubX8oV4ZTQrVXFYKxM+zEkSibT6lsC5WgojoZS2U4h1o/cG2/fE3xvDW4t28+tMfPDmwNROutCkpP8865oop1jq9PnWt5pIuY8+16Z9Kt+bi2TQTGnSAW96/dG2mIB8+vcX6lX/fYqjXrowXwybnhNX9c8Wb1kjgPo9DtwfB3c7rWx/ZATOGWUn6to+h5XX2LV85lS6yo6q+4/th2xzY+rW1wHpRrh7gFWgNAPMOxHgHsia5gG3HXbk2phWNQkNtn9keZ/f1CgBXN6uMnBOw4TNY9bZ1cze4OXR/CDqMvPQf3PhvYMGfrO/2f9r6NX1he/+vL8Kv/4Gb3oSYu+13PY7ts5p1diywak/XvWD1WrJHk1nCbzDrDnD1hDtweRBaAAAZ/ElEQVS/hIYdy1+mqlTsXVMIA94EegIG+A141BhT4f0CNSnUUKl/QOp26/7EqXTr+XT6ee8LTqWTnXEUX7IvX5ZnAHgHwKkMyMmwxlb0fARaDizdDd0TR6wJAHd+D417wc3vWIO1wLo5O30IRA23+vA7YqTyniVWrSV1BzTtBwMmlXgP5rK2zIZ5E61Ec+fsc+eiqhV7J4WfgM+BT22bRgF3GmOuLVeUZaBJQV3O1KV7eHnhNj65oyW9GrpdkDyOn59UXNyg0z3QqOuVH8gY2Pg5LPwbYKw/zC2uham9rRrJ+CWOXQM4Pw/iPrTGNuScgJi7IHbilSUHY6wms5+ehvAe1tKqzu4CqxzG7nMfGWOiS9pWETQpqMvJOp1Ln5eWcDw7l49Gd3b8us/pB2DeA9YNcq9Aa0TyfYuhXlvHHvesk2lWU9X6TyE/B5r0se6HtBp4+eUnC/KthLb2A2h3C9w81f73KFSlYu+5j46KyCgRcbU9RgFp5QtRKfvz83Lnm4d6EV67Fv+at43M0+Vb46FEgeHWBIDX/xcw1ojuikoIAD7B1nxDj22H/v+GtL3wxZ3wRgf47XXrpvGFzmTDF3dZCaHHw3DrR5oQVKHS1hTCgbewprowwArgEWNMMROlO5bWFFRprNt/jNvfW0VooDcjujZiZJdwgnwcvL6vMc6f7TQ/z7rXseZ9q/bi5gXth1m1hwZR1pxUnw+HQ+ussQ+x450br6owDu99JCJ/MsZMvsznHwE3AEeMMZHFfN4XmA/ss22aY4wpcbFfTQqqtJbsOMK7S/ewZt8xGgZ48cE9nWnXMMDZYVWclHgrOWz+AnKzIby7NQ4hK9lawKbNDc6OUFWgikgKB4wx4Zf5vA9wAmsxnkslhb8YY67oX6YmBXWlNiemM+HTdaSfyuW9uzrRu0VdZ4dUsU4dt26Kr/nAmqxw5KyKGx2tKg1731Mo9hiX+9AYswyognMcq+omKiyQeQ/1JLx2LcZ8vJbvtyQ7O6SK5R0E3R+Eh9fDY/GaENRllScp2GPUW3cR2SQiC0XkksM+RWS8iMSJSFxqaqodDqtqmhA/L764vzsdwgJ56PP1/LjtsLNDqnguLuBWiWeUVZXCZZOCiGSJSGYxjyygYTmPvR5obIzpgDUwbt6ldjTGvG+M6WyM6Vy3bg2r+iu7CfB255MxXWnb0J/xn67j201Jzg5JqUrnsknBGONnjPEv5uFnjHErz4GNMZnGmBO2198D7iJSpzxlKlUSH083vrq/B5Gh/vz3++0cyTzt7JCUqlScMEm7RUTqi206SxHpaotFxz4oh/P2cOW5IZGkn8rl2teXsWqv/rNT6iyHJQURmQmsBFqJSKKIjBWRCSIywbbLMGCriGwCpgAjTFWbnU9VWTHhQcx/sCfBPh48MnMDGw+mOzskpSoFnSVV1WjbkzMZ90kcGady+WpCd9o00AXoVfVUEV1Slary2jTwZ/bE7vh4ujLqf6v5Ym2FD9JXqlLRpKBqvAYB3nwypisNA73529dbeGvxLvILqlYNWil70aSgFNC6vj9zH+jBzdENeeXHP+j94mIWbNYuq6rm0aSglI2bqwuv3R7Nu3fGUNfPk4c+38BrP+6kQGsNqgYp11gDpaobFxdhYPsG9G9Tj3/O28KUxbs5np3Lc0PaIc6eAVWpCqBJQalieLi58OKtUfh5ufPhb/tISDvJmyM7EljLwdNvK+Vk2nyk1CWICP8Y1IZnb2rH6r3HGD99HVmOXrRHKSfTpKDUZbi6CPf0iOCJAa1Yk3CMPi8tYVtShvZOUtWWJgWlSmFsrya8c2cMBQYGT/mNPi8tYXtyprPDUsruNCkoVQoiwqD2Dfjxz314YkArTuTkMfCN5dz14WoOZ+ikeqr60GkulCqD1KwcZq9L5K3Fu/Bwc+HWmDDG9GpCw0BvZ4emVLF0mgulHKiunycT+zZj/kM9iQwN4JOVCQx/fyXr9h9jf9pJ8vILnB2iUmWiNQWl7GDd/mPc+b/VnM61koG3uyvtQwPo1iyYoR1Diajj4+QIVU1X2pqCJgWl7OToiRziEo6TeSqX+ORMNhw4zpZDGfh5uTPzvm60bagzsCrnKW1S0MFrStlJHV9PBkTWP2/b/rSTDH9vFbdNXcGbd3Tk6tb1nBSdUqWj9xSUcqDGwT7Me7AnTer6MO6TOD5btd/ZISl1WZoUlHKw+gFefHl/d/q1CuGf87byxs+7nB2SUpekSUGpClDLw4337urE0I6hvP7zH6zZd8zZISlVLE0KSlUQN1cXnr8lkka1vXli9iaOnshxdkhKXUSTglIVqJaHGy/d2oGkjNMMmLycV3/cyY/bDnM6N9/ZoSkFaO8jpSpc92bBfPtQL/4+ZzNvL9lNgYEQP09uiQnlietb4+qi6zYo59GkoJQTtKrvx5wHenI6N59Ve9P48Ld9vLd0L0cyc/jv0PZ4ubs6O0RVQ2nzkVJO5OXuSt9WIXw6NpbHrm3J3A2HuOa1paw/cNzZoakaSpOCUpXEI/1bMH1MV3LyChj6zgru+nC1LuqjKpwmBaUqkT4t6/LL41fx1KA2rNyTxtiP49iWlEGBLuqjKogmBaUqGX8vd+7r05TXh0ez7sBxBk/5jXumrSEp/ZSzQ1M1gE6Ip1QldjjjNHM2JPLSDztxEejZvA69mtfh3p5N8HDT33Sq9HQ9BaWqgfoBXjzQtzk//Kk39/VpavVOWriDcdPjdM0G5RBaU1Cqipmxej9Pzd1Kz+bB3N65ER3CAnW9BlUinTpbqWrqztjGFBh47cedPDprIyLwf0MiGdGlEW6uWvlX5aM1BaWqqPwCw5ZDGTy/IJ64/ccJDfTm+Vsi6dcqxNmhqUpI7ykoVc25ugjRjQL54v7uvH9XJ3w8Xbl32lr+NnszmTq+QZWRw5KCiHwkIkdEZOslPhcRmSIiu0Vks4jEOCoWpaozVxfhunb1+fbhXjzQtxlfrTvIgNeX8d3mZI6fPOPs8FQV48iawsfAgMt8PhBoYXuMB951YCxKVXuebq48MaA1X0/sgZeHKw9+vp5eLy7m9Z/+ICNbaw6qdByWFIwxy4DLrSQyBJhuLKuAQBFp4Kh4lKopOoYH8f0jvXlzZEdiGgcxZfEuBr+5nNQsXb9BlcyZ9xRCgYNF3ifatimlysnL3ZUbOzTk07GxzJ7QnaMncrj9vZWkZJ52dmiqknNmUihu0vhiu0KJyHgRiRORuNTUVAeHpVT10qlxbT4bG0tyxinGf7qO3UeynB2SqsScmRQSgUZF3ocBScXtaIx53xjT2RjTuW7duhUSnFLVSeeI2vx3aHv2HDnBwDeW8+Dn63W1N1UsZyaFb4C7bb2QugEZxphkJ8ajVLV2S8cwFv/lKoZ2DOO7zck8/tUmnX1VXcRhI5pFZCbQF6gjIonAvwF3AGPMVOB7YBCwG8gG7nVULEopS4ifFy8Oi6JpXR/+u3AHGdm5PDmwNZGhAc4OTVUSOqJZqRrIGMOnq/bz2k9/kH0mnwf6NuORq1vgoutDV1s6olkpdUkiwt3dI1j8eF/6tw5h8s+7eGreFs7k6cyrNZ0mBaVqsNo+HrxzZwwT+zZj5pqDdPq/n3ju23gd7FaD6SypStVwIsIT17eiY6NAFmxOZvrKBNYdOM6koe1p08Df2eGpCqY1BaUUItb8SVNGduSNER3ZnZLFTW/9xrTf91HV7juq8tGkoJQ6z+CoBvz+5NVc1bIuz34bz83vrGDKL7u0SamG0KSglLpIYC0PPri7M8/e1A6M4fWf/2DAG8vYlpTh7NCUg2lSUEoVS0S4p0cE8x/qxfwHewIw7N2VzF6XSHq2TsldXWlSUEqVKCoskPkP9qRVfT/+8tUmop/7iXunrSHthM68Wt1oUlBKlUqIvxezJ3Tns7GxPNK/BSv2pHHXh2vYn3bS2aEpO9KkoJQqNTdXF3q1qMNj17Zk6l2d2JN6ggGTlzNp4Q6ydAnQakGTglKqTPq1CmHpX/vRpUltpi7dw90frWHJziOczMlzdmiqHDQpKKXKrH6AF9PHdOXtO2LYm3qSe6etJfY/v/Dpqv06vqGK0gnxlFJ2cepMPuv2H+e9ZXtYvusoLUJ8eeW2DnRoFOjs0BQ6IZ5SqoJ5e7jSq0UdPr63K6/c1oGTOXnc+u4KXvgunn1H9WZ0VaFJQSllV64uwrBOYSx8tA+D2jfgg+X7uPXdFczfeEjvN1QB2nyklHKo7cmZ3Dc9jsTjp6jr58ng9g0Y3qWRTrZXwbT5SClVKbRp4M+yv/bj8/tiaVnPl1lrDzD0nRWs2H3U2aGpYujU2Uoph3NxEXo0q0OPZnU4knmaUR+uZvTHaxnbqwkju4QTHlzL2SEqG20+UkpVuGMnz/Dk15v5eXsKLiLENA6iT4s6jOnVhFoe+lvVEUrbfKRJQSnlNIczTjN16R7WJhxjW1ImLUJ8mTEulhB/L2eHVu1oUlBKVSnLd6Uyfvo6fDxdGdurKSO6NCLIx8PZYVUbeqNZKVWl9G5Rl7kP9qBlPT9e/GEHPSYt5r8Lt7Nu/zEdHV2BtKaglKp0dh7O4sUfdrB4xxEAWtXz447YcG7uGEqAt7uTo6uatPlIKVXlZZ7O5fvNycxYfYAthzLwcHPhqpZ1Gdm1EVe3rufs8KoUTQpKqWplc2I6czcc4rvNyRzJyuGR/i14qF9zPNy0Fbw0NCkopaqlM3kFPDlnM3PWH6JVPT8eu64lPZvXwddTu7JejiYFpVS19nN8Cv+av5XkjNO4uQgNAr1oVc+f2Ca16dKkNpEN/XFz1VrEWaVNCppalVJV0jVt69G7ZR3W7T/O8l1HSTx+iq2HMvh5ewoA4bVr8fzNkfRpWdfJkVYtmhSUUlWWp5tr4fQZZx3JPM3KvWm88fMu7v5oDaO6hfPUoLZ4e7g6MdKqQ5OCUqpaCfH3Ykh0KNe3q8+rP+7kg+X7WLknjXt6RNC3ZYjOs1QCvaeglKrWlu9K5am5WzlwLBsXgRuiGtKrRR16t6hDfX8vRMTZIVYIvdGslFI2+QWG/WknmbX2IJ+t2k/2mXwA6vh60q1pbdo08OfO2HACa1XfaTU0KSilVDFO5+az+8gJ4hKOsSkxg1V700jOOE2wjwf924TQqXEQ3ZoG0zjYx9mh2lWl6H0kIgOANwBX4H/GmEkXfD4aeBk4ZNv0ljHmf46MSSlVs3m5uxIZGkBkaEDhto0H0/lg+V4Wbj3Ml3GJeLq50LtFHQZENuDm6IY1qmurw2oKIuIK/AFcCyQCa4GRxpj4IvuMBjobYx4qbblaU1BKOUpefgE7U7L4fPUBftt9lP1p2TQOrsVNHRrSr3UIYUHehPhVzWm9K0NNoSuw2xiz1xbQLGAIEH/ZbymllJO4ubrQrmEAL9zSHmMMP28/wpuLd/Hm4t28uXg3AFe3DqF3izq0rOdH2wb+iFCt7kU4MimEAgeLvE8EYovZ71YR6YNVq/izMebghTuIyHhgPEB4eLgDQlVKqfOJCNe2rce1betxJOs0v+8+ypbETL6MO1g4e+tZXSKCiAkPokuENZq6Ks/k6sjmo9uA640x42zv7wK6GmMeLrJPMHDCGJMjIhOA240xV1+uXG0+Uko526F0a/T0wWPZHM8+w++704hPyuRMfgEi0K6hPyO7htMw0JugWh74ebnRrK6vU2OuDM1HiUCjIu/DgKSiOxhj0oq8/QB40YHxKKWUXYQGehMa6F34/q/XW72aNh5MZ9XeNH6KT+GpuVvP+06Atzt3xIZzb8+ISn1fwpE1BTesJqH+WL2L1gJ3GGO2FdmngTEm2fb6FuBvxphulytXawpKqcouv8CwNuEYri5CUvopUrNyWLPvGD/Gp+DuKrSs50eBgTq+HlzXth5RYYG0Dw3AxcVxA+mcXlMwxuSJyEPAIqwuqR8ZY7aJyHNAnDHmG+AREbkJyAOOAaMdFY9SSlUUVxehW9Pg87aN692UfUdP8smKBLYlZRDg7cHeoyf413zrd7LVs8mTnLwCcvIKaBjoTXSjQHq3qENMeBCuDkwYRengNaWUcpKCAsOyXakczjjNT/Ep5OQV4OnmgrurC1uTMkjOOE1+gSHA253oRoF8MqZrmY/l9JqCUkqpy3NxEfq2CgFgRNeLe1Zmnc7l152prNiThmcFrTCnSUEppSopPy93buzQkBs7NKywY9acsdtKKaVKpElBKaVUIU0KSimlCmlSUEopVUiTglJKqUKaFJRSShXSpKCUUqqQJgWllFKFqtw0FyKSCuwv49frAEftGE5Vp9fjfHo9ztFrcb7qcD0aG2PqlrRTlUsK5SEicaWZ+6Om0OtxPr0e5+i1OF9Nuh7afKSUUqqQJgWllFKFalpSeN/ZAVQyej3Op9fjHL0W56sx16NG3VNQSil1eTWtpqCUUuoyakxSEJEBIrJTRHaLyJPOjqciiMhHInJERLYW2VZbRH4SkV225yDbdhGRKbbrs1lEYpwXuf2JSCMRWSIi20Vkm4g8atteU6+Hl4isEZFNtuvxrG17ExFZbbseX4iIh227p+39btvnEc6M3xFExFVENojIAtv7GnktakRSEBFX4G1gINAWGCkibZ0bVYX4GBhwwbYngV+MMS2AX2zvwbo2LWyP8cC7FRRjRckDHjfGtAG6AQ/a/g3U1OuRA1xtjOkARAMDRKQb8CLwuu16HAfG2vYfCxw3xjQHXrftV908Cmwv8r5mXgtjTLV/AN2BRUXe/x34u7PjqqBzjwC2Fnm/E2hge90A2Gl7/R4wsrj9quMDmA9cq9fDANQC1gOxWAO03GzbC/+/ARYB3W2v3Wz7ibNjt+M1CMP6UXA1sACQmnotakRNAQgFDhZ5n2jbVhPVM8YkA9ieQ2zba8w1slX3OwKrqcHXw9ZcshE4AvwE7AHSjTF5tl2KnnPh9bB9ngEEV2zEDjUZeAIosL0PpoZei5qSFKSYbdrt6nw14hqJiC/wNfAnY0zm5XYtZlu1uh7GmHxjTDTWr+SuQJvidrM9V9vrISI3AEeMMeuKbi5m12p/LaDmJIVEoFGR92FAkpNicbYUEWkAYHs+Ytte7a+RiLhjJYQZxpg5ts019nqcZYxJB37FutcSKCJuto+KnnPh9bB9HgAcq9hIHaYncJOIJACzsJqQJlMzr0WNSQprgRa23gQewAjgGyfH5CzfAPfYXt+D1bZ+dvvdtl433YCMs80q1YGICPAhsN0Y81qRj2rq9agrIoG2197ANVg3WZcAw2y7XXg9zl6nYcBiY2tUr+qMMX83xoQZYyKw/jYsNsbcSQ28FkDNuNFs++81CPgDq930KWfHU0HnPBNIBnKxft2MxWr7/AXYZXuubdtXsHpo7QG2AJ2dHb+dr0UvrCr+ZmCj7TGoBl+PKGCD7XpsBZ62bW8KrAF2A18BnrbtXrb3u22fN3X2OTjouvQFFtTka6EjmpVSShWqKc1HSimlSkGTglJKqUKaFJRSShXSpKCUUqqQJgWllFKFNCmoSkdE8kVko20Gz/Ui0qOE/QNF5IFSlPuriNSIdXZLS0Q+FpFhJe+pagpNCqoyOmWMiTbWDJ5/B/5bwv6BQIlJwVmKjIpVqtLTpKAqO3+saYsREV8R+cVWe9giIkNs+0wCmtlqFy/b9n3Cts8mEZlUpLzbbOsI/CEivW37uorIyyKy1rZ2wv227Q1EZJmt3K1n9y9KRBJE5EVbmWtEpLlt+8ci8pqILAFetK3bMM9W/ioRiSpyTtNssW4WkVtt268TkZW2c/3KNmcTIjJJROJt+75i23abLb5NIrKshHMSEXnLVsZ3nJsAUCmLs0fP6UMfFz6AfKwRxzuwZqDsZNvuBvjbXtfBGlEqXDw9+EBgBVDL9v7sKOVfgVdtrwcBP9tejwf+aXvtCcQBTYDHsY1+B1wBv2JiTSiyz92cGw37MdYUzK62928C/7a9vhrYaHv9IjC5SHlBtnNbBvjYtv0NeBqojTWF99lBp4G25y1A6AXbLnVOQ7FmRHUFGgLpwDBn/zfXR+V5aLVWVUanjDV7JyLSHZguIpFYCeA/ItIHa4rjUKBeMd+/BphmjMkGMMYUnazs7ER467CSCcB1QFSRtvUArMV11gIf2SbSm2eM2XiJeGcWeX69yPavjDH5tte9gFtt8SwWkWARCbDFOuLsF4wxx22zdrYFfrembMIDWAlkAqeB/9l+5S+wfe134GMR+bLI+V3qnPoAM21xJYnI4kuck6qhNCmoSs0Ys1JE6gB1sX7d18WqOeTaZrX0KuZrwqWnMs6xPedz7t+/AA8bYxZdVJCVgAYDn4rIy8aY6cWFeYnXJy+IqbjvFRerAD8ZY0YWE09XoD9WInkIa/W0CSISa4tzo4hEX+qcRGRQMcdTqpDeU1CVmoi0xmrqSMP6tXvElhD6AY1tu2UBfkW+9iMwRkRq2cqoXcJhFgETbTUCRKSliPiISGPb8T7AmmH1Uus0Dy/yvPIS+ywD7rSV3xc4aqz1HH7E+uN+9nyDgFVAzyL3J2rZYvIFAowx3wN/wlpGExFpZoxZbYx5GmsVsEaXOidbHCNs9xwaAP1KuDaqhtGagqqMvMVaEQysX7z3GGPyRWQG8K2IxHHungPGmDQR+V1EtgILjTF/tf1ajhORM8D3wD8uc7z/YTUlrRervSYVuBlrxsy/ikgucALrnkFxPEVkNdaPrIt+3ds8A0wTkc1ANuemXn4eeNsWez7wrDFmjoiMBmaKiKdtv39iJb/5IuJluy5/tn32soi0sG37BdiENftpcec0F+uexhasWYOXXua6qBpIZ0lVqhxsTVidjTFHnR2LUvagzUdKKaUKaU1BKaVUIa0pKKWUKqRJQSmlVCFNCkoppQppUlBKKVVIk4JSSqlCmhSUUkoV+n+0bs7L/+/iCAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_losses()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing the dimension of hidden layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using our default of 128 gets 54%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:06 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.769207</th>\n",
       "    <th>2.736875</th>\n",
       "    <th>0.191667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.700270</th>\n",
       "    <th>2.432153</th>\n",
       "    <th>0.291667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.527224</th>\n",
       "    <th>2.165822</th>\n",
       "    <th>0.352083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.411032</th>\n",
       "    <th>2.005153</th>\n",
       "    <th>0.375000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>2.233243</th>\n",
       "    <th>1.797880</th>\n",
       "    <th>0.427083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>2.073015</th>\n",
       "    <th>1.878045</th>\n",
       "    <th>0.414583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>1.945470</th>\n",
       "    <th>1.800441</th>\n",
       "    <th>0.425000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.808628</th>\n",
       "    <th>1.668809</th>\n",
       "    <th>0.466667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.670717</th>\n",
       "    <th>1.669584</th>\n",
       "    <th>0.456250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.536347</th>\n",
       "    <th>1.619124</th>\n",
       "    <th>0.514583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.387313</th>\n",
       "    <th>1.561667</th>\n",
       "    <th>0.512500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.237852</th>\n",
       "    <th>1.506838</th>\n",
       "    <th>0.543750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>1.109924</th>\n",
       "    <th>1.549464</th>\n",
       "    <th>0.541667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>1.008990</th>\n",
       "    <th>1.554212</th>\n",
       "    <th>0.543750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>0.929697</th>\n",
       "    <th>1.554622</th>\n",
       "    <th>0.543750</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, MyLetterRNN(n_layers=2, dropout=0.5, n_hidden=128), loss_func=F.cross_entropy, metrics=[accuracy])\n",
    "learn.fit_one_cycle(15, max_lr=3e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doubling to 256 doesn't change performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:06 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.778366</th>\n",
       "    <th>2.746171</th>\n",
       "    <th>0.233333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.692814</th>\n",
       "    <th>2.430204</th>\n",
       "    <th>0.266667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.557370</th>\n",
       "    <th>2.121804</th>\n",
       "    <th>0.335417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.400152</th>\n",
       "    <th>1.958430</th>\n",
       "    <th>0.383333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>2.266069</th>\n",
       "    <th>1.900910</th>\n",
       "    <th>0.377083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>2.139609</th>\n",
       "    <th>1.769911</th>\n",
       "    <th>0.437500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>1.997191</th>\n",
       "    <th>1.776130</th>\n",
       "    <th>0.443750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.849745</th>\n",
       "    <th>1.595855</th>\n",
       "    <th>0.481250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.695713</th>\n",
       "    <th>1.665297</th>\n",
       "    <th>0.462500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.545231</th>\n",
       "    <th>1.611652</th>\n",
       "    <th>0.491667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.396923</th>\n",
       "    <th>1.523717</th>\n",
       "    <th>0.518750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.269320</th>\n",
       "    <th>1.593659</th>\n",
       "    <th>0.531250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>1.159536</th>\n",
       "    <th>1.635850</th>\n",
       "    <th>0.518750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>1.062906</th>\n",
       "    <th>1.653824</th>\n",
       "    <th>0.529167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>0.984885</th>\n",
       "    <th>1.646750</th>\n",
       "    <th>0.531250</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, MyLetterRNN(n_layers=2, dropout=0.5, n_hidden=128), loss_func=F.cross_entropy, metrics=[accuracy])\n",
    "learn.fit_one_cycle(15, max_lr=3e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Halving to 64 definitely does; 128 does seem to be a sweet spot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:06 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.788145</th>\n",
       "    <th>2.761199</th>\n",
       "    <th>0.089583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.748204</th>\n",
       "    <th>2.577398</th>\n",
       "    <th>0.239583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.619068</th>\n",
       "    <th>2.174743</th>\n",
       "    <th>0.316667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.437004</th>\n",
       "    <th>2.133431</th>\n",
       "    <th>0.362500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>2.297863</th>\n",
       "    <th>1.931483</th>\n",
       "    <th>0.372917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>2.146514</th>\n",
       "    <th>1.836011</th>\n",
       "    <th>0.410417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>2.041750</th>\n",
       "    <th>1.737024</th>\n",
       "    <th>0.437500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.896687</th>\n",
       "    <th>1.610484</th>\n",
       "    <th>0.477083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.764543</th>\n",
       "    <th>1.659021</th>\n",
       "    <th>0.462500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.635953</th>\n",
       "    <th>1.572796</th>\n",
       "    <th>0.493750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.525941</th>\n",
       "    <th>1.614364</th>\n",
       "    <th>0.489583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.413064</th>\n",
       "    <th>1.567258</th>\n",
       "    <th>0.497917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>1.319074</th>\n",
       "    <th>1.581343</th>\n",
       "    <th>0.483333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>1.237835</th>\n",
       "    <th>1.610361</th>\n",
       "    <th>0.502083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>1.175101</th>\n",
       "    <th>1.607811</th>\n",
       "    <th>0.502083</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, MyLetterRNN(n_layers=2, dropout=0.5, n_hidden=64), loss_func=F.cross_entropy, metrics=[accuracy])\n",
    "learn.fit_one_cycle(15, max_lr=3e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally 3 layers also gets a worse result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:14 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.777945</th>\n",
       "    <th>2.767582</th>\n",
       "    <th>0.108333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.763745</th>\n",
       "    <th>2.700912</th>\n",
       "    <th>0.177083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.724584</th>\n",
       "    <th>2.610098</th>\n",
       "    <th>0.170833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.610290</th>\n",
       "    <th>2.289089</th>\n",
       "    <th>0.293750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>2.476536</th>\n",
       "    <th>2.088696</th>\n",
       "    <th>0.347917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>2.339663</th>\n",
       "    <th>1.914408</th>\n",
       "    <th>0.404167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>2.205508</th>\n",
       "    <th>1.885391</th>\n",
       "    <th>0.425000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>2.120229</th>\n",
       "    <th>1.957143</th>\n",
       "    <th>0.341667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>2.053982</th>\n",
       "    <th>1.699707</th>\n",
       "    <th>0.397917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.987532</th>\n",
       "    <th>1.688885</th>\n",
       "    <th>0.441667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.908170</th>\n",
       "    <th>1.695390</th>\n",
       "    <th>0.462500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.848755</th>\n",
       "    <th>1.654914</th>\n",
       "    <th>0.456250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>1.779561</th>\n",
       "    <th>1.624753</th>\n",
       "    <th>0.460417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>1.697022</th>\n",
       "    <th>1.597392</th>\n",
       "    <th>0.475000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>1.646311</th>\n",
       "    <th>1.599763</th>\n",
       "    <th>0.475000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>1.582875</th>\n",
       "    <th>1.559585</th>\n",
       "    <th>0.477083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>1.531765</th>\n",
       "    <th>1.559109</th>\n",
       "    <th>0.475000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>1.491278</th>\n",
       "    <th>1.601305</th>\n",
       "    <th>0.462500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>1.446864</th>\n",
       "    <th>1.503486</th>\n",
       "    <th>0.483333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>1.387920</th>\n",
       "    <th>1.531969</th>\n",
       "    <th>0.508333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>21</th>\n",
       "    <th>1.325619</th>\n",
       "    <th>1.495371</th>\n",
       "    <th>0.520833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>22</th>\n",
       "    <th>1.257018</th>\n",
       "    <th>1.581387</th>\n",
       "    <th>0.531250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>23</th>\n",
       "    <th>1.193253</th>\n",
       "    <th>1.517283</th>\n",
       "    <th>0.537500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>24</th>\n",
       "    <th>1.138225</th>\n",
       "    <th>1.559087</th>\n",
       "    <th>0.529167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>25</th>\n",
       "    <th>1.086946</th>\n",
       "    <th>1.572238</th>\n",
       "    <th>0.539583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>26</th>\n",
       "    <th>1.040665</th>\n",
       "    <th>1.561826</th>\n",
       "    <th>0.525000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>27</th>\n",
       "    <th>1.001137</th>\n",
       "    <th>1.584307</th>\n",
       "    <th>0.520833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>28</th>\n",
       "    <th>0.970200</th>\n",
       "    <th>1.590697</th>\n",
       "    <th>0.516667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>29</th>\n",
       "    <th>0.947225</th>\n",
       "    <th>1.590535</th>\n",
       "    <th>0.516667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>30</th>\n",
       "    <th>0.927288</th>\n",
       "    <th>1.590184</th>\n",
       "    <th>0.516667</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, MyLetterRNN(n_layers=3, dropout=0.5), loss_func=F.cross_entropy, metrics=[accuracy])\n",
    "learn.fit_one_cycle(30, max_lr=3e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:14 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.777945</th>\n",
       "    <th>2.767582</th>\n",
       "    <th>0.108333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.763745</th>\n",
       "    <th>2.700912</th>\n",
       "    <th>0.177083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.724584</th>\n",
       "    <th>2.610098</th>\n",
       "    <th>0.170833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.610290</th>\n",
       "    <th>2.289089</th>\n",
       "    <th>0.293750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>2.476536</th>\n",
       "    <th>2.088696</th>\n",
       "    <th>0.347917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>2.339663</th>\n",
       "    <th>1.914408</th>\n",
       "    <th>0.404167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>2.205508</th>\n",
       "    <th>1.885391</th>\n",
       "    <th>0.425000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>2.120229</th>\n",
       "    <th>1.957143</th>\n",
       "    <th>0.341667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>2.053982</th>\n",
       "    <th>1.699707</th>\n",
       "    <th>0.397917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.987532</th>\n",
       "    <th>1.688885</th>\n",
       "    <th>0.441667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.908170</th>\n",
       "    <th>1.695390</th>\n",
       "    <th>0.462500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.848755</th>\n",
       "    <th>1.654914</th>\n",
       "    <th>0.456250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>1.779561</th>\n",
       "    <th>1.624753</th>\n",
       "    <th>0.460417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>1.697022</th>\n",
       "    <th>1.597392</th>\n",
       "    <th>0.475000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>1.646311</th>\n",
       "    <th>1.599763</th>\n",
       "    <th>0.475000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>1.582875</th>\n",
       "    <th>1.559585</th>\n",
       "    <th>0.477083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>1.531765</th>\n",
       "    <th>1.559109</th>\n",
       "    <th>0.475000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>1.491278</th>\n",
       "    <th>1.601305</th>\n",
       "    <th>0.462500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>1.446864</th>\n",
       "    <th>1.503486</th>\n",
       "    <th>0.483333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>1.387920</th>\n",
       "    <th>1.531969</th>\n",
       "    <th>0.508333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>21</th>\n",
       "    <th>1.325619</th>\n",
       "    <th>1.495371</th>\n",
       "    <th>0.520833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>22</th>\n",
       "    <th>1.257018</th>\n",
       "    <th>1.581387</th>\n",
       "    <th>0.531250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>23</th>\n",
       "    <th>1.193253</th>\n",
       "    <th>1.517283</th>\n",
       "    <th>0.537500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>24</th>\n",
       "    <th>1.138225</th>\n",
       "    <th>1.559087</th>\n",
       "    <th>0.529167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>25</th>\n",
       "    <th>1.086946</th>\n",
       "    <th>1.572238</th>\n",
       "    <th>0.539583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>26</th>\n",
       "    <th>1.040665</th>\n",
       "    <th>1.561826</th>\n",
       "    <th>0.525000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>27</th>\n",
       "    <th>1.001137</th>\n",
       "    <th>1.584307</th>\n",
       "    <th>0.520833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>28</th>\n",
       "    <th>0.970200</th>\n",
       "    <th>1.590697</th>\n",
       "    <th>0.516667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>29</th>\n",
       "    <th>0.947225</th>\n",
       "    <th>1.590535</th>\n",
       "    <th>0.516667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>30</th>\n",
       "    <th>0.927288</th>\n",
       "    <th>1.590184</th>\n",
       "    <th>0.516667</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, MyLetterRNN(n_layers=3, dropout=0.5), loss_func=F.cross_entropy, metrics=[accuracy])\n",
    "learn.fit_one_cycle(30, max_lr=3e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN From Scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build our own RNN; instead of one hot encoding we'll use a `nn.Embedding`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = (TextList\n",
    "     .from_df(df, \n",
    "              cols=[2], \n",
    "              processor=processors)\n",
    "     .split_by_idxs(train_idx=bal_idx, valid_idx=valid_idx)\n",
    "     .label_from_df(cols=0)\n",
    "     .databunch(bs=1024))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_data_set = set(tuple(_[0].data) for _ in data.valid_ds)\n",
    "for datum in data.train_ds:\n",
    "    assert tuple(datum[0].data) not in valid_data_set, datum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([19, 512]), torch.Size([512]))"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x, y = next(iter(data.train_dl))\n",
    "x.shape, y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "512"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self, n_input, n_hidden, n_output, bn=False):\n",
    "        super().__init__()\n",
    "        self.i_h = nn.Embedding(n_input,n_hidden)\n",
    "        self.bn = nn.BatchNorm1d(n_hidden) if bn else None\n",
    "        self.o_h = nn.Linear(n_hidden, n_output)\n",
    "        self.h_h = nn.Linear(n_hidden, n_hidden)\n",
    "        self.reset()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # I'm not quite sure why the batch size seems to change to 720 in validation...\n",
    "        if self.h.shape[0] != x.shape[1]:\n",
    "            self.reset(x.shape[1])\n",
    "        h = self.h\n",
    "        x = self.i_h(x)\n",
    "        for xi in x:\n",
    "            h += xi\n",
    "            h = self.h_h(h)\n",
    "            h = F.relu(h)\n",
    "            if self.bn:\n",
    "                h = self.bn(h)\n",
    "        self.h = h.detach()\n",
    "        o = self.o_h(h)\n",
    "        return o\n",
    "        \n",
    "    def reset(self, size=None):\n",
    "        size = size or 1\n",
    "        self.h = torch.zeros(size, n_hidden).cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(n_letters, n_hidden, n_output).cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, model, loss_func=F.cross_entropy, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8XHW9//HXJ5OlzdIkTdM2bdqm+0L3pi37WrEUQRaVTURREQUuKHqvwu8q6tWLogKCiEVkEQRkuywKZWuBAl3SfV/plqZpuqVp2mYyM9/fHzMdQkjaNMmseT8fj3kwc+Z7zvl8mXQ+813O95hzDhEREYCUWAcgIiLxQ0lBRETClBRERCRMSUFERMKUFEREJExJQUREwpQUREQkTElBRETClBRERCQsNdYBHK9u3bq5kpKSWIchIpJQFixYsMs5V3iscgmXFEpKSigrK4t1GCIiCcXMNreknLqPREQkTElBRETClBRERCRMSUFERMKUFEREJExJQUREwpQUREQkTElBRCQB3PPWWt5fVxXx8ygpiIgkgPvfWc9HG3ZH/DxKCiIicS4QcPgCjjRP5L+ylRREROKc1x8AID1VSUFEpMM7khQylBRERMTrS4KWgpl1MrN5ZrbEzFaY2c+bKJNhZs+Y2Xozm2tmJZGKR0QkUdWHWgqJPqZQB5ztnBsDjAWmmtmJjcp8E9jrnBsE3A38JoLxiIgkpHBLIZGTggs6EHqZFnq4RsW+CDwWev4ccI6ZWaRiEhFJREnRfQRgZh4zWwzsBN50zs1tVKQ3sBXAOecDqoGCSMYkIpJokmb2kXPO75wbCxQDk8xsZKMiTbUKGrcmMLPrzKzMzMqqqiJ/RZ+ISDxJiu6jhpxz+4BZwNRGb20D+gCYWSqQC+xpYv/pzrlS51xpYeExbzEqIpJUkqL7yMwKzSwv9LwzMAVY3ajYy8A1oedfAt5xzn2mpSAi0pFFs/soNYLHLgIeMzMPweTzT+fcq2b2C6DMOfcy8DDwdzNbT7CFcHkE4xERSUjRnJIasaTgnFsKjGti+08bPD8MfDlSMYiIJIOkG1MQEZHWq0uGMQUREWkf9f7gUKvWPhIRkXD3UaIvcyEiIu3A6/MD6j4SERGS6IpmERFpuyNjCmmeyC8Np6QgIhLn6jQlVUREjvD6AqR7UojGItJKCiIica7eH4jKeAIoKYiIxD2vLxCV8QRQUhARiXten1oKIiIS4lX3kYiIHOH1B6JyNTMoKYiIxL0js4+iQUlBRCTOeX2BqCyGB0oKIiJxT1NSRUQkLDglVUlBRETQ7CMREWlAA80iIhLm9QdIU0tBREQgNPtILQUREQEtcyEiIg1oSqqIiIRpSqqIiIRpSqqIiAAQCDjq/U5TUkVEBOoDofszq6UgIiJeXygpqKUgIiLhpKCWgoiI1PsdoKQgIiJ80lLQlFQREcHr9wNqKYiICFCXLAPNZtbHzGaa2SozW2FmNzdRJt/MXjSzpWY2z8xGRioeEZFE9MmYgkXlfJFMPT7gVufccOBE4AYzG9GozG3AYufcaOBrwL0RjEdEJOF8MiXVE5XzRSwpOOcqnHMLQ89rgFVA70bFRgBvh8qsBkrMrEekYhIRSTRJOSXVzEqAccDcRm8tAS4JlZkE9AOKoxGTiEgiqPcnWVIws2zgeeAW59z+Rm/fCeSb2WLgJmARwW6nxse4zszKzKysqqoq0iGLiMSNuvCU1OiMKaRG8uBmlkYwITzpnHuh8fuhJPGNUFkDPg49GpebDkwHKC0tdZGMWUQknnhDLYWMRG8phL7kHwZWOef+0EyZPDNLD738FvBeE60JEZEOK9oDzZFsKZwCXA0sC3UPQXC2UV8A59yDwHDgcTPzAyuBb0YwHhGRhHNkTCEtSlNSI5YUnHOzgaPWwjn3ETA4UjGIiCQ6rZIqIiJhSTklVUREWsebbFNSRUSk9cKrpKYoKYiIdHhef4A0j5GSkvhrH4mISBt5fYGoDTKDkoKISFyr9wdIi9J4AigpiIjENbUUREQkzOsLRG3mESgpiIjENa9fSUFERELUfSQiImFqKYiISJhaCiIiElbvD5CmpCAiIqDZRyIi0kCdkoKIiBxRr4FmERE5wuvXQLOIiIRo9pGIiIRpoFlERMLq/U5TUkVEJEgtBRERAcA5p2UuREQkqN7vAMhQUhAREa8/AECaJzr3ZwYlBRGRuOX1BZOCpqSKiMgnSSHVE7VzKimIiMSpenUfiYjIEXXhloK6j0REOrwj3UeafSQiIuHuo2i2FFKjdiY5qq17DrJiezV9umbSryCL7IyjfzSV+w/zzPytzP14N4XZGfTK60xRXmd65XbC6wuwq9bL7gN17D7gxe8cA7plMaAwiwHdsinO70xqFGcziEjrfDIlVUmhw3DO8cz8rfz8lZUcqveHt3fLzqCkIJPBPXIY2iOboT27MLhHNiu27+fJOZt5e/VO/AHHiKIubN59kB1LK/AF3GeOn5+ZhgP2HawPb/OkGKkpRsPSndM8FOZk0C07nW7ZGRTldmJCv66cOKAreZnpEav7msoa0jwp9C/IIiWl5YNph+v9LN1WzfxNe1iweS9rdtRQkJ1On66Z9A09SgqyGNwjm4KsdMyiN1An0l5iMSVVSeE47ag+zILNe9ldW8euA1721Nax72A9KWakeoJftqmeFPrkZzK+bx6ji/PonN70dLJ9B7385IVlvLZ8BycPLOAHnxvCzpo6Nu2uZfOug3y8q5bXllfw1Lz6T+1XkJXOt07rzxUT+1LSLQsAf8BRVVNHRfUh0lNTKMzOID8rPfwLY2+tl427DrBhZy2b99SGE4gR/LI86PVRVVPHrgN1LC+v5s2VlTz0/seYwfCeXThxQAFj+uQyuHsOAwqz6JTWuilyh7x+Ptywi7dX7+SdVTvZsf8wADkZqYwqzmV0cR5DemST5knBDFJCX+aV+w+zZc9Btu45xNY9B9m460D4as+BhVlM6JfP3oNeVpRXM2P5jk8lyLzMNAYVZjOsKIdTBxVyyqACcjqltSp+kWjyxmCgOWJJwcz6AI8DPYEAMN05d2+jMrnAE0DfUCy/c849EqmY2qLO5+eh9zZy/8z1HK4PhLfnZ6aRl5mOc456v8MfcNT7A+yu9QKQmmKM6NWFkb1z6Z6TQbfs4K/xer/jV/9axe7aOn5y3jC+fdqAJn8pOxf8sl9TWcPaygP07NKJKSO6k9Fo3rInxeiZ24meuZ2ajD8/K50JWV2Z0K9ri+rr9QVYsm0fH23YzUcbdvPE3M387YNgvVMMSgqyKMrrxOH6AAe9fg55feGWjscMM8OTYjgc9T6HLxDA6wuW9QUcWekeThtcyNnDuoPBkq37WLqtmodnbwx/2TeWme6hT34mfbpmcuawQkr7dWVCv3y6Zn26JeMPOLbvO8THu2pZt/MA63ceYP3OGv5v0XaemLOF1BRjQr98zhhayOT+BZzQq0urk5xIJMWi+8ica/of4KcKmQ0Etjnn6szsTGA08Lhzbt9R9ikCipxzC80sB1gAXOScW9mgzG1ArnPuv8ysEFgD9HTOeZs7bmlpqSsrK2th9drHu2uruOPlFXy8q5bzRvbke2cOomduJ/Iz05rtm99T62XRlr0s3LI33L2x9+Cnf/EP6JbFvZePY1RxbjSq0SZ1Pn/wS7byAOtCCWpnzWE6p3vonJZKZrqHzqEv1oBz+J3jyJ9WmifYekr3pNA53cNJAwqYPKDrZxIbBLuFyvcdwjlHwIFzweMV5mS0uRuo3h9g4ea9zFpbxbtrqlhZsR8INs1H9OrCuL55jO2Tx5jiPPoVZKrLSWLulSXbuempRbz5/dMZ3COnTccyswXOudJjlWtpS+F5oNTMBgEPAy8D/wCmNbeDc64CqAg9rzGzVUBvYGXDYkCOBf/1ZQN7AF8LY4q45eXV3PfOOmasqKR/tyweu3YSZwwpbNG+XbPSOWd4D84Z3iO8rd4fYO9BL7sPeKk+VM+Yo3QtxZuMVA/DenZhWM8uET1PpzQPAwuzI3LsNE8KkwcUMHlAAf81dRhVNXUsDCXuRVv28dS8LTzywSYg2OU0qncuY/vkcdaw7owtzjuuMQ+R9hDP3UcB55zPzC4G7nHO3Wdmi1p6EjMrAcYBcxu9dT/BBLMdyAEuc84FGpXBzK4DrgPo27dvS0/bLH/AMXfjbv61rILOaR4m9e/KxJKu5GcFu4HeXVvF9Pc28uGG3WSle/jhuUP49ukDmvxlezzSPCl0z+lE95ymu3gkugpzMvj8CT35/Ak9gWDSXrOjhqXbqlm6bR9LtlXzwKwN3PfOenp26cTUkT05b2RPSku64lGCkCiI5ymp9WZ2BXANcEFoW4tG6swsm2BL4xbn3P5Gb38eWAycDQwE3jSz9xuXc85NB6ZDsPuohTF/xuod+3lxUTkvLdrOjv2HyUz34As4/jr7YwAGdw/+Ql238wA9umTwk/OGcfmkvuR21qBkR5DmSWFk71xG9s7lysnBHx/Vh+p5Z3Ul/162g6fmbeHRDzfRKS2FIT1yGNIjh6E9chjaM4eRvXM/M7YBwckEZZv2Un2ontOHFFKYkxHtakkCi+cpqd8Argd+5Zz72Mz6ExwgPiozSyOYEJ50zr3QzHHvdMGBjfVm9jEwDJjXwrha7NmyrfzouaWkphhnDCnk9vOHM2V4D8xgWXk18z7ew/xNe6g57OP3Xx7DBWN6RTU7S3zK7ZzGxeOKuXhcMbV1Pmau2cmiLftYs6OGd9dW8dyCbeGyvfM6M7o4lxN6daFyfx3zN+1h9Y6a8PtmUNovP9w66dM1MxZVkgQSi+6jFg00f2oHs3ygj3Nu6THKGfAYsMc5d0szZf4MVDrn7jCzHsBCYIxzbldzx23tQHNVTR2vLa/g/FFFFGTr15q0jz21XlZX7GdZeXX4sXn3QbLSPYzvl8+kkq5M7N+V7IxU3lxZyYwVO8KJ4uSBBVxzcglThvdQd5Q06U8z13PXjDWs/uXUNs+Qa+lAc0tnH80CLiTYslgMVAHvOud+cJR9TgXeB5YRnJIKcBvB6ac45x40s17Ao0ARYARbDUdtgcRi9pHI8ag5XE/nNE+zM9M27arl1aXb+cfcLWyvPkzvvM589cR+XD6xD/lNdEFJx3XPW2u55611bPz1tDZPdGjv2Ue5zrn9ZvYt4BHn3M/M7KgtBefcbOCotXDObQfObWEMIgnhWBfGlXTL4sazB3P9GQN5a1Ulj324md+8vpq731rLF0YVcdWJ/RjfN09TYgWvL0BqikV15ltLk0Jq6LqDrwC3RzAekQ4j1ZPC1JFFTB1ZxJodNTwxZzMvLirnhUXlDC/qwldP7MsFY3rRRVdfd1heXyDqY5stPdsvgBnABufcfDMbAKyLXFgiHcvQnjn88qKRzLntHH518UgAbn9xORP/5y1uemoRs9bsxOf/zGxtSXL1/ugnhRa1FJxzzwLPNni9Ebg0UkGJdFTZGalcNbkfV07qy5Jt1Ty/YBsvL9nOK0u20z0ng6+d1I9rT+1PZrqWLesIvP5AVKejQgtbCmZWbGYvmtlOM6s0s+fNrDjSwYl0VGbG2D55/PKikcy7/Rwe/Op4RvTqwu/eWMuZd83i6Xlb1HLoAOp8gaiukAot7z56hOCVx70ILlXxSmibiERYRqqHqSOLePQbk3ju+pMozu/Mj19YxtR73+ftVZWxDk8iyOsLRPWua9DypFDonHvEOecLPR4FWrYIkIi0m9KSrjz/3ZN58KvjCQQc33ysjG8/XkZF9aFYhyYREIsxhZaebZeZfdXMPKHHV4HdkQxMRJpmZkwdWcSM75/Oj88bxvvrqpjy+3d55IOP8TdxoyVJXF5fnI4pANcSnI66g+DKp18iuESFiMRImieF688YyBu3nMGEkq78/JWVXPLAB7y0uJw9tc2uPi8JxBvHs4+2ELyiOczMbgHuiURQItJyfQsyeewbE3l5yXZ+9a9V3Pz0YsxgdHEeZwwp5ILRRW1ei19io97n4naguSnNLnEhItFlZnxxbG8++sk5vPi9k7n5nMGkGNz/zjrO/+Nsni3bGusQpRXq/AHS4rGl0Axdgy8SZzwpxri++Yzrm88tU4ZQVVPH959ZzI+eW8rKiv3cPm14s2sySfzxxvGU1KZoREskzhXmZPDoNyZy7Sn9eeSDTXz9kfnsO6jxhkTh9fnja0qqmdWY2f4mHjUEr1kQkTiX6knhpxeM4LdfGs28j/dw4f0f8NqyCs1USgD1fhdfA83OOY1OiSSJr5T2YWBhNrf+czHffXIhJQWZfPv0AVw6vrjNa/VLZASnpEa3p16diyIdyIR++bx965k8cNV4cjuncfuLyznlznf4v0XlsQ5NmhC3U1JFJHl4Uoxpo4o4b2RP5mzcw+/eWMMP/rmY3M5pnDWse6zDkwbqfQHSPdFtxamlINJBmRknDSzg8WsnMbyoCzf+YyErt++PdVjSQHBKqrqPRCSKsjJS+dvXJ9KlcxrXPjqfHdWHYx2SAM654IJ4CTQlVUSSRI8unXj4monUHK7n2kfnU1vni3VIHV69Pzg7LF4XxBORJDeiVxfuv2o8aypruOmpRZqyGmP1oftlKCmISMycNbQ7d1x4Au+s3slvZ6yOdTgdmtcXTArRXiVVs49E5FOuPrEfqyv285d3NzKiqAtfHNs71iF1SF61FEQkXvzsghOYWJLPfz2/lOXl1bEOp0M60lJIpLWPRCRJpaem8MBVE8jPTOc7f1/ArgN1sQ6pw1FLQUTiSmFOBtOvLmXXgTq+9+TC8MCnRIdaCiISd0YV5/KbS4ML6d01Y02sw+lQwklBLQURiScXjevNlZP78tD7G/log27NHi2akioicev/nT+cfl0z+eGzS9h/uD7W4XQIsZqSqqQgIseUmZ7K3ZeNZcf+w9zx8opYh9Mh1KmlICLxbFzffG44axAvLCzn38sqYh1O0qvXQLOIxLubzh7E6OJcbntxGTv3a+G8SNKUVBGJe2meFO6+bCyH6/385/NLcU7rI0VK0k1JNbM+ZjbTzFaZ2Qozu7mJMj8ys8Whx3Iz85tZ10jFJCJtN7Awmx99fhiz1lTxxsrKWIeTtJJxSqoPuNU5Nxw4EbjBzEY0LOCcu8s5N9Y5Nxb4CfCuc25PBGMSkXZwzUn9GNIjm1++upLD9f5Yh5OUkm5KqnOuwjm3MPS8BlgFHG1lrSuApyIVj4i0n1RPCndccALb9h7iofc2xjqcpFSXzFNSzawEGAfMbeb9TGAq8Hw04hGRtjt5UDemjerJn2atZ/u+Q7EOJ+kcGWjOSJaWwhFmlk3wy/4W51xzN4C9APigua4jM7vOzMrMrKyqqipSoYrIcbpt2nCcg1//e1WsQ0k69b7gIH5StRTMLI1gQnjSOffCUYpezlG6jpxz051zpc650sLCwvYOU0RaqTg/k++eOZBXl1Ywd6OWwGhPXr8fT4rhSbGonjeSs48MeBhY5Zz7w1HK5QJnAC9FKhYRiZzrzxhI77zO/OzlFfi0kmq78foCUZ+OCpFtKZwCXA2c3WDa6TQzu97Mrm9Q7mLgDedcbQRjEZEI6ZTm4fbzh7N6Rw2vLtWVzu3F6wtEfeYRRPB2nM652cAx2z3OuUeBRyMVh4hE3tQTelJSkMmTczdz0TjdvrM9eP0uJklBVzSLSJulpBhXTOrL/E17WVtZE+twkkIydh+JSAfypQnFpHtS+MfcLbEOJSl4/bHpPlJSEJF2UZCdwdSRPXlh4TYOeXWVc1vVq6UgIonuysl92X/Yx6tLt8c6lITn9QdIS43udFRQUhCRdjS5f1cGFmbxj3nqQmorjSmISMIzCw44L9qyj1UVzS1gIC0RqympSgoi0q6+NKGY9FQNOLdVcKDZE/XzKimISLvKy0znC6OK+L9F5Rz0+mIdTsIKdh9pTEFEksCVk/tSU+fjlSUacG4tTUkVkaQxoV8+Q3vk8LfZmwgEdMvO1qj3a6BZRJKEmfG9swayprKGV5dpPaTW8PoCUV82G5QURCRCLhjdi6E9crjnzbVaPbUVNPtIRJJKSorxg3OHsHFXLS8sLI91OAlHSUFEks65I3owpjiXe99eR51PS18cDw00i0jSMTNuPXco5fsO8cz8rbEOJ2E454JJQWMKIpJsThvcjUklXbnvnfVaKK+FfAGHcygpiEjyCbYWhlBVU8ff52yKdTgJoT40MJ9Ud14TETli8oACThvcjT/N3MDh+gCnDOrGmOJcUmPwSzgReH3BpKApqSKStH52wQhKCjK5+621XPrnDxn3izf51mNlLC+vjnVocedIUlBLQUSS1qDuObx046nsrfXy4YbdzF6/ixkrdvCdvy/g9VtOI6dTWqxDjBt1MUwKaimISFTlZ6Vz/ugi/veSUfz1mlIqqg/xP6+uinVYceXImEKGkoKIdCTj++Zz/RkDeaZsK++srox1OHHD69eYgoh0UDdPGcywnjn81/PL2FvrjXU4cSE8pqCkICIdTUaqhz98ZSz7Dnr575eWxzqcuBDLKalKCiIScyN6deGWKUN4dWmF7sHAJwPN6j4SkQ7rO6cPYGyfPH760nIO1HXsO7bFckqqkoKIxIVUTwp3XHgCew/W8+SczbEOJ6aOJAXNPhKRDm1snzxOG9yNh97fyOH6jrtOUr0/eLc6tRREpMO78axB7Drg5el5W2IdSsx4/cGEqDEFEenwJg8oYFJJV/7y3sYOew+GiurDAHTpFP1FJ5QURCTu3Hj2ICqqD3fYO7bNXL2TEUVdKMjOiPq5lRREJO6cNji4iuoDs9Z3uPs77631smDzXqYM7x6T8yspiEjcMTNuPHswW/cc4uUOdt3CzDU7CTg4Z3iPmJw/YknBzPqY2UwzW2VmK8zs5mbKnWlmi0Nl3o1UPCKSWM4Z1p1hPXP408z1+AMu1uFEzdurdlKYk8Go3rkxOX8kWwo+4Fbn3HDgROAGMxvRsICZ5QEPABc6504AvhzBeEQkgaSkGDeePYgNVbXc9846nEv+xOD1BXhvbRVnD+1OSorFJIaIJQXnXIVzbmHoeQ2wCujdqNiVwAvOuS2hcjsjFY+IJJ5pI4u4aGwv7nlrHT99aUXStxjmb9pDTZ2Pc2I0ngBRusmOmZUA44C5jd4aAqSZ2SwgB7jXOfd4NGISkfiXkmL84Stj6d6lE9Pf20hVTR33XD6WTmmeWIcWEW+tqiQ9NYVTB3eLWQwRH2g2s2zgeeAW59z+Rm+nAhOA84HPA/9tZkOaOMZ1ZlZmZmVVVVWRDllE4khKinHbtOH89xdG8PqKHXzt4XlUH6yPdVjtzjnH26t2csrAAjLTY3dTzIgmBTNLI5gQnnTOvdBEkW3A6865WufcLuA9YEzjQs656c65UudcaWFhYSRDFpE49c1T+3PfFeNYvHUfNz29KNbhtLv1Ow+wZc/BmM06OiKSs48MeBhY5Zz7QzPFXgJOM7NUM8sEJhMcexAR+YwLxvTihrMG8f66KiqqD7X6OIGA4/11VVQfip8Wx1urgkOqsRxPgMi2FE4BrgbODk05XWxm08zsejO7HsA5twp4HVgKzAP+6pzTXTZEpFlfHNsL5+DVJRWt2n95eTWXPvghVz88j/PueY85G3e3c4St8/aqSkYUdaEot3NM44hYx5VzbjZwzDlVzrm7gLsiFYeIJJeSblmMLs7l5SXb+fbpA1q8X/XBen7/5hqemLOZrlnp/OS8YTw9fytXPDSH688YyPenDInJqqQAe2q9LNyylxvPGhST8zekK5pFJOFcOKYXy8qr2Vh1oEXlF27Zy9m/n8UTczbztZNKePvWM/nOGQN59aZTuay0D3+etYFL/vwBG1p4vPY2c3Vsr2JuSElBRBLOBWN6YUaLl8C4+821eFKMV246lTsuPIHczmkAZGWkcuelo3nwqxPYtvcQF943m7dXVUYy9Ca9taoyplcxN6SkICIJp0eXTkzu35WXl2w/5pXO2/YeZPb6XVwxqS8n9Gr6S3fqyJ68dvNpDCjM5luPl/HguxuidgX1+p0HmLFiBxeO6RWzq5gbUlIQkYR04ZjebKyqZcX2xpc/fdpzC7YB8OXS4qOWK8rtzD+/cxLTRhVx52urufWfS6Jy97c/vLmGzmkevnvmwIifqyWUFEQkIZ03siepKXbULiR/wPFs2TZOHdSN4vzMYx6zc7qH+68Yxw8+N4QXFpVzxUNzqDkcuWmrS7bu49/LdvCt0wbQLQb3TmiKkoKIJKT8rHTOGFLIK0u2E2hmTaQP1u+ifN8hLpvYp8XHNTP+45zBPHDVeBZt2cf09za2V8if8dsZq+malc63TusfsXMcLyUFEUlYF47tRUX1Yco2723y/WfKtpKfmcbnRhz/rJ5po4o4f3QRf5v9MbsP1LU6Rudck+MTs9ft4oP1u7nhrEHkdEpr9fHbW+wW2BARaaMpw3vQKS2Fl5eUM6l/10+9t6fWyxsrdnD1iSVkpLZuAb3vTxnCa8sqePDdDdx+/ohj70BwYHvuxj2srNjPyu37WVmxn8x0Dz8+bxgXjumFmeGc4zevr6Z3Xmeumty3VbFFiloKIpKwsjJSmTK8B/9etoM636cHhV9cVE693x1X11Fjg7pnc/G4Yh7/aDOV+w8fs/zWPQeZ8od3ufXZJTwxZzMH6/1MG1VEt+wMbn56MZdNn8PqHft5bfkOlpVX8/3PDYm7FV/VUhCRhPaV0j68urSCSx74kHsuG8vgHjk45/jn/K2M6ZPH0J45bTr+LVMG89Licu57Zx3/c9Goo5a9+621BBy8dMMpnNCrC6me4O9uf8DxzPyt/HbGas7/42xyOqUyuHs2F49rfIuZ2FNLQUQS2ulDCvnL1ROoqD7MF+6bzSMffMyirftYU1nD5W1oJRzRp2sml03swzPzt7J1z8Fmy62trOHFReVcc1I/xvTJCycEAE+KceXkvsy89UyumNSH2joft00bjicOrktozBLtFnelpaWurKws1mGISJypqqnjx88v5e3VO8nJSMUXcMy7/Zx2GcTdUX2YM+6ayQVjevG7L39mdX8Arnu8jA837Oa9/zyLrlnpRz2ezx/4VNKIBjNb4JwrPVY5tRREJCkU5mTw12tKufNImdJMAAAIo0lEQVSSUfid45LxvdttVk/P3E5cfWI/Xli4jfU7P7s+0uKt+3hjZSXfPm3AMRMCEPWEcDziNzIRkeNkZlw+qS/zbp/CHRee0K7Hvv7MgXRK8/AfTy1i067aT71314zVFGSl8804ut6gtZQURCTpZGekktbOv8a7ZWfwpyvHU77vEOf/8X1eWlwOBC+Q+2D9br531iCyMxJ/7k7i10BEJErOGtadf998Gjc/tYibn17M7HW7WFtZQ6/cTnF3vUFrKSmIiByH3nmdefq6E7n37XXcP3M9zsFvLx0dd9cbtJaSgojIcUr1pHDruUM5aWABs9ft4pLx8Xe9QWspKYiItNLJA7tx8sBusQ6jXWmgWUREwpQUREQkTElBRETClBRERCRMSUFERMKUFEREJExJQUREwpQUREQkLOHup2BmVcDmRptzgepjbDva66aedwN2tTHcpuI63nKRqBu0vX6qW9vr1nhbc3VV3VquJfU73ro1tT1ev0+OVmawcy73mGdxziX8A5h+rG1He93Uc6AsEnEdb7lI1K096qe6tb1uR6tDw9eqW/vW73jrdrT4j1XXaH+ftKZujR/J0n30Sgu2He11c8/bqqXHOlo51a3514let8bbmqur6tZyLTne8datqe3x+nfZmrp9SsJ1H0WLmZW5Fty6LlElc/1Ut8SkusWHZGkpRML0WAcQYclcP9UtMalucUAtBRERCVNLQUREwjpEUjCzv5nZTjNb3op9J5jZMjNbb2Z/NDNr8N5NZrbGzFaY2W/bN+oWx9fudTOzO8ys3MwWhx7T2j/yFscYkc8u9P4PzcyZWUwWxI/QZ/dLM1sa+tzeMLNe7R95i+KLRN3uMrPVofq9aGZ57R95i+KLRN2+HPoeCZhZbMce2jpNKhEewOnAeGB5K/adB5wEGPAacF5o+1nAW0BG6HX3JKrbHcAPY/25Rap+off6ADMIXvPSLVnqBnRpUOY/gAeTqG7nAqmh578BfpNEdRsODAVmAaWxqNeRR4doKTjn3gP2NNxmZgPN7HUzW2Bm75vZsMb7mVkRwX9kH7ngJ/c4cFHo7e8Cdzrn6kLn2BnZWjQtQnWLGxGs393AfwIxG1SLRN2cc/sbFM0iRvWLUN3ecM75QkXnAMWRrUXTIlS3Vc65NdGI/1g6RFJoxnTgJufcBOCHwANNlOkNbGvweltoG8AQ4DQzm2tm75rZxIhGe3zaWjeAG0PN9L+ZWX7kQm2VNtXPzC4Eyp1zSyIdaCu0+bMzs1+Z2VbgKuCnEYz1eLXH3+UR1xL8pR0v2rNuMdUh79FsZtnAycCzDbqZM5oq2sS2I7+8UoF84ERgIvBPMxsQ+gUQM+1Utz8Dvwy9/iXwe4L/CGOurfUzs0zgdoJdEXGlnT47nHO3A7eb2U+AG4GftXOox6296hY61u2AD3iyPWNsrfasWzzokEmBYAtpn3NubMONZuYBFoRevkzwy7FhE7UY2B56vg14IZQE5plZgOD6JlWRDLwF2lw351xlg/0eAl6NZMDHqa31Gwj0B5aE/gEXAwvNbJJzbkeEYz+W9vi7bOgfwL+Ig6RAO9XNzK4BvgCcE+sfYA209+cWW7Ec0IjmAyihwcAQ8CHw5dBzA8Y0s998gq2BIwND00Lbrwd+EXo+BNhK6LqPJKhbUYMy3weeTqbPrlGZTcRooDlCn93gBmVuAp5LorpNBVYChbH8e4zk3yRxMNAc0/+xUfwAnwIqgHqCv/C/SfDX4uvAktAf2k+b2bcUWA5sAO4/8sUPpANPhN5bCJydRHX7O7AMWErwF05RtOoTjfo1KhOzpBChz+750PalBNe66Z1EdVtP8MfX4tAjVjOrIlG3i0PHqgMqgRmxqJtzTlc0i4jIJzry7CMREWlESUFERMKUFEREJExJQUREwpQUREQkTElBkoKZHYjy+f5qZiPa6Vj+0Kqmy83slWOt/mlmeWb2vfY4t0hjmpIqScHMDjjnstvxeKnuk8XXIqph7Gb2GLDWOfero5QvAV51zo2MRnzSsailIEnLzArN7Hkzmx96nBLaPsnMPjSzRaH/Dg1t/7qZPWtmrwBvmNmZZjbLzJ4LreP/ZIP172cdWffezA6EFqFbYmZzzKxHaPvA0Ov5ZvaLFrZmPuKThfuyzextM1towTX4vxgqcycwMNS6uCtU9keh8yw1s5+34/9G6WCUFCSZ3Qvc7ZybCFwK/DW0fTVwunNuHMFVRH/dYJ+TgGucc2eHXo8DbgFGAAOAU5o4TxYwxzk3BngP+HaD898bOv8x17gJrZVzDsGryAEOAxc758YTvH/H70NJ6cfABufcWOfcj8zsXGAwMAkYC0wws9OPdT6RpnTUBfGkY5gCjGiwcmUXM8sBcoHHzGwwwVUq0xrs86ZzruFa+fOcc9sAzGwxwTVvZjc6j5dPFg1cAHwu9PwkPrmHwz+A3zUTZ+cGx14AvBnabsCvQ1/wAYItiB5N7H9u6LEo9DqbYJJ4r5nziTRLSUGSWQpwknPuUMONZnYfMNM5d3Gof35Wg7drGx2jrsFzP03/m6l3nwzONVfmaA4558aaWS7B5HID8EeC90MoBCY45+rNbBPQqYn9Dfhf59xfjvO8Ip+h7iNJZm8QvJ8AAGZ2ZGnjXKA89PzrETz/HILdVgCXH6uwc66a4C00f2hmaQTj3BlKCGcB/UJFa4CcBrvOAK4NreuPmfU2s+7tVAfpYJQUJFlkmtm2Bo8fEPyCLQ0Nvq4kuNw5wG+B/zWzDwBPBGO6BfiBmc0DioDqY+3gnFtEcKXNywneRKbUzMoIthpWh8rsBj4ITWG9yzn3BsHuqY/MbBnwHJ9OGiItpimpIhESusvbIeecM7PLgSucc1881n4isaQxBZHImQDcH5oxtI84uaWpyNGopSAiImEaUxARkTAlBRERCVNSEBGRMCUFEREJU1IQEZEwJQUREQn7/0cKdXvZlfwAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This simple RNN seems to work *better* than the one we built using `nn.rnn`, and we're only using one layer and haven't implemented dropout.\n",
    "\n",
    "The big difference is that we're using an embedding layer instead of one-hot encoding. This gives us an extra bunch of parameters we can fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:08 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.743902</th>\n",
       "    <th>2.623438</th>\n",
       "    <th>0.239583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.569348</th>\n",
       "    <th>2.255966</th>\n",
       "    <th>0.333333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.323031</th>\n",
       "    <th>1.929247</th>\n",
       "    <th>0.397917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.109766</th>\n",
       "    <th>1.880335</th>\n",
       "    <th>0.406250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>1.942279</th>\n",
       "    <th>1.689045</th>\n",
       "    <th>0.460417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>1.745093</th>\n",
       "    <th>1.724750</th>\n",
       "    <th>0.452083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>1.561630</th>\n",
       "    <th>1.589873</th>\n",
       "    <th>0.508333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.374822</th>\n",
       "    <th>1.634860</th>\n",
       "    <th>0.525000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.204314</th>\n",
       "    <th>1.628457</th>\n",
       "    <th>0.518750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.037950</th>\n",
       "    <th>1.674552</th>\n",
       "    <th>0.552083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>0.891771</th>\n",
       "    <th>1.778297</th>\n",
       "    <th>0.552083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>0.772682</th>\n",
       "    <th>1.856418</th>\n",
       "    <th>0.541667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>0.666088</th>\n",
       "    <th>1.887347</th>\n",
       "    <th>0.581250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>0.572181</th>\n",
       "    <th>1.933961</th>\n",
       "    <th>0.545833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>0.487959</th>\n",
       "    <th>2.034596</th>\n",
       "    <th>0.568750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>0.415125</th>\n",
       "    <th>2.025807</th>\n",
       "    <th>0.556250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>0.356673</th>\n",
       "    <th>2.072717</th>\n",
       "    <th>0.558333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>0.310153</th>\n",
       "    <th>2.125560</th>\n",
       "    <th>0.560417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>0.271913</th>\n",
       "    <th>2.125447</th>\n",
       "    <th>0.560417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>0.244666</th>\n",
       "    <th>2.124661</th>\n",
       "    <th>0.558333</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(20, 7e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('rnn-bal-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Arabic',\n",
       " 'Chinese',\n",
       " 'Czech',\n",
       " 'Dutch',\n",
       " 'English',\n",
       " 'French',\n",
       " 'German',\n",
       " 'Greek',\n",
       " 'Irish',\n",
       " 'Italian',\n",
       " 'Japanese',\n",
       " 'Korean',\n",
       " 'Polish',\n",
       " 'Russian',\n",
       " 'Spanish',\n",
       " 'Vietnamese']"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the data classes; this will be useful if we want to make predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data.classes', 'wb') as f:\n",
    "    pickle.dump(data.classes, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the model data directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('models/rnn-bal-1.model', 'wb') as f:\n",
    "    pickle.dump(model.state_dict(), f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And read it back in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('models/rnn-bal-1.model', 'rb') as f:\n",
    "    state = pickle.load(f)\n",
    "    model.load_state_dict(state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batchnorm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(n_letters, n_hidden, n_output, bn=True).cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, model, loss_func=F.cross_entropy, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batch norm makes the learning surface much smoother"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl83HWdx/HXZ3LfPZLeR+hFW0ppaYotVQR0sXYFREFBxWu1sqKCoO6KrrerLCriBVRZ1BVQbkVdAbWclt4HbdP7vnPQ3Ofks3/MJGTTtE3b/OZI3s/HYx79zcx35vf5djL55Pv7XubuiIiIAITiHYCIiCQOJQUREemgpCAiIh2UFEREpIOSgoiIdFBSEBGRDkoKIiLSQUlBREQ6KCmIiEiH1HgHcKoKCwu9uLg43mGIiCSVlStXlrt70cnKJV1SKC4uZsWKFfEOQ0QkqZjZ7p6U0+UjERHpoKQgIiIdlBRERKSDkoKIiHRQUhARkQ5KCiIi0kFJQUREOigpiIgkgbv+upUXt5YFfh4lBRGRBOfu3PW3LSzdURn4uZQUREQSXF1zmDaH/KzgF6FQUhARSXBVDS0AFGSlBX4uJQURkQRXHU0K+ZlKCiIi/V5HUlBLQUREdPlIREQ6VDe2Arp8JCIivH75SC0FERHpuHyUm5nEQ1LNLNPMlpnZWjPbYGZf76bMRWa2ysxazezqoGIREUlm1Y0t5GWkkhKywM8VZEuhCbjU3c8DZgDzzWxOlzJ7gA8DDwYYh4hIUqtqaInJyCMIcI9md3egNno3LXrzLmV2AZhZW1BxiIgku+qG1pglhUD7FMwsxczWAEeAZ9196Wm+z0IzW2FmK8rKgl8QSkQkkVQ3tpAfg/4ECDgpuHvY3WcAo4ALzGzaab7PIncvcfeSoqKi3g1SRCTBVTe0xGTkEcRo9JG7HwWeA+bH4nwiIn1JdQz7FIIcfVRkZgOix1nAW4FNQZ1PRKSvqm5s7RMtheHAYjNbBywn0qfwRzP7hpldAWBms81sH3ANcK+ZbQgwHhGRpNMabqO2qTUms5kh2NFH64CZ3Tz+lU7Hy4n0N4iISDdq2pe4iMFeCqAZzSIiCS2Wi+GBkoKISEKrbozdXgqgpCAiktCqG9ovHykpiIj0e7p8JCIiHTouH6mjWUREYrk/MygpiIgktKqGFlJDRnZ6SkzOp6QgIpLAqhsjS1yYBb+XAigpiIgktOqG2C1xAUoKIiIJraohdstmg5KCiEhCa798FCtKCiIiCSyWW3GCkoKISEKrbojdCqmgpCAiktAil4/UpyAi0u81toRpbm3T6CMREYn9bGYIdjvOTDNbZmZrzWyDmX29mzIZZvY7M9tmZkvNrDioeEREkk37ukd9paXQBFzq7ucBM4D5ZjanS5l/AV5z9wnAncDtAcYjIpJU2ldI7ROjjzyiNno3LXrzLsWuBH4VPX4UeIvFai63iEiC69hLoa9MXjOzFDNbAxwBnnX3pV2KjAT2Arh7K1AFDA4yJhGRZBHrvRQg4KTg7mF3nwGMAi4ws2ldinTXKujamsDMFprZCjNbUVZWFkSoIiIJ5/W9FPpIUmjn7keB54D5XZ7aB4wGMLNUoACo7Ob1i9y9xN1LioqKAo5WRCQx9LXRR0VmNiB6nAW8FdjUpdgfgA9Fj68G/u7ux7QURET6o6qGFrLSUkhPjd3sgSB7L4YDvzKzFCLJ52F3/6OZfQNY4e5/AO4D/sfMthFpIVwbYDwiIkmluqE1prOZIcCk4O7rgJndPP6VTseNwDVBxSAiksyqG1tieukINKNZRCRhVTW0xHTkESgpiIgkrFjvpQBKCiIiCUstBRER6RDZSyG2Hc1KCiIiCaitzanR5SMREQGobW6lzWO7xAUoKYiIJKR4zGYGJQURkYTUsUJqjCevKSmIiCSgeOylAEoKIiIJqWOFVF0+EhGR6jjspQBKCiIiCUmXj0REpEN1YytmkJehjmYRkX6vuqGFvIxUQqHYbluvpCAikoCqG2I/mxmUFEREElI89lKAYLfjHG1mi82s1Mw2mNlN3ZQZaGZPmNk6M1tmZtOCikdEJJnEY4VUCLal0Arc6u5TgDnAjWY2tUuZ24A17j4d+CBwV4DxiIgkjXhsxQkBJgV3P+juq6LHNUApMLJLsanA36JlNgHFZjY0qJhERJJFdWPfayl0MLNiIvs1L+3y1FrgXdEyFwBjgVGxiElEJFG1htsoq2licG5GzM8deFIws1zgMeBmd6/u8vR3gYFmtgb4NLCayGWnru+x0MxWmNmKsrKyoEMWEYmrA0cbaW1zzhqcE/NzB3rByszSiCSEB9z98a7PR5PER6JlDdgZvXUttwhYBFBSUuJBxiwiEm87K+oAKC6MfVIIcvSRAfcBpe7+g+OUGWBm6dG7HwNe6KY1ISLSr+xuTwqDs2N+7iBbCvOA64FXo5eHIDLaaAyAu98DTAF+bWZhYCPwLwHGIyKSFHaW15GdnkJRXuz7FAJLCu7+EnDC+dnuvgSYGFQMIiLJaHdFPWMH5xC54BJbmtEsIpJgdpXXxeXSESgpiIgklNZwG3tfq49LJzMoKYiIJJSDVY20hF0tBRERiXQyAxTHYY4CKCmIiCSU3XGcowBKCiIiCWVneT1ZaSkMicNwVFBSEBFJKLsr6hg7ODsuw1FBSUFEJKHsrKjjrDhdOgIlBRGRhBFuc/ZWRiauxYuSgohIgjhwtCGuw1FBSUFEJGHsivPII1BSEBFJGLviPEcBlBRERBLGrop6MtNCDM2Pz3BUUFIQEUkYkYXw4rM6ajslBRGRBLGroi6ul45ASUFEJCFEhqM2MLYwfiOPQElBRCQhHDjaQHO4jbP6akvBzEab2WIzKzWzDWZ2UzdlCszsKTNbGy3zkaDiERFJZLsr6gHiOnENgt2juRW41d1XmVkesNLMnnX3jZ3K3AhsdPfLzawI2GxmD7h7c4BxiYgknJ3ROQrxXOICAmwpuPtBd18VPa4BSoGRXYsBeRbpas8FKokkExGRfmV3eR2ZaaG4rY7aLiZ9CmZWDMwElnZ56ifAFOAA8Cpwk7u3dfP6hWa2wsxWlJWVBRytiEjs7aqoY+ygHEKh+A1HhRgkBTPLBR4Dbnb36i5Pvw1YA4wAZgA/MbP8ru/h7ovcvcTdS4qKioIOWUQk5naWR5bMjrceJQUzG29mGdHji83sM2Y2oAevSyOSEB5w98e7KfIR4HGP2AbsBCb3PHwRkeTX1BpmV0U9k4bmxTuUHnc0PwaUmNkE4D7gD8CDwILjvSDaT3AfUOruPzhOsT3AW4AXzWwocDawo4cxnZKNB6p5YvU+ahpbqW5sobqhlZrGFhpawjS2tNHYEiYtJURJ8UDmjS/kwgmDGTUw+Kxd09jCnsp69lY20NQaJiM1hYy0EJmpKQwryGTMoGxS4tycFJFgbTtSS7jNmTw8eZJCm7u3mtlVwA/d/cdmtvokr5kHXA+8amZroo/dBowBcPd7gG8CvzSzVwED/s3dy0+5Fj2wp7KO/3llN/mZaRRkpZGflcaA7HSGp6WQmRYiMy2F2qZWXt5Wwe/XHABg1MAspo8q4JwRBZwzIp/zRg1gYE76KZ/b3Tlc3cTmwzVsPVzDlsM1bD1Sy67yOl6rbznhazNSQ0wcmsukoXkU5maQnhIiPTVyG16QyfiiXMYV5ZCd3jsDydydmqZW6pvCDM3PiOt0e5H+YtPBGgAmDzvm6nnM9fQ3SYuZXQd8CLg8+ljaiV7g7i8R+UV/ojIHgMt6GMMZeds5w9j0zbeftJy7s+VwLS9vK2f5rkrW76/mz68eAiA1ZLxlyhCunT2GiyYVdfsXfGNLmNKD1Ww4UE3pwWq2HK5h86EaqhtfH1RVmJvOhCG5vP3c4YwdlM3oQdmMHphNdkYKTS1tNLaGaWwOs+9oA1sO1bD5cA0vbyunqqGF5tY22vzYuEcUZFKUn0lBViTp5WWmEg479S1hGppbaWxpIzcjlUG56QzOSacgK42qhhaOVDdxuKaRI9VNVNQ18VpdC83hto73vHTKEC6dPIQLxxeSmZZymv/7InIimw5Vk5Eaius+Cu3MvZvfMF0LmU0FbgCWuPtDZnYW8F53/27QAXZVUlLiK1asiOk5qxpa2HigmsWbj/DYyn1U1DUzoiCTiyYV0Rxuo74pTH1LmENVDWw7UtvxSzs/M5XJw/KZNCyXs4fmMXFoHpOG5jHoNFobnbWG22hqbePA0cj5tpfVsqOsjvK6ZqoaWqiO3lJCRnZ6ClnpqWSmhahtbKWyrpnX6ptpcwgZFOZmMCQ/gyF5mQzOSe9IGmkpIZZsr+ClbeXUN4dJCRmZ0RZKWkr7zUgJGamhEGmpRm5GKvmZaeRlppGRFqKmsZWqhpaOZDY4J52ivAwKc9MpzM1gcG4Gg3PTKczJoDAvnSF5mcck2tZwG4drmqhraiUzNYXM9BBZaSlkp6fqspr0Gdfft5Sj9S089ek3BnYOM1vp7iUnLdeTpNDljQcCo9193ekGdybikRQ6a25t42+lh3lo+V427K8iMy2F7PQUsjNSKcxJ55wR+ZwzMnK5aeSArIS8/BJuc2obW8nJSCE15cRjDZpawyzdUcnyXZXUN4dpCbfR3NpGc7iNcJvT2uaEw05zuI2axpZIn01DC02tbeRlplKQHWmVpKcY5bXNlNc2UVbTRFPrMSOPSQkZQ/IyGFaQSXpKiP1HGzhY1Ui4m6aRGQzMTmdQTuTWnmjab2ZEEmBdM5X1zTR3OV96SoiMtFBHH07J2EFccnbRSf8/RIJQ8q1nueTsIdxxzXmBnaOnSaFHl4/M7Dngimj5NUCZmT3v7recUZRJKD01xNvPHc7bzx0e71BOW0rIKMg+4dW/DhmpKVw0qYiLJvXeUOD2fovK2mYq6pqpqG3iSE0Th6oaOVjVyKHqBppb2ygZO5CRA7MYOSCb/KxUGlvaIgMDmsPUNLZQUddMZV3kPTYfquHl2gqqGv5/H012egoDs9PJSOv0y96hpa2NxpY2mlrCNLSEuff5HQwvyOS6C8Zw7ezRDMnP7LX6ipxIWU0T5bXNTBke//4E6HmfQoG7V5vZx4D73f2rZhaXloIkPzMjPzON/My0Xt92sKk1TEVtZJWUQTnpPeoHaQm38bfSIzywdDc/eHYLP/rbVs4Zkc/UEflMGR65jR2UTWFuRtwnFknfs+lQZPpWIow8gp4nhVQzGw68B/hSgPGInJGM1BRGDMg6pdekpYSYP20Y86cNY1d5HY+s3MvqPUf53/WHeGjZ3o5yqSFjaH4mIwZkMm1kAXPGDeaC4kGnNSJNpF0ijTyCnieFbwBPAy+7+3IzGwdsDS4skfgoLszh82+LzJ90dw5VN1J6sJr9r0X6Nw5VNbL3tXoeXLqH+1/eBcDkYXl8YM5Y3jt7NGnqk5BTVHqomqH5GWc8AKW39CgpuPsjwCOd7u8A3h1UUCKJwMwYXpDF8IJjWx5NrWHW7ati6Y4Kni09wpefXM+iF3Zwyz9N4vLzRmhklPTYpoM1CdNKgJ53NI8CfkxkQpoDLxFZvG5fgLGJJKyM1BRmFw9idvEgbrxkAs9tLuO/nt7Mzb9bw08Xb+PNk4qYNCyPycPymDAkt9cmF0rf0hJuY9uRWt40qTDeoXTo6U/q/USWtbgmev8D0cf+KYigRJKJmXHJ5CG8eVIRf15/kJ+/uJNfv7K7YxhsSsi4eFIR15SM4tLJQ0lP1SUmidhZXkdzuI0pydZSAIrc/f5O939pZjcHEZBIsgqFjHdMH8E7po8g3Obsrqhjy+EaVu05ypOr9/O3TUcYlJPO/GnDcIeK2iYq65ppaXM+Oq+YK84bkZDzWiQ4pQcTa+QR9DwplJvZB4CHovevAyqCCUkk+aWEjHFFuYwrymX+tOF84W1n8+LWch5ZuZcnV+8nOz2FwTmRzsXqxiZu+u0a7n95F//xjinMGjso3uFLjGw6VENaijGuMDfeoXToaVL4KJENce4k0qfwDyLLXotID6SmhLhk8hAumTzkmOfCbc7jq/Zxx9ObeffdS7j8vBF87fKpDM6N7w5cErzSg9VMGJKXUJcUexSJu+9x9yvcvcjdh7j7O4F3BRybSL+QEjKuKRnN4s9dzGcuncDT6w/x9rte5KWtgSwYLAlk08EapgxLnEtHcGY7r/W7JS5EgpSTkcotl53NkzfOIz8rjQ/ct5Tv/Ln0mHWbpG94ra6ZQ9WNCdWfAD2/fNQd9YiJBGDqiHye+tQb+eafNnLvCzv4+6YjzJtQ2LGvxuRheeRl9mztKklcmw4l1kzmdmeSFE5teVUR6bGs9BT+86pzuWhiEfc8v51HVuylrjkMRBb5u/O9M3jbOcPiHKWciURb86jdCZOCmdXQ/S9/A064wIyZjQZ+DQwD2oBF7n5XlzKfB97fKZYpRIa/VvYoepE+rn1NprY2Z//RBrYeqeGuv27lht+s5N/nT2bhReM0jDVJbTpYE9ljJMEGFJwwKbj7maSwVuBWd19lZnnASjN71t03dnr/O4A7AMzscuCzSggixwqFLLJD36BsLhxfyK0Pr+U7/7uJ7WW1fOud5ybU6BXpmY0Hq5k8PC/hknpgP0nuftDdV0WPa4BSYOQJXnIdr8+DEJHjyExL4cfXzeQzl07g4RX7eP8vXmF3RV28w5JTUNPYwsaD1Zw/ZmC8QzlGTP68MLNiYCaw9DjPZwPzgcdiEY9IsguFjFsuO5u7rp1B6cEaLrvzBe5+bjstYY1USgbLd1USbnPmjhsc71COEXhSMLNcIr/sb3b36uMUu5zIstzdXjoys4VmtsLMVpSVlQUVqkjSuXLGSP56y5u5+Owibv/LJq74ycus2Xs03mHJSSzZXkF6aojzx/azloKZpRFJCA+4++MnKHotJ7h05O6L3L3E3UuKinpvW0iRvmBYQSb3Xl/CPR+YRWVdE1f97GU+/8hajlQ3xjs0OY5/bK/g/DEDerQzYKwFlhQs0ntyH1Dq7j84QbkC4M3A74OKRaQ/mD9tGH+95c0sfNM4nlyzn0u+9xw/XbyNxpZwvEOTTo7WN7PxYDVzxyXOctmdBdlSmAdcD1xqZmuitwVmdoOZ3dCp3FXAM+6unjKRM5SXmcYXF0zh2c++mQsnFHLH05v55x+9yIGjDfEOTaJe2VGJO1w4IfH6E+DMJq+dkLu/RA9mPbv7L4FfBhWHSH9UXJjDzz9YwnObj/CpB1fznnuX8ODH5jBmcHa8Q+v3XtlRQVZaCueNGhDvULqlwc0ifdjFZw/hwY+/gdqmVt5z7xK2l9XGO6R+b8n2CkqKBybs3JLEjEpEes30UQP47cI5tLa18d57l3QsryCxV17bxObDNcwdn5iXjkBJQaRfmDwsn999Yi6poRAfvG8ZVfUt8Q6pX3plR2RvskScn9BOSUGknxhflMsvPlRCRV0z3/7zxpO/QHrdP7ZXkJuRyrkjC+IdynEpKYj0I9NGFrDwonE8vGKfNvGJg1e2V3DBWYNITUncX72JG5mIBOKmt0xkXGEO//74OuqaWuMdTr9xqKqRHeV1CX3pCJQURPqdzLQUbr96Ovtea+COpzfHO5x+Y8mOSMsskTuZQUlBpF+aXTyID84dy6+W7GLlbq1WHwtLtldQkJXG1OGJtdNaV0oKIv3UF+ZPZkRBFp95aA17K+vjHU6ft3ZvFbPGDiQUSqz9E7pSUhDpp3IzUrnnA7OobWrlmnuWsEMT2wLj7uyurOOswpx4h3JSSgoi/di5owr47cI5tITbeM+9r7A5upm89K4jNU00trRRnATLjCgpiPRzU4bn87tPzCFkcO2iJazfXxXvkPqcXeWR9T7HDlZLQUSSwIQheTz8iblkp6dy7aJXeH6LNrPqTbsrIn02xUoKIpIsigtzePRf5zJ6UDYf/eVyHli6O94h9Rm7KupIDRkjBmTGO5STUlIQkQ7DC7J45Ia5XDSxkC89sZ7//HMpbW0e77CS3u6KekYNzEromcztEj9CEYmp3IxUfv7BEq6fM5ZFL+zglofXKDGcod2VdUnRnwDBbsc52swWm1mpmW0ws5uOU+7i6K5sG8zs+aDiEZGeS00J8Y0rz+Fzl03iyTUH+MYfN+KuxHA63J3d5fVJMfIIAtx5DWgFbnX3VWaWB6w0s2fdvWN5RjMbAPwMmO/ue8xsSIDxiMgpMDNuvGQCR+tb+MVLOynMTedTl06Md1hJp7KumZqm1qRpKQS5HedB4GD0uMbMSoGRQOc1e98HPO7ue6LljgQVj4icOjPjtgVTqKxr5nvPbGFQTgbve8OYeIeVVHZFRx6NVUvhdWZWDMwElnZ5ahKQZmbPAXnAXe7+61jEJCI9EwoZt189ndfqm/nyk69S39zKNSWjKchKi3doSWFPZfLMUYAYdDSbWS7wGHCzu3fdBzAVmAX8M/A24D/MbFI377HQzFaY2YqyMo2fFom1tJQQP33/+cwZN5hv/amU2d/+K596cBWLNx8hrE7oE9pVXo8ZjB6UFe9QeiTQloKZpRFJCA+4++PdFNkHlLt7HVBnZi8A5wFbOhdy90XAIoCSkhL9BIrEQXZ6Kg987A2s31/NY6v28eSa/fxx3UGumjmSO987I97hJazdFXWMKMgiIzUl3qH0SJCjjwy4Dyh19x8cp9jvgTeZWaqZZQNvAEqDiklEzoyZce6oAr52xTksu+2tLLxoHE+s3s9zm9UdeDy7KuqTpj8Bgr18NA+4Hrg0OuR0jZktMLMbzOwGAHcvBf4CrAOWAb9w9/UBxiQivSQ9NcStl01iXFEO//H79TQ0h48pc6S6kdZwWxyiSxx7KuuTpj8BAkwK7v6Su5u7T3f3GdHbn939Hne/p1O5O9x9qrtPc/cfBhWPiPS+jNQUvv3Oc9lb2cCP/771/z33xOp9zLv97yz8n5X9tt+hqqGFyrrmpJmjAJrRLCJnaO74wVw9axSLXtjB5kM1uDs/XbyNz/5uLSMHZPH3TUf4wbP9c9vPPR3DUdVSEJF+5LYFU8jLTOW2J17ltifWc8fTm7lyxgie/uxFXHfBGH66eDtPrT0Q7zBjbldF+3BUtRREpB8ZlJPObQumsHL3azy0bA+fvHg8d75nBhmpKXz9inMoGTuQzz+6lg0H+tdeDXsqk2viGigpiEgvuXrWKD72xrO44+rpfGH+5I69iNNTQ9z9gVkMzE5n4a9XUlHbFOdIY2dXeR1D8jLITo/JPOFeoaQgIr3CzPjyO6ZyTcnoY54rysvg3utnUV7bxM2/6z+rru6uqE+KjXU6U1IQkZiYPmoAX7l8Ki9uLWfRizviHU5M7KqoY0wSXToCJQURiaH3XTCGBecO43tPb2b1ntfiHU6g6ptbOVLTlFTDUUFJQURiyMz4zrumMzQ/k08/tJqqhpZ4hxSY1zuZdflIROS4CrLS+NF1MzlY1chtT7zaZzfv2VUeSQrqUxAROYlZYwdy62WT+NO6g/xu+d54hxOI3dE5CupTEBHpgRsuGs+8CYP5+lMb2VFWG+9wet2uinoGZqcl3b4TSgoiEhehkPH9a2aQkRbipt+uobm17yyc91pdMy9sKWN8UW68QzllSgoiEjfDCjL57rum8+r+Ku7865aTvyAJNLWG+cRvVlJW28QXF0yJdzinTElBROJq/rRhXDt7NPc8v50l2yviHc4ZcXdue3w9y3ZWcsfV05k1dmC8QzplSgoiEndfuXwqZw3O4ZaH13C0vjne4Zy2nz23ncdW7ePmt07kyhkj4x3OaVFSEJG4y05P5YfXzqC8tokP37+cmsbkmr/g7jy0bE/H6rA3vWVivEM6bUFuxznazBabWamZbTCzm7opc7GZVXXame0rQcUjIolt+qgB/PR957N+fxUfuX85dU2t8Q6pR9buPco19yzhi4+/yhvOGsTt755OZDfi5BRkS6EVuNXdpwBzgBvNbGo35V7stDPbNwKMR0QS3GXnDONH181k9d6jfPSXy7vd4jNRlNc2ccvDa7jypy+zq6KO777rXB78+Bwy01LiHdoZCWw9V3c/CByMHteYWSkwEtgY1DlFJPktOHc4LeE2Pvu7NXzs18u570OzE/IX7Vf/sIFnNxzmhjeP58ZLxpOXmVzzEY4nJn0KZlYMzASWdvP0XDNba2b/a2bnxCIeEUlsV84YyfeuOY9/bK/gy0+uT7ilMFrDbbywpYyrZo7k398+uc8kBAiwpdDOzHKBx4Cb3b26y9OrgLHuXmtmC4AngWN6aMxsIbAQYMyYMQFHLCKJ4F3nj2JXRT0/+ttWSsYO5NoLEue7/+r+KmoaW3njxMJ4h9LrAm0pmFkakYTwgLs/3vV5d69299ro8Z+BNDM75n/Z3Re5e4m7lxQVFQUZsogkkJveMpE3TSzkK3/YwPr9ibOV50tbywGYN0FJoccs0v1+H1Dq7j84Tplh0XKY2QXReJJ79oqI9JqUkHHXtTMpzEnnht+sTJg5DC9uK+ecEfkMykmPdyi9LsiWwjzgeuDSTkNOF5jZDWZ2Q7TM1cB6M1sL/Ai41hPt4qGIxNWgnHR++v7zOVzdyC0Pr437Vp51Ta2s3vNan7x0BMGOPnoJOOFgXXf/CfCToGIQkb5h5piBfPmfp/LVP2zg7ue3c+MlE+IWy7KdlbSEnTdN6JuXsjWjWUSSwgfnjuXy80bw/Wc284/t5XGL48Wt5WSkhigpTr51jXpCSUFEkkJkK89zOaswh888tIYj1Y1xiePlbeXMLh6UkHMneoOSgogkjdyMVO7+wCzqmlr51EOraQ3Hdg+GI9WNbD5c02f7E0BJQUSSzKSheXz7qmks21nJ95+N7R4ML22LXLZ6Yx8citpOSUFEks67zh/FdReM4e7ntrN405GYnfelbeUMykln6vD8mJ0z1pQURCQpffXyqUwelsfnH11LWU1T4Odzd17aWs6F4wcTCiXvKqgno6QgIkkpMy2FH183k5rGVj73SPDzF7YeqeVITRNv6sP9CaCkICJJbOLQPP7jHVN5fksZ//3yzl5/f3dn/9EGFm86ws8WbwP65tIWnQW+IJ6ISJDe/4YxvLCljNv/sok54wYzbWRBr7zv79fs50tPrKe202Y/F44fzKiB2b3y/olKSUF+zeRlAAAL5klEQVREkpqZcfu7pzP/rhf4zG9X88Qn51GQdeZLWf9p3UGy0lP44oLJTBqax6QheRRk950lso9Hl49EJOkNzEnnh++dyd7Ket577xIO98LEtvX7q5g7bjDvf8NYZhcP6hcJAZQURKSPmDt+MPd/+AL2Vtbz7rv/wY6y2tN+r4raJg5UNTJtZN8deno8Sgoi0me8cWIhDy2cQ0NzmKvvWcLavUdP633WH4jsB9Zb/RPJRElBRPqU6aMG8Oi/Xkh2egrv+/kr7K2sP+X3aN/Q55wRSgoiIknvrMIcHvr4HFrCzt3Pbz/l16/fX8XYwdm90mGdbJQURKRPGj0om/fMHsUjK/Zy4GjDKb12/YEqpvXDVgIoKYhIH/avF0c247nnFFoLR+ub2VvZ0C/7EyDYPZpHm9liMys1sw1mdtMJys42s7CZXR1UPCLS/4wckMXVs0bx22V7OVTVs2GqGzo6mfvfyCMItqXQCtzq7lOAOcCNZja1ayEzSwFuB54OMBYR6ac+efEEwu49bi28Gu1k1uWjXubuB919VfS4BigFRnZT9NPAY0Ds1r8VkX5j9KBs3jVzJA8t29Oj3drW769i5IAsBuakxyC6xBOTPgUzKwZmAku7PD4SuAq45ySvX2hmK8xsRVlZWVBhikgfdeMlE2htcxa9sOOkZdfvr+LcftqfADFICmaWS6QlcLO7V3d5+ofAv7l7+ETv4e6L3L3E3UuKioqCClVE+qjiwhyunDGC3yzdzcrdlcctV93Ywq6K+n7bnwABJwUzSyOSEB5w98e7KVIC/NbMdgFXAz8zs3cGGZOI9E9feNtkhhdk8f5fLOWvGw93W2bD/v47k7ldkKOPDLgPKHX3H3RXxt3Pcvdidy8GHgU+6e5PBhWTiPRfwwoyefSGuZw9NI9P/GYlDy/fe0yZDQeincxKCoGYB1wPXGpma6K3BWZ2g5ndEOB5RUS6NTg3gwc/Pod5Ewr5wmPr+Mnft+L++o5tr+6vYnhBJoW5GXGMMr4C20/B3V8CeryRqbt/OKhYRETa5WSkct+HSvi3R9fxvWe2cKSmia9efg4pIWP9/qp+ud5RZ9pkR0T6nbSUEN+75jwK8zJY9MIOymub+NY7z2VHeR1XnNfdyPn+Q0lBRPqlUMi4bcEUhuRl8K0/lbLxQDXu/XcmczslBRHp1z72pnEU5mbwuUfWAvTrOQqgpCAiwjtnjmRofibr9h1lSH5mvMOJKyUFEREi23nOHT843mHEnZbOFhGRDkoKIiLSQUlBREQ6KCmIiEgHJQUREemgpCAiIh2UFEREpIOSgoiIdLDOy8YmAzMrA3Z381QBUHWa99uP2/8tBMpPM8Su5zmVMt093pO4Ox93fizIegRZh87H/f2ziHcdOh8nymeh7/bp1WOsu59860p37xM3YNHp3m8/7vTvit6K41TKdPd4T+Lurg5B1yPIOuizSJw6JOJnoe/2mdXjZLe+dPnoqTO4/9RxyvRGHKdSprvHexJ35+PeqENP3ifIOvTk/D3RFz6LeNehpzGcTG/WQ9/tACXd5aNYMLMV7l4S7zjOVF+oR1+oA/SNeqgOiSPIevSllkJvWhTvAHpJX6hHX6gD9I16qA6JI7B6qKUgIiId1FIQEZEOfT4pmNl/m9kRM1t/Gq+dZWavmtk2M/uRmVmn5z5tZpvNbIOZ/VfvRn1MHL1eBzP7mpntN7M10duC3o/8mFgC+Syiz3/OzNzMCnsv4m7jCOKz+KaZrYt+Ds+Y2Yjej/yYWIKoxx1mtilalyfMbEDvR/7/4giiDtdEv9NtZhZY38OZxH6c9/uQmW2N3j7U6fETfm+6FdSwpkS5ARcB5wPrT+O1y4C5gAH/C7w9+vglwF+BjOj9IUlYh68Bn0v2zyL63GjgaSLzVwqTrQ5AfqcynwHuScbPArgMSI0e3w7cnoR1mAKcDTwHlCRa7NG4irs8NgjYEf13YPR44InqeaJbn28puPsLQGXnx8xsvJn9xcxWmtmLZja56+vMbDiRL+sSj/zv/hp4Z/TpfwW+6+5N0XMcScI6xFyA9bgT+AIQeAdZEHVw9+pORXNI3no84+6t0aKvAKOSsA6l7r45yLjPJPbjeBvwrLtXuvtrwLPA/NP9/vf5pHAci4BPu/ss4HPAz7opMxLY1+n+vuhjAJOAN5nZUjN73sxmBxpt9860DgCfijb1/9vMBgYX6gmdUT3M7Apgv7uvDTrQEzjjz8LMvm1me4H3A18JMNYT6Y2fqXYfJfKXaaz1Zh1irSexd2cksLfT/fb6nFY9+90ezWaWC1wIPNLp8lpGd0W7eaz9L7hUIs20OcBs4GEzGxfNxoHrpTrcDXwzev+bwPeJfJFj5kzrYWbZwJeIXLaIi176LHD3LwFfMrMvAp8CvtrLoZ5Qb9Uj+l5fAlqBB3ozxpPpzTrE2oliN7OPADdFH5sA/NnMmoGd7n4Vx6/PadWz3yUFIq2jo+4+o/ODZpYCrIze/QORX5qdm7+jgAPR433A49EksMzM2oisRVIWZOCdnHEd3P1wp9f9HPhjkAEfx5nWYzxwFrA2+kUaBawyswvc/VDAsbfrjZ+nzh4E/kSMkwK9VI9oJ+c7gLfE6o+kTnr7s4ilbmMHcPf7gfsBzOw54MPuvqtTkX3AxZ3ujyLS97CP06lnUB0piXQDiunUoQP8A7gmemzAecd53XIirYH2TpoF0cdvAL4RPZ5EpOlmSVaH4Z3KfBb4bTJ+Fl3K7CLgjuaAPouJncp8Gng0GT8LYD6wESiKRfxB/jwRcEfz6cbO8TuadxK5ejEwejyoJ/XsNq5YfXjxugEPAQeBFiKZ81+I/HX5F2Bt9If4K8d5bQmwHtgO/ITXJ/ulA7+JPrcKuDQJ6/A/wKvAOiJ/PQ0Psg5B1aNLmV0EP/ooiM/isejj64isbzMyGT8LYBuRP5DWRG+BjqIKqA5XRd+rCTgMPJ1IsdNNUog+/tHo//824COn8r3petOMZhER6dBfRx+JiEg3lBRERKSDkoKIiHRQUhARkQ5KCiIi0kFJQfoEM6uN8fl+YWZTe+m9whZZIXW9mT11stVFzWyAmX2yN84t0pWGpEqfYGa17p7bi++X6q8v7haozrGb2a+ALe7+7ROULwb+6O7TYhGf9C9qKUifZWZFZvaYmS2P3uZFH7/AzP5hZquj/54dffzDZvaImT0FPGNmF5vZc2b2qEX2CXigfT366OMl0ePa6IJ2a83sFTMbGn18fPT+cjP7Rg9bM0t4fbG/XDP7m5mtssia+FdGy3wXGB9tXdwRLfv56HnWmdnXe/G/UfoZJQXpy+4C7nT32cC7gV9EH98EXOTuM4msSPqfnV4zF/iQu18avT8TuBmYCowD5nVznhzgFXc/D3gB+Hin898VPf9J15yJrtHzFiIzzAEagavc/Xwie3h8P5qU/h3Y7u4z3P3zZnYZMBG4AJgBzDKzi052PpHu9McF8aT/eCswtdOqk/lmlgcUAL8ys4lEVo1M6/SaZ9298zr3y9x9H4CZrSGyXs1LXc7TzOsLCq4E/il6PJfX169/EPjeceLM6vTeK4mshw+R9Wr+M/oLvo1IC2JoN6+/LHpbHb2fSyRJvHCc84kcl5KC9GUhYK67N3R+0Mx+DCx296ui1+ef6/R0XZf3aOp0HKb770yLv945d7wyJ9Lg7jPMrIBIcrkR+BGRvRWKgFnu3mJmu4DMbl5vwHfc/d5TPK/IMXT5SPqyZ4jsTQCAmbUvS1wA7I8efzjA879C5LIVwLUnK+zuVUS24/ycmaURifNINCFcAoyNFq0B8jq99Gngo9E1+TGzkWY2pJfqIP2MkoL0Fdlmtq/T7RYiv2BLop2vG4kseQ7wX8B3zOxlICXAmG4GbjGzZcBwoOpkL3D31URWybyWyCY1JWa2gkirYVO0TAXwcnQI6x3u/gyRy1NLzOxV4FH+f9IQ6TENSRUJSHRnuAZ3dzO7FrjO3a882etE4kl9CiLBmQX8JDpi6Cgx3u5U5HSopSAiIh3UpyAiIh2UFEREpIOSgoiIdFBSEBGRDkoKIiLSQUlBREQ6/B+06KF3/W0OUwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:09 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.507374</th>\n",
       "    <th>2.190547</th>\n",
       "    <th>0.354167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.235302</th>\n",
       "    <th>1.959421</th>\n",
       "    <th>0.387500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.003024</th>\n",
       "    <th>1.930297</th>\n",
       "    <th>0.445833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>1.824707</th>\n",
       "    <th>1.977895</th>\n",
       "    <th>0.454167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>1.707330</th>\n",
       "    <th>2.065223</th>\n",
       "    <th>0.470833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>1.562633</th>\n",
       "    <th>2.085219</th>\n",
       "    <th>0.481250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>1.450853</th>\n",
       "    <th>2.300199</th>\n",
       "    <th>0.481250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.306937</th>\n",
       "    <th>2.536774</th>\n",
       "    <th>0.475000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.191327</th>\n",
       "    <th>2.522710</th>\n",
       "    <th>0.483333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.058434</th>\n",
       "    <th>2.229035</th>\n",
       "    <th>0.533333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>0.928646</th>\n",
       "    <th>2.419605</th>\n",
       "    <th>0.533333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>0.800899</th>\n",
       "    <th>2.562785</th>\n",
       "    <th>0.514583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>0.701665</th>\n",
       "    <th>2.489831</th>\n",
       "    <th>0.533333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>0.613110</th>\n",
       "    <th>2.690704</th>\n",
       "    <th>0.522917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>0.528683</th>\n",
       "    <th>2.681259</th>\n",
       "    <th>0.537500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>0.458904</th>\n",
       "    <th>2.672714</th>\n",
       "    <th>0.539583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>0.393891</th>\n",
       "    <th>2.499606</th>\n",
       "    <th>0.541667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>0.341363</th>\n",
       "    <th>2.683054</th>\n",
       "    <th>0.545833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>0.300728</th>\n",
       "    <th>2.611176</th>\n",
       "    <th>0.547917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>0.272150</th>\n",
       "    <th>2.567088</th>\n",
       "    <th>0.550000</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(20, 3e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case we actually get a very similar fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4VGX2wPHvSU9ISEgCBAgQmrQQQuhFqiKggAICCiKgItZV111dd62rP9uKiLoqNgQRUFhQsQAqvUkPvbeQ0IKEQEh/f3/cIUYkkIRM7kxyPs8zz8zcuXPn3Jlkztz7vu95xRiDUkopBeBhdwBKKaVchyYFpZRSeTQpKKWUyqNJQSmlVB5NCkoppfJoUlBKKZVHk4JSSqk8mhSUUkrl0aSglFIqj5fdARRVeHi4iYqKsjsMpZRyK+vWrTtpjKl8pfXcLilERUWxdu1au8NQSim3IiIHC7Oenj5SSimVR5OCUkqpPJoUlFJK5dGkoJRSKo8mBaWUUnk0KSillMqjSUEppVQetxunUFyJp8/z4dJ9PNWnMd6emguVumrpKbB+Mphc8AkE3yDHdWC++xWs2z6B4KH/d+6g3CSFLUdS+HT5AcIDfXmgW327w1HKvWWmwRdD4dCKwj/Hu0K+hBEIPheSRoV8yaMC+ATku13Bet4f1gmwbvtWBBHn7eOlxH8JKyZAo77QYhgER5bu65cCMcbYHUORtGrVyhR3RPMDU9fzw5YkalTy57m+TenRuGoJR6dUOZCTBdOHwe75MPAjuKYXZJ6FjLOQmeq4Luz9s5CVBpnnHJez1pFHYdTrAYM/s45ISsOGqfD1AxBUDVITAYF63SHuDmjYB7x8SyeOYhKRdcaYVldcrzwlhdNpmXy4dB8/bz/OzmOpTBrVhvpVAqkR4l/CUSpVRuXmwux7YfOXcOM4aH1XyW7fGMjO+D1BXEgWWef+mDhSEmDZeKgeC8NmQkBoycZxsXWfwbd/gXrdYOgXcPY4bJxqJYozCeAfCs2HQovhULWpc2MpJk0Kl3E+M4d+7yxj38lz5OQaJt7RkrqVK/DcN9t4sncjomsEl1C0SpUhxsAPT8CvH0D3p6Hz4/bGs+M7+GoUhNaBO2ZDxerOeZ21n8DcR6H+dTBkKnj7/f5Ybg7sWwjrp1jx5GZB9Tjr6CF6IPi5zneJJoUr2JqYwj9nb+HYmXT8vT3JNYYDyWnUCa/Adw93IsCn3DS3KFU4C1+Gxa9A+weh54ulfz7/UvYvgWm3WUcKd8yBsHolu/1fP4TvH4cGN8CQKZc/RXQuGeJnwIYpcHwbePlD05uhxR1Qu4Pt75cmhUL6YXMS901dT6UAb+7pXJfXftzJ8/2acmeHqBJ7DaXc3qr34ccnIHYY9H/X9i+4PziyHj4fCB5e1hFDRHTJbPfCPje8EW79tPBtBsZA4nrr6GHLLMg4A6F1rVNLzW+HitVKJr4i0qRQBPtOnKVmaADenh70fmspPp7C1w92KtHXUMptbZoBs8dAo5vg1s/A0wWPok/shMk3W20Pt38Ftdpe3fZWvgvznoLGfWHgJ+DlU7ztZKbBtq+to4eDy0E8oGo01GoPtdtb10ERVxdrIWlSKKaPlu7jxe+2M++RzjSMKKVeDUq5qp0/WD2NojpaX7b5z6e7mtOHrMRwJhGGfm61ARTH8rdgwTPQ5Gard5Wnd8nEl7wXNs+0kkPCGqvXFUClOvmSRAfrFJgTjsQ0KRRT8tkMOr+2kPb1wvnoziu+f0qVXQeWw+cDoEpjuPPb0uv6eTXOHrdiPr4DBkyE6AFFe/7SN+DnF6xG4lsmOu+oKCcLjsbDwZVwyHFJS7Yeq1AZarWzEkWt9hARUyJxaFK4Cu8t2surP+7gi3va0qFeuFNfSymXlLgRPutrndoY9SNUCLM7osI7fxqmDYVDq+CmN6HVqMI9b/FrsPAlaDYYbn6vdE+TGQMnd1uDAQ+tgoMr4LRjojTvClCztXUUcc0NVjfcYihsUnDBk4P2G9Uxio+X7eOjpfs1KSjXdmo/zLobUg47+smPgPCrHLF/co/VcOsXbPXocaeEAOAfAsP/B1+OgLmPQPpp6PRowesbA4tehsWvQvPbrIZ0D8/Sixes00WVr7EuLUday84kWkcQB1daiWLRy5CTWeykUFiaFC7Bz9uTYW1r89bPu9l34ix1KwfaHZJSf7bje5g91vpCqdUOVrxjnQ+v3QniRkCTfuBdxIGZKQkw5Wbr9h1zILhGycddGnwCrEFmc8bCT8/B+d/guuf/fK7eGPjlRVj6H6t3UN8JpZ8QClKxunUaK3qgdf/8acjNdvrLOi0piEhNYDIQAeQCE40xb120Tlfga2C/Y9H/jDEvOCumohjerjYfLNnLhJ93M35oC7vDUYW19A349SPrV25AqHXxd1wHhP1+299xPyAU/ELcq1hbTjb88m9YPh6qxcLgyVCpNqQehY1fWEXqZo+BH/4GMUOsBBHR7MrbPZcMU26xvnxGzr36Iw67efnAgA+tv4Xlb1mJ4abxv3/pG2MljOXjIe5Ox2Mu/HfgH1IqL+PMI4Vs4K/GmPUiEgSsE5EFxphtF6231BhzkxPjKJbKQb6M6liH9xfvZVTHOjSvWTofiLoKW+dYjYS1O0FAJUg7ZZ0KOX/KasQr8FeWgH8lK0GEN4Sb37Xuu6LUYzBzNBxcBq1Gww0v/94jKCgCrn0MOj5iPb5+slWe4deJUL2F9cUXPRD8Kv55uxmpMHWg1YNn+Cynn6IoNR6eVjkO/1DraCD9jNUA7ekDC56GFW9Dq7ugz39cOyGUolJraBaRr4F3jDEL8i3rCjxelKRQGg3NF6SkZdH7rSXkGvj2oU5UDnLtglfl2vEd8FEPqNIERn73537lxlhffBcSRNpv1vX5U1bySEu2Ltu/tU4j9Jtgz35czoHlMHOU9cXWd7zVhnAlaadg81dWcji+FbwDoOkAaHknRLa2TqdkpcPUQVbj5tCp0LC38/fFDivehvn/sorYhTWwynW0GQO9X3OtwXhO4lK9j0QkClgCRBtjzuRb3hWYBSQAiVgJYuvltlWaSQGschg3TljGk70bMbZLCQ+hVyUj/Qx82M26vnfx1dXAmf8v68tj5PdW33xXYIxVrvmn5606P4OnQNUmRd/GkfWwfhJsnmUN8qrcyDq1dHAF7JhrdcFsPsQpu+Ay1k+Bbx+2KrG2vQ96vVwuEgK4UFIQkUBgMfCSMeZ/Fz1WEcg1xpwVkT7AW8aYBpfYxhhgDECtWrVaHjx40KkxX+z6cYupUcmfSaPalOrrqkLIzYUv77AGWd357dV/kWeeg/+2A09fGLvM/sFa509b5Zp3zIUm/aHfO5c+/VMUGamwdbZ1eilhjbWs16vQbuzVx+sOdv8Ep/ZBm3vKTUIAF0kKIuINzAXmGWPGFWL9A0ArY8zJgtYp7SMFgKfnbOF/6xPY9GxPvHTWNtdyYbDRDS9D+/tLZpt7frK6ZHZ5Aro9VTLbLI6keKtbZcphuP7f0O6+kv8SO7YVziRBg2KO/lVuo7BJwWnfcCIiwMfA9oISgohEONZDRNo44kl2VkzF1a5uGOcyc9h8JMXuUFR+e362uhNGD7K+MEtK/eusAUxLx1ltFXZYPwU+vh6y0602kvb3O+dXbdWmmhDUHzjzZ29H4A6gu4hsdFz6iMhYEblwnDoI2CIim4AJwFDjgkOs29cLw8fTgxlrDtsdirrgt4Mw6y6o3NhqFC7pL8wb/s+aMvLbh61TVKUl67x1uuibB6FmW7h3qTUGQalS4rQuqcaYZcBl/1ONMe8A7zgrhpISWsGHYe1qMXnlQe7pXJd6OpjNXlnnrXaE3Fyrxr1PhZJ/jcDK0PMl+Pp+WPdpyc8wdimn9lmni45uhs5/g67/cJ2BVKrc0BPkhfRAt/r4eXkwbv4uu0Mp34yB7/4KSZus/uYlPalKfrG3Q53O1gCnM0nOex2wRid/0BVOH4bbv4Tu/9KEoGyhSaGQwgN9ufvauny3OYlx83dyNsP5w83VJaz9xJobt8uT0LCXc19LxBrlmpNpjQ52ls0zYcYwq7vpvUusomdK2USTQhHcfW0d2kSFMuGXPfxn3k67wyl/Dv9qzRHcoKfVM6g0hNWDLn+3BrVtn1vy29/2NfxvjFUiedT3VrkKpWykSaEIgvy8+XJse/rHVmfWugT2njhLTq7LtYuXTWePW+fbg2tYp41KsyRBh4ehSlP4/m/WALmSsuN7q2RFZCu4fYZz2kaUKiJNCsUwon1tUjOy6fHGYq5/czHbk0rwi0L9WU4WfDXSGsg15PPSr0vk6W31cEpNssZElIRd860kVy0Whs10jwlsVLmgSaEY4mpV4rVBMTxzUxNS0rJ48buLa/ypErXgWWsKw34TClft0xkiW1l1ctZ8ZJ3Guhp7foYZw61SFcNnXf0IZaVKkCaFYhARBreqyehOdbi3S12W70lm0+HTdodVNm2eCavehbZjIWawvbH0eNqqq/TtXyA7s3jb2LcYpt8O4ddY8xWUUjlkpQpLk8JVuq1NLSr6efH+4r12h1L2HNsK3zxkNcL2fNHuaKxTPDe+Ace3wYq3rrz+xQ6usKaJrFQHRsyxSnUr5WI0KVylID9vRrSP4setR9l74qzd4ZQd50/D9GHgWxFunWSd13cFDXtbhekWv27N1VBYh3+FqbdCcCTc+Q1U0GlelWvSpFACRnaMwsfTgw+X7LM7lLIhNxdm32sVghs82Zo8xpX0fg28/Kz5fwtTleXIOqvAXmAVGPGNda2Ui9KkUALCA30Z0roms9YncDQl3e5w3N+S12DXj1bl01pt7Y7mz4Ii4Prn4cBS2PD55ddN2mRNcelfySrtXbFa6cSoVDFpUigh91xbl1wD43/SMhhXZfNMWPQyNL/NqnfvquLutNo65v/LGkNxKUe3wOT+1imwkXOtU0dKuThNCiWkZmgAd3eqw/Q1h1my64Td4bingytgzn1QuyP0fcu1J0Dx8LBizEqDH//x58ePb4fJ/cDL3zpCCKlV+jEqVQyaFErQo9dfQ/0qgTwxK569J87y0dJ9bE3UORgK5eRuq6tmSG1rgJqXG8yHXbkhXPtX2DITdi/4ffnJ3fBZP/Dwto4QQuvYF6NSRVQqczSXJDtmXiuKTYdPM+C9FXnlL+JqhfC/+11krl9Xde4kfNQDMs7C3T+515dodga83wmy0uH+lXD2GEy6EXKzrclxKje0O0KlgMLPvOa0+RTKq+Y1Q5g5tj1bjqSw98Q5Jq04wJYjKUTXCLY7NNeUdd7qu596FO50w1/VXr7QdwJ82gu+ewwOLLcShSYE5ab09JETtKhViTvaR/Ho9dfg7+3J5JUH7A7JNV3oepqw1ipyV7O13REVT+320HIUxM+AzLMw4murhIVSbkiTghMF+3tzc4safL0xkd/OFbMsQln207NW6eieL1oDwtzZdc9By5FWQqgWY3MwShWfJgUnG9G+NhnZuXy1Tud3/oM1H8OKCdD6Hmj/gN3RXD3/EKs3UvVYuyNR6qpoUiiMtFNWmeMTRR+D0LhaRVpHVeKL1Ydwt0Z9p9k1H75/HBrcAL1ece2up0qVM5oUCiMrzWpAnH47pBe9i+ltbWpxIDmNlfuSnRCcm0naZM2NUDUaBn0CntrXQSlXokmhMIIjrRo8v+2HWfdYDaRF0KdZNYL9vZnw825yy/NMbSkJ8MUQq+TD7V+Cb6DdESmlLqJJobCiOlqnOnbPg0X/V6Sn+nl78lSfRqzad4qPlpXTonnpZ2DqYMg8B8O+1BpASrkoPXYvitZ3W6c/lrxuzQBWhB4zg1vV5Jcdx3l93k6Oncng2gbhdG1YTqplXphO8+ROGPYVVG1qd0RKqQLokUJRiFiTrES2htn3WZPAFPqpwisDYgir4MvHy/bzyIyNnE4rB91UjbEGde39GW56E+p1tzsipdRlaFIoKi9fGDzFmoVr+u1Wz6RCqlTBh/mPdWbWfe05cz6Lf87Zwoo9Jzlw8pwTA7bZsjdh/WS49nGIG2F3NEqpK9CkUBwVq1lF284kwszRkJNd+Kf6edOydih/7dmQ7+KTuP2j1fR6awkLdxZQftmdbZ4JPz8PzW6F7v+yOxqlVCFoUiiumq2hz39g30Lri6+IHuhWny/uacuE21pQNzyQR6ZvJPlshhMCtcnBlTDnfqjVAfq/q2MRlHIT2tB8NVreCUfjrZG5ETEQc2uRnt6hnjVPb6OIIPq8tZSXvtvOuCFuMCI267x1lHThkpp40f0kq1poaF0YOtU9ymArpQBNClev1yvWhCrfPAjhDYpV5uCaqkHc360+E37eTft6YdzaqqYTAi2i9DOwYy6cPgRnjsCZpN8TwPnf/ry+b7B1Wq1idajSBIJrWLOTBYSWfuxKqWLT+RRKwtkTMLELiAeMWQQVwou8iZxcwx0fr2b9od+Y80BHGkVULPEwC+XscVj1nlWbKCMFEKhQ2fqyv3AJqgYVaziSQA3rvg5EU8qlFXY+BU0KJSVxA3zSC2q0ghFzwNO7yJs4kZrBjROWEujrxdcPdiTIr+jbKLZT+2HF29ZE9DmZ1hiMDg9Zp8W8fEovDqWUUxQ2KTitoVlEaorIQhHZLiJbReQvl1hHRGSCiOwRkXgRiXNWPE5XvYVVJfPgMpj3z2JtonKQL2/f1oKDp9L45+wtJRxgAY5uhpl3wdtxsGEKNB8KD66FwZ9BZCtNCEqVM85sU8gG/mqMWS8iQcA6EVlgjNmWb53eQAPHpS3wnuPaPTUfCknxsOpdq6Z+i+FF3kTbumE80qMBbyzYRZ9mEfSKdkI5CGPg0EprDMHu+eATaJWvbveAlp9QqpxzWlIwxiQBSY7bqSKyHagB5E8K/YHJxjqHtUpEQkSkmuO57un6F+DYFpj7KFRuZP3aLqKxXesxb9tRHv8qnspBfrSsXalkYsvNtWo3LXsTDq+GgHBr/EDru60idUqpcq9UximISBTQAlh90UM1gPyzzyQ4lrkvTy+4dRIERcCM4dbcw0Xk7enBRyNaEx7ow/1T15GelXN1MeVkwabp8F4Haz7kM0nWGItHNkPnv2lCUErlcXpSEJFAYBbwiDHmzMUPX+Ipf2r5FpExIrJWRNaeOHHCGWGWrIBQGPqFNffCjDusidyLKCLYj5cHxHDsTAZTVx8qXhznkmH1BzChhTUXsggM+BAeXg9t7gGfgOJtVylVZjl1nIKIeGMlhKnGmP9dYpUEIH+n/Egg8eKVjDETgYlg9T5yQqglL6KZNZJ35ij44e9WI3QRta8XRsf6Ybz10y56R0dQPcT/8k84fxoOLof9S+HAUus0FkDNdlYhvwY9dWSxUuqynJYURESAj4HtxphxBaz2DfCgiEzHamBOcev2hItFD7BGPC97E86dtLp3hjeA8GsgrB54X+FLHnjp5mbcOGEpD3yxns9Gt6Fi/m6qGWetBuP9S6zL0XgwueDlBzXbWu0F9XpADfft1KWUKl1OG6cgIp2ApcBm4MJUZU8BtQCMMe87Esc7QC8gDRhljLnsIASXHadQkNwc+OEJq4H39GF+PzsmUKm2lSDCr/k9WYRfAwFhf/hF/+OWJB78YgMta/gx9QbB6+AyKwkkrofcbPDwtsp51+kMda61bmtpCaVUPjp4zRVlpsGpvXByF5zc7bjeBSf3QPb539fzr/THRJGVTvKWBQSe2ICvZIN4WuMiLiSBmu20fUApdVmFTQpa+6g0+QRYbQ0Rzf64PDcXziRclCx2w+4F1ghjhLBqMSwOH8hnR2vx0Mg7aNGgti27oJQq2/RIwdWdP21d+4dwNiOb3m8tISvb8PndbahfJcje2JRSbsP2MheqhPiHWBcg0NeLiXe0IjvXcMfHv179+AWllLqIJgU307haRSbcFktSSjoz1hy+8hOUUqoINCm4ofZ1w2gTFcq7C/dwLqPwU4EqpdSVaFJwQyLCE70bcTw1g3cW7rE7HKVUGaJJwU21rF2JAXE1+HjZfo6npuct33IkhTfm7yQrJ/cyz1ZKqUvTpODGHuregKycXD5dfgAAYwxPzIrn7V/28PeZ8ZoYlFJFpuMU3Fid8Ar0ia7GpOUH6NGoCifPZrA18Qwd6oUxe8MRjqem8/GdrfHz9rQ7VKWUm9AjBTf3bL8mVAv2Y8Qnv/KP/22mYdUgJo9uw2sDY1i+J5nnvtmKu41FUUrZR5OCm6sS5Me0Me2oXyWQjOxc3h3WAi9PDwa3rskD3eoxfc1h3vp5tyYGpVSh6OmjMqBqRT9m3deB1PRsQiv8PqfyX69vSNLpdMb/tJsVe5N5vl9T0rNy+M/8nfzfLc2oHVbBxqiVUq5Iy1yUcTm5hhlrDvP6vB2cy8ihbuUK7DiaStWKvtwUU52/3dBQ2xyUKge0zIUCwNNDuL1tLeY/2oUKvp7sOJpK/9jq1A0P5ONl+3n5++12h6iUciGaFMqJykG+/PvmaGqFBvDMTU2YNqYdozvW4bOVB3lb2xyUUg7aplCO3BRTnZtiqufdf7J3I06nZfLGgl0cOpXGE70bER6ok/MoVZ7pkUI55uPlwRuDm/NQ9/p8tS6BDq/8wsIdx+0OSyllI00K5ZyI8NeeDVnwaGeuqRrI2M/X8fScLWxNTLE7NKWUDTQpKAAaVA3is1FtuLFZNWasPcyNE5axYNsxu8NSSpUyTQoqT1igL+OGxPLrUz1oFBHE03O2kJKWZXdYSqlSpElB/UlIgA+vDIzh1LlMhkxcycHkc3aHpJQqJZoU1CXF1gzh45GtOPLbeXq+uYT3Fu3VqqtKlQOaFFSBrm1QmQWPdaFrw8q8+uMO+r2znPiE03aHpZRyIk0K6rIigv344I5WvD88juSzGdz87nJmb0iwOyyllJNoUlCF0iu6Ggse60LbOmH87at4lu0+aXdISikn0KSgCi3Y35sPRrSkfhVrPMO2xDN2h6SUKmGaFFSRVPTz5tNRrQn09WLUpF9JSjlvd0hKqRKkSUEVWbVgfyaNbs3Z9Gz+Mm0j2dorSakyo1BJQUTqiYiv43ZXEXlYREKcG5pyZY0iKvLiLdH8euAUL/+wgydmxvP95iRyc7XaqlLurLBVUmcBrUSkPvAx8A3wBdDHWYEp13dLi0iW7jrJx8v2AzBj7WH8vT0Z0romf+/VkAAfLcKrlLsp7H9trjEmW0RuAcYbY94WkQ3ODEy5hxdujsbb04N+sdU5nprOst3JfLbyABnZubw8oJnd4SmliqiwSSFLRG4D7gT6OpZ5Oyck5U4Cfb14dVBM3v1bWkQS5OfFlFUHuatTFPWrBNkYnVKqqArb0DwKaA+8ZIzZLyJ1gM8v9wQR+UREjovIlgIe7yoiKSKy0XF5pmihK1f1UPf6VPDx5MEvNpCargX1lHInhUoKxphtxpiHjTHTRKQSEGSMeeUKT5sE9LrCOkuNMbGOywuFiUW5vrBAX/47rCW7j5+l/zvL2X0s1e6QlFKFVNjeR4tEpKKIhAKbgE9FZNzlnmOMWQKcKoEYlRvq1CCcKaPbcCY9izFT1nEuIzvvsczsXO2lpJSLKuzpo2BjzBlgAPCpMaYlcF0JvH57EdkkIj+ISNMS2J5yIR3qh/P2bXEcSD7HkIkrWXPgFEkp5+n82kI6vPIL7y7cw/nMHLvDVErlU9ik4CUi1YDBwNwSeu31QG1jTHPgbWBOQSuKyBgRWSsia0+cOFFCL69KQ/t6Ybw/vCVHU9K59f2VdH5tIanpWVwTEcTr83Zy44SlzI1PJOW8tj0o5QrEmCsfxovIrcDTwHJjzH0iUhd43Rgz8ArPiwLmGmOiC/EaB4BWxpjLVlpr1aqVWbt27RVjVq4lLTObbzclsuXIGfrFVqd1VCjL95zk8a82kZSSTqCvFw90q8+9nevi4SF2h6tUmSMi64wxra64XmGSwlUEEUUBSUFEIoBjxhgjIm2AmVhHDpcNSJNC2ZKZncumhNNMXLKPBduO0b1RFd4cHEtwgPZ4VqokFTYpFLahOVJEZju6mB4TkVkiEnmF50wDVgINRSRBRO4SkbEiMtaxyiBgi4hsAiYAQ6+UEFTZ4+PlQeuoUCbe0ZLn+zVl6e4TDPt4lbY1KGWTwp4+WoBV1mKKY9FwYJgx5nonxnZJeqRQti3YdowxU9bSr3l1xg+JRURPJSlVEkr0SAGobIz51BiT7bhMAipfVYRKXcL1TaryeM+GfL0xkdfn7SRHu64qVaoKmxROishwEfF0XIYDyc4MTJVf93etx60tI/nvor0M+2iVztmgVCkqbFIYjdUd9SiQhNUeMMpZQanyTUR4bVAMrw2KIT4hhV7jlzJ/61G7w1KqXChsmYtDxph+xpjKxpgqxpibsQayKeUUIsLgVjX57uFrqR0WwL2fr2PS8v12h6VUmXc1M689VmJRKFWAOuEV+PLe9lzfuCrPfbuNV37YgXZSU8p5riYpaLcQVSr8vD15b3hLhrWtxfuL9/LXLzeRpVOAKuUUVzM1lv5cU6XG00N48eZoIir68caCXZw4m8F7w1sS6KuzuylVki57pCAiqSJy5hKXVKB6KcWoFGC1MzzUowGvDYxhxd5khk5cyYnUDLvDUqpMuezPLGOMTpulXM7g1jUJD/Lhgakb6Pv2MrpcUxkfLw8eua4BYYG+doenlFu7mjYFpWzTvVFVvhrbnpAAb77ZlMj0NYfo+eYS5mnXVaWuilML4jmDlrlQ+RljyMk17D1xjse/2sTmIyn8vVdD+sfWIKyCD37ennaHqJRLcIkqqc6gSUEVJD0rh4embWDBtmMABPl58WzfpgxqednajUqVC4VNCtp1Q5UZft6eTLyjJZsSUtiWeIY5G4/w+FebOHQqjUeva6DF9ZQqBG1TUGWKiBBbM4Tb29bii7vbMqhlJBN+3s09k9dyOi3T7vCUcnmaFFSZ5eXpweuDYniubxMW7zrB9W8u4b+L9pCRrXM1KFUQTQqqTBMRRnasw6z7OtAoIojXftxJv7eXs+VIit2hKeWSNCmociEmMoQpd7Xlk5Gt+C0tk/7vLueN+Tt1hjelLqJJQZUr3RtVZcGjXejfvDpv/7KH68YtZs2BU3aHpZTL0KSgyp3gAG/GDYll+ph2eHkKQycBWDthAAAbnElEQVSu4u2fd5OWmW13aErZTpOCKrfa1Q1j7kOd6NOsGm8s2MW1ry5k97FUu8NSylaaFFS5FuTnzYShsXw1tj0iwl2frWXfibN2h6WUbTQpqHJPRGgdFcqHI1qSmp5F37eXsWjncbvDUsoWmhSUcmhRqxLf/+VaaodV4K7P1vLG/J06pkGVO5oUlMqnWrA/M+5tx82xNXj7lz30e3s58Qmn7Q5LqVKjSUGpiwT5efPG4OZ8MrIVp89ncst/V/Dajzv0qEGVC5oUlCpA90ZVmf9oFwbG1eC/i/Zy04RlbDysRw2qbNOkoNRlBPt789qg5kwa1ZqzGdkM+O9yXv5hO+lZetSgyiZNCkoVQteGVZj3aGeGtK7JB4v30WfCUtYd/M3usJQqcZoUlCqkin7evDwghil3tSEjK5dB76/gn7M3czZDR0KrskOTglJFdG2Dysx7tDMjO0Qx7ddDDJ24UnsoqTJDk4JSxRDoa031+fGdrTl4Mo1+7yxn0vL9doel1FXTpKDUVejWqAor/tGd6xpX5YW525iy6iDuNu+5Uvk5LSmIyCciclxEthTwuIjIBBHZIyLxIhLnrFiUcqYgP2/GD42lY/1wnp6zhZGfruHk2Qy7w1KqWJx5pDAJ6HWZx3sDDRyXMcB7ToxFKacK9PXis1FteKF/U1btS2bwBytJ+C3N7rCUKjKnJQVjzBLgcrOX9AcmG8sqIEREqjkrHqWczcNDGNE+iil3teVEagY3TljGLzuO2R2WUkViZ5tCDeBwvvsJjmV/IiJjRGStiKw9ceJEqQSnVHG1qRPKtw92onqIP6MnrWXAf5fz/eYkbWtQbsHOpCCXWHbJ/xpjzERjTCtjTKvKlSs7OSylrl5UeAVm39+Bx66/htT0bO6fup7Rk9aQlHLe7tCUuiw7k0ICUDPf/Ugg0aZYlCpxft6ePNyjAT/85VqevqkJq/ef4qYJy1i5N9nu0JQqkJ1J4RtghKMXUjsgxRiTZGM8SjmFl6cHd3WqwzcPdiIkwJvhH6/m5R+2k3I+y+7QlPoTZ3ZJnQasBBqKSIKI3CUiY0VkrGOV74F9wB7gQ+B+Z8WilCuoXyWQrx/sxC0tajBxyT56vLGYbzclaluDcinibn+QrVq1MmvXrrU7DKWuypYjKfzjf5vZfCSFtnVCefHmaBpUDbI7LFWGicg6Y0yrK62nI5qVskF0jWDmPNCRf/dvyu7jZ7ntw9UcOHnO7rCU0qSglF08PYQ72kfx5b3tyMnNpf+7y/lpm45rUPbSpKCUzepXCWLOAx2JrOTP3ZPX8s/Zmzmdlml3WKqc0qSglAuoHVaBWfd1YHTHOkxfc5i+7yxj59FUu8NS5ZAmBaVchJ+3J8/0bcLMse05n5lD37eX8e7CPWTl5NodmipHNCko5WJa1KrED3/pzPVNq/L6vJ30f2c5W46k2B2WKic0KSjlgioH+fLu7XF8cEdLTpzNoP+7y3n1xx2kZ+XYHZoq4zQpKOXCbmgawU+PdmFQXCTvLdpLn7eWsubA5YoPK3V1NCko5eKCA7x5dVAMn9/VlsycXG59fyXPfL2FsxnZdoemyiBNCkq5iU4Nwpn/aGdGd6zDlFUH6fr6Ij5bcYCMbD2lpEqOJgWl3EiAjxfP9G3C7Ps7Uq9yBZ79Ziv931nO8TPpdoemyghNCkq5odiaIUwf044P7mjJoVNp9HtnOSv2nrQ7LFUGaFJQyk2JCDc0jeDLe9sT4OPJsI9W8/hXm1i487hWXlXFplVSlSoD0jKzefn7HczecISzGdm0iQpldKcoomsEE1kpwO7wlAsobJVUTQpKlSFZObnMWHOY8T/t4uTZTHw8PXj0+msY2SEKfx9Pu8NTNtKkoFQ5dj4zh+1Hz/Deor0s2HaM8EBf7u1cl2HtahHg42V3eMoGmhSUUgCs3pfM27/sYdmek4RW8OH+rvUY1bEOnh5id2iqFBU2KehPBqXKuLZ1w2hbN4x1B08x/qfdvPjddr7bnMS9netxXeMqeHlqfxP1O/1rUKqcaFk7lMmj2/DGrc05lpLO2M/Xcd24xWxO0GJ76neaFJQqR0SEgS0jWfL3brw/vCWZ2bkMfG8FU1YdJDfXvU4lK+fQpKBUOeTl6UGv6AjmPnwt7eqF8fScLfSZsJRfdhzTMQ7lnDY0K1XO5eYa5m5O4o35OzmYnEb1YD9CA324s30U11QNIiYyGBFtlHZ32vtIKVUkmdm5zN6QwJJdJ9l1LJXdx88C0KFeGP+8sTFNqwfbHKG6GpoUlFLFlp2Ty46jqaw5cIoJP+/m9PksBsZF8pceDagZqiOk3ZEmBaVUiUg5n8W7C/cwafkBcoxhQIsaPNCtPlHhFewOTRWBJgWlVIk6mpLOB0v28sXqQ2Tm5NKpfjhP39SEa6oG2R2aKoRylRSysrJISEggPV1rypckPz8/IiMj8fb2tjsU5UKOp6YzddUhPl91kNT0bK5rUoV+zWtwfZOqOkrahZWrpLB//36CgoIICwvTXhIlxBhDcnIyqamp1KlTx+5wlAs6kZrBO7/sZm58EsnnMmlYNYgnejekW8Mq+n/oggqbFMrEOIX09HRNCCVMRAgLC9OjL1WgykG+PN8/mtVP9eCd21uQkZ3D6ElrGfjeCr5YfYjzmTpNqDsqE0kB0ITgBPqeqsLw8vTgppjqLHisCy/dEs2pc5k8NXsz141bzOSVBzibkW13iKoIykxSsFNycjKxsbHExsYSERFBjRo18u5nZmYWahujRo1i586dTo5UKefx9vRgWNvaLHy8K9PuaUdYoA/PfL2V9v/3M0/P2cLWRK2x5A7KRJvC9u3bady4sU0R/dFzzz1HYGAgjz/++B+WG2MwxuDh4V552JXeW+VejDFsOHyaKSsP8t3mJDKzc2lRK4QR7WvTp1k1fL100p/S5BJtCiLSS0R2isgeEXnyEo+PFJETIrLRcbnbmfGUtj179hAdHc3YsWOJi4sjKSmJMWPG0KpVK5o2bcoLL7yQt26nTp3YuHEj2dnZhISE8OSTT9K8eXPat2/P8ePHbdwLpYpHRIirVYk3h8Sy5qnreOamJqSkZfHojE10ePkXPl2+n+ycXLvDVBdx2nwKIuIJvAtcDyQAa0TkG2PMtotWnWGMebCkXvf5b7eyLfFMSW0OgCbVK/Js36bFeu62bdv49NNPef/99wF45ZVXCA0NJTs7m27dujFo0CCaNGnyh+ekpKTQpUsXXnnlFR577DE++eQTnnzyTzlVKbcRHODN6E51GNkhiuV7T/LB4n08/+02Zqw5zBO9GtG1YWVtw3IRzjxSaAPsMcbsM8ZkAtOB/k58PZdUr149WrdunXd/2rRpxMXFERcXx/bt29m27eIcCf7+/vTu3RuAli1bcuDAgdIKVymn8vAQrm1QmSl3teG9YXGkpmczatIaeo1fyqx1CWRka48luzlz5rUawOF89xOAtpdYb6CIdAZ2AY8aYw5fvIKIjAHGANSqVeuyL1rcX/TOUqHC76UAdu/ezVtvvcWvv/5KSEgIw4cPv2SXTx8fn7zbnp6eZGdr7w1VtogIvZtVo0fjqny7KZEPluzlr19t4rlvt9KzSQSNIoJoUyeU5jVD7A613HFmUrjUseDFrdrfAtOMMRkiMhb4DOj+pycZMxGYCFZDc0kHWlrOnDlDUFAQFStWJCkpiXnz5tGrVy+7w1LKNj5eHgxsGcmAuBos23OSrzcmMm/rUWatTwCgWY1gWtQKoXlkCDdERxDoqzMIO5sz3+EEoGa++5FAYv4VjDHJ+e5+CLzqxHhsFxcXR5MmTYiOjqZu3bp07NjR7pCUcgki1mmlaxtU5rWBMZw+n8V38YnMWHuY2RuOMHnlQf45ZzM9m0RwS4saXNsgXOeWdhKndUkVES+sU0I9gCPAGuB2Y8zWfOtUM8YkOW7fAjxhjGl3ue26epfUskbfW2U3YwzrD51m9oYE5sYncToti5qh/jzUrQG3xNXAW5NDoRS2S6rTjhSMMdki8iAwD/AEPjHGbBWRF4C1xphvgIdFpB+QDZwCRjorHqWUexIRWtauRMvalXjmpqb8suMY/120l7/Piuel77fTtWFlujeqQrdGVajop8Ubr5YOXlOXpe+tckXGGBbtOsHcTUks3HmcU+cy8ff2pF/z6gxqFUlcrUpasfUith8pKKWUs4gI3RpWoVvDKuTkGjYe/o2Z6xKYs8FqhwgP9OG6xlXp2bQqHeqF4+eto6cLS5OCUsqteXoILWuH0rJ2KE/1aczCnSeYv/Uoc+OTmL7mMAE+nnRtWJle0dXo3qiK9mC6An13lFJlRpCfN/2aV6df8+pkZOewcm8y87cdY8G2Y3y/+Sg+Xh50b1iFYe1q0a5umDZSX4ImBaVUmeTr5UnXhlXo2rAK/+4fzfpDv/HD5qPM3pDAj1uPEuTnRbeGVbiuSVW6XFOZYH9tpAYtnV0iunbtyrx58/6wbPz48dx///0FPicwMBCAxMREBg0aVOB2rzQf9fjx40lLS8u736dPH06fPl3Y0JUqFzw9hNZRoTzTtwkr/9GDiXe0pHd0BMv3nOThaRto+e8F3P7hKj5Ztp+Dyedwtw44JUmPFErAbbfdxvTp07nhhhvylk2fPp3XX3/9is+tXr06M2fOLPZrjx8/nuHDhxMQEADA999/X+xtKVUe+Hl70rNpBD2bRjgaqU/z8/Zj/LT9GC/M3cYLc7dRI8SfjvXD6FAvnA71wqhS0c/usEuNHimUgEGDBjF37lwyMjIAOHDgAImJicTGxtKjRw/i4uJo1qwZX3/99Z+ee+DAAaKjowE4f/48Q4cOJSYmhiFDhnD+/Pm89e677768ktvPPvssABMmTCAxMZFu3brRrVs3AKKiojh58iQA48aNIzo6mujoaMaPH5/3eo0bN+aee+6hadOm9OzZ8w+vo1R5YjVSV+LvvRox/9EuLPlbN17o35RmNYKZt/UYj8zYSJv/+5nu/1nEg1+s58Ml+zh8Kq1Ml/wue0cKPzwJRzeX7DYjmkHvVwp8OCwsjDZt2vDjjz/Sv39/pk+fzpAhQ/D392f27NlUrFiRkydP0q5dO/r161dgieD33nuPgIAA4uPjiY+PJy4uLu+xl156idDQUHJycujRowfx8fE8/PDDjBs3joULFxIeHv6Hba1bt45PP/2U1atXY4yhbdu2dOnShUqVKrF7926mTZvGhx9+yODBg5k1axbDhw8vmfdKKTdWKyyAEe2jGNE+ipxcw7bEM6zYe5I1B35j4+HTzI1P4qXvt+PtKcREhtC2Tiht64bRsnalMtOrqWzshQu4cArpQlL45JNPMMbw1FNPsWTJEjw8PDhy5AjHjh0jIiLikttYsmQJDz/8MAAxMTHExMTkPfbll18yceJEsrOzSUpKYtu2bX94/GLLli3jlltuyavSOmDAAJYuXUq/fv2oU6cOsbGxgJbmVqognh5Cs8hgmkUGc28Xa9mBk+dYsvsECb+d59f9p/hgyT7+u2gvHgINqgTRLDKYmMhgYiJDaBQR5JbjI8peUrjML3pnuvnmm3nsscdYv34958+fJy4ujkmTJnHixAnWrVuHt7c3UVFRlyyVnd+ljiL279/Pf/7zH9asWUOlSpUYOXLkFbdzuYYyX1/fvNuenp56+kipQooKr0BU+O/l8M9lZLP+0G+sOfAbmxNOs3DHcWausyq8enkIDSOCiIkMISYymGY1gmkYEeTy3WDLXlKwSWBgIF27dmX06NHcdtttgDWDWpUqVfD29mbhwoUcPHjwstvo3LkzU6dOpVu3bmzZsoX4+HjAKrldoUIFgoODOXbsGD/88ANdu3YFICgoiNTU1D+dPurcuTMjR47kySefxBjD7NmzmTJlSsnvuFLlWAVfr7zqrmD9GEtMSWdzwmniE1KIT0jhu/hEpv16CLBKhTepVpGYyGCaVKtI42oVaehiRxSaFErQbbfdxoABA5g+fToAw4YNo2/fvrRq1YrY2FgaNWp02effd999jBo1ipiYGGJjY2nTpg0AzZs3p0WLFjRt2vRPJbfHjBlD7969qVatGgsXLsxbHhcXx8iRI/O2cffdd9OiRQs9VaSUE4kINUL8qRHiT6/oaoCVKA6dSnMkCStZzFqXwORMa5Y5D4E64RVoVK0ilQK88ff2JCLYn6iwAGqHVaBmqD++XqWXNLQgnrosfW+VKnm5uYbDv6WxPekM25JS2Z50hl3HUklNz+ZcRjYZ2b/3bhKB6sH+3NWpDqM71Sn2a2pBPKWUclEeHkLtsArUDquQd0RxgTGGU+cyOXgqjYPJ5ziYnMbB5DQqB/kWsLWSpUlBKaVciIgQFuhLWKAvcbUqlfrru3YzuFJKqVJVZpKCu7WNuAN9T5Uqf8pEUvDz8yM5OVm/xEqQMYbk5GT8/MpPzRelVBlpU4iMjCQhIYETJ07YHUqZ4ufnR2RkpN1hKKVKUZlICt7e3tSpU/yuWkoppSxl4vSRUkqpkqFJQSmlVB5NCkoppfK4XZkLETkBXL6yXMHCgZMlGI6ddF9cU1nZl7KyH6D7ckFtY0zlK63kdknhaojI2sLU/nAHui+uqazsS1nZD9B9KSo9faSUUiqPJgWllFJ5yltSmGh3ACVI98U1lZV9KSv7AbovRVKu2hSUUkpdXnk7UlBKKXUZ5SYpiEgvEdkpIntE5Em74ykqETkgIptFZKOIrHUsCxWRBSKy23Fd+sXXC0FEPhGR4yKyJd+yS8YulgmOzyleROLsi/yPCtiP50TkiONz2SgiffI99g/HfuwUkRvsifrSRKSmiCwUke0islVE/uJY7lafy2X2w+0+FxHxE5FfRWSTY1+edyyvIyKrHZ/JDBHxcSz3ddzf43g8qkQCMcaU+QvgCewF6gI+wCagid1xFXEfDgDhFy17DXjScftJ4FW74ywg9s5AHLDlSrEDfYAfAAHaAavtjv8K+/Ec8Pgl1m3i+DvzBeo4/v487d6HfPFVA+Ict4OAXY6Y3epzucx+uN3n4nhvAx23vYHVjvf6S2CoY/n7wH2O2/cD7ztuDwVmlEQc5eVIoQ2wxxizzxiTCUwH+tscU0noD3zmuP0ZcLONsRTIGLMEOHXR4oJi7w9MNpZVQIiIVMMFFLAfBekPTDfGZBhj9gN7sP4OXYIxJskYs95xOxXYDtTAzT6Xy+xHQVz2c3G8t2cdd70dFwN0B2Y6ll/8mVz4rGYCPURErjaO8pIUagCH891P4PJ/OK7IAPNFZJ2IjHEsq2qMSQLrnwOoYlt0RVdQ7O74WT3oOKXySb5TeG6zH47TDi2wfpm67edy0X6AG34uIuIpIhuB48ACrCOZ08aYbMcq+ePN2xfH4ylA2NXGUF6SwqWyp7t1u+pojIkDegMPiEhnuwNyEnf7rN4D6gGxQBLwhmO5W+yHiAQCs4BHjDFnLrfqJZa5zP5cYj/c8nMxxuQYY2KBSKwjmMaXWs1x7ZR9KS9JIQGome9+JJBoUyzFYoxJdFwfB2Zj/cEcu3AI77g+bl+ERVZQ7G71WRljjjn+kXOBD/n9VITL74eIeGN9kU41xvzPsdjtPpdL7Yc7fy4AxpjTwCKsNoUQEbkw903+ePP2xfF4MIU/vVmg8pIU1gANHK34PliNMt/YHFOhiUgFEQm6cBvoCWzB2oc7HavdCXxtT4TFUlDs3wAjHL1d2gEpF05nuKKLzqvfgvW5gLUfQx09ROoADYBfSzu+gjjOPX8MbDfGjMv3kFt9LgXthzt+LiJSWURCHLf9geuw2kgWAoMcq138mVz4rAYBvxhHq/NVsbvFvbQuWL0ndmGdo/un3fEUMfa6WD0mNgFbL8SPdf7wZ2C34zrU7lgLiH8a1iF8Ftavm7sKih3rkPhdx+e0GWhld/xX2I8pjjjjHf+k1fKt/0/HfuwEetsd/0X70gnrVEM8sNFx6eNun8tl9sPtPhcgBtjgiHkL8IxjeV2sxLUH+ArwdSz3c9zf43i8bknEoSOalVJK5Skvp4+UUkoVgiYFpZRSeTQpKKWUyqNJQSmlVB5NCkoppfJoUlAuR0RyHJUtN4nIehHpcIX1Q0Tk/kJsd5GIlIm5ekuKiEwSkUFXXlOVF5oUlCs6b4yJNcY0B/4BvHyF9UOwKka6pHyjUZVyeZoUlKurCPwGVn0bEfnZcfSwWUQuVLp9BajnOLp43bHu3x3rbBKRV/Jt71ZHzfpdInKtY11PEXldRNY4Cqjd61heTUSWOLa75cL6+Yk1z8Wrjm3+KiL1Hcsnicg4EVkIvCrWPAVzHNtfJSIx+fbpU0es8SIy0LG8p4isdOzrV47aPojIKyKyzbHufxzLbnXEt0lEllxhn0RE3nFs4zvcq4iiKg12j+LTi14uvgA5WCNTd2BVfmzpWO4FVHTcDscaySlAFH+c46A3sAIIcNy/MCp3EfCG43Yf4CfH7THAvxy3fYG1WLX2/8rvo8c9gaBLxHog3zojgLmO25OAuThq9QNvA886bncHNjpuvwqMz7e9So59WwJUcCx7AngGCMUahXth0GmI43ozUOOiZQXt0wCs6pueQHXgNDDI7s9cL65z0cNa5YrOG6tSJCLSHpgsItFYCeD/HBVic7FKB1e9xPOvAz41xqQBGGPyFwm7UPhtHVYyAauWVEy+c+vBWDVx1gCfOAquzTHGbCwg3mn5rt/Mt/wrY0yO43YnYKAjnl9EJExEgh2xDr3wBGPMbyJyE9ZkMMut0j74ACuBM0A68JHjV/5cx9OWA5NE5Mt8+1fQPnUGpjniShSRXwrYJ1VOaVJQLs0Ys1JEwoHKWL/uK2MdOWSJyAGs+i8XEwouIZzhuM7h979/AR4yxsz704asBHQjMEVEXjfGTL5UmAXcPndRTJd63qViFWCBMea2S8TTBuiBlUgeBLobY8aKSFtHnBtFJLagfRJrWkqtbaMKpG0KyqWJSCOsUx3JWL92jzsSQjegtmO1VKypGC+YD4wWkQDHNkKv8DLzgPscRwSIyDViVaat7Xi9D7EqcRY0L/GQfNcrC1hnCTDMsf2uwElj1f2fj/XlfmF/KwGrgI752icCHDEFAsHGmO+BR7DmCkBE6hljVhtjngFOYpVTvuQ+OeIY6mhzqAZ0u8J7o8oZPVJQrshfrNmnwPrFe6cxJkdEpgLfishafm9zwBiTLCLLRWQL8IMx5m+OX8trRSQT+B546jKv9xHWqaT1Yp2vOYE15WFX4G8ikgWcxWozuBRfEVmN9SPrT7/uHZ4DPhWReCCN30sevwi864g9B3jeGPM/ERkJTBMRX8d6/8JKfl+LiJ/jfXnU8djrItLAsexnrGq68QXs02ysNo3NWFWDF1/mfVHlkFZJVeoqOE5htTLGnLQ7FqVKgp4+UkoplUePFJRSSuXRIwWllFJ5NCkopZTKo0lBKaVUHk0KSiml8mhSUEoplUeTglJKqTz/D7uKEwHBw+LfAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_losses()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding a little regularisation using weight decay seems to help; we get a 59%.\n",
    "\n",
    "Maybe dropout could help more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:09 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.683858</th>\n",
       "    <th>2.500692</th>\n",
       "    <th>0.277083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.455149</th>\n",
       "    <th>2.113877</th>\n",
       "    <th>0.352083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.241700</th>\n",
       "    <th>1.903266</th>\n",
       "    <th>0.406250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.017066</th>\n",
       "    <th>1.779788</th>\n",
       "    <th>0.454167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>1.849892</th>\n",
       "    <th>1.776859</th>\n",
       "    <th>0.493750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>1.691735</th>\n",
       "    <th>1.798783</th>\n",
       "    <th>0.479167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>1.523797</th>\n",
       "    <th>1.714493</th>\n",
       "    <th>0.520833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.337385</th>\n",
       "    <th>1.671328</th>\n",
       "    <th>0.541667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.208459</th>\n",
       "    <th>1.819937</th>\n",
       "    <th>0.535417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.074747</th>\n",
       "    <th>1.773041</th>\n",
       "    <th>0.535417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>0.935993</th>\n",
       "    <th>1.755780</th>\n",
       "    <th>0.568750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>0.815807</th>\n",
       "    <th>1.749104</th>\n",
       "    <th>0.562500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>0.713215</th>\n",
       "    <th>1.812801</th>\n",
       "    <th>0.566667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>0.620390</th>\n",
       "    <th>1.839610</th>\n",
       "    <th>0.575000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>0.550001</th>\n",
       "    <th>1.766414</th>\n",
       "    <th>0.591667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>0.485581</th>\n",
       "    <th>1.801695</th>\n",
       "    <th>0.597917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>0.429470</th>\n",
       "    <th>1.961521</th>\n",
       "    <th>0.587500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>0.386420</th>\n",
       "    <th>1.903309</th>\n",
       "    <th>0.597917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>0.346661</th>\n",
       "    <th>1.830627</th>\n",
       "    <th>0.593750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>0.319924</th>\n",
       "    <th>1.856341</th>\n",
       "    <th>0.593750</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = Model(n_letters, n_hidden, n_output, bn=True).cuda()\n",
    "learn = Learner(data, model, loss_func=F.cross_entropy, metrics=[accuracy])\n",
    "learn.fit_one_cycle(20, 1e-2, wd=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## fastai Builtin\n",
    "How does fastai's built in learner compare?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How long are the names?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    16869.000000\n",
       "mean         7.409509\n",
       "std          2.050366\n",
       "min          2.000000\n",
       "25%          6.000000\n",
       "50%          7.000000\n",
       "75%          9.000000\n",
       "max         18.000000\n",
       "Name: ascii_name, dtype: float64"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.ascii_name.str.len().describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = text_classifier_learner(data, bptt=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl4VOXd//H3d7ISEkIgIUAChH0RZQso4E5dW9eqVbu4tWq1VuvS2r1P+7NPW2sXH9taW1yqqHXfahVsXUBFCGHfl0DYSQgkISH7/ftjBo2YkABzcmYmn9d1zcXMOfeZ+d5M4JNz7nPObc45REREwi3gdwEiIhKbFDAiIuIJBYyIiHhCASMiIp5QwIiIiCcUMCIi4gkFjIiIeEIBIyIinlDAiIiIJ+L9LiCcMjMzXV5ent9liIhEjQULFpQ657K8eO+YCpi8vDwKCgr8LkNEJGqY2Sav3luHyERExBMKGBER8YQCRkREPKGAERERTyhgRETEEwoYERHxhAJGREQ8EVPXwXRW9Y1N/HfVLnZW1HDS0CwGZnb1uyQREQVMNFu7s5JnCjbz4sKtlO6r+3j5wMyunDa8FycNy+SYvt3ISk3CzHysVEQ6IwVMlGlscsxasZPpczYwf+Me4gPGtJG9+NLEfgzKTOXdNSX8d9UunvhoEw+/XwRAj66JDM9OY2h2Kt1TEumWHE+35AS6dUmgV7ckendLJistiYQ4HTEVkfBRwESIuoYm9tU2UFPfGHo0EQhAQlyAxLgAcQHjrZU7eXhOERt3V5PTvQs/OHcEF4/PJTM16eP3ycvsylVT8qiua2DR5r2s3lHJ6h2VrNxRyYsLt1JZ09Di55tBr7QkjsvtzuRBPZk8uCfDs9MIBD6959PU5NhQuo/CTXtZuHkPJZV1NDQ10dDoaGhqIj4QIDUpntTkeFKT4umWHE9G10R6dE0kIyWR3unJDMlK/cz7ikjsUcD4bHNZNX+fvYF/Fmympr6pzfZj+3XnT2eN4Kxjsok/xB5HSmI8UwZnMmVw5qeWNzY59tU2UFlTz97qenZV1rCjvJYdFTVs2VPN/I1lzFqxE4DuKQlkpSYRHxcgIc4ImFFUWkX5/noA0rskkNO9CwlxRlzAiI8LUF3XQEll7cefUVnbgHOfrq1bcjz5eT2YNLAHE/N6cEzfbiQnxB3m35yIRDoFjE+WbS3nwXfX8/rS7cQFjAvG5nBsTjpJ8QGSE+JITgjgHNQ1NlHb0ERdQxMj+6Qxvn/GUY2nxAWM9C4JpHdJIDcDIP0zbbbsqWbuhjLmF5VRUVNPfWjvpLHJcc7o3ozvn8H4ARkMyuza5p5IY5OjYn89ZdV17Kmqo7gsGGIfFZXx31W7AIgPGCP7dGNMv3TG9svgtOFZ9Gy2VyYi0cncwb9eRrH8/HwX6XdTbmxy/PGtNdz/33WkJcVz5Qn9uWbKQHqnJ/tdWocrqaxlwaYyFm8pZ/HmvSzZUs6+2gbiAsaUwT35/LF9OOuY3mR0TfS7VJGYZWYLnHP5nry3VwFjZv2AfwC9gSbgIefcHw9qkw48AfQnuDf1W+fcI6F1jcDSUNNi59z5bX1mpAdMeXU9t/5zIe+sLuGSCbn85LxRdEtO8LusiNHU5FixvYJ/L9vOa0u2s2l3NQGDrLQkstKS6JWWTFZqErkZXejfM4UBPbuS1zOF7ikKIJEjFa0B0wfo45wrNLM0YAFwoXNuRbM2PwDSnXPfM7MsYDXQ2zlXZ2b7nHOph/OZkRwwK7dXcMPjC9hevp+fnncMXz6+v04dPgTnHMu3VfDWyp1s3bOfkn21lFTWsqsy+GdzQ3ulcsHYvlwwNod+PVJ8qlgkOnkZMJ6NwTjntgPbQ88rzWwlkAOsaN4MSLPg/7SpQBnQ8mlOUWLr3v1Mn13ErJU7qG9wOBxNDvZW19GjayJPXz+ZCQMy/C4z4pkZo3PSGZ3z2TGi/XWNbN5TzcbSKopKq3hr5U5+O3MNv525hvH9u3Pa8F4ck9ONUX3Sye6ma4BE/NIhYzBmlge8B4x2zlU0W54GvAKMANKALznn/hVa1wAsIhg4v3LOvdTKe18PXA/Qv3//CZs2eTY52yGt3lHJX99dzyuLtwFw2oheZKQkYBiBAHRNjOf6UwbRK63zjbV0hC17qnl18XZeWbyNlds//hGjR9dExvfvzhmjsvncyGydPCBykKg8RPbxB5ilAu8C9zjnXjho3SXAVOB2YDAwCxjjnKsws77OuW1mNgj4LzDNObf+UJ/l9SGy2oZGlm4pZ97GMtbt3PfxYZvSfXWU7qulS0Icl0/qx9dPGkRO9y6e1SGHVllTz6odlazYVsHybeW8v243W/fuJ2CQP6AHp43oxXG56Yzum056isbApHOLykNkAGaWADwPzDg4XEKuIbh34oB1ZlZEcG9mnnNuG4BzboOZvQOMAw4ZMF55dfE2npi7iUWb91LbELxWpW96MlndksnNSGFc/+4Mykzlkgm5OuMpAqQlJzAxL3iNDXwynjNzxU5mLt/Br99Y9XHb/j1SyM/L4KZThzCk12EN+YlIG7wc5DfgMaDMOXdbK23+Aux0zv3MzLKBQmAM0AhUO+dqzSwT+BC4oPkJAi050j2YpibX6vUcLy/aym3/XMTgrFROGZYV+o8rQ4daolhZVR3LtpazdGs5y7aW896aEvbXN/Klif24ddqwTnnKuHReUXmIzMxOBGYTPNX4wCXqPyB4SjLOuQfNrC/wKNAHMIJ7M0+Y2RTgr6HtAsAfnHPT2/rMIwmYxibHFQ/N5eRhmVx/8mAS4z+5Ov4/K3dy/eMLmJiXwaPXTNLV5jFq975aHnh7HU/M3URcwLhm6kC+fuJA/RIhnUJUBowfjiRgKmvqufv5pfxr6XaG9krlfy8+lvy8HszdsJurHp7H8N5pPPmNE0hN0k0PYl3x7mp+N2s1Ly3aRnJCgMsn9ucbJ2s8TWKbAqadjmaQ/z8rd/KTl5ezde9+LhqXw6wVO+mTnsw/b5hMD42rdCrrdlXy4LsbeGnhVgAuGJvDj78wUhd0SkxSwLTT0Z5FVlXbwO9nreHh94vo270Lz904RcfjO7Gte/fz99kbmDG3mOG903jiuuN11pnEHAVMO4XrNOWi0iq6JcfrGLwA8PaqXdzw+AJG9Enj8euOJ72LQkZih5cBoxmmWjAws6vCRT522ohe/OUr41m5vYKvPTyPipp6v0sSiQoKGJF2mDYymz9/eQIrtpXztenz2FxW7XdJIhFPASPSTmeMyuaBK8ezfFs5J/3mbS578EOemldMebX2aERaojEYkcO0de9+Xlq4lRcKt7C+pIrEuAC/ueQ4LhyX43dpIodNg/ztpICRjuScY+nWcn7y8nKKSqv47x2naOxOoo4G+UUikJlxXG537r3kOKpqG/jNG6v9LkkkoihgRI7S0Ow0rjtxIP8s2Exh8R6/yxGJGAoYkTC4ZdpQendL5scvLaOxKXYOO4scDQWMSBikJsXzoy+MZPm2Cp78yJ9J70QijQJGJEw+f2wfpg7pyb1vrqZ0X63f5Yj4TgEjEiZmxv+cP5r99Y18/4WlOlQmnZ4CRiSMhvRK5fvnjGTWip388vWVfpcj4itNciISZteeOJDismqmzymiX0YXrp460O+SRHyhgBHxwI+/MIote/bz89dWkJORwhmjsv0uSaTD6RCZiAfiAsb9V4xldE46335qIUu27PW7JJEOp4AR8UhKYjx/vyqfHl0Tuf4fCyirqvO7JJEOpYAR8VCvtGT++tUJlFXVcfszi2jSmWXSiShgRDw2OiedH39hJO+sLuHB99b7XY5Ih1HAiHSAr5wwgM8f14f7Zq5hXlGZ3+WIdAgFjEgHMDN+dfGx9Mvowi1PFbJbV/pLJ6CAEekgackJ/OnL49lTXc9NMwqprNFMmBLbFDAiHeiYvunce8lxFGzaw2V/ncvOihq/SxLxjAJGpINdMDaHh6+eSPHuKi760/us3lHpd0kinlDAiPjglGFZPHPjZBqaHJf85QPeX1fqd0kiYaeAEfHJMX3TefHmqfTpnsxVD8/jmfmb/S5JJKwUMCI+yunehee+OYXJg3vy3eeX8Kt/r9LFmBIzFDAiPuuWnMDDV0/kyuP78+C767lpRiH76xr9LkvkqClgRCJAQlyAey4czY8+P5I3V+zg8oc+ZG+17l0m0U0BIxIhzIyvnzSIh76az8rtlVz1yHxdKyNRTQEjEmHOGJXNn748nuVby7nu0QIdLpOopYARiUBnjMrm918aS8GmMq5/vICaeoWMRB8FjEiEOm9MX379xeOYvbaUbz25kEadXSZRRgEjEsEuze/Hj78wirdW7mTWih1+lyNyWBQwIhHu6il59E1PZsZHxX6XInJYFDAiES4uYFw+qT+z15ayaXeV3+WItJtnAWNm/czsbTNbaWbLzezWFtqkm9mrZrY41OaaZuuuMrO1ocdVXtUpEg2+NLEfcQHjyXnai5Ho4eUeTANwh3NuJHACcLOZjTqozc3ACufcGOBU4D4zSzSzHsBPgeOBScBPzSzDw1pFIlp2t2Q+N7IXzxZsobZBZ5RJdPAsYJxz251zhaHnlcBKIOfgZkCamRmQCpQRDKazgFnOuTLn3B5gFnC2V7WKRIMvHz+Asqo63ly+0+9SRNqlQ8ZgzCwPGAd8dNCqB4CRwDZgKXCrc66JYBA1v7XsFj4bTgfe+3ozKzCzgpKSkjBXLhI5ThySSf8eKcyYu8nvUkTaxfOAMbNU4HngNudcxUGrzwIWAX2BscADZtYNsBbeqsWLAJxzDznn8p1z+VlZWWGsXCSyBALGFZP681FRGet2aZIyiXyeBoyZJRAMlxnOuRdaaHIN8IILWgcUASMI7rH0a9Yul+Bejkindml+LglxplOWJSp4eRaZAdOBlc6537XSrBiYFmqfDQwHNgBvAmeaWUZocP/M0DKRTi0zNYmzR/fh+QVbdPsYiXhe7sFMBb4KnG5mi0KPc83sRjO7MdTmF8AUM1sK/Af4nnOu1DlXFlo3P/T4eWiZSKd35aT+VNQ08O9l2/0uReSQ4r16Y+fcHFoeS2neZhvBvZOW1j0MPOxBaSJR7fiBPcjp3oWXFm7jonG5fpcj0ipdyS8SZQIB4/yxfZmzrpTSfbV+lyPSKgWMSBS6cGwOjU2O1xbr3BeJXAoYkSg0vHcaI3qn8dIiBYxELgWMSJS6cFwOizbvZWOpboApkUkBIxKlzh/TFzN4WXsxEqEUMCJRqm/3LkzK68HLi7binGa7lMijgBGJYheOy2FDaRVLt5b7XYrIZyhgRKLYuaP7kBgX4KWFOkwmkUcBIxLF0lMSOHV4Fq8u2UZjkw6TSWRRwIhEuQvH5VBSWcs7q3f5XYrIpyhgRKLc6SN6MTCzK997fik7ymv8LkfkYwoYkSiXnBDHX786geq6Br45Y4GmVJaIoYARiQHDstO479IxLCzey/+8usLvckQABYxIzDjn2D7cdOpgnvyomKfmaUIy8Z8CRiSG3HHmcE4elsVPX17OwuI9fpcjnZwCRiSGxAWM+y8fS1ZaErc/s1izXoqvFDAiMaZ7SiL3XnIcRaVV/PbN1X6XI52YAkYkBk0ZkslXTujP9PeLWLBJs42LPxQwIjHq7nNG0je9C3c9u0SHysQXChiRGJWaFM9vLjmODaVV3DdTh8qk4ylgRGLY1CGZfPn4/vx9jg6VScdTwIjEuO+fO5Kc7l245cmFlFTW+l2OdCIKGJEYl5oUz4NfmcCe6npueLxAt5KRDqOAEekERuekc99lYygs3ssPX1ymGTClQyhgRDqJc4/tw63ThvLcgi1Mn1PkdznSCShgRDqRW6cN5ZzRvfnl6yt5W/PHiMcUMCKdSCBg3HfZGIZlp3HXs4sp31/vd0kSwxQwIp1MSmI8v710DGVVdbqVjHiqXQFjZoPNLCn0/FQz+7aZdfe2NBHxyuicdL42OY8nPtrE4s17/S5HYlR792CeBxrNbAgwHRgIPOlZVSLiudvPHEZmahI/emkZjU06q0zCr70B0+ScawAuAv7gnPsO0Me7skTEa92SE/jR50eydGs5Mz7a5Hc5EoPaGzD1ZnYFcBXwWmhZgjcliUhHOX9MX6YO6cm9b6xmV2WN3+VIjGlvwFwDTAbucc4VmdlA4AnvyhKRjmBm/OKC0dQ2NPHLf630uxyJMe0KGOfcCufct51zT5lZBpDmnPuVx7WJSAcYlJXKN04eyEuLtrF0S7nf5UgMae9ZZO+YWTcz6wEsBh4xs995W5qIdJQbThlMRkoCv35jld+lSAxp7yGydOdcBXAx8IhzbgLwOe/KEpGO1C05gZtPG8KcdaXMWVvqdzkSI9obMPFm1ge4jE8G+UUkhnzlhAHkdO/Cr99YRZNOW5YwaG/A/Bx4E1jvnJtvZoOAtd6VJSIdLTkhju+cMYylW8v519LtfpcjMaC9g/zPOueOc859M/R6g3Pui4faxsz6mdnbZrbSzJab2a0ttLnLzBaFHsvMrDE0zoOZbTSzpaF1BUfSORE5PBeNy2F4dhr3zVxNfWOT3+VIlGvvIH+umb1oZrvMbKeZPW9muW1s1gDc4ZwbCZwA3Gxmo5o3cM7d65wb65wbC3wfeNc513xe19NC6/MPo08icoTiAsZ3zx7Oxt3VPD1/s9/lSJRr7yGyR4BXgL5ADvBqaFmrnHPbnXOFoeeVwMrQtq25AniqnfWIiEdOH9GLiXkZ/PGtteyv0+yXcuTaGzBZzrlHnHMNocejQFZ7P8TM8oBxwEetrE8BziZ4z7MDHDDTzBaY2fWHeO/rzazAzApKSkraW5KItMLMuPPM4ZTuq+XJecV+lyNRrL0BU2pmXzGzuNDjK8Du9mxoZqkEg+O20KnOLTkPeP+gw2NTnXPjgXMIHl47uaUNnXMPOefynXP5WVntzjwROYTjB/XkhEE9ePDd9dTUay9Gjkx7A+Zagqco7wC2A5cQvH3MIZlZAsFwmeGce+EQTS/noMNjzrltoT93AS8Ck9pZq4iEwa3ThlFSWctT2ouRI9Tes8iKnXPnO+eynHO9nHMXErzoslVmZgRv7b/SOdfqVf9mlg6cArzcbFlXM0s78Bw4E1jWnlpFJDwmD+7JpIHai5EjdzQzWt7exvqpwFeB05udinyumd1oZjc2a3cRMNM5V9VsWTYwx8wWA/OAfznn3jiKWkXkCNw2bSg7K2r5p84okyMQfxTb2qFWOufmtNUm1O5R4NGDlm0AxhxFbSISBpMH92RiXgZ/eWc9l0/qR1J8nN8lSRQ5mj0Y3UtCJMaZGbdOG8aOihqe0V6MHKZDBoyZVZpZRQuPSoLXxIhIjJs6pCcTBmTwp7fXs7ms2u9yJIocMmCcc2nOuW4tPNKcc0dzeE1EooSZ8YNzR1JV28DZf3iPZwo245wOYEjbjuYQmYh0EhMGZPDv207i2Nx0vvvcEm54fAG799X6XZZEOAWMiLRLbkYKT379BH70+ZG8s7qEs/4wm+3l+/0uSyKYAkZE2i0QML5+0iBevHkKe6vrePCd9X6XJBFMASMih+2Yvul8cXwuT83fzK7KGr/LkQilgBGRI/LNUwfT0NjE9NlFfpciEUoBIyJHJC+zK+eN6cvjczexp6rO73IkAilgROSI3XzaEKrrGnnkg41+lyIRSAEjIkdsWHYaZx2TzaPvF1FRU+93ORJhFDAiclS+ddpQKmoaePzDTX6XIhFGASMiR+XY3HROHZ7F9DlFVNc1+F2ORBAFjIgctVtOH0JZVR2PaixGmlHAiMhRmzCgB58bmc1f3l6vW8jIxxQwIhIWd58zgur6Ru7/z1q/S5EIoYARkbAY0iuVKyb1Y8ZHxWwo2fepdU1NjucWbGFjaVUrW0ssUsCISNjcOm0YSfEBfv3Gqo+X1TY08p1nFnHns4u57rH51NQ3+lihdCQFjIiETVZaEt88dTBvLt/JvKIyyqvr+dr0eby8aBsXj89hfUkV97652u8ypYNo0jARCavrThzEE3OL+dkry6lvbGLj7ir+8KWxXDguh9SkeB5+v4gzRmVzwqCefpcqHtMejIiEVZfEOO44cxgrtlewo6KGx66dxIXjcoDgiQADeqRw57OL2Vera2ZinQJGRMLu4vG5/OjzI3nxpilMGZz58fKUxHjuu2wM2/bu555/rfCxQukIChgRCbu40MRkQ3qlfWbdhAE9uP7kwTw1bzNvr9rlQ3XSURQwItLhvnPGUAZlduUPb63xu5SoV7CxjNeXbqepyfldymcoYESkwyXFx3Hl8f1ZvKWcNTsr/S4nqj0+dxP/8+pyzPyu5LMUMCLii4vG5RAfMJ5bsMXvUqJaYfEexvfPwCIwYRQwIuKLnqlJnD6iFy8UbqW+scnvcqLSrooaNpftZ8KADL9LaZECRkR8c8mEXEr31fLemhK/S4lKhcV7ABjXXwEjIvIpp43oRWZqIs8W6DDZkSgs3ktiXIDROd38LqVFChgR8U1CXIALx+bwn1U7Kauq87ucqLNg0x5G53QjKT7O71JapIAREV9dkp9LfaPj5UVb/S4lqtQ2NLJ0a3nEjr+AAkZEfDaidzeOzUnXYbLDtHxbBXUNTYyP0PEXUMCISAS4ND+XFdsrWL6t3O9SokbhpuAA/3jtwYiItO78MX1JjAvw9LzNfpcSNQqL95Cb0YXsbsl+l9IqBYyI+K57SiIXj8/h8bmbeHGhDpW1xTnHgk17IvrwGGg+GBGJED87/xiKy6q589kldEtOYNrIbL9LiljbymvYWVEb0QP8oD0YEYkQyQlxPPS1fI7p242bZhTy0YbdfpcUsRYcGH+J8D0YBYyIRIzUpHgevWYSuRld+PpjBSzbqkH/lhRu2kOXhDhG9PnsdAiRxLOAMbN+Zva2ma00s+VmdmsLbe4ys0WhxzIzazSzHqF1Z5vZajNbZ2Z3e1WniESWHl0Tefy640lLjufqR+azvXy/3yVFnMLiPRyXm05CXGTvI3hZXQNwh3NuJHACcLOZjWrewDl3r3NurHNuLPB94F3nXJmZxQF/As4BRgFXHLytiMSuvt278Ni1k6ipb+Qb/yhgf12j3yVFjP11jazYVhHx4y/gYcA457Y75wpDzyuBlUDOITa5Angq9HwSsM45t8E5Vwc8DVzgVa0iEnmGZqdx/xVjWb6tgrueW4xzkTehlh+WbNlLQ5Pr3AHTnJnlAeOAj1pZnwKcDTwfWpQDND8hfguthJOZXW9mBWZWUFKiO7KKxJLTR2TzvbNH8NqS7Tzw33V+lxMRCov3ApF7B+XmPA8YM0slGBy3OecqWml2HvC+c67swGYttGnx1xfn3EPOuXznXH5WVtbRFywiEeWGkwdx0bgc7pu1hjeW7fC7HN8VbCxjYGZXenRN9LuUNnkaMGaWQDBcZjjnXjhE08v55PAYBPdY+jV7nQtsC3+FIhLpzIz/vfhYxvTrzp3PLqZ8f73fJfmmvrGJuRt2M2VwT79LaRcvzyIzYDqw0jn3u0O0SwdOAV5utng+MNTMBppZIsEAesWrWkUksiUnxHHPhaPZV9vA0/OK/S7HNwuL91JV18hJQ6PjaI2XezBTga8Cpzc7FflcM7vRzG5s1u4iYKZzrurAAudcA/At4E2CJwc845xb7mGtIhLhRuekM3lQTx77YGOnnWJ59toS4gLG5CjZg/HsVjHOuTm0PJZycLtHgUdbWP468HrYCxORqHXdiQP5+j8K+PeyHZw/pq/f5XS499aWMrZfd9K7JPhdSrtE9lU6IiLNnD6iFwMzuzJ99oZOd9ry3uo6lmzZy0lDM/0upd0UMCISNQIB49qpeSzeUv7x/bg6i/fX7cY5omb8BRQwIhJlvjghl/QuCUyfU+R3KR1q9toS0pLjGZOb7ncp7aaAEZGokpIYz5XH9+fN5TvYXFbtdzkdwjnH7LWlTB2cSXyE33+sueipVEQk5KrJeQTMeOT9jX6X0iE2lFaxde9+ThoWPeMvoIARkSjUOz2Z88b05ZmCzWzdG/t3W569JngbrJOjaPwFFDAiEqVuOX0IZnDtI/OpqIntq/tnry0lr2cK/Xqk+F3KYVHAiEhUGpSVyoNfmcD6kn3cPKMwZi++rGto4sMNu6Pq7LEDFDAiErWmDsnklxcfy+y1pfz4pWUxeW1MYfEequsao+r6lwM8u5JfRKQjXJbfj+Ld1Tzw9jr690zhplOH+F1SWM1ZWxpVt4dpTnswIhL17jhzGOeP6ctv3lhNwcaytjeIElv2VPPiwq2M79+dtOTouD1McwoYEYl6Zsavv3gcmalJ/G7WGs8+Z9aKnTz47voOGe9Zsa2Ci//8ARU19dx9zkjPP88LChgRiQldEuP45qmD+WD9buZu2B3W965vbOL/vbaCb/yjgF/9exWXPvihpxd5frC+lC/99UMCZjx345SomB65JQoYEYkZXz6+P73SgnsxBw/4l+6r5dpH5/P2ql2H9Z67Kmq48m9z+fucIq6aPID7rxjH+pJ9nHv/bP61ZHs4ywfg9aXbufrh+fROT+aFm6YwvHda2D+jo2iQX0RiRnJCHDefNoSfvrKcD9fvZsqQ4JlXNfWNXP+PAgqL9zJ/Yxmvf/ukFq8peWf1LpZsKScuYMQHjCYH0+cUUVXbwB8vH8sFY3MAGNevO7c8tZCbnyzk/fX9+ckXRpGcEHfU9Tc1Ob733BJG9u3GY9dMpHtK5E+LfCgKGBGJKV+a2I+/vLOe381a8/GZV999bgmFxXv54bkjuf8/a/n20wt55obJJDS7r9cz8zfz3eeXfOb9Bmd15clvHM+w7E/2JPr1SOHZGydz38w1PPjuehZs3MP/XTnuU22OxM7KGiprG7hkQm7UhwsoYEQkxiQnxHHz6UP48UvLmL22lAWb9vDK4m189+zhfOPkQfROT+aWpxZy38w13H3OCABeXLiF772whJOGZvLXr04gLmA0NjkamhypifEEAp+dOzEhLsDd54xgyuCe3P7MIs5/YA4/Pe8YLp/Yj+CM8YevqCQ4se+gzK5H/hcQQRQwIhJzLsvP5cF31nPXc4vZWVHLpRNy+eYpgwE4b0xfPlhfyoPvrmfy4J5U1tRzxzOLOWFgT/72tfzDPtR18rACjhSBAAANNklEQVQsXr/1JG7/52K+/8JSPli/m99fNuaI7nq8oTQUMFmxETAa5BeRmJMUH8e3Th/Czopajh/Yg3suOvZTexU/+cIxDMtO5dtPLeTWpxcxYUAG068+/HA5oFdaMv+4dhLfnjaUVxdv483lO4/ofTaUVNElIY7stOQj2j7SKGBEJCZdOiGX3102hoe+lk9i/Kf/q+uSGMefrhxPXUMTx+ak8/DVE0lJPLoDOoGAceu0ofTvkcL0ORuO6D2KSveRl9m1xUNy0UgBIyIxKT4uwMXjg7NftmRodhrv3nUq/7zhhLBdJR8XmtK5sHgvhcWHP6VzUWlVzIy/gAJGRDqxXt2SSYo/+tOLm7s0vx9pyfGHPaVzXUMTm/fsj5nxF1DAiIiEVdekeK6c1J83lu1gy572X+1fXFZNY5NjoPZgRESkNVdNyQPgsQ82tnubotAZZAoYERFpVd/uXTj32D48PW8z+2ob2rVNUek+AAZlpnpZWodSwIiIeOC6EwdSWdvAM/M3t6t9UWkVPbsmkp4Sfbflb40CRkTEA2P7dWfCgAwe+aCIxqa2Z9rcUFIVU4fHQAEjIuKZr584kM1l+3ltybY2224oVcCIiEg7nXVMb4Znp/HHt9bScIhJyipr6imprGVQVuyMv4ACRkTEM4GAcfuZw9hQWsWLC7e22m5jafB0Zu3BiIhIu505Kptjc9L543/WUtfQ8l7MhgNnkMXQRZaggBER8ZSZcceZw9iyZz/PFLR8RtmGkirMoH8Lk6BFMwWMiIjHThmWRf6ADP7vv2upqW/8zPqi0ipyM7qEZVbMSKKAERHxWHAvZjg7K2qZ8VHxZ9YXlVYxMIYusDxAASMi0gEmD+7J1CE9+cs766hqdnW/cy7m7qJ8gAJGRKSD3HHmcEr31fHA2+s+XlZSWcu+2oaYG+AHBYyISIcZ3z+DSybk8rf3NrByewXwyTTJsXaKMihgREQ61A/PHUl6lwTufmEpjU0uJu+ifIBnAWNm/czsbTNbaWbLzezWVtqdamaLQm3ebbZ8o5ktDa0r8KpOEZGOlNE1kZ+cN4rFm/fy+IcbKSqtIik+QN/0Ln6XFnZHNwn1oTUAdzjnCs0sDVhgZrOccysONDCz7sCfgbOdc8Vm1uug9zjNOVfqYY0iIh3u/DF9eb5wK/e+uZrBvVIZmNmVQMD8LivsPNuDcc5td84Vhp5XAiuBnIOaXQm84JwrDrXb5VU9IiKRwsy458LRNDlYsqU8Jg+PQQeNwZhZHjAO+OigVcOADDN7x8wWmNnXmq1zwMzQ8usP8d7Xm1mBmRWUlJSEu3QREU/065HC7WcMA2Jz/AW8PUQGgJmlAs8DtznnKlr4/AnANKAL8KGZzXXOrQGmOue2hQ6bzTKzVc659w5+f+fcQ8BDAPn5+W1PuiAiEiGumZrHnuo6Lhp38MGd2OBpwJhZAsFwmeGce6GFJluAUudcFVBlZu8BY4A1zrltEDxsZmYvApOAzwSMiEi0io8L8N2zR/hdhme8PIvMgOnASufc71pp9jJwkpnFm1kKcDyw0sy6hk4MwMy6AmcCy7yqVUREws/LPZipwFeBpWa2KLTsB0B/AOfcg865lWb2BrAEaAL+7pxbZmaDgBeDGUU88KRz7g0PaxURkTDzLGCcc3OANs+7c87dC9x70LINBA+ViYhIlNKV/CIi4gkFjIiIeEIBIyIinlDAiIiIJxQwIiLiCXMudi5+N7MSYNNBi9OB8jaWNX/d1vNM4GhuwNlSPe1tc7h9Ofj1geex1Jfmz4+mP0fTl9bW6efsk2X6btpXa1ttvPhuhjvn0tou+wg452L6ATzU1rLmr9t6DhSEu572tjncvhyiDzHTl3D152j6op+zQ/+c6buJ3e+mrUdnOET2ajuWvXqYz8NdT3vbHG5fDn79aittjlQk9KW9dbTlaPrS2jr9nIWHvptDL/fzuzmkmDpE1hHMrMA5l+93HeEQS32B2OpPLPUFYqs/sdQX8LY/nWEPJtwe8ruAMIqlvkBs9SeW+gKx1Z9Y6gt42B/twYiIiCe0ByMiIp7o1AFjZg+b2S4zO+ypAMxsgpktNbN1ZnZ/aHqCA+tuMbPVZrbczH4T3qpbrSfsfTGzn5nZVjNbFHqcG/7KW63Jk+8mtP5OM3Nmlhm+ig9ZjxffzS/MbEnoe5lpZn3DX3mL9XjRl3vNbFWoPy+aWffwV95qTV7059LQv/0mM/N8rOZo+tDK+11lZmtDj6uaLT/kv6sWeXV6WjQ8gJOB8cCyI9h2HjCZ4B2j/w2cE1p+GvAWkBR63SuK+/Iz4M5Y+W5C6/oBbxK8XiozWvsCdGvW5tvAg1HclzOB+NDzXwO/juafM2AkMBx4B8iP1D6E6ss7aFkPYEPoz4zQ84xD9fdQj069B+OCUzCXNV9mZoPN7A0zW2Bms83sM9PNmVkfgv/AP3TBv/l/ABeGVn8T+JVzrjb0Gbu87UWQR33xjYf9+T3wXaDDBh+96Iv79PTjXemg/njUl5nOuYZQ07lArre9+IRH/VnpnFvdEfWHPu+I+tCKs4BZzrky59weYBZw9pH+P9GpA6YVDwG3OOcmAHcCf26hTQ7B6Z4P2BJaBjCM4CydH5nZu2Y20dNqD+1o+wLwrdChi4fNLMO7UtvlqPpjZucDW51zi70utB2O+rsxs3vMbDPwZeAnHtbalnD8nB1wLcHfjv0Uzv74pT19aEkOsLnZ6wP9OqL+ejmjZdQxs1RgCvBss8OLSS01bWHZgd8g4wnuWp4ATASeMbNBodTvMGHqy1+AX4Re/wK4j+B/AB3uaPtjwSm5f0jwcIyvwvTd4Jz7IfBDM/s+8C3gp2EutU3h6kvovX4INAAzwlnj4Qhnf/xyqD6Y2TXAraFlQ4DXzawOKHLOXUTr/Tqi/ipgPi0A7HXOjW2+0MzigAWhl68Q/I+3+W58LrAt9HwL8EIoUOaZWRPBexeVeFl4C466L865nc22+xvwmpcFt+Fo+zMYGAgsDv2jywUKzWySc26Hx7UfLBw/Z809CfwLHwKGMPUlNJj8BWBaR/8ydpBwfzd+aLEPAM65R4BHAMzsHeBq59zGZk22AKc2e51LcKxmC0fSX68HoCL9AeTRbHAM+AC4NPTcgDGtbDef4F7KgQGvc0PLbwR+Hno+jODupkVpX/o0a/Md4Olo/m4OarORDhrk9+i7GdqszS3Ac1Hcl7OBFUBWR/58ef1zRgcN8h9pH2h9kL+I4FGYjNDzHu3pb4t1+fGFRsoDeArYDtQTTOjrCP6W+wawOPRD/5NWts0HlgHrgQf45KLVROCJ0LpC4PQo7svjwFJgCcHf2vp0RF+86s9BbTbScWeRefHdPB9avoTgfaVyorgv6wj+IrYo9OiQM+I87M9FofeqBXYCb0ZiH2ghYELLrw19J+uAa9rq76EeupJfREQ8obPIRETEEwoYERHxhAJGREQ8oYARERFPKGBERMQTChiJaWa2r4M/7+9mNipM79VowbslLzOzV9u6y7CZdTezm8Lx2SLhoNOUJaaZ2T7nXGoY3y/efXJjRk81r93MHgPWOOfuOUT7POA159zojqhPpC3ag5FOx8yyzOx5M5sfekwNLZ9kZh+Y2cLQn8NDy682s2fN7FVgppmdambvmNlzFpzHZMaBuTFCy/NDz/eFbki52Mzmmll2aPng0Ov5Zvbzdu5lfcgnN+1MNbP/mFmhBefnuCDU5lfA4NBez72htneFPmeJmf1PGP8aRdqkgJHO6I/A751zE4EvAn8PLV8FnOycG0fw7sS/bLbNZOAq59zpodfjgNuAUcAgYGoLn9MVmOucGwO8B3yj2ef/MfT5bd7PKXQfrGkE76YAUANc5JwbT3D+oftCAXc3sN45N9Y5d5eZnQkMBSYBY4EJZnZyW58nEi662aV0Rp8DRjW702w3M0sD0oHHzGwowTvFJjTbZpZzrvmcG/Occ1sAzGwRwXtBzTnoc+r45AahC4AzQs8n88lcGk8Cv22lzi7N3nsBwbk5IHgvqF+GwqKJ4J5Ndgvbnxl6LAy9TiUYOO+18nkiYaWAkc4oAEx2zu1vvtDM/g942zl3UWg8451mq6sOeo/aZs8bafnfUr37ZJCztTaHst85N9bM0gkG1c3A/QTnf8kCJjjn6s1sI5DcwvYG/K9z7q+H+bkiYaFDZNIZzSQ4fwoAZnbgtubpwNbQ86s9/Py5BA/NAVzeVmPnXDnBaZHvNLMEgnXuCoXLacCAUNNKIK3Zpm8C14bmB8HMcsysV5j6INImBYzEuhQz29LscTvB/6zzQwPfKwhOsQDwG+B/zex9IM7Dmm4DbjezeUAfoLytDZxzCwneGfdyghNy5ZtZAcG9mVWhNruB90OnNd/rnJtJ8BDch2a2FHiOTweQiKd0mrJIBwvNrrnfOefM7HLgCufcBW1tJxJtNAYj0vEmAA+Ezvzai0/TUIt4TXswIiLiCY3BiIiIJxQwIiLiCQWMiIh4QgEjIiKeUMCIiIgnFDAiIuKJ/w9RFOyb9qidfgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_end=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note we don't necessarily *expect* this to do great because the parameters are tuned to processing medium sized documents a word at a time.\n",
    "\n",
    "However it gets 67% way outperforms our RNN model without *any* parameter tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 01:07 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.760123</th>\n",
       "    <th>2.778821</th>\n",
       "    <th>0.062500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.630543</th>\n",
       "    <th>2.745926</th>\n",
       "    <th>0.075000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.513042</th>\n",
       "    <th>2.573858</th>\n",
       "    <th>0.160417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.355199</th>\n",
       "    <th>2.121467</th>\n",
       "    <th>0.300000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>2.195587</th>\n",
       "    <th>1.810203</th>\n",
       "    <th>0.387500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>2.012244</th>\n",
       "    <th>1.566222</th>\n",
       "    <th>0.472917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>1.909551</th>\n",
       "    <th>1.693872</th>\n",
       "    <th>0.460417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.752974</th>\n",
       "    <th>1.615144</th>\n",
       "    <th>0.527083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.620254</th>\n",
       "    <th>1.322627</th>\n",
       "    <th>0.581250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.416614</th>\n",
       "    <th>1.251798</th>\n",
       "    <th>0.625000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.226631</th>\n",
       "    <th>1.297575</th>\n",
       "    <th>0.610417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.056211</th>\n",
       "    <th>1.230383</th>\n",
       "    <th>0.641667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>0.922565</th>\n",
       "    <th>1.201090</th>\n",
       "    <th>0.664583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>0.791524</th>\n",
       "    <th>1.235106</th>\n",
       "    <th>0.656250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>0.657713</th>\n",
       "    <th>1.220895</th>\n",
       "    <th>0.683333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>0.552160</th>\n",
       "    <th>1.252036</th>\n",
       "    <th>0.666667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>0.460102</th>\n",
       "    <th>1.207947</th>\n",
       "    <th>0.666667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>0.411827</th>\n",
       "    <th>1.196497</th>\n",
       "    <th>0.670833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>0.370949</th>\n",
       "    <th>1.197413</th>\n",
       "    <th>0.668750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>0.330069</th>\n",
       "    <th>1.188975</th>\n",
       "    <th>0.677083</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(20, max_lr=7e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4VFX6wPHvmx7SQxI6hColBAihI0UQBaWKAjbAgqisa9td1P3ZVld2VRZxFRvFCqKIhRXBgiK9SQmEEnroIRBKIKSc3x/3EgIkISGZzEzyfp5nnrlz59w7782Feeeec+45YoxBKaWUAvBwdgBKKaVchyYFpZRSuTQpKKWUyqVJQSmlVC5NCkoppXJpUlBKKZVLk4JSSqlcmhSUUkrl0qSglFIql5ezAyiuiIgIEx0d7ewwlFLKraxevTrFGBN5pXJulxSio6NZtWqVs8NQSim3IiK7i1JOq4+UUkrl0qSglFIqlyYFpZRSudyuTUEpVX5kZmaSnJzM2bNnnR1KueHn50fNmjXx9va+qu01KSilnCY5OZmgoCCio6MREWeH4/aMMRw9epTk5GTq1q17VfvQ6iOllNOcPXuWypUra0IoJSJC5cqVS3TlpUlBKeVUmhBKV0n/nhWm+igtPZMDJ85QNdiPk2ez+HHTIVrXCSO2Zoj+o1RKKVuFSQqLklJ4+LM1l62vGebPkPha3N0xmhD/q2uYUUq5p6NHj9KjRw8ADh48iKenJ5GR1k2/K1aswMfH54r7GDlyJGPHjuWaa65xaKxlRYwxzo6hWOLj483V3NF8MO0sq3ancjDtLGczs+nVrCrrk9P4Zu0+ft+WQoi/N2O6N+CuDnXw8/Z0QORKqUslJibSpEkTZ4cBwPPPP09gYCBPPvnkReuNMRhj8PBwn9r2/P6uIrLaGBN/pW3d5yhLqGqIHzfHVue+a+sx5rqGNKoSxODWNfn43nb875HOtKwVysvfJ3Lda7+yYmeqs8NVSjlRUlISMTExjB49mri4OA4cOMCoUaOIj4+nWbNmvPjii7llO3fuzNq1a8nKyiI0NJSxY8fSokULOnTowOHDh514FFenwlQfFaZZ9RA+vKctS7an8PfZCdzxwTIe7FqfezvXI6SSVikpVRZe+G4jm/afKNV9Nq0ezHN9m13Vtps2bWLq1Km88847AIwbN47w8HCysrLo3r07gwcPpmnTphdtk5aWRteuXRk3bhyPP/44U6ZMYezYsSU+jrJUYa4UiqJj/QhmP9SJG2OqMfGXJNq98hMzVuwBYyAt2XpWSlUI9evXp02bNrmvp0+fTlxcHHFxcSQmJrJp06bLtvH396d3794AtG7dml27dpVVuKVGrxQuEVLJmzeHteLBrvV5ZW4iY7/agH/GEfr/fB34BkNkY4hqDFFN7eWmEBgF2oNJqRK52l/0jhIQEJC7vG3bNt544w1WrFhBaGgod955Z773AuRtmPb09CQrK6tMYi1NmhQK0LR6MFNHtOHuKSt45cedtOz4D+pk74Ejm2Hz/2DNRxcK+4dBZBOIyvOIbAIBlZ13AEqpUnPixAmCgoIIDg7mwIEDzJs3jxtvvNHZYTmEJoVCeHl68NqtLRg86TRdf/XnptjO+Pp7cM8ddYkJOQeHN1lJ4vAmOLwZEr6Es2kXdhAQZV1VRDaBVndAtRbOOxil1FWLi4ujadOmxMTEUK9ePTp16uTskBymwnRJLYnTGVlM+nU77/++g+wcQ0SgL18/3ImqIX4XFzQGTh6Aw4nW40jihWVj4NapcE3vMo1dKVfmSl1Sy5OSdEnVpFAM2TmGLQdPMvidJQT7eTNxWCva1g2/8oanjsBnt8KBdXDT6xB/j+ODVcoNaFJwDL1PoYx4eghNqwfz5eiO+Hh5MOS9pbw0ZxNnM7ML3zAwEkb8Dxr0hDmPwS8vaU8mpZRL0qRwFZpWD2bun6/ljna1+WDRToa+t4zDJ64wKqFPAAydDnF3w8JX4ZuHITuzbAJWSqki0qRwlQJ8vXhpQHPeuTOOzQdP0GvCQj5auouMrEKuGjy9oO9E6PYUrP0UPhsCGSfLLGallLoSTQoldGNMNeb86VoaRQXx7Dcb6f/fxWw7VMgXvQh0Gwv93oQdv8K0m+DkoTKLVymlCqNJoRQ0iArk8wfaM3l4PEdOZtD3v4uYtngnOTmFtBvE3Q3DZkDKNpjc03pWSikn06RQSkSEHk2qMPfRa2lXtzLPf7eJ295dyu6jpwveqFEvGDEHzqXD5F6wd0XZBayUolu3bsybN++idRMmTOChhx4qcJvAwEAA9u/fz+DBgwvc75V6SU6YMIH09PTc13369OH48eNFDd1hNCmUsqggP6aNbMNrt7Zg66GT3DJpKZv2nyDtTCYb96eRfu6S295rtIb7fgT/UPiwLyTOcU7gSlVAw4YNY8aMGRetmzFjBsOGDbvittWrV+fLL7+86s++NCl8//33hIaGXvX+SosmBQcQEQa3rslXD3XE0wMGvL2YNi//xE0TF9HrPwtZvfvYxRuE14N7f4QqzWDmXbDyA+cErlQFM3jwYObMmUNGRgYAu3btYv/+/bRs2ZIePXoQFxdH8+bN+eabby7bdteuXcTExABw5swZhg4dSmxsLEOGDOHMmTO55R588MHcIbefe+45ACZOnMj+/fvp3r073bt3ByA6OpqUlBQAxo8fT0xMDDExMUyYMCH385o0acL9999Ps2bN6NWr10WfU1p0mAsHahAVxJw/Xcur8zbj6+VJy1qhvD5/C7dMWsLITtE8e3PTC1OBBkTA8O/gy3vgf09A2j7o8awOtKcqjrlj4eCG0t1n1ebQe1yBb1euXJm2bdvyww8/0L9/f2bMmMGQIUPw9/dn9uzZBAcHk5KSQvv27enXr1+BU/dOmjSJSpUqsX79etavX09cXFzuey+//DLh4eFkZ2fTo0cP1q9fzyOPPML48eNZsGABERERF+1r9erVTJ06leXLl2OMoV27dnTt2pWwsDC2bdvG9OnTef/997ntttuYNWsWd955Z+n8rWx6peBgkUG+/HtwC/4xIIZbWtdk/uNduat9HaYu3sUL320iMzvnQmGfABjyKbQeAYvGw9cPQtY5p8WuVEWQtwrpfNWRMYann36a2NhYevbsyb59+zh0qOBeggsXLsz9co6NjSU2Njb3vZkzZxIXF0erVq3YuHFjvkNu57Vo0SIGDhxIQEAAgYGBDBo0iN9//x2AunXr0rJlS8BxQ3PrlUIZC/T14sX+zfDyFKYu3sW2wyeZPLzNhSlAPb3g5gkQXBMWvAQnD8JtH4FfsHMDV8rRCvlF70gDBgzg8ccfZ82aNZw5c4a4uDimTZvGkSNHWL16Nd7e3kRHR+c7VHZe+V1F7Ny5k9dee42VK1cSFhbGiBEjrrifwoYe8vX1zV329PR0SPWRXik4gYjwXN9m/HtwLIuTjhL/0k/8afofnDybeb4AdP0L9H8Ldi6Ez0v38lApdUFgYCDdunXjnnvuyW1gTktLIyoqCm9vbxYsWMDu3bsL3UeXLl349NNPAUhISGD9+vWANeR2QEAAISEhHDp0iLlz5+ZuExQUxMmTl9/T1KVLF77++mvS09M5ffo0s2fP5tprry2tw70ivVJwotviaxHi783PiYeYtWYfa/ce45WBsXRuaNcxtroTzhyH+c/AvjVQI67wHSqlrsqwYcMYNGhQbjXSHXfcQd++fYmPj6dly5Y0bty40O0ffPBBRo4cSWxsLC1btqRt27YAtGjRglatWtGsWbPLhtweNWoUvXv3plq1aixYsCB3fVxcHCNGjMjdx3333UerVq3KbBY3HSXVRazclcrYWevZkXKaJ3tdw0Pd6luXo2dPwPgm0KQvDHzH2WEqVap0lFTHcMlRUkWklogsEJFEEdkoIn/Op0w3EUkTkbX241lHxePq2kSHM+dP19I3tjqvztvCw5+t4XRGltWW0PJ2SJgFpw47O0ylVDnnyDaFLOAJY0wToD3wsIg0zafc78aYlvbjRQfG4/L8fTx5Y2hLnunThB8SDjLo7SXsSjkNbUdB9jlYPc3ZISqlyjmHJQVjzAFjzBp7+SSQCNRw1OeVFyLC/V3q8dE97Th08iy3vruUfV41oX4PWDlZu6iqcsfdqrBdXUn/nmXS+0hEooFWwPJ83u4gIutEZK6INCuLeNxB54YRzHygA2fPZXPXB8vZ33g4nDoIid86OzSlSo2fnx9Hjx7VxFBKjDEcPXoUPz+/KxcugMMbmkUkEPgNeNkY89Ul7wUDOcaYUyLSB3jDGNMwn32MAkYB1K5du/WVuoeVJyt3pTLqo1UcT8/gN78nOesTTpXHFhLi7+3s0JQqsczMTJKTk6/Yd18VnZ+fHzVr1sTb++LvCJeYo1lEvIE5wDxjzPgilN8FxBtjUgoqU157HxXmQNoZZv+xjxpbPqL/gTd4JOg/PDZ8CHUjApwdmlLKTbhC7yMBJgOJBSUEEalql0NE2trxHHVUTO6qWog/D3VrQP/hT5LlFcD1J7+m35uLWLStwNyplFJXxZFtCp2Au4Dr8nQ57SMio0VktF1mMJAgIuuAicBQo5WLBfMLxivuDm72XEqzkLPc8+FKVu1KdXZUSqlyRG9eczcp2+C/8ZzpNJY+6zqQcjKDlwbG0De2Oh4eOqKqUip/Tq8+Ug4S0RDq98B/3TQ+HRlHvahA/jxjLQPeXszxdO2uqpQqGU0K7qjdaDh1kOr75/Pl6A78e3Asmw+cZNRHq8nKOxS3UkoVkyYFd9SgpzVb2/J38fb04Lb4WvxrcHNW7Erl/d93Ojs6pZQb06Tgjjw8rKEvkldYo6cCA1rWoHdMVf7z41a2Hbp8OF6llCoKTQruquXt4BMIK94DrOExXuwfQ4CvJ09+sY6zmdlODlAp5Y40Kbgrv5DLRk+NDPLlnwObsy45jX7/XcR9H67S5KCUKhZNCu4sn9FTezevxiuDmiMIPyUeYspibWNQShWdJgV3ZndPZeVkyM7MXT2sbW3mPdaFXk2r8Nq8LQx9byl7U9OdGKhSyl1oUnB3dvdUNn1z2Vv/uiWWMd0bsGn/CQa+vZg1e445IUCllDvRpODu8nRPvVRYgA+P97qGrx7qRICvF0PfW8Z36/Y7IUillLvQpODu8umeeqkGUYHMfqgTLWqG8MiMP1i9W8dLUkrlT5NCeXBJ99T8hAf4MG1kW6oG+/HM7AQy9c5npVQ+NCmUB/l0T81PgK8Xz/VtxuaDJ5m2eFfZxaeUchuaFMqLfLqn5ueGZlXo0TiK//y0lX3HzwCQneNeI+UqpRxHk0J5UUD31EuJCM/3a0aOMfz1y3U8/OkaGjzzPXdNXs7mgyfKMGCllCvSpFCeFNI9Na9a4ZV4rGcjFicd5afEQwyOq8mm/Sfo++YiZv+RXEbBKqVckZezA1ClKG/31OaDCy06qks9ejatQvUQf/x9PEk9fY6HPl3NY5+vI+XkOe7vUq+MglZKuRK9UihPitA99TwRoX5kIP4+noDVO+nDe9rSp3lVXv4+kZfmbMLdZuVTSpWcJoXypgjdUwvi6+XJm8PiuLtDHT5YtJPPVuxxQIBKKVemSaG8uah76pFib+7pIbzQrxkd61dm3NzNfLp8N+ey9J4GpSoKTQrlURG7pxZERPjnwOZEBfnyzOwEHvxkNRlZOgS3UhWBJoXyKLd76geFdk8tTHREAD893pV/DIjh582H+duX67WNQakKQJNCeVXE7qmFERHual+HJ3s14uu1+xnw9hKW7ThaikEqpVyNJoXyqpDRU4vr4e4NeHVwLIdPnGXoe8uY+PO2UghQKeWKNCmUVx4e0P4hq3tq4pwS7UpEuDW+Fgue7MaguBqM/3ErE37aWkqBKqVciSaF8qz1CIhqBnP/ChknS7w7P29PXh3cgsGtazLhp21MWaRTfSpV3mhSKM88vaHvBDixHxb8s3R26SH865ZYrm9ahZe/T2TJ9pRS2a9SyjVoUijvarW1rhiWvwP715bKLj09hPG3taBuRAAPf7pG539WqhzRpFAR9HwOKkXAnEchp3TuNwjy8+a9u1qTlWN44OPVnDmn9zEoVR5oUqgI/MPgxldg/x/WvQulpF5kIBOHtiLx4Anu+GAZCfvS9F4Gpdycw5KCiNQSkQUikigiG0Xkz/mUERGZKCJJIrJeROIcFU+FF3ML1L8Ofv6H1cZQSro3juLt2+NIPHCSm99cxL0fruLoqYxS279Sqmw58kohC3jCGNMEaA88LCJNLynTG2hoP0YBkxwYT8UmAje9DjmZMPdvpbrr3s2rsWTsdYzt3ZhFSSk8PnOdXjEo5aYclhSMMQeMMWvs5ZNAIlDjkmL9gY+MZRkQKiLVHBVThRdeD7r8BRK/ha3zSnXXYQE+jO5an6d7N+a3rUf4eu2+Ut2/UqpslEmbgohEA62A5Ze8VQPYm+d1MpcnDlWaOj4CkY3hf0/CudOlvvu7O0TTvEYIr8/fqqOrKuWGHJ4URCQQmAU8aoy5dBJgyWeTy+odRGSUiKwSkVVHjhR/OGiVh5cP3PwfSNsDv44r9d17eAhP9GpE8rEz/P3rDZzOyCr1z1BKOY5Dk4KIeGMlhE+NMV/lUyQZqJXndU3gslZQY8x7xph4Y0x8ZGSkY4KtSOp0hFZ3wdK34GBCqe++a6NIHuhSjy9WJ3PjGwtJ2JeW+95Pmw6xZHuKtjko5aIc2ftIgMlAojFmfAHFvgXutnshtQfSjDEHHBWTyuP6F8E/1L53oXSreUSEp/o04fNRHcjKtu5jOJ5+jlW7Urnvo1Xc/v5ynv1mY6l+plKqdDjySqETcBdwnYistR99RGS0iIy2y3wP7ACSgPeBhxwYj8qrUjjc8E9IXgmrpzrkI9rWDWfSna05dOIsPccv5IGPV1Mj1J9hbWvz8bLd/JCg+V8pVyPudhkfHx9vVq1a5ewwygdj4KN+sH8djFkJQVUc8jGrdx9j0q9JeIgw5roGNIwKYsBbi9ly6CSDW9fk5YEx+Hp5OuSzlVIWEVltjIm/YjlNChVcShJM6gBN+sLgKWX2sWczs/nvL0n8d0ES1zetwuGTGWRl5/Bkr2vo3jiqzOJQqqIoalLQYS4quogGcO0TkDALkn4qs4/18/bkyRuu4anejflx0yHWJx/nVEYWI6etZPz8LWUWh1LqYpoUFHR+DCo3gP89AZlnyvSj77+2HsM71OEf/WOY/1gXbomrycRfkpi38WCZxqGUsmhSUODla927cGwXLHy1TD/aw0N4oX8Md7avg6+XJy8PjCGmRjAPfrKaCT9tJTvHvao3lXJ3mhSUpW4XaDEMFr8BhxOdFoaftyczRnWgf8saTPhpG4PeXnzRfQ5KKcfSpKAu6PUS+AbBnMdK/d6F4gj09eI/Q1ryxtCWHEg7y9D3lrEhWRODUmVBk4K6ICACrv8H7FkKaz9xdjT0b1mDb8Z0IsTfm1vfXcJLczaxcleqs8NSqlzTpKAu1upOqNMJ5v8fnHL+OFPVQvz56qGOdKofwbQluxjy7lI+Xb7b2WEpVW5pUlAXE7Eanc+dhvl/d3Y0AFQJ9mPyiDase64XXRtF8szsBO74YBnj528hM1tHYlWqNGlSUJeLvAY6/RnWz4DN3zs7mlwBvl68f3c8919bl8MnMpj4SxL3TFupQ3QrVYo0Kaj8dfkLVGsJsx+AlG3OjiaXl6cHz9zUlB8f78q4Qc35fVsKf5u1XkddVaqUaFJQ+fP2gyGfgKc3zLgDMk46O6LLDG1bmyd7NWL2H/v497wtmhiUKgWaFFTBQmvB4KlwdBt8/aA1gJ6Lebh7A25vV5tJv27n9flbNTEoVUKaFFTh6nW1uqkmfgeLCpoWw3lEhJf6xzCsbS3+uyCJcT9s1sSgVAl4OTsA5QY6PAz718DP/4CqLaBhT2dHdBEPD+HlAc3x9BDe/W0HGHiqTxNnh6WUW9IrBXVlItDvTajSDGbdC6k7nR3RZTw8hH/0j+H2drV5d+EOlu046uyQlHJLmhRU0fgEWA3PAJ/fad3H4GJEhP+7qSk1Qv156qsN7E1NZ9rinQx4a7EOk6FUEWlSUEUXXhcGT4ZDG+HbR1yy4dnfx5Pxt7Xg6KkMbpiwkBfnbCJhXxqDJi3m3d+2k6OjripVqCIlBRGpLyK+9nI3EXlEREIdG5pySQ16Qo//g4QvYelbpb//o9vhsyHw0QDYOBuyzhV7F+3qVWbOn67lhmZV6Vg/gt/+2p0ejavwytzNjJy2kpNnM0s/bqXKiSJNxykia4F4IBqYB3wLXGOM6ePQ6PKh03G6AGNg5l3W3c53f20Nu11SOdmwbBL88hJ4+oBfCKTtgYAoiLsL4oZDWJ0ShGz4ZPkeXvh2I42qBPHJfe0ID/ApedxKuYnSno4zxxiTBQwEJhhjHgOqlSRA5cZEYMAka7a2L0bA8b0l29+RrTDlRpj/jNUF9uFl8Oe1cPsXUKM1LPoPvNECPr0Vtsy1EkixQxbual+HD4bHs/3IKW5/fxkppzJKFrdS5VBRrxSWAxOAZ4C+xpidIpJgjIlxdICX0isFF5KSBO93h/B6cM8P4O1fvO2zs2DJRPh1HPhUgt7/hua3Wkknr+N7Yc1H1uPUQQiuCa2HQ6u7ILj4v00WJ6Vw74crqRbiz6gu9WgTHUaDqKBi70cpd1LUK4WiJoWmwGhgqTFmuojUBYYYY8aVPNTi0aTgYrbMhelDocXtMODty7/QC3JoI3z9EBxYC036wU2vQ2BU4dtkZ1qft2oK7FgA4gmN+0D8PVC3G3gUvd/Esh1HeXr2BnYcOY0ITBnehu6Nr/D5SrmxUk0Kl+w4DKhljFl/tcGVhCYFF7TgFfhtHPR5DdreX3jZ7Ez4fbw1F7RfiJUMmg0o/mce3Q6rp8HaTyH9KITVhfiR0PIOa7KgIjDGsP3IaR6Z/gd7j6Xz3l3xdKhfufixKOUGSvtK4VegH9Yd0GuBI8BvxpjHSxhnsWlScEE5OTBjGCT9BMPnQJ0O+Zfbvxa+GQOHNkDMYKu6KKCEX8JZGdYQHKumwO7FViN1/D1w47giX7UkH0tnxNSV7Eo5zbdjOtO0enDJYlLKBZV2Q3OIMeYEMAiYaoxpDbjWWAfKeTw8YOC7EFoHZt4NJ/Zf/H5WhjVExvvXwekjMPQz636HkiYEAC9faD4YRn4PDy23ks3yd2DtZ0XeRc2wSnw5ugMh/t488/UGsvVeBlWBFTUpeIlINeA2YI4D41Huyj8Uhn5q3ek8824rEQAkr4Z3u8Dvr0HsEKtnUeObHBNDVGPo/xbU7gg/PHV5cipEaCUfnu3blD/2HOfprzbooHqqwipqUngR6/6E7caYlSJSD3CdmVeUa4hqAgMnQfJK+N8T1jzPk3taczHc8aX1nn+YY2Pw8ID+/4Xsc/Ddo8W667p/yxr86boGfL5qL+N+2OzAIJVyXUUaJdUY8wXwRZ7XO4BbHBWUcmNN+0Pnx6x7CwBaj4DrX7QalctK5frQ41mY9xSs/xxaDC3ypo9f34jj6Zm8+9sOGlcNYmCrmg4MVCnXU6SkICI1gTeBToABFgF/NsYkOzA25a6u+z9rAL2aba2b0Zyh3QOw6RuY+1eo1w2CqhZpMxHh+X7N2HLoJH+fnUDLWmHUjQhwaKhKuZKiVh9NxRraojpQA/jOXlcgEZkiIodFJKGA97uJSJqIrLUfzxYncOXCPDytOZ6dlRDOx9D/LattY85jxapG8vQQJgxpiZenB49M/4OzmcW/g1opd1XUpBBpjJlqjMmyH9OAyCtsMw248QplfjfGtLQfLxYxFqWKJqIBXPd32PI9bPiyWJtWD/Xn34NjSdifxsipKzmdkeWgIJVyLUVNCikicqeIeNqPO4FCZzExxiwEUkscoVIl0f4hqNkG5v4FTh0u1qY3NKvK+NtasGJXKndNXq6JQVUIRU0K92B1Rz0IHAAGAyNL4fM7iMg6EZkrIs1KYX9KXczDE/q/DefSi12NBDCwVU3eur0Vf+w9zgvfbXRQkEq5jiIlBWPMHmNMP2NMpDEmyhgzAOtGtpJYA9QxxrTAasT+uqCCIjJKRFaJyKojR46U8GNVhRPZCLo/DZvnwMavir35jTHVeLhbA2auSuaJmeu0jUGVayWZea1EQ1wYY04YY07Zy98D3iKS76A1xpj3jDHxxpj4yMgrNWUolY8OY6xhuL//C5wq/g+LR3s2ZEz3Bsxak8xr87Y4IEClXENJkkIRh8MsYGORqiLW4DQi0taORWdbV47h6WVVI2WchO+fLPbmXp4ePHnDNdzZvjaTF+9k3NzNpJ3RGdxU+VOSpFBo5ayITAeWAteISLKI3Csio0VktF1kMJAgIuuAicBQo2MLKEeKagzdxsKmr2FjgbWVhXq6TxMGtqzBO79tp9urC/htq1ZnqvKl0FFSReQk+X/5C+BvjCnSzW+lSUdJVSWSnQUf9IC0ZHh4xVUPypewL40nv1hH0uFT/OuWWG5prXc+K9dWKqOkGmOCjDHB+TyCnJEQlCoxTy9rKtGzaVY31asUUyOEmaM70CY6nCe+WMfbvybpIHqqXChJ9ZFS7qlKU+j6N0iYZc3FcJWC/byZdk8b+rWozr9/2MKMlSWcq1opF6BJQVVMnR+FqrEw53FIv/p7LH29PJkwpCUd6lXmn98n8uXqZO2yqtyaJgVVMXl6W3NKn0mFuX8r0a48PISXB8bg6+XJk1+s40/T/9CJepTb0qSgKq6qza2B+zbMhM3fl2hX9SIDWf50D569uSk/bjrEi99t1DYG5ZY0KaiKrfPjUKU5zHm0RNVIYI2uek/nutzXuS4fLt3N3VNWsDc1vZQCVapsaFJQFZuXDwx4C9KPwrynS2WXT/dpwtjejVm75zhjPltDZnZOqexXqbKgSUGpai2sK4Z102HrvBLvzsNDGN21Pv8aHMu65DRGTF3B4RNnSyFQpRxPk4JSYLUtRDWDb/8Eq6bCmeMl3mWf5tUYN6g5a3YfZ8h7yziYpolBuT5NCkqBVY006D3wC7XaF15rBDPvhi1zIfvqxzga2rY2n9zXliMnMxjy3lL2Hz9TikErVfoKHebCFekwF8qhjIH9f8D6z2HDF1ZbQ6XKEDOn0qVfAAAY0ElEQVQYWgyB6nEgxR8Lcs2eYwyfvIJgf2+m39+e2pUrQcYpSF4Je5ZazyG1oPlgqNPJmgdCqVJU1GEuNCkoVZDsTEj62Wpr2DIXsjMgohG0GArNb4PQWsXa3aat2/hg+gzayGYGVt6LX0oCmGwQD4hsAsd2QeZpCKwKzQZaCaJG66tKQkpdSpOCUqXpzHFrdNV1n8OeJYBAdGcrQTTpB37BF5c3BlJ3wJ5lVvk9y+BoEgBn8SGBBkS36kFEs27WdKF+IXDuNGz9ARK+gm3zIfschEVDzC3Wo4pOTqiuniYFpRzl2C5YP9O6gkjdAV7+0PgmaNofTuy/kAROHbLK+4VC7Q5Quz3U6ch27/rcMWUtZzKz+fS+dsTUCLn8M84ct2aK2/Al7PwNTI51NdHcThDh9cr0kF1Kdiak7gTfIAiu5uxo3IYmBaUczRhIXgXrZ1iD6505Zq0PqW0ngA5WMoi4Bjwu7tOxNzWdoe8tIyMrh9kPdaRWeKWCP+fUYdj0jZUg9i6z1tVobSWHZoPK7xfjudOQsg2ObIGULfbzVisR52RZZUJqQ+12UKsd1Gpr9SDz1AGc86NJQamylHXOaiwOqwMhRZtbIenwSQa9vYSoYD9mje5ISCXvK290fA9snG0liIPrya3GatDTau+IaGhVOXkWYV/FdS4djmyGw5vg0CY4vNG6avIJBP8wqBQO/uEXL1+6zi/08i/t9NQ8X/xbLzyn7blQRjytq6PIa+zjbGSNW7V3OexZDqcOWuW8A6Bma6jV3koUNePBP7T0/xZuSJOCUm5g2Y6j3DV5OT2bVGHSna2Lt/GRrdYVSsIsOLrtwnrxtBJDREOo3MB6RDSEyg0hMOrKDdc52Vb1zOGNF778D22yfqGfn3PLyw8iG0Pl+pB5xvpiP5NqXS2lp1oN6AXxC7GShF+IVd12Os/sdV7+ENHAurqKbAyRjazl8HpWt+H8GANpe2HvCjtJLINDCVaVG2Ltp1ZbK0nUbm/tqwI23mtSUMpNvLUgiVfnbeGdO1tzY0zVq9vJmWNwdLtV3XI0yUoSKUmQuh2y8tw05xtsfZFXbmgnivrgG5LnCmCj9as96/z9FGJ9iVZpalXNnH8Or1twt1ljIOOEnSiOWcki/Vie5fMJ5DgEVbV//V9jJYCQ2pdVtV2VjFOwb/WFRJG8wppYCawuxlFNraTkF2L9TfxCrM4CvsGXPIdceM/Lt+RxOZEmBaXcxLmsHAa+vZhdKaf5/IEO+Tc8X62cHDiRnCdZJF1YTrtkUqDAKtaXZZVm9nNT68vap5D2DneRk2O1R+xdZlU3pW6Hsyes5HU2Dc6duvI+PH2t5OATYL02xnpwybPJuXwd59djXQ35VLL24x1gLXuff13Jfi/wwnJumQCoXO+qOxloUlDKjRw6cZZBby/hXHYOXz14hYbn0nIu3aoSOptmVbFc5XzV5UJOtp0gThTwnJYngdgj34qHXQ0lFz/nu87DWsZA5lnrfpRz6ZCZbjWoZ6Zbr8+dspazChgSpdOjcP0LV3WImhSUcjPbDp3klklLiAjyZdbojoQFFFCHrsq/nOwLiSLztJU4zqVbbULhda9ql0VNCjr2kVIuomGVID4Y3obkY2cY9r4OoFeheXha92EEVbGqi6o2t7reXmVCKNZHO/wTlFJF1rZuOJOHx7M3NZ3bP1jGsdPnnB2SqmA0KSjlYq5tGMnUkW1JPnaGez5cSfq5LGeHpCoQTQpKuaC2dcOZOLQV6/Ye567JK9hzVKf1VGVDk4JSLurGmKpMHNaKLQdP0v31X3nqq/WkanWScjAdJEQpF3ZzbHVa1wnj3d928Mmy3SzZfpQvHuhAVLCfs0NT5ZReKSjl4qqF+PN8v2Z8/kB7Uk5mcMcHyzl6KsPZYalySpOCUm6idZ1wJo9ow57UdPq+uYh//7CZnxMPOTssVc5oUlDKjbSvV5nP7m9PZJAv7/y2nXs/XMVTX23A3W5CVa5L2xSUcjOt64TxzZjOZGXn8K8fNvP+7ztpWj2Yu9rXcXZoqhxw2JWCiEwRkcMiklDA+yIiE0UkSUTWi0ico2JRqjzy8vTgqd5N6NIokv/7OoEXv9ukVwyqxBxZfTQNuLGQ93sDDe3HKGCSA2NRqlzy8BA+uDueER2jmbJ4J2Om/8Efe445OyzlxhyWFIwxC4HUQor0Bz4ylmVAqIiU03kFlXIcHy8PnuvblIe71+eXxMPc+s5SZq7cq1cN6qo4s6G5BpB3QPdke91lRGSUiKwSkVVHjhzJr4hSFZqI8JcbGrPimR60qxfOX2etZ8z0P8jKznF2aMrNODMp5DcfXr4/bYwx7xlj4o0x8ZGRkQ4OSyn3FeTnzYcj2/Jkr0b8b/0BnpmdwN7UdNbuPe7s0JSbcGbvo2SgVp7XNYH9TopFqXLDy9ODMdc15FxWDhN/SeKL1XsxwNgbGzOyU118vLQnuiqYM5PCt8AYEZkBtAPSjDEHnBiPUuXKY9c3Itjfm3XJaZw5l80rczfz7sId3Bxbjf+7uSnenpoc1OUclhREZDrQDYgQkWTgOcAbwBjzDvA90AdIAtKBkY6KRamKSES471prPl9jDL9tPcKsNfv4aOluDqSd5Z8DmxMZ5N6T0avSp9NxKlXBTF60k1e+T8Tfx5Nxg2K5KVY7/VUEOh2nUipf93auy/zHutAgKpAx09cwf+NBZ4ekXIgmBaUqoHqRgUy/vz2xNUJ4ZMYf/LJZB9ZTFk0KSlVQft6eTB7RhgZRgdz/0Wpmrtx75Y1UuadJQakKLCLQlxmjOtCxfmX+Oms9b/68Te+EruA0KShVwQX6ejF5eBsGtarB6z9u5e9fJ5Cdo4mhotKhs5VS+Hh58PptLagS4sekX7dz+GQGbw5rhZ+3p7NDU2VMrxSUUoB1X8PfbmzMC/2a8VPiIYa9v4ykw6ecHZYqY3qloJS6yPCO0UQF+fLXL9fTc/xvNK4axKM9G1GnciVOZWQRXycMkfyGLlPlgSYFpdRlejevRnx0OLPWJDNz1V5Gf7I6972eTarw6uBYwgJ8nBihchS9o1kpVaiMrGyWJB3lZEYWB46f4bX5W4gI9GXisFa0iQ53dniqiIp6R7NeKSilCuXr5Un3xlG5rzvUr8yYz/5g6HvLeGVgc25rU6uQrZW70YZmpVSxxNYMZc4jnXPvbXhrQZLe21COaFJQShVbsJ83k4e3YUDL6rw6bwtPfLGOs5nZzg5LlQKtPlJKXRUfLw/G39aS6IgAJvy0ja2HTvLW7XHUqRzg7NBUCeiVglLqqnl4CI/2bMTk4fHsOZpO7zd+Z+bKvVqd5MY0KSilSqxHkyr88GgXWtQM5a+z1vPAx6tJPX3O2WGpq6BJQSlVKqqH+vPpfe14pk8Tft1yhBsmLOSHBJ1h191oUlBKlRoPD+H+LvX4+uFORAT6MvqTNfzty/WcyshydmiqiDQpKKVKXdPqwXw7phNjujfg81V76fDKz0xetJNzWTnODk1dgSYFpZRDeHt68OQN1/DNw51oVTuMf8zZRIdXfmba4p06NLcL06SglHKoFrVC+XBkG6aNbEOTasE8/90mBr69WEdgdVGaFJRSDicidLsmio/vbcvEYa1IPnaGQW8v5uOlu/SmNxejSUEpVWZEhH4tqvPNw52oFxnI/32zkVvfWUrysXRnh6ZsmhSUUmWuVnglZj/Ukffuas3OlNP0+s9Cpmpbg0vQpKCUcgoRoVezqvzw6LW0iQ7nhe82ccukJSTsS3N2aBWaJgWllFPVDKvEtJFtmDCkJXtS07n5zUU8/OkaliSlaBdWJ9AB8ZRSTiciDGhVg+6No5j8+w4+WLST/204QFglb/q1qM49nevqQHtlRGdeU0q5nFMZWSxJSuGbdfv5cdMhPEV4+qYm3Nmuts4PfZWKOvOaJgWllEs7mHaWv85az8KtR4iuXIm+LaozvGM0EYG+zg7NrRQ1KWibglLKpVUN8ePDkW0Yf1sLaoVX4r8LkrhxwkJ+2nRIh+h2AIcmBRG5UUS2iEiSiIzN5/0RInJERNbaj/scGY9Syj2JCIPiavLxve344c9dCA/w4b6PVnH3lBVsOXjS2eGVKw6rPhIRT2ArcD2QDKwEhhljNuUpMwKIN8aMKep+tfpIKXUuK4ePl+3mjZ+2cioji6Fta3NT82pERwRQI9Tf2eG5pKJWHzmy91FbIMkYs8MOaAbQH9hU6FZKKXUFPl4e3Nu5LoNa1eCNn7fxybLdfLZ8D96ewh3t6vCn6xpQWdscroojk0INYG+e18lAu3zK3SIiXbCuKh4zxuzNp4xSSl0mLMCH5/s1497OddmTms6c9Qf4eNluvlydzANd6nHvtXWp5KM974vDkX+t/PqNXVpX9R0w3RiTISKjgQ+B6y7bkcgoYBRA7dq1SztOpZSbqxVeiVrhlejUIIJ7O9fl3z9s5vUft/LWr0m0iQ7nxpiqDG5dE18vT2eH6vIc2abQAXjeGHOD/fopAGPMKwWU9wRSjTEhhe1X2xSUUkWxevcxvlu3n9+3HWH7kdPUCPXnkR4NGBRXE2/Pitfx0hXaFFYCDUWkLrAPGArcnreAiFQzxpyfxLUfkOjAeJRSFUjrOmG0rhOGMYbft6Xw+o9b+dusDbz963Ye7dmQ3jHVyMjM4aOlu1i64yjD2tYmPjoMf29PQiv5ODt8p3HozWsi0geYAHgCU4wxL4vIi8AqY8y3IvIKVjLIAlKBB40xmwvbp14pKKWuhjGGnxMP8/qPW0k8cOKi90IreXM8PRMAb0/htvhajO5an5ph/uXmDmq9o1kppfKRk2OYt/EgiQdOEOTnTVSwLzc1r8bKXcdIOnKKLQdP8PnKvWRmGyKDfOkbW51b42vSpFqws0MvEU0KSil1lZKPpTN/4yFW7Ezll82HOZedQ8f6lXmga326NIxwy6sHTQpKKVUKjqef4/OVe5myeCeHTmTQuGoQ8dFhzN94CANcd00U/VpWp329ynh6uG6y0KSglFKl6FxWDt+s3cf7v+9g2+FT9GpaBR8vT35JPMTpc9lEBvlyQ7Mq1IsIpHfzqlQLca07qzUpKKWUAxhjOJWRRZCfNwBnM7P5ZfNhvl27n9+2HuFMZjaeHkKPxlEMbVuL2uEBBPp6Eezv5dQb6TQpKKVUGTPGsDf1DNNX7mHmyr0cPX0u9z0RaBsdztnMbKKC/bi+SRV6N6+am1wcTZOCUko5UUZWNit2pnIsPZNTZ7PYeyydnxMPERHoy95j6exNPYOvlwetaofSqnYYcbXDaFU71GHzRGhSUEopF2WMYe3e43y37gCrdqeyaf8JsnKs7+Jm1YPJzjHUDPMnPjqcNtFhxNYMLfFd2K5wR7NSSql8iAitaofRqnYYYLVLbNiXxoqdqfy29QiVfDzZkXKanxIPAxDs58Wfezbi3s51HR6bJgWllHIyP29P2kSH0yY6nIe7N8hdn3Iqg5U7U/l582GqhfiVSSyaFJRSykVFBPrSu3k1ejevVmafWfGGClRKKVUgTQpKKaVyaVJQSimVS5OCUkqpXJoUlFJK5dKkoJRSKpcmBaWUUrk0KSillMrldmMficgRYPdVbh4BpJRiOM6kx+KaysuxlJfjAD2W8+oYYyKvVMjtkkJJiMiqogwI5Q70WFxTeTmW8nIcoMdSXFp9pJRSKpcmBaWUUrkqWlJ4z9kBlCI9FtdUXo6lvBwH6LEUS4VqU1BKKVW4inaloJRSqhAVJimIyI0iskVEkkRkrLPjKS4R2SUiG0RkrYissteFi8iPIrLNfg5zdpz5EZEpInJYRBLyrMs3drFMtM/TehGJc17kFyvgOJ4XkX32eVkrIn3yvPeUfRxbROQG50SdPxGpJSILRCRRRDaKyJ/t9W51Xgo5Drc7LyLiJyIrRGSdfSwv2Ovrishy+5x8LiI+9npf+3WS/X50qQRijCn3D8AT2A7UA3yAdUBTZ8dVzGPYBURcsu7fwFh7eSzwL2fHWUDsXYA4IOFKsQN9gLmAAO2B5c6O/wrH8TzwZD5lm9r/znyBuva/P09nH0Oe+KoBcfZyELDVjtmtzkshx+F258X+2wbay97AcvtvPRMYaq9/B3jQXn4IeMdeHgp8XhpxVJQrhbZAkjFmhzHmHDAD6O/kmEpDf+BDe/lDYIATYymQMWYhkHrJ6oJi7w98ZCzLgFARKbtppwpRwHEUpD8wwxiTYYzZCSRh/Tt0CcaYA8aYNfbySSARqIGbnZdCjqMgLnte7L/tKfult/0wwHXAl/b6S8/J+XP1JdBDRKSkcVSUpFAD2JvndTKF/8NxRQaYLyKrRWSUva6KMeYAWP85gCinRVd8BcXujudqjF2lMiVPFZ7bHIdd7dAK65ep256XS44D3PC8iIiniKwFDgM/Yl3JHDfGZNlF8sabeyz2+2lA5ZLGUFGSQn7Z0926XXUyxsQBvYGHRaSLswNyEHc7V5OA+kBL4ADwur3eLY5DRAKBWcCjxpgThRXNZ53LHE8+x+GW58UYk22MaQnUxLqCaZJfMfvZIcdSUZJCMlArz+uawH4nxXJVjDH77efDwGysfzCHzl/C28+HnRdhsRUUu1udK2PMIfs/cg7wPheqIlz+OETEG+uL9FNjzFf2arc7L/kdhzufFwBjzHHgV6w2hVAR8bLfyhtv7rHY74dQ9OrNAlWUpLASaGi34vtgNcp86+SYikxEAkQk6Pwy0AtIwDqG4Xax4cA3zonwqhQU+7fA3XZvl/ZA2vnqDFd0Sb36QKzzAtZxDLV7iNQFGgIryjq+gth1z5OBRGPM+DxvudV5Keg43PG8iEikiITay/5AT6w2kgXAYLvYpefk/LkaDPxi7FbnEnF2i3tZPbB6T2zFqqN7xtnxFDP2elg9JtYBG8/Hj1V/+DOwzX4Od3asBcQ/HesSPhPr1829BcWOdUn8ln2eNgDxzo7/CsfxsR3nevs/abU85Z+xj2ML0NvZ8V9yLJ2xqhrWA2vtRx93Oy+FHIfbnRcgFvjDjjkBeNZeXw8rcSUBXwC+9no/+3WS/X690ohD72hWSimVq6JUHymllCoCTQpKKaVyaVJQSimVS5OCUkqpXJoUlFJK5dKkoFyOiGTbI1uuE5E1ItLxCuVDReShIuz3VxEpF3P1lhYRmSYig69cUlUUmhSUKzpjjGlpjGkBPAW8coXyoVgjRrqkPHejKuXyNCkoVxcMHANrfBsR+dm+etggIudHuh0H1LevLl61y/7VLrNORMbl2d+t9pj1W0XkWrusp4i8KiIr7QHUHrDXVxORhfZ+E86Xz0useS7+Ze9zhYg0sNdPE5HxIrIA+JdY8xR8be9/mYjE5jmmqXas60XkFnt9LxFZah/rF/bYPojIOBHZZJd9zV53qx3fOhFZeIVjEhH5r72P/+FegyiqsuDsu/j0oY9LH0A21p2pm7FGfmxtr/cCgu3lCKw7OQWI5uI5DnoDS4BK9uvzd+X+CrxuL/cBfrKXRwF/t5d9gVVYY+0/wYW7xz2BoHxi3ZWnzN3AHHt5GjAHe6x+4E3gOXv5OmCtvfwvYEKe/YXZx7YQCLDX/Q14FgjHugv3/E2nofbzBqDGJesKOqZBWKNvegLVgePAYGefc324zkMva5UrOmOskSIRkQ7ARyISg5UA/mmPEJuDNXRwlXy27wlMNcakAxhj8g4Sdn7gt9VYyQSssaRi89Sth2CNibMSmGIPuPa1MWZtAfFOz/P8nzzrvzDGZNvLnYFb7Hh+EZHKIhJixzr0/AbGmGMicjPWZDCLraF98AGWAieAs8AH9q/8OfZmi4FpIjIzz/EVdExdgOl2XPtF5JcCjklVUJoUlEszxiwVkQggEuvXfSTWlUOmiOzCGv/lUkLBQwhn2M/ZXPj3L8CfjDHzLtuRlYBuAj4WkVeNMR/lF2YBy6cviSm/7fKLVYAfjTHD8omnLdADK5GMAa4zxowWkXZ2nGtFpGVBxyTWtJQ6to0qkLYpKJcmIo2xqjqOYv3aPWwnhO5AHbvYSaypGM+bD9wjIpXsfYRf4WPmAQ/aVwSISCOxRqatY3/e+1gjcRY0L/GQPM9LCyizELjD3n83IMVY4/7Px/pyP3+8YcAyoFOe9olKdkyBQIgx5nvgUay5AhCR+saY5caYZ4EUrOGU8z0mO46hdptDNaD7Ff42qoLRKwXlivzFmn0KrF+8w40x2SLyKfCdiKziQpsDxpijIrJYRBKAucaYv9i/lleJyDnge+DpQj7vA6yqpDVi1dccwZrysBvwFxHJBE5htRnkx1dElmP9yLrs173teWCqiKwH0rkw5PFLwFt27NnAC8aYr0RkBDBdRHztcn/HSn7fiIif/Xd5zH7vVRFpaK/7GWs03fUFHNNsrDaNDVijBv9WyN9FVUA6SqpSJWBXYcUbY1KcHYtSpUGrj5RSSuXSKwWllFK59EpBKaVULk0KSimlcmlSUEoplUuTglJKqVyaFJRSSuXSpKCUUirX/wMDoISj6iGQ5AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_losses()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('fastai_bal')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pretraining the Encoder\n",
    "\n",
    "From the IMDB example we know for word level data pretraining the encoder gives much better results (albeit on *much* bigger datasets). Let's see if it improves things here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = (TextList\n",
    "           .from_df(df, cols=[2], processor=processors)\n",
    "           .random_split_by_pct(0.1)\n",
    "           .label_for_lm()\n",
    "           .databunch(bs=32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='5%'>  <col width='95%'>  <tr>\n",
       "    <th>idx</th>\n",
       "    <th>text</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>  z h i h a r e v i t c h xxbos   s m o l a k xxbos   n o s c h e n k o xxbos   c r o w n xxbos   t o k a e v xxbos   o r i o l xxbos   d j a n i b e</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>  p i s k o t i n xxbos   o ' c a l l a g h a n n xxbos   e o g h a n xxbos   e n o k i xxbos   s h a n a u r i n xxbos   c h k h a r t i s h v i l</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>j i m a xxbos   t z a g o l o v xxbos   l i c h m a n xxbos   c o w l e y xxbos   b a g d a s a r o f f xxbos   w a t e r f i e l d xxbos   n e l l i xxbos</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>  t a l m u d xxbos   m a r t z e n k o xxbos   r i p l e y xxbos   z a v o r i n xxbos   g e i g e r xxbos   v r a z e l xxbos   r e y e r xxbos   r o</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>o v s a e v xxbos   g r e a v e s xxbos   r e k u n xxbos   y u z v i s h i n xxbos   t c h e k m a s o v xxbos   s o n e xxbos   g r u s h e t s k y xxbos</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_lm, drop_mult=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4XNW18OHfmlHvsiW5Sa7YGBcsY9mYDsYkpsYpEAghhBSSXAhcUi7wJXATEkJCenJJIQECSYBQEzoYsGnGlmXcexPGXS6SJVkaaTTr+2OO7LGs6ilnNFrv88yjmT37zFmbMVra++yzt6gqxhhjzPHyuB2AMcaY3s0SiTHGmLBYIjHGGBMWSyTGGGPCYonEGGNMWCyRGGOMCYslEmOMMWGxRGKMMSYslkiMMcaEJcntAGKhoKBAhw8f7nYYxhjTqyxevHivqhZ2Va9PJJLhw4dTUVHhdhjGGNOriMiH3alnQ1vGGGPCYonEGGNMWCyRGGOMCYslEmOMMWGxRGKMMSYslkiMMcaExRKJMcaYsPSJ+0iO17x1e9he3UBLQPG3aPBnQAlo8HlLQFFVRASPCB4Bj0cA8IggAh6BJI+H1GQPaUleUpM9JHs9HNnhOPjE6/Hg9Tg/RVCUgEJAlUBA8Yjg9QhJnuBPr0ec8+KcW0jyCsleIdnrIcnrwSMgBOMQgWSPh+QkT7COx3M4VmOMCYclkk48PL+Sueuq3A4jaloTUzDxCCneYMJLTfKSmuQhLfnIzzQnEWakeslMSSIjJYnMVC/ZaUnkZaSQn5FCfkYyuRnJ5KYnk5rkdbt5xpgYsUTSiV9eUYq/JUCS13NUb6C1d+AREBFUFXV6Dy3Oc3B6Ewr+lgA+f4DG5hZ8/gBN/kCwl+D0FlqP9QeO9HTkcE8j+FOBlkDgqJ6ROudQVVoCwfM0B5RmfwB/IOB8Lod7Ny0tAZpblKaWYAx+5/OaWxR/IECzE6evOYDP30JjczDmg43NNDYHaGhqoaG5hXqfH58/0Ol/u7RkDzlpyeRlJFOYnUphVmrwZ3Yqg/PSGZyXzpC8dAqzUq1nZEwvZ4mkE/0yU7pVT1qHsZA+8x+0JaDUN/k52NBM9aFmDhxq4sChZmoONXGw0U9NQzM1h5qpbmiiqtbH4q0H2HPQd0wCSvF6GJKfTkm/DIb2S2dYv0yG9c9gZGEmQ/tlkpJkl/GMiXd95feeiTCvR8hJSyYnLZni/O4do6ocbPSzs6aBHdUNbD/QwLbqBrYdaGDrvkMs+6iamobmo85Rkp/OqMIsxg/OYfyQXMYPzmFIXjoi1osxJl5YIjExIyLkpgevoYwdmNNunZpDzVTuq2fz3jq2VNWzaW89G3bXMnfdHgLOkGF+RjKTh+YzZVg+pwzNZ1JJLhkp9k/ZGLfY/30mruRmJDMpI49JJXlHlTc0tbB210FW7jjIim3VfLC1mjfX7gGCPZfSkjzOGl3AWaMLmVScS5LXhsSMiRXRI/NQE1ZZWZnaMvKJp/pQE0u2VrOocj/vbdrH8m3VqEJOWhLnnFjERRMGcu6JRaSn2AwyY46HiCxW1bKu6lmPxPRaeRkpnDe2iPPGFgFwoL6J+Zv28fb6Kl5fs5vnl+0gPdnLjLFFXDppMDNPKrKeijFRYD0Sk5D8LQHKt+znxRU7eXXVLvbWNTEwJ43PnTqUK6eVUJSd5naIxsS97vZILJGYhNcSUOau3cMjCz7k7fVVJHmECycO4pszTmDMgGy3wzMmbtnQljEOr0eYOW4AM8cNYMveev6x4EP+tegjXli+g09MGszNM8cwoiDT7TCN6bWiNmAsImkiUi4iy0RklYj8sJO6nxERFZEy5/UFIrJYRFY4P2eE1J0nIutEZKnzKIpWG0ziGVGQyR2XjOOd/zmPr509ildW7WLmr97i1qeWU1Xrczs8Y3qlaPZIfMAMVa0TkWTgXRF5WVUXhFYSkWzgJmBhSPFe4FJV3SEiE4BXgSEh71+tqjZWZY5bfmYKt104li+dOZw/ztvEPxds5fU1u/npp0/mgnED3A7PmF4laj0SDapzXiY7j/YuyPwIuBdoDDl2iarucF6uAtJEJDVasZq+qyg7jf+9dDwv3HQmA3LS+OojFdz+zHLqfX63QzMmLHPX7mHGL+ZRubc+6ueK6lxIEfGKyFJgDzBHVRe2eX8yUKKqL3TyMZ8Glqhq6LjDQ86w1h1ia2WYCBgzIJtnbzidr50zkscXfcTFv3uHldtr3A7LmONWVedj8956krzR/xUZ1USiqi2qWgoUA9OcYSoARMQD/Br4dkfHi8h44GfA10KKr1bVicBZzuOaDo69XkQqRKSiqipxl4I3kZOa5OX2C0/isa9Ox+cPcNX9C1j84QG3wzLmuNQ1BnvV2anJUT9XTO7OUtVqYB4wK6Q4G5gAzBORSmA68FzIBfdi4FngC6q6KeSztjs/a4FHgWkdnPN+VS1T1bLCwsKIt8kkrukj+/P0N06nf1YKX3hgIeVb9rsdkjE9VuskkszU6K/sEM1ZW4Uikuc8TwdmAmtb31fVGlUtUNXhqjocWABcpqoVznEvArer6nshn5kkIgXO82TgEmBltNpg+q7Beek8fv1pDMhN49oHy3l/0z63QzKmR+p8zaQne2OymkM0zzAImCsiy4FFBK+RvCAid4nIZV0ceyNwAnBHm2m+qcCrzmcuBbYDf4liG0wfNjA3jcevn05xfjrX/a2c9zbudTskY7qtzucnOy02twrane3GdGFvnY+r/7KQnTUNvPHtcynMtgmEJv7d+OgHrN55kDe/fe5xf0Z372y3FeyM6UJBVir3XX0KDc0t3PPSGrfDMaZb6nx+slNj0yOxRGJMN5xQlMXXzh7FM0u2M3+TDXGZ+FfX6CcrRkNblkiM6aYbZ5zA0H4Z3PHvlTS12XvemHhT2+gny3okxsSXtGQvP/zEeDZV1fOXdza7HY4xnarz+cmKwT0kYInEmB4578QiLpo4kN+9sYGt+w65HY4xHaptbI7ZrC1LJMb00J2XjCfJI9z53Er6wqxH0/uoakyn/1oiMaaHBuamcfPM0cxbV8XK7QfdDseYYzQ0txBQ7BqJMfHss1OHkprk4cnFH7kdijHHaF1ny2ZtGRPHctOT+fj4gfxn6Q4am1vcDseYoxxsTSTWIzEmvl1RVkJNQzNzVu92OxRjjlLn7Kdj10iMiXOnj+rPkLx0nly8ze1QjDnK4SXk02z6rzFxzeMRPj2lmHc2VLGjusHtcIw5rM7XDNjQljG9wuVTilGFZz6wXomJH7V2jcSY3qOkXwanjezPk4u32T0lJm7YNRJjepnLy4r5cN8h20nRxI0juyNaIjGmV7hwwiCyUpPsoruJG3U+P2nJHpJjsDsiWCIxJmzpKV4unTSIF5fvPDykYIybahv9MZuxBZZIjImIK8pKaGhu4e/vf+h2KMbEdFMrsERiTERMHprPzJOK+MPcjeyt87kdjunj6hqbY7Y8ClgiMSZibr/oJBqaW/jN6+vdDsX0ccG9SBIgkYhImoiUi8gyEVklIj/spO5nRERFpCyk7HYR2Sgi60Tk4yHls5yyjSJyW7TiN6anRhVm8fnpw3h04VbW7651OxzTh8Vyd0SIbo/EB8xQ1UlAKTBLRKa3rSQi2cBNwMKQsnHAlcB4YBbwBxHxiogXuA+4EBgHXOXUNSYu3Hz+aLJSk7j7xTVuh2L6sNoY7tcOUUwkGlTnvEx2Hu3dsfUj4F6gMaTsE8DjqupT1S3ARmCa89ioqptVtQl43KlrTFzIz0zhpvNH89b6Kuat2+N2OKaPqvP5yUmUWVtOL2IpsAeYo6oL27w/GShR1RfaHDoECN3oYZtT1lG5MXHjmtOGMax/Bj95aQ3+loDb4Zg+pnV3xEQZ2kJVW1S1FCgGponIhNb3RMQD/Br4djuHSnsf10n5sR8gcr2IVIhIRVVVVc+DN+Y4pSZ5uf3CsazfXcdTdpOiibHG5gAtAU2Moa1QqloNzCN4vaNVNjABmCcilcB04Dnngvs2oCSkbjGwo5Py9s55v6qWqWpZYWFhhFpiTPd8fPxAxg7M5rHyrW6HYvqY2hiv/AvRnbVVKCJ5zvN0YCawtvV9Va1R1QJVHa6qw4EFwGWqWgE8B1wpIqkiMgIYDZQDi4DRIjJCRFIIXpB/LlptMOZ4iQifPqWYZdtq2FRV1/UBxkTIkb1IEiCRAIOAuSKynGACmKOqL4jIXSJyWWcHquoq4AlgNfAKcIMzTOYHbgReBdYATzh1jYk7nygdjEfg2Q+2ux2K6UNivYQ8QNTOpKrLgcntlN/ZQf1z27y+G7i7nXovAS9FJkpjoqcoJ40zRxfy7JLtfOuCMXg87V3iMyayjiwhnyCztozp6z41eQjbqxsor7Ql5k1suNEjsURiTBR9bPwAMlK8NrxlYibWm1qBJRJjoiojJYkLJwzipRU7aWxucTsc0wfUNSbQrC1jTNCnThlCrc/PnNW73Q7F9AGtPZJY7Y4IlkiMibrpI/szMCeNZ5fY8JaJvtpGP6lJHlKSYvfr3RKJMVHm9QizJw/hrfVVtleJibpaX2x3RwRLJMbExKdOGUJLQHl+WbsLMRgTMXWN/pheaAdLJMbExJgB2UwYksOjC7cSCLS7PJwxERHrBRvBEokxMXP92aPYsKeOl1fucjsUk8DqYrypFVgiMSZmLp44iBOKsvjtG+utV2KiptYX202twBKJMTHj9Qg3nT+a9bvreGWV9UpMdNQ2NpNtPRJjEtfFEwcxqjCT376+wXolJirqfHax3ZiE1torWbe7lletV2IiTFWD10gskRiT2C45eXCwV/KG9UpMZPn8AfwBJSvV7iMxJqG19krW7qrltdXWKzGRc3jlX+uRGJP4Ljl5MCMLM/nN6xvwtwTcDsckiMMr/9rFdmMSn9cjfPdjJ7J2Vy0PvLvF7XBMgqh1YeVfsERijGtmTRjIx8YN4Fdz1rPZ9nU3EeDGfu1gicQY14gIP549gdQkD7c9vcIuvJuw1frsGokxfU5RThrfv2Qc5ZX7+cfCD90Ox/Ryh3skiTJrS0TSRKRcRJaJyCoR+WE7db4uIitEZKmIvCsi45zyq52y1kdAREqd9+aJyLqQ94qi1QZjYuHyKcWcNbqAn728lm0HDrkdjunF6hKwR+IDZqjqJKAUmCUi09vUeVRVJ6pqKXAv8CsAVf2nqpY65dcAlaq6NOS4q1vfV9U9UWyDMVEnItzzqYkA3P7MClRtiMscnyO7I3pjet6oJRINar2CmOw8tE2dgyEvM9u+77gKeCwqQRoTJ4rzM7j1wrG8s2EvD71X6XY4ppeqbfSTkuQhNSlBEgmAiHhFZCmwB5ijqgvbqXODiGwi2CO5qZ2P+SzHJpKHnGGtO0REIh64MS64ZvowLhg3gJ+8tIbFH+53OxzTC9U2NpMT42EtiHIiUdUWZ3iqGJgmIhPaqXOfqo4CbgW+H/qeiJwKHFLVlSHFV6vqROAs53FNe+cWketFpEJEKqqqqiLUImOiR0T4xeWTGJKfzn/98wPbltf0mBubWkGMZm2pajUwD5jVSbXHgdltyq6kTW9EVbc7P2uBR4FpHZzzflUtU9WywsLC44zcmNjKTU/mD1efQvWhZm56bAktNiXY9IAbCzZCdGdtFYpInvM8HZgJrG1TZ3TIy4uBDSHveYDLCSaY1rIkESlwnicDlwChvRVjer3xg3P58ewJzN+0j1/NWed2OKYXqXWpRxLNMw4CHhYRL8GE9YSqviAidwEVqvoccKOIzASagQPAtSHHnw1sU9XNIWWpwKtOEvECrwN/iWIbjHHF5WUlfLD1APfN3cTkknxmjhvgdkimF6hr9DM4Lz3m541aIlHV5cDkdsrvDHl+cyfHzwOmtymrB6ZELkpj4tf/XjqeldsPcsu/lvLcN89kREGm2yGZOOfGplZgd7YbE7fSkr388fOnkJzk4fpHKqh37hEwpiO1jc2WSIwxRyvOz+D/rprMpqo6vvvUMrtZ0XRIVRN71pYx5vidfkIBt104lpdW7OLPb2/u+gDTJ/n8AZpbNLFmbRljIuerZ43kkpMHce8ra3l3w163wzFxyK1NrcASiTG9gohw72dO5oSiLG55Yin765vcDsnEmTqXttkFSyTG9BoZKUn85rOTqT7UxPf/bYs7mqMdXvk3xkvIgyUSY3qVcYNzuOWCMby0YhfPLdvhdjgmjhx0ttm1WVvGmC597exRTBmWzx3/XsnOmga3wzFx4vDQll0jMcZ0xesRfnn5JJpblP95arkNcRkg5GK79UiMMd0xvCCT7118Eu9s2MvfF9gWvSb0GoklEmNMN1196lDOGl3Ava+so6ah2e1wjMtqbdaWMaanRITbLhxLnc/P39+vdDsc47I6n58Ub+x3R4RuJhIRGSUiqc7zc0XkptYl4o0x7hk/OJdzTyzkwfcqaWhqcTsc46LqQ03kpMd+6i90v0fyNNAiIicADwAjCG4qZYxx2Q3nncD++ib+tWir26EYF+2obmRQbpor5+5uIgmoqh/4JPAbVb2F4H4jxhiXTR3ej6nD87n/7c00+QNuh2Ncsqsm/hNJs4hcRXDjqRecMnf6UMaYY/zXeSewo6aR/yzd7nYoxiU7ahpc2dQKup9IrgNOA+5W1S0iMgL4R/TCMsb0xLljChk3KIc/vrXJ9nnvg+p8fmob/QyM5x6Jqq5W1ZtU9TERyQeyVfWnUY7NGNNNIsI3zh3F5qp6Xlu1y+1wTIztrA6ucBDXQ1siMk9EckSkH7AMeEhEfhXd0IwxPXHRxEEM75/BH+Ztsrvd+5idNY0AcT+0lauqB4FPAQ+p6hRgZvTCMsb0lNcjfP2cUazYXsP7m/e5HY6JodY11+K6RwIkicgg4AqOXGzvlIikiUi5iCwTkVUi8sN26nxdRFaIyFIReVdExjnlw0WkwSlfKiJ/CjlminPMRhH5nYhIN9tgTMKbPXkI/TJTeOi9SrdDMTG0o7oRERiQE9+J5C7gVWCTqi4SkZHAhi6O8QEzVHUSUArMEpHpbeo8qqoTVbUUuBcIHS7bpKqlzuPrIeV/BK4HRjuPWd1sgzEJLy3Zy+emDeX1NbvZuu+Q2+GYGNlZ00BhVirJXncWK+nuxfYnVfVkVf2G83qzqn66i2NUVeucl8nOQ9vUORjyMrPt+205vaIcVX1fg4PAjwCzu9MGY/qKa04bhleEh9+vdDuUsKzecZBlH1W7HUavsLOmkUEuXR+B7l9sLxaRZ0Vkj4jsFpGnRaS4G8d5RWQpsAeYo6oL26lzg4hsItgjuSnkrREiskRE3hKRs5yyIcC2kDrbnDJjjGNAThoXnzyIJxZ9dHhF2N5k5fYavvJwBRf97h0+86f5zF27x+2Q4t7OmkYGu3R9BLo/tPUQ8BwwmOAv7uedsk6paoszbFUMTBORCe3UuU9VRwG3At93incCQ1V1MvAt4FERyQHaux7Sbi9GRK4XkQoRqaiqquqygcYkkuvOGEGtz89TFR+5HUq3rd9dy1cfqeCS379L+ZZ93DJzDGMH5vC1fyxm/sa9bocXt1SVndUNrt1DAt1PJIWq+pCq+p3H34DC7p5EVauBeXR+PeNxnGEqVfWp6j7n+WJgEzCGYA8ktCdUDLS736iq3q+qZapaVljY7VCNSQilJXlMHprH3+ZXEugFNyj6WwJ87i8LWbh5H9+6YAzv3jaDm2eO5pEvTWNE/0y+8kgFFZX73Q4zLh1s9FPf1MLg3Dgf2gL2isjnnaEqr4h8Huh0fqGIFLauECwi6QSnC69tU2d0yMuLcS7gO8d6necjCV5U36yqO4FaEZnuzNb6AvCfbrbBmD7lujNGULnvEHPXxf/Q0OIPD7C3zsc9nzqZm84fTU5acAWm/MwU/v6VaQzMSeO6hxaxfJtdM2nr8NTfvPjvkXyJ4NTfXQSHnT5DcNmUzgwC5orIcmARwWskL4jIXSJymVPnRmdq8FKCQ1jXOuVnA8tFZBnwFPB1VW39c+QbwF+BjQR7Ki93sw3G9CkXThjIwJy0XjEVeM7q3aR4PZxz4rGjB0XZafzjK6eSk57MtQ+Ws6e20YUI41frzYhu3UMC0K2ttFR1K3BZaJmI/Dfwm06OWQ5Mbqf8zpDnN3dw7NMEl65v770K4JhrLcaYoyV7PVxz2jB+/uo61u+uZcyAbLdDapeqMmfNbk4b1b/DbWIH56Xz8JemcdFv3+Gel9by68+WxjjK+LWzujWRxP/QVnu+FbEojDFR8blpQ0lJ8vDI+5Vuh9KhjXvq+HDfIWaOG9BpvROKsvjaOSN5dsl23t9kd+632lnTgEegKDvVtRjCSSR2R7kxcS4/M4XLJg3mmQ+2c7AxPvd1n7NmNwAzTyrqsu4N551ASb907vjPStt7xbGjupEBOWkkuXQzIoSXSOJ/Kogxhi+cNoxDTS08s3hb15VdMGf1biYOye3W0ExaspcfXDqejXvqeODdLTGILv7tOuju1F/oIpGISK2IHGznUUvwnhJjTJw7uTiPSSV5/H3Bh3G3KvCe2kaWflTNBV0Ma4U6/6QBfGzcAH73xga2HbBlYHZWN7o69Re6SCSqmq2qOe08slW1WxfqjTHu+8L0YWyqqmd+nF1beHPNHlTpUSIBuPPScQDc9fzqaITVa6gqO2oaXJ2xBeENbRljeomLTx5Ev8yUuLvo/vqa3QzJS2fswJ7NKCvOz+Cm80fz2urdveI+mWipPtRMY3Mgvoe2jDGJIS3ZyxVlJcxZvZsdzm56bjvU5OedDXu5YNwAjmc3iC+fOYKRBZn86IXVffbCu9sbWrWyRGJMH3H1qUMBeHThVpcjCXpnw158/kCPh7VapSR5+P4lJ7G5qj7uelqx4vaGVq0skRjTR5T0y2DG2AE8Vr4Vn7/F7XB4ffVustOSmDai33F/xnknFnHOmEJ++8YG9tX5Ihhd77DDeiTGmFj7wmnD2FffxMsrdrkaR0tAeXPtHs47sSiszZhEhDsuOYmGphZ+OWd9BCPsHXZWN5DkEQqy3LsZESyRGNOnnHlCAcP7Z/BYubvDWzuqG9hX38Tpo/qH/VknFGVzzWnDeKx8K6t21EQgut5jV03wZkSvx937wy2RGNOHeDzC5WUlLNyyn8q99a7FUdsY3HArLyM5Ip/33+ePIS89mbueXx1398pEUzxM/QVLJMb0OZ8+pRiPwFMu3uneunNjVmpkEkluRjLf/tiJLNyy39V2xdrOmkbXp/6CJRJj+pyBuWmcM6aQpxZvo8WlTa/qnUSSmeqN2GdeNW0o04b349anl/PMB4mfTFQ1uMWuyxfawRKJMX3SFWUl7DrYyDsb3NmGurVHkp0WuQUyvB7hoeumMn1kf7795LK4meYcLfvrm2jyB2xoyxjjjvNPGkC/zBSerHDnL/e6wz2SyK60lJmaxINfnMp5Jxbx/55dwV/f2RzRz48nRza0sh6JMcYFKUkeZpcO4bXVu9hf3xTz89dHKZFA8C7+P31+ChdNHMiPX1zDD55blZC7KrauUGA9EmOMa66YWkxzi/Kfpdtjfu7WWVuZKdFZ+zUlycPvrpzM56cP5eH3KznzZ3O5/ZkVbK6qi8r53HC4R+LiXu2tLJEY00eNHZjDpOJc/rXoo5hPma33+clI8Ub1/ockr4cfz57Im98+l8unFPP0B9s4/1dvcfPjSzjU5I/aeWNlZ00jyV6hINPdmxHBEokxfdrlZSWs3VXLqh0HY3reOp+/w/3ZI21EQSZ3f3Ii7906g2+cM4rnl+3gmgfKqWmIzx0ju2tnTQMDctLwuHwzIkQxkYhImoiUi8gyEVklIj9sp87XRWSFiCwVkXdFZJxTfoGILHbeWywiM0KOmSci65xjlopI1/tzGmPademkwaQmefjXoo9iet5YJpJWhdmp/M+ssdz3uVNYvq2aK+9fQFVt712fKx42tGoVzR6JD5ihqpOAUmCWiExvU+dRVZ2oqqXAvcCvnPK9wKWqOhG4Fvh7m+OuVtVS59F3NyMwJky56clcPHEQz3ywLaZ/odf7/FG50N4dF04cxAPXTqVybz1X/Pl9tsfJsvo94W8JsHrnQUYPyHI7FCCKiUSDWq9sJTsPbVMntD+d2fq+qi5R1R1O+SogTUTcHwg0JgF96cwR1De18HgM199yo0cS6uwxhfzjK9PYW+fj8j/O73U9kzU7a6nz+cNaOTmSonqNRES8IrIU2APMUdWF7dS5QUQ2EeyR3NTOx3waWKKqod/0Q86w1h1yPDviGGMOmzAkl9NH9eeh9ypjtkFUna/FtR5JqynD+vHYV6ezp9bHr3rZysHllfsB+kYiUdUWZ9iqGJgmIhPaqXOfqo4CbgW+H/qeiIwHfgZ8LaT4amfI6yzncU175xaR60WkQkQqqqrcuXvXmN7iq2eNZNfBRl5csaPryhFQ52uO6F3tx2vCkFw+P30Y/1q0lfW7a90Op9vKt+yjpF96XNyMCDGataWq1cA8YFYn1R4HZre+EJFi4FngC6q6KeSztjs/a4FHgWkdnPN+VS1T1bLCwsKw22BMIjtnTCGji7L4y9tbYjIVuN7XEtF1tsJx8/mjyUxN4icvrXE7lG5RVRZVHmDa8PCX4I+UaM7aKhSRPOd5OjATWNumzuiQlxcDG5zyPOBF4HZVfS+kfpKIFDjPk4FLgJXRaoMxfYXHI3zlrBGs3nmQ9zfti/r5gtdIIrPyb7jyM1P45owTmLeuyrW1x3piU1Ud++ubmDYi3+1QDotmj2QQMFdElgOLCF4jeUFE7hKRy5w6NzpTg5cC3yI4QwvgRuAE4I4203xTgVedz1wKbAf+EsU2GNNnfKJ0CAVZqdwf5fWpmvwBmvwBsuKkRwJw7enDKc5P5+4X17i2InJ3lW85AMC0EfHTI4naIKWqLgcmt1N+Z8jzmzs49sfAjzv46CkRCdAYc5S0ZC/XnjaMX85Zz/rdtYwZkB2V80Rzna3jlZrk5dZZY/nmY0t4+oNtXFFW4nZIHSrfso+CrFSG989wO5TD7M52Y8xhn58+jLRkT1RXzT2yqVX8JBKAS04eRGlJHr94dV1cL6GyqPIAp47oRzxNWLVEYow5LD8zhcunlPDvJTuitmJuvCZCBx09AAAT10lEQVQSEeGOS05iT62Pn7+6zu1w2rXtwCG2VzcwdXj8XB8BSyTGmDa+dOYImgMB/v7+h1H5/MOJJA6m/7Y1ZVg/vnj6cB56r5LXVu1yO5xjLDp8/0j8XB8BSyTGmDZGFGQy86QB/GPBhzQ0tUT886O1qVWk3H7RWE4uzuU7Ty7jo/2H3A7nKOVb9pOdlsSJA6Nz/ep4WSIxxhzjq2eN5MChZp6Owt7n9XE6tNUqNcnL/111Cgrc+NiSmN3t3x3lW/YzdXi/qC6/fzwskRhjjjF1eD4nF+fy4LtbCER4OmxdY3wnEoCh/TP4+WdOZtlH1fzslbVdHxADe+t8bKqqZ+rw+FgWJZQlEmPMMUSEr5w1ks1763lzbWQX2I73oa1WsyYM4ounD+eBd7fwZEXsN/9qqyLO1tcKZYnEGNOuCycMZHBuGn+J8FTgeJ211Z7bLxpL2bB8vvvUcr7wYDkbXFyPa+GW/aQle5g4JNe1GDpiicQY065kr4frzhjBwi37WbGtJmKfW+/zk54c3W12IyU1ycvj10/nB5eOY9lH1cz67Tv84LlVVB9qinksiyr3M7kkn5Sk+Pu1HX8RGWPixmenlZCVmsRf341cr6TO1xKXU387kuT18MUzRjDvu+dx1bQSHnm/ktPueZNb/rWUt9dXxWRJlSVbD7B6x0FOHRl/w1pgicQY04mctGQ+O7WEF5bvZEeEdhJ0e1Or49UvM4Ufz57ISzefxezJQ3hjzW6+8GA5p93zBve+sjZqs7tqDjVz46NLGJyXznWnj4jKOcJlicQY06nrzhiOqvLw/MqIfF5wm934WbCxp8YOzOGeT02k/Hsz+cPVp3BycS5/mLeJW55YGvHeiary3aeWsftgI7+/ajK5GfGxYnJblkiMMZ0qzs/gwgmDeLR86+F7QMJR19g7eyRtpSV7uWjiIP567VT+30VjeXH5Tm57enlEp0v/bX4lr63ezW0XjmXy0PhaFiWUJRJjTJe+fNYIahv9PFnxUdif1VuHtjpz/dmjuOn80Ty5eBt3vbA6IlOFl2+r5icvrWHmSUV8+cz4HNJqZYnEGNOlU4bmM3loHg/Nrwx7+CYREwnALTNH86UzRvC3+ZVh7wG/r87HjY8uoTArlV9cPimuVvptjyUSY0y3fOXMkXy47xCvr9kd1ucEr5EkXiJpXT34yqkl/P7Njdw3d+Nxfc66XbV84r73gtdFPjeZvIyUCEcaeZZIjDHd8vHxAxiSl84D72wJ63MStUcCwWRy9ycnMrt0MD9/dR1/fmtTj46fu3YPn/7jfHz+AP/62mlMGRaf033bskRijOmWJK+H684YTnnlfpZvqz6uz2huCeDzBxI2kQB4PcIvLp/EpZMGc8/La7u1SZiq8sC7W/jyw4sY1j+D5248g9KSvBhEGxmJ+20aYyLuiqkl/Ob1DTzw7hZ+e+UxO2l3KR632Y2GJK+HX18xiUBA+fGLa/B6hGumD2P1zoNUVB5g8YcH2Lr/ELWNzdQ2+qlt9NPUEuDj4wfw68+WkpHSu/779K5ojTGuar1B8eH5ldx24VgG5ab36Pjaxvjd1CrSkrwefnNlKf5AgB8+v5p7X1lHQ3Nwf5cheemMHpDFiIJMstOSyEpLYlRBFp+ZUoynFywd01bUvk0RSQPeBlKd8zylqv/bps7XgRuAFqAOuF5VVzvv3Q582XnvJlV91SmfBfwW8AJ/VdWfRqsNxphjBXcQ3MJjC7fyrY+d2KNj65t6z4KNkZDs9fD7q07h16+v55DPT9nwfpQNz+9xAo530fw2fcAMVa0TkWTgXRF5WVUXhNR5VFX/BCAilwG/AmaJyDjgSmA8MBh4XUTGOMfcB1wAbAMWichzrcnHGBN9Jf0yOH1UAf9euoNbLhjTo6mpfWVoK1RKkodbZ411O4yoitrFdg2qc14mOw9tU+dgyMvMkPc/ATyuqj5V3QJsBKY5j42qullVm4DHnbrGmBiaPXkIW/cf4oOtPbvoXtsLNrUyPRfVWVsi4hWRpcAeYI6qLmynzg0isgm4F7jJKR4ChN5Cu80p66jcGBNDHx8/gNQkD/9esr1Hx9X7gtcILJEklqgmElVtUdVSoBiYJiIT2qlzn6qOAm4Fvu8Ut9dX1k7KjyEi14tIhYhUVFVVHV8DjDHtyk5L5oJxA3hxxU6aW7q/6m2drxnoGxfb+5KY3EeiqtXAPGBWJ9UeB2Y7z7cBJSHvFQM7Oilv75z3q2qZqpYVFhYeZ+TGmI7MLh3C/vom3tnQ/T/U6lp7JL1seqvpXNQSiYgUikie8zwdmAmsbVNndMjLi4ENzvPngCtFJFVERgCjgXJgETBaREaISArBC/LPRasNxpiOnT2mkLyMZJ5d0u7fcu2qa2y92N57l5E3x4rmnwWDgIdFxEswYT2hqi+IyF1Ahao+B9woIjOBZuAAcC2Aqq4SkSeA1YAfuEFVWwBE5EbgVYLTfx9U1VVRbIMxpgMpSR4uOXkQTy3e1u1lT+qb/KQle0jy2qIaiSRqiURVlwPH3PqqqneGPL+5k+PvBu5up/wl4KUIhWmMCcPs0iH8Y8FWXlu1i0+dUtxl/UReZ6svsz8LjDHHbcqwfIrz03m2m7O3EmVTK3M0SyTGmOMmIswuHcJ7G/eyp7axy/qJuoR8X2eJxBgTltmTBxNQeGHZzi7r1trQVkKyRGKMCcsJRdmMG5TDiyu6TiT1lkgSkiUSY0zYZo4bwJKtBzhQ39RpPRvaSkyWSIwxYTt/bBEBhbfWd35zYp3Pb3e1JyBLJMaYsE0ckktBVipvrN3TaT2b/puYLJEYY8Lm8QjnnljIW+v24O9g7S1/S4DG5sTeZrevskRijImIGWOLONjo73Bp+daVf+0aSeKxRGKMiYgzRxeQ5BHe7GB4q9ZZ+TfbEknCsURijImInLRkpg7vx5trd7f7vvVIEpclEmNMxJx/UhHrd9ex7cChY96r89nKv4nKEokxJmLOG1sEwNx2hrdaE0m2Tf9NOJZIjDERM7Igk2H9M9q9TlJ/uEdiiSTRWCIxxkSMiHDeiUXM37SPhqaWo95r3dTKpv8mHkskxpiImjG2CJ8/wPxNe48qbx3askSSeCyRGGMi6tSR/chI8R4zvGVDW4nLEokxJqJSk7yceUIBc9fuQVUPl9f5/KQmeUi2bXYTjn2jxpiIO/fEInbUNLJxT93hMltnK3FZIjHGRNzZYwqAo1cDtpV/E1fUEomIpIlIuYgsE5FVIvLDdup8S0RWi8hyEXlDRIY55eeJyNKQR6OIzHbe+5uIbAl5rzRabTDGHJ/i/AxGFWYelUjqfX4yUyyRJKJo9kh8wAxVnQSUArNEZHqbOkuAMlU9GXgKuBdAVeeqaqmqlgIzgEPAayHHfbf1fVVdGsU2GGOO09ljCinfsp/G5uA04NpG65EkqqglEg1qHSBNdh7aps5cVW1dS2EBUNzOR30GeDmknjGmFzhnTCE+f4AFm/cBUN9k10gSVVSvkYiIV0SWAnuAOaq6sJPqXwZebqf8SuCxNmV3O8NhvxaR1AiFa4yJoFNH9CclycPb64P3k9T7Wmzqb4KKaiJR1RZneKoYmCYiE9qrJyKfB8qAn7cpHwRMBF4NKb4dGAtMBfoBt3bwmdeLSIWIVFRVdb79pzEm8tJTvJw6oh9vbwj+/1fbaD2SRBWTWVuqWg3MA2a1fU9EZgLfAy5TVV+bt68AnlXV5pDP2ukMm/mAh4BpHZzzflUtU9WywsLCCLXEGNMT54wpZOOeOrZXN1Dv85NlK/8mpGjO2ioUkTzneTowE1jbps5k4M8Ek0h7u+FcRZthLaeXgogIMBtYGfnojTGRcPaY4B9xb67dQ0NzC1mpyS5HZKIhmv3MQcDDIuIlmLCeUNUXROQuoEJVnyM4lJUFPBnMC2xV1csARGQ4UAK81eZz/ykihYAAS4GvR7ENxpgwjC7KYlBuGq+s3AnYXiSJKmqJRFWXA5PbKb8z5PnMTo6vBIa0Uz4jQiEaY6JMRDh7dCFPLv4IsAUbE5Xd2W6MiapzTiwk4Ez8t/tIEpMlEmNMVJ0xqgCPBJ/b9N/EZInEGBNVuRnJTB6aD0C2JZKEZInEGBN1Z48Ozt6yHklism/VGBN1V51awqEmP6OLstwOxUSBJRJjTNQVZadx+0UnuR2GiRIb2jLGGBMWSyTGGGPCYonEGGNMWCyRGGOMCYslEmOMMWGxRGKMMSYslkiMMcaExRKJMcaYsIiquh1D1IlIFfBhm+JcoKaLstDX3XleAOwNI9T2Yupune6W96RNsWhPZ/W68x21LUuENvX0O3Pz311H71mbgs/j5fdDR+911YY8Ve16i1lV7ZMP4P6uykJfd/N5RaRj6m6d7pb3pE2xaE9P29RVWSK0qaffmZv/7qxNnbcpXn4/HE+buvtvXVX79NDW890oe76Hz8PVnc/qqE53y2PZpu5+Tk/a1FVZIrTpeL6zcITz766j96xN3Y+ju2Ldpm7H3ieGtmJFRCpUtcztOCIl0doD1qbewtrUu/TlHkk03O92ABGWaO0Ba1NvYW3qRaxHYowxJizWIzHGGBMWSyTtEJEHRWSPiKw8jmOniMgKEdkoIr8TEQl575sisk5EVonIvZGNusu4It4mEfmBiGwXkaXO46LIR95pXFH5npz3vyMiKiIFkYu4W3FF43v6kYgsd76j10RkcOQj7zSuaLTp5yKy1mnXsyKSF/nIO40rGm263PndEBCR3nUtJZzpaIn6AM4GTgFWHsex5cBpgAAvAxc65ecBrwOpzuuiBGjTD4DvJNL35LxXArxK8N6jgt7eJiAnpM5NwJ8SoE0fA5Kc5z8DfpYAbToJOBGYB5TFsj3hPqxH0g5VfRvYH1omIqNE5BURWSwi74jI2LbHicgggv/Tvq/BfxmPALOdt78B/FRVfc459kS3FUeLUptcFcU2/Rr4HyDmFxCj0SZVPRhSNZMYtytKbXpNVf1O1QVAcXRbcbQotWmNqq6LRfyRZomk++4HvqmqU4DvAH9op84QYFvI621OGcAY4CwRWSgib4nI1KhG2z3htgngRmd44UERyY9eqN0WVptE5DJgu6oui3agPRD29yQid4vIR8DVwJ1RjLW7IvFvr9WXCP5l77ZItqlXsT3bu0FEsoDTgSdDhtJT26vaTlnrX39JQD4wHZgKPCEiI52/SmIuQm36I/Aj5/WPgF8S/J/aFeG2SUQygO8RHDaJCxH6nlDV7wHfE5HbgRuB/41wqN0WqTY5n/U9wA/8M5Ix9lQk29QbWSLpHg9QraqloYUi4gUWOy+fI/iLNbSLXQzscJ5vA55xEke5iAQIrr1TFc3AOxF2m1R1d8hxfwFeiGbA3RBum0YBI4Blzi+DYuADEZmmqruiHHtHIvFvL9SjwIu4mEiIUJtE5FrgEuB8t/4gCxHp76l3cfsiTbw+gOGEXEgD5gOXO88FmNTBcYsI9jpaL6Rd5JR/HbjLeT4G+AjnPp5e3KZBIXVuAR7v7d9TmzqVxPhie5S+p9Ehdb4JPJUAbZoFrAYKY92WaP/boxdebHc9gHh8AI8BO4Fmgj2JLxP8S/UVYJnzD/jODo4tA1YCm4D/a00WQArwD+e9D4AZCdCmvwMrgOUE/9oaFKv2RKtNberEPJFE6Xt62ilfTnD9pCEJ0KaNBP8YW+o8Yj0TLRpt+qTzWT5gN/BqLNsUzsPubDfGGBMWm7VljDEmLJZIjDHGhMUSiTHGmLBYIjHGGBMWSyTGGGPCYonE9EkiUhfj8/1VRMZF6LNanJV8V4rI812tfCsieSLyX5E4tzHtsem/pk8SkTpVzYrg5yXpkUUEoyo0dhF5GFivqnd3Un848IKqTohFfKbvsR6JMQ4RKRSRp0VkkfM4wymfJiLzRWSJ8/NEp/yLIvKkiDwPvCYi54rIPBF5ytkr458he03Ma91jQkTqnEUUl4nIAhEZ4JSPcl4vEpG7utlrep8jC05micgbIvKBBPe7+IRT56fAKKcX83On7ned8ywXkR9G8D+j6YMskRhzxG+BX6vqVODTwF+d8rXA2ao6meDKuT8JOeY04FpVneG8ngz8NzAOGAmc0c55MoEFqjoJeBv4asj5f+ucv8v1l5x1nM4nuKoAQCPwSVU9heD+N790EtltwCZVLVXV74rIx4DRwDSgFJgiImd3dT5jOmKLNhpzxExgXMjqrTkikg3kAg+LyGiCK7UmhxwzR1VD96UoV9VtACKylOB6TO+2OU8TRxa4XAxc4Dw/jSP7ojwK/KKDONNDPnsxMMcpF+AnTlIIEOypDGjn+I85jyXO6yyCieXtDs5nTKcskRhzhAc4TVUbQgtF5PfAXFX9pHO9YV7I2/VtPsMX8ryF9v8fa9YjFyc7qtOZBlUtFZFcggnpBuB3BPcaKQSmqGqziFQCae0cL8A9qvrnHp7XmHbZ0JYxR7xGcK8OAESkdUnwXGC78/yLUTz/AoJDagBXdlVZVWsIbp37HRFJJhjnHieJnAcMc6rWAtkhh74KfMnZQwMRGSIiRRFqg+mDLJGYvipDRLaFPL5F8JdymXMBejXBpf8B7gXuEZH3AG8UY/pv4FsiUg4MAmq6OkBVlxBcbfZKgps7lYlIBcHeyVqnzj7gPWe68M9V9TWCQ2fvi8gK4CmOTjTG9IhN/zUmTjg7NDaoqorIlcBVqvqJro4zxm12jcSY+DEF+D9nplU1Lm5bbExPWI/EGGNMWOwaiTHGmLBYIjHGGBMWSyTGGGPCYonEGGNMWCyRGGOMCYslEmOMMWH5/0QLd/htSfdkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_end=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:34 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.655349</th>\n",
       "    <th>2.249993</th>\n",
       "    <th>0.325342</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.187761</th>\n",
       "    <th>1.975275</th>\n",
       "    <th>0.401822</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.007302</th>\n",
       "    <th>1.886618</th>\n",
       "    <th>0.426303</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>1.868937</th>\n",
       "    <th>1.847373</th>\n",
       "    <th>0.435982</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(4, max_lr=1e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('letter_lang')\n",
    "learn.save_encoder('letter_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = \"ho\"\n",
    "N_WORDS = 4\n",
    "N_SENTENCES = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ho u s i m\n",
      "ho n o b a\n",
      "ho n e r e\n",
      "ho n a r a\n",
      "ho v a b e\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tr i n o r\n",
      "tr e r o e\n",
      "tr e n e n\n",
      "tr i r i s\n",
      "tr u p u t\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"tr\"\n",
    "print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = text_classifier_learner(data, bptt=30)\n",
    "learn.load_encoder('letter_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8VPW9//HXZyaTfWFJwg4BBCSCbBH3fanVqnWr7XWr2OvVWpfut7Xb7ab92dpFW73UvdbeWrV1qWsririALGERFBCQHYIs2cgkM/P9/TFDDDGBQHLmzCTv5+MxD2f5zpz3HEk++Z7vOd+vOecQEREBCPgdQEREUoeKgoiINFNREBGRZioKIiLSTEVBRESaqSiIiEgzFQUREWmmoiAiIs1UFEREpFmG3wEOVHFxsSsrK/M7hohIWpk3b94251zJ/tqlXVEoKytj7ty5fscQEUkrZvZhR9rp8JGIiDRTURARkWYqCiIi0kxFQUREmqkoiIhIMxUFERFppqIgIiLNVBRERNLAb/61nJnLqzzfjoqCiEiKc85x5ysrmb36I8+3paIgIpLiwpEY0ZgjN9P7SShUFEREUtzuxigAuZlBz7eloiAikuLqGiMA5KmnICIi9Xt6ClnqKYiI9Hj1OnwkIiJ71Ifjh4800CwiItQlegoaUxAREeoTA8056Xz4yMyyzWyOmS00s3fN7H/aaJNlZn81s5VmNtvMyrzKIyKSrvaMKeSl+UBzGDjFOTcBmAicaWZHtWpzNbDDOXcI8GvgFx7mERFJS3XdYUzBxdUmHoYSN9eq2XnAQ4n7jwOnmpl5lUlEJB11m7OPzCxoZpXAVuBl59zsVk0GAesAnHMRYBfQ18tMIiLppr4xSmYwQCjo/TCwp1twzkWdcxOBwcBUMxvXqklbvYLWvQnM7Bozm2tmc6uqvJ8lUEQkldQ3RpJy4Rok6ewj59xO4FXgzFYvrQeGAJhZBlAEbG/j/dOdcxXOuYqSkhKP04qIpJa6cDQpp6OCt2cflZhZr8T9HOA04L1WzZ4Grkzcvwh4xTn3iZ6CiEhPVt8YScrpqABelp4BwENmFiRefB5zzj1rZj8G5jrnngbuA/5kZiuJ9xA+72EeEZG0VN8YJS/di4JzbhEwqY3nf9DifgNwsVcZRES6g/rGSFJORwVd0SwikvLqwtGkXLgGKgoiIilvd1OUHPUUREQE4lc0J2tMQUVBRCTF1TdGNaYgIiLgnKOuMZKUKS5ARUFEJKWFIzGcS85SnKCiICKS0vbMkJr2VzSLiEjnJXOGVFBREBFJaR8XBfUURER6vLrEUpwaUxAREerDiaU41VMQEZHmnoLGFEREZLcGmkVEZI89PYW8LB0+EhHp8faMKainICIiOiVVREQ+Vt8YISsjQDBgSdmeioKISAqra4wkbTwBVBRERFJafThKTig54wmgoiAiktLqG5O3FCeoKIiIpLT4Wgo6fCQiIqinICIiLdQ3RskJqacgIiLET0ntFj0FMxtiZjPMbJmZvWtmN7XRpreZ/d3MFpnZHDMb51UeEZF0VBeOdpsxhQjwdefcWOAo4HozK2/V5rtApXPucOAK4Lce5hERSTv1jZGkTXEBHhYF59wm59z8xP0aYBkwqFWzcuDfiTbvAWVm1s+rTCIi6SQWc+xuipLXHYpCS2ZWBkwCZrd6aSFwQaLNVGAYMDgZmUREUl1DJIpzkNudrmg2s3zgCeBm51x1q5dvA3qbWSVwA7CA+GGn1p9xjZnNNbO5VVVVXkcWEUkJdc2rriWvp+Bp+TGzEPGC8Gfn3JOtX08UiasSbQ1Ynbi1bjcdmA5QUVHhvMwsIpIq9iywk9MdBpoTv+TvA5Y55+5op00vM8tMPPwSMLON3oSISI/UvMBON+kpHAtcDixOHB6C+NlGQwGcc/cAY4GHzSwKLAWu9jCPiEhaqd+zPnMSxxQ825JzbhawzwnAnXNvAaO8yiAiks7qkrzqGuiKZhGRlPXxqmsqCiIiPV5985hCNxhoFhGRzqnb01PoDnMfiYhI5+zeM9CsnoKIiOwZaNZynCIiQn1jhJxQkGBgnydydqnk9UnSUH1jhG01jVTVhqluaGLy0N4U5YT8jiUiPURdYzSpZx6BisInRKIxnpi/njtfWcn6Hbv3ei0vM8jnjhjCVccMZ2jf3C7fdm04QlVNmPrGCKUF2fTNyyTQ6i+EWMxhBvELxg9MQ1OUj+oayQwGyMwIkJURIByJUb27iV27m6hpiHBIaT4lBVld9ZVEpBN2N0aTOsgMKgrNYjHHs4s38ZuXl7NqWx0TBhdx6ZHDKM7PpLggi1AgwBPz1/Ontz7koTfXcOLoEkoLsskIGqFgADNoisZoijiaojGizhE0IxCw5v8GDIIBw4Cahgjb6xvZUd/EjrpGttWGm89J3iMUNEoLsskOBagLR6kNR6gNRwgGjLzMIAXZIfKzMuiTl0nf/EyK87MoKciiMCdEUU6IwuwMmqKOuR9u553V21m8YRdN0f1PHVU+oJDjRxdz7MhiBvbKplduJr1yQmQEA0RjjrrGCPXhKBlBo29e5kEVKBHZv7pwJKmno0IPKgr/WrqF7/x9MRkBIxgwMgKGI16JdzdF2d0YJRJzjOlXwPTLp3B6eb9P/LI7blQx3z7zUB56aw3PL97E0k3VRKLxIuAchDICZAYDhDKMgBnRmCMWc0SdI+bihSeWuF+QnUHv3Ex65YYY1ieXkoL4L/TSgixyQkG21oTZXN3All0NNESiFGSFyMvKID8rSMzFexXVDfG/7rfXNbJkwy621TZSG/7EJLOEgsb4QUVMO244w/vm0RSNEY7EaIzGyAwGKEoUkdzMDBau38nrK6q4f9Zq/ve1VXt9TmZGgMZIbK/nCrIzGFGSz4jiPEoKsshK9ECyMoKUFGQxqHcOg3rlUFqQhSPeW2loihFzjrysDPIyg3vt50g0Rl1jlMxggJwkd5tFUk29Dh95p7Qwi9PGlhKJOqIxRyQW/4s5NzNIdihITmaQwwYW8ulxA/Y5qNO/KJtvn3ko3z7z0GRFPyANTVGqG5oSh4QigOOwgUVkd/DsheNGFXP9yYdQG46waN1OqmrD7KxvYkd9I7sbo+RkBsnPyiA3M4PdTVFWb6tl9bY63l71ETvrm5rnf++oYMDIz8ogFDRqwxEamj4uOtmhAH1yM+mdl8nIknzGDypi3KAixg0qpCBbYzvS/cVXXVNPwROHD+7F4YN7+R3Dc9mheJErLcju1OfkZ2VwzCHFB/w+5+IFd3dTlK3VYTbs3M2GHbvZUt1ARsAS+QIEAkZtQ4SahniPpynqyM8KJnpD8cNeO+ob2Z44tPbOmu08vXBj83YGFGUzvDiPESV5lPXNY3DvHAYU5TCwVw7F+TqkJd1DfWOU4vzkjvH1mKIgyWFmhBLjLIXZIQ4pze+yz95WG2bxhl0s3VjNB1W1rKqq4+nKjVQ37H3ILC8zyNgBhZQPLOSwgYVMHtqbQ0rzVSgk7dQ1RshL4gypoKIgaaQ4P4uTx5Ry8pjS5uecc+ysb2LDzt1s2tXAxp27WVVVy7sbq3li3noefis+eD+gKJvjRxVz/KgSTjm0NOk/aCIHoz4cTfrYmn4yJK2ZGb3z4uMO4wYV7fVaLOZY81Eds1dv5/UVVbywZDOPzV1PUU6IK44expXHlCW9ay5yIOobo0ldYAdUFKQbCwQsfmZUST5fmDqUSDTGvA93cP8bq7lrxkqmz1zFxRWDOXv8QCYP60VWhs52ktQRTYzNaaBZxCMZwQBHjujLkSP68kFVLX+cuYrH3lnPI2+vJTsUYOrwvhx/SDGfHt+fwb27/uJEkQOxuyl+6DNPF6+JeG9kST63XXg4t5w9ltmrtjNr5TbeWLmNnz23jJ89t4wjynpz7sRBnD1+AH3yMvf/gSJdbM9aCjnqKYgkT0F2iNPK+3FaeT8A1n5UzzOLNvJU5Qa+/48l3PbcMn547mFcPGWwzl6SpKpPzJCa7DEFzZIq0sLQvrlcf/IhvPTVE3nuxuMZP7iIbz2+iP/60zw+qg37HU96kDof1lIAFQWRdpUPLOTRLx3FLWeN5dX3q/jUb2by9MKNRGMHcMm2yEHyY31mUFEQ2adAwPjPE0bw9A3HUpyfxY1/WcBJv5zBg2+spq6NeaZEusqeopDsgWYVBZEOOLR/If+88XjuuWwypQXZ/OiZpRx967/5/YyVhCPR/X+AyAGqD+vwkUhKCwaMM8cN4InrjuHJLx/D1OF9uf3F9/nUr2cy4/2tfseTbqZuT0+huxQFMxtiZjPMbJmZvWtmN7XRpsjMnjGzhYk2V3mVR6QrTR7am3uvrODhaVMJBIyrHniHLz00l6oaDUZL19jdfEpq9zl8FAG+7pwbCxwFXG9m5a3aXA8sdc5NAE4CfmVmOilc0sYJo0t44aYT+O9PH8qslVVcePebrN5W53cs6QbqutuYgnNuk3NufuJ+DbAMGNS6GVBg8RPA84HtxIuJSNrIzAhw7Ykj+b9rjqY2HOGiu99k0fqdfseSNFcfjmAG2UmefiUpYwpmVgZMAma3eukuYCywEVgM3OSciyGShiYO6cXj1x5NTmaQz09/m5nLq/yOJGmsrjFKTij4iXXaveZ5UTCzfOAJ4GbnXHWrlz8FVAIDgYnAXWZW2MZnXGNmc81sblWVftAkdY0oyefJ645hWN88pj34Dk/OX+93JElT8aU4kz/phKdFwcxCxAvCn51zT7bR5CrgSRe3ElgNfGKdS+fcdOdchXOuoqSkxMvIIp1WWpjNX//rKI4o68PXHlvIPa99gDuQNUpFiM99lOzxBPD27CMD7gOWOefuaKfZWuDURPt+wBhgVTttRdJGYXaIB6cdwTkTBnLb8+/xP88s1ZXQckDqwv70FLzc4rHA5cBiM6tMPPddYCiAc+4e4CfAg2a2GDDg2865bR5mEkmarIwgv71kIv0Ksrh31mq21jRwx+cmkh3Sug2yf7ubIkmf4gI8LArOuVnEf9Hvq81G4AyvMoj4LRAwvveZcvoXZfOz55axYefb/PGKKZQWZPsdTVJcbThKYXY3G1MQkbgvHT+C/71sCss31/DZu95g6cbW51yI7K22oYkCFQWR7uuMw/rzt2uPJubgonveZMZ7mhpD2lcbjlCQFUr6dlUURJJo3KAinvrKsYwoyePaR+axYO0OvyNJiqptiJCvnoJI99evMJuHpx1Jv8Js/vPheWzYudvvSJJiojFHXWOU/CwVBZEeoU9eJvddWUG4KcrVD75DrdZmkBb2/HvQmIJIDzKqXwF3XTqZFVtrufn/Fug6BmmW8kXBzEaaWVbi/klmdqOZ9fI2mkj3d+LoEn50Tjn/WraV7z+1hJgKgwA1DU0A5KfwQPMTQNTMDiF+lfJw4FHPUon0IJcfXcZ1J43k0dlr+cbjC4lENSdkT1fb4F9PoaNbjDnnImZ2PvAb59ydZrbAy2AiPcm3PjWGnFCQO15eTkNTlN9cMonMDB3d7alqEoeP/Dj7qKNbbDKzLwBXAucknkt+v0akmzIzbjx1FLmZQX76z2U0NM3ju2cdyvDifIJJnjpZ/NfcU/Dh7KOObvEq4FrgZ8651WY2HHjEu1giPdOXjh9BTmaQ7/1jCa+8t5XczCCHDSxk0tDeXHrkUIb1zfM7oiRBTUOK9xScc0uBGwHMrDdQ4Jy7zctgIj3VpUcO49iRxcz7cAeLN+xi8YZdPPjGGu59fRXnTRzEl08ayah+BX7HFA/VhuMDzQXZyT8g06GiYGavAucm2lcCVWb2mnPuax5mE+mxyorzKCvO48IpgwHYWt3AvbNW88jbH/KPyg2cN2Egt15weNIXdZfkqG2IL8WZ68OMuh0dySpKrJp2AfCAc24KcJp3sUSkpdLCbL571lhmffsUrjtxJE8t3Mi0B9+hThe9dUvVDRHyMzOSvhQndLwoZJjZAOBzwLMe5hGRfeiTl8m3zjyU31wykTlrtnPl/XOaz2mX7qM2HPHldFToeFH4MfAi8IFz7h0zGwGs8C6WiOzLeRMHcecXJlG5bieX3TeHXfUqDN2JX5PhQQeLgnPub865w51z1yUer3LOXehtNBHZl7PGD+Duy6awbGM1Vz04h8aILnrrLmrCTb5Mhgcdn+ZisJn93cy2mtkWM3vCzAZ7HU5E9u308n7ccckE5q/dye0vvud3HOkitQ0RX848go4fPnoAeBoYCAwCnkk8JyI++8zhA7n8qGH88fXV/HvZFr/jSBeoCaf44SOgxDn3gHMukrg9CJR4mEtEDsAtZ4+lfEAhX//bQjZqfYa0V9sQ8eVqZuh4UdhmZpeZWTBxuwz4yMtgItJx2aEgv790Mk2RGDf8ZQFNmlQvrdU0pP7ZR9OIn466GdgEXER86gsRSRHDi/O49cLDmffhDn7zr+V+x5GDFInG2N0U9WXabOj42UdrnXPnOudKnHOlzrnPEr+QTURSyLkTBnLRlMHc89oqFq/f5XccOQh14Sjgz7xH0LmV1zTFhUgK+v7Z5fTNy+Sbjy/UaappqLphz7xH6VcU9nn9tZkNMbMZZrbMzN41s5vaaPNNM6tM3JaYWdTM+nQik0iPV5Qb4mfnj+e9zTXc/eoHfseRA9S8FGeKDzS3ZX/rBkaArzvnxgJHAdebWfleH+Dc7c65ic65icB3gNecc9s7kUlEiF+/cN7Egdw1YwXvba72O44cgFofF9iB/RQFM6sxs+o2bjXEr1lol3Nuk3NufuJ+DbCM+DUO7fkC8JcDzC8i7fjhOYdRmB3iW48v0tlIaeTj9Zn9KQr73KpzrksmbTezMmASMLud13OBM4GvtPP6NcA1AEOHDu2KSCLdXp+8TH583jiuf3Q+k378MpOG9mLy0N4cN6qYI8p0lDZV1TSvz5zCZx91hpnlA08ANyem327LOcAb7R06cs5Nd85VOOcqSkp0zZxIR501vj9/vKKCz04ayLbaRn73ygouvuctXnx3s9/RpB3NYwo+HT7ydKtmFiJeEP7snHtyH00/jw4diXQ5M+P08n6cXt4PiJ/ZcuEf3uTnzy3j5DGlZGZ4/nehHKA96zOn9IR4B8PMDLgPWOacu2Mf7YqAE4GnvMoiInGF2SFuOXssH35Uz8NvrfE7jrShpiFCwCDXp1X1vPwz4VjgcuCUFqednmVm15rZtS3anQ+85Jyr8zCLiCScNKaUE0eX8Nt/r2B7XaPfcaSV2nCE/KwM4n9XJ59nRcE5N8s5Z4l1GCYmbs855+5xzt3Tot2DzrnPe5VDRD7plrPHUt8Y5beaDiPl1Pg4bTYkYaBZRFLP6H4FfGHqEB6ZvZaVW2v9jiMt1DQ0+TbIDCoKIj3WV08bTW4oyM/+uRTn9nctqiTLnsNHflFREOmh+uZnceOpo5jxfhVPVW70O44k1Pq4wA6oKIj0aNOOG07FsN58/6klbNDiPClBYwoi4ptgwPj1JROJxRxff6ySWEyHkfxW06DDRyLioyF9cvnhuYfx9qrt3Dtrld9xerzasAaaRcRnF08ZzBnl/fjli8tZtkmzqvqlKRqjoSnm27TZoKIgIsSnw7j1gvEU5oT46l8rtTiPT5qnuFBPQUT81jc/i19cGF+c585XVvgdp0dqXktBPQURSQWnju3HhZMH84dXP2DR+p1+x+lxPp42W0VBRFLED84ppzg/k2/8bSHhSNTvOD1KTfP6zDolVURSRFFOiNsuPJzlW2r57b90GCmZdPhIRFLSyWNKuaRiCPe89gHzPtSy6cni9/rMoKIgIu245TNjGVCUw3/8cTYPvrFaF7YlQbXGFEQkVRVmh/j79cdw9Mi+/OiZpVz5wBw272rwO1a3tueU1IIsjSmISAoqLcjmgS8ewU8/O465a3bwqd/M1FlJHqoNNxEMGNkh/341qyiIyD6ZGZcdNYx/3ngcoWCA21983+9I3VZtQ4SCbP9WXQMVBRHpoBEl+Vx93HBeX7GNJRt2+R2nW/J7MjxQURCRA3DpUUMpyMrg7tc+8DtKt1Tj8wI7oKIgIgegMDvEpUcN4/nFm1izrc7vON1ObUOEQh8vXAMVBRE5QNOOLSMjEGD665pmu6vVhJt8vUYBVBRE5ACVFmZz4ZRBPD5vPVtrdIpqV6rVmIKIpKNrThhJUzTGg2+s8TtKt+L3+sygoiAiB2F4cR6fHtefP731YfMkbtJ51YlTUv3kWVEwsyFmNsPMlpnZu2Z2UzvtTjKzykSb17zKIyJd69oTR1ITjvDXd9b5HaVbCEeiNEb8XXUNvO0pRICvO+fGAkcB15tZecsGZtYL+ANwrnPuMOBiD/OISBc6fHAvppb14YE31hCJaqW2zqoLx6cp77ZjCs65Tc65+Yn7NcAyYFCrZv8BPOmcW5tot9WrPCLS9a4+fjgbdu7mxXe3+B0l7TXPe9QTTkk1szJgEjC71Uujgd5m9qqZzTOzK9p5/zVmNtfM5lZVVXkbVkQ67LSx/RjWN5f7Zun01M6qTozNdPuBZjPLB54AbnbOVbd6OQOYApwNfAr4vpmNbv0ZzrnpzrkK51xFSUmJ15FFpIOCAWPascOZv3Yn8z7c4XectLZnLYXuPKaAmYWIF4Q/O+eebKPJeuAF51ydc24bMBOY4GUmEelaF00ZTGF2BvfPWu13lLTW7Q8fWXyav/uAZc65O9pp9hRwvJllmFkucCTxsQcRSRN5WRl84cihPL9kE+u21/sdJ23VhLv/4aNjgcuBUxKnnFaa2Vlmdq2ZXQvgnFsGvAAsAuYA9zrnlniYSUQ88MVjygiY8dCba/yOkrb29BT8PvvIs60752YB+50U3Dl3O3C7VzlExHsDinI4+/AB/GXOWj53xBBG9yvwO1LaqQn7vxQn6IpmEeki3zhjDHlZGVx+32wdRjoINQ0RQkEjK8PfX8sqCiLSJYb0yeVPVx9JQ1OMy+6brcnyDtCW6gZ65Wb6uuoaqCiISBca07+AB646gq3VYa64bw676jUvUkc453hj5TamlvXxO4qKgoh0rclDezP9iil8UFXLTX9d4HectLBiay1bqsOcMLrY7ygqCiLS9Y4fVcKNp4zi1ferWPuRxhf2Z+by+EwNx4/y/+JcFQUR8cSFUwZjBn9fsMHvKCnvteVVHFKaz8BeOX5HUVEQEW8M7JXD0SP68uSC9Tjn/I6TshqaosxZvZ0TUqCXACoKIuKhCyYP5sOP6pm/VvMitWfO6u2EI7GUGE8AFQUR8dCZ4/qTEwryxHwdQmrPzOVVZGYEOHJ4X7+jACoKIuKh/KwMzhzXn2cXbiQcifodJyXNXFHF1LI+5GQG/Y4CqCiIiMcumDyI6oYIryzTGlqtbd7VwPIttSlz6AhUFETEY8eMLKZfYZYOIbVh5orUORV1DxUFEfFUMGB8duIgXn1/Kx/VhnHOsWZbHc8v3tS82lhPNXN5FaUFWRzaP3UmEPR3Oj4R6REumDyY/525ikvvnc3WmjDb6xoBOPXQUu69ssL3+X78EI05Zq3cxqmH9kup76+iICKeG9O/gNPGlrJ6Wx2nHlrKpKG92bRrN3e+spJH56zl0iOH+R0x6ZZs2MXO+qaUGk8AFQURSZJ7rzxir8exmKNy3U5+8uxSjhrRl5El+T4l88fbqz4C4mMuqURjCiLii0DA+OXFE8gOBfnqXytpisb8jpRUlet2MqRPDiUFWX5H2YuKgoj4pl9hNreeP55F63fxu3+v8DtOUlWu28nEIb39jvEJKgoi4qtPjx/AxVMG8/sZK5nxfs+4lmFLdQObdjUwcUgvv6N8goqCiPjuh+cextgBhVz3yDzmrtnudxzPLVi7E0BFQUSkLflZGTw0bSoDi3K46sF3WLqx2u9Inqpct5NQ0DhsYKHfUT5BRUFEUkJxfhYPXz2V/KwMrrh/Dmu21fkdyTOV63YwdkAh2aHUmO+oJRUFEUkZg3vn8qerpxKNxbj8/tnUhiNttttV30RDU3pOsBeNORav35WSh45ARUFEUswhpQVMv6KCddt3t3lG0o66Rk7/9Wt88YE5abl4z4qtNdQ1RnteUTCzIWY2w8yWmdm7ZnZTG21OMrNdZlaZuP3Aqzwikj6OKOvDJRVDuH/WalZsqdnrtR8+/S5ba8K8vWo7LyzZ7FPCg1eZwoPM4G1PIQJ83Tk3FjgKuN7Mytto97pzbmLi9mMP84hIGvnWmWPIzQzyg6febe4RvLBkM08v3MiNp45idL98bn3+vbRbp6Fy3U6KckIML87zO0qbPCsKzrlNzrn5ifs1wDJgkFfbE5HupW9+Ft8881DeWvURzy7axI66Rr73jyWUDyjkhlMO4Xtnl7N2ez0PvrHG76gHpHLdTiYM6ZVSk+C1lJQxBTMrAyYBs9t4+WgzW2hmz5vZYe28/xozm2tmc6uqqjxMKiKp5D+mDmXcoEJ++s+lfOfJxeysb+SXF08gFAxwwugSTh5Twl2vrOSj2rDfUTukLhxh+ZaalD10BEkoCmaWDzwB3Oyca33y8XxgmHNuAnAn8I+2PsM5N905V+GcqygpSZ3FKETEW8GA8ePzxrGlOswL727mhlNGUd7i3P5bzh5LfVOUX/9ruY8pO27xhl3EHEzqqUXBzELEC8KfnXNPtn7dOVftnKtN3H8OCJlZak0ZKCK+mjy0N/914giOH1XMl08euddrh5QWcNmRQ3l09lre31zTziekjsp18UHmCT2xKFj8gNl9wDLn3B3ttOmfaIeZTU3k+cirTCKSnr7z6bH86eojCQU/+Svr5tNGU5QT4sa/LKC+se3rGlJF5dqdDOubS5+8TL+jtMvLnsKxwOXAKS1OOT3LzK41s2sTbS4ClpjZQuB3wOddOp54LCK+6Z2Xye++MInlW2v47ycWp/S1C/GZUVO3lwAeLrLjnJsF7HN43Tl3F3CXVxlEpGc4flQJ3zhjDLe/+D4Th/Ri2nHD/Y70Ceu217O5OjVnRm1JVzSLSLdw3YkjOb28Hz9/bhlzVqfeTKu3Pf8emRkBThvbz+8o+6SiICLdQiBg/OpzExjaJ5frH53Psk2pM9Pqv5Zu4Z+LN3HTqaMY0ifX7zj7pKIgIt1GYXaIey6fgnOOc+6cxR0vve/7Fc+14Qg/eGoJY/oV8J+O7BBGAAALtElEQVTHj/A1S0eoKIhItzK6XwEvf/VEzpkwkN+9spJz7pzFgrU7fMvzq5feZ1N1Az+/YDyZGan/Kzf1E4qIHKDeeZn8+pKJPPDFI6hpiHDRPW/x9MKNSc9RuW4nD765hsuPGsaUYam3HnNbVBREpNs6+dBSXvzqCUwZ1pub/m8Bj72zbp/tnXP87t8ruqRnEYs5vvvkYvoVZPPNT43p9Ocli4qCiHRrhdkhHrpqKscdUsy3nljEQ2+uabftjPe3csfLy/niA++wupMrv81cUcXSTdV868wxFGSHOvVZyaSiICLdXk5mkHuvrOD08n788Ol3+ePMVZ9o45zj9zM+oH9hNsGAMe3Bd9hZ33jQ23zozTWUFGTxmcMHdiZ60qkoiEiPkJUR5A+XTuas8f259fllzfMQ7TFn9XbmfbiD604ayfTLp7Bhx26ufWQejZHYAW9rzbY6Xl1exX9MHZoWg8stpVdaEZFOCAUD/OLCwyktyObbjy/a6xf+H179gL55mXyuYggVZX34xUXjeXvVdr73j8UHXBgefutDgmZceuTQrv4KnlNREJEepSA7xE8/O473t9Rwz2sfALBkwy5eW17FtOOGk5MZBOD8SYO54ZRDeGzuesb/6EUuvPtNfvLsUl58dzOxWPvzK9WFI/xt7jrOGj+A0sLspHynruTZ3EciIqnqtPJ+fObwAdz1ykrOGt+fu1/7gPysDC47athe7b52+mjGDypizurtVK7bySNvf8h9s1YzblAh3z+7nCNH9P3EZz+5YAM14QhXHlOWpG/TtVQURKRH+tG5hzFr5Tau//MCVmyt4ZoTRlKUs/dZQmbGGYf154zD+gPQFI3x3OJN/OL597hk+tucNb4/3z7zUIb1ja+37Jzj4TfXMH5QEZOHpvbEd+1RURCRHqk4P4vvn13O1/+2kKyMAFd3YGbVUDDAeRMHcUZ5f/74+irufvUDnlu8manD+/DZiYPolRtixdZafnnxhJRdg3l/VBREpMe6YPIgFqzbQVnfPEoKsjr8vpzMIDeeOorPVQzhsbnr+EflBr7798UA9MnL5DOHD/AqsucslRekaEtFRYWbO3eu3zFERJo551iyoZpnF29k4uBefHp86hUFM5vnnKvYXzv1FEREOsnMGD+4iPGDi/yO0mk6JVVERJqpKIiISDMVBRERaaaiICIizVQURESkmYqCiIg0U1EQEZFmKgoiItIs7a5oNrNdwIo2XioCdu3judav73ncVptiYNtBxGsrQ0fbtJevrcdt3e9s9n1l29/r2vcfP+dV/oPd960fe/Fvx8993/J+T9z3+8rX+vVhzrmS/X6acy6tbsD0jj7f8rnWr+953FYbYG5XZjuY/Pt63E7mTmXvSH7t+33vey/zH+y+T8a/HT/3fTLyp/K+70z+9m7pePjomQN4/pl9vP5MB9ocqI68v6P59/W4rfudzd6Rz9C+T7993/qxF/n93Pcd3f6+pPO+78hnHNA20u7wUTKY2VzXgYmjUlE6Zwfl91M6Z4f0zp9K2dOxp5AM0/0O0AnpnB2U30/pnB3SO3/KZFdPQUREmqmnICIizbp9UTCz+81sq5ktOYj3TjGzxWa20sx+Zy3W1zOzG8zsfTN718z+X9embt5Gl2c3sx+Z2QYzq0zczur65M0ZPNn3ide/YWbOzIq7LvFen+/Fvv+JmS1K7PeXzGxg1ydvzuBF/tvN7L3Ed/i7mXmyCLFH2S9O/KzGzMyTY/edyd3O511pZisStytbPL/Pn41OO9jToNLlBpwATAaWHMR75wBHAwY8D3w68fzJwL+ArMTj0jTK/iPgG+m67xOvDQFeBD4EitMlO1DYos2NwD3ptO+BM4CMxP1fAL9Io+xjgTHAq0BFKuVOZCpr9VwfYFXiv70T93vv6zt21a3b9xScczOB7S2fM7ORZvaCmc0zs9fN7NDW7zOzAcR/iN9y8f8TDwOfTbx8HXCbcy6c2MbWNMqeNB7m/zXwLcCzATEvsjvnqls0zUvD/C855yKJpm8Dg9Mo+zLn3Pte5O1s7nZ8CnjZObfdObcDeBk4Mxk/292+KLRjOnCDc24K8A3gD220GQSsb/F4feI5gNHA8WY228xeM7MjPE27t85mB/hK4hDA/WbW27uobepUfjM7F9jgnFvoddA2dHrfm9nPzGwdcCnwAw+ztqUr/u3sMY34X6nJ0pXZk6kjudsyCFjX4vGe7+L5d+xxazSbWT5wDPC3Fofistpq2sZze/6yyyDepTsKOAJ4zMxGJCq3Z7oo+93ATxKPfwL8ivgPuOc6m9/McoFbiB/GSKou2vc4524BbjGz7wBfAX7YxVHb1FX5E591CxAB/tyVGdvTldmTaV+5zewq4KbEc4cAz5lZI7DaOXc+7X8Xz79jjysKxHtHO51zE1s+aWZBYF7i4dPEf3m27B4PBjYm7q8HnkwUgTlmFiM+d0mVl8HpguzOuS0t3vdH4FkvA7fS2fwjgeHAwsQP2WBgvplNdc5tTvHsrT0K/JMkFQW6KH9iwPMzwKle/xHUQlfv+2RpMzeAc+4B4AEAM3sV+KJzbk2LJuuBk1o8Hkx87GE9Xn9HLwZcUu0GlNFi8Ad4E7g4cd+ACe287x3ivYE9AzpnJZ6/Fvhx4v5o4t08S5PsA1q0+Srwf+m071u1WYNHA80e7ftRLdrcADyeTvseOBNYCpR4mdvLfzd4ONB8sLlpf6B5NfEjEr0T9/t05Dt2+jt4/T/X7xvwF2AT0ES8yl5N/K/NF4CFiX/kP2jnvRXAEuAD4C4+vtgvE3gk8dp84JQ0yv4nYDGwiPhfVwO8yO5V/lZt1uDd2Ude7PsnEs8vIj4fzaB02vfASuJ/AFUmbp6cPeVR9vMTnxUGtgAvpkpu2igKieenJfb5SuCqA/nZ6MxNVzSLiEiznnr2kYiItEFFQUREmqkoiIhIMxUFERFppqIgIiLNVBSkWzCz2iRv714zK++iz4pafObUJWb2zP5mHzWzXmb25a7YtkhrOiVVugUzq3XO5Xfh52W4jyd/81TL7Gb2ELDcOfezfbQvA551zo1LRj7pWdRTkG7LzErM7AkzeydxOzbx/FQze9PMFiT+Oybx/BfN7G9m9gzwkpmdZGavmtnjFl9H4M975q5PPF+RuF+bmOhuoZm9bWb9Es+PTDx+x8x+3MHezFt8PPlfvpn928zmW3z+/PMSbW4DRiZ6F7cn2n4zsZ1FZvY/XbgbpYdRUZDu7LfAr51zRwAXAvcmnn8POME5N4n4TKU/b/Geo4ErnXOnJB5PAm4GyoERwLFtbCcPeNs5NwGYCfxni+3/NrH9/c5Pk5jL51TiV5oDNADnO+cmE1/D41eJovTfwAfOuYnOuW+a2RnAKGAqMBGYYmYn7G97Im3piRPiSc9xGlDeYobKQjMrAIqAh8xsFPEZJkMt3vOyc67lnPhznHPrAcyskvjcNrNabaeRjycWnAecnrh/NB/Pdf8o8Mt2cua0+Ox5xOfOh/jcNj9P/IKPEe9B9Gvj/WckbgsSj/OJF4mZ7WxPpF0qCtKdBYCjnXO7Wz5pZncCM5xz5yeOz7/a4uW6Vp8RbnE/Sts/M03u48G59trsy27n3EQzKyJeXK4Hfkd8zYUSYIpzrsnM1gDZbbzfgFudc/97gNsV+QQdPpLu7CXiaxYAYGZ7pjAuAjYk7n/Rw+2/TfywFcDn99fYObeL+DKd3zCzEPGcWxMF4WRgWKJpDVDQ4q0vAtMS8/djZoPMrLSLvoP0MCoK0l3kmtn6FrevEf8FW5EYfF1KfMpzgP8H3GpmbwBBDzPdDHzNzOYAA4Bd+3uDc24B8Rk1P098EZsKM5tLvNfwXqLNR8AbiVNYb3fOvUT88NRbZrYYeJy9i4ZIh+mUVBGPJFaK2+2cc2b2eeALzrnz9vc+ET9pTEHEO1OAuxJnDO0kScueinSGegoiItJMYwoiItJMRUFERJqpKIiISDMVBRERaaaiICIizVQURESk2f8HIw6IdeKxeBUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case pretraining the encoder gives a *worse* result.\n",
    "\n",
    "Maybe it's because the language model was on the entire (unbalanced) dataset? Or wasn't well trained enough?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:29 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.623937</th>\n",
       "    <th>2.702015</th>\n",
       "    <th>0.302083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.410758</th>\n",
       "    <th>2.412743</th>\n",
       "    <th>0.337500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.200685</th>\n",
       "    <th>1.927255</th>\n",
       "    <th>0.393750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.046425</th>\n",
       "    <th>2.312107</th>\n",
       "    <th>0.316667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>1.865316</th>\n",
       "    <th>2.074282</th>\n",
       "    <th>0.393750</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>1.777486</th>\n",
       "    <th>2.281461</th>\n",
       "    <th>0.360417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>1.689011</th>\n",
       "    <th>2.557259</th>\n",
       "    <th>0.302083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.596411</th>\n",
       "    <th>2.346404</th>\n",
       "    <th>0.370833</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.566185</th>\n",
       "    <th>2.441514</th>\n",
       "    <th>0.341667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.473553</th>\n",
       "    <th>1.770901</th>\n",
       "    <th>0.429167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.411398</th>\n",
       "    <th>1.677306</th>\n",
       "    <th>0.502083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.352370</th>\n",
       "    <th>1.966482</th>\n",
       "    <th>0.404167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>1.323129</th>\n",
       "    <th>3.021722</th>\n",
       "    <th>0.310417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>1.271584</th>\n",
       "    <th>2.182109</th>\n",
       "    <th>0.389583</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>1.224382</th>\n",
       "    <th>1.864778</th>\n",
       "    <th>0.450000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>1.196533</th>\n",
       "    <th>1.896098</th>\n",
       "    <th>0.447917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>1.156429</th>\n",
       "    <th>1.960691</th>\n",
       "    <th>0.435417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>1.108229</th>\n",
       "    <th>1.840390</th>\n",
       "    <th>0.456250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>1.136375</th>\n",
       "    <th>2.003758</th>\n",
       "    <th>0.429167</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>1.125715</th>\n",
       "    <th>1.961390</th>\n",
       "    <th>0.443750</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(20, max_lr=2e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## fastai: Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a bit of tuning we can make a much smaller model that trains faster and is almost as good"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:10 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.748441</th>\n",
       "    <th>2.779718</th>\n",
       "    <th>0.062500</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.560826</th>\n",
       "    <th>2.709974</th>\n",
       "    <th>0.122917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.413826</th>\n",
       "    <th>2.418473</th>\n",
       "    <th>0.325000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>2.248409</th>\n",
       "    <th>1.827642</th>\n",
       "    <th>0.397917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>2.131097</th>\n",
       "    <th>1.928447</th>\n",
       "    <th>0.385417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>2.005232</th>\n",
       "    <th>1.580826</th>\n",
       "    <th>0.510417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>1.831701</th>\n",
       "    <th>1.488690</th>\n",
       "    <th>0.510417</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.711539</th>\n",
       "    <th>1.291139</th>\n",
       "    <th>0.600000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.528764</th>\n",
       "    <th>1.403310</th>\n",
       "    <th>0.572917</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.353717</th>\n",
       "    <th>1.199514</th>\n",
       "    <th>0.627083</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.200141</th>\n",
       "    <th>1.201450</th>\n",
       "    <th>0.658333</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.080066</th>\n",
       "    <th>1.182234</th>\n",
       "    <th>0.641667</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>0.959931</th>\n",
       "    <th>1.155729</th>\n",
       "    <th>0.650000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>0.873939</th>\n",
       "    <th>1.152237</th>\n",
       "    <th>0.656250</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>0.815623</th>\n",
       "    <th>1.167501</th>\n",
       "    <th>0.654167</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = text_classifier_learner(data, bptt=30, emb_sz=200, nh=300, nl=2)\n",
    "learn.fit_one_cycle(15, max_lr=1e-2, moms=(0.2, 0.1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('fastai_min')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysing the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('fastai_min')\n",
    "None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob, target, losses = learn.get_preds(with_loss=True)\n",
    "pred = np.array([data.classes[_] for _ in prob.argmax(dim=1)])\n",
    "target = np.array([data.classes[_] for _ in target])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = list(learn.data.valid_dl)[0]\n",
    "y = np.array([data.classes[_] for _ in y])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(480, 480)"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(y), len(prob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = np.array([''.join([vocab.itos[x] for x in l if x != 1][1:]) for l in zip(*x)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I certainly think we *could* do better, but let's call it good enough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('simonis', 'Greek', 'Dutch', tensor(6.7631)),\n",
       " ('dam', 'Korean', 'Vietnamese', tensor(6.4027)),\n",
       " ('jelen', 'Dutch', 'Polish', tensor(6.1876)),\n",
       " ('cha', 'Vietnamese', 'Korean', tensor(6.1856)),\n",
       " ('hayden', 'Dutch', 'Irish', tensor(6.1785)),\n",
       " ('chmiel', 'French', 'Polish', tensor(5.8600)),\n",
       " ('blanxart', 'English', 'Spanish', tensor(5.8351)),\n",
       " ('chicken', 'Dutch', 'Czech', tensor(5.6187)),\n",
       " ('attia', 'Spanish', 'Arabic', tensor(5.4960)),\n",
       " ('ton', 'Korean', 'Vietnamese', tensor(5.4933))]"
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_val, idx = losses.topk(10)\n",
    "list(zip(names[idx], pred[idx], target[idx], loss_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [],
   "source": [
    "confuse = sklearn.metrics.confusion_matrix(target, pred, labels=data.classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [],
   "source": [
    "def most_confused(n):\n",
    "    top = []\n",
    "    for i, row in enumerate(confuse):\n",
    "        for j, cell in enumerate(row):\n",
    "            if i == j:\n",
    "                continue\n",
    "            if cell >= n:\n",
    "                top.append([data.classes[i],data.classes[j], cell])\n",
    "    return sorted(top, key=lambda x: x[2], reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the confusion is between similar language families:\n",
    "- Vietnamese and Korean and Chinese\n",
    "- Czech and Polish\n",
    "- Spanish and Italian\n",
    "\n",
    "This is a good sign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Vietnamese', 'Chinese', 13],\n",
       " ['Korean', 'Chinese', 11],\n",
       " ['Spanish', 'Italian', 8],\n",
       " ['Polish', 'Czech', 6],\n",
       " ['Chinese', 'Korean', 4],\n",
       " ['Czech', 'German', 4],\n",
       " ['English', 'French', 4],\n",
       " ['English', 'German', 4],\n",
       " ['German', 'English', 4],\n",
       " ['Vietnamese', 'Korean', 4],\n",
       " ['Arabic', 'German', 3],\n",
       " ['Czech', 'Spanish', 3],\n",
       " ['Dutch', 'English', 3],\n",
       " ['English', 'Dutch', 3],\n",
       " ['German', 'Dutch', 3],\n",
       " ['Irish', 'Chinese', 3],\n",
       " ['Korean', 'Vietnamese', 3],\n",
       " ['Polish', 'German', 3],\n",
       " ['Spanish', 'French', 3],\n",
       " ['Vietnamese', 'Irish', 3]]"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_confused(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAGYCAYAAAB20FQvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzsnXeYFMX2/j+HRclBgghIUHJQFlkQXZKAYMCIYkAFwQz6FSPXBOpVMYerovhTAQN6DdcsQRAQFclRCRfBK0paVJKEZTm/P7pXhnXDzHTvTE9zPjzz7Ex1V83bNcOereqq84qqYhiGYRjFTYlkCzAMwzAODizgGIZhGAnBAo5hGIaRECzgGIZhGAnBAo5hGIaRECzgGIZhGAnBAo5hGIaRECzgGIZhGAnBAo5hGIaRECzgGIZhGAmhZLIFGIZhGPGRVrGe6t6dntvRnZsmqOopPkgqFAs4hmEYKYru3UWpphd6bmfX/H9V80FOkdiUmmEYhpEQbIRjGIaRqgggkmwVUWMBxzAMI5WR1JmoSh2lhmEYRkpjIxzDMIxUxqbUDMMwjOJHUmpKzQKOYRhGKpNCI5zUCY2GYRhGSmMjHMMwjFRFsCk1wzAMIxGITakZhmEYRl5shGMYhpHK2JSaYRiGkRBSaErNAo5hGEbKklr7cFJHqWEYhpHS2AjHMAwjVbFs0YZhGEbCsCk1wzAMwzgQG+EYhmGkLKm1aMACjmEYRipTwu7hGIZhGMVNiuVSSx2lhmEYRkpjIxzDMIxUxpZFG4ZhGMWPLRowDMMwEkUKjXBSJzQahmEYKY2NcAzDMFIZm1IzDMMwih0xx0/DSBlEpIyIfCwiW0TkHQ/t9BWRiX5qSxYi0lFElidbhxE+LOAYKYGIXCwic0Rku4isE5HPRaSDD02fB9QAqqrq+fE2oqpvqGoPH/QUKyKiItKwsHNU9StVbZIoTYZHpIT3R4KwgGMEHhG5CXgKeBAnONQFngfO8qH5esAKVd3rQ1spj4jYNHuqkTut5uWRICzgGIFGRCoB9wGDVPV9Vd2hqtmq+rGq3uqeU0pEnhKRX93HUyJSyj3WRUTWisjNIrLRHR1d7h67F7gHuMAdOQ0UkeEi8nrE+9d3RwUl3df9ReRHEdkmIqtFpG9E+YyIeieKyGx3qm62iJwYcWyqiNwvIl+77UwUkWoFXH+u/tsi9J8tIqeJyAoR+U1E7og4v52IfCsif7jnPisih7rHprunLXSv94KI9m8XkfXAq7llbp0G7nsc576uJSJZItLF0wdr+ITYCMcwfOQEoDTwn0LOuRNoD6QDrYB2wF0Rx48AKgG1gYHAcyJymKoOwxk1va2q5VX15cKEiEg54BngVFWtAJwILMjnvCrAp+65VYEngE9FpGrEaRcDlwOHA4cCtxTy1kfg9EFtnAD5EnAJ0AboCNwjIke75+YAQ4BqOH3XDbgOQFU7uee0cq/37Yj2q+CM9q6KfGNVXQXcDrwhImWBV4HRqjq1EL2GkS8WcIygUxXIKmLKqy9wn6puVNVNwL3ApRHHs93j2ar6GbAdiPcexT6gpYiUUdV1qro0n3NOB1aq6muquldVxwHLgDMiznlVVVeo6k7g3zjBsiCygQdUNRt4CyeYPK2q29z3XwocC6Cqc1V1pvu+a4AXgc5RXNMwVd3t6jkAVX0JWAl8B9TECfBGULApNcPwjc1AtSLuLdQCfop4/ZNb9lcbeQLWn0D5WIWo6g7gAuAaYJ2IfCoiTaPQk6updsTr9THo2ayqOe7z3ICwIeL4ztz6ItJYRD4RkfUishVnBJfvdF0Em1R1VxHnvAS0BP6lqruLONdIFLnZom1KzTB84VtgF3B2Ief8ijMdlEtdtywedgBlI14fEXlQVSeo6sk4f+kvw/lFXJSeXE2/xKkpFkbi6GqkqhWBO3B+LRWGFnZQRMrjLNp4GRjuThkaRsxYwDECjapuwblv8Zx7s7ysiBwiIqeKyCPuaeOAu0Skunvz/R7g9YLaLIIFQCcRqesuWPhH7gERqSEiZ7r3cnbjTM3l5NPGZ0Bjdyl3SRG5AGgOfBKnplioAGwFtrujr2vzHN8AHP23WoXzNDBXVa/AuTf1gmeVhk/YogHD8BVVfQK4CWchwCbgZ2Aw8IF7yj+BOcAiYDEwzy2L570mAW+7bc3lwCBRArgZZwTzG869kevyaWMz0Ms9dzNwG9BLVbPi0RQjt+AsSNiGM/p6O8/x4cAYdxVbn6IaE5GzgFNwphHB+RyOy12dZwSAFLqHI6qFjqYNwzCMgFKicj0t1fmOok8sgl0fXTNXVTN8kFQoNsIxDMMwEoLtKjYMw0hlUih5pwUcwzCMVEVSy/EzdZQahmEYKY2NcIqZQytU1rJVa3pqo0HVcp7q7/NhXUgJj6P2IGjIzvEuIs2jCK/XAP70pRa+9aZIpMitPUUThO+U135I8zid9dNPa8jKyvLWSAKm1ESkDjAWZ1/aPmCUqj4tIsOBK3FWjwLc4WbzyBcLOMVM2ao16XznWACu71SfjLqV2bIzmxveczKi3Nq1AbUqlwag3KFp7NiTw5D3D8yW8vblbfNte+KE8dxy0/+Rk5ND/wFXcOttQ/M9b3d2fltFYNDVVzBh/KdUr344385ZWOh1lDokLeU1bNr69w3yu3bt4vxe3dmzZzd79+7ltDPP4eah9xSooWKZv/+XSeQ1QP59GYsGgN179x3w+pe1PzP46gFs2rCeEiVKcEn/K7jquusLvo6Sf58ciVVDfn2Rav1Qscwh+ZZHex2Zx3tfGCaJuYezF7hZVeeJSAVgrohMco89qaqPRdOITaklkMkrsrj38xUHlD06ZRVD3l/KkPeX8u3q35m5+veo2srJyeHGGwbx4cefM3/R97zz1jh++P77mPRcfOllvPvBpzHVCZuGUqVK8dYH45kwfTbjp81i2uRJzJv9XUwakn0NfmgoWbIk9z7wCDPmLOazyTN49aWRLF+WWp+lHxr86Ac/riNaBCfgeH0UhZs3cJ77fBvwAwemaooKCzgJ5Pv129m+u+AclB2OrsL0VZujamv2rFk0aNCQo44+mkMPPZTzL7iQTz7+MCY9mR06cViV+LOUhEGDiFCuvJPGbG92Nnv3Zsf8F2Oyr8EPDTWOqMmx6a0BKF+hAo2aNGX9r7FlB7J+cPDjOoKMiNQHWuMkcwUYLCKLROQVETmssLoWcAJC8yPK88fObNblM+2TH7/++gtHHlnnr9e1ax/JL78kIlVX+DTk5ORwSud2tG5ahw6du9E6o53fMgslCP0Yyf9+WsOSRQs5zvohrn5I6HWITw8nQe6ciMdV5IObV+894EZV3YqTu68BTrbzdcDjhckNVcARkXNcs6z8MvgWVXd7AeX3iUh37+oKp1ODqlGPbgDyyxCRoLnc0GlIS0tj/LRZfLd4FQvnz2b5D/k5DhQfQejHXHZs387ASy/g/hGPUaFixYS+d1j6IbHX4X06zdWWpaoZEY9R+VzDITjB5g1Vfd+91g2qmqOq+3BSKRUanUMVcICLgBnAhXkPiEj+d2uLQFXvUdUvvAorjBICJ9Q/jBk//hZ1ndq1j2Tt2p//ev3LL2upVatWITX8J2waKlWqTPvMTkydPNEveVERhH4EyM7OZsAlF9C7z0WcfuY5CX//sPRDoq8jEfdwxDnpZeAHN7dhbnnkEtxzgCWFtROagOMO9TJxHB0vdMu6iMiXIvImTlJHROQDEZkrIkvzDhtF5HERmScik0Wkuls2WkTOc5+3FZFvRGShiMxyV2t4plXtiqzdspPNO7KjrpPRti3//e9K1qxezZ49e3jn7bc4vdeZfsg5qDRsztrEli1/ALBr505mTJtCg0bxerPFRxD6UVUZMugqGjVpyjWDb0zoe+cSln4IwnUUA5k4poZdRWSB+zgNeEREFovIIuAkHLfZAglNwMHxSxmvqiuAvzzYcYZ4d6pqc/f1AFVtA2QAN8h+299ywDxVPQ6YBgyLbFwcX/i3gf9T1VZAd/abYUXFzScdzcNnNaN25dK8fFErujdxfLE6NqjKV6uiH92As5rmyaef5YzTe5J+TDN6n9+H5i1axNTGwH596dGlAytXLKd5w3qMHf3KQadh44b1XHhWT3p0zKBX90w6dulG956npdQ1+KFh1sxveOetN5gx/Uu6ZmbQNTODLyZ8nlANYekHP64jFhK0Sm2GqoqqHquq6e7jM1W9VFWPccvPVNV1hWoNS7ZoEfkUeEpVJ4nIDUAdHO+OYap6UsR5w3GGfgD1gZ6qOlNEcoBSqrpXHH/491U1XURG46SoXw68oKqZUWi5CtcbvkyVI9qcPOIjT9dW0D6caCloD0wsFLR/JJU05LcPJ1by24cTC16vAfzpy7z7T2Ilv304MbcRgO+U134oaB9OtGQen8HcuXPivsGTVuUoLd/zPk8aALa+dVlCskWHYuOnO0rpiuM1r0AajovhZzgOjrnndcEZmZygqn+KyFSgdAHN5o3Ekk9Z/hWdG26jACrXbxaOiG4YhuGRsEypnQeMVdV6qlpfVesAq4EOec6rBPzuBpumQPuIYyXcdsAxsJqRp+4yoJaItAUQkQoiEoqAbRhGiuLfsuiEEJZfmBcBI/KUvYdjr7sqomw8cI17g2s5MDPi2A6ghYjMBbYAF0Q2pqp7xLEK/peIlMG5f9Mdx2bYMAwj4QjR3YMJCqEIOKraJZ+yZ4Bn8pTtBk4toI3y7tO785T3j3g+mwNHRYZhGEkllQJOWKbUDMMwjIATihGOYRjGwUoqjXAs4BQzDaqW87ysuepFr3qqv3nc5Z7q+4Efy4G9Ur1iqWRL8AU/+tJrG2t/i2kLWr4cWaWMp/pB6AevS7P98PRJpYBjU2pJYuKE8Rzbogktmjbk0UfyrnfIn9pVy/HZsFOY++Q5zH7ibK47zdnLesf56ax8sQ/fPnom3z56Jj1bH1lsGvxuwzSESwM4yVDP7NaeK/uemxQNye6HQVdfQcN6NTkho1XMdcOOBZwkEK9fRk7OPu4YO5s2Q/7DSXd8wlU9m9L0yEoAPPvJ95xw60eccOtHTJi/ttg0+NmGaQiXhlzGvPQcDRrFnD83NP3g1ZMnJlJsWbQFnCQQr1/G+j92smC1k1F6+669LP9lC7WqxGc/7Ydnh9c2TEO4NACs+3UtUyeNp0/f/jHXDUs/ePXkiZVEpLbxCws4ScAPv4y61cvT6qgqzF7pWIlffUpTvnvsLEZem0nlcocmRIPXNkxDuDQAPHD3bdx2zz8pUSL2Xy1h6odEkbsPxwKOz4jIESLyloisEpHvReQzEblKRD4p4Pz/JyLN8zuWbLz6ZZQrXZI3bzmJ216dxbad2fy/ictoef17tL/1Q9b/sZOHLit6kYIfnh1e2zAN4dIwZeJnVK1WnZatjiv65GLSEIR+MAomJQKO68XwH2CqqjZwMz/fAdQoqI6qXqGqxWMk7hEvfhkl04Q3b+7K21/9yEezfgJg45Zd7NunqMKrX6wgo2H1YtXgVxumIVwa5s2ayeQJn9Iloyk3Xn0ZM7+exs3XDUiohiD0Q6KxEY7/nARkq+oLuQWqugD4CigvIu+KyDIRecMNTojIVBHJcJ9vF5EHxPGxmSkiNdzy6iLynojMdh+Zbnln2e/5MF9c3xsRudU9b5GI3BvvxXjxyxh5bQeW//IH//pkvyvlEZX3Ly89s11dlv78e7Fq8KsN0xAuDbfcdR8zFvyXqXOW8dSLY2mf2ZnHn4/eHiAs/ZBwUmjRQKrsw2kJzC3gWGugBfAr8DWOUVDexJvlgJmqeqeIPAJcCfwTeBp4UlVniEhdYALQDLgFGKSqX4tj7LZLRHoAjXD8dQT4SEQ6qer0WC8m0i8jJyeHfv0HROWXcULTw7m4c0OW/PQb3z7q/CcY/uY8zu9wFMfWr4qq8tOm7dzw4jfFpsHPNkxDuDR4JSz9MLBfX2ZMn8bmzVk0b1iPoXcN47L+0Y/0wkxK+OG4/jZHqeqQPOVdcMzVTnZfjwS+VtXXXeuBW1R1jojsBkqrqroJOE9W1StEZCNOoMqlOtAUGITjmfMGji/OWhF5DCeb9B/uueWBh1T15Xz0/uWHU6du3TYrVv3k6frDsPHTCBdB2PgZBLxu/OySeTzz58Xvh3NI9QZa5az490zlsvHlPuaHE8FS9lsH5CXSVSuH/K8pW/dH1shzSuB44+T93zNCHEO304CZItIdZ1TzkKq+WJTYSD+cNm0ygh/RDcNIWVJpUUOq3MOZApQSkStzC8Txpenssd2JwOCINtPdnw1UdbGqPgzMwRn1TAAGuFNsiEhtETnc4/sbhmF4whYN+Iw7OjkHONldFr0UGM6B02HxcAOQ4S4C+B64xi2/UUSWiMhCHN+bz1V1IvAm8K2ILAbeBSp4fH/DMIyDhlSZUkNVfwX65HPopYhzBkc87xLxvHzE83dxggWqmkUeozW3/PoCNDyNs9DAMAwj6ZgBm2EYhpE4UifeWMAxDMNIWSS1Fg1YwClmclTZujPbUxtelzW/8M2PnuoDXN62nuc2ks2mbXs8t1G9QtF56gojCL5AgOfvpB9Lmr1qKFXS+y3orTv3eqrv1WOpROrECl+wgGMYhpHCpNIIJyVWqYWNX9b+zDmnn0yHjGPo1K4Vo57/V1ztxGMUNW7E7dx9Vlse7n/KX2UfjXyIhy49mUcuP41X7ryGndu2RtWWH0ZTXtvwy+zKi2mYHxqSbTyWzO+knxq8fha7du3ijO4d6NmpLd1ObM3jI+6Lqx2/zOyiwZZFG4VSsmRJ7n3gEWbMWcxnk2fw6ksjWb4sMUZT7U7tzVWPHpi5oHFGB2579XNue/Uzqtc5ii/eGBmVBj+Mpry24ZfZVbymYX5oCILxWDK/k35q8PpZlCpVirc+GM+E6bMZP20W0yZPYt7s72Jqw08zu7BhAScJ1DiiJsemtwagfIUKNGrSlPW/xralKF6jqAat2lGuQuUDypq27UhaSWd2tV7zdP7YtD4qDX4YTXltww8NXkzD/NAQBOOxZH4n/dTg9bMQEcqVd3ZR7M3OZu/e7JhHAH6Z2UVNCiXvtICTZP730xqWLFrIcRntYqpXXEZR3332Ls2O95rAIbXwYhrmB0EwHoskCN/JeDX4QU5ODqd0bkfrpnXo0LkbrQPyf7MgbEotiUj+Rm2NfWh3uIjc4ofGXHZs387ASy/g/hGPUaFixZjqFodR1KTXniMtLY02J5/lqZ1UwqtpmB8EwXgslyB8J71o8IO0tDTGT5vFd4tXsXD+bJb/sLToShEk0sTNj2BjASdORGI3aksW2dnZDLjkAnr3uYjTzzwn5vp+G0XNGv8eS7/5kkvufjKlVr14xatpmB8EwXgMgvGd9KrBTypVqkz7zE5MnTwxpnqpaOKWKEIVcCjYqO1k2W+o9ouIvAogIpeIyCy3/EURSXPLTxGReeIYtk2OaL+5OMZuP4pjmRAXqsqQQVfRqElTrhl8Y1xt+GkU9cN305jy5iiueOhFDi2d+injY8GraZgfBMF4LAjfST80eGVz1ia2bHEcSHbt3MmMaVNo0KhJTG0k2sTNRjjJI1+jNlW9R1XTcbJLbwaeFZFmOHnUMt1jOUBfEamOk5+tt6q2As6PaKop0BPHhG2YiBwSj8hZM7/hnbfeYMb0L+mamUHXzAy+mPB5TG1EGkWlH9OM3uf3icooauy9/8dT153Hxv+tZvh5mcz89N+8//Rwdv+5nZE39+PRgb349+N3RaVhYL++9OjSgZUrltO8YT3Gjo79F7XXNvzQ4BWvGuL9LP1sI5nfST81eP0sNm5Yz4Vn9aRHxwx6dc+kY5dudO95Wkxt+PF5xkIqBZyUMGCLFinAqM09JsDHwHuq+qqIDMaZbtvonlIGGIcTsC5U1b556g/HGT094L7+AcfIbW0+7/WXAduRdeq2mbv0v56uq2KZuOLaX1imAQfLNLAfr7v8vX4n/dAQhkwDmcdnMHdu/AZspWo00loXPeVJA8Cap3slxIAtbCOcpUCbAo4NB9aqau4mFAHGqGq6+2iiqsPd8oKicDRmb6jqKFXNUNWMqtWqxXoNhmEY0WPLopNGvkZtIjIMOBnH/yaXycB54pqoiUgVEakHfAt0FpGjcssTpt4wDCNGUmlKLVS51FRVReQc4CkRGQrsAtYAZYFawCy3cz9S1XtE5C5gooiUALKBQao6050Se98t34gTrAzDMIKFZYtOLoUYteV37tvA2/mUfw58nqdseJ7XLeNXaRiGcfARuoBjGIZxsCBACg1wLOAYhmGkLmYxbUSQJuLLElIv+LGk+fh7v/BUf9JtXTxr8LoEtWIZ+7rn4vU7uTs7J+ka/CDxiXMObsK2Si1lSLb/CcTnHXJEpdKMubItn96UycdDMrk0sy4APY+pwcdDMvn+wR60rB3df+MgeI8EwYMFgvF9SMb3yW8NfrQRlOuIFhHvj0RhAScJBMH/BOLzDsnZt4+HP13G6U98zYXPzaRv+7o0OLwcK9dv54bX5jNnze9RtxUE75EgeLAE4fuQrO+T3xrCch2xkErLoi3gJIEg+J9AfN4hm7bt4ftftwGwY08OqzbtoEbF0vy4aQers/6MqS2R5HuPBMGDJQjfh2R9n/zWEJbriBofRjc2wgk5QfM/iZfah5WmWa0KLPz5j7jbCJL3SLI8WILwfQjC9ykI/eAHQdAQVEIXcEQkx83+vNTN9nyTu4GzqHp3RHHOaBE5z6vGIPmfxEvZQ9N4pm86D328jB2747+BHBTvkWR6sATh+5Ds75NfGsJyHdEiQIkS4vmRKEIXcICdbm60FjgZAk4DhkVRr8iA4xdB8T+Jl5IlhGcuSefjBeuYtHRj0RWiIJneI8n2YAnC9yEIHi5B6Ac/SLQGm1ILCKq6ESdr82Bx6C8iz+YeF5FPRKSLiIwAyrgjozfcY5eJyCJ3lPRaRLOdROQbcTxx4hrtBMH/xAv/PK8FqzbuYPSMnzy1EwTvkSB4sATh+5DM75OfGsJyHbGQSosGQr8xQVV/dKfUDi/knKEiMtj1xUFEWgB34njlZOVJ4FkT6IDjjfMR8G6smiL9MnJycujXf4An/5N42xjYry8zpk9j8+Ysmjesx9C7hnFZ/8LdLo+rV5mzj6vN8nXb+M8NJwDw5ISVHFqyBHed2Ywq5Q7lhf7HsWzdNq545W/WRAewccN6bhp0BTk5Oezbt49eZ/f25D0STz/kerA0a9GSrplOdvY77rmf7j1PjbqNePrRz2vwo41kfZ/81hCW6wgrofLDARCR7apaPk/ZH0AT4FQgQ1UHu+WfAI+p6tTIeiJyPXCEqt6Zp53RwCRVzR0FbVPVCvlo+MsPp07dum1WrPI2EvCKH5v0wrDx06v/Cnj3YAmKH45X/PhOBaEvvF6H12vw6odTplZjbTjwOU8aAJb8s4f54fiBiByN412zEdjLgddcuqBqROeJk+8XJdIPp3q16jEqNgzDiA4nl1rqTKmFOuCIYxf9AvCsOkO5NUC6iJQQkTo4VtG5ZMt+y+jJQB8Rqeq2Y544hmEYHgnjPZwyIrIAOARnRPMa8IR77GtgNbAYWALMi6g3ClgkIvNUta+IPABME5EcYD7QP0H6DcMwoiQxIxT3D/SxwBHAPmCUqj7t/jH+NlAf5w/6PqpaYLqR0AUcVS1wUtUd5fQt4NjtwO0Rr8cAY/Kc0z/P6wPuFRmGYSSaBM2I7QVuVtV5IlIBmCsik3D+EJ+sqiNc08uhRPwezUvoAo5hGMbBRCJGOKq6DljnPt8mIj8AtYGzgC7uaWOAqRQScEJ9D8cwDMPwFxGpD7QGvgNquMEoNygVuP0EbISTEnhdzuuH74jXZc1PzVjtWcO17b37+njlyCplPNUPy3LiIGjwA6/X4fX/Zo7XbSn+ZQqoJiJzIl6PUtVRf3s7kfLAe8CNqro11tGVBRzDMIwUJXdZtA9kFbUPx13F+x7whqq+7xZvEJGaqrpORGribD8pEJtSSxJeDZr8MA7zqiFeA7UJz9zJyMsyGXP9GX87Nuc/r/DEWc3YuTV6Xx1wsk6f2a09V/Y9N6Z6ftUPg+lXEMzPwqDBj/+bQUOcqPYy8IOqPhFx6COgn/u8H1CoD4MFnCTgh0GTV+MwPzTEa6DWotvZnDvsb6N1tm1ax08LvqFC9Zox6QAY89JzNGjUNOZ6ftQPg+lXEMzPwqLBD1O/WEhQ8s5M4FKgq5tzcoGInAaMAE4WkZU4yZILjdAWcJKAHwZNXo3D/NAQr4HakS3aUrp85b+VT315BJ363xLzFMG6X9cyddJ4+vTtH1M9v+qHwfQrCOZnYdHgh6lfLCQi04CqzlBVUdVj3Wz86ar6mapuVtVuqtrI/flbYe1YwEkCfhs0xWMc5pcGrwZquaz6bgrlq9ag+lGxjzIeuPs2brvnn5QoEd/X2Wv9IBhuBcGAzTT8nUSY+pk9QTEj+03Wch9DPbS13f1ZS0QKzPwsIvVFZEm87xOJnwZN8RqH+aXBq4EaQPbunXz3zoucePH1MdedMvEzqlarTstWx8Vc14/6EA7TryCYn4VFQy7JNPULKqm6Sm1nrpWAX6jqr4BnN89o8MugyYtxmN8mUZEGak2axZaK/Y91P7Nl41peu/FsALZlbeD1Ib25+LG3KXdY4clP582ayeQJnzJt8gR279rF9u3buPm6ATz+/CtRvbfX+hAO068gmJ+FRQMk0NRPEv/HjRdScoRTECKyRkTuFZF5IrJYRJq65dVFZJJb/qKI/CQi1fLU/WsEIyItRGSWO3paJCKN3NPSROQlceyrJ4pIXJsy/DBo8moc5ocGPwzUAKrXb8y1Y7/mipcmc8VLk6lQrQaXPPlekcEG4Ja77mPGgv8ydc4ynnpxLO0zO8cULLzWh3CYfgXB/CwsGhJp6ucsi7YpteIm150z93FBxLEsVT0OGAnc4pYNA6a45f8B6hbR/jXA0+4oKgNY65Y3Ap5z7av/AHrHIz7SoCn9mGb0Pr9PzAZNucZhM6Z/SdfMDLpmZvDFhM8TqmHjhvVceFZPenTMoFf3TDp26RaVgdqnj93MW7dfyO+/rGHUgC4snhSzh12g8KMvB/brS48uHVi5YjnNG9biQlmwAAAgAElEQVRj7OjYgp5XDX5cg2lw8Pp/M8ykpAFbfiZrbvkaHJfOX0TkeOABVe3uZo8+R1VXu+f9BjR23Ty3q2p5N13DJ6raUkQuxnH8HAu8r6or3eOTVLWR28btwCGq+s98dPhqwBaETAObtu4u+qRCsEwDDmHJNGA4eP2/2aNzexbMmxv3GKP8kU31mOv/vsUgVmYO7WwGbHGS+5sxh/33qGL6QFX1TeBMYCcwQUS65mk7b/t565sBm2EYCcGm1ILHDKAPgIj0AA4r7GRxXEJ/VNVncHbSHlvsCg3DMOIgEftw/CJVA07eezhF5Z+4F+ghIvOAU3HSbG8r5PwLgCXuVFxTnKk1wzAMwwMpuSy6IJM1Va0f8XwO+30atgA9VXWviJwAnKSqu93zyrs/1wAt3ecPAQ/laf633OPuOY/5cCmGYRjxk+ApMa+kZMCJg7rAv0WkBLAHuDLJegzDMDzjY7bohHBQBBxVXYljGGQYhmEkiYMi4KQ6fixr9kr1iqU81b/9pAaeNTQd9I6n+r++crFnDV7ZvXdfsiUAtrQ6F6/L1L3+30zzYXSSSiOcVF00kPKEwffDaxvx+obUrlKWD4d2Y+aI0/nmwdO4useB2Q0Gn9qU38ZeTJXy0QXJVO2HSILgp+NHG0HQEJS+jBZbFm0USlh8P7y2Ea9vyN6cfdw9bh7th35Kj/smMrB7I5rUcpIj1q5Sli4ta/Jz1o6EXIMfbfjhn5JsPx0/2giCBghGX8aCLYs2CiUsvh9e24jXN2TDll0s+slxBN2+ay8rft1KzcPKAvDAxccx7K35+Wb9LY5r8KMNP/xTku2n40cbQdAAwejLsGIBJwmExffDT++QeH1D6lQrx7H1DmPuqixOaV2bdb/vZOnPf0RdPyz94JUg9EMQNPhBQjX4MJ1mU2pRko8vTv1ifK8uIvKJH22FxffDL++QeH1DypUqyZjrO3LHG3PZu0+5+cwWPPj+opjeOwz94AdB6IcgaPCDRGoQvE+nJbJ/Un2VWqG+OCJSUlX3JlJQNITF98OPNuL1DSmZJoy5oSPvfruGT+aspdmRlahbvTxf/fNUAGpVKcvU+0+h+/AJbNyyq1ivIZn94BdB6IcgaPCDRGtIoUVqqT3CyQ8R6S8i74jIx8BEt+xWEZntetvc65bVF5Ef8vO3EZGGIvKFiCwUx0Mnd01veRF5V0SWicgbEuefBmHx/fDahhffkGcGtmfFr1t4fvwyAH5Yu4Umg98n/eaPSL/5I3797U+63D2+0GDjxzX40UYi/VMKIgj9EAQNfhAEDUEl1Uc4Zdx8ZwCrVTX3T8MTgGNV9Tc3WWcjoB3OxtyPRKQT8D+3/CJVvVJE/o3jb/M68AYwQlX/IyKlcQJzHZzNoy2AX4GvgUycxKAxEem5kZOTQ7/+Azz5fsTTRhA05PqGNGvRkq6ZTmb0O+65n+49Ty203vGNq3Nhh6NY+r/fmXa/c+797yzki0Wx3Wj34xr8aCPefohkYL++zJg+jc2bs2jesB5D7xrGZf0HJOwa/GgjCBogGH0ZCyVSaIiTkn44ueTniyMi/YHOqnq5+/oxHOvo3DvJ5XHypE0mH38b4GngB1U9Mk+7XYA7VfVk9/VI4GtVfT0fXb764YQBr74hEI6Nn370Q6mS3icmbOOng9eNn177MfP4DObOnRN3xKhYt5m2v/1VTxoAJg0+ISF+OKk+wimIyE0YAjykqi9GnuAuMMjrb1OGwr1zovbDAUYBtGmTkboR3TAMw0dCdw8nHyYAA0SkPICI1BaRwws6WVW3AmtF5Gz3/FIiUjYxUg3DMKLHWdZsq9QCg6pOFJFmwLdux24HLsEZoRTEpcCLInIfkA2cX+xCDcMw4qBE6tzCSe2Ak/f+jVs2Ghidp+xpnHszecnX38bNLt01z7k/AlMjzhkch2TDMAxfseSdhmEYhpGHlB7hGIZhHOyk0ADHAo4RHZu27i76pELwwwdm2XPebqWd8MAUzxq+vTPvTGtsBMHbyNiP12XNXpdV7/O4hlVw0tukCjalZhiGYSQECzhJIghGU17r79q1izO6d6Bnp7Z0O7E1j4+4L+Y2wPEPObNbe67se27MdeMxL6tRsRSjLmvNe9cdz7vXtuOi4509vhVLl2TkJel8OLg9Iy9Jp0Lp6CcAkv1ZmIbgaPDDwC0WSoj3R6KwgJMEgmA05YeGUqVK8dYH45kwfTbjp81i2uRJzJv9XUxtAIx56TkaNGoacz2Iz7wsZ5/yxMSV9H7+Oy57eS4XtD2So6uV5fIO9Zi1+nfOenYms1b/zuUd6kWlIQifhWkIjgavBm4x4cMeHDNgCzlBMJryQ4OIUK68szJ9b3Y2e/dmx/zlXffrWqZOGk+fvv1jqpdLPOZlWdv3sGz9dgD+3JPD6k07qF6xFF2aVOPjhesA+HjhOk5qUi0qDUH4LExDcDR4NXCLFfPDMQolCEZTfplE5eTkcErndrRuWocOnbvROkbjsAfuvo3b7vknJUp4/yrGY15Ws1JpmtSswJK1W6la/lCytu8BnKBUpdyhUbURhM/CNARHg1EwKRNwRKSGiLwpIj+KyFwR+VZEEm8c4gNBMJryyyQqLS2N8dNm8d3iVSycP5vlPyyNuu6UiZ9RtVp1WrY6Lub3zUs85mVlDknjsT4teWz8SnbsiX+1URA+C9MQHA2JRHCyRXt9JIqUCDiu78wHwHRVPVpV2wAXAkcWXvOv+oFKjRsEoym/TaIqVapM+8xOTJ08Meo682bNZPKET+mS0ZQbr76MmV9P4+brok8Dn0s85mUlSwiP9WnJ54s3MGXZJgA2b99DtfLOqKZa+UP5bceeqNoKwmdhGoKjIdHYlJr/dAX2qOoLuQWq+pOq/ktE0kTk0QiDtavhL0voL0XkTWCxa7i2TET+n4gscQ3UuovI1yKyUkTaufXaicg3IjLf/dnELe8vIu+LyHj3/EfivZggGE35oWFz1ia2bHFcH3bt3MmMaVNo0KhJ1PVvues+Ziz4L1PnLOOpF8fSPrMzjz//Skwa4jUvG3ZmU1Zn/cnrM/f/cpm2IoszWtUE4IxWNZm6PCuqtoLwWZiG4GgwCiZVNn62AOYVcGwgsEVV24pIKeBrEcn9M7sd0FJVV7t2BA1xEnFeBcwGLgY6AGcCdwBnA8uATqq6V0S6Aw/iGLMBpOOYsO0GlovIv1R1/2+sKAmC0ZQfGjZuWM9Ng64gJyeHffv20evs3nTveVpMbXglHvOy9DqV6NWqJis2bOetq9sC8OzkH3l1xk88fF5Lzm5dk3VbdnHbO0ui0hCEz8I0BEeDVwO3WAnylF9eUsKATURuAI5S1SHu6+dwAsUe4CfgWOBP9/RKwNXusWGqepJbpz4HGq6NBSao6hsicjTwvqqmi0gd4BkcN1AFDlHVpq6xW6aqXunW/xx4QFX/5vgZRgO2IGQaqFjG299HPZ/4yrMGr5kGjHDhNdNAl8zjmT8vfgO2Kkc1127D/uYBGTPvXt4muQZsIlLonVfXNyZRLGX/KANVHSQi1YA5OFbR16vqhMgKrkNnpBEbHGigti/i9T7298X9wJeqeo4bpKYWUN8M2AzDSDqpZDFd2D2cpcAS9+fSPK+jm2vwjylAaRG5NqIs1xRtAnCtiBwCICKNRaSch/eqBOSug+zvoR3DMAwjggJHOKpap6BjiUZV1XXgfFJEbgM24YxebgfeAeoD89zVbJtw7sXEyyPAGBG5CSfQGYZhBJbUGd9EuWhARC4EjlbVB0XkSKCGqs4tXmkHoqrrcJZC58cd7iOSqRxomLaGAw3X+ud3TFW/BRpHtHO3Wz6aCGM3Ve0V0wUYhmEUA6m0aKDIZdEi8ixwEo7tMjg3518ouIZhGIaRCJyNn6mTvDOaEc6JqnqciMwHUNXfRCS6nB+GYRiG4RJNwMkWkRI4S4QRkao4q7qMg4jqFUslW4Jn/FjSfFjHoZ7q//5VfCn3w4jXJcVezdOCoMHz6CLB2Z69Ek2mgeeA94DqInIvMAN4uFhVHQSEwffjYNZw5OGVGP/slcwfdxNz3xjCoD6ZABzbqCbTXrqOmWNuYMYrg8loHlX2pZTtB7/b8MNLJgwaYiFUqW1UdSxwF/AY8Btwvqq+VdzCwkxYfD8OZg17c/Yx9JlPaX3RE3S+8jmu7t2epvUP54FBp/LAy1/Qvt8z3P/SJB4YVHTmhVTuB7/b8OolExYNYSXaXGppQDbO7v1Uyb8WWMLi+3Ewa1i/eRsLVji+O9v/3MOyNZuoVb0iqlCxXGkAKpUvzbqsovdHp3I/+N2GVy+ZsGiIhVAZsInIncA4oBZOduY3ReQfxS0szITF98M0ONQ94jDSG9di9tKfufWpj3lw8Gms/GAoD11/GveMHJ8QDUHohyB4yRxsGlJtlVo0o5VLgLaqepeq3omTEPOy4pUVP8XpmyMi2/1oJyy+H6YBypU5lHEP9eXWpz5m25+7uerc9tz29Cc0OnsEtz39CSPv6F1kG2HoB7/a8MrBqCFUIxyc5JiRq9lKAj8WjxxvROubIyJJzZIdFt+Pg11DybQSjHvwEt6esIAPpznGc31Pa8MHU53MT+9NXkxG86ITdqR6P/jZhldMQ7ApMOCIyJMi8gTORs+lro/MS8Bi4I9ECYyRwnxz+ovIOyLyMTARQERulf0+Ovfm1hGRS0RklogsEJEXJY+Bm4hUc0dOp8cjMiy+Hwe7hhfuPI/lP23kmbf2Jwxfl7WVjq2PBqBLRgP++3PRnjqp3g9+tuGVg1GD+PAo8j1EXhGRjSKyJKJsuIj84v6eXCAiRa6QKewv/dyGlwKRSzZmRqEvWRTmmwNwAnCsu3m1B44FQTucPv9IRDrh5GK7AMeKIFtEngf6AmPBmbIDPgLuUtVJ8YgMi+/HwazhxGPr0ffU41j833XMHHMDAMNemMCgh97j0SFnUDItjd17shk84j/Ffg1+tBEEDeDdSyYsGqJFJGHZokcDz+L+HozgSVV9LNpGUsIPJ1qK8M15Duisqpe7xx4DzmP/aK088BBQBicv20a3vAwwTlWHi8huYCUwSFWnFaIjdH44hoNt/PSPZG+6DIKGzOMzmDs3fj+c6g1a6FkPvu1JA8DLFx5TpB+Oa9fyiaq2dF8PB7bHEnCiWaXWQETecqedVuQ+on2DBLMUOC73haoOAroB1d2iSH8cAR5S1XT30VBVX3bLx0SUN1HV4W6dvcBcoGdhIlR1lKpmqGpG9WrVCzvVMAzDE0ne+DnYjQ2viMhhRZ0czaKB0cCrOL+ITwX+DQR142dhvjl5mQAMEJHyACJSW0QOByYD57nPEZEqIlLPraPAAKCpiHj7U9cwDMMHfFqlVk1E5kQ8rorirUcCDYB0YB3weFEVolmtVVZVJ4jIY6q6CrhLRLx79RYDRfjmlMlz7kQRaQZ863b4duASVf1eRO4CJoqTQy4bGISzWg9VzRHHruFjEdmqqs8n6voMwzDy4tMtnKxYLaZVdcN+DfIS8ElRdaIJOLvd5carROQaHDfMw2MRlkiK8M0Znefcp4Gn82njbeBvE6OqWt79uYciptUMwzDCjIjUdH/fApxDFE7Q0QScITg31G8AHsCxYI5+yYZhGIZRLAiSkFVqIjIO6IIz9bYWGAZ0EZF0nFsNa4Cri2qnyICjqt+5T7ex34TNMAzDSDYJyvasqhflU/xyrO0UGHBE5D+4HjgFCDg31jcz4mPT1t2e6pc6xHu+1VIlvbWxe693C6WKZQ7x3IZX1k95wFP9bk9O96xh8pBOntvwitflxOB9SXEQNBixUdgI59mEqTAMwzDiItG54rxQ4J+tqjq5sEciRYYRrwZNu3bt4ozuHejZqS3dTmzN4yPui6n+L2t/5pzTT6ZDxjF0ateKUc//K2YNXo2q/NAAyTcei7cf/nFKYz65rj2v9W/zV1nD6uV4sW86Y/u34eFzWlD20Oj/Ak/VfgibBr/aiJYSPjwShXnbJAE/DJpKlSrFWx+MZ8L02YyfNotpkycxb/Z3RVd0KVmyJPc+8Agz5izms8kzePWlkSxfllijKj80BMF4LN5++GzJBm5698CFPUN7NmbktNVcNnou01dm0bdtdI6hqdwPYdOQSAM2IXzZog2f8cOgSUQoV748AHuzs9m7NzumL06NI2pybHprAMpXqECjJk1Z/+uvMWnwalTlh4YgGI/F2w8L125h667sA8rqVinDgrVbHG0//UHnxtWiaiuV+yFsGhJtwJZKRB1wRKRUcQo5mPDLoCknJ4dTOrejddM6dOjcjdYZ7eLS87+f1rBk0UKOi7O+H8SrIQjGY37yY9YOOjSsCsBJTapRo2J0/+3C0A9h0ZDo6wiVAZuItBORxThJKxGRViIS32R7AJBCTNRE5Jt468aCXwZNaWlpjJ82i+8Wr2Lh/Nks/2FpzG3s2L6dgZdewP0jHqNCxYox1/cDLxqCYDzmJw+OX0Hv1rV4+dLWlD00jeyc6JLrhqEfwqIh0deRSgEnmo2fzwC9cIzNUNWFInJSsapKMCKSpqo5qnpiIt7Pb4OmSpUq0z6zE1MnT6RJs+jToGdnZzPgkgvo3eciTj/TF1PUmPGqIQjGY37yv992MuSdxQDUOawMJx4d3fRQGPohLBoSeR1O8s0QrFKLPEdV8+bX974APsmISBcR+VJE3sQxlftrBCMiNUVkumsqtEREOkbUe0BEForITNcbJ2b8MGjanLWJLVscZ4VdO3cyY9oUGjRqEnV9VWXIoKto1KQp1wy+Mab39gs/NATBeMxPKpd19hoJ0O+EunywYF3hFVzC0A9h0RCE6wgq0QScn0WkHaAikiYiNwJBtSeIlXbAnaraPE/5xcAEVU0HWgEL3PJywExVbQVMB66M500jDZrSj2lG7/P7xGzQtHHDei48qyc9OmbQq3smHbt0o3vPIg33/mLWzG945603mDH9S7pmZtA1M4MvJnwek4aB/frSo0sHVq5YTvOG9Rg7+pWY6vuhwWtf+vFZxNsPw3s15cW+6dStUob/XHM8vY45gpObVmfcwAzeHJhB1vbdfLpkQ9EN+XAdyeyHsGnwo41YSKUptSIN2Nw0/c8A3d2iL4DBqlq0d24AEZHtqlpeRLoAw1T1pHyOdQJeAV4HPlDVBe7x3UBpNyv1BcDJqnpFPu/hqwGbZRpwCEKmAa+720979mvPGizTQHA0eMWrAdsRjVrqZU+951nHo72aFmnA5gdF/hZR1Y2qeqGqVnMfF6ZqsMmHHfkVqup0oBNOZuzXROQy91C27o/QORRwD8wM2AzDMP5OkYsGXJ+Dvw2DVDUag56UxDVc+0VVXxKRcjguonm9vA3DMJKKQEKyRftFNKvUvoh4XhrH9+DnAs4NC12AW0UkG8eY7bLCTzcMw0gOqbR7Pxp7ggOMyETkNWBSsSkqZiJM1KYCUws4NgYYU1Bd9/m7wLvFKNUwDKNIUmiAE1dwPAqo57cQwzAMI9xEcw/nd/bfwykB/AYMLU5RhmEYRtGIJMbx0y8KDTjibGFthbNaC2BfxCotI0FUjzKfVpDxY/mp12Wwfmjw2oYfS5oPazvYcxu/z/Zmd5Xs5cRB0RAEUijeFD6l5gaX/7hpX3Is2PhHEDw3wqAhLP4n8dQ/skZlxo+6gfnv3cXcd+9k0EVdADimcW2mjrmZ2f++g3efupoK5Uon5Br8aMM0hJto7uHMEpHjil3JQUQQPDfCoiEM/ifx1t+bs4+hT7xP697/pPNlj3H1BZ1oevQRjLznYu565kPa9nmQj75cyJB+3Yr9GvxowzTERyplGigw4IhI7nRbB5ygs1xE5onIfBGZlxh54SQInhth0RAG/5N466/P2sqCZWsB2P7nbpatXk+t6pVpVO9wZsz9LwBTZi7j7G7pxX4NfrRhGmIndx+O10eiKGyEM8v9eTbQBDgNOB84z/1pxEkQPDfCosErYemHujWrkN7kSGYvWcP3q9bRq8sxAJx78nEcWeOwhGgIQj+EQUOsOBmjvT0SRWEBRwBUdVV+jwTpi5qITM/1ReTiKM6vLyJL3OcZIvJMcWvMJQieG2HR4JUw9EO5Mocy7rEruPWx99i2YxdXD3+Dq/t04us3bqN82VLsiWKxRRj6ISwawkxhq9Sqi8hNBR1U1SeKQY8f1MfJ9vxmtBVUdQ4wp7gE5SUInhth0eCVVO+HkiVLMO6xK3n78zl8OGUhACvWbOCM654DoGHdwzm1Y9GZilO9H8KkISYSfA/GK4WNcNKA8kCFAh5BZQTQ0fWyGeKOZL5y7z/NE5G/may53jifuM/bicg37r2qb0SkiVveX0TeF5HxIrJSRB6JV2AQPDfCosErqd4PLwzry/LV63nm9Sl/lVU/zEmIISIMvbInL707o9ivwY82TEN8iA//EkVhI5x1qnpfwpT4x1DgFlXtBSAiZXFsBHaJSCNgHFBYGu5lQCdV3Ssi3YEHgd7usXSgNbAbWC4i/1LVmPPKRfpl5OTk0K//AE+eG/G0ERYNA/v1Zcb0aWzenEXzhvUYetcwLus/IKEaktUPJ6YfTd9ex7N4xS/MfMvZiz3s2Y9oWOdwrr7A2e/z4ZQFjP1wZrFfgx9tmIbwU6AfjojMV9XWCdYTN3l8biIDTiXgWZxgkQM0VtWyIlIf+ERVW0bWEZE6OP4/jXAyLByiqk1FpD+QqapXuu1+Djygqn/789FvPxzDIQgbP4NAEDZ+Gv7g1Q/nyCbH6OCRH3jW8Y9uDZPuh1P04v3UYAiwASdjQgZwaBHn3w98qaotgTNwMmTnEumEZn44hmEknVTah1PglJqq/pY4Gb6yjQPvMVUC1qrqPhHph3NvqjAqsT+VT3//5RmGYfhHKq2ASyUrhWhZBOwVkYUiMgR4HugnIjOBxhTg8hnBI8BDIvI1RQcnwzAMI0qiMWBLCSK8bLL5+3TgsRHP/+GetwZo6T6fiuuNo6rf4gSmXO52y0cDoyPer5dv4g3DMOLAyTSQbBXRE5qAYxiGcdCR4EwBXrGAYxiGkcKkkh9OGO/hGIZhGAHERjiGkWKs/+Zpz20c1uNBT/V/n3iHZw1hwOvesH0eHcZS7R6OjXCSRBBMnsKg4WA2YIsknn44snoFxj/el/mvXsXcV65k0LltAXjt7rOZOWogM0cNZNmb1zFz1MCEXcfB+ll4ISzZoo1iIggmT2HRcDAbsEUSTz/szdnH0Be+oPXlo+g8aAxXn3UcTetV49L7P6D9VS/T/qqX+WD6cj78anlCruNg/iwOFizgJIEgmDyFRcPBbMAWSTz9sP63HSxYuQGA7Tv3sOx/m6lVrfwB5/Tu0ox/T1kaVXtB6IcgaPD6nYwNoYQPj0RhAScJBMHkKSwavGL94FC3RiXSG9Zg9g+//lWWeWwdNvy+g1W//B5VG0HohyBoSCRCak2pBXLRQG4izmTrKC6CYPIUFg1esX6AcqUPYdy953Lr81+w7c89f5X36dqCd6Ic3UAw+iEIGhJKiPxwjGIiCCZPYdHglYO9H0qmlWDcvb15+4ulB9yrSSshnNWhCe9++UPUbQWhH4KgwSiYwAYcESkvIpNd07TFInKWW15fRJaJyBgRWSQi77qeN4jIPSIyW0SWiMgocf80EZGpIvKwiMwSkRUi0tEtTxORR906i0Tkare8pohMd03clkSc30NEvnU1vSMicY3CgmDyFBYNXjnY++GFW09n+f+yeObdWQeUd21zFCt+3swvWduibisI/RAEDYmmhIjnR8K0JuydYmcXcI6qHgecBDyeG0CAJsAoVT0W2Apc55Y/q6ptXWuBMkBkvrOSqtoOuBEY5pYNBLaoalugLXCliByFY1E9QVXTcWwNFohINeAuoLuraQ5QoAV3YUQaNKUf04ze5/fxZPIUTxth0TCwX196dOnAyhXLad6wHmNHv5LQa/CjjWT1w4ktj6Rvj2PonF7/r2XQPY9vAMD5JzWPerGAX9dxMH8W8ZJq93AKNGBLJiKyHTgMeBLoBOzDCTJH4fjTTFfVuu65XYEbVPVsEekN3AaUBaoA/1LVESIyFbhTVb8WkRrA16raUETexUns+af71pWAq3GC3SvA68AHqrpARHrhJO9c6557KPCtqv5tk4IZsBUPZsDm4LUfAI44/WFP9W3jp4PXz6JL5vHMnxe/AVv9ZsfqnaM/9qQB4Kr29RNiwBbIRQMufYHqQBtVzRaRNew3Q8sbJVVESuNYEWSo6s8iMpz8zdMijdMEuF5VJ+R9cxHpBJwOvCYijwK/A5NU9aKihKvqKGAUQJs2GcGL6IZhhAbLpeYPlYCNbrA5CagXcayuiJzgPr8ImMH+4JLl3ls5L4r3mABcKyKHAIhIYxEpJyL13Pd+CXgZOA6YCWSKSEP33LIi0righg3DMBJBKk2pBW6EIyIlcUYjbwAfi8gcYAGwLOK0H3BM1V4EVgIjVfVPEXkJWAysAWZH8Xb/D6gPzHPvD20Czga6ALeKSDawHbhMVTeJSH9gnIiUcuvfBayI/2oNwzDiRwj2qCEvgQs4QAtglapmASfkPSgi9YF9qnpN3mOqehdOEMhb3iXieRZOkEFV9wF3uI9IxriPvO1MwVlcYBiGYcRIoAKOiFwD3ICzkswwDMMoDAn4xtQ8BCrgqOoLwAtFnLMG1xraMAzjYCd1wk3AAk4Y2afel06+s2ht0ScVwiVt6hV9UhEEYUmy1zb8WE4chKXVfmjwuqz54SkrPWu4vWsjz20km60793qqn+PVECfFSKX7TaEiHs+M0f+8lZtObcOwi3v8VTZn8qfcc9HJXHXCUaz5YVFMGoLg+xEE/5MgXEcQ+iGe+h88/g8e6dOe5646/a+yKWOe4vlrzmDktWcy9h+Xs3XzhoRdgx9teK2/a9cuzujegZ6d2tLtxNY8PuK+mNuIFseAzTINGEUQj2fGiaefx/89eeBahtpHN+G6ES/QKL1dTNKPV9MAACAASURBVG0FwfcjCP4nkPzrCEI/xFs/vce5XPLAyweUnXjeFVz3wsdcO/IjGh9/EtNefy4h1+BHG35oKFWqFG99MJ4J02czftospk2exLzZ38XURiyID49EYQEnScTjmdG49fGUq1jpgLKaRzXkiHoNYn7/IPh+BMH/BJJ/HUHoh3jr1z+mLWUqHPidLF1uf4rB7F1/Rn1TO5X7IRIRoVx5pw/2Zmezd292sd7YT8Q+HBF5RUQ2isiSiLIqIjJJRFa6Pw8rqh0LOAcpQfD9CIL/iR8EwYMlCBoimfzqEzzRtxOLpnzMSZf9X8I0BKUfcnJyOKVzO1o3rUOHzt1onRHbDEQAGQ2ckqdsKDBZVRsBk93XhZKyAcfNt5b7/DQ3ytZNpqZUIgi+H0HwP/GDIHiwBEFDJN0uv4mb3pjOsV3PYNZHryVMQ1D6IS0tjfHTZvHd4lUsnD+b5T/Elgg1egQR74+iUNXpwG95is9i/37FMTib5gslZQNOLiLSDfgXcIqq/i/KOgf96rwg+H4Ewf/ED4LgwRIEDflxzEln8P2MiQnTELR+qFSpMu0zOzF1cnR9ECu5mQa8PuKkhqquA3B/Hl5UhZQOOK5PzUvA6aq6yi2r5/roLHJ/5maVHi0iT4jIl8DDbs60V1wvnPlyoN/OV67nzTwROdEt7+L66rwrjh/PGxLNnwYBJQi+H0HwP/GDIHiwBEFDLpt/WfPX8+UzJ1OtztEJ0xCEftictYktW/4AYNfOncyYNoUGjZrE1EYs+DTCqSYicyIeVxWH1lT+S78U8CHQRVUj86w9C4xV1TEiMgB4hv1DvcY4fjY5IvIgMEVVB4hIZWCWiHwBbAROVtVdItIIGAfkpu1ujZN651fgayATJ3FozAzs15cZ06exeXMWzRvWY+hdw7is/4BC64y6+3pWzJvJ9j9+59Yz2nPmlUMoV7ES4x4fzvY/fuOZmwZQp3Ezhjxd9BRGpO9HTk4O/foPiMv3I9Zr8FtDGK4jCP0Qb/13HxrCmkWz+HPL7zzetyMnXXoDK2dNI2vtaqRECSofXoteN9ybkGvwow0/NGzcsJ6bBl1BTk4O+/bto9fZvene87SY2kgCWXHYE2wQkZqquk5EauL87iyUQPrhRIOI/AlMwcm79n8R5VlATTfL9CHAOlWtJiKjgS9VdYx73hycDNO5O7eqAD1xgsmzQDqOlUFjVS0rIl1wPHVOduuPxPHVeT0fbfv9cOrUbbN4+Y+ertU2fvpDWDZ+BgHb+Omwaevuok8qhNO7nsiiBXPjnilp0LyVjnjzc08aAPq0rl2kH46bx/IT1+AScWxbNrueY0OBKqp6W2FtpPKU2j6gD9BWRArbNh0ZUXdEPBegt6qmu4+6qvoDMATYgOP0mYFjtJZL5Lcr0lfnwDdUHaWqGaqaUbVa9eivyDAMIxbEtym1wt9GZBzwLdBERNaKyEBgBHCyiKwETnZfF0oqT6nhWhL0Ar4SkQ2q+jLwDXAh8BqOiVtBU14TgOtF5HpVVRFprarzcXx41qrqPhHpB9iftIZhHNQUYjzZLZZ2UjrgAKjqbyJyCjDdnU67AXhFRG7F8be5vICq9wNPAYvcm/9rgF44rqHvicj5wJccOCoyDMMIDOaHkyBUtXzE85+BoyIOd83n/P55Xu8Ers7nvJXAsRFF/3DLpwJTI84bHJdwwzAMH0mlxbIpG3AMwzCM1LInSKXRmGEYhpHC2AjHMAwjhUmhGTULOMVNCfG+d8PrPpofftnqqT7A0YeX81Tfjz0wXvFqlgVQPST7cLx+Hn7sobng1dme6r99eVvPGrxSvWIpT/UPSfMWLZxFA6kTcWxKLUkEwWhq25Y/uPXaSzm3awbndmvLwrmzYqrvh3GZ1za81vfLLCvZpl9+tJEsI7rrO9VnzCXpPNN7/47+W7s24MlzW/DkuS0YdeGxPHlu9Lv9w/BZhBULOEkgCEZTAI/eO5QTO3fn/SlzePvzrzm6YeOY6ns1LvOjDa/1/TDLCoLpVyob0U1ekcW9n684oOzRKasY8v5Shry/lG9X/87M1b8Xqwa/6vvVRiwkwg/HLyzgJIEgGE1t37aVebO+5uwLLgPgkEMPpUKlyjFp8Gpc5kcbXuuLD2ZZQTD9SmUjuu/Xb2f77oKnOzscXYXpqzYXqwa/6vvVRvSIL/8ShQWcJBAEo6lf/reGw6pWY/gt13HRaR247/bB7Pzz4Nzj6tUsKwimX2EwosuP5keU54+d2ayLMmfZwfhZ2AgniYhIjogsEJElIvKOiJQt4vzt7s9aIvJuIefVlwh7VS8EwWgqJ2cvy5Ys5LxLBjLusxmUKVOOV0c+GZOGsODVLCsIpl9hMKLLj04NqkY9uvFDQ1g+i6ASuoAD7HSTcbYE9gDXRFNJVX9V1fOKV5pDEIymDj+iNocfUZtjWjsJYruddhbLliyMSUPYiNcsKwimX2EwostLCYET6h/GjB/zGk0Wn4ZU+yxyV6l5fSSKMAacSL4CGgKIyE3uqGeJiNyY98TIEYyItBCRWe5IaZHriwOQJiIvichSEZkoImXiERUEo6lqh9egRq3arFnlpJmf9fU0jipGk6ig4odZVhBMv8JgRJeXVrUrsnbLTjbvyE6YhpT7LHyYTrMpNR8Qx0b6VGCxiLTBSeJ5PNAeuFJEWhdS/RrgaVVNx7EoyDWkaQQ8p6otgD+A3vFoizR5Sj+mGb3P7+PJaCreNm4f/gh33ngFfU45kRXfL2bgoJtjqj+wX196dOnAyhXLad6wHmNHvxJTfT/a8Fp/44b1XHhWT3p0zKBX90w6dukWs1mW188iKN8Hr30Zr4abTzqah89qRu3KpXn5olZ0b1INgI4NqvLVquhHN140+FXfrzbCSsoasBWEiOQAi92XXwE3A9cCVVX1Hvec+4FNqvrM/2/vvMOsKLI+/P4ICpJcFXNAEUFEiUYQFRXdFYyYXQVzWtO3urrqimlxTWvOijlhxEVBRAVRgoCYA2tas4AJJY/n+6PqMj3DMNN9u+fOvUO9z3Ofud23q/p0mD5dVafOT9KvZtY8Ki4k6RDgXOBe4Akzm+F/H21m7XwdfwMam9klVdhQLsC2/vrdP/r481o73jgUw8TPYiCTiZ8pJ/oVC8UgqFcfJn6mpefWPZg6dUrebYxNOnWxG4aNTm3Hbh1Xr1GALQvqYwsnN4bTxcz+YmYLSZjfzsweBPYE5gGjJOWyTycWYGsdBNgCgUAtEsKii49xwN6SVpLUDNgH1/qpEkkbAZ+Y2XXAcCrKFQQCgUBRIFxwRdpPoVgucqmZ2TRJdwO53C13eHXPZXEgcJikRcC3wEVAy9q1MhAIBOo39c7hRIXZKq2/Grh6Wdub2WdAJ/99CDCk0qY/5H7321yZjcWBQCCQP4XsEktLvXM4gUAgsDxRSnNKg8MJBAKBEia0cAJFxabr1P3wUxZ6OGlDceuLlk0WLFj8e6ryWYRFpw1r/kPff6a24cfn/566jkB8lpcotaKjGDQ3isGGutJgybJ8fbDhqy+/YJ89dqVXj83pvVVnbrvp+oLbkG/5dVu3YORVh/LG0GOZetcxnLSvc2T3nb83E287iom3HcUHD57IxNuOKsgxZFVHHEotSq3eTfwsNrp372GvTppSYV1ZWRmbd9yEEc+NZp1116XXNltyz/0PsWnHjrHrTVtHoW1YVgvn1fHjaNasOSccM4gJU6rP5VbVW3WpnYdiseGXeRXTxXz37Td89+23bNGlK7/OmcOuvbfm7oceo32Hqsu3bNo48+NIWj7XwllzlWasuWpzps/4juZNV+C1WwZxwD8e54PPZy3Z9rLjd+bn3xYw5L7xFeqo3MIp9LVIO/GzQ6eudvsTL+ZbfAm9268SJn7WV4pBc6MYbIC602DJqnx9sWGNNddiiy4u21PzFi1o174D3379dUFtyLf8tz/8xvQZ3wHw67yFfPC/2ay9WsVg1f123JRHX6w5C3gxXIv6THA4dUAxaG4Ugw1ZUB/OQzHYEOV/n3/GO2+9SbcS1AVaf41WdNl4DV5/v9xZ9txiPb778Tc+/qpm1dBiuxY1EpJ31g6VdG6ekZRMnrL6untIui6r+mqiGDQ3isGGLKgP56EYbMjx26+/ctSfD+Tiy66kRctkwSZ1fR6aNWnMQxfuy5k3vcCcuQuXrD+gz2YMi9G6ycKGrOpIgjL4FIqScThU1Ln5ATgpq4rNbIqZnZJVfTVRDJobxWBDFtSH81AMNgAsWrSIIw87kP0OOJg99twnUdksbEhTvlHDBjx04X488sK7PP3Kh0vWN2wg9urVnsdeer/Wbciyjri4oAGl/hSKUnI4USYA6wBI2lHSf3I/SLpB0kD//TJJ73lNmyv9uv19K+lNSeMq1yFpK0mvSXrD/23v1w+U9ISkkZJmSLo8X+OLQXOjGGzIgvpwHorBBjPj9JOOpV37Dhx/8lJyUQWxIU35W87cgw//N4vrHptcYX2f7hvy0Rez+WrWnIIcQ1Z11FdKbh6OpIbAzsCdNWy3Ci5JZwczs0gX3D+A3czsq2V0y30A9DazxZJ2Af5Jue5NF6ArLmv0h5KuN7MvqqijWqJ6GWVlZRwx8MhUmhv51FEMNoDTYBk/biyzZ8+i48YbcPZ5F3D4wCMLZkMxnIdisGHyxNcY9vADbLpZJ/r0dMFKf//Hxeyy2x8LZkO+5bfrtC6H9t2ctz/+fkno8wV3vsyoSR+z/04dYwULZHUMWdWRhNKZ9llCYdERnZs2wFSgr5mVSdoR+KuZ9fPb3QBMAe73200BRuC0bhZKugVoCzyK07qZHa1D0nrAdTixNcNp3nTwraaeZnaM389zwKVmVjHOkuLTwykGimHiZ6CcymHRSakqLLrQ1IeJn2nDojfdvKsNfeql1HZsu/EfQlh0JeZ5Bc4NgBUoH8NZTMXjaAJgZouBrYDHgb2BkX798cB5wHrAdEmrVtrPxcBLfqyof64+T9DDCQQCgTwpJYcDgJn9DJwC/FVSY+BzoKOkFSW1wnW3Iak50MrMngVOw3WHIamtmU3y6p+zcI4nSisgF8M4sLaPJxAIBNJQSgJsJTeGA2Bmb0h6EzjIzO6T9CjwFjADyOnctACeltQE1815ul9/haR2ft0Y4E1gh0j1lwP3SDoDSD+FNxAIBGqRkC26Fqisc2Nm/SPfzwLOqqLYUjPXzGzfKrZ72X8wswnAJpHfzvfr7wbujtTTL6bpgUAgUGuUkL8pvS61QCAQCJQmJdPCCQQCgUAVlFATJzicQCAQKFFcaprS8TjB4SwHFMMcmLSCXwC/zFucqnzLpulv99TCZY3S92JP+/yn1HVsu3Hl2QClRxZzaG557ZNU5Y/fbqPUNixPhDGcOqIYRJ7qWvwsC9Gv+fPn03+XXuzWe0t23q4rV112UeI60p6HLI4ji2sx7O6bGdSvJ0f278XF/3cMCxfMT1S+GO7JurLhocv+xvl7bcm/Bu6+ZN3wm4cw5M+7cvmgP3HXucczb84vtWpDXoRs0YGaKCsr47RTTuLpZ57jjbfeY9jDD/H+e+8VvI5D/nw4jz01IlGZLG1o1KgRF156OeOnvM2zY8Yz9Pab+fCDZMew4oor8vBTIxk17nVGjp3M2DGjmfb6pER1pD0PWRxHWhtmfvcNT95/O7c89gJ3PTOe33//nReffTJ2+WK4J+vShq3+uB/HXjG0wrpNevTirKHPcdbQZ2m93oa88MDNBTuOJIRs0YFqKRaRp7oWP8tC9EsSzZq7iPnFixaxePGixKng056HLI4jrQ0AZWWLWTB/PmWLF7Ng3lxWXX3N2GWL4Z6sSxvadt6KZi0qplbssOX2NGzkumE36NiFn2Z+W7DjSEQJeZzgcOqAkhN5KoAN+Yp+gXuj3H2HrejaYT167bAzXfOoIyvSHEcaWq+xFgcMOomDdu7CgN6b0axFS7bsuVPs8sVwTxaDDcti0rOPsenWO9S8YS3aUB8oKYcj6VxJ73q5gemSts64/tdq+P3XLPZTiiJPtWlDGtEvgIYNGzJy7GQmvf0xb77xOh++Hz87cJakPY40zPn5J1598TkeHD2VYWPfYf68uYwe/mjs8sVwTxaDDVUx+r4badiwId133avObFg2WSS2CXo4SyFpW6Af0M3MtgB2ARJLA1SHmW2XZX3LotREnmrThrSiX1FatVqZbXr25uUxz6eqJx+yPI58mDphLGutswErr7IajRo3Zvtd+vHuG6/HLl8M92Qx2FCZySMf593XXuKw8/8d22kU+n8zBA3UDmsBs8xsAYCZzTKzryV9Julfkib7z8YAkvpLmuSF1F6QtIZfP1jSXZJelvSJpCVKn7kWjKS1JI2LSFpvH9nmUi/eNjFXZ1Lqi8hTMYh+zZ41k59/dmHC8+fNY/zYF2nbrn1edeVLFseRljXWWpf33pzC/HlzMTOmTRzH+m03qbmgpxjuyWKwIcr7k8by4oO3cfSQW1mhSdM6saG+UUoO53lgPUkfSbpJUrRD9Rcz2wq4AbjGrxsPbGNmXYGHqZhrrQOwGy7X2gU+63SUQ4BRXg6hMzDdr28GTDSzzsA44Jh8DiQq0NRl803Zb/8DUok85VvHUUccSt8dezHjow/puPEG3Hv3XQW1ISf6NX7cS/Tp2YM+PXvwwqjnEtnw/XffctBeu9F3+x7026Un2++4M7vs9qdEdaQ9D1kcR1obNu3cnR12689x+/XhqD235/fff6ffAYfHLl8M92Rd2nDvhadyzYkD+P5/nzJ4QE8mjniUJ64dzIK5v3Lz/x3BFUf149GrzivYccQli3iBQnbEl4wAGyxR+9we2Ak4DjgbGAz0MbNPvOP41sxWlbQ5cBWuZbQC8KmZ7S5pMLDIzC71db4P7GpmX0r61cyaS+oN3IUTcXvKzKb7bRcATbyC6IG+3NFV2FlUAmzFMPEzreAXwIJF6SZdhomf5dSHiZ9ZUNcTP9MKsG22RTd7cMTYVDYAdFm/ZRBgq4yZlZnZy2Z2AXAy5dLPUa+Z+349cIOZbY5zTrGF1MxsHNAbp4tzn6Tcq+IiK/fQQYAtEAjUOSFooBaQ1N7r2OToghNfAzgw8neC/x4VUjsi4b42AL43s9uBO4FueRkdCAQCgSWUUi615sD1klbGyUr/F9dt1Q9YUdIknAM92G8/GBgm6StgIrBhgn3tCJwpaRHwKxC/MzwQCAQKSBBgqwXMbCqwVNiyD1W80cwurLT908BS03vNbHCl5U6R783933uAe6oo2zzy/THgsYSHEQgEAplSQv6mdBxOIBAIBCpRwDAzSZ8Bc3Dj14vzCTIoeYdjZm3q2oZAIBBYTtjJzGblW7jkHc7ywL9enJGq/N/6tKt5oxpIG1rdsmnlqU55EH/uXa2RNjx85i8Lat6oBoohpPnLH+alrmPdVer+gqYNax79/nepyv8yP/10gVISYCuZKLX6Rj56GU9ddQ6XH7ANNx67x5J1L95zDTcd35+bT9iTe88ZxC+z4/8DlLqeTlZ11LUNWWj6pLUhi/I5ysrK2HPnbTjm0H3rxIZiOA/D77+dk/fZgZP26c3T992WVx1xEAVNbWPA85Km+rmGiQkOpw7IVy+jS999OezSOyus227A0Zx4yzOccPNwNtl6J8bef2Ot2hClrvV0sqijGGzIQtOnGM5Djntuv5G27TokLlcM1yILGz6f8T7PP34/Vz34HNcNe5Ep40bz9efpJpgWgNUkTYl8qnIoPc2sG/BH4CQ/QT4RweHUAfnqZbTZfEuatmhVYV2TZksC51g0f27sBIP1QU8nizqKwYYsNH2K4TwAfPP1l7w8eiQHHDowcdliuBZZ2PDFpzNov0V3Vmy6Eg0bNWKzHtsyYcyziepIQkapbWblJqv7z1LNMjP72v/9HngSlxosEcHh1AFZ62WMGXo1Vx/am7defIadDj+1TmzIh2LQPykGGyC9pk8xnAeAS88/i7P+cQkNGiR/tBTDtcjChg027sC70ybyy08/sGDeXKa+MoZZ3yUT5EtEAZKpSWomqUXuO9AXeCepqbXicHwm5t0qrTvNZ2mudu6KpDaSDqkNu4qFrPUydh50Bmc8MI4t+vRn8vD76sSGfCgG/ZNisAHSa/oUw3l48flnWXW11nTqnF9ijmK4FlnYsN5Gm7DvoJP5x7EHcsEJh7Bh+81o2LD24rMKlNpmDWC8pDeBycAIMxuZ1NbaauE8BBxUad1BwFAzG1BD2Ta4bM31ltrSy9h8p/68Nz6eFkx90dOpbxos+Wr6FMN5mDZ5ImNGjWDHHh047bjDmfjqWP7vxCMLakMxnAeAvvsewjWPjuayu5+iecuVWXv9JIlOig8z+8TMOvvPZrnkx0mpLYfzGNBP0orgWi3A2sCXkt7x6xpKukLS617B8zhf9jJge69Fc7qkgZKekDRS0gxJl+d2IulmP8D1rqQLI+s/k/RPSRP8790kjZL0saTjI9udGdn/hX5dM0kjvObNOz4rNJK6SxrrIzRGSVor35OTpV7G7K8+W/L9w4ljWG29eGGexaDZUQz6J8VgQxaaPsVwHv563kWMn/5fXp7yAdfcei/b9NyBq26KL7NQDNciq/+Ln2bPBGDmN18yYcyz9P5T7YnylZIAW62088xstqTJwO649DIHAY9QMavzUcDPZrald0yvSnoeJznwVzPrByBpIC5RZ1dclucPJV1vZl8A55rZD162YIykLczsLV//F2a2raR/A3cDPXEZo98FbpHUF2iHG/gSMNxHXbQGvjazPfz+W3nZg+uBvcxspndClwLxX98iRPUyysrKOGLgkbH0Mh4bcjqfvTWZuT//yFWHbs9Ofz6FGZPHMuvLT1GDBqy8+tr0O+XCGutJY0OUo444lPHjxjJ79iw6brwBZ593AYcPjH9KsrAhbR3FYMP3333LGScdTVlZmdOx2Xu/xJo+xXAe0lIM1yKr83DZGUcz5+cfaNioMcf/fQjNW66cuI64lM4snFrUw5F0GLCHmR0saTru4fwD8B8z6+THcrYA5voirXAyAgtZ2uH0NLNj/PJzwKVmNt63Vo7FOc61gL+Y2cM+BUNPM/tK0pHAtpHy//P7PQ8YAOTERZoDQ4BXgFHAo97WVyR1Al4DcrGNDYFvzKzvMo49Uz2c+jDxM+2EyfpCFhM/W7dcMQNL0lFfJn6mJe3EzzMO6suMd9/M22d06tzNnhg1PpUNAO3XalYQPZzazDTwFHC1pG5AUzOb5rvWcgjnIEZFC0nasYq6ltKvkbQh8FdgSzP7UdLdVK1583ul8r/jjlvAEDO7tfLOJHUH/gQM8a2uJ4F3zWzbao/Y40MKbwPo3r1H6SjcBQKB0qOEmji1FhZtZr8CL+OUMx+qYpNRwAk5eWdJm/hwuzlAixi7aAn8BvwsaQ3cZKQkjAKOlNTc738dSatLWhuYa2b3A1fitHA+BFpL2tZv21hSYfsbAoFAoBIuqrl0BNhqO5faQ8ATLB2xBnAHLiJtmlzc4Uxgb+AtYLEPv7sb+LGqis3sTUlv4MZkPgFeTWKYmT0vaVNggg97/BU4DNgYuELS78Ai4AQzWyhpAHCdpFa483aN33cgEAjUDQUe9E9LrTocM3uSSIPPzD4DOvnvvwN/95/K7Fxp+e5IHf0i3wcuY79tIt/vrlQ++tu1wLWVin+Ma/1UrnM6TnY6EAgEAnkQskUHAoFACVNCDZzgcAKBQKCkKSGPE3KpBQKBQKAghBZOLTNt2tRZTRuruok4qwF5K+jFKT84gzpquXywIdiwvNqwQYr6ocBRZmkJDqeWMbPW1f0uaUqaCVdpywcbgg3BhuK1Id5+ansP2REcTiAQCJQoMdUFioYwhhMIBAKBghBaOHVPWsHzLATTgw3BhmBDcdpQMyXUxKm15J2BQCAQqF226NLdnhnzWup62qzWpOSTdwYCgUCglimloIEwhhMIBAKBghBaOIFAIFDClFADJ7RwCo2XsG4QWW4gaaW6tCmQP5KWksWIypgXO5JWknS+pNv9cjtJ/WoqFygSMpCXLnmJ6UC1jAF2wckhAKwEPA9sl6QSSb2AdmY2VFJroLmZfZqgfENgDSL3gJn9L0H5fYF/AatTPh3AzKxlgjpWBPbDyVRE7bgoQR3bVVH+3gTl18HN9o6WHxe3PHC+pAVm9qKv72/AjsAtCWxoDRzD0scRW687xXEMBaYCOXHBL4FhwH/i7jtiQ9prkcX9kPf1THsdvC7XP4G1zeyPkjri1IbvjGt/fSc4nMLTxIvTAU6oLmkLR9IFQA+gPe6B0Ri4H+gZs/xfgAuA73AKqACGk96Oy+VAfzN7P0GZyjwN/Ix74CXWXpZ0H9AWmI5TggV3HLEecpL+BRwIvFepfBKHsyfwH0lnArsDHfy6JDyNkzZ/IWJHbFIeR1szO1DSwQBmNs/rUyW1IdW18KS9H9Jez1TXASeDMhQ41y9/BDwC1LLDKZ1OteBwCs9vkrqZ2TRYImedVCB+H6ArMA3AzL6WFEclNcepQHszm51wv1G+S+lsANY1s91TlO8BdLT8Y/v3xp2HxA+3HGY2S9KeuIfUVGBAHvasZGZ/y9cG0h3HQklNcQ9mJLUlj4c96a8FpL8f0l7PtNdhNTN7VNI5AGa2WFI+jis2orSi1ILDKTynAcMkfe2X18K9lSVhoZmZpNxDolnC8l/g3iQT47vSAKZIegR4isgDysyeSFDda5I2N7O387EFeAdYE/gmz/Kf4FqH+bxNz8E/pD0rABsBAyQl6lrEtZD+ZGbPJrXDk/dx4Fq6I4H1JD2AayUPzKOetNcC0t8Pac4DpL8Ov0lalXLnvQ15/p8loYT8TXA4hcbMXpfUAdcdJuADM1uUsJpHJd0KrCzpGOBI4PaaCkk6w3/9BHhZ0ggqOourY+y7f+T7XKBvZNlwkuI12fG237YRMEjSJ96O3DhQtV17kp7x5VsA70maXOk4QWPZRQAAGOxJREFU4nZpzQWmSxpTqfwpNRU0syQtyiqJOC0Bf5e0ACdrnnQ8LM1xjJY0DdjG7/dUM8snQ/Jq5Hkt0t4PEfI6DxlehzOA4UBbSa8CrYEBMcsuFwSHUyAk9TGzFyMthBztJCVqGZjZlZJ2BX7BOa5/mNnoGEVzD8n/+c8K/hMbMxuUZPtlkDYK6soMbAD3cBiepgI/3nEosKGZXSxpPWAtM5tcU9ksnJYn7+OQ1BOYbmYjJB2Ge+Bea2bVSWpUxeB89u/JKiour/OQ1XUws2mSdqD8ZfLDPF4mE1NKXWohtU2BkHShmV0gaWgVP1vCiKRmwHwzK5PUHneDP1eImztiwz24t+Gf/PIfgKsSHkdb4EszWyBpR1zQwr25OmOU3xD4xszm++WmwBpm9lmig0mBpJtxgRd9zGxTfx6eN7MtE9SRe+j/5h/63YBrkkQN5oukt4DO+HMP3AXsa2Y71Pa+q7Al1f2Qwf5TXQdJ+wMjzWyOpPN8+Uty47W1Qeeu3W3UyxNT17PWyisUJLVNmIdTIMzsAv93UBWf2A9pzzhgRR8C+gIwCBchEwtJoyWtHFn+g6RRCW3YIvogMLMfcYEMSXgcKJO0MS6SZ0PgwQTlh1EeZQcusmhY3MJ+zsljkt6T9Enuk2D/AFub2UnAfFhyHhK1GoGbgbmSOgNnAZ8D98UtnPI4FvuB/r2A68zsWspbwrGRtI2k1yX9KmmhpDJJvySsJtX9kMH1THUdgPO9s+kF7Abc4+sMeILDKTCSVpV0naRpkqZKutYPNCaqxszmAvsC15vZPkDHBOVbV+EsVk9oQwP/Nu8MklYheRft72a2GHcc15jZ6bggirg0MrOFuQX/PcnDfijugbAY2An3hp/kAQOwSG5OU26guDUVnWAcog/9a/N46Kc5jjk+qurPwAh/LI0T7DvHDcDBwAygKXC0X5eEtPdD2uuZ9jrkItL2AG42s6dJ/vKRHGXwKRDB4RSeh4GZuAluA/z3RxLWIUnb4sYORvh1SR72ZZLWj1S2ARUjruJwFS6q6GJJFwGvAVckrGOR3PyPwymfaJjkYTfThyQDIGkvkkkCNzWzMTgH/rmZDQb6JCgPcB3wJLCGpEuB8bjJf0nIPfQPI7+HfprjOBA3wH6kmX0LrEPy6wiAmf0XaGhmZWY2FDcBNglp74e01zPtdfjKB/McADwrN5G11p+xJeRvQtBAHbCKmV0cWb5E0t4J6zgNOAd40szelbQR8FKC8ucC4yWN9cu9gWOTGGBm90qagvuHFq7f/70kdeC6Ao8HLjWzT/2YzP0Jyh8PPCDpBm/DF7iHVVzmy6UZmiHpZOArErb0zOwBSVOBnb0Ne1vy+UkHAocAR5nZt/5lIMlDP+/j8Pt7HGjnV83COdCkzJW0Ai5K7HJceHTScP2090Pa65n2OhyAm/x7pZn9JGkt4MwE5RNT6NQ0aQlBAwVG0pXAFOBRv2oAsFlujCdhXc3M7Lc87ViN8lDYCUlDYSXdZ2Z/rmldIZDUHHcvz0lYbkvgfWBl4GKgFXC5mSUahVXKNENpSXMccmH1x+JehNpKagfcYmY7J7RhA1zmihWA070NN/lWT0HI6nqmtKGg90KXbt3t+bHpD2+NloUJGggOp0CoYqx/M8r7+RsAvyaI9cd3p92Ju5nX94Ocx5nZiTHL50J5NzKzi/yb3JpxQnkjdUwzs26R5YbA22ZW41iSpEfN7ACVz7+oQNx5F8og91ZaFEkzZGabSFobGGZmNaYZkjTezHpp6UmkifPS5Yuk6cBWwCQz6+rXvW1mm+dRV1NgfTP7MGG5TO6HfMnqOqS5F/KlS7fuNnrspNT1rN6ycRBgq09YdnMuAK7BRcEM93W/Kal3gvI34UN5gYuAObgIoRpDeX0f99+Bpj4KKdegX0h8Sd1T/d+08y/S5t7qgeterJzsMckDLu80Q2bWy//N696QdI2ZnabyibCV648zAXaBmS2U75eR1KiqumLY0h83P2oFYENJXYCLYtqQ6n5Iex7SXocIaVNO5UcJdakFh1MH+OiudkCT3DpLlqEYM/tCFTtvk+Rs2trMukl6w9f1o+9/j7PfIcAQSUPM7JwE+4zW8Y3/m3RyYWXS5t56ANfH/jbJI8typEoz5Mcc3jKzTnnsOxeBlWYi7FhJuReIXYETgWfyqGcwrqX0MoCZTZfUJk7BDO6HLM4DsKSlnm8W9bQpp+o9weEUGElH497o1sVl1t0GmECyaJov5FLBm3cUp+D6ruOSRSjvc1W1quI4ziq6Lpb8RLKupLS5t2aaWapMA+SZZiiHmf0u6U1J6yd4sOXKTvV/x9a0bTWcDRyFc7rHAc8Cd+RRz2Iz+1l5jGAvqyuLmPdDVefBv9StZ2ZvJbAjbRb1VPdCvpRQAyc4nDrgVFzX1UQz20kur9qFCes4HrgWF8L6JU5P56QE5XOhvKv7UN4BwHkJbYhG3zTBvd1OJYbjzLB7sRcwUNKn5Jd76wJJd+A0ivJKQGr5pxmKshbwrlwesiVBIDV1BS1rzCNSvsbzYGa/4x6KaR+M70g6BGjoAw9OwYXK10hW94Okl3HSEI1wL3MzJY01szOqLVhOqizqGd0LiSmlKLXgcArPfDObLwlJK5rZB3LpaWLjI8oOzdeALEJ5zSyaxBO5HGKXJ6lDbrJoZeZY/BQ9S6ltJmQQTr+mMRXfaGM5HN9KHGVmuwBpHixJXzhypM5BJpfOZTDl41g5p71Rwqr+ghsPW4DLDjAKuCQPezoD2/vFcUlaKEArM/vF9yIMNZdKKkn5vLOo5zCXDHUS/tkqaRUz+yFNndUjVEJtnOBwCs+XcmllngJGS/oR+LqGMhVQemXCO3EZCm6MrBtsbqJcvnwJJB2HmAasB/yIe9CtDHwj6XvgmFxXybIws8+rCkNNsP/O+URjRfZfJmmupFZmlveDKt8usQzGwMBFO56Oa53mpd3iHe+FZnYm5eJj+dRzKu6+zjn8ByTdZmbXx6yikdzclwPytCNNFnUkHYcLwpmHe4HJdQ0mdd71luBwCoy5NDQAgyW9hJsrMDJhNWmVCXcDuku62solgPckQcZfSddT3p3TAOgCvJnQjpG4yaujfJ19cRPnHsVF0m1dgw2plE+BiZI6WvIJq1HmA29LGk3F7rAapQGyGsuS0125HtgUFyXWEPgtZvmfzey5OPtZFt7xdk9Th+coXEDLb0BOwXMC7tjicBGuZTXenAzIRrhUO3HJO4u656+4OXX5yDvkRRBgCyyTyhFJKQZ70yoTfo9LO/KApK1xfddJb9spke+LgYfM7NWEdfQws+NzC2b2vKR/mtkZfo5NTaQNQ+0FHJFiDAhcaqFceqGc84h1LjMcy7oBOAiXuLQHLtvCxjHLviTpClyrIvpWnzTD8RuShnsboo43iSCfqPgCVUaC+9LMhhFJ3mpmn+DmacUtn2/XZo6PcZo8gWUQHE4BSRORVIm0yoQys1+A/pIG40JZWyWpwMzu8V1YmNnMPO34QdLfcPnlwKUW+dF30cSJmksbhpp3SLVc3rZ1c92SfsC/Nc7ppHkZyAsz+6+khmZWBgyVFGvAnvJWZHTSn5E8p9wqwOxK5WKPh3mGApMk5VLr7I3r8ouFXEqdS3BdWiNxsgunmVms9Dj+fj4L2IyKUxbinotzcJGTk0gohJeG0MIJVEdVEUlmZnslqONU8lAmlEv7viYRkSozG+wf2LEmjsrFvV4AnOz320DSYtyYUNIZ/of4up7ydY336xri+uFrIu8wVN/aHJHn/BdwD6aDIssrAN1xY0hDSSCTkAF55zEzs52yMMAyEOYzs6t9pFkv3P0wyMzeSFBFXzM7S9I+uDHF/XE5BuPmY3sAl0i3Hy4S9Ahcct243Aq8SLp5XfWa4HAKT7TZLtw/18FJKkjRFXMN8HdbOm/bCCq+4VbHabgxki3N54jyfeU3SzrdzP4d1xjf1/2XZfxcYw6uNGGoGbQ2VzCzLyLL43000g91MOHvz7hxtJNxAQDr4VL8x0LSHiz9Vp/o5UHSurixlp64ls14nEDflzHKNsE94DfGPaxvMidTkJRcZuc/4bp4f0g4L2hVM7tT0qm+u3usyhPcxmFxghDszAhRaoFlYmZj5dJ+HIJ7i/8UuCVOWUkdfBh1t6p+j9Hv3qaqMFMzm6KYs8Jx4wO7RgdGzewTOYXE54HYDkfSJriB1jZUjLarsQsjo5DkvOa/eP4QXTCzkyOLrfO0J1/2NqfdMh//QuMjvq6tqaCkW4CVcPoxd+DmZMXOqRdhKC4cen+/fJhft2uMsvfgWuqv4ELdN8W92CTlGUkf4LrUTvRdZPMTlM+F43/jnfDXuAnacXlJ0rG4TA3RLrXaC4susWzRweEUCP9wPQjXmpmNa7orYZfGGbjMvldV8Vucfvcm1fzWNKYNjauKwjGzmZKSCncNwznbO0gYbZdRSHKaQeJJko4xswpdeD40Np8HdhqOYGnnMrCKdVWxnZltIektM7tQ0lUkG3fJ0dqcBk6OuyXFdRodc+HpPmQ/r/NnZmf7yLZf/P3xG05MLS6XSGoF/B+utdYS12KMyyH+bzTlUwiLjhAcTuH4APcG1998ynZJSW5mzOxY/zfffvfXl/GQPAo3DyMOC/P8rSoWm1kaCd68Q5L9dmPl0uq3M7MXJK2EGz+Kw+nAU3Kz63Mty+7AirjB7lpHTqzsEFyyzGiKnha4l5o4zPN/58plN56Nk3ZOyizfyn3IL+derOKwZKKvmS1O2A22BEmHR75Hf7p36a2Xxsxyom8/41p8iTCzfM5bKgotoJaW4HAKx364Fs5LkkbiIrPyvlfkcqm1oWJXVE3/WKcBT0o6lHIH0wM34L3PMktVpLOq1qoX1begquIZSSfi0uzk0wURDUlOjCJaMEBbXKqgW3AZGKrFzL4HtpPUBzf+AS4I4cV87cmD13ABAqtRsdU7B4g7w/4/chORr8A5TiO/NDdH4sKz/+3reA2XySEO0XtKVMxEHns+EhWznTfBXcdpxHQ4fizyWmBb3KD/BOB0H14dC0mdcHLv0fGwWPvPmxLyOEEPp8D4AeW9cW+AfXD910+a2fMJ6rgP94CcTnlXlMV9s5e0E+VZAd4t8EMyakdVwlRmNaRVySCsPFdPZlow9QE/96lJki5KSesuKzBAUn8zyyfzdCb47rH7Yo7JIWkicCPlrbSDgL+YWbUTkCPlL8DNb+uIS4L6R1wwyYCEpsemW/ceNu6111PX06JJg6CHUx/xs6gfwE26XAU3yHo2bsA9Lj1w/d55vS2Y2Uskk6SuFVJ0QTwFdAOQ9LiZxZ7cV4lMtGDqCmWQqcBHiJ2Ii5Y0nPT4zWYWd7B9jKTdzOyzSvUOwiWErTOHg5uE2a7GrcqRmd0XWb5fTqo6LgNwc3/eMLNBktYgv8zb9ZYGdW3A8oyZ/WBmtyaYWJbjHdx8mpJE0lmR7/tX+u2fcaqIfE8zIDtWFbVghlG3D8hEmFkLM2tZxadFgm6oe3FdgtfjusQ2pVxfJg6n43ICLnmwy4n0nQHskKCe1Eh6RtJw//kP8CGROWcxeEnS2ZLaSNrA36cjJK2iqhPNVmaeuezbiyW1xGX0qPWAASn9p1CEFk4JoXJFwxbAez6cNzr2EavroAg4iPLM0udQcZLk7jhF0eqwZXxPSlQL5ljcGMzy9kba3sw6R5ZfkhQ7J56ZPSs3Afk5SXsDR+PGUnqb2Y8Z21oTUQG2xcDnceYBRTjQ/z3W/809io8kXrTZFD8edjtujPRXChCxWEJDOMHhlBjDcWqEr1RavwPwVeHNyRst43tVy1WRG2SODjDnysbJuBBNS3O7Dx5ojUto+pOZPRbrKOoHb0jaxswmAsjl1kuUE8/MxkgaiEuR9Bqwc4IuucywSrkJJTWUdKiZPVBdOUlbAl/kunglHYEL8vkMGBw3iMXMTvRfb/GBQS2rmvdWqkjaHRdU0RC4w8wuS1pHcDilxV64TAEVbmI/3+ACEuSdqmOqa6HU2GIxs7ihy8uiprQ0y5PD2Ro4XFIuCGN94H15cTerIZFpZBxJuJDwnYHv5QbGkkSY5Y3vvjoJF2U4HDcR+CScSOB03JhpddwK7OLr6g0MwWXA6ALchhubiWvLOpRrCyGptyWUj09MAZo4fqL1jbiJvF/iplgMt4SZ1oPDKS2yyBRQDFTXQkkaWp0PxZSWpq7JO4EpZJrxOg334TSVJuC69M7EvUTsZWbTY5RvGGnFHAjcZmaPA4/7SMZY+EmnBwLvEYkeBWrV4RQotc1WwH9zIeKSHsa9AAeHU4/JIlNAnZNBCyUtxZSWpk4xL+ImaXUqzh1JHXZeQDaKZCq4A5gFrG9mc2KWbyipkbn8bTtTPoYDyZ6Re+PGxBbUuGVGFFAPZx2cImqOL6lBr6oqgsMpLbLIFBAorrQ0dYqkPXGTRtfGRVVtALxP+WTWUiCaqaBM0qcJnA24eTdjJc3CZV54BZZkV0+SNukTXALRgjmcadOmjmraWKtlUFUTSVGNq9vM7LbIclVuLXHATpj4WUL4uP4ncSlklsoUYGbf1pVtpYR/m38K92BYKi2NmX1XV7YVGh+R1gd4wcy6+knBB+fSKJUCksooT20kXGt/LsnmI22DS+b6vJUrjm4CNLeYYnSSHsfNwxlDAfVwCoGkbXEBFLv55XMAzGxIonqCwyk9iiVTQKlTKS3NcnkeJU0xsx7e8XQ1J9sw2cy2qmvbSg0f3VYZq/XUNgXAT4r+CNfl+BXwOnCImb2bpJ7QpVaCFEumgFLHO5jlzslU4idJzXHdSA9I+h43hyWQnJXNyUQsQU4mouTxSVVPBkbhwqLvSupsILRwAoHlGrkM2fNx3U+H4VLyP5AggWrAI2mamXWrtO6NXJ6+QHA4gcByyTLysOUGhucDHwPnmtmYghpWgqhcJqIXFSdltwDKzIkEBghdaoHAckl182f8JL9OuAmTnZa1XWAJWchELBeEFk4gEKgSSceZ2a11bUcpoYqCfk2BRglDtOs1weEEAoFABigi6GdmbX0G7VvMrEZBv+WFIE8QCAQC2XAS0BP4BcDMZgCr16lFRUZwOIFAIJANC8xsYW6h1AT9CkFwOIFAIJANJS3oVwjCGE4gEAhkgKQGOEG/vrgQ81E43ZjwkPUEhxMIBAKBghDm4QQCgUAKJD1qZgfkROsq/16TiN3yRHA4gUAgkI45knoC/QlBAtUSHE4gEAik4y3gSpy8wSPAQzGVRpc7whhOIBAIZIDPMnCQ/zTBCbs9bGYf1alhRURwOIFAIJAxkroCdwFbFIGketEQ5uEEAoFABkhqLKm/pAeA53CCZfvVsVlFRWjhBAKBQAr8JM+DgT2AycDDwFM5qepAOcHhBAKBQAokvQQ8CDwehOuqJzicQCAQCBSEMIYTCAQCgYIQHE4gEAgECkJwOIF6i6QySdMlvSNpmKSVUtS1o6T/+O97Sjq7mm1XlnRiHvsYLOmvcddX2uZuSQMS7KuNpHeS2hgIpCE4nEB9Zp6ZdTGzTsBC4Pjoj3Ik/h8ws+Fmdlk1m6wMJHY4gUB9JzicwPLCK8DG/s3+fUk3AdOA9ST1lTRB0jTfEmoOIGl3SR9IGg/sm6tI0kBJN/jva0h6UtKb/rMdcBnQ1reurvDbnSnpdUlvSbowUte5kj6U9ALQvqaDkHSMr+dNSY9XarXtIukVSR9J6ue3byjpisi+j0t7IgOBfAkOJ1Dv8cqLfwTe9qvaA/eaWVfgN+A8YBcz6wZMAc6Q1AS4HZeQcXtgzWVUfx0w1sw6A92Ad4GzgY996+pMSX2BdsBWQBegu6Tekrrj0qB0xTm0LWMczhNmtqXf3/s4/ZUcbYAdcPNBbvHHcBTws5lt6es/RtKGMfYTCGROSN4ZqM80lZRLovgKcCewNvC5mU3067cBOgKvSgJYAZgAdAA+9br0SLofOLaKffQBDgcwszLgZ0l/qLRNX/95wy83xzmgFsCTZjbX72N4jGPqJOkSXLddc5zIV45Hzex3YIakT/wx9AW2iIzvtPL7Dvm9AgUnOJxAfWaemXWJrvBOJToDXMBoMzu40nZdyC7VvIAhZnZrpX2clsc+7gb2NrM3JQ0Edoz8Vrku8/v+i5lFHROS2iTcbyCQmtClFljemQj0lLQxgKSVJG0CfABsKKmt3+7gZZQfA5zgyzaU1BKYg2u95BgFHBkZG1pH0urAOGAfSU0ltcB139VEC+AbSY2BQyv9tr+kBt7mjYAP/b5P8NsjaRNJzWLsJxDInNDCCSzXmNlM31J4SNKKfvV5ZvaRpGOBEZJmAeOBTlVUcSpwm6SjgDLgBDObIOlVH3b8nB/H2RSY4FtYvwKHmdk0SY8A04HPcd1+NXE+MMlv/zYVHduHwFhgDeB4M5sv6Q7c2M40uZ3PBPaOd3YCgWwJqW0CgUAgUBBCl1ogEAgECkJwOIFAIBAoCMHhBAKBQKAgBIcTCAQCgYIQHE4gEAgECkJwOIFAIBAoCMHhBAKBQKAgBIcTCAQCgYLw/8rh4YY/ITTFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6,6))\n",
    "plot_confusion_matrix(confuse, data.classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's set up everything from scratch so we could set it up in an external app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unidecode import unidecode\n",
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Arabic',\n",
       " 'Chinese',\n",
       " 'Czech',\n",
       " 'Dutch',\n",
       " 'English',\n",
       " 'French',\n",
       " 'German',\n",
       " 'Greek',\n",
       " 'Irish',\n",
       " 'Italian',\n",
       " 'Japanese',\n",
       " 'Korean',\n",
       " 'Polish',\n",
       " 'Russian',\n",
       " 'Spanish',\n",
       " 'Vietnamese']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('data.classes', 'rb') as f:\n",
    "    classes = pickle.load(f)\n",
    "classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LetterTokenizer(BaseTokenizer):\n",
    "    \"Character level tokenizer function.\"\n",
    "    def __init__(self, lang): pass\n",
    "    def tokenizer(self, t:str) -> List[str]:\n",
    "        t = unidecode(t).lower() ## Decode in tokenizer (ideally would be a separate preprocessor)\n",
    "        out = []\n",
    "        i = 0\n",
    "        while i < len(t):\n",
    "            if t[i:].startswith(BOS):\n",
    "                out.append(BOS)\n",
    "                i += len(BOS)\n",
    "            else:\n",
    "                out.append(t[i])\n",
    "                i += 1\n",
    "        return out\n",
    "            \n",
    "    def add_special_cases(self, toks:Collection[str]): pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "itos = [UNK, BOS] + list(string.ascii_lowercase + \" -'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab=Vocab(itos)\n",
    "tokenizer=Tokenizer(LetterTokenizer, pre_rules=[], post_rules=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>cl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>Arabic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>Chinese</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>Czech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>Dutch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>English</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td>French</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>German</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td>Greek</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td>Irish</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td>Italian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td></td>\n",
       "      <td>Japanese</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td></td>\n",
       "      <td>Korean</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td></td>\n",
       "      <td>Polish</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td></td>\n",
       "      <td>Russian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td></td>\n",
       "      <td>Spanish</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td></td>\n",
       "      <td>Vietnamese</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   text          cl\n",
       "0            Arabic\n",
       "1           Chinese\n",
       "2             Czech\n",
       "3             Dutch\n",
       "4           English\n",
       "5            French\n",
       "6            German\n",
       "7             Greek\n",
       "8             Irish\n",
       "9           Italian\n",
       "10         Japanese\n",
       "11           Korean\n",
       "12           Polish\n",
       "13          Russian\n",
       "14          Spanish\n",
       "15       Vietnamese"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "empty = pd.DataFrame({'text':'', 'cl':classes})\n",
    "empty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "processors = [TokenizeProcessor(tokenizer=tokenizer, mark_fields=False),\n",
    "            NumericalizeProcessor(vocab=vocab)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = TextList.from_df(empty, processor=processors).no_split().label_from_df(cols='cl').databunch(bs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = text_classifier_learner(data, bptt=30, emb_sz=200, nh=300, nl=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('fastai_min')\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check it's not in the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grep -ir '^Wu' data/names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Category Chinese,\n",
       " tensor(1),\n",
       " tensor([1.5628e-04, 8.2566e-01, 1.5067e-04, 7.8611e-04, 5.4162e-03, 6.0296e-06,\n",
       "         1.6915e-03, 7.5540e-05, 1.2069e-03, 2.6427e-05, 2.3748e-03, 1.0973e-01,\n",
       "         4.8861e-02, 1.4435e-04, 4.1366e-05, 3.6734e-03]))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict('Wu') # Chinese"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predictions(name):\n",
    "    return sorted(zip(classes, (_.item() for _ in learn.predict(name)[2])), key=lambda x: x[1], reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does it do in practice?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Polish', 0.9237067699432373),\n",
       " ('Czech', 0.0443655289709568),\n",
       " ('Vietnamese', 0.00837066862732172),\n",
       " ('Spanish', 0.006321582943201065),\n",
       " ('Chinese', 0.0038041360676288605)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions(\"Wojtyła\")[:5] # Polish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Czech', 0.866197407245636),\n",
       " ('Polish', 0.10485668480396271),\n",
       " ('Russian', 0.026793140918016434),\n",
       " ('Korean', 0.00042316922917962074),\n",
       " ('Japanese', 0.00041837719618342817)]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions(\"Dvořák\")[:5] # Czech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Italian', 0.8921791911125183),\n",
       " ('Russian', 0.0403725765645504),\n",
       " ('Japanese', 0.025237590074539185),\n",
       " ('Spanish', 0.015365079045295715),\n",
       " ('Arabic', 0.01298986654728651)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions(\"Gaddafi\")[:5] # Arabic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Dutch', 0.35977253317832947),\n",
       " ('German', 0.21167784929275513),\n",
       " ('French', 0.18858234584331512),\n",
       " ('English', 0.1542830914258957),\n",
       " ('Irish', 0.02631647326052189)]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions('Goethe')[:5] # German"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes it does bad even if it's in the source data (it may not have ended up in training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/names/Korean.txt:Kim\r\n",
      "data/names/Japanese.txt:Kimio\r\n",
      "data/names/Japanese.txt:Kimiyama\r\n",
      "data/names/Japanese.txt:Kimura\r\n",
      "data/names/Vietnamese.txt:Pham\r\n",
      "data/names/Vietnamese.txt:Kim\r\n",
      "data/names/English.txt:Kimber\r\n",
      "data/names/English.txt:Kimble\r\n",
      "data/names/French.txt:Pascal\r\n"
     ]
    }
   ],
   "source": [
    "!grep -Er 'Pascal|Pham' data/names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Spanish', 0.9103199243545532),\n",
       " ('Italian', 0.07529251277446747),\n",
       " ('Polish', 0.005148978438228369),\n",
       " ('Greek', 0.0036756773479282856),\n",
       " ('Czech', 0.0034040361642837524)]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions(\"Pascal\")[:5] # French"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Dutch', 0.2937869727611542),\n",
       " ('Vietnamese', 0.13536876440048218),\n",
       " ('English', 0.11861108988523483),\n",
       " ('French', 0.07520488649606705),\n",
       " ('Irish', 0.0743907243013382)]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions(\"Pham\")[:5] # Vietnamese"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But sometimes it gets it right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grep -ir '^Meijer' data/names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Dutch', 0.9885940551757812),\n",
       " ('German', 0.008121304214000702),\n",
       " ('Czech', 0.0013009845279157162),\n",
       " ('Korean', 0.00039360582013614476),\n",
       " ('English', 0.0003091120161116123)]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions(\"Meijer\")[:5] # Dutch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Polish', 0.9900423288345337),\n",
       " ('Czech', 0.004603931214660406),\n",
       " ('Chinese', 0.002563303103670478),\n",
       " ('Korean', 0.0009222071967087686),\n",
       " ('Dutch', 0.0005194866680540144)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions('Wójcik')[:5] # Polish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model is not bad; but definitely sub-human."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does it think about our ambiguous \"Michel\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Czech', 0.6636908054351807),\n",
       " ('German', 0.11761205643415451),\n",
       " ('English', 0.061124321073293686),\n",
       " ('Irish', 0.04792550206184387),\n",
       " ('Polish', 0.027553826570510864),\n",
       " ('French', 0.026092179119586945),\n",
       " ('Dutch', 0.020014718174934387)]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions('Michel')[:7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting from a pretrained custom model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self, n_input, n_hidden, n_output, bn=False, use_cuda=False):\n",
    "        super().__init__()\n",
    "        self.i_h = nn.Embedding(n_input,n_hidden)\n",
    "        self.bn = nn.BatchNorm1d(n_hidden) if bn else None\n",
    "        self.o_h = nn.Linear(n_hidden, n_output)\n",
    "        self.h_h = nn.Linear(n_hidden, n_hidden)\n",
    "        self.use_cuda = use_cuda\n",
    "        self.reset()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # I'm not quite sure why the batch size seems to change to 720 in validation...\n",
    "        if self.h.shape[0] != x.shape[1]:\n",
    "            self.reset(x.shape[1])\n",
    "        h = self.h\n",
    "        x = self.i_h(x)\n",
    "        for xi in x:\n",
    "            h += xi\n",
    "            h = self.h_h(h)\n",
    "            h = F.relu(h)\n",
    "            if self.bn:\n",
    "                h = self.bn(h)\n",
    "        self.h = h.detach()\n",
    "        o = self.o_h(h)\n",
    "        return o\n",
    "        \n",
    "    def reset(self, size=None):\n",
    "        size = size or 1\n",
    "        self.h = torch.zeros(size, n_hidden)\n",
    "        if self.use_cuda:\n",
    "            self.h = self.h.cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_letters = len(vocab.itos)\n",
    "n_hidden = 128\n",
    "n_output = len(classes)\n",
    "model = Model(n_letters, n_hidden, n_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('models/rnn-bal-1.model', 'rb') as f:\n",
    "    state = pickle.load(f)\n",
    "    model.load_state_dict(state)\n",
    "model = model.cpu()\n",
    "model = model.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "for param in model.parameters():\n",
    "    param.requires_grad = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = 'Wójcik' # Polish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxbosWojcik'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decode = BOS + unidecode(name)\n",
    "decode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxbos', 'w', 'o', 'j', 'c', 'i', 'k']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens = tokenizer.process_all([decode])[0]\n",
    "tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 24, 16, 11, 4, 10, 12]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nums = vocab.numericalize(tokens)\n",
    "nums"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1],\n",
       "        [24],\n",
       "        [16],\n",
       "        [11],\n",
       "        [ 4],\n",
       "        [10],\n",
       "        [12]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([nums]).transpose(1,0)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ -8.0967,  -6.4112,   7.0235,   4.0332,   1.4341, -15.9867,  -0.3718,\n",
       "          -9.9789, -17.9649,  -7.5314,  -2.7700,  -0.1450,  -3.1552,   1.9744,\n",
       "         -14.5163, -15.3465]])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = model(x).detach()\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2.5545e-07, 1.3781e-06, 9.4171e-01, 4.7340e-02, 3.5193e-03, 9.5652e-11,\n",
       "        5.7831e-04, 3.8891e-08, 1.3231e-11, 4.4956e-07, 5.2559e-05, 7.2556e-04,\n",
       "        3.5759e-05, 6.0412e-03, 4.1617e-10, 1.8145e-10])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probs = F.softmax(result[0], dim=0)\n",
    "probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Czech: Probability 94.17%\n",
      "Dutch: Probability 4.73%\n",
      "Russian: Probability 0.60%\n"
     ]
    }
   ],
   "source": [
    "for prob, idx in zip(*probs.topk(3)):\n",
    "    print(f'{classes[idx]}: Probability {prob:0.2%}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_probs(name):\n",
    "    decode = BOS + unidecode(name)\n",
    "    tokens = tokenizer.process_all([decode])[0]\n",
    "    nums = vocab.numericalize(tokens)\n",
    "    x = torch.tensor([nums]).transpose(1,0)\n",
    "    model.reset()\n",
    "    result = model(x).detach()\n",
    "    probs = F.softmax(result[0], dim=0)\n",
    "    return probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_top_probs(name, n=3):\n",
    "    probs = get_probs(name)\n",
    "    for prob, idx in zip(*probs.topk(n)):\n",
    "        print(f'{classes[idx]}: Probability {prob:0.2%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In reality the model doesn't do great by human standards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Irish: Probability 72.69%\n",
      "English: Probability 16.42%\n",
      "Japanese: Probability 3.84%\n"
     ]
    }
   ],
   "source": [
    "print_top_probs('Goethe') # German"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "German: Probability 51.32%\n",
      "English: Probability 34.13%\n",
      "Chinese: Probability 9.00%\n"
     ]
    }
   ],
   "source": [
    "print_top_probs('Jinping') # Chinese"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Korean: Probability 41.02%\n",
      "Russian: Probability 22.39%\n",
      "Dutch: Probability 16.08%\n"
     ]
    }
   ],
   "source": [
    "print_top_probs('Kim') # Korean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Korean: Probability 41.59%\n",
      "Chinese: Probability 34.24%\n",
      "Vietnamese: Probability 14.89%\n"
     ]
    }
   ],
   "source": [
    "print_top_probs('Đặng') # Vietnamese"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Arabic: Probability 91.50%\n",
      "Russian: Probability 4.34%\n",
      "Czech: Probability 2.74%\n"
     ]
    }
   ],
   "source": [
    "print_top_probs('Zahir') # Arabic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's also possible to use `learn.load` to load in the model, if you make some fake data.\n",
    "\n",
    "We need at least 2 rows or it will complain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0\n",
       "0   \n",
       "1   "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "empty = pd.DataFrame([[' ']]*2)\n",
    "empty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "processors = [TokenizeProcessor(tokenizer=tokenizer, mark_fields=False),\n",
    "            NumericalizeProcessor(vocab=vocab)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = TextList.from_df(empty, processor=processors).no_split().label_const().databunch(bs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(n_letters, n_hidden, n_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = learn.load('rnn-bal-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.model = learn.model.eval().cpu()\n",
    "for param in learn.model.parameters():\n",
    "    param.requires_grad = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, _ = data.one_item('Dvořák') # Czech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[1.2497e-05, 3.4860e-06, 5.5647e-01, 1.3278e-04, 1.5334e-03, 4.7975e-04,\n",
       "         7.6774e-05, 6.1160e-06, 4.5827e-08, 2.0946e-05, 1.2422e-04, 5.2584e-06,\n",
       "         4.1456e-01, 2.5337e-02, 1.2437e-03, 2.2771e-11]])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.model.reset()\n",
    "probs = F.softmax(learn.model(x.cpu()))\n",
    "probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Czech: Probability 55.65%\n",
      "Polish: Probability 41.46%\n",
      "Russian: Probability 2.53%\n"
     ]
    }
   ],
   "source": [
    "for prob, idx in zip(*probs[0].topk(3)):\n",
    "    print(f'{classes[idx]}: Probability {prob:0.2%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using the model to find similar names\n",
    "\n",
    "The idea is to dig into the representation in the 50 dimensional activation and use this to compare names.\n",
    "\n",
    "Two names are similar if they are close together in this embedding space.\n",
    "It's not totally obvious that the RMS distance is appropriate for this, but it's what we'll use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.callbacks.hooks import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "      <th>valid</th>\n",
       "      <th>bal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Ahn</td>\n",
       "      <td>ahn</td>\n",
       "      <td>False</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Baik</td>\n",
       "      <td>baik</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Bang</td>\n",
       "      <td>bang</td>\n",
       "      <td>False</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Byon</td>\n",
       "      <td>byon</td>\n",
       "      <td>False</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Cha</td>\n",
       "      <td>cha</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       cl  name ascii_name  valid  bal\n",
       "0  Korean   Ahn        ahn  False   13\n",
       "1  Korean  Baik       baik   True    0\n",
       "2  Korean  Bang       bang  False   13\n",
       "3  Korean  Byon       byon  False   15\n",
       "4  Korean   Cha        cha   True    0"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('names_clean.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = TextList.from_df(df, cols='ascii_name', processor=processors).no_split().label_from_df('cl').databunch(bs=1024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model = Model(n_letters, n_hidden, n_output).cuda()\n",
    "# learn = Learner(data, model)\n",
    "# learn = learn.load('rnn-bal-1')\n",
    "learn = text_classifier_learner(data, bptt=30, emb_sz=200, nh=300, nl=2)\n",
    "learn.load('fastai_min')\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the structure of our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('0', MultiBatchRNNCore(\n",
       "    (encoder): Embedding(31, 200, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(31, 200, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(200, 300)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(300, 200)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "    )\n",
       "  )), ('1', PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(600, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.4)\n",
       "      (2): Linear(in_features=600, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1)\n",
       "      (6): Linear(in_features=50, out_features=16, bias=True)\n",
       "    )\n",
       "  ))]"
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(learn.model.named_children())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's capture the output of the 50 dimensional embedding near the end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = 17"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=600, out_features=50, bias=True)"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(learn.model.modules())[layer]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [],
   "source": [
    "def embed(x):\n",
    "    #with hook_output(list(learn.model.children())[-1]) as hook_a: \n",
    "    with hook_output(list(learn.model.modules())[layer]) as hook_a:\n",
    "        preds = learn.predict(x)\n",
    "        return hook_a.stored[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 32.2 s, sys: 2.22 s, total: 34.4 s\n",
      "Wall time: 34.4 s\n"
     ]
    }
   ],
   "source": [
    "%time df = df.assign(embed = df.name.apply(embed))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cl</th>\n",
       "      <th>name</th>\n",
       "      <th>ascii_name</th>\n",
       "      <th>valid</th>\n",
       "      <th>bal</th>\n",
       "      <th>embed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Ahn</td>\n",
       "      <td>ahn</td>\n",
       "      <td>False</td>\n",
       "      <td>13</td>\n",
       "      <td>[tensor(0.7824, device='cuda:0'), tensor(2.431...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Baik</td>\n",
       "      <td>baik</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "      <td>[tensor(0., device='cuda:0'), tensor(0., devic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Bang</td>\n",
       "      <td>bang</td>\n",
       "      <td>False</td>\n",
       "      <td>13</td>\n",
       "      <td>[tensor(0., device='cuda:0'), tensor(1.9753, d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Byon</td>\n",
       "      <td>byon</td>\n",
       "      <td>False</td>\n",
       "      <td>15</td>\n",
       "      <td>[tensor(0., device='cuda:0'), tensor(4.9263, d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Korean</td>\n",
       "      <td>Cha</td>\n",
       "      <td>cha</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "      <td>[tensor(0., device='cuda:0'), tensor(0., devic...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       cl  name ascii_name  valid  bal  \\\n",
       "0  Korean   Ahn        ahn  False   13   \n",
       "1  Korean  Baik       baik   True    0   \n",
       "2  Korean  Bang       bang  False   13   \n",
       "3  Korean  Byon       byon  False   15   \n",
       "4  Korean   Cha        cha   True    0   \n",
       "\n",
       "                                               embed  \n",
       "0  [tensor(0.7824, device='cuda:0'), tensor(2.431...  \n",
       "1  [tensor(0., device='cuda:0'), tensor(0., devic...  \n",
       "2  [tensor(0., device='cuda:0'), tensor(1.9753, d...  \n",
       "3  [tensor(0., device='cuda:0'), tensor(4.9263, d...  \n",
       "4  [tensor(0., device='cuda:0'), tensor(0., devic...  "
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {},
   "outputs": [],
   "source": [
    "def closest(name, n=10):\n",
    "    e = embed(name)\n",
    "    dist = [d(e, _) for _ in df.embed]\n",
    "    for idx in np.argsort(dist)[:10]:\n",
    "        print(f'{df.name.iloc[idx.item()]} ({df.cl.iloc[idx.item()]}): {dist[idx]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's not immediately clear in what sense these are similar; but it doesn't seem random to me"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ahn (Korean): 0.0\n",
      "Hor (Chinese): 74.33894348144531\n",
      "Gul (Russian): 78.02725219726562\n",
      "Noh (Korean): 88.23501586914062\n",
      "Hon (Russian): 89.40960693359375\n",
      "Ryu (Korean): 90.15037536621094\n",
      "Byon (Korean): 92.82701110839844\n",
      "Jermy (English): 93.4688720703125\n",
      "Bishop (English): 96.69548034667969\n",
      "Heron (English): 97.36801147460938\n",
      "CPU times: user 1.46 s, sys: 188 ms, total: 1.65 s\n",
      "Wall time: 1.65 s\n"
     ]
    }
   ],
   "source": [
    "%time closest('Ahn')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Turner (English): 34.640953063964844\n",
      "Raizer (Russian): 41.138641357421875\n",
      "Reuter (German): 45.08790588378906\n",
      "Gunter (English): 46.127220153808594\n",
      "Mendel (German): 48.00371551513672\n",
      "Render (English): 48.202239990234375\n",
      "Raeburn (English): 52.79540252685547\n",
      "Rosenberger (German): 53.13862228393555\n",
      "Rebinder (Russian): 53.502662658691406\n",
      "Rosser (English): 54.7049446105957\n"
     ]
    }
   ],
   "source": [
    "closest('Ruder')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Oelberg (German): 28.07500457763672\n",
      "Mordberg (Russian): 29.14569854736328\n",
      "Gramberg (Russian): 30.82413101196289\n",
      "Engman (Russian): 31.429853439331055\n",
      "Burman (English): 33.723426818847656\n",
      "Bumgarner (German): 33.76372528076172\n",
      "Egger (German): 35.34405517578125\n",
      "Großer (German): 35.70815658569336\n",
      "Ranger (English): 35.80017852783203\n",
      "Grainger (English): 36.01886749267578\n"
     ]
    }
   ],
   "source": [
    "closest('Gugger')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Manus (Irish): 49.601768493652344\n",
      "Jemaitis (Russian): 56.250274658203125\n",
      "Horos (Russian): 63.35132598876953\n",
      "Klimes (Czech): 73.13825225830078\n",
      "Bertsimas (Greek): 73.65045166015625\n",
      "Tsogas (Greek): 79.87809753417969\n",
      "Simonis (Dutch): 85.69441223144531\n",
      "Honjas (Greek): 86.7238998413086\n",
      "Mihelyus (Russian): 87.06715393066406\n",
      "Grotus (Russian): 88.79036712646484\n"
     ]
    }
   ],
   "source": [
    "closest('Thomas')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "East (English): 59.85662841796875\n",
      "Gammer (English): 60.93962097167969\n",
      "Gale (English): 61.32642364501953\n",
      "Gass (German): 65.68402099609375\n",
      "Abrams (English): 68.60626983642578\n",
      "Groer (Russian): 72.51294708251953\n",
      "Bannister (English): 72.60062408447266\n",
      "Glencross (English): 73.78807830810547\n",
      "Moss (English): 74.06442260742188\n",
      "Gander (English): 75.61122131347656\n"
     ]
    }
   ],
   "source": [
    "closest('Ross')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wan (Chinese): 55.373172760009766\n",
      "Wei (Chinese): 63.00840759277344\n",
      "Won (Chinese): 96.11245727539062\n",
      "Gwang (Korean): 103.33255004882812\n",
      "Gwock (Chinese): 118.81124114990234\n",
      "Weng (Chinese): 131.87939453125\n",
      "Wane (English): 141.58985900878906\n",
      "Twigg (English): 147.58078002929688\n",
      "Wain (English): 153.88088989257812\n",
      "Gowing (English): 156.93133544921875\n"
     ]
    }
   ],
   "source": [
    "closest('Wu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cheryshev (Russian): 4.010871410369873\n",
      "Dobryshev (Russian): 23.908416748046875\n",
      "Chalyshev (Russian): 23.938615798950195\n",
      "Chanyshev (Russian): 28.119571685791016\n",
      "Cherushov (Russian): 28.66720962524414\n",
      "Tchanyshev (Russian): 30.515729904174805\n",
      "Chehov (Russian): 30.542831420898438\n",
      "Tchalyshev (Russian): 34.40039825439453\n",
      "Yachmentsev (Russian): 36.298770904541016\n",
      "Jerebyatiev (Russian): 36.888675689697266\n"
     ]
    }
   ],
   "source": [
    "closest('Chebyshev')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}