RNNs v01.ipynb

rnns-v01.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "RNNs v01.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "https://gist.github.com/nirshlezinger1/f50e562745367272ed4b7c28b01ca18d#file-rnns-v01-ipynb",
      "authorship_tag": "ABX9TyN7N+hyM2/cCcRF+u0KJrhM",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/nirshlezinger1/f50e562745367272ed4b7c28b01ca18d/rnns-v01.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gB85tJFQGkX9"
      },
      "source": [
        "## Training Recurrant Neural Networks\n",
        "In this notebook we train RNNs for the task of *sentiment analysis*. Such tasks correspond to identifying the sentiment of the writer from a piece of text, and can be viewed as *many-to-one* time sequence processing. In particular, we will train two RNN types - a vanilla RNN and a GRU - for classifying tweets to either *happy* or *sad*.\n",
        "\n",
        "Since we are dealing with text processing, we will need to convert words into vectors which can be fed as input to the neural network. For this reason we include the package *torchtext* which provides these capbilities, as we show next."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IjDXFqbEZ2j7"
      },
      "source": [
        "import csv\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "import torchtext\n",
        "import random\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt \n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BgV5OgklHkCr"
      },
      "source": [
        "# Sentiment140 Dataset\n",
        "The data set we will use is the Sentiment140 data set, which contains tweets with either a positive or negative emotion. This dataset was collected by a group of students from Stanford who were working on machine learning projects.\n",
        "\n",
        "This dataset should be downloaded manually, and is available in [http://help.sentiment140.com](https://help.sentiment140.com). Once the data is downloaded, it should be placed in a Google Drive folder, and the drive should be mounted to the project. I have placed the dataset under *Colab Notebooks/Sentiment140* directories in my Google Drive storage."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DvrO_qZo3dD4"
      },
      "source": [
        "def get_data():\n",
        "    # This is a very large file, so we will not load it into RAM\n",
        "    return csv.reader(open(\"drive/MyDrive/Colab Notebooks/Sentiment140/training.1600000.processed.noemoticon.csv\", \"rt\", encoding=\"latin-1\"))"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M6nQDSm7I-2o"
      },
      "source": [
        "Let's see what this data looks like, printing the first 10 entries."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sVkJ19B23eAA",
        "outputId": "d4f7e97e-437a-41e7-a63e-ca7e4dd63b2e"
      },
      "source": [
        "for i, line in enumerate(get_data()):\n",
        "    if i > 10:\n",
        "        break\n",
        "    print(line)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['0', '1467810369', 'Mon Apr 06 22:19:45 PDT 2009', 'NO_QUERY', '_TheSpecialOne_', \"@switchfoot http://twitpic.com/2y1zl - Awww, that's a bummer.  You shoulda got David Carr of Third Day to do it. ;D\"]\n",
            "['0', '1467810672', 'Mon Apr 06 22:19:49 PDT 2009', 'NO_QUERY', 'scotthamilton', \"is upset that he can't update his Facebook by texting it... and might cry as a result  School today also. Blah!\"]\n",
            "['0', '1467810917', 'Mon Apr 06 22:19:53 PDT 2009', 'NO_QUERY', 'mattycus', '@Kenichan I dived many times for the ball. Managed to save 50%  The rest go out of bounds']\n",
            "['0', '1467811184', 'Mon Apr 06 22:19:57 PDT 2009', 'NO_QUERY', 'ElleCTF', 'my whole body feels itchy and like its on fire ']\n",
            "['0', '1467811193', 'Mon Apr 06 22:19:57 PDT 2009', 'NO_QUERY', 'Karoli', \"@nationwideclass no, it's not behaving at all. i'm mad. why am i here? because I can't see you all over there. \"]\n",
            "['0', '1467811372', 'Mon Apr 06 22:20:00 PDT 2009', 'NO_QUERY', 'joy_wolf', '@Kwesidei not the whole crew ']\n",
            "['0', '1467811592', 'Mon Apr 06 22:20:03 PDT 2009', 'NO_QUERY', 'mybirch', 'Need a hug ']\n",
            "['0', '1467811594', 'Mon Apr 06 22:20:03 PDT 2009', 'NO_QUERY', 'coZZ', \"@LOLTrish hey  long time no see! Yes.. Rains a bit ,only a bit  LOL , I'm fine thanks , how's you ?\"]\n",
            "['0', '1467811795', 'Mon Apr 06 22:20:05 PDT 2009', 'NO_QUERY', '2Hood4Hollywood', \"@Tatiana_K nope they didn't have it \"]\n",
            "['0', '1467812025', 'Mon Apr 06 22:20:09 PDT 2009', 'NO_QUERY', 'mimismo', '@twittera que me muera ? ']\n",
            "['0', '1467812416', 'Mon Apr 06 22:20:16 PDT 2009', 'NO_QUERY', 'erinx3leannexo', \"spring break in plain city... it's snowing \"]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d2WWkznT4NWh"
      },
      "source": [
        "The columns we care about is the first one and the last one. The first column is the label (the label 0 means \"sad\" tweet, 4 means \"happy\" tweet), and the last column contains the tweet. Our task is to predict the sentiment of the tweet given the text.\n",
        "\n",
        "We will need to split the text into words. We will do so by splitting at all whitespace characters. There are better ways to perform the split, but let's keep our dependencies light."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jxK6_gno6Cqt"
      },
      "source": [
        "def split_tweet(tweet):\n",
        "    # separate punctuations\n",
        "    tweet = tweet.replace(\".\", \" . \") \\\n",
        "                 .replace(\",\", \" , \") \\\n",
        "                 .replace(\";\", \" ; \") \\\n",
        "                 .replace(\"?\", \" ? \")\n",
        "    return tweet.lower().split()"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ennt_YnM6L93"
      },
      "source": [
        "**Word Embedding:** To apply neural networks to text, we need to convert words into vectors. There are obviously many ways to do that, and the family of methods which convert words into such feature representations are reffered to as *word embedding*. There are many techniques in natural language processing for word embedding, where the main rationale is to have words of similar meanings being mapped into vectors of small distances. \n",
        "\n",
        "While word embedding is not the focus of our course, it is neeeded for the current example. Therefore, in our demnostration we will use the GloVe word embedding, which also has several variations which differ in the data used to obtain the mapping and the dimensionality of the vectors. In particular, we will use the mapping trained on Wikipedia 2014 corpus (\"6B\") with embedding size 50. More information about GloVe is available here [https://nlp.stanford.edu/projects/glove/](https://nlp.stanford.edu/projects/glove).\n",
        "\n",
        "Fortunately, GloVe is available by PyTorch's *torchtext* package (requiring the downloading of a $~823$MB file).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "90FsZdk_6MWP",
        "outputId": "4c50de4c-31cb-49e3-b1ce-d1e647cb0e11"
      },
      "source": [
        "glove = torchtext.vocab.GloVe(name=\"6B\", dim=50)\n"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            ".vector_cache/glove.6B.zip: 862MB [02:41, 5.33MB/s]                          \n",
            "100%|█████████▉| 398011/400000 [00:11<00:00, 35164.89it/s]"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WNuz5bm1KKjy"
      },
      "source": [
        "Since tweets often have mispellings, we'll need to ignore words that do not appear in the Glove embeddings. Let's sanity check that there are enough words for us to work with."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UEDHp8t-KFEH",
        "outputId": "dc480244-4c2e-4918-a12f-da6efc94b8d8"
      },
      "source": [
        "for i, line in enumerate(get_data()):\n",
        "    if i > 30:\n",
        "        break\n",
        "    print(sum(int(w in glove.stoi) for w in split_tweet(line[-1])))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "21\n",
            "23\n",
            "17\n",
            "10\n",
            "22\n",
            "4\n",
            "3\n",
            "21\n",
            "4\n",
            "3\n",
            "9\n",
            "4\n",
            "19\n",
            "15\n",
            "19\n",
            "18\n",
            "18\n",
            "4\n",
            "9\n",
            "13\n",
            "11\n",
            "23\n",
            "8\n",
            "9\n",
            "4\n",
            "11\n",
            "13\n",
            "6\n",
            "23\n",
            "20\n",
            "13\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z1SS4xSf6PyS"
      },
      "source": [
        "# Training Data\n",
        "We will only use  $1/29$  of the data in the file, so that this demo runs relatively quickly. So we only look at one in every $29$ tweets. For these tweets, we use $60\\%$ as training, $20\\%$ as validation, and $20\\%$ as test.\n",
        "\n",
        "Since we are going to store the individual words in a tweet, we will defer looking up the word embeddings. Instead, we will store the index of each word in a PyTorch tensor. Our choice is the most memory-efficient, since it takes fewer bits to store an integer index than a 50-dimensional vector or a word."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8zSZJXY06VpI"
      },
      "source": [
        "def get_tweet_words(glove_vector):\n",
        "    train_data, valid_data, test_data = [], [], []\n",
        "    for i, line in enumerate(get_data()):\n",
        "        if i % 29 == 0:\n",
        "            tweet = line[-1]\n",
        "            idxs = [glove_vector.stoi[w]        # lookup the index of word\n",
        "                    for w in split_tweet(tweet)\n",
        "                    if w in glove_vector.stoi] # keep words that has an embedding\n",
        "            if not idxs: # ignore tweets without any word with an embedding\n",
        "                continue\n",
        "            idxs = torch.tensor(idxs) # convert list to pytorch tensor\n",
        "            label = torch.tensor(int(line[0] == \"4\")).long()\n",
        "            if i % 5 < 3:\n",
        "                train_data.append((idxs, label))\n",
        "            elif i % 5 == 4:\n",
        "                valid_data.append((idxs, label))\n",
        "            else:\n",
        "                test_data.append((idxs, label))\n",
        "    return train_data, valid_data, test_data\n",
        "\n",
        "train_data, valid_data, test_data = get_tweet_words(glove)"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g10dgnTON22l"
      },
      "source": [
        "Here's what an element of the training set looks like:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qVOnnDJ46jVE",
        "outputId": "ed13feaa-a67d-4857-d121-785a14cd9e8f"
      },
      "source": [
        "tweet, label = train_data[0]\n",
        "print(tweet)\n",
        "print(label)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([     2,     11, 190100,      1,      7,  70483,      2,     81, 107356,\n",
            "           405,    684,   9912,      3,    245,    122,      4,     88,     20,\n",
            "             2,     89,   1968])\n",
            "tensor(0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PQnuJOg2qdYH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bec13d68-a407-495f-dd15-90b32a6dfd65"
      },
      "source": [
        "print(len(train_data))\n",
        "print(len(valid_data))\n",
        "print(len(test_data))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "32873\n",
            "10966\n",
            "10972\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pXK0jc6pByWV"
      },
      "source": [
        "# Train and Validation \n",
        "Next, we formulate the training fucntion, which also implements validation. \n",
        "We begin with defining an intermediate function for computing the accuracy of a given model over a dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Z48ie49B6hW"
      },
      "source": [
        "def get_accuracy(model, data_loader):\n",
        "    correct, total = 0, 0\n",
        "    for tweets, labels in data_loader:\n",
        "        output = model(tweets)\n",
        "        pred = output.max(1, keepdim=True)[1]\n",
        "        correct += pred.eq(labels.view_as(pred)).sum().item()\n",
        "        total += labels.shape[0]\n",
        "    return correct / total"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IJ2ox5piOioq"
      },
      "source": [
        "The trainin function uses the Adam optimizer with the cross entropy loss (since we are dealing with a classification problem). \n",
        "The training function also computes the validation error on each epoch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e1JdXFrXB7T4"
      },
      "source": [
        "def train(model, train_loader, valid_loader, num_epochs=5, learning_rate=1e-5):\n",
        "    criterion = nn.CrossEntropyLoss()\n",
        "    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
        "    train_acc, valid_acc = [], []\n",
        "    epochs = []\n",
        "    # Loop over epochs\n",
        "    for epoch in range(num_epochs):\n",
        "        # Loop over batches\n",
        "        for tweets, labels in train_loader:\n",
        "            optimizer.zero_grad()\n",
        "            pred = model(tweets)\n",
        "            loss = criterion(pred, labels)\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "        # Save error on each epoch\n",
        "        epochs.append(epoch)\n",
        "        train_acc.append(get_accuracy(model, train_loader))\n",
        "        valid_acc.append(get_accuracy(model, valid_loader))\n",
        "        print(\"Epoch %d; Loss %f; Train Acc %f; Val Acc %f\" % (\n",
        "              epoch+1, loss, train_acc[-1], valid_acc[-1]))\n",
        "    # plotting\n",
        "    plt.title(\"Training Curve\")\n",
        "    plt.plot(epochs, train_acc, label=\"Train\")\n",
        "    plt.plot(epochs, valid_acc, label=\"Validation\")\n",
        "    plt.xlabel(\"Epoch\")\n",
        "    plt.ylabel(\"Accuracy\")\n",
        "    plt.legend(loc='best')\n",
        "    plt.show()"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9RGMZtcICwp7"
      },
      "source": [
        "# Batching\n",
        "Unfortunately, we will not be able to use DataLoader with a batch_size of greater than one. This is because each tweet has a different shaped tensor. PyTorch implementation of DataLoader class expects all data samples to have the same shape. So, if we just create a DataLoader for batching, it will throw an error when we try to iterate over its elements.\n",
        "\n",
        "How to deal with this?\n",
        "\n",
        "\n",
        "1.   Zero pad - we can pad shorter sequences with zero inputs. Pytorch provides several utilizies to that aim, including \n",
        "`torch.nn.utils.rnn.pad_sequence`.\n",
        "\n",
        "2.   Batch together sequences of the same length. \n",
        "\n",
        "Here, we will adopt the second approach. The reason to do that is that our network will end up looking only at the hidden state of the RNN after the entire sequence has been processed, so we should use the output corresponding to the final word and not the zero padding in the end.\n",
        "To that aim, we will define a dedicate function which essentially first stores the tweets in bins by lengths, and then converts each bin into batches (where the final tweets which do not fit into the batches are dropped).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RNSyq7NjCzPe"
      },
      "source": [
        "class TweetBatcher:\n",
        "    def __init__(self, tweets, batch_size=64, drop_last=False):\n",
        "        # store tweets by length\n",
        "        self.tweets_by_length = {}\n",
        "        for words, label in tweets:\n",
        "            # compute the length of the tweet\n",
        "            wlen = words.shape[0]\n",
        "            # put the tweet in the correct key inside self.tweet_by_length\n",
        "            if wlen not in self.tweets_by_length:\n",
        "                self.tweets_by_length[wlen] = []\n",
        "            self.tweets_by_length[wlen].append((words, label),)\n",
        "         \n",
        "        #  create a DataLoader for each set of tweets of the same length\n",
        "        self.loaders = {wlen : torch.utils.data.DataLoader(\n",
        "                                    tweets,\n",
        "                                    batch_size=batch_size,\n",
        "                                    shuffle=True,\n",
        "                                    drop_last=drop_last) # omit last batch if smaller than batch_size\n",
        "            for wlen, tweets in self.tweets_by_length.items()}\n",
        "        \n",
        "    def __iter__(self): # called by Python to create an iterator\n",
        "        # make an iterator for every tweet length\n",
        "        iters = [iter(loader) for loader in self.loaders.values()]\n",
        "        while iters:\n",
        "            # pick an iterator (a length)\n",
        "            im = random.choice(iters)\n",
        "            try:\n",
        "                yield next(im)\n",
        "            except StopIteration:\n",
        "                # no more elements in the iterator, remove it\n",
        "                iters.remove(im)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z3ZyWPy5SnQ0"
      },
      "source": [
        "Let's see how this shapes our data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r4BuIwUKSqaI",
        "outputId": "d26267a5-5431-477f-c906-c5459c603000"
      },
      "source": [
        "for i, (tweets, labels) in enumerate(TweetBatcher(train_data, drop_last=True)):\n",
        "    if i > 5: break\n",
        "    print(tweets.shape, labels.shape)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([64, 9]) torch.Size([64])\n",
            "torch.Size([64, 33]) torch.Size([64])\n",
            "torch.Size([64, 2]) torch.Size([64])\n",
            "torch.Size([64, 24]) torch.Size([64])\n",
            "torch.Size([64, 27]) torch.Size([64])\n",
            "torch.Size([64, 27]) torch.Size([64])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K_rCuVzwSs5R"
      },
      "source": [
        "Finally, we can load the data in mini-batches of idential sizes to be processed during training. In particular, we use $B=64$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2miP3i_PD_0y"
      },
      "source": [
        "# Divide data into batches\n",
        "train_loader = TweetBatcher(train_data, batch_size=64, drop_last=True)\n",
        "valid_loader = TweetBatcher(valid_data, batch_size=64, drop_last=False)\n",
        "test_loader = TweetBatcher(test_data, batch_size=64, drop_last=False)"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TPz-RilH7YMu"
      },
      "source": [
        "# Recurrent Neural Networks\n",
        "\n",
        "Before defining our models, we will define a word embedding layer, using PyTorch *nn.Embedding* layer. Our motivation of using an embedding layer which utilizes GloVe is that it lets us look up the embeddings of multiple words simultaiously, so that our network can train and infer faster. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7JjVEE7g7Xke",
        "outputId": "3996e5c5-f565-46e3-df04-eede754f8dc2"
      },
      "source": [
        "# Create an `nn.Embedding` layer and load data from pretrained `glove.vectors`\n",
        "glove_emb = nn.Embedding.from_pretrained(glove.vectors)\n",
        "\n",
        "# Example: we use the forward function of glove_emb to lookup the\n",
        "# embedding of each word in the first tweet in the trainin set\n",
        "tweet, label = train_data[0]\n",
        "tweet_emb = glove_emb(tweet)\n",
        "tweet_emb.shape"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([21, 50])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rzDn8YqYDt1L"
      },
      "source": [
        "**Model 1: Vanilla RNN:** Our first model is a vanilla RNN with a fully-connected layer applied to the last hidden state, mapping its value into a $2\\times 1$ vector (which is converted to a distribution in the computation of the cross entropy loss)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z-g6lzqWDQU7"
      },
      "source": [
        "class TweetRNN(nn.Module):\n",
        "    def __init__(self, input_size, hidden_size, num_classes):\n",
        "        super(TweetRNN, self).__init__()\n",
        "        self.emb = nn.Embedding.from_pretrained(glove.vectors)\n",
        "        self.hidden_size = hidden_size\n",
        "        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)\n",
        "        self.fc = nn.Linear(hidden_size, num_classes)\n",
        "    \n",
        "    def forward(self, x):\n",
        "        # Look up the embedding\n",
        "        x = self.emb(x)\n",
        "        # Forward propagate the RNN\n",
        "        out, _ = self.rnn(x)\n",
        "        # Pass the output of the last time step to the classifier\n",
        "        out = self.fc(out[:, -1, :])\n",
        "        return out"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FyfIfMuZVF8e"
      },
      "source": [
        "Applying an RNN with hidden state size $=50$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 646
        },
        "id": "EwMEW0OYDoXv",
        "outputId": "cfa6fa95-e7b1-45ad-9209-0a6a1b654f01"
      },
      "source": [
        "model = TweetRNN(50, 50, 2)\n",
        "# Train and produce training curve and validation curve\n",
        "train(model, train_loader, valid_loader, num_epochs=20, learning_rate=2e-4)\n",
        "# Compute test accuracy\n",
        "get_accuracy(model, test_loader)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1; Loss 0.684299; Train Acc 0.665827; Val Acc 0.668977\n",
            "Epoch 2; Loss 0.502785; Train Acc 0.673797; Val Acc 0.675816\n",
            "Epoch 3; Loss 0.715875; Train Acc 0.669670; Val Acc 0.667062\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r100%|█████████▉| 398011/400000 [00:29<00:00, 35164.89it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 4; Loss 0.495388; Train Acc 0.681956; Val Acc 0.681105\n",
            "Epoch 5; Loss 0.609030; Train Acc 0.681042; Val Acc 0.678096\n",
            "Epoch 6; Loss 0.668073; Train Acc 0.685358; Val Acc 0.682838\n",
            "Epoch 7; Loss 0.504567; Train Acc 0.686145; Val Acc 0.682747\n",
            "Epoch 8; Loss 0.572168; Train Acc 0.681641; Val Acc 0.682291\n",
            "Epoch 9; Loss 0.515267; Train Acc 0.676884; Val Acc 0.678643\n",
            "Epoch 10; Loss 0.574880; Train Acc 0.691469; Val Acc 0.691045\n",
            "Epoch 11; Loss 0.508762; Train Acc 0.691658; Val Acc 0.693234\n",
            "Epoch 12; Loss 0.555889; Train Acc 0.693989; Val Acc 0.692413\n",
            "Epoch 13; Loss 0.673776; Train Acc 0.688666; Val Acc 0.686941\n",
            "Epoch 14; Loss 0.678226; Train Acc 0.687973; Val Acc 0.681196\n",
            "Epoch 15; Loss 0.562337; Train Acc 0.694556; Val Acc 0.692960\n",
            "Epoch 16; Loss 0.544935; Train Acc 0.698431; Val Acc 0.697793\n",
            "Epoch 17; Loss 0.583147; Train Acc 0.695029; Val Acc 0.696061\n",
            "Epoch 18; Loss 0.555708; Train Acc 0.700762; Val Acc 0.695513\n",
            "Epoch 19; Loss 0.632018; Train Acc 0.703062; Val Acc 0.699708\n",
            "Epoch 20; Loss 0.583777; Train Acc 0.706212; Val Acc 0.702900\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd1yW5f7A8c8XkOFEFByAYgqZG8RtjtTSSk0zzWzYsKzTsl2/SrN16nROUzsNy/KYIy3TcpTm3qg4cCKi4mQKiuzv74/70QgBWQ/zer9ezwue+7nv675uLL5c63uJqmIYhmEYBeVQ1hUwDMMwKhYTOAzDMIxCMYHDMAzDKBQTOAzDMIxCMYHDMAzDKBQTOAzDMIxCMYHDMPIhIktE5L6SPtcwKjIx6ziMykZEzmd7Wx1IBTJt7x9R1ZmlX6viEZHawGRgOOABnAEWAW+pakxZ1s2oekyLw6h0VLXmpRdwDBic7djloCEiTmVXy4ITEWdgBdAaGAjUBroBsUDnIpRXIZ7bKL9M4DCqDBHpIyJRIvKiiJwGvhWRuiLyq4hEi0i87XufbNesEpGHbN+PFZF1IvKB7dwjIjKoiOc2E5E1IpIkIstFZIqI/C+Pqt8LNAGGqepeVc1S1bOq+qaqLraVpyLSIlv500XkrXyee5+I3JrtfCfbzyDI9r6riGwQkQQR2SkifYr78zcqDxM4jKqmIVZXT1PgYaz/B761vW8CXAQ+y+f6LsABoD7wPjBNRKQI5/4AbAHqAZOAe/K5Z39gqaqez+ecq8n53LOA0dk+vwmIUdXtIuIN/Aa8ZbvmOWC+iHgW4/5GJWICh1HVZAETVTVVVS+qaqyqzlfVZFVNAt4Geudz/VFV/UpVM4HvgEZAg8KcKyJNgE7A66qapqrrgIX53LMecKpwj3mFvz03VuAaIiLVbZ/fhRVMAO4GFqvqYlvr5g8gBLi5mHUwKgkTOIyqJlpVUy69EZHqIvKFiBwVkURgDeAuIo55XH/60jeqmmz7tmYhz20MxGU7BnA8nzrHYgWd4vjbc6tqOLAPGGwLHkOwgglYrZI7bN1UCSKSAPQsgToYlYQZJDOqmpzTCJ8FrgW6qOppEekA7ADy6n4qCacADxGpni14+OZz/nLgLRGpoaoX8jgnGWsG2SUNgahs73ObPnmpu8oB2GsLJmAFsRmqOu4qz2FUUabFYVR1tbDGNRJExAOYaO8bqupRrK6fSSLiLCLdgMH5XDID65f5fBFpKSIOIlJPRF4RkUvdR6HAXSLiKCIDyb+77ZLZwI3Ao/zV2gD4H1ZL5CZbea62AXafXEsxqhwTOIyq7iPADYgBNgFLS+m+Y/hrSu1bwBys9SZXUNVUrAHy/cAfQCLWwHp9YLPttKewgk+CrewFV6uAqp4CNgLdbfe/dPw4MBR4BYjGClrPY35fGDZmAaBhlAMiMgfYr6p2b/EYRnGZvyAMowyISCcRaW7rdhqI9Rf+VVsJhlEemMFxwygbDYGfsKbaRgGPquqOsq2SYRSM6aoyDMMwCsV0VRmGYRiFUiW6qurXr69+fn5lXQ3DMIwKZdu2bTGqekWqmSoROPz8/AgJCSnrahiGYVQoInI0t+Omq8owDMMoFBM4DMMwjEIxgcMwDMMolCoxxpGb9PR0oqKiSElJufrJxlW5urri4+NDtWrVyroqhmHYWZUNHFFRUdSqVQs/Pz/y3ofHKAhVJTY2lqioKJo1a1bW1TEMw86qbFdVSkoK9erVM0GjBIgI9erVM603w6giqmzgAEzQKEHmZ2kYVUeVDhyGYRiVVXRSKm8sCiM1I7PEyzaBo4zExsbSoUMHOnToQMOGDfH29r78Pi0tLd9rQ0JCePLJJ0uppoZhVDSnz6Uw6suNzNpyjIOnz5d4+VV2cLys1atXj9DQUAAmTZpEzZo1ee655y5/npGRgZNT7v88wcHBBAcHl0o9DcOoWKLik7nrq83Enk/l+we60NanTonfw7Q4ypGxY8cyfvx4unTpwgsvvMCWLVvo1q0bgYGBdO/enQMHDgCwatUqbr31VsAKOg888AB9+vThmmuu4ZNPPinLRzAMowxFxlxg5H83kpCcxv8e6kLnZh52uY9dWxy2DWo+BhyBr1X1nzk+/xDoa3tbHfBSVXfbZ/cBr9o+e0tVv7MdXwU0wtonGuBGVT1bnHq+sSiMvScTi1PEFVo1rs3Ewa0LfV1UVBQbNmzA0dGRxMRE1q5di5OTE8uXL+eVV15h/vz5V1yzf/9+Vq5cSVJSEtdeey2PPvqoWU9hGFXMoTNJjPl6MxlZyqyHu9K6ccm3NC6xW+AQEUdgCjAAa6OarSKyUFX3XjpHVSdkO/8JIND2vQcwEQgGFNhmuzbedvoYVa2UWQvvuOMOHB0dATh37hz33Xcfhw4dQkRIT0/P9ZpbbrkFFxcXXFxc8PLy4syZM/j4+JRmtQ3DKENhJ89xz7QtODoIsx/uSkCDWna9nz1bHJ2BcFWNABCR2VjbY+7N4/zRWMEC4CbgD1WNs137BzAQmGWPihalZWAvNWrUuPz9a6+9Rt++ffn555+JjIykT58+uV7j4uJy+XtHR0cyMjLsXU3DMMqJ0OMJ3DttMzVdnJg5rivN6te4+kXFZM8xDm/geLb3UbZjVxCRpkAz4M8CXvutiISKyGuSxwICEXlYREJEJCQ6Orqoz1Cmzp07h7e39djTp08v28oYhlHubI2M4+6vN1OnejXmPNKtVIIGlJ/B8TuBeapakAnHY1S1LXC97XVPbiep6peqGqyqwZ6eV+xDUiG88MILvPzyywQGBppWhGEYf7M+PIZ7p23Bq7YLPz7SHV+P6qV2b7vtOS4i3YBJqnqT7f3LAKr6bi7n7gD+oaobbO9HA31U9RHb+y+AVao6K8d1Y4FgVX08v7oEBwdrzo2c9u3bx3XXXVfEpzNyY36mhlE6Vu4/yyP/28Y19Wsw48EueNZyufpFRSAi21T1irn/9mxxbAX8RaSZiDhjtSoW5lKxlkBdYGO2w8uAG0WkrojUBW4ElomIk4jUt11XDbgV2GPHZzAMwyhXlu45xcMzQghoUJNZ47raLWjkx26D46qaISKPYwUBR+AbVQ0TkclAiKpeCiJ3ArM1W9NHVeNE5E2s4AMw2XasBlYAqWYrcznwlb2ewTAMozz5JfQEz8zdSXufOnx7f2fquJXNtHu7ruNQ1cXA4hzHXs/xflIe134DfJPj2AWgY8nW0jAMo/ybu/U4L/60iy7NPJh2XydquJRd4g+TcsQwDKOc+35jJK//EkavAE++uLsjbs6OZVofEzgMwzDKsS/XHOadxfsZ0KoBn90ViItT2QYNMIHDMAyjXFJVPlkRzofLD3JLu0Z8NKoD1RzLxwqK8lGLKqhv374sW7bsb8c++ugjHn300VzP79OnD5emFN98880kJCRccc6kSZP44IMP8r3vggUL2Lv3r8X7r7/+OsuXLy9s9Q3DsKPElHSenbuTD5cf5PYgHz65M7DwQSMzHXb9CHZYcmECRxkZPXo0s2fP/tux2bNnM3r06Kteu3jxYtzd3Yt035yBY/LkyfTv379IZRmGUfK2HIlj0Edr+WXnSZ7q58+/RrTD0aGQO2xmpMLce+GnhyBq69XPLyQTOMrIiBEj+O233y5v2hQZGcnJkyeZNWsWwcHBtG7dmokTJ+Z6rZ+fHzExMQC8/fbbBAQE0LNnz8tp1wG++uorOnXqRPv27bn99ttJTk5mw4YNLFy4kOeff54OHTpw+PBhxo4dy7x58wBYsWIFgYGBtG3blgceeIDU1NTL95s4cSJBQUG0bduW/fv32/NHYxhVUlpGFu8t3c+oLzfi5CjMfaQbEwYE4FDYoJGeArPHwIHFcPMH4Nu5xOtqxjgAlrwEp3eXbJkN28Kgf+b5sYeHB507d2bJkiUMHTqU2bNnM3LkSF555RU8PDzIzMykX79+7Nq1i3bt2uVaxrZt25g9ezahoaFkZGQQFBREx47WbOXhw4czbtw4AF599VWmTZvGE088wZAhQ7j11lsZMWLE38pKSUlh7NixrFixgoCAAO69914+//xznn76aQDq16/P9u3bmTp1Kh988AFff/11SfyUDMMAws8m8dTsUMJOJjIq2JfXBreiZlGm26Ylw+zRELEaBn8MHceWeF3BtDjKVPbuqkvdVHPnziUoKIjAwEDCwsL+1q2U09q1axk2bBjVq1endu3aDBky5PJne/bs4frrr6dt27bMnDmTsLCwfOty4MABmjVrRkBAAAD33Xcfa9asufz58OHDAejYsSORkZFFfWTDMLJRVb7fGMktn6zjZMJFvrinI++NaFe0oJF6Hn4YCUfWwG1T7RY0wLQ4LPm0DOxp6NChTJgwge3bt5OcnIyHhwcffPABW7dupW7duowdO5aUlJQilT127FgWLFhA+/btmT59OqtWrSpWXS+lbjdp2w2jZJxNSuGFebtYdSCa3gGe/GtEO7xquxatsJREmHmHNZ4x7Etod0fJVjYH0+IoQzVr1qRv37488MADjB49msTERGrUqEGdOnU4c+YMS5Ysyff6Xr16sWDBAi5evEhSUhKLFi26/FlSUhKNGjUiPT2dmTNnXj5eq1YtkpKSrijr2muvJTIykvDwcABmzJhB7969S+hJDcPIblnYaQZ+tJaNh2OZPLQ10+/vVPSgcTEBZgyDEyEw4hu7Bw0wLY4yN3r0aIYNG8bs2bNp2bIlgYGBtGzZEl9fX3r06JHvtUFBQYwaNYr27dvj5eVFp06dLn/25ptv0qVLFzw9PenSpcvlYHHnnXcybtw4Pvnkk8uD4gCurq58++233HHHHWRkZNCpUyfGjx9vn4c2jCrqQmoGb/66l9lbj9O6cW0+vrMDLbyKsVtfcpwVNM6EwcjvoeUtJVfZfNgtrXp5YtKqlw7zMzWMvG0/Fs+EOaEci0tmfO/mTOgfgLNTMTp9LsTA97dBzEEY9T8IuLHkKmuTV1p10+IwDMOwo4zMLD79M5zPVobTsLYrs8d1pcs19YpXaNIZ+H4oxB+Bu2ZD8xtKprIFZAKHYRiGnUTGXODpOaGEHk9geKA3k4a2prZrMVOhJ56C7wZD4gkY8yM061UylS2EKh04VJU8tiw3CqkqdHkaRmGERMZx7zdbcHIQPh0dyOD2jYtf6LkoK2icPwt3/wRNuxW/zCKw66wqERkoIgdEJFxEXsrl8w9FJNT2OigiCdk+u09EDtle92U73lFEdtvK/ESK+Jvf1dWV2NhY8wuvBKgqsbGxuLoWcVaIYVQymVnKqwv2ULe6M8sm9CqZoBF/FL4dBBdi4Z4FZRY0wI4tDhFxBKYAA4AoYKuILFTVyyvaVHVCtvOfAAJt33sAE4FgQIFttmvjgc+BccBmrE2iBgL5z1vNhY+PD1FRUURHRxfxCY3sXF1d8fHxKetqGEa5MGfrcfafTmLqmCAa1XErfoGxh+G7IZB2Hu5dAN5BxS+zGOzZVdUZCFfVCAARmQ0MBfJaCj0aK1gA3AT8oapxtmv/AAaKyCqgtqpush3/HriNIgSOatWq0axZs8JeZhiGka/ElHT+/fsBOvt5MKhNw+IXGHPI6p7KTIP7FkGj3FMQlSZ7Bg5v4Hi291FAl9xOFJGmQDPgz3yu9ba9onI5bhiGUS5M+TOcuOQ0pt/aqvhjqGf3WS0NFO77FRq0KpE6Fld5WTl+JzBPVTNLqkAReVhEQkQkxHRHGYZRGiJjLvDN+iOMCPKhrU+d4hUWfQCm3wLiAGMXl5ugAfYNHCcA32zvfWzHcnMnMKsA156wfX/VMlX1S1UNVtVgT0/PQlbdMAyj8N5ZvA9nRweev+na4hWUkmilRhcHuH8xeAaUTAVLiD0Dx1bAX0SaiYgzVnBYmPMkEWkJ1AU2Zju8DLhRROqKSF3gRmCZqp4CEkWkq2021b3AL3Z8BsMwjALZEB7D73vP8FjfFkXPOwXWjn2/PAZxEXDHdKjXvMTqWFLsNsahqhki8jhWEHAEvlHVMBGZDISo6qUgcicwW7PNi1XVOBF5Eyv4AEy+NFAOPAZMB9ywBsULPTBuGIZRkjKzlMm/7sXb3Y0HexZz0s2GT2HfIrjxLfDrWTIVLGF2XQCoqouxpsxmP/Z6jveT8rj2G+CbXI6HAG1KrpaGYRjFMzfEmn475a4gXKs5Fr2gI2tg+URoNRS6PV5yFSxh5WVw3DAMo0JKTEnng2UH6ORXl5vbFmP67bkT8OP9UK8FDJ0C5TirhQkchmFUSueS01mx74zds0NMWWlNv3391tZFn36bkQY/3gcZKVamW5dipFovBSZwGIZR6aSkZ3L/9C08+F0I/1yy327B42jsBb5dF8ntxZ1+u+wVa/e+oZ+BZzFnZJUCEzgMw6hUVJUX5+9i+7EEerSoxxdrIvh4xSG73OudxftwchReKM70251zYOtX1phG62ElVzk7MoHDMIxK5dM/w/kl9CQvDriGGTdVY0SQNx8tP8R/Vx8u0ftsOBzDsrAz/KM4029P74FFT0HTntD/jRKtnz1V6bTqhmFULot2nuQ/fxxkeKA3489PQabN4F/enfD1v4t/LgFXJwfG9ih+jrrMLOXNX/cVb/rtxQSYcze41rH2CnesOL+OTYvDMIxKYcexeJ77cSfBTevyXvvTyI4ZEDAISTzBU8cn8GvdD/nx19+YveVYse/1Y8hx9p1K5OWbWxZt+m1WFvw8Hs4dh5HfQa0Gxa5TaTKBwzCMCi8qPplx32+jQW1XvhxxDdV+fQq8Wlm/lJ/cDje+RWvC+c3l/6i16CGWr1lX5HslpaTzwe8HCG5al1vaNipaIev+AweXwE3vQJOuRa5LWTGBwzCMCu18agYPfRdCakYm34wNxmPNa5AcA8P+C04uUM0Nuj+BPLWTjJ7P0c9pJ31WDOb49IestROFNGXlYWLOp/H64CJmvw1fAX++BW3vgM4PF/76csAEDsMwKqzMLOXJWTs4dPY8U+4KokXMStg9F3q9AI3a//1k1zo49X+NzCdCWVZ9MA2O/ETmxx1g2f9BclzuN8jhaOwFvll3hNuDfGjn4174Ciccg/kPgdd1MPjjcr3ILz8mcBiGUWG9s3gff+4/y6QhrenlLfDrBGjUAa5/Js9rang04vqnpvFo3a/4JaMrumkqfNweVr8Pqefzvd+7i/db028HFmH6bXoKzL0XsjJg5AxwrlH4MsoJEzgMw6iQZm4+yrR1Rxjb3Y97ujSBX5+G1ESri8qxWr7X1natxgfjBvOlx/MMyXifOK+usPJt+KQDbPovZKRecc3Gw7EsDTvNY32a06Ao02+Xvggnd1j1q9+i8NeXIyZwGIZR4aw7FMPrv4TR91pPXru1FeyeZ2WU7ft/VjdQAdSt4cyMB7uQ7O5Pr2MPcXDwz+DZ0voF/2kwhM6CLGtvOWv6rZX99qHrryl8hbfPgG3Toecz0PKWwl9fzpjAYRhGhRJ+9jyPztxGC8+afDI6EMfzp2Hxc+DTGbo/UaiyPGu5MPOhrnjUcGbEonTCBvwP7vkZqnvAgvHwaUeYP46wORNpfGYlk693w7Wws29PhsJvz0Kz3nDDq4W8uHwSeycAKw+Cg4M1JCSkrKthGEYxxV1IY9jU9VxIzeDnx3rgW9cNfhhlpSMfv67IXUDH45IZ9cVGUjKymPNwV/y9asLeX2DHDLLO7sMhMdvsKyc3qO9vtWw8W9q+XgvufuCQ42/x5Dj4srfVcnlkDdSoX/SHLwMisk1Vg684bgKHYRgVQWpGJvd8vYXQqARmjetKx6Z1rS6ghY/DwPeg6/hilX8k5gKjvtiIAnMf6Uaz+tbg9XtL9zNj1W5+vqMe/hIFZ/dD9D5rT/CcAcUzADyvA6+W1tetX1lB7f6l4NOxWPUrC3kFDruucReRgcDHWDsAfq2q/8zlnJHAJECBnap6l+34e8ClzsA3VXWO7fh0oDdwzvbZWFUNteNjGIZRxlSVV37aw5bIOD4ZHWgFjYTjsPRl8Lu+RNZDNKtfg5kPdWHUl5sY89Um5jzSDVWYtvYIg4MC8O/Y/sqLUs5ZAeTsPojeb309sgZ2zf7rnFv+UyGDRn7sFjhExBGYAgwAooCtIrJQVfdmO8cfeBnooarxIuJlO34LEAR0AFyAVSKyRFUTbZc+r6rz7FV3wzDKl89XH2b+9iie7u/PkPaNrZQdCx8H1EpFnrOLqIj8G9RixoOdGf3lJsZ8vZmm9arj6JDP9FvXOuDb2XpldzHBCiiaVSFXhl+NPQfHOwPhqhqhqmnAbGBojnPGAVNUNR5AVc/ajrcC1qhqhqpeAHYBA+1YV8Mwyqmle07x/tIDDGnfmKf6+VsHQ6ZBxCprX+66fiV6v9aN6/D9g12Iu5DG2kMxRZt+6+YOTbpA024VdpFffuwZOLyB49neR9mOZRcABIjIehHZZOvaAtgJDBSR6iJSH+gL+Ga77m0R2SUiH4qIS243F5GHRSREREKio6NL5okMwyhVu6ISeHpOKIFN3Hl/RDsrxUdcBPzxOjTvBx3H2uW+HXzd+e6BTozp0oRxvYow/baSK+s8vk6AP9AH8AHWiEhbVf1dRDoBG4BoYCOQabvmZeA04Ax8CbwITM5ZsKp+afuc4ODgyj8DwDAqmdPnUnjouxDq1XDhy3uCrSy0WZmw4B/gUA2GfGrXv+Y7NvWgY1MPu5VfkdmzxXGCv7cSfGzHsosCFqpquqoeAQ5iBRJU9W1V7aCqAwCxfYaqnlJLKvAtVpeYYRiVzPvL9pOYks60scF41rJ1LGz6HI5tgEHvQZ2cHRhGabFn4NgK+ItIMxFxBu4EFuY4ZwFWawNbl1QAECEijiJSz3a8HdAO+N32vpHtqwC3AXvs+AyGYZSBIzEXWLDjBPd0bUrLhrWtg9EHYMVkuPZmaH9n2VawirNbV5WqZojI48AyrOm436hqmIhMBkJUdaHtsxtFZC9WV9TzqhorIq7AWlvK4kTgblXNsBU9U0Q8sVohoUDxJm8bRiW1LOw0u6ISeHbAtTg4VKwB2s/+DMfZyYGHezW3DmRmWBsfOdeAWz+qlAPOFYldxzhUdTGwOMex17N9r8Aztlf2c1KwZlblVuYNJV9Tw6g8LqRmMHnRXuaEWHNTGru7MaZL0zKuVcFFxlxgQegJxnb3+6uLav1HcHI7jPi2wu2WVxmZXFWGUYnsikrg1k/XMXfbcR7r05wuzTx4b8l+Ys5fme21vPpsZThODsIjvW2zmU7vgVX/hNbDoc3wsq2cAZjAYRiVQmaWMnVVOMOnbiAlPZNZ47rywsCWvD2sDRfTM3ln8b6yrmKBHI29wM87TnBXlyZ41XKFjDSri8qtLtzy77KunmFT1tNxDcPITXIcLH3JWuTmHQx+Pa1XgzZXrJI+mXCRCXNC2XwkjlvaNuKdYW2pU93aj6KFVy0e7nUNU1Ye5o6OvnRrXq8MHqbgpthaG4/2to1trPkXnNkNd86yMtYa5YIJHIZR3uz71drJ7mIcXDvI6qo58Jv1mav7X0HErye/nfHg5Z/3kJmlfHBHe24P8r5iH+zH+/rzS+hJXvtlD4ufvB5np/LZ0XAsNpmftp/g7q5N8artCqd2wtp/Q/u7oOXNZV09IxsTOAyjvEiOgyUvWntmN2wL9/xkfQU4FwWR6yFyDUSug/2/AtBda/KlS1v8uwyinrcPaOMrZhy5OTvy5tA23D99K1+tjeAffcvn7nNTVobj4CA82sfW2tg+A5xcYOC7ZVsx4womcBhGebB/sbX1aXIs9H4Jrn8WnJz/+ryOD7QfBe1HseNYPO/M+gPfxO084B1Fl/TdyNrXYe3r1lhA0x5Wxli/nuDVChwc6NvSi4GtG/LJikMMad8YX4/qZfesuTgel8z87VGM6dLkr7xQh1dAs15W3iejXDGBwzDK0qWxjF1zrPGLMfOgUbtcT83MUqauDOejFYdoWLs+L4x7gTZ+tn7/hONWSyRyHUSuvdwioX4APLQCXGvz+uBWrDkUzcSFYUy7L/iKLq2yNHVVOA4iPNrH1hqKi7BeXR4t24oZuTKBwzDKyoElsOhpSI6B3i/C9c/9vZWRTVR8MhPmhLI1Mp4h7Rvz5m1tqONW7a8T3H2hw2jrBZBwDA79Dr89Z01lHfgOjd3deGZAAG/9to9lYWcY2KZhKTzk1UXFJ/NjSBR3dWlCwzq21kb4Cutri35lVzEjTyZwGEZpuxhvbUC0cxZ4tYYxc6FRLpsE2fwSeoJXF+xBFT4c1Z5hgT5Xv4d7E+j0EJzeDZv/C4F3Q4NWjO3ux7xtUbyxKIzr/etTw6XsfwVMWXnY1tpo/tfBw3+Ce1PwMJlpy6PyOb3CMCqrg8tgajfYNRd6vQAPr7ocNLKylMSUdE4kXGT/6US2RsYxYU4oT80OJaBBLZY8dX3BgkZ2/SaCa21Y/Dyo4uTowNvD2nDqXAofLT9Y4o9XWCcSLjJv23FGdvKhUR0362BGmrWLXot+JrVIOVX2f24YRiWVkJzGygNnOZecTtqFeLoe/IB2Mb9x0rkZXzT4hD0HriFp50aSUjJISsngfGrGFWU4CDzd35/H+7bAybEIf+dV94B+r1vTe/fMh7Yj6NjUg9GdfflmfSTDg3y4rlHtEnjaopm6MhyAx/pkm+l1fDOknYcW/cuoVsbVmMBhGHby0fJDTN8QSR+HUN6t9jWeJDDN4XbmVBuNa4YbtVwdqF+zBrVcq1HL1cn66uL01/euTjStV52m9WoUryJB98G272DZ/4H/jeBamxcHtmRZ2Bn+7+fdzBvfvUySIJ5MuMjckOOMDPalsbvbXx8cXgEOTtbMMKNcMoHDMOwkLDKKb+pO54aLv5Pl2RKH2+bzoHcQD5Z2RRwcrXQdX/eD1e/BTW/jXt2Zlwe15Pl5u5gbcpw7Ozcp7VoxdZWttZFzXUn4CvDtYnWxGeWSGeMwDDtITsvgpujv6HNxOVz/LA6PrAHvoLKrkE8wBN5jDZSf3Q/AiI4+dG7mwT+X7ie2lJMgnjp3kblboxjR0Rfv7K2N82fh9C5obpJgl2cmcBiGHew8fo5eEkp8wx7WGIOTS1lXCfpPAueasPg5UEVEeOu2NpxPyeDdJftLtSqfrzpMlivTtlYAACAASURBVCr/6Nv87x8c/tP6asY3yjW7Bg4RGSgiB0QkXEReyuOckSKyV0TCROSHbMffE5E9tteobMebichmW5lzbLsLGka5sv/QQQIcTlD92nL0l3ON+tDvNWuBYNhPAAQ0qMW4Xtcwb1sUW47ElUo1Tp9LYfaW49wR7INP3Rwr2MNXQPX60DD3RZBG+WC3wCEijsAUYBDWpkyjRaRVjnP8gZeBHqraGnjadvwWIAjoAHQBnhORSx2e7wEfqmoLIB5Kv8vYMK4mI3wVAG4ty9kCto73W9N/l70KqecBePIGf7zd3Xh1wW7SMrLsXoXPV4WTpfr3mVQAWVlWi6P5DVdkADbKF3v+63QGwlU1QlXTgNnA0BznjAOmqGo8gKqetR1vBaxR1QxVvQDsAgba9hm/AZhnO+87rH3HDaPcUFW8YjZxwbEONGhb1tX5OwdHuPkDSDoJa94HrCSIbwxpzcEz55m27ohdb38mMYVZW49ze5DPlfmyTu+yVtGb1eLl3lUDh4gMFpGiBBhv4Hi291G2Y9kFAAEisl5ENonIQNvxnViBorqI1Af6Ar5APSAh2/7juZVpGGXqSPR5OukuYj27lM+/nH07Q4e7YeMUiD4AQP9WDbixVQM+WXGIqPhku93681WHycrS3DP0HralGTED4+VeQf6rHgUcEpH3RaRlCd/fCfAH+gCjga9ExF1Vf8faq3wDMAvYCGQWpmAReVhEQkQkJDo6umRrbRj5OLhvJ40lDpeAcvwLsP8kcK5xeUU5wMQhrQGYtHCvXW55NjGFWVuOMTzImyb1csnOG77CGtuo6WWX+xsl56qBQ1XvBgKBw8B0Edlo+6Vc6yqXnsBqJVziYzuWXRSwUFXTVfUIcBArkKCqb6tqB1UdAIjts1jAXUSc8inzUr2/VNVgVQ329PS82mMaRolJPWD95ezZ7qYyrkk+anpC31fhyGrYuwAAb3c3nu7vz/J9Z/g97HSJ3/Lz1YfJyFIe7+t/5YcpidaKcdNNVSEUqB2tqolY4wqzgUbAMGC7iDyRz2VbAX/bLChn4E5gYY5zFmC1NrB1SQUAESLiKCL1bMfbAe2A31VVgZXACNv19wG/FOQZDKO01I/eSLRjQxzqNSvrquQv+AFrDGbZ/10eKH+gZzOubVCLNxbtJTntyhQoRXU2MYUfNh9jWGAerY3ItZCVAc1N4KgICjLGMUREfgZWAdWAzqo6CGgPPJvXdbZxiMeBZcA+YK6qhonIZBEZYjttGRArInuxAsLzqhpru89a2/EvgbuzjWu8CDwjIuFYYx7TCvvQhmEvickptEnbyZn6Xcp/gj5HJ7jlA0g8AWs/AKCaLQniiYSLfLziUInd6os1EbbWRh67D4avsNaY+HYpsXsa9lOQlCO3Y01/XZP9oKomi0i+U2FVdTHWWEX2Y69n+16BZ2yv7OekYM2syq3MCKwZW4ZR7hzetZ5ASeZUi75lXZWCadLV2tN7w2fQYQzU9yfYz4ORwT5MW3uE4YE+XNvwar3S+TublMLMzUe5rYM3fvVzybulCuHLrdxUeexHYpQvBemqmgRsufRGRNxExA9AVVfYpVaGUUEl77NWPvsElePxjZwGvAHVqsOSFy4PlL806DpquTrx9JxQ5m49zsEzSWRmaZGK/3J1BGkZWTx+Qx6tjbgISDhqxjcqkIK0OH4Eumd7n2k71skuNTKMCsz99HoiHPy4pl7jsq5KwdX0gr6vwNIXYd9CaDUUjxrOvDOsLS/M38UL83cBUMPZkbY+dejgW5cOvnVo7+tOw9qu+W5BG52Uyv9srY1mubU2wOz2VwEVJHA42RbwAaCqaSbNh2FcKSs1mRape9hafzgVbt+6Tg/Bjhmw9BUrT5RzDQa1bcRNrRsSEXOBnccT2BmVwM7jCUxbF0F6ptX68KrlQgdfd9r7utPB1522PnWo7frXlrZfrb1KawOs9Rt1m5nd/iqQggSOaBEZoqoLAURkKBBj32oZRsVzYs9qfEmHa/qUdVUKz9EJbv4XfDsI1v7bSswIODgILbxq0sKrJrd3tHYfTM3IZO/JRFswOcfO4wn8vvcMYM0HaO5Zk/Y+7rTxrs2MjUcZ2sGbazxr5n7fjFRrt78Od5XKYxoloyCBYzwwU0Q+w1pPcRy41661MowKKCnsD9LVEe/2FbTLpWl3aDcKNnxqDZTXa57raS5OjgQ2qUtgk7qXj51LTmfXiQRCj1ktk9UHzzJ/exSODpJ/a+PYJkhPNtlwK5irBg5VPQx0FZGatvfn7V4rw6iAap5az27xJ7BxBV75PGAy7F9sDZSPmVfgKcV1qlfjen9Prve3FtuqKifPpXAxLYPmebU2wLbbXzWz218FU6AdAG3ZalsDrpcGwlR1sh3rZRgVy8V4fC4eYIf7PQSV9/Ub+anV0BooX/Yy7P8Nrru1SMWIyN83aMpL+J/WlGCXfIKLUe4UZAHgf7HyVT2B1VV1B9DUzvUyjArlwoFVOKBk+vUq66oUX+eHwasVLH0Z0uyX8JCk03Bmt0lqWAEVZB1Hd1W9F4hX1TeAblipQQzDsInf8wfn1ZXGrStBl4ujk5V6/dwxWPcf+93n8m5/FXRMqAorSOBIsX1NFpHGQDpWvirDMGyqn1jHVm1Ju6b1y7oqJcOvB7S9A9Z/DHF22qMjfAXU8Cp/e5YYV1WQwLFIRNyBfwHbgUjgh3yvMIyq5FwUHhePEl4zmOrOBRo2rBj6vwEOTrB8YsmXnZUFESvNbn8VVL7/YrYNnFaoaoKqzsca22iZPd+UYVR1mYdXAZDWpBJ0U2VXxxt6PA17f4HI9SVb9qlQSI413VQVVL6BQ1WzsPYNv/Q+VVXP2b1WRpWnqhyOPs+RmAtlXZWrStq7nGitjc+1Hcu6KiWv+xNQ29uaZZVVgvuRX0ozck0FSQZp/E1B2tUrROR24CdbNlvDKHGZWcq+U4lsjYxjy5E4tkbGEXM+Dbdqjix6oictvMrpdE1VnI+vZU1Wa4Ka1ivr2pQ85+pWl9VPD8HOWRA4pmTKPbwCGrW3NpQyKpyCBI5HsNKeZ4hICtaUXFXV2natmVGppWVksftEApuPxLH1SBwhkfEkpVpbrvjUdaOXvyeBTdz5cPkhnpi1g58f645rNccyrnUuog9QPTWGndXuYHDdAqxbqIjajoAtX8CKN6DV0OKvuUg5B8e3QM+nS6Z+RqkryMrx4iXjNwwgOS2D7UcT2BIZx5Yjsew4lkBqhtX10cKrJoM7NKZLMw86+XnQONvCMe+6bjwwPYT3lu5n4uDWZVX9vEWsAuCiT898s8RWaCJw07swrT+s+xD6vVa88o6sAc00u/1VYFcNHCKS64qmnBs75XHtQOBjwBH4WlX/mcs5I7H2/FBgp6reZTv+PnAL1jjMH8BTqqoisgprOvBFWxE3qurZq9XFKH2RMRf4YcsxthyJY8+Jc2RkKQ4CrRvX4e6uTenk50Env7rUq+mSZxk3tGzA/T38+HZ9JD1b1KffdQ1K8QmuLvXQSk5lNaBp8+vKuir25dvJmp674VPoeB+4Nyl6WeHLwbkW+Jr92CqqgnRVPZ/te1es3fe2Afku9xQRR6yB9QFAFLBVRBaq6t5s5/gDLwM9VDVeRLxsx7sDPbD2GgdYB/TG2r4WYIyqhhSg7kYZSUhOY8zXm4lOSqW9bx0e6X0NnZvVI6iJO7Wypd0uiJcGtWRzRBzPz9vFkqeup0FtVzvVupAyM3A8upb1WV3p2LTu1c+v6PpPgn2/wh8T4Y5vi1aGqpVmpFkvcCzcfwdG+XHVCdSqOjjbawDQBogvQNmdgXBVjbDt5zEbGJrjnHHAFFWNt93rUstBsYKUM+CCtQf5mYI8kFH2srKUZ+fu5GxSCj+O78aP47vz/E0t6R3gWeigAVY21k9GB3IxLZMJc0KLvBNdiTu5HaeMC2yiDW2865R1beyvjg/0eBLCfoJjm4tWRmy4tSLdTMOt0Iqy8iYKKEi73BsrBXv267xznBMABIjIehHZZOvaQlU3AiuBU7bXMlXdl+26b0UkVERekzw6lkXkYREJEZGQ6Ojogj2ZUSK+WhvBiv1nefWWVrT3dS+RMlt41eSNIa3ZcDiWL9YcLpEyiy1iNVkICQ26ls+Be3vo8RTUagRLXyra9Fyz21+lUJAkh5+KyCe212fAWqwV5CXBCfAH+gCjga9ExF1EWmAFJx+sYHODiFxaXTVGVdsC19te9+RWsKp+qarBqhrs6Wmm/JWWLUfieH/ZAW5p24h7u5VsLsw7gn24tV0j/v37QbYfK0ij176yIlayT5vi7+dX1lUpPc41rC6rk9th99zCXx++HDyaQ12/Eq6YUZoK0uIIwRrT2AZsBF5U1bsLcN0JwDfbex/bseyigIWqmq6qR4CDWIFkGLBJVc/b9v9YgpVcEVU9YfuahJX6xIywlRMx51N5YtZ2fOu68c/b25b4LCMR4e1hbWlUx5WnZu8gMSW9RMsvlLQLcHwrazPbENS0ZFpVFUbbkdA4CJZPsn4OBZWeApHrTGujEihI4JgH/E9Vv1PVmcAmEalegOu2Av4i0sy2R/mdwMIc5yzAam0gIvWxuq4igGNAbxFxEpFqWAPj+2zv69vOrwbcCuwpQF0MO8vMUp6eHUp8cjpTx3Qs0lhGQdRxq8bHdwZyMiGF//t5D2W2JvXYRhyy0tiQ1bpqDIxn5+AAA9+FpFNWEsSCOrYRMi6aabiVQEECxwog+8omN2D51S5S1QzgcWAZsA+Yq6phIjJZRIbYTlsGxIrIXqwxjedVNRYrWB0GdgM7sabpLsIaKF8mIruAUKwWzFcFeAbDzj77M5x14TFMHtKaVo3tuza0Y9O6PDMggEU7TzJvW5Rd75WniNWkSzWO12xPozqVdOFffpp0hdbDYf0ncK6A/waHV4CjM/j1tG/dDLsryHRc1+zbxarq+QK2OFDVxcDiHMdez/a9Yq1KfybHOZlYK9ZzlncBqIQJgSq29eExfLTiIMMDvRnVyffqF5SA8b2bs+5QDBMXhhHUtG7+25PaQ8QqdhNAK78qvMPAgDesXQKXvwG3F+DvN7PbX6VRkBbHBREJuvRGRDry1+I7o4o7k5jCU7N30MKzJm8Na1Nqq6cdHYQPR3XAxcmBJ2ftIDUjs1TuC8CFWDi9iz/TWhHUpIp1U2Xn3sRKgrh7Lhzfmv+5iSfhbJjppqokChI4ngZ+FJG1IrIOmIPVBWVUcRmZWTwxawcXUjOZOiao1PeiaFjHlX+NaE/YyUTeX3qg9G4caSVNWJ/VpuqNb+TUcwLUbGBNz81vvMns9lepFGQB4FagJfAoMB64TlW32btiRvn3nz8OsuVIHO8Mb4N/g7JJada/VQPGdvdj2rojrNxfSplnIlaR4liDA44taNWoiuf6dKkJ/SbCiRDYPS/v88JXWAGmQZvSq5thNwVZx/EPoIaq7lHVPUBNEXnM/lUzyrM/959h6qrDjO7chGGBPrmfpArno+FiAqRfLNn9HLJ5aVBLWjasxXM/7uRsYsrVLyiuiNXscmxLax8PnJ3M7nW0H22lSF8+CdKSr/w8K9O2218/K2GiUeEVpG9hnKpm38wpXkTGAVPtVy2jPIuKT2bCnJ20alSbiYNb5X6SKvz8COya8/fjDk7g6AJOzn99dXK98tilr651oG4z8LjG9mpmHcvGtZojn90VyK2fruOZuTv5/oHOODjY6RdU/FGIP8KyzF5Ve3wjOwcHK3vu9Jth42fQ+4W/f35yB1yMN91UlUhBAoejiMilTZxsyQud7Vsto7xKy8ji8R92kJmlTB0TlHeqjV1zraARdC94toSMVMhMy/Y1JZdjtq9pyZARZ32fHAcXcnRBVa9nBZFsAaWFRzPevbERE36L4su1EYzv3dw+P4AjqwFYndGa50zg+ItfD2uvjnUfQuDdULvxX5+FrwDE7PZXiRQkcCwF5ojIF7b3j2Ct5DaqoH8u2U/o8QSmjgnCr36N3E86FwWLn4cm3eDWj8ChmHmcUs9DfCTERViv+CPW12MbYfePWDkxrXQDA6vXIGKFJ/GRranrHWBlYW2ebyLnwolYxQXn+oSneFe9FeNXM2AyHFgCKybDsP/+dfzwCmjcAWpUwh0Sq6iCBI4XgYexBsYBdgEN7VYjo9xauucU36w/wtjuftzcNo/1C1lZsOAxyMqA2z4vftAAawC2YRvrlVNGqtV9ZAsoDmfDSQzdRu3IHbgfXYas+xBun2btYldcWVkQsZrdzh3w9aiOV61ykt69vKjrB93+YbU6Oo8D745WF1XUVuj5zFUvNyqOguwAmCUim4HmwEigPjDf3hUziu/nHVEs3XOaHi3q08vfM+8WQgEcjb3A8z/uor2vO6/cnE9y5C1fWt05gz+2xiPszckFPAOsF1ZqAef2cfT5YhPD2njwr9Q3kJ/HQ436cE2f4t3r7F5IjmGZ47V09DfdVLnq+QzsmAlLX4EHlkLEatAsaNG/rGtmlKA8A4eIBGBlrB0NxGCt30BVTUdlBfHF6gjCz55nWZi1lUnTetXp5e9J7wBPujWvRw2Xgq27SEnP5LGZ23FwEKbcFZj3TKLoA7B8IgQMhKD7SuoxCq1jUw+e7ufPv/84SK/b/s2Qiw/C7Lvh/sXQqN3VC8iLbZvYpRda8mhVX7+RF9facMOrsOhJCPvZmk3lUht8gsu6ZkYJyu83x36sFOq3qmo4gIhMKJVaGcV2NimF/aeTeGHgtdzcphGrD0az5mA087dHMWPTUao5Ch2b1qV3gBe9AurTqlHtPFd9v/nrXsJOJjLtvmB86uaRbSYz3ZpFVa06DP6kzKddPta3BevCY3h58TE6PjgD7/lDYeYIePD3oqf0PrKa8zX9OJVSz8yoyk/g3bDlK2unQM00u/1VQvlNQh+OtYnSShH5SkT6AWYSdgWx8XAsANe3sLqo7uvux7Sxndjx+gB+eKgLD/RsRkJyOu8t3c8tn6yj8zsreGZuKL+EniDuQtrlcn4JPcHMzccY37t5/vt9r/nAmnY5+COoVfb7gjs6CP8Z1QEHESYsiSZzzDxrPGTGcCtlSGFlpEHkeva6BlHd2ZGWDctmwWOF4OAIA9+xdvpLPGGm4VZCebY4VHUBsEBEamBt+fo04CUinwM/q+rvpVRHowjWHYqhj9thWp+OAe97L7cAXJwc6d6iPt1b1OflQddxJjGFNQejWXMohpX7z/LT9hOIQFvvOnRrXo8ZG4/S2c+D524MyPtmUdtgzb+g3Z3WlMxywtvdjUlDWvPsjzuZdsCLh++aA98PhR9Gwn0LrU2JCurENki/wB8p19Hexx0nR7PwL1/NekHLW2H/ryY/VSVUkJQjF1T1B1UdjLUZ0w6smVZGOaWqrAuP4f9c5uLw65Pw69PW6t1cNKjtyh3Bvnw6OpCQVwew4B89mNA/gGqODny1JoLqztZ+33n+okxLhp8ftrYTHfSeHZ+qaIYHeXNT6wZ8sOwgB5xbWzOsTm6HH8da3WsFFbEKFQfmxfmZabgFNeRTuGsu1C3ZnSCNsleoP5tUNd62Jav5E6Ici4i5QOy5JJqlHbQymG6bDvPut7pq8uHoIHTwdefJfv7Mf7Q7O16/kT8m9KZhnXymnS6fBLHhcNsUcCt/v1BFhHeGtaW2mxMT5oSS5n8z3PJvOPS7FVALuhFUxCoueLQhPquGGd8oqOoeEHBTWdfCsAO7trdFZKCIHBCRcBF5KY9zRorIXhEJE5Efsh1/33Zsn22/c7Ed7ygiu21lXj5u/GV9eAxt5AhOWalw0zvWa+8vMPMOSE0qcDl13KpRt0Y+SQIO/wlbvoAujxZ/qqsd1avpwrvD27H3VCKfrDgEwQ9A7xdhx/9g5dtXLyA1CU6EcLCGtbtAoAkcRhVnt8BhS00yBRgEtAJGi0irHOf4Ay8DPVS1NdY4CiLSHegBtAPaAJ2wto8F+BwYh7U3uT8w0F7PUFGtOxRD/5oR1hvfLtairNv+a+33/N1guBBT/JtcjIcF/4D6AdB/YvHLs7MBrRowMtiHqavC2X4sHvq8bKVDWfMvawZQfo5ugKwM/kxrxTX1a+CRXzA1jCrAni2OzkC4qkaoahowG2uQPbtxwBRVjQdQ1UtJiRRwxcqJ5QJUA86ISCOgtqpusuXO+h64zY7PUOFkZGaxMSKW3m4R4NEcanpZH3QYDXfOhLP74JuBkHC8eDda/IKVQ2rYF1CtYmyd+tqtrWhUx41n5oSSnJ4Jt3wIAYOs9Ch7F+Z9YcRq1NGF+We9CTLrNwzDroHDG8j+2ynKdiy7ACBARNaLyCYRGQigqhux9iA/ZXstU9V9tuujrlJmlbbrxDmSUtLxT9lj5YrK7tpBcM/PcP4sfHOTtWCvKMJ+tnZ96/UCeAdd/fxyopZrNf49sj1H45J5d/F+cHSCEd+ATyeY/5DVsshNxCpSGnXiVLKY8Q3DwM5jHAXghNXd1AdrhfpXIuIuIi2A67BmcXkDN4jI9YUpWEQeFpEQEQmJjo4u4WqXX+sPxdBcTuKclgBNulx5QtPucP9vVi6pbwZaU2kLI+k0/DrBykN0/bMlU+lS1PWaejzYoxkzNh1l9cFocK4Od82xJhHMuhPO7P37BefPwtkwDte0Vj5X+R3/DAP7Bo4TgG+29z62Y9lFAQtVNV1VjwAHsQLJMGCTqp5X1fNY2Xi72a73uUqZANhmfwWrarCnp2eJPFBFsC48hiEetoZezhbHJQ3bWnmEXGtbYx6XtvW8GlX45XFIT7G6qBxLd6vYkvLcTdfi71WTF+bt5FxyujX7556fwMkN/ne7ld33kiPWNrGrM1tTy8UJf6+aZVRrwyg/7Bk4tgL+ItJMRJyBO4GcHckLsFobiEh9rK6rCOAY0FtEnESkGtbA+D5VPQUkikhX22yqe4Ff7PgMFUpyWgbbj8XT1+2wtWdFvRZ5n+xxDTywzEpEOHMk7Pnp6jfY9i2E/2Glz67vX3IVL2Wu1Rz5z8gOxJ5P4/WFe6yD7k3g7vmQdt4KHhfjreMRK8G1Dr+d9aRDE3f7bRBlGBWI3QKHqmYAjwPLgH3AXFUNE5HJIjLEdtoyIFZE9mKNaTyvqrHAPOAwsBvYCexU1UW2ax4DvgbCbeeYvUFsthyJIz1T8U/dA75dr54vqlZDGPublYBu3gOw9eu8z409DMv+z9qMp9NDJVvxMtDWpw5P9vPnl9CT/LrrpHWwYRtrAkFcBMwabW13G7Ga9CY92X822YxvGIaNXfsaVHUxsDjHsdezfa/AM7ZX9nMysTaMyq3MEKwpukYO68NjaOSUiFvSUWhSwF/ubu5w90/WAsHfnoXkeOj13N+DTmYG/DzeSlQ3dIq1VWgl8Fif5qzYf5ZXF+yhs58HXrVdrVQZw76wAumM4XDuOEcDHiJLMTOqDMOmcvwGMABYeyiGkV62IZ+8xjdy41wdRv0P2o+GlW/B0pesTYsu2fAxRG2Bm/8NdSrPJDYnRwf+M7I9KemZvDh/F3ppFXmb4TDwn3DMmmW1Ias1ItDBt/ytjDeMsmACRyURnZTK/tNJ9HWLACdXaNS+cAU4VoOhU6HrP2Dzf60U6ZnpcGoXrHwXWt1WMrvolTPNPWvy8qDrWHkgmtlbs80e7zre2lfi2ptZcbY2/l41qeNmUoMbBti5q8ooPRsOW6vBA9LCrKmyTkVY3ezgADe9be0NvWIypCRYM4yq14NbPyzzPTbs5Z6uTflj7xne/HUv3ZvXo2k9W9bcXs+TlaXsmPw7t7TLY6tcw6iCTIujklgfHkND10zcYvdAk65FL0jEWp8x+GMIX25tlzr0M2vKaiXl4CC8P6Idjg7Cs3N3kpn1V+LDw9HnSUzJMPmpDCMbEzgqAVVl3aEYRjU+i2RlWDOqiqvjWLjrRys1tv+A4pdXzjV2d2Py0NaEHI3nq7URl49vP2ZNyzUL/wzjL6arqhI4EnOBk+dS6NvkCCDg26lkCvbvXzLlVBC3dfDm97Az/Of3g/QO8OS6RrXZdjQe9+rVuKZ+ITZ9MoxKzrQ4KoH14db4xrWpe8CrFbiZv46LQkR4e1hbartVY8KcUFIzMtl+LIFAX/c892M3jKrIBI5KYF14DE3cnXE9vS33/FRGgXnUcOa929uy/3QSkxftJfzsedNNZRg5mMBRwWVmKRsOx3K7TxKSllS49RtGrvpd14A7O/kyc/MxALNi3DByMIGjgtt94hxJKRncUOOwdaA4M6qMy169tRU+dd1wEGhvFv4Zxt+YwfEKbt0hK2V8QGoY1GoMdXyvcoVREDVdnPhmbCf2nkykhov538QwsjP/R1Rw68JjaNWoNi4nt1qtDTOIW2ICGtQioEGtsq6GYZQ7pquqAktOy2D70QRuaZIBiVFmfMMwjFJhAkcFtjUynrTMLG6oYVuwZmZUGYZRCkzgsKeds2HjFLsVvz48BmdHB/xTwsC5Fni1ttu9DMMwLjFjHPZyMQEWPw8ZKVa6cjvkelp3KIaOTevidGKztVq8gm7lahhGxWLXFoeIDBSRAyISLiIv5XHOSBHZKyJhIvKD7VhfEQnN9koRkdtsn00XkSPZPutgz2cosq1fQ2oiZKbBnvklXnzM+VT2nkrkhmYucCasZPJTGYZhFIDd/kQVEUdgCjAAiAK2ishCVd2b7Rx/4GWgh6rGi4gXgKquBDrYzvHA2ib292zFP6+q8+xV92JLuwCbpoL/jZB4EkJ/gM7jSvQWGw7HAtCv5lFAzfoNwzBKjT1bHJ2BcFWNUNU0YDYwNMc544ApqhoPoKpncylnBLBEVZPtWNeStf17SI6F65+DDnfBye1w9v/bu/foqKp7gePfHwnhHUhIgMj7EaQgNWBAykMRrUVrlVpLQezV1talXtu6Wr3q6rreXlddq7UPXFrv9Yqt2vKsVin20oq1XgUkvMMbIQXy4JkECI9Anr/7xz6Jw2QSZpI5M6P8zSHN0wAAFD5JREFUPmvNysk++5y952SGH2fvffbeHdUiVu8tI7VjMoPObgNJcuuGG2NMDPgZOPoCAUuqUeKlBRoODBeR1SKSJyLTQ5xnFrAoKO1pEdkqInNFpEOowkXkPhHZICIbSktLW/seIldbBaufg4GT3Sin0TOhXTJsWRi1IlSVVQVlTByaQbviPMj6PKTY7K3GmNiI96iqZCAbmArMBuaJSOP8DiKSBYwG3gk45glgBDAOSAceC3ViVX1JVXNVNTczM9Of2oeyZTGcPgTX/Mj93jUThn0RtiyButqoFFFYXsnBk+eYMrQ7HNxoz28YY2LKz8BxEAic/6KflxaoBFimqjWquh/YgwskDWYCb6lqTUOCqh5Wpwp4BdcklhjqamHVXLhsDAy57pP0nDvhzBHY935UilnpTaN+XephqD0H/e35DWNM7PgZONYD2SIyWERScE1Oy4LyLMXdbSAiGbimq30B+2cT1Ezl3YUgboGEGcB2PyrfKjuXwon9rm8jcOqP4dOhUzrkL4hKMav3ltG3RyeyKja7BOsYN8bEkG+BQ1VrgYdwzUy7gD+q6g4ReUpEbvWyvQOUi8hO4H3caKlyABEZhLtj+SDo1AtEZBuwDcgAfurXe4hIfT2s/BVkfg4uv/nCfckpMPoO2L0czp1oUzFuGvUyJg/LQIrXQtpg6NanTec0xphI+PrEmKouB5YHpT0ZsK3AD71X8LEHaNqZjqpOi3pFo2HP3+DYTrh9HrRz8fjV1fs5UF7Jk7eMpF3OnbDuJdj+Joy7t9XFbD9YwanztUwa1hNW5F0S64EbYxKLPWocDaqw8peQNghG3Q7AmapafvHOx5ytriO9Swrfn5bjlnXNX9imwLHK69+YnF4BlWXWTGWMibl4j6r6bNj/gRvdNOnhxmk/lm4+yNnqOnIHpjH373t4/+NS10l+cAOU7ml1Uav2lvG5rFTSyze5BBtRZYyJMQsc0fDhL6FblgsMuOcs5ucVMjIrlT/cezWf65PK9xdvprjvLe5hvVY+03Guuo6NhSeYkp0BRWugUxr0zL74gcYYE0UWONqqeB0cWAkTvwfJ7lnEjYUn2H3kNHdNGEinlCT+55tXkdRO+M6bxdQOud4961FfF3FR6w8cp7qunknDMqAoz81P1c7+hMaY2LJ/ddpq5a/cUNur7mlMmp9XSLcOydyWcxkA/dM78/zsMew9dppXKyfC6cOw7/8iLqphGvVxmXVQXmD9G8aYuLDA0RZHtrnRVBMebJzyo/xMFcu3HeH2sX0vWKt6SnYmj35pBM/sH8z55FTXSR6hVQVljB3Yg85HNrgECxzGmDiwwNEWK3/tFlAa/53GpNc3llBdV8+cCQObZL//2iHcMLo/r1ddTd2ut92aHWE6fraaHYdOMXmY17+R1ME9oW6MMTFmgaO1ygpgx1suaHRKA6C+Xlm4tojxg9MZ3rtbk0NEhGfuuJK81Okk1VVxYv2SsItb3TAMNzsTitdC37GNfSrGGBNLFjhaa/Vc9w/3hAcbkz7cW0rR8UruCnG30aBrh2QeuWcWBfTj8IevcL4mvE7y1QVldOuYzOhe7eFQvs1PZYyJGwscrXGy2I2MGns3dO3VmDw/r4iMrilMH9XyFCCDM7uin5/NyNpdzF28HPcAffNUlZV7y5g4tCdJhzdDfY09v2GMiRsLHK3x0fPu58TvNSYdPHmOf+w+yszc/qQkX/yyZt9wL/W0o9vHr/P7NYUt5i067qZRn9wwDBegf+JMCmyMubRY4IjUmWOw6TW4chb0+GTW+MXrilBg9vgB4Z0nNQsZdj13dviIp/+ynXX7jzebtWGakcbnNzJHQOf0trwLY4xpNQsckcr7L6irhsmfzMtYXVvPonXFXHd5L/qndw77VJIzm/S6Ur6SWsCDCzZxpOJ8yHyrvGnUB/fs5B44tGG4xpg4ssARiXMnYN3LMHIG9BzamLxi5xHKzlTxzRY6xUO6/MvQoTv/0T+fyupaHliwkaraCzvL3TTq5Uwa1hMp3Q1VFda/YYyJKwsckVj3MlSfhik/uiB5fl4h/dI6cc3wCJeobd8RRn+N1P1/49kZQ9lcdJL/fHvnBVl2HKqg4lyN10y1xiXaiCpjTBz5GjhEZLqIfCwiBSLyeDN5ZorIThHZISILvbTrRCQ/4HVeRGZ4+waLyFrvnEu81QX9V3XGNVMNnw59rmhMLjh2mrx9x7nz6gEktZMWTtCMnDlQe44bWcP91w5l4doilqwvatzd0L8xcajXv9G1j5u+3Rhj4sS3wCEiScALwE3ASGC2iIwMypMNPAFMUtVRwMMAqvq+quaoag4wDagEVniH/RyYq6rDgBNA6xe3iMSm1+DccbcsbID5eUW0TxJm5vZv5sCL6HuVm+E2fyGPfulypmRn8O9Ld5Bf7J4qX11Qxog+3cjs1gGK1rr+DWlFgDLGmCjx845jPFCgqvtUtRpYDNwWlOe7wAuqegJAVY+FOM8dwF9VtdJbZ3wa8Ia37zXcuuP+qq1yQ3AHTYH+4xqTK6tr+dOmEm66IouMrq18ilvETcdetIakE/t4btYYeqV24IH5Gyk5Ucn6A9406hUlUFFkHePGmLjzM3D0BYoDfi+h6VKww4HhIrJaRPJEZHqI88wCFnnbPYGT3nrmzZ0TABG5T0Q2iMiG0tLSVr8JwE1IePowXHPh3cbbWw5x+nxti0+Kh+XKWSDtYMti0rqk8OJdV3H8bDVff3EN1bUB06iDBQ5jTNzFu3M8GcgGpgKzgXki0qNhp4hkAaOBdyI9saq+pKq5qpqbmRlhp3WgulpY/axrUhp87QW75ucVMbx3V8YNSmv9+QFSL4MhU2HLIqiv54q+3fnZ10ZzuOI87ZOE8YPT3fxU7btA79FtK8sYY9rIz8BxEAhs+O/npQUqAZapao2q7gf24AJJg5nAW6pa4/1eDvQQkYb5ykOdM7p2vAknDri+jYC+hS3FJ9l2sIK7JgxEotHnkDMHKordolDAV8f04+EbsvnmhEF0Tkl2I6r65TYuTWuMMfHiZ+BYD2R7o6BScE1Oy4LyLMXdbSAiGbimq30B+2fzSTMV6iZ1eh/X7wFwN/BnPyoPQH29W6ip10g3mirAH/IK6ZySxFfHhGwpi9yIL0OHVHfX4Xn4huE8+ZWRcP4UHN1hz28YYxKCb4HD64d4CNfMtAv4o6ruEJGnRORWL9s7QLmI7MQFhEdVtRxARAbh7lg+CDr1Y8APRaQA1+fxW7/eAx8vh9Ld7rmNgCVaT1ZW8/aWQ8wY05duHdtHp6z2neCK22Hnn6Hq9IX7StaD1lv/hjEmIfja7qGqy4HlQWlPBmwr8EPvFXzsAUJ0fKvqPtyILf+t+Q2kDXZPigd4Y2MJVbX13HV1GzvFg115J2x81QWPMXd9kl6U5zrP++VGtzxjjGmFeHeOJ7Y7XoHb513Qr6DqFmsaO6AHIy9LjW55/cdD+lDIX3RhetEa6DMaOjRdHMoYY2LNAkdLUrMueG4D4KN/lrOv7Gzbh+CG0vBMR+EqOL7fpdXVwMGN1r9hjEkYFjgiND+vkLTO7bl5dJY/BVw5CxC3UBTAka1QU2nzUxljEoYFjggcPXWeFTuP8vXc/nRsn+RPId37wZBrG5/psAf/jDGJxgJHBBavK6auXrkz3MWaWitnDpwshKKPXODoMdA9JGiMMQnAAkeYauvqWbSuiCnZGQzK6OJvYSNugZRusHmBCxx2t2GMSSAWOML03u5jHDl1PvLFmlojpTOMmgHbXoezxyxwGGMSigWOMM3PKySre0emjegVmwJz5kC9N9OKjagyxiQQCxxh2F92lpV7y5g9fgDJSTG6ZAMmuIcPO3aHjMtjU6YxxoTBZswLw8K1hSS3E2aNa+ViTa0hAl95Fs6WXTDdiTHGxJsFjos4X1PH6xtLuHFUb3qldoxt4UOmxrY8Y4wJg/1X9iL+d+thTlbWRH9eKmOM+ZSywHER89cWMiSzC18Y2jPeVTHGmIRggaMFOw5VsLnoJHOujtJiTcYY8xlggaMF8/OK6Ni+HXeM7RfvqhhjTMKwwNGCgT07861Jg+neOUqLNRljzGeAr4FDRKaLyMciUiAijzeTZ6aI7BSRHSKyMCB9gIisEJFd3v5BXvqrIrJfRPK9V45f9b//2qE8Nn2EX6c3xphPJd+G44pIEvAC8EWgBFgvIstUdWdAnmzgCWCSqp4QkcDHsn8PPK2q74pIV6A+YN+jqvqGX3U3xhjTPD/vOMYDBaq6T1WrgcXAbUF5vgu8oKonAFT1GICIjASSVfVdL/2Mqlb6WFdjjDFh8jNw9AWKA34voeka4sOB4SKyWkTyRGR6QPpJEXlTRDaLyC+8O5gGT4vIVhGZKyIdQhUuIveJyAYR2VBaWhqt92SMMZe8eHeOJwPZwFRgNjBPRHp46VOAR4BxwBDgHu+YJ4ARXno68FioE6vqS6qaq6q5mZmZPr4FY4y5tPgZOA4CgZM79fPSApUAy1S1RlX3A3twgaQEyPeauWqBpcBYAFU9rE4V8AquScwYY0yM+Bk41gPZIjJYRFKAWcCyoDxLcXcbiEgGrolqn3dsDxFpuFWYBuz08mV5PwWYAWz38T0YY4wJ4tuoKlWtFZGHgHeAJOB3qrpDRJ4CNqjqMm/fjSKyE6jDjZYqBxCRR4D3vACxEZjnnXqBF1AEyAfu9+s9GGOMaUpUNd518F1ubq5u2LAh3tUwxphPFRHZqKq5TdIvhcAhIqVAYSsPzwDKolidaLP6tY3Vr22sfm2T6PUbqKpNRhddEoGjLURkQ6iImyisfm1j9Wsbq1/bJHr9mhPv4bjGGGM+ZSxwGGOMiYgFjot7Kd4VuAirX9tY/drG6tc2iV6/kKyPwxhjTETsjsMYY0xELHAYY4yJiAUOz8UWnRKRDiKyxNu/tmFhqRjVrb+IvB+w4NUPQuSZKiIVAQtcPRmr+nnlHxCRbV7ZTZ62FOc57/ptFZGxMazb5QHXJV9ETonIw0F5Ynr9ROR3InJMRLYHpKWLyLsistf7mdbMsXd7efaKyN0xrN8vRGS39/d7y5uQNNSxLX4WfKzfT0TkYMDf8OZmjr3oAnM+1W9JQN0OiEh+M8f6fv3aTFUv+RduSpR/4mbhTQG2ACOD8jwIvOhtzwKWxLB+WcBYb7sbbjLI4PpNBf4Sx2t4AMhoYf/NwF9xU8VMANbG8W99BPdgU9yuH3ANbuLO7QFpzwCPe9uPAz8PcVw6bj63dCDN206LUf1uxK2TA/DzUPUL57PgY/1+AjwSxt+/xe+6X/UL2v8r4Ml4Xb+2vuyOwwln0anbgNe87TeA6715tHynbkbgTd72aWAXTdc2SXS3Ab9XJw83iWVWHOpxPfBPVW3tTAJRoaofAseDkgM/Y6/hJvEM9iXgXVU9rm4BtHeB6SHyRb1+qrpC3WzVAHm4Ga/jopnrF45wvutt1lL9vH83ZgKLol1urFjgcMJZdKoxj/flqQB6xqR2AbwmsjHA2hC7vyAiW0TkryIyKqYVAwVWiMhGEbkvxP5wrnEszKL5L2w8rx9Ab1U97G0fAXqHyJMo1/HbuDvIUC72WfDTQ15T2u+aaepLhOs3BTiqqnub2R/P6xcWCxyfIuLWXv8T8LCqngravQnX/HIl8DxuyvpYmqyqY4GbgH8VkWtiXP5FiZve/1bg9RC74339LqCuzSIhx8qLyI+BWmBBM1ni9Vn4b2AokAMcxjUHJaLZtHy3kfDfJQscTjiLTjXmEZFkoDtQHpPauTLb44LGAlV9M3i/qp5S1TPe9nKgvbg1TmJCVQ96P48Bb9F0ga1wrrHfbgI2qerR4B3xvn6eo/LJejNZwLEQeeJ6HUXkHuAWYI4X3JoI47PgC1U9qqp1qlqPW4YhVLnxvn7JwO3AkubyxOv6RcIChxPOolPLgIYRLHcA/2juixNtXpvob4FdqvrrZvL0aehzEZHxuL9tTAKbiHQRkW4N27hO1OAFtpYB/+KNrpoAVAQ0y8RKs//Ti+f1CxD4Gbsb+HOIPA1r2KR5TTE3emm+E5HpwL8Bt6pqZTN5wvks+FW/wD6zrzZTbjjfdT/dAOxW1ZJQO+N5/SIS7975RHnhRv3swY24+LGX9hTuSwLQEdfEUQCsA4bEsG6Tcc0WW3GLV+V79b0fuN/L8xCwAzdKJA+YGMP6DfHK3eLVoeH6BdZPgBe867sNyI3x37cLLhB0D0iL2/XDBbDDQA2unf1eXJ/Ze8Be4O9Aupc3F3g54Nhve5/DAuBbMaxfAa5/oOEz2DDK8DJgeUufhRjV7w/eZ2srLhhkBdfP+73Jdz0W9fPSX234zAXkjfn1a+vLphwxxhgTEWuqMsYYExELHMYYYyJigcMYY0xELHAYY4yJiAUOY4wxEbHAYUwUiEhd0Ay8UZt1VUQGBc6yaky8Jce7AsZ8RpxT1Zx4V8KYWLA7DmN85K2t8Iy3vsI6ERnmpQ8SkX94E/K9JyIDvPTe3loXW7zXRO9USSIyT9x6LCtEpFPc3pS55FngMCY6OgU1VX0jYF+Fqo4GfgM866U9D7ymqp/HTRb4nJf+HPCBuskWx+KeHgbIBl5Q1VHASeBrPr8fY5plT44bEwUickZVu4ZIPwBMU9V93kSVR1S1p4iU4abEqPHSD6tqhoiUAv1UtSrgHINwa3Bke78/BrRX1Z/6/86MacruOIzxnzazHYmqgO06rH/SxJEFDmP8942An2u87Y9wM7MCzAFWetvvAQ8AiEiSiHSPVSWNCZf9r8WY6OgkIvkBv/9NVRuG5KaJyFbcXcNsL+17wCsi8ihQCnzLS/8B8JKI3Iu7s3gAN8uqMQnD+jiM8ZHXx5GrqmXxrosx0WJNVcYYYyJidxzGGGMiYnccxhhjImKBwxhjTEQscBhjjImIBQ5jjDERscBhjDEmIv8PtB1aQ0SJ1E4AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.6987787094422165"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RdPUiPazDyEJ"
      },
      "source": [
        "**Model 2:GRU:** Our second model is a GRU with a fully-connected layer applied to the last hidden state, mapping its value into a  2×1  vector (which is converted to a distribution in the computation of the cross entropy loss)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bvyUm6eEPPE"
      },
      "source": [
        "class TweetGRU(nn.Module):\n",
        "    def __init__(self, input_size, hidden_size, num_classes):\n",
        "        super(TweetGRU, self).__init__()\n",
        "        self.emb = nn.Embedding.from_pretrained(glove.vectors)\n",
        "        self.hidden_size = hidden_size\n",
        "        self.rnn = nn.GRU(input_size, hidden_size, batch_first=True)\n",
        "        self.fc = nn.Linear(hidden_size, num_classes)\n",
        "    \n",
        "    def forward(self, x):\n",
        "        # Look up the embedding\n",
        "        x = self.emb(x)\n",
        "        # Forward propagate the GRU \n",
        "        out, _ = self.rnn(x)\n",
        "        # Pass the output of the last time step to the classifier\n",
        "        out = self.fc(out[:, -1, :])\n",
        "        return out"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qYT_x3TVVLnW"
      },
      "source": [
        "Applying the GRU to our data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 646
        },
        "id": "Fwq2BeECESOP",
        "outputId": "cf206e19-3780-4640-ee72-8d52c5770d8e"
      },
      "source": [
        "modelGRU = TweetGRU(50, 50, 2)\n",
        "# Train and produce training curve and validation curve\n",
        "train(model, train_loader, valid_loader, num_epochs=20, learning_rate=2e-5)\n",
        "# Compute test accuracy\n",
        "get_accuracy(model, test_loader)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1; Loss 0.579916; Train Acc 0.706622; Val Acc 0.703082\n",
            "Epoch 2; Loss 0.489776; Train Acc 0.705740; Val Acc 0.703538\n",
            "Epoch 3; Loss 0.477442; Train Acc 0.707598; Val Acc 0.703082\n",
            "Epoch 4; Loss 0.576201; Train Acc 0.706937; Val Acc 0.701076\n",
            "Epoch 5; Loss 0.479983; Train Acc 0.707504; Val Acc 0.701897\n",
            "Epoch 6; Loss 0.587661; Train Acc 0.707850; Val Acc 0.703903\n",
            "Epoch 7; Loss 0.498737; Train Acc 0.708512; Val Acc 0.703812\n",
            "Epoch 8; Loss 0.569685; Train Acc 0.708606; Val Acc 0.701258\n",
            "Epoch 9; Loss 0.531579; Train Acc 0.707094; Val Acc 0.702717\n",
            "Epoch 10; Loss 0.648734; Train Acc 0.707567; Val Acc 0.704541\n",
            "Epoch 11; Loss 0.588156; Train Acc 0.708197; Val Acc 0.704541\n",
            "Epoch 12; Loss 0.587276; Train Acc 0.708669; Val Acc 0.703721\n",
            "Epoch 13; Loss 0.597835; Train Acc 0.708323; Val Acc 0.702170\n",
            "Epoch 14; Loss 0.623071; Train Acc 0.709362; Val Acc 0.703082\n",
            "Epoch 15; Loss 0.463287; Train Acc 0.709835; Val Acc 0.702353\n",
            "Epoch 16; Loss 0.559710; Train Acc 0.709362; Val Acc 0.703812\n",
            "Epoch 17; Loss 0.554710; Train Acc 0.710118; Val Acc 0.703356\n",
            "Epoch 18; Loss 0.559829; Train Acc 0.710874; Val Acc 0.703447\n",
            "Epoch 19; Loss 0.508009; Train Acc 0.709740; Val Acc 0.701988\n",
            "Epoch 20; Loss 0.560618; Train Acc 0.711379; Val Acc 0.703538\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzddXzVZfvA8c+1prtH1wDp0SAgSAiCEgoIghioj1g/u/N57MCmDERAEREDEZCSHimdAwYCg8GIsb5/f9xnOGFsZ9upbdf79dqLE9+4zmE71/necd1ijEEppZRylp+3A1BKKZW3aOJQSimVLZo4lFJKZYsmDqWUUtmiiUMppVS2aOJQSimVLZo4lMqEiMwVkZGu3lapvEx0HofKb0TkXLq7hYEEIMVxf4wxZqrno8odESkOvAQMAEoDx4CfgFeMMSe8GZsqePSKQ+U7xpiiaT/AQeD6dI9dTBoiEuC9KJ0nIkHAQqAR0AsoDrQDTgKtc3C8PPG6le/SxKEKDBHpIiJRIvK4iBwFPheRUiLys4hEi8gpx+3QdPssFpE7HLdHicifIvKWY9v9ItI7h9vWFJGlInJWRBaIyEci8vUVQr8VqAbcaIzZZoxJNcYcN8a8bIz51XE8IyJ10h3/CxF5JZPXvV1E+qbbPsDxHrRw3G8rIitE5LSIbBKRLrl9/1X+oYlDFTQVsU091YG7sH8DnzvuVwMuAB9msn8bYCdQFngDmCQikoNtvwHWAGWAF4ARmZyzO/CbMeZcJttk5dLXPQ0Ymu75nsAJY8x6EakC/AK84tjnEeB7ESmXi/OrfEQThypoUoHnjTEJxpgLxpiTxpjvjTFxxpizwKtA50z2P2CMmWCMSQG+BCoBFbKzrYhUA1oBzxljEo0xfwJzMjlnGeDv7L3My/zrdWMTVz8RKex4fhg2mQAMB341xvzquLqZD0QA1+UyBpVPaOJQBU20MSY+7Y6IFBaRz0TkgIicAZYCJUXE/wr7H027YYyJc9wsms1tKwMx6R4DOJRJzCexSSc3/vW6jTF7gO3A9Y7k0Q+bTMBelQx2NFOdFpHTQEcXxKDyCe0kUwXNpcMI/w+oD7QxxhwVkWbABuBKzU+u8DdQWkQKp0seVTPZfgHwiogUMcacv8I2cdgRZGkqAlHp7mc0fDKtucoP2OZIJmCT2BRjzJ1ZvA5VQOkVhyroimH7NU6LSGngeXef0BhzANv084KIBIlIO+D6THaZgv0w/15EwkTET0TKiMhTIpLWfLQRGCYi/iLSi8yb29JMB3oA9/DP1QbA19grkZ6O44U4OthDMzyKKnA0caiC7j2gEHACWAX85qHz3sI/Q2pfAWZg55tcxhiTgO0g3wHMB85gO9bLAqsdmz2ATT6nHceenVUAxpi/gZVAe8f50x4/BPQHngKisUnrUfTzQjnoBEClfICIzAB2GGPcfsWjVG7pNwilvEBEWolIbUezUy/sN/wsrxKU8gXaOa6Ud1QEZmGH2kYB9xhjNng3JKWco01VSimlskWbqpRSSmVLgWiqKlu2rKlRo4a3w1BKqTxl3bp1J4wxl5WaKRCJo0aNGkRERHg7DKWUylNE5EBGj2tTlVJKqWzRxKGUUipbNHEopZTKlgLRx5GRpKQkoqKiiI+Pz3pjlaWQkBBCQ0MJDAz0dihKKTcrsIkjKiqKYsWKUaNGDa68Do9yhjGGkydPEhUVRc2aNb0djlLKzQpsU1V8fDxlypTRpOECIkKZMmX06k2pAqLAJg5Ak4YL6XupVMFRoBOHUkrlV1uPxDJu4W7OJyS7/NiaOLzk5MmTNGvWjGbNmlGxYkWqVKly8X5iYmKm+0ZERHD//fd7KFKlVF707vxdTFy2jxQ31CMssJ3j3lamTBk2btwIwAsvvEDRokV55JFHLj6fnJxMQEDG/z3h4eGEh4d7JE6lVN6z6dBpFmw/ziM96lE8xPUjHfWKw4eMGjWKu+++mzZt2vDYY4+xZs0a2rVrR/PmzWnfvj07d+4EYPHixfTt2xewSWf06NF06dKFWrVqMW7cOG++BKWUD3hn/i5KFQ5kVAf3jHLUKw7gxZ+2su3IGZces2Hl4jx/faNs7xcVFcWKFSvw9/fnzJkzLFu2jICAABYsWMBTTz3F999/f9k+O3bsYNGiRZw9e5b69etzzz336HwKpQqodQdiWLIrmid6h1E02D0f8Zo4fMzgwYPx9/cHIDY2lpEjR7J7925EhKSkpAz36dOnD8HBwQQHB1O+fHmOHTtGaGioJ8NWSvmId+bvomzRIG5tV91t59DEATm6MnCXIkWKXLz97LPP0rVrV3744QciIyPp0qVLhvsEBwdfvO3v709ysutHUSilfN+qfSdZvuckz/RpQOEg9328ax+HD4uNjaVKlSoAfPHFF94NRinl04wxvDN/F+WLBTO8rfuuNkATh0977LHHePLJJ2nevLleRSilMrVi70nW7I/hP13rEBLo79ZzFYg1x8PDw82lCzlt376dBg0aeCmi/EnfU6W8wxjDwE9W8HdsPIse6eKyxCEi64wxl4391ysOpZRygymrDnD3lHVcSExx+7mW7Ipm/cHT3HeN+682QDvHlVLKpYwxvDlvJx8v3gtAyZ+28trAJm493zvzdxFaqhCDW1Z123nS08ShlFIukpySylM//MW3EVEMbV2NEoUC+XTJXtrXKUu/ppXdcs6F24+zOSqWNwY2ISjAM41ImjiUUsoFLiSmcN8361m44zj3d6vLQ93rkpxqWBsZw1Oz/qJpaAmqlymS9YGyITXVXm1UL1OYG1tUcemxM6N9HEoplUun4xIZPmk1f+w8zss3XMXD19ZDRAj09+P9Ic3wExg7bQOJyakuPe/v246y7e8zPNCtLoH+nvs418ShlFK5cOT0BQZ/upK/omL5aFgLRlwyhyK0VGHeGNTUNif9tsNl501NNbw7fze1yhWhfzPPXW2AJg6v6dq1K/PmzfvXY++99x733HNPhtt36dKFtCHF1113HadPn75smxdeeIG33nor0/POnj2bbdu2Xbz/3HPPsWDBguyGr5QCdh87y8BPVnA0Np4vRrfiusaVMtyu11UVubVddSb+uZ8/dhxzybl/+etvdh47y4Pd6+Hv59mF1DRxeMnQoUOZPn36vx6bPn06Q4cOzXLfX3/9lZIlS+bovJcmjpdeeonu3bvn6FhKFWTrDpxi0KcrSU41TB/Tlva1y2a6/VPXNaBBpeL837ebOBqbu2WWU1IN7y3YRb0KRel7hWTlTpo4vGTQoEH88ssvFxdtioyM5MiRI0ybNo3w8HAaNWrE888/n+G+NWrU4MSJEwC8+uqr1KtXj44dO14suw4wYcIEWrVqRdOmTRk4cCBxcXGsWLGCOXPm8Oijj9KsWTP27t3LqFGjmDlzJgALFy6kefPmNG7cmNGjR5OQkHDxfM8//zwtWrSgcePG7NjhusttpfKihduPccvEVZQqHMj3d7enUeUSWe4TEujPh8Oak5CcygPTN5CSmvPJ13M2HWZv9Hke6l4PPw9fbYCOqrLmPgFH/3LtMSs2ht6vXfHp0qVL07p1a+bOnUv//v2ZPn06N910E0899RSlS5cmJSWFbt26sXnzZpo0yXgM+Lp165g+fTobN24kOTmZFi1a0LJlSwAGDBjAnXfeCcAzzzzDpEmTGDt2LP369aNv374MGjToX8eKj49n1KhRLFy4kHr16nHrrbfyySef8OCDDwJQtmxZ1q9fz8cff8xbb73FxIkTXfEuKZXnfBtxiCdn/UXDSsX5/LZWlC0anPVODrXLFeXl/lfxf99t4oM/dvNg93rZPn9ySirvL9hNg0rF6dmoYrb3dwW94vCi9M1Vac1U3377LS1atKB58+Zs3br1X81Kl1q2bBk33ngjhQsXpnjx4vTr1+/ic1u2bKFTp040btyYqVOnsnXr1kxj2blzJzVr1qRePfuLPHLkSJYuXXrx+QEDBgDQsmVLIiMjc/qSlcqzjDF8vHgPj83cTPvaZZh2V9tsJY00A1uGMqB5FcYt3M2qfSezvf+sDYeJPBnHQ93reuVqA/SKw8rkysCd+vfvz0MPPcT69euJi4ujdOnSvPXWW6xdu5ZSpUoxatQo4uNz1hY6atQoZs+eTdOmTfniiy9YvHhxrmJNK92uZdtVQZSaanj5l218vjySfk0r89bgprmabPfyDVex4dBpHpy+kV8f6ETpIkFO7ZeYnMq4hbtpXKUE1zaskOPz55ZecXhR0aJF6dq1K6NHj2bo0KGcOXOGIkWKUKJECY4dO8bcuXMz3f/qq69m9uzZXLhwgbNnz/LTTz9dfO7s2bNUqlSJpKQkpk6devHxYsWKcfbs2cuOVb9+fSIjI9mzZw8AU6ZMoXPnzi56pUrlXYnJqTw4YyOfL4/ktg41eO/mZrmeoV0kOIAPhzUn5nwij363CWeLzc5cF0XUqQsX54l4iyYOLxs6dCibNm1i6NChNG3alObNmxMWFsawYcPo0KFDpvu2aNGCm2++maZNm9K7d29atWp18bmXX36ZNm3a0KFDB8LCwi4+PmTIEN58802aN2/O3r17Lz4eEhLC559/zuDBg2ncuDF+fn7cfffdrn/BSuUh5xKSGf3FWuZsOsLjvcJ4rm9DlzUPNapcgqf7NGDhjuNMXh6Z5fYJySl8+MdumlcrSZf65VwSQ05pWXXlMvqeqvzkxLkEbvt8Ldv+PsP/BjTmpnDXFxA0xjBmyjoW7TzO9/e0p0nolYfZf7Uykud+3MqU21vTqa5nEodXyqqLSC8R2Skie0TkiQyef1dENjp+donI6XTP/SYip0Xk50v2qSkiqx3HnCEizjUOKqWUkw6ejGPQJyvYffws40e0dEvSABAR3hjUhHJFgxk7bQNn45My3C4+KYWPFu2hdY3SdKyT+XwRT3Bb4hARf+AjoDfQEBgqIg3Tb2OMecgY08wY0wz4AJiV7uk3gREZHPp14F1jTB3gFHC7O+JXShVMW4/EMuCTFZyKS2LqHW3p1sC9ndAlCwcxbmhzok5d4OkftmTY3zF19UGOnUngIS/3baRx5xVHa2CPMWafMSYRmA70z2T7ocC0tDvGmIXAv3pxxb5j1wAzHQ99CdyQ0wALQjOdp+h7qfKDFXtPcPNnqwj0F2be3Y6W1Ut55LzhNUrz8LX1mLPpCN9FRP3rubjEZD5ZvIf2tcvQrnYZj8STFXcmjirAoXT3oxyPXUZEqgM1gT+yOGYZ4LQxJm08aGbHvEtEIkQkIjo6+rLnQ0JCOHnypH7guYAxhpMnTxISEuLtUJTKsV//+ptRk9dSqUQI39/TnroVinn0/Hd3rk2HOmV4bs4Wdh/75zvzlJUHOHEukYevzf5kQXfxlXkcQ4CZxhiXrbFojBkPjAfbOX7p86GhoURFRZFRUlHZFxISQmhoqLfDUCpHpqyM5Lk5W2lRrRSTRoZTsrDnu079/YR3b2pG7/eXcd83G/jxvg4kpxo+XbKXq+uVI7xGaY/HdCXuTByHgfQ9SqGOxzIyBPiPE8c8CZQUkQDHVUdmx8xUYGAgNWvWzMmuSik3O34mnnfm7+JgTBz/6VqHDm7qEDbG8O78XYz7Yw/dwsrz4bAWFApy/5rdV1K+eAjv3NyMkZPX8PLP26hcshCn4pJ86moD3Js41gJ1RaQm9sN9CDDs0o1EJAwoBazM6oDGGCMii4BB2D6TkcCPrgxaKeU9FxJTmLBsH58u2UtSSipligRzy8TVdG9Qnqeua0CtckVddq7klFSe/XEL09Yc4qbwUP57Y2MCPLgY0pV0rleOMZ1r8dmSfQQH+NEtrDzNquasGra7uC1xGGOSReQ+YB7gD0w2xmwVkZeACGPMHMemQ4Dp5pLOBhFZBoQBRUUkCrjdGDMPeByYLiKvABuASe56DUopz0hNNczZdITXf9vB37Hx9GpUkSevC6NC8RAmL9/Px4v20uPdpYxoV50HutXNdVNSfFIK90/bwO/bjvGfrrV5pEd9nxitlOaRHvVZvS+GjYdO85CPXW1AAZ4AqJTyDesOxPDSz9vZdOg0V1UpzjN9GtK21r9HD0WfTeCd+buYsfYgxUICebB7XYa3rZ6j5VJjLyRx55cRrD0Qw3N9G3JbB99ssj51PpGdx85e9l540pUmAGriUEp5RdSpOF6bu4OfN/9N+WLBPNqzPgNbhGZa0mPH0TO88vN2/txzglrlivD0dQ24Jqy801cLx87Ec+ukNew7cY53bmrG9U0ru+rl5EuaODRxKOUTzsYn8cnivUz8cz9+AnddXZsxV9eiSLBzLefGGP7YcZxXf93OvujzdKxTlmf6NiCsYvFM99sbfY5bJ63hdFwin40Ip2Nd78/A9nWaODRxKOVVKamGbyMO8fbvOzlxLpEbm1fh0Z71qVyyUI6Ol5SSytRVB3hv4W7OXEji5lbVePjaepQrdvkaGRsOnmL0F2vx9xO+uK01V1XJesU+pYlDE4dSXrR8zwle/nkbO46eJbx6KZ7p29BlI4VOxyUybuEevloZSUigP/d2rc3oDjUJCbTDahftPM69X6+nXLFgvhrdmhpli7jkvAWBJg5NHEp53L7oc/z31+0s2H6c0FKFeKJ3GH0aV3LLCCZ7rh0s2H7s4rkSk1N5bOZm6lUoxhejW1G+mFY3yA5NHJo4lPKor1ZG8tJP2wgJ9Oc/XetwW4caF68C3GnFnhO85Li6AWhfuwyfjWhJsZBAt587v7lS4vCVkiNKqXxk6a5oXpizlS71y/P6wCYZ9ju4S/s6Zfnl/k7MXHeI/SfieOjaugQHeG82eH6kiUMp5VIHTp5n7LQN1KtQjA+HNadwkOc/Zvz9hJtbVfP4eQsK78+vV0rlG+cTkhkzZR0A40eEeyVpKPfTxKGUcgljDI/O3MSuY2f5cFhzqpUp7O2QlJto4lBKucTHi/fy619HeaJ3mMfWxFbeoYlDKZVri3Ye563fd9KvaWXu7FTL2+EoN9PEoZTKlf0nznP/tA00qFic1wc28akqs8o9NHEopXLsXEIyd30VQYCf8NmIll5dBEl5jg55UErlSGqq4eEZG9l34jxfjW5N1dLaGV5Q6BWHUipHPly0h9+3HePJ3mFuW9pV+SZNHEqpbFuw7RjvLtjFjc2rcHtH31wISbmPJg6lVLbsjT7HQzM20qhycf43oLF2hhdAmjiUyoeMMSzdFc20NQeJjUty2XHPxCdx51cRBAX48dmIcI8ULVS+RzvHVb5zITGFv2Mv8HdsPKGlClG9TMFaf2Hn0bO88ss2lu0+AcCLP23lhmZVGN62eq4WMErrDD9wMo6pd7ShSg4XYFJ5nyYOlackJKdwLDaBI7EX+Dv2AkdOx9skcTqeI7H29ul037DLFQtm8SNdnF6WNC87cS6Bd+fvYtqagxQNDuCZPg0Ir1Ga6WsOMnvjYaavPUTzaiW5tV11el9VKdtXC+8t3M2C7cd54fqGtK1Vxk2vQuUFuh6H8lnL95xg/rZjF68ejpyO58S5hMu2K1EokEolQqhcstC//k1ONTw2czMPdq/Lg93reeEVeEZCcgqfL4/koz/2EJeUwvA21Xigez1KFwm6uE1sXBIz10fx9aoD7D9xntJFgrgpvCq3tKnm1DDaeVuPMmbKOga1DOXNQTrJr6DQhZw0ceQZqamG9xfu5v2Fuykc5P9PQihRiEol//m3UolCVC4ZkmkF1nunrmPxzmgWP9KF8sXz1+pvxhjmbjnK/+Zu51DMBbrWL8fTfRpQp3yxK+6TmmpYsfckU1ZFMn/bMQxwTf3yDG9Xnc51y+Hnd3lC2H3sLDd8tJw65YsyY0w77dcoQDRxaOLIE2IvJPHwjI0s3HGcgS1CefXGq3L1QRV54jzXvruEQS1D+d+AJi6M1Ls2R53mlZ+3syYyhvoVivF0nwZcXS97hQWPnL7AtDUHmbbmECfOJVCtdGGGt63G4JZVKeW4Wom9kMQNHy3nbHwSP43tSKUS2q9RkGji0MTh83YePcuYKRFEnbrAc9c3ZETb6i5pEnnxp618uSKS3x68mnoVrvxtPC84GhvPG/N2MGv9YcoUCeLhHvW4ObwqAf45HyCZmJzKvK1HmbLyAGsiYwgK8OP6JpUZ3rYa4xbuZtnuE3xzZ1ta1yztwlei8gJNHJo4fNrPm4/w2MzNFAkO4ONbWtCqhus+pE6dT+TqNxfRqkZpJo9q5bLjelJcYjLjl+7jsyX7SEk13NaxBv/pWofiLl5He8fRM3y96gA/rD/M+cQUAF6+4SpGtK3u0vOovEEThyYOn5Scksqb83by2dJ9tKhWkk+Gt6SCG/oiPl2yl9fm7uCbO9rQPg+Vx0hNNczeeJg3ftvJ0TPxXNe4Ik/0auD2RZLOxicxe8NhklIMt3WooZ3hBZQmDk0cPifmfCJjp61n+Z6T3NKmGs9f34igAPfMSY1PSqHb20soWTiQn+7rmGEnsK/ZdOg0z/64hc1RsTQJLcGzfRu69EpMqaxcKXHk/8Htyif9FRXL3V+vI/pcAm8MbMJNraq69Xwhgf482rM+D87YyOyNhxnQItSt58sNYwyT/tzPa3N3UKZoEO/c1JQbmlXJE8lOFQxuLTkiIr1EZKeI7BGRJzJ4/l0R2ej42SUip9M9N1JEdjt+RqZ7fKiI/CUim0XkNxHJO+0OCoCZ66IY+OkKjDF8N6ad25NGmn5NK9O4SgnemreT+KQUj5wzu2LjkhgzZR2v/LKdrmHl+f3BzgxoEapJQ/kUtyUOEfEHPgJ6Aw2BoSLSMP02xpiHjDHNjDHNgA+AWY59SwPPA22A1sDzIlJKRAKA94GuxpgmwGbgPne9BuVaicmpPPfjFh75bhMtq5ViztiONK1a0mPn9/MTnrquAUdi4/l8eaTHzuuszVGn6fPBMv7YcZxn+jRg/IiWlCjs2s5vpVzBnVccrYE9xph9xphEYDrQP5PthwLTHLd7AvONMTHGmFPAfKAXII6fImJ764oDR9z1ApTrHD8Tz7AJq/hq5QHu7FSTKbe3pmzRYI/H0a52GbqFlefjRXs4mcEsdG8wxvDlikgGfbKS1FTDt3e3445OtbRDWvksdyaOKsChdPejHI9dRkSqAzWBPzLb1xiTBNwD/IVNGA2BSVc45l0iEiEiEdHR0bl5HSqX1h2Ioe8Hf7L1yBnGDW3O030a5mreQW49eV0YcUkpfPDHHq/FkOZMfBL3fbOB5+dspWPdsvxyfydaVCvl7bCUypSvlFUfAsw0xmTa8CwigdjE0RyojG2qejKjbY0x440x4caY8HLlsjejVrmGMYYpqw4wZPwqQgL9mXVve/o1reztsKhTvhg3t6rK16sOsC/6nNfi2Hokln4f/MlvW4/yRO8wJt4afnHGtlK+zJ2J4zCQvtcz1PFYRobwTzNVZvs2AzDG7DV2HPG3QHtXBaxcwxjDyr0nufOrdTw7ewsd6pTlp/s60qBScW+HdtGD3esSFODHG7/t9Pi5jTFMXX2AGz9eQXxSKtPvasvdnWtrB7jKM9w5HHctUFdEamI/9IcAwy7dSETCgFLAynQPzwP+KyJp1+w9sFcWIUBDESlnjIkGrgW2u+8lqOw4G5/EDxsOM2XlAXYfP0eJQoE80qMe93Spg7+PfSiWLxbC3Z1r8878XURExhDuofkR5xKSeWrWX8zZdISr65Xj3ZuaUsYLfT1K5YbbEocxJllE7sMmAX9gsjFmq4i8BEQYY+Y4Nh0CTDfpZiIaY2JE5GVs8gF4yRgTAyAiLwJLRSQJOACMctdrUM7ZcfQMU1Ye4IcNh4lLTKFxlRK8MagJ/ZpW9ulKqnd0qsnXqw7w6q/bmXVPe7d3Ru84eoZ7v15P5MnzPNKjHvd2qaNXGSpP0pnjKkcSk1P5betRvk5XGK9f08qMaFvdo0Nsc2vG2oM8/v1ffDSsBX2aVHLLOYwxfBcRxbM/bqF4oUDGDWlOu9q6EJLyfTpzXLlERqW4n7ou7F+luPOSQS2rMvnPSF7/bQfdG5YnOMC1V0hxick8M3sLs9YfpkOdMrx3c3PKFdOmKZW3aeLwYfdP28C5hGTeHNTEq+3gqamG5XtPMGXlARZst4v/dAsrz/C21bn6Cov/5BX+fsKT14Ux6vO1fL3qILd3rOmyY+84eoax32xgT/Q5Huxel7HX1PW5vh6lckITh4/acjiWOZvs3MZ+Hy7n0+EtaRxawqMxxF5IYua6KKauOsA+x3KjYzrXZlhr55YbzSs61ytHxzpl+eCP3QxqEZrr2donzyXwjmPt79JFgpgyug0d62plHJV/+Mo8DnWJCcv2UTQ4gG/uaAPAwE9X8F3EoSz2cp35247R7e3FvPzzNkoWDuS9m5ux8slreLxXWL5KGgAi9qoj9kISHy3O+aTAhOQUPluyly5vLmb62kOMaFud3x/qrElD5Tt6xeGDjpy+wM+b/2ZU+xq0r1OWOfd1YOy0DTw6czObo2J5tm9Dt5Ufj72QxEs/beP79VE0rFScSSNb5anO7pxqVLkEA5qH8sXySEa0rZ6t5GiM4bctR/nf3B0cjIlzau1vpfIyveLwQV+siATgtg41AChTNJivRrdmzNW1mLLqAMMmrOL4mXiXn3fprmh6vbeU2RsPc3+3usz+T4cCkTTSPNKzHiLw1u/OTwr8KyqWm8ev4p6p6wkJ9OOr0a35/LbWmjRUvqZXHD7mbHwS01YfpPdVFQkt9c+33gB/P568rgGNQ0vw2MzN9P3gTz4Z3oKW1XM/ce18QjL//XU7U1cfpE75ovwwoiVNQgtOwkhTqUQh7uhUk48W7eX2jjUzfQ+OnYnnjd92MmtDFKULB/HqjVfleu1vpfIK/S33MTPWHuJsQjJ3dqqV4fN9m1Tmh3s7UDjInyHjVzFlZSS5mYuzet9Jer+/jG/WHOSuq2vx89iOBTJppLm7c23KFAni1V+2Z/i+XkhM4f0Fu+ny5mJ+2nSEu66uxaJHu3BLm+qaNFSBkeVvuohcLyL6F+EBySmpfL48ktY1SmfaRFS/YjF+vK8jneqW49kft/LozM3ZXpgoPimFV37expAJqwD4dkw7nrqugU/P9PaEYiGBPNi9Lqv3x7Bg+/GLj6emGmatj6LrW4t5d/y0YhoAACAASURBVMEuuoaVY8HDnXmydwOKh+iaGapgcSYh3AzsFpE3HHWllJvM3XKUw6cvcEenrOcSlCgUyMRbw3mgW11mroti8KcrOXz6glPn2XjoNH3GLWPin/sZ3qY6cx/opGtZpzOkdTVqlS3Ca3O3k5ySSkRkDDd+vJyHv91E+eLBfDumHR/f0pJqZfLX6DKlnOVUyRERKY5daOk2wACfA9OMMWfdG55r5IWSI8YYbvhoOWfik1n4cOdsTapbsO0YD83YSGCAHx8ObU77OhkP/0xMTmXcwt18smQv5YsF88agJnSqqyXnMzJv61HGTFnHVVWKs+XwGSoWD+GxXvV17W9VoFyp5IhTTVDGmDPATOwqfpWAG4H1IjLWpVEWYGsjT7EpKpbRHWtm+4Ope8MK/HhfB8oUCWL4pNVMWLrvsvb5bUfO0P+j5Xy4aA83Nq/CvIeu1qSRiR4NK9C2Vmn2Hj/Pg93r8scjuva3UmmyHFUlIv2wVxp1gK+A1saY4yJSGNiGXStc5dKEZfsoVTiQQS1Cc7R/rXJF+eE/HXhs5iZe/XU7mw/H8vrAxgT5+/HZ0n28t2AXJQoFMeHWcK5tWMHF0ec/IsLno1qTmJyq634rdQlnhuMOBN41xixN/6AxJk5EbndPWAXL/hPnWbD9GPd1rUOhoJx3ThcNDuCjYS34bOk+3vhtB7uOniUkyJ9Nh07Tp0klXu5/FaXzYCFCbykU5J+r/w+l8itnEscLwN9pd0SkEFDBGBNpjFnorsAKkkl/7iPQz48R7arn+lgiwt2da9OocnHGTtsAwAdDm3O9DyzZqpTKH5xJHN/x7+VZUxyPtXJLRAXMqfOJzFwXxQ3NK1O+WIjLjtupbjkW/V8X/ES0qUUp5VLOJI4AY0xi2h1jTKKIaHuHi3y96gDxSanccYUJf7mRF9fHUEr5PmdGVUU7OsgBEJH+wAn3hVRwxCel8OXKA3SuV456FbS2kVIqb3DmiuNuYKqIfAgIcAi41a1RFRBzNh7hxLmEK5YXUUopX5Rl4jDG7AXaikhRx/1zbo/KR2w9EkvJwkFUKVnI5cc2xjDxz32EVSxGhzq6/rRSKu9wqjquiPQBGgEhInYClDHmJTfG5XWJyanc9dU6ggL8+HZMO5evE71kVzS7jp3j7cFNSXtPlVIqL3CmyOGn2HpVY7FNVYOB3I8b9XFBAX6MG9qMo7HxjJi0mtNxiVnvlA0Tl+2nQvFgHSarlMpznOkcb2+MuRU4ZYx5EWgH1HNvWL6hZfXSTLg1nH3R5xn5+VrOJSS75Ljbjpzhzz0nGNm+httW8lNKKXdx5lMrbam5OBGpDCRh61UVCB3rluXDYc3ZcjiWO75cm+3y5RmZ+Oc+Cgf5c0vrfH/hppTKh5xJHD+JSEngTWA9EAl8486gfE2PRhV556amrN4fwz1fryMxOTXHxzp2Jp6fNh3hpvCqOjFPKZUnZZo4HAs4LTTGnDbGfI/t2wgzxjznkeh8SP9mVXj1hsYs2hnNQzM2kpKas1X3vlgRSUqqYXSHrNfcUEopX5TpqCpjTKqIfAQ0d9xPABI8EZgvGtamGucTknn11+0UDvLn9YFNslVm+3xCMlNXHaBno4q6CJBSKs9yZjjuQhEZCMwyuVncOp+48+panE1IZtzC3RQJDuD56xs6PZz2u4hDnIlPdkt5EaWU8hRnEscY4GEgWUTisUNyjTGmuFsj82EPda/L+YRkJv25n2IhAfxfj/pZ7pOSapi8PJIW1UrSsnopD0SplFLukWXnuDGmmDHGzxgTZIwp7rjvVNIQkV4islNE9ojIExk8/66IbHT87BKR0+meGykiux0/I9M9HiQi4x3b73BcDXmUiPBMnwYMaVWVD/7Yw6dL9ma5z+9bj3IwJk7Liyil8jxnVgC8OqPHL13YKYP9/IGPgGuBKGCtiMwxxmxLd4yH0m0/FkdfioiUBp4HwrFrnK9z7HsKeBo4boyp5+i8L53Va3AHEeHVGxtzPjGF1+buoGhwAMPbXnl47YRl+6hWujA9GlX0YJRKKeV6zjRVPZrudgjQGlgHXJPFfq2BPcaYfQAiMh3oj11uNiNDsckCoCcw3xgT49h3PtALmAaMBsLAdt7jxUq9/n7COzc15UJiMs/+uIUiwf7c2PzypV/XHTjF+oOneeH6hvjrmtVKqTzOmaaq69P9XAtcBZxy4thVsJV000Q5HruMiFQHagJ/ZLavYz4JwMsisl5EvhORDBfQFpG7RCRCRCKio6OdCDdnAv39+HBYC9rVKsMj323mty1HL9tm4rJ9FA8JYHB4VbfFoZRSnpKTehdRQAMXxzEEmGmMyWpadgAQCqwwxrQAVgJvZbShMWa8MSbcGBNerlw510Z7iZBAfybcGk6T0BLcP20DS3f9k6gOnoxj3taj3NK2OkWCnaopqZRSPs2ZIocfiMg4x8+HwDLsDPKsHAbSf8UOdTyWkSHYZqis9j0JxAGzHI9/B7RwIha3KxIcwBejWlO7fFHumhLB2sgYACYv34+/nzCqfQ3vBqiUUi7izBVHBLZPYx32G/7jxpjhTuy3FqgrIjUdS80OAeZcupGIhAGlHMdOMw/oISKlRKQU0AOY55hH8hPQxbFdN67cZ+JxJQoHMuX21lQuWYjRn6/lz90n+DbiENc3rUyF4q5bT1wppbzJmbaTmUB8WjOSiPiLSGFjTFxmOxljkkXkPmwS8AcmG2O2ishLQIQxJi2JDAGmp59caIyJEZGXsckH4KW0jnLgcWCKiLwHRAO3OfdSPaNs0WC+vr0Ngz9dyYjJqzEG7uioQ3CVUvmHZDUZXERWAd3TVv5zrAT4uzGmvQfic4nw8HATERHh0XMeOHmewZ+u5KoqJZg8qpVHz62UUq4gIuuMMeGXPu7MFUdI+uVijTHnREQLLWWhepkiLHm0K7q4n1Iqv3Gmj+O8iFzsgBaRlsAF94WUfxQK8ick0N/bYSillEs5c8XxIPCdiBzB1qmqiF1KVimlVAGUZeIwxqx1jHxKq+S30xiT5N6wlFJK+Spn5nH8ByhijNlijNkCFBWRe90fmlJKKV/kTB/HncaYi1VrHYUG73RfSEoppXyZM4nDX9KtVOSoehvkvpCUUkr5Mmc6x38DZojIZ477Y4C57gtJKaWUL3MmcTwO3AXc7bi/GTuySimlVAHkTFn1VGA1EIldY+MaYLt7w1JKKeWrrnjFISL1sIsrDcUuljQDwBjT1TOhKaWU8kWZNVXtwJZQ72uM2QMgIg9lsr1SSqkCILOmqgHA38AiEZkgIt2wM8eVUkoVYFdMHMaY2caYIdj1vRdhS4+UF5FPRKSHpwJUSinlW5zpHD9vjPnGGHM9diW+DdiRVkoppQqgbK05bow55VjLu5u7AlJKKeXbspU4lFJKKU0cSimlskUTh1JKqWzRxKGUUipbNHEopZTKFmeKHCqlcuLsUTi0JnfHKBcG5eq5Jh6lXEQTh1Lu8u1IOLQqd8fwD4YhU6Huta6JSSkX0MShlDscXG2TRpenIKxPzo6RmgRz7ofpw+Cmr6B+b9fGqFQOaeJQyh1WjINCpaD9fRBUJOfHGTkHptwIM0bA4C+gQV+XhahUTmnnuFKudmI37PgFWt2Zu6QBNvmMmA2VmsJ3I2HrbNfEqFQuaOJQytVWfAABwdD6Ltccr1BJGPEDVGkJM0fDlu9dc1ylckgTh1KudO44bJoOTYdC0XKuO25IcRj+PVRtA9/fAZu/dd2xlcomTRxKudLqzyAlEdqPdf2xg4vB8JlQvQPMugs2fuP6cyjlBE0cSrlKwjlYO9F2YJep7Z5zBBWBYd9Czath9r2wfop7zqNUJtyaOESkl4jsFJE9IvJEBs+/KyIbHT+7ROR0uudGishux8/IDPadIyJb3Bm/Utmy4WuIPw3tH3DveYIKw7AZULsrzLkPIj537/mUuoTbhuOKiD/wEXAtEAWsFZE5xphtadsYYx5Kt/1YoLnjdmngeSAcMMA6x76nHM8PAM65K3alsi0lGVZ+BNXaQdVW7j9fYCEYMg2+HQE/PwipydD6TvefVynce8XRGthjjNlnjEkEpgP9M9l+KDDNcbsnMN8YE+NIFvOBXgAiUhR4GHjFbZErlV3bZkPsQWh/v+fOGRgCN38N9XrDr4/Aqk89d25VoLkzcVQBDqW7H+V47DIiUh2oCfzhxL4vA28DcZmdXETuEpEIEYmIjo7OfvQq91JTISne21G4nzGw/H0oWw/q9fLsuQOC7azysL7w2+Ow4kPPnl8VSL4yc3wIMNMYk5LZRiLSDKhtjHlIRGpktq0xZjwwHiA8PNy4KE6VHTNugZ2/QuEyULwyFK/i+Df97SpQrBIEF/V2tDm3fwkc3Qz9PgA/L4w3CQiys8q/vx1+f9qWKun4UJa7KZVT7kwch4Gq6e6HOh7LyBDgP5fs2+WSfRcD7YBwEYnExl5eRBYbY9Jvq3zBid02adTrZRPDmSMQe9hWi70Qc/n2ISWgWOXLE0uNju4boeQqy9+HohWgyc3ei8E/EAZOBr+7YMELts/j6ke9F48viI+FmH1Qubm3I8l33Jk41gJ1RaQmNhEMAYZdupGIhAGlgJXpHp4H/FdESjnu9wCeNMbEAJ849qsB/KxJw0etnQR+gXD9OChW4d/PJV2wieTiz+F/3z62Bc4ds9uWrQf/WQMinn8Nzjj6F+z9A7o9Z5uNvMk/AG4cD34B8McrkJoCnR/33ffO3b6/E3bPg9rXwLUvQcXG3o4o33Bb4jDGJIvIfdgk4A9MNsZsFZGXgAhjzBzHpkOA6cYYk27fGBF5GZt8AF5yJA2VFySet5PTGva7PGmAHRFUpnbmVxLJibD6U5j/LBzfDhUaui/e3FjxAQQWgfDR3o7E8g+AGz6xyWPx/wCBLo97OyrP27PAJo16veHgSvi0EzQdAl2fhpJVs95fZcqtfRzGmF+BXy957LlL7r9whX0nA5MzOXYkcFWug1Su99d3kBBri/zlVECQbfqZ/xxs+9E3E0dslK0b1fouW4zQV/j5Q78PISUJlrxmE3j5Bt6OynNSkuG3p6BUTbjpS0iKgz/ftaPOtsyCNmOg08O+9X+Wx+jMceVaxtjZ0+UbQbW2uTtWsQpQvb1NHL5o1Sf29ba9x9uRXM7PD3q/DkFFYeFL3o7GsyImw4md0OMV23xYqJRtqhq7Dq4aaK8S329mR6AlJ3g72jxJE4dyrUNrbLt/6ztc07beoB9Eb4foXbk/litdOA3rvrAfRCWreTuajBUuDR0ftIMUDqzMevv8IC4GFv/XlmS5dAGtklXhxk/g7mW20vDvT8OH4bZgZGqqd+J1p8Q4OPO3Ww6ticMdUpJtGYi3w2BcC7uGgikgI4LXToDg4tD4Jtccr8H19t/tPnbVETEZEs9BBw9O+MuJNvdA0Yqw4PmC8Tu45HU7mqrn/678xaViYxgxy5aqDykBs+6E8Z1h32KPhup2qz6GD1rYQScuponDlYyBHb/CJ+1sGYgSVcE/yC7AM7E7HFjh7Qjd61y0TZJNh7puXkaJKhDa2reaq5ITbMd9ra6+P1InqDB0fRIOrbZXHvlZ9E5YMwFajISKTnR/1r4G7loKAybYK8iv+sPXA+FoLkvgJV2Ak3th/zI4vD53x8qp8yftMPFaXe2wdhfzlQmAeV9UBPz+LBxcAWXqwM1T7aWySbUjjBa9Cp/3hvrXQfcXoFx9b0fseuu/tJPPWt3u2uM27Ae/P2PH5Jeu5dpj58Tmb+1w4RvzSImPZsNte/6CF6FuTzvyKj+a97StHtz1aef38fODJjfZJtG1E2DpW/BpR/vl55qnoUTov7dPOJvBEPJLhpann6ck/nDPcs8PTlj2tr0i7vZc1tvmgJgCcPkaHh5uIiIi3HPwk3th4Yv2G3GR8tDlCWhxq52QlV5inL10/PM9SDpvt+nyJBSr6J64PC01Bd5rAmVqwcifXHvsUwfg/SbQ/UXbZu9NqanwcRvb6TpmWd6ZI7Ftji2I2O8D+7uX3+yeD1MH2Q7x3KyFcuEULHvHrqsCEHadbfpKSwwJZy7fp3DZjCsjFClrZ/NXbmGbxTz1u3LqgO27aXIz9M9dCRoRWWeMCb/scU0cOXQuGpa+Ydu6/YNtW3e7+7Juojl/Apa+aUce+QfZX/L2Y+0iPXnZjl9g+jBbN6lhZrUsc2h8F0DgrkWuP3Z27JwL04bAgInQZLB3Y8kOY2xz6ZkjcP96O5cmv0hJgk/a29ny9662Q7lz6/RB+ONViFxmqwJcWiYnLUEUq2SLTV7Jyo9h3pN2DZV6PXMflzNmjbFFN8eut029uXClxJFPr1ndKPG848rhfTs+vOVI6PxExhPdMlKkrB0m2fou+ONl25kXMdnO8G056vIrlbxi7UT7R1S/T9bb5kTD/raUxumD3h3FtHyc7btqdIP3YsgJEbj2Rfiij/027e0rN1eKmAwndtky865IGmB/xwZ8lvvjtL7TxjfvKdvf4Kr4ruToX7B5BnR4INdJIzPaOe6slGRY96UdJfXHK1CrM9y7Cvq+63zSSK9MbVuY7o6FtqzGr4/Ax21tk0Jeuwo8sceW3Wh5m/vazxv0s/9ud3EzWHZERdg+rLb35s0EX6Oj7eP48x07bDU/iIuBRf+FWl2gfm9vR3M5/0Do+Sqc3GO/XLnbghftSDE3fzHQxJEVY2Dnb/BpB/jpfvtNZPQ8GDIVytXL/fFDw2HULzB0uu1I+3YETOoBB1fl/tieEjHZlrhoedlCja5TpjZUaOzd0VXL37d/lHm5j6D78xB/xs6kzg8Wv2b7HXr+13f7m+r2gNrd7Cz+8yfdd579y2DPfI/MitfEkZmodfBFX5h2s21HvWkK3P577mdEX0rEflu6Z4UtCnj6IEzuCdNvgVORrj2XqyXGwcav7RWBuzv6G/a3w0rdMC49Syf32qud8Nvzdgn4Co3siKHVn9mSKe4UFwPHtrrv+Md32G/xLW+zr8tXidjElnDOTk50B2PsXJ3iVWwzuJtp4riS1BT4frQtXXDdW/Cf1XZYqDu/1fg7vrXfvx66PmMnJH070rebrrbMtKNOWt3h/nOldbpv/9n957rUyg9ts0ObMZ4/t6t1fcr+u+h/7jvH+RMw6Vo7tNVdTTS/P21LqqS9Hl9WPswOU4+YDMe2Zb19dm2fA4fX2ffCAwMfNHFciZ+/XZbz/g22g8uTbdpBRaDzo7Yz8++N9hfCFxljJ1yVb2hrSrlbuXpQLszzzVXnou1cnKZD8sfw6ZJV7e/0pm9s5WFXSzhrh8bGRkG19vDL/9lila4s67Hrd1sBt8vjdsBJXtDlSVtVYd6Trv0ymJJk65GVa2CvJj1AE0dmKjb27jDZJjdDUDH74eyLoiLsynetbvdc+3LD/nBgOZw77pnzgZ0YlhwP7XIxP8DXdPo/9xRATE6AGcPh78128MetP9rmveXv2zkNrlhKOCXJjlIqUyd3FZg9rXBpmzz2LYZdv7nuuBum2M737s/bL7weoInDlwUXs99yt85yb6daTq2dYBObJ1e+a9gfMLDDQ81Viedt4q5/nWsGQ/gKdxRATE2BWXfZD8b+H9p+O/8A6PO2rU67dRZMuSH3I7rWToSTu6HHq+4f3upqrW63oyjnPW3XnMmtxPN2gEC1dh5d714Th69rdQekJMKGr7wdyb+dPwFbf7CJzZNXZeUbQunanmuu2jDVlpBo7+PFDHPClQUQjYG5j9mJZz1egWbpFvsUsfMKBk22za6TeuR80EdcjF2gqlZXz02ocyX/QNtRHrMX1ozP/fFWfWzL33R/0aOjyjRx+LryYVCjk+1US03xdjT/WP+VTWie6BRPT8Redexf5v65CCnJtlM8tJXrR9L5AlcWQFz8mr0S6PDAlUt+XDXQNl2dj7az2HPSd7fov7YPpVcm1W99Xd1roU53WPKG/QKWU+dP2onIYX2hWhvXxecETRx5Qas77BDd3fO9HYmVmmLLxtfoZBObpzXsDybFljlxp+1z4PQBe7WRVz+kstJsOJSpayeOpSTn7BhrJtg5Cs2H22++maneHu5YAIGF7VD3HdlIWMe32y9Q4aPz/oqGPf9rixAuejXnx1j2tq1756ZChpnRxJEXhPWx5Tw8MfPUGbt/h9iDnr/aSFOpKZSs7t7mqpRk+42wdO3LFwTKT/wDbKfqiZ12lFV2/TUTfn3Ulprp+75zCbZsXZs8yoXBjFucG/xhjO0QDy4KXfLA8NuslKtv/37WfZGzuS6nDtg+xma3eKXStiaOvMA/0Nax2rPAlhb3tjUTbCLz1geqiJ1Ts2+xXUfBHdZ9blcevPZFj41U8ZqwvrY5btH/7IROZ+1ZCD/cba8iBk3KXrmZouVh1M+2Q/fXR2zZ/MyG6+7+3Za16fwEFCnj/Hl8WZcn7PDc33IwPHfRf0H87CgtL9DEkVe0GGk/wCImezeOk3th70LvF2RseINd+8OVwxrTXDhl/zBrdLIfqvmdiF0j5uwRWONkYb+odTBjhL1qGDotZ5POgorYuVKt7rTrgM+8LePhusmJjuG3de38k/yicGk7YW//Elt12VlphQzb3O3WQoaZ0cSRVxSvZD/ENnxtVxjzlrS6VC3cWJfKGVVaQvFQ9zRXLXnDJo+83AGbXRcLIL6b9aCD6J12gl/RcjD8e1u/K6f8/OG6N+1IrG2z7Sp8l55/7UQ7T6Hnq3mzuGRmwkdD2fp2FnxygnP7eKiQYWY0ceQlre6wH2hbvvfO+RPjbOIK62sTmTelNVftWWiL9rlK9C47TLLFrb6/LKyrOVMAMTYKpgywXx5G/JCzytCXErEjsQZ/AUc22FIlaU2y50/ajvfa3WyxwPzm4vDcfc4Nz92/1FHI8P/cXsgwM5o48pIaHW3TgLc6ybd8D/Gnfae5oEE/SEmw7d+u8vszEFAIrnnWdcfMK7IqgBgXY5NGwhl7peHqZXwb3Qgj50DcSZh4ra1MsNhRHNCXq9/mVt3uNikuecOWt7kSY2D+8/ZK2wOFDDOjiSMvEbFXHUc22DZmTzLGjuIo1wCqd/Dsua+kahs7gc1VzVV7FsDuebZOWNFyrjlmXnOlAogJ52DqYDtxb+h0qNTEPeev1hZuX2BHT33R1zaNtrrdO8O+PanHq3ZhuMyG5277EY6sdxQyzGTVQQ/QxJHXNLnZ1hjy9FXH4XXw9ybP1qXKip8fNLjezm9JPJ+7Y6Uk2zIQpWraTseCKn0BxLQqrsmJ8O2t9kNr8OdQw81fHMrWscmjQiMoVNprI4c8qlw9O0hg/ZdwdMvlz6ck2RVDyzWw1Rq8TBNHXhNS3CaPLd97tn7V2ok2YXmyLpUzGvaD5Au5nxy57nOI3mE7aQOCXRNbXtXp/2wNsoUv2SGys++xI+muH+e5IdhFy9m1b+7fYEcfFQSdH7Od3hlVz/VCIcPMaOLIi1rdYdv2N37tmfOdPwlbZtlvOiHFPXNOZ1VrD4XL2lneORUXY5sIal6dvyf7Oatwaej4AOyaaxcx2zLTzghvMcKzcfj5+97vmzsVLg1dn7Yd4OlLwHipkGFmNHHkRRUa2n6GtZM8U79qwxSbqLw1Uzwz/gHQoC/smpfzYcpL3rCLUfUsQMNvs5JWAHH379DuPluDSrlfy9vsAJh56YbneqmQYWbcmjhEpJeI7BSRPSLyRAbPvysiGx0/u0TkdLrnRorIbsfPSMdjhUXkFxHZISJbReQ1d8bv01rdYeso7Vno3vOkpkDEJKje0XfrAzXsb+v+7P0j+/tG77Kd/i1GQsWrXB9bXhVUGAZOsBMDr33ZZz6w8j3/ADuC7NR+O7rNi4UMM5ONGgHZIyL+wEfAtUAUsFZE5hhjLq6baIx5KN32Y4HmjtulgeeBcMAA60RkDpAAvGWMWSQiQcBCEeltjMnGtMt8IqwvFK1gP/TquXF8+54FtsDitS5e8MeVanSCkJKwbU72m5p+f9oW3Ov6tHtiy8tqXm1/lGfV6WYnYy59E45t8Vohw8y484qjNbDHGLPPGJMITAf6Z7L9UGCa43ZPYL4xJsYYcwqYD/QyxsQZYxYBOI65Hgh12yvwZQFBtuzH7vkQs99951kzwTZZ+HLpDf9AG9/Ouc7PvgXYvcA2xXR+rOAOv1W+qadjeO7mGbbqsBcKGWbGnYmjCnAo3f0ox2OXEZHqQE0gra0hy31FpCRwPZBhW42I3CUiESISER2dyaSavKzlKFvozF31q2L22SsOb9elckbD/pAQC/uWOLd92vKjpWtB6zHujU2p7Cpb1/YtBZfwyeHIvtI5PgSYaYxxqqdXRAKwVyfjjDEZlos1xow3xoQbY8LLlcun3yaLV7ZNMxumuKd+1erxNjG19HJdKmfU6mwrjW53cjJgxGRbSjwvLj+qCobuL8DDW+3fuY9xZ+I4DFRNdz/U8VhGhvBPM5Uz+44Hdhtj3nNBnHlb6ztt/aqtP7j2uKvHw+pPbAkKH/zFvUxAsF3jescv9moiM3Extvptzc52H6V8kYhnl2XOBncmjrVAXRGp6ejIHgJcNtheRMKAUsDKdA/PA3qISCkRKQX0cDyGiLwClAC8VxrSl9ToZKtrOrMYjrNWfgxz0xbnyaTgna9p2N8m0chlmW+3+DVbb6kgVb9VyoXcljiMMcnAfdgP/O3At8aYrSLykoj0S7fpEGC6Mf9MlTTGxAAvY5PPWuAlY0yMiIQCTwMNgfWOYbw+OLnAgy7Wr1qfszWcL7V8nJ252qAf3PRl3mrGqX2Nnd2eWe2q6J12FnzLUbakhVIq28Rkd+WpPCg8PNxERER4Owz3iY+FtxtAoxvgho9zfpxl78DCF6HRABgw3vc7xDPy3W125u0juzIuzfD1QDi0Fu5fD0XKej4+pfIQEVlnjAm/9HFf6RxXuRFSApo66ldltQjPlSx5wyaNxoNhwIS8mTTANlfFnYADKy5/bvd8IinSIAAACShJREFUO0qs82OaNJTKBU0c+UWrOyA53i60lB3G2I7iRa/ajvAbP8ve2tG+pu61dj2NS5urLg6/re31tQyUyus0ceQXFRrZgn8Rk2xFU2cYY0s1L3ndTjLq/5FPVN7MlaAiNnls/+nf78PaSXBil51YlZf6bZTyQZo48pNWt9uFdvY6Ub/KGFjwPCx729Zpuv6DvJ800jTsD+eOQtQaez8uBhb/D2p18ZnqokrlZZo48pMG/aBI+ayH5hpjq28ufx/Cb4e+79lFkfKLuj3AP/if5qrF/7PDb7X6rVIukY8+LZStXzXS1l86FZnxNsbA3Mdh1Ud2pbs+b+evpAF2DYc63WzRw+PbbTNVy9tsOXqlVK7ls08MRcvbrly/KjUVfvk/WPOZrYPT67X8+w28QT84EwXTb7FzO7T6rVIuo4kjvylRxZbRWD8FkuL/eTw1FX5+0Haed3jALpGaX5MGQP1e4BcIMXuhy+NQpIy3I1Iq39DEkR+1vhMuxPxTvyo1BeaMhfVf2vWkfWglMbcpVArq9bSrqbW609vRKJWv5OEB++qKanaGMnVtaY0mN8Hse2HzdOj8BHR5Iv8njTQDJ9qkqcNvlXIpveLIj9LqVx2OgK/626TR9Wno+mTBSRoAgYUguKi3o1Aq39HEkV81HWKXRI1cBt2et2U2lFLKBbSpKr8qVBKuHwepydBsqLejUUrlI5o48rMmg70dgVIqH9KmKqWUUtmiiUMppVS2aOJQSimVLZo4lFJKZYsmDqWUUtmiiUMppVS2aOJQSimVLZo4lFJKZYsYY7wdg9uJSDRwIIe7lwVOuDAcV9P4ckfjyx2NL3d8Pb7qxphylz5YIBJHbohIhDEm3NtxXInGlzsaX+5ofLnj6/FdiTZVKaWUyhZNHEoppbJFE0fWxns7gCxofLmj8eWOxpc7vh5fhrSPQymlVLboFYdSSqls0cShlFIqWzRxOIhILxHZKSJ7ROSJDJ4PFpEZjudXi0gND8ZWVUQWicg2EdkqIg9ksE0XEYkVkY2On+c8FZ/j/JEi8pfj3BEZPC8iMs7x/m0WkRYejK1+uvdlo4icEZEHL9nGo++fiEwWkeMisiXdY6VFZL6I7Hb8W+oK+450bLNbREZ6ML43RWSH4//vBxEpeYV9M/1dcGN8L4jI4XT/h9ddYd9M/9bdGN+MdLFFisjGK+zr9vcv14wxBf4H8Af2ArWAIGAT0PCSbe4FPnXcHgLM8GB8lYAWjtvFgF0ZxNcF+NmL72EkUDaT568D5gICtAVWe/H/+ih2YpPX3j/gaqAFsCXdY28ATzhuPwG8nsF+pYF9jn9LOW6X8lB8PYAAx+3XM4rPmd8FN8b3AvCIE///mf6tuyu+S55/G3jOW+9fbn/0isNqDewxxuwzxiQC04H+l2zTH/jScXsm0E1ExBPBGWP+Nsasd9w+C2wHqnji3C7UH/jKWKuAkiJSyQtxdAP2GmNyWknAJYwxS4GYSx5O/zv2JXBDBrv2BOYbY2KMMaeA+UAvT8RnjPndGJPsuLsKCHX1eZ11hffPGc78redaZvE5PjduAqa5+ryeoonDqgIcSnc/iss/mC9u4/jjiQXKeCS6dBxNZM2B1Rk83U7+v727C5miiuM4/v2hRg8aYglPhYVZXkklIhFiXURIRgi9gIpQqRBKRt1UF95FV11EaBJkr4QXEb15YSUpRFBhJGpFkQ/RRfL4qEGKFGJP/y7OWZjWHd2hnZmtfh8YdubMeXbOnucM/51zZs9IByV9IGlBowWDAHZL+krSQz3291PHTVhF+QnbZv0BjEbEeF4/Coz2yDMs9biOdAXZy4XaQp025a60V0q6+oah/m4BJiLicMn+NuuvLw4c/yKSZgBvA49FxKmu3ftJ3S83AluB9xou3tKIWAQsBx6WdGvDx78gSRcBK4C3euxuu/7+JlKfxVDeKy9pM/AHsKMkS1tt4QXgWmAhME7qDhpGqzn/1cbQn0sOHMkR4KrC9pyc1jOPpKnATOCXRkqXjjmNFDR2RMQ73fsj4lREnM7ru4BpkmY3Vb6IOJJfjwHvkroEivqp47otB/ZHxET3jrbrL5vodN/l12M98rRaj5IeBO4C1uTgdo4+2kItImIiIiYj4k9ge8lx266/qcA9wJtledqqvyocOJIvgfmSrsnfSlcBO7vy7AQ6d7DcB+wtO3EGLfeJvgx8FxHPluS5vDPmIukm0v+2kcAmabqkSzrrpEHUb7qy7QTuz3dX3QycLHTLNKX0m16b9VdQbGMPAO/3yPMRsEzSrNwVsyyn1U7SHcATwIqI+K0kTz9toa7yFcfM7i45bj/nep1uB76PiJ977Wyz/ippe3R+WBbSXT8/kO642JzTniKdJAAXk7o4xoB9wLwGy7aU1G1xCDiQlzuBDcCGnGcT8C3pLpEvgCUNlm9ePu7BXIZO/RXLJ2Bbrt+vgcUN/3+nkwLBzEJaa/VHCmDjwFlSP/t60pjZHuAw8DFwac67GHip8LfrcjscA9Y2WL4x0vhApw127jK8Eth1vrbQUPneyG3rECkYXNFdvrx9zrneRPly+mudNlfI23j9/dPFU46YmVkl7qoyM7NKHDjMzKwSBw4zM6vEgcPMzCpx4DAzs0ocOMwGQNJk1wy8A5t1VdLc4iyrZm2b2nYBzP4jfo+IhW0XwqwJvuIwq1F+tsIz+fkK+yRdl9PnStqbJ+TbI+nqnD6an3VxMC9L8ltNkbRd6XksuyWNtPah7H/PgcNsMEa6uqpWFvadjIjrgeeB53LaVuD1iLiBNFnglpy+Bfgk0mSLi0i/HgaYD2yLiAXAr8C9NX8es1L+5bjZAEg6HREzeqT/BNwWET/miSqPRsRlkk6QpsQ4m9PHI2K2pOPAnIg4U3iPuaRncMzP208C0yLi6fo/mdm5fMVhVr8oWa/iTGF9Eo9PWoscOMzqt7Lw+nle/4w0MyvAGuDTvL4H2AggaYqkmU0V0qxf/tZiNhgjkg4Utj+MiM4tubMkHSJdNazOaY8Ar0p6HDgOrM3pjwIvSlpPurLYSJpl1WxoeIzDrEZ5jGNxRJxouyxmg+KuKjMzq8RXHGZmVomvOMzMrBIHDjMzq8SBw8zMKnHgMDOzShw4zMyskr8ARXq+4CpsJpAAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7043383157127233"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    }
  ]
}