Created
December 4, 2023 14:21
-
-
Save georgehc/24ed6def3c903e32ea12eb5796d76e40 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# 95-865: Sentiment Analysis with IMDb Reviews\n", | |
| "\n", | |
| "Author: George H. Chen (georgechen [at symbol] cmu.edu)\n", | |
| "\n", | |
| "Last updated: Dec 4, 2023\n", | |
| "\n", | |
| "This demo shows how to train an LSTM model for sentiment analysis with IMDb reviews. This is a binary classification task: for each review, we classify it as having positive or negative sentiment." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 1, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "%matplotlib inline\n", | |
| "import matplotlib.pyplot as plt\n", | |
| "import numpy as np\n", | |
| "import random\n", | |
| "import os\n", | |
| "\n", | |
| "os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8' # to help make code deterministic\n", | |
| "\n", | |
| "from glob import glob\n", | |
| "\n", | |
| "import torch\n", | |
| "torch.use_deterministic_algorithms(True) # to help make code deterministic\n", | |
| "torch.backends.cudnn.benchmark = False # to help make code deterministic\n", | |
| "import torch.nn as nn\n", | |
| "from torchinfo import summary\n", | |
| "\n", | |
| "np.random.seed(0) # to help make code deterministic\n", | |
| "torch.manual_seed(0) # to help make code deterministic\n", | |
| "random.seed(0) # to help make code deterministic\n", | |
| "\n", | |
| "from UDA_pytorch_utils import UDA_pytorch_classifier_fit, \\\n", | |
| " UDA_plot_train_val_accuracy_vs_epoch, UDA_pytorch_classifier_predict, \\\n", | |
| " UDA_compute_accuracy, UDA_get_rnn_last_time_step_outputs\n", | |
| "\n", | |
| "# these next two lines are needed on my old Intel Mac laptop due to some weird software update issue and also a memory issue\n", | |
| "import os\n", | |
| "os.environ['LOKY_MAX_CPU_COUNT'] = '1'" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## Load the dataset\n", | |
| "\n", | |
| "Here, we downloaded the IMDb dataset from: http://ai.stanford.edu/~amaas/data/sentiment/\n", | |
| "\n", | |
| "We place the file `aclImdb_v1.tar.gz` into `./data/` and uncompress the file within that directory so that after uncompressing, you should have access to the directories `./data/aclImdb/train`, `./data/aclImdb/test`, and other files such as the \"README\" file `./data/aclImdb/README`." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 2, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "train_dataset = []\n", | |
| "\n", | |
| "for filename in sorted(glob('./data/aclImdb/train/pos/*.txt')):\n", | |
| " with open(filename, 'r', encoding='utf-8') as f:\n", | |
| " train_dataset.append((f.read(), 1)) # 1 means `positive` sentiment\n", | |
| "\n", | |
| "for filename in sorted(glob('./data/aclImdb/train/neg/*.txt')):\n", | |
| " with open(filename, 'r', encoding='utf-8') as f:\n", | |
| " train_dataset.append((f.read(), 0)) # 0 means `negative` sentiment" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 3, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "# proper training data points: 20000\n", | |
| "# validation data points: 5000\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "proper_train_size = int(len(train_dataset) * 0.8)\n", | |
| "val_size = len(train_dataset) - proper_train_size\n", | |
| "print('# proper training data points:', proper_train_size)\n", | |
| "print('# validation data points:', val_size)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 4, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "proper_train_dataset, val_dataset = torch.utils.data.random_split(train_dataset,\n", | |
| " [proper_train_size,\n", | |
| " val_size])" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 5, | |
| "metadata": { | |
| "scrolled": true | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "(\"Master cinéaste Alain Resnais likes to work with those actors who are a part of his family.In this film too we see Resnais' family members like Pierre Arditi, Sabine Azema, André Dussolier and Fanny Ardant dealing with serious themes like death,religion,suicide,love and their overall implications on our daily lives.The formal nature of relationship shared by these people is evident as even friends, they address each other using a formal you.In 1984,while making L'amour à mort,Resnais dealt with time,memory and space to unravel the mysteries of a fundamental question of human existence :Is love stronger than death ? It was 16 years ago in 1968 that Resnais made a somewhat similar film Je t'aime Je t'aime which was also about love and memories.Message of this film is loud and clear :true and deep love can even put science to shame as dead lovers regain their lost lives leaving doctors to care for their reputation.L'amour à mort is like a game which is not at all didactic.It is a film in which the musical score is in perfect tandem with its images.This is one of the reasons why this film can easily be grasped.\",\n", | |
| " 1)" | |
| ] | |
| }, | |
| "execution_count": 5, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "proper_train_dataset[0] # this is a tuple of the format (text, label)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "`torchtext` provides a wrapper around spaCy that we could use." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 6, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "from torchtext.data import get_tokenizer" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 7, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "tokenizer_cased = get_tokenizer('spacy', language='en_core_web_sm')" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 8, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "['Master',\n", | |
| " 'cinéaste',\n", | |
| " 'Alain',\n", | |
| " 'Resnais',\n", | |
| " 'likes',\n", | |
| " 'to',\n", | |
| " 'work',\n", | |
| " 'with',\n", | |
| " 'those',\n", | |
| " 'actors',\n", | |
| " 'who',\n", | |
| " 'are',\n", | |
| " 'a',\n", | |
| " 'part',\n", | |
| " 'of',\n", | |
| " 'his',\n", | |
| " 'family',\n", | |
| " '.',\n", | |
| " 'In',\n", | |
| " 'this',\n", | |
| " 'film',\n", | |
| " 'too',\n", | |
| " 'we',\n", | |
| " 'see',\n", | |
| " 'Resnais',\n", | |
| " \"'\",\n", | |
| " 'family',\n", | |
| " 'members',\n", | |
| " 'like',\n", | |
| " 'Pierre',\n", | |
| " 'Arditi',\n", | |
| " ',',\n", | |
| " 'Sabine',\n", | |
| " 'Azema',\n", | |
| " ',',\n", | |
| " 'André',\n", | |
| " 'Dussolier',\n", | |
| " 'and',\n", | |
| " 'Fanny',\n", | |
| " 'Ardant',\n", | |
| " 'dealing',\n", | |
| " 'with',\n", | |
| " 'serious',\n", | |
| " 'themes',\n", | |
| " 'like',\n", | |
| " 'death',\n", | |
| " ',',\n", | |
| " 'religion',\n", | |
| " ',',\n", | |
| " 'suicide',\n", | |
| " ',',\n", | |
| " 'love',\n", | |
| " 'and',\n", | |
| " 'their',\n", | |
| " 'overall',\n", | |
| " 'implications',\n", | |
| " 'on',\n", | |
| " 'our',\n", | |
| " 'daily',\n", | |
| " 'lives',\n", | |
| " '.',\n", | |
| " 'The',\n", | |
| " 'formal',\n", | |
| " 'nature',\n", | |
| " 'of',\n", | |
| " 'relationship',\n", | |
| " 'shared',\n", | |
| " 'by',\n", | |
| " 'these',\n", | |
| " 'people',\n", | |
| " 'is',\n", | |
| " 'evident',\n", | |
| " 'as',\n", | |
| " 'even',\n", | |
| " 'friends',\n", | |
| " ',',\n", | |
| " 'they',\n", | |
| " 'address',\n", | |
| " 'each',\n", | |
| " 'other',\n", | |
| " 'using',\n", | |
| " 'a',\n", | |
| " 'formal',\n", | |
| " 'you',\n", | |
| " '.',\n", | |
| " 'In',\n", | |
| " '1984,while',\n", | |
| " 'making',\n", | |
| " \"L'amour\",\n", | |
| " 'à',\n", | |
| " 'mort',\n", | |
| " ',',\n", | |
| " 'Resnais',\n", | |
| " 'dealt',\n", | |
| " 'with',\n", | |
| " 'time',\n", | |
| " ',',\n", | |
| " 'memory',\n", | |
| " 'and',\n", | |
| " 'space',\n", | |
| " 'to',\n", | |
| " 'unravel',\n", | |
| " 'the',\n", | |
| " 'mysteries',\n", | |
| " 'of',\n", | |
| " 'a',\n", | |
| " 'fundamental',\n", | |
| " 'question',\n", | |
| " 'of',\n", | |
| " 'human',\n", | |
| " 'existence',\n", | |
| " ':',\n", | |
| " 'Is',\n", | |
| " 'love',\n", | |
| " 'stronger',\n", | |
| " 'than',\n", | |
| " 'death',\n", | |
| " '?',\n", | |
| " 'It',\n", | |
| " 'was',\n", | |
| " '16',\n", | |
| " 'years',\n", | |
| " 'ago',\n", | |
| " 'in',\n", | |
| " '1968',\n", | |
| " 'that',\n", | |
| " 'Resnais',\n", | |
| " 'made',\n", | |
| " 'a',\n", | |
| " 'somewhat',\n", | |
| " 'similar',\n", | |
| " 'film',\n", | |
| " 'Je',\n", | |
| " \"t'aime\",\n", | |
| " 'Je',\n", | |
| " \"t'aime\",\n", | |
| " 'which',\n", | |
| " 'was',\n", | |
| " 'also',\n", | |
| " 'about',\n", | |
| " 'love',\n", | |
| " 'and',\n", | |
| " 'memories',\n", | |
| " '.',\n", | |
| " 'Message',\n", | |
| " 'of',\n", | |
| " 'this',\n", | |
| " 'film',\n", | |
| " 'is',\n", | |
| " 'loud',\n", | |
| " 'and',\n", | |
| " 'clear',\n", | |
| " ':',\n", | |
| " 'true',\n", | |
| " 'and',\n", | |
| " 'deep',\n", | |
| " 'love',\n", | |
| " 'can',\n", | |
| " 'even',\n", | |
| " 'put',\n", | |
| " 'science',\n", | |
| " 'to',\n", | |
| " 'shame',\n", | |
| " 'as',\n", | |
| " 'dead',\n", | |
| " 'lovers',\n", | |
| " 'regain',\n", | |
| " 'their',\n", | |
| " 'lost',\n", | |
| " 'lives',\n", | |
| " 'leaving',\n", | |
| " 'doctors',\n", | |
| " 'to',\n", | |
| " 'care',\n", | |
| " 'for',\n", | |
| " 'their',\n", | |
| " 'reputation',\n", | |
| " '.',\n", | |
| " \"L'amour\",\n", | |
| " 'à',\n", | |
| " 'mort',\n", | |
| " 'is',\n", | |
| " 'like',\n", | |
| " 'a',\n", | |
| " 'game',\n", | |
| " 'which',\n", | |
| " 'is',\n", | |
| " 'not',\n", | |
| " 'at',\n", | |
| " 'all',\n", | |
| " 'didactic',\n", | |
| " '.',\n", | |
| " 'It',\n", | |
| " 'is',\n", | |
| " 'a',\n", | |
| " 'film',\n", | |
| " 'in',\n", | |
| " 'which',\n", | |
| " 'the',\n", | |
| " 'musical',\n", | |
| " 'score',\n", | |
| " 'is',\n", | |
| " 'in',\n", | |
| " 'perfect',\n", | |
| " 'tandem',\n", | |
| " 'with',\n", | |
| " 'its',\n", | |
| " 'images',\n", | |
| " '.',\n", | |
| " 'This',\n", | |
| " 'is',\n", | |
| " 'one',\n", | |
| " 'of',\n", | |
| " 'the',\n", | |
| " 'reasons',\n", | |
| " 'why',\n", | |
| " 'this',\n", | |
| " 'film',\n", | |
| " 'can',\n", | |
| " 'easily',\n", | |
| " 'be',\n", | |
| " 'grasped',\n", | |
| " '.']" | |
| ] | |
| }, | |
| "execution_count": 8, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "tokenizer_cased(proper_train_dataset[0][0])" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Note that the above code cell output just consists of tokens (already converted to strings) found by spaCy. You learned how to do precisely this back in the first demo of this course!\n", | |
| "\n", | |
| "You'll notice that the above tokens keeps track of whether words are uppercase or lowercase. We will be using a word embedding that is \"uncased\" so we will convert everything to lowercase with the follow function." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 9, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "def tokenizer(text):\n", | |
| " return [token.lower() for token in tokenizer_cased(text)]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 10, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "proper_train_dataset_as_tokens_without_labels = [tokenizer(text) for text, label in proper_train_dataset]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 11, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "['master',\n", | |
| " 'cinéaste',\n", | |
| " 'alain',\n", | |
| " 'resnais',\n", | |
| " 'likes',\n", | |
| " 'to',\n", | |
| " 'work',\n", | |
| " 'with',\n", | |
| " 'those',\n", | |
| " 'actors',\n", | |
| " 'who',\n", | |
| " 'are',\n", | |
| " 'a',\n", | |
| " 'part',\n", | |
| " 'of',\n", | |
| " 'his',\n", | |
| " 'family',\n", | |
| " '.',\n", | |
| " 'in',\n", | |
| " 'this',\n", | |
| " 'film',\n", | |
| " 'too',\n", | |
| " 'we',\n", | |
| " 'see',\n", | |
| " 'resnais',\n", | |
| " \"'\",\n", | |
| " 'family',\n", | |
| " 'members',\n", | |
| " 'like',\n", | |
| " 'pierre',\n", | |
| " 'arditi',\n", | |
| " ',',\n", | |
| " 'sabine',\n", | |
| " 'azema',\n", | |
| " ',',\n", | |
| " 'andré',\n", | |
| " 'dussolier',\n", | |
| " 'and',\n", | |
| " 'fanny',\n", | |
| " 'ardant',\n", | |
| " 'dealing',\n", | |
| " 'with',\n", | |
| " 'serious',\n", | |
| " 'themes',\n", | |
| " 'like',\n", | |
| " 'death',\n", | |
| " ',',\n", | |
| " 'religion',\n", | |
| " ',',\n", | |
| " 'suicide',\n", | |
| " ',',\n", | |
| " 'love',\n", | |
| " 'and',\n", | |
| " 'their',\n", | |
| " 'overall',\n", | |
| " 'implications',\n", | |
| " 'on',\n", | |
| " 'our',\n", | |
| " 'daily',\n", | |
| " 'lives',\n", | |
| " '.',\n", | |
| " 'the',\n", | |
| " 'formal',\n", | |
| " 'nature',\n", | |
| " 'of',\n", | |
| " 'relationship',\n", | |
| " 'shared',\n", | |
| " 'by',\n", | |
| " 'these',\n", | |
| " 'people',\n", | |
| " 'is',\n", | |
| " 'evident',\n", | |
| " 'as',\n", | |
| " 'even',\n", | |
| " 'friends',\n", | |
| " ',',\n", | |
| " 'they',\n", | |
| " 'address',\n", | |
| " 'each',\n", | |
| " 'other',\n", | |
| " 'using',\n", | |
| " 'a',\n", | |
| " 'formal',\n", | |
| " 'you',\n", | |
| " '.',\n", | |
| " 'in',\n", | |
| " '1984,while',\n", | |
| " 'making',\n", | |
| " \"l'amour\",\n", | |
| " 'à',\n", | |
| " 'mort',\n", | |
| " ',',\n", | |
| " 'resnais',\n", | |
| " 'dealt',\n", | |
| " 'with',\n", | |
| " 'time',\n", | |
| " ',',\n", | |
| " 'memory',\n", | |
| " 'and',\n", | |
| " 'space',\n", | |
| " 'to',\n", | |
| " 'unravel',\n", | |
| " 'the',\n", | |
| " 'mysteries',\n", | |
| " 'of',\n", | |
| " 'a',\n", | |
| " 'fundamental',\n", | |
| " 'question',\n", | |
| " 'of',\n", | |
| " 'human',\n", | |
| " 'existence',\n", | |
| " ':',\n", | |
| " 'is',\n", | |
| " 'love',\n", | |
| " 'stronger',\n", | |
| " 'than',\n", | |
| " 'death',\n", | |
| " '?',\n", | |
| " 'it',\n", | |
| " 'was',\n", | |
| " '16',\n", | |
| " 'years',\n", | |
| " 'ago',\n", | |
| " 'in',\n", | |
| " '1968',\n", | |
| " 'that',\n", | |
| " 'resnais',\n", | |
| " 'made',\n", | |
| " 'a',\n", | |
| " 'somewhat',\n", | |
| " 'similar',\n", | |
| " 'film',\n", | |
| " 'je',\n", | |
| " \"t'aime\",\n", | |
| " 'je',\n", | |
| " \"t'aime\",\n", | |
| " 'which',\n", | |
| " 'was',\n", | |
| " 'also',\n", | |
| " 'about',\n", | |
| " 'love',\n", | |
| " 'and',\n", | |
| " 'memories',\n", | |
| " '.',\n", | |
| " 'message',\n", | |
| " 'of',\n", | |
| " 'this',\n", | |
| " 'film',\n", | |
| " 'is',\n", | |
| " 'loud',\n", | |
| " 'and',\n", | |
| " 'clear',\n", | |
| " ':',\n", | |
| " 'true',\n", | |
| " 'and',\n", | |
| " 'deep',\n", | |
| " 'love',\n", | |
| " 'can',\n", | |
| " 'even',\n", | |
| " 'put',\n", | |
| " 'science',\n", | |
| " 'to',\n", | |
| " 'shame',\n", | |
| " 'as',\n", | |
| " 'dead',\n", | |
| " 'lovers',\n", | |
| " 'regain',\n", | |
| " 'their',\n", | |
| " 'lost',\n", | |
| " 'lives',\n", | |
| " 'leaving',\n", | |
| " 'doctors',\n", | |
| " 'to',\n", | |
| " 'care',\n", | |
| " 'for',\n", | |
| " 'their',\n", | |
| " 'reputation',\n", | |
| " '.',\n", | |
| " \"l'amour\",\n", | |
| " 'à',\n", | |
| " 'mort',\n", | |
| " 'is',\n", | |
| " 'like',\n", | |
| " 'a',\n", | |
| " 'game',\n", | |
| " 'which',\n", | |
| " 'is',\n", | |
| " 'not',\n", | |
| " 'at',\n", | |
| " 'all',\n", | |
| " 'didactic',\n", | |
| " '.',\n", | |
| " 'it',\n", | |
| " 'is',\n", | |
| " 'a',\n", | |
| " 'film',\n", | |
| " 'in',\n", | |
| " 'which',\n", | |
| " 'the',\n", | |
| " 'musical',\n", | |
| " 'score',\n", | |
| " 'is',\n", | |
| " 'in',\n", | |
| " 'perfect',\n", | |
| " 'tandem',\n", | |
| " 'with',\n", | |
| " 'its',\n", | |
| " 'images',\n", | |
| " '.',\n", | |
| " 'this',\n", | |
| " 'is',\n", | |
| " 'one',\n", | |
| " 'of',\n", | |
| " 'the',\n", | |
| " 'reasons',\n", | |
| " 'why',\n", | |
| " 'this',\n", | |
| " 'film',\n", | |
| " 'can',\n", | |
| " 'easily',\n", | |
| " 'be',\n", | |
| " 'grasped',\n", | |
| " '.']" | |
| ] | |
| }, | |
| "execution_count": 11, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "proper_train_dataset_as_tokens_without_labels[0]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "We now build the vocabulary. We already saw how to manually do this with spaCy in the second demo of this course! `torchtext` provides a helper function to do this automatically for us." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 12, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "from torchtext.vocab import build_vocab_from_iterator" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 13, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "vocab = build_vocab_from_iterator(proper_train_dataset_as_tokens_without_labels,\n", | |
| " specials=[\"<unk>\"])" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 14, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "vocab.set_default_index(vocab[\"<unk>\"])" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 15, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "[1259,\n", | |
| " 59266,\n", | |
| " 11261,\n", | |
| " 16475,\n", | |
| " 1225,\n", | |
| " 7,\n", | |
| " 171,\n", | |
| " 20,\n", | |
| " 162,\n", | |
| " 169,\n", | |
| " 43,\n", | |
| " 31,\n", | |
| " 5,\n", | |
| " 186,\n", | |
| " 6,\n", | |
| " 33,\n", | |
| " 233,\n", | |
| " 3,\n", | |
| " 10,\n", | |
| " 12,\n", | |
| " 24,\n", | |
| " 110,\n", | |
| " 80,\n", | |
| " 77,\n", | |
| " 16475,\n", | |
| " 53,\n", | |
| " 233,\n", | |
| " 1068,\n", | |
| " 45,\n", | |
| " 6177,\n", | |
| " 54976,\n", | |
| " 2,\n", | |
| " 36409,\n", | |
| " 55541,\n", | |
| " 2,\n", | |
| " 15191,\n", | |
| " 29297,\n", | |
| " 4,\n", | |
| " 12106,\n", | |
| " 32513,\n", | |
| " 1943,\n", | |
| " 20,\n", | |
| " 614,\n", | |
| " 1318,\n", | |
| " 45,\n", | |
| " 361,\n", | |
| " 2,\n", | |
| " 2310,\n", | |
| " 2,\n", | |
| " 1752,\n", | |
| " 2,\n", | |
| " 131,\n", | |
| " 4,\n", | |
| " 78,\n", | |
| " 579,\n", | |
| " 10754,\n", | |
| " 26,\n", | |
| " 281,\n", | |
| " 3045,\n", | |
| " 474,\n", | |
| " 3,\n", | |
| " 1,\n", | |
| " 11373,\n", | |
| " 887,\n", | |
| " 6,\n", | |
| " 646,\n", | |
| " 5281,\n", | |
| " 40,\n", | |
| " 149,\n", | |
| " 94,\n", | |
| " 8,\n", | |
| " 3721,\n", | |
| " 19,\n", | |
| " 67,\n", | |
| " 379,\n", | |
| " 2,\n", | |
| " 39,\n", | |
| " 5559,\n", | |
| " 267,\n", | |
| " 96,\n", | |
| " 787,\n", | |
| " 5,\n", | |
| " 11373,\n", | |
| " 25,\n", | |
| " 3,\n", | |
| " 10,\n", | |
| " 52626,\n", | |
| " 247,\n", | |
| " 26695,\n", | |
| " 13321,\n", | |
| " 35429,\n", | |
| " 2,\n", | |
| " 16475,\n", | |
| " 3345,\n", | |
| " 20,\n", | |
| " 68,\n", | |
| " 2,\n", | |
| " 1897,\n", | |
| " 4,\n", | |
| " 844,\n", | |
| " 7,\n", | |
| " 8888,\n", | |
| " 1,\n", | |
| " 4146,\n", | |
| " 6,\n", | |
| " 5,\n", | |
| " 8964,\n", | |
| " 905,\n", | |
| " 6,\n", | |
| " 415,\n", | |
| " 2081,\n", | |
| " 93,\n", | |
| " 8,\n", | |
| " 131,\n", | |
| " 3535,\n", | |
| " 85,\n", | |
| " 361,\n", | |
| " 60,\n", | |
| " 9,\n", | |
| " 18,\n", | |
| " 3207,\n", | |
| " 168,\n", | |
| " 617,\n", | |
| " 10,\n", | |
| " 5174,\n", | |
| " 13,\n", | |
| " 16475,\n", | |
| " 104,\n", | |
| " 5,\n", | |
| " 631,\n", | |
| " 730,\n", | |
| " 24,\n", | |
| " 7627,\n", | |
| " 8295,\n", | |
| " 7627,\n", | |
| " 8295,\n", | |
| " 72,\n", | |
| " 18,\n", | |
| " 99,\n", | |
| " 51,\n", | |
| " 131,\n", | |
| " 4,\n", | |
| " 1851,\n", | |
| " 3,\n", | |
| " 758,\n", | |
| " 6,\n", | |
| " 12,\n", | |
| " 24,\n", | |
| " 8,\n", | |
| " 1313,\n", | |
| " 4,\n", | |
| " 790,\n", | |
| " 93,\n", | |
| " 304,\n", | |
| " 4,\n", | |
| " 929,\n", | |
| " 131,\n", | |
| " 70,\n", | |
| " 67,\n", | |
| " 282,\n", | |
| " 1027,\n", | |
| " 7,\n", | |
| " 930,\n", | |
| " 19,\n", | |
| " 358,\n", | |
| " 1799,\n", | |
| " 9250,\n", | |
| " 78,\n", | |
| " 430,\n", | |
| " 474,\n", | |
| " 1200,\n", | |
| " 5721,\n", | |
| " 7,\n", | |
| " 468,\n", | |
| " 21,\n", | |
| " 78,\n", | |
| " 2703,\n", | |
| " 3,\n", | |
| " 26695,\n", | |
| " 13321,\n", | |
| " 35429,\n", | |
| " 8,\n", | |
| " 45,\n", | |
| " 5,\n", | |
| " 501,\n", | |
| " 72,\n", | |
| " 8,\n", | |
| " 30,\n", | |
| " 38,\n", | |
| " 37,\n", | |
| " 16907,\n", | |
| " 3,\n", | |
| " 9,\n", | |
| " 8,\n", | |
| " 5,\n", | |
| " 24,\n", | |
| " 10,\n", | |
| " 72,\n", | |
| " 1,\n", | |
| " 638,\n", | |
| " 626,\n", | |
| " 8,\n", | |
| " 10,\n", | |
| " 425,\n", | |
| " 21189,\n", | |
| " 20,\n", | |
| " 105,\n", | |
| " 1241,\n", | |
| " 3,\n", | |
| " 12,\n", | |
| " 8,\n", | |
| " 36,\n", | |
| " 6,\n", | |
| " 1,\n", | |
| " 1037,\n", | |
| " 152,\n", | |
| " 12,\n", | |
| " 24,\n", | |
| " 70,\n", | |
| " 727,\n", | |
| " 35,\n", | |
| " 19175,\n", | |
| " 3]" | |
| ] | |
| }, | |
| "execution_count": 15, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "# we can now convert any piece of text first into tokens and then from tokens into indices into the vocabulary\n", | |
| "vocab(tokenizer(proper_train_dataset[0][0]))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 16, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "'master'" | |
| ] | |
| }, | |
| "execution_count": 16, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "# we can also look up what any specific word index refers to\n", | |
| "vocab.lookup_token(1259)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 17, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "proper_train_encoded = [vocab(tokens) for tokens in proper_train_dataset_as_tokens_without_labels]\n", | |
| "# note that another way to have written the above line is to instead write the line below (but this would repeat the work of tokenization which we already did):\n", | |
| "# proper_train_encoded = [vocab(tokenizer_lower_case(text)) for text, label in proper_train_dataset]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 18, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "[1259,\n", | |
| " 59266,\n", | |
| " 11261,\n", | |
| " 16475,\n", | |
| " 1225,\n", | |
| " 7,\n", | |
| " 171,\n", | |
| " 20,\n", | |
| " 162,\n", | |
| " 169,\n", | |
| " 43,\n", | |
| " 31,\n", | |
| " 5,\n", | |
| " 186,\n", | |
| " 6,\n", | |
| " 33,\n", | |
| " 233,\n", | |
| " 3,\n", | |
| " 10,\n", | |
| " 12,\n", | |
| " 24,\n", | |
| " 110,\n", | |
| " 80,\n", | |
| " 77,\n", | |
| " 16475,\n", | |
| " 53,\n", | |
| " 233,\n", | |
| " 1068,\n", | |
| " 45,\n", | |
| " 6177,\n", | |
| " 54976,\n", | |
| " 2,\n", | |
| " 36409,\n", | |
| " 55541,\n", | |
| " 2,\n", | |
| " 15191,\n", | |
| " 29297,\n", | |
| " 4,\n", | |
| " 12106,\n", | |
| " 32513,\n", | |
| " 1943,\n", | |
| " 20,\n", | |
| " 614,\n", | |
| " 1318,\n", | |
| " 45,\n", | |
| " 361,\n", | |
| " 2,\n", | |
| " 2310,\n", | |
| " 2,\n", | |
| " 1752,\n", | |
| " 2,\n", | |
| " 131,\n", | |
| " 4,\n", | |
| " 78,\n", | |
| " 579,\n", | |
| " 10754,\n", | |
| " 26,\n", | |
| " 281,\n", | |
| " 3045,\n", | |
| " 474,\n", | |
| " 3,\n", | |
| " 1,\n", | |
| " 11373,\n", | |
| " 887,\n", | |
| " 6,\n", | |
| " 646,\n", | |
| " 5281,\n", | |
| " 40,\n", | |
| " 149,\n", | |
| " 94,\n", | |
| " 8,\n", | |
| " 3721,\n", | |
| " 19,\n", | |
| " 67,\n", | |
| " 379,\n", | |
| " 2,\n", | |
| " 39,\n", | |
| " 5559,\n", | |
| " 267,\n", | |
| " 96,\n", | |
| " 787,\n", | |
| " 5,\n", | |
| " 11373,\n", | |
| " 25,\n", | |
| " 3,\n", | |
| " 10,\n", | |
| " 52626,\n", | |
| " 247,\n", | |
| " 26695,\n", | |
| " 13321,\n", | |
| " 35429,\n", | |
| " 2,\n", | |
| " 16475,\n", | |
| " 3345,\n", | |
| " 20,\n", | |
| " 68,\n", | |
| " 2,\n", | |
| " 1897,\n", | |
| " 4,\n", | |
| " 844,\n", | |
| " 7,\n", | |
| " 8888,\n", | |
| " 1,\n", | |
| " 4146,\n", | |
| " 6,\n", | |
| " 5,\n", | |
| " 8964,\n", | |
| " 905,\n", | |
| " 6,\n", | |
| " 415,\n", | |
| " 2081,\n", | |
| " 93,\n", | |
| " 8,\n", | |
| " 131,\n", | |
| " 3535,\n", | |
| " 85,\n", | |
| " 361,\n", | |
| " 60,\n", | |
| " 9,\n", | |
| " 18,\n", | |
| " 3207,\n", | |
| " 168,\n", | |
| " 617,\n", | |
| " 10,\n", | |
| " 5174,\n", | |
| " 13,\n", | |
| " 16475,\n", | |
| " 104,\n", | |
| " 5,\n", | |
| " 631,\n", | |
| " 730,\n", | |
| " 24,\n", | |
| " 7627,\n", | |
| " 8295,\n", | |
| " 7627,\n", | |
| " 8295,\n", | |
| " 72,\n", | |
| " 18,\n", | |
| " 99,\n", | |
| " 51,\n", | |
| " 131,\n", | |
| " 4,\n", | |
| " 1851,\n", | |
| " 3,\n", | |
| " 758,\n", | |
| " 6,\n", | |
| " 12,\n", | |
| " 24,\n", | |
| " 8,\n", | |
| " 1313,\n", | |
| " 4,\n", | |
| " 790,\n", | |
| " 93,\n", | |
| " 304,\n", | |
| " 4,\n", | |
| " 929,\n", | |
| " 131,\n", | |
| " 70,\n", | |
| " 67,\n", | |
| " 282,\n", | |
| " 1027,\n", | |
| " 7,\n", | |
| " 930,\n", | |
| " 19,\n", | |
| " 358,\n", | |
| " 1799,\n", | |
| " 9250,\n", | |
| " 78,\n", | |
| " 430,\n", | |
| " 474,\n", | |
| " 1200,\n", | |
| " 5721,\n", | |
| " 7,\n", | |
| " 468,\n", | |
| " 21,\n", | |
| " 78,\n", | |
| " 2703,\n", | |
| " 3,\n", | |
| " 26695,\n", | |
| " 13321,\n", | |
| " 35429,\n", | |
| " 8,\n", | |
| " 45,\n", | |
| " 5,\n", | |
| " 501,\n", | |
| " 72,\n", | |
| " 8,\n", | |
| " 30,\n", | |
| " 38,\n", | |
| " 37,\n", | |
| " 16907,\n", | |
| " 3,\n", | |
| " 9,\n", | |
| " 8,\n", | |
| " 5,\n", | |
| " 24,\n", | |
| " 10,\n", | |
| " 72,\n", | |
| " 1,\n", | |
| " 638,\n", | |
| " 626,\n", | |
| " 8,\n", | |
| " 10,\n", | |
| " 425,\n", | |
| " 21189,\n", | |
| " 20,\n", | |
| " 105,\n", | |
| " 1241,\n", | |
| " 3,\n", | |
| " 12,\n", | |
| " 8,\n", | |
| " 36,\n", | |
| " 6,\n", | |
| " 1,\n", | |
| " 1037,\n", | |
| " 152,\n", | |
| " 12,\n", | |
| " 24,\n", | |
| " 70,\n", | |
| " 727,\n", | |
| " 35,\n", | |
| " 19175,\n", | |
| " 3]" | |
| ] | |
| }, | |
| "execution_count": 18, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "proper_train_encoded[0]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 19, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "['master', 'cinéaste', 'alain', 'resnais', 'likes', 'to', 'work', 'with', 'those', 'actors', 'who', 'are', 'a', 'part', 'of', 'his', 'family', '.', 'in', 'this', 'film', 'too', 'we', 'see', 'resnais', \"'\", 'family', 'members', 'like', 'pierre', 'arditi', ',', 'sabine', 'azema', ',', 'andré', 'dussolier', 'and', 'fanny', 'ardant', 'dealing', 'with', 'serious', 'themes', 'like', 'death', ',', 'religion', ',', 'suicide', ',', 'love', 'and', 'their', 'overall', 'implications', 'on', 'our', 'daily', 'lives', '.', 'the', 'formal', 'nature', 'of', 'relationship', 'shared', 'by', 'these', 'people', 'is', 'evident', 'as', 'even', 'friends', ',', 'they', 'address', 'each', 'other', 'using', 'a', 'formal', 'you', '.', 'in', '1984,while', 'making', \"l'amour\", 'à', 'mort', ',', 'resnais', 'dealt', 'with', 'time', ',', 'memory', 'and', 'space', 'to', 'unravel', 'the', 'mysteries', 'of', 'a', 'fundamental', 'question', 'of', 'human', 'existence', ':', 'is', 'love', 'stronger', 'than', 'death', '?', 'it', 'was', '16', 'years', 'ago', 'in', '1968', 'that', 'resnais', 'made', 'a', 'somewhat', 'similar', 'film', 'je', \"t'aime\", 'je', \"t'aime\", 'which', 'was', 'also', 'about', 'love', 'and', 'memories', '.', 'message', 'of', 'this', 'film', 'is', 'loud', 'and', 'clear', ':', 'true', 'and', 'deep', 'love', 'can', 'even', 'put', 'science', 'to', 'shame', 'as', 'dead', 'lovers', 'regain', 'their', 'lost', 'lives', 'leaving', 'doctors', 'to', 'care', 'for', 'their', 'reputation', '.', \"l'amour\", 'à', 'mort', 'is', 'like', 'a', 'game', 'which', 'is', 'not', 'at', 'all', 'didactic', '.', 'it', 'is', 'a', 'film', 'in', 'which', 'the', 'musical', 'score', 'is', 'in', 'perfect', 'tandem', 'with', 'its', 'images', '.', 'this', 'is', 'one', 'of', 'the', 'reasons', 'why', 'this', 'film', 'can', 'easily', 'be', 'grasped', '.']\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "# we can reconstruct any original review from the encoded version of the review\n", | |
| "print([vocab.lookup_token(word_idx) for word_idx in proper_train_encoded[0]])" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 20, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "proper_train_labels = [label for text, label in proper_train_dataset]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 21, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "val_encoded = [vocab(tokenizer(text)) for text, label in val_dataset]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 22, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "val_labels = [label for text, label in val_dataset]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 23, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "proper_train_dataset_encoded = list(zip(proper_train_encoded, proper_train_labels))\n", | |
| "val_dataset_encoded = list(zip(val_encoded, val_labels))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## Setting up a recurrent neural net for sentiment analysis that uses pre-trained word embeddings" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "We first load in pre-trained GloVe embeddings only for tokens that we encountered in the proper training data. Note that these embeddings are for \"uncased\" words (i.e., we should convert the tokens to lowercase first)." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 24, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "from torchtext.vocab import GloVe\n", | |
| "pretrained_embedding = GloVe(name='6B', dim=100)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 25, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "torchtext.vocab.vectors.GloVe" | |
| ] | |
| }, | |
| "execution_count": 25, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "type(pretrained_embedding)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 26, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "tensor([ 0.2309, 0.2828, 0.6318, -0.5941, -0.5860, 0.6326, 0.2440, -0.1411,\n", | |
| " 0.0608, -0.7898, -0.2910, 0.1429, 0.7227, 0.2043, 0.1407, 0.9876,\n", | |
| " 0.5253, 0.0975, 0.8822, 0.5122, 0.4020, 0.2117, -0.0131, -0.7162,\n", | |
| " 0.5539, 1.1452, -0.8804, -0.5022, -0.2281, 0.0239, 0.1072, 0.0837,\n", | |
| " 0.5501, 0.5848, 0.7582, 0.4571, -0.2800, 0.2522, 0.6896, -0.6097,\n", | |
| " 0.1958, 0.0442, -0.3114, -0.6883, -0.2272, 0.4618, -0.7716, 0.1021,\n", | |
| " 0.5564, 0.0674, -0.5721, 0.2374, 0.4717, 0.8277, -0.2926, -1.3422,\n", | |
| " -0.0993, 0.2814, 0.4160, 0.1058, 0.6220, 0.8950, -0.2345, 0.5135,\n", | |
| " 0.9938, 1.1846, -0.1636, 0.2065, 0.7385, 0.2406, -0.9647, 0.1348,\n", | |
| " -0.0072, 0.3302, -0.1236, 0.2719, -0.4095, 0.0219, -0.6069, 0.4076,\n", | |
| " 0.1957, -0.4180, 0.1864, -0.0327, -0.7857, -0.1385, 0.0440, -0.0844,\n", | |
| " 0.0491, 0.2410, 0.4527, -0.1868, 0.4618, 0.0891, -0.1819, -0.0152,\n", | |
| " -0.7368, -0.1453, 0.1510, -0.7149])" | |
| ] | |
| }, | |
| "execution_count": 26, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "pretrained_embedding['cat']\n", | |
| "# note that if you ask for a word embedding for a word that GloVe does not keep\n", | |
| "# track of, you'll get all zeros" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 27, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "embedding_matrix = torch.zeros(len(vocab), pretrained_embedding.dim)\n", | |
| "for i, token in enumerate(vocab.lookup_tokens(range(len(vocab)))):\n", | |
| " embedding_matrix[i] = pretrained_embedding[token]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "The next code cell constructs a PyTorch recurrent neural net model for sentiment analysis. Unfortunately, the code involved doesn't readily work with ``nn.Sequential`` that we have previously been using (for somewhat complicated reasons; in short, it has to do with an input batch of time series possibly having varying lengths, and accounting for varying lengths properly is not easy to do with ``nn.Sequential``). We will instead build the PyTorch model using another standard approach by creating a Python class that inherits from the `nn.Module` class.\n", | |
| "\n", | |
| "To illustrate how this works, here was how we created the multilayer perceptron model for MNIST digits in the previous demo:\n", | |
| "\n", | |
| "```\n", | |
| "deeper_model = nn.Sequential(nn.Flatten(),\n", | |
| " nn.Linear(in_features=784, out_features=512),\n", | |
| " nn.ReLU(),\n", | |
| " nn.Linear(in_features=512, out_features=10))\n", | |
| "```\n", | |
| "\n", | |
| "An alternative way to code the same model is as follows:\n", | |
| "\n", | |
| "```\n", | |
| "class DeeperModel(nn.Module):\n", | |
| " def __init__(self, num_in_features, num_intermediate_features, num_out_features):\n", | |
| " super().__init__()\n", | |
| " self.flatten = nn.Flatten()\n", | |
| " self.linear1 = nn.Linear(num_in_features, num_intermediate_features)\n", | |
| " self.relu = nn.ReLU()\n", | |
| " self.linear2 = nn.Linear(num_intermediate_features, num_out_features)\n", | |
| "\n", | |
| " def forward(self, inputs):\n", | |
| " flatten_output = self.flatten(inputs)\n", | |
| " linear1_output = self.linear1(flatten_output)\n", | |
| " relu_output = self.relu(linear1_output)\n", | |
| " linear2_output = self.linear2(relu_output)\n", | |
| " return linear2_output\n", | |
| "\n", | |
| "deeper_model = DeeperModel(784, 512, 10)\n", | |
| "```\n", | |
| "\n", | |
| "**Importantly, in the above code, the `forward` function specifies how the neural net processes input data. Note that the only argument it takes (aside from `self`) is the input data (`inputs`), which for MNIST digits we saw will be of the format (batch size, 1, 28, 28). Consequently, the example input data batch supplied to the `summary` function only needs this 4D table.**\n", | |
| "\n", | |
| "The code below creates the PyTorch neural net model corresponding to the architecture:\n", | |
| "\n", | |
| "1. `Embedding` layer (for every time series: convert the word index at every time step into a 100-dimensional GloVe word embedding)\n", | |
| "2. `LSTM` layer with 32 output nodes (for every time series: put the time series through the LSTM's `for` loop and grab only the last time step's output, which has 32 numbers)\n", | |
| "3. `Linear` layer with 2 output nodes (now every input data point to the linear layer is just a 1D table of 32 numbers, which this linear layer converts to 2 output numbers corresponding to the two classes)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 28, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "class EmbeddingLSTMLinearModel(nn.Module):\n", | |
| " def __init__(self, embedding_matrix, num_lstm_output_nodes, num_final_output_nodes):\n", | |
| " super().__init__()\n", | |
| " self.embedding_layer = nn.Embedding.from_pretrained(embedding_matrix)\n", | |
| " self.lstm_layer = nn.LSTM(embedding_matrix.shape[1], num_lstm_output_nodes)\n", | |
| " self.linear_layer = nn.Linear(num_lstm_output_nodes, num_final_output_nodes)\n", | |
| "\n", | |
| " def forward(self, text_encodings, lengths):\n", | |
| " embeddings = self.embedding_layer(text_encodings)\n", | |
| "\n", | |
| " rnn_last_time_step_outputs = \\\n", | |
| " UDA_get_rnn_last_time_step_outputs(embeddings, lengths, self.lstm_layer)\n", | |
| "\n", | |
| " return self.linear_layer(rnn_last_time_step_outputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "**Notice that the `forward` function takes _two_ inputs (aside from `self`): `text_encodings` and `lengths`.** Keep in mind that `forward` takes in a batch of input time series, and these input time series could have different lengths. These different lengths are precisely what is stored in `lengths` as a 1D table of integers. Meanwhile, `text_encodings` is a 2D table where the number of rows is the maximum number of time steps in the input batch of time series, and the number of columns is the number of time series in the input batch. See the lecture slides for how `text_encodings` gets filled in (basically we pad all the time series in the batch to be of the same length as the longest time series; the padded entries will of course get ignored by the LSTM layer since it will know the correct length to use per time series). **When we give this neural net model an example data batch using the `summary` function, we need to specify two inputs corresponding to `text_encodings` and `lengths`.**" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 29, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "simple_lstm_model = EmbeddingLSTMLinearModel(embedding_matrix, 32, 2)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 30, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "==========================================================================================\n", | |
| "Layer (type:depth-idx) Output Shape Param #\n", | |
| "==========================================================================================\n", | |
| "EmbeddingLSTMLinearModel [5, 2] --\n", | |
| "├─Embedding: 1-1 [7, 5, 100] (9,066,700)\n", | |
| "├─LSTM: 1-2 [18, 32] 17,152\n", | |
| "├─Linear: 1-3 [5, 2] 66\n", | |
| "==========================================================================================\n", | |
| "Total params: 9,083,918\n", | |
| "Trainable params: 17,218\n", | |
| "Non-trainable params: 9,066,700\n", | |
| "Total mult-adds (Units.MEGABYTES): 73.35\n", | |
| "==========================================================================================\n", | |
| "Input size (MB): 0.00\n", | |
| "Forward/backward pass size (MB): 0.03\n", | |
| "Params size (MB): 36.34\n", | |
| "Estimated Total Size (MB): 36.37\n", | |
| "==========================================================================================" | |
| ] | |
| }, | |
| "execution_count": 30, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "# example where there are 5 input time series of lengths 3, 2, 5, 1, 7;\n", | |
| "# we specify these time series using a 2D table that is padded and a\n", | |
| "# 1D table of lengths (see lecture slides for details)\n", | |
| "summary(simple_lstm_model,\n", | |
| " input_data=[torch.zeros((7, 5), dtype=torch.long),\n", | |
| " torch.tensor([3, 2, 5, 1, 7], dtype=torch.long)])\n", | |
| "\n", | |
| "# note: the LSTM's output is in a compressed format (called a \"packed sequence\")\n", | |
| "# that appears to put all the outputs of all the time series together (the output\n", | |
| "# shape in this case appears to be 18 by 32) but it actually does keep track of\n", | |
| "# which of the time steps correspond to which input time series (i.e., it knows\n", | |
| "# that 3 of the rows correspond to the 0-th time series, 2 of the rows correspond\n", | |
| "# to the 1st time series, 5 of the rows correspond to the 2nd time series, 1 row\n", | |
| "# corresponds to the 3rd time series, and 7 rows correspond to the 4th time\n", | |
| "# series); my helper code automatically maps these rows back to the correct\n", | |
| "# format so that the final linear layer recognizes that there are 5 input data\n", | |
| "# points" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 31, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "os.makedirs('./saved_model_checkpoints', exist_ok=True)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 32, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 0%| | 0/30 [00:00<?, ?it/s]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.6021\n", | |
| " Validation accuracy: 0.6048\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 3%|████▋ | 1/30 [20:28<9:53:47, 1228.52s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.7117\n", | |
| " Validation accuracy: 0.7176\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 7%|█████████▍ | 2/30 [38:56<9:00:10, 1157.51s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.7664\n", | |
| " Validation accuracy: 0.7738\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 10%|██████████████ | 3/30 [57:23<8:30:31, 1134.49s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.7790\n", | |
| " Validation accuracy: 0.7804\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 13%|██████████████████▌ | 4/30 [1:15:49<8:06:42, 1123.18s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8116\n", | |
| " Validation accuracy: 0.8112\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 17%|███████████████████████▏ | 5/30 [1:34:01<7:43:22, 1112.11s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8180\n", | |
| " Validation accuracy: 0.8110\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 20%|███████████████████████████▊ | 6/30 [1:52:29<7:24:11, 1110.47s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8469\n", | |
| " Validation accuracy: 0.8390\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 23%|████████████████████████████████▍ | 7/30 [2:10:50<7:04:30, 1107.43s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8613\n", | |
| " Validation accuracy: 0.8534\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 27%|█████████████████████████████████████ | 8/30 [2:29:11<6:45:22, 1105.59s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8655\n", | |
| " Validation accuracy: 0.8602\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 30%|█████████████████████████████████████████▋ | 9/30 [2:47:35<6:26:42, 1104.88s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8715\n", | |
| " Validation accuracy: 0.8658\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 33%|██████████████████████████████████████████████ | 10/30 [3:05:52<6:07:29, 1102.49s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8804\n", | |
| " Validation accuracy: 0.8658\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 37%|██████████████████████████████████████████████████▌ | 11/30 [3:24:09<5:48:38, 1100.98s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8810\n", | |
| " Validation accuracy: 0.8672\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 40%|███████████████████████████████████████████████████████▏ | 12/30 [3:42:40<5:31:11, 1103.96s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8847\n", | |
| " Validation accuracy: 0.8686\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 43%|███████████████████████████████████████████████████████████▊ | 13/30 [4:01:07<5:13:00, 1104.72s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8923\n", | |
| " Validation accuracy: 0.8734\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 47%|████████████████████████████████████████████████████████████████▍ | 14/30 [4:19:31<4:54:34, 1104.64s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8914\n", | |
| " Validation accuracy: 0.8688\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 50%|█████████████████████████████████████████████████████████████████████ | 15/30 [4:37:58<4:36:20, 1105.38s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9000\n", | |
| " Validation accuracy: 0.8746\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 53%|█████████████████████████████████████████████████████████████████████████▌ | 16/30 [4:56:28<4:18:14, 1106.73s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9010\n", | |
| " Validation accuracy: 0.8794\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 57%|██████████████████████████████████████████████████████████████████████████████▏ | 17/30 [5:14:37<3:58:38, 1101.40s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9050\n", | |
| " Validation accuracy: 0.8784\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 60%|██████████████████████████████████████████████████████████████████████████████████▊ | 18/30 [5:32:52<3:39:53, 1099.50s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9061\n", | |
| " Validation accuracy: 0.8762\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 63%|███████████████████████████████████████████████████████████████████████████████████████▍ | 19/30 [5:51:04<3:21:08, 1097.15s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9087\n", | |
| " Validation accuracy: 0.8752\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 67%|████████████████████████████████████████████████████████████████████████████████████████████ | 20/30 [6:09:24<3:03:01, 1098.16s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8938\n", | |
| " Validation accuracy: 0.8650\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 70%|████████████████████████████████████████████████████████████████████████████████████████████████▌ | 21/30 [6:27:40<2:44:35, 1097.31s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9158\n", | |
| " Validation accuracy: 0.8804\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 73%|█████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 22/30 [6:45:55<2:26:13, 1096.72s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8961\n", | |
| " Validation accuracy: 0.8608\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 77%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 23/30 [7:04:17<2:08:07, 1098.23s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9204\n", | |
| " Validation accuracy: 0.8756\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 80%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 24/30 [7:22:33<1:49:46, 1097.75s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9193\n", | |
| " Validation accuracy: 0.8804\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 83%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 25/30 [7:40:50<1:31:26, 1097.28s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9045\n", | |
| " Validation accuracy: 0.8628\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 87%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 26/30 [7:59:06<1:13:07, 1096.93s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9193\n", | |
| " Validation accuracy: 0.8720\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 90%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 27/30 [8:17:31<54:58, 1099.40s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9250\n", | |
| " Validation accuracy: 0.8742\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 93%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 28/30 [8:35:59<36:44, 1102.06s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.8967\n", | |
| " Validation accuracy: 0.8512\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "\r", | |
| " 97%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 29/30 [8:54:23<18:22, 1102.75s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| " Proper training accuracy: 0.9267\n", | |
| " Validation accuracy: 0.8710\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [9:12:57<00:00, 1105.93s/it]\n" | |
| ] | |
| }, | |
| { | |
| "data": { | |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAB7L0lEQVR4nO3deVhUZfvA8e/MsCOLgGyKiLu477tmuWambVKZtqhlWWpWb/maP5cWs8WsTMtKbXF7rVwq09DMNDVXXNJcUVxABBRwYZs5vz8OjIxsAwycAe7Pdc0FnHnmzH0EOTfPcj86RVEUhBBCCCGqEL3WAQghhBBClDdJgIQQQghR5UgCJIQQQogqRxIgIYQQQlQ5kgAJIYQQosqRBEgIIYQQVY4kQEIIIYSochy0DsAemUwmLl68iIeHBzqdTutwhBBCCGEFRVFITU0lODgYvb7wPh5JgPJx8eJFQkJCtA5DCCGEECVw7tw5atWqVWgbSYDy4eHhAaj/gJ6enhpHI4QQQghrpKSkEBISYr6PF0YSoHzkDHt5enpKAiSEEEJUMNZMX5FJ0EIIIYSociQBEkIIIUSVIwmQEEIIIaocmQNUCkajkczMTK3DEJWMo6MjBoNB6zCEEKJSkwSoBBRFIS4ujqtXr2odiqikvL29CQwMlDpUQghRRiQBKoGc5Mff3x83Nze5SQmbURSFGzduEB8fD0BQUJDGEQkhROUkCVAxGY1Gc/Lj6+urdTiiEnJ1dQUgPj4ef39/GQ4TQogyIJOgiylnzo+bm5vGkYjKLOfnS+aYCSFE2ZAEqIRk2EuUJfn5EkKIsiVDYEIIIYQoN0aTwq7oJOJT0/D3cKFDmA8Gffn/0ScJkBBCCCHKxfrDsUz/6QixyWnmY0FeLkwdFE7/ZuW76EOGwDRkNCnsOJXImqgL7DiViNGkaB1SlXPmzBl0Oh1RUVFWv2batGm0atWqzGISQoiypsX9Z/3hWJ79bp9F8gMQl5zGs9/tY/3h2DKPITfpAdKIFlnwE088wddffw2Ag4MDISEh3H///UyfPh13d/cyec+yoNPpWLVqFUOGDCn1uUJCQoiNjcXPz8/q17z88su88MILpX5vIYTQghb3H6NJYfpPR8gvzVIAHTD9pyP0CQ8st+Ew6QHSgJZZcP/+/YmNjeX06dO8+eabzJs3j5dffjnftlqvQCrN+1v7WoPBQGBgIA4O1v8tUK1aNSmBIISokLS6/+yKTsrznrkpQGxyGruik8rk/fMjCZANKIrCjYwsqx6paZlMXftPgVkwwLS1R0hNy7TqfIpSvG5LZ2dnAgMDCQkJ4dFHH2XYsGGsXr1afd/soZ2FCxdSt25dnJ2dURSFmJgYBg8eTLVq1fD09GTo0KFcunTJfM6c133++eeEhITg5ubGQw89lKdS9qJFi2jSpAkuLi40btyYefPmmZ/LGYr63//+xx133IGLiwvfffddnvjr1KkDwH333YdOpzN/XVDs69evp1u3bnh7e+Pr68s999zDqVOn8rxvzhDYH3/8gU6nY9OmTbRr1w43Nze6dOnCsWPH8lxvjieeeIIhQ4bw/vvvExQUhK+vL2PHjrVIwmJjYxk4cCCurq6EhYWxdOlS6tSpw5w5c4rx3RNCiJIrqhcG1F6YshgOi08tOPkpSTtbkCEwG7iZaST8/zbY5FwKEJeSRvNpv1nV/siMfrg5lfzb6OrqanGjPnnyJP/73//44YcfzAX4hgwZgru7O1u2bCErK4vnnnuOiIgI/vjjjzyv++mnn0hJSWHkyJGMHTuWJUuWAPDFF18wdepU5s6dS+vWrdm/fz+jR4/G3d2dxx9/3HyeV199lQ8++IBFixbh7OycJ97du3fj7+/PokWL6N+/v0WRwPxiv379OhMnTqR58+Zcv36d//u//+O+++4jKioKvb7g/H/y5Ml88MEH1KhRgzFjxvDUU0/x119/Fdh+8+bNBAUFsXnzZk6ePElERAStWrVi9OjRAIwYMYKEhAT++OMPHB0dmThxornasxBCG/ayGqm8FKcXpnM92/Zy+3u42LSdLUgCVIXt2rWLpUuXctddd5mPZWRk8O2331KjRg0AIiMjOXjwINHR0YSEhADw7bff0rRpU3bv3k379u0BSEtL4+uvv6ZWrVoAfPLJJwwcOJAPPviAwMBA3njjDT744APuv/9+AMLCwjhy5Aiff/65RQI0YcIEc5v85MSVs1dWbrfHDvDAAw9YtPnqq6/w9/fnyJEjNGvWrMD3eeutt+jZsycAr732GgMHDiQtLQ0Xl/z/c1avXp25c+diMBho3LgxAwcOZNOmTYwePZp///2XjRs3snv3btq1awfAl19+SYMGDQp8fyFE2bKn1UhlLel6Bqv3X+DLbdFWtS+LXpgOYT4EebkUmIDpgEAvNQktL5IA2YCro4EjM/pZ1XZXdBJPLNpdZLvFT7a36gfB1bF42yT8/PPPVKtWjaysLDIzMxk8eDCffPKJ+fnQ0FCLBOLo0aOEhISYkx+A8PBwvL29OXr0qDkBql27tjn5AejcuTMmk4ljx45hMBg4d+4cI0eONPeIAGRlZeHl5WURX06CUBK3xw5w6tQppkyZws6dO0lISMBkMgEQExNTaALUokUL8+c5+3HFx8dTu3btfNs3bdrUojcqKCiIQ4cOAXDs2DEcHBxo06aN+fn69etTvXr1Yl6hEMIWcubB3D7QkzMPZv5jbaxKgrTsQSrqvbOMJrYcv8zKPefZ9O8lMo3WD2uVRS+MQa9jWMfavP/b8TzP5UQ9dVB4ufbASQJkAzqdzuphqO4NahDk5UJcclq+47A5WXD3BjXK5AehV69ezJ8/H0dHR4KDg3F0dLR4/vbVYIqi5FuVuKDjOXKe0+l05qTjiy++oGPHjhbtbt/nqjSr0fJ77aBBgwgJCeGLL74gODgYk8lEs2bNyMjIKPRcuf9dcq4l5zqKap/zmpz2Bc3TKu78LSFE6dlqNZKWPUiFvXd9/2qs3HOeH/df4HJquvn55jW9eKBtTeb/cYr4lPRC7z9l1Quz87Q6wdnVycDNDKP5eKBGPW+SAJUzg17H1EHhPPvdPnRg8UNYHlmwu7s79evXt7p9eHg4MTExnDt3ztwLdOTIEZKTk2nSpIm5XUxMDBcvXiQ4OBiAHTt2oNfradiwIQEBAdSsWZPTp08zbNiwUl+Do6MjRqOxyHaJiYkcPXqUzz//nO7duwOwbdu2Ur9/cTVu3JisrCz2799P27ZtAXW+0u2TxIUQZc/aeTAr95yjZ6MaVHdzwuW2nnZb9SCVREHvHZucxpjv9lkc83V3YkjrmjzUrhaNAz0BCPR0yff+k6Os7j+7zySx7WQCDnodv47rTmxymuZzryQB0kD/ZkHMf6xNngxeqyy4ML1796ZFixYMGzaMOXPmmCdB9+zZ02K4ysXFhccff5z333+flJQUxo0bx9ChQ83zdKZNm8a4cePw9PRkwIABpKens2fPHq5cucLEiROLFVOdOnXYtGkTXbt2xdnZucChpOrVq+Pr68uCBQsICgoiJiaG1157reT/GCXUuHFjevfuzdNPP23ufXvppZdwdXWVPb9ElVfew0jWzm957cdD5s9dHPV4uzrh7eaIt6sjUeeualLPprDeq9zualyDoe1r06uRP04Olos9Crr/ADzZtU6Z3X8+2ngCgIfa1aKOnzt1/LSvPScJkEb6NwuiT3ig3a9A0Ol0rF69mhdeeIEePXqg1+vp37+/xbwhUOe03H///dx9990kJSVx9913WyxzHzVqFG5ubrz33nv85z//wd3dnebNmzNhwoRix/TBBx8wceJEvvjiC2rWrMmZM2fybafX61m+fDnjxo2jWbNmNGrUiI8//pg77rij2O9ZWt988w0jR46kR48eBAYGMnPmTP75558CJ1ULURVoMYxk7fwWTxcHrmcYMZoU0jJNxGWmEZdSdPJUliupiuq9yjGqe71C3/v2+89fJxL4397zbDwaz2sDmuRJmkord+/Pc3dYPwJR1nSKTETIIyUlBS8vL5KTk/H09LR4Li0tjejoaMLCwuTmlW3atGmsXr26WNtJVHXnz58nJCSEjRs3WqzCyyE/Z6KyK2goJ+dPwLIaRjKaFLrN+r3I1UjbXr0TvQ5S07NIvpHJlRsZXLmRyaajl/hmx9ki3+ejh1sxuFVNm8a+JuoC45dH2fy9r6dn0fO9P0i4ls4bQ5oxvFNoKaLM67Ev/2bbyQQe6VCbmfc3t+m5b1fY/ft2UghRiHLw+++/s3btWqKjo9m+fTsPP/wwderUoUePHlqHJkS507IgX848zPzcPg9Tp9Ph6eJIiI8bLWp507NhDQZYmZSVxUqqsqql4+7swLi71J6Zjzed4EZGVrFjK0ju3p+xverZ7Ly2IAmQEOUgMzOT//73vzRt2pT77ruPGjVqmIsiClHVaL0tQkGrdgO9XIrsecqpZ1PYZAUvV4cyWUkVl3yz0Od1qEOIJXnvh9vXJsTHlcup6Sz660zJAszHnI3qsveH2oVQq7qbzc5rCzIHSJTatGnTmDZtmtZh2LV+/frRr591taKEqOy03BYhLdPIlDWHAXi8Syj9mwYVax5mYSt5cyTfzOLr7Wd4qluYzeL+Ye95Xvn+gPlrW68idnLQ81KfRkxYEcVnW04xrGNtvN2cShXzrugk/jqZaJe9PyA9QEIIIcqZltsizNt8krOJNwjwdOblvo3oXM+Xwa1q0rmer9WJQ85KqkAvy/iCvFzoGx4AwIyfj7Dgz1P5vbzYVuyO4eXvD2BS4JEOtZn3aN73tqb3qij3tgymcaAHqWlZzP+j9LF/tMl+e39AeoCEEEKUs6K2RYCSD+UU5mT8NeZvUW/s0wY1xcOl5EPQBa3k1evgw8jjfPz7Sd5e9y+ZRoWxvUq+8um7nWd5fbXaYzWicyjT722KTqejXzPbryLW63X8p38jnlq8h8Xbz/BE1zoEebmW6Fw5vT+OBvvs/QFJgIQQosor71o8Br2OJ7vW4e11/xbY5okudWwag6IovL76EJlGhV6NatC/WWDRLyqCQa/Ld7n5xL6NcDDomR15nPc2HCPTaGL8XQ2KXfdr8V/RTPvpCABPdQ1jyj1NzOco6L1Lq1cjf9rXqc7uM1f4eNMJZt7fougX5cPee39AEiAhhKjStKjFk55l5Md9FwBwdtCTnnVrm5mcr7/7+ywPt6+Nl5ttFgr8uO8CO08n4eKoZ8bgZmVehHTcXQ1wMOh4d/0x5mw8QZZR4aW+Da1+3y+3nubNX44C8EzPurzWv3G5FE7V6XS82r8xD362g//tOc+o7nWpV6Nasc6Ru/fnuTvss/cHZA6QEEJUCkaTwo5TiayJusCOU4lWLSHPqcVz+1BUzpYO6w/Hlkmsczae4N+4VHzdndj6n14sG92Jjx5uxbLRndjx2l3Uqu7KuaSbvPz9AZvsmXf1RgZvrVOTiXF3NSDEp3x6JJ67oz6vD1S3DJq7+STvrP/XquuZ/8cpc/Lzwp31yy35ydGujg+9m/hjNCnMzmfz0qLY88qv3CQBEla74447LCo316lThzlz5hT6mpxK0qVlq/MIYa9KksDkWH84lm6zfueRL3YyfnkUj3yxk26zfi80gdGqFs/es1f4PHseztv3N8ff08ViIrJPNSfmD2uLk0FP5JFLfLH1dKnf851f/yXpegYNA6oxunvdUp+vOEZ1r8u07LpDn285zRs/Hy00Cfp40wlmrVeHBl/s3ZCX+jbSZMucl/s1QqeDXw7FcvD8Vatf9/fpRLafsv/eH5AhsCpj0KBB3Lx5k40bN+Z5bseOHXTp0oW9e/fSpk0bq8+5e/fuUu3enp+CqkrHxsYWuOeXEBVdaYahrN2Y02RSiEm6wb9xqfwbl8L2kwlW1+Kx1VyTmxlGXl6prma6v3VN+jXNfx5O81pe/N+gcF5ffZhZ64/RunZ12tcp2YToPWeSWL77HABv3dccR0P5/93/RNcwHAx6Xl99mIV/RZNlMjFtUFMUyDX3ypm/TiYwd7OaHL7Sr1GpJk+XVuNAT+5rVZMf91/g3fXH+G5UR6te99GmnD2/7Lv3ByQB0l5qHOxZBO2eBI/ST8oryMiRI7n//vs5e/YsoaGWZc4XLlxIq1atipX8ANSoUcOWIRYqZ1NVISqb0uwsbk0vzosrDjBv80lOxF/nZqax2PHZshbPrPX/Ep1wnUBPF6be27TQtsM61mb3mSTWRF3k+aX7+GVcd/yqORfr/TKNJiavUldQRbQLKXESZQuPdQrF0aDjtR8P8c2Os5y+fJ2T8dfy3V9s8t1NGN2jfHuq8vNin4b8dPAi204msO1EAt0a+BXaPnfvj5bJm7VkCExrqXGw5R31Yxm655578Pf3Z/HixRbHb9y4wYoVKxgyZAiPPPIItWrVws3NjebNm7Ns2bJCz3n7ENiJEyfo0aMHLi4uhIeHExkZmec1r776Kg0bNsTNzY26desyZcoUMjMzAVi8eDHTp0/nwIED6HRqGfqceG8fAjt06BB33nknrq6u+Pr68vTTT3Pt2jXz80888QRDhgzh/fffJygoCF9fX8aOHWt+LyHsgbXDUNfSsohLTuPfuBR2nk5k/eE4VuyOYcrqQ0Vujnkz08jBCynczDTi5KCnabAnD7SpxWMda1sVo61q8Ww/mcDi7WcAePfBFni5Fj65WafT8fZ9zanvX41LKemMX76/2MNxX22L5tilVHzcnXhtQOOShm4zEe1r8/6DLdEB204mFLi5aohPyZae21qIjxvDOqp/ML+7oej5Szm9P0PbhVDT2z6uoTDSA2QLigKZN0r22qybtz5mXC/+6x3dwIrxYQcHB0aMGMHixYv5v//7P/OY8sqVK8nIyGDUqFEsW7aMV199FU9PT3755ReGDx9O3bp16dix6K5Pk8nE/fffj5+fHzt37iQlJSXfnd49PDxYvHgxwcHBHDp0iNGjR+Ph4cF//vMfIiIiOHz4MOvXrzcP1Xl5eeU5x40bN+jfvz+dOnVi9+7dxMfHM2rUKJ5//nmLBG/z5s0EBQWxefNmTp48SUREBK1atWL06NFFXo8Q5cHaLSGaTdtQqvd5sksdhnUKpY6vGw7ZQ0BGk8Kmf+OJS07LNwEDcHHU07q2d6neGyAlLZNXvj8IqD07PRpa13vs7uzA/GFtuHfuX/x1MpGPNp1gYp+GVr32XNIN82Tc/97dhOrupatqbCtDWtfkjZ+PcPVm/n+M6VCT3j7hgWVaisBaz99Zn//tOcfB88msPxzHgOb590ZazP2pAL0/IAmQbWTegLeDS3eOhf1L9rr/XgQn6+bhPPXUU7z33nv88ccf9OrVS33bhQu5//77qVmzJi+//LK57QsvvMD69etZuXKlVQnQxo0bOXr0KGfOnKFWrVoAvP322wwYMMCi3euvv27+vE6dOrz00kusWLGC//znP7i6ulKtWjUcHBwKHfJasmQJN2/e5JtvvjHPQZo7dy6DBg1i1qxZBASolVirV6/O3LlzMRgMNG7cmIEDB7Jp0yZJgESZsbaejsmkcCQ2hWW7Yqw+t0Gvw9vVES83R7xdHfF2cyIzy8TWkwlFvrZv00Dq+1suZbZmS4e0TBMvroji40dal2ruzJs/H+HC1ZvU9nHjv3c3KdZrGwR4MPP+5kxYEcUnv5+gXWj1IhMoRVGYuvYf0jJNdAzz4YE2tt2VvTR2RScVmPxA2cy9Kg2/as6M6l6Xjzed4L3fjtEnPMCcROc2Z2PF6v0BSYCqlMaNG9OlSxcWLlxIr169OHXqFFu3buW3337DaDTyzjvvsGLFCi5cuEB6ejrp6elWT3I+evQotWvXNic/AJ07d87T7vvvv2fOnDmcPHmSa9eukZWVhaenZ7Gu4+jRo7Rs2dIitq5du2IymTh27Jg5AWratCkGg8HcJigoiEOHDhXrvUTVUpqCgEVNZD6XdIO/Tiaw9WQC208mcOWG9cOxCx9vR6/G/nlWAxlNCt1m/V5gL44OdYuEgioq52zpkF/c97WuyZdbo/n1cBzjlu0vcRK06egl/rfnPDodvP9QS9ydi3/bGdK6JrvOJLH07xjGL9/PL+O6E1zITXbDP3H8/m88jgYdb93XXJNVVAXRch+0khrdPYxvd5zh9OXr/LDvPBHtLYdPd55OZMfpitX7A3aQAM2bN4/33nuP2NhYmjZtypw5c+jevXuB7T/99FPmzp3LmTNnqF27NpMnT2bEiBEWbX744QemTJnCqVOnqFevHm+99Rb33Xdf2V2Eo5vaE2Ota5fUB0DcIVj3Ctz9HgQ2V49VC1Af1r53MYwcOZLnn3+eTz/9lEWLFhEaGspdd93Fe++9x4cffsicOXNo3rw57u7uTJgwgYyMDKvOm9/Y8O2/dHbu3MnDDz/M9OnT6devH15eXixfvpwPPvigWNegKEqBv9ByH799p3WdTofJZLr9JUIAZbMSKzY5jTHf7aNGNScuX7P8v+TuZKBjmA97zl4hJS0r3/PmJDA9G+VNfqDwXhxrN8csaEsHg15H+zo+PPPtXn49HMcLS/fzyaPFS4KuXM/gtR/VPzpGdQsr1dYW/3dPOAfPX+XwhRSeX7qPFc90zjeWa+lZTFurVk8e07Nenp4vrWm5D1pJebg4MrZXfd785ShzNp5gcKuauDje+uPyo+zen4j2Faf3BzSeBL1ixQomTJjA5MmT2b9/P927d2fAgAHExOTfLTx//nwmTZrEtGnT+Oeff5g+fTpjx47lp59+MrfZsWMHERERDB8+nAMHDjB8+HCGDh3K33//XXYXotOpw1DWPnzqQu3O6qNWB/UctTrcOuZT1/pzFfMvm6FDh2IwGFi6dClff/01Tz75JDqdjq1btzJ48GAee+wxWrZsSd26dTlx4oTV5w0PDycmJoaLF28lgjt27LBo89dffxEaGsrkyZNp164dDRo04OzZsxZtnJycMBoLX6kSHh5OVFQU16/fmjP1119/odfradjQuvkBQuRW3IKA6VlG4lPTOHEplZ2nE5n046EC59EAXL6WgUEH7etUZ0LvBnw/pjNRU/uy8MkOvPtgC3TcSlhyFCeByW9jzuJsjpmzrcLtm4L2auzP58PVmjzr/4nj+aX7yMiy/o+IKWsOczk1nfr+1XipbyOrX5cfF0cD8x5ti4eLA/tirvLOr/lvozH7t+PEpaQR6utmlyuRcvZBK+g7qqNs9kErrcc6hRKcvX/btztu/d626P25w/7+vQujaQ/Q7NmzGTlyJKNGjQJgzpw5bNiwgfnz5zNz5sw87b/99lueeeYZIiIiAKhbty47d+5k1qxZDBo0yHyOPn36MGnSJAAmTZrEli1bmDNnTpGrmqqCatWqERERwX//+1+Sk5N54oknAKhfvz4//PAD27dvp3r16syePZu4uDiaNLFuvL537940atSIESNG8MEHH5CSksLkyZMt2tSvX5+YmBiWL19O+/bt+eWXX1i1apVFmzp16hAdHU1UVBS1atXCw8MDZ2fLpa/Dhg1j6tSpPP7440ybNo3Lly/zwgsvMHz4cPPwlxDWsmYl1rhl+wnzO05qWhZXb2ZyI6P4y8m/eLw9dzb2z3O8oGGowGJsR1FYL05p9Wrsz+cj2vLMt3vZ8M8lxi7dx6ePtsHJofC/n386cJGfD8Zi0OuYPbSlRY9BSdX2deP9h1ryzLd7+WpbNO3rVLf49zl8IZnF26MBeGNwM5u8p63ZotdOCy6OBib0ach/vj/I3M0nqFvDnWvpWXy+RS1UGdE+pNBhSXukWQ9QRkYGe/fupW/fvhbH+/bty/bt2/N9TXp6Oi4uln/luLq6smvXLvPy5h07duQ5Z79+/Qo8Z855U1JSLB7lxiMQer5WpjWAbjdy5EiuXLlC7969qV1bHcudMmUKbdq0oV+/ftxxxx0EBgYyZMgQq8+p1+tZtWoV6enpdOjQgVGjRvHWW29ZtBk8eDAvvvgizz//PK1atWL79u1MmTLFos0DDzxA//796dWrFzVq1Mg3aXVzc2PDhg0kJSXRvn17HnzwQe666y7mzp1b/H8MUeUVtRILIMOocOzSNS4mp5mTH70Oqrs5UsPDuto0qWkFz/np3yyIba/eabElxLZX7yzWXlwF9eLYQq9G/iwY3hYnB7U689gieoLiU9KYskatvzO2V31a1PK2WSz9mgYyunsYAK+sPMip+GvsOJXIqv0XGL9sPyYFBrUMtnqlmRZs0WunhQfa1CLI04Xkm1mM/HoP45dHcSRWvV+GBxVvLqc90Cm22GilBC5evEjNmjX566+/6NKli/n422+/zddff82xY8fyvOa///0vixYt4ueff6ZNmzbs3buXgQMHEh8fz8WLFwkKCsLJyYnFixfz6KOPml+3dOlSnnzySdLT0/ONZdq0aUyfPj3P8eTk5DwTdNPS0oiOjiYsLCxPMiaErcjPWflZ+vdZ/ptdLK8wz/asx4DmgXi7OuHl5oiHswN6vY4dpxJ55IudRb5+2ehOdrGqpzS2HL/M6G/2kJFloneTAOYNy9sTpCgKo77ew6Z/42ka7MnqsV1tXn0502jikQU72XP2Cg56HVm56gPpgHfub05EB+vqHGmpNJPutbD+cCxjvtuX73M6sIvkLSUlBS8vr3zv37fTvBDi7RP7CpvgOmXKFAYMGECnTp1wdHRk8ODB5iGc3Kt9inNOUIfJkpOTzY9z586V8GqEEBVFRpaJL7ee5o2fj1jVvkfDGrSo5U1tXze8XB3RZ9+oKuqcjpLo2bAGX45oh7ODno1HL/Hckr2kZxkt9jF759d/2fRvPE4GPbOHtiqTrSccDXoeaKuuOM26rTiiArz246Ey28jVlsqy187WcoaKC1MWe8eVJc3mAPn5+WEwGIiLs6yAHB8fX+A8DldXVxYuXMjnn3/OpUuXCAoKYsGCBXh4eODnp5boDgwMLNY5AZydnfPMMxFCaKOs/ypWFIUN/1xi5q9HOZuoFjC9vRcht6KWklfUOR0l1aNhDb58vB2jvt7DxqPxPDB/Owmp6cSlWPawD2wRRKNAjzKJwWhS+HhT4Ys07KmYYGVgbdFOe6lfZA3NeoCcnJxo27Ztnu0SIiMjLYbE8uPo6EitWrUwGAwsX76ce+65B71evZTOnTvnOedvv/1W5DmFENorya7mtytsV/XDF5J55IudjPluL2cTb1DDw5l3H2jBxw+31nwlVkXSvUENvnq8PQ56HYcvpORJfgBW779QZr0wxbkZC9uoiPWLiqLpKrCJEycyfPhw2rVrR+fOnVmwYAExMTGMGTMGUIemLly4wDfffAPA8ePH2bVrFx07duTKlSvMnj2bw4cP8/XXX5vPOX78eHr06MGsWbMYPHgwa9asYePGjWzbtk2TaxRCWKc0m4LmPkd+tXwm3NWAPWev8P2+8ygKODvoebpHXcb0rGcuzDdfb78rsexR53q+eLo6knS94FphZdULUxlvxvauItYvKoqmCVBERASJiYnMmDGD2NhYmjVrxrp168y7lcfGxlrUBDIajXzwwQccO3YMR0dHevXqxfbt26lTp465TZcuXVi+fDmvv/46U6ZMoV69eqxYscKq7RyKQ6O546KKqGo/X0UtRbdmf6TCihG++uOtCuCDWwXzn/6N8xRss0UCkzOnoyrYFZ1UaPJTlkMilfFmbO9y5rqVtOq4PdK8EvRzzz3Hc889l+9zt+9c3qRJE/bv31/kOR988EEefPBBW4SXR0514Rs3buDqWrFqHoiK48YNdW7K7dWsKytrhzS+2XGGzvV88XBxxMPFgWpO6kqswhKoHI4GHUtHd6J9nYJ/QVelBKa0tOyFqYw3Y3tXGee6aZ4AVTQGgwFvb2/i4+MBtSaNPe0zIyo2RVG4ceMG8fHxeHt7W6xurMysvUnevgpFp4Nqzg44G/QkFNIbAZBpVMgyVq2etbKkZS9MZbwZVwS2KNppTyQBKoGcncpzkiAhbM3b29v8c1bZpWcZOXDuqlVtAz1dyDSaSE3LIsNoQlEgNS2LVCvfS+aE2I7WvTCV7WZcUVSmuW6SAJWATqcjKCgIf39/cwVqIWzF0dGxSvT8ZBpN/LjvPB9vOsmFqzcLbZtzM9326p3mX7RpmUZS0jJJTcti+6lEpqwuupihzAmxHXvohalMN+OKpLIMFUsCVAoGg6FK3KiEKI6i6vgYTQproi7w0aYT5jo8/h7O3NnYnxW71SKk1txMXRwNuDga8PeAOr7uzNt8UuaElDN76IWpLDdjUf4kARJC2ExBy9CnDgqnb3ggvxyKZc7G45y6fB0AX3cnnr2jHo91CsXF0cAdjWqU6GZqD70RVZX0woiKSrO9wOxZcfYSEUKoClqGnpOQ1PR24cJVNbHxcnXkmZ51ebxzHXMdnhylqQRdWAImc0KEqPyKc/+WBCgfkgAJUTxGk0K3Wb8Xuat6NScDo3rU5aluYXi6lM0S/4q2waQQwnaKc/+WITAhRB7FTSKKquOT48OHW9EnvGxXt8mcECGENSQBEqISKq9hJKNJ4WhsCiv3nLPq3DcyjNZfhBBClCFJgISoZEozD6ao/bg+ebQ1wd6u7IpO4u/Tiew5c4XU9CyrY5Nl6CJfqXGwZxG0exI8Klj9q4ocexUnCZAQlUhpNhQtaj8ugOeX5t2KppqzA21Dvdl39mqByVCVWoYuN8TiS42DLe9AowEV79+sIsdexUkCJEQlYU0C89oPh7h8LZ30TBM3M4zczFQfaZlGYhJvWDWPp5qzgS71/OgQ5kOnur40DvTAwaA3J1+53w+q4DL00twQq2zyVIHX4qRd1ToCUUKSAAlRSVgzEfnqzUymrP6nVO/z5pDmDGldM89xeyiKpzlFgWtx6udZaerXxdkrsCr1JqTGqY+sNFj7gnrs78+h4zPq5x6B9vtvcO5vOLwKov+E+Oz/Tyc33nrenmMXZpIACVFJWLvPVfOantStUQ03J7WSsmv241JqGt/tjCny9QGeBc/jqZJF8UwmOLEB/sm+IabGqscX9gO9Azh5gFt1cK8BLt7g6l3wx9RL2SctYY9IRepB2rNITfZyO7BUfQD0fA16TSr/uPKjKBB/BI7+BEd/hkuH8rb5/Q31AfYVuyiQJEBCVAI3MrLY/K91m/P+9+7wfJeJG00Km47Gl3o7iSqxDN2YCWe2qTfEf3+51etzO1MWpF1RH0mnrT//wgHgXRu8Q8CvMQS3At+64FNPTZQKUpGG31oNg9Nb4NwOMDiBMUM97hEE938Bfg3KPgYo+LpNJri4D46uVb/PFt8/vfo9Ceuhxv7nu+rhoNZw93vq903YPUmAhKjAFEVh3aE43vrlCBeLGP4qKoGR7SSyFXRDzLwJpzarN8Nj6yznfji6Q+1O6g3R0Q1+fQUGvAvVQyEtFQyOoNOrr7l5NfvjFfXzi/vgyhnLGLJuQsIx9ZF7aAXAzQ9864NvPfXhU0/92qdu6a+7vIbfjJmwYZKa/Di4Qv+Z8PMEqBao9qDtnA8PLynbGHLkvm43Pzj7163ENvXirXYGZ6h3JzQZlN02+//RxSg1AXJwg9j9sHcRDP60fGIvrYrUY1gGJAESooI6cSmVqWv/YfupRABqertyT8sgFmxR/1ItSQJjN/N4tPzFnPuG6OgGJ35TewFObITM67fauflC44HQ5F418XFwVo9fjFI/hnRUewmseb/U7B6k2APw0zjoOQmc3CD5HNxIhJRYSDoF1y7BjQT1cW5n3nM5e6kffxwNHsHg4qEmFV41Cx52c/YCvb74/04lZcyCH0bBvz+rScUjS8E1O5noM12dD3TsF9j+CXQdV/bxKCb14x/vqHN7bibdes6pGjTspyY99XuDs0fB5+kzHda/BlFL1MS0+0tlG7ctVKU5Z/mQBEgIO1VQMcOUtEw+2niCr7efIcuk4OygZ0zPejx7Rz1cHA20DvEuVQJjF/N4tPzFnJasfvz1VbV3JmdoBsCzpnozbDIIQjqBwQa/QvObMNuof/7JU1qKOhSTePLWx9zzjtKzY084rj6s5VRNHcoB+PenwmMrDZMRVo+BI6tB7wgR36m9Kqlx6ryZundA/3fgl4mwcRrUagehXWz3/jlykk5jJvw8Xj12/Ff1o7MnNOgNLR6GsJ7gWETtKo9ANfbwe9VE8peXYNMMqB4Gze63fezCZiQBEsIO5VfMMNDThX5NA/jlUBwJ19IB6BsewJR7wgnxcTO3s0UCUyXm8eSWfEHtkTj6szoEArd6WLxC1ESs5SMQ3LroVV05N8SySNxcPNXEKHdylHMzT0uB05th22xoPRxcq0P6NVCM6o3eYvgt+2PmDfUcGddune/P99UH2HYyr8mk9u4cWqlODh/6NTTsqz7nEXjrfdo9BTE74dD/YOWTMGYrVPO3TQw58puAnSM9BXwbqj0/1sgde/tRkHgKds6DVWPUn52Q9raJOT/F6SnNuK4mzef3wKXDcPU8XMme1xR74Fa7KrSCTTZDzYdshiq0VFAxw9zq+rkz9d6m9GxYo9ziKnPJF+H0HxC9Rb2RX7ukDjs0fQD8m4BnkO1/MSdFq0MWOz+DjNSC25XXqp7SDv1djIIFPeHpLdYNv12JUYfW0lPhwj7468Nbz/k1gLtnQ90exY/jdiaTOsdn39egM8BDiyB8cMHtM67DF3fB5aNQpzsMX22b3rYcSWfgx1FwfrfaE2XKhEEfQ1BL9fnSJAEmIywfpvYoudeAUZvUuWBl4fbvd1a6Op8s8VR2L+Gp7M9PWc5nKkxF+VkvgGyGKkQFVVgxwxweLg78Mq47rk6GcourzBgzsyed/gwHlln2RIA6AThnEnBgC+j1X3WYxNG15O+ZeVOd5Lr/W3X4KIezBzToq04o3jIr7w2xPOTuTSgP1WurD1BXnf31Idw1FbZ/DAknYMkD0GsydHkB9CX8eVMU+PU/2cmPHu5fUHjyA+DkDkO/gQV3wJmt8MfbcNf/lez9b5dxQ51ndX63Oser39tqchbU0rqksSh6AzzwJSwaAHEHYelQGPkbuHiV/ty5KQrE7FA/X/cyXItX54zlzGnKj2t19fvsHqCuVDu+HlIuQEBzGPwJoCu/n3U7mH8kCZAQdsSaYoapaVlEnbtq30NUhf11l5mm9vDkrKa6eeXWcw5uENpJ/SW9d7E65HT5uDr5OO4gLHtYXTVU7071F2fD/lAtn16w/N7/YpSa9BxaeWueDzo1oWozHBoNVOd7XIxSEyBb3RDLky2G3+rdCa0ehZ/GqzfIjVPVFVH3faZO7i0ORYENk2H3F4AOBs+D5g9a99oaDeHej+GHkbD1A3VSubXDUgXJuA5LI9SkytEdHvteTYJszbkaPLoCvrgTLv8L/3schq1UVwOWVmpcdm/dR7eGac/vvvW8o5vae5ezOtC8UrDerZVrOQ53g++fVOsaJZ2GZg+UPj5rGLPUSecakwRICDuRlmlk3SHruqmtLXqomdv/uktLyV5N9ROciMy7mqrR3epqqro91dVUF6PUBOieOeAfDme3wb/r4NivkHJeXSV07BdAByEd1Nc3ulu9aeZ+/zrd4Mha2P8NxOUqXucVotahaT1MTbYqi9L0IOVOnjwC4ZHl6vDgr6/B+V0wv6u60qn9aOtWjSmKOpF5Z/aS8EEfQatHihdT8wfVlVm7FsCPT8Mzf5Z8OCn9Gix5CGK2q8UpH/teLV2QMwHb1r0QnsFqErRwgJrwr3sF7vmweJXBb5eVrq6gO7O14DZdXlB7Sq2Ru3TCuv9A2B3gXoZ/WOXMV9v9xa1J57mTt3KefyRzgPIhc4CELRS0iut2Jy6lsnRXDD/uu0DyzUyrzr1sdKey7wEqzRh9ztyEHq9CbJR6A7h9NVXje9TVVLU7553fUdBcFkVRE5lj69RH7smboP7F22iAuqx60/Rb8ztAXeXU+B61tyesZ8FDOlW8NkoeV8/BmrHq3CxQ5+QMmVd04rj5bbUnDWDgB+oE4ZLISleHky7sVXsEn9pwq+SAtdJT4bsH1R4TZ0947MeynZyc27FfYdkjgAJ931QTlJI4/Qf88jIknlC/Dm6jJv2b3yz5/KXUONj1pboqL/EENH9IHb4rK5tnFjz5HGwy/6g4929JgPIhCZAorfxWcQXlWoqu9vbEsmxXDLvP3BoCCvZyISUti2vZu6rX4ArDHDaxJOsuLlPdXMxw26t3Fr2qqywm1CqKOnx0LV6dpHw9/tbnV85C8nm1jsq1S+pNJ7fqdaDpfWrSE9ym8L+ErY09+bx6gzn2q1pVWMlnN3qfeurwVuth4N+4mP8IAlAnMe/5CiL/T1055uQB/d9WV5vl93388z34/U318/7vQKdnS/f+V8/B593V4dL2o9SEylppyWryc36XWvNo+Cqo1bZ08RTXjnlq4Ud06tL/JvdY/9rUS7Dhv3D4e/XragHqvKVmD6h/ABRn0ntBLuyFL3ur84ceWaGWYSgLF/bBN/eqvxvqdFOrqdtq8nk2SYBKSRIgURoFreLKqa7cq3EN9p29au7tMeh19G7izyMdatO9QQ0ij8SZd1UP10Xzi/NkBqa/xRElDID5j7WxriBhcVYEZdywTGauXYLYQ7BvMYR2UzeszHnOmF6Mf41cer5qfdd8SUROs1zFlOf9y2l1S2WWeApWP3dr7kmDvuoNzDPoVtKqGNUECKDPDOg63jbvfWIjLHkQUOD+L6HFQ0W/5uZV+O5+9Qbv4g0jVqu9SOVNUdSJyru/VOfoPLmu6DhMRtj9lbq/WHqKOoG8/Wi4c/KtCdXFXfVXmN+mqJPfPYLguZ2Fb7lSElkZsPhudcgrqBXc/T581ds2seciq8CE0Ehhq7hyjm3+9zKgVm5+pEMIQ9uF4J9rg9Hc1ZidU9RhIy+u09gznZf7NeKuMEe4nlB0MDlbNSQcu5XUXMud5OT6WNgS8LPb8h5z9lJrs5gfAerKLIOTOqfn+mX1JlieK6k6jYGmQ9TPcyoqa7GSqzLzrafevHd8qvbwnPgN5nVSb2Z+9S2HN+583XbJD6jFCXu8om478dM4CGxeeI/ejST49j51CNa1OoxYc+tnobzpdNB/lrpE/eRGWPowjN4EXrXyb39hL/w8UY0d1B7Te2bnTZpsWXOq13/Vye5JpyByCtz7SenPmdvGaWry4+yl1oC6edW25y8B6QHKh/QAiZLacSqRR77YWWS71wY05unuddHnN4x1JQaOrEE5vh7l7Hb0FLKs1ZYcXNQEJj2l4DYdx0DvaUUvQ7flX6YlofX7VwXx/8KqZ27dpP2bQvw/6udl1dtnMqo9Oqf/AL+GMHqzuuLqdjeS4JvB6spBN181+Qlsbvt4iistBRb2U3eWD2gGT61XV6blDPc6uKhVpPcsBBQ1Wej9f9D2yZKXISiOM3+pvTSg/pvVvcM25z36E6x4TP384aXqFjJSB0gI+2btRGajSeHg+at8s+OMVecN8nKxTH4URe21OLAM9n4NWTfRcWsPL5sJaA4N+qg9Njk9Nzm9OM6eao/Q7ftS3d6LUpoaPKLy8G+szmfZNlv9mc1Jfur3VieiX4yy/aoevQEe+Ao+665u9fHTOPXr3POQrieoyc+lw+rmpo//BAHhtouhNFw8s5fH36XG9/1Tag/OlnfUa9u1QO09BXUrjr5v2L4KdmHqdFXnWO3+EtaOg+d2qDWZSiPpNKweq37e+Xk1+YHyr3mVD0mAhChAUROZL6Wk8efxy2w5fpltJxO4esO6FVwA/h7ZQ16pl9SS/1HLbt1AQF3F1KAPVK8LW2ZmJyEt1OeqFXJTSY2DazkJzMH8E5jCbkj5PV+SejhluR1ERXj/qmL/d9m9FbnkLl5ZFvOu3P3UStKLB8LhH9RVhE0Gqb0JTe5VKzzHHwF3fzX5sbeJ79611RIDiweqQ4hklxTY/Jb60a+ROsk7rLs28fWeBsc3wNWzsOkNGFCKej2ZaWoNpPRktY5T72m2itImZAgsHzIEJorajqKmtwsXrlrW4vFwcaBbfV+2n0oyT3DObxVXiKeBzfemYTiwVL1RKEb1BAYn9a+jlo+qxegMDqUbyintMJAMI4mi5LeTvY1X9RRox6fq6ii9ozpfZfUY8K4DV8+ofyQ8/tOtulD2JjUODq5QV9Xl0DtC2yegRYRapVnL5P3kRvjuAUCnlh2o3bFk5/n5RTVBdvWBMdvAq6ZNw8yPDIEJUQrWbEeRk/y0DPGmZwM/ejaqQcta3jgY9ObkCcBfd5UJDj8SaWxDTRJ5wLCVCGUXhu+Tb52sZju18m6z+9XJmvZCelFEUWzVY1gSnZ5Tt4I4+pNarRrU5McjCB7/WZ2Uba/y24zVlKkWCNz9hfYrFuv3VguFRi2Btc/DM1vVKunFcXBldu+gDu7/olySn+KSBEiI21izHQXAguFt6ds0b3KQexVX9RR1ddVnjnMI0Wev3MoEPIKh5cPqDuOF/ZVamiSktAmMHYzRC1Gga5fUSfkX9qn7WYG61P3u99U95VLj7Dd5b/ekOk8K1N3Z171kfysW+72l9gQlHFcLWvaeav1rLx9Xt1IB6PGyuoLPDkkCJMRtrN1m4mamMf8nUuPor9tJ39AV6hi/AiH6BBSDE7qwHmrS0/Q+61Z1lHZrA0lgRHkp7x7D/HpR0q7CimHq51r3ohRGy54za7lWV+cirXhM3XcsfLB18WXcgP+NULe7qdMd7rDT7wGSAAmRh4ezdf8tzBOZcyiKukfPT+Mh6XTO1EYznTFD/YuqZrvyWdIqRHkq74Q7dy/Kub/VHeftrRelomsyCMKHqFtlrH1eLTtQ2IauigK/vASXj6orTB/4yq5/10kCJEQuV29kMGfTiULb5GxH0SEse2dlYyb8sxp2fGK5N1Wd7mqy89eH8otZCFurCL0o1rD3uXZ3v6fuAxd3CP6aoxajLMj+7+DAUrVq9QNfgUdAuYVZEpIACZEt4Vo6w7/axdHYFNydDFzPMJq3r8iRU21k6qBwDBmpsO8b2Dlf3aEcwMEVWj+m7n3kW09dSfXXhxXzF7MQouzZ+1B1NX8Y8C78OBq2vAuNB+VfWiDusLrdB0Cvydot4y8GSYCEAC6lpPHoFzs5dfk6ftWcWTKqI9EJ15j+0xGykmPNS9kdvIJ4+y4fel2cBz8tvlU12b0GdHgG2o8ENx9Nr0WIKsfee1EquuYPwaHv4cQGdSjsqQ2WQ1tpKeq8n6w0qN8Huk3ULtZikDpA+ZA6QFXL+Ss3GPbl35xNvEGQlwtLRnWkbg21vL7RpPDPnj9pse5ejnf7iPop29Ef/gFM2buO+zVUq5u2iMh/mWgZlXsXQohylXxB3fctPUXdjb5zdnVnRVErWv/zI3jWgjFbNf0jUOoACWGl6ITrDPtiJxeT06jt48aSUR0J8XEzP2/Q62jBcQAabsu1sWOd7mri06Av6G+f7pyLvXdvCyGENbxqQp8Z8PMEtUJ0rXZw8ne1gOs/P4LeQa3QXYF6wCUBElXW8UupDPvyby6nplOvhjtLRnUi0Ctni4o4SI2FnZ/BweXZr9BBvV5qb0/dO6RHRwhRtbR9Qt1+5MxWWPcqxO4HXXYa0WcGhHTQNLziKuRP1/Ixb948wsLCcHFxoW3btmzdurXQ9kuWLKFly5a4ubkRFBTEk08+SWJiovn5xYsXo9Pp8jzS0qyr7SKqhsMXkon4fAeXU9NpHOjBimc630p+AHYvhAV35Ep+ABQ49bu6A/aeReUdshBCaEung3s/Bkc3NfkBULKg8T1qZe4KRtMEaMWKFUyYMIHJkyezf/9+unfvzoABA4iJicm3/bZt2xgxYgQjR47kn3/+YeXKlezevZtRo0ZZtPP09CQ2Ntbi4eJSzDLeotLaF3OFR77YyZUbmbSs5cXypzvhV835VgOTERJzLYVvEaF+HPSxui/W01vUOT1CCFGVpMapE57bPXXrmJuvWpE79sCtfeEqCE2HwGbPns3IkSPNCcycOXPYsGED8+fPZ+bMmXna79y5kzp16jBu3DgAwsLCeOaZZ3j33Xct2ul0OgIDrR+eSE9PJz093fx1SkpKSS5H2CGjSWFXdBLxqWn4e7igKAqjv9nD9Qwj7etUZ+ET7fFwyVXYy5ip9vD886Nay2LIZ1CjkbpxoSxlF0JUZflV376RCF/fo35uz9W386FZApSRkcHevXt57bXXLI737duX7du35/uaLl26MHnyZNatW8eAAQOIj4/n+++/Z+DAgRbtrl27RmhoKEajkVatWvHGG2/QunXrAmOZOXMm06dPL/1FCbuy/nAs0386ku++Xt3q+7FgRFvcnHL9F8hKh5VPwrFf1Al9D3wFTYeotXyEEKKqy119++J+dUJ0BS7yqtkQWEJCAkajkYAAy0qRAQEBxMXl343WpUsXlixZQkREBE5OTgQGBuLt7c0nn3xibtO4cWMWL17M2rVrWbZsGS4uLnTt2pUTJwqu7jtp0iSSk5PNj3PnztnmIoVmcnZkL2hT06HtalkmPxk3YNkjavJjcIaHl6rJD0iNESGEAPV3YHCr7Ed2p0JOz3hwqwr3O1LzSdA6nc7ia0VR8hzLceTIEcaNG8f//d//sXfvXtavX090dDRjxowxt+nUqROPPfYYLVu2pHv37vzvf/+jYcOGFknS7ZydnfH09LR4iIrLaFKY/tMRCipwpQNm/vovRlN2i/RUWPIQnNqkTu4b9j9o2O/WC3KWslew/9xCCCEKptkQmJ+fHwaDIU9vT3x8fJ5eoRwzZ86ka9euvPKKuhdJixYtcHd3p3v37rz55psEBQXleY1er6d9+/aF9gCJymVXdFKBPT+gbm0Rm5zGrugkOgcbYMmDcH43OHnAsJUQ2rn8ghVCiIqoEvSMa9YD5OTkRNu2bYmMjLQ4HhkZSZcuXfJ9zY0bN9DfVnTOYFDLcRdU0FpRFKKiovJNjkTlFJ9qXcmDqwmx8PUgNflx8YbH10jyI4QQ1qgEPeOargKbOHEiw4cPp127dnTu3JkFCxYQExNjHtKaNGkSFy5c4JtvvgFg0KBBjB49mvnz59OvXz9iY2OZMGECHTp0IDg4GIDp06fTqVMnGjRoQEpKCh9//DFRUVF8+umnml2nKF/+HkWXPKjBFXpunwrJJ9R9vIavhsBmZR+cEEIIu6BpAhQREUFiYiIzZswgNjaWZs2asW7dOkJDQwGIjY21qAn0xBNPkJqayty5c3nppZfw9vbmzjvvZNasWeY2V69e5emnnyYuLg4vLy9at27Nn3/+SYcOFatCpSi5DmE+BHm5FDgMVpMElrvOxC05FjyCYcQaqNGwnKMUQgihJdkMNR+yGWrF9/X2M0xd+0+e46G6Syx1eouaugTwrg2P/wTV65R/gEIIIWxONkMVVd7es1cAcHbQ45mVyDCHTfxlbMo857nU4Ar41ocRa9UN/oQQQlQ5kgCJSuffuBR+OngRgJVjOsPFA7RY9yNjXTfhmJkM/uHqsFc1f40jFUIIoRVJgESl88Fvx1EUGNg8iBa1vCFBTYYcM5MhqBUMXwVuPprGKIQQQluaF0IUwpaizl0l8sglAnRXeK1VOpzdDuteVp+sHgb93oKrMRVu0z4hhBC2JT1AolL54LdjAMyouZuQlWMtn7wSDYuz942rYJv2CSGEsC1JgESlsfN0IltPJOBo0NFs8ATIGKLu75V1U21QgTftE0IIYVsyBCYqBUVReH+D2vsT0T6EmiFhcOxXNfnxa6w2qsCb9gkhhLAtSYBEpfDH8cvsOXsFZwc9L9zZAJKiYc9C9cmOz2gbnBBCCLsjQ2CiwjOZbvX+jOgcSoCnC/zwFpgyod6d0PhuuHZJen2EEEKYSQIkKrz1/8Txz8UU3J0MPHtHfYg9AIdWqk/2nnZr0z4hhBAimwyBiQrNaFKYHXkcgJHdwvBxd4KN09Unmz14a9KzEEIIkYskQKJCW73/Aifjr+Hl6sioHnUh+k84tQn0DnDnZK3DE0IIYackARIVVkaWiTmb1N6fZ3rWxdPZASKnqk+2ewp86moYnRBCCHsmCZCosP635xznkm7iV82ZJ7rUgSNr4OI+cHSHHq9oHZ4QQgg7JgmQqJDSMo188vsJAMb2qoebAfj9DfXJLs/LRqdCCCEKJQmQqJC+23mWSynpBHu58GjH2rD/W0g8CW5+0Pl5rcMTQghh5yQBEhXOtfQs5v1xCoDxvRvgbEqHP95Rn+zxCrh4ahidEEKIikASIFHhLNoWTdL1DML83HmgTS34ez5ciwPv2tDuSa3DE0IIUQFIAiQqlOQbmSzYehqACb0b4JB+FbZ9pD7Z63VwcNYuOCGEEBWGJECiQvn8z1OkpmXRONCDQS2CYdtsSE+GgObQ/CGtwxNCCFFByFYYwu4ZTQq7opM4dfkaX26NBmBin4boU87D3wvURr2ngl7yeSGEENaRBEjYtfWHY5n+0xFik9PMxxwNOowmRZ34bEyH0G5Qv7eGUQohhKhoJAESdmv94Vie/W4fym3HM40KHy5dS3/npegA+kwHnU6DCIUQQlRUMmYg7JLRpDD9pyN5kp8crzisQIcJpfEgqNWuXGMTQghR8UkCJOzSrugki2Gv3NrqjtHHsBejoiOq4bhyjkwIIURlIAmQsEvnr9wo4BmFVx2XA7DCeAcx+prlF5QQQohKQ+YACbtyNvE63+w4y9K/z+b7/F36fXTQHyNNceSjrAeY4+FSzhEKIYSoDCQBEmUuZxl7fGoa/h4udAjzwaC/NWnZZFLYcuIy32w/wx/HL6NkT/wx6LNXe2XTY+I/DisAWGTsj94rmA5hPuV6LUIIISoHSYBEmcpZxp6VHMswh028mXUXDl5BTB0UTud6fny/9zzf7jjDmcRbQ153NKrB453rcDPDyNil+wBQgPv022ikP89VxZ3PsgYxa1C4RSIlhBBCWEsSIFFmci9jb6q7ygSHH4k0tuWf5OqM+W4fTgY9GUYTAB4uDgxtF8JjnUIJ83M3n2O+vg3TfzqCITmGqY5fA/CdwwPMeqgH/ZsFaXFZQgghKgFJgESZKGoZO0CG0URD/2o83rUOQ1rVxN05749j/2ZB9AkP5NLKiXgevUmGix/PvjgLg7Nb2QUvhBCi0pMESJSJXdFJZCXH0lR3FYBHDJsA+I/DcuIVb0zoScGN/mGNCEmvBrsdQGcAvQPoDaDTmz836AwEn14JgFOHkSDJjxBCiFKSBEiUifjUNIY5bGKCw48Wx3saDlk2PPBr8U7sGQQXo9TPPQLVhxBCCFFMkgCJMuHv4cKbWXcRaWzLSw7/407DAQD+MLYgSfHAARNpONIjvBaB1RzAZAJTFihGMBnVz+OPQuIJyxP/POHW5z1fg16Tyu+ihBBCVBqSAIky0SHMBwevIK6npNFTf9B8/L2sCP5RwtABgV4uPPDwnVDQSq7UOPUBEHsAfhoHgz6GoJbqMen9EUIIUUKSAIkyYdDrmDoonJTlH2HQKewyNqKD4RgAOenO1KKWsec3xBXUEoJblUnMQgghqg5JgESZqWNIpJ5hGwDzs+5lu3KKeMWbQC8Xpg4Kl2XsQgghNCMJkCgzCevfobHOyL9u7Xh6+LPEp6bxcT6VoK3iEajO+ZFhLyGEEDYgCZAoExfOnqT91V9BBy53vUbner6lO6FHoEx4FkIIYTOyG7woE+d+nomzLoujzi2o07aP1uEIIYQQFjRPgObNm0dYWBguLi60bduWrVu3Ftp+yZIltGzZEjc3N4KCgnjyySdJTEy0aPPDDz8QHh6Os7Mz4eHhrFq1qiwvQdzmyqVztIpfA4Cp28saRyOEEELkpWkCtGLFCiZMmMDkyZPZv38/3bt3Z8CAAcTExOTbftu2bYwYMYKRI0fyzz//sHLlSnbv3s2oUaPMbXbs2EFERATDhw/nwIEDDB8+nKFDh/L333+X12VVeafWvoOLLpN/HRoR3nWQ1uEIIYQQeegURSlsu6Yy1bFjR9q0acP8+fPNx5o0acKQIUOYOXNmnvbvv/8+8+fP59SpU+Zjn3zyCe+++y7nzp0DICIigpSUFH799VaF4f79+1O9enWWLVtmVVwpKSl4eXmRnJyMp6dnSS+vSrp5NR5lTjPcSOfvzp/Rsd8jWockhBCiiijO/VuzHqCMjAz27t1L3759LY737duX7du35/uaLl26cP78edatW4eiKFy6dInvv/+egQMHmtvs2LEjzzn79etX4DkB0tPTSUlJsXiIkjmxdhZupPOvvh5t7xqqdThCCCFEvjRLgBISEjAajQQEBFgcDwgIIC4uLt/XdOnShSVLlhAREYGTkxOBgYF4e3vzySefmNvExcUV65wAM2fOxMvLy/wICQkpxZVVXVnXkqh3egkAsS2ex8HBoHFEQgghRP40nwSt01nWg1EUJc+xHEeOHGHcuHH83//9H3v37mX9+vVER0czZsyYEp8TYNKkSSQnJ5sfOcNponhO/fIB7tzkOLXpNGC41uEIIYQQBdKsDpCfnx8GgyFPz0x8fHyeHpwcM2fOpGvXrrzyyisAtGjRAnd3d7p3786bb75JUFAQgYGBxTongLOzM87OzqW8oqpNSUum5r+LADjRaAwNnR01jkgIIYQomGY9QE5OTrRt25bIyEiL45GRkXTp0iXf19y4cQO93jJkg0EdZsmZy925c+c85/ztt98KPKewjZj1H1NNuc5pJZhOA5/UOhwhhBCiUJpWgp44cSLDhw+nXbt2dO7cmQULFhATE2Me0po0aRIXLlzgm2++AWDQoEGMHj2a+fPn069fP2JjY5kwYQIdOnQgODgYgPHjx9OjRw9mzZrF4MGDWbNmDRs3bmTbtm2aXWell3Gd6ge/AGB/nZE84OmmcUBCCCFE4TRNgCIiIkhMTGTGjBnExsbSrFkz1q1bR2hoKACxsbEWNYGeeOIJUlNTmTt3Li+99BLe3t7ceeedzJo1y9ymS5cuLF++nNdff50pU6ZQr149VqxYQceOHcv9+qqKS5vnE2BK5owSQPt7ntY6HCGEEKJImtYBsldSB6gYMm+S8k44nsYklgS8zLBnp2gdkRBCiCqqQtQBEpXDlb++wtOYxHnFj5Z3jyn6BUIIIYQdkARIlFxWOvq/PgLgN++HaRZaQ+OAhBBCCOsUOwGqU6cOM2bMKHC/LlF13Nj1LV6Z8VxSvKnf/1mtwxFCCCGsVuwE6KWXXmLNmjXUrVuXPn36sHz5ctLT08siNmHPjJlk/fkBAKtcH6R745oaBySEEEJYr9gJ0AsvvMDevXvZu3cv4eHhjBs3jqCgIJ5//nn27dtXFjEKO5S5fzmeaRe5rHhS865nC620LYQQQtibEs8BatmyJR999BEXLlxg6tSpfPnll7Rv356WLVuycOFCZHFZJWYycvP39wD4n8Ng+repq3FAQgghRPGUuA5QZmYmq1atYtGiRURGRtKpUydGjhzJxYsXmTx5Mhs3bmTp0qW2jFXYCdPhH/G8cZYrSjU8uo/B0SBz6YUQQlQsxU6A9u3bx6JFi1i2bBkGg4Hhw4fz4Ycf0rhxY3Obvn370qNHD5sGKuyEycT1jbPwAJbo7uHJzo2LfIkQQghhb4qdALVv354+ffowf/58hgwZgqNj3k0vw8PDefjhh20SoLAvytG1eKScIEVxRekwGndnTYuJCyGEECVS7LvX6dOnzVtVFMTd3Z1FixaVOChhh1LjYM9Cbh74ETfgW6U/D/dornVUQgghRIkUe/JGfHw8f//9d57jf//9N3v27LFJUMK+GE0KB48egy2zcLt6gmuKC0nNRlHDw1nr0IQQQogSKXYCNHbsWM6dO5fn+IULFxg7dqxNghL2Y/3hWLrN+p1Jqw6aj31n7ENILan7I4QQouIq9hDYkSNHaNOmTZ7jrVu35siRIzYJStiHzbsPMPfHrfhjZKThVwAyFANbjc1I/ukXQh2706t9S42jFEIIIYqv2AmQs7Mzly5dom5dy9ovsbGxODjIhNjKwmhSuPDrB/zsvMbiuJPOyBLnmQB8uX4oPdouwKCXIohCCCEqlmJnLH369GHSpEmsWbMGLy8vAK5evcp///tf+vTpY/MAhQbi/yXhtw950PgLZOc2KYornrqb/DfzKQ6Y6qnNFG+aRifRuZ6vhsEKIYQQxVfsBOiDDz6gR48ehIaG0rp1awCioqIICAjg22+/tXmAopwoCpzcBDs/hVO/EwCgg0OmOnyVdTfRSiBrnP+PA6Z6/KOEmV8Wn5qmWchCCCFESRU7AapZsyYHDx5kyZIlHDhwAFdXV5588kkeeeSRfGsCCTuSGgd7FkG7J8EjUD2WcQMOLoedn0HCseyGOhJD+jDmZCd2K40AHU110fme0t/DpVxCF0IIIWypRJN23N3defrpp20diyhrqXGw5R1oNAAUE+z6AvYugptX1OedPKDNcOjwNOuO69h94h/zS+MVb+Zk3U+84g2oI2OBXi50CPMp/+sQQgghSqnEs5aPHDlCTEwMGRkZFsfvvffeUgclytjvb8DpP8CUpX7tHQodx0Drx8hyrMZb646y6K8z5uY64DLVmZP1oPlrgKmDwmUCtBBCiAqpRJWg77vvPg4dOoROpzPv+q7TqTdCo9Fo2whF6aTGqQ+TEX5+UT12cqP6MbAldBgFrYaB3kBKWibPf72HP49fBuDlvg2pV6MaM34+Qmzyrbk+gV4uTB0UTv9mQeV9NUIIIYRNFDsBGj9+PGFhYWzcuJG6deuya9cuEhMTeemll3j//ffLIkZRGnsWqcNe+Yk7AMkXQG/gbOJ1Rn69h5Px13Bx1PPh0FYMaK4mOH2bBrIrOon41DT8PdRhL+n5EUIIUZHplJwuHCv5+fnx+++/06JFC7y8vNi1axeNGjXi999/56WXXmL//v1lFWu5SUlJwcvLi+TkZDw9PbUOp3RyeoB2fwX7v1GPDfoYgrILGHoEsvOyI2O+28vVG5kEerrw5ePtaFbTS7uYhRBCiBIozv272D1ARqORatWqAWoydPHiRRo1akRoaCjHjh0r4tWi3HkEqo/LR28dC2oJwa0AWLE7hsmr9pFlUmhZy4sFI9oR4Ckru4QQQlRuxU6AmjVrxsGDB6lbty4dO3bk3XffxcnJiQULFuSpDi3sRFoyyoW95snLB88n09jfxLvr/+XLbery9ntaBPH+Qy1xcTRoF6cQQghRToqdAL3++utcv34dgDfffJN77rmH7t274+vry4oVK2weoCi9fVvW0kYxccbkz2pTN5b8eI6UtRtIzzIBMKF3A8bf1cA8kV0IIYSo7IqdAPXr18/8ed26dTly5AhJSUlUr15dbqB2aP3hWOK3rqGNA2wxtTQvZSc7+RnVPYwJvRtqGKEQQghR/vTFaZyVlYWDgwOHDx+2OO7j4yPJjx0ymhSm/3SErnr1+/WXqVmeNr8cjMVoKtY8eCGEEKLCK1YC5ODgQGhoqNT6qSB2RSehSz5PPX0sRkXHDlPTPG1ik9PYFZ2kQXRCCCGEdoqVAIE6B2jSpEkkJclN097Fp6bR1aD2/hxQ6pGKW4HthBBCiKqk2HOAPv74Y06ePElwcDChoaG4u7tbPL9v3z6bBSdKx9/DhW7Zw19bTc0LbSeEEEJUJcVOgIYMGVIGYYiy0KGON8nZPUB/GfPO/5ENTYUQQlRVxU6Apk6dWhZxiDJguHwEH1K4rjizX2lg8ZxsaCqEEKIqK/YcIFGBnP4DgLMercm8LdcN9HJh/mNtZENTIYQQVVKxe4D0en2hS95lhZgdObUZgH0Gdd+vEZ1CaVunumxoKoQQosordgK0atUqi68zMzPZv38/X3/9NdOnT7dZYKKUstLh7HYAliXWB2BEl1Dq+3toGZUQQghhF4qdAA0ePDjPsQcffJCmTZuyYsUKRo4caZPARCmd2wVZN0l3qcE/V4MJ8nKhXo1qWkclhBBC2AWbzQHq2LEjGzdutNXpRGmdVoe/jrm1BXR0b+An1bqFEEKIbDZJgG7evMknn3xCrVq1bHE6YQvZE6A33GwEQI+GNTQMRgghhLAvxR4Cu33TU0VRSE1Nxc3Nje+++86mwYkSunkFLu4H4PsrDdDpoGs9P42DEkIIIexHsROgDz/80CIB0uv11KhRg44dO1K9enWbBidKKHorKCaSq9XlUpoPLWt6Ud3dSeuohBBCCLtR7AToiSeeKIMwhE1lz//Z79AKkOEvIYQQ4nbFngO0aNEiVq5cmef4ypUr+frrr4sdwLx58wgLC8PFxYW2bduydevWAts+8cQT6HS6PI+mTW/tcr548eJ826SlVaENP7Pn/6xKVqs/d28gCZAQQgiRW7EToHfeeQc/v7zzSfz9/Xn77beLda4VK1YwYcIEJk+ezP79++nevTsDBgwgJiYm3/YfffQRsbGx5se5c+fw8fHhoYcesmjn6elp0S42NhYXlyqy4eeVs5B0GkVnYNPNBlRzdqB1bW+toxJCCCHsSrEToLNnzxIWFpbneGhoaIGJS0Fmz57NyJEjGTVqFE2aNGHOnDmEhIQwf/78fNt7eXkRGBhofuzZs4crV67w5JNPWrTT6XQW7QIDA4sVV4WW3fsT69GMa7jRuZ4vjgbZ8UQIIYTIrdh3Rn9/fw4ePJjn+IEDB/D19bX6PBkZGezdu5e+fftaHO/bty/bt2+36hxfffUVvXv3JjQ01OL4tWvXCA0NpVatWtxzzz3s37+/0POkp6eTkpJi8aiwshOgv0zq7u89GsjqLyGEEOJ2xU6AHn74YcaNG8fmzZsxGo0YjUZ+//13xo8fz8MPP2z1eRISEjAajQQEBFgcDwgIIC4ursjXx8bG8uuvvzJq1CiL440bN2bx4sWsXbuWZcuW4eLiQteuXTlx4kSB55o5cyZeXl7mR0hIiNXXYVdMJojeAsAPV9XtL2T+jxBCCJFXsVeBvfnmm5w9e5a77roLBwf15SaTiREjRhR7DhCQpzqxoihWVSxevHgx3t7eDBkyxOJ4p06d6NSpk/nrrl270qZNGz755BM+/vjjfM81adIkJk6caP46JSWlYiZBlw7BjUSyHNzZk1aP2j5u1PFz1zoqIYQQwu4UOwFycnJixYoVvPnmm0RFReHq6krz5s3zDEMVxc/PD4PBkKe3Jz4+Pk+v0O0URWHhwoUMHz4cJ6fC69vo9Xrat29faA+Qs7Mzzs7O1gdvr7KHv065tSLrmgPdZfhLCCGEyFexE6AcDRo0oEGDBiV+YycnJ9q2bUtkZCT33Xef+XhkZGS+G67mtmXLFk6ePGnVxquKohAVFUXz5s1LHGuFcUqt//NbWhNAhr+EEEKIghR7DtCDDz7IO++8k+f4e++9l2c5elEmTpzIl19+ycKFCzl69CgvvvgiMTExjBkzBlCHpkaMGJHndV999RUdO3akWbNmeZ6bPn06GzZs4PTp00RFRTFy5EiioqLM56y0MtMgZgcAa1IbYtDr6FLf+knpQgghRFVS7B6gLVu2MHXq1DzH+/fvz/vvv1+sc0VERJCYmMiMGTOIjY2lWbNmrFu3zjycFhsbm2dpfXJyMj/88AMfffRRvue8evUqTz/9NHFxcXh5edG6dWv+/PNPOnToUKzYKpxzOyErjRvO/pxMq0m72t54ujhqHZUQQghhl3SKoijFeYGrqytRUVE0atTI4vi///5L69atuXnzpk0D1EJKSgpeXl4kJyfj6empdTjW2TgNtn3ITo++PHz5CV7s3ZDxvUs+RCmEEEJUNMW5fxd7CKxZs2asWLEiz/Hly5cTHh5e3NMJW8meAL02tSEA3RvKBGghhBCiIMUeApsyZQoPPPAAp06d4s477wRg06ZNLF26lO+//97mAQor3EiCi1EARKY1wdPFgZa1vDUNSQghhLBnxU6A7r33XlavXs3bb7/N999/j6urKy1btuT333+vOMNFlU30n4BCgltdLqdV5+4Gfhj0RddSEkIIIaqqEi2DHzhwIAMHDgTUScdLlixhwoQJHDhwAKPRaNMAhRVOq8vfd9ACkOXvQgghRFFKvEvm77//zmOPPUZwcDBz587l7rvvZs+ePbaMTVgre/7P6mR10rMUQBRCCCEKV6weoPPnz7N48WIWLlzI9evXGTp0KJmZmfzwww8yAVorSdFw5QwmnQM7jE2oW8OdWtXdtI5KCCGEsGtW9wDdfffdhIeHc+TIET755BMuXrzIJ598UpaxCWtk9/6cdWvKDVzoIcNfQgghRJGs7gH67bffGDduHM8++2yptsAQNpadAG1KV7e/6CHL34UQQogiWd0DtHXrVlJTU2nXrh0dO3Zk7ty5XL58uSxjE0UxGSF6CwDrrjfG0aCjY5hsfyGEEEIUxeoEqHPnznzxxRfExsbyzDPPsHz5cmrWrInJZCIyMpLU1NSyjFPkJ+4g3LxChkM1Dij1aBtaHXfnEu9vK4QQQlQZxV4F5ubmxlNPPcW2bds4dOgQL730Eu+88w7+/v7ce++9ZRGjKEj27u//OLXAiIEeDWX+jxBCCGGNEi+DB2jUqBHvvvsu58+fZ9myZbaKSVgre/7PumvqvmwyAVoIIYSwTqkSoBwGg4EhQ4awdu1aW5xOWCPzJsTsBGBTZlN83J0ID5JK3EIIIYQ1bJIACQ3E7ABjOilO/pxWguhW3w+9bH8hhBBCWEUSoIoqe/hrt64FoJP5P0IIIUQxSAJUUWVPgF6bPf9Htr8QQgghrCcJUEV0PVFdAg/8ZWxGowAPAjxdNA5KCCGEqDgkAaqIov8A4KJLPRLwkurPQgghRDFJAlQRZc//+SND3YC2uyx/F0IIIYpFEqCKRlHg1B8AbEgLx9lBT4cwH21jEkIIISoYSYAqmqTTkByDUefALlMjOoT54OJo0DoqIYQQokKRBKiiyR7+OuEUzk1cpPqzEEIIUQKSAFUkqXHw92cA/HqzCQDdZQK0EEIIUWySAFUkKRch4TgAWzKb4u/hTKMAD42DEkIIISoeSYAqkoRjAKTrXTmo1KV7gxrodLL9hRBCCFFcDloHIIqQGqc+AI7+DMA5XRBNdGcZ5O+iPucRqGGAQgghRMUjCZC927MItrxjcai+8TS/OE+GPwDlNeg1SZPQhBBCiIpKEiB71+5JaDQAo6KQ9d1QnG/G82XWAA769OXjh9tI748QQghRAjIHyN55BLI+KYBuXyfCjSQAdpqasCk5mPVJAZIACSGEECUgCZCdW384lme/20dWShzOuiwALiveXE838ux3+1h/OFbjCIUQQoiKRxIgO2Y0KUz/6QgKUFsXD0Cy4sZFxdfcZvpPRzCaFI0iFEIIISomSYDs2K7oJGKT04BbCdBhUxiXqQ6AAsQmp7ErOkmrEIUQQogKSRIgOxafmmb+PCcBOqv4F9pOCCGEEEWTBMiO+Xu4mD+vrb8EwDkloNB2QgghhCiaJEB2rEOYD0FeLui41QMUk6sHSAcEebnQIcxHmwCFEEKICkoSIDtm0OuYOigcyJsA5WyAMXVQOAa9bIchhBBCFIckQHauf7MgPn+4CQG6q8CtBCjQy4X5j7Whf7MgDaMTQgghKiapBF0B9A3OACBFcSMwIJDP7m1GhzAf6fkRQgghSkgSoIrgyhlA7f1pV8eHzvV8C28vhBBCiELJEFhFkCsBCvV10zYWIYQQohKQBKgiyJUA1fZx1zYWIYQQohLQPAGaN28eYWFhuLi40LZtW7Zu3Vpg2yeeeAKdTpfn0bRpU4t2P/zwA+Hh4Tg7OxMeHs6qVavK+jLKVnYCdE56gIQQQgib0DQBWrFiBRMmTGDy5Mns37+f7t27M2DAAGJiYvJt/9FHHxEbG2t+nDt3Dh8fHx566CFzmx07dhAREcHw4cM5cOAAw4cPZ+jQofz999/ldVk2Z0yKBnJ6gCQBEkIIIUpLpyiKZjtpduzYkTZt2jB//nzzsSZNmjBkyBBmzpxZ5OtXr17N/fffT3R0NKGhoQBERESQkpLCr7/+am7Xv39/qlevzrJly6yKKyUlBS8vL5KTk/H09CzmVdmYomB6MxC9MY3Bhk9ZM+UxbeMRQggh7FRx7t+a9QBlZGSwd+9e+vbta3G8b9++bN++3apzfPXVV/Tu3duc/IDaA3T7Ofv161foOdPT00lJSbF42I1rl9Ab0zAqOpx9a2sdjRBCCFEpaJYAJSQkYDQaCQiw3NsqICCAuLi4Il8fGxvLr7/+yqhRoyyOx8XFFfucM2fOxMvLy/wICQkpxpWUsez5PxcVP2r5atwbJYQQQlQSmk+C1uksi/kpipLnWH4WL16Mt7c3Q4YMKfU5J02aRHJysvlx7tw564IvD7lXgMkEaCGEEMImNCuE6Ofnh8FgyNMzEx8fn6cH53aKorBw4UKGDx+Ok5OTxXOBgYHFPqezszPOzs7FvIJykisBquMrS+CFEEIIW9CsB8jJyYm2bdsSGRlpcTwyMpIuXboU+totW7Zw8uRJRo4cmee5zp075znnb7/9VuQ57VauJfDSAySEEELYhqZbYUycOJHhw4fTrl07OnfuzIIFC4iJiWHMmDGAOjR14cIFvvnmG4vXffXVV3Ts2JFmzZrlOef48ePp0aMHs2bNYvDgwaxZs4aNGzeybdu2crkmWzMlRaMnuwq0LIEXQgghbELTBCgiIoLExERmzJhBbGwszZo1Y926deZVXbGxsXlqAiUnJ/PDDz/w0Ucf5XvOLl26sHz5cl5//XWmTJlCvXr1WLFiBR07dizz6ykLOQlQgmMQPu5ORbYXQgghRNE0rQNkr+ymDlDmTXgrEIAIr6WseHGgdrEIIYQQdq5C1AESVriq9n6lKK74+BU+MVwIIYQQ1pMEyJ7lngDtJyvAhBBCCFuRBMieZSdAZ5UAQmUXeCGEEMJmJAGyZ7k2QZVd4IUQQgjbkQTIjilX1ATonOwCL4QQQtiUJEB2LCtRTYAu6gII9nbVOBohhBCi8pAEyF4pCvqrZwHI8AzFoC96fzQhhBBCWEcSIHt1LR6DMQ2josPZt7bW0QghhBCViiRA9ip7BVgsvtTy89I2FiGEEKKSkQTIXuXsAm+SCdBCCCGErUkCZK9yEiDFnzq+UgNICCGEsCVJgOxUzhJ4qQEkhBBC2J4kQHYqZwn8OcWfEBkCE0IIIWxKEiA7pSSdAeC6WwgujgZtgxFCCCEqGUmA7FHmTZxuxKmf+4ZpG4sQQghRCUkCZI+uxgCQorji6xugcTBCCCFE5SMJkD3KXgF2TvEn1E9WgAkhhBC2JgmQPcq1BL62LIEXQgghbE4SIHuUKwEKlRVgQgghhM1JAmSHshJPA9lDYFIDSAghhLA5SYDsUE4NoATHILzdnDSORgghhKh8JAGyN4qCQ7K6CszkXUfbWIQQQohKShIge3P9Mg7Gm5gUHS416mgdjRBCCFEpSQJkb7InQF/El1q+XtrGIoQQQlRSkgDZm5waQCaZAC2EEEKUFUmA7E2SOgH6rOJPqNQAEkIIIcqEJEB2xpSdAMXIEnghhBCizEgCZGcyEtQaQLH6QAI8XDSORgghhKicJAGyM7rsOUAZnrXR63XaBiOEEEJUUpIA2ZPMNJxvXgLAwSdM42CEEEKIyksSIHtyVS2AmKq44lMjUONghBBCiMpLEiB7krMEXlaACSGEEGVKEiB7knsXeEmAhBBCiDIjCZAdUa7cWgJfW5bACyGEEGVGEiA7knFZXQJ/TvGnVnVXjaMRQgghKi9JgOyIMVHtAbruVgtnB4PG0QghhBCVlyRA9kJRcEpVV4Ep1etoG4sQQghRyUkCZC+uX8bBeBOTosOthtQAEkIIIcqSJED2InsF2EV8qVnDS9tYhBBCiEpOEiB7kVMDyORPqI8sgRdCCCHKkiRA9sKiBpAsgRdCCCHKkuYJ0Lx58wgLC8PFxYW2bduydevWQtunp6czefJkQkNDcXZ2pl69eixcuND8/OLFi9HpdHkeaWlpZX0ppZKZvQu8JEBCCCFE2XPQ8s1XrFjBhAkTmDdvHl27duXzzz9nwIABHDlyhNq1a+f7mqFDh3Lp0iW++uor6tevT3x8PFlZWRZtPD09OXbsmMUxFxeXMrsOW8hIOI0jkOQUjIeLo9bhCCGEEJWapgnQ7NmzGTlyJKNGjQJgzpw5bNiwgfnz5zNz5sw87devX8+WLVs4ffo0Pj4+ANSpUydPO51OR2Cg9ZuJpqenk56ebv46JSWlmFdSevqrZwHI8so/8RNCCCGE7Wg2BJaRkcHevXvp27evxfG+ffuyffv2fF+zdu1a2rVrx7vvvkvNmjVp2LAhL7/8Mjdv3rRod+3aNUJDQ6lVqxb33HMP+/fvLzSWmTNn4uXlZX6EhISU7uKKKzMNl5uXAHD0q1u+7y2EEEJUQZolQAkJCRiNRgICAiyOBwQEEBcXl+9rTp8+zbZt2zh8+DCrVq1izpw5fP/994wdO9bcpnHjxixevJi1a9eybNkyXFxc6Nq1KydOnCgwlkmTJpGcnGx+nDt3zjYXaa3kc+hQuKa44FcjqHzfWwghhKiCNB0CA3W4KjdFUfIcy2EymdDpdCxZsgQvL7VWzuzZs3nwwQf59NNPcXV1pVOnTnTq1Mn8mq5du9KmTRs++eQTPv7443zP6+zsjLOzs42uqASS1C0wzin+1Parpl0cQgghRBWhWQ+Qn58fBoMhT29PfHx8nl6hHEFBQdSsWdOc/AA0adIERVE4f/58vq/R6/W0b9++0B4gzckSeCGEEKJcaZYAOTk50bZtWyIjIy2OR0ZG0qVLl3xf07VrVy5evMi1a9fMx44fP45er6dWrVr5vkZRFKKioggKst+hJWN2D9BZJYBQH0mAhBBCiLKmaR2giRMn8uWXX7Jw4UKOHj3Kiy++SExMDGPGjAHUuTkjRowwt3/00Ufx9fXlySef5MiRI/z555+88sorPPXUU7i6ugIwffp0NmzYwOnTp4mKimLkyJFERUWZz2mP0uJPARCnD6CGh4ZDcUIIIUQVoekcoIiICBITE5kxYwaxsbE0a9aMdevWERoaCkBsbCwxMTHm9tWqVSMyMpIXXniBdu3a4evry9ChQ3nzzTfNba5evcrTTz9NXFwcXl5etG7dmj///JMOHTqU+/VZS0k6A0B6tdoFzn8SQgghhO3oFEVRtA7C3qSkpODl5UVycjKenp5l+2aKQuabQTgab/J6rcW8Oeq+sn0/IYQQopIqzv1b860wqrzrCTgab2JSdLj7h2kdjRBCCFElSAKktewVYLH4UKuGt6ahCCGEEFWFJEBay06Azin+hPq6axuLEEIIUUVIAqQxU/YS+BiT1AASQgghyoskQBpLu6wugT+PP8HerhpHI4QQQlQNkgBpLCvhNADX3GrhaJBvhxBCCFEe5I6rMYfkswCYvOtoG4gQQghRhUgCpKXMNFzT4gFw8qurcTBCCCFE1SEJkJaSz6FD4Zrigl9AsNbRCCGEEFWGJEBayrUEvrZvNW1jEUIIIaoQSYC0lJ0AxSiyBF4IIYQoT5IAaSg9exf4GMWf2j6SAAkhhBDlRRIgDaVdVpfAJzkF4+7soHE0QgghRNUhCZCWsofAMj1raxuHEEIIUcVIAqQVRcH1+jkADD6yC7wQQghRniQB0sqNRJyMNzApOqoFSgIkhBBClCdJgLSSvQlqHNWpVaO6xsEIIYQQVYskQFrJVQMo1Ndd21iEEEKIKkYSII1kJqorwGJM/oTKEnghhBCiXEkCpJGbl9QaQLGGIHzcnTSORgghhKhaJAHSiDFRnQOUVi0EnU6ncTRCCCFE1SIJkEYcU2LUT6rX0TQOIYQQoiqSBEgLWem4pV8CwNm/rsbBCCGEEFWPJEBauHoOPQrXFWdq+NfSOhohhBCiypEESAu5d4H3kyXwQgghRHmTBEgDpuwiiOdkF3ghhBBCE5IAaeD6pZMAnCeAYG9XjaMRQgghqh5JgDSQcVmtAZTqWguDXpbACyGEEOVNEiAN6K+eBSDLq7bGkQghhBBVkyRA5U1RcLt+HgBHP1kCL4QQQmhBEqDydiMRZ9MNTIoOjwBJgIQQQggtSAJU3rKXwMdRnVr+PtrGIoQQQlRRkgCVMyXXEvg6vrIEXgghhNCCJEDl7Ga8ugIsxuRPiNQAEkIIITQhCVA5u3lJTYCSnINxcTRoHI0QQghRNUkCVN4SjwNgcJf5P0IIIYRWJAEqZ86p5wDwqOapcSRCCCFE1SUJUDkyZqThlpkIQLJTDYwmReOIhBBCiKpJEqDykBrH9q0b+e+7s83/4KdOHOWpmV+wfetGSI3TNDwhhBCiqnHQOoCq4OS6j+ly9FO65Do2y/FLyAQ2wcmLY6kf8bZW4QkhhBBVjuY9QPPmzSMsLAwXFxfatm3L1q1bC22fnp7O5MmTCQ0NxdnZmXr16rFw4UKLNj/88APh4eE4OzsTHh7OqlWryvISCmU0KUw41ZqB6W9xb/obzMqMAODVzNEMTH+Le9LfYsKp1jIcJoQQQpQjTROgFStWMGHCBCZPnsz+/fvp3r07AwYMICYmpsDXDB06lE2bNvHVV19x7Ngxli1bRuPGjc3P79ixg4iICIYPH86BAwcYPnw4Q4cO5e+//y6PS8pjV3QSh1Pc+EcJ46BSjz9NLQA4bKrDP0oYh5UwDqe4sSs6SZP4hBBCiKpIpyiKZl0PHTt2pE2bNsyfP998rEmTJgwZMoSZM2fmab9+/XoefvhhTp8+jY9P/svIIyIiSElJ4ddffzUf69+/P9WrV2fZsmVWxZWSkoKXlxfJycl4epZutdaaqAuMXx5l/rqpLppfnCczMP0t/lHCzMc/ergVg1vVLNV7CSGEEFVZce7fmvUAZWRksHfvXvr27WtxvG/fvmzfvj3f16xdu5Z27drx7rvvUrNmTRo2bMjLL7/MzZs3zW127NiR55z9+vUr8JygDqulpKRYPGzF38PF4ut4xZs5WfcTr3gX2k4IIYQQZUezSdAJCQkYjUYCAgIsjgcEBBAXl/+qqNOnT7Nt2zZcXFxYtWoVCQkJPPfccyQlJZnnAcXFxRXrnAAzZ85k+vTppbyi/HUI8yHIy4W45DQU4DLVmZP1oPl5HRDo5UKHMCmMKIQQQpQXzSdB63Q6i68VRclzLIfJZEKn07FkyRI6dOjA3XffzezZs1m8eLFFL1BxzgkwadIkkpOTzY9z586V4oosGfQ6pg4KV+O67bmcr6cOCsegLzg+IYQQQtiWZgmQn58fBoMhT89MfHx8nh6cHEFBQdSsWRMvLy/zsSZNmqAoCufPnwcgMDCwWOcEcHZ2xtPT0+JhS/2bBTH/sTYEelkOcwV6uTD/sTb0bxZk0/cTQgghROE0GwJzcnKibdu2REZGct9995mPR0ZGMnjw4Hxf07VrV1auXMm1a9eoVq0aAMePH0ev11OrVi0AOnfuTGRkJC+++KL5db/99htdunTJ95zlpX+zIPqEB7IrOon41DT8PdRhL+n5EUIIIcqfpoUQJ06cyPDhw2nXrh2dO3dmwYIFxMTEMGbMGEAdmrpw4QLffPMNAI8++ihvvPEGTz75JNOnTychIYFXXnmFp556CldXVwDGjx9Pjx49mDVrFoMHD2bNmjVs3LiRbdu2aXadOQx6HZ3r+WodhhBCCFHlaZoARUREkJiYyIwZM4iNjaVZs2asW7eO0NBQAGJjYy1qAlWrVo3IyEheeOEF2rVrh6+vL0OHDuXNN980t+nSpQvLly/n9ddfZ8qUKdSrV48VK1bQsWPHcr8+IYQQQtgnTesA2Stb1gESQgghRPmoEHWAhBBCCCG0IgmQEEIIIaocSYCEEEIIUeVIAiSEEEKIKkcSICGEEEJUOZIACSGEEKLKkQRICCGEEFWOpoUQ7VVOaaSUlBSNIxFCCCGEtXLu29aUOJQEKB+pqakAhISEaByJEEIIIYorNTXVYuP0/Egl6HyYTCYuXryIh4cHOp3lZqUpKSmEhIRw7ty5KlUluqpeN1Tda5frluuuCqrqdUPlvHZFUUhNTSU4OBi9vvBZPtIDlI/cu8sXxNPTs9L8wBRHVb1uqLrXLtddtch1Vz2V7dqL6vnJIZOghRBCCFHlSAIkhBBCiCpHEqBicnZ2ZurUqTg7O2sdSrmqqtcNVffa5brluquCqnrdULWvHWQStBBCCCGqIOkBEkIIIUSVIwmQEEIIIaocSYCEEEIIUeVIAiSEEEKIKkcSoGKaN28eYWFhuLi40LZtW7Zu3ap1SOVq5syZ6HQ6JkyYoHUoZSorK4vXX3+dsLAwXF1dqVu3LjNmzMBkMmkdms39+eefDBo0iODgYHQ6HatXrzY/l5mZyauvvkrz5s1xd3cnODiYESNGcPHiRe0CtpHCrjvH0aNHuffee/Hy8sLDw4NOnToRExNT/sHayMyZM2nfvj0eHh74+/szZMgQjh07ZtFGURSmTZtGcHAwrq6u3HHHHfzzzz8aRWw71lx7bs888ww6nY45c+aUX5BlwJrrvnbtGs8//zy1atXC1dWVJk2aMH/+fI0iLj+SABXDihUrmDBhApMnT2b//v10796dAQMGVOhfiMWxe/duFixYQIsWLbQOpczNmjWLzz77jLlz53L06FHeffdd3nvvPT755BOtQ7O569ev07JlS+bOnZvnuRs3brBv3z6mTJnCvn37+PHHHzl+/Dj33nuvBpHaVmHXDXDq1Cm6detG48aN+eOPPzhw4ABTpkzBxcWlnCO1nS1btjB27Fh27txJZGQkWVlZ9O3bl+vXr5vbvPvuu8yePZu5c+eye/duAgMD6dOnj3mPxIrKmmvPsXr1av7++2+Cg4M1iNS2rLnuF198kfXr1/Pdd99x9OhRXnzxRV544QXWrFmjYeTlQBFW69ChgzJmzBiLY40bN1Zee+01jSIqP6mpqUqDBg2UyMhIpWfPnsr48eO1DqlMDRw4UHnqqacsjt1///3KY489plFE5QNQVq1aVWibXbt2KYBy9uzZ8gmqHOR33REREZX++x0fH68AypYtWxRFURSTyaQEBgYq77zzjrlNWlqa4uXlpXz22WdahVkmbr/2HOfPn1dq1qypHD58WAkNDVU+/PBDbQIsI/ldd9OmTZUZM2ZYtGvTpo3y+uuvl3d45Up6gKyUkZHB3r176du3r8Xxvn37sn37do2iKj9jx45l4MCB9O7dW+tQykW3bt3YtGkTx48fB+DAgQNs27aNu+++W+PItJecnIxOp8Pb21vrUMqMyWTil19+oWHDhvTr1w9/f386duyY7zBZRZacnAyAj48PANHR0cTFxVn8nnN2dqZnz56V7vfc7dcO6vd9+PDhvPLKKzRt2lSr0MpUftfdrVs31q5dy4ULF1AUhc2bN3P8+HH69eunVZjlQjZDtVJCQgJGo5GAgACL4wEBAcTFxWkUVflYvnw5+/btY/fu3VqHUm5effVVkpOTady4MQaDAaPRyFtvvcUjjzyidWiaSktL47XXXuPRRx+tVJsn3i4+Pp5r167xzjvv8OabbzJr1izWr1/P/fffz+bNm+nZs6fWIZaaoihMnDiRbt260axZMwDz77L8fs+dPXu23GMsK/ldO6hD3w4ODowbN07D6MpOQdf98ccfM3r0aGrVqoWDgwN6vZ4vv/ySbt26aRht2ZMEqJh0Op3F14qi5DlWmZw7d47x48fz22+/Vei5D8W1YsUKvvvuO5YuXUrTpk2JiopiwoQJBAcH8/jjj2sdniYyMzN5+OGHMZlMzJs3T+twylTOZPfBgwfz4osvAtCqVSu2b9/OZ599VikSoOeff56DBw+ybdu2PM9V9t9z+V373r17+eijj9i3b1+lutbcCvqef/zxx+zcuZO1a9cSGhrKn3/+yXPPPUdQUFDl7vXXcvytIklPT1cMBoPy448/WhwfN26c0qNHD42iKnurVq1SAMVgMJgfgKLT6RSDwaBkZWVpHWKZqFWrljJ37lyLY2+88YbSqFEjjSIqHxQwBygjI0MZMmSI0qJFCyUhIaH8Aytjt193enq64uDgoLzxxhsW7f7zn/8oXbp0KefobO/5559XatWqpZw+fdri+KlTpxRA2bdvn8Xxe++9VxkxYkR5hlhmCrr2Dz/80Px7LffvOr1er4SGhmoTrA0VdN03btxQHB0dlZ9//tni+MiRI5V+/fqVZ4jlTuYAWcnJyYm2bdsSGRlpcTwyMpIuXbpoFFXZu+uuuzh06BBRUVHmR7t27Rg2bBhRUVEYDAatQywTN27cQK+3/O9hMBgq5TL4omRmZjJ06FBOnDjBxo0b8fX11TqkMufk5ET79u3zLBc+fvw4oaGhGkVVeoqi8Pzzz/Pjjz/y+++/ExYWZvF8WFgYgYGBFr/nMjIy2LJlS4X/PVfUtQ8fPpyDBw9a/K4LDg7mlVdeYcOGDRpFXXpFXXdmZiaZmZlV8vedDIEVw8SJExk+fDjt2rWjc+fOLFiwgJiYGMaMGaN1aGXGw8PDYqwYwN3dHV9f3zzHK5NBgwbx1ltvUbt2bZo2bcr+/fuZPXs2Tz31lNah2dy1a9c4efKk+evo6GiioqLw8fEhODiYBx98kH379vHzzz9jNBrN80R8fHxwcnLSKuxSK+y6a9euzSuvvEJERAQ9evSgV69erF+/np9++ok//vhDu6BLaezYsSxdupQ1a9bg4eFh/l56eXnh6upqrvH19ttv06BBAxo0aMDbb7+Nm5sbjz76qMbRl05R1+7r65snuXd0dCQwMJBGjRppEbJNFHXdnp6e9OzZk1deeQVXV1dCQ0PZsmUL33zzDbNnz9Y4+jKmcQ9UhfPpp58qoaGhipOTk9KmTZs8SyirgqqwDD4lJUUZP368Urt2bcXFxUWpW7euMnnyZCU9PV3r0Gxu8+bNCpDn8fjjjyvR0dH5Pgcomzdv1jr0UinsunN89dVXSv369RUXFxelZcuWyurVq7UL2AYK+l4uWrTI3MZkMilTp05VAgMDFWdnZ6VHjx7KoUOHtAvaRqy59ttVhmXw1lx3bGys8sQTTyjBwcGKi4uL0qhRI+WDDz5QTCaTdoGXA52iKEpZJ1lCCCGEEPZE5gAJIYQQosqRBEgIIYQQVY4kQEIIIYSociQBEkIIIUSVIwmQEEIIIaocSYCEEEIIUeVIAiSEEEKIKkcSICGEEEJUOZIACSGEFXQ6HatXr9Y6DCGEjUgCJISwe0888QQ6nS7Po3///lqHJoSooGQzVCFEhdC/f38WLVpkcczZ2VmjaIQQFZ30AAkhKgRnZ2cCAwMtHtWrVwfU4an58+czYMAAXF1dCQsLY+XKlRavP3ToEHfeead55++nn36aa9euWbRZuHAhTZs2xdnZmaCgIJ5//nmL5xMSErjvvvtwc3OjQYMGrF27tmwvWghRZiQBEkJUClOmTOGBBx7gwIEDPPbYYzzyyCMcPXoUgBs3btC/f3+qV6/O7t27WblyJRs3brRIcObPn8/YsWN5+umnOXToEGvXrqV+/foW7zF9+nSGDh3KwYMHufvuuxk2bBhJSUnlep1CCBvRejt6IYQoyuOPP64YDAbF3d3d4jFjxgxFURQFUMaMGWPxmo4dOyrPPvusoiiKsmDBAqV69erKtWvXzM//8ssvil6vV+Li4hRFUZTg4GBl8uTJBcYAKK+//rr562vXrik6nU759ddfbXadQojyI3OAhBAVQq9evZg/f77FMR8fH/PnnTt3tniuc+fOREVFAXD06FFatmyJu7u7+fmuXbtiMpk4duwYOp2OixcvctdddxUaQ4sWLcyfu7u74+HhQXx8fEkvSQihIUmAhBAVgru7e54hqaLodDoAFEUxf55fG1dXV6vO5+jomOe1JpOpWDEJIeyDzAESQlQKO3fuzPN148aNAQgPDycqKorr16+bn//rr7/Q6/U0bNgQDw8P6tSpw6ZNm8o1ZiGEdqQHSAhRIaSnpxMXF2dxzMHBAT8/PwBWrlxJu3bt6NatG0uWLGHXrl189dVXAAwbNoypU6fy+OOPM23aNC5fvswLL7zA8OHDCQgIAGDatGmMGTMGf39/BgwYQGpqKn/99RcvvPBC+V6oEKJcSAIkhKgQ1q9fT1BQkMWxRo0a8e+//wLqCq3ly5fz3HPPERgYyJIlSwgPDwfAzc2NDRs2MH78eNq3b4+bmxsPPPAAs2fPNp/r8ccfJy0tjQ8//JCXX34ZPz8/HnzwwfK7QCFEudIpiqJoHYQQQpSGTqdj1apVDBkyROtQhBAVhMwBEkIIIUSVIwmQEEIIIaocmQMkhKjwZCRfCFFc0gMkhBBCiCpHEiAhhBBCVDmSAAkhhBCiypEESAghhBBVjiRAQgghhKhyJAESQgghRJUjCZAQQgghqhxJgIQQQghR5fw/IZzK3NqZ7wcAAAAASUVORK5CYII=", | |
| "text/plain": [ | |
| "<Figure size 640x480 with 1 Axes>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "num_epochs = 30 # during optimization, how many times we look at training data\n", | |
| "batch_size = 128 # during optimization, how many training data to use at each step\n", | |
| "learning_rate = 0.001 # during optimization, how much we nudge our solution at each step\n", | |
| "\n", | |
| "proper_train_accuracies, val_accuracies = \\\n", | |
| " UDA_pytorch_classifier_fit(simple_lstm_model,\n", | |
| " torch.optim.Adam(simple_lstm_model.parameters(),\n", | |
| " lr=learning_rate),\n", | |
| " nn.CrossEntropyLoss(), # includes softmax\n", | |
| " proper_train_dataset_encoded, val_dataset_encoded,\n", | |
| " num_epochs, batch_size,\n", | |
| " rnn=True,\n", | |
| " save_epoch_checkpoint_prefix='./saved_model_checkpoints/imdb_lstm')\n", | |
| "\n", | |
| "UDA_plot_train_val_accuracy_vs_epoch(proper_train_accuracies, val_accuracies)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 33, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "The model at the end of epoch 22 achieved the highest validation accuracy: 0.880400\n" | |
| ] | |
| }, | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "<All keys matched successfully>" | |
| ] | |
| }, | |
| "execution_count": 33, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "best_epoch_idx = np.argmax(val_accuracies)\n", | |
| "print('The model at the end of epoch %d achieved the highest validation accuracy: %f'\n", | |
| " % (best_epoch_idx + 1, val_accuracies[best_epoch_idx]))\n", | |
| "simple_lstm_model.load_state_dict(torch.load('./saved_model_checkpoints/imdb_lstm_epoch%d.pt' % (best_epoch_idx + 1)))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## Finally evaluate on test data" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 34, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "test_dataset = []\n", | |
| "\n", | |
| "for filename in sorted(glob('./data/aclImdb/test/pos/*.txt')):\n", | |
| " with open(filename, 'r', encoding='utf-8') as f:\n", | |
| " test_dataset.append((f.read(), 1)) # 1 means `positive` sentiment\n", | |
| "\n", | |
| "for filename in sorted(glob('./data/aclImdb/test/neg/*.txt')):\n", | |
| " with open(filename, 'r', encoding='utf-8') as f:\n", | |
| " test_dataset.append((f.read(), 0)) # 0 means `negative` sentiment" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 35, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "test_encoded = [vocab(tokenizer(text)) for text, label in test_dataset]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 36, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "test_labels = [label for text, label in test_dataset]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 37, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "predicted_test_labels = UDA_pytorch_classifier_predict(simple_lstm_model,\n", | |
| " test_encoded,\n", | |
| " rnn=True)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 38, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Test accuracy: 0.88348\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "print('Test accuracy:', UDA_compute_accuracy(predicted_test_labels, test_labels))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 39, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "tensor([1])" | |
| ] | |
| }, | |
| "execution_count": 39, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "UDA_pytorch_classifier_predict(simple_lstm_model,\n", | |
| " [vocab(tokenizer('this movie rocks'))],\n", | |
| " rnn=True)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 40, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "tensor([0])" | |
| ] | |
| }, | |
| "execution_count": 40, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "UDA_pytorch_classifier_predict(simple_lstm_model,\n", | |
| " [vocab(tokenizer('this movie sucks'))],\n", | |
| " rnn=True)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 41, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "tensor([0])" | |
| ] | |
| }, | |
| "execution_count": 41, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "UDA_pytorch_classifier_predict(simple_lstm_model,\n", | |
| " [vocab(tokenizer('this sucks'))],\n", | |
| " rnn=True)" | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "anaconda-cloud": {}, | |
| "kernelspec": { | |
| "display_name": "Python 3 (ipykernel)", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 3 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython3", | |
| "version": "3.11.5" | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 1 | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment