Skip to content

Instantly share code, notes, and snippets.

@georgehc

georgehc/UDA Recurrent Neural Nets.ipynb Secret

Last active August 28, 2025 18:14
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save georgehc/3f0fd7947329bfda500e2620b53f945f to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save georgehc/3f0fd7947329bfda500e2620b53f945f to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
UDA Recurrent Neural Nets.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# UDA Sentiment Analysis with IMDb Reviews\n",
"\n",
"Author: George H. Chen (georgechen [at symbol] cmu.edu)\n",
"\n",
"Last updated: Fri Apr 18, 2025\n",
"\n",
"This demo shows how to train an LSTM model for sentiment analysis with IMDb reviews. This is a binary classification task: for each review, we classify it as having positive or negative sentiment."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import random\n",
"import os\n",
"\n",
"os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8' # to help make code deterministic\n",
"\n",
"from glob import glob\n",
"\n",
"import torch\n",
"torch.use_deterministic_algorithms(True) # to help make code deterministic\n",
"torch.backends.cudnn.benchmark = False # to help make code deterministic\n",
"import torch.nn as nn\n",
"from torchinfo import summary\n",
"from transformers import AutoTokenizer, AutoModel\n",
"\n",
"np.random.seed(0) # to help make code deterministic\n",
"torch.manual_seed(0) # to help make code deterministic\n",
"random.seed(0) # to help make code deterministic\n",
"\n",
"from UDA_pytorch_utils import UDA_pytorch_classifier_fit, \\\n",
" UDA_plot_train_val_accuracy_vs_epoch, UDA_pytorch_classifier_predict, \\\n",
" UDA_compute_accuracy, UDA_get_rnn_last_time_step_outputs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load the dataset\n",
"\n",
"Here, we downloaded the IMDb dataset from: http://ai.stanford.edu/~amaas/data/sentiment/\n",
"\n",
"We place the file `aclImdb_v1.tar.gz` into `./data/` and uncompress the file within that directory so that after uncompressing, you should have access to the directories `./data/aclImdb/train`, `./data/aclImdb/test`, and other files such as the \"README\" file `./data/aclImdb/README`."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"train_dataset = []\n",
"\n",
"for filename in sorted(glob('./data/aclImdb/train/pos/*.txt')):\n",
" with open(filename, 'r', encoding='utf-8') as f:\n",
" train_dataset.append((f.read(), 1)) # 1 means `positive` sentiment\n",
"\n",
"for filename in sorted(glob('./data/aclImdb/train/neg/*.txt')):\n",
" with open(filename, 'r', encoding='utf-8') as f:\n",
" train_dataset.append((f.read(), 0)) # 0 means `negative` sentiment"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# proper training data points: 20000\n",
"# validation data points: 5000\n"
]
}
],
"source": [
"proper_train_size = int(len(train_dataset) * 0.8)\n",
"val_size = len(train_dataset) - proper_train_size\n",
"print('# proper training data points:', proper_train_size)\n",
"print('# validation data points:', val_size)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"proper_train_dataset, val_dataset = torch.utils.data.random_split(train_dataset,\n",
" [proper_train_size,\n",
" val_size])"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"(\"Master cinéaste Alain Resnais likes to work with those actors who are a part of his family.In this film too we see Resnais' family members like Pierre Arditi, Sabine Azema, André Dussolier and Fanny Ardant dealing with serious themes like death,religion,suicide,love and their overall implications on our daily lives.The formal nature of relationship shared by these people is evident as even friends, they address each other using a formal you.In 1984,while making L'amour à mort,Resnais dealt with time,memory and space to unravel the mysteries of a fundamental question of human existence :Is love stronger than death ? It was 16 years ago in 1968 that Resnais made a somewhat similar film Je t'aime Je t'aime which was also about love and memories.Message of this film is loud and clear :true and deep love can even put science to shame as dead lovers regain their lost lives leaving doctors to care for their reputation.L'amour à mort is like a game which is not at all didactic.It is a film in which the musical score is in perfect tandem with its images.This is one of the reasons why this film can easily be grasped.\",\n",
" 1)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"proper_train_dataset[0] # this is a tuple of the format (text, label)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We're going to use a [tiny version of BERT](https://huggingface.co/prajjwal1/bert-tiny) to get word embeddings."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"word_embedding_model_name = \"prajjwal1/bert-tiny\"\n",
"tokenizer = AutoTokenizer.from_pretrained(word_embedding_model_name)\n",
"max_doc_len = 512 # BERT/BERT-Tiny only supports documents with up to 512 tokens!\n",
"word_embedding_dim = 128 # this is what BERT-Tiny uses"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you want to try out the full version of BERT, set `word_embedding_model_name` to `\"bert-base-uncased\"` in the code snippet above and change `word_embedding_dim` to 768 above."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"[3040,\n",
" 25022,\n",
" 26737,\n",
" 2618,\n",
" 15654,\n",
" 24501,\n",
" 28020,\n",
" 7777,\n",
" 2000,\n",
" 2147,\n",
" 2007,\n",
" 2216,\n",
" 5889,\n",
" 2040,\n",
" 2024,\n",
" 1037,\n",
" 2112,\n",
" 1997,\n",
" 2010,\n",
" 2155,\n",
" 1012,\n",
" 1999,\n",
" 2023,\n",
" 2143,\n",
" 2205,\n",
" 2057,\n",
" 2156,\n",
" 24501,\n",
" 28020,\n",
" 1005,\n",
" 2155,\n",
" 2372,\n",
" 2066,\n",
" 5578,\n",
" 12098,\n",
" 23194,\n",
" 2072,\n",
" 1010,\n",
" 18578,\n",
" 17207,\n",
" 14545,\n",
" 1010,\n",
" 7213,\n",
" 4241,\n",
" 24137,\n",
" 14355,\n",
" 1998,\n",
" 17813,\n",
" 12098,\n",
" 28210,\n",
" 7149,\n",
" 2007,\n",
" 3809,\n",
" 6991,\n",
" 2066,\n",
" 2331,\n",
" 1010,\n",
" 4676,\n",
" 1010,\n",
" 5920,\n",
" 1010,\n",
" 2293,\n",
" 1998,\n",
" 2037,\n",
" 3452,\n",
" 13494,\n",
" 2006,\n",
" 2256,\n",
" 3679,\n",
" 3268,\n",
" 1012,\n",
" 1996,\n",
" 5337,\n",
" 3267,\n",
" 1997,\n",
" 3276,\n",
" 4207,\n",
" 2011,\n",
" 2122,\n",
" 2111,\n",
" 2003,\n",
" 10358,\n",
" 2004,\n",
" 2130,\n",
" 2814,\n",
" 1010,\n",
" 2027,\n",
" 4769,\n",
" 2169,\n",
" 2060,\n",
" 2478,\n",
" 1037,\n",
" 5337,\n",
" 2017,\n",
" 1012,\n",
" 1999,\n",
" 3118,\n",
" 1010,\n",
" 2096,\n",
" 2437,\n",
" 1048,\n",
" 1005,\n",
" 21518,\n",
" 1037,\n",
" 22294,\n",
" 1010,\n",
" 24501,\n",
" 28020,\n",
" 9411,\n",
" 2007,\n",
" 2051,\n",
" 1010,\n",
" 3638,\n",
" 1998,\n",
" 2686,\n",
" 2000,\n",
" 4895,\n",
" 22401,\n",
" 2140,\n",
" 1996,\n",
" 15572,\n",
" 1997,\n",
" 1037,\n",
" 8050,\n",
" 3160,\n",
" 1997,\n",
" 2529,\n",
" 4598,\n",
" 1024,\n",
" 2003,\n",
" 2293,\n",
" 6428,\n",
" 2084,\n",
" 2331,\n",
" 1029,\n",
" 2009,\n",
" 2001,\n",
" 2385,\n",
" 2086,\n",
" 3283,\n",
" 1999,\n",
" 3380,\n",
" 2008,\n",
" 24501,\n",
" 28020,\n",
" 2081,\n",
" 1037,\n",
" 5399,\n",
" 2714,\n",
" 2143,\n",
" 15333,\n",
" 1056,\n",
" 1005,\n",
" 6614,\n",
" 2063,\n",
" 15333,\n",
" 1056,\n",
" 1005,\n",
" 6614,\n",
" 2063,\n",
" 2029,\n",
" 2001,\n",
" 2036,\n",
" 2055,\n",
" 2293,\n",
" 1998,\n",
" 5758,\n",
" 1012,\n",
" 4471,\n",
" 1997,\n",
" 2023,\n",
" 2143,\n",
" 2003,\n",
" 5189,\n",
" 1998,\n",
" 3154,\n",
" 1024,\n",
" 2995,\n",
" 1998,\n",
" 2784,\n",
" 2293,\n",
" 2064,\n",
" 2130,\n",
" 2404,\n",
" 2671,\n",
" 2000,\n",
" 9467,\n",
" 2004,\n",
" 2757,\n",
" 10205,\n",
" 12452,\n",
" 2037,\n",
" 2439,\n",
" 3268,\n",
" 2975,\n",
" 7435,\n",
" 2000,\n",
" 2729,\n",
" 2005,\n",
" 2037,\n",
" 5891,\n",
" 1012,\n",
" 1048,\n",
" 1005,\n",
" 21518,\n",
" 1037,\n",
" 22294,\n",
" 2003,\n",
" 2066,\n",
" 1037,\n",
" 2208,\n",
" 2029,\n",
" 2003,\n",
" 2025,\n",
" 2012,\n",
" 2035,\n",
" 2106,\n",
" 28804,\n",
" 1012,\n",
" 2009,\n",
" 2003,\n",
" 1037,\n",
" 2143,\n",
" 1999,\n",
" 2029,\n",
" 1996,\n",
" 3315,\n",
" 3556,\n",
" 2003,\n",
" 1999,\n",
" 3819,\n",
" 18231,\n",
" 2007,\n",
" 2049,\n",
" 4871,\n",
" 1012,\n",
" 2023,\n",
" 2003,\n",
" 2028,\n",
" 1997,\n",
" 1996,\n",
" 4436,\n",
" 2339,\n",
" 2023,\n",
" 2143,\n",
" 2064,\n",
" 4089,\n",
" 2022,\n",
" 15517,\n",
" 1012]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokenizer.encode(proper_train_dataset[0][0],\n",
" add_special_tokens=False) # we'll talk about special tokens later (for now, don't add them)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"proper_train_encoded = [tokenizer.encode(text, add_special_tokens=False)[:max_doc_len] for text, label in proper_train_dataset]"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"[3040,\n",
" 25022,\n",
" 26737,\n",
" 2618,\n",
" 15654,\n",
" 24501,\n",
" 28020,\n",
" 7777,\n",
" 2000,\n",
" 2147,\n",
" 2007,\n",
" 2216,\n",
" 5889,\n",
" 2040,\n",
" 2024,\n",
" 1037,\n",
" 2112,\n",
" 1997,\n",
" 2010,\n",
" 2155,\n",
" 1012,\n",
" 1999,\n",
" 2023,\n",
" 2143,\n",
" 2205,\n",
" 2057,\n",
" 2156,\n",
" 24501,\n",
" 28020,\n",
" 1005,\n",
" 2155,\n",
" 2372,\n",
" 2066,\n",
" 5578,\n",
" 12098,\n",
" 23194,\n",
" 2072,\n",
" 1010,\n",
" 18578,\n",
" 17207,\n",
" 14545,\n",
" 1010,\n",
" 7213,\n",
" 4241,\n",
" 24137,\n",
" 14355,\n",
" 1998,\n",
" 17813,\n",
" 12098,\n",
" 28210,\n",
" 7149,\n",
" 2007,\n",
" 3809,\n",
" 6991,\n",
" 2066,\n",
" 2331,\n",
" 1010,\n",
" 4676,\n",
" 1010,\n",
" 5920,\n",
" 1010,\n",
" 2293,\n",
" 1998,\n",
" 2037,\n",
" 3452,\n",
" 13494,\n",
" 2006,\n",
" 2256,\n",
" 3679,\n",
" 3268,\n",
" 1012,\n",
" 1996,\n",
" 5337,\n",
" 3267,\n",
" 1997,\n",
" 3276,\n",
" 4207,\n",
" 2011,\n",
" 2122,\n",
" 2111,\n",
" 2003,\n",
" 10358,\n",
" 2004,\n",
" 2130,\n",
" 2814,\n",
" 1010,\n",
" 2027,\n",
" 4769,\n",
" 2169,\n",
" 2060,\n",
" 2478,\n",
" 1037,\n",
" 5337,\n",
" 2017,\n",
" 1012,\n",
" 1999,\n",
" 3118,\n",
" 1010,\n",
" 2096,\n",
" 2437,\n",
" 1048,\n",
" 1005,\n",
" 21518,\n",
" 1037,\n",
" 22294,\n",
" 1010,\n",
" 24501,\n",
" 28020,\n",
" 9411,\n",
" 2007,\n",
" 2051,\n",
" 1010,\n",
" 3638,\n",
" 1998,\n",
" 2686,\n",
" 2000,\n",
" 4895,\n",
" 22401,\n",
" 2140,\n",
" 1996,\n",
" 15572,\n",
" 1997,\n",
" 1037,\n",
" 8050,\n",
" 3160,\n",
" 1997,\n",
" 2529,\n",
" 4598,\n",
" 1024,\n",
" 2003,\n",
" 2293,\n",
" 6428,\n",
" 2084,\n",
" 2331,\n",
" 1029,\n",
" 2009,\n",
" 2001,\n",
" 2385,\n",
" 2086,\n",
" 3283,\n",
" 1999,\n",
" 3380,\n",
" 2008,\n",
" 24501,\n",
" 28020,\n",
" 2081,\n",
" 1037,\n",
" 5399,\n",
" 2714,\n",
" 2143,\n",
" 15333,\n",
" 1056,\n",
" 1005,\n",
" 6614,\n",
" 2063,\n",
" 15333,\n",
" 1056,\n",
" 1005,\n",
" 6614,\n",
" 2063,\n",
" 2029,\n",
" 2001,\n",
" 2036,\n",
" 2055,\n",
" 2293,\n",
" 1998,\n",
" 5758,\n",
" 1012,\n",
" 4471,\n",
" 1997,\n",
" 2023,\n",
" 2143,\n",
" 2003,\n",
" 5189,\n",
" 1998,\n",
" 3154,\n",
" 1024,\n",
" 2995,\n",
" 1998,\n",
" 2784,\n",
" 2293,\n",
" 2064,\n",
" 2130,\n",
" 2404,\n",
" 2671,\n",
" 2000,\n",
" 9467,\n",
" 2004,\n",
" 2757,\n",
" 10205,\n",
" 12452,\n",
" 2037,\n",
" 2439,\n",
" 3268,\n",
" 2975,\n",
" 7435,\n",
" 2000,\n",
" 2729,\n",
" 2005,\n",
" 2037,\n",
" 5891,\n",
" 1012,\n",
" 1048,\n",
" 1005,\n",
" 21518,\n",
" 1037,\n",
" 22294,\n",
" 2003,\n",
" 2066,\n",
" 1037,\n",
" 2208,\n",
" 2029,\n",
" 2003,\n",
" 2025,\n",
" 2012,\n",
" 2035,\n",
" 2106,\n",
" 28804,\n",
" 1012,\n",
" 2009,\n",
" 2003,\n",
" 1037,\n",
" 2143,\n",
" 1999,\n",
" 2029,\n",
" 1996,\n",
" 3315,\n",
" 3556,\n",
" 2003,\n",
" 1999,\n",
" 3819,\n",
" 18231,\n",
" 2007,\n",
" 2049,\n",
" 4871,\n",
" 1012,\n",
" 2023,\n",
" 2003,\n",
" 2028,\n",
" 1997,\n",
" 1996,\n",
" 4436,\n",
" 2339,\n",
" 2023,\n",
" 2143,\n",
" 2064,\n",
" 4089,\n",
" 2022,\n",
" 15517,\n",
" 1012]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"proper_train_encoded[0]"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"master cineaste alain resnais likes to work with those actors who are a part of his family. in this film too we see resnais ' family members like pierre arditi, sabine azema, andre dussolier and fanny ardant dealing with serious themes like death, religion, suicide, love and their overall implications on our daily lives. the formal nature of relationship shared by these people is evident as even friends, they address each other using a formal you. in 1984, while making l ' amour a mort, resnais dealt with time, memory and space to unravel the mysteries of a fundamental question of human existence : is love stronger than death? it was 16 years ago in 1968 that resnais made a somewhat similar film je t ' aime je t ' aime which was also about love and memories. message of this film is loud and clear : true and deep love can even put science to shame as dead lovers regain their lost lives leaving doctors to care for their reputation. l ' amour a mort is like a game which is not at all didactic. it is a film in which the musical score is in perfect tandem with its images. this is one of the reasons why this film can easily be grasped.\""
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# we can reconstruct any original review from the encoded version of the review\n",
"tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(proper_train_encoded[0]))"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"proper_train_labels = [label for text, label in proper_train_dataset]"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"val_encoded = [tokenizer.encode(text, add_special_tokens=False)[:max_doc_len] for text, label in val_dataset]"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"val_labels = [label for text, label in val_dataset]"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"proper_train_dataset_encoded = list(zip(proper_train_encoded, proper_train_labels))\n",
"val_dataset_encoded = list(zip(val_encoded, val_labels))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setting up a recurrent neural net for sentiment analysis that uses pre-trained word embeddings"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We now load in the PyTorch BERT-Tiny model (or if you changed `word_embedding_model_name` above then the next code cell will just load in whichever model you specified)."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"word_embedding_model = AutoModel.from_pretrained(word_embedding_model_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The next code cell constructs a PyTorch recurrent neural net model for sentiment analysis. Unfortunately, the code involved doesn't readily work with ``nn.Sequential`` that we have previously been using (for somewhat complicated reasons; in short, it has to do with an input batch of time series possibly having varying lengths, and accounting for varying lengths properly is not easy to do with ``nn.Sequential``). We will instead build the PyTorch model using another standard approach by creating a Python class that inherits from the `nn.Module` class.\n",
"\n",
"To illustrate how this works, here was how we created the multilayer perceptron model for MNIST digits in the previous demo:\n",
"\n",
"```\n",
"deeper_model = nn.Sequential(nn.Flatten(),\n",
" nn.Linear(in_features=784, out_features=512),\n",
" nn.ReLU(),\n",
" nn.Linear(in_features=512, out_features=10))\n",
"```\n",
"\n",
"An alternative way to code the same model is as follows:\n",
"\n",
"```\n",
"class DeeperModel(nn.Module):\n",
" def __init__(self, num_in_features, num_intermediate_features, num_out_features):\n",
" super().__init__()\n",
" self.flatten = nn.Flatten()\n",
" self.linear1 = nn.Linear(num_in_features, num_intermediate_features)\n",
" self.relu = nn.ReLU()\n",
" self.linear2 = nn.Linear(num_intermediate_features, num_out_features)\n",
"\n",
" def forward(self, inputs):\n",
" flatten_output = self.flatten(inputs)\n",
" linear1_output = self.linear1(flatten_output)\n",
" relu_output = self.relu(linear1_output)\n",
" linear2_output = self.linear2(relu_output)\n",
" return linear2_output\n",
"\n",
"deeper_model = DeeperModel(784, 512, 10)\n",
"```\n",
"\n",
"**Importantly, in the above code, the `forward` function specifies how the neural net processes input data. Note that the only argument it takes (aside from `self`) is the input data (`inputs`), which for MNIST digits we saw will be of the format (batch size, 1, 28, 28). Consequently, the example input data batch supplied to the `summary` function only needs this 4D table.**\n",
"\n",
"The code below creates the PyTorch neural net model corresponding to the architecture:\n",
"\n",
"1. Determine word embeddings (for every time series: convert the token ID at every time step into a word embedding)\n",
"2. `LSTM` layer with 32 output nodes (for every time series: put the time series through the LSTM's `for` loop and grab only the last time step's output, which has 32 numbers)\n",
"3. `Linear` layer with 2 output nodes (now every input data point to the linear layer is just a 1D table of 32 numbers, which this linear layer converts to 2 output numbers corresponding to the two classes)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"class WordEmbeddingLSTMModel(nn.Module):\n",
" def __init__(self, word_embedding_model, word_embedding_dim, num_rnn_output_nodes, num_final_output_nodes):\n",
" super().__init__()\n",
" self.word_embedding_model = word_embedding_model\n",
" self.lstm_layer = nn.LSTM(word_embedding_dim, num_rnn_output_nodes)\n",
" self.linear_layer = nn.Linear(num_rnn_output_nodes, num_final_output_nodes)\n",
"\n",
" def forward(self, text_encodings, lengths):\n",
" # the attention mask here is just a tensor the same shape as `text_encodings`\n",
" # that indicates which entries of `text_encodings` correspond to actual\n",
" # tokens whereas which ones are padding entries\n",
" attention_mask = torch.zeros_like(text_encodings)\n",
" for doc_idx in range(text_encodings.shape[1]):\n",
" attention_mask[:lengths[doc_idx], doc_idx] = 1\n",
"\n",
" # warning, word embedding models from Hugging Face expect the rows\n",
" # to be data points and columns to be time steps so we transpose\n",
" embeddings = self.word_embedding_model(text_encodings.T,\n",
" attention_mask=attention_mask.T)[0]\n",
" # embeddings.shape is (number of docs, number of time steps, word embedding dim)\n",
" embeddings = embeddings.transpose(0, 1) # swap axes 0 and 1 so that shape is (number of time steps, number of docs, word embedding dim)\n",
"\n",
" rnn_last_time_step_outputs = \\\n",
" UDA_get_rnn_last_time_step_outputs(embeddings, lengths, self.lstm_layer)\n",
"\n",
" return self.linear_layer(rnn_last_time_step_outputs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Notice that the `forward` function takes _two_ inputs (aside from `self`): `text_encodings` and `lengths`.** Keep in mind that `forward` takes in a batch of input time series, and these input time series could have different lengths. These different lengths are precisely what is stored in `lengths` as a 1D table of integers. Meanwhile, `text_encodings` is a 2D table where the number of rows is the maximum number of time steps in the input batch of time series, and the number of columns is the number of time series in the input batch. See the lecture slides for how `text_encodings` gets filled in (basically we pad all the time series in the batch to be of the same length as the longest time series; the padded entries will of course get ignored by the LSTM layer since it will know the correct length to use per time series). **When we give this neural net model an example data batch using the `summary` function, we need to specify two inputs corresponding to `text_encodings` and `lengths`.**"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"word_embedding_lstm_linear_model = WordEmbeddingLSTMModel(word_embedding_model, word_embedding_dim, 32, 2)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"==============================================================================================================\n",
"Layer (type:depth-idx) Output Shape Param #\n",
"==============================================================================================================\n",
"WordEmbeddingLSTMModel [5, 2] --\n",
"├─BertModel: 1-1 [5, 128] --\n",
"│ └─BertEmbeddings: 2-1 [5, 7, 128] --\n",
"│ │ └─Embedding: 3-1 [5, 7, 128] 3,906,816\n",
"│ │ └─Embedding: 3-2 [5, 7, 128] 256\n",
"│ │ └─Embedding: 3-3 [1, 7, 128] 65,536\n",
"│ │ └─LayerNorm: 3-4 [5, 7, 128] 256\n",
"│ │ └─Dropout: 3-5 [5, 7, 128] --\n",
"│ └─BertEncoder: 2-2 [5, 7, 128] --\n",
"│ │ └─ModuleList: 3-6 -- 396,544\n",
"│ └─BertPooler: 2-3 [5, 128] --\n",
"│ │ └─Linear: 3-7 [5, 128] 16,512\n",
"│ │ └─Tanh: 3-8 [5, 128] --\n",
"├─LSTM: 1-2 [18, 32] 20,736\n",
"├─Linear: 1-3 [5, 2] 66\n",
"==============================================================================================================\n",
"Total params: 4,406,722\n",
"Trainable params: 4,406,722\n",
"Non-trainable params: 0\n",
"Total mult-adds (Units.MEGABYTES): 33.61\n",
"==============================================================================================================\n",
"Input size (MB): 0.00\n",
"Forward/backward pass size (MB): 0.91\n",
"Params size (MB): 17.63\n",
"Estimated Total Size (MB): 18.54\n",
"=============================================================================================================="
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# example where there are 5 input time series of lengths 3, 2, 5, 1, 7;\n",
"# we specify these time series using a 2D table that is padded and a\n",
"# 1D table of lengths (see lecture slides for details)\n",
"summary(word_embedding_lstm_linear_model,\n",
" input_data=[torch.zeros((7, 5), dtype=torch.long),\n",
" torch.tensor([3, 2, 5, 1, 7], dtype=torch.long)])\n",
"\n",
"# note: the LSTM's output is in a compressed format (called a \"packed sequence\")\n",
"# that appears to put all the outputs of all the time series together (the output\n",
"# shape in this case appears to be 18 by 32) but it actually does keep track of\n",
"# which of the time steps correspond to which input time series (i.e., it knows\n",
"# that 3 of the rows correspond to the 0-th time series, 2 of the rows correspond\n",
"# to the 1st time series, 5 of the rows correspond to the 2nd time series, 1 row\n",
"# corresponds to the 3rd time series, and 7 rows correspond to the 4th time\n",
"# series); my helper code automatically maps these rows back to the correct\n",
"# format so that the final linear layer recognizes that there are 5 input data\n",
"# points"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"os.makedirs('./saved_model_checkpoints', exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
" 0%| | 0/30 [00:00<?, ?it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.8860\n",
" Validation accuracy: 0.8550\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 3%|████▊ | 1/30 [00:45<22:01, 45.55s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9394\n",
" Validation accuracy: 0.8740\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 7%|█████████▋ | 2/30 [01:31<21:23, 45.85s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9570\n",
" Validation accuracy: 0.8688\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 10%|██████████████▌ | 3/30 [02:18<20:46, 46.18s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9826\n",
" Validation accuracy: 0.8698\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 13%|███████████████████▎ | 4/30 [03:04<20:06, 46.40s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9902\n",
" Validation accuracy: 0.8708\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 17%|████████████████████████▏ | 5/30 [03:51<19:24, 46.57s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9939\n",
" Validation accuracy: 0.8706\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 20%|█████████████████████████████ | 6/30 [04:38<18:40, 46.67s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9952\n",
" Validation accuracy: 0.8726\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 23%|█████████████████████████████████▊ | 7/30 [05:25<17:54, 46.74s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9969\n",
" Validation accuracy: 0.8782\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 27%|██████████████████████████████████████▋ | 8/30 [06:12<17:09, 46.79s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9976\n",
" Validation accuracy: 0.8776\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 30%|███████████████████████████████████████████▌ | 9/30 [06:59<16:23, 46.82s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9974\n",
" Validation accuracy: 0.8738\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 33%|████████████████████████████████████████████████ | 10/30 [07:46<15:36, 46.83s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9980\n",
" Validation accuracy: 0.8708\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 37%|████████████████████████████████████████████████████▊ | 11/30 [08:33<14:49, 46.83s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9969\n",
" Validation accuracy: 0.8656\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 40%|█████████████████████████████████████████████████████████▌ | 12/30 [09:19<14:02, 46.83s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9981\n",
" Validation accuracy: 0.8740\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 43%|██████████████████████████████████████████████████████████████▍ | 13/30 [10:06<13:16, 46.83s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9982\n",
" Validation accuracy: 0.8636\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 47%|███████████████████████████████████████████████████████████████████▏ | 14/30 [10:53<12:29, 46.84s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9982\n",
" Validation accuracy: 0.8628\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 50%|████████████████████████████████████████████████████████████████████████ | 15/30 [11:40<11:42, 46.84s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9976\n",
" Validation accuracy: 0.8676\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 53%|████████████████████████████████████████████████████████████████████████████▊ | 16/30 [12:27<10:55, 46.85s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9987\n",
" Validation accuracy: 0.8692\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 57%|█████████████████████████████████████████████████████████████████████████████████▌ | 17/30 [13:14<10:08, 46.84s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9983\n",
" Validation accuracy: 0.8702\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 60%|██████████████████████████████████████████████████████████████████████████████████████▍ | 18/30 [14:00<09:21, 46.83s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9986\n",
" Validation accuracy: 0.8624\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 63%|███████████████████████████████████████████████████████████████████████████████████████████▏ | 19/30 [14:47<08:35, 46.83s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9990\n",
" Validation accuracy: 0.8720\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 67%|████████████████████████████████████████████████████████████████████████████████████████████████ | 20/30 [15:34<07:48, 46.82s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9985\n",
" Validation accuracy: 0.8590\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 70%|████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 21/30 [16:21<07:01, 46.82s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9990\n",
" Validation accuracy: 0.8692\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 73%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 22/30 [17:08<06:14, 46.82s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9991\n",
" Validation accuracy: 0.8686\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 77%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 23/30 [17:54<05:27, 46.82s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9991\n",
" Validation accuracy: 0.8724\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 80%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 24/30 [18:41<04:41, 46.84s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9991\n",
" Validation accuracy: 0.8722\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 83%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 25/30 [19:28<03:54, 46.83s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9991\n",
" Validation accuracy: 0.8648\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 87%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 26/30 [20:15<03:07, 46.82s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9991\n",
" Validation accuracy: 0.8736\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 27/30 [21:02<02:20, 46.80s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9992\n",
" Validation accuracy: 0.8698\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 93%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 28/30 [21:48<01:33, 46.80s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9991\n",
" Validation accuracy: 0.8540\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" 97%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 29/30 [22:35<00:46, 46.80s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Proper training accuracy: 0.9993\n",
" Validation accuracy: 0.8528\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [23:22<00:00, 46.75s/it]\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAAB0cUlEQVR4nO3deVhU1RsH8O/MsAy7IMqiCLiDuORG4palqLm3SJtpqWVZubSammaplWlWJqWmZmpalpblz8Q1FQ3FPUxzCxWQQGWVbeb+/jjO4Mg6MDN3YL6f55kHuHPuvefCMPedc95zjkKSJAlERERENkQpdwWIiIiILI0BEBEREdkcBkBERERkcxgAERERkc1hAEREREQ2hwEQERER2RwGQERERGRz7OSugDXSarVISkqCm5sbFAqF3NUhIiKiSpAkCVlZWfD394dSWX4bDwOgUiQlJSEgIEDuahAREVEVXL58GQ0bNiy3DAOgUri5uQEQv0B3d3eZa0NERESVkZmZiYCAAP19vDwMgEqh6/Zyd3dnAERERFTDVCZ9hUnQREREZHMYABEREZHNYQBERERENocBEBEREdkcBkBERERkcxgAERERkc1hAEREREQ2hwEQERER2RwGQERERGRzZA2A/vjjDwwaNAj+/v5QKBTYtGlThfvs2bMHHTp0gFqtRuPGjfHll1+WKPPjjz8iNDQUjo6OCA0NxcaNG81QeyIikptGK+HA+XT8fOwqDpxPh0YrWWRfufe31XObkqxLYeTk5KBt27Z45pln8PDDD1dY/uLFi3jwwQcxduxYrF69Gvv378eLL76IevXq6fc/cOAAoqKi8N5772HYsGHYuHEjhg8fjn379iE8PNzcl0REVONotBLiLl5HalYe6rup0TnYCyplxUsJyH3uraeS8e7mBCRn5Om3+XmoMWNQKPqF+ZltX7n3t9Vzm5pCkiR5Qq+7KBQKbNy4EUOHDi2zzJtvvolffvkFp0+f1m8bN24cjh8/jgMHDgAAoqKikJmZif/973/6Mv369YOnpye+++67Uo+bn5+P/Px8/c+6xdQyMjK4FhgRVVp1bubVDUKqur8pbkhynHvrqWS8sPoI7r6B6c4a/VT7Mo9RnX3l3t9Wz11ZmZmZ8PDwqNT9u0YthnrgwAFERkYabOvbty++/vprFBYWwt7eHgcOHMCkSZNKlFm4cGGZx507dy7effddc1SZSBY19RO9NexfVTXxU3VZN6SUjDy8sPpIpW5Icpxbo5Xw7uaEEvsCgARxQ313cwL6hPqW+NtXZ1+596/J5y7SaDHzl7+qvL851KgWoObNm2PUqFF4++239dtiY2PRtWtXJCUlwc/PDw4ODli5ciWeeOIJfZm1a9fimWeeMWjluRNbgMgcavIn+qqSu3m8uvtX53de0z5Va7QSun64Eyl3/K7u3t/XQ419b95f5u+gWuf+YAdSMkt/TwYADyd7vHx/U2i0Eoq0EgqKtCjUaFGklfBveg5+/+tamfvq3NvYC3VdHQ22pWfn4+CF61Xa15L792zuDR93NZQKBRQKBZQKIDUzHzGnK77u7s284eXigCKtBI1GgkaSoNFKSMvOx4krGRXuH+bvDk8XB/15lQoFbuYU4MjlmxXuG+DpBHuVEvlFWuQXaW5/1aKgSFvhvgDw3dh70aVJ3UqVLU2tbQECSi5xr4vf7txeWpm7t93J0dERjo4lX6hEVVWTP9FXVXXPbQ37V+V3rtFKmFHOJ1sAeHvjKXi7OsLDyR4ujnZwcbSDq6N4+zXmU7UkSSjUSLhVqEF+oQbZ+UWYtulUueee/P1x/H4qBVn5GmTlFSIrrwhZ+YW4nlOAnHxNmdclAUjOyEPEBzvg466Gh5M93J3s4XH74aa2w1d7LpR77td+OI5dZ/5D5q1C3MgtwM3cQtzMLUR6dj4KK0h8zbhViPd/O11umYpUJtAwx76m2H/P2bQq77v3n6rvCwCnkjKrvO/lG7eqde7UrNIDcnOoUQGQr68vUlJSDLalpqbCzs4OdevWLbeMj4+PxepJtUdVWgTkupFXt4n67mMZc90VnRsApvx0EoVFErQQn0Z1jyKthEKNFgtizpa7/+s/nMCZlCwoS/kwo5UkLN17sdz9Z/6SgN4hPrBTlRz8WpnfeWSoL5IybuH8fzm48F82zv+XjfOpOfg7JRM3cgtL/b3oXM8pwCNfHiix3UGlRIGm7E/GuiDknlnbUKSVkFeogbEDZnILNNh4LMm4ne5wLTMf18ppqSlPdr4G6w9drvK52zeqgyBvFziolLBTKWCvUsJBpcS1zDxsqsQ1jYoIQrC3i8G2i2k5WBl7qUr7WnL/xzsFoKGXMyRJglYSr/HE67n46cjVCvd9onMAGtdzhZ1SAZVSAZVSCTulAhfTchC953yF+7/Uqwma1neD9o5zn0vNwpI/Lla479sPhuCeRnXgoFLC0V55+6sKJ6/cxLjVRyrcv76busIyplKjAqAuXbpg8+bNBtu2bduGjh07wt7eXl8mJibGIA9o27ZtiIiIsGhdqearSouAJfroZ/6SgBA/d2TcEp/idY8TVzIM6lra/skZediekIK+JmoJKSjS4lxqNjYfTyr33ABwI7cQL687Wm6Z8mTlF+GT7f9Uef+UzDyEvLMVPu5qeLs6wtvVAd6ujvByccDqg/+WGzy9tPYoVEogv6jqGQOezvaQAGTnFaHodhRTXvBzp8y8ohLblArATqlAgabiOg1u6497G9eFm9ru9sMeF//LxmsbTlS478xBoQjwckbGrUKDx6mrGTh06UaF+/cL80WXxnVRx9kens4O8HR2wMW0bLyy7liF+77et2Wp3SEarYQ/L15HSkZeqX83Xffd9IGhpeay/P5XSpX2teT+7w9rXWrdD5xPr3Df94aW3Fe3/6ZjVyvcf1KfFqWee/Px5Ar3Hd0tuNRz+7qr4eehrnD/zsFepTxrHrLmAGVnZ+PcuXMAgHvuuQcLFixAr1694OXlhUaNGmHKlCm4evUqVq1aBUAMgw8LC8Pzzz+PsWPH4sCBAxg3bhy+++47/TD42NhY9OjRA7Nnz8aQIUPw888/Y9q0aUYNgzemD5Fqp8rkNvQO8UF6TgFSMvJwLVM84v+9UalPpt4uDnBV28FepYSdSgmH259ucwqKcDo5y+TXczdPZ3s0q++GJvVd0ay+K5rWd0UzH1ccS7yJF9eUfd0THmgGF0c7nE7OREJyJs7/l43CStyAdZrUc4GPu/r2p1KF/hPqtcx8HKtEfkGXJl4IqlvyU/Wl9BwcOF+9LofKsFcpEFjXBU3quaBJPVc0rueKWwVFmP7zXxXuq8ttkCQJ+UVa5OQXYd8/aZiw/liF+374cGvc27gu1Paq2w/xyfrghet4fOnBSp/7ThqthG63c4DKuyGVlQN04Hy6bOcGiv9HARgcw5i8q6rsK/f+tnruyjLm/i1rALR792706tWrxPaRI0di5cqVGDVqFC5duoTdu3frn9uzZw8mTZqEv/76C/7+/njzzTcxbtw4g/03bNiAadOm4cKFC2jSpAlmz56Nhx56qNL1YgBUe1SlC0v35lxei4ZSAUgSSn3ztgSVQoF6bqL1oq6r+FSdX6SpVGJoeRQw/prc1XZoUMcJp1MqDtzKSnCszs3UmP0XRrVFgJcL0rLzkZadj/TsAhy6dL1SORPTBoRgVERQiS606tzMqxsIVHf/6tyQ5Dz3nceoaSPvanrd5f69VaTGBEDWigFQ7WDsP5okSfgnNRvr4hKxfP+lSp1DpVSgnqsjfNwdUd9dDUio1CiN94eGoaWvGwo1Iv9FPCQkJGXgs53nKtz/u7Hh6NLE22BbZW9IMZN64lJ6Ds7/l41/rmXjn9QsnEvNxsW0nErll3QO8kS3ZvUQ4ueOED83NKjjBK0EWW/k1dm/usEXULM/VZtiLh45zq1TE+dequl1l/v3Vh4GQNXEAKjmq0wXVp9QX5xOzsSfF68j7mI6Dl26ges5BZU+x3tDWuGJ8ECDf9ya/In+p/grmPzD8VKfu9Onj7XDkHYNTHpuOfc3RXeM7vw19VO1XLMxV/fcRHdjAFRNDIBqtsp0YTnaKWGvVCC7wHAYsNpeiSb1XPFXJYaBltUiYA2BQFVuSKZqCamJ3Qqmyk2orZ+qrfncRHdiAFRNDICsi7FvrpW9kQOAq6MdOgZ5Ijy4LjoHe6F1Aw+olAqTJGjWtE/0pmoJkftGXhMnkCQi02AAVE0MgKxHZW5KkiQhOSMPJ69m4NTVDGw/fa1SI6le79sC43o2MVuCptyBQFVYYpSGNWNLBlHNxgComhgAWYey8nh0+of5IrdAg1NXM5BuRO6OTkVTrttqi4CtXjcR1XwMgKqJAZD8KpPHcyeVUoHmPm5o3cAdrfzd8dmOc7ieU1CtrhxdPWyxRcBWr5uIarZavRYY2Ya4i9crFfw82zUIg9s1QEtfN6jtVfrtPu5qvLD6SIl5bXS38BmDSp+l9W4qpaJaC/PVVLZ63URkO0ouikMks+z8Iqw6cKlSZdsG1EG7gDoGwQ8A9AvzQ/RT7eHrYbiujK+HutbnsRARUcXYAkRWI79Ig7V/JmLRznOVzukpb+G8fmF+6BPqy64cIiIqgQEQyU6jlbDp6FUsiDmLqzdvAQCC6jrj5q1CZOQWVmvhPHblEBFRaRgAkdmVlVArSRJiEq7h421ncPZaNgDAx90RE3s3xyMdGmLH6WsmyeMhIiK6GwMgMquyhlQ/3rkRdp1JxdHEmwAADyd7vHhfE4yMCNLn8+jyeO7e35dDsomIqJo4DL4UHAZvGhXN4wOIpSdGdwvGcz2awMPJvtQyHJJNRESVwWHwJDuNVsK7mxPKDX6cHVTYMbkn/Oo4lXss5vEQEZGpcRg8mUVl5vHJLdDgUnquhWpERERUjAEQmUVqVuVmcK5sOSIiIlNiAERmUd78PFUpR0REZEoMgMgs2gZ4wEFV9stLATEarKJ5fIiIiMyBARCZnFYr4e2fTqJAoy31ec7jQ0REcmMARCY3b9sZbDqWBDulAi/f3xR+XI+LiIisDIfBk0l9e/BfRO8+DwCY+1BrPNoxABN7N+c8PkREZFUYAJHJxCRcw4yfTwEAJvVujkc7BgDgPD5ERGR92AVGJnHs8k28/N0RaCUgqmMAXnmgqdxVIiIiKhMDIKq2f9NzMHrlIeQVatGzeT28PywMCgW7uIiIyHoxAKJqSc/Ox8jlcUjPKUBYA3csfrI97MsZ/k5ERGQNeKeiKrtVoMGYVYdxKT0XDeo4YfmoTnBxZFoZERFZPwZAVCUarYQJ647iaOJNeDjZ45tnO3FWZyIiqjH4cZ0qpNFKBsPYOwV54r1fE7At4RocVEosfbojmtZ3k7uaRERElcYAiMq19VQy3t2cYLCyu5vaDll5RQCABVFtuZwFERHVOAyAqExbTyXjhdVHIN21XRf8PNK+AQa28bd8xYiIiKqJOUBUKo1WwrubE0oEP3fafz4dGm15JYiIiKwTAyAqVdzF6wbdXqVJzshD3MXrFqoRERGR6TAAolKlZpUf/BhbjoiIyJowAKJSVXZIO4e+ExFRTcQAiErVOdgLfh5qlLWghQKAn4eaI8CIiKhGkj0AWrx4MYKDg6FWq9GhQwfs3bu33PJffPEFQkJC4OTkhBYtWmDVqlUlyixcuBAtWrSAk5MTAgICMGnSJOTlsavGGCqlAjMGhZb6nC4omjEoFCol1/wiIqKaR9YAaP369Zg4cSKmTp2Ko0ePonv37ujfvz8SExNLLR8dHY0pU6Zg5syZ+Ouvv/Duu+9i/Pjx2Lx5s77MmjVr8NZbb2HGjBk4ffo0vv76a6xfvx5Tpkyx1GXVGv3C/BD9VHs42hm+THw91Ih+qj36hfnJVDMiIqLqUUiSJNs45vDwcLRv3x7R0dH6bSEhIRg6dCjmzp1bonxERAS6du2KefPm6bdNnDgRhw8fxr59+wAAL730Ek6fPo0dO3boy7z66quIi4srs3UpPz8f+fn5+p8zMzMREBCAjIwMuLu7V/s6a7IijRbt3t2G7AINXo1sjo6BXugc7MWWHyIisjqZmZnw8PCo1P1bthaggoICxMfHIzIy0mB7ZGQkYmNjS90nPz8farVh0q2TkxPi4uJQWFgIAOjWrRvi4+MRFxcHALhw4QK2bNmCAQMGlFmXuXPnwsPDQ/8ICAiozqXVKieuZiC7QAMPJ3u8eF9TdGlSl8EPERHVeLIFQGlpadBoNPDx8THY7uPjg5SUlFL36du3L5YtW4b4+HhIkoTDhw9j+fLlKCwsRFpaGgDgsccew3vvvYdu3brB3t4eTZo0Qa9evfDWW2+VWZcpU6YgIyND/7h8+bLpLrSG2/+P+L1GMPAhIqJaRPalMBQKw5uqJEkltulMnz4dKSkpuPfeeyFJEnx8fDBq1Ch89NFHUKlUAIDdu3dj9uzZWLx4McLDw3Hu3DlMmDABfn5+mD59eqnHdXR0hKOjo2kvrJbYd04EQF2bestcEyIiItORrQXI29sbKpWqRGtPampqiVYhHScnJyxfvhy5ubm4dOkSEhMTERQUBDc3N3h7ixv09OnTMWLECIwZMwatW7fGsGHDMGfOHMydOxdardbs11Wb5BYU4UjiDQBANwZARERUi8gWADk4OKBDhw6IiYkx2B4TE4OIiIhy97W3t0fDhg2hUqmwbt06DBw4EEqluJTc3Fz99zoqlQqSJEHGfO8aKe7idRRqJDSo44TAus5yV4eIiMhkZO0Cmzx5MkaMGIGOHTuiS5cuWLJkCRITEzFu3DgAIjfn6tWr+rl+zp49i7i4OISHh+PGjRtYsGABTp06hW+++UZ/zEGDBmHBggW455579F1g06dPx+DBg/XdZFQ5+293f3Vr6l1mtyQREVFNJGsAFBUVhfT0dMyaNQvJyckICwvDli1bEBgYCABITk42mBNIo9Fg/vz5OHPmDOzt7dGrVy/ExsYiKChIX2batGlQKBSYNm0arl69inr16mHQoEGYPXu2pS+vxtt3Lh0A0LUZu7+IiKh2kXUeIGtlzDwCtVVadj46vr8dAHB4Wm94uzJJnIiIrFuNmAeIrFvsedH6E+LnzuCHiIhqHQZAVCrd/D/dmtaVuSZERESmxwCISpAkifP/EBFRrcYAiEr4Nz0XV2/egr1Kgc7BXnJXh4iIyOQYAFEJutaf9o084ewg+2ThREREJscAiEq4c/4fIiKi2ogBEBnQaCX9CLAIBkBERFRLMQAiAwlJmci4VQhXRzu0beghd3WIiIjMggEQGdDl/9zbuC7sVHx5EBFR7cQ7HBkozv/h/D9ERFR7MQAivbxCDeIuXQcAdOP6X0REVIsxACK9+H9voKBICx93RzSp5yp3dYiIiMyGARDp3Tn7s0KhkLk2RERE5sMAiPQ4/w8REdkKBkAEALiZW4CTVzMAcP0vIiKq/RgAEQDgwPl0SBLQrL4rfNzVcleHiIjIrBgAEQBw9XciIrIpDIAIAPN/iIjItjAAIly+notL6blQKRUIb+wld3WIiIjMjgEQIfa8aP1pF1AHbmp7mWtDRERkfgyACPvOidXfmf9DRES2ggGQjdNqJcQy/4eIiGwMAyAbd+ZaFtJzCuDsoEK7gDpyV4eIiMgiGADZON3or/BgLzjY8eVARES2gXc8G8f5f4iIyBYxALJhBUVa/HnhOgCgWzMGQEREZDsYANmwo4k3cKtQA29XB7TwcZO7OkRERBbDAMiG7b+j+0uhUMhcGyIiIsthAGTDmP9DRES2igGQjcrMK8TxKxkAGAAREZHtYQBko/68cB0arYTG3i5oUMdJ7uoQERFZFAMgG7Wf3V9ERGTDGADZKOb/EBGRLWMAZINSMvJwLjUbSgXQpXFduatDRERkcQyAbJCu+6t1Aw94ONvLXBsiIiLLkz0AWrx4MYKDg6FWq9GhQwfs3bu33PJffPEFQkJC4OTkhBYtWmDVqlUlyty8eRPjx4+Hn58f1Go1QkJCsGXLFnNdQo3D/B8iIrJ1dnKefP369Zg4cSIWL16Mrl274quvvkL//v2RkJCARo0alSgfHR2NKVOmYOnSpejUqRPi4uIwduxYeHp6YtCgQQCAgoIC9OnTB/Xr18eGDRvQsGFDXL58GW5unOkYACRJ0uf/dGMARERENkohSZIk18nDw8PRvn17REdH67eFhIRg6NChmDt3bonyERER6Nq1K+bNm6ffNnHiRBw+fBj79u0DAHz55ZeYN28e/v77b9jbV657Jz8/H/n5+fqfMzMzERAQgIyMDLi7u1f18qzSudQs9F7wBxztlDg+IxJqe5XcVSIiIjKJzMxMeHh4VOr+LVsXWEFBAeLj4xEZGWmwPTIyErGxsaXuk5+fD7VabbDNyckJcXFxKCwsBAD88ssv6NKlC8aPHw8fHx+EhYVhzpw50Gg0ZdZl7ty58PDw0D8CAgKqeXXWa98/ovWnc7AXgx8iIrJZsgVAaWlp0Gg08PHxMdju4+ODlJSUUvfp27cvli1bhvj4eEiShMOHD2P58uUoLCxEWpq4sV+4cAEbNmyARqPBli1bMG3aNMyfPx+zZ88usy5TpkxBRkaG/nH58mXTXaiV2XcuHQDzf4iIyLbJmgMEoMQinJIklbkw5/Tp05GSkoJ7770XkiTBx8cHo0aNwkcffQSVSrRmaLVa1K9fH0uWLIFKpUKHDh2QlJSEefPm4Z133in1uI6OjnB0dDTthVkZjVbCgfNp2PvPfwA4/J2IiGybbC1A3t7eUKlUJVp7UlNTS7QK6Tg5OWH58uXIzc3FpUuXkJiYiKCgILi5ucHbW7Ro+Pn5oXnz5vqACBB5RSkpKSgoKDDfBVmxraeS0e3DnXjq6zjkF2kBAM9/G4+tp5JlrhkREZE8ZAuAHBwc0KFDB8TExBhsj4mJQURERLn72tvbo2HDhlCpVFi3bh0GDhwIpVJcSteuXXHu3DlotVp9+bNnz8LPzw8ODg6mvxArt/VUMl5YfQTJGXkG269l5uGF1UcYBBERkU2SdR6gyZMnY9myZVi+fDlOnz6NSZMmITExEePGjQMgcnOefvppffmzZ89i9erV+OeffxAXF4fHHnsMp06dwpw5c/RlXnjhBaSnp2PChAk4e/YsfvvtN8yZMwfjx4+3+PXJTaOV8O7mBJQ2zE+37d3NCdBoZRsISEREJAtZc4CioqKQnp6OWbNmITk5GWFhYdiyZQsCAwMBAMnJyUhMTNSX12g0mD9/Ps6cOQN7e3v06tULsbGxCAoK0pcJCAjAtm3bMGnSJLRp0wYNGjTAhAkT8Oabb1r68mQXd/F6iZafO0kAkjPyEHfxOro0YU4QERHZDlnnAbJWxswjYM1+PnYVE9Ydq7Dcp4+1w5B2DcxfISIiIjOqEfMAkfnVd1NXXMiIckRERLUFA6BarHOwF/w8yg5uFAD8PNToHOxluUoRERFZAQZAtZhKqcCMQaGlPqebaWnGoFColKXPu0RERFRbMQCq5R4I8YGbumSuu6+HGtFPtUe/MD8ZakVERCQv2WeCJvPa9XcqsvKKUNfFAZ8+1g7pOQWo7ya6vdjyQ0REtooBUC33/WGxrtkjHRqiW7N6MteGiIjIOrALrBZLzczDrjNi7a9HO9beFe6JiIiMxQCoFvvxyFVotBI6BnqiaX1XuatDRERkNRgA1VKSJOGH291fw9n6Q0REZIABUC116NINXEjLgbODCgPacKQXERHRnRgA1VK65OeBbfzg4shcdyIiojsxAKqFsvIK8duJZABAVCd2fxEREd2NAVAt9OuJZNwq1KBJPRe0b+Qpd3WIiIisDgOgWuj7O5KfFQpOdkhERHQ3BkC1zD/XsnA08SbslAo81L6h3NUhIiKySgyAapn1h0Trz/0t66Oem6PMtSEiIrJODIBqkYIiLTYevQqAc/8QERGVhwFQLbLz72tIzylAPTdH3NeC634RERGVhQFQLaLr/nq4fUPYqfinJSIiKgvvkrVESkYe9pwVC58O78jkZyIiovIwAKolfjxyBVoJ6Bzkhcb1uPApERFReRgA1QJaraSf++dRtv4QERFViAFQLRB36Tr+Tc+Fq6MdFz4lIiKqBAZAtcD3t5OfB7X1g7MDFz4lIiKqCAOgGi4zrxBbTomFTx/l3D9ERESVwgCohtt8PAl5hVo0q++KewLqyF0dIiKiGoEBUA2n6/7iwqdERESVxwCoBvs7JRPHr2TATqnAsPYN5K4OERFRjcEAqAb7/tAVAEDvEB94u3LhUyIiospiAFRD5RdpsPGoCICGd+LcP0RERMZgAFRDbU9IxY3cQvi4O6JHMy58SkREZAwGQDWUbuZnLnxKRERkPN45a6Ckm7fwxz+6hU859w8REZGxGADVQBvir0CSgPBgLwR5u8hdHSIiohpH9gBo8eLFCA4OhlqtRocOHbB3795yy3/xxRcICQmBk5MTWrRogVWrVpVZdt26dVAoFBg6dKiJa215Gq2EA+fTsenoVXx78BIAtv4QERFVlawLR61fvx4TJ07E4sWL0bVrV3z11Vfo378/EhIS0KhRoxLlo6OjMWXKFCxduhSdOnVCXFwcxo4dC09PTwwaNMig7L///ovXXnsN3bt3t9TlmM3WU8l4d3MCkjPy9NsUAOxUnPiQiIioKhSSJElynTw8PBzt27dHdHS0fltISAiGDh2KuXPnligfERGBrl27Yt68efptEydOxOHDh7Fv3z79No1Gg549e+KZZ57B3r17cfPmTWzatKnS9crMzISHhwcyMjLg7u5etYszka2nkvHC6iMo7Y+kABD9VHv0C+MK8ERERMbcv2XrAisoKEB8fDwiIyMNtkdGRiI2NrbUffLz86FWqw22OTk5IS4uDoWFhfpts2bNQr169TB69OhK1SU/Px+ZmZkGD2ug0Up4d3NCqcGPzrubE6DRyhbDEhER1UiyBUBpaWnQaDTw8fEx2O7j44OUlJRS9+nbty+WLVuG+Ph4SJKEw4cPY/ny5SgsLERaWhoAYP/+/fj666+xdOnSStdl7ty58PDw0D8CAqwjtybu4nWDbq+7SQCSM/IQd/G65SpFRERUC8ieBH33Ap6SJJW5qOf06dPRv39/3HvvvbC3t8eQIUMwatQoAIBKpUJWVhaeeuopLF26FN7e3pWuw5QpU5CRkaF/XL58ucrXY0qpWWUHP1UpR0RERIJsSdDe3t5QqVQlWntSU1NLtArpODk5Yfny5fjqq69w7do1+Pn5YcmSJXBzc4O3tzdOnDiBS5cuGSREa7VaAICdnR3OnDmDJk2alDiuo6MjHB2tby2t+m7qigsZUY6IiIgE2VqAHBwc0KFDB8TExBhsj4mJQURERLn72tvbo2HDhlCpVFi3bh0GDhwIpVKJli1b4uTJkzh27Jj+MXjwYPTq1QvHjh2zmq6tyuoc7AU/DzXKGuulAODnoUbnYC9LVouIiKjGM7oFKCgoCM8++yxGjRpV6lB1Y0yePBkjRoxAx44d0aVLFyxZsgSJiYkYN24cANE1dfXqVf1cP2fPnkVcXBzCw8Nx48YNLFiwAKdOncI333wDAFCr1QgLCzM4R506dQCgxPaaQKVUYMagULyw+ggUgEEytC4omjEoFColh8MTEREZw+gWoFdffRU///wzGjdujD59+mDdunXIz8+v0smjoqKwcOFCzJo1C+3atcMff/yBLVu2IDAwEACQnJyMxMREfXmNRoP58+ejbdu26NOnD/Ly8hAbG4ugoKAqnb8m6Bfmh+in2sPL1cFgu6+HmkPgiYiIqqjK8wAdP34cy5cvx3fffYeioiI88cQTePbZZ9G+fXtT19HirGkeIJ3Nx5Lw8rqjCKrrjLkPtUHnYC+2/BAREd3BIvMAtW3bFp9++imuXr2KGTNmYNmyZejUqRPatm2L5cuXQ8b5FWulpIxbAIC2AXXQpUldBj9ERETVUOVRYIWFhdi4cSNWrFiBmJgY3HvvvRg9ejSSkpIwdepUbN++HWvXrjVlXW3alRsiAGro6SRzTYiIKqbValFQUCB3NagWcnBwgFJZ/TFcRgdAR44cwYoVK/Ddd99BpVJhxIgR+OSTT9CyZUt9mcjISPTo0aPalaNil2/kAgACPJ1lrgkRUfkKCgpw8eJF/TQkRKakVCoRHBwMBweHiguXw+gAqFOnTujTpw+io6MxdOhQ2NvblygTGhqKxx57rFoVI0PFLUAMgIjIekmShOTkZKhUKgQEBJjkkzqRjlarRVJSEpKTk9GoUaMyJ06uDKMDoAsXLuhHaZXFxcUFK1asqHKlyJAkSbiiawHyYhcYEVmvoqIi5Obmwt/fH87O/MBGplevXj0kJSWhqKio1EaYyjI6NE9NTcWff/5ZYvuff/6Jw4cPV7kiVLa07ALkFWqhUAB+HgyAiMh6aTQaAKh29wRRWXSvLd1rraqMDoDGjx9f6lpZV69exfjx46tVGSqdLv/Hz10NBzs2JxOR9atO1wRReUz12jL6bpqQkFDqXD/33HMPEhISTFIpMsT8HyIiItMyOgBydHTEtWvXSmxPTk6GnZ1sa6vWapevixaghsz/ISIiC7l06RIUCgWOHTtW6X1mzpyJdu3ama1OpmR0ANSnTx9MmTIFGRkZ+m03b97E22+/jT59+pi0ciSwBYiIbI1GK+HA+XT8fOwqDpxPh0Zr3sl1R40aBYVCAYVCAXt7ezRu3BivvfYacnJyzHpeU1MoFNi0aZNJjhUQEIDk5GSj1tJ87bXXsGPHDpOc39yMbrKZP38+evTogcDAQNxzzz0AgGPHjsHHxwfffvutyStIKB4BxkkQicgGbD2VjHc3JyA5I0+/zc9DjRmDQs26/mG/fv2wYsUKFBYWYu/evRgzZgxycnIQHR1domxhYWG1RiBVV3XOX9l9VSoVfH19jTq2q6srXF1dq1QvSzO6BahBgwY4ceIEPvroI4SGhqJDhw749NNPcfLkSQQEBJijjjaPLUBEZCu2nkrGC6uPGAQ/AJCSkYcXVh/B1lPJZju3o6MjfH19ERAQgCeeeAJPPvmkvjVF17WzfPlyNG7cGI6OjpAkCYmJiRgyZAhcXV3h7u6O4cOHG6SJ6Pb76quvEBAQAGdnZzz66KO4efOmwblXrFiBkJAQqNVqtGzZEosXL9Y/p+uK+v7773HfffdBrVZj9erVJeqvWxh82LBhUCgU+p/LqvvWrVvRrVs31KlTB3Xr1sXAgQNx/vz5EufVdYHt3r0bCoUCO3bsQMeOHeHs7IyIiAicOXOmxPXqjBo1CkOHDsXHH38MPz8/1K1bF+PHj0dhYaG+THJyMgYMGAAnJycEBwdj7dq1CAoKwsKFC4346xmvSkk7Li4ueO6550xdFyqFVivh6u0AiHMAEVFNI0kSbhVWbriyRithxi9/obTOLgmAAsDMXxLQtal3pdZDdLJXVWvEkJOTk8GN+ty5c/j+++/x448/QqVSAQCGDh0KFxcX7NmzB0VFRXjxxRcRFRWF3bt3l9hv8+bNyMzMxOjRozF+/HisWbMGALB06VLMmDEDixYtwj333IOjR49i7NixcHFxwciRI/XHefPNNzF//nysWLECjo6OJep76NAh1K9fHytWrEC/fv30dSyr7jk5OZg8eTJat26NnJwcvPPOOxg2bBiOHTtW7gSWU6dOxfz581GvXj2MGzcOzz77LPbv319m+V27dsHPzw+7du3CuXPnEBUVhXbt2mHs2LEAgKeffhppaWnYvXs37O3tMXnyZKSmppb3pzGJKmctJyQkIDExscRaL4MHD652pahYalY+CjRaqJQK+Lqr5a4OEZFRbhVqEPrO7yY5lgQgJTMPrWduq1T5hFl94exQtdtcXFwc1q5diwceeEC/raCgAN9++y3q1asHAIiJicGJEydw8eJFfQ/It99+i1atWuHQoUPo1KkTACAvLw/ffPMNGjZsCAD4/PPPMWDAAMyfPx++vr547733MH/+fDz00EMAgODgYCQkJOCrr74yCIAmTpyoL1MaXb3q1KlTouvq7roDwMMPP2xQ5uuvv0b9+vWRkJBQbt7P7Nmz0bNnTwDAW2+9hQEDBiAvLw9qden3KE9PTyxatAgqlQotW7bEgAEDsGPHDowdOxZ///03tm/fjkOHDqFjx44AgGXLlqFZs2Zlnt9UqjQT9LBhw3Dy5EkoFAr9qu+6KLu6ExORId0cQP511LBTcQ4gIiJz+fXXX+Hq6oqioiIUFhZiyJAh+Pzzz/XPBwYGGgQQp0+fRkBAgEH6R2hoKOrUqYPTp0/rA6BGjRrpgx8A6NKlC7RaLc6cOQOVSoXLly9j9OjR+hYRQMyo7eHhYVA/XYBQFXfXHQDOnz+P6dOn4+DBg0hLS9Ov3ZaYmFhuANSmTRv9935+IicrNTUVjRo1KrV8q1atDFqj/Pz8cPLkSQDAmTNnYGdnZzC9TtOmTeHp6WnkFRrP6ABowoQJCA4Oxvbt29G4cWPExcUhPT0dr776Kj7++GNz1NGm6RKgG9Zh/g8R1TxO9iokzOpbqbJxF69j1IpDFZZb+UwndA72qtS5jdGrVy9ER0fD3t4e/v7+JRKFXVxcDH6WJKnULraytuvonlMoFPqgY+nSpQgPDzcod2fQUNr5jVHavoMGDUJAQACWLl0Kf39/aLVahIWFlejZududvxfdtZS38O3dv8c7r1vXiHK3srabktEB0IEDB7Bz507Uq1cPSqUSSqUS3bp1w9y5c/HKK6/g6NGj5qinzbp8nfk/RFRzKRSKSndDdW9WD34eaqRk5JWaB6QA4OuhRvdm9SqVA2QsFxcXNG3atNLlQ0NDkZiYiMuXL+tbgRISEpCRkYGQkBB9ucTERCQlJcHf3x+AuI8qlUo0b94cPj4+aNCgAS5cuIAnn3yy2tdgb29fqZ6Y9PR0nD59Gl999RW6d+8OANi3b1+1z2+sli1boqioCEePHkWHDh0AiHylu5PEzcHoPhWNRqMf4ubt7Y2kpCQAonntzkxwMg19CxBHgBFRLadSKjBjUCgAEezcSffzjEGhZgl+qqJ3795o06YNnnzySRw5cgRxcXF4+umn0bNnT4PuKrVajZEjR+L48ePYu3cvXnnlFQwfPlyfpzNz5kzMnTsXn376Kc6ePYuTJ09ixYoVWLBggdF1CgoKwo4dO5CSkoIbN26UWc7T0xN169bFkiVLcO7cOezcuROTJ082/pdQTS1btkTv3r3x3HPPIS4uDkePHsVzzz0HJycnsy+nYnQAFBYWhhMnTgAAwsPD8dFHH2H//v2YNWsWGjdubPIK2jq2ABGRLekX5ofop9rD18MwodbXQ43op9qbdR4gY+kmHfT09ESPHj3Qu3dvNG7cGOvXrzco17RpUzz00EN48MEHERkZibCwMINh7mPGjMGyZcuwcuVKtG7dGj179sTKlSsRHBxsdJ3mz5+PmJgYBAQE6OfqK41SqcS6desQHx+PsLAwTJo0CfPmzTP6fKawatUq+Pj4oEePHhg2bBjGjh0LNze3MpOqTUUhGdnR9vvvvyMnJwcPPfQQLly4gIEDB+Lvv/9G3bp1sX79etx///3mqqvFZGZmwsPDAxkZGXB3d5e1Lt0/2onL12/hh3Fd0Cmo4j5vIiI55eXl4eLFiwgODq7WDUyjlRB38TpSs/JQ302NzsFeVtPyY4yZM2di06ZNRi0nYeuuXLmCgIAAbN++3WAUnk55rzFj7t9G5wD17VuczNa4cWMkJCTg+vXr8PT05Oq/Jlak0SLpppgMLIBdYERkQ1RKBbo0qSt3NcgCdu7ciezsbLRu3RrJycl44403EBQUhB49epj1vEZ1gRUVFcHOzg6nTp0y2O7l5cXgxwxSMvOg0UpwUClR363kpFdEREQ1XWFhId5++220atUKw4YNQ7169fSTIpqTUS1AdnZ2CAwM5Fw/FqLL/2ng6QRlDWz6JSKydTNnzsTMmTPlroZV69u3r0HvkqUYnQQ9bdo0TJkyBdevXzdHfegOxSPAmABNRERkSkbnAH322Wc4d+4c/P39ERgYWGJypSNHjpiscrbuMhdBJSIiMgujA6ChQ4eaoRpUGrYAERERmYfRAdCMGTPMUQ8qxZXruhYgBkBERESmxNU1rZiuBSjAi11gREREpmR0C5BSqSx3yDtHiJlGQZEWKZliDiC2ABEREZmW0S1AGzduxE8//aR/rF+/Hm+99Rb8/PywZMkSc9TRJiVn3IJWAhztlKjnyjmAiIis2X333YeJEyfqfw4KCsLChQvL3Ue3lEZ1meo4tsboFqAhQ4aU2PbII4+gVatWWL9+PUaPHm2Sitm6KzeK8384ySQRkfkMGjQIt27dwvbt20s8d+DAAURERCA+Ph7t27ev9DEPHTpUYpR0dZW1rEZycjI8PT1Nei5bYLIcoPDw8FJfPFQ1l68z/4eIbFxWCrBrrvhqRqNHj8bOnTvx77//lnhu+fLlaNeunVHBDwDUq1cPzs6Wef/29fWFoyN7CoxlkgDo1q1b+Pzzz9GwYUNTHI5g2AJERGSTslKAPR+YPQAaOHAg6tevj5UrVxpsz83Nxfr16zF06FA8/vjjaNiwIZydndG6dWt899135R7z7i6wf/75Bz169IBarUZoaChiYmJK7PPmm2+iefPmcHZ2RuPGjTF9+nQUFhYCAFauXIl3330Xx48fh0KhgEKh0Nf37i6wkydP4v7774eTkxPq1q2L5557DtnZ2frnR40ahaFDh+Ljjz+Gn58f6tati/Hjx+vPZSuM7gK7e9FTSZKQlZUFZ2dnrF692qSVs2WXdSPAOAkiEdVkkgQU5lZt36JbxV8Lcozf394ZqEQKgZ2dHZ5++mmsXLkS77zzjv4e98MPP6CgoABjxozBd999hzfffBPu7u747bffMGLECDRu3Bjh4eEVHl+r1eKhhx6Ct7c3Dh48iMzMTIN8IR03NzesXLkS/v7+OHnyJMaOHQs3Nze88cYbiIqKwqlTp7B161Z9b4uHh0eJY+Tm5qJfv3649957cejQIaSmpmLMmDF46aWXDAK8Xbt2wc/PD7t27cK5c+cQFRWFdu3aYezYsRVeT21hdAD0ySefGARASqUS9erVQ3h4OPsgTegKZ4EmotqgMBeY41+9YyzvV7X93k4CHCqXh/Pss89i3rx52L17N3r16iVOu3w5HnroITRo0ACvvfaavuzLL7+MrVu34ocffqhUALR9+3acPn0aly5d0veUzJkzB/379zcoN23aNP33QUFBePXVV7F+/Xq88cYbcHJygqurK+zs7ODr61vmudasWYNbt25h1apV+hykRYsWYdCgQfjwww/h4+MDQDRmLFq0CCqVCi1btsSAAQOwY8cOmwqAjO4CGzVqFEaOHKl/jBgxAv369aty8LN48WIEBwdDrVajQ4cO2Lt3b7nlv/jiC4SEhMDJyQktWrTAqlWrDJ5funQpunfvDk9PT3h6eqJ3796Ii4urUt3kVJwDxC4wIiJza9myJSIiIrB8+XIAwPnz57F37148++yz0Gg0mD17Ntq0aYO6devC1dUV27ZtQ2JiYqWOffr0aTRq1MggTaRLly4lym3YsAHdunWDr68vXF1dMX369Eqf485ztW3b1iABu2vXrtBqtThz5ox+W6tWraBSqfQ/+/n5ITU11ahz1XRGtwCtWLECrq6uePTRRw22//DDD8jNzcXIkSMrfaz169dj4sSJWLx4Mbp27YqvvvoK/fv3R0JCAho1alSifHR0NKZMmYKlS5eiU6dOiIuLw9ixY+Hp6YlBgwYBAHbv3o3HH38cERERUKvV+OijjxAZGYm//voLDRo0MPZyZZFXqEFqVj4AtgARUQ1n7yxaYior+5p4AEDKSWDL68CD8wDf1mKbq494VPbcRhg9ejReeuklfPHFF1ixYgUCAwPxwAMPYN68efjkk0+wcOFCtG7dGi4uLpg4cSIKCgoqdVxJkkpsu3t078GDB/HYY4/h3XffRd++feHh4YF169Zh/vz5Rl2DJElljhy+c7u9vX2J57RarVHnqumMbgH64IMP4O3tXWJ7/fr1MWfOHKOOtWDBAowePRpjxoxBSEgIFi5ciICAAERHR5da/ttvv8Xzzz+PqKgoNG7cGI899hhGjx6NDz/8UF9mzZo1ePHFF9GuXTu0bNkSS5cuhVarxY4dO4y7UBldvSm6v1wcVPB0tq+gNBGRFVMoRDdUZR9ejYFGXcSjYWdxjIadi7d5Na78sYycQmT48OFQqVRYu3YtvvnmGzzzzDNQKBTYu3cvhgwZgqeeegpt27ZF48aN8c8//1T6uKGhoUhMTERSUnEgeODAAYMy+/fvR2BgIKZOnYqOHTuiWbNmJUalOTg4VDjZcGhoKI4dO4acnOKcqf3790OpVKJ58+aVrrMtMDoA+vfffxEcHFxie2BgoFFNdQUFBYiPj0dkZKTB9sjISMTGxpa6T35+PtRqtcE2JycnxMXFlZm9npubi8LCQnh5eZVZl/z8fGRmZho85HRn/g/nACIisgxXV1dERUXh7bffRlJSEkaNGgUAaNq0KWJiYhAbG4vTp0/j+eefR0pK5Uem9e7dGy1atMDTTz+N48ePY+/evZg6dapBmaZNmyIxMRHr1q3D+fPn8dlnn2Hjxo0GZYKCgnDx4kUcO3YMaWlpyM/PL3GuJ598Emq1GiNHjsSpU6ewa9cuvPzyyxgxYoQ+/4cEowOg+vXr48SJEyW2Hz9+HHXr1q30cdLS0qDRaEr8QXx8fMp8YfXt2xfLli1DfHw8JEnC4cOHsXz5chQWFiItLa3Ufd566y00aNAAvXv3LrMuc+fOhYeHh/4REBBQ6eswB+b/EBEBcPMFer4lvlrI6NGjcePGDfTu3VufijF9+nS0b98effv2xX333QdfX18MHTq00sdUKpXYuHEj8vPz0blzZ4wZMwazZ882KDNkyBBMmjQJL730Etq1a4fY2FhMnz7doMzDDz+Mfv36oVevXqhXr16pQ/GdnZ3x+++/4/r16+jUqRMeeeQRPPDAA1i0aJHxv4zaTjLS66+/LgUGBko7d+6UioqKpKKiImnHjh1SYGCg9Oqrr1b6OFevXpUASLGxsQbb33//falFixal7pObmys988wzkp2dnaRSqSR/f3/pjTfekABI165dK1H+ww8/lDw9PaXjx4+XW5e8vDwpIyND/7h8+bIEQMrIyKj09ZjS3C2npcA3f5Vm/HxKlvMTEVXVrVu3pISEBOnWrVtyV4VqqfJeYxkZGZW+fxvdAvT+++8jPDwcDzzwAJycnODk5ITIyEjcf//9RuUAeXt7Q6VSlWjtSU1NLbOZzsnJCcuXL0dubi4uXbqExMREBAUFwc3NrURe0scff4w5c+Zg27ZtaNOmTbl1cXR0hLu7u8FDTro5gDgJIhERkXkYHQA5ODhg/fr1OHPmDNasWYOffvoJ58+fx/Lly+Hg4GDUcTp06FBiNsyYmBhERESUu6+9vT0aNmwIlUqFdevWYeDAgVAqiy9l3rx5eO+997B161Z07NjRuAu0ApwDiIiIyLyMHgav06xZMzRr1qxaJ588eTJGjBiBjh07okuXLliyZAkSExMxbtw4AMCUKVNw9epV/Vw/Z8+eRVxcHMLDw3Hjxg0sWLAAp06dwjfffKM/5kcffYTp06dj7dq1CAoK0rcwubq6wtXVtVr1tZQrzAEiIiIyK6NbgB555BF88MEHJbbPmzevxNxAFYmKisLChQsxa9YstGvXDn/88Qe2bNmCwMBAAGKF2ztHlmk0GsyfPx9t27ZFnz59kJeXh9jYWAQFBenLLF68GAUFBXjkkUfg5+enf3z88cfGXqoscguKkJ4j5pZgCxAREZF5KCSplBmaylGvXj3s3LkTrVu3Nth+8uRJ9O7dG9euXTNpBeWQmZkJDw8PZGRkWDwf6Oy1LER+8gfc1XY4MbOvRc9NRFRdeXl5uHjxIoKCguDkxFZsMr1bt27h0qVL+lUk7mTM/dvoFqDs7OxSc33s7e1lnz+nNriiT4Bm6w8R1Ty65RUqO0sykbF0r607l/KoCqNzgMLCwrB+/Xq88847BtvXrVuH0NDQalWGgMvXRQI083+IqCays7ODs7Mz/vvvP9jb2xsMUCGqLq1Wi//++w/Ozs6ws6tyGjOAKgRA06dPx8MPP4zz58/j/vvvBwDs2LEDa9euxYYNG6pVGWILEBHVbAqFAn5+frh48WKJpRyITEGpVKJRo0bVXinB6ABo8ODB2LRpE+bMmYMNGzbAyckJbdu2xc6dO2WfP6c20LcAcQ4gIqqhHBwc0KxZM3aDkVk4ODiYpGWxSu1HAwYMwIABAwAAN2/exJo1azBx4kQcP368woXaqHxXbrIFiIhqPqVSWSJBlciaVDmE2rlzJ5566in4+/tj0aJFePDBB3H48GFT1s0mFecAMQAiIiIyF6NagK5cuYKVK1di+fLlyMnJwfDhw1FYWIgff/yRCdAmkJlXiIxbYlV7LoNBRERkPpVuAXrwwQcRGhqKhIQEfP7550hKSsLnn39uzrrZnCu3W3+8XBzg4li97HYiIiIqW6Xvstu2bcMrr7yCF154odpLYFDprnARVCIiIouodAvQ3r17kZWVhY4dOyI8PByLFi3Cf//9Z8662ZziRVAZABEREZlTpQOgLl26YOnSpUhOTsbzzz+PdevWoUGDBtBqtYiJiUFWVpY562kTLt9uAQrgCDAiIiKzMnoUmLOzM5599lns27cPJ0+exKuvvooPPvgA9evXx+DBg81RR5vBFiAiIiLLqNZMQi1atMBHH32EK1eu4LvvvjNVnWzW5eu3c4A4BJ6IiMisTLJIi0qlwtChQ/HLL7+Y4nA2SZIkXL3BWaCJiIgsgavUWYmMW4XIyi8CwFmgiYiIzI0BkJXQ5f94uzpCba+SuTZERES1GwMgK6HL/wnwYvcXERGRuTEAshLFI8DY/UVERGRuDICsRPEcQGwBIiIiMjcGQFaCLUBERESWwwDISjAHiIiIyHIYAFkBSZLYAkRERGRBDICsQHpOAW4VaqBQAP511HJXh4iIqNZjAGQFdK0/Pm5qONpxDiAiIiJzYwBkBZj/Q0REZFkMgKwA83+IiIgsiwGQFeAcQERERJbFAMgKsAWIiIjIshgAWYErt3OAGjIHiIiIyCIYAMlMq5Vw5aZoAQpgCxAREZFFMACS2X/Z+Sgo0kKlVMDPg3MAERERWQIDIJlduZ0A7euuhp2Kfw4iIiJL4B1XZpev3+7+Yv4PERGRxTAAkpmuBYgjwIiIiCyHAZDM9C1ADICIiIgsRvYAaPHixQgODoZarUaHDh2wd+/ecst/8cUXCAkJgZOTE1q0aIFVq1aVKPPjjz8iNDQUjo6OCA0NxcaNG81V/Wq7clPXAsQuMCIiIkuRNQBav349Jk6ciKlTp+Lo0aPo3r07+vfvj8TExFLLR0dHY8qUKZg5cyb++usvvPvuuxg/fjw2b96sL3PgwAFERUVhxIgROH78OEaMGIHhw4fjzz//tNRlGaV4EkQGQERERJaikCRJkuvk4eHhaN++PaKjo/XbQkJCMHToUMydO7dE+YiICHTt2hXz5s3Tb5s4cSIOHz6Mffv2AQCioqKQmZmJ//3vf/oy/fr1g6enJ7777rtK1SszMxMeHh7IyMiAu7t7VS+vQhqthJbT/4dCjYTYt+6Hfx0GQURERFVlzP1bthaggoICxMfHIzIy0mB7ZGQkYmNjS90nPz8farXhXDlOTk6Ii4tDYWEhANECdPcx+/btW+YxdcfNzMw0eFjCtcw8FGok2KsU8HHnHEBERESWIlsAlJaWBo1GAx8fH4PtPj4+SElJKXWfvn37YtmyZYiPj4ckSTh8+DCWL1+OwsJCpKWlAQBSUlKMOiYAzJ07Fx4eHvpHQEBANa+uci7fXgLDv44TVEqFRc5JREREVpAErVAY3vglSSqxTWf69Ono378/7r33Xtjb22PIkCEYNWoUAEClUlXpmAAwZcoUZGRk6B+XL1+u4tUYh/k/RERE8pAtAPL29oZKpSrRMpOamlqiBUfHyckJy5cvR25uLi5duoTExEQEBQXBzc0N3t7eAABfX1+jjgkAjo6OcHd3N3hYwuXbcwBxCDwREZFlyRYAOTg4oEOHDoiJiTHYHhMTg4iIiHL3tbe3R8OGDaFSqbBu3ToMHDgQSqW4lC5dupQ45rZt2yo8phzYAkRERCQPOzlPPnnyZIwYMQIdO3ZEly5dsGTJEiQmJmLcuHEARNfU1atX9XP9nD17FnFxcQgPD8eNGzewYMECnDp1Ct98843+mBMmTECPHj3w4YcfYsiQIfj555+xfft2/Sgxa6LLAQrwYgsQERGRJckaAEVFRSE9PR2zZs1CcnIywsLCsGXLFgQGBgIAkpOTDeYE0mg0mD9/Ps6cOQN7e3v06tULsbGxCAoK0peJiIjAunXrMG3aNEyfPh1NmjTB+vXrER4ebunLqxBbgIiIiOQh6zxA1soS8wAVarRoMe1/0EpA3NsPoD6HwRMREVVLjZgHyNalZORBKwEOdkp4uzrKXR0iIiKbwgBIJrr8n4aeTlByDiAiIiKLYgAkk+L8HyZAExERWRoDIJkUzwHEBGgiIiJLYwAkE7YAERERyYcBkEyK5wBiCxAREZGlMQCSCVuAiIiI5MMASAb5RRpcy8oDwBwgIiIiOTAAkkHSzTxIEuBkr4KXi4Pc1SEiIrI5DIBkcGf+j0LBOYCIiIgsjQGQDJj/Q0REJC8GQDLgHEBERETyYgAkA7YAERERyYsBkAw4BxAREZG8GADJgC1ARERE8mIAZGG3CjRIy84HAAQwACIiIpIFAyALu3pTdH+5OdrB3clO5toQERHZJgZAFnb5dvdXA0/OAURERCQXBkAWdkWfAM3uLyIiIrkwALKw4gRojgAjIiKSCwMgC9JoJRy7fEN8r5Gg0Uoy14iIiMg2MQCykK2nktHtw53486IIgFYd/BfdPtyJraeSZa4ZERGR7WEAZAFbTyXjhdVHkJyRZ7A9JSMPL6w+wiCIiIjIwhgAmZlGK+HdzQkorbNLt+3dzQnsDiMiIrIgBkBmFnfxeomWnztJAJIz8hB38brlKkVERGTjGACZWWpW2cFPVcoRERFR9TEAMrP6bmqTliMiIqLqYwBkZp2DveDnoUZZcz4rAPh5qNE52MuS1SIiIrJpDIDMTKVUYMagUAAoEQTpfp4xKBQqJZfFICIishQGQBbQL8wP0U+1h6+HYTeXr4ca0U+1R78wP5lqRkREZJu4HLmF9AvzQ59QX8RdvI7UrDzUdxPdXmz5ISIisjwGQBakUirQpUlduatBRERk89gFRkRERDaHARARERHZHAZAREREZHNkD4AWL16M4OBgqNVqdOjQAXv37i23/Jo1a9C2bVs4OzvDz88PzzzzDNLT0w3KLFy4EC1atICTkxMCAgIwadIk5OVxpmUiIiISZA2A1q9fj4kTJ2Lq1Kk4evQounfvjv79+yMxMbHU8vv27cPTTz+N0aNH46+//sIPP/yAQ4cOYcyYMfoya9aswVtvvYUZM2bg9OnT+Prrr7F+/XpMmTLFUpdFREREVk7WAGjBggUYPXo0xowZg5CQECxcuBABAQGIjo4utfzBgwcRFBSEV155BcHBwejWrRuef/55HD58WF/mwIED6Nq1K5544gkEBQUhMjISjz/+uEEZIiIism2yBUAFBQWIj49HZGSkwfbIyEjExsaWuk9ERASuXLmCLVu2QJIkXLt2DRs2bMCAAQP0Zbp164b4+HjExcUBAC5cuIAtW7YYlLlbfn4+MjMzDR5ERERUe8kWAKWlpUGj0cDHx8dgu4+PD1JSUkrdJyIiAmvWrEFUVBQcHBzg6+uLOnXq4PPPP9eXeeyxx/Dee++hW7dusLe3R5MmTdCrVy+89dZbZdZl7ty58PDw0D8CAgJMc5FUO2SlALvmiq9ERFQryJ4ErVAYzoQsSVKJbToJCQl45ZVX8M477yA+Ph5bt27FxYsXMW7cOH2Z3bt3Y/bs2Vi8eDGOHDmCn376Cb/++ivee++9MuswZcoUZGRk6B+XL182zcVR7ZCVAuz5oGoBEIMnIiKrJNtM0N7e3lCpVCVae1JTU0u0CunMnTsXXbt2xeuvvw4AaNOmDVxcXNC9e3e8//778PPzw/Tp0zFixAh9YnTr1q2Rk5OD5557DlOnToVSWTLmc3R0hKOjo4mvkGoFrQa4Gi++TzsDuHgDLvUAu0q+XnTBU4v+gJuv+epJRERGkS0AcnBwQIcOHRATE4Nhw4bpt8fExGDIkCGl7pObmws7O8Mqq1QqAKLlSFfm7iBHpVJBkiR9GaIKZaUA6eeBHe8Blw+IbT89V/y8oxvg6gO41Adc64mgyKW+CJBc6xd/X5grT/2JiKhcsq4FNnnyZIwYMQIdO3ZEly5dsGTJEiQmJuq7tKZMmYKrV69i1apVAIBBgwZh7NixiI6ORt++fZGcnIyJEyeic+fO8Pf315dZsGAB7rnnHoSHh+PcuXOYPn06Bg8erA+WiCq0dwEQ91XZz+dniUf6ucod78AiIPwFQKkSLUFsDSIikpWsAVBUVBTS09Mxa9YsJCcnIywsDFu2bEFgYCAAIDk52WBOoFGjRiErKwuLFi3Cq6++ijp16uD+++/Hhx9+qC8zbdo0KBQKTJs2DVevXkW9evUwaNAgzJ492+LXRzXUxT+AE+vE9051gY6jgL3zgUGfAb5tROCjVAGSFsj5TzyyU4u/TzoKZCUbHvPkD+IBAD3fBHq9bdFLIiIiQwqJ/UIlZGZmwsPDAxkZGXB3d5e7OmRJh74G/vcGoC0C/NsDj60Fsq8BS3oCz+0B/NtVfIyslOKk58t/iuPZOxd3hzXqAjw4D/BtbbbLICKyRcbcv2UfBUZkFTSFwG+vAr9NFsFP2CPAM1sAdz/jj+XmKwIl/3ZAQLjY9sT3QMQrgMoBSDwAfNkd+Hk8R4cREcmEARBR7nVg9UPAoWUAFMAD7wAPLwPsncTzbr5Az7eql7fj6AZEvge8dAhoNQyABBxdDXzWHtjzEVDAZGkiIktiAFSTcE4Z0/vvDLD0fpH34+Aqury6vwrcOReVmy/Qa0rVAqC7gyfPIODRlcCz24CGnYDCHGDXbODzDsCx7wCt1hRXJT++VonIyjEAqkmqMyEflXR2G7CsN3DjIlCnETB6G9DyQdOeo6zgqVE4MDoGeGQ54NEIyEoCNo0Dlt4HXNxbXK6mBhJ8rRKZX019f7ASDIDI9kgSsP8zYO1wID8TCOwKjN0F+LSybD0UCiDsYdEt1nsm4OgOJB8HvhkIfPcEkHaOgQQRlY3vD9Ui6zB4qoQ7RxSd+lF8PfO/4uc5p4xxivKBzROB42vFz+1HAg9+DNg5yFcnezXQbRJwzwhg91zg8ArgzG/AP78DoaVPCmqV7nyt/vWT+Hp+Z/HzfK0SkRXhMPhSWNUw+F1zRYRflp5viS4WKltWiggqWg4Uo7yuxAEKFdBvLtD5OcN8H2twab/hDNQAEBgBhI8D6gRabyDB1yrVRLr3h47PVO3/qrr7V+V8WSkAJGDn+8C57WJusRa3u++t9f3BQoy5fzMAKoVVBUC6F/vNf4HvnzZ8LmQQ0Gc24BUoT91qiqRjYh4fl/pATiqg9hCJyE3ul7tmpasokGgTBQz7yvoCN93yIZteEK9XnYgJQNhDNv/GbPUsfSO3Frr3h8rO82Xq/Y3FDxrl4jxAtYluTpnkE8XbOjwDQAGc3gx8/xRw/aJctbN+WdeAU7e7Y3JSgbrNgDE7rTf4AcQN6Lk94jHoM7HNt03x8yfWA9ERwJFVQGGePHUsjUt9IPZzEfw4exdvP7QMcPK0rZtqTWTr+SSaQpEfaO06PgMMXiRase9mpxZTahTkWL5eNRBzgGoCSQJOfl/8c4dRovXnxzFAygnx6WPol6YfwVQTabVA8jHg5I/AuRixgrtOvZYi36cgW7zJW+sNubSWksGfA2p34M+vgCPfAqkJwC8vA9vfBTqNATqNFouwymn3XODs/wCVI9D3fWDjOBG4pZwANk8ARmy0vlYrKibVkikYKkPfsn4Z2HN7KaWve4uvSnsRSNirxVeVg/hq51D8MxSApAHsHMXSOIAYwKBjztZOhUpMnSFpxAe58zuBwV8AR1YCVw4BBz4DTm0AHpgOtHkMULKdoyzsAiuFVXWBAUDiQWB5X8DeReSBhD8n/rkyrgA/jBIvegDoOhG4fzqgssK41pz97PlZwIXdwNmtwD8xYumKitSUZuLSmtdv3RStP3FLgIzLYpvKAWg9HOjyYvFoNkt2aST8Anw/Qnw/9EugSS9x7sb3Ad8OBYrygCFfAPc8Zd56kHF0gYAkia7L/04D3V8TH7CA2tttue0dIPZT8x3fXO8vWo34f7r4B+DdAhj0KbCin3h/8GsrBh/EzAQybq+h6dsG6DsbCO5h+rpYKeYAVZPVBUC/TgIOLwfaPgEMizZ8rqgAiHkH+PP29sBuYm4ZNx/L17M8pu5nv34BOPu7eFzaB2gLi8s6uIr1thq0BwLuFUHC5ldEd5JfW1GmpryxlxfEaIqA078AB74Arh4u3t74PuDe8YBLPTGvkLlzE64liPmUCnOAe18UyeV32rcQ2D5D5F6Nj6sZv3e5WDoPx9bySYrygbil4pp1LTf1Q0WLav8PRVChKRCvVUc3EbgXFYivmoLin7OvAblpYlvSUeDfWHGsnm8BLfqb7/1lxyyxMLO9C/DcLlHPu18vhXnAn1+KcvmZYluLB4E+swDvZqavk5VhAFRNVhUAFRUA85sDt26ILoSyclf+2gj8/JLo3nH1EUFQUDfT1aOqb8wFuSJYObdd3ATvfweo11x0k9jd8dD/rDbcprIX3SZXDgPLHhCtHElHgfR/DM/j1Rho3g9oFilGTNk5Fj9n6SRFOVyOE4HQ6V+KuzI8AkTwN3aXCAbNIfc6sLQXcOOS+JT51MaSLZCaIvG3Sz4mRuJFrWZXWFks/VrNShGtCRtfAKSi4u0DPhGvmZryQaEikiT+N2LeEa9VAKjfSnTVOnmZ5sMZILqnolabJx3hzFbguyjx/cNfA60fKb98TlrxtBqSBlDaAR1HA/e9BTh7iTK1MPHdmPu3FfaVkIHzO0Tw4+oDBPcsu1yrYYBPGLB+hGjG/maQWNMqYoJp+oB1CZK6Tzd30hQCN/4Frp8H0s8VP/47C2TflVC5c5bx51Y5FC8RocuFUqiAhh2BkMEi8PFuavxxa5OAzuJx5bDIE/r7t+LusY3PA/0/Kk5ENtUbnaYI2PCsuKHUaQQ8srL07leVnej+WtIT+PtXEayHPWSaOlijmnRTUXsAf3wsgp/AbsC/+8T242tF/S0VqJrzd3YlHtg2VSxCDIj30vunAe2eBJQqEcCYQvP+IgduwzPAUz8BQV1Nc1xADHTZ+Jz4vvNzFQc/AODiDQyYL8pvmy7mFYv7CjixDujxuthe3vu6DWAAZO1OrBdfwx4R/6zl8W4GjN0B/DpZvMi3zxQtA0MXi5tfdd5kdA2FV+NFd0v6+duPc+IGKGmMvTIxUkjtXrKJWVNgWO7unwFxvsa9gIiXKj6PKRYzrSn+iTFMmAeAtLMibwAwbZfGjpnAhV2AvbNYQ82lbtllfcPEGmt7PgS2vC666XSfQmubytxU8rOLPzAkHQNST4ucPl3rxD8xxWXN2QoT844YKODqA/R8A1i1T7S8Xjkkupdb9DPPee9mjhvxzcvAjneBkz+In+2cgIiXga4TAEfX4nLVfX/Q7X/P7Ry4s/8DvnsMeGYL4Nu6etcAiC6tH0YCeRlAg45A5Gzj9q/XAnjye5En+fs04NpJYNs0MTqzwzPVr18Nxi6wUlhNF1heJvBxMxEUPLcb8L+ncvtJEhC/EvjfGyJ4qBMIDF8lniuvqVeSgNz02y0450WXRerf4o0547Jhns3d7J0BryZA3SZA3abi4VRH9FWrby/xUNk8HK1W1Pvmv0BmEqDJB1L+Eq1HNTGPx5LunI1Z9zt39bmdGK4Eer4O3Del+p/sT3wP/DRWfP/Iisq16BQVAF/1EC2UbaKAh5ZUrw7WStclMnq7eO3f2SqafkF8vbtltDzmysP5JwZYc7sl4akfRQvy4RXiPeDQUvHz83stM4qoul1/d364s3cG9n0CHFws3jsBoO3jYoCIRwNT1rqkwlvAtw8BibFiWojR2wCv4Ood85dXgCPfAM51gef/ADwaVv1YWg1wMBrYuwC4lV68vRa9r7ILrLb4+1fxD+zdHPBrV/n9FArxRuDfDvh+pAgkvo4Un34AoDBXvOHoAp30c8WfRvMyKn+eZn1FC0zdpoCbX+Vuqn5tK36DUyoBpVp8cqnXQmxz9RUBUGX2t2WlvXk9+o34tHdqg2iBufaXaBVUe1TtHElHxRB8AOg2ufLdWXYOoivs696iZTPsEaB5ZNXqYG2uXwDO7wZSThZ3tXzdB0A5ny9d6okPDe7+ojW0TgBw67q4eQMiOXfgQsDTDBOdZv8HbHpRfB8+Dmh6ewh4rykir+vEeuDaKSBho1ivzhzuDNaPrxNft7wuPkQ5eQKeQaJV26W++F051y17hKuuBUlbIKaJyPlPbA/qDkS+b7n3DHsn4PHvgJUDxO/v26HAs9uqPijl6BoR/EABPLysesEPIHoR8rMMgx9AfFDSqW2J7+VgAGTNdN1frYdX7RO7/z3Akz+Ita8SY4G9H4vtK/qXs5NCJM/WbQy4+Ys3nToB4p9m53u16pOCzbB3Em+egV2ArVNEYL0kQbQKGttEn/0fsO4pEZg3ixS5FMZo2EGMFDuwCPh1IvDiQdFKYk0q01Wckyamp0g8IB5Xj6BksHPHz/VCxLpudZuK/y2vJqKF9G5Jx0QA5OAqRiYdWCQCWFOSJBHA5qSKevWeafi8s5f4sLRrNrBrDhAyxDxTaxxeUXIE2pU48SiVQrwfudQDXOsVB0au9YBbtz+47V0gvno1ASLfE6OfLJ1w71RHtKgt7yu6NFc/DDzzm/EfOFJOiqV7ANFqa6rJWzs+I7oaIQFrHxOtka0eEl2DgE29pzMAslaZyWJ0BlC5hLeynPpJBD9l8W0tXvy6biuvYHHDvFvSMREAVbUFxlT97Db0z1ltd/7OFAoxYaL/PaJV8PoFMXR9wPzKz82jKRS5CJlXxGvloaUV56WVptdUkaR946LIQRm00PhjmNPd+SiSJG5kiQfF/1LiQZFXdTeXemLeFWcvkXcSOft2IqzC+A8LfecAW167PXJpupjLxVTiV9yesNIBeHhp6f/v974ghlKnnwOOfwe0H2G68+t0fEbkGG19u/g9quUgMQHhrRsiTyo/U7Tm5KQBkMTQ89w00Y1aGntnoOMYIHSw+OAm12hDN18xavfrviLn5rvHRVBU2u+6NHkZYumjojygaR+RtGzKuulei+2eAPYtEK/ph7+2uUkTmQNUCqvIAYpdJEYuBISLfuSqurOZ+cIukRjd/0MxPw5Q+TdmWxhKbityrwM/PSdmygZEAPTgxxW/Of/2quhKc3ATyfa67smquLgX+Gag+H7kr0Bw96ofy9SuHhFD+yNeEblviQeBrOSS5eqFAI3uFdMuNLpXjIQDqve/cmfr08W9wE9jxPYHPwY6j63OVQlp/wBfdgeKbomuIV23eGl070EeAcDL8YZTS5iKLpdMYSdGopX1O9NqRG5Szn9AdqoIiI6vFbMgl8UaunKST4jusPxM0Ro1/NuKW9MkCVj/lGip9QgQeT/mGjCQeABYfjvR/YnvgeZ9zXMeC2IOUG2gG8nT+tHqHadEgDNTBD/GvjGzBab2cPYSb3b75osujqOrgaTjwPBvRP5FaeJv5xEBotWgOsEPIAKeDs+I1ohfXgZeiAUcnKt3zOrQfVBIPS1aXAAg9rPi5xV2Yl6cwC5iks2AcPPclNx8i2/abR4V+Xs73xMDGjwa3u66qCJNoQg2im6JKTXuHV9++U6jRRdcxmUxqCL8+aqfuzRZKSLnBwA6jgQOfV12WaVKLPXiWr94pvPg7iUT/u/uopebXxuRE/TtQ8CZLWJJmCGLym+Ziv1MBD8qB/E/ac7Rkp7BQMNOYtTfn1/WigDIGLbV3lVT/HdG/EMr7UT3lDXQvTFbw5sKVZ9SKZrVR2wUCbjXTgJL7hML7N4t8U/R+gOI7qvq3ITv1GcW4N5AdIXtMmEXT1Uc+lq02mwaV5xAe6duE4AxMaLOLfqXf1My5YeF7q8C7Z8Wk1tuePZ2rlEV7Z4rEtjVdYCh0RV3d9g7iaHxAPDHPNMusClJYob7vJtigEfXycb/znQLRfu3Kw56dF30/u2s570qqBvw6ApAoQSOrRYTwpbl0n6xvh8A9PsAaNDBvHVz8xX5gVCI1rT/zlS4S23CAMganbjd+tO0d/lzqxiLrTh0t8b3AeP2ilbB/EzR9P77VNFakJUi8jPWPSGmQAgZJNaJMhW1uxjlBIghy1cOl1vcbG5cEjOV6+immxj0meiSeW6PmDSuskz5YUGhAAYsAJo8IEZvro0CbiYaf5x/Y4sThAd9Wvnh4PeMEKOxcv4TE2yayskNokVEaS9GJNZpWLs/YLUcIF5PALD/U2D/ZyXLZKWISRQljZgmouOzlqmbZ5DongPE+oI2hAGQtZGk4om7qtv9dTe24lBp3P2BUb8CXW5PKnlgkZhJ/OoR4OAXIum0fqhY5NTUSZLNI8WbvaQFfh4v1mqyFEkSw6+juwFJR0Ru07CvgIG3h6FbS2uCyh54dKWYlycnFVjzqFgQt7LyMoCfngcgidmPWw017tz3vS2+37/QuPOWJesa8L/bXV893yzu0qqOmvDhrv0IoPft1p2Y6cCxteL7rBRg52wxujL7msgtG/iJZRO4dd2bx74zzd+4hmAAZG0ux4l+fwfX4qicyNxU9mKk0fBvAUd3kRy57knxnIMr8Ngaw9lzTanfB2IE1X9/i2UZLOHWTeDH0WKZkIIs0QL2wj6g7WMArHCdMrW7yNty8xe/p/VPiYklK+O318Tq4J5BYgCEsVo/Im7KeRlA7OfG738nSRJDu2/dECPmuk2s3vF0asqHu64Tij9o/PwScOZ/IgD64yPg6iERhEd9Czi4WLZewT3E37gwBzi2xrLnlhEDIGujm/snZJC8SaFkmwI6i/wQryYAbq+/1v5pMSt50rHipFNTcvYCHpwnvt+3QMx/Yk6X9gHRXYFTP4o15XpNA0b9JgIEwHpbEzwaiCUNHNyAS3tF8nhFg3hPbhADKhQqMW2Bo5vx51WqgPuniu8PRou5oKrq1I8iwVdpJ15nKvuqH6smUijE6Lu2T4iurh9GGSZ/D1kkz4rtCkVxK1DcEjHqzgYwALImRQVioUjA9N1fRJVxeAWw/kkxM7jOwcUiQXhJT/G8OYQOFSvFa4vEJ+ObV4Bdc00bcBUVADEzgJUDxVxGXo3FFBM9XzccmmzNrQm+rYHhK0VAc2KdSGwuy83LYl1AQCS8B3Su+nlbDhS5UYU5IkitiuxUMbcRAPR4Q6wPZ4sUCjGYoFEXMc/P0dvLFDXuJYJwc33QqEib4SJB/sYlw7XoajEGQNbk/A4xFX5FK78TmUvHZ4oTf3VJm3cmA3c00+KJCoWYlFHtIdagi/1UTEZoqhvBf2fFEhz7FwKQRHLv83uBhh1Nc3xLatq7OE9pz4diuYS7aTXAxnFA/u0FNKs7kZ5CIdbSAkSLRcYV4/bXjfq6dUMEcd0nV68+Nd3Rb4uXTNG5sMv8HzTK4+BSPOHln19a/vwy4DxA1kQ3+ivsYfNMPU9UkdImxrTU+mtuvkDfucDPL1b/BqCbULDDKDHa6PepYv4bJ08R0IUONkmVZdPh9hp/e+eL+W/c/YEmvYqfj/0M+HefWIz4oSWmeT9pcj8Q2E0cd89HwOBSRjKV5a+fbLvr62665SiKbolE/PiV1jGHUaexwIEvRDCW+jdQv6U89bAQ3mWtRV6meKMGRFMkka3JSgF8QosnZgOATS+IZTdcfcSiwL6txYSAbn7l39R1y1lc/KN4mYXGvcTN193P/NdiCb2mATf+FYvcfv808OzvIp9q11wxuSUgkp7LmtzSWAoF8MB0scbV0dUiobcyx85OFYnYgGiJMnb9udrozg8adk4iALKGhZ49A8Xgm79/FblAA6vY3VlDMACyFrqV3+s2M27ldyJzsXQycGmLY6YmiMfdFEoxIsqjoVjzyaPh7cft7y/tF+USY8WMur3fFaue16a1jpRKMYdOZpK4zjWPAg9+BBxZKZ4PGVT5dd4qq9G9YhHcf7aJ/KOHl5VfXj/q6zrg0xroZuNdXzVB+PPifnT8O+CBd0pftLeW4FpgpZBlLbBVQ0WzY6+pxbOvEtmSO9etSzwAbH0LaPOYCGCyr4mh6znXgIyrYmLGynDzE4uS1m1i/IKkNUXudeDrSCD9H0DlCGjyxezeLx0yzzIKyceBr3oAUAAv7C9/Hp9TP4nJ/ZR2wNhdYmkIMnTn+m/W8PqUJCA6QnzwiJwNRLwkd42MwrXAapqsFODiHvE9R3+RrSotQLn3hZLdAlqtmBDw5mWxTlXGFfE4v0OsXn6nrGTgx9sz6lrD4pjmoCkQS3RsekEsLQGIbvSbieJh6sDPr60YtZewSUzg9/ja0stl/1c86qv7qwx+ynLn+m/WQDckfvME4NBS8T+oVMldK7NgAGQNTv0oZsJt2BnwCpa7NkTWTaksvqkHdCrefmcLUtIx4NcJ1pFYam6ldR0eXCwegHkCv15TgdO/AGd+A67EAw1LWbNqy6tiBXefMNMuoULm13q4mDLixiXR3Wmq9f+sDAMga6Ab/cXkZyKhKvlHco5gk5NuRBFguVXR6zUH2j4uZg3eOQt4+mfD5//aCCT8LOYrGroYsHMwfR3IfBycxUjD/Z+KIfG1NACSPSNw8eLFCA4OhlqtRocOHbB3795yy69ZswZt27aFs7Mz/Pz88MwzzyA9Pd2gzM2bNzF+/Hj4+flBrVYjJCQEW7ZsMedlVN1/Z8W8J0o7oNUwuWtDZB2seTJCayPXqug93xSLmV7YLUbb6eSkAb+9Kr7v/mpxnahm6TRGDDa4sFsMia+FZA2A1q9fj4kTJ2Lq1Kk4evQounfvjv79+yMxsfTVjvft24enn34ao0ePxl9//YUffvgBhw4dwpgxY/RlCgoK0KdPH1y6dAkbNmzAmTNnsHTpUjRoUMnVjy3t5O3WnyYPAC7e8taFqLaw1uUsahPPQDHPEgDseK94WY4tr4mur/qtqj8BI8mnTqM7Von/St66mImso8DCw8PRvn17REdH67eFhIRg6NChmDu35BTvH3/8MaKjo3H+fPE0/Z9//jk++ugjXL58GQDw5ZdfYt68efj7779hb1+5ybby8/ORn1+8CnVmZiYCAgLMPwpMkoBP24oJzR7+Wiw6SERUVZYeUZSVAnzaTkzoN+wrsbhnwibR9TV2h1g+g2qui3uBbwYC9s7A5AQxkaiVM2YUmGwtQAUFBYiPj0dkZKTB9sjISMTGxpa6T0REBK5cuYItW7ZAkiRcu3YNGzZswIABA/RlfvnlF3Tp0gXjx4+Hj48PwsLCMGfOHGg0ZS/uNnfuXHh4eOgfAQEBprnIinDldyIyJUt3Hbr5AuHPie93zRXBDwB0m8TgpzYI6iZa8gpziyfXrEVkC4DS0tKg0Wjg4+NjsN3HxwcpKaWv/xMREYE1a9YgKioKDg4O8PX1RZ06dfD555/ry1y4cAEbNmyARqPBli1bMG3aNMyfPx+zZ88usy5TpkxBRkaG/qFrTTI7XfdXy4Fc+Z2IaqauEwFHd+DmJfGzZzDnMqstDFaJX1rrVomXPQlaoVAY/CxJUoltOgkJCXjllVfwzjvvID4+Hlu3bsXFixcxbtw4fRmtVov69etjyZIl6NChAx577DFMnTrVoJvtbo6OjnB3dzd4mJ2mUEwSBnD0FxHVTFkpYq6hsDu678MeBlJPy7eqOZlW60dF19fNf4Gzv8tdG5OSbRi8t7c3VCpVidae1NTUEq1COnPnzkXXrl3x+usisa5NmzZwcXFB9+7d8f7778PPzw9+fn6wt7eHSlU8cVNISAhSUlJQUFAABwcrGY557vbK7y71ufI7EdVMpc1BtPdj8QBq7+STtsTBGWg/Eti/UAyJb1l70jVkawFycHBAhw4dEBMTY7A9JiYGERERpe6Tm5sL5V1r+egCHV0ud9euXXHu3DlotVp9mbNnz8LPz896gh+guPuLK78TUU3V8RnguT3iMej26vCDPive1vEZeetHpqEbEn9xj2jdqyVk7QKbPHkyli1bhuXLl+P06dOYNGkSEhMT9V1aU6ZMwdNPP60vP2jQIPz000+Ijo7GhQsXsH//frzyyivo3Lkz/P39AQAvvPAC0tPTMWHCBJw9exa//fYb5syZg/Hjx8tyjaXKzwL+5srvRFTDyTUHEVlWnQCg5e3BRn/WniHxsjY9REVFIT09HbNmzUJycjLCwsKwZcsWBAYGAgCSk5MN5gQaNWoUsrKysGjRIrz66quoU6cO7r//fnz44Yf6MgEBAdi2bRsmTZqENm3aoEGDBpgwYQLefPNNi19fmU7/KoaN1m3KkRJERGT9wscBpzcDx9cBvWfUiCHxFeFq8KUw+2rw3w4Dzu8E7nsbuM+KAjMioqqytlXNybQkCfiyG3DtFNDnPaDrK3LXqFQ1Yh4gm5V8UgQ/ANCGK78TUS3B5UtqtzuHxB+qHUPiGQBZ2ol14mv9UMCrsbx1ISIiqiz9kPhE4OxWuWtTbQyALO2f26PemvaRtx5ERETGsHcqXv9t/6di9u8aPNcTAyBLyEoRk4IlbAbSzohtbj5iGycLIyKimqLjaLHW2+U/xRxQNfj+xQloLKG0ycJ+f7v4e04WRkRENUGdACBkIJDws9w1qTYGQJbQ8RmgRX/xffJxYPMrYrIw3bwZTBokIiJrl5UiHk16FwdAv04EGrQHPJsAjToDDTqKhOnKHEvmUYMMgCzBzbfkH1g3WRgREVFNUFpvRtJR8dBxcAPqhwA+oYBPmBjw4xNact6grBRxrBb9GQARERGRFbuzN+NKHLDldSBkMFB4C7h+EbhxCSjIEs9diTPc171BcTDkEybyiGTGAMjS3HxFzg+7vYiIqCYprTej+6vFvRmaQiD9HHDtL/FITQCuJQAZiUDmVfE4Z7j+J5KPl398M2IAZGm6ycKIiIhqE5W96P6qHwK0fqR4e16GWER173zgn22G+2y+Y0ZpCw8IYgBERERExjGmN0PtATS6Fxj8efGweSsYEMQAiIiIiIxTld4MKxsQxIkQiYiIyOYwACIiIiLLsoIBQewCIyIiIsuyggFBbAEiIiIim8MAiIiIiGwOAyAiIiKyOQyAiIiIyOYwACIiIiKbwwCIiIiIbA4DICIiIrI5DICIiIjI5jAAIiIiIpvDAIiIiIhsDpfCKIUkSQCAzMxMmWtCRERElaW7b+vu4+VhAFSKrKwsAEBAQIDMNSEiIiJjZWVlwcPDo9wyCqkyYZKN0Wq1SEpKgpubGxQKhcFzmZmZCAgIwOXLl+Hu7i5TDS3PVq8bsN1r53Xzum2BrV43UDuvXZIkZGVlwd/fH0pl+Vk+bAEqhVKpRMOGDcst4+7uXmteMMaw1esGbPfaed22hddte2rbtVfU8qPDJGgiIiKyOQyAiIiIyOYwADKSo6MjZsyYAUdHR7mrYlG2et2A7V47r5vXbQts9boB2752gEnQREREZIPYAkREREQ2hwEQERER2RwGQERERGRzGAARERGRzWEAZKTFixcjODgYarUaHTp0wN69e+WukkXNnTsXCoUCEydOlLsqZlVUVIRp06YhODgYTk5OaNy4MWbNmgWtVit31Uzujz/+wKBBg+Dv7w+FQoFNmzbpnyssLMSbb76J1q1bw8XFBf7+/nj66aeRlJQkX4VNpLzr1jl9+jQGDx4MDw8PuLm54d5770ViYqLlK2sic+fORadOneDm5ob69etj6NChOHPmjEEZSZIwc+ZM+Pv7w8nJCffddx/++usvmWpsOpW59js9//zzUCgUWLhwoeUqaQaVue7s7Gy89NJLaNiwIZycnBASEoLo6GiZamw5DICMsH79ekycOBFTp07F0aNH0b17d/Tv379GvyEa49ChQ1iyZAnatGkjd1XM7sMPP8SXX36JRYsW4fTp0/joo48wb948fP7553JXzeRycnLQtm1bLFq0qMRzubm5OHLkCKZPn44jR47gp59+wtmzZzF48GAZampa5V03AJw/fx7dunVDy5YtsXv3bhw/fhzTp0+HWq22cE1NZ8+ePRg/fjwOHjyImJgYFBUVITIyEjk5OfoyH330ERYsWIBFixbh0KFD8PX1RZ8+ffRrJNZUlbl2nU2bNuHPP/+Ev7+/DDU1rcpc96RJk7B161asXr0ap0+fxqRJk/Dyyy/j559/lrHmFiBRpXXu3FkaN26cwbaWLVtKb731lkw1spysrCypWbNmUkxMjNSzZ09pwoQJclfJrAYMGCA9++yzBtseeugh6amnnpKpRpYBQNq4cWO5ZeLi4iQA0r///muZSllAadcdFRVV6//eqampEgBpz549kiRJklarlXx9faUPPvhAXyYvL0/y8PCQvvzyS7mqaRZ3X7vOlStXpAYNGkinTp2SAgMDpU8++USeCppJadfdqlUradasWQbl2rdvL02bNs3S1bMotgBVUkFBAeLj4xEZGWmwPTIyErGxsTLVynLGjx+PAQMGoHfv3nJXxSK6deuGHTt24OzZswCA48ePY9++fXjwwQdlrpn8MjIyoFAoUKdOHbmrYjZarRa//fYbmjdvjr59+6J+/foIDw8vtZusJsvIyAAAeHl5AQAuXryIlJQUg/c5R0dH9OzZs9a9z9197YD4u48YMQKvv/46WrVqJVfVzKq06+7WrRt++eUXXL16FZIkYdeuXTh79iz69u0rVzUtgouhVlJaWho0Gg18fHwMtvv4+CAlJUWmWlnGunXrcOTIERw6dEjuqljMm2++iYyMDLRs2RIqlQoajQazZ8/G448/LnfVZJWXl4e33noLTzzxRK1aPPFuqampyM7OxgcffID3338fH374IbZu3YqHHnoIu3btQs+ePeWuYrVJkoTJkyejW7duCAsLAwD9e1lp73P//vuvxetoLqVdOyC6vu3s7PDKK6/IWDvzKeu6P/vsM4wdOxYNGzaEnZ0dlEolli1bhm7duslYW/NjAGQkhUJh8LMkSSW21SaXL1/GhAkTsG3bthqd+2Cs9evXY/Xq1Vi7di1atWqFY8eOYeLEifD398fIkSPlrp4sCgsL8dhjj0Gr1WLx4sVyV8esdMnuQ4YMwaRJkwAA7dq1Q2xsLL788staEQC99NJLOHHiBPbt21fiudr+PlfatcfHx+PTTz/FkSNHatW13qmsv/lnn32GgwcP4pdffkFgYCD++OMPvPjii/Dz86vdrf5y9r/VJPn5+ZJKpZJ++ukng+2vvPKK1KNHD5lqZX4bN26UAEgqlUr/ACApFApJpVJJRUVFclfRLBo2bCgtWrTIYNt7770ntWjRQqYaWQbKyAEqKCiQhg4dKrVp00ZKS0uzfMXM7O7rzs/Pl+zs7KT33nvPoNwbb7whRUREWLh2pvfSSy9JDRs2lC5cuGCw/fz58xIA6ciRIwbbBw8eLD399NOWrKLZlHXtn3zyif597c73OqVSKQUGBspTWRMq67pzc3Mle3t76ddffzXYPnr0aKlv376WrKLFMQeokhwcHNChQwfExMQYbI+JiUFERIRMtTK/Bx54ACdPnsSxY8f0j44dO+LJJ5/EsWPHoFKp5K6iWeTm5kKpNPz3UKlUtXIYfEUKCwsxfPhw/PPPP9i+fTvq1q0rd5XMzsHBAZ06dSoxXPjs2bMIDAyUqVbVJ0kSXnrpJfz000/YuXMngoODDZ4PDg6Gr6+vwftcQUEB9uzZU+Pf5yq69hEjRuDEiRMG73X+/v54/fXX8fvvv8tU6+qr6LoLCwtRWFhok+937AIzwuTJkzFixAh07NgRXbp0wZIlS5CYmIhx48bJXTWzcXNzM+grBgAXFxfUrVu3xPbaZNCgQZg9ezYaNWqEVq1a4ejRo1iwYAGeffZZuatmctnZ2Th37pz+54sXL+LYsWPw8vKCv78/HnnkERw5cgS//vorNBqNPk/Ey8sLDg4OclW72sq77kaNGuH1119HVFQUevTogV69emHr1q3YvHkzdu/eLV+lq2n8+PFYu3Ytfv75Z7i5uen/lh4eHnByctLP8TVnzhw0a9YMzZo1w5w5c+Ds7IwnnnhC5tpXT0XXXrdu3RLBvb29PXx9fdGiRQs5qmwSFV23u7s7evbsiddffx1OTk4IDAzEnj17sGrVKixYsEDm2puZzC1QNc4XX3whBQYGSg4ODlL79u1LDKG0BbYwDD4zM1OaMGGC1KhRI0mtVkuNGzeWpk6dKuXn58tdNZPbtWuXBKDEY+TIkdLFixdLfQ6AtGvXLrmrXi3lXbfO119/LTVt2lRSq9VS27ZtpU2bNslXYRMo62+5YsUKfRmtVivNmDFD8vX1lRwdHaUePXpIJ0+elK/SJlKZa79bbRgGX5nrTk5OlkaNGiX5+/tLarVaatGihTR//nxJq9XKV3ELUEiSJJk7yCIiIiKyJswBIiIiIpvDAIiIiIhsDgMgIiIisjkMgIiIiMjmMAAiIiIim8MAiIiIiGwOAyAiIiKyOQyAiIiIyOYwACIiqgSFQoFNmzbJXQ0iMhEGQERk9UaNGgWFQlHi0a9fP7mrRkQ1FBdDJaIaoV+/flixYoXBNkdHR5lqQ0Q1HVuAiKhGcHR0hK+vr8HD09MTgOieio6ORv/+/eHk5ITg4GD88MMPBvufPHkS999/v37l7+eeew7Z2dkGZZYvX45WrVrB0dERfn5+eOmllwyeT0tLw7Bhw+Ds7IxmzZrhl19+Me9FE5HZMAAiolph+vTpePjhh3H8+HE89dRTePzxx3H69GkAQG5uLvr16wdPT08cOnQIP/zwA7Zv324Q4ERHR2P8+PF47rnncPLkSfzyyy9o2rSpwTneffddDB8+HCdOnMCDDz6IJ598EtevX7fodRKRici9HD0RUUVGjhwpqVQqycXFxeAxa9YsSZIkCYA0btw4g33Cw8OlF154QZIkSVqyZInk6ekpZWdn65//7bffJKVSKaWkpEiSJEn+/v7S1KlTy6wDAGnatGn6n7OzsyWFQiH973//M9l1EpHlMAeIiGqEXr16ITo62mCbl5eX/vsuXboYPNelSxccO3YMAHD69Gm0bdsWLi4u+ue7du0KrVaLM2fOQKFQICkpCQ888EC5dWjTpo3+excXF7i5uSE1NbWql0REMmIAREQ1gouLS4kuqYooFAoAgCRJ+u9LK+Pk5FSp49nb25fYV6vVGlUnIrIOzAEiolrh4MGDJX5u2bIlACA0NBTHjh1DTk6O/vn9+/dDqVSiefPmcHNzQ1BQEHbs2GHROhORfNgCREQ1Qn5+PlJSUgy22dnZwdvbGwDwww8/oGPHjujWrRvWrFmDuLg4fP311wCAJ598EjNmzMDIkSMxc+ZM/Pfff3j55ZcxYsQI+Pj4AABmzpyJcePGoX79+ujfvz+ysrKwf/9+vPzyy5a9UCKyCAZARFQjbN26FX5+fgbbWrRogb///huAGKG1bt06vPjii/D19cWaNWsQGhoKAHB2dsbvv/+OCRMmoFOnTnB2dsbDDz+MBQsW6I81cuRI5OXl4ZNPPsFrr70Gb29vPPLII5a7QCKyKIUkSZLclSAiqg6FQoGNGzdi6NChcleFiGoI5gARERGRzWEARERERDaHOUBEVOOxJ5+IjMUWICIiIrI5DICIiIjI5jAAIiIiIpvDAIiIiIhsDgMgIiIisjkMgIiIiMjmMAAiIiIim8MAiIiIiGzO/wHS5/N3OXmuqgAAAABJRU5ErkJggg==",
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"num_epochs = 30 # during optimization, how many times we look at training data\n",
"batch_size = 32 # during optimization, how many training data to use at each step\n",
"learning_rate = 0.0001 # during optimization, how much we nudge our solution at each step\n",
"\n",
"proper_train_accuracies, val_accuracies = \\\n",
" UDA_pytorch_classifier_fit(word_embedding_lstm_linear_model,\n",
" torch.optim.Adam(word_embedding_lstm_linear_model.parameters(),\n",
" lr=learning_rate),\n",
" nn.CrossEntropyLoss(), # includes softmax\n",
" proper_train_dataset_encoded, val_dataset_encoded,\n",
" num_epochs, batch_size,\n",
" rnn=True,\n",
" save_epoch_checkpoint_prefix='./saved_model_checkpoints/imdb_bert_tiny_lstm_linear_full_finetune')\n",
"\n",
"UDA_plot_train_val_accuracy_vs_epoch(proper_train_accuracies, val_accuracies)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The model at the end of epoch 8 achieved the highest validation accuracy: 0.878200\n"
]
},
{
"data": {
"text/plain": [
"WordEmbeddingLSTMModel(\n",
" (word_embedding_model): BertModel(\n",
" (embeddings): BertEmbeddings(\n",
" (word_embeddings): Embedding(30522, 128, padding_idx=0)\n",
" (position_embeddings): Embedding(512, 128)\n",
" (token_type_embeddings): Embedding(2, 128)\n",
" (LayerNorm): LayerNorm((128,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (encoder): BertEncoder(\n",
" (layer): ModuleList(\n",
" (0-1): 2 x BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSdpaSelfAttention(\n",
" (query): Linear(in_features=128, out_features=128, bias=True)\n",
" (key): Linear(in_features=128, out_features=128, bias=True)\n",
" (value): Linear(in_features=128, out_features=128, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=128, out_features=128, bias=True)\n",
" (LayerNorm): LayerNorm((128,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=128, out_features=512, bias=True)\n",
" (intermediate_act_fn): GELUActivation()\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=512, out_features=128, bias=True)\n",
" (LayerNorm): LayerNorm((128,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" )\n",
" )\n",
" (pooler): BertPooler(\n",
" (dense): Linear(in_features=128, out_features=128, bias=True)\n",
" (activation): Tanh()\n",
" )\n",
" )\n",
" (lstm_layer): LSTM(128, 32)\n",
" (linear_layer): Linear(in_features=32, out_features=2, bias=True)\n",
")"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"best_epoch_idx = np.argmax(val_accuracies)\n",
"print('The model at the end of epoch %d achieved the highest validation accuracy: %f'\n",
" % (best_epoch_idx + 1, val_accuracies[best_epoch_idx]))\n",
"word_embedding_lstm_linear_model.load_state_dict(torch.load('./saved_model_checkpoints/imdb_bert_tiny_lstm_linear_full_finetune_epoch%d.pt' % (best_epoch_idx + 1)))\n",
"word_embedding_lstm_linear_model.eval()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Finally evaluate on test data"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"test_dataset = []\n",
"\n",
"for filename in sorted(glob('./data/aclImdb/test/pos/*.txt')):\n",
" with open(filename, 'r', encoding='utf-8') as f:\n",
" test_dataset.append((f.read(), 1)) # 1 means `positive` sentiment\n",
"\n",
"for filename in sorted(glob('./data/aclImdb/test/neg/*.txt')):\n",
" with open(filename, 'r', encoding='utf-8') as f:\n",
" test_dataset.append((f.read(), 0)) # 0 means `negative` sentiment"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"test_encoded = [tokenizer.encode(text, add_special_tokens=False)[:max_doc_len] for text, label in test_dataset]"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"test_labels = [label for text, label in test_dataset]"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"predicted_test_labels = UDA_pytorch_classifier_predict(word_embedding_lstm_linear_model,\n",
" test_encoded,\n",
" rnn=True)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test accuracy: 0.86036\n"
]
}
],
"source": [
"print('Test accuracy:', UDA_compute_accuracy(predicted_test_labels, test_labels))"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([1], device='cuda:0')"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"UDA_pytorch_classifier_predict(word_embedding_lstm_linear_model,\n",
" [tokenizer.encode('this movie rocks', add_special_tokens=False)[:max_doc_len]],\n",
" rnn=True)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([0], device='cuda:0')"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"UDA_pytorch_classifier_predict(word_embedding_lstm_linear_model,\n",
" [tokenizer.encode('this movie sucks', add_special_tokens=False)[:max_doc_len]],\n",
" rnn=True)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([0], device='cuda:0')"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"UDA_pytorch_classifier_predict(word_embedding_lstm_linear_model,\n",
" [tokenizer.encode('this sucks', add_special_tokens=False)[:max_doc_len]],\n",
" rnn=True)"
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python [conda env:base] *",
"language": "python",
"name": "conda-base-py"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment