Skip to content

Instantly share code, notes, and snippets.

@vhf

vhf/how-to-make-a-racist-ai-without-really-trying.ipynb

Forked from rspeer/how-to-make-a-racist-ai-without-really-trying.ipynb
Created July 27, 2017 08:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save vhf/bac72ae8642b4a7e890207a7734d4315 to your computer and use it in GitHub Desktop.
Save vhf/bac72ae8642b4a7e890207a7734d4315 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
how-to-make-a-racist-ai-without-really-trying.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# How to make a racist AI without really trying\n",
"\n",
"A cautionary tutorial."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*This post is part of the [ConceptNet blog](http://blog.conceptnet.io), written by Rob Speer from [Luminoso](http://luminoso.com).*\n",
"\n",
"## Let's make a sentiment classifier!\n",
"\n",
"Sentiment analysis is a very frequently-implemented task in NLP, and it's no surprise. Recognizing whether people are expressing positive or negative opinions about things has obvious business applications. It's used in social media monitoring, customer feedback, and even automatic stock trading (leading to bots that [buy Berkshire Hathaway when Anne Hathaway gets a good movie review](https://www.theatlantic.com/technology/archive/2011/03/does-anne-hathaway-news-drive-berkshire-hathaways-stock/72661/)).\n",
"\n",
"It's simplistic, sometimes too simplistic, but it's one of the easiest ways to get measurable results from NLP. In a few steps, you can put text in one end and get positive and negative scores out the other, and you never have to figure out what you should do with a parse tree or a graph of entities or any difficult representation like that.\n",
"\n",
"So that's what we're going to do here, following the path of least resistance at every step, obtaining a classifier that should look very familiar to anyone involved in current NLP. For example, you can find this model described in the [Deep Averaging Networks](http://cs.umd.edu/~miyyer/pubs/2015_acl_dan.pdf) paper (Iyyer et al., 2015). This model is not the point of that paper, so don't take this as an attack on their results; it was there as an example of a well-known way to use word vectors.\n",
"\n",
"Here's the outline of what we're going to do:\n",
"\n",
"* Acquire some typical **word embeddings** to represent the meanings of words\n",
"* Acquire **training and test data**, with gold-standard examples of positive and negative words\n",
"* **Train a classifier**, using gradient descent, to recognize other positive and negative words based on their word embeddings\n",
"* Compute **sentiment scores** for sentences of text using this classifier\n",
"* **Behold the monstrosity** that we have created\n",
"\n",
"And at that point we will have shown \"how to make a racist AI without really trying\". Of course that would be a terrible place to leave it, so afterward, we're going to:\n",
"\n",
"* **Measure the problem** statistically, so we can recognize if we're solving it\n",
"* **Improve the data** to obtain a semantic model that's more accurate _and_ less racist"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Software dependencies\n",
"\n",
"This tutorial is written in Python, and relies on a typical Python machine-learning stack: `numpy` and `scipy` for numerical computing, `pandas` for managing our data, and `scikit-learn` for machine learning. Later on we'll graph some things with `matplotlib` and `seaborn`.\n",
"\n",
"You could also replace `scikit-learn` with TensorFlow or Keras or something like that, as they can also train classifiers using gradient descent. But there's no need for the deep-learning abstractions they provide, as it only takes a single layer of machine learning to solve this problem."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib\n",
"import seaborn\n",
"import re\n",
"import statsmodels.formula.api\n",
"\n",
"from sklearn.linear_model import SGDClassifier\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.metrics import accuracy_score"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Configure how graphs will show up in this notebook\n",
"%matplotlib inline\n",
"seaborn.set_context('notebook', rc={'figure.figsize': (10, 6)}, font_scale=1.5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1: Word embeddings\n",
"\n",
"Word embeddings are frequently used to represent words as inputs to machine learning. The words become vectors in a multi-dimensional space, where nearby vectors represent similar meanings. With word embeddings, you can compare words by (roughly) what they mean, not just exact string matches.\n",
"\n",
"Successfully training word vectors requires starting from hundreds of gigabytes of input text. Fortunately, various machine-learning groups have already done this and provided pre-trained word embeddings that we can download.\n",
"\n",
"Two very well-known datasets of pre-trained English word embeddings are **word2vec**, pretrained on Google News data, and **GloVe**, pretrained on the Common Crawl of web pages. We would get similar results for either one, but here we'll use GloVe because its source of data is more transparent.\n",
"\n",
"GloVe comes in three sizes: 6B, 42B, and 840B. The 840B size is powerful, but requires significant post-processing to use it in a way that's an improvement over 42B. The 42B version is pretty good and is also neatly trimmed to a vocabulary of 1 million words. Because we're following the path of least resistance, we'll just use the 42B version.\n",
"\n",
"> **Why does it matter that the word embeddings are \"well-known\"?**\n",
">\n",
"> I'm glad you asked, hypothetical questioner! We're trying to do something extremely typical at each step, and for some reason, comparison-shopping for better word embeddings isn't typical yet. Read on, and I hope you'll come out of this tutorial with the desire to use [modern, high-quality word embeddings](https://github.com/commonsense/conceptnet-numberbatch), especially those that are aware of algorithmic bias and try to mitigate it. But that's getting ahead of things.\n",
"\n",
"We download glove.42B.300d.zip from [the GloVe web page](https://nlp.stanford.edu/projects/glove/), and extract it into `data/glove.42B.300d.txt`. Next we define a function to read the simple format of its word vectors.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(1917494, 300)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def load_embeddings(filename):\n",
" \"\"\"\n",
" Load a DataFrame from the generalized text format used by word2vec, GloVe,\n",
" fastText, and ConceptNet Numberbatch. The main point where they differ is\n",
" whether there is an initial line with the dimensions of the matrix.\n",
" \"\"\"\n",
" labels = []\n",
" rows = []\n",
" with open(filename, encoding='utf-8') as infile:\n",
" for i, line in enumerate(infile):\n",
" items = line.rstrip().split(' ')\n",
" if len(items) == 2:\n",
" # This is a header row giving the shape of the matrix\n",
" continue\n",
" labels.append(items[0])\n",
" values = np.array([float(x) for x in items[1:]], 'f')\n",
" rows.append(values)\n",
" \n",
" arr = np.vstack(rows)\n",
" return pd.DataFrame(arr, index=labels, dtype='f')\n",
"\n",
"embeddings = load_embeddings('data/glove.42B.300d.txt')\n",
"embeddings.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: A gold-standard sentiment lexicon\n",
"\n",
"We need some input about which words are positive and which words are negative. There are many sentiment lexicons you could use, but we're going to go with a very straightforward lexicon (Hu and Liu, 2004), the same one used by the Deep Averaging Networks paper.\n",
"\n",
"We download the lexicon from Bing Liu's web site (https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html#lexicon) and extract it into `data/positive-words.txt` and `data/negative-words.txt`.\n",
"\n",
"Next we define how to read these files, and read them in as the `pos_words` and `neg_words` variables:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def load_lexicon(filename):\n",
" \"\"\"\n",
" Load a file from Bing Liu's sentiment lexicon\n",
" (https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html), containing\n",
" English words in Latin-1 encoding.\n",
" \n",
" One file contains a list of positive words, and the other contains\n",
" a list of negative words. The files contain comment lines starting\n",
" with ';' and blank lines, which should be skipped.\n",
" \"\"\"\n",
" lexicon = []\n",
" with open(filename, encoding='latin-1') as infile:\n",
" for line in infile:\n",
" line = line.rstrip()\n",
" if line and not line.startswith(';'):\n",
" lexicon.append(line)\n",
" return lexicon\n",
"\n",
"pos_words = load_lexicon('data/positive-words.txt')\n",
"neg_words = load_lexicon('data/negative-words.txt')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Train a model to predict word sentiments"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our data points here are the embeddings of these positive and negative words. We use the Pandas `.loc[]` operation to look up the embeddings of all the words.\n",
"\n",
"Some of these words are not in the GloVe vocabulary, particularly the misspellings such as \"fancinating\". Those words end up with rows full of `NaN` to indicate their missing embeddings, so we use `.dropna()` to remove them."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"pos_vectors = embeddings.loc[pos_words].dropna()\n",
"neg_vectors = embeddings.loc[neg_words].dropna()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we make arrays of the desired inputs and outputs. The inputs are the embeddings, and the outputs are 1 for positive words and -1 for negative words. We also make sure to keep track of the words they're labeled with, so we can interpret the results."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"vectors = pd.concat([pos_vectors, neg_vectors])\n",
"targets = np.array([1 for entry in pos_vectors.index] + [-1 for entry in neg_vectors.index])\n",
"labels = list(pos_vectors.index) + list(neg_vectors.index)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> **Hold on. Some words are neither positive nor negative, they're neutral. Shouldn't there be a third class for neutral words?**\n",
">\n",
"> I think that having examples of neutral words would be quite beneficial, especially because the problems we're going to see come from assigning sentiment to words that shouldn't have sentiment. If we could reliably identify when words should be neutral, it would be worth the slight extra complexity of a 3-class classifier. It requires finding a source of examples of neutral words, because Liu's data only lists positive and negative words.\n",
">\n",
"> So I tried a version of this notebook where I put in 800 examples of neutral words, and put a strong weight on predicting words to be neutral. But the end results were not much different from what you're about to see.\n",
">\n",
"> **How is this list drawing the line between positive and negative anyway? Doesn't that depend on context?**\n",
">\n",
"> Good question. Domain-general sentiment analysis isn't as straightforward as it sounds. The decision boundary we're trying to find is fairly arbitrary in places. In this list, \"audacious\" is marked as \"bad\" while \"ambitious\" is \"good\". \"Comical\" is bad, \"humorous\" is good. \"Refund\" is good, even though it's typically in bad situations that you have to request one or pay one.\n",
"> \n",
"> I think everyone knows that sentiment requires context, but when implementing an easy approach to sentiment analysis, you just have to kind of hope that you can ignore context and the sentiments will average out to the right trend."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the scikit-learn `train_test_split` function, we simultaneously separate the input vectors, output values, and labels into training and test data, with 10% of the data used for testing."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"train_vectors, test_vectors, train_targets, test_targets, train_labels, test_labels = \\\n",
" train_test_split(vectors, targets, labels, test_size=0.1, random_state=0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we make our classifier, and train it by running the training vectors through it for 100 iterations. We use a logistic function as the loss, so that the resulting classifier can output the probability that a word is positive or negative."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"SGDClassifier(alpha=0.0001, average=False, class_weight=None, epsilon=0.1,\n",
" eta0=0.0, fit_intercept=True, l1_ratio=0.15,\n",
" learning_rate='optimal', loss='log', n_iter=100, n_jobs=1,\n",
" penalty='l2', power_t=0.5, random_state=0, shuffle=True, verbose=0,\n",
" warm_start=False)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = SGDClassifier(loss='log', random_state=0, n_iter=100)\n",
"model.fit(train_vectors, train_targets)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We evaluate the classifier on the test vectors. It predicts the correct sentiment for sentiment words outside of its training data 95% of the time. Not bad."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0.95022624434389136"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"accuracy_score(model.predict(test_vectors), test_targets)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's define a function that we can use to see the sentiment that this classifier predicts for particular words, then use it to see some examples of its predictions on the test data."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sentiment</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>fidget</th>\n",
" <td>-9.931679</td>\n",
" </tr>\n",
" <tr>\n",
" <th>interrupt</th>\n",
" <td>-9.634706</td>\n",
" </tr>\n",
" <tr>\n",
" <th>staunchly</th>\n",
" <td>1.466919</td>\n",
" </tr>\n",
" <tr>\n",
" <th>imaginary</th>\n",
" <td>-2.989215</td>\n",
" </tr>\n",
" <tr>\n",
" <th>taxing</th>\n",
" <td>0.468522</td>\n",
" </tr>\n",
" <tr>\n",
" <th>world-famous</th>\n",
" <td>6.908561</td>\n",
" </tr>\n",
" <tr>\n",
" <th>low-cost</th>\n",
" <td>9.237223</td>\n",
" </tr>\n",
" <tr>\n",
" <th>disapointment</th>\n",
" <td>-8.737182</td>\n",
" </tr>\n",
" <tr>\n",
" <th>totalitarian</th>\n",
" <td>-10.851580</td>\n",
" </tr>\n",
" <tr>\n",
" <th>bellicose</th>\n",
" <td>-8.328674</td>\n",
" </tr>\n",
" <tr>\n",
" <th>freezes</th>\n",
" <td>-8.456981</td>\n",
" </tr>\n",
" <tr>\n",
" <th>sin</th>\n",
" <td>-7.839670</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fragile</th>\n",
" <td>-4.018289</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fooled</th>\n",
" <td>-4.309344</td>\n",
" </tr>\n",
" <tr>\n",
" <th>undecided</th>\n",
" <td>-2.816172</td>\n",
" </tr>\n",
" <tr>\n",
" <th>handily</th>\n",
" <td>2.339609</td>\n",
" </tr>\n",
" <tr>\n",
" <th>demonizes</th>\n",
" <td>-2.102152</td>\n",
" </tr>\n",
" <tr>\n",
" <th>easygoing</th>\n",
" <td>8.747150</td>\n",
" </tr>\n",
" <tr>\n",
" <th>unpopular</th>\n",
" <td>-7.887475</td>\n",
" </tr>\n",
" <tr>\n",
" <th>commiserate</th>\n",
" <td>1.790899</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sentiment\n",
"fidget -9.931679\n",
"interrupt -9.634706\n",
"staunchly 1.466919\n",
"imaginary -2.989215\n",
"taxing 0.468522\n",
"world-famous 6.908561\n",
"low-cost 9.237223\n",
"disapointment -8.737182\n",
"totalitarian -10.851580\n",
"bellicose -8.328674\n",
"freezes -8.456981\n",
"sin -7.839670\n",
"fragile -4.018289\n",
"fooled -4.309344\n",
"undecided -2.816172\n",
"handily 2.339609\n",
"demonizes -2.102152\n",
"easygoing 8.747150\n",
"unpopular -7.887475\n",
"commiserate 1.790899"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def vecs_to_sentiment(vecs):\n",
" # predict_log_proba gives the log probability for each class\n",
" predictions = model.predict_log_proba(vecs)\n",
"\n",
" # To see an overall positive vs. negative classification in one number,\n",
" # we take the log probability of positive sentiment minus the log\n",
" # probability of negative sentiment.\n",
" return predictions[:, 1] - predictions[:, 0]\n",
"\n",
"\n",
"def words_to_sentiment(words):\n",
" vecs = embeddings.loc[words].dropna()\n",
" log_odds = vecs_to_sentiment(vecs)\n",
" return pd.DataFrame({'sentiment': log_odds}, index=vecs.index)\n",
"\n",
"\n",
"# Show 20 examples from the test set\n",
"words_to_sentiment(test_labels).ix[:20]"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"More than the accuracy number, this convinces us that the classifier is working. We can see that the classifier has learned to generalize sentiment to words outside of its training data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Get a sentiment score for text\n",
"\n",
"There are many ways to combine sentiments for word vectors into an overall sentiment score. Again, because we're following the path of least resistance, we're just going to average them."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import re\n",
"TOKEN_RE = re.compile(r\"\\w.*?\\b\")\n",
"# The regex above finds tokens that start with a word-like character (\\w), and continues\n",
"# matching characters (.+?) until the next word break (\\b). It's a relatively simple\n",
"# expression that manages to extract something very much like words from text.\n",
"\n",
"\n",
"def text_to_sentiment(text):\n",
" tokens = [token.casefold() for token in TOKEN_RE.findall(text)]\n",
" sentiments = words_to_sentiment(tokens)\n",
" return sentiments['sentiment'].mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are many things we could have done better:\n",
"\n",
"- Weight words by their inverse frequency, so that words like \"the\" and \"I\" don't cause big changes in sentiment\n",
"- Adjust the averaging so that short sentences don't end up with the most extreme sentiment values\n",
"- Take phrases into account\n",
"- Use a more robust word-segmentation algorithm that isn't confused by apostrophes\n",
"- Account for negations such as \"not happy\"\n",
"\n",
"But all of those would require extra code and wouldn't fundamentally change the results we're about to see. At least now we can roughly compare the relative positivity of different sentences:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"3.889968926086298"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text_to_sentiment(\"this example is pretty cool\")"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"2.7997773492425186"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text_to_sentiment(\"this example is okay\")"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"-1.1774475917460698"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text_to_sentiment(\"meh, this example sucks\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Behold the monstrosity that we have created"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Not every sentence is going to contain obvious sentiment words. Let's see what it does with a few variations on a neutral sentence:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"2.0429166109408983"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text_to_sentiment(\"Let's go get Italian food\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1.4094033658140972"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text_to_sentiment(\"Let's go get Chinese food\")"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0.38801985560121732"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text_to_sentiment(\"Let's go get Mexican food\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is analogous to what I saw when I experimented with analyzing restaurant reviews using word embeddings, and found out that [all the Mexican restaurants were ending up with lower sentiment](https://blog.conceptnet.io/2017/04/24/conceptnet-numberbatch-17-04-better-less-stereotyped-word-vectors/) for no good reason.\n",
"\n",
"Word vectors are capable of representing subtle distinctions of meaning just by reading words in context. So they're also capable of representing less-subtle things like the biases of our society.\n",
"\n",
"Here are some other neutral statements:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"2.2286179364745311"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text_to_sentiment(\"My name is Emily\")"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1.3976291151079159"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text_to_sentiment(\"My name is Heather\")"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0.98463802132985556"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text_to_sentiment(\"My name is Yvette\")"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"-0.47048131775890656"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text_to_sentiment(\"My name is Shaniqua\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Well, dang.\n",
"\n",
"The system has associated wildly different sentiments with people's names. You can look at these examples and many others and see that the sentiment is generally more positive for stereotypically-white names, and more negative for stereotypically-black names.\n",
"\n",
"This is the test that Caliskan, Bryson, and Narayanan used to conclude that [semantics derived automatically from language corpora contain human-like biases](http://opus.bath.ac.uk/55288/), a paper published in *Science* in April 2017, and we'll be using more of it shortly."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 6: Measure the problem\n",
"\n",
"We want to learn how to not make something like this again. So let's put more data through it, and statistically measure how bad its bias is.\n",
"\n",
"Here we have four lists of names that tend to reflect different ethnic backgrounds, mostly from a United States perspective. The first two are lists of predominantly \"white\" and \"black\" names adapted from Caliskan et al.'s article. I also added typically Hispanic names, as well as Muslim names that come from Arabic or Urdu; these are two more distinct groupings of given names that tend to represent your background.\n",
"\n",
"This data is currently used as a bias-check in the ConceptNet build process, and can be found in the `conceptnet5.vectors.evaluation.bias` module. I'm interested in expanding this to more ethnic backgrounds, which may require looking at surnames and not just given names.\n",
"\n",
"Here are the lists:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"NAMES_BY_ETHNICITY = {\n",
" # The first two lists are from the Caliskan et al. appendix describing the\n",
" # Word Embedding Association Test.\n",
" 'White': [\n",
" 'Adam', 'Chip', 'Harry', 'Josh', 'Roger', 'Alan', 'Frank', 'Ian', 'Justin',\n",
" 'Ryan', 'Andrew', 'Fred', 'Jack', 'Matthew', 'Stephen', 'Brad', 'Greg', 'Jed',\n",
" 'Paul', 'Todd', 'Brandon', 'Hank', 'Jonathan', 'Peter', 'Wilbur', 'Amanda',\n",
" 'Courtney', 'Heather', 'Melanie', 'Sara', 'Amber', 'Crystal', 'Katie',\n",
" 'Meredith', 'Shannon', 'Betsy', 'Donna', 'Kristin', 'Nancy', 'Stephanie',\n",
" 'Bobbie-Sue', 'Ellen', 'Lauren', 'Peggy', 'Sue-Ellen', 'Colleen', 'Emily',\n",
" 'Megan', 'Rachel', 'Wendy'\n",
" ],\n",
"\n",
" 'Black': [\n",
" 'Alonzo', 'Jamel', 'Lerone', 'Percell', 'Theo', 'Alphonse', 'Jerome',\n",
" 'Leroy', 'Rasaan', 'Torrance', 'Darnell', 'Lamar', 'Lionel', 'Rashaun',\n",
" 'Tyree', 'Deion', 'Lamont', 'Malik', 'Terrence', 'Tyrone', 'Everol',\n",
" 'Lavon', 'Marcellus', 'Terryl', 'Wardell', 'Aiesha', 'Lashelle', 'Nichelle',\n",
" 'Shereen', 'Temeka', 'Ebony', 'Latisha', 'Shaniqua', 'Tameisha', 'Teretha',\n",
" 'Jasmine', 'Latonya', 'Shanise', 'Tanisha', 'Tia', 'Lakisha', 'Latoya',\n",
" 'Sharise', 'Tashika', 'Yolanda', 'Lashandra', 'Malika', 'Shavonn',\n",
" 'Tawanda', 'Yvette'\n",
" ],\n",
" \n",
" # This list comes from statistics about common Hispanic-origin names in the US.\n",
" 'Hispanic': [\n",
" 'Juan', 'José', 'Miguel', 'Luís', 'Jorge', 'Santiago', 'Matías', 'Sebastián',\n",
" 'Mateo', 'Nicolás', 'Alejandro', 'Samuel', 'Diego', 'Daniel', 'Tomás',\n",
" 'Juana', 'Ana', 'Luisa', 'María', 'Elena', 'Sofía', 'Isabella', 'Valentina',\n",
" 'Camila', 'Valeria', 'Ximena', 'Luciana', 'Mariana', 'Victoria', 'Martina'\n",
" ],\n",
" \n",
" # The following list conflates religion and ethnicity, I'm aware. So do given names.\n",
" #\n",
" # This list was cobbled together from searching baby-name sites for common Muslim names,\n",
" # as spelled in English. I did not ultimately distinguish whether the origin of the name\n",
" # is Arabic or Urdu or another language.\n",
" #\n",
" # I'd be happy to replace it with something more authoritative, given a source.\n",
" 'Arab/Muslim': [\n",
" 'Mohammed', 'Omar', 'Ahmed', 'Ali', 'Youssef', 'Abdullah', 'Yasin', 'Hamza',\n",
" 'Ayaan', 'Syed', 'Rishaan', 'Samar', 'Ahmad', 'Zikri', 'Rayyan', 'Mariam',\n",
" 'Jana', 'Malak', 'Salma', 'Nour', 'Lian', 'Fatima', 'Ayesha', 'Zahra', 'Sana',\n",
" 'Zara', 'Alya', 'Shaista', 'Zoya', 'Yasmin'\n",
" ]\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we'll use Pandas to make a table of these names, their predominant ethnic background, and the sentiment score we get for them:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def name_sentiment_table():\n",
" frames = []\n",
" for group, name_list in sorted(NAMES_BY_ETHNICITY.items()):\n",
" lower_names = [name.lower() for name in name_list]\n",
" sentiments = words_to_sentiment(lower_names)\n",
" sentiments['group'] = group\n",
" frames.append(sentiments)\n",
"\n",
" # Put together the data we got from each ethnic group into one big table\n",
" return pd.concat(frames)\n",
"\n",
"name_sentiments = name_sentiment_table()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A sample of the data:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sentiment</th>\n",
" <th>group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>mohammed</th>\n",
" <td>0.834974</td>\n",
" <td>Arab/Muslim</td>\n",
" </tr>\n",
" <tr>\n",
" <th>alya</th>\n",
" <td>3.916803</td>\n",
" <td>Arab/Muslim</td>\n",
" </tr>\n",
" <tr>\n",
" <th>terryl</th>\n",
" <td>-2.858010</td>\n",
" <td>Black</td>\n",
" </tr>\n",
" <tr>\n",
" <th>josé</th>\n",
" <td>0.432956</td>\n",
" <td>Hispanic</td>\n",
" </tr>\n",
" <tr>\n",
" <th>luciana</th>\n",
" <td>1.086073</td>\n",
" <td>Hispanic</td>\n",
" </tr>\n",
" <tr>\n",
" <th>hank</th>\n",
" <td>0.391858</td>\n",
" <td>White</td>\n",
" </tr>\n",
" <tr>\n",
" <th>megan</th>\n",
" <td>2.158679</td>\n",
" <td>White</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sentiment group\n",
"mohammed 0.834974 Arab/Muslim\n",
"alya 3.916803 Arab/Muslim\n",
"terryl -2.858010 Black\n",
"josé 0.432956 Hispanic\n",
"luciana 1.086073 Hispanic\n",
"hank 0.391858 White\n",
"megan 2.158679 White"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"name_sentiments.ix[::25]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can visualize the distribution of sentiment we get for each kind of name:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(-10, 10)"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAnwAAAGCCAYAAAB6nR6iAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8XXd9//HXvVd77+Ehb389E++ZPZ2EDEYg7NHSMkpb\nRigtlBHaMn60pbS00FIIUCCQkBDItjOcxE7ixHt+PWVJ1t5b9+re+/vjypKvlqVg6dx79X4+Hn7I\n53vOld9WYulzv9MVDAYRERERkdjldjqAiIiIiEwsFXwiIiIiMU4Fn4iIiEiMU8EnIiIiEuNU8ImI\niIjEOBV8IiIiIjFOBZ+IiIhIjItzOsBwjDFzgJ8AVwNzrLWlF9x7N3AvsACoBn4DfNla6x/hc6UA\n3wFuAXKAI33Pb53Iv4OIiIhIpIi4Hj5jzFuBV4Gzw9y7Gvgp8A0gD3gb8D7gS6N8yv8ANgE3A4XA\n/cAfjDHmkgYXERERiVARV/AR6oW7Evj5MPc+BTxhrX3QWttjrT0I/AvwKWPMkL+LMSabUEH4VWvt\ncWttt7X2h8BR4GMT91cQERERiRwRV/BZa//XWnt8hNsbgF2D2nYBuYSGeAdbDcSP8JoNf0xOERER\nkWgRcQXfReQDjYPa6vs+FozwPCO8ZrjnRURERGJORC7aeJOCl/r53l5/MC7O8ybjiIiIiEwq10g3\noq3gqyE0fHuhvL6P1SM8T99rzg16zXDPh2lq6hxvPhERERFH5Oenj3gv2oZ0dzJ07t0VQBVwapjn\ndwM9w7xmM/DSJU8nIiIiEoGirYfvu8CLxph3Ab8DlgOfBb5jrQ0CGGN+BlRbaz9vrW0xxvwY+Jox\n5iBQAXwCmA38wIm/gIiIiMhki7iCzxhjgVkM9D5aY0wQ+Lm19qPGmHuA+4CfERqy/R7wzxd8ihLC\ney4/DXwbeBlIB/YBN1trh+zzJyIiIhKLXMHgeNc6TB11dW364oiIiEhUyM9PH3HRRrTN4RMRERGR\ncVLBJyIiIhLjVPCJiIiIxDgVfCIiIiIxTgWfiIiISIxTwSciIiIS41TwiYiIiMQ4FXwiIiIiMU4F\nn4iIiEiMU8EnIiIiEuNU8ImIiIjEOBV8IiIiIjFOBZ+IiIhIjFPBJyIiIlHD5/VTdrqRlqZOp6NE\nlTinA4iIiIiMRUNdO7//1X66O30ArL1iNmuumO1sqCihHj4RERGJCrt3nO0v9gB27zxLZ4fXwUTR\nQwWfiIiIRIXO9vDiLhAI0tPtG+FpuZAKPhEREYkKC5cXhl0XFKeTnZvqUJroojl8IjIhmntaONJg\nyUvOZWH2PKfjiEgMWHL5NOLjPZy29WRmJ7Ni/UynI0UNVzAYdDpDxKqra9MXR+RNON1Syvf2/g++\nQGio5YrpG3i3eZvDqUREYlt+frprpHsa0hWRS+7p0uf7iz2AHedeo7mnxcFEIiJTmwo+EbnkvIHw\nSdRBgnj9mlgtIuIUFXwicsldNX0jLgZGFhbnLKQgJc/BRCIiU5vm8I1Cc/hE3rwTTac5UH+Y/ORc\nNhSvJcET73QkEZGYNtocPhV8o1DBJyIiItFCizZEREREpjAVfCIiIiIxTgWfiIiISIxTwSciIiIS\n41TwiYiIiMS4qDtL1xhzFfDMMLfigZ9Zaz886PkPAT8BegY9/xtr7QcmJKSIiIhIBIm6gs9a+yKQ\ndGGbMaYIOATcP8LLzlprZ09sMhEREZHIFCtDuj8g1GO33ekgIiIiIpEm6nr4BjPG3A5sBuaN8li6\nMeaRvud8wFPAvdbaxkmIKCIiIuKoqC74jDFu4BvAN621rSM8Vg8cAf4duBtYBvwK+D/g1tE+f3Z2\nCnFxnksXWGSK6fJ1k+hJwO2OlcEEEZHoFNVHqxlj3gH8D1Bsre0ex+vuAB4FSqy15SM9p6PVRN6c\ndl8HPzn0S441nSAzIZ17zNu4LH+p07FERGJaLB+t9j7g9+Mp9vqc7Ps4/RLnERHgsdPPcKzpBAAt\n3jZ+euTX9Pi9DqcSEZm6orbgM8akA7cAv7vIcx8zxgzefmVx38dTE5FNZKorbzsXdt3t76a+q8Gh\nNCIiEs1z+FYCCcDeCxuNMeuAnwE3WWvL+p75tjGmEngBWAr8E6E9++omNbHIFLEoez6lrWX915kJ\nGRSlFDiYSERkaovmgm9a38faQe0pgCFU6GGt/Z4xJh74PlACNAE/Be6bpJwiU86WOTfQ5e9hX+1B\n8lNyefuC2/G4tQBKRMQpUb1oY6Jp0YaIiIhEi1hetCEiIiIiF6GCT0RERCTGqeATERERiXEq+ERE\nRERinAo+ERERkRingk9EREQkxqngExEREYlxKvhERMRxQb+fQE+P0zFEYlY0n7QhIiIxoOXll6h7\n8AECnZ2krVpN0Uc+ijsx0elYIjFFBZ+IiDimt7mJmp/fD34/AO2736Bpxkxyb7/T2WASkYLBIHtf\nLePM8XoyspJZf/UcMrKSnY4VFVTwiYiIY3oqyvuLvf62sjKH0kik2/daOa9tPwNAbVUbddVtvPvP\n1uFyjXiimPTRHD4REXFM0tx5uBKTwtpSlixxKI1EujMn6sOuW5q6aKzvcChNdFHBJyIijvGkpDL9\nU39F0py5xOXkkHPb7WRefa3TsSRCZWWHD9/GxblJS9d8z7FwBYNBpzNErLq6Nn1xREREIkRbSzeP\nP3iApvpO4uLdbL5hPksun+Z0rIiRn58+4ti2Cr5RqOALV1bTxqMvn6Glw8vGpUVcv3qG05FERGSK\nCQaDNDd2kZqWQEKiliJcaLSCT18pGZMer5/vPLCP9i4fAKcrW0lO9LBpWbHDyUREZCpxuVxk56Y4\nHSPqaA6fjMmJc839xd55ewdNnhUREZHIpIJPxqQwO4XB/cRFOXqHJSIiEg1U8MmY5Gcl87ar5xLn\nCZV986dncvO6EodTiYjIVNfV6aWnu9fpGBFPizZGoUUbQ3V0++jo8lGQrd49ERGZHNXnWgj4gxTP\nzOzfZNnvD/DsH45y6lgdbo+LFetmsv7quQ4ndZYWbcglk5oUT2pSvNMxRERkCvD7Azzx4EEqSpsA\nKChOZ+O1cyk/00R7aw+njtUBEPAH2fNKGbMX5FE4LcPJyBFLBZ+IiIhEpNITDf3FHoSOU3v0l/tH\nfL6xrkMF3wg0h09EREQiUlend8zPuj0uZszOnsA00U09fCIi4ihvbS2NT/wBf0sLGRs3k75uvdOR\nJELMWZDHa9tP4+3xhxpcwKDZ9Tl5KSQkxbNm8yzSM5OGfA4JUcEnIiKOCfh8VPy/b9DbFBq26zh4\nADwe0levcTiZRILU9ETuet9KDu0+h98fpKA4nZ3PnsTvD1V9ZnkR1922yOGU0UEFn4iIOKbrxPH+\nYu+8ttdfU8En/XLz07h6i+m/Lpmbw9mTDWRkJ1MyN8fBZNFFBZ+MS21TJ60dPuZOy8DtHnH1t4jI\nmMTnDP2BHZ+T60ASiRYZWcksX6Oz3MdLBZ+M2QPPnmDr6+UECZ2yce+7V5Kdnuh0LBGJYglFxWTf\nfAtNzzwFwSAJxdPIvnmL07FEYo42Xh6FNl4eUNXQwRf/57WwtpvWzuSe6xc4lEhEYomvvo7e1jaS\nZs/G5dYGEjLg6P4qDu0+hyfezepNs+jq9HHmeB2ZWcms2FBCSmqC0xEjhjZelj9aY1vPmNpERN6M\n+Lx84vPynY4hEaaitJEXnrT9108+dJAL+6nOlTVz94c133MsorLgM8aUAtMB/6Bbl1lrjw/z/I3A\n14AlQAvwJPAZa23nxCaNHQtnZJGTkUhj60CRt2FJoYOJREQk1pWdagy7HjwoWV/TTlN9B9l5qZOY\nKjpFZcHX56PW2vsv9pAxZgHwB+Be4H+BQuAh4PvAhycyYCyJj3PzN+9ZxROvnqWl3cumZUWsWqh3\n4zK6+q5G0hPSSPRoyEVExu9ihZzb7SIpRcd9jkU0F3xj9efAMWvtv/ddnzHG3Ac8ZIy511pb72C2\nqJKflcwHt2i/I7m4pu5mfnjgfsrbK0nyJPLOhXexvni107FEJMosXFbIubImTh6pxeV2YZYVcfZU\nA53tXlwuWHvlbJJT9IZyLKK54HunMebzhIZ2TwL3WWsfHea5DcCuQW27CP3dVwNPT2hKkSnosTPP\nUN5eCUC3v4cH7MNclr+E5Lhkh5OJSDTxeNzccPsSrrhhAW63i4TEOHp6erEHqykozqBous7NHato\nLfgOECryPgj0AJ8CHjHGbLLWvjro2XygcVDb+V69gtH+kOzsFOLiPJcgrsjU0rg//J+cN+CDZB/5\nWaP+kxMRGVV9bTsP/fgNWlu6cbldXH/rYjZdO8/pWFEhKgs+a+0dg5r+0RjzduCjwOCCbzSjbrvS\n1KQ1HSJvhslcyPGG0/3XeUk5JHrTqKtrczCViES7rb8/QmtLNwDBQJDnnjzKjLlZGtbtk5+fPuK9\nqCz4RnCS0PDuYDXA4G3b8/o+Vk9oIpEp6qZZ1xAI+tlXd4j85DzumLcFt0t7q8mAnvJy2vftIT4v\nn/S163DFxdKPI5ko7W3dYdcBf5CuTp8KvjGIun9hxpg5wOeAL1prmy+4tRjYPsxLdgK3DWq7gtBQ\n8OsTElJkinO73Nw650ZunXOj01EkAnUcPsS5f/sXCAQAaNv9OjlbbqW3pYXUpctwJyU5nFAi1ZwF\neVRXtPZf5xakkqMtWcYk6go+Qj12dwLZxphPAd3AZ4GFwDuMMeuAnwE3WWvLgB8AnzLGfLrv9yWE\n9uT7kbW2xYm/gIjIVNa09Zn+Yg+gY99eOvbtBcCTkcHML3yJhALN95RwwWCQU8fqwtrWbJ7tTJgo\nFHVjLH2bJd8IpAHHgDrgJuBqa60FUgADJPQ9XwrcAtwDNBHqBXwK+MxkZxcREXC5Rzz9CX9rK01b\nn5rENBItaipbqa0Knwd86litQ2miTzT28GGtPQoMXrhx/t4LgGtQ24vA+olPFtvau3xs33eOlnYv\n65cWMm9aptORRCQKZd+0hc4jhwn29g57P9ChBXMy1HC7Zujc5bGLyoJPJl8gEOTbv9xDRV0HAM/t\nOce9716BKcl2OJlEMn/Aj8etrY0kXMqixcz62j/QsX8fcbm51D/8W3w1fWvoXC4yr7zK2YASkfIK\n0ygoTg/r5Wtt6nIwUXRRwSdjcqKiub/YAwgEg2zfV6mCT4bV6eviZ0d/zaH6o+QkZXOPeStLco3T\nsSSCJBQWkXDTFgCSFxqan3sWf0sL6Rs2krJQ/6/I8Dxx4T16NZWttDR1kpmd4lCi6KG+UBmTpISh\n7w2SEvV+QYb32JlnOFh/hCBBGrob+cnhX+L1e52OJREqLj2DvDvfSuEHPqRiT0aVMOjnjss1/FCv\nDKWCT8ZkVlE6q01+/3Vacjw3rZ3pYCKJZGdby8OuO3u7qOtqcCiNiMSKVRtLiIsfKF2WrpxGanqi\ng4mih7poZMw+cdcyjpU109Lew2XzcklJinc6kkSoBVlzKW0t679OT0ijMCV/lFeIiFxc0fRM3vPn\n6yk/3UhmdjLFM7OcjhQ1XMHgqKeLTWl1dW364oi8CV6/lwePP8reukPkJ+dy98I7mZs5y+lYIiIx\nLT8/fcQ9j1TwjUIFn4iIiLNqKls5tOccHo+b5Wumk5mdTM25VtIzk8jISnY6XkQZreDTkK6IiIhE\npMb6Dh79xV78/lD/y4kjNcTFeeju8uFywdor57B6k0YPxkIFn4xZS4eXbW+U09LhZePSIhbP0pYs\nMjx/wM+TpdvYV3eIvORc7pp3K0WpOipLRMbn5NHa/mIPoNcXoNcXOpYvGIQ3Xi5lyYpiklMSnIoY\nNVTwyZj4AwG++Ys91DSGdsDfcaCKz7xrBUvn5DicTCLRM2df4MnSZwGo6qihsr2ar278PG6XNgYQ\nkbFLSR29kAsEgnR3+lTwjYG++8qYnChv6S/2AILASwcqnQskEe1ww9Gw64buRqo6ahxKIyLRyiwr\nIjN7YJ5eWkb4Fix5hWlk56VOdqyopB4+GZPU5KFbsKQN0yYCUJRayJkLtmVJ8CSQk6QpADKg/cB+\nmp/disvjIfvmW0gxi5yOJBGooa6dlguOT+vsCN/APT0zabIjRS318MmYzCxIY9Oyov7rrLQEbl5X\n4mAiiWRvmXsTJekzAEiOS+Ld5m0kx+kbs4R0ny2l8t+/S+fhQ3Qc2M+5f/0OnScs3aWlBAMBp+NJ\nBCk9UR92HfCHb55x5ng9bS3dkxkpaqmHT8bsT9+yhOtXz6C5vYcls3NIjNdxNjK8rMRM/mbtX9LU\n3UxqfCoJHvUGy4D2vXtCM+77BHt7qfjWNwBImDadGZ/9PHGZmU7Fkwgylm1XtL3c2KiHT8bM1xug\nuqGTmsYuWjt0LqpcXHZSloo9GSI+f+QV297KczQ98+QkppFIZpYVMWvewOLAGXPCp4bMW5SvvfjG\nSD18Mmb/8ut92PJmAH738mn+9r2rmVWU7nAqEYk2Ges30L7nDTr27xv2vq++fth2mXo8cW5uvfsy\nWpq68HhcpGUk0VDbTunJBjKzk5lr8pyOGDVU8MmYnDrX0l/sAXh9AZ7dXcFHblvsYCoRiUauuDim\nf+qv8dbUEAwGqPjOt/A3D3x/SV+73sF0EqlOH68nMzuZkrk55BakOR0n6qjgkzEZboZEcNhWEZGx\nSSgsBGDmvV+g8fHH6G1pJmPTZtLXrHU4mUSSitImHn/wQP+CjUWXFXHtrVrVPV4q+GRM0lPimVWY\nztmaNgAS4txcv3qGw6lEJBYkFBZR9JE/dTqGRKh9r5WFrc49dqAan89PXkEay1ZNJyFRpcxY6Ksk\nF/Xjx4/y8sEqAAqzk9m4tIgNSwspyE5xOJmIiMQ6v3/oaNKpo3WcOlpH2elG7nrvyv723l4/HW1e\nMrKScLlckxkz4qngk1EdO9vUX+wB1DR1kZQYp2JPREQmxWVrZlBV3sxwu69UlbfQ3NhJVk4KpSfr\nee6xY/R095KZk8wtb19Gdq5O4ThPBZ+Mqn6YDS23vVFOVloC6xYXOpBIRESmkqIZGcxfXEBNVRsB\nf4D21p7+ey4XPPuHo7jdLhrrO/H29ALQ0tjFzudOcdvdlzkVO+Ko4JNRLZ+XS2KChx6vv7+tvqWb\nHzx6GI/bxWoz8n5aIiIif6wnHjpIbWVb/7XH4+of5g0GobaqbdjXNdV3Dts+VWnjZRlVZmoCf/Oe\nlSycmTXk3mtHax1IJCIiU0VLU1dYsQfDz+kbzuz5uRMRKWqp4JOLml2UwQe3mCHtOemJDqSRSOYL\n9OIL9DodQ0RiRFJyHB7P2BdfFM/MJCc/lcvXzWTDtXMnMFn00ZCujElxbipb1pXw1K4yAApzktmy\nvsThVBJJHjn5ONsrdgAurp15BeuKVnGo/ih5yblcnr8Ut0vvL0VkfBKT4ll31VxefeEUwSC43S4C\ngaE9fC5X6Bi2a241Wp07AhV8MmaNbQMLODq7fPT4/KM8LVPJofqjbCvb3n/9zNnn2Xr2hf7NudcU\nruDDS9/jVDwRiWIr1s9k/uJ8Wpq6aG3p5oUnbP+9aTMzufGupQCkpCY4FTEq6C23jElZTRu7Lpiz\n19bVyzO7yh1MJJGkvO3ckLYLT2J5o2YfDV1NkxlJRGJIWkYS02dls/iyYjZeN4+c/FRmL8jlxrcu\nISU1QcXeGKjgkzHp6hk6L6tzmDaZmkzO/DE8paP4ZGRdJ07Qtvt1At1dTkeRCHba1vHKc6dorOug\n9EQDzz9mL/4iATSkK2O0YEYW0/JSqazvAELzJa66rNjhVBIp5mbO5j2L3s62su24cLEkdxHbK3YQ\nCAYAWFlwGbnJOQ6nlEhV9d8/oG3XqwB4MjKY+YUvkVCgLZ9kqEN7wkcTyk430tLURWZ2skOJokdU\nFnzGmALgW8DNQBpwBPiitfbZYZ79EPAToGfQrd9Yaz8wwVFjhtvt4m/es5Ln95yjucPLxqWFLJgx\ndKsWmbo2T1vP5mnr+683Fa/lQP1h8pNzWZG/3MFkEsl6ysv6iz0Af2srTVufovC9+vYsQ8XFeYa0\neeI0WDkWUVnwAY8CrcAqoBn4CvCoMWahtbZymOfPWmtnT2K+mJSeksDtm2f3r5QSGc20tCKmpRU5\nHUMinL+jY0hboEMb5gr0dPs4dqCanu5eFiwtIDs3lZUbZlJxtgl/b2j0YM7CPBIShhaBMlTUFXzG\nmExCPXrfsdZW97V9C/gCsB54xMF4Me21IzU88NwJ2jt9rF9SyAe3LCJe76xE5I+QvNCQUDwNb1Xf\ne3WXi8wrr3I2lDjO7w/wyM/30tQQKv737ypn4fIiyk41kJGVRG5+KhVnmjhzvJ6flb7Ctbca5i3S\nNIDRRF3BZ61tAf5kUPP53RVHWjaabox5BNgM+ICngHuttY0TkzL2tLT38KPHjuDv2/9o56FqinNT\nuG3jbGeDiUhUc7ndzPz839L03DZ6m5vJ3LSZ5AULnY4lDqs409Rf7AH09gY4sndgAK+pobN/HZjP\n6+fFZ04we0EeHo86IUYSdQXfYMaYDEJz9B611r4xzCP1hHoE/x24G1gG/Ar4P+DW0T53dnbKsPMF\npqKyhs7+Yq+/ra6D/Px0enx+Xt53jrZOL5svm06+Js+KyDjUHt5L20vb8bW2kuAKMGPNctwJ2mZj\nKmtr6h79gUGL/rs7fXS1e5kzP3/iQkU5VzAYvVslGGNmAY8B1cBd1tqhk0GGf90dhOYBllhrR9xM\nrq6uLXq/OJdYe5ePz31/B96+eRPnzS5Kwx+A8tp2AJISPHzxA2uYnpfqREwRiTK9Lc2c/vxnwT+w\nkXvuW99O7m23O5hKnBYMBnn0F/uoqmgBIC7eTa8vcJFXwdVbFrJkxbSJjhex8vPTR5xgH7U9fMaY\ntYSKvd8Cf2Wt9Y3j5Sf7Pk5n5GFgAd44VsuBUw1My0vlz+5YwsMvnqGqoYPz7xNKq9vDnu/2+nlh\nzznee5OGZERkZP7OTjqPHMLX2BRW7AH0lJY6E0oihsvl4vZ3X87Zkw30dPdSMjeHV54/xYkjtaO+\nbvfOs1O64BtNVBZ8xphlhObhfd1a+92LPPsxoNNa+7MLmhf3fTw1QRFjwrO7K/jF1uP918vm5LB8\nbk7/XnwjCWiDXQEq26s5WH+EvORcVuQvw+PW9AgJ6Tl3jvJv/xOB8yt03W4IDPTeJJtFDiWTSFFV\n0cLBNyoAWL5mBqnpidxwxxLamruprmwd8XVdHT4e+NEuSubmkFeYxusvldLd5WPRZcVsvHbelN5h\nIuoKPmOMB/gp8D/DFXvGmHXAz4CbrLVlQALwbWNMJfACsBT4J+Bn1tq6SQsehbbvC9/h5tCZRg6d\nGbrOJTUpjo7u0KkbiQkerls5fVLySeQ62nCc/zzw4/6Nl1cXXM5Hlr3X4VQSKZqeemKg2IOwYg8A\nTbyf0pobO/n9r/YR8Ic6D0pP1PPOP1lLVk4Kiy4vDiv4UtIS6Gz39l/7/QGa6jtpqu/E5aJ/NOrA\n6xVk5SSzdAr/fIq6gg/YSGj/vWXGmL8edO/nwC8AQ6jQw1r7PWNMPPB9oARoIlQw3jdpiaNUatLo\n/3vEx7lZY/K5+5p5HDjdSEeXj7WLCsjL0qKNqe7Z8hf7iz2A3bX7ubPrVnKTsx1MJZHC3zn6KEHL\n88+Sfe31k5RGIs1pW9df7AH4/UEe+sluXG5YvnoGN921BHuwmuy8VFJSE3h1++mw588bvEShqqJF\nBV80sda+DFysTzbsvrX2n4F/nrBQMequK+fwr7/ZP2ShxnlXXFbM+28yAFx1ueZMiMjYZF55NR0H\n9g/9idzHFRc/yYkkkqRlJA1p8/lC8zx37zxLemYSbS3dVJY309sbJBgY2zSiwmkZlzRntBlzv7kx\n5jljzLBnaRljlhljXr90sSQSmJJsvv3xTXzszqV8/U/WsWFpYf+9zNQEbl5X4mA6iWTXz7wKt2vg\n28vqgsvVuyf90lasJOOKETZXdrvJecsd9LY003Ho4LAncUhsm7con5K5I5+93dYS2rLF5w0MKfbO\nz9HLyklm43VzSU1PwBPnZunKaSxdObU7Ji66LYsx5vxP9VJgDaF97cI+B/AW4J+ttUPL8iimbVmG\nOnWuheZ2L8vm5JCo42xkFJXt1RyqP0peSmjRxoUFoEjdr39F09anh95wu0lfvzF0vq7fjysxkWmf\n+BSpS5dNfkhxVENtO80NnTzz6JExv2bDtXOZZ/JJz0zC5Zp6CzT+2G1ZjgPxhLY5HKkXzwW8OP5o\nEm3mTc90OoJECZ2lK6NJX7+Rpue2DdmShUCAtld29F8Ge3qof+g3KvimoNyCNHIL0rii08vrL5Xi\n9wcoKE6nsqyl/5nU9ATyi9Jpa+lm3qICVqybOSULvbEYS8GXAawFXgI+Cwy3HrqJ0DYpEuO6enrp\n6uklZ5g5FiIiY5U0ezYZGzfRunPH0FW6g/S2NE9SKok0Pd0+Du+ppKdvJ4heXyBsE+Z5i/LZfP0C\nJyNGjYsWfNZaL7DDGPNh4AFrbc/Ex5JI9NRrZTzy0ml8vQHMzCw+9fblpCRpcrWIjF/XyRO0vvzS\nsPfcqalh27ZkbNw8WbEkwhzZXxV2pm5tVVvY/UO7K1m1cRbJKTqK72LGvErXWvtTY0yKMWYVkMsw\nK2Wttc9cynASOeqbu3jwhZP9i+pseTNP7SrjbVfNczaYRJTqjhoauptZkDWXBI/eDMjIesrODmlz\nJSWRuvxy8t5xN22v7KT7bCkpixaTpS1apqyertEP0QoEgnR1+lTwjcGYCz5jzG2E9rnLJLzYC/Zd\nBwHN4o9Bh0438PsdpUP3NKrvHP4FMiU9cvJxtpVtByAjIZ2/XvnnZCZmkuCJ14INGSJ50RLCdsYF\ngt3dtO9+naxrriX3LXc4mE4ixYKlhRx4vQJ/3z578fGe/i1aAPIK08jR2e1jMp59+L4FHAa+B9SB\nzs+aCk5UNPOvD+4fdrusy+fnTX4giUiN3U08WzawbqvV28a/7f1vWrytZCVmco95K8vzljiYUCJN\n4rRpFP/Zx6l/+EF8dRccehQI0PLiC6ToeDUBcvPTeOv7V3FkXyVx8R6Wr55OZVkzp4/Xk5mdzMoN\n2h5srMatDS7yAAAgAElEQVRT8M0F1lhrx74+WqLerqO1Q4q99OR4bl4/kysuK3YmlEScdl8HwUHv\nAVu8ofVdzT0t/PTIA/zj5i+R6NGwiwxIX7uO+MJCyu77Sli7O0mn9ciA/KJ0rt5i+q8zspJZsLSQ\nN3aU8tgD+8nKTWHDNXPJ0ClPoxrPOMsZQtuzyBSSk544pK2ty8fW1yuobdKQroTMTJvO9LSR3wB0\n9XZT39UwiYkkWiSVzCJt9Zr+a09aGtk33uxgIok0Pd0+Xt56gkd+voddL56ht9fPGztK2bOzjIa6\nDk4dq+PJ3x5yOmbEG0/B9xXgn4wx2i5/Crl6xXTmFKcPaW/p8PL06+UOJJJI5HK5+NSKj3JjyTWs\nLricFfnLw+5nJmRQlFLgUDqJdPEFA6f4uNMzcCerp0YGbPvDUQ7uPkf1uVZ27zzLjm0nOXmkNuyZ\nxroOWpu7HEoYHcYzpPsuYAFwzhhzEmgfdD9ordXa+RiTkhTHlz6whhf2nuPnzxwPu9fZty+SCEB6\nQhp3zb8VAJ/fx8Mn09lXd5D85FzevuB2PG6t6ZJwAa8Xb00NTU8+3t/mq6qk6Zknyb/7HgeTSaTo\n9fkpO9UY1nZ0f9WQqUYJiXGkpGrKyGjGU/DlAef6fskU4nK5uGrFNLbtrqCqbz8klwuu1Bw+GUG8\nJ553mbt4l7nL6SgSoep+8wDNz20jOMymy776wSd4ylTliXOTmp5AR5u3v21wsRef4OHqLQuJi9eb\nytFc9CzdqUxn6YZr7fTy3O4Kmtu9bFxaiCnR6L6IjF/7gX1Ufu+7I94v/tgnSV+zdhITSSQrPVHP\ns48dxdvjx+12EQiE/2h++4dWUVCU4VC6yPLHnqUbxhiTDiwHioGnrLUdxhiPtdZ/kZdKtAvCHZvn\n4HbrnEIRefN6yspGvunxEF+g+Z4yYPaCPD7wyY00N3ZRVdHCjm0n++/lFaap2BujMffwGWM8hPbi\n+wsggdA+fAsAH/AMcL21tnKCcjpCPXwh1Y2d/OcjB6mo6yA3I5GP3r6UhTOznI4lIlGq6/Qpyv/p\n6yPez9h8JUUf/pNJTCTR5NiBKk4eqyMrJ5lVG2dp7t4FRuvhG88q3S8Dfwp8HdgMnF8O0wxUAf/4\nZgNKZPvF1uNU1IXOtWxo7eFHjx1BUwFE5M1yJyRA3CgDTPr+IiPo7vJx8lgd5acbOW3rqRt0tq6M\nbDwF3weBj1tr/9Fa+wp9J21Ya9uALwJvmYB8EgEqasMXZNe3dNPQ2u1QGhGJds3PbYPeEVb5u1xk\nbLpicgNJ1Hj9pTOUnw6t2u1o62HbH47g82pG2ViMZw5fAfDqCPeqAA2ix6ilc3LYeag6rO2++9/g\nb9+3iuJcnWEoIU+e2cbzFS8T54rj1jk3kByXxP66w+Ql53J9yVWkxqc4HVEiRDAwSg9eMEjnscOk\nLNLRajJUbXV4j563x09rcxe5BWkOJYoe4yn4SoGrCJ24MdhGoOJSBJLI894bF9LV08veEwNbJbR3\n+dj6ejkf2KJvygL76w7z2Jln+q9/ZR8Ou2+bTnLvmr+Y7FgSobKuu562Xa8S9HqHve+tjKnp4HIJ\nTZ+VTW3lQNGXkpZAVq7eTI7FeAq+R4DvG2NKgG2AC1hmjNkCfA34jwnIJxEgOTGOm9bODCv4ALrV\njS59TrUM9z5wQGlrGdUdtRSlavWlhI5Tm/WVr9P2xi48ySnUPfJbgl0DRzWmXrbCwXQSidpaujl+\nuIbEBA+Llhdx9lQDmdnJbL5hPh7PeGanTV3jKfi+ChQSOmLtq4QKvt8BfuB/CS3mkBi1YEYWBVnJ\n1PYdXeNywdUrpjmcSiLF3IxZPDvKfbfLrSFdCZNQWEjubbcDkDRnDvW/e5jelhYyNm4i84orHU4n\nkaS1uYsHf7Ibb09o3md6ZhLv/dh64hPGvbPclDbmr5a11gf8qTHm74E1hObsNQFvWGtrR32xRL0e\nn5+2zvCdzmuburT5sgCwomA5W2ZdxwsVO4hzx7Fp2jp2Vu6i3Rda3b1l1nWkJ2iOjYTzt7fjSkwk\nac5cZnz6c07HkQhlD1b3F3sQ6u17/gnLXJPPXJOH260evrHQSRuj0D58Aw6dbuBffrM/rG31wnw+\n+bblDiWSSNfj93Ky+TR5ybkUpuQ7HUciiL+zk6r//i86Dx3EnZJC/jvvIfOKq5yOJRFq986z7Hpx\n+Gkj8xfnc+OdSyc5UeS6JCdtGGMygL8CVgKZhIZ0w1hrr3szASXyFeWm4HKFb481LU8rdGVkiZ4E\nluZqUY8M1fTUE3QeOghAoLOTmp/dT9urr+Dv7CRj02ayb7jJ4YTipGAwyN5XyzhxpJa09ESWr5lB\nSloCne1DF/mcPFrHyg1tJCbFk56Z5EDa6DGeAfCfAjcCzwHl9O3DJ1NDXmYy91y/gIe3n6bH52fJ\n7Gy2rC9xOpaIRKGe8kFHqwUCdB47CkBd2VncySlkbtZefFPV4T2VvLY91KPXWNdBXXUbN961hJqK\nVk4craGxrjPs+Yfu300wCNNnZbHlbctISNTcvuGM56tyHXCntXa0udkSw25cM5PL5+XS0NrN4lk5\nTseRCGUbT7K9Ygcet4frS65idobeGEi4lKXL6Th4YMT7Hfv3quCbws6eagi77ur08ftfhqYUTSvJ\nHPL8+ZGnc2ebObTnHKs2zprwjNFoPAVfE3BuooJI5PvdS6d5bOdZAsEgJYVpfOZdK8hI0RmGMqC8\nrZL/2P8jAsEAAIfqj/LlDfeSnaSzl2VA1nXX4+9op+21V/GkpdF9+lTY/YRi7QAwlWXnpVDWd5rG\nYJVlLaO+trmhc9T7U9l4lrZ8B/iSMUaD5FNQTVMnf9hRSqDvrVRZTTtPv1Z2kVfJVLOv9kB/sQfg\nDfjYenY7/oD2bJQBLrebvDvfypx/+hYlf/f35L3jneAJ9T8kzZtP9s1bHE4oTlq1cRbFM0I9eW73\niGsQhjV7Qd5ERIoJ4+nh+yXwbqDKGHMc6Bj8wGQt2jDGpBAqQG8BcoAjwJettVtHeH418C1CC056\ngO3Ap6y19cM9L0PVN3cPmbRZ17cnn8h5w/XkbT+3g2NNx/n0qo9raxYZVtfJE+APbbvhrarC39KC\nJ0WLwqaqpOR47nrfStrbemhp7OQPD+zvH7Z1uVxcuLtIXLyH4hkZ9PT0svjyYuYa7QgwkvH08P0S\nuAx4CTgKnB3m12T5D2ATcDOhzaDvB/5gjDGDHzTG5ABPAW8AcwkVfVnAg5MVNhYsmJFJRmr48O1q\no1MTJNy0tGlkJgw9Vrums47tFTscSCSRrru0lI59e/uvA50dNG19ZpRXyFSRlp7I9FnZ3PL25cyc\nm8OchXlcd9siEhI9/c8sXVnMW951OW//wGqWXK6pAKMZTw/flcDt1trnJirMWBhjsoH3Ae+01h7v\na/6hMeZjwMeATw96yXsIbSHzJWttL9BijPkCsM8Yc7m1dj8yqvYuHycrWvjoWxbz8sFqzla34Q8E\nePlgFdnpiSycqflZArWd9Xxv7w/xBXzD3m/1tg3bLlNbwNsztK1naJtMXbPm5zJrfi4Azz9xDG/P\nwBSRw3srWbN5tlbmjsF4vkK1QCScaL0aiAd2DWrfBWwY5vkNwJ6+Yu+8A0B33z0VfKM4eraJf3to\nP15fAJcLrlkxnerG0KTYuuZuTlQ0862PbSIzVYs3pro9tftHLPZcuFhbuGqSE0k0SJ6/gMSZM+kp\nLw81eDxkXn2No5kkcjXVhy/K6PUFaGvpJrdA00UuZjxDuvcBXzHGOD2x4vwA/eAlPPXAcGOM+YOf\ntdYG+9o0JnkRD794Cq8vNAk/GIQX94fX/F5fgCNnhl9NJVNLWvzI3xpmpk9nQfbcSUwj0cLldjPj\n3i+Qd/e7yL7xZkr+9u9JWThkdo4IALPmhW8Jlp6ZRLYOARiT8fTw3UFoDl+NMeYUQxdtBK21my9Z\nsjdnvJtBj/p8dnYKcXGe0R6JeZ094asr/YGhX7LF8/PIz0+frEgSoW7Jvopdtbs51TR0Om9ZWwX+\npC6K0vUeS4aTTtGsdzodQqLATbcvJSkpgaMHq8jJTeH62xaTV6ifP2MxnoIvCyjr++Wkmr6PuYTv\nC5gHVI/wfPGFDcYYF6HVvcM936+pSfv5bFhSyCMvnh72nssFW9aXkJUUR12d5mcJ/PWKj3Oy+TS/\nP/U0Z1oHCj8XLtpavHi69f+JDNXb0kLzs1vpbW0hY8MmUhYtdjqSRLClq6exdHVogUYQ9PPnAqN1\nvoy54LPWXntJ0vzxdhPaWmUD8NsL2jcDfxjm+Z3APxpj4q215ycYrQGSgJcnMmgseMvGWWSlJbDz\nYDW2vDns3mVzc7n7mvkOJZNI5Ha5WZg9n7ctiOO7e36IPxjqIV5ftJqsxKE75IsE/X7Kv/0NfDWh\n99+tO15m+l9/ltSlyxxOJhJbRi34jDEJ1lrv+d9f7JOdf3YiWWtbjDE/Br5mjDkIVACfAGYDPzDG\nTAeeBT5srX0F+BXwFeAbxpj7gAxCe/g9bq09NtF5o53L5eLKy6axdlEB9/7nTjq6B9a+rNJ+RzKC\nc+1V/cUewLmOKgfTSCTrOnG8v9gDIBikdcdLKvhkCJ+3l2MHq+ls9zJ/cYEWaozTxRZtdBnTv9la\nN9B1kV+T5dOEirqXgQbg7cDN1tqzhFbwGiAVQgUicCOwitAQ7gHgNKHtWmQMenx+PG4Xn3nXCpbN\nyWFmQRrvum4+V16mPY9keK9X7w27Lm87R3VHzQhPy1TmSR36Q9tbVUXV//xg1PN2ZWoJBoP8/oH9\nvLz1JHteKeOh+3dTNWjUSUZ3sSHd+4D2C34/3kURE8Ja2wP8Vd+vwfdKCe27d2HbIWBSTgGJJYFg\nkF9sPc6L+yrxuF1sWV/CZ961wulYEgUyE8M3X/a4PKSOsopXppaAz0vDw7+l/eB+EqdNJ23lKtr3\n7gnddLvpKS+jp7yMttdeZdpf/jVpl+n7zlRXU9lKbeXAXL1AIMihvZUUax/YMRu14LPWfu2Cyx8D\nFdbawODnjDGZwKJLnE0c9saxWp7fE1oX4w8E+f2OUjp7eklK8LBucSEz8tWdLsO7dc6NnGo+Q4u3\nDRcubptzo45Vk371D/+W5q1PA+Crria+oIDE2XPwNdQTaAufgN+6c4cKPsHjGTogWVvZSmtzFxlZ\nyQ4kij7jWaV7BigC6oa5V0Lo+LLsSxFKIsPZmqErn7a9UQHAU6+V8fn3rGL+dE3El6GKUwv52sYv\ncKb1LLlJueQm61uDDOg4GL7fva+2dsRnPelDj+qTqSe/KJ1Z83M5e7Khv621uZvHf3OAez66DpfL\nNcqrBcZQ8Bljvtz3WxfwOWPM4P33XMBGxreJs0SBpbNzePLV4Xfh6fUHeX7PORV8MqJ4TzwLs7WK\nW4ZKnDYdX/Wou2IBEJeTQ86WWyYhkUSDLW9bxq9/tIvmxoElA82NXTTWd5CrEaeLGksP31xgE6H5\ne/eO8Ewb8PVLFUoiw5LZObz/ZsPW18sJBIPUNoWvy4mP0zsqGV4gGODZshfZX3eYvORc3jL3JvKS\ncy7+QpkS8u++B29NDd5zFbhTUwl0hPcjZFx1Demr15C80OCOj3copUQat9tFQXFGWMEXF+cmLT3R\nwVTRwxUMjm0dhjEmAMxgYOPjftZa/9BXRL+6uraIWKQSKe776euUVoWGeeM8Lv7+A2uYqR3OZRjb\nyrbzyMnH+68LU/L50vrP4nZpIEAG+BobiMvIpHn789Q//BDBnh5Sli6j+GOfxJOseVkyVGtzF4//\n5gDNjV3ExbnZdP18lq7UjhHn5eenj9gTM56Nl/Wdegpr7/Jxrq69/7rXH8SWN6vgk2Htqz0Udl3T\nWUd1Ry3T0oocSiSRKD4nF4Ds628k84qrCHR3E5epaSIysoysZO756Dqa6jtJTU8gMUk9wGM15oKv\n7ziydxM64SKXQVufEDpL972XMJtEkDNVrfh6wzs8j5U1c8OamQ4lkkgTDAaxTSdp6WklOymTM60D\n9+Ld8WQlavK9jMydmIg7UUNzcnEul4ucfG3zNF7jWaX7r8BfAh2ENjsePNyp4c8YNrMgDY/bhT8w\n8J95dpF692TAjw//gj21oY1yEz2J5Cbl0NDdSLw7jncsuJ2U+BSHE4qITF3jKfjeCfwd8C1rrYq7\nKSYrLZGP3LaYXz93kvZOH+sWF3DzOvXuSci59qr+Yg+gx99DMBjasjMQCNAbiMlpviIiUWM8BV8a\n8ICKvalr49Ii1i8pJBAIEjfMJpgydXn9Q4/R9gZ8APgJ8Mipx1lbtJJU9fKJiDhiPD+1XyJ0Hq1M\nUYFgkCNnGnntSA3tXT6n40gEmZ1RQkn69BHv9wZ6aelpHfG+iIhMrPH08H0O+KExphB4jYEzdvtZ\na49fqmASeb7/8EH2nqgHIDUpjr97/2qKczVxVkKTqP9y5Z/xn/t/zOmWs0PuF6cWUpxa6EAyERGB\n8RV8h/s+XsHICzQ8f1wciVSl1a39xR5AR3cvz7xezge36AhlCXG7PJxtrQhri3fHs6bwcm6bc5OO\nPhIRcdB4Cr6PoJW4U5bXFxjS1uPTRHwZ4CLU03fhdwlfwMe+usOY7AWsLVrpWDYRkaluPBsv3z+B\nOSTCzZ+RSUlBGmW1oZF8j9vFNStGnrMlU0+CJ4FrZmxmW9n2sPau3i5+aX/LsrzFJMclOZRORGRq\nG/PRagDGmBTgA4QWbxQBf07oqLUrrLUvTkhCB+lotXCd3T5e3F9FS0cP65cUMrtIG+nKUEcbjvOL\nYw/R1NMc1v536z7N9LRih1KJiMS+S3K0mjFmLvA8MB04CcwDkoGFwDZjzJ3W2if/yKwSwVKS4tmy\nvsTpGBLBXql6g9+fepJ2X0dYe25SthZtiIg4aDzbsvwLUAbMtdYuAroBrLXHgH8AvnTp44lItKjv\nauAXRx+k1dtGoG/T5YyEdJbnLeb2uTfz3T0/5Cs7v8ljp5/uvy8iIpNjPIs2rgFutNaWDXPvV8Df\nXJJEErGa2np4elcZrR1eNi4rYvncXKcjSQQpbS0nOGhdlz8YIM4VxwP2Ebr9PQA8WfosafFpXDNz\nsxMxRUSmpPH08AWAkXZOTUAreGNarz/AN3+xm2deL+fVIzX862/2c+BUg9OxJAJ4/V58gV7mZMzC\n7Qr/ltLh62Bv3cH+Yu+8o412MiOKiEx54+nhO0ToLN0PXtB2vsj7U2DvpQolkedEeTN1zd1hbTsP\nVXHZPPXyTVWBYIBf20fYWfU6cS4PN826jg8tuYffn3qKxu5mAow8bDs9bdokJhURkfH08H0TeK8x\n5oQx5odAPPA1Y8wu4JPAVycgn0SI9NSEoW0pQ9tk6nijZh8vV75GIBjAG/Dx2Jmn8QX8fOLyjwzZ\nc8/FwMKxlLhkNhSvmey4IiJT2pgLPmvtE8D1hFbovoPQqRpbgCrgWmvtsxOSUCLCjPw0rrp8oFcm\nNyORLeu0YncqK287N6Tt50d/zX2vfYeWnhY8roGDdy7cJ6Czt4tnB+3VJyIiE2s8Q7oAXcB7rLVN\nAMaYDwNrAO23MAV86JZFXLdqOq0dXkxJNvFx4+kgllizKGcBz5W/NOy9Y00nw64Dg6b4VnZUT1gu\nEREZasw/sY0x7wJ2AvP7rj8P/DewGfg/Y8yHJiKgRJaSwnSWzc1VsScszV3E3QvupCA5j7T41HG9\ndkmOzmAWEZlM4/mp/Xng7621rxtjXMBngG9aa1cAnwA+NREBRSRyXTNzM1/Z+Hk+fvmHw+bpxbnC\nBw9ykrJYlL2AnKRsbiy5hptmXTPJSUUkVnR3+Th5tJa66jano0SV8RR8Bnig7/drgXzgR33Xz9HX\n8yciU8/sjBK2zL6etPhUMhMyeOfCu1hVcDkpcckUpuRzfcnVVLRX0tjdxJ7a/VR11DgdWUSiUG1V\nK//3X6+y9dEjPHT/bnY+e/LiLxJgfAWf94LnbwBOWGvP9l0njPNziUgMOdl8hqdKn6Xd10GLt5Vf\nH3+EPbX76eztoqazjodPPNZ/3FpDdxMPnnjU4cQiEo327CzD5/X3Xx94o4KO9p5RXiHnjadIOwB8\n0hizFvg48MgF9+4Ajl/KYCISPfbXHQo7ZcMf9IfdH3xd01E3KblEJLb09PSGXQeDhBWAMrLxFHxf\nIbTB8quEVut+B8AY8xbgW8C/XvJ0IhIV8pPHtwH38rzFE5RERGLZ4suLw66nzcwkKyfFoTTRZczb\nslhrtxtjZhCay3fYWtvZd+sY8BZr7dMTEXA4xpiVhIrMNYAf2AF8xlp7eoTn7wfeD/gG3foLa+2P\nhr5CRMZjQ/FaDtYf5UijxYWLFfnLONlyhjZvOy5c3FByNW3edsraKjDZ87l93hanI0uE8dZU429t\nJWnuPFwez8VfIFPSwqWFJCZ6OHagmpz8VC5fN9PpSFHDFQxG1xG4xphi4DChLWHuA1KB/wMKrLUr\nR3jN/QDW2g+N58+qq2uLri+OiMNqO+tJ8MSTlZiJ1+/jdEspeck55I2zB1CmltoHfknztmcAiC8s\nZOa9XyAuK9vhVBKJWpu7ePzBgzQ3dBIX52bT9fNZulJHNZ6Xn5/uGuleNC60mE5o/uAXrbWd1to6\n4L+AFcYYfYcQcVBBSh5ZiZkAJHjiWZSzYEixd6j+KL+2v+PFip34Ar3DfRqZQrxVlf3FHoCvpobG\np59yMJFEstdfKqW5ITTA2NsbYMezJ+npHjx4J8MZ70kbjrPWvgH8yaDmuUBr36+RXGaM2QEsBWqB\nnwDfttZqtqfIJRIMBqnsqCYtPo3MxHQA6jobePHcTnoDvaTGp/Bk6cApjMebTvGny9/vVFyJAL3N\nzUPbmhodSCLRoKW5K+za3xugva2HxKR4hxJFj4gr+IwxcUDaSPettWHfHYwxy4GvAl8epXg7A6QD\nX+z7/c3AL4EAobmAw8rOTiEuTnNJRMaipbuVf9j+75xtrsDtcvO2JbewZf7V/PPL/0GbN7Qly+Cx\nhn11h0hID5KZlDH5gSUiBLJWUZuXh7e+vr9txk3XkZuf7mAqiVTLV86g5tyR/uvc/FTMoiJc7hFH\nMqVPxM3hM8bcAGwd5ZFka21337PXAw8B/2Wt/btx/jn/AtxprZ030jOawycydg+ffIxny17sv3bh\nYl3RKl6r3j3ia+JcHr555ZdJjkuejIgSoXz1dTQ++QS9Lc1kbNxM+uo1TkeSCBUMBtm/q5zTx+vJ\nzEpm7ZWzycjS94/zRpvDF3E9fNbabQztCBjCGPMnwL8Bf2mt/fGb+KNOEpoPKCKXQENX+DBckOCw\nxZ7b5SYQDABww6xrVOwJ8Xn5FL7/g07HkCjgcrlYsb6EFetLnI4SdSKu4BsLY8z7gH8GbrHWvnSR\nZz3AN4CHrbWvXnBrMaGiT0QugZX5y9lXd2jUZ4pTC7l7wR1UdtQwN3MWszK0pYKIyGSIuCHdizHG\nzASOAO+01j45wjN/AbzDWntN3/XDhM76vRs4DdwK/Ar4pLX2JyP9WRrSFRmfVypfZ1f1Hio7qvuP\nUjvvHQtu53jTaQ7UH8btcnPl9I28c+GdDiUVEYk9UTWkOwYfJLSo4xFjzOB7H7XW/hzII1Tgnfdh\nQr1824B84CzwcWvtTyc+rsjUMTdzFnvrDjLcG8nHT2+ly98NQCAYYHvFDlbmL2NB9ojTaEVE5BKJ\nuh6+yaQePpGxCwaDfP2171DTOfZzct9t3sYV0zdMYCoRkakj1jZeFpEI1NjdNK5iL84dx+KcIb30\nIiIyAaJxSFcc1NjaTWunl5LCdNwu7XskAzISM0iLTx0yd+9CawtX0tDdRII7nptmXUtusg7HERGZ\nDBrSHYWGdMM99MIpnnztLMEgTM9L5XPvXklmaoLTsSSCHKo/yi+OPUSrt23IvVvn3Mits2/ApTcK\nIiITQkO68kerbuzkiVdDxR7AufoOnn6tzNlQEnGW5S3mHzb9HQsHLcRw4WJT8VoVeyIiDlHBJ2PS\n0No9pK2+pWuYJ2Wq87g93DL7BuJcA8cSritaRXZSloOpRESmNs3hkzFZOCOTzLQEWtq9/W3rFhc6\nmEgi2cLsefz9hs9xsP4o+cm5LMnV4gwRESdpDt8oNIcvXE1jJ4+9UkpLu5dNy4rYsLTI6UgiIiLS\nZ7Q5fCr4RqGCT0RERKJFrJ20ISIiMSQYCNBx6CD+1hZSL19BXHqG05FEYo4KPhERcVTl979Hx/59\nALhTUpj5hS+ROG2aw6lEYotW6YqIiGO6S0v7iz2AQGcnzduecTCRSGxSwSciIo4JeHvG1CYifxwV\nfCIi4pjk+QtILJk10ODxkHX1dc4FEolRWqU7Cq3SFRGZeP6uLlpfepHe1hbS160n6cICUETGTNuy\nvEkq+ERERCRa6CxdERERkSlMBZ+IiIhIjNM+fCIyIfbXHWJ/3WHyknO4duYVJMclOx1JIlgwEIBA\nAFecfiyJTAT9yxKRS+61qt387Oiv+6+PNp7gs6s/4WAiiWStr+6k7jcP4G9vJ33tOgo/9BHc8QlO\nxxKJKRrSFZFLbmfVrrDr0y2l1HTUOpRGIllvSzPVP/lf/K2tEAjQ9tqrND3ztNOxRGKOCj4RueTS\n4lPDrt0uN8nxGtKVoXrKy8DvD28rLXUmjEgMU8EnIpfcltnXk3LBnL0bSq4mIyHdwUQSqZJmz8WV\nED58m2wWOZRGJHZpH75RaB8+kTevq7eL402nyE/OY1pakdNxJIJ1HD5E/UO/pre5hYxNm8h7+ztx\nudUfITJe2nj5TVLBJyIiItFCGy+LiIiITGEq+ERERERinAo+ERERkRingk9EREQkxqngE5EJ0dXb\nzYG6w1S2VzsdRURkytPRaiJyyZW3VfK9vT+ks7cLgJtnXccd87Y4nEpEZOqKqoLPGDMbOAN4gQu3\nTKm21s4e5XV/BfwZMAsoA/7bWvvdiUsqMrU9Vbqtv9gD2Fr2AtfM3KzNl0VEHBJVBd8FjLW2dEwP\nGkTARlsAABXzSURBVPMB4OvAncAOYB3wuDGmyVr704mLKDJ1tfs6wq4DwQBdvi4VfCIiDpkKc/j+\nEvixtfZ5a63XWvv/27vzKDmqeoHj35AESAKYEAYQMSwCPzU+QcAHIiibBFRUfCj63IKoIKLwEJen\nPBAXcDmsrogiyqKiRyVxAYmKCIKAKLL+2DdRCFkQTAIS8v641Umn7Z6ZhJlMT833c86cmap7q+p2\nUl31q7vUvQw4EzhiiMsl1da2679wueUNxq/PBhPWH6LSSJKGa8B3QkTcExFzIuKiiJjaLlNErAFs\nA1zVknQV8MKIGD/YBZVGor//86Hllh95/BEeX/zEEJVGktRVTboRMQZYq5csjwPXAL8B3gmsC3wJ\nmBURz83MR1ryrwuMBua2rH+YEuxOBhZ0OtikSeMZM2b0Cn0GSfC3P/9tueVFix9n8ZoL6Zk4eYhK\nJEkjW1cFfMCuwMW9pI/LzBc3Lf81It5JCeBeA5y9gsfrda7cefM6xoKSerHZ2ptx29y7ly4/Y/W1\nWePxtZg9+9GhK5Qk1VxPT+d+0l0V8GXmLKDjxL8dtpkXEXOAZ7VJngM8SanJa7Zetf7hlSmnpN69\nctM9WfTkIv40+3p6xq3H/lvuy+jVrC2XpKHSVQFfXyJiT+BlmXlM07oeSgB3e2v+zHwiIv4I7Aic\n25S0M3BNZi4a5CJLI9LY0WM5IPbjgNhvqIsiSWKYBXzAPOB/I2Ie8FVgInA6JdibCRARhwH7Z+au\n1TYnAWdGxE+A3wEvB6YD71ilJZckSRoiw2qUbmb+kfI+vQOAB4EEFgK7ZebjVbb1gC2atjkf+CDw\ndeBR4CvA4Zn5w1VYdEmSpCEzasmSXsctjGizZz/qP44kSRoWenrW7jgOYrg16UoaJq596C9cN/sG\nesZNZvdn78L4sb72UpKGigGfpAF3xQNXc84tP1i6fMvc2zhq+8OGsESSNLINqz58koaHK/9+zXLL\nd/3j3n+bfUOStOoY8EkacGuNXX7CnNVGrcb4seOGqDSSJAM+SQNun033YMKYZX329tpkN9ZZvfMb\n4CVJg8tRur1wlK608hY9uYjb5t9Jz7jJbDhhg6EujiTVXm+jdA34emHAJ0mShoveAj6bdCVJkmrO\ngE+SJKnmDPgkSZJqzoBPkiSp5gz4JEmSas6AT5IkqeacS1fSoLjzkXu4bvYNrDduMjtuuB1jR48d\n6iJJ0ohlwCdpwF03+wbOuP5sllBeZXn9wzdx6NbvHOJSSdLIZZOupAH32/t/vzTYA7hxzi3MXjBn\nCEskSSObAZ+kATd2teUbD0YxijGrjR6i0kiSDPgkDbhXbLLbckHfThu9mElrThzCEknSyOZcur1w\nLl1p5c1dNI8b5yQ94yYTk7Zg1KiOUzxKkgZAb3PpGvD1woBPkiQNF70FfDbpSpIk1ZwBnyRJUs0Z\n8EmSJNWcAZ8kSVLNGfBJkiTVnAGfJElSzRnwSZIk1ZwBnyRJUs0Z8EmSJNWcAZ8kSVLNjek7S3eJ\niKOBo9skrQFMz8xvt9nmLOBtwL9akg7LzG8MeCElSZK6SC3m0o2IacB3gKmZ+XCb9LMAMnP6iuzX\nuXQlSdJwUeu5dCNiHeAM4P3tgj1JkqSRbtg16bZxLHBzZp7fR74XRsTlwFTgIeBbwOczc/FgF1CS\nJGkodV3AFxFjgLU6pWfm/Ka8U4BDgV362O1dwNrAx6u/pwHnAU8Bn+u00aRJ4xkzZnS/yy5JktSN\nuq4PX0TsCVzcS5ZxmbmoyvslYIvM3HsljnMS8NrMfE6nPPbhkyRJw0Vvffi6roYvM2cBHQvcUNUE\nHgB8eCUPdTvwrJXcVpIkadjouoBvBewOTAYu6C1TRIwGTgB+lJlXNiU9jxL0SZIk1dpwHqW7I3Bv\nZs5tTYiIwyLiEoBqUMYWwNejGBsRrwUOAk5clQWWJEkaCsO5hm8jymjbdtajBHkNB1Jq+WYBPcA9\nwHvbvaRZkiSpbrpu0EY3cdCGJEkaLmr94mVJkiT1zoBPkiSp5gz4JEmSas6AT5IkqeYM+CRJkmrO\ngE+SJKnmDPgkSZJqzoBPkiSp5gz4JEmSas6AT5IkqeYM+CRJkmrOgE+SJKnmDPgkSZJqzoBPkiSp\n5gz4JEmSas6AT5IkqeYM+CRJkmrOgE+SJKnmDPgkSZJqzoBPkiSp5gz4JEmSas6AT5IkqeYM+CRJ\nkmrOgE+SJKnmDPgkSZJqzoBPkiSp5gz4JEmSas6AT5IkqeYM+CRJkmpuzFAXoJ2I2B04EyAzN21J\nGw0cB7wReCZwO/CFzDyvl/1tDpwC7ACMAv4AHJ6Zdw5G+SVJkrpJ19XwRcQJwDeAmzpk+TjwDuBN\nwGTgE8BZEbFrh/2NBX4OzAemAlsBs4FfVGmSJEm11nUBH/AYsA1wbWtCRIwCDgNOzMxrM/OJzLwA\n+BlweIf9TQO2BP4nMx/OzPnAUcBzgFcOxgeQJEnqJl0X8GXmZzLzHx2SnwP0AFe1rL8K2LHDNjsC\nd2TmnKZjzAXu6GUbSZKk2lilffgiYgywVqf0qvatNz3V77kt6x9uSmu3TWv+xjbr93E8SZKkYW9V\nD9rYFbi4U2JEjMvMRSu57yUDvU1Pz9qjVrIskiRJXWOVBnyZOYsySnZlPVj9ntyyfr2mtHbbtOZv\nbPP3p1EWSZKkYaHr+vD14S5KkNba925n4Hcdtvk9sHlELG2+jYgNgC162UaSJKk2uvI9fJ1k5pKI\nOBn4YERcAtwAvB54BSXoAyAibgGOzswfUpqQbwROjYjDKDWMpwHXA7NW7SeQJEla9boq4IuITYCs\nFscAoyOi0afv3Zl5NvAFYE3gAsqAjAT2z8zmkbsBTATIzMUR8SrgS8DdlH57s4BXZebiwf1EkiRJ\nQ2/UkiUrM9ZBkiRJw8Vw68MnSZKkFWTAN4JExCURcU4/8h0bEd9fFWVqc+y7I+LT1d9HR8TtQ1EO\n9V9ELImIdw3Qvpb+/2v4iIhNImJRROw21GUBiIiXVeXZfKjLooFRXRuO7iXd+0UfuqoPn5YXEeOB\nvwJrABtXM4SsCtOAb1ZlWAI8AWzUPFtJlbYWZdT0hMwc8HcWZuanAW/+Q6gaHLUL8K9q1RPAfcC5\nwAmZaZ+QmqvOgfsz861t0i4Dbs/M6ZS+1V0hMy+li8qjZSLiD5Tz6b9a1l8EvJByr1nStP7/gCOB\nBb3tt/V+ERGvBh7IzH+bpnWksoavu70VeAC4GThwVRwwIp4B/CfLvyB7blWWVm+gzH2sevtuZq6Z\nmWsC6wIfBP6PzvNXS1InM4A9I2JsY0VVufFyYALwopb8rwQuZNlDZ38dB2z7NMpZO9bwdbf3UWpS\nFgKHRsRJjSefqubtSOAQ4L7M3DMiAjiJErCtDlwHHNHyhDMqIr4ATK+WLwDen5kLq+U9KE/s9zZt\n81NKwHlqS/mmAzOBpc157WoDmmsBImLNqoyvo4ykfhA4gza1RRHxCeBdmblxRGxKeQ/jvsDHga0p\ngfAbgbcB76Wcz2dk5kc7/HvqacrMJ4ELI+JWytzWy6mmT/w08CaWvRD9tMw8tSnPNOAzwPOAv1Xp\np7U7XkR8hHKe75KZtw7wx9EAafp+viIzZ0XEtsCJwDaU7+X1wIcz87KI2BX4DfAq4ATKWxXuBw7O\nzF9V+9sQOIUyO9ME4FbgI9XL+4mIs4BxwOXAUcAkyntVp2fmQ03H2DIzb69aIz4P7A+MrbY7PDPv\nGLx/FXUwg3KN2JnyfwTlvvMA8CdKgHctQERMptzPTgNeAoyJiC8Cb6Hc484GDqvexvEJlt0v7gee\nBXw1It6fmVtHxDjgs8BrgA0pb+04ITO/M/gfuTtYw9elImIXYCrwbcpJPYXS1NrsIGA/ynsIAX5I\nqY2bQjmh7wJ+1LLNqykX12cDu1NO/k81pe/Fv09/9xNgs4jYrql8m1Oeni5YwY92BOWLvm1mjqfU\nEh7e5rN18iHgzZTPtxrwa2A+sHG1r49ExH+sYJnUTxExLiLeQAn2vtsmy+GUh4M9gLWB9wOnNPp2\nRcQLKOfMyZSb9FuB4yOiXXPh24CPANMM9oad8yhB1QaU12fNBM6LiNFNeT5MefCbXKXPiIh1qrQz\nqu22otQqXwj8qCkdyvWrB3gu5Vq5bbXPdk6nBJ/bUAKBucDPI8J74CqWmddT7k2valr9Skrw99vq\n74a9gaeAX1TLB1LuTxtQrveHtORvHGPj6s/3ZubW1d+nUyZtaFybPgZ8MyJe9vQ/1fBgDV/3eh9w\nYWb+FSAiLgAOpVz4Gi7KzJualncCnmzU1lUDL94eERtmZmMaub821bZcXw3ieB3lKRlKwNfaVPdP\n4HuUAPOP1bp3UALMFW3SnUj5Ai8EyMxrqvL1ty/YuZl5N0BEXAwckJmnVGm/jojZwPMpNQoaGG+O\niP2rv1en9OM7BriiTd7TgDMzc161/PPq/2QHygX9IODGzDy3Sr8yIvYDWvuH7gV8kfK+zD8P6KfR\nymg+B5qtDrTrKD+Rcp78q/pun1D9UBoiAPhiZt5Vrfsk8AFgH+D7lJr70Zn5WJV+LuUG/Xzgymr7\nJ4FjM/Mp4N6IuJTSB2w5VS3RAcB+mflAte5DlCbENemjb5gGxUxKoNa47+xDCdb/ApwUEetWfdb3\nAS7NzPnVeXNFZs6otvlFRMylBPszeztYRKxLqRXcJzPvrFb/OCJmAO8GLh24j9a9DPi6UEQ8kzKD\nyBubVp9BaUrbJDPvqda1NkfsBBwTEVMpF7LG02tz5+UbWra5jVLbR0RsSakpu6RNsb4JXBQRRwKP\nA2+vfka3ydubL1Oe2h6IiN9SntbOBR7q5/Z3N/29ALinJX0BpalHA+e7jSb6qsl2KuVp+cWUp+xm\nEykX7D0pNXhQBh01zsEtKU/3S2Vma43yNpSHjlMz8/KB+hB6WpaeA82q7hrtHEX5rh8UEbMoN+SZ\nLS+7v7nxR2bOi4j5VNci4AXAZ6pWhbWatmm+lt1VBXsNC4CN2pRlM8p1aul5l5kPAud3KLsG3wzg\nAxGxGeV6vTFwcXUe3AdMqyosprH8wL07W/azkP4NztmScj+cWXWHaliNZQ8QtWd1dnc6mNLP5NsR\nMb+6EP6A8v91SFO+Jxp/RMRWlC/RFcDmVQf7fdvsu11NWmM2k72AKzPz0dYM1Uwm91EC0V0pT+79\nnYt4aVCYmfdl5jbVPi6j9L+7LSK27+e+nupjWYMoM5/MzOsoTev7V+dds/MpAdvuwPjqPPx7U/pi\n+r7u7E6pPT68zf41DGTmOZSm0w9QHhC/SamBb35AbH1YHAU8VQ0c+yUwG5iamWvQpuaO/n/3G0Gm\n97vucSmlK87e1c8fmloFLqrWbU/pBzyjabuVvd43+qjv2BiAVv2snpkjpknXL0CXqUYuvYcysGFr\nlvU72YbS0f2giFijzabbUppXjm96fcsO7Q7RsrwVJZCD9v33mn2DUi3+VuBbHfIsBMY3fZ7VKE/Y\njeUJETEuM6/OzOOB7YA/U2oLNfxMaFl+CfCtzLw5M5+KMl3ihk3pt1L6XC0VEa+pXqHQcGpmHki5\n8J/f4XxXF4uInsx8LDMvyMxDKR3vX0a5pjVs2ZR/XUrt8L2UwTwTgRObuqK0u5b1152U5t+l511E\nrBcRR1XH1SqWmf+idE/ag/KA19xV6UJgzyrthkaz/9N0JyXwX24EcERMqVotRoQR80GHkdcD6wOn\nZOZ9zQkRcRqln0NrMxosa97dOSJ+Sande3m1bgrLmkI3j4iDKQHbVErwdlp10u8GHN9L2c4BjqXU\nLG7XIc8twFuqG/2DlH43zefZT4CHI+LwzHwI2ILSjDMkL3rWiqsG7BwHXEUZCd7sDmCHiFidMrDj\nBMq5N6VKPwM4IiIOpdT6TAXOYvl+o40amXdRHgZOpvRf1TAQEVOA2yNiOqWm9inKQK1FlC4YjUFV\nH4iIayndOY6h9Ae+kNKhfjHw0oi4nnIda7yzrXEe9VtmPhIR5wEfj4irKTXOn6IEFCevzGfUgJhB\nefPDGOATTet/RRmM83b+fdDhilgAbBURk6qm4m8Ax0bEnyh9BV8K/JhSC31uL/upDWv4us/7gJ+2\nBnsAVYD0I9rc/DLzakoN4JmUV13sQwkeL6N0bm0Ef99n2eswflnt7/OUmpnFwDWdClbVHF4EXNvo\n/NzGFygDJm6i3PznsGzoPZRXuawB3BIRCygX+HOAr3U6robcm6PMWrAoIhZS+njeTBlQ0drEcgjl\nhj4f+A6l/80pwJsi4mvVaNu9KK/RaXRV+HRmnt160MycD/w38O4OAwbUhapXOh1AafafBzxMabV4\nTcvL279OeQCcSxmx+eqqVvBvlJvwx6q0wyjB//nA6e1GdPfDIZS+WtdSrn2bUM7fxb1upcH0C0o/\n3ydZNhiQqkvR7yk1sjPab9ovp1HeEnBjtXwkpS/phcCjlH7IxzQNIKu9UUuW+KJ8SdKq0fqOvCEu\njjRiWMMnSZJUcwZ8kiRJNWeTriRJUs1ZwydJklRzBnySJEk1Z8AnSZJUcwZ8kiRJNWfAJ0mSVHMG\nfJIkSTXnXLqS1A8RMRH4CvBqyjSE51CmLvweZbrCjwIvoMwN/Cngy5l5dERMokxfuC+wLvAA8F3g\n2Mx8otr33cCVmfmmpuO9izL38GaZeXdEnAXsQJku6mRgK+A+4Lh2U9NJUjNr+CSpf75CCdrew7K5\npz/VkqcH2BvYBTixWjezWvduyvygx1Dmh/3iSpRhI8ocs+8BXgRcDpwVES9eiX1JGkGs4ZOkPkTE\neGB/4MTM/F61+oiI2B7YsinrFGBaZt5SbbcT8FJg/8ycWeW5MyKeBxwZER/MzMdWoCjrAB/KzD9W\n+z8UeCPwZuDqlfx4kkYAa/gkqW9TgLHAtS3rf9ayvKAR7FW2r35f3pLvD8DqwPNXsByLmsuQmf8E\nEth8BfcjaYQx4JOkvk2ufrfWxs1uWX60ZXmd6vc/OuRbewXL8Whmts6H+RgwaQX3I2mEMeCTpL49\nXv0e37J+cmvGFo9Uv5/Rsr6xPL/6vQQY1ZKnXTA4oc26tYE5fZRD0ghnwCdJfbuDEpRt17L+9X1s\nd031e+eW9TsBC4CbquX5lBG8rXlajY+IpWWIiAlAADf3UQ5JI5yDNiSpD5k5LyIuBg6JiCsp/eYO\noY8m2cy8IiJ+C5wUEY8BtwJ7AO8DTs7MhVXWq4F3RMS+lCBwP2Bqm10+Vu3ro5Taw49R+hae83Q/\no6R6s4ZPkvrnQMpgi+8Dl1Bq6E6p0hb1st3rgIso7+e7GfgwcBxwdFOeY4FfAedSgr8tqnWt/gl8\nBjgd+BPl9TBvy0xr+CT1atSSJa39fyVJrSJiTWBCZs5pWvdZ4EhgXGYuHuTjnwXsnZkbDuZxJNWT\nTbqS1D9nAntExHRKs+t2wMHAtwc72JOkp8uAT5L652Dgc5TpztYD7ge+DnxyKAslSf1hk64kSVLN\nOWhDkiSp5gz4JEmSas6AT5IkqeYM+CRJkmrOgE+SJKnmDPgkSZJq7v8Be+8XknlWNmsAAAAASUVO\nRK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f3d22c2c1d0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot = seaborn.swarmplot(x='group', y='sentiment', data=name_sentiments)\n",
"plot.set_ylim([-10, 10])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that as a bar-plot, too, showing the 95% confidence intervals of the means."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAF8CAYAAACKSxzUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYXGWZ9/FvZyMJCVuMC/IGcOF2hBHEHcWdsKi84r4i\noCwCiuA64oAoiMsoizIDghEXxOVVB3AAE1BRFBQFHRW8cQFlEYwQNCEEstT7x3MaKkV30tVd1XXS\n9f1cV19dZ627kuqq33me55wz0Gg0kCRJUj1M6nUBkiRJeoDhTJIkqUYMZ5IkSTViOJMkSaoRw5kk\nSVKNGM4kSZJqZEqvC+iUxYuXek0QSZK0QZg7d/bAcMtsOZMkSaoRw5kkSVKNGM4kSZJqxHAmSZJU\nI4YzSZKkGjGcSZIk1YjhTJIkqUYMZ5IkSTViOJMkSaoRw5kkSVKNGM4kSZJqZMLcW1OSJPXOihX3\nsHLlyra3mzp1KtOnz+hCRRsuw5kkSRqTBQs+y6JFF9FoNNredmBgErvttgcHHHBQFyrbMNmtKUmS\nxmThwgtHFcwAGo01LFx4YYcr2rAZziRJ0pjMn78XAwOjixQDA5OYP3+vDle0YRsYbdKtm8WLl06M\nFyJJ0gZoqDFny5Yt5cgjD7t/+qSTTmPWrNlrrdOvY87mzp09MNwyx5xJkqQxmz59xnpD1qxZs5k9\ne5NxqmjDZbemJElSjRjOJEmSasRwJkmSVCO1HHMWEdsDHwV2ATYGrgWOy8zzelqYJElSl9Wu5Swi\nZgKXAX8AtgE2Bb4JfDMiHt/D0iRJkrquduGM0lL2PuDozFyamfcCnwEmAzv0tDJJkqQuq123ZmYu\nBs4anI6IOcC/ATcDl/aqLkmSpPFQu3DWLCLuBaYBVwG7ZeYdw627+eYzmTJl8rjVJkmS1m3atDVr\nTc+ZM4tNN509zNoaVOtwlpkbRcRc4DDg8ojYJTOvH2rdJUuWj29xkiRpnZYuXbbW9B13LOO+++o4\nomr8zZ07fEit/b9QZi7OzA8CtwCH9LgcSZKkrqpdOIuIvSPixoiY3rJoI2BVL2qSJEkaL3Xs1rwC\nmAl8JiLeAywHDgYeA3yrl4VJkiR1W+1azqqzNZ8HPBL4M3A78Dpgn8y8spe1SZIkdVsdW87IzN8C\ne/a6DkmSpPFWu5YzSZKkfmY4kyRJqhHDmSRJUo0YziRJkmrEcCZJklQjhjNJkqQaMZxJkiTViOFM\nkiSpRgxnkiRJNWI4kyRJqhHDmSRJUo0YziRJkmrEcCZJklQjhjNJkqQaMZxJkiTViOFMkiSpRgxn\nkiRJNWI4kyRJqhHDmSRJUo0YziRJkmrEcCZJklQjhjNJkqQamdLrAiRJ6ndnnnxxr0voipUrV6w1\n/aXTv8fUqdN7VE13HPiOPTq+T1vOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1\nYjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmq\nEcOZJElSjRjOJEmSamRKrwvQ2K1YcQ8rV65se7upU6cyffqMLlQkSZJGy3C2gVuw4LMsWnQRjUaj\n7W0HBiax2257cMABB3WhMkmSNBqGsw3cwoUXjnrbRmMNCxdeaDiTNCRb5aXeMJxt4ObP34tFiy6m\n0VjT9raDLWeS1MpWeal3DGcbuAMOOIjXve6NDzq6XbZsKUceedj90yeddBqzZs1eax2PbiUNx1Z5\nqXcMZxPA9Okz1huyZs2azezZm4xTRZI2dLbKS71jOJMkPYit8lLvGM4kSUOyVV7qDS9CK0mSVCOG\nM0mSpBoxnEmSJNWI4UySJKlGanlCQEQ8FPgYsDswC7gWODozL+1pYZIkSV1W15az84AtgZ2BhwLf\nB86LiC17WpUkSVKX1S6cRcSmlJayd2TmbZm5gtKKtjHwtJ4WJ0mS1GW169bMzH8Ab26Z/ajq903j\nXI4kSdK4ql04axURmwCfB87LzJ8Pt97mm89kypTJ41dYzU2btvYtV+bMmcWmm84eZm1JGhk/WzSc\nVatX0lizeq15K1etWOc0wMCkyUyZPLWrtXXT3Lmdf//XOpxFxNbAd4DbgNeva90lS5aPS00biqVL\nl601fccdy7jvvtr1YkvawPjZoqHkDT/i5tt/u971rvjVV4eYO8BWD3s8se2unS9sHCxevHRU260r\n1NX2LyoingL8DPgRsFdm3t3jkiRJ0hBGEsyG1xjj9hNPLVvOImIH4GLgw5l5cq/rkSRJw9vqYdtz\n8+3XAo1RbF1azvSA2oWziJgMfAE402AmSVL9xba78uh5T3/QmLOR2NDHnHVD7cIZ8AzK9c12iIh3\ntCz7UmYe2IOaJEnSOkyZPBUMWR1Ru3CWmZcDA72uQ5IkqRdqe0KAJElSPzKcSZIk1YjhTJIkqUYM\nZ5IkSTViOJMkSaoRw5kkSVKNGM4kSZJqxHAmSZJUI4YzSZKkGjGcSZIk1Uhbt2+KiCcDzwW2BLYA\n7gRuBX6QmT/veHWSJEl9Zr3hLCImAW8GPgBsRbnv5XLgLmAzYCbQiIibgeOBBZnZ/m3pJUmStO5u\nzYh4BPAT4FTgYmBvYNPMnJWZW2XmLGBT4KXV8lOBH1fbSZIkqU3razn7BfAD4OWZectQK2TmUuAC\n4IKI+BDwMeDnwCM7WKckSVJfWF84+1Bmnj6SHUXEQBXg3hARB4+9NEmSpP6zzm7N5mAWEX+KiDlD\nrRcRO1FODBjc7oyOVShJktRHRnJCwLOrh9sAz4qIJS2rDAC7U8aeSZIkaQxGcimNLwLzgAbwrSGW\nD1S/v9mpoiRJkvrVesNZZm4TEY8EbgJeBbS2nAEsycyrO12cJElSvxnRRWgz85aIeB7w48xc1eWa\nJEmS+taI7xCQmZdFxLzqLgFzeKA7s3mdz3ayOEmSpH4z4nAWEfsDpwNTh1mlARjOJEmSxqCde2se\nTbnY7MeAxZQwJkmSpA5qJ5w9AtgjM//QrWIkSZL63TovQtviOuAh3SpEkiRJ7bWcvQs4MSIOzczr\nulXQeDviE+f3uoSuWLNqxVrT7//MxUyaMr1H1XTPKe/eu9clSJLUUe2Es08ADwd+ExF3A8taljcy\n05udS5IkjUE74ew31Y8kSZK6pJ3rnO3fzUIkSZLUXssZABHxOGBnytmbn8vMuyJis8y8q+PVSZIk\n9Zl2LkI7k3IT9H0odwdoAN+OiLnA5RHx7MzM7pQpSZLUH9ppOTsReDqwL/A94Ppq/o3AZcBHgJd3\nsjhJ2lBd9c6397qErrhn9eq1pq/5939jxuTJPaqme57yyVN7XYL6WDvXOXslcHBmnpOZfx2cmZkr\ngY8Cz+90cZIkSf2mnXC2CXDtMMv+AcwYezmSJEn9rZ1w9gfKeLOh7Ab8aezlSJIk9bd2xpx9Efh4\nRGwPXEIJdntExDbAocD7Ol+eJElSf2nnOmefqs7YfBcweM2zzwB3Acdn5me6UJ8kSVJfaadbk8w8\nnnLz8x2BXYEdgLmZ+dEu1CZJktR32r4IbWauAn7dhVokSZL6XjsXod0KOA54IrAp5UK0a8nMR3Wu\nNEmSpP7TTsvZF4AAvgXcSblDgCRJkjqonXD2VOCFmfnTbhUjSZLU79o5IeAW4O5uFSJJkqT2wtkx\nlOucbdmtYiRJkvpdO92aPwPeCdwUEX9niFY0TwiQJEkam3bC2bnAlsBpeEKAJElSV7QTznYEnpOZ\nV3WrGEmSpH7XzpizG4EVXapDkiRJtBfO3gWcGBGOK5MkSeqSdro1TwAeCvw+Ipbx4BMCGpn5yI5V\nJkmS1IfaCWe/wpMAJEmSumrE4Swz9+tiHZIkSWI94SwitgN+n5mN6vE6Zeb1nSgqIrYFPg88B9g2\nM2/sxH4lSZLqbn0tZ9cBjwD+BvyO4bs1B6plk8daUETsA5wOXDzWfUmSJG1o1hfODgD+UT3ev8u1\nDNoC2BWYB+w7Ts8pSZJUC+sMZ5n5habJG4CfZOaq1vUi4qGUQDVmmfm5ap/zOrE/SZKkDUk7Z2t+\nH3g4sHiIZY8Azga+2YGaRmXzzWcyZcqYe1W1gZk7d3avS5A0AfnZopHqxntlveEsIhZUDweAUyPi\nnpZVBoAnAvd2uLa2LFmyvJdPrx5ZvHhpr0uQNAH52aKRGu17ZV2hbiQtZ38BdqkePxtYM8Q6S4C3\nt12ZJEmS1rLecJaZHwSIiBuAp2Tm37tdlCRJUr9q5yK023azEEmSJLURziJiY+AI4OnAHMpYs2aN\nzHzmWAuKiAS25oGbsmdENIAvZeaBY92/JElSnbVztuZ/AW+gXJh2MV26z2ZmRjf2K0mStCFoJ5zt\nCeyXmV/sVjGSJEn9btL6V7nfFOCH3SpEkiRJ7YWzi4DndasQSZIktdet+Z/AyRHxOOCnwLLWFTJz\nYacKkyRJ6kfthLPBLs2dW+Y3KGduNgDvnyRJkjQG7YQzuzQlSZK6rJ2L0F7WzUIkSZLUXssZEbEV\ncDila/PhwIuA24BXZ+aXO1+eJElSf2nnDgFPBL5HGVv2MyCAqcA2wJkRgQGtN9asXgmN1WvPW3Xv\nOqcBGJjMpMlTu1maJElqUzstZ/8BfB94Y2beHRFLATLz9xHxLuCdgOFsnP3zL1dwz+Lr1rveHb/9\n5hBzB5gx93FsMu8ZnS9MkiSNSjvXOXsqcGxm3j3EsguBx3WmJLVjJMFseI0xbi9JkjqtnXC2fB3L\nNgNWjrEWjcKMuf/Cg+9BP1ID1faSJKku2unW/CXwsYh4ZVPrWSMipgDvAX7S8eq0XpvMewazHvnk\nB405GxHHnEmSVDvthLNjgUuAWyLip8BGwFmUEwM2AZ7T+fI0EiVgGbIkSZoIRtytmZlXUi6hcS7w\nEODPwGzgW8ATM/OarlQoSZLUR9q6zllmXg+8tUu1SJIk9b12L0L7SuCyzPxbNdbsA8BTgEsz81Pd\nKFCSJKmfjLhbMyIOB74EPKKadSLlRICVwPura51JkiRpDNq5lMZbgcMz81dVq9lbgA9k5kurx/t1\noT5JkqS+0k442xZYWD3ehXKG5jnV9M8pt3GSJEnSGLQTzpYBM6vHuwP/m5m3V9OzgVWdLEySJKkf\ntXNCwFXAMRHxbeBg4JNNy/YFftPJwiRJvXXfmjWsbjTWmrdizep1TgNMHhhg2qR2jv0lNWsnnL0P\n+A7wGuAK4BSAiHgV8G7gxR2vTpLUE5fedSe/vHvZetdb8LfbHjRvANhx41m8YLMtulCZNPGNOJxl\n5q+BrSNibmYublp0OfCEzLy249VJknpiJMFsOI1qe8OZNDrrbHeOiK9HxMbN81qCGZl5a3Mwi4iN\nI+JrnS1TkjSedtp4FgOj3Hag2l7S6Kyv5Wxz4JqIODozv7G+nUXEK4ATKLd2kiRtoF6w2Rbsuslm\nDxpzNhKOOZPGZn3hbHfgo8BXIuIE4H8o3Zh/Bf4BbApsCTwL2JNyuY1Tgfd2q2BJ0vgwYEm9sc5w\nlplrgPdExOeAo4E3AUdQhhQMGgDuAi4A9s7M7FKtkiRJE96ITgioAte+ETEJ2JFyC6fNKKHsr8Cv\nqiAnSZKkMWjrxudVALum+pEkSVKHtRXOImIX4OnAHHjQiTyNzDy6U4VJkiT1oxGHs4h4P3D8OlZp\nUMalSZIkaZTaaTk7BDgN+FDrtc4kSZLUGe2cJ70F8CmDmSRJUve0E86uBh7drUIkSZLUXrfmYcCn\nI2Iq8FPgQTdey8z7OlWYJElSP2onnH0P2Aj4zjrWmTy2ciRJkvpbO+HsNNa+M4AkSZI6bMThLDM/\nONyyiJgBPK0TBUmSJPWzti5COygipjVNDgDPBr4JzOpEUZK6Z8WKe1i5cmXb202dOpXp02d0oSJJ\nUrN2LkK7BXAGsDuw8RCrXNepoiR1x4IFn2XRootoNNofoTAwMInddtuDAw44qAuVSZIGtXMpjY8D\nO1PGnq0CPgOcBdxR/X5Ox6uT1FELF144qmAG0GisYeHCCztckSSpVTvhbHfgTZn5b8B9wCmZeTDw\nGOAJlHtuSqqx+fP3YmCgnT/7BwwMTGL+/L06XJEkqVU7Y84eBtxQPV4FTAfIzH9GxFHA6cD/dLY8\nSZ10wAEH8brXvfFBY86WLVvKkUcedv/0SSedxqxZs9daxzFnkjQ+2glni4HtgFuA2yhdnNdWy/5O\naUGTVHPTp89Yb8iaNWs2s2dvMk4VSZKatRPOvg18LSKeClwMnFSdtXkn5e4BN3a+PEmSpP7STjh7\nLzATuAc4EXge5UQAgCXAaztbmiRJUv9p5yK0dwMHNM3aKSJ2AKYBv8vM5Z0uTpIkqd+M9iK0WwKP\nAH6Tmfd2tiRJkqT+1VY4i4jDgfdTztwEeGxELAXOBfbJzKUdrk+SJKmvjPiCRxHxNuCTlMtlvAFo\nbjF7CHB8Z0uTJEnqP+1cjfJw4KjMPDAzzwVWA2TmYuAo4JVdqE+SJKmvtNOtOQ+4aJhlf6S0no1Z\nRMwE/gPYE9iCci21YzJzUSf2L0mSVGfttJzdTLlN01D+Fbh97OUA5Z6du1BuF/Uw4GzggoiIDu1f\nkiSpttppOVsEnB4Rs4FLgAaweUTsBZwKfHOsxUTE5pTxbK/KzOur2WdExCHAIcCRY30OSZKkOmv3\nIrTbAV+gBLMB4GfV74WUszjH6knA1Gq/zX6GN1aXJEl9YKDRaLS1QXX7pqcBm1DuDHBlZl7diWIi\n4rXAV4AZmbmiaf4JwGsy89HDbbtq1erGlCmTO1GGNKz9Pn9Er0voitX3ruLWb/zu/uktX/k4Jm80\nqssg1tbZ+5/S6xIkqdnAcAtG/OkbEVMot2h6IrBp006fPDgcLDMPGHrrjlhnilyyxBsUSBre4sVe\nhlFSfcydO3vYZe0cGp8B7Af8lnKz8/aa3EZm8KSCOcAtTfMfAtzWheeTJEmqlXbC2T6UrsVvdKsY\n4BeUi9s+nbVPMHgmcEEXn1eSJKkW2gln9wLXdKsQgMz8R0QsAI6LiF9TLt9xKLANcHo3n1uSJKkO\n2rnO2VmUy1l025HApcDlwB3Ay4HdM/PP4/DckiRJPdVOy9lHgEsiIoFfAne3rtCJEwIy817giOpH\nkiSpr7QTzs6ijAX7LeXK/d04IUCSJKmvtRPO9qZcuX/MdwKQJEnS0NoZc7YU+N9uFSJJkqT2wtmp\nwOHdKkSSJEntdWs+DNgzIv4v8CsefEJAIzNf37HKJEmS+lA74eyl1e8G8IQhlnuCgCRJ0hiNOJxl\n5rbdLESSJEntjTmTJElSlxnOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiT\nJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZ\nJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjO\nJEmSasRwJkmSVCOGM0mSpBoZaDQava6hIxYvXjoxXojUA0uX/pMDD9z3/ukzz/wis2dv0sOKJGli\nmzt39sBwy2w5kyRJqhHDmSRJUo0YziRJkmrEcCZJklQjhjNJkqQaMZxJkiTViOFMkiSpRgxnkiRJ\nNWI4kyRJqhHDmSRJUo0YziRJkmrEcCZJklQjhjNJkqQaMZxJkiTViOFMkiSpRgxnkiRJNTKl1wUM\nJSKeDywAyMxteluNJEnS+Kldy1lEnAicBVzb61okSZLGW+3CGbAM2Am4uteFSJIkjbfadWtm5gkA\nEdHrUiRJksbduIaziJgCzBpueWbeNdp9b775TKZMmTzazaW+Nm3amrWm58yZxaabzu5RNZLU38a7\n5ey5wKLhFkbEjMxcMZodL1myfLQ1SX1v6dJla03fcccy7ruvjqMeJGlimDt3+APgcQ1nmXkJMDCe\nzylJkrQh8dBYkiSpRgxnkiRJNVKrszUjYmsgq8kpwOSIGByDdmBmfqk3lUmSJI2PWoWzzPwzML3X\ndUiSJPWK3ZqSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlG\nDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1\nYjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1MqXXBUga\nXytW3MPKlSvXmrds2dJ1TgNMnTqV6dNndLU2SRIMNBqNXtfQEYsXL50YL0TqogULPsuiRRcxmr/7\ngYFJ7LbbHhxwwEFdqEyS+svcubMHhltmt6bURxYuvHBUwQyg0VjDwoUXdrgiSVIrw5nUR+bP34uB\ngdH92Q8MTGL+/L06XJEkqZXdmlKfGWrM2Ug45kySOmdd3ZqeECD1menTZxiyJKnG7NaUJEmqEcOZ\nJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjO\nJEmSamTC3PhckiRpIrDlTJIkqUYMZ5IkSTViOJMkSaoRw5kkSVKNGM4kSZJqxHAmSZJUI4azGoqI\nH0TEl0ew3rER8bXxqGmI574xIo6vHn8gIv7QizrUnohoRMRbOrSv+98D2nBExNYRsSIintfrWgAi\n4tlVPY/qdS0au+pz4QPrWO73xQhM6XUBE0FEzARuATYCtsrMO8fpqXcHPlfV0ADuA7bMzDta6psF\n3AZsnJkDnS4iM48H/JLusYj4AbArsLKadR9wE3AOcGJmelHDCa56D9ycmW8YYtnlwB8ycz9g+jiX\nNqzM/CE1qkdFRPyU8l56ecv87wJPoHzXNJrm/ztwFLB8Xftt/b6IiBcDt2bm1R0sf4Nny1lnvAG4\nFbgO2H88njAiNgWeCixqmn1nVUurVwLLxqMu9dy5mTk9M6cDWwDvBP4dOKK3ZUnawJwPvDAipg7O\nqBoingNsDDyxZf29gIt54OBwpI4Ddh5DnROSLWedcRildeIe4NCI+NTgEUXVonUUcAhwU2a+MCIC\n+BQlXE0DfgW8o+XIYSAiPgHsV02fB7wtM++ppl9AOQr+S9M236GEw1Na6tsPuAC4vztrqCPs5iPr\niJhe1fhSYDPgduBMhmiBiYgPAm/JzK0iYhvgBuAlwNHAjpTQ+irgjcBbKe+7MzPzfcP8e6oDMnMV\ncHFEXA88unV5REyhHMG+BngI5f/41Mw8pWmd3YETgH8B/lotP3Wo54uI91Le67tm5vUdfjnqkKa/\n0d0y85KI2Bn4JLAT5W/z18B7MvPyiHgu8H3gRcCJQAA3Awdn5qXV/h4OnAw8l/KlfT3w3sy8pFp+\nNjAD+DHwLmBz4EfAfpn5t6bneGxm/qFq6f848ApgarXdEZn5x+79q2gI51M+H55F+f+B8r1zK3AN\nJYxdDRARcyjfZ6cCzwCmRMSngddTvuO+BByematbvi9uBh4J/FdEvC0zd4yIGcBHgb2BhwM3Ur53\nvtj9l1wftpyNUUTsCmwPfIHyBpxH6W5s9mZgH2C3avr/UVq55lHefDcA32rZ5sWUD8H/Azyf8kb9\ncNPy+azdagbw38C2EfGkpvoeRTkqOa/Nl/YOyh/lzpk5k9L6dsQQr2047wZeS3l9k4DvAXcBW1X7\nem9E/GubNakNETEjIl5JCWbnDrHKEZQw/wJgNvA24OTBsUgRsQPlfXMS5Qv1DcBHImKoLrM3Au8F\ndjeYbXC+QglADwPmUg7kvhIRk5vWeQ/lQG1Otfz8iNikWnZmtd12lNbai4FvNS2H8hk2F3gc5fNy\n52qfQzmDEhR3onxx3wlcGBF+X42jzPw15bvpRU2z96IEtcuqx4P2ANYAF1XT+1O+nx5G+bw/pGX9\nwefYqnr41szcsXp8BvB0Hvhcej/wuYh49thf1YbDlrOxOwy4ODNvAYiI84BDKR9Qg76bmdc2Te8C\nrBpsBasG9e8bEQ/PzNuqdW5pasH4dXWCwEspR55QwllrV9XdwFcpYfAX1bw3UcJgu92am1H+2O4B\nyMyfV/WNdNzSOZl5I0BELAJenZknV8u+FxGLgcdTjtLVOa+NiFdUj6dRxp0dA1wxxLqnAgsyc0k1\nfWH1//I0ygfwm4HfZuY51fIrI2IfoHVM43zg08CLMvOXHX01Go3m90CzacBQA7E3o7xPVlZ/3ydW\nP5RGfgA+nZk3VPM+BLwd2BP4GqVVfHJmLquWn0P5Qn08cGW1/Srg2MxcA/wlIn5IGbe0lqoF5tXA\nPpl5azXv3ZSutOmsZzyTOu4CSqga/N7ZkxKq/xf4VERsUY2x3hP4YWbeVb1nrsjM86ttLoqIOymh\n/IJ1PVlEbEFpbdszM/9Uzf52RJwPHAj8sHMvrd4MZ2MQEY8AXkb5cBp0JqUraevM/HM1r7U5fhfg\nmIjYnvKBM3hE2Dwo9jct2/ye0opGRDyW0gL1gyHK+hzw3Yg4CrgX2Lf6mTzEuutyGuVo6NaIuIxy\nFHQO8LcRbn9j0+PlwJ9bli+ndHWos84d7Kquui23pxyJPoVyBNtsM8oH7AspLWNQTmoZfB8+lnLk\nfL/MbG2t3YlykHBKZv64Uy9CY3L/e6BZNWxhKO+i/L2/OSIuoXyBXpCZq5vWuW7wQWYuiYi7qD6P\ngB2AE6oW+1lN2zR/nt1QBbNBy4Eth6hlW8pn1f3vu8y8Hfj6MLWru84H3h4R21I+r7cCFlXvgZuA\n3avGhd1Z+6SwP7Xs5x5GdtLHYynfhxdUQ4IGTeKBoN8XbCYem4MpYyK+EBF3VR9Y36D8ux7StN59\ngw8iYjvKG/4K4FHVwO2XDLHvoVqoVlS/5wNXZubS1hUy82eUM/ReRhkDsjIzfzTC13N/gMvMmzJz\np2ofl1PGi/0+Ip48wn2tWc+0uiwzV2XmryhdzK+o3nvNvk4JV88HZlbvxdualq9m/Z8Rz6e0zB4x\nxP61AcjML1O6D99OOaD7HKV1u/mArvXgbgBYU52YtBBYDGyfmRsxRIsYI//7HwyEfjfVww8pw1H2\nqH5+2tS5kvsXAAAGaElEQVTS/t1q3pMpY1bPb9putJ/3g2Oqnz54YlP1My0z+6pb0z+AUarOYDmI\nMmh+Rx4YI7ETZQD1myNioyE23ZnSvfCRpktuPG2op2iZ3o4SumDo8WbNzqI0Db8B+Pww69wDzGx6\nPZMoR62D0xtHxIzMvCozPwI8CfglpRVOG6aNW6afAXw+M6/LzDURsTVljOCg6yljhO4XEXtXp74P\nOiUz96d8UH99mPe8aiwi5mbmssw8LzMPpQzsfjblc23QY5vW34LS6voXyokimwGfbBqSMdTn2Uj9\nidIFev/7LiIeEhHvqp5X4ygzV1KG6LyAciDWPFznYuCF1bLfDHZ7j9GfKAF9rTNBI2Je1RPQN/rq\nxXbYy4CHAidn5k3NCyLiVEq/fGs3EjzQxfmsiFhIaTV7TjVvHg90Bz4qIg6mhKvtKUHr1OoN+jzg\nI+uo7cvAsZQWuycNs87vgNdXX8i3U8aINL8f/hv4e0QckZl/Ax5D6cboyUVvNTrVCSHHAT+jnBXc\n7I/A0yJiGuWkgRMp77951fIzgXdExKGU1pTtgbNZe6zjYEvHWyjh/STKmEttACJiHvCHiNiP0gK6\nhnIi0ArKUITBk3beHhFXU4Y1HEMZw3oxZcD2auCZEfFrymfZ4HWxBt9HI5aZ/4iIrwBHR8RVlJbc\nD1MCwEmjeY0as/MpVwCYAnywaf6llJM89uXBJ7S1YzmwXURsXnWXngUcGxHXUMa2PRP4NqVl95x1\n7GdCseVs9A4DvtMazACqMPMthviSysyrKC1rCyiXJtiTEvQupwycHAxqX+OByxcsrPb3cUprx2rg\n58MVVrXIfRe4enBQ7RA+QRmMfy3lS/oOHjhdGsrlNzYCfhcRyykfxF8GTh/ueVULr41ytfUVEXEP\nZVzidZTB+q1dDYdQvnzvAr5IGTNyMvCaiDi9OutyPuXyJ4Nd9sdn5pdanzQz7wJeBxw4zGB01VB1\nKZ5XU7q+lwB/p/QI7N1yMevPUg7Y7qScvffiqrXtr5QvzfdXyw6nBPWvA2cMdWbvCBxCGV90NeXz\nb2vK+3f1OrdSt1xEGZO6igdONKMaVvMTSivn+UNvOiKnUs4U/201fRRl3OPFwFLKmNljmk5M6gsD\njYYXDZckPVjrNch6XI7UN2w5kyRJqhHDmSRJUo3YrSlJklQjtpxJkiTViOFMkiSpRgxnkiRJNWI4\nkyRJqhHDmSRJUo0YziRJkmrEe2tKmnAiYjPgP4EXU2539mXKLdK+Srkt2vuAHSj3Cv0wcFpmfiAi\nNqfcJu0lwBbArcC5wLGZeV+17xuBKzPzNU3P9xbKvUi3zcwbI+Jsyg3A30a5J+R2wE3AcUPd/kqS\nmtlyJmki+k9KwDqIB+5H++GWdeYCewC7Ap+s5l1QzTuQcs/AYyj3i/z0KGrYknLPyYOAJwI/Bs6O\niKeMYl+S+ogtZ5ImlIiYCbwC+GRmfrWa/Y6IeDLw2KZV5wG7Z+bvqu12AZ4JvCIzL6jW+VNE/Atw\nVES8MzOXtVHKJsC7M/MX1f4PBV4FvBa4apQvT1IfsOVM0kQzD5gKXN0y/39appcPBrPKk6vfP25Z\n76fANODxbdaxormGzLwbSOBRbe5HUp8xnEmaaOZUv1tbuRa3TC9tmd6k+v3PYdab3WYdSzOz9f54\ny4DN29yPpD5jOJM00dxb/Z7ZMn9O64ot/lH93rRl/uD0XdXvBjDQss5QwW3jIebNBu5YTx2S+pzh\nTNJE80dKgHpSy/yXrWe7n1e/n9UyfxdgOXBtNX0X5UzO1nVazYyI+2uIiI2BAK5bTx2S+pwnBEia\nUDJzSUQsAg6JiCsp47wOYT3dkpl5RURcBnwqIpYB1wMvAA4DTsrMe6pVrwLeFBEvoQS2fYDth9jl\nsmpf76O0yr2fMhbuy2N9jZImNlvOJE1E+1MG8n8N+AGl5evkatmKdWz3UuC7lOufXQe8BzgO+EDT\nOscClwLnUILaY6p5re4GTgDOAK6hXNLjjZlpy5mkdRpoNFrHq0rShi0ipgMbZ+YdTfM+ChwFzMjM\n1V1+/rOBPTLz4d18HkkTk92akiaiBcALImI/Stfjk4CDgS90O5hJ0lgZziRNRAcDH6PcUukhwM3A\nZ4EP9bIoSRoJuzUlSZJqxBMCJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk18v8B\nXsuNWSGwQeAAAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f3d282b06a0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot = seaborn.barplot(x='group', y='sentiment', data=name_sentiments, capsize=.1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And finally we can break out the serious statistical machinery, using the [statsmodels](http://www.statsmodels.org/stable/index.html) package, to tell us how big of an effect this is (along with a bunch of other statistics)."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<table class=\"simpletable\">\n",
"<caption>OLS Regression Results</caption>\n",
"<tr>\n",
" <th>Dep. Variable:</th> <td>sentiment</td> <th> R-squared: </th> <td> 0.208</td>\n",
"</tr>\n",
"<tr>\n",
" <th>Model:</th> <td>OLS</td> <th> Adj. R-squared: </th> <td> 0.192</td>\n",
"</tr>\n",
"<tr>\n",
" <th>Method:</th> <td>Least Squares</td> <th> F-statistic: </th> <td> 13.04</td>\n",
"</tr>\n",
"<tr>\n",
" <th>Date:</th> <td>Thu, 13 Jul 2017</td> <th> Prob (F-statistic):</th> <td>1.31e-07</td>\n",
"</tr>\n",
"<tr>\n",
" <th>Time:</th> <td>11:31:17</td> <th> Log-Likelihood: </th> <td> -356.78</td>\n",
"</tr>\n",
"<tr>\n",
" <th>No. Observations:</th> <td> 153</td> <th> AIC: </th> <td> 721.6</td>\n",
"</tr>\n",
"<tr>\n",
" <th>Df Residuals:</th> <td> 149</td> <th> BIC: </th> <td> 733.7</td>\n",
"</tr>\n",
"<tr>\n",
" <th>Df Model:</th> <td> 3</td> <th> </th> <td> </td> \n",
"</tr>\n",
"<tr>\n",
" <th>Covariance Type:</th> <td>nonrobust</td> <th> </th> <td> </td> \n",
"</tr>\n",
"</table>"
],
"text/plain": [
"<class 'statsmodels.iolib.table.SimpleTable'>"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ols_model = statsmodels.formula.api.ols('sentiment ~ group', data=name_sentiments).fit()\n",
"ols_model.summary().tables[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The F-statistic is the ratio of the variation between groups to the variation within groups, which we can take as a measure of overall ethnic bias.\n",
"\n",
"The probability, right below that, is the probability that we would see this high of an F-statistic given the null hypothesis: that is, given data where there was no difference between ethnicities. The probability is very, very low. If this were a paper, we'd get to call the result \"highly statistically significant\".\n",
"\n",
"Out of all these numbers, the F-value is the one we really want to improve. A lower F-value is better."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"13.041597745167659"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ols_model.fvalue"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 7: Trying different data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have the ability to measure prejudicial badness in our word vectors, let's try to improve it. To do so, we'll want to repeat a bunch of things that so far we just ran as individual steps in this Python notebook.\n",
"\n",
"If I were writing good, maintainable code, I wouldn't have been using global variables like `model` and `embeddings`. But writing ad-hoc spaghetti research code let us look at what we were doing at every step and learn from it, so there's something to be said for that. Let's re-use what we can, and at least define a function for redoing some of these steps:"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def retrain_model(new_embs):\n",
" \"\"\"\n",
" Repeat the steps above with a new set of word embeddings.\n",
" \"\"\"\n",
" global model, embeddings, name_sentiments\n",
" embeddings = new_embs\n",
" pos_vectors = embeddings.loc[pos_words].dropna()\n",
" neg_vectors = embeddings.loc[neg_words].dropna()\n",
" vectors = pd.concat([pos_vectors, neg_vectors])\n",
" targets = np.array([1 for entry in pos_vectors.index] + [-1 for entry in neg_vectors.index])\n",
" labels = list(pos_vectors.index) + list(neg_vectors.index)\n",
"\n",
" train_vectors, test_vectors, train_targets, test_targets, train_labels, test_labels = \\\n",
" train_test_split(vectors, targets, labels, test_size=0.1, random_state=0)\n",
" \n",
" model = SGDClassifier(loss='log', random_state=0, n_iter=100)\n",
" model.fit(train_vectors, train_targets)\n",
" \n",
" accuracy = accuracy_score(model.predict(test_vectors), test_targets)\n",
" print(\"Accuracy of sentiment: {:.2%}\".format(accuracy))\n",
" \n",
" name_sentiments = name_sentiment_table()\n",
" ols_model = statsmodels.formula.api.ols('sentiment ~ group', data=name_sentiments).fit()\n",
" print(\"F-value of bias: {:.3f}\".format(ols_model.fvalue))\n",
" print(\"Probability given null hypothesis: {:.3}\".format(ols_model.f_pvalue))\n",
" \n",
" # Show the results on a swarm plot, with a consistent Y-axis\n",
" plot = seaborn.swarmplot(x='group', y='sentiment', data=name_sentiments)\n",
" plot.set_ylim([-10, 10])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Trying word2vec\n",
"\n",
"You may think this is a problem that only GloVe has. If the system weren't trained on all of the Common Crawl (which contains lots of unsavory sites and like 20 copies of Urban Dictionary), maybe it wouldn't have gone bad. What about good old word2vec, trained on Google News?\n",
"\n",
"The most authoritative source for the word2vec data seems to be [this file on Google Drive](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing). Download it and save it as `data/word2vec-googlenews-300.bin.gz`."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy of sentiment: 94.30%\n",
"F-value of bias: 15.573\n",
"Probability given null hypothesis: 7.43e-09\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAnwAAAGCCAYAAAB6nR6iAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4nFed/v/3NPXe3buPW1wSO7bTq1NJARJaKCG0pSxl\nCWXhxy6wLG3hSwkdAiGEQAgbUtj04jjVabbjdlzkJsuSR71rNOX3x8iyRtLIUmLpmRndr+vKJT1l\n7Nu6FOkz5znnc1yRSAQRERERSV1upwOIiIiIyNhSwSciIiKS4lTwiYiIiKQ4FXwiIiIiKU4Fn4iI\niEiKU8EnIiIikuJU8ImIiIikOK/TAYZijJkF/B44F5hlrd3f79q7gJuBeUANcBfwNWttKM6flQX8\nD3AZUARs773/0bH8N4iIiIgkioQb4TPGXAu8ABwY4tq5wG3At4ES4K3ADcBXh/kjbwHOAC4ByoE/\nAPcbY8xJDS4iIiKSoBKu4CM6Cnc2cPsQ1z4F/J+19m/W2m5r7evAD4FPGWMG/VuMMYVEC8L/tNbu\nstZ2WWt/BewAPjZ2/wQRERGRxJFwBZ+19nfW2l1xLq8BNg44txEoJvqId6DTAF+c16x5MzlFRERE\nkkXCFXwnUAo0DDhX1/uxLM79xHnNUPeLiIiIpJyEXLTxBkVO9v3BYCji9XreYBwRERGRceWKdyHZ\nCr5aoo9v+yvp/VgT5356X3N4wGuGuj9GY2PHaPOJiIiIOKK0NDfutWR7pPscg+fenQUcAfYOcf8r\nQPcQrzkT2HDS04mIiIgkoGQb4fsR8LQx5h3AP4BTgH8D/sdaGwEwxvwRqLHWfsFa22yMuRX4ujHm\ndaAK+DgwE/ilE/8AERERkfGWcAWfMcYCMzg++miNMRHgdmvth40x7wS+AfyR6CPbnwA/6PdHTCd2\n5PKzwPeAZ4BcYBNwibV2UJ8/ERERkVTkikRGu9Zh4vD7W/XFERERkaRQWpobd9FGss3hExEREZFR\nUsEnIiIikuJU8ImIiIikOBV8IiIiIilOBZ+IiIhIilPBJyIiIpLiVPCJiIiIpDgVfCIiIiIpTgWf\niIiISIpTwSciIiKS4lTwiYiIiKQ4FXwiIiIiKU4Fn4iIiEiKU8EnIiIikuJU8ImIiIikOBV8IiIi\nIilOBZ+IiIhIilPBJyIiIpLiVPCJiIiIpDgVfCIiIiIpTgWfiIiISIpTwSciIiKS4lTwiYiIiKQ4\nFXwiIiIiKU4Fn4iIiEiKU8EnIiIikuJU8ImIiIikOBV8IiIiIilOBZ+IiIhIilPBJyIiIpLivE4H\nGC1jzDnAI0Nc8gF/tNbeOOD+DwC/B7oH3H+XtfZ9YxJSREREJIEkXcFnrX0ayOh/zhhTAWwF/hDn\nZQestTPHNpmIiIhIYkqVR7q/JDpit97pICIiIiKJJulG+AYyxrwFOBOYM8xtucaYe3rv6wEeAm62\n1jaMQ0QRERERRyV1wWeMcQPfBr5jrW2Jc1sdsB34KXAdsAS4E/gTcPlwf35hYRZer+fkBRYRERFx\ngCsSiTid4Q0zxrwd+A0wyVrbNYrXXQXcC0y31h6Kd5/f35q8XxwRERGZUEpLc13xriX7HL4bgPtG\nU+z12tP7ccpJziMiIiKScJK24DPG5AKXAf84wX0fM8YMbL+ysPfj3rHIJiIiIpJIknkO3wogDXit\n/0ljzOnAH4F11tqDvfd8zxhTDTwFLAb+m2jPPv+4JhYRERFxQDIXfJN7Px4dcD4LMEQLPay1PzHG\n+ICfAdOBRuA24BvjlFNERETEUUm9aGOsadGGiIiIJItUXrQhIiIiIieggk9EREQkxangExEREUlx\nKvhEREREUpwKPhEREZEUp4JPREREJMUlcx8+ERFJAeFAgNYXnyfY3EzuytNJq6hwOpJIylEfvmGo\nD5+IyNiKRCIc+s636Nob3eLc5fMx7QtfJmPWbIeTiSQf9eETEZGE1LV3T1+xBxDp6aHpyccdTCSS\nmlTwiYiIc9xD/Boa6pyIvCn6v0pERByTOXsOmQsW9h270tMpvPBiBxOJpCbN4RuG5vCJiIy9SDBI\n26uvEGxuIufUlfiKi52OJJKUhpvDp4JvGCr4REREJFlo0YaIiIjIBKaCT0RERCTFqeATERERSXEq\n+ERERERSnAo+ERERkRSngk9EREQkxangExEREUlxKvhEREREUpwKPhEREZEUp4JPREREJMWp4BMR\nERFJcSr4RERERFKc1+kAIiIysUXCYdq3biHU3Ez2shV48/KcjiSSclTwiYiIo6pv+THtWzYD4M76\nK9O+9BXSJ09xOJVIatEjXRERcUzX/n19xR5AuKODpscecTCRSGpSwSciIo4JBwIjOicib44KPhER\ncUzm3HmkT59x/ITHQ8F5FzgXSCRFuSKRiNMZRs0Ysx+YAoQGXFpqrd01xP0XA18HFgHNwIPA56y1\nHcP9PX5/a/J9cUREkkyos5OWZzcQbGoib/Ua0qdNdzqSSFIqLc11xbuWzIs2Pmyt/cOJbjLGzAPu\nB24GfgeUA3cDPwNuHMuAIiJyYp7MTAovWud0DJGUNhEe6X4U2Gmt/am1tsNauw/4BnCDMabE4Wwi\nIiIiYy6ZC77rjTHbjTHNxphXjDFXx7lvDbBxwLmNREc3TxvThCIiIiIJIFkf6W4B9gDvB7qBTwH3\nGGPOsNa+MODeUqBhwLm63o9lw/0lhYVZeL2ekxBXRERExDlJWfBZa68acOpbxpi3AR8GBhZ8wxl2\nUUZj47BrOkREREQSRmlpbtxryfxId6A9RFfuDlQLFA84d2zuXs2YJhIRERFJAElX8BljZhljfmaM\nKRhwaSHRom+g54jO4+vvLKKPgl8ag4giIiIiCSXp+vAZY7KAXcDTROfudQH/BnwFWArkA38E1llr\nDxpjZgLbgK8CvwSmA/cAT1hrPznc36U+fCIiIpIshuvDl3QjfL3Nki8GcoCdgB9YB5xrrbVAFmCA\ntN779wOXAe8EGoH1wEPA58Y7u4iIiIgTkm6EbzxphE9ERESSRUqN8ImIiIjI6KjgExEREUlxKvhE\nREREUpwKPhEREZEUp4JPREREJMWp4BMRERFJcSr4RERERFKcCj4RERGRFKeCT0RERCTFqeATERGR\npBIKhdFOYaPjdTqAiIiIyEh0d/Xw+P07ObC3nty8dM65dD7TZxc7HSspaIRPREREksLGp/dzYG89\nAK0t3Tx67w56ekIOp0oOKvhEREQkKRytaYk5DnQHaWnsdChNclHBJyIiIklhyvSCmOPMbB8FxVkO\npUkumsMnIiIiSWHlmTPp6gxSaf3kF2Zy1sXz8Hg0djUSLq1yic/vb9UXR0RERJJCaWmuK941lcUi\nIiIiKU4Fn4iIOKrr4AGan91AT53f6SgiKUtz+ERExDEN//cAdf97d/TA42Hyv3ySnOUrnA0lkoI0\nwiciIo4I9wSo/+f9x0+EQtTf9w/nAomkMBV8IiLijFCYSE9PzKlgc7NDYURSmwo+ERFxhDsjg9zV\na2LOhZqbaNv0mkOJRFKXCj4REXFM3pozBp1reeF5B5KIpDYVfCIi4hhvYeHgcwUFQ9wpIm+GVumK\niIhj0idPoeCCC2l64nEAfKWlFK671OFUkqgC3UHWP7SLyl1+8gsyOXvdPKbMGPymQQbTThvD0E4b\nIiLjI3CkmmBzM5lz5+HyaixChvbs43vY8lJV33FGppf3fmItXq/HwVSJQzttiIhIQkubNJmsBQtV\n7Mmwag7HruLu6gzS3NDpUJrkooJPREREkkLFlPyY44xMH/lFmQ6lSS56KyUiIiJJYdVZM+loD7DP\n+skrzOScdfP1OHeENIdvGJrDJyIikhiCPSE62gPkFWhEL57h5vAl5QifMaYM+C5wCZADbAe+Yq19\nfIh7PwD8HugecOkua+37xjiqiIiIvEl7dhxl/UOWQHeIotJsLnvbEhV+o5SUBR9wL9ACnAo0Af8B\n3GuMmW+trR7i/gPW2pnjmE9EREROgp6eUF+xB9Dgb+fF9fu4+OpFDidLLkm3aMMYk090RO8z1toa\na20X0dG+bGC1o+FERETkpOpoC/QVe8fs2XGUO3/9IocPNDqUKvkk3QiftbYZuGnA6dm9Hw/FeVmu\nMeYe4EygB3gIuNla2zA2KUVERORkyCvIoLAki8a6jpjzTQ2dPPKPberDN0JJV/ANZIzJIzpH715r\n7ctD3FJHdETwp8B1wBLgTuBPwOXD/dmFhVn6JupnT1UTdzy0k6bWLs5fOY2rzp7jdCQRSVKRUIiO\nQ4dILyvHm6W5WDK8Gz6yhsf/uZNd22sI9oT7znd1BnFH3JSW5jqYLjkk9SpdY8wM4AGgBrjGWts+\nwtddRXQe4HRrbbxRQa3S7acrEOTmnz9He1ew79xNVyzkzFMmOZhKRJJR9+HDHP7xDwk21ONKz6Di\n/TeSNmkywZZmMucb3D6f0xElQT33+B42x+y04eN9n1iLx5t0M9TGRMqt0gUwxqwiWuz9Hfi0tbZn\nFC/f0/txCvEfA0s/ew43xxR7AJt216ngE5FR8//trwQb6gGIdHdRc+tviASjP1+8RcVM++K/4ysu\ndjKiJKhVZ8+ktaWL/bvryclL5/zLF6jYG6Gk/CoZY5YQnYf3bWvtx4cr9owxHzPGDGy/srD3496x\nyphqKgqzGPi2oaI4y5EsIpLcevxHY46PFXsAwYZ6Gh9+cLwjSZJoqOvgYGUD4XCElqYuKq3f6UhJ\nI+lG+IwxHuA24DfW2h8Ncf104I/AOmvtQSAN+J4xphp4ClgM/DfwR2v1nTJSJQWZXHf+XO7ZUElP\nMMz8aQVcunq607FEJAnlrDiVxof+L+71UGvLOKaRRNfd1cOGR3dzaF8jRCIxc/i2vnqYFWumk52b\n7mDC5JB0BR+wlmj/vSXGmM8MuHY7cAdgiBZ6WGt/YozxAT8DpgONRAvGb4xb4hRx6erpnLNsMh3d\nPZTka5K1iLwxJde8FXd6Ou2vbyFt0mQ6dm4nWF/fdz3vjLMcTCeJ5tnH97J729Ehr0Ui0T59cmJJ\nvWhjrGnRhojI2Av4azn6p9sJtbSQf+HFFJx1ttORJIHc/vPnaWsZuFlWVE5eOu/9+NpxTpS4hlu0\nMeI5fMaYJ4wxBXGuLTHGvPRGwomIyMQViUSovfV3dGzbSvehg/j/fDtdB/Y7HUsSSGlF/JYrbS3d\ndLQNXQxKrBM+0jXGHJuodR4w2xhTN+AWF3AucMrJjSYiIqmuq3Ivnbt39R1HAgGaHn+Mig9+yMFU\nkkjOumgunR0Baqpa8HjdhILhmOuBQAgtITyxkczh2wX4gAgQbxTPBTx9skKJSHILR8I8cuBJNvm3\nUpJZzFWzL6Usq8TpWJIsXHGfSskElJOXwbU3nEpPIETlLj9PPLCz71rFlDwKilTujcRICr48YBWw\nAfg3YKjlU41E26SIiPDYgfXcX/kwAIdaD1PVepivrbkZtyspO0HJGOlpaKD5+WdxZ2cTbo/2zXel\npVFw4UUOJ5NE5Evz4HYPeDOgNwcjdsKCz1obAJ41xtwI/MVaq4flIjKsLXXbY479nfXUtB9lck6F\nQ4kk0UTCYap+8F16amv7zuWevpqSa9+Or7TUwWSSyLZvOhJzXFPVTFNDh0b5RmDEbVmstbcZY7KM\nMacCxTCoDy/W2kdOZjgRSU7lWaXsaznQd+xz+yjMyHcwkSSarv37Yoo9gMCRatzZ2Q4lkmSQlh67\nv73LFR35kxMbzSrdK4AqovP4Hib6CPch4MF+H0VEuHL2ur45e163l+vnX02mV70b5ThvQcGgx3Hd\nhw6x/ytfJFBb41AqSXSnnTEjpsA7ZeVUsnPUdHkkRtN4+bvANuAngJ/oIg4RkUFqO/zUdzYCEAwH\n2dVYyRmTT3c4lSQSX1ExRVdeRcMD90W75/YKtbbS+MjDlL/3/Q6mk0RVNimP93xsNVX7G8kvzKRs\nUp7TkZLGaAq+2cBKa+32E94pIhPa4wefJhQ53v3+pdpXuWrOJRRlFDqYShJNydXXkjZpMjW//kXM\n+XBXp0OJJBE1N3ay4ZFd+GvamDqzgLPXzWfeonKnYyWd0RR8+4i2Z5EJJBKJ0NkdIisjGXfhE6eE\nI+FB57Srjwwld+UqGh64j0D14egJl4v8s891NpQklEfv3Ya/pg2APTv8dHX2UFKeS35hJmZJBR6v\nVv+PxGh+i/8H8N/GmBustY1jFUgSx4GaVn513zZqGjqYWprDe9fN5/V99TS3BVi7uIIFMzRaI4OF\nI2Fm5c/ANu4h0jvzY3npEoozixxOJonI5XYz7QtfpumpJwg2N5G3ei2Zc+c5HUsSRKA72FfsHVO1\nv4mq/U0A7N9dx+XXLXUiWtIZ8V66xpi/AcuAqcAeoG3ALRFr7ZknN56zJvpeul/73YtU+dv7jn1e\nNz29Hc5dwOfesZzFs/RLXGL95vXb2eR/HQCPy8NlMy9k3Yzz8bi1kk5ERicSiXDnbzbS3BD/Mf+7\nP7qa/EItCoPh99IdzQhfCXC49z9JcaFwOKbYA/qKPYiu2NmwpVoFn8SobqvpK/YAQpEQNR1HVexJ\nn3BPgMZHHqarci+Z8+ZTeNE6XN7YX0WhjnYaH3qQ7urDZJ+yjPxzzsWlBrspbd/uOp5/Yi/tbd3M\nX1JBaUUur79chQuYv7ic3dtqaWroJC3dQ6A7FPNaj+f490Z7azdHqpoprcghv1C9+fobTR++88cy\niCQWj9uNmVaAPdQU956cTE3plFj9F2ocs7NhN4daq5mWO9mBRJJojt7+R1qeewaA9s2b6NixnWBj\nA+GuLgrOu4Ciy6+k+ue30LlzR/SeTa8R7uig6LLLnYwtY6izI8Cj927v2yN3+2vVMdfr/e0sO30a\nk7uDFBRn8eLT+/ruLanI4R93bCIj08fMecW88twBwqHow7mz181jyalTxvcfk8BGPRPfGJMLnAJM\nAh6y1rYbYzzW2sE/6SWpffgti7jj0V1UVrdgphcQDsPL9igA+TlprDt9usMJJdFMy53CnPxZ7G3e\n13euraedX275Pd9Y+yWN9E1wkUiElhefjznXsW1r3+d1/3s3webmvmLvmJYXnlPBl8L8NW19BVw8\nmzce6vv89HNm0tzYRSgYZs+O6O+k1uYu/DWtMa/Z+PQ+Fi2fhNutRR0wioLPGOMh2ovvk0Aa0ad6\n84wxPcAjxpgLrbXVw/0ZklyK8jL41NtiJ8PurW6mpS3AoplFpKu7uQzhk8tv4r9e/AH1XcfXdjV1\nN1PTcZQpOZMcTCZOc7lcePPzCTY0xL2n6fFHB53zFhSMZSxxWGlFDh6Pi1BoZNPmX9qwn5EsP+gJ\nhAiHI6jeixrNl+FrwIeAbwJnAsdmUDYBR4BvndxokojmTM5nxfxSFXsSV5onjYVF82POZXjSKc7Q\nfE+B0ne8+/icvRHMy3NnZ1Ny7dvGOJU4KTMrjYuuWkRufgYej4sFSytYvGIybrcLt2fw98hIOzzN\nW1SG16vfVceM5pHu+4F/sdbeCWCMiQBYa1uNMV8B7huDfCKShK6cfQlHO+rY1bSX3LQc3mneSoZX\n2x8J5J62krbNa2h97pnjv7ndbggPfqSXMWs2Uz//Rdzp+t5JdbNNKbNNacy5tefPBlxsfHofW16u\nAqLvEQYWfNH9dL2cfs5M0tK8HD7QSGlFLotWaN5wf6Mp+MqAF+JcOwJofxMRASA3LYdPn/pROno6\nSfekae6exGh7eWPsiSGKPVwuiq++RsXeBOZLi5YoZ140l5LyHF55dj+trd1E+j36nTy9gMvfvgS3\nx43HE31oaU6pcCRvohtNwbcfOIfojhsDrQWqTkYgEUkdWT71xpLBvHn59NT54173lZVT8aGPkjl7\n9jimkkRUW91CS1MnLz2zn9bmrr7zhcVZfY9+jxWGMrzRfJXuAX5mjJkOPEa09+4SY8ylwNeBW8Yg\nn4iIpJiS695BzW9+SSQYBK8Xl9tNJBDou1506eUq9oQNj+xi66tDrwVtaeqktrqV4vIWXn32ALVH\nWpk0NZ/zLzfk5GWMc9LkMJqdNnzAL4APEC32js2kDAG/Az5urR1+XXWSmeg7bQwU6AnxwvZamtsD\nrFpQRkWRmlqKyBsTbG6m68A+MmbMoqfOT909fyfU3Eze2jMovOwKNVqe4Fqbu/jTL+LNIjtu4Ore\nabMKufIdy8YyWkIbbqeNERd8xxhjJgEric7ZawRetra3OVuKUcF3XCQS4dt3vMqeqmYgus3al95z\nKrMmaeqmiIicXPX+Nu763cujfp3X5+bD/3bOGCRKDidrazUArLVHgPvfVCJJOnurW/qKPYhus/bE\nK1XcdOUiB1NJogqEevj77vvY5N9KaWYxb59/FTPz1KhbREamuDSH8il51B5uGdXryidrECKe0TRe\nzgM+DawA8jn+SLePtfaCkxdNEslQT1dcbj1ykaE9uP8xnql+EYjutPGrLbfxzTO+jNetydUiMjJX\nXr+Upx607N0Zf4GP2wPFpbn4a6Jz+M67zIxjwuQymp++twEXA08Ah4jutCETxJzJ+SycUciOA9Hd\nE9J9Hi46barDqSRR7WrcG3PcEmiltsOvnTZkSN2HDlF/3z8ItjSTt+YMCs7X2IFAWrqX1efOptL6\n4zZbDofg/MsNxWU54xsuCY2m4LsAuNpa+/hYhZHE9tnrl/GK9dPc1s1ppozifK2EkqFNz53K/paD\nfceZ3gxKM4sdTCSJKtzdTdUPvkeoLboPatfePbgzMshbe4bDySQR5Bdmct5lhhfWVxLoDlFUkoW/\npq3velZOGgXFWkA4EqMp+BqBw2MVRBKf1+Nm9aJyp2NIErhy9jrquxrYXm8pSM/nneZa0jxpTseS\nBNS5Z3dfsXdM26ZXVfAJAE0NHezfXU9Gpo8FSytYsWYGG9dXUmnryCvM4IwL5vY1XJbhjabg+x/g\nq8aYD1lru054t6S8jq4eGlq6mVyajdvl4mhTJ92BENM0tD7hZfuy+PiyD9IT6sHr9qrFhsTlKysb\ntF+Wr0xvLCXaHeKfd22hpSlacjTWdeDxeDh73XzSM3xsfukQ9/9lMytWT+O0M2c6GzYJjKbg+zPw\nLuCIMWYX0D7whvFatGGMySJagF4GFAHbga9Zax+Nc/9pwHeJLjjpBtYDn7LW1o1H3mQX6AmR5ovd\nGmvD5mrueHQXgWCYsoJMZlTk8tLOaHeeuVPy+ez1y8hM1wT9ic7n8TkdQRJcWmkZRVddQ8P990I4\nTPq06RRdcpnTsSQBNNV39BV7x2x56RAN/jYq+/363rhhP+VT8pk6s3C8IyaV0RZ8S4EngQacXbRx\nC3AqcAlwEHg/cL8xZpm11va/0RhTBDxEtDn024AM4A/A34DzxzFz0jlY28pvHtjOYX87sybl8u6L\n5vHa7nrqWzp5dVcdPcFon+2jTZ0cberse92ew82s31TNpavVhiNV1XU2cH/lQ/g761leuoSLpp+L\n2zX6xypHO/ykedIoSM8fg5SSDCKRCJ3bt/XtpxuoraGnrg5Pjp4UTHQ5een40jz0BEJ95wLdoZhi\n7xh/TasKvhMYTcF3NvAWa+0TYxVmJIwxhcANwPXW2l29p39ljPkY8DHgswNe8m6iLWS+aq0NAs3G\nmC8Bm3oLxM3jlT3Z/PaBHRz2Rwdy9x1p5ft3biIQHNlmKnXNnSe+SZJSOBLmZ5t/y9GO6A/dAy2H\nONByiObuFnLSsrl85sVMzxt6BXckEqEz2InH7eWXW/7ArsY9uHBxztS1XD//mvH8Z0iC6KrcS+fu\nXX3HkUCApiceo+KDH3IwlSQCX5qXcy+dz9MP7ybQHRz23knT9KbxREZT8B0Fht7UbnydBviAjQPO\nbwTWDHH/GuDV3mLvmC1AV+81FXxDCIbCVPnbYs4NV+x5PS6CvdvbuIBVC8rGMp44qKb9aF+xd8wm\n/9a+z/c07eebZ3yZDE86Gw6/wGb/VsqySlhUbLh79/3UddaTn5ZHcyDaUDVChPVVz7GyfDmz82eO\n5z9FEpXmfEqveYvKmTWvhIf+vpVD+xsHXfeleVh7/mwqpqjgO5HRFHzfAP6jd9HGoPl746i092PD\ngPN1wFBVRunAe621EWNMQ5z7heiK3LlT82N21xhoUnEW2Rk+ls8rYdGMQh7aeJCuQIjzlk/BTNfQ\neqoqSM/H5/bSEx76HXdnsJOd9btoDDTz993RTXl2Nu7mueqNBCPRRzPHir3+/B31KvgmoMw5c8k0\nC+i0OwFw+Xzkn3ues6EkoXh9HtZeMAf/XzbT1dETcy3YE2Lm3BKHkiWX0RR8VxGdw1drjNnL4EUb\nEWvtmSct2Rsz2nmFw95fWJiF1+sZ7paU9qX3n87P797M7kONLJlTgsfj4plN0UHeorwMvvmxMykv\nOt7/aNXSKU5FlXGVy42nvoM/vHYXgVAPuek5tHbHjgbftfsfFGTGvuM+VuwNJd2bzlnzT6UgI3dM\nEktiK/7Wf3Lwzr9y5IF/Eu4OUPOzn7Dgy18gb+ECp6NJgigtzWXO/1fKrT99ltrq428YIxHIzc2g\nuFRzPk9kNAVfAdEFEgdPdOMYq+39WExsX8ASoCbO/THt/Y0xLqKre4e6v09jY8cbT5kC3MAnr10S\nc+78ZZNpbg+waEYh7lAIv7916BdLSluWt4z/PnM+zd0t5KXl8vPNt7KvX6PllkAbHT2x8zhdxL7D\nmp0/E6/LQ4Y3g3Uzzqen1YW/Vd9PE0nzMxuov+8fhAPdEIkQ7g4A0NPczK6f/5ppX/x3XGlpausz\nAVVaP7u21ZKVk8aK1dPJ7W30v3TVVB69d3vffVNmFBAmot9FvUpL479pHnHBZ61NlBWtrxBtrbIG\n+Hu/82cC9w9x/3PAt4wxPmvtsbHglURX6z4zlkFTTXNbd3SnjfZu0rxuFs0scjqSOCjTm0mmNxOA\ndy94O9/a+MOY6wNH9C6efj5HOmo42HKY+YVzuW7+VWT71CF/ouquPkztbbcSb8+s7qpD7PnER/GV\nllJ+44fImq89UieKfbv8PHzPtr7jA3vqOf8yAy6YbUp4yzuXUbnLT35BJmnpXv76u5eIRCIsWzWN\nhcu0fWMc21DjAAAgAElEQVQ8rki8DeoAY0yatTZw7PMT/WHH7h1rxpifA+cAbwWqgI8D/wksBoLA\n48CN1trnjTH5wE7gDqLzEPN6P2+11l453N/j97dqv+BeoXCYr/7mRWobo6M2LuCz71jGklnaLkui\nq2+/89KPqWqLv67r40s/yOKSBdR1NvDM4RcIRUKcOXk1FdmaSjsRNa1/iqO3/2FE93oLC5n13R/g\ncmtHhYng4Xu2Dtl6BaC4NJvTz51F1b5G3B43mzceirl+zXuWM2lawXjETEilpblxh8NP9H9PpzHm\n2E/jLqDzBP+Nl88SLeqeAeqJ9te7xFp7gOgKXgNkA1hrm4GLifbtqyG6QreSaLsWGaFdh5r7ij2I\nPpp7ZssR5wJJQnG5XHxi2U1Mz52KCxfuIX60/HzLrfzXCz/guy/9mEcPPsUThzbwvZd/Ql3nwPVX\nMhEEakb+8yPY2EiwOf4CMkkt2Tnpca/V+9t58O6tvP7K4UHFHjDkSl6JOtEj3W8Abf0+T4gRL2tt\nN/Dp3v8GXttPdACq/7mtwLjsApJqWjsCVPnb8XkH/wLPzdTeqHLc7qa9HGytAqKtVgByfNm093T0\nHR/pqI15TXcowMu1r3HpzAvHN6w4qm3LJpoefXjE96dVTMJbMHFHbSaa5auncWBv/aBdNkaiRFt7\nxjVswWet/Xq/w1uBKmvtoGZsvY9NtZwqxWzcUctvH9hBMBQmI83DzIpc9tdEJ8ZmpHm4+PShm+vK\nxLS/ZfC77baeE3dwyvBkjEUcSWCdO3YMf4PLRfay5XTt3UPalKmUvfsGLdyYQHLyMnjXR06npqoF\nl9vFg3e/TndX/MbLbnf0e2PxisnMmq8WLfGMZpXuPqAC8A9xbTrR7cvUfC1FhCMR7nxsN8FQtL7v\nCoQ4UHN8FVRXIMTrexu48DRNupeoeQWzeeLQhhPe17/pckV2OasnnTrW0STBpM+cOez1zIWLmPLJ\nQQ9wZAJxu91Mnh4d1V151gyefWxv3HvD4Qi+NA9zF5bpjcEwTljwGWO+1vupC/i8MWbgW3YXsJYT\nzweUJBIKRWjpiF2DM/B5/o4DjVx4mkb5JGpp6WKunn0ZT1U9Q3tPJ8FI7Dvy0swSVpYvZ93089jV\ntJdwJMyiYoPXPZr3nZIKcletpmvvHpqfXg9uN5FA7M8aX7EWg8lxQ60tdXuihV24d4ennkCIF56q\n5JobVoxntKQykp+0s4EziP6+vznOPa3AN09WKHGez+tmpSnjpZ1H494zTXMlZIB1M89n3czzeWT/\nk9xb+WDf+ZLMYr625vO4XdH3hUtKFjoVURKAy+2m7N3vpeS6d+Jyu6m+5ce0v74FAHdmJoUXXeJw\nQkkks+aVsHH9PoL9tvc8Vuj1197WPZ6xks6wbVn6M8aEgakcb3zcx1obv4V+EpvobVm6e0I8+MIB\n9te0MmtSLvc/u59wv6/I286dzRVrZzqWTxJXOBLm77vv57Wjr1OUUcgNC69T+xWJKxIO075lM8Hm\nZnKWr8Cbr31RJVZtdQvPPLqbo0fiN1g+7cwZnH72rHFMlXiGa8symsbLemQ7waT7PFxz9mwAXq+s\njyn2APYN8z+eTGy2cQ9PH36ecCRMc6CFB/c/xo2L1QlJYgX8R/Hm5eNOTydnuR7FSXwuFyw7fVrM\nLhsDzZijzQCGM+KCr3c7sncR3eGimAGtT4jupfuek5hNEsiM8ly8HhfBfsPocybnOZhIEtnjB58m\nHDn++OXl2k1cNfsyijO1rkugp76Owz/5EYHDVbgzMym74f1kLzmFUEc7aaUaCZbjenpC/POvWzhS\nFe3DWFiSRUtjJ6EhHuk21nVQPlmjw/GMZrb0/wP+FWgn2ux44Fd7Qj/+THV52Wl85C2L+esTu2lu\nD7B2cQUXr5rmdCxJKvoRIVF1f7+bwOFoz8ZwZye1v/8tkUgEQiEy5s5jyqc+gyc72+GUkgh2ba3t\nK/YgWtQNxe12MXVmIS1NnTT425k0LZ/0DN94xUwKoyn4rgf+HfiutVY/uSeglQvKWLlA777lxKbl\nTGZHw66+4xxfDsWZetwiUQN32YgEj6/o7tqzm8aHH6TkrW8f71iSgNpbR7oQI8ILT1Wye3t0oaEv\nzcPl153C5Am8zdpAo5mXlwP8RcWeiJzIvpaDMcdtPW0c7RiqhadMRNnLlg97PVBbM05JJNHNWVAa\n01svXpu9cJi+Yg+ibVo2Pr1vrOMlldGM8G0guh/t/rGJIiKpIsMbu3uGCxfpnvj7Y8rEUnzFWyAS\noW3Ta6SVltG+fRuR7uPbaGkBhxwTDIbp301khI1FAOjs6BmDRMlrNAXf54FfGWPKgRc5vsduH2vt\nrkGvEpEJZ07+TF6vO76arjCjgPx0LfKRKJfXS8k1b6XkmrcS7u5m7+c/E3uDniNJr3276kZ8b0am\nj67O40XeglMqxiJS0hpNwbet9+NZxP/f0fPm4kgi21pZz11P7qWlvZu1Syq47ry5fXsYivTXf/4e\nQENXI/6OekqztIOCxOrcs5tIZ2fMubbNr5F3xpkOJZJEkld44r22K6bkUT4lj6WrpmJfr6XB38a0\nWUUsWDppHBImj9EUfB9E77smrLbOHm6553UCPdFWGw9vPERRXgYXr9RKXYna3VjJjoZdTMmpGLRd\nmgsXPo+2UJPB0srKoxOz+j2rSyvXyIxEmcUV7N9Vz4G99QBMmV5A9aGmvm+XOQtKWXfN4r77Tztj\nhhMxk8JoGi//YQxzSILbd6Slr9g7xh5sUsEnADxf/RJ/2vm3vuNTihfic3vpCUdXX0aIcNu2v3DT\nkhvISVO7DTnOV1pKyVuvo/7e/yUSDJI5bz6Fl1zmdCxJEB6vm8WnTqbqQAOhYISa6hbWXjCHYE+Y\n/MJMZpsSpyMmjVG95TbGZAHvI7p4owL4KNGt1s6y1j598uNJopheloPH7SLUb7uNWZNyHUwkieSJ\nQxtijrfW7+TMSat4vuYVQpHozou7mvZy/76HeZd5qxMRJYEVXXY5+eeeS7i9A19pqdNxJME8/8Re\nQsHo755QMMzLzxygqDSb/IIMyifnkZt/4se+Moq2LMaY2cAO4BbgHOAyIBOYDzxmjNFbshSWn5PO\nTVcsJD87DbfLxZrF5axT42Xp5XHHTt+NEOGZIxv7ir1jDrdWj2csSSKerGwVezKk9rbYXnyB7iA1\nVc3YrbU8cNeWmFW8Et9oRvh+CBwEzrbWHjTGtAJYa3caY/4L+Crw4BhklASxZnEFqxeVEwpH8Hq0\ntbIct27G+dy69Q4iJ5jmu6Bo3jglEpFUMW9ROdteG/rNYlN9Bw/8dQtp6V6WrpzCJDVajms0Bd95\nwMXW2oNDXLsT+OJJSSQJzeVy4fVoZa7EOrVsKRWnl/Fs9Ys8VfVszDU3LrJ92aysWM6lMy90KKGI\nJKuW5uOruAes7wGgan8jAPv31HH9B1dRWJw1nvGSxmiGacJAS5xraWgFr8iENjmngrfPu4r5BXNi\nzoeJEI6EuWr2ZYNW74qIDKfmcDOHKhv7jod7ehsORdi3Szv6xDOagm8r0b10+zv2pf8Q8NpJSSQi\nScvlcvGJ5TdRmhnbb6892IG/c+QNVEVEAIIDukOcSE6eFnDEM5q3298B7jPGnAE8AfiArxtjFgDL\niS7ikBRT19zJoaNtzJ2ST25WGi3tAdo6e5hcotYaEquqtZonDz1DKBJiet5U/J31fddyfNmUZWlC\nvsQXCYWIBIO407UFnxw3eXo+hSVZNNZ1nPDe6bOLmLNAP2ficY1mdYsx5lzgS8DpQB7QCDwPfM9a\n++xwr01Gfn/rhH5M/dSmw9z+sCUSgTSvm5WmjBd31BIKR5hZkctnr19Gblaa0zElATR1N/ONF75P\ndygAgBs3OWnZtARa8bo8XDP3cs6fdrbDKSVRNT/7DP6//YVwezs5p62k4sYPqfCTPt1dPWzfdITO\njh4ys3y88FRl37Vpswo544K5RIhQXJrjYMrEUFqaG3eS/Wgn1HQC77bWNgIYY24EVgLlbzyeJKJg\nKMzfn9rbN18iEAzz3Laavuv7a1p5aONBrjtvrkMJJZFs9m/rK/YAwoRpCbQCEIyEeHD/45w1eQ0+\nj8+piJKggk2N1P7x9xCKtvBpe/klGqdOo/jKqxxOJokiPcPHijXT+47LJ+exb1cd+YWZLFhagden\nXV1HYsQFnzHmHcAdwFrgJWPMF4BvEd1j90ZjzMe1G0fqCIUidHQHh73naGPnsNdl4shPG74Jd3tP\nB89Wv0hJZjELi+YP6tsnE0+orY3aP/2R9q1b+oq9Y7oPHHAolSSDydMLmDxd7VdGazSLNr4A/H/W\n2peMMS7gc8B3rLXLgY8DnxqLgOKM9DQPqxaUxZxL88Z+u5xmNFdCok4pWcSMvOONuDO9sROn3bj5\n2+77+MWW3/P9l39KoN9ooExMR/98O20vbyTS1TXoWtbChQ4kEklto3mka4C/9H6+CigFftt7/ATw\n45OYSxLATVcsJC8rjT2Hm1kxv5QVc4v5xzP7aWrt5pzlk1mzSBucS1RLoJXDbUf6jjuDXeT4smnr\naceNizDHV9odaqvmlaNbWDtppRNRJUF07Nwx+KTLRfqMmeSdfe74BxJJcaMp+AIcHxG8CNhtrT02\n7p7G6EYLJQk8/sphHnulCoADta3UNnSwdV89gZ4w6WkeVpoysjLUV01gR8MuguHYKQBtPe1AtA/f\nQF3BwaM6MrFkzJxF+5bNsScjEbr376N5/ZMUXrTOmWCS8FqaOjlY2UBeQSbTZhXicmkzgJEYTZG2\nBfiEMWYV8C/APf2uXQXsOpnBxFmhcJj7n9vXdxyJwHNbawj09kTacaCRhzYOtemKTESjabmS5c1k\nRdkpY5hGkkHZe95LxpyhF311WjvOaSRZVB9s4i+/2ciGR3bzz7u28NSD+l4ZqdEUfP9BtMHyC0RX\n6/4PgDHmSuC7wP876enEMeEw9ASHb3h5pK59nNJIoptbMIsLpp2N2xX9kVIyoPFyljeT86eexUXT\nz+XmlZ+iID3fiZiSQHzFJUz/8leZ+e3vgyd2EU/GrFkOpZJE99qLBwmFjj812LmlhrYWPTEYiRE/\nj7PWrjfGTCU6l2+btfZYF8SdwJXW2ofHIuBQjDEriBaZK4EQ8CzwOWttZZz7/wC8F+gZcOmT1trf\nDn6F+Lxuzl42mSdfPXz8nMdNT+h4Ebh0TvFQL5UJ6m3z3sIlMy6gJ9xDli+Lv9p72OR/ndLMEq6f\nfw1zCmY6HVESUFppKZNu+ghH77qTUGsreavXUHDxJU7HkgQVGmIgIhye0C1zR2xUjZcTgTFmEtFW\nML8GvgFkA38Cyqy1K+K85g8A1toPjObvmuiNl7dW1vOze7bS3RMiN9PH+Sum8NirVQR6QiycUchn\nrlumuRMictJEwmFcbk0Hl/i2vXaYpx/e3XdcVJLNOz60ysFEiWW4xsvJ+H/WFKLzB79ire2w1vqB\nXwDLjTGFzkZLHZFIhNsesnT3RPtjtXb28MDz++noChIMRXi9soGnN1c7G1IS2obDL3DLpt/yF3sP\nTd3NTseRJKBiT06kpqol5ripsYOuzoEP72QoSbfE0lr7MnDTgNOzgZbe/+JZaox5FlgMHAV+T3RL\nuNAwr5mweoJh6gfMixg4ar61soFzl08Zx1SSLJ6qepa/7bq377iyeT9fXvUZjQiLyJvS0hTb8D8c\nitDRFiAjU7v4nEjCFXzGGC8Qd0M8a23TgPtPAf4T+Nowxds+IBf4Su/nlwB/BsJE5wIOqbAwC693\n4u4IsHxeKZt2++Nenz+ziNLS4XdYkIlpy5atMceH244QSG9nav4khxKJSCpYetpUag5v7zsuKc9h\n/sJyvZkcgYQr+IDzgEfjXTTGZFpru3o/vxC4G7jFWvujeK+x1n59wKn7jDG/Bj7CMAVfY2NHvEsT\nwgcuM9yd6eVATSsLphcSjoR58tXDhCNQUZTFmYvK8PtbnY4pCSjXE/tGwOvy0NPuwh/Q94uIvHGz\nF5ZyZvtcKnf5yS/MZOWZM6mra3M6VsIYbhAm4Qo+a+1jwAlLdWPMTUR39/hXa+2tb+Cv2kN0PqDE\nkZeVxgcvj25x1Nkd5OafP9f3WLemoYOXrZ9zlk12MKEkqstnXczepv00djfhdrm5as5l5PiynY4l\nIknO5XKxdNVUlq6a6nSUpJNwBd9IGGNuAH4AXGat3XCCez3At4H/tda+0O/SQqJFn4zA3sPNdHTH\n7qTw+t56FXwypPKsUr6+9oscaK2iOKOQ/PQ8pyOJiExoSVfwGWOmEV2Ve328Ys8Y80ng7dba86y1\nIWPMXODXxpjrgErgcqILPz4xXrmT3aTibFyu6I4bx0wu0YiNxOdxe5idP8PpGCIiQhIWfMD7iS7q\nuMcYM/Dah621twMlQP89e24kOsr3GFAKHAD+xVp729jHTQ3F+Rm868J5/H19Jd09IRbPKuLS1dOd\njiUiIiIjkHSNl8fTRG+8DPDSzqNsraxnWlkO5y6fQjgcoasnRH52mtPRRCSFBGpqCLY0kzlnLi7P\nxO2OIPJmDNd4ORlH+GScPLzxIH994vg0x9crG8jO8NLcHmDt4grOWqoWGyLy5h298w6aHo82Z/CV\nlTP15i/hK1QffZGTSQWfxLV+U+xOGq9X1vd9vuNAI2k+N6cvLB/vWCKSQrqrq/uKPYCeo7U0Pvwg\nZe98t4OpRFKPCj6JKytj+G+Pl3ceVcEnQwpHwjxy4Ek2+7dSklnMW2ZfSllWidOxJAGFmpsGnQs2\nDT4nIm+ONi6UuK45exZeT/xvkZKCzHFMI8nksQPrub/yYQ62HubVo1v4+ebfEY6EnY4lCShz3ny8\nRcUx5/LWrHUojUjq0gifxLVkVjHf/dha7KFGppXmsHHHUf75/AHCkQjTy3K0Slfi2lK3PebY31lP\nTftRJudUOJRIEpXL62XaF79Mw0P/R6ipmbwzziBn+QqnY4mkHBV8MqzC3HTWLKqgqa2bnmCYZXOL\nWTijkAtPm6q9C2VIh9uO0BmM3eDc5/ZRmJHvUCJJdL7iEsrf8z6nY4ikNBV8ckLhcITv/fk1ahqi\newu/truOwtwMTjOlDieTRHO47Qjff/mn9ISP78qS5k7juvlXk+nVFAAREadoDp+cUGV1S1+xd8xz\nW484lEYS2QtHXo4p9o75575HeObwC0O8QkRExoMKPjmh3CzfoHPVde30BDUJX2JleNIHnQuEAzR1\nN3On/V8OtVYP8SoRERlrKvjkhMqLsjh7QJPl2sZOHn35kEOJJFGtKFuK2xX/x8re5n3jmEZERI5R\nwScjsnJB2aBzldUtDiSRRLalbtuw7Vdm5Wllt4iIE7RoQ0Zk1qQ80rxuAv0e486fVuBgIklE7T0d\ng865cJHhTeeKWeuYkTfNgVQiIqIRPhmRnEwfH792CZNLssnO8HLRyqlceNoUp2NJgjm94jS8rtiN\n7yNESHP7OGeKmumKiDjFFYlEnM6QsPz+Vn1xREbpQMshfrH597T2tMWc//fTP8uUnElxXiUiIm9W\naWlu3Aa5GuETkZNqRt40VpSdEnMuw5NBSWZxnFeIiMhYU8EnIiddRVZ5zHFBej7pnjSH0oiIiBZt\nyIg1tnbz8MaDtLQHWLukglNma8RGhvaaf0vMcU1HLbUdfsqztDuLiIgTVPDJiARDYb5zxyv4m7oA\neGF7LZ+5bhlL56jom+iernqexw+uB5eLdTPOY+2kVUPus7ytfifdoW4eOfAUXcEuzpq8muUDHv2K\niMjY0KKNYWjRxnE79jfw/b9sijl3+sIyPnb1EocSSSLY3VjJj177Zcy5vLRcWgKtQ97vdXsJ9tt6\n7V+XfwRTNHdMM4qITBRatCFvWm724PlXuVmakzXR7WraO+hcvGIPiCn2AF7zv37SM4mIyGAq+GRE\nppbmcM6yyX3HxXnpXHq6dk2Y6GbkTn1Try/OKDxJSUREZDh6pDsMPdId7GBtKy3tAcz0QnxevV8Q\nuHfvgzx56BlcLhdTsydR2XJgyPuyvVnMK5zDZv9WIkSYnT+TTyz7IBnejHFOLCKSmoZ7pKuCbxgq\n+ERGJhQO9X3+0IEnePzgerpDgZh7Pn/aJ5mVP52Grka6gt1MzqkY75giIiltuIJPq3RF5E3zuI9v\np3bFrItp7m7h2eoX+85leNKZlB3tzVekx7giIuNOz+RE5KS7YtbFffPzvC4vV8+5jAxvusOpREQm\nLhV8InLSbax5lfquRgCCkSBPVj1DOBJ2OJWIyMSlgk9ETrrN/m0xx0c76qhpP+pQGhERUcEno9IV\nCNLU1u10DElwZVklMcc+t4+C9HyH0oiIiBZtyIg9+tIh/v70XgI9YRbOKOQT155CVoa+hWSwK2at\n40BrFTXttaS5fbx93lVk+TKdjiUiMmGpLcsw1JbluLrmTr74y+fp/+3yljNmcu05s50LJQktEolQ\n23GU/PQ8Mr0q9kRExlrKtGUxxswE9gEBoH8xVmOtnTnM6z4NfASYARwEfm2t/dHYJU09NfUdDHxv\nUF3X7kwYSQoul4uK3lYsIiLirKQq+Pox1tr9I7rRmPcB3wSuBp4FTgf+aYxptNbeNnYRU8ucKflk\npXvp6D6+F+rSOcUOJhIREZGRStaCbzT+FbjVWvtk7/Ezxphbgc8AKvhGKDPdy+fesZx7NlTS3NbN\nGUsmcXa/vXVFREQkcSXVHL5+j3T/ApwB5AAvA5+z1m4b4v50oB14n7X2z/3Ovwv4E5Brre2I9/dp\nDp+IiIgki6SZw2eM8RIt4uLpJlrgPQl8ECgCbgEeM8YssNY2D7i/CPAADQPO1xFtSVMMxC34Cguz\n8Ho98S6LiIiIJIWEKviA84BHh7meaa1d1e/4sDHmg0QLuKuA20f59w07gtfYGLcWnJA6unpYv7ma\n5rYAqxeVM2tSntORREREpFdpaW7cawlV8FlrHwPiDkfGeU2jMaYemDLE5XogSHQkr7+S3vN1byTn\nRBSORPjun1/j0NE2AB5/pYqb37WC+dMKHE4mIiIiJ5JUO20YYy4yxnxjwLlSogXcnoH3W2sDwCvA\nmgGXzgJettZ2jVXWVLP7UFNfsQcQCkd4atNhBxOJiIjISCXUCN8INAJfNsY0Ar8ACoBfES327gcw\nxnwSeLu19rze1/wQuNUY8w9gA3Au8AHg/eOaPMmlpw2ey5ju0/xGERGRZJBUI3zW2leI9tN7B1AL\nWKATON9ae2yD1xJgbr/X3AX8G/BroBX4OfBpa+3d4xg96c2syGPFvOP7o2ZneFm3apqDiURERGSk\nkqoty3hTW5ZYkUiE7fsbaWrrZtncEnIyfU5HEhERkV5J05ZFEpvL5WLxrCKnY4iIiMgoJdUjXRER\nEREZPRV8IiIiIilOBZ+IiIhIilPBJyIiIpLiVPCJiIiIpDgVfCIiIiIpTgWfiIiISIpTwSciIiKS\n4lTwiYiIiKQ47bQhI9bR1cP6TdU0twdYvaicWZPynI4kCSgSiVDVVk22L4uijEKn44iICNpLd1ja\nS/e4cCTCf976ElX+NgDcLhc3v2s5Zrp+octxbYF2frrpN1S1VePCxQXTz+atc690OpaIyIQw3F66\neqQrI7L7UFNfsQfRAnD95moHE0kieuLQBqraot8XESI8fvBpqttqHE4lIiIq+GREMtIGP/3P8Hkc\nSCKJrL6rYdC5P++8m9r2ow6kERGRY1TwyYjMqMhlzuTjc/a8HjcXnDbVwUSSiE4tWzbo3L6Wg/xs\n862EI2EHEomICKjgkxFq7QhwoLa17zgYCrNt3+DRHJnYlpUu5sbF7ybTkxFzvr6rgSPttQ6lEhER\nFXwyIvtrWgmGYtew7K5qdiiNJLKV5ctZXnZKzLk0T5pW7IqIOEgFn4zIjPJcvJ7YxT/9H/GK9Hfl\n7HVMy50CQKY3g3fOv5ZMb8YJXiUiImNFBZ+MSF52Gh95y2KK8tLxelycvXQSF6+a5nQsSVBZ3iym\n504ly5tJaWYJ5dmlTkcSEZnQ1IdvGOrDN7RIJILLFbfVjwj37n2QRw482Xecl5bLN8/4Ml63er2L\niIwV9eGTk0rFnpzIrsa9McctgVZqO/wOpRERERV8InLSTc+NbdmT6c2kNLPYoTQikkoO7Wvg6Yd3\nsenFQ/QEQk7HSRp6viIiJ91bZq+joauBbfWWgvR83rXgraR50pyOJSJJbs+Oozx67/a+44OV9Vz1\nruUOJkoeKvhE5KTL8mXxL8s+SE+oB6/bq2kAInJSbHstdkvPwweaaGrooKAoy6FEyUMFn4iMGZ/H\n53QEEUkhaWmxW3q6XOBL0zafI6E5fCIiIpIUTj1jRkyBt+S0KWTnpDuYKHmoLcsw1JZF5M0JR8K4\nXXpfKSInT2dHgEP7GskvzKRcGwDEGK4tiwq+YajgE3ljOoOd3L7jb2zxb6M4s4h3mmtZWDTf6Vgi\nIilNffhEZFw9UPkIm/1biRChrrOeW7feQSDU43QsEZEJSwWfiJx0+1sOxRx3BDvxd9Y5lEZERJJu\nla4x5qvAV4e4lA58wFp72xCv+QPwXmDgEMMnrbW/PekhRSa4uQWz2N9ysO8415dDWZb20xURcUrS\nFXzW2v8C/qv/OWPMJcAfgX8O89LbrbUfGMNoItLrilkX09bTzqajWynNKub6+Vfj0z66IiKOSfqf\nwMaYPOA3wKestXpmJJIA0jxpvHfh9bx34fVORxEREVKg4AP+A9hhrb3rBPctNcY8CywGjgK/B75n\nrdVGfCIiIpLSEq7gM8Z4gZx41621Tf3unQ58HDj7BH/sPiAX+Erv55cAfwbCwHfjvaiwMAuvVx28\nRUREJLklXB8+Y8xFwKPD3JJpre3qvfcWYK619tI38Pf8ELjaWjsn3j3qwyciIiLJYrg+fP9/e3ce\n/VVd53H8Se6ghiJp5mCS+BqlGU1tNNMyNdFcRhtzmaz0qIkrjpp2ylxyofKoiFOTmaYpmk7HAkpR\nqQ0rPsUAABDLSURBVDFHcyuqMZe3uaOWG1ISmIrMH+/PFy63729h//4ur8c5v/P93v1z4X7vfd/P\n2nE5fBExGehxpPWSE3gAcMpCHuox4D0Lua2ZmZlZn9FxAd8C2AkYBIzvbiVJKwCjgRsj4p7Kok3J\noM/MzMys0fpyx8vbAs9ExLT6AknHSrodoDTK2Bj4jtJKkv4VOAy4YGkm2MzMzGxZ6Ms5fOuTrW3b\nWYcM8loOJXP5JgODgaeBo9p10mxmZmbWNB3XaKOTuNGGmZmZ9RXdNdroy0W6ZmZmZtYLDvjMzMzM\nGs4Bn5mZmVnDOeAzMzMzazgHfGZmZmYN54DPzMzMrOEc8JmZmZk1nAM+MzMzs4ZzwGdmZmbWcA74\nzMzMzBrOAZ+ZmZlZwzngMzMzM2s4B3xmZmZmDeeAz8zMzKzhHPCZmZmZNZwDPjMzM7OGc8BnZmZm\n1nAO+MzMzMwazgGfmZmZWcM54DMzMzNrOAd8ZmZmZg3ngM/MzMys4RzwmZmZmTWcAz4zMzOzhnPA\nZ2ZmZtZwDvjMzMzMGs4Bn5mZmVnDOeAzMzMzazgHfGZmZmYNt+KyTkA7knYCrgCIiPfWlq0AnAXs\nD7wbeAw4PyKu7WZ/Q4ExwDZAP+BeYFREPLEk0m9mZmbWSTouh0/SaOC7wENdrPJl4HPAgcAg4Ezg\nSkk7drG/lYCbgOnAcGAT4CXg5rLMzMzMrNE6LuADZgBbAFPqCyT1A44FLoiIKRHxRkSMB34KjOpi\nfyOAYcB/RMTLETEdOBl4H/CJJXECZmZmZp2k4wK+iDg3Iv7SxeL3AYOB+2rz7wO27WKbbYHHI+KV\nyjGmAY93s42ZmZlZYyzVOnySVgRW72p5yX3rzuDyOa02/+XKsnbb1NdvbfOuHo5nZmZm1uct7UYb\nOwK3dbVQ0moR8fpC7nvO4t5m8OA1+i1kWszMzMw6xlIN+CJiMtlKdmG9UD4H1eavU1nWbpv6+q1t\n/rQIaTEzMzPrEzquDl8PniSDtHrdu+2B/+1im18CQyXNLb6VtC6wcTfbmJmZmTVGR/bD15WImCPp\nIuAkSbcDvwc+CXycDPoAkPQIcFpE/JAsQn4QuFjSsWQO41jgAWDy0j0DMzMzs6WvowI+SRsCUSZX\nBFaQ1KrTd0REXA2cD6wKjCcbZASwX0RUW+4KGAgQEbMl7QH8J/AUWW9vMrBHRMxesmdkZmZmtuz1\nmzNnYdo6mJmZmVlf0dfq8JmZmZnZAnLAtxyRdLuka3qx3hmSrl8aaWpz7KcknVO+nybpsWWRDus9\nSXMkHb6Y9jX3/9/6DkkbSnpd0seWdVoAJH2kpGfosk6LLR7l3nBaN8v9vOhBR9Xhs/lJ6g88B6wC\nbFBGCFkaRgCXlzTMAd4A1q+OVlKWrU62mh4QEYu9z8KIOAfww38ZKo2jdgDeLLPeAKYC44DREeE6\nIQ1XroFnI+LgNsvuBB6LiEPIutUdISLuoIPSY/NIupe8nv6tNv8W4J/JZ82cyvyvACcCM7vbb/15\nIWlP4PmI+LthWpdXzuHrbAcDzwMPA4cujQNKeifwL8zfQfa0kpa6T5FjH1uzXRcRq0bEqsDawEnA\nV+h6/Gozs65MAHaRtFJrRsnc+CgwAPhAbf1PAJOY99LZW2cBWy5COhvHOXyd7RgyJ2UWcLSkC1tv\nPiXn7URgJDA1InaRJOBCMmBbGfgdcELtDaefpPOBQ8r0eOC4iJhVpncm39ifqWzzEzLgvLiWvkOA\nicDc4rx2uQHVXABJq5Y07kO2pH4BuIw2uUWSzgQOj4gNJL2X7IdxL+DLwOZkILw/8BngKPJ6viwi\nvtjFv6ctooh4C5gk6VFybOv5lOETzwEOZF6H6GMj4uLKOiOAc4FNgT+W5WPbHU/SqeR1vkNEPLqY\nT8cWk8rv8+MRMVnSlsAFwBbk7/IB4JSIuFPSjsD/AHsAo8leFZ4FjoyIn5X9rQeMIUdnGgA8Cpxa\nOu9H0pXAasBdwMnAWmS/qodExIuVYwyLiMdKacQ3gP2Alcp2oyLi8SX3r2JdmEDeI7Yn/48gnzvP\nA78hA7wpAJIGkc+zscCHgBUlXQJ8mnzGXQ0cW3rjOJN5z4tngfcA/yXpuIjYXNJqwNeAvYH1yF47\nRkfE95f8KXcG5/B1KEk7AMOBq8iLeghZ1Fp1GLAv2Q8hwA/J3Lgh5AX9JHBjbZs9yZvrPwA7kRf/\n2ZXlu/L3w9/9GNhI0laV9A0l357GL+CpnUD+0LeMiP5kLuGoNufWlS8AB5Hn9w7g58B0YIOyr1Ml\n/dMCpsl6SdJqkj5FBnvXtVllFPlysDOwBnAcMKZVt0vS+8lr5iLyIX0wcJ6kdsWFnwFOBUY42Otz\nriWDqnXJ7rMmAtdKWqGyzinki9+gsnyCpDXLssvKdpuQucqTgBsryyHvX4OBfyTvlVuWfbZzKRl8\nbkEGAtOAmyT5GbiURcQD5LNpj8rsT5DB3y/K95bdgLeBm8v0oeTzaV3yfj+ytn7rGBuUr0dFxObl\n+6XkoA2te9OXgMslfWTRz6pvcA5f5zoGmBQRzwFIGg8cTd74Wm6JiIcq09sBb7Vy60rDi89KWi8i\nWsPIPVfJbXmgNOLYh3xLhgz46kV1fwV+QAaYvy7zPkcGmAtapDuQ/AHPAoiIX5X09bYu2LiIeApA\n0m3AARExpiz7uaSXgM3IHAVbPA6StF/5vjJZj+904O42644FroiIV8v0TeX/ZBvyhn4Y8GBEjCvL\n75G0L1CvH7orcAnZX+ZvF+vZ2MKoXgNVKwPtKsoPJK+TN8tve3T5IwsiALgkIp4s874KHA/sDlxP\n5tyvEBEzyvJx5AN6M+Cesv1bwBkR8TbwjKQ7yDpg8ym5RAcA+0bE82XeF8gixFXpoW6YLRETyUCt\n9dzZnQzW/w+4UNLapc767sAdETG9XDd3R8SEss3NkqaRwf7E7g4maW0yV3D3iHiizP6RpAnAEcAd\ni+/UOpcDvg4k6d3kCCL7V2ZfRhalbRgRT5d59eKI7YDTJQ0nb2Stt9dq5eXf17b5A5nbh6RhZE7Z\n7W2SdTlwi6QTgb8Bny1/K7RZtzvfJN/anpf0C/JtbRzwYi+3f6ryfSbwdG35TLKoxxaf61pF9KXI\ndjj5tvxB8i27aiB5w96FzMGDbHTUugaHkW/3c0VEPUd5C/Kl4+KIuGtxnYQtkrnXQFWprtHOyeRv\n/TBJk8kH8sRaZ/cPt75ExKuSplPuRcD7gXNLqcLqlW2q97InS7DXMhNYv01aNiLvU3Ovu4h4Abih\ni7TbkjcBOF7SRuT9egPgtnIdTAVGlAyLEczfcO+J2n5m0bvGOcPI5+HEUh2q5R3Me4FoPGdnd6Yj\nyXomV0maXm6E/03+f42srPdG64ukTcgf0d3A0FLBfq82+26Xk9YazWRX4J6IeK2+QhnJZCoZiO5I\nvrn3diziuUFhREyNiC3KPu4k69/9QdLWvdzX2z1M2xIUEW9FxO/IovX9ynVXdQMZsO0E9C/X4Z8q\ny2fT831nJzL3eFSb/VsfEBHXkEWnx5MviJeTOfDVF8T6y2I/4O3ScOxW4CVgeESsQpucO3r/228F\nmX7edY47yKo4u5W/eyulAreUeVuT9YAnVLZb2Pt9q476tq0GaOVv5YhYbop0/QPoMKXl0ufJhg2b\nM6/eyRZkRffDJK3SZtMtyeKV8yrdt2zT7hC16U3IQA7a19+r+i6ZLX4w8L0u1pkF9K+czzvIN+zW\n9ABJq0XE/RFxHrAV8Fsyt9D6ngG16Q8B34uIhyPibeVwietVlj9K1rmaS9LepQuFlosj4lDyxn9D\nF9e7dTBJgyNiRkSMj4ijyYr3HyHvaS3DKuuvTeYOP0M25hkIXFCpitLuXtZbT5DFv3OvO0nrSDq5\nHNeWsoh4k6yetDP5gletqjQJ2KUs+32r2H8RPUEG/vO1AJY0pJRaLBeWmxPtQz4JvAsYExFTqwsk\njSXrOdSL0WBe8e72km4lc/c+WuYNYV5R6FBJR5IB23AyeBtbLvqPAed1k7ZrgDPInMWtuljnEeDT\n5UH/Alnvpnqd/Rh4WdKoiHgR2JgsxlkmHT3bgisNds4C7iNbglc9DmwjaWWyYcdo8tobUpZfBpwg\n6Wgy12c4cCXz1xtt5cgcTr4MXETWX7U+QNIQ4DFJh5A5tW+TDbVeJ6tgtBpVHS9pClmd43SyPvAk\nskL9bODDkh4g72OtPtta11GvRcSfJV0LfFnS/WSO89lkQHHRwpyjLRYTyJ4fVgTOrMz/GdkY57P8\nfaPDBTET2ETSWqWo+LvAGZJ+Q9YV/DDwIzIXelw3+2kM5/B1nmOAn9SDPYASIN1Im4dfRNxP5gBe\nQXZ1sTsZPN5JVm5tBX/XM687jFvL/r5B5szMBn7VVcJKzuEtwJRW5ec2zicbTDxEPvxfYV7Te8iu\nXFYBHpE0k7zBXwN8u6vj2jJ3kHLUgtclzSLreD5MNqioF7GMJB/o04Hvk/VvxgAHSvp2aW27K9mN\nTquqwjkRcXX9oBExHfh34IguGgxYBypdOh1AFvu/CrxMllrsXeu8/TvkC+A0ssXmniVX8I/kQ/hL\nZdmxZPB/A3BpuxbdvTCSrKs1hbz3bUhev7O73cqWpJvJer5vMa8xIKVK0S/JHNkJ7TftlbFkLwEP\nlukTybqkk4DXyHrIp1cakDVevzlz3FG+mZktHfU+8pZxcsyWG87hMzMzM2s4B3xmZmZmDeciXTMz\nM7OGcw6fmZmZWcM54DMzMzNrOAd8ZmZmZg3ngM/MzMys4RzwmZmZmTWcAz4zMzOzhvNYumZmvSBp\nIPAtYE9yGMJryKELf0AOV/hF4P3k2MBnA9+MiNMkrUUOX7gXsDbwPHAdcEZEvFH2/RRwT0QcWDne\n4eTYwxtFxFOSrgS2IYeLugjYBJgKnNVuaDozsyrn8JmZ9c63yKDt88wbe/rs2jqDgd2AHYALyryJ\nZd4R5Pigp5Pjw16yEGlYnxxj9vPAB4C7gCslfXAh9mVmyxHn8JmZ9UBSf2A/4IKI+EGZfYKkrYFh\nlVWHACMi4pGy3XbAh4H9ImJiWecJSZsCJ0o6KSJmLEBS1gS+EBG/Lvs/GtgfOAi4fyFPz8yWA87h\nMzPr2RBgJWBKbf5Pa9MzW8FesXX5vKu23r3AysBmC5iO16tpiIi/AgEMXcD9mNlyxgGfmVnPBpXP\nem7cS7Xp12rTa5bPv3Sx3hoLmI7XIqI+HuYMYK0F3I+ZLWcc8JmZ9exv5bN/bf6g+oo1fy6f76zN\nb01PL59zgH61ddoFgwPazFsDeKWHdJjZcs4Bn5lZzx4ng7KtavM/2cN2vyqf29fmbwfMBB4q09PJ\nFrz1der6S5qbBkkDAAEP95AOM1vOudGGmVkPIuJVSbcBIyXdQ9abG0kPRbIRcbekXwAXSpoBPArs\nDBwDXBQRs8qq9wOfk7QXGQTuCwxvs8sZZV9fJHMPv0TWLbxmUc/RzJrNOXxmZr1zKNnY4nrgdjKH\nbkxZ9no32+0D3EL2z/cwcApwFnBaZZ0zgJ8B48jgb+Myr+6vwLnApcBvyO5hPhMRzuEzs271mzOn\nXv/XzMzqJK0KDIiIVyrzvgacCKwWEbOX8PGvBHaLiPWW5HHMrJlcpGtm1jtXADtLOoQsdt0KOBK4\nakkHe2Zmi8oBn5lZ7xwJfJ0c7mwd4FngO8BXl2WizMx6w0W6ZmZmZg3nRhtmZmZmDeeAz8zMzKzh\nHPCZmZmZNZwDPjMzM7OGc8BnZmZm1nAO+MzMzMwa7v8BNk8m3QxaBN4AAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f3bf6ddc400>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Use a ConceptNet function to load word2vec into a Pandas frame from its binary format\n",
"from conceptnet5.vectors.formats import load_word2vec_bin\n",
"w2v = load_word2vec_bin('data/word2vec-googlenews-300.bin.gz', nrows=2000000)\n",
"\n",
"# word2vec is case-sensitive, so case-fold its labels\n",
"w2v.index = [label.casefold() for label in w2v.index]\n",
"\n",
"# Now we have duplicate labels, so drop the later (lower-frequency) occurrences of the same label\n",
"w2v = w2v.reset_index().drop_duplicates(subset='index', keep='first').set_index('index')\n",
"retrain_model(w2v)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So: word2vec is even worse. With an F-value over 15, it has even larger differences in sentiment between groups.\n",
"\n",
"In retrospect, expecting _news_ to be safe from algorithmic bias was rather a lot to hope for."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Trying ConceptNet Numberbatch\n",
"\n",
"Now I can finally get to discussing my own word-embedding project.\n",
"\n",
"ConceptNet, the knowledge graph I work on with word-embedding features built in, has a training step that adjusts the embeddings to identify and remove some sources of algorithmic racism and sexism. This step is based on Bolukbasi et al.'s \"[Debiasing Word Embeddings](https://arxiv.org/abs/1607.06520)\", and generalized to address multiple forms of prejudice at once. As far as I know, we're the only semantic system that has anything of the sort built in.\n",
"\n",
"From time to time, we export pre-computed vectors from ConceptNet, a release we give the name [ConceptNet Numberbatch](https://github.com/commonsense/conceptnet-numberbatch). The April 2017 release was the first to include this de-biasing step, so let's load its English vectors and retrain our sentiment model with them.\n",
"\n",
"Download [`numberbatch-en-17.04b.txt.gz`](http://conceptnet.s3.amazonaws.com/downloads/2017/numberbatch/numberbatch-en-17.04b.txt.gz), save it in the `data/` directory, and retrain the model:"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy of sentiment: 97.46%\n",
"F-value of bias: 3.805\n",
"Probability given null hypothesis: 0.0118\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAnwAAAGCCAYAAAB6nR6iAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8XHW9//HXZGmWJmnTNt0ppdB+2LeWfREBKSCCK8gV\nFcQFFcSLov6U666oVxRRr4obq3q5KgIi+yKyyb6Wfkr3vU3SJE2aZp/fH+ckTaYzSaZNMjMn7+fj\n0Ucy53vOzCfpZOY93/P9fk8sHo8jIiIiItGVl+kCRERERGR4KfCJiIiIRJwCn4iIiEjEKfCJiIiI\nRJwCn4iIiEjEKfCJiIiIRJwCn4iIiEjEFWS6gGTMbC/g98BbgL3cfWWvtvOBK4G5wEbgNuCr7t6Z\n4r5KgR8CZwATgEXh/g8M588gIiIiki2yrofPzN4FPA2sStL2FuBG4GpgEvBu4ALgqn7u8mfAscBC\nYApwA3CXmdmQFi4iIiKSpbIu8BH0wp0A3Jyk7TLgH+7+f+7e6u6vAj8CLjOznX4WM6skCIRfd/cl\n7t7i7r8C3gAuGb4fQURERCR7ZF3gc/ffuvuSFM1HA88kbHsGmEhwijfRfKAwxTFH706dIiIiIrki\n6wLfAKqALQnbasKvk1PsT4pjku0vIiIiEjlZOWljF8WHev+Ojs54QUH+LpYjIiIiMqJiqRpyLfBt\nIjh929uk8OvGFPsTHrMu4Zhk+/dRV9ecbn0iIiIiGVFVVZ6yLddO6T7JzmPvjgc2AMuS7P880Jrk\nmOOAfw15dSIiIiJZKNd6+K4FHjOz84C/AQcBnwN+6O5xADO7Cdjo7l9w9wYz+x3wDTN7FVgLfAqY\nDfwyEz+AiIiIyEjLusBnZg7syY7eRzezOHCzu3/MzN4PfBO4ieCU7XXANb3uYhZ9ey7/E/gB8DhQ\nDrwELHT3ndb5ExEREYmiWDye7lyH0aO6ulG/HBEREckJVVXlKSdt5NoYPhERERFJkwKfiIiISMQp\n8ImIiIhEnAKfiIiISMQp8ImIiIhEnAKfiIiISMQp8ImIiIhEnAKfiIiISMQp8ImIiIhEnAKfiIiI\nSMQp8ImIiIhEnAKfiIiISMQp8ImIiIhEnAKfiIiISMQp8ImIiIhEnAKfiIiISMQp8ImIiIhEnAKf\niIiISMQp8ImIiIhEnAKfiIiISMQp8ImIiIhEnAKfiIiISMQp8ImIiIhEnAKfiIiISMQp8ImIiIhE\nnAKfiIiISMQp8ImIiIhEnAKfiIiISMQp8ImIiIhEnAKfiIiISMQVZLqAdJnZicD9SZoKgZvc/aKE\n/S8Efg+0Jux/m7t/aFiKFBEREckiORf43P0xoLj3NjObCrwG3JDisFXuPnt4KxMRERHJTlE5pftL\ngh67f2a6EBEREZFsk3M9fInM7B3AccDe/exWbma3h/u1A/cCV7r7lhEoUURERCSjcjrwmVkecDXw\nPXffmmK3GmAR8FPgfcCBwB+BW4Az+7v/yspSCgryh65gERERkQyIxePxTNewy8zsvcCvgWnu3pLG\ncWcDdwCz3H1Nqv2qqxtz95cjIiIio0pVVXksVVuuj+G7ALgznbAXWhp+nTHE9YiIiIhknZwNfGZW\nDpwB/G2A/S4xs8TlV/YLvy4bjtpEREREskkuj+E7DBgDvNh7o5kdCdwEnObuq8N9fmBm64FHgQOA\n7xKs2Vc9ohWLiIiIZEAuB77p4dfNCdtLASMIerj7dWZWCPwcmAXUATcC3xyhOkVEREQyKqcnbQw3\nTdoQERGRXBHlSRsiIiIiMgAFPhEREZGIU+ATERERiTgFPhEREZGIU+ATERERiTgFPhEREZGIU+AT\nERERiTgFPhEREZGIU+ATERERiTgFPhEREZGIU+ATERERiTgFPhEREZGIU+ATERERiTgFPhEREZGI\nU+ATERERiTgFPhEREZGIU+ATERERiTgFPhEREZGIU+ATERERiTgFPhEREZGIU+ATERERiTgFPhER\nEZGIU+ATERERiTgFPhEREZGIU+ATERERiTgFPhEREZGIU+ATERERiTgFPhEREZGIU+ATERERiTgF\nPhEREZGIK8h0AbvCzFYCM4DOhKaD3X1Jkv3fBnwD2B9oAO4BrnD35uGtVERERCTzcjLwhT7m7jcM\ntJOZzQXuAq4EfgtMAf4M/By4aDgLFBEREckGo+GU7ieAxe7+U3dvdvcVwDeBC8xsUoZrExERERl2\nuRz4zjWzRWbWYGbPm9k5KfY7GngmYdszBL2b84e1QhEREZEskKundF8BlgIfBlqBy4DbzexYd386\nYd8qYEvCtprw6+T+HqSyspSCgvwhKFdEREQkc3Iy8Ln72QmbvmNm7wE+BiQGvv7E+2usq9OcDhER\nEckNVVXlKdty+ZRuoqUEM3cTbQImJmzrHru3cVgrEhEREckCORf4zGwvM/u5mY1PaNqPIPQlepJg\nHF9vxxOcCn52GEoUERERySqxeLzfs5pZx8xKgSXAYwRj91qAzwFfAQ4GxgE3Aae5+2ozmw28DlwF\n/BKYBdwOPOzul/b3WNXVjbn1yxEREZFRq6qqPJaqLed6+MLFkt8GlAGLgWrgNOAt7u5AKWDAmHD/\nlcAZwPuBOuCfwL3AFSNdu4iIiEgm5FwP30hSD5+IiIjkikj18ImIiIhIehT4RERERCJOgU9EREQk\n4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9E\nREQk4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCJO\ngU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERE\nRCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCKuINMF7Aozmwx8H1gIlAGLgK+4+0NJ9r0Q\n+D3QmtB0m7t/aJhLFREREcm4nAx8wB3AVuBwoB74GnCHmc1z9/VJ9l/l7rNHsD4RERGRrJFzp3TN\nbBxBj95n3X2ju7cQ9PaNBY7KaHEiIiIiWSjnevjcvQG4OGHznPDrmhSHlZvZ7cBxQDtwL3Clu28Z\nnipFREREskfOBb5EZlZBMEbvDnd/LskuNQQ9gj8F3gccCPwRuAU4s7/7rqwspaAgf2gLFhERERlh\nsXg8nukadpmZ7Qn8HdgIvNPdtw3yuLMJxgHOcvdUvYJUVzfm7i9HRERERpWqqvJYqracG8PXzcyO\nAJ4B/gWcOdiwF1oafp0x5IWJiIiIZJmcPKVrZgcSjMP7lrtfO8C+lwDN7n5Tr837hV+XDVOJIiIi\nMsTi8TivPreOFUuqqags4YjjZ1NWUZzpsnJCzgU+M8sHbgR+nSzsmdmRwE3Aae6+GhgD/MDM1gOP\nAgcA3wVucvfqEStcREREdssrz67lyYeDvpr1axrYtH4r5118BLFYyjOZEsq5wAccQ7D+3oFm9tmE\ntpuBWwEjCHq4+3VmVgj8HJgF1BEExm+OWMUiIiKy25Yn9NPU1TRTV9PMhKqxGaood+Rc4HP3x4GB\nonyfdne/Brhm2IoSERGRYVc+vpiN67b23M7Pj1FaNiaDFeWOQU/aMLOHzWx8irYDzezZoStLRERE\npK8jT9iLivHBmL28/BjHnLw3xSWFGa4qNwy4LIuZzQq/XQksIFjXrs99AGcB17h7pEZOalkWERGR\n7NLVFad2cxNlFUWUlKp3r7f+lmUZzCndJUAhEAdS9eLFgMfSL01ERERk8PLyYlRNLc90GTlnMIGv\nAjiCYL27zwFbk+xTR7BMioiIiIhkmUFfacPMPgz8yd1bh7ek7KFTuiIiIpIr+julm9al1cysFNgX\nmEiSmbLufv+uFJitFPhEREQkV+zuGD4AzOztBOvcjaNv2IuHt+NA/i7WKCIiIiLDJJ11+L4PvA5c\nB1QTBDwRERERyXLpBL45wAJ3XzRcxYiIiIjI0Bv0wsvACoLlWUREREQkh6QT+L4GfNfMKoerGBER\nEREZeumc0j0PmAusM7OlQFNCe9zdjxuyykREZNSId3UR7+wkr1AnkkSGQzqBbxKwLvwnIiIyJLY+\n9STVt/2Jzm1NlC84kikXfoS8MbpklshQSmsdvtFG6/CJiAyvjvp6ln/xc9DZ2bNt4rvew8S3vyOD\nVYnkpiFZh6+bmZUDBwHTgHvdfZuZ5bt75wCHioiI9NG6dnWfsAfQunJlZooRibBBT9ows3wz+yHB\nGnyPA7cBU8xsD+A1M5s+TDWKiEhEFc+eQyzh9G3JvvtmqBqR6Epnlu5XgY8C3wKOA7aH2+uBDcB3\nhrY0ERGJuvyyMqZ/+jMU7TGL/IoKKheewfi3npLpsiRLtbd38sg/FvO7ax/nzzc8z8Z1DZkuKWcM\negyfma0E/p+7/zG83Qgc4u7LzewY4E53rxquQjNBY/hERESyx1OPLOOlf6/puV0ytpAPfuoY8vPT\n6b+Krv7G8KXzG5oMPJ2ibQNQkU5RIiIiIulYv6a+z+3t29qpr23OUDW5JZ3AtxI4MUXbMcDa3a5G\nREREJIUp0/r2LY0pKqCisiRD1eSWdGbp3g783MxmAQ8CMeBAMzsd+Abws2GoT0RyVEtHK8saVjCp\nZCJTSiM12kNEMuSIE2bT2NDCyqW1lI8r5sSF8ygszM90WTkhnTF8hcAvgAsJwl73eeJO4LfAp9y9\naxhqzBiN4RPZNeuaNvCTF3/FtvbgVMsZs0/lrDmnZbgqEYmKzs4ujdtLor8xfGkvvGxm04AFBGP2\n6oDn3H3zblWYpRT4RHbNr1+9mZeqX+25nRfL47vHXUX5mLIMViUiuS4ej/PKs2tZvqSGceOLOeKE\nvSgfV5zpsrLGkC687O4bgLt2qyIRibTGtr6X2u6Kd7GtvVmBT0R2yyvPruXJh5cBsHFtA5s3NHLe\nR48gFkuZcyQ06MBnZhXA5cBhwDh2nNLt4e4nD11pIpKrjp62gGUNK3puz66YxdSxkzNYkYhEwXKv\n7nO7rraZLTXbmFilD5MDSecE+I3AFwlC4hpgVZJ/IiIcMeVQ9q2cS0FeAeOLKnjHnIWZLklEIqBi\nfN8ZufkFeZSVF2WomtySzindk4Fz3P2h4SpGRKLhHysfZHHdmwDUt27lpkV/4lvHfpn8PM2mE5Fd\nd8QJs9m0fisNddvJz49x7Ml7U1RcmOmyckI6ga8OWDdchYhIdCypW9bndkNbIxubNzOjbFqGKhKR\nKKgYX8L5Hz+SLdXbGFteRHGJwt5gpXNK94fAVWam6TAi0q9Z5TP63C4pKGZSycQMVSMiURKLxZg4\nuUxhL03p9PD9ATgf2GBmS4BtiTuM1KQNMyslCKBnABOARcBX3f2BFPvPB75PMOGkFfgncJm714xE\nvSKjzVlzFlKzfQuLtjjji8bxfnsXRfljMl2WiOS4eDzO6y+uZ7lXM66yhPnHzdYYvkFKN/AdDDwC\nbAEyuUbdz4DDgYXAauDDwF1mdoi7e+8dzWwCcC/B4tDvAYqBG4D/A946gjWLRF5LRwu1LXVMGzuF\nTx96MW2dbRTkFZAX0wKpIrL7Xn1uHU88tBSAdavq2bhuK+d+ZIGWZRmEdALfCcA73P3h4SpmMMys\nErgAONfdl4Sbf2VmlwCXAP+ZcMh/ECwhc5W7dwANZvYl4KUwIL48UrWLRNlzG1/kD/4XWjvbmFhc\nyYkzj2XV1jVMKpnIKbNOpKxwbKZLFJEct2xx3+s8bKneRl1NMxOq9PoykHQC32Zg/XAVkob5QCHw\nTML2Z4Cjk+x/NPBCGPa6vQK0hG0KfCK7qb2znT8t+RutnW0A1LbUcfvSu3vavW4pX1hwWabKE5GI\nKB9XzMZ1W3tu5+fHKC3TcJHBSOc8yzeBr5lZpmN091XYtyRsrwGSrexalbivu8fDbVoJVmQIbOto\nZnvH9pTtq7auYeO2SF6BUURGUOWkvhGkZOwYTd4YpHR6+M4mGMO3ycyWsfOkjbi7Hzdkle2adMcV\n9rt/ZWUpBQVaN0xkIFWUs8+E2SzdsjJpe34sj1lTqxhXXD6yhUnOaKtvoGPrVkpn7ZHpUiSLbVzb\n0Od209ZWYl0waYpeWwaSTuAbTzBBYvUw1TJYm8KvE+m7LuAkYGOK/fss/mVmMYLZvcn271FX17zr\nVYqMMh/Z7wL+vvw+1m/bxB7lM3hh08ts62gmRoyFs0+hrTFGdWNjpsuULFR759+ovfsu6OykaM/Z\nzPzs58gv1xu47KygsG8nTF5ejG3b24hX67UFoKoq9d/NoAOfu2fLjNbnCZZWORr4S6/txwF3Jdn/\nSeA7Zlbo7u3htgUEs3UfH85Co6a1vZOnXt/I1qY2jthvMtMmZvrsvmSTcUUVfGC/9/XcfuvM4/jn\nuieZVT6To6bOz2Blks3aNm+m9q47IB6ccGldtZIt991D1XvPzXBlko0WHLcn61fX07I9eDs//JhZ\nlI7VGL7BiMXjqc9qmtkYd2/r/n6gO+ved7iZ2f8AJwLvBtYCnwK+DhwAdAAPARe5+1NmNg5YDNxK\nMA6xIvy+0d3P6u9xqqsbM7n0TFaJx+N895bnWRYOli3Iz+NLHzicOdMrMlyZZKOl9Sv42Uu/pr0r\nmCt13PQj+Y9935vhqiQbbXv9Ndb9+Id9tpXNX8D0T16aoYokG9VubmKZV1NWUcRecyexeUMj4ypL\nGD+hNNOlZZWqqvKU69MMNGlju5l1T2xoAbYP8G+k/CdBqHscqCVYX2+hu68imMFrwFgAd28A3kaw\nbt9Gghm6ywmWa5FBWrZua0/YA+jo7OLhF9ZmsCLJZg+seqQn7AE8uf5Z6lrqM1iRZKuSufPIL+/7\nwbHs8AUZqkay0bpVdfz5hud5/olV/POeJfzlhud58uFlPPnQUrbU7HQNCElhoFO63wSaen2fFT1e\n7t4KXB7+S2xbSbDuXu9trwEjchWQqMrL2/lDQ36SbTK6tXd1QDzeJ+wBxInT0dWZoaokm+WNGcP0\ny69g8y030tnUyLgT3kLFUclW2JLR6pXn1tLVtSN+NG5tBaC+tpnqTU1MmlxGW1sn+x86DTtwaqbK\nzHr9Bj53/0avm78D1rp7V+J+4WnTfYe4Nskic6ZXsP/sShatrAOgaEw+py7QbDrZ4a5l9/LQmn8R\nJ86+lXP7tM0dP4eqUl1LV6D+sUfZ+q/HyCstZeLZ76SzeRubb76Jji21AGz5+52UzDNaV62ibeMG\nyg45lLEHHZzhqiVTtje3sTZ830mmuamN1U3Bymsb1zZQXFLInnvrtSaZdGbprgCmAtVJ2mYRXL6s\nciiKkuz02fcdwotv1tDQ1Mrh86qYUFGc6ZIkSyze8ib3rtpxEZ7Xat/o017XUk9XvEuXWBvlGl94\nns033dBzu3nxG9DZt+c33t7Ouh/9N/H2YFB+w6MPM+XCixl3/AkjWapkWFNjK9UbG9m8fisd7Tv1\nM6W0YkmNAl8KAwY+M/tq+G0M+LyZJZ4wjwHHkN4izpKDGpraWLK6noZtrUwcV6zAJz3WNK7rt72m\nZQsbt21meplOt4xm2156se+GzuSn+bvDXreGfz6iwDeKLHl9E4/cvZiurjjpXiJ3rK66kdJgevjm\nAMcSjN+7MsU+jcC3hqooyT4dnV1879YXqN3aAsBzXs1n3nMwh86dlOHKJBtY5T7EiBFPMcy3MK+Q\nyuJxI1yVZJsx06YNvFMSecX6cDmaPPXwsp4xe/0sJJJUbbUmcaQyYK+cu1/o7vMIevJmEsyC7fPP\n3ce5+w/7uRvJcUvW1PeEvW5Pvd7vutUyisyqmMkH9zuXqaWTmVw6iXP2PoOJxcEIj4JYPu/e5yxK\nCkoyXKVk2vi3nkLpAQcGN/IHdxWjWGEhE846exirkmzS1RXvWWNvV2xIuBKH7JDOwss6ZTuKjSsr\nSrJNXeeyw1HT5nPUtGCB5ec2vURtSzDQuiPeyYubX+HEmcdksjzJAnnFxcz8z8/TXltLXlERa6+9\nhtaVK3baL1ZURNX7P0BeQT6l+x1AwfjxGahWMiEvL8a8A6ew+JVd61CYPE1XaEll0IEvvBzZ+QRX\nuJhIwtInBNfS/cAQ1iZZZMaksUwoL2JLY2vPtoP31ulcSe6BVY/2ub2kfhmbm6uZXFqVmYIky8RZ\nd92PaV25glhpKfHmvpexrDj+RMafcGKGapNMO3HhPCZMGsum9VuZtsc4ajY19RsAx08spb62makz\nKzjxtHkjWGluSWeW7o+BzwDbCBY7TjyznhVr9MnQ6+qK869X1vcJewD3PL2KA/eakKGqJJs1tG7d\naVtLR2uSPWU02nzrzbQsXwawU9gDetpkdMrPz+OQI3cs+7VxbUPKwFcxvpjzP3YknZ1d5OfrRGR/\n0gl85wJfBr7v7gp3o0RDUys/+OOLbKjd+UX5jVV13Pvv1Zx+1KwMVCbZbEJxJY3tTX22lRZqDJ8E\nWlat6rc93qYPB7JDa2tHyrat9S001G1nXKVeXwaSThwuA/6ksDe63P/smqRhr9ujL/W/HIeMTmfs\ndUqf2wdN2p9JJVobSwKl++43QPv+I1SJ5ILps8b3u9zKS0+v5pG7F7NhjS7f2J90evj+RXA92pXD\nU4pko7qm/j9pF48Z3Ew7ib7Xat7gvlWP0NHVwUkzj+MD+76HJ9Y/y+TSSZw795xMlydZZPL5H4B4\nF82LFlG0xyy2r1pJvHnHcholpgs3SaCttYO/3foi25raACgqLqC1ZUePX15ejEUvbwDAX9vIOy84\njKkztARUMukEvs8DvzKzKcC/2XGN3R7uvmSoCpPscPT+U3j69U1J22IxeOfxc0a4IslGm5tr+NWr\nN9IVD1bEv+mN/+1pW7l1Ndvam/nUIR/JVHmSZfLLypj28U8C0LzEWfuDq/u0Nz3/LOWHz89EaZJl\nFr+ykdrNOz4M9A57QJ9r7Mbj8PSjyxlbNoaqqeUcNH8m+QUa19ctncD3evj1eFJP0FB3T8QcvPck\nLnvPQTz+6gZeWVpLZ68/rvNO3kcLLwsAi7cs6Ql7ybxeu5jq5lpdT1d21rXz86a9tjYDhUg2at6W\n3njODWuCdfiWvlFN7eZtnPKO/ocPjCbpBL6PoJm4o9Jhc6sYN7aIF5fU9Nm+drNWNJfA1LFTBtyn\nIE+fByXQsXUrNX/9P1pXraKwauelevIKtcanBHZn5u2bizZx0hmmXr5QOgsv3zCMdUiWq27YvtO2\nl5bWcNcTKzh5/kzGFhdmoCrJFvMq9+aUPU7k0bVP0BXvYmLxBGpadvTSFOUXMa6oIoMVSjbZcP0v\n2L74DQBa16zeqT2vvGykS5Is1d8M3YEUFReSl5/mxXgjLJ0ePsysFPgQweSNqcAngE3A8e7+2NCX\nJ9mgaXs7t97vSbff/q8VvPhmDV+98IgMVCbZ5N1zz+L02afQGe/k2hd+2aettbOVDds2MaNs166l\nKrkvHo/TvmkjseLinrCXSkdDA/HOTojFiOXlsX35clpWLKNk7jzGTJ3G1qeepKNuC+Xzj6Bojz36\nvS/JbTNmjefV53ZtNYijT5pDLKbA1y2dK23MAR4BZgBLgb2BEmAe8KCZnePu9wxLlZJR/160iabt\nqT9lrdzYyKqNjew5VZe0Ge2619pr6dx53E1eWqtASZS019aw7ic/om39eigsJK+4hK6Wnc8adGtb\nu5all32SWF4eJXPnse3VV3raCqdMoX1TMJFsyz13M/OKKynVrN7Imjx9184MHHLkTPY7RB8we0vn\nFfhHwGpgjrvvC7QAuPti4NvAVUNfnmTS9tYObn1gCf94amW/+8WA0uK0Oosl4iqLdl4WoTBfp/1H\nq5q//TUIewDt7f2GPYCubU3E29roamnpE/aAnrAHQGcn9Y88NNTlShapq0lvrPiEqrG84/2HcOzJ\n+wxTRbkrnXfpk4C3ufvOAy7gj8AXh6QiyRo33ruYZ97YPOB+Jx02g6rxWuVcdhhbOLbP7fxYHhVj\n1AM8WvUJaUMslq/JQFFWNbWc/Pw8OjtTrwLQ2wGHTmfm7Mphrio3pdPD1wXsfIHMwBg0gzdyXlpa\n02/7jEml/NeHF/DBhTZCFUmuqG/tu+J9Z7yL6u39P58kusoOPazP7VjBrp8RGDNt+o77KSqi8m2n\n7/J9SfZr2tpKV5Kle5KZPK2c/Q+bPvCOo1Q6f3WvEVxL98O9tnWHvI8CLw5VUZIdplSWsmbzTutr\n98jPy2OvaZp5KYF1TRu4e/n91LdupaG1caf28oRePxk9Kk8/E4DGF55nzOQplB97HOt/9hPoSD42\nuLBqCvHODmL5+Ux4+zvIKymhZcVySubNY+yBB7Pt5ZfoqNvC2EMPo3CC1naMsjcXbSI+yO6k4tJC\n8vI0SSOVdALf94A7zexY4GGgEPiGme0LHAqcMQz1SQZNm9h/4CvU2kYSau9s56cv/ZrGttTPl6b2\nZiq0NMuoFMvLY8KZZzHhzLMA6GptTRn2AAoqx7HHF77cZ1v5/AU935cddvjwFCpZp6R08Gsyrl1Z\nR0dHJwUFOs2fzKDfsd39H8ApBDN030twVY3TgQ3AW91dI2cjZtm6hn7b87W+kYRWbF3db9gDKCko\nHqFqJNvlFRVRsm/qKyAUqNdOQnvvu/PC3Kl0dcZZtVRXaUkl3S6a7cB/uPtEdy8kmKixFhh4mX3J\nOQNNxCgt0sxcCZQWDDxpp7mj/5mZMrpM+/gnqTjuBMbMmBlcmLuX3uP0ZHSr2dT/B8lEtboCVEqD\nDnxmdh7wJLBPePsLwPXAccAtZnbhcBQomXPeyXMZWxKEuoIkvXnra5v53q0v8OdHl7G+ehu//fsi\nvnfrCzz43Brigx10IZEwoXh8pkuQHFNQUcHUiy6m6rzzSRyklezqGzI6jZtQmtb+VdO0GkAq6XTR\nfAH4L3d/1sxiwBXA99z9v8Kwdxlww9CXKJmyZE0928IFlzs6dw5wm+u2s7luO0vW1PPwC2tpaevs\nOS4Wi3HK/JkjWq9kTmlhKWPyCmnrak+9T0F6L9wyOhSMS/JhQR8YJVQ5sZQjTpjN80+soqsrTuWk\nUupqmnv5VPYVAAAeBklEQVTaK8YXs725nXhXnIMWzGCvuZMyWG12SyfwGfCn8PsjgCrgN+Hth4Gf\nDGFdkgXu+feqQe/bHfa6Pbd4swLfKLNP5RwW1e58Cb5urUmuviHS2bjzal/xQa65JqNDR3snXV3B\nh4AxYwqwg6awetkWyiqKOPmsfRk/oZR4HPLzNZGwP+n8dtp67X8q8Ka7dyeCMWnel+SA3bkGoRZi\nHn3OnftO9iifAezcm1dVMompYydnoizJcvljd16uJ79Urx8SqN3cxItPr+m5vWn9VvzVTWxvbqd6\nYxP3/uV1YrGYwt4gpPMbegX4tJkdAXwSuL1X29nAkqEsTDLvzKP37HO7eEzfqe5FhcHTp7SogLce\nNqNnnN/UCaWcffzsEalRskdV6US+sOAyrpx/KYdMOqBPW0PrVl7e/Drb2ptTHC2jVbJFmOMxvXlL\nYGt9S7/tDXXb2ZLm5ddGq3RO6X4NuAu4HFgG/BDAzM4Cvg9cNOTVSUadMn8me04px9fUMWf6OCZW\nFHH3U6to2NbGsQdO5ZB9JrG+ZhvTJ42lqDCfd56wF3WNrcycXEbebvQOSm5a37SRX77ye2pb6sij\n7/9/W1cb1792I4V5hXz0wAs4cFLqJTlkdOmor99pW7y1/zd5GT1m7DmeouICWluSr9uYX5BHWbmW\nfBqMQQc+d/+nmc0kGMv3urt3f1RfDJzl7vcNR4HJmNlhBCFzAdAJPAFc4e7LU+x/A/BBIHFE+aXu\n/pudj5Bu+8wcxz4zx/XcvujMvm/Uva+0UV46hvI0FsmUaPnLm3dR21IHQFeKKy22d7Vz+9K7Ffik\nR/E+cymYMIGOLVt6tlUcdUwGK5JsMqaogLPPP5Tnn1xFy/Z2iksKWO47LtNYUlJIUbGWCBuMtH5L\n7r4VeDZh21KCxZhHhJlNAx4iWBLmncBY4BbgL8Bh/Rx6s7tfOOwFioxSG5o3DWq/utade3Rk9Mor\nLGSPK/8fW+79Bx0N9VQcc5yupCF9TJpSxsJ3BcNEbr/lhT5tTY2tbKnZxoRJunTjQHIxFs8gGD/4\nFXfvBJrN7BfA7WZW6e51mS0vOmoatnPL/UtYubGRfWeN5+T5M3n4+bU0NLVxzIFTqW1o4eEX1jKm\nMJ9zjt+LEw/RYqmjVXtnO60dbYPat62znermWqpKdTUFCRRWVTHlgx8eeEcZ9crKi/rczsuPUVJa\nmKFqckvOBT53fw64OGHzHGBr+C+Vg83sCeAAYDPwe+AHYWiUJK6/cxFLw8urPfPGZl5YUt2zHp+v\n2dFLs62lgxvvWcze0yuYUVWWkVolsxZtWUJL5+DGXcWJ83LNa5w66y3DXJWIRM2C42ezenkdba3B\nmL7Djp6V1vV2R7OsC3xmVgCkTA3u3ud8kJkdBHwd+Go/4W0FUA58Jfx+IfAHoItgLGBSlZWlo/Yi\nzG3tnT1hr1uyxZe7xYGNDa0cuv+0Ya5MslFVZ8XAO/Wy1+TpVFVpRXwRSc/qpbU9YQ9g/ap6qt6t\n15LByLrAB5wEPJCq0cxK3L0l/P4U4M/Az9z92lTHuPs3EjbdaWbXAx+nn8BXVze6l5CYPmks69OY\n7j6loojq6sZhrEiy1dS8Gew9bi+WNawAIEaMeIqJG4dUHcjsMXP0XJE+2jZuoKOhgZJ95hLLH50f\ntKV/tZubePqxvnMzN6xtYMkbG6nUGD6Afj9IZ13gc/cHgQHX9DCziwmu7vEZd//dLjzUUoLxgJLC\nsQdO5c+PLuu5PW1iKRtqgxBcPCafYw6YypOvb6SoII9zTpjDzMk6nTta5cXyuPywj/Na7WK2d2zn\n9jfvpqmj74eFjx34QaaNncIULcAsCTb/4RbqH34QgMKqycy88ksUTpiQ4aokW3R1xbn/9tdZ8WbN\nTm15eTGKNYZvULIu8A2GmV0AXAOc4e7/GmDffOBq4K/u/nSvpv0YwdnFueixl9b3ud0d9iC4lNq4\nsWP4xRUahyWB/Lx8DqkKZtI9uuYJmpr6Br7pZVOZXFqVidIki7WuX9cT9gDaqzdTd/+9TH7/f2Sw\nKskmq5fXJg17AOMnlGgM3yDl3HLmZrYH8Avg/FRhz8wuNbNHAcJxffsA11ug0MzOIZj4cc0IlZ2T\nmrYnLlvY1+rNTSNUieSaU/fs+0FgXuU+CnuSVGdDw07bki3GLKPXtsbUKwBsqWlma/32Eawmd+Vi\nD9+HCSZ13G5miW0fc/ebgUkEIa/bRQS9fA8CVcAq4JPufuPwl5u7jj94Gvc/u+Mahnkx6Oo1LGv/\n2ZUZqEpywYIphzK2sJSXql+jqmQix08/OtMlSZYqmTuPgokT6ait7dlWcbQWXpYdZu8zkaceyae9\nLfm8zN257vtoEovHU8+8HO2qqxtH9S+nqyvOYy+vZ8naevaoKuP2fy3vM1P3A2+byynz98hghSIS\nBe21NWy59x46tzZQcfSxWnhZdlKzqZGXn1lL49YWNq5toDu67LPfZN52zv6ZLS6LVFWVp0y/Cnz9\nGO2Br7dXl9fy49te7rPt8HlVXPrugzJUkWS79q4OVjasYmLJBCYUqzdYRIbGlpptrFpaS8X4Evaa\nN4m8PPXwdesv8OXiKV3JgJlVZeTnxejsdU531hTNypXkNm7bxHUvXk9DWyMxYpw953ROm/3WTJcl\nIhEwYdJYXUptF+TcpA3JjMryIj50ujG2uIBYDOZbFQuPnJXpsiRL/WPFgzS0BevsxYnz9xX309Q2\n+DUdRURkaCnwyaAV5ueRlxcjBhQW5KFedEmlvrXvzMvOeCdN7ZrVLSKSKQp8MigNTa389u43aGxu\npysOT7++qc8MXpHejph6WJ/be5RNZ+rYKRmqRkRENIZPBmX15qY+4/cAVmzQpbEkuRNmHENBXiEv\nV79KVckkTttT4/dERDJJgU8GZa9pFYwpyKOto6tnm80an8GKJNsdM20Bx0xbkOkyREQEndKVQSor\nKeTSdx/EzKoyyksLWXjkHpxy+MxMlyUiIiKDoHX4+qF1+ERERCRX9LcOn3r4RERERCJOgU9EREQk\n4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9E\nREQk4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCJO\ngU9EREQk4hT4RERERCJOgU9EREQk4hT4RERERCKuINMFpMPMZgMrgDYg3qtpo7vP7ue4y4GPA3sC\nq4Hr3f3a4atUREREJHvkVODrxdx95aB2NPsQ8C3gHOAJ4EjgbjOrc/cbh69EERERkeyQq4EvHZ8B\nfufuj4S3Hzez3wGfBRT4REREJPJydQzf1Wa2ysxqzew+Mzsg2U5mVgQcCjyT0PQMcLCZlQ53oSIi\nIiKZllU9fGZWAJT1s0sr8BzwCPARYALwM+BBM9vX3RsS9p8A5ANbErbXEITdiUBzqgerrCyloCA/\nrZ9BREREJNtkVeADTgIe6Ke9xN2P6HV7nZl9hCDAnQ3cnObjxftrrKtLmQVFREREskpVVXnKtqwK\nfO7+IBBL85g6M6sFZiRprgU6CHryepsUbq/ZlTpFREREcklOjeEzs1PN7JsJ26oIAtzSxP3dvQ14\nHjg6oel44Dl3bxmuWkVERESyRVb18A1CHfD/zKwO+AUwHvgVQdi7C8DMLgXe6+4nhcf8CPidmf0N\n+BfwFuBC4MMjWrmIiIhIhuRUD5+7P0+wnt55wCbAge3AW929NdxtErBPr2NuAz4HXA80Av8DXO7u\nfx7B0kVEREQyJhaP9ztvYVSrrm7UL0dERERyQlVVecp5EDnVwyciIiIi6VPgExEREYk4BT4RERGR\niFPgExEREYk4BT4RERGRiFPgExEREYk4BT4RERGRiFPgExEREYk4BT4RERGRiFPgExEREYk4BT4R\nERGRiFPgExEREYk4BT4RERGRiFPgExEREYk4BT4RERGRiFPgExEREYk4BT4RERGRiFPgExEREYk4\nBT4RERGRiFPgExEREYk4BT4RERGRiFPgExEREYk4BT4RERGRiFPgExEREYk4BT4RERGRiFPgExER\nEYk4BT4RERGRiFPgExEREYk4BT4RERGRiFPgExEREYm4gkwXkC4zuwq4KklTEXChu9+Y5JgbgA8C\n7QlNl7r7b4a8SBEREZEsEovH45muYbeZ2ULgJuAAd69J0n4DgLtfmM79Vlc35v4vR0REREaFqqry\nWKq2nD+la2YVwK+By5KFPREREZHRLudO6SbxNeANd79tgP0ONrMngAOAzcDvgR+4e+dwFygiIiKS\nSVkX+MysAChL1e7u9b32nQV8CjhhgLtdAZQDXwm/Xwj8AegCvp/qoMrKUgoK8gddu4iIiEg2yrox\nfGZ2KvBAP7uUuHtLuO/PgH3c/fRdeJwfAee4+96p9tEYPhEREckV/Y3hy7oePnd/EEhZcLewJ/A8\n4Au7+FBLgRm7eKyIiIhIzsi6wJeGk4GJwB397WRm+cDVwF/d/eleTfsRhD4RERGRSMvlWbpHA6vd\nfUtig5ldamaPAoSTMvYBrrdAoZmdA1wMXDOSBYuIiIhkQi738E0nmG2bzCSCkNftIoJevgeBKmAV\n8MlkizSLiIiIRE3WTdrIJpq0ISIiIrki0gsvi4iIiEj/FPhEREREIk6BT0RERCTiFPhEREREIk6B\nT0RERCTiFPhEREREIk6BT0RERCTiFPhEREREIk6BT0RERCTiFPhEREREIk6BT0RERCTiFPhERERE\nIk6BT0RERCTiFPhEREREIk6BT0RERCTiFPhEREREIk6BT0RERCTiFPhEREREIk6BT0RERCTiFPhE\nREREIk6BT0RERCTiFPhEREREIk6BT0RERCTiFPhEREREIk6BT0RERCTiFPhEREREIk6BT0RERCTi\nFPhEREREIk6BT0RERCTiCjJdQDJmdjLwOwB3n53Qlg98AzgXmAYsBf7b3f/Qz/3NAa4FjgJiwL+B\ny919+XDULyIiIpJNsq6Hz8yuBn4DLEqxy1eADwPvByYCXwduMLOTUtxfIfAPoB44AJgHVAP3hG0i\nIiIikZZ1gQ9oAg4FXkhsMLMYcClwjbu/4O5t7n4HcDdweYr7WwjMBf7T3WvcvR74PLA3cOZw/AAi\nIiIi2STrAp+7f8fdt6Zo3huoAp5J2P4McHSKY44Glrl7ba/H2AIs6+cYERERkcgY0TF8ZlYAlKVq\nD3vf+lMVft2SsL2mV1uyYxL37z5m8gCPJyIiIpLzRnrSxknAA6kazazE3Vt28b7jQ31MVVV5bBdr\nEREREckaIxr43P1Bglmyu2pT+HViwvZJvdqSHZO4f/cxG3ejFhEREZGckHVj+AawgiCkJY69Ox74\nV4pjngTmmFnP6VszmwLs088xIiIiIpGRlevwpeLucTP7MfA5M3sUeA14N/A2gtAHgJktBq5y9z8T\nnEJ+HfiJmV1K0MN4HfAq8ODI/gQiIiIiIy+rAp+Z7Ql4eLMAyDez7jF9H3P3m4H/BoqBOwgmZDjw\nXnfvPXPXgPEA7t5pZm8HfgasJBi39yDwdnfvHN6fSERERCTzYvH4rsx1EBEREZFckWtj+EREREQk\nTQp8o4iZPWpmtwxiv6+Z2f+ORE1JHnulmX07/P4qM1uaiTpk8MwsbmYfHaL76vn/l9xhZnuaWYuZ\nvTXTtQCY2YlhPXMyXYsMjfC14ap+2vV+MYCsGsMnfZlZKbAOKAJmhlcIGQkLgd+GNcSBNmB676uV\nhG1lBLOmx7r7kK9Z6O7fBvTmn0Hh5KgTgPZwUxuwBrgVuNrdNSYk4sLnwFp3vyBJ2+PAUne/kGBs\ndVZw98fIonpkBzP7N8Hz6T0J2+8DDiZ4r4n32v5fwBVAc3/3m/h+YWZnAevdfafLtI5W6uHLbhcA\n64E3gItG4gHNbBxwJH0XyN4S1pLofQTXPpZo+6O7F7t7MTAB+BzwX6S+frWISCp3AqeaWWH3hrBz\n4y3AWOCwhP3PBO5lx4fOwfoGcPhu1Bk56uHLbp8m6EnZDnzKzH7U/ckn7Hm7ArgEWOPup5qZAT8i\nCGxjgJeBzyZ8womZ2X8DF4a37wAuc/ft4e1TCD6xr+51zN8JAudPEuq7ELgL6Dmdl6w3oHcvgJkV\nhzW+k2Am9Sbg1yTpLTKzrwMfdfeZZjabYB3GdwBfAQ4hCMLnAh8EPknwfP61u38pxe9TdpO7dwD3\nmtkSgmtb9xFePvHbwPvZsSD6de7+k177LAS+A+wHbAjbr0v2eGb2RYLn+QnuvmSIfxwZIr3+Pt/m\n7g+a2eHANcChBH+XrwJfcPfHzewk4BHg7cDVBKsqrAU+4e4Phfc3FbiW4OpMY4ElwBfDxfsxsxuA\nEuAJ4PNAJcG6qhe6++ZejzHX3ZeGZyN+ALwXKAyPu9zdlw3fb0VSuJPgNeJ4gv8jCN531gMvEgS8\nFwDMbCLB+9l1wDFAgZn9FPgAwXvczcCl4WocX2fH+8VaYAbwCzO7zN0PMbMS4HvA2cBUglU7rnb3\nm4b/R84O6uHLUmZ2AnAAcCPBk3oWwanW3i4G3kWwDiHAnwl642YRPKFXAH9NOOYsghfXPYCTCZ78\n3+rVfho7X/7ub8BeZja/V31zCD493ZHmj/ZZgj/0w929lKCX8PIkP1sqVwLnE/x8ecDDQD0wM7yv\nL5rZQWnWJINkZiVm9j6CsPfHJLtcTvDh4BSgHLgMuLZ7bJeZHUjwnPkxwZv0BcB3zSzZ6cIPAl8E\nFirs5Zw/EISqKQTLZ90F/MHM8nvt8wWCD34Tw/Y7zawibPt1eNw8gl7le4G/9mqH4PWrCtiX4LXy\n8PA+k/kVQfg8lCAIbAH+YWZ6Dxxh7v4qwXvT23ttPpMg/P0z/L7b6UAXcE94+yKC96cpBK/3lyTs\n3/0YM8NvP+nuh4Tf/4rgog3dr01fBn5rZifu/k+VG9TDl70+Ddzr7usAzOwO4FMEL3zd7nP3Rb1u\nHwt0dPfWhRMvPmRmU929+zJy63r1trwaTuJ4J8GnZAgCX+Kpum3AnwgC5vPhtg8TBMx0T+mOJ/gD\n3g7g7s+F9Q12LNit7r4SwMweAM5z92vDtofNrBrYn6BHQYbG+Wb23vD7MQTj+L4KPJVk3+uA37l7\nXXj7H+H/yVEEL+gXA6+7+61h+9Nm9i4gcXzoacBPCdbLfGlIfxrZFb2fA72NAZINlB9P8DxpD/+2\nrw7/EZyIAOCn7r4i3PZN4DPAGcD/EvTc57t7U9h+K8Eb9P7A0+HxHcDX3L0LWG1mjxGMAesj7CU6\nD3iXu68Pt11JcAqxmAHGhsmwuIsgqHW/75xBENZfAX5kZhPCMetnAI+5e334vHnK3e8Mj7nHzLYQ\nhP27+nswM5tA0Ct4hrsvDzffbmZ3Ah8DHhu6Hy17KfBlITObRnAFkXN7bf41wam0Pd19Vbgt8XTE\nscBXzewAghey7k+vvQcvv5ZwzJsEvX2Y2VyCnrJHk5T1W+A+M7sCaAU+FP7LT7Jvf35O8KltvZn9\nk+DT2q3A5kEev7LX983AqoT2ZoJTPTJ0/th9ij48ZXsAwaflIwg+Zfc2nuAF+1SCHjwIJh11Pwfn\nEny67+HuiT3KhxJ86PiJuz8xVD+E7Jae50Bv4XCNZD5P8Ld+sZk9SPCGfFfCYvdvdH/j7nVmVk/4\nWgQcCHwnPKtQ1uuY3q9lK8Kw160ZmJ6klr0IXqd6nnfuvgm4LUXtMvzuBD5jZnsRvF7PBB4Inwdr\ngIVhh8VC+k7cW55wP9sZ3OScuQTvh3eFw6G65bHjA0TkqTs7O32CYJzJjWZWH74Q/h/B/9clvfZr\n6/7GzOYR/BE9BcwJB9i/I8l9J+tJ676ayWnA0+7emLhDeCWTNQRB9CSCT+6DvRZxTyh09zXufmh4\nH48TjL9708wWDPK+uga4LcPI3Tvc/WWCU+vvDZ93vd1GENhOBkrD5+HGXu2dDPy6czJB7/HlSe5f\ncoC730Jw6vQzBB8Qf0vQA9/7A2Lih8UY0BVOHLsfqAYOcPcikvTcMfi//e6Qqfe77PEYwVCc08N/\n/+51VuC+cNsCgnHAd/Y6bldf77vHqB/dPQEt/DfG3UfNKV39AWSZcObSxwkmNhzCjnEnhxIMdL/Y\nzIqSHHo4wemV7/ZavuWoZA+RcHseQZCD5OP3evsNQbf4BcDvU+yzHSjt9fPkEXzC7r491sxK3P1Z\nd/8uMB94iaC3UHLP2ITbxwC/d/c33L3LgsslTu3VvoRgzFUPMzs7XEKh20/c/SKCF/7bUjzfJYuZ\nWZW7N7n7He7+KYKB9ycSvKZ1m9tr/wkEvcOrCSbzjAeu6TUUJdlr2WAtJzj92/O8M7NJZvb58HFl\nhLl7O8HwpFMIPuD1Hqp0L3Bq2PZa92n/3bScIPj3mQFsZrPCsxajwqj5QXPIu4HJwLXuvqZ3g5ld\nRzDOIfE0Guw4vXu8md1P0Lv3lnDbLHacCp1jZp8gCGwHEIS368In/VuB7/ZT2y3A1wh6Fuen2Gcx\n8IHwjX4Twbib3s+zvwE1Zna5u28G9iE4jZORhZ4lfeGEnW8AzxDMBO9tGXCUmY0hmNhxNcFzb1bY\n/mvgs2b2KYJenwOAG+g7brS7R+ajBB8GfkwwflVygJnNApaa2YUEPbVdBBO1WgiGYHRPqvqMmb1A\nMJzjqwTjge8lGFDfCRxnZq8SvI51r9nW/TwaNHdvMLM/AF8xs2cJepy/RRAofrwrP6MMiTsJVn4o\nAL7ea/tDBJNxPsTOkw7T0QzMM7PK8FTxb4CvmdmLBGMFjwNuJ+iFvrWf+4kM9fBln08Df08MewBh\nQPorSd783P1Zgh7A3xEsdXEGQXh8nGBwa3f4+192LIdxf3h/PyDomekEnktVWNhzeB/wQvfg5yT+\nm2DCxCKCN/9adky9h2AplyJgsZk1E7zA3wL8MtXjSsadb8FVC1rMbDvBGM83CCZUJJ5iuYTgDb0e\nuIlg/M21wPvN7JfhbNvTCJbR6R6q8G13vznxQd29HvgP4GMpJgxIFgqXdDqP4LR/HVBDcNbi7ITF\n268n+AC4hWDG5llhr+AGgjfhL4dtlxKE/9uAXyWb0T0IlxCM1XqB4LVvT4Lnb2e/R8lwuodgnG8H\nOyYDEg4pepKgR/bO5IcOynUEqwS8Ht6+gmAs6b1AI8E45K/2mkAWebF4XAvli4jIyEhcIy/D5YiM\nGurhExEREYk4BT4RERGRiNMpXREREZGIUw+fiIiISMQp8ImIiIhEnAKfiIiISMQp8ImIiIhEnAKf\niIiISMQp8ImIiIhEnK6lKyIyCGY2Hvgf4CyCyxDeQnDpwj8RXK7wS8CBBNcG/hbwc3e/yswqCS5f\n+A5gArAe+CPwNXdvC+97JfC0u7+/1+N9lODaw3u5+0ozuwE4iuByUT8G5gFrgG8kuzSdiEhv6uET\nERmc/yEIbR9nx7Wnv5WwTxVwOnACcE247a5w28cIrg/6VYLrw/50F2qYTnCN2Y8DhwFPADeY2RG7\ncF8iMoqoh09EZABmVgq8F7jG3f8Ubv6smS0A5vbadRaw0N0Xh8cdCxwHvNfd7wr3WW5m+wFXmNnn\n3L0pjVIqgCvd/fnw/j8FnAucDzy7iz+eiIwC6uETERnYLKAQeCFh+90Jt5u7w15oQfj1iYT9/g2M\nAfZPs46W3jW4+zbAgTlp3o+IjDIKfCIiA5sYfk3sjatOuN2YcLsi/Lo1xX7ladbR6O6J18NsAirT\nvB8RGWUU+EREBtYafi1N2D4xcccEDeHXcQnbu2/Xh1/jQCxhn2RhcGySbeVA7QB1iMgop8AnIjKw\nZQShbH7C9ncPcNxz4dfjE7YfCzQDi8Lb9QQzeBP3SVRqZj01mNlYwIA3BqhDREY5TdoQERmAu9eZ\n2QPAJWb2NMG4uUsY4JSsuz9lZv8EfmRmTcAS4BTg08CP3X17uOuzwIfN7B0EIfBdwAFJ7rIpvK8v\nEfQefplgbOEtu/sziki0qYdPRGRwLiKYbPG/wKMEPXTXhm0t/Rz3TuA+gvX53gC+AHwDuKrXPl8D\nHgJuJQh/+4TbEm0DvgP8CniRYHmYD7q7evhEpF+xeDxx/K+IiCQys2JgrLvX9tr2PeAKoMTdO4f5\n8W8ATnf3qcP5OCISTTqlKyIyOL8DTjGzCwlOu84HPgHcONxhT0RkdynwiYgMzieA7xNc7mwSsBa4\nHvhmJosSERmM/9++HZAAAAAwCOvf2h6ypRDOTboAAHNOGwAAc4IPAGBO8AEAzAk+AIA5wQcAMCf4\nAADmAkuHnvhvfbKAAAAAAElFTkSuQmCC\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f3bf6d93668>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"retrain_model(load_embeddings('data/numberbatch-en-17.04b.txt'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So have we entirely fixed the problem by switching to ConceptNet Numberbatch? Can we stop worrying about algorithmic racism? **No.**\n",
"\n",
"Have we made the problem a lot smaller? **Definitely.**\n",
"\n",
"The ranges of sentiments overlap a lot more than they did in the word vectors that came directly from GloVe or word2vec. The F-value is less than a third of what it was for GloVe, and a quarter of what it was for word2vec. And in general, we see much smaller differences in sentiment that come from comparing different given names, which is what we'd hope for, because names really shouldn't matter to the task of sentiment analysis.\n",
"\n",
"But there is still a small correlation. Maybe I could have picked some data or training parameters that made the problem look completely solved. That would have been a bad move, because the problem _isn't_ completely solved. There are more causes of algorithmic racism than the ones we have identified and compensated for in ConceptNet. But this is a good start."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### There is no trade-off\n",
"\n",
"Note that the accuracy of sentiment prediction went *up* when we switched to ConceptNet Numberbatch.\n",
"\n",
"Some people expect that fighting algorithmic racism is going to come with some sort of trade-off. There's no trade-off here. You can have data that's better and less racist. You can have data that's better *because* it's less racist. There was never anything \"accurate\" about the overt racism that word2vec and GloVe learned."
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Other approaches\n",
"\n",
"This is of course only one way to do sentiment analysis. All the steps we used are common, but you probably object that you wouldn't do it that way. But if you have your own process, I urge you to see if your process is encoding prejudices and biases in the model it learns.\n",
"\n",
"Instead of or in addition to changing your source of word vectors, you could try to fix this problem in the output directly. It may help, for example, to build a stronger model of whether sentiment should be assigned to words at all, designed to specifically exclude names and groups of people.\n",
"\n",
"You could abandon the idea of inferring sentiment for words, and only count the sentiment of words that appear exactly in the list. This is perhaps the most common form of sentiment analysis -- the kind that includes no machine learning at all. Its results will be no more biased than whoever made the list. But the lack of machine learning means that this approach has low recall, and the only way to adapt it to your data set is to edit the list manually.\n",
"\n",
"As a hybrid approach, you could produce a large number of inferred sentiments for words, and have a human annotator patiently look through them, making a list of exceptions whose sentiment should be set to 0. The downside of this is that it's extra work; the upside is that you take the time to actually see what your data is doing. And that's something that I think should happen more often in machine learning anyway.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment