Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@pb111

pb111/Naïve Bayes Classification with Python and Scikit-Learn.ipynb

Created May 20, 2019 15:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pb111/9e3816d2584a85ef7bff8d70bed20b1b to your computer and use it in GitHub Desktop.
Save pb111/9e3816d2584a85ef7bff8d70bed20b1b to your computer and use it in GitHub Desktop.
Download ZIP
Naïve Bayes Classification with Python and Scikit-Learn
Display the source blob
Display the rendered blob
Raw
Naïve Bayes Classification with Python and Scikit-Learn.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Naive Bayes Classification with Python and Scikit-Learn\n",
"\n",
"\n",
"In this project, I implement Naive Bayes Classification algorithm with Python and Scikit-Learn. I build a Naive Bayes Classifier to predict whether a person makes over 50K a year. I have used the **Adult Data Set** for this project. I have downloaded this dataset from the UCI Machine Learning Repository website. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Table of Contents\n",
"\n",
"1.\tIntroduction to Naive Bayes Classification algorithm\n",
"2.\tNaive Bayes algorithm intuition\n",
"3.\tThe problem statement\n",
"4.\tDataset description\n",
"5.\tImport libraries\n",
"6.\tImport dataset\n",
"7.\tExploratory data analysis\n",
"8.\tDeclare feature vector and target variable\n",
"9.\tSplit data into separate training and test set\n",
"10.\tFeature engineering\n",
"11.\tFeature scaling\n",
"12.\tModel training\n",
"13.\tPredict the test-set results\n",
"14.\tCheck the accuracy score\n",
"15.\tConfusion matrix\n",
"16.\tClassification metrices\n",
"17.\tCalculate class probabilities\n",
"18.\tROC - AUC\n",
"19.\tk-Fold Cross Validation\n",
"20.\tResults and conclusion\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Introduction to Naive Bayes Classification algorithm\n",
"\n",
"\n",
"In machine learning, Naïve Bayes classification is a straightforward and powerful algorithm for the classification task. Naïve Bayes classification is based on applying Bayes’ theorem with strong independence assumption between the features. Naïve Bayes classification produces good results when we use it for textual data analysis such as Natural Language Processing.\n",
"\n",
"\n",
"Naïve Bayes models are also known as `simple Bayes` or `independent Bayes`. All these names refer to the application of Bayes’ theorem in the classifier’s decision rule. Naïve Bayes classifier applies the Bayes’ theorem in practice. This classifier brings the power of Bayes’ theorem to machine learning.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Naive Bayes algorithm intuition\n",
"\n",
"\n",
"Naïve Bayes Classifier uses the Bayes’ theorem to predict membership probabilities for each class such as the probability that given record or data point belongs to a particular class. The class with the highest probability is considered as the most likely class. This is also known as the **Maximum A Posteriori (MAP)**. \n",
"\n",
"The **MAP for a hypothesis with 2 events A and B is**\n",
"\n",
"**MAP (A)**\n",
"\n",
"= max (P (A | B))\n",
"\n",
"= max (P (B | A) * P (A))/P (B)\n",
"\n",
"= max (P (B | A) * P (A))\n",
"\n",
"\n",
"Here, P (B) is evidence probability. It is used to normalize the result. It remains the same, So, removing it would not affect the result.\n",
"\n",
"\n",
"Naïve Bayes Classifier assumes that all the features are unrelated to each other. Presence or absence of a feature does not influence the presence or absence of any other feature. \n",
"\n",
"\n",
"In real world datasets, we test a hypothesis given multiple evidence on features. So, the calculations become quite complicated. To simplify the work, the feature independence approach is used to uncouple multiple evidence and treat each as an independent one.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. The problem statement\n",
"\n",
"\n",
"In this project, I try to make predictions where the prediction task is to determine whether a person makes over 50K a year. I implement Naive Bayes Classification with Python and Scikit-Learn. So, to answer the question, I build a Naive Bayes classifier to predict whether a person makes over 50K a year."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Dataset description\n",
"\n",
"\n",
"I have used the **Adult Data Set** for this project. I have downloaded this dataset from the UCI Machine Learning Repository website. The data set can be found at the following url:-\n",
"\n",
"\n",
"https://archive.ics.uci.edu/ml/datasets/Adult\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import warnings\n",
"\n",
"warnings.filterwarnings('ignore')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Import dataset"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"data = 'C:/datasets/adult.data'\n",
"\n",
"df = pd.read_csv(data, header=None, sep=',\\s')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Exploratory data analysis\n",
"\n",
"\n",
"Now, I will explore the data to gain insights about the data. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(32561, 15)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# view dimensions of dataset\n",
"\n",
"df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are 32561 instances and 15 attributes in the data set."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### View top 5 rows of dataset"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" <th>4</th>\n",
" <th>5</th>\n",
" <th>6</th>\n",
" <th>7</th>\n",
" <th>8</th>\n",
" <th>9</th>\n",
" <th>10</th>\n",
" <th>11</th>\n",
" <th>12</th>\n",
" <th>13</th>\n",
" <th>14</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>39</td>\n",
" <td>State-gov</td>\n",
" <td>77516</td>\n",
" <td>Bachelors</td>\n",
" <td>13</td>\n",
" <td>Never-married</td>\n",
" <td>Adm-clerical</td>\n",
" <td>Not-in-family</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>2174</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>50</td>\n",
" <td>Self-emp-not-inc</td>\n",
" <td>83311</td>\n",
" <td>Bachelors</td>\n",
" <td>13</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Exec-managerial</td>\n",
" <td>Husband</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>13</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>38</td>\n",
" <td>Private</td>\n",
" <td>215646</td>\n",
" <td>HS-grad</td>\n",
" <td>9</td>\n",
" <td>Divorced</td>\n",
" <td>Handlers-cleaners</td>\n",
" <td>Not-in-family</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>53</td>\n",
" <td>Private</td>\n",
" <td>234721</td>\n",
" <td>11th</td>\n",
" <td>7</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Handlers-cleaners</td>\n",
" <td>Husband</td>\n",
" <td>Black</td>\n",
" <td>Male</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>28</td>\n",
" <td>Private</td>\n",
" <td>338409</td>\n",
" <td>Bachelors</td>\n",
" <td>13</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Prof-specialty</td>\n",
" <td>Wife</td>\n",
" <td>Black</td>\n",
" <td>Female</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" <td>Cuba</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3 4 5 \\\n",
"0 39 State-gov 77516 Bachelors 13 Never-married \n",
"1 50 Self-emp-not-inc 83311 Bachelors 13 Married-civ-spouse \n",
"2 38 Private 215646 HS-grad 9 Divorced \n",
"3 53 Private 234721 11th 7 Married-civ-spouse \n",
"4 28 Private 338409 Bachelors 13 Married-civ-spouse \n",
"\n",
" 6 7 8 9 10 11 12 \\\n",
"0 Adm-clerical Not-in-family White Male 2174 0 40 \n",
"1 Exec-managerial Husband White Male 0 0 13 \n",
"2 Handlers-cleaners Not-in-family White Male 0 0 40 \n",
"3 Handlers-cleaners Husband Black Male 0 0 40 \n",
"4 Prof-specialty Wife Black Female 0 0 40 \n",
"\n",
" 13 14 \n",
"0 United-States <=50K \n",
"1 United-States <=50K \n",
"2 United-States <=50K \n",
"3 United-States <=50K \n",
"4 Cuba <=50K "
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# preview the dataset\n",
"\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Rename column names\n",
"\n",
"We can see that the dataset does not have proper column names. The columns are merely labelled as 0,1,2.... and so on. We should give proper names to the columns. I will do it as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Index(['age', 'workclass', 'fnlwgt', 'education', 'education_num',\n",
" 'marital_status', 'occupation', 'relationship', 'race', 'sex',\n",
" 'capital_gain', 'capital_loss', 'hours_per_week', 'native_country',\n",
" 'income'],\n",
" dtype='object')"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"col_names = ['age', 'workclass', 'fnlwgt', 'education', 'education_num', 'marital_status', 'occupation', 'relationship',\n",
" 'race', 'sex', 'capital_gain', 'capital_loss', 'hours_per_week', 'native_country', 'income']\n",
"\n",
"df.columns = col_names\n",
"\n",
"df.columns"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>age</th>\n",
" <th>workclass</th>\n",
" <th>fnlwgt</th>\n",
" <th>education</th>\n",
" <th>education_num</th>\n",
" <th>marital_status</th>\n",
" <th>occupation</th>\n",
" <th>relationship</th>\n",
" <th>race</th>\n",
" <th>sex</th>\n",
" <th>capital_gain</th>\n",
" <th>capital_loss</th>\n",
" <th>hours_per_week</th>\n",
" <th>native_country</th>\n",
" <th>income</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>39</td>\n",
" <td>State-gov</td>\n",
" <td>77516</td>\n",
" <td>Bachelors</td>\n",
" <td>13</td>\n",
" <td>Never-married</td>\n",
" <td>Adm-clerical</td>\n",
" <td>Not-in-family</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>2174</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>50</td>\n",
" <td>Self-emp-not-inc</td>\n",
" <td>83311</td>\n",
" <td>Bachelors</td>\n",
" <td>13</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Exec-managerial</td>\n",
" <td>Husband</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>13</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>38</td>\n",
" <td>Private</td>\n",
" <td>215646</td>\n",
" <td>HS-grad</td>\n",
" <td>9</td>\n",
" <td>Divorced</td>\n",
" <td>Handlers-cleaners</td>\n",
" <td>Not-in-family</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>53</td>\n",
" <td>Private</td>\n",
" <td>234721</td>\n",
" <td>11th</td>\n",
" <td>7</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Handlers-cleaners</td>\n",
" <td>Husband</td>\n",
" <td>Black</td>\n",
" <td>Male</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>28</td>\n",
" <td>Private</td>\n",
" <td>338409</td>\n",
" <td>Bachelors</td>\n",
" <td>13</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Prof-specialty</td>\n",
" <td>Wife</td>\n",
" <td>Black</td>\n",
" <td>Female</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" <td>Cuba</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" age workclass fnlwgt education education_num \\\n",
"0 39 State-gov 77516 Bachelors 13 \n",
"1 50 Self-emp-not-inc 83311 Bachelors 13 \n",
"2 38 Private 215646 HS-grad 9 \n",
"3 53 Private 234721 11th 7 \n",
"4 28 Private 338409 Bachelors 13 \n",
"\n",
" marital_status occupation relationship race sex \\\n",
"0 Never-married Adm-clerical Not-in-family White Male \n",
"1 Married-civ-spouse Exec-managerial Husband White Male \n",
"2 Divorced Handlers-cleaners Not-in-family White Male \n",
"3 Married-civ-spouse Handlers-cleaners Husband Black Male \n",
"4 Married-civ-spouse Prof-specialty Wife Black Female \n",
"\n",
" capital_gain capital_loss hours_per_week native_country income \n",
"0 2174 0 40 United-States <=50K \n",
"1 0 0 13 United-States <=50K \n",
"2 0 0 40 United-States <=50K \n",
"3 0 0 40 United-States <=50K \n",
"4 0 0 40 Cuba <=50K "
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# let's agian preview the dataset\n",
"\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the column names are renamed. Now, the columns have meaningful names."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### View summary of dataset"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 32561 entries, 0 to 32560\n",
"Data columns (total 15 columns):\n",
"age 32561 non-null int64\n",
"workclass 32561 non-null object\n",
"fnlwgt 32561 non-null int64\n",
"education 32561 non-null object\n",
"education_num 32561 non-null int64\n",
"marital_status 32561 non-null object\n",
"occupation 32561 non-null object\n",
"relationship 32561 non-null object\n",
"race 32561 non-null object\n",
"sex 32561 non-null object\n",
"capital_gain 32561 non-null int64\n",
"capital_loss 32561 non-null int64\n",
"hours_per_week 32561 non-null int64\n",
"native_country 32561 non-null object\n",
"income 32561 non-null object\n",
"dtypes: int64(6), object(9)\n",
"memory usage: 3.7+ MB\n"
]
}
],
"source": [
"# view summary of dataset\n",
"\n",
"df.info()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are no missing values in the dataset. I will confirm this further."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Types of variables\n",
"\n",
"\n",
"In this section, I segregate the dataset into categorical and numerical variables. There are a mixture of categorical and numerical variables in the dataset. Categorical variables have data type object. Numerical variables have data type int64.\n",
"\n",
"\n",
"First of all, I will explore categorical variables."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explore categorical variables"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"There are 9 categorical variables\n",
"\n",
"The categorical variables are :\n",
"\n",
" ['workclass', 'education', 'marital_status', 'occupation', 'relationship', 'race', 'sex', 'native_country', 'income']\n"
]
}
],
"source": [
"# find categorical variables\n",
"\n",
"categorical = [var for var in df.columns if df[var].dtype=='O']\n",
"\n",
"print('There are {} categorical variables\\n'.format(len(categorical)))\n",
"\n",
"print('The categorical variables are :\\n\\n', categorical)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>workclass</th>\n",
" <th>education</th>\n",
" <th>marital_status</th>\n",
" <th>occupation</th>\n",
" <th>relationship</th>\n",
" <th>race</th>\n",
" <th>sex</th>\n",
" <th>native_country</th>\n",
" <th>income</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>State-gov</td>\n",
" <td>Bachelors</td>\n",
" <td>Never-married</td>\n",
" <td>Adm-clerical</td>\n",
" <td>Not-in-family</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Self-emp-not-inc</td>\n",
" <td>Bachelors</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Exec-managerial</td>\n",
" <td>Husband</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Private</td>\n",
" <td>HS-grad</td>\n",
" <td>Divorced</td>\n",
" <td>Handlers-cleaners</td>\n",
" <td>Not-in-family</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Private</td>\n",
" <td>11th</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Handlers-cleaners</td>\n",
" <td>Husband</td>\n",
" <td>Black</td>\n",
" <td>Male</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Private</td>\n",
" <td>Bachelors</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Prof-specialty</td>\n",
" <td>Wife</td>\n",
" <td>Black</td>\n",
" <td>Female</td>\n",
" <td>Cuba</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" workclass education marital_status occupation \\\n",
"0 State-gov Bachelors Never-married Adm-clerical \n",
"1 Self-emp-not-inc Bachelors Married-civ-spouse Exec-managerial \n",
"2 Private HS-grad Divorced Handlers-cleaners \n",
"3 Private 11th Married-civ-spouse Handlers-cleaners \n",
"4 Private Bachelors Married-civ-spouse Prof-specialty \n",
"\n",
" relationship race sex native_country income \n",
"0 Not-in-family White Male United-States <=50K \n",
"1 Husband White Male United-States <=50K \n",
"2 Not-in-family White Male United-States <=50K \n",
"3 Husband Black Male United-States <=50K \n",
"4 Wife Black Female Cuba <=50K "
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# view the categorical variables\n",
"\n",
"df[categorical].head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Summary of categorical variables\n",
"\n",
"\n",
"- There are 9 categorical variables. \n",
"\n",
"\n",
"- The categorical variables are given by `workclass`, `education`, `marital_status`, `occupation`, `relationship`, `race`, `sex`, `native_country` and `income`.\n",
"\n",
"\n",
"- `income` is the target variable."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explore problems within categorical variables\n",
"\n",
"\n",
"First, I will explore the categorical variables.\n",
"\n",
"\n",
"### Missing values in categorical variables"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"workclass 0\n",
"education 0\n",
"marital_status 0\n",
"occupation 0\n",
"relationship 0\n",
"race 0\n",
"sex 0\n",
"native_country 0\n",
"income 0\n",
"dtype: int64"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check missing values in categorical variables\n",
"\n",
"df[categorical].isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are no missing values in the categorical variables. I will confirm this further."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Frequency counts of categorical variables\n",
"\n",
"\n",
"Now, I will check the frequency counts of categorical variables."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Private 22696\n",
"Self-emp-not-inc 2541\n",
"Local-gov 2093\n",
"? 1836\n",
"State-gov 1298\n",
"Self-emp-inc 1116\n",
"Federal-gov 960\n",
"Without-pay 14\n",
"Never-worked 7\n",
"Name: workclass, dtype: int64\n",
"HS-grad 10501\n",
"Some-college 7291\n",
"Bachelors 5355\n",
"Masters 1723\n",
"Assoc-voc 1382\n",
"11th 1175\n",
"Assoc-acdm 1067\n",
"10th 933\n",
"7th-8th 646\n",
"Prof-school 576\n",
"9th 514\n",
"12th 433\n",
"Doctorate 413\n",
"5th-6th 333\n",
"1st-4th 168\n",
"Preschool 51\n",
"Name: education, dtype: int64\n",
"Married-civ-spouse 14976\n",
"Never-married 10683\n",
"Divorced 4443\n",
"Separated 1025\n",
"Widowed 993\n",
"Married-spouse-absent 418\n",
"Married-AF-spouse 23\n",
"Name: marital_status, dtype: int64\n",
"Prof-specialty 4140\n",
"Craft-repair 4099\n",
"Exec-managerial 4066\n",
"Adm-clerical 3770\n",
"Sales 3650\n",
"Other-service 3295\n",
"Machine-op-inspct 2002\n",
"? 1843\n",
"Transport-moving 1597\n",
"Handlers-cleaners 1370\n",
"Farming-fishing 994\n",
"Tech-support 928\n",
"Protective-serv 649\n",
"Priv-house-serv 149\n",
"Armed-Forces 9\n",
"Name: occupation, dtype: int64\n",
"Husband 13193\n",
"Not-in-family 8305\n",
"Own-child 5068\n",
"Unmarried 3446\n",
"Wife 1568\n",
"Other-relative 981\n",
"Name: relationship, dtype: int64\n",
"White 27816\n",
"Black 3124\n",
"Asian-Pac-Islander 1039\n",
"Amer-Indian-Eskimo 311\n",
"Other 271\n",
"Name: race, dtype: int64\n",
"Male 21790\n",
"Female 10771\n",
"Name: sex, dtype: int64\n",
"United-States 29170\n",
"Mexico 643\n",
"? 583\n",
"Philippines 198\n",
"Germany 137\n",
"Canada 121\n",
"Puerto-Rico 114\n",
"El-Salvador 106\n",
"India 100\n",
"Cuba 95\n",
"England 90\n",
"Jamaica 81\n",
"South 80\n",
"China 75\n",
"Italy 73\n",
"Dominican-Republic 70\n",
"Vietnam 67\n",
"Guatemala 64\n",
"Japan 62\n",
"Poland 60\n",
"Columbia 59\n",
"Taiwan 51\n",
"Haiti 44\n",
"Iran 43\n",
"Portugal 37\n",
"Nicaragua 34\n",
"Peru 31\n",
"France 29\n",
"Greece 29\n",
"Ecuador 28\n",
"Ireland 24\n",
"Hong 20\n",
"Trinadad&Tobago 19\n",
"Cambodia 19\n",
"Thailand 18\n",
"Laos 18\n",
"Yugoslavia 16\n",
"Outlying-US(Guam-USVI-etc) 14\n",
"Honduras 13\n",
"Hungary 13\n",
"Scotland 12\n",
"Holand-Netherlands 1\n",
"Name: native_country, dtype: int64\n",
"<=50K 24720\n",
">50K 7841\n",
"Name: income, dtype: int64\n"
]
}
],
"source": [
"# view frequency counts of values in categorical variables\n",
"\n",
"for var in categorical: \n",
" \n",
" print(df[var].value_counts())"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Private 0.697030\n",
"Self-emp-not-inc 0.078038\n",
"Local-gov 0.064279\n",
"? 0.056386\n",
"State-gov 0.039864\n",
"Self-emp-inc 0.034274\n",
"Federal-gov 0.029483\n",
"Without-pay 0.000430\n",
"Never-worked 0.000215\n",
"Name: workclass, dtype: float64\n",
"HS-grad 0.322502\n",
"Some-college 0.223918\n",
"Bachelors 0.164461\n",
"Masters 0.052916\n",
"Assoc-voc 0.042443\n",
"11th 0.036086\n",
"Assoc-acdm 0.032769\n",
"10th 0.028654\n",
"7th-8th 0.019840\n",
"Prof-school 0.017690\n",
"9th 0.015786\n",
"12th 0.013298\n",
"Doctorate 0.012684\n",
"5th-6th 0.010227\n",
"1st-4th 0.005160\n",
"Preschool 0.001566\n",
"Name: education, dtype: float64\n",
"Married-civ-spouse 0.459937\n",
"Never-married 0.328092\n",
"Divorced 0.136452\n",
"Separated 0.031479\n",
"Widowed 0.030497\n",
"Married-spouse-absent 0.012837\n",
"Married-AF-spouse 0.000706\n",
"Name: marital_status, dtype: float64\n",
"Prof-specialty 0.127146\n",
"Craft-repair 0.125887\n",
"Exec-managerial 0.124873\n",
"Adm-clerical 0.115783\n",
"Sales 0.112097\n",
"Other-service 0.101195\n",
"Machine-op-inspct 0.061485\n",
"? 0.056601\n",
"Transport-moving 0.049046\n",
"Handlers-cleaners 0.042075\n",
"Farming-fishing 0.030527\n",
"Tech-support 0.028500\n",
"Protective-serv 0.019932\n",
"Priv-house-serv 0.004576\n",
"Armed-Forces 0.000276\n",
"Name: occupation, dtype: float64\n",
"Husband 0.405178\n",
"Not-in-family 0.255060\n",
"Own-child 0.155646\n",
"Unmarried 0.105832\n",
"Wife 0.048156\n",
"Other-relative 0.030128\n",
"Name: relationship, dtype: float64\n",
"White 0.854274\n",
"Black 0.095943\n",
"Asian-Pac-Islander 0.031909\n",
"Amer-Indian-Eskimo 0.009551\n",
"Other 0.008323\n",
"Name: race, dtype: float64\n",
"Male 0.669205\n",
"Female 0.330795\n",
"Name: sex, dtype: float64\n",
"United-States 0.895857\n",
"Mexico 0.019748\n",
"? 0.017905\n",
"Philippines 0.006081\n",
"Germany 0.004207\n",
"Canada 0.003716\n",
"Puerto-Rico 0.003501\n",
"El-Salvador 0.003255\n",
"India 0.003071\n",
"Cuba 0.002918\n",
"England 0.002764\n",
"Jamaica 0.002488\n",
"South 0.002457\n",
"China 0.002303\n",
"Italy 0.002242\n",
"Dominican-Republic 0.002150\n",
"Vietnam 0.002058\n",
"Guatemala 0.001966\n",
"Japan 0.001904\n",
"Poland 0.001843\n",
"Columbia 0.001812\n",
"Taiwan 0.001566\n",
"Haiti 0.001351\n",
"Iran 0.001321\n",
"Portugal 0.001136\n",
"Nicaragua 0.001044\n",
"Peru 0.000952\n",
"France 0.000891\n",
"Greece 0.000891\n",
"Ecuador 0.000860\n",
"Ireland 0.000737\n",
"Hong 0.000614\n",
"Trinadad&Tobago 0.000584\n",
"Cambodia 0.000584\n",
"Thailand 0.000553\n",
"Laos 0.000553\n",
"Yugoslavia 0.000491\n",
"Outlying-US(Guam-USVI-etc) 0.000430\n",
"Honduras 0.000399\n",
"Hungary 0.000399\n",
"Scotland 0.000369\n",
"Holand-Netherlands 0.000031\n",
"Name: native_country, dtype: float64\n",
"<=50K 0.75919\n",
">50K 0.24081\n",
"Name: income, dtype: float64\n"
]
}
],
"source": [
"# view frequency distribution of categorical variables\n",
"\n",
"for var in categorical: \n",
" \n",
" print(df[var].value_counts()/np.float(len(df)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we can see that there are several variables like `workclass`, `occupation` and `native_country` which contain missing values. Generally, the missing values are coded as `NaN` and python will detect them with the usual command of `df.isnull().sum()`.\n",
"\n",
"But, in this case the missing values are coded as `?`. Python fail to detect these as missing values because it do not consider `?` as missing values. So, I have to replace `?` with `NaN` so that Python can detect these missing values.\n",
"\n",
"I will explore these variables and replace `?` with `NaN`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explore workclass variable"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['State-gov', 'Self-emp-not-inc', 'Private', 'Federal-gov',\n",
" 'Local-gov', '?', 'Self-emp-inc', 'Without-pay', 'Never-worked'],\n",
" dtype=object)"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check labels in workclass variable\n",
"\n",
"df.workclass.unique()"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Private 22696\n",
"Self-emp-not-inc 2541\n",
"Local-gov 2093\n",
"? 1836\n",
"State-gov 1298\n",
"Self-emp-inc 1116\n",
"Federal-gov 960\n",
"Without-pay 14\n",
"Never-worked 7\n",
"Name: workclass, dtype: int64"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check frequency distribution of values in workclass variable\n",
"\n",
"df.workclass.value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are 1836 values encoded as `?` in workclass variable. I will replace these `?` with `NaN`."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"# replace '?' values in workclass variable with `NaN`\n",
"\n",
"\n",
"df['workclass'].replace('?', np.NaN, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"Private 22696\n",
"Self-emp-not-inc 2541\n",
"Local-gov 2093\n",
"State-gov 1298\n",
"Self-emp-inc 1116\n",
"Federal-gov 960\n",
"Without-pay 14\n",
"Never-worked 7\n",
"Name: workclass, dtype: int64"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# again check the frequency distribution of values in workclass variable\n",
"\n",
"df.workclass.value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we can see that there are no values encoded as `?` in the `workclass` variable.\n",
"\n",
"I will adopt similar approach with `occupation` and `native_country` column."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explore occupation variable"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['Adm-clerical', 'Exec-managerial', 'Handlers-cleaners',\n",
" 'Prof-specialty', 'Other-service', 'Sales', 'Craft-repair',\n",
" 'Transport-moving', 'Farming-fishing', 'Machine-op-inspct',\n",
" 'Tech-support', '?', 'Protective-serv', 'Armed-Forces',\n",
" 'Priv-house-serv'], dtype=object)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check labels in occupation variable\n",
"\n",
"df.occupation.unique()\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Prof-specialty 4140\n",
"Craft-repair 4099\n",
"Exec-managerial 4066\n",
"Adm-clerical 3770\n",
"Sales 3650\n",
"Other-service 3295\n",
"Machine-op-inspct 2002\n",
"? 1843\n",
"Transport-moving 1597\n",
"Handlers-cleaners 1370\n",
"Farming-fishing 994\n",
"Tech-support 928\n",
"Protective-serv 649\n",
"Priv-house-serv 149\n",
"Armed-Forces 9\n",
"Name: occupation, dtype: int64"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check frequency distribution of values in occupation variable\n",
"\n",
"df.occupation.value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are 1843 values encoded as `?` in `occupation` variable. I will replace these `?` with `NaN`."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"# replace '?' values in occupation variable with `NaN`\n",
"\n",
"df['occupation'].replace('?', np.NaN, inplace=True)\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Prof-specialty 4140\n",
"Craft-repair 4099\n",
"Exec-managerial 4066\n",
"Adm-clerical 3770\n",
"Sales 3650\n",
"Other-service 3295\n",
"Machine-op-inspct 2002\n",
"Transport-moving 1597\n",
"Handlers-cleaners 1370\n",
"Farming-fishing 994\n",
"Tech-support 928\n",
"Protective-serv 649\n",
"Priv-house-serv 149\n",
"Armed-Forces 9\n",
"Name: occupation, dtype: int64"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# again check the frequency distribution of values in occupation variable\n",
"\n",
"df.occupation.value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explore native_country variable\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['United-States', 'Cuba', 'Jamaica', 'India', '?', 'Mexico',\n",
" 'South', 'Puerto-Rico', 'Honduras', 'England', 'Canada', 'Germany',\n",
" 'Iran', 'Philippines', 'Italy', 'Poland', 'Columbia', 'Cambodia',\n",
" 'Thailand', 'Ecuador', 'Laos', 'Taiwan', 'Haiti', 'Portugal',\n",
" 'Dominican-Republic', 'El-Salvador', 'France', 'Guatemala',\n",
" 'China', 'Japan', 'Yugoslavia', 'Peru',\n",
" 'Outlying-US(Guam-USVI-etc)', 'Scotland', 'Trinadad&Tobago',\n",
" 'Greece', 'Nicaragua', 'Vietnam', 'Hong', 'Ireland', 'Hungary',\n",
" 'Holand-Netherlands'], dtype=object)"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check labels in native_country variable\n",
"\n",
"df.native_country.unique()\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"United-States 29170\n",
"Mexico 643\n",
"? 583\n",
"Philippines 198\n",
"Germany 137\n",
"Canada 121\n",
"Puerto-Rico 114\n",
"El-Salvador 106\n",
"India 100\n",
"Cuba 95\n",
"England 90\n",
"Jamaica 81\n",
"South 80\n",
"China 75\n",
"Italy 73\n",
"Dominican-Republic 70\n",
"Vietnam 67\n",
"Guatemala 64\n",
"Japan 62\n",
"Poland 60\n",
"Columbia 59\n",
"Taiwan 51\n",
"Haiti 44\n",
"Iran 43\n",
"Portugal 37\n",
"Nicaragua 34\n",
"Peru 31\n",
"France 29\n",
"Greece 29\n",
"Ecuador 28\n",
"Ireland 24\n",
"Hong 20\n",
"Trinadad&Tobago 19\n",
"Cambodia 19\n",
"Thailand 18\n",
"Laos 18\n",
"Yugoslavia 16\n",
"Outlying-US(Guam-USVI-etc) 14\n",
"Honduras 13\n",
"Hungary 13\n",
"Scotland 12\n",
"Holand-Netherlands 1\n",
"Name: native_country, dtype: int64"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check frequency distribution of values in native_country variable\n",
"\n",
"df.native_country.value_counts()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are 583 values encoded as `?` in `native_country` variable. I will replace these `?` with `NaN`.\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"# replace '?' values in native_country variable with `NaN`\n",
"\n",
"df['native_country'].replace('?', np.NaN, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"United-States 29170\n",
"Mexico 643\n",
"Philippines 198\n",
"Germany 137\n",
"Canada 121\n",
"Puerto-Rico 114\n",
"El-Salvador 106\n",
"India 100\n",
"Cuba 95\n",
"England 90\n",
"Jamaica 81\n",
"South 80\n",
"China 75\n",
"Italy 73\n",
"Dominican-Republic 70\n",
"Vietnam 67\n",
"Guatemala 64\n",
"Japan 62\n",
"Poland 60\n",
"Columbia 59\n",
"Taiwan 51\n",
"Haiti 44\n",
"Iran 43\n",
"Portugal 37\n",
"Nicaragua 34\n",
"Peru 31\n",
"France 29\n",
"Greece 29\n",
"Ecuador 28\n",
"Ireland 24\n",
"Hong 20\n",
"Trinadad&Tobago 19\n",
"Cambodia 19\n",
"Thailand 18\n",
"Laos 18\n",
"Yugoslavia 16\n",
"Outlying-US(Guam-USVI-etc) 14\n",
"Honduras 13\n",
"Hungary 13\n",
"Scotland 12\n",
"Holand-Netherlands 1\n",
"Name: native_country, dtype: int64"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# again check the frequency distribution of values in native_country variable\n",
"\n",
"df.native_country.value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Check missing values in categorical variables again"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"workclass 1836\n",
"education 0\n",
"marital_status 0\n",
"occupation 1843\n",
"relationship 0\n",
"race 0\n",
"sex 0\n",
"native_country 583\n",
"income 0\n",
"dtype: int64"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df[categorical].isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we can see that `workclass`, `occupation` and `native_country` variable contains missing values."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Number of labels: cardinality\n",
"\n",
"\n",
"The number of labels within a categorical variable is known as **cardinality**. A high number of labels within a variable is known as **high cardinality**. High cardinality may pose some serious problems in the machine learning model. So, I will check for high cardinality."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"workclass contains 9 labels\n",
"education contains 16 labels\n",
"marital_status contains 7 labels\n",
"occupation contains 15 labels\n",
"relationship contains 6 labels\n",
"race contains 5 labels\n",
"sex contains 2 labels\n",
"native_country contains 42 labels\n",
"income contains 2 labels\n"
]
}
],
"source": [
"# check for cardinality in categorical variables\n",
"\n",
"for var in categorical:\n",
" \n",
" print(var, ' contains ', len(df[var].unique()), ' labels')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that `native_country` column contains relatively large number of labels as compared to other columns. I will check for cardinality after train-test split."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explore Numerical Variables"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"There are 6 numerical variables\n",
"\n",
"The numerical variables are : ['age', 'fnlwgt', 'education_num', 'capital_gain', 'capital_loss', 'hours_per_week']\n"
]
}
],
"source": [
"# find numerical variables\n",
"\n",
"numerical = [var for var in df.columns if df[var].dtype!='O']\n",
"\n",
"print('There are {} numerical variables\\n'.format(len(numerical)))\n",
"\n",
"print('The numerical variables are :', numerical)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>age</th>\n",
" <th>fnlwgt</th>\n",
" <th>education_num</th>\n",
" <th>capital_gain</th>\n",
" <th>capital_loss</th>\n",
" <th>hours_per_week</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>39</td>\n",
" <td>77516</td>\n",
" <td>13</td>\n",
" <td>2174</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>50</td>\n",
" <td>83311</td>\n",
" <td>13</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>13</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>38</td>\n",
" <td>215646</td>\n",
" <td>9</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>53</td>\n",
" <td>234721</td>\n",
" <td>7</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>28</td>\n",
" <td>338409</td>\n",
" <td>13</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" age fnlwgt education_num capital_gain capital_loss hours_per_week\n",
"0 39 77516 13 2174 0 40\n",
"1 50 83311 13 0 0 13\n",
"2 38 215646 9 0 0 40\n",
"3 53 234721 7 0 0 40\n",
"4 28 338409 13 0 0 40"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# view the numerical variables\n",
"\n",
"df[numerical].head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Summary of numerical variables\n",
"\n",
"\n",
"- There are 6 numerical variables. \n",
"\n",
"\n",
"- These are given by `age`, `fnlwgt`, `education_num`, `capital_gain`, `capital_loss` and `hours_per_week`.\n",
"\n",
"\n",
"- All of the numerical variables are of discrete data type."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explore problems within numerical variables\n",
"\n",
"\n",
"Now, I will explore the numerical variables.\n",
"\n",
"\n",
"### Missing values in numerical variables"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"age 0\n",
"fnlwgt 0\n",
"education_num 0\n",
"capital_gain 0\n",
"capital_loss 0\n",
"hours_per_week 0\n",
"dtype: int64"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check missing values in numerical variables\n",
"\n",
"df[numerical].isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that all the 6 numerical variables do not contain missing values. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. Declare feature vector and target variable"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"X = df.drop(['income'], axis=1)\n",
"\n",
"y = df['income']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 9. Split data into separate training and test set"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
"# split X and y into training and testing sets\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)\n"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"((22792, 14), (9769, 14))"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check the shape of X_train and X_test\n",
"\n",
"X_train.shape, X_test.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 10. Feature Engineering\n",
"\n",
"\n",
"**Feature Engineering** is the process of transforming raw data into useful features that help us to understand our model better and increase its predictive power. I will carry out feature engineering on different types of variables.\n",
"\n",
"\n",
"First, I will display the categorical and numerical variables again separately."
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"age int64\n",
"workclass object\n",
"fnlwgt int64\n",
"education object\n",
"education_num int64\n",
"marital_status object\n",
"occupation object\n",
"relationship object\n",
"race object\n",
"sex object\n",
"capital_gain int64\n",
"capital_loss int64\n",
"hours_per_week int64\n",
"native_country object\n",
"dtype: object"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check data types in X_train\n",
"\n",
"X_train.dtypes"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['workclass',\n",
" 'education',\n",
" 'marital_status',\n",
" 'occupation',\n",
" 'relationship',\n",
" 'race',\n",
" 'sex',\n",
" 'native_country']"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# display categorical variables\n",
"\n",
"categorical = [col for col in X_train.columns if X_train[col].dtypes == 'O']\n",
"\n",
"categorical"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['age',\n",
" 'fnlwgt',\n",
" 'education_num',\n",
" 'capital_gain',\n",
" 'capital_loss',\n",
" 'hours_per_week']"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# display numerical variables\n",
"\n",
"numerical = [col for col in X_train.columns if X_train[col].dtypes != 'O']\n",
"\n",
"numerical"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Engineering missing values in categorical variables"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"workclass 0.055985\n",
"education 0.000000\n",
"marital_status 0.000000\n",
"occupation 0.056072\n",
"relationship 0.000000\n",
"race 0.000000\n",
"sex 0.000000\n",
"native_country 0.018164\n",
"dtype: float64"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# print percentage of missing values in the categorical variables in training set\n",
"\n",
"X_train[categorical].isnull().mean()"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"workclass 0.055984555984555984\n",
"occupation 0.05607230607230607\n",
"native_country 0.018164268164268166\n"
]
}
],
"source": [
"# print categorical variables with missing data\n",
"\n",
"for col in categorical:\n",
" if X_train[col].isnull().mean()>0:\n",
" print(col, (X_train[col].isnull().mean()))"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [],
"source": [
"# impute missing categorical variables with most frequent value\n",
"\n",
"for df2 in [X_train, X_test]:\n",
" df2['workclass'].fillna(X_train['workclass'].mode()[0], inplace=True)\n",
" df2['occupation'].fillna(X_train['occupation'].mode()[0], inplace=True)\n",
" df2['native_country'].fillna(X_train['native_country'].mode()[0], inplace=True) "
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"workclass 0\n",
"education 0\n",
"marital_status 0\n",
"occupation 0\n",
"relationship 0\n",
"race 0\n",
"sex 0\n",
"native_country 0\n",
"dtype: int64"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check missing values in categorical variables in X_train\n",
"\n",
"X_train[categorical].isnull().sum()"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"workclass 0\n",
"education 0\n",
"marital_status 0\n",
"occupation 0\n",
"relationship 0\n",
"race 0\n",
"sex 0\n",
"native_country 0\n",
"dtype: int64"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check missing values in categorical variables in X_test\n",
"\n",
"X_test[categorical].isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As a final check, I will check for missing values in X_train and X_test."
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"age 0\n",
"workclass 0\n",
"fnlwgt 0\n",
"education 0\n",
"education_num 0\n",
"marital_status 0\n",
"occupation 0\n",
"relationship 0\n",
"race 0\n",
"sex 0\n",
"capital_gain 0\n",
"capital_loss 0\n",
"hours_per_week 0\n",
"native_country 0\n",
"dtype: int64"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check missing values in X_train\n",
"\n",
"X_train.isnull().sum()"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"age 0\n",
"workclass 0\n",
"fnlwgt 0\n",
"education 0\n",
"education_num 0\n",
"marital_status 0\n",
"occupation 0\n",
"relationship 0\n",
"race 0\n",
"sex 0\n",
"capital_gain 0\n",
"capital_loss 0\n",
"hours_per_week 0\n",
"native_country 0\n",
"dtype: int64"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check missing values in X_test\n",
"\n",
"X_test.isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are no missing values in X_train and X_test."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Encode categorical variables"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['workclass',\n",
" 'education',\n",
" 'marital_status',\n",
" 'occupation',\n",
" 'relationship',\n",
" 'race',\n",
" 'sex',\n",
" 'native_country']"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# print categorical variables\n",
"\n",
"categorical"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>workclass</th>\n",
" <th>education</th>\n",
" <th>marital_status</th>\n",
" <th>occupation</th>\n",
" <th>relationship</th>\n",
" <th>race</th>\n",
" <th>sex</th>\n",
" <th>native_country</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>32098</th>\n",
" <td>Private</td>\n",
" <td>HS-grad</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Craft-repair</td>\n",
" <td>Husband</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>United-States</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25206</th>\n",
" <td>State-gov</td>\n",
" <td>HS-grad</td>\n",
" <td>Divorced</td>\n",
" <td>Adm-clerical</td>\n",
" <td>Unmarried</td>\n",
" <td>White</td>\n",
" <td>Female</td>\n",
" <td>United-States</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23491</th>\n",
" <td>Private</td>\n",
" <td>Some-college</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Sales</td>\n",
" <td>Husband</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>United-States</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12367</th>\n",
" <td>Private</td>\n",
" <td>HS-grad</td>\n",
" <td>Never-married</td>\n",
" <td>Craft-repair</td>\n",
" <td>Not-in-family</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>Guatemala</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7054</th>\n",
" <td>Private</td>\n",
" <td>7th-8th</td>\n",
" <td>Never-married</td>\n",
" <td>Craft-repair</td>\n",
" <td>Not-in-family</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>Germany</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" workclass education marital_status occupation \\\n",
"32098 Private HS-grad Married-civ-spouse Craft-repair \n",
"25206 State-gov HS-grad Divorced Adm-clerical \n",
"23491 Private Some-college Married-civ-spouse Sales \n",
"12367 Private HS-grad Never-married Craft-repair \n",
"7054 Private 7th-8th Never-married Craft-repair \n",
"\n",
" relationship race sex native_country \n",
"32098 Husband White Male United-States \n",
"25206 Unmarried White Female United-States \n",
"23491 Husband White Male United-States \n",
"12367 Not-in-family White Male Guatemala \n",
"7054 Not-in-family White Male Germany "
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_train[categorical].head()"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [],
"source": [
"# import category encoders\n",
"\n",
"import category_encoders as ce"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [],
"source": [
"# encode remaining variables with one-hot encoding\n",
"\n",
"encoder = ce.OneHotEncoder(cols=['workclass', 'education', 'marital_status', 'occupation', 'relationship', \n",
" 'race', 'sex', 'native_country'])\n",
"\n",
"X_train = encoder.fit_transform(X_train)\n",
"\n",
"X_test = encoder.transform(X_test)"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>workclass_1</th>\n",
" <th>workclass_2</th>\n",
" <th>workclass_3</th>\n",
" <th>workclass_4</th>\n",
" <th>workclass_5</th>\n",
" <th>workclass_6</th>\n",
" <th>workclass_7</th>\n",
" <th>workclass_8</th>\n",
" <th>workclass_-1</th>\n",
" <th>education_1</th>\n",
" <th>...</th>\n",
" <th>native_country_39</th>\n",
" <th>native_country_40</th>\n",
" <th>native_country_41</th>\n",
" <th>native_country_-1</th>\n",
" <th>age</th>\n",
" <th>fnlwgt</th>\n",
" <th>education_num</th>\n",
" <th>capital_gain</th>\n",
" <th>capital_loss</th>\n",
" <th>hours_per_week</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>32098</th>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>45</td>\n",
" <td>170871</td>\n",
" <td>9</td>\n",
" <td>7298</td>\n",
" <td>0</td>\n",
" <td>60</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25206</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>47</td>\n",
" <td>108890</td>\n",
" <td>9</td>\n",
" <td>1831</td>\n",
" <td>0</td>\n",
" <td>38</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23491</th>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>48</td>\n",
" <td>187505</td>\n",
" <td>10</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>50</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12367</th>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>29</td>\n",
" <td>145592</td>\n",
" <td>9</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7054</th>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>23</td>\n",
" <td>203003</td>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>25</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 113 columns</p>\n",
"</div>"
],
"text/plain": [
" workclass_1 workclass_2 workclass_3 workclass_4 workclass_5 \\\n",
"32098 1 0 0 0 0 \n",
"25206 0 1 0 0 0 \n",
"23491 1 0 0 0 0 \n",
"12367 1 0 0 0 0 \n",
"7054 1 0 0 0 0 \n",
"\n",
" workclass_6 workclass_7 workclass_8 workclass_-1 education_1 \\\n",
"32098 0 0 0 0 1 \n",
"25206 0 0 0 0 1 \n",
"23491 0 0 0 0 0 \n",
"12367 0 0 0 0 1 \n",
"7054 0 0 0 0 0 \n",
"\n",
" ... native_country_39 native_country_40 \\\n",
"32098 ... 0 0 \n",
"25206 ... 0 0 \n",
"23491 ... 0 0 \n",
"12367 ... 0 0 \n",
"7054 ... 0 0 \n",
"\n",
" native_country_41 native_country_-1 age fnlwgt education_num \\\n",
"32098 0 0 45 170871 9 \n",
"25206 0 0 47 108890 9 \n",
"23491 0 0 48 187505 10 \n",
"12367 0 0 29 145592 9 \n",
"7054 0 0 23 203003 4 \n",
"\n",
" capital_gain capital_loss hours_per_week \n",
"32098 7298 0 60 \n",
"25206 1831 0 38 \n",
"23491 0 0 50 \n",
"12367 0 0 40 \n",
"7054 0 0 25 \n",
"\n",
"[5 rows x 113 columns]"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_train.head()"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(22792, 113)"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_train.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that from the initial 14 columns, we now have 113 columns."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similarly, I will take a look at the `X_test` set."
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>workclass_1</th>\n",
" <th>workclass_2</th>\n",
" <th>workclass_3</th>\n",
" <th>workclass_4</th>\n",
" <th>workclass_5</th>\n",
" <th>workclass_6</th>\n",
" <th>workclass_7</th>\n",
" <th>workclass_8</th>\n",
" <th>workclass_-1</th>\n",
" <th>education_1</th>\n",
" <th>...</th>\n",
" <th>native_country_39</th>\n",
" <th>native_country_40</th>\n",
" <th>native_country_41</th>\n",
" <th>native_country_-1</th>\n",
" <th>age</th>\n",
" <th>fnlwgt</th>\n",
" <th>education_num</th>\n",
" <th>capital_gain</th>\n",
" <th>capital_loss</th>\n",
" <th>hours_per_week</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>22278</th>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>27</td>\n",
" <td>177119</td>\n",
" <td>10</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>44</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8950</th>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>27</td>\n",
" <td>216481</td>\n",
" <td>13</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7838</th>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>25</td>\n",
" <td>256263</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16505</th>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>46</td>\n",
" <td>147640</td>\n",
" <td>3</td>\n",
" <td>0</td>\n",
" <td>1902</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19140</th>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>45</td>\n",
" <td>172822</td>\n",
" <td>7</td>\n",
" <td>0</td>\n",
" <td>2824</td>\n",
" <td>76</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 113 columns</p>\n",
"</div>"
],
"text/plain": [
" workclass_1 workclass_2 workclass_3 workclass_4 workclass_5 \\\n",
"22278 1 0 0 0 0 \n",
"8950 1 0 0 0 0 \n",
"7838 1 0 0 0 0 \n",
"16505 1 0 0 0 0 \n",
"19140 1 0 0 0 0 \n",
"\n",
" workclass_6 workclass_7 workclass_8 workclass_-1 education_1 \\\n",
"22278 0 0 0 0 0 \n",
"8950 0 0 0 0 0 \n",
"7838 0 0 0 0 0 \n",
"16505 0 0 0 0 0 \n",
"19140 0 0 0 0 0 \n",
"\n",
" ... native_country_39 native_country_40 \\\n",
"22278 ... 0 0 \n",
"8950 ... 0 0 \n",
"7838 ... 0 0 \n",
"16505 ... 0 0 \n",
"19140 ... 0 0 \n",
"\n",
" native_country_41 native_country_-1 age fnlwgt education_num \\\n",
"22278 0 0 27 177119 10 \n",
"8950 0 0 27 216481 13 \n",
"7838 0 0 25 256263 12 \n",
"16505 0 0 46 147640 3 \n",
"19140 0 0 45 172822 7 \n",
"\n",
" capital_gain capital_loss hours_per_week \n",
"22278 0 0 44 \n",
"8950 0 0 40 \n",
"7838 0 0 40 \n",
"16505 0 1902 40 \n",
"19140 0 2824 76 \n",
"\n",
"[5 rows x 113 columns]"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_test.head()"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(9769, 113)"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_test.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We now have training and testing set ready for model building. Before that, we should map all the feature variables onto the same scale. It is called `feature scaling`. I will do it as follows."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 11. Feature Scaling"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [],
"source": [
"cols = X_train.columns"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.preprocessing import RobustScaler\n",
"\n",
"scaler = RobustScaler()\n",
"\n",
"X_train = scaler.fit_transform(X_train)\n",
"\n",
"X_test = scaler.transform(X_test)\n"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [],
"source": [
"X_train = pd.DataFrame(X_train, columns=[cols])"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [],
"source": [
"X_test = pd.DataFrame(X_test, columns=[cols])"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th></th>\n",
" <th>workclass_1</th>\n",
" <th>workclass_2</th>\n",
" <th>workclass_3</th>\n",
" <th>workclass_4</th>\n",
" <th>workclass_5</th>\n",
" <th>workclass_6</th>\n",
" <th>workclass_7</th>\n",
" <th>workclass_8</th>\n",
" <th>workclass_-1</th>\n",
" <th>education_1</th>\n",
" <th>...</th>\n",
" <th>native_country_39</th>\n",
" <th>native_country_40</th>\n",
" <th>native_country_41</th>\n",
" <th>native_country_-1</th>\n",
" <th>age</th>\n",
" <th>fnlwgt</th>\n",
" <th>education_num</th>\n",
" <th>capital_gain</th>\n",
" <th>capital_loss</th>\n",
" <th>hours_per_week</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.40</td>\n",
" <td>-0.058906</td>\n",
" <td>-0.333333</td>\n",
" <td>7298.0</td>\n",
" <td>0.0</td>\n",
" <td>4.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.50</td>\n",
" <td>-0.578076</td>\n",
" <td>-0.333333</td>\n",
" <td>1831.0</td>\n",
" <td>0.0</td>\n",
" <td>-0.4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.55</td>\n",
" <td>0.080425</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>2.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>-0.40</td>\n",
" <td>-0.270650</td>\n",
" <td>-0.333333</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>-0.70</td>\n",
" <td>0.210240</td>\n",
" <td>-2.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>-3.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 113 columns</p>\n",
"</div>"
],
"text/plain": [
" workclass_1 workclass_2 workclass_3 workclass_4 workclass_5 workclass_6 \\\n",
"0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"1 -1.0 1.0 0.0 0.0 0.0 0.0 \n",
"2 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"3 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"4 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"\n",
" workclass_7 workclass_8 workclass_-1 education_1 ... \\\n",
"0 0.0 0.0 0.0 1.0 ... \n",
"1 0.0 0.0 0.0 1.0 ... \n",
"2 0.0 0.0 0.0 0.0 ... \n",
"3 0.0 0.0 0.0 1.0 ... \n",
"4 0.0 0.0 0.0 0.0 ... \n",
"\n",
" native_country_39 native_country_40 native_country_41 native_country_-1 \\\n",
"0 0.0 0.0 0.0 0.0 \n",
"1 0.0 0.0 0.0 0.0 \n",
"2 0.0 0.0 0.0 0.0 \n",
"3 0.0 0.0 0.0 0.0 \n",
"4 0.0 0.0 0.0 0.0 \n",
"\n",
" age fnlwgt education_num capital_gain capital_loss hours_per_week \n",
"0 0.40 -0.058906 -0.333333 7298.0 0.0 4.0 \n",
"1 0.50 -0.578076 -0.333333 1831.0 0.0 -0.4 \n",
"2 0.55 0.080425 0.000000 0.0 0.0 2.0 \n",
"3 -0.40 -0.270650 -0.333333 0.0 0.0 0.0 \n",
"4 -0.70 0.210240 -2.000000 0.0 0.0 -3.0 \n",
"\n",
"[5 rows x 113 columns]"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_train.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We now have `X_train` dataset ready to be fed into the Gaussian Naive Bayes classifier. I will do it as follows."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 12. Model training"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"GaussianNB(priors=None, var_smoothing=1e-09)"
]
},
"execution_count": 57,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# train a Gaussian Naive Bayes classifier on the training set\n",
"from sklearn.naive_bayes import GaussianNB\n",
"\n",
"\n",
"# instantiate the model\n",
"gnb = GaussianNB()\n",
"\n",
"\n",
"# fit the model\n",
"gnb.fit(X_train, y_train)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 13. Predict the test set results"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['<=50K', '<=50K', '>50K', ..., '>50K', '<=50K', '<=50K'],\n",
" dtype='<U5')"
]
},
"execution_count": 58,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_pred = gnb.predict(X_test)\n",
"\n",
"y_pred"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 14. Check accuracy score"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model accuracy score: 0.8083\n"
]
}
],
"source": [
"from sklearn.metrics import accuracy_score\n",
"\n",
"print('Model accuracy score: {0:0.4f}'. format(accuracy_score(y_test, y_pred)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, **y_test** are the true class labels and **y_pred** are the predicted class labels in the test-set."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Compare the train-set and test-set accuracy\n",
"\n",
"\n",
"Now, I will compare the train-set and test-set accuracy to check for overfitting."
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['>50K', '<=50K', '>50K', ..., '<=50K', '>50K', '<=50K'],\n",
" dtype='<U5')"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_pred_train = gnb.predict(X_train)\n",
"\n",
"y_pred_train"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training-set accuracy score: 0.8067\n"
]
}
],
"source": [
"print('Training-set accuracy score: {0:0.4f}'. format(accuracy_score(y_train, y_pred_train)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Check for overfitting and underfitting"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training set score: 0.8067\n",
"Test set score: 0.8083\n"
]
}
],
"source": [
"# print the scores on training and test set\n",
"\n",
"print('Training set score: {:.4f}'.format(gnb.score(X_train, y_train)))\n",
"\n",
"print('Test set score: {:.4f}'.format(gnb.score(X_test, y_test)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The training-set accuracy score is 0.8067 while the test-set accuracy to be 0.8083. These two values are quite comparable. So, there is no sign of overfitting. \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Compare model accuracy with null accuracy\n",
"\n",
"\n",
"So, the model accuracy is 0.8083. But, we cannot say that our model is very good based on the above accuracy. We must compare it with the **null accuracy**. Null accuracy is the accuracy that could be achieved by always predicting the most frequent class.\n",
"\n",
"So, we should first check the class distribution in the test set. "
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<=50K 7407\n",
">50K 2362\n",
"Name: income, dtype: int64"
]
},
"execution_count": 63,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check class distribution in test set\n",
"\n",
"y_test.value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the occurences of most frequent class is 7407. So, we can calculate null accuracy by dividing 7407 by total number of occurences."
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Null accuracy score: 0.7582\n"
]
}
],
"source": [
"# check null accuracy score\n",
"\n",
"null_accuracy = (7407/(7407+2362))\n",
"\n",
"print('Null accuracy score: {0:0.4f}'. format(null_accuracy))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that our model accuracy score is 0.8083 but null accuracy score is 0.7582. So, we can conclude that our Gaussian Naive Bayes Classification model is doing a very good job in predicting the class labels."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, based on the above analysis we can conclude that our classification model accuracy is very good. Our model is doing a very good job in terms of predicting the class labels.\n",
"\n",
"\n",
"But, it does not give the underlying distribution of values. Also, it does not tell anything about the type of errors our classifer is making. \n",
"\n",
"\n",
"We have another tool called `Confusion matrix` that comes to our rescue."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 15. Confusion matrix\n",
"\n",
"\n",
"A confusion matrix is a tool for summarizing the performance of a classification algorithm. A confusion matrix will give us a clear picture of classification model performance and the types of errors produced by the model. It gives us a summary of correct and incorrect predictions broken down by each category. The summary is represented in a tabular form.\n",
"\n",
"\n",
"Four types of outcomes are possible while evaluating a classification model performance. These four outcomes are described below:-\n",
"\n",
"\n",
"**True Positives (TP)** – True Positives occur when we predict an observation belongs to a certain class and the observation actually belongs to that class.\n",
"\n",
"\n",
"**True Negatives (TN)** – True Negatives occur when we predict an observation does not belong to a certain class and the observation actually does not belong to that class.\n",
"\n",
"\n",
"**False Positives (FP)** – False Positives occur when we predict an observation belongs to a certain class but the observation actually does not belong to that class. This type of error is called **Type I error.**\n",
"\n",
"\n",
"\n",
"**False Negatives (FN)** – False Negatives occur when we predict an observation does not belong to a certain class but the observation actually belongs to that class. This is a very serious error and it is called **Type II error.**\n",
"\n",
"\n",
"\n",
"These four outcomes are summarized in a confusion matrix given below.\n"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Confusion matrix\n",
"\n",
" [[5999 1408]\n",
" [ 465 1897]]\n",
"\n",
"True Positives(TP) = 5999\n",
"\n",
"True Negatives(TN) = 1897\n",
"\n",
"False Positives(FP) = 1408\n",
"\n",
"False Negatives(FN) = 465\n"
]
}
],
"source": [
"# Print the Confusion Matrix and slice it into four pieces\n",
"\n",
"from sklearn.metrics import confusion_matrix\n",
"\n",
"cm = confusion_matrix(y_test, y_pred)\n",
"\n",
"print('Confusion matrix\\n\\n', cm)\n",
"\n",
"print('\\nTrue Positives(TP) = ', cm[0,0])\n",
"\n",
"print('\\nTrue Negatives(TN) = ', cm[1,1])\n",
"\n",
"print('\\nFalse Positives(FP) = ', cm[0,1])\n",
"\n",
"print('\\nFalse Negatives(FN) = ', cm[1,0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The confusion matrix shows `5999 + 1897 = 7896 correct predictions` and `1408 + 465 = 1873 incorrect predictions`.\n",
"\n",
"\n",
"In this case, we have\n",
"\n",
"\n",
"- `True Positives` (Actual Positive:1 and Predict Positive:1) - 5999\n",
"\n",
"\n",
"- `True Negatives` (Actual Negative:0 and Predict Negative:0) - 1897\n",
"\n",
"\n",
"- `False Positives` (Actual Negative:0 but Predict Positive:1) - 1408 `(Type I error)`\n",
"\n",
"\n",
"- `False Negatives` (Actual Positive:1 but Predict Negative:0) - 465 `(Type II error)`"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.axes._subplots.AxesSubplot at 0xf202a9c6a0>"
]
},
"execution_count": 66,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAWYAAAENCAYAAAA8Fc+PAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XecFeXZ//HPdxcUVLqAKIqKGEs0CWLXPMaCvSTRiBqDLZhH/GmKjWhi7KZoLE80YohiRU1MLLFhQbFgQwUVFTUWFEVFESvt+v0x9+KCu2dnlz27s4fvO695nZn7nDNzjTlc3Fxzzz2KCMzMrDiqWjsAMzNblBOzmVnBODGbmRWME7OZWcE4MZuZFYwTs5lZwTgxm5kVjBOzmVnBODGbmRVMu3IfoONq+/nWQvuaF57fv7VDsALqt8LuWtJ9NCbnfP7GtUt8vHJwj9nMrGDK3mM2M2tJUtvvbzoxm1lFqVLbT2tt/wzMzGpxj9nMrGCkQl7PaxQnZjOrMO4xm5kViksZZmYF44t/ZmYF4x6zmVnBODGbmRWME7OZWcEID5czMysU95jNzAqmqqrtp7W2fwZmZotwj9nMrFBcyjAzKxgnZjOzgpFLGWZmxeIes5lZwVRVVbd2CEvMidnMKopLGWZmBeNShplZwTgxm5kVjEsZZmYFI9+SbWZWLH4Yq5lZwbiUYWZWML74Z2ZWNC5lmJkVTNvvMDsxm1mFqWr7mdmJ2cwqS9vPy07MZlZZwjVmM7OCaft52YnZzCpMVdvPzBVQjTEzq0XKvzS4K70mabKkpyU9kdq6SxoraWp67ZbaJekCSS9LmiRpYK39DE2fnyppaEPHdWI2s8pSrfxLPt+LiG9HxKC0fQJwT0QMAO5J2wA7AwPSMgy4GLJEDpwMbApsApxck8zr48RsZpWlGXvM9dgTGJ3WRwN71Wq/IjITgK6S+gA7AmMjYmZEfAiMBXYqdQAnZjOrLGrE0rAA7pL0pKRhqa13REwHSK+9UvsqwJu1vjsttdXXXi9f/DOzytKIi38p2Q6r1TQyIkbW2t4yIt6W1AsYK+mFUruroy1KtNfLidnMKksjKhQpCY8s8f7b6XWGpH+R1YjfldQnIqanUsWM9PFpwKq1vt4XeDu1b7NY+7hScbmUYWYVJaqrci+lSFpeUqeadWAw8CxwM1AzsmIocFNavxn4SRqdsRkwK5U67gQGS+qWLvoNTm31co/ZzCpL8w1j7g38K0283w64JiLukPQ4cL2kQ4E3gH3S528DdgFeBj4DDgaIiJmSTgMeT587NSJmljqwE7OZVZZmuiU7Il4FvlVH+wfAdnW0BzC8nn39Hfh73mM7MZtZZamAO/+cmM2ssrT9vOzEbGYVxrPLmZkVTP5brQvLidnMKot7zPbCQxcw+9PPmT9/AfPmL2Cr3U5kg3VX48IzD2X55Tvw+rT3OPiovzD7k89p376a/zvrMAZuuCYLFgTH/G404ydMAWDv3TfjuCO/T3V1FXfc+xQnnnlNK5+ZLYlzTrmOCeOfp2v3Fbj0+mMXee+GK8Zx6fm3csPdp9Cl2/JEBBf98SYef2gKy3ZYhmN+ty8D1u0LwKXn38pjD05hwYJg4KZrc8Sxe6IKSDxlVQH/eXyDSTPYad/T2WznEWy124kAXPyHYZx09hg2Hnw8N9/xBL84fDcADtlvWwA2Hnw8ux1wJmf/5sdIonvXFTjz1wewy36ns9H2x9JrxS5ss+X6rXY+tuR22H0QZ17406+1z3jnIyY++hK9Vuq6sO3xh17grTff47J/n8DPT9qbC876JwDPPfMazz3zGn8d8ytGXn8MLz3/JpOefKXFzqGtiirlXorKibkMBqzZhwcfzXrC946fxF67bALAOgP6ct9DzwHw3gcfM+vjz9howzVZY7VeTP3vdN6fOTv7zoOT2WvnTVsneGsWGw7sT6cuy32t/a/n3sRhR++2SK/34fufY4ddByGJdTfox6effMEH732MBHO+nMu8ufOZO2ce8+bNp1uPTi15Gm1T+WeXK7smJWZJv23uQNqqiOCWq0bw0H/O4JD9sx7x8y9OY7cdNgLgB7tuRt8+PQCYPOV1dh+8EdXVVfRbtSff+eYa9F25B6+8/i7f6L8yq/VdkerqKvYYPIi+K3dvtXOy8njk/udYsWcX+q+98iLtH8yYRc/eX/WgV+zVhQ/em8V6G67OtwetxZAdT2HIjqcyaPNvsNoavVs67LaneWeXaxVNrTEfBpzanIG0Vdv+8HdMf/dDevbozK1X/5oXX36bw4+9hHNOGcqIn/+A/4ydyJy58wAYfd041llrFR669QzeeOt9Jjz5EvPmzeejWZ9y1Il/56q/HM2CBQuY8ORU1litVwNHtrbki8/ncM2ouzn7L8O+9l52w9hiJN56833e+O+7XHP7bwA44YiRTJr4ChsO7F/ucNu2BubAaAvqTcySPq7vLaBjqZ3WnkqvXbdBtFthrSYHWHTT3/0QyEoTN9/5OBt/uz/njfwPu//4LADWWmMldt722wDMn7+A4069cuF377vxFF5+7R0Abrt7IrfdPRGAQ/bflvkLFrTkaViZTZ/2Ae+8PZOf7XcuAO/NmMURB/yZC684ihV7d+W9dz9a+Nn3Z8yix4qduef2J1lng350XG5ZADbe4hu8MPkNJ+aGFLgnnFepv1o+AgZEROfFlk7A9FI7jYiRETEoIgZVclJeruOyrLB8h4Xr22+9Ic+9OI2ePToDIIkTjvo+l151DwAdOyzDch2zP2Tbbr0B8+bP54WpbwEs/E7XLssz7MAduOzae1v6dKyM1hjQhxvuPoUrbz2RK289kZ69unDR1b+g+4qd2fy76zH2P08QEUyZ/DrLr9CBHj0702ulbkye+Crz581n3tz5TJr4Kquu4X9JNahK+ZeCKlXKuALoB7xbx3seywX06tmF60b+EoB27aq57t8PMfb+Zxh+yE4c/pPBANx0x2Nccf04AHqu2JlbrhzBggXB2+/O5NCfX7RwX3/63VA2WG81AM4670Ze/u87LXsy1qzO/PVVTHriFWZ99Cn773waBx4+mJ33qvuC7iZbrctjD73AQXuezbId2nPM7/YFYOvtNuTpx19m2L7nIMGgLdZh8+96tE6DCpxw81Kd9a1m1HG1/cp7AGuTXnh+/9YOwQqo3wq7L3FWXfOwG3LnnFf/tk8hs7hvMDGzylIBF/9ynYGkiaW2zcwKo8JrzAtFxMBS22ZmhdH2O8y5e8z9JG2f1jvWPAfLzKxwloY7/yT9FPgHcElq6gv8u5xBmZk1WQWUMvL0mIcDWwIfA0TEVMCDKc2skELKvRRVnhrzlxExp2bSFUntAA+BM7NialfchJtXnh7z/ZJ+DXSUtANwA3BLecMyM2uipaHGDJwAvAdMBg4HbgNOKmdQZmZNVgE15jyljD2BKyLi0nIHY2a2xIqbb3PL02PeA3hJ0pWSdk01ZjOzQloqnmASEQcDa5HVlvcHXpH0t3IHZmbWJNVV+ZeCynvn31xJt5ONxuhIVt44rJyBmZk1SXHzbW55bjDZSdLlwMvA3sDfgD5ljsvMrGkqYFRGnh7zQcAY4PCI+LK84ZiZLaEC147zajAxR8SQlgjEzKxZVEBirreUIenB9Dpb0se1ltklngdoZtaqmvuWbEnVkp6SdGvaXkPSo5KmSrpO0jKpfdm0/XJ6f/Va+xiR2l+UtGNDx6w3MUfEVum10+LP/IuIzrnOyMyspVUr/5LP0cCUWtu/B/4cEQOAD4FDU/uhwIcRsRbw5/Q5JK0HDAHWB3YCLpJUXeqAeS7+XZmnzcysEJrxzj9JfYFdyQY9oGzSoG3JZtwEGA3sldb3TNuk97dLn98TGBMRX0bEf8kGUmxS8hRynOYiT39MN5hslON7ZmYtr3lvyT4POA5YkLZ7AB9FxLy0PQ1YJa2vArwJkN6flT6/sL2O79R9CvW9kWois4ENa9eXyZ6afVOeMzIza3HKv0gaJumJWsuwhbuRdgNmRMSTi+19cdHAe6W+U6d6R2VExFnAWZLOiogRpXZiZlYUjbnVOiJGAiPreXtLYA9JuwAdgM5kPeiuktqlXnFf4O30+WnAqsC0VFnoAsys1V6j9nfqVKrHvE5avUHSwMWXUjs1M2s1zXSDSUSMiIi+EbE62cW7eyPiAOA+spvtAIbyVQXh5rRNev/eiIjUPiSN2lgDGAA8VurYpcYx/xIYBpxTV8xkBXAzs2LJP9qiqY4Hxkg6HXgKGJXaRwFXSnqZrKc8BCAinpN0PfA8MA8YHhHzSx2gVCljWHr93pKehZlZS6kqw1wZETEOGJfWX6WOURUR8QWwTz3fPwM4I+/x8gyX26fmqdiSTpJ0o6Tv5D2AmVlLqoCpMnINl/tNRMyWtBWwI9k4vb+WNywzs6ZZWhJzTS1kV+DiiLgJWKZ8IZmZNZ2k3EtR5Zld7i1JlwDbA7+XtCwVMeOpmVWictSYW1qeU/gRcCewU0R8BHQHji1rVGZmTaSq/EtR5Zn28zNJrwA7plmRxkfEXeUPzcys8Qpcocgtz6iMo4GrgV5puUrS/yt3YGZmTdG8U2W0jjw15kOBTSPiUwBJvwceAS4sZ2BmZk1RCT3mPIlZfDUyg7ReAaduZpVoaUnMlwGPSvpX2t6Lr25BNDMrlKry35Jddnku/p0raRywFVlP+eCIeKrcgZmZNUVF95gldQB+BqwFTAYuqjU5tJlZIVV0Yia79XouMB7YGVgX+HlLBGVm1lSVnpjXi4gNACSNooH5Q83MiqDIw+DyKpWY59asRMS8It9XbmZWoxJSVanE/C1JH6d1AR3TtoCIiM5lj87MrJEqelRGRFS3ZCBmZs2h0nvMZmZtjhOzmVnBODGbmRVMJYzKyDO73O/ztJmZFUFVdf6lqPJMFb1DHW07N3cgZmbNoRKe+Vfqluz/BY4A+kuaVOutTsDD5Q7MzKwpKuGei1I15muA24GzgBNqtc+OiJlljcrMrIkqIC+XHMc8C5gl6XxgZkTMBpDUSdKmEfFoSwVpZpZXRSfmWi4GBtba/rSOtnp9/sYpTQjLKt34d6a2dghWQP1WWPJ9LC2JWRERNRsRsUCSh9mZWSG1K/DTr/PKcwqvSjpKUvu0HA28Wu7AzMyaokqReymqPIn5Z8AWwFvANGBTYFg5gzIza6ql4inZETEDGNICsZiZLbEKqGSUHMd8XET8QdKFwNf6/BFxVFkjMzNrgiKXKPIq1WOekl6faIlAzMyaQ3OVKNJzTx8AliXLlf+IiJMlrQGMAboDE4EDI2KOpGWBK4CNgA+AfSPitbSvEcChwHzgqIi4s9SxS41jviW9jl6y0zMzazntmq92/CWwbUR8Iqk98KCk24FfAn+OiDGS/kqWcC9Orx9GxFqShgC/B/aVtB5ZOXh9YGXgbklrR8T8es+hvjck3UIdJYwaEbFHo0/TzKzM1EyljDRM+JO02T4tAWwL7J/aRwO/I0vMe6Z1gH8A/6fs/vA9gTER8SXwX0kvA5sAj9R37FKljD+l1x8AKwFXpe39gNdynZmZWQtrTClD0jAWHWU2MiJG1nq/GngSWAv4C/AK8FFEzEsfmQasktZXAd6Ehc9JnQX0SO0Tah2j9nfqVKqUcX8K7LSI+G6tt26R9ECpnZqZtZbGjMpISXhkiffnA9+W1BX4F7BuXR9Lr3X9lRAl2uuV5xx6SlqzZiMVvnvm+J6ZWYsrxw0mEfERMA7YDOha6+7nvsDbaX0asCpAer8LMLN2ex3fqfsccsT0C2CcpHGSxgH3AT/P8T0zsxbXTvmXUiT1TD1lJHUEticbrXYfsHf62FDgprR+c9omvX9vqlPfDAyRtGzq2A4AHit5Dg2dZETcIWkAsE5qeiEVsc3MCqcZ7+jrA4xOdeYq4PqIuFXS88AYSacDTwGj0udHAVemi3szSTfmRcRzkq4HngfmAcNLjciAHIlZ0nJkw0P6RcRPJQ2Q9I2IuLVJp2pmVkbNdYNJREwCvlNH+6tkoyoWb/8C2KeefZ0BnJH32HlKGZcBc4DN0/Y04PS8BzAza0mVMFdGnsTcPyL+AMwFiIjPqfsqo5lZq6tqxFJUeeZVnpMK3wEgqT/ZHTFmZoVT6XNl1DgZuANYVdLVwJbAQeUMysysqSphovySiTndTvgC2d1/m5GVMI6OiPdbIDYzs0argLxcOjFHREj6d0RsBPynhWIyM2uySihl5PnLZYKkjcseiZlZM6iEURl5aszfA34m6TWyJ2SLrDO9YTkDMzNrioovZSQ7lz0KM7NmUl3V9ksZpeZj7kD2INa1gMnAqFpT3ZmZFVKRSxR5leoxjya7qWQ8Wa95PeDolgjKzKypKr2UsV5EbAAgaRQNzIZkZlYElTAqo1Rinluzkmbjb4FwzMyWTKWXMr4l6eO0LqBj2q4ZldG57NGZmTVSRSfmiKhuyUDMzJpD+wovZZiZtTkV3WM2M2uLnJjNzAqm2onZzKxY3GM2MyuYSh/HbGbW5rR3j9nMrFhcyjAzKxiXMszMCsajMszMCsalDDOzgqn4p2SbmbU11a4xm5kVSwV0mJ2YzayyuMZsZlYwlZCYK6HXb2a2ULUi91KKpFUl3SdpiqTnJB2d2rtLGitpanrtltol6QJJL0uaJGlgrX0NTZ+fKmloQ+fgxGxmFaVdVf6lAfOAX0XEusBmwHBJ6wEnAPdExADgnrQN2UOrB6RlGHAxZIkcOBnYFNgEOLkmmdfHidnMKkqV8i+lRMT0iJiY1mcDU4BVgD2B0eljo4G90vqewBWRmQB0ldQH2BEYGxEzI+JDYCywU8lzaNKZm5kVVLXyL5KGSXqi1jKsrn1KWh34DvAo0DsipkOWvIFe6WOrAG/W+tq01FZfe7188c/MKkpj5sqIiJHAyFKfkbQC8E/g5xHxsVRvV7uuN6JEe72cmJvZ/Pnz+eEPf0nv3t255JKTiQjOO+9K7rjjIaqqqthvv535yU/24NFHJ3PEEafTt29vAHbYYXOOPHK/Vo7emstlZ49h0iPP06nbCpx6+XEAvDH1La469wbmzplHVXUVB/zih6y5bj8+nf0Zl589hhlvf0D7Zdpx8PFDWGXNPrzzxgwuOeWKhft87+0P2POQndhhn/9prdNqE5qzDCCpPVlSvjoibkzN70rqExHTU6liRmqfBqxa6+t9gbdT+zaLtY8rdVwn5mZ2xRW30L9/Xz755DMAbrzxHqZPf5/bb7+YqqoqPvjgo4WfHTRoPS655OTWCtXKaMudN2bbH2zFqDOvWdj2j7/ewu5Dd2SDzdZl0oTn+cdfb+W484dz21V3s+qAVRh+xiFMf/1drj7vRo758/+y0mq9OHnUMQAsmL+AY/Y+hYFbb9Bap9RmNNdwOWVd41HAlIg4t9ZbNwNDgbPT60212o+UNIbsQt+slLzvBM6sdcFvMDCi5Dk0ENiOki6WdLOkm9J6yaL10uydd95n3LjH2XvvwQvbrr32NoYPH0JVVfafukePrq0VnrWgtb/Vn+U7LbdImyQ+/+wLAD7/5Au69ugMwNuvvcu6AwcA0Kdfbz54ZyazZs5e5LtTJk6l58o96LFS9xaIvm1rXxW5lwZsCRwIbCvp6bTsQpaQd5A0FdghbQPcBrwKvAxcChwBEBEzgdOAx9NyamqrV709ZknnAWsDV5B1xSHrgh8laeeIOLqhs1ranHnmpRx77MF8+unnC9vefPMdbrttPGPHTqB7986cdNLhrL76ygA8/fSL7LHH/6NXr+4cf/whDBjQr7VCtxaw75F7cd6xl3DDRbcQsYARfzkKgFX7r8zEByYzYMM1eXXK63zw7od8+N5HdOneaeF3H7vnKTbd7jutFXqb0lw95oh4kLrrwwDb1fH5AIbXs6+/A3/Pe+xSPeZdImKXiBgTEQ+mZQywK7BLqZ3WvtI5cuR1eWNp0+677zG6d+/CN7+51iLtc+bMZdlll+HGG//Mj360I7/+9fkArL9+f+69dxQ333whBx64O8OHn9EaYVsLGnfTQ+x75J788R+/Zd/he3H5H7I/GzsfsB2fzv6MUw79E/f+80FWW2sVqqu/+qM5b+48nnn4OTba5tutFXqb0lzD5VpTqRrzF5I2iYjHFmvfGPii1E4XvdL5Utuf6imHiROncO+9j/HAA0/y5Zdz+OSTzzjmmHPo3bsHgwdvAWQX+EaMyBLzCit89c/c//mfQZxyysXMnDmL7t27tEr8Vn6P3PkE+x31fQAGfe9bjP5jlpg7Lt+BQ0ZkF34jghOGnM6KfXos/N7kR19gtQGrLNKDtvpVwhjgUudwEHChpOcl3ZWWKcCF6T2r5Ve/GsoDD1zOvfeO4txzj2OzzTbkT3/6FdtvvxkTJkwC4LHHnl1YxnjvvQ/J/uUDkya9xIIFC+jWrXOrxW/l16VHZ158+hUAXpg4lV59ewLw2ezPmTd3HgDjb53A2hv2p+PyHRZ+77F7JrLJdgO/vkOrk5R/Kap6e8zpjpdNJa1ENhhawLSIeKelgqsEw4btzTHHnMPo0Tex3HIdOOOMrK54550Pce21t1FdXU2HDsty7rnHUWJ8pLUxI0+5kheffplPZn3KsXufwh4H78jQY3/EtRf+mwXz59N+mfb85Jh9AJj++ruMOvMaqqqr6NOvNwcdv+/C/Xz5xRyef+IlDvzVPq11Km1OkUsUeamm11Y+S0cpwxpn/DtTWzsEK6CtV9p1idPqxPf/kzvnDFxxyY9XDrnKMZImlto2MysKKXIvRZXrBpOIGFhq28ysKArZBW6kvD3mfpK2T+sdJfnysJkVUiVc/GswMUv6KfAP4JLU1Bf4dzmDMjNrKjViKao8pYzhZJM7PwoQEVMl9Sr9FTOz1lFd5IybU57E/GVEzKkZyiWpHQ1MWWdm1lqKXKLIK0+N+X5JvwY6StoBuAG4pbxhmZk1TSWUMvIk5hOA94DJwOFkMyidVM6gzMyaqhISc55SRs1zrC4tdzBmZkuqEu78y9Nj3gN4SdKVknZNNWYzs0KqhB5zg4k5Ig4G1iKrLe8PvCLpb+UOzMysKaoUuZeiynvn31xJt5ONxuhIVt44rJyBmZk1xVIxKkPSTpIuJ3tcyt7A34A+ZY7LzKxJqhqxFFWeHvNBwBjg8Ij4srzhmJktmUroMTeYmCNiSEsEYmbWHCogL5d8GOuDEbGVpNkseqefyJ476MdtmFnhVMJwuVJPMNkqvXomOTNrMyohMee5+HdlnjYzsyKohHHMeS7+rV97I91gslF5wjEzWzJFfjJJXvX2mCWNSPXlDSV9nJbZwLvATS0WoZlZI1RCj7nexBwRZ6X68h8jonNaOkVEj4gY0YIxmpnlVglPMMkzXG6EpG7AAKBDrfYHyhmYmVlTVLd2AM2gwcQs6TDgaLJHSj0NbAY8Amxb3tDMzBqvyD3hvPLclXg0sDHwekR8D/gO2fzMZmYF1ParzHlGZXwREV9IQtKyEfGCpG+UPTIzsyZQgRNuXnl6zNMkdSV7MvZYSTcBb5c3LDOzppGqci8N70t/lzRD0rO12rpLGitpanrtltol6QJJL0uaJGlgre8MTZ+fKmloQ8fNMx/z9yPio4j4HfAbYBSwV4NnZGbWKpq1lHE5sNNibScA90TEAOCetA2wM9kgiQHAMOBiyBI5cDKwKbAJcHJNMq9Pnjv/utcsZM/9exA/JdvMCkpU5V4akkafzVyseU9gdFofzVcd1ZrH8EVETAC6SuoD7AiMjYiZEfEhMJavJ/tF5KkxTwRWBT4k+yumKzBd0gzgpxHxZI59mJm1iDwliiXUOyKmA0TEdEm9UvsqwJu1PjcttdXXXq88Z3AHsEtErBgRPci669cDRwAX5TkLM7OWk7+UIWmYpCdqLcOW8MCLixLt9cqTmAdFxJ0L9xZxF/Dd1FVfNsf3zcxajBrxv4gYGRGDai0jcxzi3VSiIL3OSO3TyKoLNfqSDZSor71eeRLzTEnHS+qXluOADyVVAwtyfN/MrMU0JjE30c1AzciKoXw1d9DNwE/S6IzNgFmp5HEnMFhSt3TRb3Bqq1eeGvP+ZFcU/522H0xt1cCPGnEyZmZll/UZm2tfuhbYBlhR0jSyXHg2cL2kQ4E3gH3Sx28DdiF7PupnwMEAETFT0mnA4+lzp0bE4hcUFz1uRL4BFpJWiIhPGnNSmZc8gsO+Zvw7U1s7BCugrVfadYnvDvl03gO5c87y7b5byLtR8gyX20LS88DzaftbknzRz8wKqQVKGWWXp8b8Z7JxeB8ARMQzwHfLGZSZWdNVNWIppjw1ZiLiTS06ZdP88oRjZrZkitwTzitPYn5T0hZASFoGOAqYUt6wzMyaRhUw72eexPwz4HyyO1WmAXcBw8sZlJlZU6kCpsrP8wST94EDWiAWM7NmUME9Zkm/LfG9iIjTyhCPmdkSqfRSxqd1tC0PHAr0AJyYzayAKjgxR8Q5NeuSOpE9YupgYAxwTn3fMzNrTXmm8yy6kjXmNAfzL8lqzKOBgWk+UTOzQqroxCzpj8APgJHABk27HdvMrGVVQo251F8tvwJWBk4C3pb0cVpmS/q4ZcIzM2usCr7zLyKKG7WZWT2Wljv/zMzaECdmM7NCqYQasxOzmVWUSrglu8GJ8iX9PiKOb6jNGiZpWM5nitlSxL8LW1yeC3w71NG2c3MHspRYkifwWuXy78IWUWoc8/8CRwD9JU2q9VYn4OFyB2ZmtrQqVWO+BrgdOAs4oVb77IYeJGhmZk1XbykjImZFxGtkczHPjIjXI+J1YK6kTVsqwArjOqLVxb8LW0Sei39Pkc2REWm7CngiIga2QHxmZkudPBf/FLWyd0QswMPszMzKJk9iflXSUZLap+Vo4NVyB9ZUkr4vKSStk+OzB0laeQmOtY2kW+tpnyXpKUlTJJ3cxP0/nF5Xl7R/rfZBki5oatyLHeMOSR/VdR5tXYF+CyFp91ptt0rapqnHquf45fyNDJU0NS1Dm2OfVlqexPwzYAvgLbJn/m1KsYf37Ac8CAzJ8dmDyCZqKofxEfEdYBDwY0kbNXYHEbFFWl0d2L9W+xMRcVSzRAl/BA5spn0VTVF+C9OAE8u07xqrU4bfSJr692SyP/ebACdL6rYHbJdrAAAGFElEQVSk+7XSGkzMETEjIoZERK+I6B0R+0fEjJYIrrEkrQBsSfaUlSGLvXecpMmSnpF0tqS9yZLm1ZKeltRR0muSVkyfHyRpXFrfRNLDqQf8sKRv5I0pIj4FniQbdthB0mUpjqckfS/tf31Jj6U4JkkakNprplo9G9g6vf+Lmt6ZpKoUc9da5/mypN6Sekr6p6TH07JlPfHdA8zOez5tRcF+C88AsyR97Z4ASRtJul/Sk5LulNQntW+cfguPSPqjpGdT++qSxkuamJaav7zL9RvZERgbETPTXOxjgZ1ynLMtiYiocwGOS68XAhcsvtT3vdZcgB8Do9L6w2QXLSG7IeZhYLm03T29jgMG1fr+a8CKaX0QMC6tdwbapfXtgX+m9W2AW+uIY2E72WO4XgPWJ5tK9bLUvg7wBtAh/Tc+ILUvA3RM65/UdZzF9n8+cHBa3xS4O61fA2yV1lcDptQ6r7/VF2+lLEX7LQBbA/entltTe/sUS8/Uvi/w97T+LLBFWj8beDatLwd0SOsDyC7El+03AhwDnFRrv78Bjmnt/38rfSl1EW9Ken2ixGeKZj/gvLQ+Jm1PJPsDdFlEfAYQjR+H3QUYnXqyQfYHqiFbKxvRsgA4OyKek3Q6WRImIl6Q9DqwNvAIcKKkvsCNETG1EbFdB/wWuIysZ3hdat8eWE9fTejSWVKniHgCOKwR+2+rivRbICLGS0LS1rWavwF8Exib/n+qBqan3m2niKi5kesaYLe03h74P0nfBuaT/X4asiS/kbpmBCo9lMuWWKn5mG9Jr6NbLpymk9QD2Bb4pqQg+5GHpOPIflx5fkzz+Kq806FW+2nAfRHxfUmrk/WuGjI+InZbrK3Oaa8i4hpJjwK7AndKOiwi7s1xDMiS+lqSegJ7Aaen9ipg84j4POd+KkYBfws1ziCrNc+rCRV4LiI2Xyz+UjXcXwDvAt9K8X2R47hL8huZRtb7rtGXxp2zNUG9NWZJt0i6ub6lJYPMaW/giojoFxGrR8SqwH+BrYC7gEMkLQcLL2hAVlvtVGsfrwE1F+l+WKu9C9nFT8guEjXVA2TPT0TS2mT/fHxR0prAqxFxAXAzsOFi31s8zoUi+/flv4Bzyf4p+kF66y7gyJrPpR7W0qKQv4WIuAvoRpZUAV4EekraPMXSXtL6kdVyZ0vaLH2udo28CzA9smGrB8LCqdTK9Ru5ExgsqVv6C2NwarMyKnXx709kT8P+L/A5cGlaPiGrfxXNfmQ/vtr+CewfEXeQJbwnJD1NVjcDuBz4a80FH+AU4HxJ48n+mVjjD8BZkh6CJZpT8CKgWtJksn9OHhQRX5LVFp9Nsa0DXLHY9yYB89LFql/Usd/ryGqq19VqOwoYlC4gPU82uqbmQtbfaj6UzvUGYDtJ0yTtuATnVxRF/i2cQdbrJCLmkP0l8ntJzwBPk42Aguyi5UhJj5D1rGel9ouAoZImkJUxPk3tZfmNpFLPacDjaTm1CeUfa6Q8d/49EBHfbajNzJqPpBUiPQBZ0glAn4g4upXDshaSZxxzz/RPbQAkrQH0LF9IZgbsmnrvz5KN6Di9oS9Y5cjTY96JbJKVmrv9VgcOjwjXmczMyqDBxAwgaVmy2ifAC6kuamZmZdBgKSNdvT4WODIingFWk7T4MDAzM2smeWrMlwFzgJqxltNwvcvMrGzyJOb+EfEHYC5AGoze9p8PbmZWUHkS85w0rrNmovz+gGvMZmZlkmfC+5OBO4BVJV1NNmPXQeUMysxsaVZyVIay2U36Ap8Bm5GVMCZExPstE56Z2dInzzjmJyOi0ZO8m5lZ0+SpMU+QtHHZIzEzMyBfj/l5snljXyObMEVkE1YtPgOamZk1gzyJuV9d7RHxelkiMjNbytU7KkNSB7JpANcCJpM9pmdefZ83M7PmUW+PWdJ1ZDeVjCd7TtrrnnbQzKz8SiXmyRGxQVpvBzwWEQNbMjgzs6VRqVEZc2tWXMIwM2s5pXrM8/nqsTUCOpLdaFIzKqNzi0RoZraUyTUfs5mZtZw8N5iYmVkLcmI2MysYJ2Yzs4JxYjYzKxgnZjOzgnFiNjMrmP8Pvk7mO2/RoMUAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 2 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# visualize confusion matrix with seaborn heatmap\n",
"\n",
"cm_matrix = pd.DataFrame(data=cm, columns=['Actual Positive:1', 'Actual Negative:0'], \n",
" index=['Predict Positive:1', 'Predict Negative:0'])\n",
"\n",
"sns.heatmap(cm_matrix, annot=True, fmt='d', cmap='YlGnBu')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 16. Classification metrices"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Classification Report\n",
"\n",
"\n",
"**Classification report** is another way to evaluate the classification model performance. It displays the **precision**, **recall**, **f1** and **support** scores for the model. I have described these terms in later.\n",
"\n",
"We can print a classification report as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" precision recall f1-score support\n",
"\n",
" <=50K 0.93 0.81 0.86 7407\n",
" >50K 0.57 0.80 0.67 2362\n",
"\n",
" micro avg 0.81 0.81 0.81 9769\n",
" macro avg 0.75 0.81 0.77 9769\n",
"weighted avg 0.84 0.81 0.82 9769\n",
"\n"
]
}
],
"source": [
"from sklearn.metrics import classification_report\n",
"\n",
"print(classification_report(y_test, y_pred))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Classification accuracy"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {},
"outputs": [],
"source": [
"TP = cm[0,0]\n",
"TN = cm[1,1]\n",
"FP = cm[0,1]\n",
"FN = cm[1,0]"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Classification accuracy : 0.8083\n"
]
}
],
"source": [
"# print classification accuracy\n",
"\n",
"classification_accuracy = (TP + TN) / float(TP + TN + FP + FN)\n",
"\n",
"print('Classification accuracy : {0:0.4f}'.format(classification_accuracy))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Classification error"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Classification error : 0.1917\n"
]
}
],
"source": [
"# print classification error\n",
"\n",
"classification_error = (FP + FN) / float(TP + TN + FP + FN)\n",
"\n",
"print('Classification error : {0:0.4f}'.format(classification_error))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Precision\n",
"\n",
"\n",
"**Precision** can be defined as the percentage of correctly predicted positive outcomes out of all the predicted positive outcomes. It can be given as the ratio of true positives (TP) to the sum of true and false positives (TP + FP). \n",
"\n",
"\n",
"So, **Precision** identifies the proportion of correctly predicted positive outcome. It is more concerned with the positive class than the negative class.\n",
"\n",
"\n",
"\n",
"Mathematically, precision can be defined as the ratio of `TP to (TP + FP)`.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Precision : 0.8099\n"
]
}
],
"source": [
"# print precision score\n",
"\n",
"precision = TP / float(TP + FP)\n",
"\n",
"\n",
"print('Precision : {0:0.4f}'.format(precision))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Recall\n",
"\n",
"\n",
"Recall can be defined as the percentage of correctly predicted positive outcomes out of all the actual positive outcomes.\n",
"It can be given as the ratio of true positives (TP) to the sum of true positives and false negatives (TP + FN). **Recall** is also called **Sensitivity**.\n",
"\n",
"\n",
"**Recall** identifies the proportion of correctly predicted actual positives.\n",
"\n",
"\n",
"Mathematically, recall can be given as the ratio of `TP to (TP + FN)`.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Recall or Sensitivity : 0.9281\n"
]
}
],
"source": [
"recall = TP / float(TP + FN)\n",
"\n",
"print('Recall or Sensitivity : {0:0.4f}'.format(recall))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### True Positive Rate\n",
"\n",
"\n",
"**True Positive Rate** is synonymous with **Recall**.\n"
]
},
{
"cell_type": "code",
"execution_count": 73,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"True Positive Rate : 0.9281\n"
]
}
],
"source": [
"true_positive_rate = TP / float(TP + FN)\n",
"\n",
"\n",
"print('True Positive Rate : {0:0.4f}'.format(true_positive_rate))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### False Positive Rate"
]
},
{
"cell_type": "code",
"execution_count": 74,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"False Positive Rate : 0.4260\n"
]
}
],
"source": [
"false_positive_rate = FP / float(FP + TN)\n",
"\n",
"\n",
"print('False Positive Rate : {0:0.4f}'.format(false_positive_rate))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Specificity"
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Specificity : 0.5740\n"
]
}
],
"source": [
"specificity = TN / (TN + FP)\n",
"\n",
"print('Specificity : {0:0.4f}'.format(specificity))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### f1-score\n",
"\n",
"\n",
"**f1-score** is the weighted harmonic mean of precision and recall. The best possible **f1-score** would be 1.0 and the worst \n",
"would be 0.0. **f1-score** is the harmonic mean of precision and recall. So, **f1-score** is always lower than accuracy measures as they embed precision and recall into their computation. The weighted average of `f1-score` should be used to \n",
"compare classifier models, not global accuracy.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Support\n",
"\n",
"\n",
"**Support** is the actual number of occurrences of the class in our dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 17. Calculate class probabilities"
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[9.99999426e-01, 5.74152436e-07],\n",
" [9.99687907e-01, 3.12093456e-04],\n",
" [1.54405602e-01, 8.45594398e-01],\n",
" [1.73624321e-04, 9.99826376e-01],\n",
" [8.20121011e-09, 9.99999992e-01],\n",
" [8.76844580e-01, 1.23155420e-01],\n",
" [9.99999927e-01, 7.32876705e-08],\n",
" [9.99993460e-01, 6.53998797e-06],\n",
" [9.87738143e-01, 1.22618575e-02],\n",
" [9.99999996e-01, 4.01886317e-09]])"
]
},
"execution_count": 76,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# print the first 10 predicted probabilities of two classes- 0 and 1\n",
"\n",
"y_pred_prob = gnb.predict_proba(X_test)[0:10]\n",
"\n",
"y_pred_prob"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Observations\n",
"\n",
"\n",
"- In each row, the numbers sum to 1.\n",
"\n",
"\n",
"- There are 2 columns which correspond to 2 classes - `<=50K` and `>50K`.\n",
"\n",
" - Class 0 => <=50K - Class that a person makes less than equal to 50K. \n",
" \n",
" - Class 1 => >50K - Class that a person makes more than 50K. \n",
" \n",
" \n",
"- Importance of predicted probabilities\n",
"\n",
" - We can rank the observations by probability of whether a person makes less than or equal to 50K or more than 50K.\n",
"\n",
"\n",
"- predict_proba process\n",
"\n",
" - Predicts the probabilities \n",
" \n",
" - Choose the class with the highest probability \n",
" \n",
" \n",
"- Classification threshold level\n",
"\n",
" - There is a classification threshold level of 0.5. \n",
" \n",
" - Class 0 => <=50K - probability of salary less than or equal to 50K is predicted if probability < 0.5. \n",
" \n",
" - Class 1 => >50K - probability of salary more than 50K is predicted if probability > 0.5. \n",
" \n"
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Prob of - &lt;=50K</th>\n",
" <th>Prob of - &gt;50K</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>9.999994e-01</td>\n",
" <td>5.741524e-07</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>9.996879e-01</td>\n",
" <td>3.120935e-04</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1.544056e-01</td>\n",
" <td>8.455944e-01</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.736243e-04</td>\n",
" <td>9.998264e-01</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>8.201210e-09</td>\n",
" <td>1.000000e+00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>8.768446e-01</td>\n",
" <td>1.231554e-01</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>9.999999e-01</td>\n",
" <td>7.328767e-08</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>9.999935e-01</td>\n",
" <td>6.539988e-06</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>9.877381e-01</td>\n",
" <td>1.226186e-02</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>1.000000e+00</td>\n",
" <td>4.018863e-09</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Prob of - <=50K Prob of - >50K\n",
"0 9.999994e-01 5.741524e-07\n",
"1 9.996879e-01 3.120935e-04\n",
"2 1.544056e-01 8.455944e-01\n",
"3 1.736243e-04 9.998264e-01\n",
"4 8.201210e-09 1.000000e+00\n",
"5 8.768446e-01 1.231554e-01\n",
"6 9.999999e-01 7.328767e-08\n",
"7 9.999935e-01 6.539988e-06\n",
"8 9.877381e-01 1.226186e-02\n",
"9 1.000000e+00 4.018863e-09"
]
},
"execution_count": 77,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# store the probabilities in dataframe\n",
"\n",
"y_pred_prob_df = pd.DataFrame(data=y_pred_prob, columns=['Prob of - <=50K', 'Prob of - >50K'])\n",
"\n",
"y_pred_prob_df"
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([5.74152436e-07, 3.12093456e-04, 8.45594398e-01, 9.99826376e-01,\n",
" 9.99999992e-01, 1.23155420e-01, 7.32876705e-08, 6.53998797e-06,\n",
" 1.22618575e-02, 4.01886317e-09])"
]
},
"execution_count": 78,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# print the first 10 predicted probabilities for class 1 - Probability of >50K\n",
"\n",
"gnb.predict_proba(X_test)[0:10, 1]"
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {},
"outputs": [],
"source": [
"# store the predicted probabilities for class 1 - Probability of >50K\n",
"\n",
"y_pred1 = gnb.predict_proba(X_test)[:, 1]"
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Text(0,0.5,'Frequency')"
]
},
"execution_count": 80,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAasAAAEdCAYAAACygkgFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xv8ZnO9///H0wyDGaNxLNGM49AoYpAchmijiFJfttOWU/SjlJ3soqYQld1hl0ROISU75FAqO7QlMnZR4xSZyWHGcYw5MBiv3x/v92XWrLmO4/O5rjWX5/12u26fz1rv93ut93qvda3XOryvtRQRmJmZVdlSva6AmZlZKw5WZmZWeQ5WZmZWeQ5WZmZWeQ5WZmZWeQ5WZmZWeX0RrCRNkXRir+uxJJE0VNL5kp6RFJJ26HWdACSNyfXZtt5wD+ozUdKDvZh3nv/Bkl4ZgOlcKOmGFnkWWtbyvCXtkNfFmp1Mp5ckHSPpUUmvSpo4SPMYkHWUp9VyPb1RVTZYNVtp+QtzQGHUFsC32pzutrn8mNdfyyXa3sB+wB7AW4Bbe1udhh4h1e/2djJ7/b4uZwDvbpJ+K2ldPA5N27rVdLpC0hrAt4HTgLeS6lV1nwI+2utKFBUOUsqfw0r53iLpZ5Kez5+fSlqtlKe870bSFyS9JOnAZvUYOnCL1DsR8VSv69CIpGUi4qVe16OO9YHHImLAg9RALnNEzAemD8S0qqSK20VEzAZmN0l/iTbWRavpdNE6pAPyqyNiWq8r04ykpYFXImJmF+b1JuDViHi+w6KbAcV2fK2ukpYCrgVeBd4HCPg+cJWkbaLO0ydyme8BBwK7R8Rvms49Iir5AS4EbmiQFsABheEpwImF4T2BPwNzgeeAPwHvAsbkssXPTbmMgH8H/gG8BDwEHFua78rA5cAc4AngZOBHxXoCNwHn5bRpwFN5/H6ks4OZwNPAdcAGhXK1uu0H/DrX/T5gAumo8Jd5vvcA27Vou6bLkutYbIMpDaZTq9OBwP8ALwAPA/vXybN/oY5n5LT1gJ/ndTAD+A3wjtI8/h/wIPAi6cj9g3l625amv22hzGrABXkdvAjcDxzSbP3mcvsCf8llpgDfBIYX0ocBZ+V1NCP/fxrwYIv2DtIR8c/z8j8OfKZOnk8Cl+bpX57Hj83bQm0Hfw2wXqHcwcArwM7A5Fz3PwGbFfKMAi4B/pnX0f3AcYDK3yfgM8BjpO3r58AqhTwTi8tam3dheIe8HGs2a+vydPK49wF/yPV7LK+/lQvp40jb/XO5De8FDmzR7u8H7gTmAU+Sdo7DC3Uo129Mg+nU3V8Uvks/JH2HXiB9p74KDGvSTp2sj2NI2+KrwAjq7Pdovd1um9t2Vv7cBezSpN0m5Hr9jPR9W7pFO7+23pvk+ZecZ2xpnQawQ3nfDSwLXEE6+Nms2fxfK9tOpl586q208gIXhqeQgxXwZtIO+nhgbWAjUgB4BzCEBTvDLXLelXK5/y+vwCNIZx1H5o3j0MJ8rgYeAHbMK+IC0o6nHKxmAT8A3k7eOQMfA3YH1iUFzquBvwPL5PQxuV4PAXsBGwBXknZ8NwAfyuN+Tro01nADa7UswEqkSyIP5zZYtcF0anV6nBSMxgKnkL5Y40t5Hs0b4Tq53VfPG+JZue3HAt8FnqnNL7fDq6SAMBb4cK5Tw2AFLEfakf0faQe+DumLsm+L9XswKQAdmMtsD9wNXFxY3m+Rdnp7AhvmNnqe9oLVs6QdzwakwPUK8OFSnmdynnVzvuWAqaQDgc3z50ZS8F6mUO9X8/JOAN5JOoKdBixf2OY/RzryXTuvh9nAx0rfp+dJ2907SDugv5POOhYnWDVr6/J03ksKBMeQtsct8nL+nrwDz+viUtJ3Zh1gN9LRdqM2f2du42+RvuO7kYLDxTl9BGl7CtJ29mZgSJ3pNNxf5PSlSNv8VqRt8YO57b/cpJ06WR9XApvmdTKU0n6PFtttXg/PkgLY+vnzIZoc0JIC8Lak7+bT+fN94D0N8tfW+xTS9+NW4N9YOPh+GfhHnbKPsPCJROTt4BbSvnTttmPCYASagfjklfYKC444i59mwepdND+K2rZeem7Ur5fGfau2AvJGEMBOhfSlc7lysHoAWKrF8q2Up7dNHh6Th4tnQFvkcccVxtWWb+Mm0266LHl4Iq13wrU6nVwafytwSSnPSaU8E4Hb6nxJXjvLIx193lrKczTNg9WhpMBb9yivyfqdAhxZGrd9zjsKGJ6ne3gpz6Q22ikoBL087lLgllKe80p5DiXtxItnN6uTDjQOysMH19nuRpG+B4c1qdN3gN+Wvk+zgRUL42pHw+vX2yZoEqxatHV5OjcBp5fyvC2X3TQPzwQObtbOpfIXA38qjduTFNhH16tvg+k03V80KPNp4O+N2qmD9fEcMKKU70IW3p+02m5HUTp76eRD2oftDvyEdEb7EPCV2jaR84wlHfCOz58vkQL8yYU851D6LufxdwBnlr4H80gHsqt0UtfKdrDIbicddZQ/zdxNupzwN0lXSvqUpLWaFZA0knS0+PtS0s3AGEnLk474AG6rJUbEy6SdWdmdEfFqaR6b5vo8LGkW6SgQYHSp7F2F/2v3B+6uM26hG5cdLkun/lga/gML2qPmT6XhLYDNJc2ufUhnnGNIgZ88jT+Uyt3Soi6bA/dExKPtVBxA0qqkdv5mqT6/ylnWI53tDGPRjiat6lOzOG00jrQsT9dGRMQTpMtG4xpNPyJmkM4u3w7p2r+kEyT9RdLTedmOZNFt655Y+J5Ire03arpkr98WwLGltr8np9W2hTOAcyXdlHsTbtZimuOov42LRdu9mZb7C0mHS7pd0hO57qexaNsW87e7Pu6NdH+v0XRabrd5WzgX+LWkX+X5jm134SPi5Yi4NiL+lbTfuAM4iXTps5bn/oj4QURMyp8vA6cDn8732lrOpjR8LelgvaMe3FUPVi9ExIPlT7MCkW7I70a69HAHqdfbA5J2b2N+5UZVG3nqmbPQRFKA+E0uewiwJQvOmpYplX25zrzqjWu17tpZlsVVb1pzSsNLkS5vlQ80xpKOvGvTaac9yzotU2urT5XqsglpZ/lXFizT4tSnnnbaqNH82mmX4vSPA/6DdJn1faRlO5dFt61eWQr4GotuC+uTd7wRcTLp0ujPgI2B2ySd0mK6jdqo7XXYan8h6aPAmcBlpHtk7yKdeTTbSbe7PuptD0XtbLdExOGkg7jfki4V/03Sx1tMGwAl20n6Aemy8ATSWeCnWhS9lXQ1YtU8PI10+bOsdjug6ErSJdqPSzond7RoqerBarFE8qeI+GpEbE864vpYTq71wBpSyP886Z7LhNKktgcejoi5LDgS3LqWKGkoaSNpZSPSSv1CRNwYEfeSTt8HMoAAbS9Lp8rdkLcmHdk3M4l09PtYnQOOWu/NycA2pXLl4bI7gXFNfutTb/0+Qbo0OrbewU9EvEi6T/RSnfm/p0V9ahanjSaTlmWV2ghJq5N22pMbTT/35tqwMP3tgesj4ryI+HM+oFufRW2Uz7xrasvWqp6NLNLWDUwCxjVo+9fOLCLiHxHx/Yj4CPBF4Kgm05zMotv4BFKgumfR7I212F9sD/w5Ir4ZEXdGxN9JVweaaXd9tKpXO9ttLe/fch13I3XwOqLZtCVtIulrpHumvyLd4zuAdMn02Ii4q1l5UtB+gXS/C9JZ+tqSXltOSRsBa1Hn6kREXEu6/Lgf8CNJrbah/gtWkt4j6SRJW0l6m6SdSDdjaxvwVNJ17fdLWk3Sinn8acAx+ZR//XxkchSp5w95I70GOFPSBElvB84GRtL6SG4q6TrtMZLWzXX6ThvlFlfTZVkMh0raT9IGkr5C2hF/u0WZ75F2YlflI7cxSr/LOVVSbSf5LWDrPG4DSR8iHZU28xNSe14taWdJa0vaSdI+Ob3R+v0C8ElJJ0raWNJYSXtJOhsgIuaQOsWcIumDOf3rpKDQjt0lHZ3b+xhgH1r/9u9S4CngMkmbSdoc+Cmpt9xlhXwBfF3S9pLeAVxEOiq/NKffD+wgacfcjrUOAWUBXJSXf3vSGcN1edteHI3auuyLwJ6SvqV0OXxdSbtKOk/ScpJGSDpT0nvz+nwXsCvNg843gM0kfVPShpJ2JZ3J/Dgi/tmk3ELa2F/cD7xD0p653p8inRU00+76aEfT7VbSepK+lr9boyVtDWxHk7aTtB3poG8T4PPA6hFxQERcn880y/k/LWnv3M5jJX2SdKnwzFjw84sbSJ2ALpG0paStSPcVbyMF/0VExP+Q7pvuQfoONL+k2MkNrm5+WPzegONIXainkwLEVNKGvUwh//GkHcJ8Fu66/llSb7SXSV1U63Vd/2/STfEnSZcDLgeuKeS5CTi3Tp0/QjrNfpHUTXYCqQPJwTl9DIt20V6TRbt+vjmP27lJ27WzLBNpv4PFgXm5al1nD6yTZ9s65UcDPybtkGvr4hIKPYBIvfgeyum3k26SN+xgUWiDi0hHdS+Suvgf3Gz95vF7ke79zCX1xPoL8MVC+nKkA5CZ+XMO7XddPxa4Kk97GvDZZttsYfxY0vZa6zx0LfW7rv8L6QxoHuly1fhCnhVJl8+eJ/U4PJP004kp5e8T6ScN00hHxVdS6Ala3iZo0cGiyXdpoenkcdvl+c9iQdf0b5N6wC1LCrwP5/X5JClYr9Wi3Ytd158i9W4b3qy+dabRdH9Butx3NqnH3fO5nkeTTsgatVPb66Od/R5NtlvSj7SvIF1NmUfquftDCh1p6sxjZVKAandf/FlSAJ5L+l7cCRxOqRNZrsvleR0/n9fhaq2+B6Rek0+RfsKxbKN61LqN2mLIp673kbr/tjojWOIoPZngYVI32HY7GrzhSApSAL+k13Ux61d98QSLbsmXTlYjnRmtQOrCOoZ0NGRmZoOkq/esJO0r6V5JcyQ9lK+dku853CdprqQbJY0ulBmm9MDV5yVNl/SZ0jQblh0EQ0jdLe8i/ahxHWDHiPjrIM7TzOwNr2uXASW9j9R9cx/S703ekpPmke5ZHEbqwHAy6bLTu3O500g/Pvwg6V7FjaT7E9cr9aJqWNbMzPpDN4PVraRf8J9XGn8EKfi8Jw8PJ904f1dE3CfpMdJjSn6T008m/bp631Zlu7JgZmY26Lpyzyp3RBhP6m78IKn3z1WkXibjKDy1ISLmSHqI9PuTJ4A1WPipDneResfQrCyp40OxDkeQf3swfPjwzTfcsN0eyWZmBnDnnXc+HRGrts458LrVwWJ1UhfQj5C6sL4M/IJ0/2cEqdti0UxSB4YRheFyGi3KLiQiziF1RWb8+PExaVK9pySZmVkjkqb2at7d6mDxQv773YiYFulZaN8k/U5iNumHtUUjSX31ZxeGy2m0KGtmZn2iK8Eq0sMWH6X+Exsmk35JDbx232ldYHIuN62Ynv+f3KrsQNbfzMx6q5td1y8gPQJoNUmjSL/4v5b0K/qN8+M8liU9muXuQgeJi4ATJY2StCHpl9MX5rRWZc3MrA90M1idTHpMzAOkR638GTg10kNN9wZOJb1kbCvSI3hqvkTqnj6V9Iypb0TE9QBtlDUzsz7whnzckjtYmJl1TtKdETG+F/Puu6eum5lZ/3GwMjOzynOwMjOzynOwMjOzyntDviLkr4/NZMwJ1/W6Gkw5/QO9roKZ2RLBZ1ZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5DlZmZlZ5XQtWkm6S9KKk2flzfyFtP0lTJc2RdJWklQppK0m6MqdNlbRfaboNy5qZWX/o9pnV0RExIn/GAkgaB5wNHAisDswFvl8ocybwUk7bHzgrl2mnrJmZ9YGhva4AKQBdExG/B5B0EnCvpBWAV4G9gY0jYjZwi6SrScHphGZlI2JWD5bFzMwGQbfPrE6T9LSkP0jaIY8bB9xVyxARD5HOpDbIn/kR8UBhGnflMq3KmplZn+jmmdXngHtIwWRf4BpJmwIjgJmlvDOBFYD5TdJoUXYhko4AjgAYMnLVxV4IMzPrvq6dWUXE7RExKyLmRcSPgD8A7wdmAyNL2UcCs1qk0UZ6cf7nRMT4iBg/ZPkVX9/CmJlZV/Wy63oAAiYDm9RGSloHGAY8kD9DJa1fKLdJLkOLsmZm1ie6EqwkvUnSLpKWlTRU0v7A9sCvgR8De0jaTtJw4CvAFfksbA5wBfAVScMlbQPsCVycJ92wbDeWy8zMuqNb96yWBk4BNiTdh7oP2Csi7geQdCQp8KwM3AB8rFD2E8D5wJPAM8BRETEZICImtyhrZmZ9oCvBKiKeArZokn4pcGmDtGeBvRanrJmZ9Qc/bsnMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCrPwcrMzCqv68FK0vqSXpR0SWHcfpKmSpoj6SpJKxXSVpJ0ZU6bKmm/0vQaljUzs/7QizOrM4E7agOSxgFnAwcCqwNzge+X8r+U0/YHzspl2ilrZmZ9YGg3ZyZpX+A54FZgvTx6f+CaiPh9znMScK+kFYBXgb2BjSNiNnCLpKtJwemEZmUjYlYXF83MzAZR186sJI0EvgIcV0oaB9xVG4iIh0hnUhvkz/yIeKCQ/65cplXZ8vyPkDRJ0qT5c2e+/gUyM7Ou6eZlwJOB8yLikdL4EUA5eswEVmiR1qrsQiLinIgYHxHjhyy/4mJU38zMeqUrlwElbQrsDLyrTvJsYGRp3EhgFukyYKO0VmXNzKxPdOue1Q7AGOCfkiCdEQ2R9HbgemCTWkZJ6wDDgAdIwWqopPUj4u85yybA5Pz/5CZlzcysT3QrWJ0D/LQw/O+k4HUUsBrwR0nbAf9Huq91Ra2DhKQrgK9IOgzYFNgTeE+ezo+blTUzs/7QlXtWETE3IqbXPqTLdy9GxFMRMRk4khR4niTdb/pEofgngOVy2k+Ao3IZ2ihrZmZ9oKtd12siYmJp+FLg0gZ5nwX2ajKthmXNzKw/+HFLZmZWeW0HK0mflLTKYFbGzMysnk7OrHYGpki6VtI+koYNVqXMzMyK2g5WEfFBYDTwK+BYYLqkcyVtP1iVMzMzgw7vWUXEMxFxZkRsDUwAtgBulDRF0hckjRiUWpqZ2Rtaxx0sJO0k6QLgJuAJ4CDSg2XfRTrrMjMzG1Btd12XdAawL+nZexcBJ0bEY4X024AZA15DMzN7w+vkd1bLAh+KiDvqJUbEy5LGD0y1zMzMFugkWJ1GernhaySNApaLiMcBIuK+AaybmZkZ0Nk9q6uANUvj1gSuHLjqmJmZLaqTYDU2Iv5aHJGHNxzYKpmZmS2sk2D1pKT1iiPy8DMDWyUzM7OFdRKszgd+Lml3SW+XtAfw38C5g1M1MzOzpJMOFqcDLwNnAGsBj5AC1TcHoV5mZmavaTtYRcSrwDfyx8zMrGs6ep+VpLGk18gv9FiliDh/ICtlZmZW1MkTLD4PfBG4i4V/bxWk+1lmZmaDopMzq2OBLSPi7sGqjJmZWT2d9AZ8AfATKszMrOs6CVYnAd+V9BZJSxU/g1U5MzMz6Owy4IX572GFcSLdsxoyUBUyMzMr6yRYrT1otTAzM2uik99ZTQXIl/1Wj4hpg1YrMzOzgrbvN0l6k6RLgReBB/O4D0o6ZbAqZ2ZmBp11sPgB6S3Bo4GX8rg/AvsMdKXMzMyKOrlntROwRn4jcABExFOSVhucqpmZmSWdnFnNBFYpjpD0NqCte1eSLpE0TdLzkh6QdFghbSdJ90maK+lGSaMLacMknZ/LTZf0mdJ0G5Y1M7P+0EmwOpf0ipAdgaUkbQ38iHR5sB2nAWMiYiTwQeAUSZtLWgW4gvQ7rpWAScBlhXITgfVJlx93BI6XtCtAG2XNzKwPdHIZ8GukzhVnAkuTngd4NvCddgpHxOTiYP6sC2wOTI6IywEkTQSelrRhRNwHHAR8LCJmADMk/RA4GLge+HCLsmZm1gfaPrOK5NsR8faIGB4RG+XhaHcakr4vaS7psU3TgF8C40gPx63NZw7wEDBO0ihgjWJ6/n9c/r9h2TrzPkLSJEmT5s+d2W6VzcysAjp56vp7G6VFxO/amUZEfELSMcDWwA7APNLrRp4qZZ0JrMCCV5HMrJNGi7LleZ8DnAMw7C3rtx1gzcys9zq5DHheaXhVYBngUWCddicSEfOBWyQdABwFzAZGlrKNBGbltNrwi6U0WpQ1M7M+0ckTLBZ63JKkIcCJLH5gGEq6ZzUZ+LfCdIfXxkfEDEnTSC98/G3OskkuQ7Oyi1knMzOroMV+Yno+QzoVOL5VXkmrSdpX0ghJQyTtAvwr8DvgSmBjSXtLWpb0gse7Cx0kLgJOlDRK0obA4Sx4qG6rsmZm1gde7+s93ge82ka+IF3yexSYAZwBHBsRv4iIp4C9SYFvBrAVsG+h7JdInSamAjcD34iI6yH9KLlFWTMz6wOddLB4hBR0apYHlgU+0apsDioTmqTfAGzYIG0ecEj+dFTWzMz6QycdLA4oDc8BHoiI5wewPmZmZovopIPFzYNZETMzs0Y6uQx4MQtfBqwrIg56XTUyMzMr6aSDxXPAXqRX2D+ay+6Zxz9U+JiZmQ2oTu5ZbQB8ICL+tzZC0rbASRGxy4DXzMzMLOvkzOrdwG2lcbeTHp1kZmY2aDoJVn8GvippOYD891TgL4NRMTMzs5pOgtXBwDbATElPkB4Yuy2Fxx2ZmZkNhk66rk8B3iNpLdJrO6ZFxD8Hq2JmZmY1HT1uSdLKpFd7TIiIf0paQ9Kag1IzMzOzrO1gJWkCcD+wP+k18pBeN3/WINTLzMzsNZ2cWX0b2CcidgVeyeNuB7Yc8FqZmZkVdBKsxkTE/+T/a0+yeInOfqtlZmbWsU4CzT2SdomIXxfG7Qz8dYDrZGZmJWNOuK7XVeipToLVccC1kq4DlpN0NrAH6ZFLZmZmg6bty4ARcRvwTtIr488HHga2jIg7BqluZmZmQJtnVpKGAP8D7BIRXx/cKpmZmS2srTOriJgPrN1ufjMzs4HUSfD5MnCWpNGShkhaqvYZrMqZmZlBZx0szs1/D2JB13Xl/4cMZKXMzMyKWgYrSW+OiOmky4BmZmZd186Z1QPAyIiYCiDpioj48OBWy8zMbIF27jepNLzDINTDzMysoXaCVbTOYmZmNnjauQw4VNKOLDjDKg8TEb8bjMqZmZlBe8HqSdITK2qeKQ0HsM5AVsrMzKyo5WXAiBgTEWs3+bQMVJKGSTpP0lRJsyT9WdJuhfSdJN0naa6kGyWNLpU9X9LzkqZL+kxp2g3LmplZf+jWD3qHAo8AE4AVSS9v/JmkMZJWAa7I41YCJgGXFcpOJL3kcTSwI3C8pF0B2ihrZmZ9oCvvooqIOaSgU3OtpIeBzYGVgckRcTmApInA05I2jIj7SD9C/lhEzABmSPohcDBwPfDhFmXNzKwP9ORRSZJWBzYgPcF9HHBXLS0HtoeAcZJGAWsU0/P/4/L/DcvWmecRkiZJmjR/7syBXSAzMxtUXQ9WkpYGfgz8KJ/9jADK0WMmsEJOo5ReS6NF2YVExDkRMT4ixg9ZfsXXtxBmZtZVXQ1W+aG3FwMvAUfn0bOBkaWsI4FZOY1Sei2tVVkzM+sTXQtWkgScB6wO7B0RL+ekycAmhXzDgXVJ96JmANOK6fn/ya3KDtJimJlZD3TzzOosYCNgj4h4oTD+SmBjSXtLWhb4InB3oYPERcCJkkZJ2hA4HLiwzbJmZtYHuhKs8m+fPg5sCkyXNDt/9o+Ip4C9gVOBGcBWwL6F4l8idZqYCtwMfCMirgdoo6yZmfWBbnVdn8qiD8Qtpt8AbNggbR5wSP50VNbMzPqD3/JrZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV17VgJeloSZMkzZN0YSltJ0n3SZor6UZJowtpwySdL+l5SdMlfabdsmZm1h+6eWb1OHAKcH5xpKRVgCuAk4CVgEnAZYUsE4H1gdHAjsDxknZts6yZmfWBrgWriLgiIq4CniklfRiYHBGXR8SLpOC0iaQNc/pBwMkRMSMi7gV+CBzcZlkzM+sDVbhnNQ64qzYQEXOAh4BxkkYBaxTT8//jWpUd5DqbmVkXVSFYjQBmlsbNBFbIaZTSa2mtyi5E0hH5ntmk+XPLRczMrMqqEKxmAyNL40YCs3IapfRaWquyC4mIcyJifESMH7L8iq+70mZm1j1VCFaTgU1qA5KGA+uS7kXNAKYV0/P/k1uVHeQ6m5lZF3Wz6/pQScsCQ4AhkpaVNBS4EthY0t45/YvA3RFxXy56EXCipFG548ThwIU5rVVZMzPrA908szoReAE4ATgg/39iRDwF7A2cCswAtgL2LZT7EqnTxFTgZuAbEXE9QBtlzcysDwzt1owiYiKpa3m9tBuAut3NI2IecEj+dFTWzMz6QxXuWZmZmTXlYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpXnYGVmZpU3tNcVMDOrsjEnXNfrKhgOVj1VhS/BlNM/0OsqmJm15GBlVjE+iDFblIPVG1wVdozgnWPVVGW7MKtxsDLLvIM2qy4HK6sEBwoza8Zd1802PMWHAAAO0klEQVTMrPL6IlhJWknSlZLmSJoqab9e18nMzAZOv1wGPBN4CVgd2BS4TtJdETG5t9UyM7OBsMSfWUkaDuwNnBQRsyPiFuBq4MDe1szMzAZKP5xZbQDMj4gHCuPuAiYUM0k6AjgiD86b+rXd/9al+lXdKsDTva5ERbgtFnBbLOC2WGBsr2bcD8FqBDCzNG4msEJxREScA5wDIGlSRIzvTvWqzW2xgNtiAbfFAm6LBSRN6tW8l/jLgMBsYGRp3EhgVg/qYmZmg6AfgtUDwFBJ6xfGbQK4c4WZWZ9Y4oNVRMwBrgC+Imm4pG2APYGLmxQ7pyuVWzK4LRZwWyzgtljAbbFAz9pCEdGreQ8YSSsB5wPvA54BToiIS3tbKzMzGyh9EazMzKy/LfGXAc3MrP85WJmZWeX1bbBq93mBSr4m6Zn8+bokdbu+g6mDtvispL9JmiXpYUmf7XZdB1unz5GUtIyk+yQ92q06dksnbSFpM0m/lzRb0hOSPtXNug6mDr4fwyT9IC//s5KukfTWbtd3MEk6WtIkSfMkXdgi76clTZc0U9L5koYNZt36Nlix8PMC9wfOkjSuTr4jgL1I3d3fCewOfLxbleySdttCwEHAKGBX4GhJ+3atlt3RblvUfBZ4shsV64G22kLSKsD1wNnAysB6wG+6WM/B1u428Slga9J+Yg3gOeC73apklzwOnELqsNaQpF2AE4CdgDHAOsCXB7VmEdF3H2A4aePboDDuYuD0OnlvBY4oDB8K3NbrZehFW9Qp+1/Ad3u9DL1qC2Bt4F5gN+DRXte/V20BfBW4uNd1rkA7nAV8vTD8AeD+Xi/DILXLKcCFTdIvBb5aGN4JmD6YderXM6tGzwusd7Q0Lqe1yrek6qQtXpMvhW5Hf/24utO2+C7weeCFwa5YD3TSFu8GnpV0q6Qn8+Wvt3WlloOvk3Y4D9hG0hqSliedhf2qC3Wsonr7zdUlrTxYM+zXYNXW8wIb5J0JjOij+1adtEXRRNL2ccEg1KlX2m4LSR8ChkbEld2oWA90sl2sCfwb6TLY24CHgZ8Mau26p5N2eAD4J/AY8DywEfCVQa1dddXbb0Lr/cpi69dg1cnzAst5RwKzI5/b9oGOn50o6WjSvasPRMS8Qaxbt7XVFvm1M18HjulSvXqhk+3iBeDKiLgjIl4k3Zt4j6QVB7mO3dBJO5wFLEu6bzec9OScN+qZVb39JgziM1n7NVh18rzAyTmtVb4lVUfPTpR0CPnGaUT0Ww+4dttifdJN4/+VNJ20U3pL7vk0pgv17IZOtou7geLBW+3/frj60Ek7bEK6j/NsPoj7LrBl7oDyRlNvv/lERDwzaHPs9Y28QbxB+FPSpYrhwDak09RxdfIdSbqJ/lZSD5/JwJG9rn+P2mJ/YDqwUa/r3Mu2IL06582Fz4dJvaTeDAzp9TL0YLt4LzCD9BbupYFvAf/b6/r3oB0uAH4OrJjb4fPAY72u/wC3xVDS2eNppI4my5Iuh5fz7Zr3FW8n9R7+HW102npddet14wxio68EXAXMIV1n3i+P3450ma+WT6RLPs/mz9fJj6Hql08HbfEw8DLpFL/2+UGv69+LtiiV2YE+6w3YaVsAR5Hu1cwArgHW6nX9u90OpMt/Pyb9lOE54BZgy17Xf4DbYiLpzLn4mUi6VzkbeFsh72eAJ0j37y4Ahg1m3fxsQDMzq7x+vWdlZmZ9xMHKzMwqz8HKzMwqz8HKzMwqz8HKzMwqz8HKzMwqz8HK6pI0RlJIGpqHfyXp37ow34mSLhns+eR5HSzplsUsu0Ozd1zl9x6dVC+vpMmSdmhStittXZifJF0gaYakPw3gdF/Xuiy2oZmD1RJM0hRJLxReiHeBpBGDMa+I2C0iftRmnXYejDosSSLiyIg4uUHauIi4Cerv0Ntt6wG0LfA+YM2I2LKL822qWRsOtrxeXs7frdpnnUL6ppLulDQ3/920kHahpFMKw+MkTZN0XLeXo584WC359oiIEcBmwBbAieUM+cj5DbeuJQ3pdR2WEKOBKRExp9cVqRmsdSdp1Q7eqHBZRIwofP6Rp7EM8AvgEtKjhn4E/CKPL89vU+BG4NSI+M+BWYo3pjfcDqxfRcRjpCdAbwwg6SZJp0r6AzAXWEfSipLOy0d5j0k6pbZTkDRE0hmSnpb0D9KL5V6Tp3dYYfhwSfdKmiXpHqXXnl9MeizLNflI9Pic9935XUjPSbqreAlM0tqSbs7T+S3Q8KGgtctpkj6f6zlF0v6F9AslnSXpl5LmADvmZb5I0lNKryw/sRS4Jem7Sq/mvk/SToWEjxWW8R+SFnmDdIu6nFLOn9OmSNpZ0q6k58vtk9vrrgZtfUiuxwxJv5Y0ulZxSd9SesfUTEl3S9q4wTzXkHS10uvYH5R0eB5/KHAusHWuwyJve5W0Xl5HM/OyXlZI+46kRyQ9n88wtqs3/5z3ci14DfrvVXgbb4N1Vz5D2V3SX/J2dKukdxbSPpe36VmS7i+ux5JDgIclfVnS2o3q2sIOpGfofTsi5kXEf5Ee2/be0vJuCdwAfD4ivreY87KaXj+Lyp/F/wBTgJ3z/2uRHsJ7ch6+ifScs3GkL9bSpOefnU16YOdqwJ+Aj+f8RwL35emsRDoaDPJDLPP0Dsv/f5T0nLgtSF/S9YDR5Trl4bcCzwDvJx0cvS8Pr5rT/wh8ExgGbE96xcAlDZZ3B+CVQv4JpOe5jc3pF5IeQrpNnteywEWko+AVSE9SfwA4NOc/OE/v07l99snlV8rpHwDWzcs4gRT0N+ugLqcU8j7aYL1NLC9vqa33Ah4kvTtpKOnM+dactgtwJ/CmXMeNgLc0aLubge/nNtkUeIr0ZP1aO9zSZDv7CfCFQptuW0g7gPTMvKHAcaSHmy5bb9lIgWKF3F7fBv5SSKu37optuBnpmXxbAUNI79eakqc1FngEWCPnHQOs22R53k163cczpO38IGD5Up6JuT7Pkr5XRxXSPg38qpT/WuC4wrL8Jpc9sNf7iX75+MxqyXeVpNpDNW8mvYK85sKImBwRr5AC0G7AsRExJyKeJD09e9+c9/+RjhQfiYhnSU9dbuQw0uu974jkwYiY2iDvAcAvI+KXEfFqRPwWmAS8X+lts1sAJ0U6Qv096SGprdTy3wxcl+te84uI+ENEvEp6KO8+wH9ExKyImAL8J3BgIf+TeblfjojLgPvJZ5URcV1EPJSX8WbSDqh85tCsLgPh48BpEXFvXo9fBTbNZ1cvk3b+G5IevnxvREwrT0DSWqT7Up+LiBcj4i+ks6kDy3kbeJl0qXCNXP61TikRcUlEPBMRr0S6zFULHouIiPPzephHCgabaOF3Yr227iK9N6vocODsiLg9IuZHuqc3jxR45uf5vl3S0hExJSIearQwEXFbRBxFesvCWaTvwKOSzi1k+xkp+K+a5/1FSf+a09p5YeO787g36vuuBpyD1ZJvr4h4U0SMjohPRETxFeyPFP4fTTp7mJYvozxHOstaLaevUcrfKPhAOvtquDMoGQ18tDbPPN9tgbfkec6Ihe+VNJsvDfKvURguLsMqwDKlaU4lne3VPBYRUUpfA0DSbpJuy5fOniOdHRYvU7aqy0AYDXyn0HbPks6i3hoRvwO+B5wJPCHpHEnlFwmS6/RsRBRfjFduh2aOz/P8k1JPxkNqCZKOy5coZ+b6rUidS7lKl5lPl/SQpOdJZ0WU8j5SLlcwGjiutB2tRQqgDwLHkgLgk5J+KqnleshB827gL8BLwDsKafdExOM5MN4KfAf4SE5u54WNZwJ3AL+VNKpVXaw1B6v+VtwJP0I6El0lB7c3RcTIiKjdN5hG+vLXvK3JdB8hXR5rNc9a3osL83xTRAyPiNPzPEcpvZm3nfnSIP/jDeb/NAvOCor5HysMv1Va6Ib724DHJQ0jvbvoDGD1iHgT8EsWfuFgq7q0o9VrDx4hXaottt9yeQdKRPxXRGxOuty7AfDZOtN4HFhJUvHIv9wOjSsYMT0iDo+INUhnet/P97G2Az5HOpscldtoJvVfyrgfsCewMymgjcnji3mbtcUjpE4KxXZYPiJ+kut4aURsS1rXAXyt0YQkrSzpaKVu+r8jXcLcMSK2atYMhbpOBt5Z2m7eycIvbJxPej/cP4FfNziIsA44WL1B5MtDvwH+U9JISUtJWlfShJzlZ8AnJa2ZjwRPaDK5c4F/l7R5vsm/Xu2mP+n9NusU8l4C7CFpl3x0vaxSR4k186XDScCXJS0jaVtgjzYWp5Z/O2B34PIGyzw/L9epklbIdfxMrlPNanm5l5b0UdKln1+SzsiGke7tvCJpN+BfFrcuTTwBjFHj3po/AP6j1hlBqcPIR/P/W0jaStLSpPtlL5J2kguJiEeAW4HTcvu/EziU9G6mliR9VNKaeXAGacc9n3TZ6xVSGw2V9EUWPeOoWYF0sPQMsDwLX65uxw+BI/PyStJwSR/I63WspPfmA4wXgReo0w55WQ4lndVNAL5Mei/X8RFxbynfnpJG5XltCXySdO8T0j3F+aTtZpiko/P43xWnEREvk+7vPg38snRgYx1ysHpjOYi0E76HtNP5b9LlOEg7g18DdwH/R3qVe10RcTlwKnAp6dLHVaR7YpDudZ2YL9X8e95R7knq9fYU6Qj5syzY9vYj3TR/FvgSqUNEM9Nz3R8n7WyPjIj7muQ/hrQj/wfpvt6lwPmF9NtJr7F/Oi/TR/I9mFmkHdTP8vz2A65+nXWppxbcnpH0f+XEiLiSdJbw03z57G+ke4+QAsMPcx2mkgLBGQ3m86+ks5nHgSuBL+X7h+3YArhd0mxSG3wqIh4mbS+/InVamUoKFI0u5V2U8zxG2v5ua3PeAETEJNK9o++RlvdBUscQSAcVp5PW4XTSAcjnG0zqj6TOQB/N9yTrBjXSfawHSdv3RcDX8n0yIuIlUseXg0gvYTyEdDn+pTr1fon0pukXSb1kl+tgsa3AL1+0JYZSl/dLImLNVnnNrL/4zMrMzCrPwcrMzCrPlwHNzKzyfGZlZmaV52BlZmaV52BlZmaV52BlZmaV52BlZmaV9/8DI5e2Q8DTcY8AAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# plot histogram of predicted probabilities\n",
"\n",
"\n",
"# adjust the font size \n",
"plt.rcParams['font.size'] = 12\n",
"\n",
"\n",
"# plot histogram with 10 bins\n",
"plt.hist(y_pred1, bins = 10)\n",
"\n",
"\n",
"# set the title of predicted probabilities\n",
"plt.title('Histogram of predicted probabilities of salaries >50K')\n",
"\n",
"\n",
"# set the x-axis limit\n",
"plt.xlim(0,1)\n",
"\n",
"\n",
"# set the title\n",
"plt.xlabel('Predicted probabilities of salaries >50K')\n",
"plt.ylabel('Frequency')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Observations\n",
"\n",
"\n",
"- We can see that the above histogram is highly positive skewed.\n",
"\n",
"\n",
"- The first column tell us that there are approximately 5700 observations with probability between 0.0 and 0.1 whose salary \n",
" is <=50K.\n",
"\n",
"\n",
"- There are relatively small number of observations with probability > 0.5.\n",
"\n",
"\n",
"- So, these small number of observations predict that the salaries will be >50K.\n",
"\n",
"\n",
"- Majority of observations predcit that the salaries will be <=50K."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 18. ROC - AUC\n",
"\n",
"\n",
"\n",
"### ROC Curve\n",
"\n",
"\n",
"Another tool to measure the classification model performance visually is **ROC Curve**. ROC Curve stands for **Receiver Operating Characteristic Curve**. An **ROC Curve** is a plot which shows the performance of a classification model at various \n",
"classification threshold levels. \n",
"\n",
"\n",
"\n",
"The **ROC Curve** plots the **True Positive Rate (TPR)** against the **False Positive Rate (FPR)** at various threshold levels.\n",
"\n",
"\n",
"\n",
"**True Positive Rate (TPR)** is also called **Recall**. It is defined as the ratio of `TP to (TP + FN)`.\n",
"\n",
"\n",
"\n",
"\n",
"**False Positive Rate (FPR)** is defined as the ratio of `FP to (FP + TN)`.\n",
"\n",
"\n",
"\n",
"In the ROC Curve, we will focus on the TPR (True Positive Rate) and FPR (False Positive Rate) of a single point. This will give us the general performance of the ROC curve which consists of the TPR and FPR at various threshold levels. So, an ROC Curve plots TPR vs FPR at different classification threshold levels. If we lower the threshold levels, it may result in more items being classified as positve. It will increase both True Positives (TP) and False Positives (FP).\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAgEAAAEdCAYAAACVGrQcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd4FOX2wPHvCSHUhN67CtKLBBQRBbmKXVGvvaAIKnbFrldBro0iTbGAoiIWrg3bFZWLiPxARUVE6U2Q3kMJJHt+f7yTsFk2BUgymc35PE+ezMzOzJ6deuadd+YVVcUYY4wxxU+c3wEYY4wxxh+WBBhjjDHFlCUBxhhjTDFlSYAxxhhTTFkSYIwxxhRTlgQYY4wxxZQlAYVARJJE5EMR2SEiKiIN/Y7JTyLSS0TS/I7jUIlIV2/91fU7llghIitE5JFC/L5pIjI2YtiTIrLeW7e9RORxEVlSgDHUE5FvRGSXiAT+GW0Raegtu5Oi9R/BfAt0PRQEERkvIl/n07wKZd/INQnwfpR6f+kislpE3hCROlHGrSEio7zg94nIRhH5j4i0jTJuvIjcJiI/iMhOEdkuIr+IyMMiUim/fmARcTPQCegM1AL+yu8vEJEyIvKQiPwsIilewjFPREaKSNP8/r4j9C5w0PZTELyDvorIHRHDD+dANRO3/v7O1yAjhMWW8ZcmIn+JyAsikliQ352fRKSFiLzpHTNSRWSllwx38zGsC4G7w2I8HngQ6Itbt+8CQ4ATCjCGh4DqQFvvO/NVxDE7zVvuL4pIlfz+rmz8hftds/MysoiclM3FUUGvh/AY4kSkv4j87iVn20RkrogMKozvz0YH4LmC/pL4PI73HXAJUAI4GngemAScmDGCiNTDHSQ34E5684GawF3ALBG5QFX/641bEvgUd2IcCHwLbASae9PuAoYf4W/LMxFJUNV9BfgVjYH5qjrvSGaSXZwikoRbhrWAAcAPwDbgKOAfuGV8yZF8d35S1T3AnkL8yj3AYyLypqpuOdyZeMt+Xf6FlavzcesyHmgJjANKAb0LMYbDIiI9gI9wx4Q+wCKgHHAm8BLQxI+4oqz/xkBIVT+OGJ5yJN/jHePSNPrb2BoDP6jq4iP8jpyOWxnH7HigPTAWqAecfRjzOiSqmk4+7CeqmsIRrodD8C/gDuA24P+A0rh9rlCSkHAZ60JVNxbKF6pqjn/AeODriGG3AQokhQ2bjFvxSVHm8bn3WRmv/x4gBHTK5jsr5RBPPG6FLQVSgTXAqLDPFbgqYpqvgfFh/SuAQcALwGbgR+AtYEqU7/sCeCes/zTge9yJZQ3wGlAlh3hXeDFl/E3zhifiDoYbgb3AT8DpYdM19Ma/0lt+u4Ah2XzHKGA30CCbzyWs+zjvN23A7WA/AmdEifmRiGFjM2L3+k/ylsNO728u0CPs84eAZd462gh8Gbb+e+EOkJnrG5gArPKW60JvGwmPe7y3HvsCK4EdwMdAtVy232nevFcAI6Is35PChv0b+NNbln8BLwIVwj7v6k1TF1eKtgp4KOL7SgFbgZsi9pcF3npeDDwMxOcQ80GxecOHAnPzutyAbkA6UC9iPtd66yzR66/hLd+N3vDvgZPDxi8JDANWe+tzLWH7RJT4ywLrgS9y278jtzXgCtwV5HZgE/AZ0CRi+py2rbrA+960e7zx7o3YHsaGbVPh+6Z6wx8HlkR8Z477PQe2z9u83xQCykf57RrxN94bXgt4B5e87/HiTI6y7Z0NzPC2pVuzWb7jOfiY/bC3LZQhh2MLcIy3/LbhtuMpQKuIeV0CLPFimAmcR9j2SvR9q7q3zNZ70y0Erg8bN9oxMst6yOjHJccLvLj/BxwdEd/luPNDRnznRMYTZZn9SjbH17BxGgEf4EoCdwPzgKtzWvbk/Xib5XyUzb4R7y2D5d5vmw/cGDGvG3DHsL3evKYDdXP8XTl9mM2Pqo276kwDyoUdjNKJOHGETdPFWwnnhS3wr3P77mzm9bq3QK/GlUqcANwVsZPlJQnY4S3QJrgSiB7eb6gTNl4N73ee6fWf6q3823DZfAdvI5xO2Akr4rur4YoYp+NKRip7wyd5cfQAmgEjgH1A04gdaTVwFe6qvlGU+cd5K/vlPC6/rriTQHPvtw/yvrdJxPLJNgnAlQhtwZ0YGnt/PYEu3ucXesv3XKA+rtjzTrJPAmoC9+N2mEbe700BrovYDrcDb+My9BNxycDrufzeaV7sl4X/TqIfqB7BbasNge64A83rEctO8XYq4ClgQcT3XYzbASuFHbhWesunEXAW7qT9RA4xR4vtKNxOP+YQl9sC4LGI+X8HvOJ1lwH+wB34k3EngYdxJ9hm3jh347bDrt767ADcmUP8F0TGn8O4K8h6oLsOd9A+GmiHu7hYDCTkcduajNvf23rLsRtweeT24HVXwF39pXnLsmbYOgs/+eS63+O2zx3Ah953tyJKoud9z0zcRUdNLwbBJT6/4pLrVrhjxlagasS2twB30m1ENgd3oicBd3vTJ5LNsQV3vFsHjPFiOBZ3gbEZL9n21kkIt+0f662P5eSQBOC2sT+Bn3Elk0cBp+P2yRIcSCI6kPUYGbkeHsed+P+LK91oA/wCfBs2TnsvvkFefBfgEofckoAvcCfoOjmM0wq4BWiN2z5vw2073bJb9uT9eJvlfJTNvjEe+M1bdo2AS3HJWu+w354GXAM08OK9IbvtJHO+edhJx3szTsHtCBnZ2pCwcTp6w3pmM4/K3uf3ev27gZG5fXeU+RzjzefiHMbJaxLwTcQ4cbgM//6IHWctUCLsAPJ0xHT1ve9sm8sy/DrK7zgrYryfgVcjdqRHc1km1b3x7ooY/ra3zlKAlFzmMRd4OGL55JQEVPK+s2s287sLV/xbMpvPexGWBGQzzgjgq4hluBEoFTbsAWBtLvOZxoGD/kxgcsTyzenA0BN3Mozz+ruSNQlo6vUfHzbNZGCS113W29YjM/9rgG05fG9GbLu99bfX659KlKvLXJbb3bgkJOM3HOvNq0PYulhNxAnL+67hYfOcSjaJbpQY7vO+o3Iexj1oW4v4POPY0TmP29Zc4PG8bA/ZbYscfPKZRi77vbd9bstt/WQTQ3dvXs3DhpXCHXv+FbHtXZ2H+Y8n6/GmOe7KeFbE9vVoxHSPZ4wTNky8ae/0+icAMyPGuZWck4De3jacXdJykjd+w1zWw+O4c1G1sGGX4U76pb3+t4DvIuZzE7nv602B3715LcRdbF5JDiV23nQf4yXU0ZZ9Dtto5PH2myjjrcDbN3An/RDeRWLYOP8CfvW6e+IulA4qjc/pL69PB8zGZbcdgSeAWcCjYZ9LLtNrRL9EGZYXx3n/pxzGtJF+CO9R1RBuA7o6bPDVwFvq7nGBdwXkVbxLEZEU3FUUuCuEvGru/Z8eMXw60CKnOKPIbtnfhVtnj+HuxbqRRap5FcwWeJVfUrzvbJCnyAFV3YpLCr4UkS9E5AEROTZslPdwRcgrvUpKV+dUoc2rlPOAiPwqIpu8mG6KEtOfqpoa1r8Gd/WSV3cB54jIqdnEcaGITBeRv70Y3gIScFcnB1HVBbirh2u86asCZ+AOIOCWaxng/Yht5iWggohUyyXe63DrsDWuxKg8MFlE4rzvy8tyG49LFHt4/X1wtxR+9Pozrr62RcTYhQPb9Gu4q4olXgWzi0QkIYe4czseZD+hSFuv8uByEdmJKzUh7Dfltm0NBx4Skdki8oyInHy4sYTJ637/p7r72IeqBbBZVTPmibedz+bQjwcZunqx7sGd3JbhbrXkNK8OQPuI37kTd1LP+J3NcbdFws3IJZb2wB+qujqPsefkb816r3wNbnurHhbfrIhp/i+3mXr7cisv1tG4/X4srj5bGQARKSsiT4vIfBHZ4i2fs8jh2HkIx9vc1muy9zt/ilg/D3Fg3XyFW8/LReQdEenrHZNylNckYI+qLlHV31X1X7gri+fDPl+My1JaZjN9xvCFYf8jN+78ohx8ECoZZbxdUYa9DrQQkfYi0hJ3AH4j7PM44BlvePhfY1xx0pGKlhxFizPcRlyxYfPwgaq6TlWX4O7BhRuPO8Df5/1viyuGDD+oh8hlGapqH9wO8xVwCvC7iNzofbYGl1lfj7t18yiw0Ks8Gs09uBrao3D3XtvidsDIE01kxaVo6zpbqjobd991GBHbvldLfBIuEeuJSzhv8j7O6YT3OnCpd1K8HLcu/ut9lvEd/yTr9tIKt83kVklxjbffLVLVKbii6264q0LIw3JTVxHuP0Afr7LaNcDLYd8Rhyuqjdymm+ESBlT1V9yVSH/cOhgB/OpVSI0mYz9vns3nUYlIWVyCr7htpyPuxKQZvym3bUtVX8MdYF/E3Wf/QkQmHEocUeR1v89tX81JtIuiwzkeZMi4cGuGu1Vymqouy2VeccA3HPw7j8VdhWcXU14czjTRRDsGQNb9+bC+S51fVHWUql6O26fac6BS9WDc7ZOBuP2wLa5ORU7Hh/HkfryF3Ndrxu87kazrpiXuIgEvAU3GHb8W4Y5fS0SkfV5mfKgeB64VkWTvy7fgdoZbsjkwPIQ7GX3l9U8AThWRTtFmLtk/Iviz9//0HGLbgKu3kDGvUuTxYKSq873vuMb7+1VVfwsb5SeghXdgjvw7lCuA+d7/yKuULmGf5YlXgjERuFJEGuVhkpOBF1R1srqnFdbi7tGFy7IMPe2ifPfvqjpMVc/E1VzvG/ZZqqr+V1Xvw530yuLuz2UX039VdZy3Ey7h0EpWDsUDuIPadRHDTwI2qeojqjpbVRfhKpnl5m3cfdazcSVHE1U14x0I83HFoEdls82kZzPP7GTMt6z3P6/L7SXcPfSbcKVCb4V99hNu/e+IEl/mo5CqmqKqH6rq7bgDTTNc8hfNFNw29HC0D3PYv5vh6tA8rKr/U9U/cbeesiR6uW1bqrpWVV9T1WtwRdFX5pCw5EV+7ffZmQ9UFZHM45R33OrIIR4PwmRcuK2IKD3LyU+4i7M1UX5nxtX3fNyjzuEi+yPNwV1cZbc/ZZzYS+Qxzpz8gXvqLNzh1vD/0/ufUcpwMq5k+F1VnYu76s7tKZe8HG/zYo73v36UdbM0YyRVTVfV6d7Fenvv+yJLgLI4rCTAKzr5FFc5JMMtuIp1U0XkDHEvxOggIhNxWVMvdY+GgbuS+AZXnNxfRJJFpIE33Ud4xatRvncJ7gD2gohcJSJHe98R/gz418BNItLJu5ofT86ZWqTXcVd0V5K1FADc/ZfzReQ5r9jyaC/mcRlFRnnhrbRJ3u/oISJNRWQELqsbfAixZngYVxozS0T6eSUZDcU9j30F7so+w0LcQbGVuPc3vM3BO9/XuKvb00XkWBF5jrDiKxE5xitqPclbb51wCcwf3ue9RaSPiLQRkQa4ZZnIgSLUSAtxxZfdRKSJuGdzjz+M5ZArVV2FKwm4J0oM1bzYjxKRa4B+eZjfFlwN9n/hrlrfCPssBXgSeFJEbvWWZQsRuUxEnslDuJVFpKaI1PaW8TDcyXVmWMy5LjdVneGNOwR4T1W3h338Fq5i12fe+m4oIseLyIMicgGAiNwrIld6sTfCXYWn4642oi2T3bh77d1E5GsROdNbpq1EpD8HF9dmWImrg3Gbt291xx0rMq/sctu2RGS0iJzlTd8CV3HtL1yx9uHKl/0+B1NxxcETRaSzd9x6A/eY2ph8mH9ejcYdCz4SkS7etnCSiPxbRDIeB38O6OQNayIiPTl4X4r0Nm7dThaRf4hIIxHpLiKXep+vxB2jzhKR6iJS4Qh+wzCgs4gM9OI7Lyy+bEsIROR9EbnHO2808H7vm8B+3P4Nbh86X0Q6egnbyxx8sRQpL8fbXHnnvleBV8TdAjvG2weuF5H7vd9wvojc5R3/6+MS43pkf9zNnPkhVTIJG94Zt1C7hw2rhbtNsBKX3W3C1TpuF2X6eFzx5k+4opAduJqeDwEVc4inJK5ewgrvO1bjVWDyPq8JfOLN7y/ceweiVQzM7kmGqt589wM1onzexZvfTi/uP3H3IXN65OugZQgkceARwVSyf0Qw1xrW3vhlcbXbf8VVKEvF1Yp9mbBHfHBXTjNxjyGtwJ3oIpdPIm4H2Io76TxO1oqBtXCPymQ8MvY38Are43S4A+9Mb/rduHuSvcPm34usTwdUwN3r3YGrifx8xjrOZRlehfdYVw7LZRphlbC8YeVxGXKW5et953pvvX6OSwYzKywRUTEwbLrzveHzsomht7de9nrLZDZwcw4xZ6z7jL8Qrtb2h0DLQ1luYePe4c3roMdygSq4k80a3La/xvuudt7nN+KuRHZw4DGn8/OwTbbCJRl/e/Nd5W034Y8friBrDeiLcQntXtzx4BRcCUivPG5bz+OSkz3eMvkMdxUfdXuI3Ba9YY9z8COCOe735KFCWC7bZOQjgt8S/RHBHGt65yUWcji24JL9tzhwXFqJK7ltFDbOZRx4RHs2B7b/nB4RrIlLbDZ563ZBxjr1Pr/P2+7SyeURwYh4D6pUyIFHBFNx9QEu8cZpn8My6YMrqV7LgUfPPyJsf8GdUL/01v9a3DtZxpH10eksy568HW9XEOV8FDkclzzc5y27jPPrt8A/vc9PxiWUGY+dL8aVfOZYoTfj8RZjTAwTkWdxj7q28jsWYwqTV6r3Gu69Dtv8jqeoyesbA40xAeQVrbbCXenc5XM4xhQ475bT/3AVbzvgKnVOsgQgOksCjIltH+PqCbzLwXVcjIlFrXH1ACrjbglPwD0qbaKw2wHGGGNMMWVNCRtjjDHFlN0OKARVq1bVhg0b+h2GMcYEypw5czapam5v1jRHwJKAQtCwYUN++uknv8MwxphAEZGVfscQ6+x2gDHGGFNMWRJgjDHGFFOWBBhjjDHFlCUBxhhjTDFlSYAxxhhTTFkSAHgtvP0kIqkiMj6Xce8SkXUisl1EXhXX5KcxxhgTOJYEOH8Dg3BNNWZLRHrgWmXqjmsp6yhcS1LGGGNM4Nh7AgBV/QBARJKBujmMei0wTlXne+M/gWt284ECD9IYU+SEQsr+UIi0dCUtpOzYs5+U1DRUQVH3P7wbCKnrxhsWUq9Jd9y4q7fuJiHeXZ+FVAmFDkyHesO8eYa8if5Yu5Mq5RJIVyU9dODvj7U7SCwVT3wJAciMIaM7oy/rcM0yTvhwsgwPm9b7KON3Zsx5x9/LSarViFev60D5Una6KYpsrRyaFrgGWTLMBWqISBVV3Rw+ooj0BfoC1K9fv/AiNCYGpIeUTSmp7N2fzuqtezKHpXttoKeH3Mlw7/50Vm3eTWLpeNK9k2nGSTLjZBsKKfvSQ8xbs51aFcqQHgplTp8eUpZtSmHb7v0kxMd5J7Swk2zIxZM5r8yTtbIpZZ9/C6iIS9+9na1Tx7Jr/v+oceVg0q9J9jskkw1LAg5NeWB7WH9GdyKQJQlQ1ZeBlwGSk5OtlSYTc0Ih5a+tu1mzdQ/KgZN0KOxKdPGGFJK8E3Qo5E7QGeOs2baHvftDlCwhzFuzg5IlhJTUNJZt3OX3TztkZUqWIL6EECfC9j37aVozEYA4EURwfwhxAojg/XOfc+BzBFBYs20P7RtUIk5AMubhTX9gnuLNw322YedeWtetSIk4oYSI+x8n7Nizn6Orl3ffDYCbznW5+Rzo9oZnxON9kDGpiIR1HxhHMkJXZeoXHzNswIPs3bGN62/rT6+br6BsqRIFs+DNEbMk4NCkAElh/RndO32IxZgCoaqs3Lybn1dt5bPf1lKjQmnS0l2R98+rtrJi826SSsezY29agcdSr3IZUveHCCk0rZlIXJxQwjsRxsW5k+LufemUiBMaVilHibjwE+WB7jiBtJC7im9Quaw3H3eSjIsTVJWmNZMyT+RCxAncm8eBE7n7vEzJEpSzYu5MI0eO5F933EFycjLjxo2jdevWfodkcmFb76GZD7QB3vP62wDrI28FGFOUpYeUZRtTmLV8C0s3pLBg3Q527Enjz3U7KJcQT0pq7if3yASgTd0KJJYumXmSdidjIT2k7ExNo3mtpAMn6LAT8PY9+2lUtRwVy5YkPQRHVStHmZIlqF+5LGUTSmRepZqiS1XZsmULVapU4corr0RVueWWW4iPt9NLENhaAkQkHrcsSgAlRKQ0kKaqkUfDN4DxIvIWsBZ4BBhfmLEak5vd+9LYuDOVbxdtZP6aHSTEx7EvLcSMJZtITUvP8V52ZAJwdLVytK5bkROOqkyJuDhKlnAn7+PqV6JK+QQSSsTZiboYW7ZsGX369GH79u3MmjWLKlWqcMcdd/gdljkElgQ4jwCPhfVfBQwQkVeBP4DmqrpKVf8rIs8C/wPKAO9HTGdMoft51VYm/bSaOSu3sGh9Sp6nq1OxDLUqlOac1rUomxBPo2rlqFwugdoVylC6pJ3cTfbS09MZNWoUDz/8MCVKlGDw4MHExdkT50Ek4Y99mIKRnJys1pSwORLLNqbwf8s2sz8txM69aXw892+qlEtg9vIt2U5TtXwC1RNL06J2Esc1qESp+DhKloijZZ0KNKparhCjN7FkzZo1XHTRRcyePZuzzz6bF198kbp1c3qy+vCJyBxVtUcLCpCVBBhTBC3bmMIfa3fwwv+W8sfaHVHHWRLRf3H7upzcpBonHFWZ6omlCz5IUyxVqVKFhIQEJk6cyGWXXWYlRgFnSYAxPlNVvvh9HdMWbuD9n9eQHsq+dO6aTg1IKBFHiRJCfJzQvVkNaiSVpk7FMoUYsSlufvzxRwYOHMjEiRNJTEzk22+/tZN/jAh0EiAiVYHTcLX0KwLbcC/w+VpVN/oZmzGR9u5P58+1O5j71zam/LGemUs3U7V8Qo4V9RpVLcd1nRtybuvaVCxb0g68plDt3r2bxx57jGHDhlGzZk2WLl1K27ZtbTuMIYFMAkSkCTAQOB34FfgTlwAkAn2A50VkCvCYqi70LVBT7G3ZtY/Xvl/OqKmRhfdOZALwr3Oa06ZeRdrVq0hcnB1ojX+mTZvGDTfcwNKlS+nbty/PPvssFSpU8Dssk88CmQTg3tc/FLhOVfdEfug94tcTeBPoWMixmWLury276T9pbraV9upXLkvFsiXp2LAyl3aoR40KpSmXEE8JO+mbIkJVeeKJJwCYOnUq3bp18zkiU1Ds6YBCYE8HxK609BCrt+5hyh/rmLZwI/NWb2dnxLP28XFCw6rleOai1hxXv6IVpZoi67PPPqNt27bUqVOHtWvXUqFCBcqWLetbPPZ0QMELaklAJhHpB7yjqtk/K2VMPklNS+e171cw/OtF7N0fynHcS5Lrcv8ZTalSvlQhRWfM4dm4cSN33nknEydO5I477mD48OHUqlXL77BMIQh8EgCcAzwrIl/jiv8/UVVr3svku+e+WsSIbxZH/axx9fKUTSjBuW1q06FhZVrVqWD39E2Rp6q8++673HbbbWzfvp0BAwbwwAPWMnpxEvgkQFXPEpHqwOXAA8DLIjIJeENVZ/obnYkFyzftotuQaVmGdWlclZtPOZrjj6pi9/JNYI0cOZI777yTjh07Mm7cOFq2bOl3SKaQBT4JAFDVDcAIYISItAVeB/qIyApcc76jVHW3jyGaAFJVbnv7Fz79bW2W4d/d1416lf27T2rMkQiFQmzZsoWqVaty9dVXExcXR79+/ShRwpr7LY5ipmKgiJyCe+f/hbh3BbwOrALuACqqale/YrOKgcGiqoyeuoShXy3KMnzoP9twUfuCeT2qMYVhyZIl9OnTh507dzJr1qwi39KfVQwseEV7C8gDEXkadytgD65OQDtVXRX2+feAVRo0eZKSmkbLx77MMuzyjvV56sJWPkVkzJFLS0tjxIgRPProo5QsWZKhQ4falb8BYiAJwL0p8DJV/b9oH6rqPhE5oZBjMgGzYede3v3hryxX/23qVWTIxa1pXCPRx8iMOTJr1qyhZ8+e/Pjjj5x33nm88MIL1KlTx++wTBERC0nA7mgJgIgMUdX+AKr6e+GHZYJg++79vDZzOcO/zlrr/4aTGvHIOc19isqY/FO1alXKli3LO++8wyWXXGLvqTBZBL5OgIjsUNWkKMO3qGplP2KKZHUCip5lG1N4efoy3vnxryzDr+3UgH8m16NlHXs9qgmu2bNnM2DAAN59910SExNR1UCe/K1OQMELbEmAiFzjdcaLyNVA+BZ+FLCp8KMyRd3ufWnc//48Ppn7d5bhpzatzoNnNrWifxNou3bt4tFHH2X48OHUqVOHZcuW0aZNm0AmAKZwBDYJwDUUBJAA9A0brsB64LpCj8gUSekhZfayzXz95wZe/X55ls8u61CPB85sSsWyCT5FZ0z+mDp1Kn369GHZsmXcfPPNPP300yQlHVRIakwWgU0CVLULuKcDVNVecWWieuSjeUyYteqg4Zd1qMcTF7SkZIk4H6IyJn+pKk8++SQlSpRg2rRpnHLKKX6HZAIisElABksATKS/tuxm5DeLmTRndZbh5RJKcHqLmvTvcSx1KpbxKTpj8s8nn3xCu3btqFu3LhMmTKBChQqUKWPbtsm7QCYB4ZUBRSSEuwWQZRRAVdUehC1Gvpy/jme+WMCyTbsO+uzXf51mRf4mZmzYsIHbb7+dd999N7PBn5o1a/odlgmgQCYBQJuw7sa+RWGKhN/XbOecUTMOGn7SMVV58KymtKhtNf1NbFBV3nrrLe644w5SUlJ44oknuO+++/wOywRYIJMAVQ2v3VXG3gNQfP28aisXvpC1najRV7TjnNa1fYrImIIzfPhw7r77bk444QTGjRtH8+b2LgtzZAKZBET4VkTWABOBieGvDDaxa932vZw5Yjpbd+/PHDbgvBZce2JD/4IypgCEQiE2b95MtWrVuPbaa0lISOCmm26y1/6afBELVaNrAo8CbYE/ROQ7EblZRKr4HJcpIMOmLOSEp77JkgB8e29XSwBMzFm0aBHdunXjjDPOIC0tjcqVK3PLLbdYAmDyTeCTAFXdr6ofq+plQA3gJVyDQmv8jcwUhMcnz2fk1CWZ/QPOa8Hyp86iQZVyPkZlTP5KS0vj2WefpU2bNvz222/ceuutduI3BSIWbgcAICIJwOnA+cBxwCx/IzL57edVWxk/c0Vm//wBPShXKmY2YWMZhANaAAAgAElEQVQAWL16NRdccAFz5syhZ8+ePP/889SqVcvvsEyMCvwRVEROB64ALgCWAO8Ad6qqlQTEiPSQcte7vzI57FW/y548i7g4exWqiT3VqlUjKSmJSZMmcdFFF9krf02BCnwSAIwG3gaOV9WFfgdj8s/mlFROe246W3btyzJ80k2dLAEwMeX//u//GDBgAJMmTSIxMZGpU6f6HZIpJgKfBKhqE79jMPlr7/50zh01g8UbUrIMP7paOT67vQulS9q9URMbUlJSeOSRRxg5ciT16tVj+fLltG7d2u+wTDESyCRARB5Q1ae97n9lN56qDiy8qEx+mLF4E1eNm51lWHKDSkzscwIJ8YGvx2pMpq+++oq+ffuyYsUKbrnlFp566ikSE60VS1O4ApkEAEeHdR/xGwNFpDIwDlexcBPwoKpOjDJeKWAE0BMoCXwP3GT1D47c4vU7OXPEd6SFDrwB+uxWtXj+yuN8jMqYgqGqPPPMMyQkJDB9+nS6dOnid0immApkEqCqfcK6r86HWT4P7MM9YtgW+ExE5qrq/Ijx7gA6Aa2B7cArwCjgwnyIoVjanJLKRWNmsmLz7sxhyQ0qcW6b2vbcv4k5H374IR06dKBu3bq89dZbJCUlWYM/xleBL18VkQ3ZDP872vAo45UDLgIeVdUUVZ0BTAaiJReNgC9Vdb2q7sU9idDi8CIv3valhRj21SLaD/o6MwGoXC6Bz24/if/cfKIlACamrFu3jn/+859ceOGFDB06FIAaNWpYAmB8F8iSgAgH7UUiEg+UyuP0TYB0VV0UNmwuEK1B7nHACBGpDWwDrgS+iDZTEekL9AWoX79+HkMpHrbs2sdxT3yVZdgNJzXi4bOb2eNQJqaoKm+++SZ33nknu3fv5sknn6R///5+h2VMpsAmASLyP1wTwqVFJPJ5mrrk/WVB5XFF++G2A9Fq6CwCVuHeRpgOzANujTZTVX0ZeBkgOTk5sqnjYi08AWhcvTxv9z2BquXzmrMZExwZDf507tyZsWPH0rRpU79DMiaLwCYBwARAcPfo3wobrsB64KtoE0WRAiRFDEsCdkYZdwxQGqgC7ALuw5UEHJ/nqIu5V2ccaADynNa1GH2FVfwzsSUUCrFp0yaqV69Or169KFOmDH379iUuLvB3X00MEtVgX6SKSMsjaUrYqxOwFWihqou9YW8Af6vqAxHj/g48rKofe/0VvWmrqeqm7L4jOTlZf/rpp8MNMWZEPv634umzfYzGmPy3cOFCbrjhBnbt2sUPP/xAfHyQr7P8JyJzVDXZ7zhiWSC3UBG5XFXf9nqPE5Gol5Oq+kZu81LVXSLyATBQRG7APR1wPnBilNF/BK4RkWnAbqAfLlnINgEwTkpqWpYE4Lv7uvkYjTH5a//+/QwdOpTHH3+csmXL8txzz1mDPyYQApkEAL1wrwoG6JPNOArkmgR4+gGvAhuAzcDNqjpfRLoAX6hqeW+8/sBIYDGQAPyOe2eAycVlL/9fZvc395xCvcplfYzGmPyzevVqzjvvPH755RcuvvhiRo0aRc2aNf0Oy5g8CWQSoKo9wrqP+C0bqroF1wBR5PDvcBUHM/o3454IMHmkqvQYPp1F690rgO8741iOrlY+l6mMCY5q1apRpUoV3n//fS680F4ZYoIl8DVVRKSyiJT1uuNE5GoRuVzsWbMi4fTnDiQALesk0a/rMT5HZMyRmzFjBqeddho7duygVKlSfPXVV5YAmEAKfBIAfA5kPHfzb+Bh4CFgiG8RGQBue/uXzEaAerarw6e32atRTbDt3LmTW2+9lS5durBkyRJWrlzpd0jGHJFYSAKOBX7xuq8CegBdgcv9CsjAb6u38clc99LG1nUr8NylbX2OyJgj8+WXX9KyZUteeOEFbr/9dubNm0erVq38DsuYIxLIOgER0oGSItIE2KmqK71bAXbj2Sfrd+zlvNHfZ/b/56ZoD1oYExyqypAhQyhbtiwzZszgxBNtmzaxIRaSgC9x7/Cv6v0HaA6s9S2iYu6acT9kdr90dXtrAtgE1vvvv0/Hjh2pV69eZoM/pUuX9jssY/JNLBydbwC+Bt4EnvKGVQcG+hZRMdbwgc9YuN69bPHeHsfSo4U9KmWCZ+3atVx00UVcfPHFPPfccwBUr17dEgATcwJfEqCqe4AXIob9z6dwirWLx8zM0t+ny1E+RWLM4VFVxo8fz913382ePXt4+umnueeee/wOy5gCE/gkwHt17924N/1lqQegqqf6ElQxdPyTX7N+R2pmv70S2ATRsGHD6N+/P126dGHs2LE0adLE75CMKVCBTwKAibiT/yTcq3xNIVJVWj8+hZ2paZnD/hx4ho8RGXNo0tPT2bhxIzVr1uT6668nKSmJ3r17W4M/pliIhSTgJKC6qu71O5DiZn96iK6Dp2VJAJY9eRZxcfaeJhMMf/75J71792bPnj38+OOPVKpUiT59snsTuTGxJxZS3XlAbb+DKG527t1P44e/YM22PQBULpfAiqfPtgTABML+/fsZNGgQbdu2ZdGiRdxzzz3W4I8plmKhJOAr4AsRGQesC/8gL60ImsNz/fgfM7vPaFGTF69u72M0xuTdX3/9xTnnnMNvv/3GpZdeysiRI6levbrfYRnji1hIArrjWv87N2L4obQiaA7BK9OX8eOKrQC8dl0Huh1rB1ATHDVq1KBmzZoMHDiQ888/3+9wjPFV4JOA/GhF0OSdqvLvz//M7LcEwATB9OnTGTBgAB9++CFJSUl8+eWXfodkTJEQC3UCEJFKXsuBd3v9NUXE6gnks7T0EP8Y9m1m//wBPXIY2xj/7dixg379+nHKKaewfPlyVq1a5XdIxhQpgU8CRKQLsAjoDQzwBjcFXvQtqBikqhzz8Bcs3bgLgH/3bEm5UoEvSDIx7PPPP6dly5a8+OKL3HnnncybN4+WLVv6HZYxRUosHMVHAFeq6hQR2eoNmwV09DGmmHPac9Mzu09pUo0rj2/gYzTG5ExVGT58OImJicycOZMTTjjB75CMKZJiIQlopKpTvG71/u8DSvoUT8yZvmgjSzakANCzXR1rFtgUSarKe++9R6dOnahfv35mgz+lSpXyOzRjiqzA3w4AFojIPyKGnQr87kcwsWb11t1c8+qBVgEtATBF0d9//80FF1zAZZddxogRIwCoVq2aJQDG5CIWSgL6Ax+LyMdAGRF5Hujp/ZkjsG33Pk565kBbTJ/dfpKP0RhzMFVl3Lhx9O/fn9TUVIYMGcIdd9zhd1jGBEbgSwJU9Xtc40FLce8FWAt0UtXZvgYWcJtTUmk78KvM/oHnt6BF7Qo+RmTMwYYNG0afPn1o27Yt8+bN45577iE+PhaubYwpHDGxt6jqauBJABFJVNWdPocUaKpKtyHTMvtPaVKNazo19C0eY8Klp6ezYcMGatWqRe/evalUqRK9evWyBn+MOQyB3WtE5EoROS2sv52IrAC2ich8EWnsX3TBFQopXYdMY8de1yjQ4+c25/Xr7UELUzTMnz+fzp07c+aZZ5KWlkbFihW5/vrrLQEw5jAFec+5D9gY1j8WmA4cB8wAhvgRVND9Y9i3rNzsWmTu2LAyvTo38jkiY2Dfvn0MHDiQdu3asXTpUu6//35r8MeYfBDk2wH1gd8ARKQu0AY4XVU3i8i9wGI/gwuiVZt3s2yTexlQh4aVeO+mTj5HZAysWrWKc845h3nz5nH55ZczYsQIqlWr5ndYxsSEIJcEpHHgXQAnAgtUdbPXnwKU8SWqAOv9+oGWASfddKKPkRhzQM2aNalTpw6TJ09m4sSJlgAYk4+CnAR8BzwhIs2BW4FPwz5rCqz3JaqAWrYxhcXeC4EGnt/C52hMcTdt2jS6du3K9u3bSUhI4IsvvuDccyMbCjXGHKkgJwF3ACcAc3ClAk+HfXYtMCXaROZge/enc+rQAw0D2ZMAxi/bt2/nxhtvpFu3bqxevZrVq1f7HZIxMS2wdQJU9S/g5Gw+u7+QwwksVaXpo//N7P93T2tgxfjjk08+4eabb2bt2rX079+fAQMGULZsWb/DMiamBbIkQESq5vN4lUXkQxHZJSIrReSKHMY9TkSmi0iKiKwXkUC/nuyM4d9ldp/TuhZXdKzvYzSmuFJVRo8eTeXKlZk1axaDBw+2BMCYQhDUkoDvROQr4E3gJ1XNaDgIERGgPXAN0B3Iyw3u53GNDtXAvX3wMxGZq6rzw0fykor/AncB/wESgLpH/nP8s3C9e69SkxrlGX3FcT5HY4oTVeWdd96hc+fOWRr8SUhI8Ds0Y4qNQJYE4E7Uy3CvCd4hIr94V+e/ANuB8bhHBHM9q4lIOeAi4FFVTVHVGcBk4Oooo98NfKmqb6lqqqruVNU/8+cnFb5zRh0oBXi3rz0OaArP6tWrOe+887jiiisYOXIkAFWrVrUEwJhCFsiSAFVNBYYDw0WkEdAKqAhsBX5T1ZWHMLsmQLqqLgobNhc4Jcq4JwDzRGQmcAwwG7hFVVdFjigifYG+APXrF70i9r+37eH3NTsy+yuVs4OvKXihUIixY8dy7733sn//foYNG8btt9/ud1jGFFuBTALCqepyYPkRzKI8rvQg3HYgMcq4dXGlC6cB84BngbeBzlHiehl4GSA5OVkjP/fbqzMOLLJ5j5/uYySmOBk6dCj33Xcfp556Kq+88gpHHXWU3yEZU6wFPgnIBylAUsSwJCBaI0R7gA9V9UcAERkAbBKRCqoamUgUae/+9BcAd3RvTGLpkrmMbczhS0tLY8OGDdSuXZs+ffpQrVo1rr32Wlz1HWOMn4JaJyA/LQLiIxocagPMjzLub0D4VX1Gd6COZjMWb2Kn10BQ7y7WNoApOL/99hudOnXK0uBPr169LAEwpogo9kmAqu4CPgAGikg5EekMnI978iDSa0BPEWkrIiWBR4EZqrqt8CI+MqGQctW42QDExwlJVgpgCkBqaiqPPfYY7du3Z+XKlTz88MPW4I8xRVBM3Q4QkeqquuEwJu0HvApsADYDN6vqfBHpAnyhquUBVHWqiDwEfAaUxbVWmO07BYqijk9+k9n93KVtfYzExKqVK1dy1lln8ccff3DVVVcxfPhwqlSp4ndYxpgoAp8EiEgFYBRwCZAOlBORc4FkVX0sL/NQ1S3ABVGGf4erOBg+bAww5kjj9sPMJZvYlJIKQJ2KZTi3TW2fIzKxqFatWjRq1IjBgwdz1lln+R2OMSYHsXA7YAyQCjTGvfAH3KN7l/sWURH1zH8XZHZ//8CpPkZiYs3UqVM5+eSTMxv8+fTTTy0BMCYAYiEJ+AfuWf2/8CrqebcEavgaVRETCilzV7sHGIbbbQCTT7Zt20afPn3o3r0769atY82aNX6HZIw5BLGQBOwAKocPEJF6WFPCWfx3/rrM7n80t/zIHLmPP/6Y5s2b8+qrr3Lfffcxd+5cmjdv7ndYxphDEPg6AbgKfZO8CntxItIBeAp4yd+wig5Vpd9bP2f2ly8VC6vd+ElVGTNmDNWqVWPy5MkkJyf7HZIx5jDEwtngKVxdgHFAaWAiLgF4zs+gipKBn/6R2f2fm6yNAHN4VJUJEyZw8skn06BBA9566y0SExPtff/GBFgs3A6ooqpDVLWJqpZW1caqOoSIWwTF2WvfrwCgdoXSJDe0xWIO3apVqzj77LO55pprGD16NABVqlSxBMCYgIuFJGBZNsMXZTO8WBkX1kbAR7cc1MSBMTkKhUKMGTOGFi1aMH36dEaOHMnTTz/td1jGmHwSC0nAQe8fFZHyQMiHWIoUVWXMtKWZ/dWTSvsYjQmioUOH0q9fPzp16sTvv//ObbfdZm/+MyaGBLZOgIgsxz0SWEZEIksDqgLvF35URcuoqUsyXw70w0PdfY7GBEVaWhrr16+nTp069O3bl5o1a3LVVVfZ+/6NiUGBTQKAG3ClAJOBPmHDFVivqtEaACpWhn3l7ohULV/KSgFMnsydO5frr7+etLQ05syZQ4UKFbj66qv9DssYU0ACmwSo6jcAIlJTVXf4HU9RM2fllszuV3vZ41smZ3v37mXQoEE888wzVK5cmeeff96K/Y0pBgKbBGRQ1R0i0hLogrsNIGGfDfQtMJ/1eWMOAO0bVKJ13Yo+R2OKspUrV3LGGWewYMECrr32WoYOHWoN/hhTTAQ+CRCR3rgGhL4BTgO+AroDn/gZl9+27HLNKHRsZI8EmpzVrl2bxo0b89xzz3HGGWf4HY4xphDFwtMBDwBnqeq5wB7v/yXALn/D8s/7c1Zndt90ytE+RmKKqilTptC5c2e2bdtGyZIlmTx5siUAxhRDsZAE1FDVaV53SETigM+I0jRwcfH1n67ZhBJxQoUyJX2OxhQlW7du5brrrqNHjx5s3ryZtWvX+h2SMcZHsZAErBaRBl73YuBs4ARgv38h+euL311jQZPsFcEmzAcffEDz5s158803efDBB/n1119p1qyZ32EZY3wU+DoBwFCgJbASGARMAkoCd/sZVFHQsEo5v0MwRYSq8sorr1CrVi0+//xz2rVr53dIxpgiIPBJgKqOC+v+VEQqAaVUdbuPYflm5eYDVSEqlbVbAcWZqvLGG29wyimn0LBhw8wGf0qWtO3CGOPEwu2ALFR1LxAvIk/5HYsfvpzvbgVUKlvS3vBWjK1cuZIzzzyTXr168cILLwBQuXJlSwCMMVkEOgkQkWtF5DkR6Sci8SKSJCKDgRXAcT6H54t3fvgLgA7WWmCxFAqFGD16NC1atGDGjBmMGjXKGvwxxmQrsLcDRORZ4GpgJnA5rjJgJ2AOcJKqzvUxPF/s3Z/Osk3udkDTWkk+R2P8MGTIEO6//3569OjBSy+9RIMGDXKfyBhTbAU2CQAuA05W1cUi0gyYD1yuqu/6HJdvpi/amNndr6u9H6C42L9/P+vWraNevXrceOON1K1bl8svv9xuBxljchXk2wEVVXUxgKr+CewuzgkAwMvTXWOKlyTXpXRJe+97cfDzzz/TsWNHzj77bNLS0qhQoQJXXHGFJQDGmDwJchIgIlJPROqLSH0gLbzfG1ZsqCo/rdwKwGUdi9VPL5b27NnDgw8+SMeOHVm3bh0DBgwgPj7IBXvGGD8E+ahRDlcBMPySZ2VYtwLF5nJ44fqdmd3t6lmDQbFsxYoV9OjRg0WLFnH99dczZMgQKlWq5HdYxpgACnISYM86hRk2ZREAtSuUtqLgGKWqiAh16tShWbNmjB49mtNOO83vsIwxARbY2wGqmp7bn98xFqbd+9zPbVGngs+RmILw5ZdfcuKJJ2Y2+PPRRx9ZAmCMOWKBTQJMVjOWbAKgZ7s6Pkdi8tOWLVu49tprOeOMM9i+fTvr1q3zOyRjTAyxJCAGzP1rW2Z3kxrlfYzE5Kf//Oc/NGvWjIkTJ/LII4/wyy+/0LRpU7/DMsbEkCDXCTCeIVMWZnYfXc2SgFigqrz66qvUq1ePKVOm0KZNG79DMsbEoJgoCfBeGdxJRC72+suISJlDmL6yiHwoIrtEZKWIXJHL+AkiskBEVh9p7PlhwTr3ZMAd3RtbpcAAyzjxr1ixAhFhwoQJzJo1yxIAY0yBCXwSICItgAXAm8B4b3B34NVDmM3zwD6gBnAlMMabb3buBTYccrAF4Js/17NxZyoAV3eyV8QG1fLlyzn99NPp3bs3Y8aMAVyDP/bsvzGmIAU+CQDGAINU9RhgvzdsGtAlLxOLSDngIuBRVU1R1RnAZFy7BNHGbwRcBRSJVgq/WeBykYplS1K1fCmfozGHKj09nREjRtCyZUtmz57NmDFjeOqpIrFpGWOKgVhIAloBr3vdCqCqKUDZPE7fBEhX1UVhw+YC2ZUEjAIeAvbkNFMR6SsiP4nITxs3bsxp1CMycfYqAC7tUK/AvsMUnCFDhnDnnXdyyimnMH/+fG666Sbi4mJhtzTGBEEsHG1WAu3CB4hIMrA0j9OXB7ZHDNsOJEaOKCI9gXhV/TC3marqy6qarKrJ1apVy2MohyYU0szui4+rWyDfYfLf/v37WbXKJW8333wz77zzDp999hn16lkiZ4wpXLFww/FfwGci8gKQICL3ArcAN+dx+hQgst3dJGBn+ADvtsGzwFlHFm7+Wb31QGFE4xoH5SymCJozZw7XX389oVCIX375haSkJC699FK/wzLGFFOBLwlQ1cnAeUA94HvgWOASVf0ij7NYBMSLSOOwYW1wTROHaww0BL4TkXXAB0AtEVknIg0P+wccgbd/dFeTrevaWwKLuj179nD//ffTsWNHNm7cyKBBg6zSnzHGd4E/ColIJVX9EfjxcKZX1V0i8gEwUERuANoC5wMnRoz6Oy7RyHAiMBo4Dii4m/45GDPN3fE4o2VNP77e5NHy5cvp0aMHixcv5oYbbmDw4MFUrGiNPBlj/Bf4kgBgjYhMFpFLD+XdABH6AWVwj/29DdysqvNFpIuIpACoapqqrsv4A7YAIa+/0NspyHgsEODc1rUL++tNHqi6Oht169alZcuWfP3117zyyiuWABhjioxYSAIaAV8DdwHrReRNETlTRPLcjLCqblHVC1S1nKrWV9WJ3vDvVDXqK/hUdZqq+lYbb+bSTZnd9Srn9UEIU1g+//xzjj/+eLZu3UrJkiX54IMP6N69u99hGWNMFoFPAlR1vaqOVNUTcEX5C4EhwN/+RlawZix2ScCFx1mDQUXJpk2buOqqqzj77LPZvXs3GzYUiXdKGWNMVIFPAiJU8P4SgV0+x1Kg5q1xTzXWt1KAIkFVeeedd2jWrBnvvfcejz32GD///DPHHnus36EZY0y2YqFiYBPgcuAKXAIwCbhMVWf6GlgB824307xW5NONxi9vvvkmjRo1Yty4cbRq1crvcIwxJleBTwJwTwV8CNwOfO1HJb3CpqosXO9eY9C2vlUy84uqMm7cOLp3706jRo2YMGECSUlJlCiR5+ooxhjjq1i4HVBDVXup6pfFIQEA2LXvwM+sZu0F+GLp0qV0796dPn368NJLLwFQqVIlSwCMMYESyJIAEblcVd/2ei/JrvlcVX2j8KIqPP/9fR0AjaqWs6aDC1lGgz+PPPIIJUuW5KWXXuKGG27wOyxjjDksgUwCgF645/kB+mQzjgIxmQRMXbAegKTSQV19wTV48GAefPBBzjnnHMaMGUPdutZmgzEmuAJ5FlHVHmHdeWoyOJZ8Ps+VBJzUuKrPkRQP+/btY+3atTRo0IB+/fpx9NFHc/HFF1spjDEm8AJfJ0BEor4uWERmFXYsheGvLbszu6/p1NC/QIqJH374gfbt23P22WeTlpZGUlIS//znPy0BMMbEhMAnAUDTbIY3KdQoCskHP68BIE6gRlJpn6OJXbt376Z///506tSJrVu38tRTT1mDP8aYmBPYo5qIvOp1JoR1Z2gI/Fm4ERWOXfvSADilSTWfI4ldy5cv57TTTmPp0qXceOONPPPMM1SoYC01GmNiT2CTAGBNNt0KzAHeLdxwCsd33uuCT21Ww+dIYo+qIiLUrVuXdu3aMXbsWLp27ep3WMYYU2ACmwSo6qPg7v2r6md+x1NY/ly7A4A6Fe1WQH765JNPGDhwIFOmTKFSpUpMmjTJ75CMMabABbJOgIh0DuvdKSInR/vzLcACsnb7nszulrWteDo/bNy4kSuuuILzzjuP1NRUNm7c6HdIxhhTaIJaEjCOAxUC38pmHAXqF044hWP6ogMnqOpWKfCIqCpvv/02t99+Ozt27GDgwIHcf//9JCQk+B2aMcYUmkAmAaraNKy7np+xFKZJP60GILlBJZ8jiQ0TJ07kmGOOYdy4cbRo0cLvcIwxptAFMgnIiYh0AdJU9f/8jiW/ZTQa1KNFTZ8jCaZQKMQrr7zCaaedxlFHHcWECRNITEy09/0bY4qtQNYJCCci07wTPyLSH/gA+EBE7vc3svy1e18aO/e6xwO7Na3uczTBs3jxYk499VRuuukmXnnlFQAqVqxoCYAxplgLfBIAtAIyrvpvBLoCxwP9/AqoIPyyaltm9zHVy/sYSbCkpaUxePBgWrduza+//srYsWN58skn/Q7LGGOKhFi4HRAHhETkKCBeVecDiEhlf8PKXys3u9cFWwJwaIYMGcKDDz7I+eefzwsvvEDt2rX9DskYY4qMWEgCZgLDgdrAhwBeQrDZz6Dy20MfzgPgVLsVkKvU1FTWrl1Lw4YN6devH02aNKFnz572vn9jjIkQC7cDegF7gYXAY96w5sAovwLKbzv37s/sttcF52zWrFkcd9xxWRr8ufDCCy0BMMaYKAKfBKjqRlW9T1UfVtWd3rBPVXWo37Hll/D6AJ2PseaDo9m1axd33XUXJ554Ijt37mTIkCHW4I8xxuQi8EmAiMSLyKMiskhEdnn/HxWRkn7Hll/m/+1eFdygSlmfIymali1bRqtWrRg+fDg333wzv//+O2eeeabfYRljTJEXC5dKzwCdgTuBlUAD4BGgInCPj3Hlm19WbQXgqKrlfI6kaMlo8Kd+/fp06NCB8ePHc/LJMfe2aGOMKTCBLwkALgHOUdXPVXW+qn4OnA9c5nNc+WbKH+sBOLZmks+RFB0ff/wxycnJbNmyhfj4eN59911LAIwx5hDFQhJQAghFDAsBMVcT7OTGVh9g/fr1XHrppVxwwQWkpaWxadMmv0MyxpjAioUk4D/AZBHpLiKNReQfuEcF3/c5rnyhqpndTWsV35IAVeXNN9+kefPmfPTRRwwaNIiffvqJJk2a+B2aMcYEVizUCbgX92jgOKAW8DfwDjDAz6DyS0Z7AQCVyxXvFu7ee+89mjZtytixY2nWrJnf4RhjTOAFviRAVVNV9SFVbaiqpVS1kao+qKp78zoPEaksIh96TxesFJErshnvXhH5XUR2ishyEbk3/35JdMs27gKgVHzgV9UhC4VCvPDCCyxbtgwRYcKECUyfPt0SAGOMySeBPbN4Rf/TRWSLiHwtIvWPYHbPAwP4sQQAABfcSURBVPuAGsCVwBgRida2rADXAJWAM4BbRaRAKyCu2boHgE5HVynIrylyFi5cSNeuXbnlllsYN24cABUqVLAGf4wxJh8FNgkARgNrcG8M3IR7dfAhE5FywEXAo6qaoqozgMnA1ZHjquqzqvqzqqap6kLgY9zjiQVmyYYUAMqXioU7N7lLS0vj6aefpk2bNsybN4/XXnuNQYMG+R2WMcbEpCCfWdoD9VR1j4j8D1hwmPNpAqSr6qKwYXOBU3KaSNx7aLsAL2XzeV+gL0D9+odfSBEX5x5yqFq+1GHPI0gGDx7MQw89RM+ePXn++eepVauW3yEZY0zMCnISkKCqewBUdaeIlDnM+ZQHtkcM2w4k5jLd47iSlNeifaiqLwMvAyQnJ2u0cfJiww5XtaFd/YqHO4siLzU1lb///ptGjRpx66230qxZMy644AK/wzLGmJgX5CSglIj8K6y/TEQ/qjowD/NJASKfvUsCdkYZFwARuRVXN6CLqqbmMd7DsmSjux0QqyUBM2fOpHfv3sTFxTF37lwSExMtATDGmEIS5DoB7wGNw/7+E9F/TB7nswiIF5HGYcPaAPOjjSwi1wMPAN1VdfXhhZ53qfvde5DqVDzcgo6iKSUlhdtvv52TTjqJ3bt3M2zYMGvwxxhjCllgj7qqelDFvcOczy4R+QAYKCI3AG1xrx0+MXJcEbkSeBLopqrL8uP7c5PuvSyoTELs1IpftmwZp556KqtWreLWW2/l3//+N4mJud19McYYk9+CXBKQn/oBZYANwNvAzao6X0S6iEhK2HiDgCrAjyKS4v29WJCBhUIuCYiT4L8FOePth/Xr1+fEE0/ku+++Y+TIkZYAGGOMTwJbEpCfVHULcNCNaFX9DldxMKO/UWHGBQdKAkrEBTsJeP/993niiSeYOnUqlStXZuLEiX6HZIwxxZ6VBBRx6V5JQImAlgSsW7eOiy++mIsvvpi4uDg2b97sd0jGGGM8lgQUcZm3AwK2plSV119/nebNm/Ppp5/y1FNPMXv2bBo3bpz7xMYYYwpFTNwOEJFuwGVADVW9QESOAxJV9VufQztiXg4QyNsB77//Pi1atGDs2LEce+yxfodjjDEmQsCuLw8mIv1wLQj+BXTzBu8D/u1bUPkoo05AECoGhkIhRo8ezdKlSzMb/Pn2228tATDGmCIq8EkAcA/wD1UdBIS8YX8CMdHUXFCeDliwYAEnn3wy/9/e3cdZVdV7HP98YQhzBjRDFB9mvCKYUqAhKSoIeW+3ArkW1ytpPpSkEfBKp7SuXb0KXAqT7hU1H0BRGCXLREK9vky9PpBU1qtEHgpDQXkQlWcmcnDmd/9Y68DhdGY4Z+Y8zJnze79e+wWzz5q9f+vsM2f/9lpr7zVx4kRmzw4PUezevTudSq0fwznnykhH+IbuBqyJ/088nreC0BpQ8tr73QG7d+9m6tSpDBgwgOXLl3P//fczefLkYoflnHMuAx0hCVgEfDtl3Xig5McDmBkxB6Cd5gDcfPPNfO9732PUqFGsWLGCiy++GLXzVgvnnHNBRxgYOBF4TNLXgG6SlhFaAT5f3LDarnFPVwDt6sS6a9cu1q9fT+/evZkwYQL9+vVj1KhRxQ7LOedclko+CTCzdZIGAoOBasIAwcVm1ljcyNquPXYFLFq0iMsuu4zOnTuzZMkSunXr5gmAc86VqI7QHYCZNZnZr8xsnpkt6ggJALCnK6A9tALs2LGDCRMmMGTIEBoaGpgxY4ZP+OOccyWu5L/FJb3B3gGB+zCzYwscTk61l6cFrlq1iuHDh7N27VquvPJKpkyZQmVlZVFjcs4513YlnwQAY1N+7kUYJzCvCLHkVLG7A8wMSdTU1DB06FDGjx/P4MGDixKLc8653Cv57gAzeyZlqSNMBnRJsWNrq6akgYGFZGY8/PDDnHTSSWzatImKigrq6uo8AXDOuQ6m5JOAZuwCSrorAJK6AwqYBWzYsIHRo0dz3nnn0aVLF7Zs2VKwfTvnnCusku8OkHR9yqoDgRHAU0UIJ6cK2R1gZsyePZva2lref/99pk2bRm1trQ/+c865DqwjfMOnTktXD9wO3Ff4UHJr74OCCtMSsGDBAvr378+sWbPo27dvQfbpnHOueEo6CZDUGfgl8FMz+1ux48m1xjzPG9DY2Mhtt93GiBEjOO6446irq6OystKf9++cc2WipL/t4/MAbu2ICQDkd0zA8uXLOfPMM7nyyiuZM2cOAN26dfMEwDnnykhH+MZ/XFLJPyI4nabENMI5PEoNDQ1MnjyZk08+mddee426ujpuvPHG3O3AOedcySjp7oCoE/CIpEWERwbveXCQmX21aFHlQD4eFjR9+nSuv/56xowZwy233ELPnj1ztm3nnHOlpSMkAa8BPyx2EPmwtyWgbUlA8oQ/EydOpH///owYMSIXITrnnCthJZsESPpSnCvgumLHki+NTeHftrQEPP/884wdO5YuXbrw6quvUlVV5QmAc845oLTHBNxV7ADyrakNzwnYvn0748aNY9iwYTQ1NXHbbbfRuXPnXIfonHOuhJVsSwBQ/Kn18iwxJiDbWQRXrVrFsGHDWL9+PbW1tUyaNMkn/HHOOfd3SjkJ6CxpOC0kA2b2bAHjybm9LQEZlm9qolOnTtTU1DB8+HDGjx/PqaeemscInXPOlbJSTgK6AvfQfBJglPj8AZneHWBmPPTQQ0yZMoXnnnuOHj167Ln33znnnGtOKScB9WZW0if5/cnk7oB169Yxbtw4Fi5cyKBBg9i2bRs9evQoVIjOOedKWCkPDOzwWro7wMyYOXMmJ554Ik8//TTTp09n8eLF9O7du8BROuecK1Wl3BJQNgMDm2sJeOKJJxg4cCAzZ870k79zzrmslWwSYGbdih1DvlmiOyDmAI2NjcyYMYORI0fSp08f5s6dS2VlZdZ3DzjnnHPg3QEASDpE0nxJ9ZLWSLqgmXKSNE3SprjcpDyegRuTnhOwdOlSTj/9dGpra5k7dy4AVVVVngA455xrtZJtCcix24EG4DDgJMKkRK+Y2bKUcpcD5wIDCHcf/BJ4HbgzH0E1NhnWuJulC+/hk9+4n4MOOoh58+Zx/vnn52N3zjnnykzZtwRIqgRGA9eZ2U4zWwT8ArgoTfFLgOlmttbM1gHTgUvzFVuTGdt/O58lv5jFeeedx4oVKxgzZoxf/TvnnMsJbwmAvkCjma1MWvcKcFaasv3ia8nl+qXbqKTLCS0HVFdXtyqwik6dqBkymiGnDeKBaRNatQ3nnHOuOZ4EQBWwLWXdNiDdwMPUstuAKkmyxCi+yMzuBu4GOOWUU/Z5LVND+x7KH6eMas2vOuecc/tV9t0BwE6ge8q67sCODMp2B3amJgDOOedcKfAkAFYCFZL6JK0bAKQOCiSuG5BBOeecc67dK/skwMzqgUeASZIqJZ0B/AswN03xOUCtpCMlHQF8C7ivYME655xzOVT2SUD0DeDDwDvAPGCcmS2TNETSzqRydwELgVeBpcDjcZ1zzjlXcnxgIGBmmwn3/6euf5EwGDDxswHXxMU555wrad4S4JxzzpUpTwKcc865MuVJgHPOOVem5Le455+kd4E1bdhED+C9HIVTKsqtzuVWX/A6l4u21LnGzA7NZTBuX54ElABJvzOzU4odRyGVW53Lrb7gdS4X5VjnUuLdAc4551yZ8iTAOeecK1OeBJSGu4sdQBGUW53Lrb7gdS4X5VjnkuFjApxzzrky5S0BzjnnXJnyJMA555wrU54EOOecc2XKk4B2QNIhkuZLqpe0RtIFzZSTpGmSNsXlJkkqdLy5kEWdr5a0VNIOSW9IurrQseZKpnVOKv8hSX+StLZQMeZaNnWW9ElJL0jaKWmjpG8WMtZcyOJz3VXSnbGemyUtlHRkoePNBUkTJP1O0vuS7ttP2askvS1pm6R7JXUtUJiuGZ4EtA+3Aw3AYcCFwB2S+qUpdzlhtsMBQH9gJHBFoYLMsUzrLOBi4CPAZ4EJksYULMrcyrTOCVcTprcuZRnVWVIP4EnC1NwfBY4DnipgnLmS6TH+JjCY8Hd8BLAVuLVQQebYemAKcG9LhST9M/Bd4GzgGOBY4MZ8B+da5ncHFJmkSmAL8HEzWxnXzQXWmdl3U8q+BNxnZnfHny8DvmZmpxU47DbJps5pfncG4XM7Mf+R5k62dZb0D8ATQC0w08yOKmS8uZDlZ3sqcLSZXVT4SHMjy/reAewws2vizyOAH5nZ8QUOO2ckTQGOMrNLm3n9QWC1mV0bfz4beMDMDi9clC6VtwQUX1+gMfGlEb0CpLt66Bdf21+59i6bOu8Ruz6GAMvyGFu+ZFvnW4FrgV35DiyPsqnzacBmSS9Jeic2j1cXJMrcyaa+9wBnSDpC0oGEVoP/LUCMxZTu++swSR8tUjwOTwLagypgW8q6bUC3DMpuA6pKcFxANnVOdgPhMzs7DzHlW8Z1lvQFoMLM5hcisDzK5jgfBVxCaCavBt4A5uU1utzLpr4rgTeBdcB24ARgUl6jK75031+w/797l0eeBBTfTqB7yrruwI4MynYHdlrp9elkU2cgDD4ijA0YYWbv5zG2fMmozrFJ+SagpLo7mpHNcd4FzDezl83sb4S+4tMlHZTnGHMpm/reARxAGP9QCTxCx28JSPf9BS383bv88ySg+FYCFZL6JK0bQPom72Xxtf2Va++yqTOSvkocUGRmpTpSPtM69yEMmnpR0tuEk0OvOKL6mALEmUvZHOclQHIym/h/KbVyZVPfAYTxPZtjUnsr8Kk4QLKjSvf9tdHMNhUpHgdgZr4UeQF+Qmj6rATOIDST9UtT7uvACuBIwojiZcDXix1/nut8IfA2cEKxYy5EnYEK4PCk5YuE0deHA52LXYc8HudPEwbVnQR0Af4beLHY8eexvrOBnwMHxfpeSxhAWPQ6tKLOFYRWje8Dc+P/K9KU+2z8Wz6RcLfPs8APih1/uS9FD8AXAzgEeBSoJ/QTXhDXDyE09yfKidBUvDkuNxHv8Ci1JYs6vwHsJjQlJpY7ix1/Puuc8jvDgLXFjr0QdQbGEfrItwALCXcLFL0O+agvoRvgAcItoFuBRcCnih1/K+t8A6HlJnm5gTC2YydQnVS2FthIGAcxG+ha7PjLffFbBJ1zzrky5WMCnHPOuTLlSYBzzjlXpjwJcM4558qUJwHOOedcmfIkwDnnnCtTngQ455xzZcqTANeuSaqTdEOx49gfSX+WNKSF15+SdGEhYyoESQdI+pOknsWOJVeSj6WCOZK2xsmNhkna71M6JV0iqVWPAZbUS9JySR9qze87lw1PAlxBSFotaZeknUnLEUWKpU5SQ4xhczxB923LNs3seDN7MW5/iqT7Ul7/jJk90JZ9pJJUIckk1ce6rJX0Q0kZ/V1L+kdJq9sYxjjgaTN7J27zbEnPSdou6S9t3DaShkpaLGlbPFaLJH2yrdttSfKxJDys6SzgCDM73cyeM7P9ztxpZveb2edgn+N0TIb730B4eNBlrYnfuWx4EuAK6Rwzq0pa1hcxlqlmVgUcTXj64r1FjKWt+sW6fBq4iDAbX6FcQXhUbEI9MAv4Tls3LOkjwC+AHxEeM3sUMAVoaOu2s1ADvGFmfy3gPiE8TfCKAu/TlSFPAlxRSeok6eE4Qc7WeBV5QjNle0p6IpbbLOmFpNeOkjRf0ruS3pA0PpP9m1k94VnvH4/bOUDSDEkbJK2T9KNEs+x+9r82NhWPBK4BLoxX57+Pry+SdKmkD8er5I8l/e7hsZXko/HnUZJeiftZJOnjGdZlJfAS4fn7iW2PlbRC0g5JqySNjesPIjyatzqpZaZnPB7XxrLvSfpJPBmnOx7HEpKo3yXF8GszqyM87rmtjgc+MLOfmVmTmf3VzJ40s6VJdXtB0o9jS8EKScOT4jtY0ux4LNdKmpTcSiLpCoWujB2SlkoaENcnjuXlwJ3AkPj+XJfaeiKpRtKj8XP3nqRbkmJ7LhZLfE6Wxe2Mjvv9XNJ2ukraknSsFwMfk3RkDt5H55rlSYBrDx4jzJ53OLCUfa8sk10NvA4cGsteByCpc9zGy4TJlf4JuFrS2fvbsaRuwAXAH+Kq64FTgP7AyYRJYP69pf0nM7PHCHM6PBBbOwamvL6L8Gz5LyWtPh94xsw2SRoEzATGEp4vfy+wQBn0D8fk6QwguRl+IzCCMG3r14BbJfU3s23AOcCbSS0z7xCe7T4CGEq48q4HZjSzy08AfzGzxv3F1kp/BjrHE/lnJR2cpszpwJ+AHsBkYH5SuTrCFMW9Ccd0BPAVAElfAv6DMEFVd8JETZuTN2xmdwMTCBMZVZnZ5OTXJVUAjxPe72MICdFP08Q4NP7bL27n58Ac4MtJZUYCqxMJjpk1ED5rA3AujzwJcIX0aLy63SrpUYB4hXefme2wMI/8DcBASZVpfn83YfbEajNrMLPn4/rTgO5mNjWu/wtwDzCmhVi+K2krYfrXrsBX4/oLgRvM7N14UpxEaGJvaf/ZepB9k4AL4jqAy4Efm9nLZtZoZoluikEtbG+JpHpgOfBL4K7EC2a20Mxet+BZ4BnCZDbNuQK41szWJR2Pf1P6cQYHk8e54M1sC3Am4XvqHuDdeNV9aFKxDcCtZrbbzB4knDg/F6+gzwauii0IbwP/w97PxFjCDHa/j+/NSjN7K8sQBxOSj++YWb2Z7TKzX2X4u3OBcyRVxZ8v4u+T3x2E99i5vPEkwBXSuWZ2cFzOhXAVL+kmSa9L2s7eq9h086r/AFgDPBObq6+O62sIzdqJBGMroUn+8BZi+UGMo5eZnWtmiebrXnEfCWsIrQst7T9bTwMHSxooqTfQD1iQVJfvpNSlV1IM6fQHEi0ag4EDEy9IGinpN7H7YivwGdK/twnVwMKkfb9KmBUu3ej/LXG/rSJpVlJXxDXpypjZMjO7xMyOJNSzmjBGIGGt7TsL2hpColZDSO42JtXlduCwWO5oYFVrY0/axurWtITEhOO3wBckHUI4Lg+mFOtGmGHQubypKHYAruxdDHyeMKhtDaEJ/F3CtMn7MLPtwFXAVZI+AfyfpN8CbwGvmVnasQRZ2kA4gfw5/lxNmN622f2naRFocWpOM/tA0s8IrQHbgAVxbAKxLjea2bRsgjazJmCepHMJzdzflvRh4GHC1e/jZrZb0mPsfW/TxbmWMP3tbzLY7RKgt6TOrTwRjiVckWdafoWkOew78PGolGLVwHrC+/hX4JD43qR6i9BN0BZvATUZ1L+5z8P9hC6BKuCF2FoBQOz+ORZ4pY0xOtcibwlwxdYNeB/YRLiC/a/mCko6R1JvSSKcPBvjshhokPQthYF9nSV9QtLA5rbVgnnA9ZJ6xGbn6wh9yy3tP9VG4JhYrjkPEsYCJHcFANwNjJc0SEFV3G+67pF0vg98PcbeFfgQIalqVBi0mDxOYiPQI46LSLgTmCqpOta5p6RR6XZkZquBN4E977PCwMIDgC7hRx0gqUuGse9D0omSahOD42JMY4BfJxXrJWmCwm14Ywgn9ifjlfbzwM2Suse4jpOU6J+fBVwj6eT4PveRdHSWIS4mfG6nSjpQYdDnGamFYoKwiXBST/YIcCph3MGclNdOA1aa2bosY3IuK54EuGKbTbhyWw8sI4xub87xwLPATuBXwC1mtsjMPiC0JnwKWA28R+gX796KeG4kXH29SrjS/Q3hxNrs/tNs4yHCyXdzbKlI5yXgA8Igw6cSK+MV+DjgDkJz+0r2HUDWIjP7I+Hk9G0z20pouZhPGPT2r4QBlImyS4GfA6tjk3lPQlP7k4Qujx0xzpbGI9zF3jETEFp0dhFu7Ts2/r9VD80h9IkPBl6OYx5eAv5I6OpJeInQnbKZMH5hdBxLAOF9qySMldgC/IzYRWRm84BphGO1nXBCTnsXRHPi524kcAKhVeBNwnuczn8CD8b3+Yvx9+sJg0Sr47/JLiQkZM7llfbtTnPOuczFq/4/AGfFgZSF3PdY4MtmNqyQ+80lSZMIA00vTVrXizCA86R4l4BzeeNjApxzrRbvIMjFWIyyo/BciK8QuoX2iE8MPLEoQbmy490BzjlXYJLGEboPFphZS11gzuWVdwc455xzZcpbApxzzrky5UmAc845V6Y8CXDOOefKlCcBzjnnXJnyJMA555wrU/8PFW+2KbaPvt0AAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# plot ROC Curve\n",
"\n",
"from sklearn.metrics import roc_curve\n",
"\n",
"fpr, tpr, thresholds = roc_curve(y_test, y_pred1, pos_label = '>50K')\n",
"\n",
"plt.figure(figsize=(6,4))\n",
"\n",
"plt.plot(fpr, tpr, linewidth=2)\n",
"\n",
"plt.plot([0,1], [0,1], 'k--' )\n",
"\n",
"plt.rcParams['font.size'] = 12\n",
"\n",
"plt.title('ROC curve for Gaussian Naive Bayes Classifier for Predicting Salaries')\n",
"\n",
"plt.xlabel('False Positive Rate (1 - Specificity)')\n",
"\n",
"plt.ylabel('True Positive Rate (Sensitivity)')\n",
"\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ROC curve help us to choose a threshold level that balances sensitivity and specificity for a particular context."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ROC AUC\n",
"\n",
"\n",
"**ROC AUC** stands for **Receiver Operating Characteristic - Area Under Curve**. It is a technique to compare classifier performance. In this technique, we measure the `area under the curve (AUC)`. A perfect classifier will have a ROC AUC equal to 1, whereas a purely random classifier will have a ROC AUC equal to 0.5. \n",
"\n",
"\n",
"So, **ROC AUC** is the percentage of the ROC plot that is underneath the curve."
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ROC AUC : 0.8941\n"
]
}
],
"source": [
"# compute ROC AUC\n",
"\n",
"from sklearn.metrics import roc_auc_score\n",
"\n",
"ROC_AUC = roc_auc_score(y_test, y_pred1)\n",
"\n",
"print('ROC AUC : {:.4f}'.format(ROC_AUC))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Interpretation\n",
"\n",
"\n",
"- ROC AUC is a single number summary of classifier performance. The higher the value, the better the classifier.\n",
"\n",
"- ROC AUC of our model approaches towards 1. So, we can conclude that our classifier does a good job in predicting whether it will rain tomorrow or not."
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cross validated ROC AUC : 0.8938\n"
]
}
],
"source": [
"# calculate cross-validated ROC AUC \n",
"\n",
"from sklearn.model_selection import cross_val_score\n",
"\n",
"Cross_validated_ROC_AUC = cross_val_score(gnb, X_train, y_train, cv=5, scoring='roc_auc').mean()\n",
"\n",
"print('Cross validated ROC AUC : {:.4f}'.format(Cross_validated_ROC_AUC))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 19. k-Fold Cross Validation"
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cross-validation scores:[0.81359649 0.80438596 0.81184211 0.80517771 0.79640193 0.79684072\n",
" 0.81044318 0.81175954 0.80210619 0.81035996]\n"
]
}
],
"source": [
"# Applying 10-Fold Cross Validation\n",
"\n",
"from sklearn.model_selection import cross_val_score\n",
"\n",
"scores = cross_val_score(gnb, X_train, y_train, cv = 10, scoring='accuracy')\n",
"\n",
"print('Cross-validation scores:{}'.format(scores))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can summarize the cross-validation accuracy by calculating its mean."
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Average cross-validation score: 0.8063\n"
]
}
],
"source": [
"# compute Average cross-validation score\n",
"\n",
"print('Average cross-validation score: {:.4f}'.format(scores.mean()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Interpretation\n",
"\n",
"\n",
"- Using the mean cross-validation, we can conclude that we expect the model to be around 80.63% accurate on average.\n",
"\n",
"- If we look at all the 10 scores produced by the 10-fold cross-validation, we can also conclude that there is a relatively small variance in the accuracy between folds, ranging from 81.35% accuracy to 79.64% accuracy. So, we can conclude that the model is independent of the particular folds used for training.\n",
"\n",
"- Our original model accuracy is 0.8083, but the mean cross-validation accuracy is 0.8063. So, the 10-fold cross-validation accuracy does not result in performance improvement for this model."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 20. Results and conclusion\n",
"\n",
"\n",
"1.\tIn this project, I build a Gaussian Naïve Bayes Classifier model to predict whether a person makes over 50K a year. The model yields a very good performance as indicated by the model accuracy which was found to be 0.8083.\n",
"2.\tThe training-set accuracy score is 0.8067 while the test-set accuracy to be 0.8083. These two values are quite comparable. So, there is no sign of overfitting.\n",
"3.\tI have compared the model accuracy score which is 0.8083 with null accuracy score which is 0.7582. So, we can conclude that our Gaussian Naïve Bayes classifier model is doing a very good job in predicting the class labels.\n",
"4.\tROC AUC of our model approaches towards 1. So, we can conclude that our classifier does a very good job in predicting whether a person makes over 50K a year.\n",
"5.\tUsing the mean cross-validation, we can conclude that we expect the model to be around 80.63% accurate on average.\n",
"6.\tIf we look at all the 10 scores produced by the 10-fold cross-validation, we can also conclude that there is a relatively small variance in the accuracy between folds, ranging from 81.35% accuracy to 79.64% accuracy. So, we can conclude that the model is independent of the particular folds used for training.\n",
"7.\tOur original model accuracy is 0.8083, but the mean cross-validation accuracy is 0.8063. So, the 10-fold cross-validation accuracy does not result in performance improvement for this model."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment