Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Watson Conversation K-Fold Example
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"# K-Fold testing of a Conversation workspace.\n",
"\n",
"This example notebook is used to show you how to generate a K-fold test in conversation, and run the cross validation test. \n",
"\n",
"## Prerequisites \n",
"\n",
"For this example to work, you will need to install the libraries below or you have not already done so.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-02T08:21:43.751823Z",
"start_time": "2017-12-02T08:21:37.322848Z"
},
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"!pip3 install --upgrade watson_developer_cloud\n",
"!pip3 install --upgrade numpy\n",
"!pip3 install --upgrade pandas\n",
"!pip3 install --upgrade scikit-learn\n",
"!pip3 install --upgrade matplotlib"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"The `ctx` object you get from the Conversation service credentials window. For this example you should create a new service for testing, as the test will generate 10 workspaces. \n",
"\n",
"You have to set the language of your workspaces. This is a 2 digit language identifier. Check the <A HREF=\"https://console.bluemix.net/docs/services/conversation/lang-support.html#supported-languages\">supported languages</A> page for what codes can be used. \n",
"\n",
"The `number_of_folds` is set to `5` so you can test this with a free conversation service. For production testing, you should set the `number_of_folds` to `10`. "
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T06:34:00.942514Z",
"start_time": "2017-12-03T06:34:00.938045Z"
},
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"ctx = {\n",
" \"url\": \"https://gateway.watsonplatform.net/conversation/api\",\n",
" \"username\": \"USERNAME\",\n",
" \"password\": \"PASSWORD\"\n",
"}\n",
"\n",
"language = 'en'\n",
"\n",
"number_of_folds = 5"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"## Initialise setup."
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T06:34:02.098244Z",
"start_time": "2017-12-03T06:34:02.092375Z"
},
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"from watson_developer_cloud import ConversationV1\n",
"from sklearn.model_selection import KFold\n",
"\n",
"conversation = ConversationV1( username=ctx.get('username'), password=ctx.get('password'), version='2017-05-26')"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"## Sample questions. \n",
"\n",
"For this test we are going to use pre-generated questions. Here you would create your code to read an existing workspace, either online or local. \n",
"\n",
"The questions below are taken from <A HREF=\"https://github.com/jt-nti/conversation-starter\">Conversation Starter</A>. The file is a `\"question\",\"intent\"` csv format."
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T06:34:04.837106Z",
"start_time": "2017-12-03T06:34:04.010390Z"
},
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"import csv\n",
"import requests\n",
"\n",
"intents_url = 'https://raw.githubusercontent.com/jt-nti/conversation-starter/master/training/intents.csv'\n",
"\n",
"rows = list(csv.reader(requests.get(intents_url).text.split('\\n')))\n",
"\n",
"# Remove last blank example. \n",
"del(rows[-1])\n",
"\n",
"examples = pd.DataFrame(rows, columns=['question','intent'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"## Generate Folds.\n",
"\n",
"What happens next is we randomise the questions, and split them evenly into `number_of_folds` buckets. "
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T06:34:07.098529Z",
"start_time": "2017-12-03T06:34:07.089789Z"
},
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"bucket = np.arange(len(examples))\n",
"folds = []\n",
"\n",
"kf = KFold(n_splits=number_of_folds, shuffle=True)\n",
"for train_index, test_index in kf.split(bucket):\n",
" train, test = bucket[train_index], bucket[test_index]\n",
" fold = { \n",
" 'test': test,\n",
" 'train': train\n",
" }\n",
" folds.append(fold)"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"## Create intents for fold.\n",
"This method will go through CSV lines and create an `intents[]` object for when the workspace is created.\n"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T06:34:08.205614Z",
"start_time": "2017-12-03T06:34:08.181909Z"
},
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"def createIntents(train_list=None):\n",
" results = []\n",
" for i in train_list:\n",
" row = {}\n",
" question = examples.iloc[i]['question']\n",
" intent = examples.iloc[i]['intent']\n",
" \n",
" if not any(intent in x['intent'] for x in results):\n",
" row = { 'intent': intent, \n",
" 'examples': [ {'text': question } ] } \n",
" else:\n",
" row = [d for d in results if d.get('intent') == intent][0]\n",
" results[:] = [d for d in results if d.get('intent') != intent]\n",
" e = {'text': question}\n",
" row['examples'].append(e)\n",
" \n",
" results.append(row)\n",
" \n",
" return results"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"## Create the workspaces. \n",
"Using each 'train' part of the folds, we will create 10 workspaces, and hold the workspace ID for each. We don't do testing straight away as each workspace needs time to train. "
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T06:34:24.725802Z",
"start_time": "2017-12-03T06:34:18.492122Z"
},
"deletable": false,
"editable": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Created workspace fold 0: 23876ee0-500b-4c4d-be52-8ffc41757b14\n",
"Created workspace fold 1: 941a033d-84d7-4d6c-b592-c03716029260\n",
"Created workspace fold 2: 366b9bb1-ebb7-4d1f-a7b9-7410a64ff911\n",
"Created workspace fold 3: 777c0144-3ce8-47f2-8c11-551a669ef196\n",
"Created workspace fold 4: 7e27f2d1-198f-410c-804c-f38bcec51810\n"
]
}
],
"source": [
"workspaces = []\n",
"\n",
"dialog_nodes = [\n",
" {\n",
" 'dialog_node': 'anything_else',\n",
" 'description': 'Required to stop the endless loop error.',\n",
" 'conditions': 'anything_else',\n",
" 'parent': None, \n",
" 'previous_sibling': None,\n",
" 'output': {'text': {'values': ['OK'], 'selection_policy' : 'sequential'}}, \n",
" 'context': None,\n",
" 'metadata': None,\n",
" 'go_to': None\n",
" }\n",
"]\n",
"\n",
"i = 0\n",
"for fold in folds:\n",
" intents = createIntents(fold['train'])\n",
" response = conversation.create_workspace(name='Fold {}'.format(i),\n",
" description='K-Fold Testing workspace.',\n",
" language=language,\n",
" intents=intents,\n",
" dialog_nodes=dialog_nodes,\n",
" metadata={})\n",
" workspaces.append(response['workspace_id'])\n",
" print('Created workspace fold {}: {}'.format(i,response['workspace_id']))\n",
" i = i + 1"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"## Check workspace status. \n",
"Before you start running the testing, you want to make sure that they are all ready to understand. "
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T06:35:48.438669Z",
"start_time": "2017-12-03T06:35:42.848806Z"
},
"deletable": false,
"editable": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Fold: 0. Workspace: 23876ee0-500b-4c4d-be52-8ffc41757b14. Status: Available\n",
"Fold: 1. Workspace: 941a033d-84d7-4d6c-b592-c03716029260. Status: Available\n",
"Fold: 2. Workspace: 366b9bb1-ebb7-4d1f-a7b9-7410a64ff911. Status: Available\n",
"Fold: 3. Workspace: 777c0144-3ce8-47f2-8c11-551a669ef196. Status: Available\n",
"Fold: 4. Workspace: 7e27f2d1-198f-410c-804c-f38bcec51810. Status: Available\n"
]
}
],
"source": [
"i = 0\n",
"for workspace in workspaces:\n",
" response = conversation.get_workspace(workspace_id=workspace, export=False)\n",
" print('Fold: {}. Workspace: {}. Status: {}'.format(i,workspace, response['status']))\n",
" i = i + 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run the test.\n",
"\n",
"**IMPORTANT!** Do not run the next piece until the status is `Available` for all workspaces above. \n",
"\n",
"Now walk through each workspace and test each fold. Generate a final report, and save test information. "
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T06:39:56.374343Z",
"start_time": "2017-12-03T06:37:32.535594Z"
},
"deletable": false,
"editable": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running Folds\n",
"Fold 0. Questions = 26\n",
"X.........X.........X.....\n",
"Fold 1. Questions = 25\n",
"X.........X.........X....\n",
"Fold 2. Questions = 25\n",
"X.........X.........X....\n",
"Fold 3. Questions = 25\n",
"X.........X.........X....\n",
"Fold 4. Questions = 25\n",
"X.........X.........X....\n",
"\n",
"Done\n"
]
}
],
"source": [
"import json\n",
"import time\n",
"\n",
"wsid = 0\n",
"col_list = [ 'Question', 'Expected Intent', 'Matched', 'Found@',\n",
" 'I1', 'C1', 'I2', 'C2', 'I3', 'C3', 'I4', 'C4', 'I5', 'C5',\n",
" 'I6', 'C6', 'I7', 'C7', 'I8', 'C8', 'I9', 'C9', 'I10', 'C10'\n",
" ]\n",
"\n",
"fold_results = []\n",
"print('Running Folds')\n",
"\n",
"for fold in folds:\n",
" results = []\n",
" workspace = workspaces[wsid]\n",
" test_set = fold['test']\n",
"\n",
" print('Fold {}. Questions = {}'.format(wsid, len(test_set)))\n",
"\n",
" counter = 0\n",
" for t in test_set:\n",
" question = examples.iloc[t]['question']\n",
" expected_intent = examples.iloc[t]['intent']\n",
" \n",
" msg = {'text': question }\n",
" \n",
" if counter % 10 == 0: \n",
" print('X',end='')\n",
" else:\n",
" print('.',end='')\n",
" counter = counter + 1\n",
" \n",
" try: \n",
" response = conversation.message(workspace_id=workspace, input=msg, alternate_intents=True)\n",
" except:\n",
" print('E', end='')\n",
" time.sleep(5)\n",
" try:\n",
" response = conversation.message(workspace_id=workspace, input=msg, alternate_intents=True)\n",
" except:\n",
" print('E', end='')\n",
" time.sleep(5)\n",
" response = conversation.message(workspace_id=workspace, input=msg, alternate_intents=True)\n",
" \n",
" intents = response['intents']\n",
" \n",
" found_at = [i for i,_ in enumerate(intents) if _['intent'] == expected_intent]\n",
" if found_at == []: \n",
" found_at = ''\n",
" else:\n",
" found_at = found_at[0]\n",
"\n",
" row = { \n",
" 'Question': question,\n",
" 'Expected Intent': expected_intent,\n",
" 'Matched': any(expected_intent in x['intent'] for x in intents),\n",
" 'Found@': found_at,\n",
" }\n",
" \n",
" i = 1\n",
" for intent in intents:\n",
" row['I{}'.format(i)] = intent['intent']\n",
" row['C{}'.format(i)] = intent['confidence']\n",
" i = i + 1\n",
" \n",
" results.append(row)\n",
" print('')\n",
" \n",
" fold_result = pd.DataFrame(results, columns=col_list)\n",
" fold_result.to_csv('fold{}.csv'.format(wsid))\n",
" wsid = wsid + 1\n",
" \n",
"print('')\n",
"print('Done')"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"## Clean up. \n",
"When all done and you want to remove the workspaces in your test service. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T06:41:54.065178Z",
"start_time": "2017-12-03T06:41:53.000228Z"
},
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"for workspace in workspaces:\n",
" response = conversation.delete_workspace(workspace_id=workspace)\n",
" print('Deleted: {}'.format(workspace))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"---\n",
"\n",
"# Understanding the results you got. \n",
"\n",
"At this point you will have four csv files called `foldX.csv` (where `X` = `0` to `number_of_folds`). The next step is to load these, and get the average across all folds. \n",
"\n",
"Fields in the reports:\n",
"\n",
"| Field | Description | \n",
"| :-| :-|\n",
"| Question | The question that was used to test with. |\n",
"| Expected Intent | The intent that was expected to be returned. |\n",
"| Matched | This will be `true` if expected intent shows up in the top 10 intents returned. |\n",
"| Found@ | This tells you what position it was found at. 0 = Top. | \n",
"| I**x** | Intent found at recall **x** |\n",
"| C**x** | Confidence of Intent found at recall **x** |\n",
"\n",
"--- \n",
"\n",
"First load the reports into memory. For this example, we are only going to bother with the main answer found. For production however, you would also examine the top 5 intents, so as to see why a particular question failed."
]
},
{
"cell_type": "code",
"execution_count": 90,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T08:47:20.356008Z",
"start_time": "2017-12-03T08:47:20.311001Z"
},
"deletable": false,
"editable": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Question</th>\n",
" <th>true_value</th>\n",
" <th>predicted_value</th>\n",
" <th>confidence</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>adios</td>\n",
" <td>cs-goodbye</td>\n",
" <td>cs-goodbye</td>\n",
" <td>0.701070</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>good night</td>\n",
" <td>cs-goodbye</td>\n",
" <td>cs-goodbye</td>\n",
" <td>0.411922</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>got to go</td>\n",
" <td>cs-goodbye</td>\n",
" <td>cs-goodbye</td>\n",
" <td>0.901277</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>gtg</td>\n",
" <td>cs-goodbye</td>\n",
" <td>cs-goodbye</td>\n",
" <td>0.119673</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>have to go</td>\n",
" <td>cs-goodbye</td>\n",
" <td>cs-goodbye</td>\n",
" <td>0.934752</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>have to run</td>\n",
" <td>cs-goodbye</td>\n",
" <td>cs-goodbye</td>\n",
" <td>0.951673</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>my time is up</td>\n",
" <td>cs-goodbye</td>\n",
" <td>cs-goodbye</td>\n",
" <td>0.704113</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>need to go</td>\n",
" <td>cs-goodbye</td>\n",
" <td>cs-goodbye</td>\n",
" <td>0.924854</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>over and out</td>\n",
" <td>cs-goodbye</td>\n",
" <td>cs-goodbye</td>\n",
" <td>0.742759</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>see you soon</td>\n",
" <td>cs-goodbye</td>\n",
" <td>cs-goodbye</td>\n",
" <td>0.941084</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Question true_value predicted_value confidence\n",
"0 adios cs-goodbye cs-goodbye 0.701070\n",
"1 good night cs-goodbye cs-goodbye 0.411922\n",
"2 got to go cs-goodbye cs-goodbye 0.901277\n",
"3 gtg cs-goodbye cs-goodbye 0.119673\n",
"4 have to go cs-goodbye cs-goodbye 0.934752\n",
"5 have to run cs-goodbye cs-goodbye 0.951673\n",
"6 my time is up cs-goodbye cs-goodbye 0.704113\n",
"7 need to go cs-goodbye cs-goodbye 0.924854\n",
"8 over and out cs-goodbye cs-goodbye 0.742759\n",
"9 see you soon cs-goodbye cs-goodbye 0.941084"
]
},
"execution_count": 90,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"reports = [] \n",
"drop_fields = [ 'C2', 'C3','C4','C5','C6','C7','C8','C9','C10',\n",
" 'I2','I3','I4','I5','I6','I7','I8','I9','I10',\n",
" 'Found@', 'Matched'\n",
" ]\n",
"\n",
"for x in range(number_of_folds):\n",
" df = pd.read_csv('fold{}.csv'.format(x),header=0,index_col=0)\n",
" df.drop(drop_fields, axis=1, inplace=True)\n",
" df = df.rename(columns={'Expected Intent': 'true_value', 'I1': 'predicted_value', 'C1': 'confidence'})\n",
" \n",
" reports.append(df)\n",
" \n",
"# example\n",
"reports[0].head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"--- \n",
"\n",
"Now get the overall details of the reports. These are just some sample metrics. "
]
},
{
"cell_type": "code",
"execution_count": 139,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T10:45:47.662314Z",
"start_time": "2017-12-03T10:45:47.604257Z"
},
"deletable": false,
"editable": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Report</th>\n",
" <th>Total</th>\n",
" <th>Correct</th>\n",
" <th>Incorrect</th>\n",
" <th>Accuracy</th>\n",
" <th>Avg Postive Confidence</th>\n",
" <th>Avg Negative Confidence</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>report 1</td>\n",
" <td>26</td>\n",
" <td>23</td>\n",
" <td>3</td>\n",
" <td>0.884615</td>\n",
" <td>0.739708</td>\n",
" <td>0.530551</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>report 2</td>\n",
" <td>25</td>\n",
" <td>22</td>\n",
" <td>3</td>\n",
" <td>0.880000</td>\n",
" <td>0.764297</td>\n",
" <td>0.244852</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>report 3</td>\n",
" <td>25</td>\n",
" <td>23</td>\n",
" <td>2</td>\n",
" <td>0.920000</td>\n",
" <td>0.784364</td>\n",
" <td>0.598231</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>report 4</td>\n",
" <td>25</td>\n",
" <td>19</td>\n",
" <td>6</td>\n",
" <td>0.760000</td>\n",
" <td>0.771111</td>\n",
" <td>0.473936</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>report 5</td>\n",
" <td>25</td>\n",
" <td>20</td>\n",
" <td>5</td>\n",
" <td>0.800000</td>\n",
" <td>0.794046</td>\n",
" <td>0.418658</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Total/Average</td>\n",
" <td>126</td>\n",
" <td>107</td>\n",
" <td>19</td>\n",
" <td>0.848923</td>\n",
" <td>0.770705</td>\n",
" <td>0.453246</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Report Total Correct Incorrect Accuracy Avg Postive Confidence \\\n",
"0 report 1 26 23 3 0.884615 0.739708 \n",
"1 report 2 25 22 3 0.880000 0.764297 \n",
"2 report 3 25 23 2 0.920000 0.784364 \n",
"3 report 4 25 19 6 0.760000 0.771111 \n",
"4 report 5 25 20 5 0.800000 0.794046 \n",
"0 Total/Average 126 107 19 0.848923 0.770705 \n",
"\n",
" Avg Negative Confidence \n",
"0 0.530551 \n",
"1 0.244852 \n",
"2 0.598231 \n",
"3 0.473936 \n",
"4 0.418658 \n",
"0 0.453246 "
]
},
"execution_count": 139,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"records = []\n",
"for index, report in enumerate(reports):\n",
" tp = report[report['true_value'] == report['predicted_value']]\n",
" fp = report[report['true_value'] != report['predicted_value']]\n",
"\n",
" record = {\n",
" 'Report': 'report {}'.format(index+1),\n",
" 'Total': len(report),\n",
" 'Correct': len(tp),\n",
" 'Incorrect': len(fp),\n",
" 'Accuracy': len(tp) / len(report),\n",
" 'Avg Postive Confidence': tp['confidence'].mean(),\n",
" 'Avg Negative Confidence': fp['confidence'].mean()\n",
" }\n",
" records.append(record)\n",
"\n",
"df = pd.DataFrame(records,\n",
" columns=['Report','Total','Correct','Incorrect','Accuracy','Avg Postive Confidence', 'Avg Negative Confidence']\n",
")\n",
"\n",
"record = [{ 'Report': 'Total/Average',\n",
" 'Total': df['Total'].sum(),\n",
" 'Correct': df['Correct'].sum(),\n",
" 'Incorrect': df['Incorrect'].sum(),\n",
" 'Accuracy': df['Accuracy'].mean(),\n",
" 'Avg Postive Confidence': df['Avg Postive Confidence'].mean(),\n",
" 'Avg Negative Confidence': df['Avg Negative Confidence'].mean()\n",
"}]\n",
"\n",
"dft = pd.DataFrame(record,\n",
" columns=['Report','Total','Correct','Incorrect','Accuracy','Avg Postive Confidence', 'Avg Negative Confidence']\n",
")\n",
"\n",
"df = df.append(dft)\n",
"\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the report above, we have a good results. The accuracy is > 0.7. The accuracy of correct answers is high, while for wrong answers is low (could be a little better)."
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"---\n",
"\n",
"# Building a confusion matrix.\n",
"\n",
"In this example, we will take the combined results and create a confusion matrix. This will allow us to see where one intent may be interfering with another. \n",
"\n",
"We start by combining the reports."
]
},
{
"cell_type": "code",
"execution_count": 147,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T11:12:48.414809Z",
"start_time": "2017-12-03T11:12:48.406854Z"
},
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"df = pd.DataFrame([],columns=['Question','true_value','predicted_value','confidence'])\n",
"\n",
"for report in reports:\n",
" df = df.append(report,ignore_index=True)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"For the confusion matrix, we need to specify the fields that are going to be scanned. For this demo, we do this by getting the unique list from the records. "
]
},
{
"cell_type": "code",
"execution_count": 161,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T12:08:10.599467Z",
"start_time": "2017-12-03T12:08:10.594985Z"
},
"deletable": false,
"editable": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"cs-goodbye\n",
"cs-hello\n",
"cs-help\n",
"cs-no\n",
"cs-thankyou\n",
"cs-yes\n"
]
}
],
"source": [
"class_names = examples['intent'].unique()\n",
"\n",
"print('\\n'.join(class_names))"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"This next piece of code is for creating the confusion matrix, and is taken from <A HREF=\"http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py\">sckit-learn example</A>."
]
},
{
"cell_type": "code",
"execution_count": 174,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T12:25:18.997787Z",
"start_time": "2017-12-03T12:25:18.960539Z"
},
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import numpy as np\n",
"import itertools\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.metrics import confusion_matrix\n",
"import matplotlib as mpl\n",
"\n",
"mpl.rcParams['figure.figsize'] = (10,10)\n",
"\n",
"def plot_confusion_matrix(cm, classes=None,\n",
" normalize=False,\n",
" title='Confusion matrix',\n",
" cmap=plt.cm.Blues):\n",
" \"\"\"\n",
" This function prints and plots the confusion matrix.\n",
" Normalization can be applied by setting `normalize=True`.\n",
" \"\"\"\n",
" plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
" plt.title(title)\n",
" plt.colorbar()\n",
" tick_marks = np.arange(len(classes))\n",
" plt.xticks(tick_marks, classes, rotation=45)\n",
" plt.yticks(tick_marks, classes)\n",
"\n",
" thresh = cm.max() / 2.\n",
" for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
" plt.text(j, i, cm[i, j],\n",
" horizontalalignment=\"center\",\n",
" color=\"white\" if cm[i, j] > thresh else \"black\")\n",
"\n",
" plt.tight_layout()\n",
" plt.ylabel('True label')\n",
" plt.xlabel('Predicted label')"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false,
"run_control": {
"frozen": true
}
},
"source": [
"This last piece of code will display the confusion matrix. "
]
},
{
"cell_type": "code",
"execution_count": 175,
"metadata": {
"ExecuteTime": {
"end_time": "2017-12-03T12:25:20.382179Z",
"start_time": "2017-12-03T12:25:19.989951Z"
},
"deletable": false,
"editable": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAq4AAALGCAYAAACEWBMxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3XmcVNW19vHnYVKMoDIo0ECCjaK2\nIrPzPIvjq4iRKA5viMZozHAzqFETozfR+8Y4JBpzFYkxgsZ5xqigogIigwLG2UhjVJwxKNCu9486\nkIZAdwPVfWp3/b73Ux/qnDp1zqqdau/q1Wvv44gQAAAAUOpa5B0AAAAA0BAkrgAAAEgCiSsAAACS\nQOIKAACAJJC4AgAAIAkkrgAAAEgCiSsAAACSQOIKAACAJJC4AgAAIAkkrgAAAEhCq7wDAAAAwJpr\n2f6rEUsX5R2GYtF7D0XEgU1xLRJXAACABMXSRVqvzzF5h6HPZ/yuU1Ndi1YBAAAAJIGKKwAAQJIs\nubxqkOX1aQEAAJAsKq4AAAApsiQ77yiaFBVXAAAAJIHEFQAAAEmgVQAAACBVTM4CAAAASg+JKwAA\nAJJAqwAAAECqWFUAAAAAKD1UXAEAAJLEnbMAAACAkkTiCgAAgCTQKgAAAJAqJmcBAAAApYfEFQAA\nAEmgVQAAACBFFqsKAAAAAKWIxBUAAABJoFUAAAAgSWZVAQAAAKAUUXEFAABIFZOzAAAAgNJD4goA\nAIAk0CoAAACQKiZnAQAAAKWHxBUAAABJoFUAAAAgSWZVAQAAAKAUUXEFAABIkcXkLAAAAKAUkbgC\nAAAgCbQKAAAApIrJWQAAAEDpIXEFAABAEmgVAAAASBLruAIAAAAliYorAABAqlqwjisAAABQckhc\nAQAAkARaBQAAAFJkMTkLAAAAKEUkrgAAAEgCrQIAAACpMqsKAAAAACWHiisAAECSuHMWAAAAUJJI\nXAEAAJAEWgUAAABSxeQsAAAAoPSQuAIAACAJtAoAAACkilUFAAAAgNJDxRUAACBFNpOzAAAAgFJE\n4goAAIAk0CoAAACQKiZnAQAAAKWHxBUAAABJoFUAAAAgVawqAAAAAJQeKq4AAABJMpOzAAAAgFJE\n4goAAIAk0CoAAACQKiZnAQAAAKWHxBUAAABJoFUAAAAgRRarCgAAAACliIorAABAkljHFQAAAChJ\nJK4AAABIAokrgGbPdlvb99j+2Pat63CeEbbHFzO2vNjezfbf844DwDqy8380IRJXACXD9nG2n7W9\n0Pbbth+wvWsRTn20pM0kdYyIYWt7koi4KSL2L0I8jcp22O5d1zER8URE9GmqmACgGEhcAZQE29+X\n9FtJF6uQZPaU9HtJhxfh9F+V9FJELC3CuZJnm4m5AJJE4gogd7Y3kvQLSadHxO0R8VlELImIeyLi\nv7Jj1rP9W9vzs8dvba+Xvban7Xm2f2D73axae1L22s8lnSdpeFbJPcX2Bbb/XOv6X8uqlK2y7RNt\nv2b7U9uv2x5Ra/+Ttd63s+2pWQvCVNs713ptgu0LbU/KzjPedqfVfP5l8f+oVvxH2D7Y9ku2P7B9\ndq3jh9h+2vZH2bFX2W6TvfZ4dtjM7PMOr3X+H9v+p6TRy/Zl76nMrjEg2+5m+z3be67T/7AAGp9b\n5P9oQiSuAErBTpLWl3RHHcecI2lHSf0kbS9piKRza73eRdJGkioknSLpd7Y3iYjzVajijouIDSPi\nuroCsf0VSVdIOigi2knaWdKMVRzXQdJ92bEdJf1G0n22O9Y67DhJJ0naVFIbST+s49JdVBiDChUS\n7T9K+oakgZJ2k/Qz272yY2skfU9SJxXGbh9J35akiNg9O2b77POOq3X+DipUn0fVvnBEvCrpx5L+\nbHsDSaMljYmICXXECwBNjsQVQCnoKGlBPX/KHyHpFxHxbkS8J+nnko6v9fqS7PUlEXG/pIWS1raH\n80tJ29puGxFvR8TsVRwzVNLLEXFjRCyNiJslvSjp0FrHjI6IlyJikaRbVEi6V2eJpIsiYomksSok\npZdHxKfZ9eeokLArIqZFxDPZdd+Q9AdJezTgM50fEV9k8awgIv4o6RVJkyV1VeEXBQAoKSSuAErB\n+5I61dN72U3Sm7W238z2LT/HSonvvyRtuKaBRMRnkoZLOlXS27bvs71VA+JZFlNFre1/rkE870dE\nTfZ8WWL5Tq3XFy17v+0tbd9r+5+2P1GhorzKNoRa3ouIz+s55o+StpV0ZUR8Uc+xAEpB3isKsKoA\ngDL0tKQvJB1RxzHzVfgz9zI9s31r4zNJG9Ta7lL7xYh4KCL2U6Hy+KIKCV198SyLqXotY1oTV6sQ\n1xYR0V7S2SrctbwuUdeLtjdUYXLcdZIuyFohAKCkkLgCyF1EfKxCX+fvsklJG9hubfsg25dkh90s\n6VzbnbNJTudJ+vPqzlmPGZJ2t90zmxj202Uv2N7M9uFZr+sXKrQcfLmKc9wvactsCa9WtodL2kbS\nvWsZ05poJ+kTSQuzavBpK73+jqTN1/Ccl0t6NiL+rwq9u9esc5QAGped/8QsJmcBKEcR8f8kfV+F\nCVfvSXpL0nck3Zkd8ktJz0qaJel5Sc9l+9bmWg9LGpeda5pWTDZbZHHMl/SBCr2jKyeGioj3JR0i\n6QcqtDr8SNIhEbFgbWJaQz9UYeLXpypUg8et9PoFksZkqw4cU9/JbB8u6UD9+3N+X9KAZaspAECp\ncESdfz0CAABACWqxyddivT3Prf/ARvb5nd+cFhGD6jvOdksVChDVEXFItlLKWBUm6E6TdHxELK7r\nHFRcAQAAUpX3xKw1m5z1XUlza23/WtJlEdFb0ocqLGVYJxJXAAAANCrb3VVYRvB/s21L2lvSX7ND\nxqjuCbqSSFwBAADQ+H6rwlyAZZNdO0r6qNYyhvO04nKCq8T9qgEAABLlJl5HdTU62X621va1EXHt\nsg3bh0h6NyKmreutpElcAQAAsC4W1DM5axdJh9k+WIVbW7dXYQm+jW23yqqu3dWAdbBJXJuAW7UN\nt2mXdxjNRr+te+YdQrNSEr+rA6vBujfFx898cT333LQFEdE5j2tbJVNxrVNE/FTZetlZxfWHETHC\n9q2SjlZhZYGRku6q71wkrk3AbdppvT71LqWIBnr8qSvyDqFZadWSVneUrqU1q7r3A9YFP/PF1ba1\nV771Mxrux5LG2v6lpOkq3LmvTiSuAAAAaBIRMUHShOz5a5KGrMn7SVwBAABSZJVd7wd/LwAAAEAS\nSFwBAACQBFoFAAAAkuQkVhUoJiquAAAASAIVVwAAgERRcQUAAABKEIkrAAAAkkCrAAAAQKJoFQAA\nAABKEIkrAAAAkkCrAAAAQKJoFQAAAABKEBVXAACAFDl7lBEqrgAAAEgCiSsAAACSQKsAAABAgiwz\nOQsAAAAoRSSuAAAASAKtAgAAAImiVQAAAAAoQVRcAQAAEkXFFQAAAChBJK4AAABIAq0CAAAAiaJV\nAAAAAChBJK4AAABIAq0CAAAAKXL2KCNUXAEAAJAEKq4AAACJYnIWAAAAUIJIXMtYixbW0zf/WLdd\nfuryfRecfqhm3Xmept92rr799T1yjC5dp406Rb16dNGQAX3zDqXZGP/Qg+pb1UdVW/XWpZf8Ku9w\nksd4Fg8/78XH9xN1IXEtY985bi/9/fV3lm8ff9iO6t5lY21/5IXqf9QvdeuD03KMLl0jjh+pO+6+\nP+8wmo2amhqddebpuuueBzR91hzdOvZmzZ0zJ++wksV4Fhc/78XF93PNWJad/6MpkbiWqYpNN9aB\nu1Zp9B1PLd83atiuuvjaBxQRkqT3PlyYV3hJ23W33bXJJh3yDqPZmDpliiore6vX5purTZs2Gjb8\nWN17z115h5UsxrO4+HkvLr6fqA+Ja5m69L+O0jmX36kvv4zl+3p176yj9x+oJ2/6ke686jRV9uyc\nY4RAwfz51erevcfy7YqK7qqurs4xorQxnihlfD9Rn7JMXG1PsD0oe152ZcWDdttW737wqabPfWuF\n/eu1aaUvFi/RriMu0ejbn9Ifzh+RU4QAAKAh8m4TaOpWAZbDKkM79dtch+yxnQ7ctUrrtWmt9l9Z\nX9f/8gRVv/Oh7nxkpiTprkdn6g8XfCPnSAGpW7cKzZv371+yqqvnqaKiIseI0sZ4opTx/UR9mqzi\navsE27Nsz7R9o+1htl/Ith9fzXsGZ++ZYftS2y9k+9e3Pdr287an296rnv1tbY+1Pdf2HZLarnSd\ny2zPtv2I7c62K20/V+v1LZZt2x5oe6LtabYfst21kYas0Zx35d3qfeDPtNXQ83XCT0ZrwtSXdPK5\nf9I9E2Zpj8FbSJJ2G7iFXvnHuzlHCkiDBg/WK6+8rDdef12LFy/WrePGaughh+UdVrIYT5Qyvp9r\nwSXwaEJNkrjarpJ0rqS9I2J7Sd+VdJ6kA7Lt1X0rR0v6VkT0k1RTa//pkiIitpP0dUljbK9fx/7T\nJP0rIraWdL6kgbXO9RVJz0ZElaSJks6PiFclfWy7X3bMSZJG224t6UpJR0fEQEnXS7po7UemtPzP\n9Q/riH36aeotZ+vCMw/Tab/4S94hJemk44/TPnvuopdf+rv6VPbUmNHX5R1S0lq1aqXLLr9Khw49\nQP2221pHDTtG21RV5R1WshjP4uLnvbj4fqI+XjaDvFEvYp8hqUtEnFNr3zWSKiXdIun2iHh/pfds\nLGlmRHw12+4r6S8RsW1WNb0yIh7NXntChaT156vZ/wtJV9Ta/5ykURHxrO0aSetFxFLbm2ex9LM9\nQtIQSd+X9FL2vKukpyS9loXZUtLbEbH/Kj7zKEmjJEmtNxy4ftXIdRhB1PbeM1fkHUKz0qplWba6\nIxFLa77MO4Rmh5/54mrb2tMiYlAe127dqTI2Ofy/87j0Ct67fniTjUFuPa4RcartHSQNlTTN9kBJ\n/yOpv6T5ko7LK7Ts39tUqM4+KmlaRLxvu5uk2RGxU70nibhW0rWS1GKDTRv/twMAAFBezC1fG8uj\nkobZ7ihJtjvYroyIyRFxnqT3JPWIiJMiol9EHBwRH0n6NEtuJenYWud7QtKI7FxbSuop6e917H9c\nWSJse1tJtW9x0kLS0dnz4yQ9KUkR8bmkhyRdrULLgrJzdba9U3au1lkbBAAAABpZk1RcI2K27Ysk\nTcz+ND9dUnvbW6jQ1vuIpJmreOspkv5o+0sV+k8/zvb/XtLVtp+XtFTSiRHxhe3V7b9ahR7VuZLm\nSqp9S6jPJA2xfa6kdyUNr/XaTZKOlDQ++xyLbR8t6QrbG6kwfr+VNHudBggAAAD1arJWgYgYI2nM\nGr5tdkT0lSTbP5H0bHauz1WYMLXyNVa3f5FWrNjWfm3DOq6/q6TREVFT6/gZknZfg88AAADQKMqt\nVaDU13EdavunKsT5pqQTm+rC2QSwSkl7N9U1AQAAsHolnbhGxDhJ43K69pF5XBcAAKChyq3iypoY\nAAAASAKJKwAAAJJQ0q0CAAAAWDXLtAoAAAAApYjEFQAAAEmgVQAAACBV5dUpQMUVAAAAaSBxBQAA\nQBJoFQAAAEiRuQEBAAAAUJKouAIAACSKiisAAABQgkhcAQAAkARaBQAAABJFqwAAAABQgkhcAQAA\nkARaBQAAAFJVXp0CVFwBAACQBiquAAAAiWJyFgAAAFCCSFwBAACQBFoFAAAAEmSbVgEAAACgFJG4\nAgAAIAm0CgAAACSKVgEAAACgBFFxBQAASBQVVwAAAKAEkbgCAAAgCbQKAAAApKq8OgWouAIAACAN\nJK4AAABIAq0CAAAAiWJVAQAAAKAEUXFtAn236qGHJ16WdxjNxhVPvpZ3CM3KmbtunncIzUqrltQD\nionxBOpgKq4AAABASSJxBQAAQBJoFQAAAEiQJZVZpwAVVwAAAKSBxBUAAABJoFUAAAAgSWZVAQAA\nAKAUUXEFAABIVJkVXKm4AgAAIA0krgAAAEgCrQIAAACJYnIWAAAAUIJIXAEAAJAEWgUAAABSZFYV\nAAAAAEoSFVcAAIAEWVKLFuVVcqXiCgAAgCSQuAIAACAJtAoAAAAkislZAAAAQAkicQUAAEASaBUA\nAABIFLd8BQAAAEoQFVcAAIAUcecsAAAAoDSRuAIAACAJtAoAAAAkyGJyFgAAAFCSSFwBAACQBFoF\nAAAAkmRaBQAAAIBSRMUVAAAgUWVWcKXiCgAAgDSQuAIAACAJJK7Q559/rgP23Fl77jxQuw3ZXr++\n6Od5h5Scj959W//7g2/otycfqMtPOUhP3X6DJOlfn3yk6380Ur8Zua+u/9FILfr043wDTdBpo05R\nrx5dNGRA37xDaTbGP/Sg+lb1UdVWvXXpJb/KO5zkMZ7FxXiuGdu5P5oSiSu03nrr6bZ7x2vCU9P0\n6KRn9djfxuvZKZPzDispLVq21EGn/lRnXf+gTr3yVj1z1016982X9fjYP6iy/876/pi/qbL/zpo4\n9g95h5qcEceP1B133593GM1GTU2NzjrzdN11zwOaPmuObh17s+bOmZN3WMliPIuL8UR9SFwh29pw\nww0lSUuWLNGSpUvKbnmNddW+46aq2KJKkrTeBhuqc89KfbLgHc196hH13/9ISVL//Y/U3El/yzPM\nJO262+7aZJMOeYfRbEydMkWVlb3Va/PN1aZNGw0bfqzuveeuvMNKFuNZXIwn6kPiCkmF33L32mWQ\ntqms0B577aOBg4fkHVKyPvznPL39yhx132p7Lfxwgdp33FSS1K5DZy38cEHO0aHczZ9fre7deyzf\nrqjorurq6hwjShvjWVyM5xpyYVWBvB9NqWwSV9sTbA9ag+O/ZvuF7Pmetu9tvOjy17JlSz026VnN\nnPu6pk97VnPnvJB3SEn6YtFn+svPv6Oh3z5H63+l3QqvOY+fcAAAmpGySVzRMBttvLF22W0PPfq3\n8XmHkpyapUv0lwu+o+33OUxVux0gSdpwk0765P13JUmfvP+uNty4Y54hAurWrULz5r21fLu6ep4q\nKipyjChtjGdxMZ6oT5KJq+0TbM+yPdP2jbaH2X4h2368jrcOsz3F9ku2d8vO1dL2pbanZuf8Vj3X\n7mD7zuzYZ2wnP9V5wYL39PFHH0mSFi1apImPPaIttuiTc1RpiQjd/j9na9OvVmrXo09evn+rnfbW\n9PF3SJKmj79DW++8T14hApKkQYMH65VXXtYbr7+uxYsX69ZxYzX0kMPyDitZjGdxMZ5rxiq/VQWS\nu3OW7SpJ50raOSIW2O4gaaKkAyKi2vbGdby9VUQMsX2wpPMl7SvpFEkfR8Rg2+tJmmR7vKRYzTl+\nLml6RBxhe29Jf5LUr0gfLxfv/PNtnXHqKaqpqVF8+aUOO/Jo7X/Q0LzDSsqbL0zTjL/dqc169dGV\n3zpUkrT/yT/QHsd+Szf/8rua9uCt2njTCh37s8tzjjQ9Jx1/nJ54YqLeX7BAfSp76uxzz9fIk07J\nO6xktWrVSpddfpUOHXqAampqNPLEk7VNVVXeYSWL8SwuxhP1ccTq8rPSZPsMSV0i4pxa+66RVCnp\nFkm3R8T7q3jfBEnnRMQk25tJmhQRvW3/VVJfSf/KDt1I0rckvSTp3ojY1vaekn4YEYfYni7pqIh4\nLTvvW5KqIuKTla43StIoSereo+fA52a/UrxBKHN/nPJm3iE0K2fuunneITQrrVom+YcsAGupbWtP\ni4gGz6Eppq9U9ImtT7smj0uvYNrP9m6yMWgW/4WNiFNVqML2kDTNdkfbo23PsF17Acgvsn9r9O9q\nsyWdERH9skeviFjnBs+IuDYiBkXEoI6dOq3r6QAAAMpeionroyr0qnaUlvecVkbE5Ig4T9J7knpE\nxElZInpwPed7SNJptltn59vS9lfqOP4JSSOyY/eUtGDlaisAAACKL7ke14iYbfsiSRNt10iaLqm9\n7S1UqJ4+ImnmGpzyfyV9TdJzLnQYvyfpiDqOv0DS9bZnqdBeMHKNPwQAAEARlNsNg5JLXCUpIsZI\nGrOG79mz1vMFKiSriogvJZ2dPWr7WNK22TETJE3Inn+guhNbAAAANIIUWwUAAABQhpKsuAIAAKD8\nbshIxRUAAABJoOIKAACQIpff5CwqrgAAAEgCiSsAAACSQKsAAABAgiwmZwEAAAAlicQVAAAASaBV\nAAAAIElmVQEAAACgFFFxBQAASFSZFVypuAIAAKDx2F7f9hTbM23Ptv3zbH8v25Ntv2J7nO029Z2L\nxBUAAACN6QtJe0fE9pL6STrQ9o6Sfi3psojoLelDSafUdyISVwAAgETZzv1RnyhYmG22zh4haW9J\nf832j5F0RH3nInEFAABAo7Ld0vYMSe9KeljSq5I+ioil2SHzJFXUdx4SVwAAAKyLTrafrfUYtfIB\nEVETEf0kdZc0RNJWa3MhVhUAAABIkUtmVYEFETGoIQdGxEe2H5O0k6SNbbfKqq7dJVXX934qrgAA\nAGg0tjvb3jh73lbSfpLmSnpM0tHZYSMl3VXfuai4AgAAJMhSKnfO6ippjO2WKhRNb4mIe23PkTTW\n9i8lTZd0XX0nInEFAABAo4mIWZL6r2L/ayr0uzYYrQIAAABIAhVXAACARCXSKlA0VFwBAACQBBJX\nAAAAJIFWAQAAgESVWacAFVcAAACkgYorAABAopicBQAAAJQgElcAAAAkgVYBAACAFJnJWQAAAEBJ\nInEFAABAEmgVAAAASJBlVhUAAAAAShEVVwAAgESVWcGVxLUptLTVrm3rvMNoNr6/R++8Q2hWxk3/\nR94hNCvD+/fMOwQAaLZoFQAAAEASqLgCAAAkqkWZ9QpQcQUAAEASSFwBAACQBFoFAAAAElVmnQJU\nXAEAAJAGKq4AAAAJssWdswAAAIBSROIKAACAJNAqAAAAkKgW5dUpQMUVAAAAaSBxBQAAQBJoFQAA\nAEgUqwoAAAAAJYjEFQAAAEmgVQAAACBRZdYpQMUVAAAAaaDiCgAAkCBLssqr5ErFFQAAAEkgcQUA\nAEASaBUAAABIFLd8BQAAAEoQiSsAAACSQKsAAABAimxu+QoAAACUIiquAAAAiSqzgisVVwAAAKSB\nxBUAAABJoFUAAAAgQZbUosx6Bai4AgAAIAkkrgAAAEgCrQIAAACJKrNOASquAAAASAMVVwAAgERx\n5yyUpfEPPai+VX1UtVVvXXrJr/IOJ3mM57q57sIf6owDBuicY/dbvu/3Z5+un404SD8bcZB+cPgu\n+tmIg3KMMG18P4uL8SwuxhN1IXGFampqdNaZp+uuex7Q9FlzdOvYmzV3zpy8w0oW47nudh06TD+4\nfMwK+7598e904U0P6MKbHtCgvQ7UoL0OzCm6tPH9LC7Gs7gYT9SHxBWaOmWKKit7q9fmm6tNmzYa\nNvxY3XvPXXmHlSzGc931GbCDvtJ+41W+FhGa+rf7tMP+hzVxVM0D38/iYjyLi/FcM3ZpPJoSiSs0\nf361unfvsXy7oqK7qqurc4wobYxn43pp+hS179BJXXr2yjuUJPH9LC7Gs7gYT9SnLBNX2xNsD1qD\n479m+4XGjAlAwzwz/m7tcADVVgAoR6wqAHXrVqF5895avl1dPU8VFRU5RpQ2xrPx1CxdqmkTHtQF\nY+7NO5Rk8f0sLsazuBjPNcctXxNj+wTbs2zPtH2j7WG2X8i2H6/jrcNsT7H9ku3dsnO1tH2p7anZ\nOb+1iuudaPuurGr7su3zG+3DNZFBgwfrlVde1huvv67Fixfr1nFjNfQQKlpri/FsPLOnPqmuX61U\nh8265h1Ksvh+FhfjWVyMJ+qTdMXVdpWkcyXtHBELbHeQNFHSARFRbXvVszsKWkXEENsHSzpf0r6S\nTpH0cUQMtr2epEm2x0uKld47RNK2kv4laart+yLi2SJ/vCbTqlUrXXb5VTp06AGqqanRyBNP1jZV\nVXmHlSzGc91dfe4ZenHa01r40Yf63iE76Ihvfk97HH6sJo+/h0lZ64jvZ3ExnsXFeK658qq3So5Y\nOSdLh+0zJHWJiHNq7btGUqWkWyTdHhHvr+J9EySdExGTbG8maVJE9Lb9V0l9VUhIJWkjSd+S9JKk\neyNiW9snSto7Ik7IzvULSR9ExG9XusYoSaMkqUfPngNfevXNIn5yoHjGTf9H3iE0K8P798w7BABN\nqG1rT4uIBs+bKaYOvbaJ/S+4KY9Lr2DciQOabAySbxVYWUScqkIVtoekabY72h5te4bt+2sd+kX2\nb43+XXm2pDMiol/26BUR41d1mXq2FRHXRsSgiBjUuVPndftQAAAASLtVQNKjku6w/ZuIeD9rFdgk\nIiZLmmz7IEk9IuKkBp7vIUmn2X40IpbY3lLSqtbh2C+71iJJR0g6uQifBQAAYI2U2y1fk05cI2K2\n7YskTbRdI2m6pPa2t1ChevqIpJlrcMr/lfQ1Sc+58E14T4XEdGVTJN0mqbukP6fc3woAAJCKpBNX\nSYqIMZLG1Hvgiu/Zs9bzBSokq4qILyWdnT1q+1iFyVjLzIuIVSW0AAAAaCTJJ64AAADlyJJalFen\nAInrmoqIGyTdkHMYAAAAZYfEFQAAIEV22U3OanbLYQEAAKB5InEFAABAEmgVAAAASFSZdQpQcQUA\nAEAaSFwBAACQBFoFAAAAEsWqAgAAAEAJouIKAACQoHK8cxYVVwAAACSBxBUAAABJWG2rgO32db0x\nIj4pfjgAAABoqHKbnFVXj+tsSaFCC8Uyy7ZDUs9GjAsAAABYwWoT14jo0ZSBAAAAAHVpUI+r7WNt\nn5097257YOOGBQAAgPq4BB5Nqd7E1fZVkvaSdHy261+SrmnMoAAAAICVNWQd150jYoDt6ZIUER/Y\nbtPIcQEAAKAOttSizCZnNaRVYIntFipMyJLtjpK+bNSoAAAAgJU0JHH9naTbJHW2/XNJT0r6daNG\nBQAAAKyk3laBiPiT7WmS9s12DYuIFxo3LAAAANSnzDoFGtTjKkktJS1RoV2Au20BAACgyTVkVYFz\nJN0sqZuk7pL+YvunjR0YAAAAUFtDKq4nSOofEf+SJNsXSZou6b8bMzAAAADUrdxu+dqQP/u/rRUT\n3FbZPgAAAKDJrLbiavsyFXpaP5A02/ZD2fb+kqY2TXgAAABYnTIruNbZKrBs5YDZku6rtf+ZxgsH\nAAAAWLXVJq4RcV1TBgIAAAAsbv1+AAAgAElEQVTUpd7JWbYrJV0kaRtJ6y/bHxFbNmJcAAAAqINl\nbvm6CjdIGi3Jkg6SdIukcY0YEwAAAPAfGpK4bhARD0lSRLwaEeeqkMACAAAATaYh67h+YbuFpFdt\nnyqpWlK7xg0LAAAAdTKrCqzK9yR9RdKZKvS6biTp5MYMCgAAAFhZvYlrREzOnn4q6fjGDQcAAABY\ntbpuQHCHCjccWKWI+D+NEhEAAAAapNxu+VpXxfWqJouimfsypEWLa/IOo9lo26Zl3iE0K8P798w7\nhGbl1XcW5h1Cs/LVThvkHQKAElLXDQgeacpAAAAAsGYasjxUc1JunxcAAACJInEFAABAEhqyHJYk\nyfZ6EfFFYwYDAACAhrHKb3JWvRVX20NsPy/p5Wx7e9tXNnpkAAAAQC0NaRW4QtIhkt6XpIiYKWmv\nxgwKAAAAWFlDWgVaRMSbK5WiWdsJAAAgZy3Kq1OgQYnrW7aHSArbLSWdIemlxg0LAAAAWFFDEtfT\nVGgX6CnpHUl/y/YBAAAgR1RcVxIR70o6tgliAQAAAFar3sTV9h8lxcr7I2JUo0QEAAAArEJDWgX+\nVuv5+pKOlPRW44QDAACAhrDLbx3XhrQKjKu9bftGSU82WkQAAADAKqzNLV97Sdqs2IEAAAAAdWlI\nj+uH+nePawtJH0j6SWMGBQAAgPqxqkAtLjRObC+pOtv1ZUT8x0QtAAAAoLHVmbhGRNi+PyK2baqA\nAAAA0DBlNjerQT2uM2z3b/RIAAAAgDqstuJqu1VELJXUX9JU269K+kySVSjGDmiiGAEAAIA6WwWm\nSBog6bAmigUAAAANZEktyqxXoK7E1ZIUEa82USwAAADAatWVuHa2/f3VvRgRv2mEeAAAAIBVqitx\nbSlpQ2WVVwAAAJSWtbmTVMrqSlzfjohfNFkkAAAAQB3q7XEFAABAaSqzuVl1Vpj3abIoAAAAgHqs\nNnGNiA+aMhAAAACgLnXe8hUAAAClyXbZreNabpPRAAAAkCgSVwAAACSBVgEAAIBElVmnABVXAAAA\npIGKKwAAQKJaUHEFAAAASg+JKzRv3ls67KB9tOPA7bTToL665ndX5B1S8sY/9KD6VvVR1Va9dekl\nv8o7nOQxnsX1pz9epcP3Hqwj9hmi/zr9JH3x+ed5h5Ss00adol49umjIgL55h9IsMJ6oD4kr1Kpl\nK1148aV6ZtrzGv/YJF137dV6ce6cvMNKVk1Njc4683Tddc8Dmj5rjm4de7PmzmE81xbjWVzvvD1f\nN11/jcbd97jufGSKvqyp0QN3/zXvsJI14viRuuPu+/MOo9lgPNeMJbXI1nLN89GUSFyhLl27avv+\nAyRJ7dq105Z9ttLb86tzjipdU6dMUWVlb/XafHO1adNGw4Yfq3vvuSvvsJLFeBbf0qVL9cXni7R0\n6VItWvQvdd6sa94hJWvX3XbXJpt0yDuMZoPxRH1IXLGCf7z5hmbNnKGBg3fIO5RkzZ9fre7deyzf\nrqjorupqfhFYW4xncW3WtZtO/NaZ2neHbbTXgN5q124j7bLHPnmHBQANQuKK5RYuXKiRxx2jiy/5\njdq3b593OAAawccffajHxt+nh55+Xo9Oe1mLFn2me24bm3dYANaSnf+jKZG4QpK0ZMkSjTxumI4e\n/nUdeviReYeTtG7dKjRv3lvLt6ur56mioiLHiNLGeBbXM09OUEWPr6pDx85q3bq19jnoMM2YNjnv\nsACgQcoicbV9gu1ZtmfavtH2MNsvZNuPr+Y9E2z/2vYU2y/Z3i3bv77t0baftz3d9l5N+2mKLyJ0\n5mnf1JZ9ttbpZ34v73CSN2jwYL3yyst64/XXtXjxYt06bqyGHnJY3mEli/Esrq7dumvW9KlatOhf\nighNfnKCNu/dJ++wAKwNF9ZxzfvRlJp94mq7StK5kvaOiO0lfVfSeZIOyLbr+v+ArSJiiKSzJJ2f\n7TtdUkTEdpK+LmmM7fUb7QM0gclPT9K4m/+sJyY+pt13HKjddxyohx9kVufaatWqlS67/CodOvQA\n9dtuax017BhtU1WVd1jJYjyLq++Awdrv4CN0zIG76sh9d9CXX4aGjTgp77CSddLxx2mfPXfRyy/9\nXX0qe2rM6OvyDilpjGfzZLuH7cdsz7E92/Z3s/0dbD9s++Xs303qPVdENH7EObJ9hqQuEXFOrX3X\nSKqUdIuk2yPi/VW8b4KkcyJiku3NJE2KiN6275B0ZUQ8mh33hKTTI2LWSu8fJWmUJHXv0XPgrBdf\na5wPWIbatmmZdwjAar36zsK8Q2hWvtppg7xDAOrUbv2W0yJiUB7XruizXXz793fkcekVnLvvFnWO\nge2ukrpGxHO220maJukISSdK+iAifmX7J5I2iYgf13WtZl9xXZWIOFWFKmwPSdNsd8z+/D/Ddu1S\n4xfZvzVaw9vjRsS1ETEoIgZ16tS5OIEDAADU4hL4v/pExNsR8Vz2/FNJcyVVSDpc0pjssDEqJLN1\nKofE9VFJw2x3lJaXpSsjYnJEnCfpPUk9IuKkiOgXEQfXc74nJI3IzrWlpJ6S/t6I8QMAAJSyTraf\nrfUYtboDbX9NUn9JkyVtFhFvZy/9U9Jm9V1ojaqIKYqI2bYvkjTRdo2k6ZLa295ChZtOPCJp5hqc\n8veSrrb9vKSlkk6MiC/qeQ8AAEBztaAh7RK2N5R0m6SzIuIT11pLKyLCdr39q80+cZWkiBijf5ei\nG/qePWs9XyDpa9nzzyUxkwEAAOSqcMvXvKNoGNutVUhab4qI27Pd79juGhFvZ32w79Z3nnJoFQAA\nAEBOXCitXidpbkT8ptZLd0samT0fKane+3mXRcUVAACgOUqk4rqLpOMlPW97RrbvbEm/knSL7VMk\nvSnpmPpOROIKAACARhMRT0qrXX5gnzU5F60CAAAASAIVVwAAgETVnplfDqi4AgAAIAkkrgAAAEgC\nrQIAAAAJSmkd12Kh4goAAIAkUHEFAABIkaUym5tFxRUAAABpIHEFAABAEmgVAAAASFSLMusVoOIK\nAACAJJC4AgAAIAm0CgAAACSIdVwBAACAEkXiCgAAgCTQKgAAAJCoMltUgIorAAAA0kDFFQAAIElW\nC5VXyZWKKwAAAJJA4goAAIAk0CoAAACQIIvJWQAAAEBJInEFAABAEmgVAAAASJG55SsAAABQkqi4\nAgAAJKpFmc3OouIKAACAJJC4AgAAIAm0CgAAACSIdVwBAACAEkXiCgAAgCTQKgAAAJAoVhUAAAAA\nShAVVwAAgESVWcGVxBXpWbS4Ju8QmpW2bVrmHUKzUrnZhnmH0KzM/3BR3iE0O5u2Xy/vEIC1RqsA\nAAAAkkDFFQAAIEFW+VUgy+3zAgAAIFEkrgAAAEgCrQIAAAApsuQyW1aAiisAAACSQMUVAAAgUeVV\nb6XiCgAAgESQuAIAACAJtAoAAAAkyJJaMDkLAAAAKD0krgAAAEgCrQIAAACJKq9GASquAAAASAQV\nVwAAgESV2dwsKq4AAABIA4krAAAAkkCrAAAAQJIsl1mvABVXAAAAJIHEFQAAAEmgVQAAACBBVvlV\nIMvt8wIAACBRVFwBAAASxeQsAAAAoASRuAIAACAJtAoAAAAkqrwaBai4AgAAIBEkrgAAAEgCrQIA\nAAApMqsKAAAAACWJiisAAECCuHMWAAAAUKJIXAEAAJAEEldo3ry3dNhB+2jHgdtpp0F9dc3vrsg7\npKQxnsU3/qEH1beqj6q26q1LL/lV3uEkj/Esrk8+/kinn3yc9tu5n/bfpb+emzo575CSddqoU9Sr\nRxcNGdA371CSYTv3R1MicYVatWylCy++VM9Me17jH5uk6669Wi/OnZN3WMliPIurpqZGZ515uu66\n5wFNnzVHt469WXPnMJ5ri/Esvl+c81/afe/99PBTM3TvY5PVe8s+eYeUrBHHj9Qdd9+fdxgoYSSu\nUJeuXbV9/wGSpHbt2mnLPlvp7fnVOUeVLsazuKZOmaLKyt7qtfnmatOmjYYNP1b33nNX3mEli/Es\nrk8/+VhTn3lSx4w4UZLUpk0btd9o43yDStiuu+2uTTbpkHcYKGEkrljBP958Q7NmztDAwTvkHUqz\nwHiuu/nzq9W9e4/l2xUV3VVdzS8Ca4vxLK633nxDHTp20o/O/JYO3XtH/fR7p+lfn32Wd1goIy6B\nR1MqucTV9om2u9XafsN2pyKde2ExztNcLVy4UCOPO0YXX/IbtW/fPu9wksd4As3f0pqlmj1rhkac\n+H91z6PPqO0GX9E1V/5P3mEBzVbJJa6STpTUrb6DUFxLlizRyOOG6ejhX9ehhx+ZdzjJYzyLp1u3\nCs2b99by7erqeaqoqMgxorQxnsXVtWuFunSrUL+BQyRJBx16pGbPmpFzVCgndv6PptSoiavtE2zP\nsj3T9o22h9l+Idt+fBXHHy1pkKSbbM+w3TZ76Qzbz9l+3vZW2bFDbD9te7rtp2z3yfafaPt22w/a\nftn2Jau4TqfsvUNt/8n2EbVeu8n24bbXtz06u+Z023vVOv9VtY6/1/aexRy3phYROvO0b2rLPlvr\n9DO/l3c4yWM8i2vQ4MF65ZWX9cbrr2vx4sW6ddxYDT3ksLzDShbjWVydN+uirt2667VXXpIkPfX4\nY+q95dY5RwU0X42WuNquknSupL0jYntJ35V0nqQDsu3/+C9lRPxV0rOSRkREv4hYlL20ICIGSLpa\n0g+zfS9K2i0i+mfnvbjWqfpJGi5pO0nDbS9v6LK9maT7JJ0XEfdJuk6FKq9sbyRp5+z10wshxXaS\nvi5pjO31121UStPkpydp3M1/1hMTH9PuOw7U7jsO1MMPMqtzbTGexdWqVStddvlVOnToAeq33dY6\natgx2qaqKu+wksV4Ft/5F/8/fe+0k3TwHkM054VZ+vZZ/5V3SMk66fjjtM+eu+jll/6uPpU9NWb0\ndXmHhBLTmLd83VvSrRGxQJIi4gPbkyTdYPsWSbevwbmWHTtN0v/Jnm+kQjK5haSQ1LrW8Y9ExMeS\nZHuOpK9Keis75hFJp0fExCyuibZ/b7uzpKMk3RYRS23vKunK7JgXbb8pacuGBmx7lKRRktS9R881\n+KhNb8edd9UHny3NO4xmg/EsvgMPOlgHHnRw3mE0G4xncW2z3fa66+FJeYfRLIy+8S95h5CUwi1f\nm3p6VL6atMc1Ik5VoQrbQ9I02x2zP8fPsF1XSeqL7N8a/TvZvlDSYxGxraRDJa2/iuNXfs9SFZLf\nA1Y6/58kfUPSSZKur+djLNWK47bKKmxEXBsRgyJiUKdOnes5JQAAAOrTmInro5KG2e4oSbY72K6M\niMkRcZ6k9yT1iIiTsraAZb/+fyqpXQPOv5GkZWu4nNjAmELSyZK2sv3jWvtvkHSWJEXEspW4n5A0\nIot9S0k9Jf1d0huS+tlukbUgDGngtQEAALAOGq1VICJm275I0kTbNZKmS2qf/WnfKvzJfuYq3nqD\npGtsL5K0Ux2XuESFVoFzVehJbWhcNba/Lulu259GxO8j4h3bcyXdWevQ30u62vbzKlRZT4yIL7J2\nh9clzZE0V9JzDb02AABAMTX1rP68OSLyjiF3tjeQ9LykAct6Y4up/4BB8eiT3Lsapaltm5Z5hwCs\n1vwPF9V/ENbIpu3XyzuEZqXd+i2nRcSgPK69RdX2cdm48XlcegWHbtelycagFNdxbVK291Whcnpl\nYyStAAAAKI7GXFUgCRHxNxVWHQAAAEiIZVYVAAAAAEpP2VdcAQAAUlVuk7OouAIAACAJJK4AAABI\nAq0CAAAACeKWrwAAAECJInEFAABAEmgVAAAASJFZVQAAAAAoSVRcAQAAEkXFFQAAAChBJK4AAABI\nAq0CAAAAiTLruAIAAAClh8QVAAAASaBVAAAAIEGW1KK8OgWouAIAACANVFwBAAASxeQsAAAAoASR\nuAIAACAJtAoAAAAkilu+AgAAACWIxBUAAABJoFUAAAAgUawqAAAAAJQgKq4AAAAJ4s5ZAAAAQIki\ncQUAAEASaBUAAABIkpmcBQAAAJQiElcAAAAkgVYBAACAFJlbvgIAAAAliYorAABAosqs4ErFFQAA\nAGmg4orktG3TMu8QgNVatLgm7xCalW6btM07hGZn9rxP8g4BWGskrgAAAAkq3PK1vJoFaBUAAABA\nEkhcAQAAkARaBQAAABJVXo0CVFwBAACQCCquAAAAqSqzkisVVwAAACSBxBUAAABJoFUAAAAgUS6z\nXgEqrgAAAEgCiSsAAACSQKsAAABAosrsjq9UXAEAAJAGKq4AAACJKrOCKxVXAAAApIHEFQAAAEmg\nVQAAACBVZdYrQMUVAAAASSBxBQAAQBJoFQAAAEiQxS1fAQAAgJJE4goAAIBGY/t62+/afqHWvg62\nH7b9cvbvJg05F4krAABAily45Wvejwa4QdKBK+37iaRHImILSY9k2/UicQUAAECjiYjHJX2w0u7D\nJY3Jno+RdERDzsXkLAAAgEQlPDVrs4h4O3v+T0mbNeRNJK4AAABYF51sP1tr+9qIuLahb46IsB0N\nOZbEFQAAAOtiQUQMWsP3vGO7a0S8bburpHcb8iZ6XAEAAFLlEnisnbsljcyej5R0V0PeROIKAACA\nRmP7ZklPS+pje57tUyT9StJ+tl+WtG+2XS9aBQAAANBoIuLrq3lpnzU9F4krAABAkswtXwEAAIBS\nRMUVAAAgUQ28c1WzQcUVmjfvLR120D7aceB22mlQX13zuyvyDil54x96UH2r+qhqq9669JIG9Zuj\nDoxn8fDzXnx8P9fNz390uvYbVKljDthx+b6PP/pA3/7G4Tpyr/769jcO1ycff5hjhCglJK5Qq5at\ndOHFl+qZac9r/GOTdN21V+vFuXPyDitZNTU1OuvM03XXPQ9o+qw5unXszZo7h/FcW4xncfHzXlx8\nP9fdoUcdpytvuG2FfTdcfZmG7LKH7nhsuobssoduuPqynKJDqSFxhbp07art+w+QJLVr105b9tlK\nb8+vzjmqdE2dMkWVlb3Va/PN1aZNGw0bfqzuvadBy9NhFRjP4uLnvbj4fq67ATvsovYbb7LCvokP\n369DjjpOknTIUcdpwvj78git5OW9fOu6LeO6dkhcsYJ/vPmGZs2coYGDd8g7lGTNn1+t7t17LN+u\nqOiu6moSg7XFeDYeft7XHd/PxvHBgvfUadMukqSOnTfTBwveyzkilAoSVyy3cOFCjTzuGF18yW/U\nvn37vMMB0Ij4eUcqbJfdBCSsHokrJElLlizRyOOG6ejhX9ehhx+ZdzhJ69atQvPmvbV8u7p6nioq\nKnKMKG2MZ/Hx8148fD8bR4dOnbXg3X9Kkha8+09t0rFzzhGVsLz7BJr4l4pml7jaPsH2LNszbd9o\ne5jtF7Ltx1dxfKXt52ptb7Fs2/ZA2xNtT7P9kO2u2f4zbc/JrjO26T5d44gInXnaN7Vln611+pnf\nyzuc5A0aPFivvPKy3nj9dS1evFi3jhuroYcclndYyWI8i4uf9+Li+9k49tj3IN17218kSffe9hft\nsd/BOUeEUtGs1nG1XSXpXEk7R8QC2x0kTZR0QERU29545fdExKu2P7bdLyJmSDpJ0mjbrSVdKenw\niHjP9nBJF0k6WdJPJPWKiC9Wdc7UTH56ksbd/GdtU7Wddt9xoCTpZxdcqP0O5D8Ua6NVq1a67PKr\ndOjQA1RTU6ORJ56sbaqq8g4rWYxncfHzXlx8P9fd2WeerGnPPKmPPnxfB++0tUad9VONPO37+ul3\nRuquW25U14oe+u+rbsg7zJJVbnfOckTkHUPR2D5DUpeIOKfWvmskVUq6RdLtEfH+Kt43QtIQSd+X\n9FL2vKukpyS9lh3WUtLbEbG/7QclLZR0p6Q7I2LhKs45StIoSereo+fAWS++tvIhWEtt27TMOwRg\ntRYtrsk7hGaFn/fimz3vk7xDaFYG9dpoWkQMyuPaVX0HxM33/ccfk5vc9j3bNdkYNLtWgZVFxKkq\nVGF7SJpmu6Pt0bZn2L4/O+w2SQdJOkTStCy5taTZEdEve2wXEftnxw+V9DtJAyRNtf0fleuIuDYi\nBkXEoE6d6M0BAABYV80tcX1U0jDbHSXJdgfblRExOSLOk/SepB4RcVKWjB4sSRHxuaSHJF0taXR2\nrr9L6mx7p+xcrW1X2W6RneMxST+WtJGkDZvyQwIAAEiFW77m/WhKzarHNSJm275I0kTbNZKmS2pv\newsVKqiPSJq5mrffJOlISeOzcy22fbSkK2xvpMJY/VaFVoI/Z/ss6YqI+KgxP9f/b+/e4y4f6/2P\nv94GGYaRGpRsJHLMMJIQoUSO2yYmQoqcqnGqybkm5w42Kc12SNio0B78bO1ODpMcM+UQEZIUtcvW\nJMzM+/fHdd1jmQMz4577u773ej897od71vrea13znXV/12d9rs/1uSIiIiJikAWuALYvBC6chx/d\nBLjA9vQCtbpYa9PZHBsRERERA2jQBa7zQtJVlAVcWzQ9loiIiIg51Vs9BRK4AmA7HbgjIiIiulwC\n14iIiIg2amDnqqYNtq4CERERETFIJXCNiIiIiFZIqUBERERES/Xalq/JuEZEREREKyRwjYiIiIhW\nSKlARERERAuJgd9ytWnJuEZEREREKyTjGhEREdFSPZZwTcY1IiIiItohgWtEREREtEJKBSIiIiLa\nqsdqBZJxjYiIiIhWSOAaEREREa2QUoGIiIiIlsqWrxERERERXSgZ14iIiIiWys5ZERERERFdKIFr\nRERERLRCSgUiIiIiWqrHKgWScY2IiIiIdkjgGhERERGtkFKBiIiIiLbqsVqBZFwjIiIiohWScY2I\niIhoIZGdsyIiIiIiulIC14iIiIhohZQKRERERLSRsuVrRERERERXSuAaEREREa2QUoGIiIiIluqx\nSoFkXCMiIiKiHZJxjYiIiGirHku5JuMaEREREa2QwDUiIiIiWiGlAgPg7l/c+eelFlvwsabHMQfe\nCPy56UEMIjmf/Svns3/lfPavnM/+15ZzukJzT62e2/I1gesAsD2i6THMCUl32F6/6XEMFjmf/Svn\ns3/lfPavnM/+l3Mas5JSgYiIiIhohWRcIyIiIloqW75GLxvf9AAGmZzP/pXz2b9yPvtXzmf/yzmN\nmch202OIiIiIiLm09shRnvDDiU0Pg7eOGHrnQNUjp1QgIiIiooVEz+0/kFKBiIiIiGiHZFwjYtCR\nJKcOKmJQy+951WMp12Rco99IL61t7Pw+Zjar8yMpv4/zqO98ShoK0PdmltdhtI2kEZIWbXoc3UzS\n2pKGJmjtTXmjjNesIzgY3nebbSdomLXOLIGk1SWtDGB7Ws7ZvKmvtw8C4yWdJWkTSYvmjW3uSDqk\nnr//krRyPkwNHBXLARcAWyd4nTVJ2wHnAe/ouC3XzR6Si1K8ZjVo2Aa4QtIXJZ3cd3vDQ+sqfRfX\njqD1EOA/gdMl/UzSUvVc5vdyDkkaUv+/IXACcHm9aydgdN/98eok7Q7sDXwTeAz4ErBevS+BwXzm\n4gngWmBPYMsEry8naQ3gdGBv27dKWlrSEr2eKFEX/DeQ8gYZr5mkjYCTgcOBhYENJS3WcX/PXlBm\nsFTfNzXQ2gf4oO2dgTuBqyUtbHtaQ+NrDUkrSFrN9lRJbwU+C0y0fQ1wGHA3sCUwrMlxtsyawOW2\n77H9KeA2SgZ7wXwInb8krSHpAADb3wCuBj4BfEDSwo0Orgt0vIe8CXgAWFDS8cB3gHslvSWv0d6R\nwDXmyQzB6DDgc8DrgfcC+9ieLGktSOYVoE4BXlynuQCeAu4F/g/A9ieBJ4GDmxlh62wLXCdpNeDP\nwCPAFpLeY/tF2xdT3uTWbXKQLfMI8GZJSwHYPhl4GFi60VENcnVWYB1gE0n7Adi+APg5cA6lbKDX\nZw6WAbD9I+AfwBmUWYFtgKsoH1KjR6SrQMyTOjWzKTANeAa4BvgDsLHtv0vaAvhXScfa/luTY21a\nrbV8QtIEYD9JU4CJwAqUC+6EeugkYHJDw2wV21+X9DrgYmA0JeP6GWCfGnjdD7wZ+Etzo+x+kkYD\nCwGPUgKAnYE9Jd1FCfxXB55vbICDVF+du6ThwPO2L63Xhc0lfcL2N4HvU64Pj9qe2uiAG1TL0I6R\ndBMw1PaHJb3O9vOS1qWco0uaHWWzem1OM4FrvBarAOvbPlDSeGBj4I2SNqbUxx2VoFUrAldJ2sP2\nNyS9AHwK+BNwJGUq9l2U2Y9tKEFYzEbnwjbbX5W0IHAZsDtwJiXzfxYlcN3f9q8kLZDyi5nVoPWL\nwLeAscBRwH6U1+W7KTMou9tO8N/PatC6A3AEMETSj4FvU64Du0l6P7Aa8Enbv2xwqI2SNIpShjYa\n2A3YVNLitp+tJWoXAWNs39rkOGNgJXCNOdYZNFS/BLat9azHUd74zqdkYMfavnYWP9NTbD8q6bvA\nBZL2tn1eLbM4iZIl/DdKecW/AB+y/WBzo+1uHVmqkZSpw0m2T5f0InApJXg9jlJ+sTw125qgdWa1\nZGVDYAfb93YETmNtH1YXCC7R6x8855e6yOg4ygeFvwBfpQRnJ1Fqi7cHzrR9Y2OD7A5DKIux3gR8\nENitBq3rArcAO9q+p9ffZ3os4ZrANeZcDRo2ALYDTrB9u6THgNNsHwyMk3RaPfSFXr6YzJAZPKlO\nA15SM6/n1hLhE4GzbJ/f5Fjbor7+tqW8kV0NnCXpINtn1A8D11De8C8FPgrsK+lo2881N+qutQVl\nivVGSQ/anihpL2BCXaV9HpCgdT6or9WpwOPAfXXKe3/gRuDxWt96ZpNjbJqk9wBvodT5XkN5La5j\n+zlJmwMfo2Sj74Gso+g1WZwVr0hVx02TgXcCZ0s6iXJReUHSEvX+F22/AL17MekMWiUtJWmI7dMo\n01qX1NXw5wLXAx+TNCydF16dpDdRMlRbAv8NvECpC8b2VylT3kvbfoxyrk9M0Dprtg+j1FZ/CFil\nllNMpGS1bmh0cIOYpJcVsDsAABMBSURBVDUpLcf+DrwIrCtpsVqO8Y16W5QFvx+w/QjwBUpWesM6\nU3AG8B3bf21ygNGcZFxjtvoK4Ov3GwNvAP5oexuVpvl7AacBawAPAWdnWvZlfVoPp6xqX1jSuZQ3\npiHAhZL2tf01SRfZ/nuDw22Tp4E7gDHAZsD2tp+WtD3wM9un9B1o++GGxtiVOsosBCxk+wXbR9WW\nQscCJ0u6J7WC893qlLrWiykfEMYAt0l6qn6/X4Nj6yaPAktLemu9Tj5LqcN+krJ2oufL0KZT7y3O\nSsY1ZknSkpT+eKvWIPViYAfgOEnn2X7Y9vHAHsDHKathl07msJC0K6VH657AqpQg63nbX6Zkqc+W\ntJDtZxodaAtIWlzSMNtTgCUpi9gOsP1IXdh2OrBik2NsgWX7vqllPJtJGmf788CzlKApiYz5RLUX\nq+3vAT8CPmL7bMq1YCjwfuAg2z9pbpTNkrSupPMkvd72/ZRZlZMkLWL7QkoLvI8maI1cqGImNdP6\nt9op4GbKBWQv2zfVgHa8pFNtf9b2r2ud646UViW5mBQjgDMlHQT8kbI5Ayq7Y42T9HXbmRachVoS\nMLy+tranZFoelXQFZfHKKpS2YqYsbDvS9p2NDbjLSfoksJOknerClpHAVygdBbC9v6Sl+0p8on/V\nhUTbSfqr7a9RWuFtAlzg0m+Y+iG2168HTwKLAKfU3+0fUno0L0m5hrpj3UDeZ3pYAtd4GUlvoGQD\nz7J9mqSnKftC/wS4ibJi+2xg144fWw0YOeCD7RKz+fT/NKVrwDOUzOs0SWMpDd7HOC2GXsmBwChJ\nXwH2pwSr04B/pwSxo4FNKX1aL3LZ+jEZmFmQ9FFKTeWuNWgdRqmjHFM/iC5oe4rtp5od6eDSUZqx\nDqXDxaOUDiwbAv8D7C7pFtuX1h+Z0tBQu4btPwJ71Bm+zSg74K0F/BM4wj3cy/bV9dZEZwLXeBnb\nf5F0LzBG0ou2L6hvdmdLusOlL+biwPqShtep7l8Bm7vss91T6hT23+v3H6bUsD5GmQLcm1LHtrGk\nFSjtmvZIHfArs32cpFMo7dV+ZftagFrndi4wwqVBe+fPJGidgcrOdUtSsv0jJO0IHAJ8xWUDhwUo\nq9ujn0haBVjN9tX1ptMoNZkTgIsk7U153/0nsD5waa9/6KrlZQu4bN88HPi97fMlXUdZP/EJSSvU\nRZcRqXGNl6g0c4fSamh5yoX2XbbPogQRd9RFRrsCJ9t+pq6Yn9KjQevbKf1Z36bSzH0cpdbyHMrq\n7I8Dw4GDKC3E9rR9b0PDbQXVrS1tj6WUqawrac36OruZsn/7WEkrKNtgzpakA4ETgCUomepjgP+l\nBLE7SVre9rReDpjmk+WB/6o17lA+GEw/x7YvdGk1thewjaS1e/XfQNIWkka5mKqyLfYPKfW+2H7S\nZYvXBYE3NjnWbibK4qymvwZSMq4xne0pkrYEvkyZ5t6LUhz/OZdemVMpjbJH2Z5UMwU9m7Gx/UBd\nDXwy8HtmbuZ+lO1jYPq2r/9ocLitUN/Ahtieavt4SacDnweOl3S/7Rskres0xp8tlR2ZDgS2tf24\npMuBR1z6hW4BLErJ+EU/Umkp9mOVLUq/W68NPwT+WRdnmVLD+Zztn0qaCKxNmbHqKfU1ehJll7a+\npMkpwLdsX9Nx3KqUmvb8vsd0CVxjRu8B/tv29cD1kk4Avq3SvuksSVf2ZVd7NVMA09+kptk+WNKJ\nlAzrzZq5mfsw2+claJ29zqnSOm3Yd/sito+U9AXKh4OjgHsStL6qNwOX1aB1IeCBWm95ELAPsJ/t\npxsd4SBTX8PT6oeu6+sMzLWUDwmrU/5NJlMSZIeqbEiyHNBziwolLQV8BtilLsBcE1gMONr27+ox\nqpnYByVtmRrs6JRSgZjRfcCikpYGsH0CJTszttYf/QFeHmD0ovomtUD9/mjKZgKzaub+0+ZG2f06\nFrHMatrwFknvtX0c8DCwUMPDbYvHKHu6v932i/X87gy8Axhte1LD4xt06jneBDhc0shal70FpdXY\n/cC/AocC42z/nrJ4c1fbDzQ26AZI2pQyk/cMsFktPRtH2Qxj83rMAvV89l1fE7S+CnXB10BK4NrD\n+oJPSe+WtKOkD1AavK9EWQG7dl0V+1tqTWtfZqwXs60d56vv99Q1o9VXk3kfpZn7WvXie6vTCP8V\n1TeoHShbXC4NL5s2PNf2T+txh9r+RWMDbZeJwO3APpK2k7QHpRvDqXk9zh81IBtPmda+WNKHbN9G\naRP4JWBn20/Y/mVfiZXtyU2OeaCptLY7g/Jh/vuUjgFX2N6ZEtRvoNLlYhqU5EBTY43ullKBHlaD\nhvdRLrgXUPZ//gLwTcpiou2At1GmcG5pbKBdYIaVv8tSeg5i+0VJmwHvs32sSu/bMcABlC1J4xXM\n4bThkF6upZ5btv9P0tcpQdNBlOzWx122z4x+VuswP0PZFONGle4iH5GE7e/UhMD0JFGPfugfBuwL\nHGL7buDujvveAxwHfNplk5GIV5TAtUfVrOEilC0GP2P7e5Iuo2S+rrB9gErbqzfYfrTBoXaFjhrM\nNHN/jTrKAzYF1uOlacMjgKWAjSiLAy+smesErXPJ9pPAOZLOr3/O67Gf1WvoAsA6lHrVfwNutP2f\nkqYBB9YPXZf2Hd+LQWtlyqYsw+ClD6OStgYOBg63/YMmB9hmvVa4l1KBHlVrCZ+jNMYeIWmo7d9Q\n2ubsq9Kj9dleD1pnWCzU18z94565mftVdYo7NVmvogatczVtGPPG9gsJWvtXxzVhWJ3y/y6lRGhR\nSQcD2L6M0nP4ob6f6+GglVoWcTmwkaTVa9C6EeV6Os72/2t2hNEmybj2kI5M10rAwpTa1XsouxBN\nBH5J2cHlWbKTy4yr3dPMvZ9k2jDaquMaui2lO8B9lE0y/qN+cN1a0uG2v2z7koaH222upPRhHi/p\nZsqGLAfXWuB4DZSds2Kw6rjgfhn4NWXq5pOUbNfYeuFdGTix1xYOzEpH0HogsCWl3+K/A78DvkcJ\nYg+WdLXtxxsbaPtk2jBaqV5D30/pQbo3ZfvhEyW9wfYpdbHmNspOTzOx/YSkU4EfA8sA37d9a8PD\nihZK4NpDJL2FUjv4Edu3S/ocpQH04ZRWQ/8CPGv77h6vx5pOaebe72xPrudxI0mP275/hmnDZGCi\nK9UP9yOADwMrUPpe7w58Q9IU21+SdKPtPzU5zm5VEyI/bXoc0W6pcR3kOms0gb8ATwF97UZOppQF\njLP9mO2b6tRtT9djzWBWzdyfV2nmfgplqivN3OfelZTrz3hJJwOXABclaI1u09dPFMrugsCllH7W\nBwBjbf8Y+DmldGClBK0x4Jpu4jrAlQoJXAe5OrW1saQ962KspygLX0bUQ66gNMOOWUsz9/nAZfe1\nUymLWu4Gds8CjegmtVVb32Yj20o6Q9KngGVt/xX4X2BYLR0YAmyelmMR819KBQapjkUEoyg7leyk\nsnf2ycDZlCb5fwN2ojQnj1mbSGnPtI/K3uLDgU8Du+VN6rXJtGF0qzq7cq2kqykZ1i9QPuSvSCkL\n2Af4CbAnMBI41vaDzYw2orckcB2katC6FfANSourP1EWZX0W+AjwfsrmAgfWptmpaZ2FNHOP6C31\nWviipP2Ai4F3A6fbvkxlK+xPUTZtOcD2hZKWsf2nXEOjKb3VUwCU37PBq05rvWD7HElDgK0oq+IP\nt311s6NrH0kLQ5q5RwxWkoYCi9n+c92AZQngGmCS7X3qMSMoCYA1gJ2B5xOwRlPWWXeUf3DDz5se\nBssOX/hO2+sPxHOlxnVwWwzYBaDuPvQzYBJwbM3GxlxIM/eIQW994LDaAu9+yoLW3YB3SfosQF2M\neSpl45F/JmiNJknd8TWQErgOYrVrwGRJV9Sb1gReAH4ErNLYwCIiupDtm4CVKGVVR9XA9EHKWoA9\nJZ1Qj3s6Na0RzUjgOgjMsC2pallA37TXRwBLmgCcR1mc9SSwahNjjYjoRn3XTeDblAVZm0t6R90O\n+wHK7NVukt7W2CAjIouz2q6je8AWwDO27wSmSloOmAAcanuXuqhgCrAOpf/gLs2NOiKiu9RyKmxf\nB1wn6XTgeGCMpLUppVcjbT/f4DAjZtJrW74m49pyNWjdATgTWBqm7+5yCnC+7RvrcU8BiwBbUFo5\n3dfQkCMiusIsZqsWrN8vYvtI4EHK9q5fA0jQGtG8ZFxbrjbJ/gywi+1fS1qTkhk42vbv6jFy8QdJ\nJ9rOFqUR0dNmM1s1pc5WXSNpjO3PSVqD0p3lobS8imheMq4t1JclkLQpZXOBZ4DNJJ0LjKOUCGxe\nj1mg80KboDUi4lVnq861fUM97j7bD/X9TFPjjZitprd7HeBKhWRcW6hecLcHPg/sC0ymbEF6he3r\nJI0GNpF0Sd1bOyIiOszhbNUCtqc1Oc6IeLkEri0kaRglYD3E9t2Uvd777nsPcBzw6QStEREv6SgP\n2BRYj5dmq44AlqJs7/xZ4MIErdEWvbU0K6UCbWVgBDAMXmrjImlrSgbhcNs/aG54ERHdp2O26gzg\np8D3gbUos1U7A4cCG0haMEFrRHdKxrWFbE+WdDmwkaTHbd8vaSNgb2Cc7dsaHmJERNfJbFVE+yVw\nba8rgU8A4yXdDOwOHJygNSJitmaarbI9tc5WHUxmq6KFBnrL1aYlcG0p209IOhX4MbAM8H3btzY8\nrIiIrpXZqoj2S+DaYrYnU+q0IiJizmS2KqLFErhGRETPyGxVDC7quS1fE7hGRERPyWxVRHslcI2I\niIhoIdF7i7PSxzUiIiIiWiGBa0RERES0QgLXiIiIiGiFBK4RERER0QoJXCOi1SRNlXS3pHskfVfS\noq/hsd4r6Zr6/Q6Sxr7CsUtKOmgenuMESUfM6e0zHPMtSbvMxXOtKOmeuR1jRES3SuAaEW33nO2R\nttcCXgAO6LxTxVxf62xPsH3KKxyyJDDXgWtERH+Smv8aSAlcI2IwuQl4W800PiDp28A9wPKStpJ0\ni6S7ama2b7/6rSX9WtJdwM59DyRpH0lfq98vI+kqSZPq10bAKcDKNdt7ej3uSEm3S/qlpM93PNbR\nkh6sOzW9/dX+EpL2q48zSdIVM2SR3yfpjvp429Xjh0g6veO5P/FaT2RERDdK4BoRg4KkBYFtgF/V\nm1YBvm57TWAycAzwPtvrAXcAh0laBPgPYHtgFLDsbB7+TOAG2+sA6wH3AmOBh2u290hJW9Xn3AAY\nCYyStKmkUZRtRUcCHwTeOQd/nSttv7M+3/3AxzruW7E+x7bAOfXv8DHgGdvvrI+/n6SV5uB5IqLl\n1AX/DaRsQBARbTdU0t31+5uA84A3A4/Z/nm9fUNgDWCiyrzWwsAtwGrAI7Z/AyDpYmD/WTzHFsBe\nALanAs9Iev0Mx2xVv35R/zyMEsguDlxl+x/1OSbMwd9pLUlfpJQjDAOu77jvO7anAb+R9Nv6d9gK\neEdH/evw+twPzsFzRUS0RgLXiGi752yP7LyhBqeTO28C/sf26BmOe9nPvUYCTrb9zRmeY8w8PNa3\ngJ1sT5K0D/Dejvs8w7Guz/1J250BLpJWnIfnjojoWikViIhe8HNgY0lvA5C0mKRVgV8DK0pauR43\nejY//yPgwPqzQyQNB56lZFP7XA/s21E7u5ykpYEbgZ0kDZW0OKUs4dUsDjwpaSFgjxnu21XSAnXM\nbwUeqM99YD0eSatKWmwOnici2qwLFmYN9OKsZFwjYtCz/XTNXF4q6XX15mNsPyhpf+BaSf+glBos\nPouH+DQwXtLHgKnAgbZvkTSxtpu6rta5rg7cUjO+fwf2tH2XpMuBScBTwO1zMORjgVuBp+v/O8f0\nO+A2YAngANv/lHQupfb1LpUnfxrYac7OTkREe8iecdYpIiIiIrrduqPW9w0Tb2t6GAwfOuRO2+sP\nxHMl4xoRERHRQqpfvSQ1rhERERHRCglcIyIiIqIVUioQERER0VY9ViuQjGtEREREtEIyrhEREREt\nNdBbrjYtGdeIiIiIaIUErhERERHRCikViIiIiGipgd5ytWnJuEZEREREKyRwjYiIiIhWSKlARERE\nREv1WKVAMq4RERER0Q7JuEZERES0VY+lXJNxjYiIiIhWSOAaEREREa2QUoGIiIiIlsqWrxERERER\n/UjS1pIekPSQpLHz+jgJXCMiIiJivpE0BDgb2AZYAxgtaY15eayUCkRERES0kGjNlq8bAA/Z/i2A\npMuAHYH75vaBknGNiIiIiPlpOeDxjj//vt4215JxjYiIiGihu+668/qhC+mNTY8DWETSHR1/Hm97\n/Px4ogSuERERES1ke+umxzCHngCW7/jzW+ptcy2lAhERERExP90OrCJpJUkLA7sDE+blgZJxjYiI\niIj5xvYUSYcA1wNDgPNt3zsvjyXb/Tq4iIiIiIj5IaUCEREREdEKCVwjIiIiohUSuEZEREREKyRw\njYiIiIhWSOAaEREREa2QwDUiIiIiWiGBa0RERES0QgLXiIiIiGiF/w/4NAX6Wrfz7wAAAABJRU5E\nrkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x11b09f390>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"cm_true = df['true_value'].tolist()\n",
"cm_predicted = df['predicted_value'].tolist()\n",
"cnf_matrix = confusion_matrix(cm_true, cm_predicted,labels=class_names)\n",
"\n",
"np.set_printoptions(precision=2)\n",
"\n",
"plt.figure()\n",
"plot_confusion_matrix(cnf_matrix, classes=class_names, title='Confusion matrix')\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So you can see that `cs-help` is a well trained intent, while `cs-goodbye` is well trained, it is also pulling in possible weaker clusters. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.