Skip to content

Instantly share code, notes, and snippets.

@Orbifold

Orbifold/GraphSageLinkPrediction.ipynb

Last active March 26, 2024 06:30
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Orbifold/5e267774dcafa58e7e3cafce7e9c73b6 to your computer and use it in GitHub Desktop.
Save Orbifold/5e267774dcafa58e7e3cafce7e9c73b6 to your computer and use it in GitHub Desktop.
Download ZIP
Using GraphSage for link predictions. So much fun.
Display the source blob
Display the rendered blob
Raw
GraphSageLinkPrediction.ipynb
{
"cells": [
{
"cell_type": "markdown",
"source": [
"# Graph Link Prediction using GraphSAGE"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"This article is based on the paper [\"Inductive Representation Learning on Large Graphs\" by Hamilton, Ying and Leskovec](https://papers.nips.cc/paper/6703-inductive-representation-learning-on-large-graphs.pdf).\n",
"The [StellarGraph](https://www.stellargraph.io) implementation of the [GraphSAGE](http://snap.stanford.edu/graphsage/) algorithm is used to build a model that predicts citation links of [the Cora dataset](https://graphsandnetworks.com/the-cora-dataset/). \n",
"\n",
"The way link prediction is turned into a supervised learning task is actually very savvy. Pairs of nodes are embedded and a binary prediction model is trained where '1' means the nodes are connected and '0' means they are not connected. It's like embedding the adjacency matrix and finding a decision boundary between two types of elements. The entire model is trained end-to-end by minimizing the loss function of choice (e.g., binary cross-entropy between predicted link probabilities and true link labels, with true/false citation links having labels 1/0) using stochastic gradient descent (SGD) updates of the model parameters, with minibatches of 'training' links fed into the model."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"import networkx as nx\n",
"import pandas as pd\n",
"import os\n",
"\n",
"import stellargraph as sg\n",
"from stellargraph.data import EdgeSplitter\n",
"from stellargraph.mapper import GraphSAGELinkGenerator\n",
"from stellargraph.layer import GraphSAGE, link_classification\n",
"\n",
"import tensorflow.keras as keras # DO NOT USE KERAS DIRECTLY\n",
"from sklearn import preprocessing, feature_extraction, model_selection\n",
"\n",
"from stellargraph import globalvar"
],
"outputs": [],
"execution_count": 31,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Dataset\n",
"\n",
"[The Cora dataset](https://graphsandnetworks.com/the-cora-dataset/) is the hello-world dataset when looking at graph learning. We have described in details in [this article](https://graphsandnetworks.com/the-cora-dataset/) and will not repeat it here. You can also find in the article a direct link to download the data.\n",
"\n",
"The construction below recreates the steps outlined in the article.\n",
"\n"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"data_dir = os.path.expanduser(\"/Users/swa/Desktop/LargeFiles/Graphs/cora\")\n",
"cora_location = os.path.expanduser(os.path.join(data_dir, \"cora.cites\"))\n",
"g_nx = nx.read_edgelist(path=cora_location)\n",
"\n",
"cora_data_location = os.path.expanduser(os.path.join(data_dir, \"cora.content\"))\n",
"node_attr = pd.read_csv(cora_data_location, sep='\\t', header=None)\n",
"values = { str(row.tolist()[0]): row.tolist()[-1] for _, row in node_attr.iterrows()}\n",
"nx.set_node_attributes(g_nx, values, 'subject')\n",
"\n",
"g_nx_ccs = (g_nx.subgraph(c).copy() for c in nx.connected_components(g_nx))\n",
"g_nx = max(g_nx_ccs, key=len)\n",
"print(\"Largest connected component: {} nodes, {} edges\".format(\n",
" g_nx.number_of_nodes(), g_nx.number_of_edges()))"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Largest connected component: 2485 nodes, 5069 edges\n"
]
}
],
"execution_count": 3,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"The features of the nodes are taken into account in the model:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"feature_names = [\"w_{}\".format(ii) for ii in range(1433)]\n",
"column_names = feature_names + [\"subject\"]\n",
"node_data = pd.read_csv(os.path.join(data_dir, \"cora.content\"), \n",
" sep=\"\\t\", \n",
" header=None, \n",
" names=column_names)\n",
"node_data.drop(['subject'], axis=1, inplace=True)\n"
],
"outputs": [],
"execution_count": 4,
"metadata": {}
},
{
"cell_type": "code",
"source": [
"node_data.index = node_data.index.map(str)\n",
"node_data = node_data[node_data.index.isin(list(g_nx.nodes()))]"
],
"outputs": [],
"execution_count": 5,
"metadata": {}
},
{
"cell_type": "code",
"source": [
"node_data.head(2)"
],
"outputs": [
{
"output_type": "execute_result",
"execution_count": 7,
"data": {
"text/plain": [
" w_0 w_1 w_2 w_3 w_4 w_5 w_6 w_7 w_8 w_9 ... w_1423 \\\n",
"31336 0 0 0 0 0 0 0 0 0 0 ... 0 \n",
"1061127 0 0 0 0 0 0 0 0 0 0 ... 0 \n",
"\n",
" w_1424 w_1425 w_1426 w_1427 w_1428 w_1429 w_1430 w_1431 \\\n",
"31336 0 0 1 0 0 0 0 0 \n",
"1061127 0 1 0 0 0 0 0 0 \n",
"\n",
" w_1432 \n",
"31336 0 \n",
"1061127 0 \n",
"\n",
"[2 rows x 1433 columns]"
],
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>w_0</th>\n",
" <th>w_1</th>\n",
" <th>w_2</th>\n",
" <th>w_3</th>\n",
" <th>w_4</th>\n",
" <th>w_5</th>\n",
" <th>w_6</th>\n",
" <th>w_7</th>\n",
" <th>w_8</th>\n",
" <th>w_9</th>\n",
" <th>...</th>\n",
" <th>w_1423</th>\n",
" <th>w_1424</th>\n",
" <th>w_1425</th>\n",
" <th>w_1426</th>\n",
" <th>w_1427</th>\n",
" <th>w_1428</th>\n",
" <th>w_1429</th>\n",
" <th>w_1430</th>\n",
" <th>w_1431</th>\n",
" <th>w_1432</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>31336</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1061127</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>2 rows × 1433 columns</p>\n",
"</div>"
]
},
"metadata": {}
}
],
"execution_count": 7,
"metadata": {
"collapsed": false,
"outputHidden": false,
"inputHidden": false
}
},
{
"cell_type": "markdown",
"source": [
"Define a set of node features that will be used by the model as the difference between the set of all node features and a list of user-defined node attributes to ignore:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"feature_names = sorted(set(node_data.columns))"
],
"outputs": [],
"execution_count": 8,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"We need to convert node features that will be used by the model to numeric values that are required for GraphSAGE input. Note that all node features in the Cora dataset, except the categorical \"subject\" feature, are already numeric, and don't require the conversion."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"node_features = node_data[feature_names].values"
],
"outputs": [],
"execution_count": 12,
"metadata": {}
},
{
"cell_type": "code",
"source": [
"node_features.shape"
],
"outputs": [
{
"output_type": "execute_result",
"execution_count": 13,
"data": {
"text/plain": [
"(2485, 1433)"
]
},
"metadata": {}
}
],
"execution_count": 13,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Add node data to g_nx:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"for nid, f in zip(node_data.index, node_features):\n",
" g_nx.node[nid]['label'] = \"paper\"\n",
" g_nx.node[nid][\"feature\"] = f"
],
"outputs": [],
"execution_count": 17,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Splitting a graph\n",
"\n",
"Splitting graph-like data into train and test sets is not as straightforward as in classic (tabular) machine learning. If you take a subset of nodes you also need to ensure that the edges do not have endpoints across the other set. That is, edges should connect only to train or test nodes but not having endpoints in each set. So, this is in general a little tricky but the StellarGraph framework makes it easy by giving us a method to do this in one line of code. Actually the splitting happens in a slightly different fashion. Instead of taking a subset of nodes all the nodes are kept in both training and test but the edges are randomly sampled. Each of these graphs will have the same number of nodes as the input graph, but the number of links will differ (be reduced) as some of the links will be removed during each split and used as the positive samples for training/testing the link prediction classifier."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"From the original graph G, extract a randomly sampled subset of test edges (true and false citation links) and the reduced graph G_test with the positive test edges removed. Define an edge splitter on the original graph `g_nx`:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"edge_splitter_test = EdgeSplitter(g_nx)"
],
"outputs": [],
"execution_count": 18,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Randomly sample a fraction p=0.1 of all positive links, and same number of negative links, from G, and obtain the reduced graph G_test with the sampled links removed:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"G_test, edge_ids_test, edge_labels_test = edge_splitter_test.train_test_split(\n",
" p=0.1, method=\"global\", keep_connected=True\n",
")"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"** Sampled 506 positive and 506 negative edges. **\n"
]
}
],
"execution_count": 19,
"metadata": {
"collapsed": false,
"outputHidden": false,
"inputHidden": false
}
},
{
"cell_type": "markdown",
"source": [
"The reduced graph G_test, together with the test ground truth set of links (edge_ids_test, edge_labels_test), will be used for testing the model.\n",
"\n",
"Now repeat this procedure to obtain the training data for the model. From the reduced graph G_test, extract a randomly sampled subset of train edges (true and false citation links) and the reduced graph G_train with the positive train edges removed:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"edge_splitter_train = EdgeSplitter(G_test)\n",
"G_train, edge_ids_train, edge_labels_train = edge_splitter_train.train_test_split(\n",
" p=0.1, method=\"global\", keep_connected=True\n",
")"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"** Sampled 456 positive and 456 negative edges. **\n"
]
}
],
"execution_count": 20,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Defining the GraphSage model"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Convert G_train and G_test to StellarGraph objects (undirected, as required by GraphSAGE) for ML:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"G_train = sg.StellarGraph(G_train, node_features=\"feature\")\n",
"G_test = sg.StellarGraph(G_test, node_features=\"feature\")"
],
"outputs": [],
"execution_count": 21,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Summary of G_train and G_test - note that they have the same set of nodes, only differing in their edge sets:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"print(G_train.info())"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"StellarGraph: Undirected multigraph\n",
" Nodes: 2485, Edges: 4107\n",
"\n",
" Node types:\n",
" paper: [2485]\n",
" Attributes: {'feature', 'subject'}\n",
" Edge types: paper-default->paper\n",
"\n",
" Edge types:\n",
" paper-default->paper: [4107]\n",
"\n"
]
}
],
"execution_count": 22,
"metadata": {}
},
{
"cell_type": "code",
"source": [
"print(G_test.info())"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"StellarGraph: Undirected multigraph\n",
" Nodes: 2485, Edges: 4563\n",
"\n",
" Node types:\n",
" paper: [2485]\n",
" Attributes: {'feature', 'subject'}\n",
" Edge types: paper-default->paper\n",
"\n",
" Edge types:\n",
" paper-default->paper: [4563]\n",
"\n"
]
}
],
"execution_count": 23,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Next, we create the link mappers for sampling and streaming training and testing data to the model. The link mappers essentially \"map\" pairs of nodes `(paper1, paper2)` to the input of GraphSAGE: they take minibatches of node pairs, sample 2-hop subgraphs with `(paper1, paper2)` head nodes extracted from those pairs, and feed them, together with the corresponding binary labels indicating whether those pairs represent true or false citation links, to the input layer of the GraphSAGE model, for SGD updates of the model parameters.\n",
"\n",
"Specify the minibatch size (number of node pairs per minibatch) and the number of epochs for training the model:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"batch_size = 20\n",
"epochs = 20"
],
"outputs": [],
"execution_count": 24,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Specify the sizes of 1- and 2-hop neighbour samples for GraphSAGE:\n",
"\n",
"Note that the length of `num_samples` list defines the number of layers/iterations in the GraphSAGE model. In this example, we are defining a 2-layer GraphSAGE model."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"num_samples = [20, 10]"
],
"outputs": [],
"execution_count": 25,
"metadata": {}
},
{
"cell_type": "code",
"source": [
"train_gen = GraphSAGELinkGenerator(G_train, batch_size, num_samples).flow(edge_ids_train,edge_labels_train)\n",
"test_gen = GraphSAGELinkGenerator(G_test, batch_size, num_samples).flow(edge_ids_test, edge_labels_test)"
],
"outputs": [],
"execution_count": 26,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Build the model: a 2-layer GraphSAGE model acting as node representation learner, with a link classification layer on concatenated `(paper1, paper2)` node embeddings.\n",
"\n",
"GraphSAGE part of the model, with hidden layer sizes of 50 for both GraphSAGE layers, a bias term, and no dropout. (Dropout can be switched on by specifying a positive dropout rate, 0 < dropout < 1)\n",
"Note that the length of layer_sizes list must be equal to the length of num_samples, as len(num_samples) defines the number of hops (layers) in the GraphSAGE model."
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"layer_sizes = [20, 20]\n",
"assert len(layer_sizes) == len(num_samples)\n",
"\n",
"graphsage = GraphSAGE(\n",
" layer_sizes=layer_sizes, generator=train_gen, bias=True, dropout=0.3\n",
" )"
],
"outputs": [],
"execution_count": 32,
"metadata": {}
},
{
"cell_type": "code",
"source": [
"x_inp, x_out = graphsage.build()"
],
"outputs": [],
"execution_count": 33,
"metadata": {
"scrolled": true
}
},
{
"cell_type": "markdown",
"source": [
"Final link classification layer that takes a pair of node embeddings produced by graphsage, applies a binary operator to them to produce the corresponding link embedding ('ip' for inner product; other options for the binary operator can be seen by running a cell with `?link_classification` in it), and passes it through a dense layer:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"prediction = link_classification(\n",
" output_dim=1, output_act=\"relu\", edge_embedding_method='ip'\n",
" )(x_out)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"link_classification: using 'ip' method to combine node embeddings into edge embeddings\n"
]
}
],
"execution_count": 34,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Stack the GraphSAGE and prediction layers into a Keras model, and specify the loss"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"model = keras.Model(inputs=x_inp, outputs=prediction)\n",
"\n",
"model.compile(\n",
" optimizer=keras.optimizers.Adam(lr=1e-3),\n",
" loss=keras.losses.binary_crossentropy,\n",
" metrics=[\"acc\"],\n",
" )"
],
"outputs": [],
"execution_count": 35,
"metadata": {
"scrolled": true
}
},
{
"cell_type": "markdown",
"source": [
"Evaluate the initial (untrained) model on the train and test set:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"init_train_metrics = model.evaluate_generator(train_gen)\n",
"init_test_metrics = model.evaluate_generator(test_gen)\n",
"\n",
"print(\"\\nTrain Set Metrics of the initial (untrained) model:\")\n",
"for name, val in zip(model.metrics_names, init_train_metrics):\n",
" print(\"\\t{}: {:0.4f}\".format(name, val))\n",
"\n",
"print(\"\\nTest Set Metrics of the initial (untrained) model:\")\n",
"for name, val in zip(model.metrics_names, init_test_metrics):\n",
" print(\"\\t{}: {:0.4f}\".format(name, val))"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\n",
"Train Set Metrics of the initial (untrained) model:\n",
"\tloss: 0.6847\n",
"\tacc: 0.6316\n",
"\n",
"Test Set Metrics of the initial (untrained) model:\n",
"\tloss: 0.6795\n",
"\tacc: 0.6364\n"
]
}
],
"execution_count": 36,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"Let's go for it:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"history = model.fit_generator(\n",
" train_gen,\n",
" epochs=epochs,\n",
" validation_data=test_gen,\n",
" verbose=2\n",
" )"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"WARNING:tensorflow:From /Users/swa/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use tf.cast instead.\n",
"Epoch 1/20\n",
"51/51 [==============================] - 2s 47ms/step - loss: 0.6117 - acc: 0.6324\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.7215 - acc: 0.6064 - val_loss: 0.6117 - val_acc: 0.6324\n",
"Epoch 2/20\n",
"51/51 [==============================] - 3s 53ms/step - loss: 0.5301 - acc: 0.7263\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.5407 - acc: 0.7171 - val_loss: 0.5301 - val_acc: 0.7263\n",
"Epoch 3/20\n",
"51/51 [==============================] - 3s 53ms/step - loss: 0.4952 - acc: 0.7441\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.4658 - acc: 0.7928 - val_loss: 0.4952 - val_acc: 0.7441\n",
"Epoch 4/20\n",
"51/51 [==============================] - 3s 56ms/step - loss: 0.4618 - acc: 0.7836\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.3992 - acc: 0.8344 - val_loss: 0.4618 - val_acc: 0.7836\n",
"Epoch 5/20\n",
"51/51 [==============================] - 3s 53ms/step - loss: 0.5118 - acc: 0.7856\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.3644 - acc: 0.8980 - val_loss: 0.5118 - val_acc: 0.7856\n",
"Epoch 6/20\n",
"51/51 [==============================] - 3s 57ms/step - loss: 0.4758 - acc: 0.7964\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 8s - loss: 0.3553 - acc: 0.8936 - val_loss: 0.4758 - val_acc: 0.7964\n",
"Epoch 7/20\n",
"51/51 [==============================] - 3s 57ms/step - loss: 0.4806 - acc: 0.8024\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 8s - loss: 0.2726 - acc: 0.9243 - val_loss: 0.4806 - val_acc: 0.8024\n",
"Epoch 8/20\n",
"51/51 [==============================] - 3s 53ms/step - loss: 0.5067 - acc: 0.8093\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.2734 - acc: 0.9309 - val_loss: 0.5067 - val_acc: 0.8093\n",
"Epoch 9/20\n",
"51/51 [==============================] - 3s 55ms/step - loss: 0.5595 - acc: 0.8014\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.2268 - acc: 0.9627 - val_loss: 0.5595 - val_acc: 0.8014\n",
"Epoch 10/20\n",
"51/51 [==============================] - 3s 52ms/step - loss: 0.5354 - acc: 0.7984\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.2051 - acc: 0.9616 - val_loss: 0.5354 - val_acc: 0.7984\n",
"Epoch 11/20\n",
"51/51 [==============================] - 3s 50ms/step - loss: 0.5476 - acc: 0.8123\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.1896 - acc: 0.9627 - val_loss: 0.5476 - val_acc: 0.8123\n",
"Epoch 12/20\n",
"51/51 [==============================] - 3s 54ms/step - loss: 0.5359 - acc: 0.8113\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.1985 - acc: 0.9594 - val_loss: 0.5359 - val_acc: 0.8113\n",
"Epoch 13/20\n",
"51/51 [==============================] - 3s 54ms/step - loss: 0.5721 - acc: 0.8103\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.1794 - acc: 0.9737 - val_loss: 0.5721 - val_acc: 0.8103\n",
"Epoch 14/20\n",
"51/51 [==============================] - 3s 53ms/step - loss: 0.5526 - acc: 0.8152\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.1600 - acc: 0.9825 - val_loss: 0.5526 - val_acc: 0.8152\n",
"Epoch 15/20\n",
"51/51 [==============================] - 3s 53ms/step - loss: 0.6594 - acc: 0.8034\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.1576 - acc: 0.9781 - val_loss: 0.6594 - val_acc: 0.8034\n",
"Epoch 16/20\n",
"51/51 [==============================] - 3s 54ms/step - loss: 0.5591 - acc: 0.8132\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.1426 - acc: 0.9825 - val_loss: 0.5591 - val_acc: 0.8132\n",
"Epoch 17/20\n",
"51/51 [==============================] - 3s 52ms/step - loss: 0.5637 - acc: 0.7935\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.1359 - acc: 0.9792 - val_loss: 0.5637 - val_acc: 0.7935\n",
"Epoch 18/20\n",
"51/51 [==============================] - 3s 53ms/step - loss: 0.6060 - acc: 0.8083\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.1306 - acc: 0.9912 - val_loss: 0.6060 - val_acc: 0.8083\n",
"Epoch 19/20\n",
"51/51 [==============================] - 3s 53ms/step - loss: 0.5586 - acc: 0.7955\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.1258 - acc: 0.9857 - val_loss: 0.5586 - val_acc: 0.7955\n",
"Epoch 20/20\n",
"51/51 [==============================] - 3s 51ms/step - loss: 0.6495 - acc: 0.7964\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
" - 7s - loss: 0.1193 - acc: 0.9923 - val_loss: 0.6495 - val_acc: 0.7964\n"
]
}
],
"execution_count": 37,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"You can use tensorboard to see pretty dataviz or you can use a normal Python plot:"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"def plot_history(history):\n",
" metrics = sorted(history.history.keys())\n",
" metrics = metrics[:len(metrics)//2]\n",
" \n",
" f,axs = plt.subplots(1, len(metrics), figsize=(12,4))\n",
"\n",
" for m,ax in zip(metrics,axs):\n",
" # summarize history for metric m\n",
" ax.plot(history.history[m])\n",
" ax.plot(history.history['val_' + m])\n",
" ax.set_title(m)\n",
" ax.set_ylabel(m)\n",
" ax.set_xlabel('epoch')\n",
" ax.legend(['train', 'test'], loc='upper right')\n",
"\n",
"plot_history(history) "
],
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"<Figure size 864x288 with 2 Axes>"
],
"image/png": [
"iVBORw0KGgoAAAANSUhEUgAAAtoAAAEWCAYAAABYLDBhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd4lMX2wPHvSa8kpNASIKFIEaSF0ERAVMCKDbFjAftVrw1791qv/rwqiogdFBEVFaUoiEgNvZeEFmoIJJCQvvP7YxYNIUCA3ewmOZ/n2Se7b9n3bMTJ2XlnzogxBqWUUkoppZRr+Xg6AKWUUkoppaojTbSVUkoppZRyA020lVJKKaWUcgNNtJVSSimllHIDTbSVUkoppZRyA020lVJKKaWUcgNNtJVSSilVJYjIJhE5x9NxKFVRmmgrpZRSSinlBppoK6WUUkop5QaaaCsFiMhwEUkVkQMiskpELi21b6iIrC61r6Nze0MRmSAiGSKSKSLveO4TKKVUzSEigSLylohsdz7eEpFA574YEflJRLJEZK+I/CkiPs59j4jINmd7vlZE+nr2k6jqzs/TASjlJVKBnsBO4ErgCxFpBpwJPAMMBFKApkCRiPgCPwG/A9cDJUBS5YetlFI10uNAV6A9YIAfgCeAJ4EHgHQg1nlsV8CISAvgbqCzMWa7iCQAvpUbtqpptEdbKcAY840xZrsxxmGM+RpYDyQDtwKvGmMWGGuDMWazc18D4CFjTK4xJt8YM8uDH0EppWqSa4HnjDG7jTEZwLPYTg+AIqA+0NgYU2SM+dMYY7AdIoFAaxHxN8ZsMsakeiR6VWNooq0UICI3iMgS563GLKANEAM0xPZ2l9UQ2GyMKa7MOJVSSgG2o2NzqdebndsAXgM2AFNEJE1EhgMYYzYA92HvUu4Wka9EpAFKuZEm2qrGE5HGwIfYW4rRxphIYAUgwFbscJGytgKNRESHXymlVOXbDjQu9bqRcxvGmAPGmAeMMU2Ai4F/HxqLbYwZY4w503muAV6p3LBVTaOJtlIQim1wMwBE5CZsjzbAKOBBEekkVjNnYj4f2AG8LCKhIhIkIj08EbxSStVAY4EnRCRWRGKAp4AvAETkQmdbLUA2dsiIQ0RaiMjZzkmT+UAe4PBQ/KqG0ERb1XjGmFXAG8AcYBfQFvjLue8b4EVgDHAA+B6IMsaUABcBzYAt2Ik3V1V68EopVTO9gJ2gvgxYDixybgNoDkwDcrDt+nvGmOnY8dkvA3uwE9/rAI9WbtiqphE7P0AppZRSSinlStqjrZRSSimllBtooq2UUkoppZQbaKKtlFJKKaWUG2iirZRSSimllBtUmxrAMTExJiEhwdNhKKXUSVm4cOEeY0zs8Y+sPrTdVkpVVRVts6tNop2QkEBKSoqnw1BKqZMiIpuPf1T1ou22UqqqqmibrUNHlFJKKaWUcgO3JdoiMlpEdovIiqPsFxF5W0Q2iMgyEelYat+NIrLe+bjRXTEqpZRSSinlLu7s0f4E6H+M/QOwqzc1B4YBIwBEJAp4GugCJANPi0htN8aplFJKKaWUy7ltjLYxZqaIJBzjkEuAz4xdmnKuiESKSH2gNzDVGLMXQESmYhP2se6KVSnlWUVFRaSnp5Ofn+/pUNwuKCiI+Ph4/P39PR2KUkqdtJrSbp9qm+3JyZBxwNZSr9Od2462/QgiMgzbG06jRo3cE6VSyu3S09MJDw8nISEBEfF0OG5jjCEzM5P09HQSExM9HY5SSp20mtBuu6LNrtKTIY0xI40xScaYpNjYGlUVS6lqJT8/n+jo6GrbWB8iIkRHR1f7HiClVPVXE9ptV7TZnky0twENS72Od2472nalVDVWnRvr0mrK51RKVX81oT071c/oyUR7InCDs/pIVyDbGLMDmAycJyK1nZMgz3NuU0opr2SMIa+wmD05BezNLfR0ONXW1r0HeeXXNezM1jsCSqmqwZ3l/cYCc4AWIpIuIreIyO0icrvzkElAGrAB+BC4E8A5CfJ5YIHz8dyhiZFKKXU8xQ4HGQcKWLfzAGt3HmDL3oNkHCggJ7+I4hLHUc/Lysrivffeq9A1ShyGnPwidu3Pp/c5/Zi7egvrd+ewPSuPrIOaaLtLdl4RI2akMjct09OhKKU87ETa7NLOP/98srKy3BBR+dxZdeTq4+w3wF1H2TcaGO2OuJRS1VN+UQl7cgrIOliEwxhCAvzw8xFyC4oPS34DfH0IDvAlyN+XYH9fggN88fORvxvtO++887D3LS4uxuBDbmExBwtLyC0oJr/IgcEAMPLL8YQG+BIS6EdogC/+vlV66otXa1W/FuGBfszbuJeBHcqdI6+UqiGO1Wb7+R09vZ00aZK7QztMtVmCXSlV8xhj2J9fxJ6cQnILihERIoP9iQkLIDjgn+atuMRBXlEJeUUl5BeWkFfkIDuv6O/9fj4+PHL/g6SmptL2jHYEBPgTEBBIWEQk69etZeIfKdx3y7Xs3LGN4sICht5xN7ffNoyQAF+aNmlCSkoK2btz6D5gAGeeeSazZ88mLi6OH374geDgYE/8aqolXx8hKaE28zZqj7ZSNd3w4cNJTU2lffv2+Pv7ExQURO3atVmzZg3r1q1j4MCBbN26lfz8fO69916GDRsGQEJCAikpKeTk5DCgEtpsTbSVUl7l2R9Xsmr7/mMeY7DJc1GJwRiDiODvK/j5+lDetJXWDWrx9EWnEx70Tx3UEoch35l85xWWcP9jz7B61Sq+nPQHC+bM4u4br2Li9Dmc1qwpIQF+fPbpxzSoG0tBfj6dO3fm5usGEx4Ufdh11q9fz9ixY/nwww8ZNGgQ3377Ldddd50LfivqkC5Nopm+NoOMAwXEhgd6OhylFBVrt0/UoXb7aF5++WVWrFjBkiVLmDFjBhdccAErVqz4uwzf6NGjiYqKIi8vj86dO3P55ZcTHV35bbYm2krVIKkZOcxJzSQqNICmsWE0jg4hyN/XLddyOAzbs/NIzchlc2YuoQF+1I8MIi4ymHoRQQT6nfh1HcZQVGIodjjA2B5Ofz9ffH1OfFa4r48QGuhHaKBtBkv2hxHk70OzOmFsqRVIcnJnzklu+/eM82feeI/vvvsOgK1bt7J+/fojGu3ExETat28PQKdOndi0adMJx+WNRKQ/8H+ALzDKGPNymf1vAn2cL0OAOsaYSHfEkpwYBcCCTXs5v219d1xCKVUFJScnH1br+u233/aKNlsTbaWqMYfDsHjrPqas2sXUVbtIy8g9bL+PQHztEJrEhtI0NoymsWF/P48JC6hQWaPcgmI27sklNSOH1Az7My0jl417csgvOvrkw5iwQOIig6gfEcyNrf3JOFBAgK/w0Hkt8Pfzwc9HEBEcxrA/zw4POVhYjI8ItUP8iQ4LdMuXhJAAP8IC/QkPC/v788+YMYNp06YxZ84cQkJC6N27d7l1VQMD/+lh9fX1JS8vz+XxVTYR8QXeBc7FLiC2QEQmGmNWHTrGGHN/qePvATq4K562cREE+/syLy1TE22lvMSxep4rS2ho6N/PvanN1kRbqWomv6iEWev3MHXVLn5bs4s9OYX4+Qhdm0RzY7cEep0WS05B8d+JcZrz59y0zMMS4/Agv8MS76axoYQE+JGWkUPann8S6h2lSq0dStybxobSvWn03+cnxoSSV1jC9qw8tmfn25/O5xsycjhYGM6O7MMbuEPDQRwOW0kk0M+XBhHB1A71x9fH9RMOw8PDOXDgQLn7srOzqV27NiEhIaxZs4a5c+e6/PpeLBnYYIxJAxCRr4BLgFVHOf5q4Gl3BePv60OnxrWZt1GLUSlVk1WVNlsTbaWqgb25hfy22vZa/7l+D3lFJYQH+tGrRSzntq5L7xZ1iAj2P+ycNnERh70+NNQjrVSvdGpGDrM3ZDJh0eFrRoUH+tGkThjdmkT/nYg3qcBQlISY0HK3r169mub1a1FUYigqcVBU4qDQOQYbY6gdGkxYoJ9bF0eIjo6mR48etGnThuDgYOrWrfv3vv79+/P+++/TqlUrWrRoQdeuXd0WhxeKA7aWep0OdCnvQBFpDCQCvx/tzURkGDAMoFGjRicVUHJiFG9OW0fWwUIiQwJO6j2UUlVbVWmzxVbZq/qSkpJMSkqKp8NQqtJs2pPLVOeQkJTNe3EYqB8RxDmt6nJu67p0bRJNgJ9ren5zCorZmJFLbmExTWJDiQ0LdGnSu3r1alq1auWy9/N25X1eEVlojEnyUEhHJSJXAP2NMbc6X18PdDHG3F3OsY8A8caYeyry3ifbbs9Ny2TwyLl8eEMS57aue/wTlFIuV5Pa7VNps7VHW6kqxBjDzPV7+L9p61i0xRbcb1kvnLv7NOPc1vVoE1fLLb2+YYF+tI2POP6BqjraBjQs9Treua08gznK+giu1L5hJAG+PszfmKmJtlLKq2mirVQVcCjBfmvaOhZvySIuMpgnLmhFv9Pr0TAqxNPhqeptAdBcRBKxCfZg4JqyB4lIS6A2dkVgtwry96V9w0jm6zhtpZSX00RbKS9mjOFPZ4K9yJlgv3RpW67oFO+yYSFKHYsxplhE7gYmY8v7jTbGrBSR54AUY8xE56GDga9MJY1HTE6MYsQfqeQUFBMWqH/KlFLeSVsnpbxQ2QS7QUQQL17ahis7NdQEW1U6Y8wkYFKZbU+Vef1MZcaUnBjFO9M3sHDzPnqdFluZl1ZKqQrTRFspL2KMYdaGPbw1bT0LN+/7O8G+olP8SS3wolR11alxbXx9hPkbMzXRVkp5LU20VbWyPD2bbVkHCfL3Jdjfl+AA+zOozPMTWUmwqMRBXlEJ+YXO5bqdS3bnFZXYJbwLHYQE+hIXGUz9iKDDlvmuKGMMf23I5K1p60jZvI/6EUG8MLANVyZpgq1UeUID/WgTF6HjtJVSXk0TbVVt/LpiB3d8uYiKjBAN8PUhyN/nsOQ7wM/HJs7O5PnQ8xLHiQ05DQ/y+zvpbhAZ7HwE0SDCPq8XEYS/rx3+UV6C/fzANgzSBLvSZWVlMWbMGO68884TPvett95i2LBhhIToxNTK1CUxik/+2kR+UYlbVglVSnmvqtJma6KtqoX5G/fyr6+W0L5hJM9f0obCEscRPdClk+hDvdH5pfYXljhsL7i/L0F/J+A+R/SIl95/aF9OQRHbsvLZ4VzxcFtWPjuy81iyNYt9B4sOi1UE6oQHUj8imBKHYfm2bE2wvUBWVhbvvffeSTfa1113nSbalaxLYhQjZ6axeEsW3ZpGezocpVQlqipttibaqspbt+sAt366gPjIYD66sTNRoZ5ZKa5T4/K35xWWsD3bJuA7svLZ5kzGd2TncyC/iOcvOZ1BnRtqgu1hw4cPJzU1lfbt23PuuedSp04dxo0bR0FBAZdeeinPPvssubm5DBo0iPT0dEpKSnjyySfZtWsX27dvp0+fPsTExDB9+nRPf5QaI6lxFCL2i7Ym2krVLFWlzdZEW1VpO7LzuHH0fAL9ffn05mSPJdnHEhzgS9PYMJrGhnk6lKrhl+Gwc7lr37NeWxjw8jEPefnll1mxYgVLlixhypQpjB8/nvnz52OM4eKLL2bmzJlkZGTQoEEDfv75ZwCys7OJiIjgv//9L9OnTycmJsa1catjigjxp2W9WszflAk093Q4StVcHmi3q0qb7dY6YSLSX0TWisgGERlezv7GIvKbiCwTkRkiEl9qX4mILHE+JpY9V6nsvCKGjF7AgfxiPrmpsy7colxmypQpTJkyhQ4dOtCxY0fWrFnD+vXradu2LVOnTuWRRx7hzz//JCJCV8v0tC6JUSzcvI/CYoenQ1HKuzhKYNsiT0dRKby5zXZbj7aI+ALvAucC6cACEZlojFlV6rDXgc+MMZ+KyNnAf4DrnfvyjDHt3RWfqtryi0oY9lkKaXty+OSmZE5voAlPtXGcnufKYIzh0Ucf5bbbbjti36JFi5g0aRJPPPEEffv25amnnirnHVRl6ZIYxSezN7F8WzadGtf2dDhKeY/5I+HX4TDkZ0g4073X8nC77c1ttjt7tJOBDcaYNGNMIfAVcEmZY1oDvzufTy9nv1JHcDgMD4xbyryNe3n9ynb0aKa369WpCw8P58CBAwD069eP0aNHk5OTA8C2bdvYvXs327dvJyQkhOuuu46HHnqIRYsWHXGuqlydE6MAtMyfUqU5HDD/Q/t8zruejcVNqkqb7c4x2nHA1lKv04EuZY5ZClwG/B9wKRAuItHGmEwgSERSgGLgZWPM92UvICLDgGEAjRo1cv0nUF7HGMNzP63i5+U7ePz8VlzSPs7TIalqIjo6mh49etCmTRsGDBjANddcQ7du3QAICwvjiy++YMOGDTz00EP4+Pjg7+/PiBEjABg2bBj9+/enQYMGOhmyksWEBdKsThjzNmZyR++mng5HKe+wcQbsTYW6bWDtL5CZCtHV6/8P17TZ9Zn+2zTwcV86LKYiRYdP5o1FrgD6G2Nudb6+HuhijLm71DENgHeARGAmcDnQxhiTJSJxxphtItIE2+vd1xiTerTrJSUlmZSUFLd8FuU93v8jlZd/WcPNPRJ58sJWiFR84RnlvVavXk2rVq08HUalKe/zishCY0ySh0LyCFe12499t5yJS7az9OnzTmgxKqWqra+uhS1zYdh0eLsjJN0E57/m0ktUi3Z7/3bI3QN1Wx8z2T6VNtudQ0e2AQ1LvY53bvubMWa7MeYyY0wH4HHntiznz23On2nADKCDG2NVVcCERem8/MsaLjyjPk9coEm2UsrqkhhFTkExq3fs93QoSnle1lZYOwk63gCRjaDtFbD4S8jL8nRk3qWkGHIzIDDcrT3a7ky0FwDNRSRRRAKAwcBh1UNEJEZEDsXwKDDaub22iAQeOgboAZSeRKlqmJnrMnh4/DK6NYnmjUHt8NFeK6WUU7JznPbctEwPR6KUF1j4CRhje7EBut4JRbmw6FOPhuV1cjPAOCC8nlsv47ZE2xhTDNwNTAZWA+OMMStF5DkRudh5WG9grYisA+oCLzq3twJSRGQpdpLky2WqlagaZMW2bO74YiHN6oTxwQ2ddGGXaspdw9i8TU35nJWpfkQwjaJCdEKkUsWFNqE+rb/tzQaofwYk9IR5I20vrgtV2fbMUWIT7aAI8A8+5qGn+hndumCNMWYSMKnMtqdKPR8PjC/nvNlAW3fGpqqGLZkHGfLxAiJDAvj05mRqBfl7OiTlBkFBQWRmZhIdHV2thwQZY8jMzCQoKMjToVQ7yYlR/LZ6Fw6H0TtequZaPdEmkJ1vPXx71zvhq6th9Q/Q5nKXXKpKt9u5GWBKIOzYvdmuaLN1ZUjltTJzCrjx4/kUlTj4algX6tbS5KS6io+PJz09nYyMDE+H4nZBQUHEx8cf/0B1QpIToxi/MJ0NGTmcVjfc0+Eo5RkLPoLaidD07MO3n9YfoprAnPdclmhX2XbbOGD/DvANgOzNxz38VNtsTbSVVzpYWMzNn6awPSuPMUO70KyO/uGszvz9/UlMTPR0GKoK65oYDcC8tExNtFXNtGslbJkN570APmVGBvv4QJc74JeHYOt8aJh8ypersu327P/BlCfglqnQ0P1VU9y6BLtSJ6O4xMHdYxazPD2L/13dgU6NozwdklLKyzWMCqZerSDm6ThtVVMtGAV+QdD+2vL3t7/GjkmupgvYVEhRHvz1NiT2csmXjYrQHm3lcYXFDlZsz2bhpn0s2LSXRVv2sSenkBcvbcN5p7t3NrBSqnoQEZITo5iblokxpuqNGVXqVOTvh6Vf22EhIUfpnAoMg443wpx3IGvLP5Mla5JFn0PubjhrdKVdUhNtVemyDxaxcMteUjbtI2XTPpamZ1FQ7ACgcXQIZ50Wy3mt69K/TX0PR6qUqkq6NIli4tLtbMo8SGJMqKfDUaryLPvalvDrfMuxj+tym+3RnvcB9Hvx2MdWN8UF8Ndb0KgbJJxZaZfVRFu5lTGGLXsP2qR68z5SNu1l/e4cAPx8hNPjIriua2M6J9SmY+Pa1AnXCY9KqZPTxVlPe/7GTE20Vc1hjB020qADxHU69rER8dD6Elj0GfQebhdrqSmWjIH92+Di/0El3vHSRFu5RU5BMU/9sIKZ6/awJ6cAgPAgPzo1rs3ADnF0alybdvGRBAdoTWyllGs0jQ0jOjSAeRv3clXnGnhbXNVMm2ZBxhq45L2KHd/tLlg5wa4W2fV298bmLUqKYNZ/7ReRshVZ3EwTbeVy+UUlDP00hQWb9nJRuwYkJdQmqXEUzeuEaX1bpdTJ27MBpj0NF74JYXWO2H1onPa8NJ0QqWqQBaMgKBLaXFax4+OTID4Z5o2A5KHgUwM6vJZ/Y8elD3i1UnuzQauOKBcrLnHwr7GLmZOWyetXtuPNq9pzbZfGtKgXrkm2UuoUGVj7C/zxylGPSE6MYltWHun7DlZiXEp5yP4dsOYn6HDdcVc4PEy3O2HfJvv/U3XnKIE/34B6bW098UqmibZyGWMMj05YzpRVu3j24tMZ2CHO0yEppVxARPqLyFoR2SAiw49yzCARWSUiK0VkjFsCiWkOnW6EhZ9AZmq5hyQ7x2kv2KS92qoGWPQZOIoh6eYTO6/lRRDRCOZWcLiJuxljJ2lOftwmxq608jvI3ABnPVTpvdmgibZyEWMML01azTcL07nvnObc2D3B0yEppVxARHyBd4EBQGvgahFpXeaY5sCjQA9jzOnAfW4LqNdw8A2E354td3fLerWoFeTHfK2nraq7kiJY+DE07QvRTU/sXF8/6DIMNv8F25e4J76KMgamPgWTH7OlB3/+t93mCg4HzHwdYlvaLxceoIm2con3ZqTy4Z8bGdI9gXv7Nvd0OEop10kGNhhj0owxhcBXwCVljhkKvGuM2QdgjNnttmjC60L3u2HVD5CecsRuXx+hc4KO01Y1wNpf4MAO6HzryZ3f8QYICPNsr7bDAZMehNlv289x5v32jtVRvkifsDU/QcZq6PngkatlVhJNtNUp+3LeZl6bvJZLO8Tx1IWtdaEIpaqXOGBrqdfpzm2lnQacJiJ/ichcETnqQEgRGSYiKSKSkpGRcXIRdb8HQmNtL1g5PV/JiVGk7cll94H8k3t/paqCBR9CREM4rd/JnR8UAR2uhxXf2rHela2kGH64y07m7P4vOP916Pu0HQYz602Y9dapvb8xMPM1iGpa8YmibqCJtjolPy7dzhPfr6Bvyzq8esUZOuFRqZrJD2gO9AauBj4UkcjyDjTGjDTGJBljkmJjY0/uaoHh0OsRe9t73eQjdndpEg3Ago37Tu79lfJ2GWth40xIuunUqoZ0uc2OiV7woetiq4jiQvj2Flg6Bno/Buc+Z8dPi9iEu83ltsJQyscnf411k2HnMuj5gEcrq2iirU7aH+sy+Pe4JXRuHMW713bE31f/OSlVDW0DGpZ6He/cVlo6MNEYU2SM2Qiswybe7tNpCEQ1gWnPHDF56vQGtQgJ8GXexky3hqCUx6SMBh9/6HDDqb1PVCK0vMC+X2ElVeopyodx18Oq7+G8F6D3I4dPUvTxhUs/gObnwU/32x73E3WoNzuyEZwxyHWxnwTNjNRJWbh5L7d/vpDmdcIZNSSJIP8aUIdTqZppAdBcRBJFJAAYDEwsc8z32N5sRCQGO5Qkza1R+fpD36fs+MulYw/b5e/rQ6fGtXVCpKqeCnPtKoenD4Swk7wrVFq3uyBvHyz76tTf63gKc2HMIFj3K1zwXzsMrDy+/nDlp3a59AnDYP3UE7tO2nTYlmLHfPv6n3rcp8CtifbxSkKJSGMR+U1ElonIDBGJL7XvRhFZ73zc6M441YlZs3M/N328gLq1Avn05mRqBXn2H7FSyn2MMcXA3cBkYDUwzhizUkSeE5GLnYdNBjJFZBUwHXjIGOP+7uTWA+1Kb9NfgqK8w3YlJ0SxZucBsg4Wuj0MpSrV8m+gYP/JT4Isq1E3qN8e5o6wkxPdJT8bPr8MNv0JA9+Hzrcc+/iAELjmK6h7Onx9PWyeXfFr/fEa1IqD9teeWswu4LZEuyIloYDXgc+MMWcAzwH/cZ4bBTwNdMHOeH9aRGq7K1ZVcVsyD3L9R/MJCfDj81u6EBse6OmQlFJuZoyZZIw5zRjT1BjzonPbU8aYic7nxhjzb2NMa2NMW2NMJXSNYW83n/sc7N8G894/bNff47Q36ThtVY0YA/NHQd020LCLa95TxPZq71kHqb+55j3LOrgXPr3Y9jJf8TG0v7pi5wVFwHUTICIexlwFO5Ye/5xNs2DLbOhxL/h5PkdxZ492RUpCtQZ+dz6fXmp/P2CqMWavs1zUVKDyl/NRh9m9P5/rPppHUYmDz29JpmFUiKdDUkrVdAlnQvN+8Oeb9o+50xnxEQT4+TAvTcdpq2pk63zYtdz2ZruywlfrgRBe39axdrUDu+CTC2D3ahjsHPJyIkJj4IbvbdL9+WWwZ/2xj5/5GoTWseULvYA7E+2KlIRaChyquXIpEC4i0RU8V52AtIwc9uWe/C3U7INF3DB6PntyCvjkpmSa1w13YXRKKXUKznkGCg/YZZadgvx9ad8wkvm6QqSqThaMgsBa0PZK176vXwAkD4W0GbBrpeveNzsdPh4A+zbDtd+cfCnCiHi4/nv75eKzgZC1tfzjti6wn6H7PSe2JL0beXoy5INALxFZDPTCzmSv8NqbLqnHWgPM3rCHs9/4gw7PT6XDc1O4YsRsHh6/lA/+SGXaql2kZeRQXHL0cVkHC4u5+dMFpGXk8uENSbRvWG7VLqWU8oy6raHdNTB/pP2D7tQ1MYoV27LJKSj2YHBKAekL4bfnITP15N8jJ8NW6mh3NQSGuS62QzrdBH7BrlvAZm8ajB4AuRlw/XfQpNepvV9MMzuMpOAAfD7Q/j7KmvkqBEed+JL0buTnxvc+bkkoY8x2nD3aIhIGXG6MyRKRbThnsJc6d0bZCxhjRgIjAZKSkly0Xmf18/7MNGLCArm9VxNSM3JJzcjh9zW7GZeS/vcx/r5Co6gQmsaG0SQ2jKaxoTSJDaNRVAgPfrOUxVv28d61HenRLMaDn0QppY6iz2OwYjxMfxEuGwlAcmI0jt83kLJpL71b1PFwgKrGWjIGfrwXSgrtXZcWA6DrHZDQ88S4vA/mAAAgAElEQVSGfyz+3L7H8SYRnqyQKDt2evGX0PeZU6tokrHWjskuKYQbf4QG7V0TY/0z4Npxtlf7i8tgyE92SAnYpeTXT4Gzn3TPF5GT5M5E+++SUNgEezBwTekDnGWg9hpjHMCjwGjnrsnAS6UmQJ7n3K9O0NqdB5i5LoMHzzuNW3s2OWxf9sEiUvfkkOZMvtMyckjNyGX62t0UlRz+veWVy9vSv039ygxdKaUqLiIOutwOf/0fdLsb6p9Bx8aR+PkI8zdqoq08oKTYLroy5x1IPAsGvAorJkDKR7B2EtRtaxPutlccf9Keo8Qu3pLQE2JbuC/mrnfamtopH0HvI4rFVcyOZfD5pbYe9pCf7R0nV2rUFa76AsYOthMkr5tgK5TMfM0m3clDXXu9U+S2RNsYUywih0pC+QKjD5WEAlKcs9V7A/8REQPMBO5ynrtXRJ7HJusAzxljdKDdSfhoVhpB/j5c26XxEfsiQvzp2Kg2HRsdXtCluMTB1n15zsQ7h8SYMM5tXbeyQlZKqZNz5v2w6FOb3Fz/HSEBfrSNj9B62qry5e2D8TdD6u+QfBv0e9HWcz77cej5b1uib+4I+OFO+++18612uEPYUb4Qrp8C2VvgvOfdG3dMc7tQzIJR0OM+8A868pjiAjiwAw7shP3b7fP92+3rAztsZZCgSLhxIkQ3dU+czc+xd67G3wzjboCzn4A1P9kVYw/1cHsJMaZ6jLhISkoyKSkpng7Dq2QcKKDHy78zqHM8Lwxs6+lwlFLHICILjTFJno6jMrml3Z79Dkx53E6catqH//yymtGzNrLs6X4EB+jCWl4nPcVWi2l+rmuraHhSxjrb25q1BS54AzodZSkQY+zEvbkjYP1k8A2AtoOg6+1Qr8zf7C8ut5MU71vu/gVYUqfbMdCdb4WQGGdSvQP274AD2+FgOZV8/IJs1ZJaDSCyMfR51K7K6G4LP7HDcgLCAWN/PyFR7r8uFW+z3Tl0RHnY53M3U+RwcHOPRE+HopRSlSN5KMz7AKY+BYl/0CUxig/+SGPx1n10b6pzTLzKjqXw6UVQdNDWhD73eWjkotrQnrJuCnx7ix0KcuOP0Ljb0Y8VgaZ97GPPelsLfskYWPKFHSLS7S5bujJrE2yYBr0frZxVDpv0hnpn2F5tBEJjoVZ9W/kjPskm0+H17bZw5yO4tme+KHUaYhfCmfqUvaNVSUn2idBEu5rKLyrhi7mb6duyLk1ivWdSgFJKuZVfoL2N/N0wWPEtSacNRATmb9yribY3ObATxl5tK0T0fRpmvQmjz4OWF9pyjTHNPR3hiTHGzg+Y9gzUawODx0Jkw+Oe9reY5rb3++wnYNFnMG+k7RWPamJXOBRf6FhJi2SLwE2TIC8Lwut5fAnz4+pxr/1iUvYugJfwdHk/5SYTFm1jb24ht/bU3mylVA3T9kr7R/f356jl56B1/Vo6TtubFOXBV9fYRO6ar+xQiX8tgj5P2KEU73aBn/4NObs9HWnFFOXBhGF2rHXrS+DmySeWZJcWXNsmjvcusSsohkTbJctbX2x7kCtLYLj9DN6eZB8S19FrY9Ue7WrI4TCMmpVG27gIuiR6320UpZRyKx8fOOdZW/5rwUckJ/Zi7PwtFBY7CPDT/iWPMgZ+uAu2LbKVIw71QgaEQq+H7FCAP16BhR/Dsq+h+7/sEApXlWvLTrcTFPP22VVF67e31TFO1v7t9kvD9sW2N7rng64ZQuHrD20us49dq2xVHVUlaaJdDc1Yt5u0jFz+b3B7pLpMLlFKqRPRrK8dazrzNXqcdw4f/+Vg+bYsOjXWzgeP+uNVWPGtHR7S6sIj94fFwgWv21KNvz0LM176p9RchxvA9wTTlsKDsPkvm1xv+A32rD18f1CEHXbQpDc06WOrZFT07+bWBfD1tVCYa5cWb3nBicVWUa4uj6cqlSba1dCHMzdSPyKI89tq3WulVA12zrMwshfdd34BdGHexr2aaHvSigk2cW53jS0ddywxzeCqz2HrfDvR7af7bXWOc56BFucfPRk2xlbnSP3NJteb50BJga2K0bg7dLwBmp79z5CMtOmQ9octDQdQK96uYNikNyT2gvCjlLY9tAhNrQa2wo0mw+ooNNGuZlZsy2ZOWiaPDmiJv6/eIlVK1WAN2kPbKwlZOJKuMe2Zk5rJnb2beTqqmmnbQvj+DmjUDS56q+K9xg2T4aZf7AIvU5+2wzQadbMVShp2tsfkZNix3YeS65xddntsK1uFpunZNsn2Dz78vdteYR/G2OXC02bYx5qfYcmX/7xHk972kdDDLlFeehGaKz/1ykoXyntool3NfDRrI6EBvgxOroT6lUop5e3OfgJW/cDjId9x0frBpGzaS1KCJkaVKnsbjL3GLsZy1RfHXwWxLBE7LKN5P1j8GUz/D3x0DjQ7F3J32zKBYCcSNuljhw01Pdv2Nlf0/aOb2kfnW+wqjDuX/ZN4L/wY5o2wlT9qxdmFY0ovQqPUMWiiXY3szM7nx6Xbub5bYyKC9X9+pZSidgJ0vpU2896ne3hfnvh+BT/ec6be8asshbm2TF1hLtzwPYSeQolFXz+7emLbQTDnXVvnObqZrVbS7OxTn9h4iI8vNOhgH2feD0X5sHWeTbq3LYReD0PH60/9OqpG0ES7Gvlk9iYcxugCNUopVVrPB5HFX/BO+BeclX4bn/y1iaFnNfF0VNWfwwHf3Qa7VsA146BOK9e8b2AY9H7EPiqDf5Bz3Havyrmeqlb0K301kVtQzJh5m+nfph4No0I8HY5SSnmP0GgY8Cq1MxcxJfw5vp82ne1ZeZ6Oqvqb/gKs/hHOe9Eusa5UDaSJdjXxTcpW9ucXc2tP7aVRSqkjtL8aueEH6vod5Gt5nB/GfuDpiKq3pV/Bn2/Y1Qy73uHpaJTyGE20q4ESh2H0X5vo2CiSjo1qezocpZTyTok98b19Jjm1mnLHrqfZ9PXDduKbcq0t82DiPbY+9QVvuGYBF6WqKE20q4Gpq3ayZe9BhmpvtlJKHVtEHFF3TeMn/34krP6Aks8vg4O6PLvL7NtsS/BFxMOgz7Qqh6rxNNGuBkb9uZGGUcGcd3o9T4eilFJeLyAomNirR/BI0VDMpr/gg16wfYmnw6r6Cg7YCiMlRXbyo9aXVkoT7apu8ZZ9pGzex809EvH10dtzSilVEV2aRFPc7noGFT1NUXERjO5nV/vzBkX5dtXBJWM9HUnFOUrg21shYy0M+hRimns6IqW8gibaVdyoWRsJD/LjyqSGng5FKaWqlMfOb0mqfwvuDH0TE9/Zrlz484NQXOi5oBwlMGEoLPwEvr8dfnnE9hB7u6lPwbpf4fxXoWkfT0ejlNdwa6ItIv1FZK2IbBCR4eXsbyQi00VksYgsE5HzndsTRCRPRJY4H++7M86qauveg/yyfAfXdGlEWKCWRFdKuUcF2vIhIpJRqs2+1RNxnqjosECGD2jJ1C0Ovjv9Heh+Dyz4ED69EPbvqPyAjIHJj8HqiXaJ8a53wbz34QsvHke+f7vtyZ7zDiQPg85V4j+9UpXGbYm2iPgC7wIDgNbA1SLSusxhTwDjjDEdgMHAe6X2pRpj2jsft7srzqrsk9mb8BFhSPcET4eilKqmKtiWA3xdqs0eValBnoKrkhrSsVEkL/y6nqwzn4IrPoadK2BkL9g8p3KDmfOOTay73gU9/gX9X4KBI2wVj5G9YdfKyo3nWIoLYNab8L8kWDURznoY+v3H01Ep5XXc2aOdDGwwxqQZYwqBr4BLyhxjgFrO5xHAdjfGU63szy/i6wVbufCM+tSPCPZ0OEqp6qsibXmV5eMjvDCwLdl5Rbzy61pocxncOg0CwmzP9rwPbE+zuy0fD1OegNMvhfNe+Gd7+2vgpkk2sR11Lqz6wf2xHM/6qfBeN5j2jF0t8a55cPbjdol0pdRh3JloxwFbS71Od24r7RngOhFJByYB95Tal+gcUvKHiPQs7wIiMkxEUkQkJSMjw4Whe7+v528lp0AXqFFKuV1F2nKAy51DAMeLyFEnjXhju926QS1u6p7A2PlbWLh5H9RtDUN/h2bnwi8P22XECw+6L4CNf9rx4Y17wMD3wafMn+b4JBg2wy5hPu4GmP6SXd68su1NgzGD4csrbG3sa7+Fq8dCVGLlx6JUFeHpyZBXA58YY+KB84HPRcQH2AE0cg4p+TcwRkRqlT3ZGDPSGJNkjEmKjY2t1MA9qbjEwcd/baRrkyjaxEV4OhyllPoRSDDGnAFMBT492oHe2m7fd+5p1KsVxBPfr6C4xAHBkTB4DPR5HJaNg1F9IWOd6y+8ayV8dS1ENYHBX4J/UPnH1aoPQ36GdtfAH6/A19fZcnqVofAg/P4CvNsVNs6Ec56FO+ZA83Mq5/pKVWHuTLS3AaV7NeKd20q7BRgHYIyZAwQBMcaYAmNMpnP7QiAVOM2NsVYpk1bsZHt2Preeqb3ZSim3O25bbozJNMYUOF+OAjpVUmwuExboxzMXt2b1jv18MnuT3ejjA70ehuvGQ84uO0562Teuu2j2NvjySggIgWvHQ/BxVvb1D4KB70H/l22Fj1Hn2l5mdzEGVn4H73SGma9B64vhnhQ48z7wC3DfdZWqRtyZaC8AmotIoogEYCc7TixzzBagL4CItMIm2hkiEuucgIOINAGaA25sTaoOYwyj/kyjSUwoZ7es4+lwlFLV33HbchGpX+rlxcDqSozPZfqdXo8+LWJ5c+o6dmTn/bOj2Tlw259Q/wyYcKutcV2Uf2oXy8+2SXb+frj2G4isYIlWEeh6B1z3LRzYASP7QOr0U4ulPLtXw6cXwTdD7BeAm36By0dBrQauv5ZS1ZjbEm1jTDFwNzAZ2+iOM8asFJHnRORi52EPAENFZCkwFhhijDHAWcAyEVkCjAduN8Z4aW2jyrVg0z6WpWdz85mJ+OgCNUopN6tgW/4vEVnpbMv/BQzxTLSnRkR49uI2FDsMz/246vCdEXFw44/Q415b4/qjcyAz9eQuVFxgh4vsWQuDv4B6bU/8PZr2gWHTIby+Lf835z3XTNrMy4JfhsOIHrBzOZz/uh0f3rj7qb+3UjWQmMqYTV0JkpKSTEpKiqfDcLthn6WwYNNeZg/vS3CAr6fDUUq5iIgsNMYkeTqOyuSt7fa70zfw2uS1fHxTZ/q0KOfO4dpf7QRJRwlc8j9bKaSiHA67IM2K8XDpSGh31akFW3AAvrsd1vxkx29f+ObRx3mXVVxge8X3b7ePvRth/geQuwc6DYGzn4TQ6FOLT6lqqqJtttbiqUI27sll6upd3N2nmSbZSinlJkN7NmHConSe+mEFU+/vRZB/mfa2RX+4fRaMv8kOrdg825bk8ws8/ptPe9om2X2fPvUkGyAwHAZ9DjNfhRn/sb3kV30JgWF20Z3922wSfWD7Pwn1ocfBPUe+X8Mudrx4g/anHptSShPtquSjWWn4+/hwfbfGng5FKaWqrQA/H54f2IZrPpzHO79v4MF+LY48KLIhDJlka0nPfRfSF8CVn0DthKO/8bwPYPbbdvXEM+93XcA+PtB7ONQ9HSbcBm+2BlNO+b/gKKgVZ8dZx3X853mtBhDewFY2CdJKVkq5kibaVcSiLfsYM28LVyc3ok54BW8LKqWUOindm8ZwWYc4PpiZysAOcTSrE3bkQX4BdvXGxt3h+zvh/bNsVZBWFx557KqJ8Msj0PJCGPCqndToaq0ugqHNYMkYCIkqk0jXB39d3EypyubpOtqqAvIKS3hw3FLqRwQzfEBLT4ejlFI1wmMXtCLY35cnv1/BMecztboQbvvDLtzy9bXw62NQXPjP/i1z7bjs+M62coePG4f+1WkF5z1ve8zPGAQJZ9oa3ZpkK+URmmhXAa/8uoa0Pbm8duUZhAf5ezocpZSqEWLCAnm4f0vmpGXy7aKyy0CUEZUIt0yB5GF2KMnHAyBrq13kZsxVtnf56q804VWqhtFE28vN3rCHT2ZvYkj3BLo3jfF0OEopVaNck9yIzgm1ee7HlezMPk7tbL9AOP81O1Y7Yy28fyZ8fin4+tu611rBQ6kaRxNtL7Y/v4iHxi+jSUwoj/TXISNKKVXZfHyE165oR2GJg0cnLDv2EJJDTr/UDiWJbAh5++CacbbHWylV42ii7cWe/3EVO7LzeH1QOy3np5RSHpLg7OyYvjaDbxamV+yk6KYwdDrcv8JW+FBK1UiaaHupaat28c3CdO7o3ZSOjWp7OhyllKrRbuyWQHJiFM//uIrtWXnHPwHskJGQKPcGppTyappoe6G9uYUMn7CclvXCubfvaZ4ORymlajwfH+H1K9pRYgyPfFvBISRKqRpPE20vY4zhye9XkJ1XyJtXtSfAT/8TKaWUN2gUHcKjA1ry5/o9fLVgq6fDUUpVAbpgjZeZuHQ7Py/fwUP9WtCqfi1Ph6OqOmPgwE7YvRJ2r4ZdqyBrMyT2gk43Qng9T0eoVJVybZfG/LJiJy/8tIqezWOIrx3i6ZCUUl5ME20vsmt/Pk/9sJIOjSK57awmng5HVdSeDZC91dbH9Qty/gwEv2DwD7I/ff3dsxJcaXlZNpnevcr5WA27VkJ+1j/HhNW1yfWMl2Dmq9DqYkgeCo26uSe+3D2w6nvY8Bs06wtJt7j/96CUG/n4CK9cfgb935rJw+OX8cUtXfDx0X/TSqnyaaLtJYxz3F9BcQlvXNkOP18dMuL1ti+Gma/Dmp+Of6z4lEq8g0ol5KV/BlXgmFI/D+49PKneX2pBjcBadoW40wdCndPt8zqt/6njm5kKCz6CJV/Aygn2mM63wBlXQWA5S02fiPz9sOZnWDEeUqeDKYGQGFg7CdZMgkvehVr1T+0aSnlQw6gQHr+gNY99t5wv52/h+q6NPR2SUspLVSjRFpFLgd+NMdnO15FAb2PM9+4Mrib5asFWZqzN4JmLWtMk9hQTnarIGMjaAptnw+ZZkJsJiT2haV+IbeFdvaBb58Mfr8KGqRAUAb2GQ+JZUJz/z6MoH4rzyvzMh6K8cn4WwME95Z9TUnDsWHwDIKYFNO4BdVvbZLpOa4iIP/bvLLop9H8Jzn4Cln8DCz6En/8N056BdlfbpDu2RcV/J0V5sH4KLB9vfxbnQ0Qj6H4PtL0C6raBBaNgypMwohtc+KatNexOWVshIFSrPpQhIvcCHwMHgFFAB2C4MWaKRwOrYq5ObsgvK3bwn0mr6dU8lkbROoREKXUkqcjMaRFZYoxpX2bbYmNMB7dFdoKSkpJMSkqKp8M4KVv3HqT/WzNp1zCy5tyGNAYyN8Dmv2DTXzbB3u+sTxsUaZOjvWn2da04aHo2NDsHmvSCYA+UOzQGNs2ywy02zoSQaOh2F3S+1Sbb7uJwlEre8w5P5APDIKop+LrgxpQxkL4A5n9oh3qUFNovD52HQovzy79GSRGkzYAV38Lqn6DwAITG2gS6zRXQMPnIZH/PepgwDLYvsr3nA16F4MhTj/+wa2ywQ2NWTLC9/+2vhq53QUwz117HxURkoTEmqRKus9QY005E+gG3AU8CnxtjKr3Yc1VutwG2Z+XR782ZtG5Qi7FDu9aMtlspBVS8za7oX+jyxjHosBMXcDgMD3yzFBHh1SvOqL4NtcNhhzgc6rHePBtyM+y+0DrQuDsk3Gd/xrYCHx/bw73hN0j9DVb9AIs/t0Mw4pLseN9m50CDDuDjxsV8jLHXn/k6bJljxzif9yIk3WR7S93NxwcCQuzDnURsYtwwGfq9BIs/g5SPYdz1EN7Aft6ON9pEesscOyxk1Q9wMBMCI+D0S2xyndDz2Il/THO4ZQr8+Ya9K7DpL7h0hE3qT1XWVvjjFVgyxibYZ95n41v8hf0sLS+wPewNu3jXHZLKd+jDn49NsFeK1OxfyMlqEBnMkxe25uFvl/HZnE0M6aGrPyqlDlfRHu3RQBbwrnPTXUCUMWbIcc7rD/wf4AuMMsa8XGZ/I+BTINJ5zHBjzCTnvkeBW4AS4F/GmMnHulZV7RkZ9WcaL/y8mlevOINBSQ09HY7rFBfAzhWwZbZNprbM+WdSXq14SOhhk+rGZ9phDMf7O19SDNtSYMM0m3xvXwwY27vdpLcdYtKsL9Rq4Jr4jYG1v8DM12zva614m7h1uM6Oj64JHCWw7lc75CP1d/Dxtz35OTvtWPIWA+ywkGbn2MmfJyp9IUwYCntTodvdcPaTdnz6icrZbRP3lNH2ddIt0PPfEFbHvj6wyw6NWTDKLocdl2QT7lYXufdL2gmqxB7tj4E4IBFoh217ZxhjOrn72mVV1Xa7NGMMN32ygLlpmfx671kkxFTCF3CllMdVtM2uaKIdir29eA5ggKnAi8aY3GOc4wusA84F0oEFwNXGmFWljhkJLDbGjBCR1sAkY0yC8/lYIBloAEwDTjPGlBztelWxwd6w+wDnvz2Ls5rH8OENSVTZTiVHiR0SsG2hTUq3LYJdK+zwA7DDGxp3t+OIG3eH2i6YOJSbCWnTbQK44Teb/IHtDa97up1sF97AVtio1QDC69vnx0sIHQ5Y/QPMfAN2LYfIxjZpa3cN+AWcetxV1Z4NNpHdnw4tL7JJ9qlOmgQozIWpT9kkOLYVXDYS6p9RsXPz9sFfb8O89+0Xuw7XwlkPQ+RRvrAW5tre7rnv2WFJkY3t8J/217rms5yiSky0fYD2QJoxJktEooB4Y8wyd1+7rKrYbpdnZ3Y+5775By3qhvP1bd3wra53JpVSf3Npon2SAXQDnjHG9HO+fhTAGPOfUsd8gG3sX3Ee/4YxpnvZY0VksvO95hztelWtwS4qcXD5iNls3XuQyfefRZ3wk+jJ8wRjbCm7bYucifVi+yjMsfsDwqFBezukI64jNOzq/goTxtgydqm/2THDe9Ng/47yJxKGRJdKwMsk4wczYdZbsGctRDeHng9A2ytdMwZaHdv6qfDDXbaSytmPQ/d/Hb23uSAH5o2Av/4HBfuhzeXQ5zF7Z6QiHCW2Asrs/8HWeXZOQNLN0OU2j9YVr8REuwewxBiTKyLXAR2B/zPGbHb3tcuqau32sXy7MJ0HvlnKExe04taeWp5VqerOpWO0RWQqcKUxJsv5ujbw1aEk+ijigNJLZ6UDXcoc8wwwRUTuAUKxPeaHzp1b5ty4cuIaBgwDaNSoUUU+itcYMSOVZenZvHtNR+9OsvP322Rk28J/kuuDe+w+3wCo19ZWqYjrCHGdbILqU8mlCUWgXhv76HGv3WaM7fE8sMM+9u8o83w77Fxmhx1Q6stmndZwxWhoPdCrhhVUe83PhTvnwo/32son6ybDwBEQVWrMa1E+pHwEf/7X/htscT70edz+dz8RPr522Eiri2wFmdn/g1lvwpx3oO0g28tdt/Xx38eY8qvM+AVWPOn3jBFAOxFpBzyArTzyGdDLo1FVcZd1jGPS8h28NnktfVrWoWlNrB6llDpCRbvqYg4l2QDGmH0iUscF178a+MQY84azR/tzEanwX01jzEhgJNieERfEUylWbMvm7d/Wc3G7BlxwhhfWE3Y4bDWQJV/Cyu9tAoFAbEs4rZ9Nqht0tEM0TmZsbmUQsZVLQqJsnEdTUgQ5u+zqiSVFdqJcZX9RUFZIFAz6DJZ9DZMegvfPhP4vQ7vBdkLjzNdsrfDEXtD3KYh3Qedvw2S46nN7F2TOe/bf/JIv7AI+/iHHKMnoTLDL06QP3ODVlU+LjTFGRC4B3jHGfCQit3g6qKpORHjpsrac9+ZMHvxmKeNv765DSJRSFU60HSLSyBizBUBEEjisG7Bc24DSgyXjndtKuwXoD2CMmSMiQUBMBc+tkvKLSvj3uCVEhQbw3CXHSAA9ITsdloy1ica+TXbRk3aDoc1ldihIYLinI3Q9X39bczoi3tORKLBfkNoNtuP5v78DJt4NU56wE2njO9te7iZu6HiNagIXvG6HoKR8ZMsVlhTaSZ8hUcdYPKjMCqD+QXY4knc74Byedz3Q0zlm29/DMVULdWsF8ezFp3Pf10sY9Wcat/Xy6jsbSqlKUNFE+3Fgloj8gS0N1RPnkI1jWAA0F5FEbJI8GLimzDFbgL7AJyLSCggCMoCJwBgR+S92MmRzYH4FY/Vq3yxMZ92uHEYPSSIyxAsm1xUX2FX8Fn9hJxZibKm13o/ZW+vuLiunVHkiG8INE+3ExXW/2qokp/Vzf1m+kCg46yH7qL6uwrbFNxtjdjqrP73m4ZiqjUvaN2DS8h28MXUdZ7esQ/O61bCDQilVYRW6R26M+RVIAtZiq4E8AOQd55xi4G5gMrAaGOes1/qciFzsPOwBYKiILHW+7xBjrQTGAauAX4G7jlVxpCoZn7KVlvXC6dPCFSNvTsGOpfb2/BstYPxNkLEWej0M9y6FG3+Edldpkq08y8cHut8NQ36CFv1reu1rlzHG7AS+BCJE5EIg3xjz2fHOE5H+IrJWRDaIyPBjHHe5iBgRcfvETm8kIrx4aVtCA3x58JulFJc4PB2SUsqDKjoZ8lbgXuwQjiVAV2AOcPaxznPWxJ5UZttTpZ6vAnoc5dwXgRcrEl9VsW7XAZamZ/PEBa08U8rv4F673Pbiz2HncvANhFYX2trQib108p9SNYCIDML2YM/A3qH8n4g8ZIwZf4xzfLHrKPxdrlVEJpYu1+o8Lhz7t2Kem8KvEmLDA3nukjbcM3YxH8xM464+3r0qqVLKfSo6dOReoDMw1xjTR0RaAi+5L6zq6duF6fj5CAM7HFFA5dQ4HFCQDXlZttJG6Ud+lt2+bxOsn2LHndZvB+e/bsuihUS5NhallLd7HOhsjNkNICKx2LUKjppoY9c02GCMSXOe8xVwCfauY2nPA68A1XrsTUVceEZ9flmxg9enrAXgzt5Nq+5aCUqpk1bRRDvfGJMvIohIoDFmjYi0cGtk1UxxiYMJi7fRp2UdYsJOslLHyu/sJK2/E+hDCXUWx5yb6h8KodF2xbwO19qSfEqpmsrnUNQ/61cAACAASURBVJLtlMnxhxEet1yriHQEGhpjfhaRoybaVbks64kQEd64sj1+Pst4bfJaVmzL5rUr2xEWqHX5lapJKvp/fLqIRALfA1NFZB9Q6YsbVGUz12eQcaCAKzqdZHWLLfNg/C0QGmsXVwmuDbUT7c/g2hAcWeq58xEUabd7awk+pZQn/OpcBGys8/VVlBnid6KclUv+Cww53rFVtSzryQgO8OX/BrenbVwE//llNRt25zDyhiQS/7+9O4+Pqjz7P/65sofsIQESQoCwo6yGHXcruBSq1SrWllq1q91sbe3z+Fh/9unzs8tjrZW2WuWnrW3dF6oIbqhFBEHKDoEkICSsAUxIIPv9+2MmOsSEBMjJTDLf9+s1r5y5z33mXDmTubk4c51za5p2kbDRrkTbOXeFf/EuM1sCpOC7SFHa6ZkPSkhPiDm1iyCry+HZm3y3oPvGUohL7vgARSQsOOduM7PP88n1MQ85555vY7O2brmaBJwJvOUvj+gDLDCzWc657jH14ykyM24+J48RWcl85x+rmfXAUu6fMy74F8SLSKc46Zk5nHNvO+cWOOdqvQioOzpcVcvrm/Yze2w2MVEnecidg5d+4Juo4/OPKMkWkdPmnHvWOXer/9FWkg0Bt2s1sxh8t2tdEPB65c65DOfcAOfcAHwz+4Z9kh1o+pAMFtwynX5pPfjqoyuZt6QQ57r1CX0R4RQSbTl5/1y3m9qGRq4+q1/bnZtb+wRseBbO/yn0m9DxwYlIWDCzI2ZW0cLjiJlVnGjbdt6uVdrQL70Hz35zKrPGZPPrxQV88/HVVNbUBzssEfGQrsroBE+vKmFkVjIjs0/ybPTBIlj4I+g/Habf6k1wIhIWnHOnNXNKW7drbdZ+3unsqzuLj4nkvmt8ddv/s3AzV8xT3bZId6Yz2h7bsreC9aXlJ38RZH2try47IgqufFD3uBYR6SbMjJvOzuPxGydRVlnDrAeWsmTL/rY3FJEuR4m2x5runT17bPbJbfjW/8Du1TDrft9FkCIi0q1MHRxQt/3YSh54c5vqtkW6GSXaHqpraOT5f+/mwhG96Hky984ufhuW3gfj58LI2d4FKCIiQdVUtz17TDa/eXWr6rZFuhkl2h56Z+sByipruOpkLoI8egie/zr0HAwz/693wYmISEiIj4nkt9eM5Y7LRvDa5n1cMe9dtpdVBTssEekAuhjSQ0+vKqFnQgznDcts3wbOwYLvQFUZXPckxOjiGBGRcNBUtz0yK5lb/vFvZj2wlDkTcxneJ4lhfZIY3CuR2ChdqyPS1SjR9sihqlre2LKPL08ZQHRkO784+OD/wZaX4OJfQNYYbwMUEZGQ46vbnsZPnl3Ho+/uoLahEYDICCMvI4FhfZL8yXcyw/skkZMWj3+SIBEJQUq0PbJgTSl1Da79dxvZvwUW/QcMugAmf8vb4EREJGTlpPXgbzdNpr6hkR0Hq9iy9wgFe4+wec8R1pZ8xEvr9nzcNzE2iqG9Ez9OvIf1SWJkdjLJcdFB/A1EpIkSbY88s7qEM/smMyKrHffOrquGZ2/0lYp87k8QodJ5EZFwFxUZweBeSQzulcTloz9pr6ypZ+s+X/K9ZU8FW/Ye4ZUNe/jH+zsBSImP5uXvTicnrUeQIheRJkq0PbB5TwUbSiu467Mj27fB63fBvg1w3dOQ1NvT2EREpGtLjI1ifG4a43PTPm5zzrH/SA3rS8r51t9Xc9/r2/jN1SpBFAk2T0+dmtlMMysws0Izu72F9b81szX+x1Yz+yhgXUPAugVextnRnvmghOhIY9bYvm133voqrPgjTPoGDL3Y++BERKTbMTN6J8dx0cjefGXqAJ5bXcLWfUeCHZZI2PMs0TazSGAecAkwEphjZsed4nXO/cA5N9Y5Nxb4PfBcwOpjTeucc7O8irOj1TU08sK/S7lweG/SE2JO3LlyP7z4Leh1Blz0fzonQBER6da+ee4gEmKi+M3igmCHIhL2vDyjPREodM4VO+dqgSeAE82+Mgf4h4fxdIq3Cg5wsKqWq/PbuAiysRGe/wbUHIGrHoHouM4JUEREurW0hBi+fm4er27ax+qdh4MdjkhY8zLR7gvsCnhe4m/7FDPrDwwE3gxojjOzVWa23Mw+18p2X/P3WXXgwIGOivu0PPPBLjISYzlnaBv3zl7xRyh6A2b8AnqN6JzgREQkLNwwbSAZibH88pUtmtZdJIhC5fYW1wLPOOcaAtr6O+fygeuA+8xsUPONnHMPOefynXP5mZntnBTGQwcra3hj836uGJd94ntn71kLr/0Mhl0G+Td2XoAiIhIWEmKj+M4Fg1mx/RDvbCsLdjgiYcvLRLsUCJx7PMff1pJraVY24pwr9f8sBt4CxnV8iB3rxTW7qW90J55yvboCnr0JEjJg1u9BEw2IiIgH5kzMJSctnl8t2kJjo85qiwSDl4n2SmCImQ00sxh8yfSn7h5iZsOBNOC9gLY0M4v1L2cA04BNHsbaIZ75oITROSkM65PUcoctL8MfJsPBQrjiT5DQs3MDFBGRsBETFcEPLx7Kxt0VvLx+T9sbiEiH8yzRds7VA7cAi4HNwFPOuY1mdreZBd5F5FrgCXd8EdkIYJWZrQWWAPc450I60d64u5xNeypangmyvBSe+CI8cR3EpcANiyDvvM4OUUREwsysMX0Z3ieJ/321gDr/dO4i0nk8nbDGObcQWNis7c5mz+9qYbtlwCgvY+toz3xQQkxkBJ8dnf1JY2MDvP9nePPnvuWL7oIpt0CkpsYVERHvRUYYt80Yxo2PreLpVSVcNyk32CGJhBXNDNkBausbeXHNbi4a2Yu0pntn714D//we7FkDgy6Ey/4X0gcGN1AREQk7FwzvRX7/NH73xlauGNeX+JjIYIckEjZC5a4jXdqSgv0cqqrl6rP6QU0lLPoP+PP5ULEbrpoP1z+rJFtERILCzPjJJcPZV1HDY+/tCHY4ImFFiXYHeOaDEjKTYjmn8X2YNwmWz4Pxc+GWlXDm53VnERERCaoJA9I5f1gmf1hSSPnRumCHIxI2lGifprLKGjZv2czjifcT+dQXITYJvvoqfPY+iE8NdngiIiIA3DZjOBXV9Tz4TlGwQxEJG0q0T0djA9sW/JpF0T9iSMVyuPBn8PV3IHdSsCMTERE5zsjsZGaPzWb+u9vZX1Ed7HBEwoIS7VO1dz3u4QuZsvXXbIsZScS3l8PZt0JUTLAjExERadGtnxlKfYPj/je3BTsUkbCgRPtUNNTDX6+g4fAuvlt7Cxsu+H+QnhfsqEREPGNmM82swMwKzez2FtZ/w8zWm9kaM1tqZiODEaecWP+eCcyZmMsT7+9iR1lVsMMR6faUaJ+KXcuh6gDP9fk+i2w6s8b0DXZEIiKeMbNIYB5wCTASmNNCIv1359wo59xY4FfAvZ0cprTTdy4YTHRkBPe+tjXYoYh0e0q0T0XBK7jIGH63I5fPnNGblB6agEZEurWJQKFzrtg5Vws8AcwO7OCcqwh4mgAEzvYrIaRXchxfnT6ABWt3s3F3ebDDEenWlGifioJXOJgxgdJjUS1PuS4i0r30BXYFPC/xtx3HzL5tZkX4zmh/t6UXMrOvmdkqM1t14MABT4KVtn3tnEGkxEfzm8UFwQ5FpFtTon2yyrbBoSIW14+nd3Is5wzJDHZEIiIhwTk3zzk3CPgJcEcrfR5yzuU75/IzMzV+BktKfDTfPG8QSwoOsKL4YLDDEem2lGifrIKFAPx571AuH51NZIQmoxGRbq8U6BfwPMff1pongM95GpGctrlTBtA7OZZfLS7AOVX6iHhBifbJKlhEZdoIdjT0ZPqQjGBHIyLSGVYCQ8xsoJnFANcCCwI7mNmQgKeXAbp/XIiLj4nkexcO5YMPD/PG5v3BDkekW1KifTKOHoJdy1nXYwpREcaEAenBjkhExHPOuXrgFmAxsBl4yjm30czuNrNZ/m63mNlGM1sD3ArMDVK4chKuzs9hYEYCv15cQEOjzmqLdLSoYAfQpWx7FVwjzx8dzZh+qSTG6vCJSHhwzi0EFjZruzNg+XudHpSctujICH548VBu+fu/eXFNKVeO1wX+Ih1JZ7RPRsFCGhN78/y+TKbk9Qx2NCIiIqft0jOzOLNvMve+tpWa+oZghyPSreiUbHvV10Dhm+zNuZT6MmPqICXaIiLS9UVEGLfNGM7c+e/z5UfeJyU+mvpGR11DI3UNjdQ3NC076ht9P5va6xsbqa1vJDE2inuvGctknYQSOY6nibaZzQR+B0QCDzvn7mm2/rfA+f6nPYBezrlU/7q5fHJ7qP92zj3mZaxt2rEUao/wr8h8YqIiGN8/LajhiIiIdJRzhmRw1Vk5rN55mPJjdURFGlEREcRERhATFUGP2Chi/G1RkUZMpO9nVKSvzztbD3DTY6v4+82TGJ2TGuxfRyRkeJZoB0zZ+xl8kxusNLMFzrlNTX2ccz8I6P8dYJx/OR34GZCPb3axD/zbHvYq3jYVvAJR8TxRlsdZuYnERUcGLRQREZGOZGb85uoxp7z93vJqrn5wGV+e/z5PfX0KQ3sndWB0Il2XlzXabU7Z28wc4B/+5RnAa865Q/7k+jVgpoexnphzsHURdQPOZc3eGqaobERERORjfVLi+NuNk4mJjOD6h1ew8+DRYIckEhK8TLTbNWUvgJn1BwYCb57Mtp02le++DVC+i4KU6TiH6rNFRESaye3Zg8dvmkRtQyNffGQ5e8urgx2SSNCFyl1HrgWecc6d1OXOnTaVb8EiAF6pGU2PmEjVn4mIiLRgaO8kHrthIoer6rj+kRUcqqoNdkgiQeVlon0yU/ZeyydlIye7rfcKFkLffF7dCfkD0omJCpX/n4iIiISWMf1SeXhuPrsOHWXu/PepqK4LdkgiQeNlxtjmlL0AZjYcSAPeC2heDFxsZmlmlgZc7G/rfEf2wu7VVA64iG37K1U2IiIi0obJeT354/Xj2byngpseXcWxWt2fW8KTZ4l2O6fsBV8C/oRzzgVsewj4Ob5kfSVwt7+t8231lY2sjJ0CqD5bRESkPS4Y3pvfXjOWlR8e4huPf0BtfWOwQxLpdJ7eR7utKXv9z+9qZdv5wHzPgmuvgkWQksurB9JIiqvhjOyUYEckIiLSJXx2TDaVNfX89Ln1/ODJNdw/ZxyRERbssEQ6jWaGPJHao1C8BMbPZdnGQ0wa2FMDhIiIyEmYMzGXyup6frFwM4mxUdzz+VGY6d9SCQ9KtE9k+9tQX01Z3wv48J2jzJ0yINgRiYiIdDk3n5PHkeo67n+zkMS4KO64bISSbQkLSrRPpGAhxCTxdu1QYDNTB6s+W0RE5FT84DNDqaiu55Gl20mKi+L7Fw0NdkginlOi3ZrGRl999uALeXd7BT0TYhjaS1PKioiInAoz487LR1JZU899r28jKS6aG6cPDHZYIp5Sot2a3f+Gqv24YZfw3sKDTM7rSYTqs0VERE5ZRIRxz5WjqKqp5+cvbSIpNoovTOjX9oYiXZRmXmlNwUKwCHalT2NPeTVTdFs/ERGR0xYVGcF9147l7CEZ3P7cOp5cuZPGRtf2hiJdkBLt1mxdBLlTWLrb9+HX/bNFREQ6RmxUJA9+6SzyB6Tzk2fXM+O+d3hxTSkNSrilm1Gi3ZKPdsK+DTDsEpYVldE7OZaBGQnBjkpERKTb6BETxT9unszvrh2LGXzviTVcdO/bPLVqF3UNmtxGugcl2i0p8M0G6YbOZHnxQaYOytBtiERERDpYZIQxe2xfFn3vHP50/Xh6xETy42fWcd6v3+Lx5R9SU6+p26VrU6LdkoKF0HMI2xr6UFZZq/psERERD0VEGDPPzOKl70xn/lfyyUyK5Y4XNnDOr5Ywf+l2jtUq4ZauSYl2c9UVsGMpDJvJssIyQPXZIiIincHMuGB4b57/1lQev3ESA3omcPdLmzj7V2/yp7eLqKypD3aIIidFiXZzRW9AYx0Mu5RlRQfplx5PTlqPYEclIiISNsyM6UMyePLrU3jq61MYkZXMPa9sYfov3+T+N7ZRfqwu2CGKtIsS7eYKFkF8Gg19J7Bi+yGm5mUEOyIRkaAys5lmVmBmhWZ2ewvrbzWzTWa2zszeMLP+wYhTuqeJA9P5642TeOHb08jvn8a9r21l+j1v8pvFBZQcPhrs8EROSBPWBGqoh22LYcgMNu87SvmxOk27LiJhzcwigXnAZ4ASYKWZLXDObQro9m8g3zl31My+CfwKuKbzo5XubGy/VB6eO4GNu8uZt6SQeW8V8sCSQvL7pzF7bDaXjsqiZ2JssMMUOY4S7UC7VsCxw7767CJfffaUPCXaIhLWJgKFzrliADN7ApgNfJxoO+eWBPRfDlzfqRFKWDkjO4U/fPEsdh06yoK1u3lxTSn/9eJG7vrnJs4eksHssdl8ZmQfEmM7NsVpbHTsLj9Gn+Q4oiJVECDto0Q70NZXICIaBl3Isve3MCgzgV7JccGOSkQkmPoCuwKelwCTTtD/RuAVTyMSAfql9+Db5w/m2+cPZsveCl5cs5sFa3bzgyfXEhe9ngtH9Gb2mGzOHZZJbFTkSb12Q6Nje1kVG0rLWe9/bNpdQWVNPX1T4/nK1AFcM7EfyXHRHv120l14mmib2Uzgd0Ak8LBz7p4W+nwBuAtwwFrn3HX+9gZgvb/bTufcLC9jBaDgFRh4NnXRiazcfogrx+d4vksRke7CzK4H8oFzT9Dna8DXAHJzczspMunuhvdJZvjMZH48Yxirdx7mxTW7eXndHl5et4fkuCguOTOL2WOzmZTXk8iI4+fFaGh0FB+o/Dih3uBPqqv8txSMjYpgRFYyV4zrS15mAos27OUXCzdz3+tbuWZCLjdMG0C/dN00QVrmWaLdnro+MxsC/BSY5pw7bGa9Al7imHNurFfxfUrZNjhYCBO/zrqScqpqG3RbPxERKAX6BTzP8bcdx8wuAv4TONc5V9PaiznnHgIeAsjPz9d829KhzIyz+qdzVv907rx8JO8WHeTFNaW8tG43T67aRa+kWC4fnc3wPkls2lPx8ZnqY3W+pDouOoKRWclcdVYOZ/ZNYVROCoMzE48rFblh2kDWl5TzyNJi/vLeDh5dtp1LzszixrMHMj43LUi/uYQqL89ot1nXB9wMzHPOHQZwzu33MJ4TK/B/0zlsJu+t9tVnT1J9tojISmCImQ3El2BfC1wX2MHMxgEPAjODOo6LBIiKjODcoZmcOzST6isaeHPLfl5cU8rjyz+ktqGR+OhIzshO5poJ/RjlT6rzMhLaVX89KieF+64dx08uGc6jy3bw9xU7eXn9HsbnpnLT2XnMOKPPp86cS3jyMtFuT13fUAAzexdfecldzrlF/nVxZrYKqAfucc694GGssHUR9D4TUnNZVrScEVnJpCfEeLpLEZFQ55yrN7NbgMX4xun5zrmNZnY3sMo5twD4NZAIPG1m0FnlfiLtFBcdyaWjsrh0VBblx+ooq6xhQM+E006Gs1Li+eklI/juBUN4etUu5r+7g2/9bTX90uO5YepAvjChX4dflCldS7Df/ShgCHAevq8j3zGzUc65j4D+zrlSM8sD3jSz9c65osCNO6zW7+gh2PkenP1Dqusa+ODDw1w/WbeBFREBcM4tBBY2a7szYPmiTg9K5BSlxEeTEt+xFzEmxEbxlWkD+dKUAby2aR+PLC3m7pc28dvXt3LdxFzmTh1Admp8h+5TugYvE+321PWVACucc3XAdjPbii/xXumcKwVwzhWb2VvAOOC4RLvDav22vQauEYZewr93fkRNfaPqs0VEROSkREYYM8/sw8wz+7Bm10c8/K9iHl66nYeXbufSUVncMG0A4/ql4v/mR8KAlzeC/Liuz8xi8NX1LWjW5wV8Z7Mxswx8pSTFZpZmZrEB7dM4vra7YxUshMTekD2O94rKiDCYMDDds92JiIhI9za2XyoPXDeet287jxumDuCtLfu58g/L+Ny8d3ludQk19Q3BDlE6gWeJtnOuHmiq69sMPNVU12dmTbV7i4GDZrYJWALc5pw7CIwAVpnZWn/7Pc1mIes49bVQ+AYMnQERESwrOsionFTdG1NEREROW05aD+64fCTL/+NCfj77DCpr6rn1qbVMu+dN7n21gH0V1cEOUTzkaY12O+r6HHCr/xHYZxkwysvYPvbhUqg9AsMu5WhtPWt2fcTN5+R1yq5FREQkPCTERvGlKQO4fnJ/lhaW8ei7O/j9kkL+8FYRl47KYu7UAYzPVVlJdxPsiyGDr+AViIqDgeeycsdh6hudpl0XERERT5gZZw/J5OwhmXx4sIq/vPchT63cxYK1uxmdk8LcKQO4fEzWSc9mKaHJyxrt0OccFCyCvPMhpgfLisqIjjTyB+iG8yIiIuKt/j0T+K+mspLPnUlVTT0/fFplJd1JeJ/R3rcRynfCOT8CYHnRQcb1S6NHTHgfFhEREek8CbFRfGlyf66flMvSwjIeW/ZJWcklo7KYnJdOclw0SXFRJMdHkxwXTbJ/OTYqQuUmISy8M8qt/tkgh86g/Fgd60vL+c4FQ4Ibk4iIiISl5mUlf33vQ55ctYt/rt3d6jYxkREBCXgUSXHRJMdHkRwXTUZiLOcP76Xa7yAK70R7zBxIGwhJfXh/0z4aHUzR/bNFREQkyPr3TOCOy0fy45nDOXy0liPVdZQfq6eiuo6KY3UcqW5arudIdR0V1fX+9jr2VlRTcayOQ1W1PLCkkL6p8Vw6qg+Xjc5mTE6Kku5OFN6JdkoOjLoKgGVFZcRGRTAuNzXIQYmIiIj4xERF0Ds5jt7JcSe9bUV1Ha9v2sfL6/bw6LId/Plf28lJi+ey0VlcPiqbM/smK+n2WHgn2gHeKzrIhAHpuspXREREuoXkuGiuHJ/DleNzKD9Wx6sb9/Ly+j088q/tPPh2MbnpPXxJ9+gsRmYp6faCEm3gYGUNW/Ye4bYZ2cEORURERKTDpcRHc3V+P67O78dHR2tZvHEvL63bw0PvFPPHt4oYmJHAZaOyuGx0FsP7JCnp7iBKtIHlxYcA1WeLiIhI95faI4ZrJuRyzYRcDlXVsmjDXl5ev5s/vFXIA0sKyctMYPrgDPIyEsjLTGRgRgJ9U+OJiFDyfbKUaOOrz06MjWJ035RghyIiIiLSadITYrhuUi7XTcqlrLLGl3Sv28OzH5RQVdvwcb+YqAgG9kxgYEYCeZlNPxPJy0ggLSEmiL9BaFOiDbxXfJCJA9OJigzv+XtEREQkfGUkxnL95P5cP7k/zjkOHKmhuKyK7WVVFB+oZHtZFVv3H+H1zfuob3Qfb5faI5q8jAQGZiSSl5lATlo8fVPjyU6Np3dyHJFhfCY87BPtveXVFB+oYs6E3GCHIiIiIhISzIxeyXH0So5jct7xpbV1DY2UHD7G9rJKig9U+ZLxA1W8W1jGs6tLjusbFWH0SYmjb6ov+e4bkIQ3LcdFd98bUYR9ov1ecRmg+mwRERGR9oiOjGBghq985ILhx687WlvP7o+OUXL4GLs/qqb0o6OUHj5G6UfHWLH9EHvWHCPgZDgAGYkxZKfGk50ST1ZqHFkpcfRJiff99N/aMCaqa1YdhH2ivazwICnx0YzMSg52KCIiIiJdWo+YKAb3SmJwr6QW19c3NLK3oprSw8fYXX7s4yS89KNqCg9U8q9tB46rDW+SkRjrT8Djjv+ZHE92ahzZqfFEh2AJcNgn2u8VH2RKXk9dSSsiIiLisajICHLSepCT1qPVPkeq69hbXs2e8upPflYcY095NTsPHmVF8UEqquuP2yY60ujfM4FBmQkMykz0PXr5asaT46K9/rVaFdaJ9q5DRyk5fIybz84LdigiIiIiAiTFRZMUF82Q3i2fFQeoqqlnb4UvES/96Bjby6oo2l9J4f5K3ti8/7iLNXsnx36SfGcmMKiXbzkrJc7z+4WHdaK9rMhXnz1V9dkiIiIiXUZCbNTHyXNzdQ2N7Dx0lKL9lRQdqKLoQCVFByp5YU0pRwLOhPeIiWRQZiLPfWuqZ2UnnibaZjYT+B0QCTzsnLunhT5fAO4CHLDWOXedv30ucIe/23875x7r6PhSe8Rw8cjeDO716TdJRERERLqe6MiIFpNw5xwHKmso2v9J8n24qtbT2m7PEm0ziwTmAZ8BSoCVZrbAObcpoM8Q4KfANOfcYTPr5W9PB34G5ONLwD/wb3u4I2OccUYfZpzRpyNfUkRERERCkJnRKymOXklxnXa3OS8vz5wIFDrnip1ztcATwOxmfW4G5jUl0M65/f72GcBrzrlD/nWvATM9jFVEREREpEN5mWj3BXYFPC/xtwUaCgw1s3fNbLm/1KS922JmXzOzVWa26sCBAx0YuoiINDGzmWZWYGaFZnZ7C+vPMbPVZlZvZlcFI0YRkVAU7BsORgFDgPOAOcCfzSy1vRs75x5yzuU75/IzMzM9ClFEJHwFlAFeAowE5pjZyGbddgJfAf7eudGJiIQ2LxPtUqBfwPMcf1ugEmCBc67OObcd2Iov8W7PtiIi4r02ywCdczucc+uAxmAEKCISqrxMtFcCQ8xsoJnFANcCC5r1eQHf2WzMLANfKUkxsBi42MzSzCwNuNjfJiIinatdpXztpZI/EQknniXazrl64BZ8CfJm4Cnn3EYzu9vMZvm7LQYOmtkmYAlwm3PuoHPuEPBzfMn6SuBuf5uIiHRhKvkTkXDi6X20nXMLgYXN2u4MWHbArf5H823nA/O9jE9ERNqkUj4RkVMU7IshRUQktLWnDFBERFpgvpPKXZ+ZHQA+PIVNM4CyDg7ndIVaTKEWD4ReTIqnbaEWU6jF0985F5K1FGZ2KXAfvll+5zvnfmFmdwOrnHMLzGwC8DyQBlQDe51zZ7TjdbvLuK142hZqMYVaPBB6MSmeE2vXmN1tEu1TZWarnHP5wY4jUKjFFGrxQOjFpHjaFmoxhVo80n6h9t4pnraFWkyhFg+EKkJHWAAAB9dJREFUXkyKp2OodERERERExANKtEVEREREPKBEGx4KdgAtCLWYQi0eCL2YFE/bQi2mUItH2i/U3jvF07ZQiynU4oHQi0nxdICwr9EWEREREfGCzmiLiIiIiHhAibaIiIiIiAfCJtE2s5lmVmBmhWZ2ewvrY83sSf/6FWY2wMNY+pnZEjPbZGYbzex7LfQ5z8zKzWyN/3FnS6/VwXHtMLP1/v2tamG9mdn9/mO0zszGexjLsIDffY2ZVZjZ95v18fwYmdl8M9tvZhsC2tLN7DUz2+b/mdbKtnP9fbaZ2VwP4/m1mW3xvyfPm1lqK9ue8P3t4JjuMrPSgPfm0la2PeHnsgPjeTIglh1mtqaVbT05RnLyQmnM9u8v5MbtUBqz/fsL+ritMfuUY9KY7RXnXLd/4JtkoQjIA2KAtcDIZn2+BfzJv3wt8KSH8WQB4/3LScDWFuI5D3ipk4/TDiDjBOsvBV4BDJgMrOjE928vvpvDd+oxAs4BxgMbAtp+BdzuX74d+GUL26UDxf6faf7lNI/iuRiI8i//sqV42vP+dnBMdwE/asf7esLPZUfF02z9/wJ3duYx0uOk38OQGrP9+wi5cTtUx+yA97DTx22N2acck8Zsjx7hckZ7IlDonCt2ztUCTwCzm/WZDTzmX34GuNDMzItgnHN7nHOr/ctHgM1AXy/21cFmA39xPsuBVDPL6oT9XggUOedOZQa50+Kcewc41Kw58G/lMeBzLWw6A3jNOXfIOXcYeA2Y6UU8zrlXnXP1/qfLgZzT3c/pxtRO7flcdmg8/s/0F4B/nO5+xFMhNWZDlx23gzVmQ5DGbY3ZpxZTO2nMPgXhkmj3BXYFPC/h0wPkx338H4ByoKfXgfm/7hwHrGhh9RQzW2tmr5hZm9MZdwAHvGpmH5jZ11pY357j6IVraf1D1tnHCKC3c26Pf3kv0LuFPsE6Vl/FdwarJW29vx3tFv9Xo/Nb+ao2GMfobGCfc25bK+s7+xhJy0J2zIaQGrdDdcyG0Bq3NWa3j8ZsD4RLoh2SzCwReBb4vnOuotnq1fi+chsD/B54oRNCmu6cGw9cAnzbzM7phH2ekJnFALOAp1tYHYxjdBzn++4qJO6RaWb/CdQDf2ulS2e+v38EBgFjgT34vvoLBXM48ZmRkPsMSGgJsXE7JP9eQ3nc1pjdKo3ZHgmXRLsU6BfwPMff1mIfM4sCUoCDXgVkZtH4Buu/Oeeea77eOVfhnKv0Ly8Eos0sw6t4/Psp9f/cDzyP72uiQO05jh3tEmC1c25f8xXBOEZ++5q+fvX/3N9Cn049Vmb2FeBy4Iv+f0g+pR3vb4dxzu1zzjU45xqBP7eyr84+RlHAlcCTrfXpzGMkJxRyY7Z/PyE1bofomA2hN25rzG6DxmzvhEuivRIYYmYD/f/TvhZY0KzPAqDpKuOrgDdb++M/Xf6ao0eAzc65e1vp06ep3tDMJuJ7r7xM/BPMLKlpGd/FGhuadVsAfNl8JgPlAV/HeaXV/8129jEKEPi3Mhd4sYU+i4GLzSzN/xXcxf62DmdmM4EfA7Occ0db6dOe97cjYwqsA72ilX2153PZkS4CtjjnSlpa2dnHSE4opMZsCL1xO4THbAi9cVtjdtsxacz2SnuvmuzqD3xXX2/Fd8Xsf/rb7sb3hw4Qh+9rrkLgfSDPw1im4/vqah2wxv+4FPgG8A1/n1uAjfiu6l0OTPX4+OT597XWv9+mYxQYkwHz/MdwPZDvcUwJ+AbglIC2Tj1G+P6x2APU4atHuxFfHegbwDbgdSDd3zcfeDhg26/6/54KgRs8jKcQX91c099S050YsoGFJ3p/PYzpr/6/kXX4BuKs5jH5n3/qc+lFPP72R5v+dgL6dsox0uOU3seQGbP9+wupcbu1v1eCOGb79xnUcbuV8Uhjdtsxacz26KEp2EVEREREPBAupSMiIiIiIp1KibaIiIiIiAeUaIuIiIiIeECJtoiIiIiIB5Roi4iIiIh4QIm2SAcxs/PM7KVgxyEiIm3TmC2dQYm2iIiIiIgHlGhL2DGz683sfTNbY2YPmlmkmVWa2W/NbKOZvWFmmf6+Y81suZmtM7Pn/TOGYWaDzex1M1trZqvNbJD/5RPN7Bkz22Jmf2uaAU1ERE6NxmzpypRoS1gxsxHANcA059xYoAH4Ir7ZzFY5584A3gZ+5t/kL8BPnHOj8c2a1dT+N2Cec24MMBXfrFYA44DvAyPxzVo1zfNfSkSkm9KYLV1dVLADEOlkFwJnASv9Jy7igf1AI/Ckv8/jwHNmlgKkOufe9rc/BjxtZklAX+fc8wDOuWoA/+u975wr8T9fAwwAlnr/a4mIdEsas6VLU6It4caAx5xzPz2u0ey/mvVzp/j6NQHLDegzJiJyOjRmS5em0hEJN28AV5lZLwAzSzez/vg+C1f5+1wHLHXOlQOHzexsf/uXgLedc0eAEjP7nP81Ys2sR6f+FiIi4UFjtnRp+p+bhBXn3CYzuwN41cwigDrg20AVMNG/bj++mkCAucCf/INyMXCDv/1LwINmdrf/Na7uxF9DRCQsaMyWrs6cO9VvW0S6DzOrdM4lBjsOERFpm8Zs6SpUOiIiIiIi4gGd0RYRERER8YDOaIuIiIiIeECJtoiIiIiIB5Roi4iIiIh4QIm2iIiIiIgHlGiLiIiIiHjg/wPNTDTixRQIJgAAAABJRU5ErkJggg==\n"
]
},
"metadata": {
"needs_background": "light"
}
}
],
"execution_count": 39,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"So, how well does our model perform?"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"train_metrics = model.evaluate_generator(train_gen)\n",
"test_metrics = model.evaluate_generator(test_gen)\n",
"\n",
"print(\"\\nTrain Set Metrics of the trained model:\")\n",
"for name, val in zip(model.metrics_names, train_metrics):\n",
" print(\"\\t{}: {:0.4f}\".format(name, val))\n",
"\n",
"print(\"\\nTest Set Metrics of the trained model:\")\n",
"for name, val in zip(model.metrics_names, test_metrics):\n",
" print(\"\\t{}: {:0.4f}\".format(name, val))"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\n",
"Train Set Metrics of the trained model:\n",
"\tloss: 0.0549\n",
"\tacc: 0.9978\n",
"\n",
"Test Set Metrics of the trained model:\n",
"\tloss: 0.6798\n",
"\tacc: 0.7925\n"
]
}
],
"execution_count": 40,
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"There is space for improvements but this article is in the first place a conceptual invitation not a way to accuracy paradise."
],
"metadata": {}
}
],
"metadata": {
"kernelspec": {
"name": "python3",
"language": "python",
"display_name": "Python 3"
},
"language_info": {
"name": "python",
"version": "3.7.2",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
},
"kernel_info": {
"name": "python3"
},
"nteract": {
"version": "0.15.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment