Skip to content

Instantly share code, notes, and snippets.

@ceceshao1

ceceshao1/keras_comet_example.ipynb

Last active December 6, 2018 13:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ceceshao1/8c320a3be4974826e9fcf2307a12d88b to your computer and use it in GitHub Desktop.
Save ceceshao1/8c320a3be4974826e9fcf2307a12d88b to your computer and use it in GitHub Desktop.
Download ZIP
Keras MNIST sample notebook with Comet.ml
Display the source blob
Display the rendered blob
Raw
keras_comet_example.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Keras MNIST example — Comet integration"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This Jupyter notebook demonstrates using the keras deep learning framework with [comet.ml](https://comet.ml). Adopted from the keras end-to-end example from the Comet.ml documentation: https://www.comet.ml/docs/python-sdk/keras/\n",
"\n",
"\n",
"![comet logo](https://comet.ml/images/logo_comet_light.png)\n",
"\n",
"In this example, we build a keras model, and train it on the MNIST dataset.\n",
"\n",
"keras is a framework built on top of lower level libraries, such as TensorFlow, or the Cognitive Toolkit. \n",
"\n",
"To find out more, you might find these links helpful:\n",
"\n",
"* https://keras.io/\n",
"* https://en.wikipedia.org/wiki/MNIST_database\n",
"* http://jupyter.org/\n",
"\n",
"Let's get started!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Imports\n",
"\n",
"First, we import the comet_ml library, followed by the keras library, and others if needed. The only requirement here is that **comet_ml be imported first**. If you forget, just restart the kernel, and import them in the proper order."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# import comet_ml in the top of your file\n",
"from comet_ml import Experiment"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using TensorFlow backend.\n"
]
}
],
"source": [
"import keras\n",
"import numpy as np\n",
"from keras.datasets import mnist\n",
"from keras.models import Sequential\n",
"from keras.layers import Dense, Dropout\n",
"from keras.callbacks import EarlyStopping\n",
"from keras.preprocessing import image\n",
"from keras.optimizers import RMSprop"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Dataset\n",
"\n",
"As a simple demo, we'll start with the the MNIST dataset. In keras, we use the `load_data` method to download and load the data:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# the data, shuffled and split between train and test sets\n",
"(x_train, y_train), (x_test, y_test) = mnist.load_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's see what we have here. \n",
"\n",
"* x_train are the training inputs\n",
"* y_train are the training targets\n",
"* x_test are the test/validation inputs\n",
"* y_test are the test/validation targets\n",
"\n",
"These are numpy tensors, so we can get the shape of each:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(60000, 28, 28)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x_train.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That is, there are 60,000 training inputs, each 28 x 28. These are pictures of numbers.\n",
"\n",
"To visualize the patterns, we write a little function."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def array_to_image(array, shape, scale):\n",
" img = image.array_to_img(array.reshape([int(s) for s in shape]))\n",
" x, y = img.size\n",
" img = img.resize((x * scale, y * scale))\n",
" return img"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We call it by providing a vector, a shape (rows, cols, color depth), and a scaling factor:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAIwAAACMCAAAAACLqx7iAAACe0lEQVR4nO3bS6iNURiH8Z9LBuTSmVBKYkCJGKCkJEmKgZhQJpghIxMzA1IYIAMjZSBTjCjXgTollwmZu8xc00EOJqt9nOzV+fb+9uBV7zM47b6z1urp39vXWu9emyRJkiRJkiRJkuS/YVKbyVNg9vhnh2A6LIGDcAZ2wzc4Bce7rDe5jcygSZkaoWSmNh24AKbBOlgPc2BnbcZrOA874As8hwe1GaGSSZkaoWSavIFXwR3+edtW+QX74Gvn2Vv4AK9q00IlkzI1Qsk0KeAhGIZFtSHD8BE2wg+aV/sYoZJJmRqhZJpsId7DUdgGTylbA/AMNlPetsvgSH8yoZJJmRqhZHo7xM2ibGYvwX7YC1cHIhMqmZSpEUqm8SEOfO58+tT5dACuUXa+bQiVTMrUCCXTbxttBtyEDbAVbreVCZVMytQIJdOqD7wYnlDOb/fgMVyE3z2vFyqZlKkRSqZVAaM0eS/DzM6zY3AF3vWyVKhkUqZGKJn2BQyWw1nY1Hl2CU7Am4arhEomZWqEkhlQAaN8Mbed8kKeBHcpTbYGhEomZWqEkhlkAY/xndLg+Alb4P7E00IlkzI1Qsn01karsgJ2werxq76Ahw1XCZVMytQIJdOqgJfAYcpJbt74/45SDnFNG8ShkkmZGqFk+thCzIM9lLuSC7sMeUw5v93oZeVQyaRMjVAyjQt4LuWiwwVY2mXIMJyG6/TzxVyoZFKmRiiZCQp4iNIPW0n322iPKB20WzDSQiZUMilTI5RM1wJeS7l+tgbmdxkyAufgJH9f+21DqGRSpkYoma6HuB2dP2O8pFx+GKX8SujjgGVCJZMyNULJJEmSJEmS9MEfGk5MiN8lROoAAAAASUVORK5CYII=\n",
"text/plain": [
"<PIL.Image.Image image mode=L size=140x140 at 0x115F14860>"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"array_to_image(x_train[0], (28, 28, 1), 5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Often, we need to do a little data preparation to get it ready for the learning model. Here, we flatten the inputs, and put input values in the range 0 - 1:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"# run this cell first to create unflatted copy of the x_test data split for plotting later \n",
"x_test_original = x_test.copy()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"60000 train samples\n",
"10000 test samples\n"
]
}
],
"source": [
"# preprocess and normalize the data \n",
"x_train = x_train.reshape(60000, 784)\n",
"x_test = x_test.reshape(10000, 784)\n",
"x_train = x_train.astype(\"float32\")\n",
"x_test = x_test.astype(\"float32\")\n",
"x_train /= 255\n",
"x_test /= 255\n",
"print(x_train.shape[0], \"train samples\")\n",
"print(x_test.shape[0], \"test samples\")"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"# set random seed for reproducibility \n",
"# see: https://stackoverflow.com/questions/21494489/what-does-numpy-random-seed0-do\n",
"np.random.seed(45)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And we examine the targets:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(60000,)"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_train.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that they are just 60,000 values. These are the integer representation of the picture. In this example, we wish to have 10 outputs representing, in a way, the probability of what the picture represents. \n",
"\n",
"To turn each number 0-9 into a 10-output vector for training, we use the `keras.utils.to_categorical` function to turn it into a so-called \"one hot\" representation:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"# convert class vectors to binary class matrices\n",
"y_train = keras.utils.to_categorical(y_train, 10)\n",
"y_test = keras.utils.to_categorical(y_test, 10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We then can check to see if the picture above is labeled correctly:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_train[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Indeed, the first pattern is a 5. We can also visualize this vector like so:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAMgAAAAUCAAAAADtzy0WAAAAPElEQVR4nO3PsREAMAjDQMz+OycLpOSICn1JgU+pBedxy/BGD//7xhAaQ2gMoTGExhAaQ2gMoTGExhCaC5mgASiEiqdjAAAAAElFTkSuQmCC\n",
"text/plain": [
"<PIL.Image.Image image mode=L size=200x20 at 0xB2BBFA978>"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"array_to_image(y_train[0], (1, 10, 1), 20)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see the \"one hot\" representation showing that y_train[0][5] is 1.0, and all of the rest are zeros."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Model Definition"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# starting parameter set\n",
"# these parameters can be adjusted with different values. Feel free to experiment\n",
"batch_size = 128\n",
"num_classes = 10\n",
"epochs = 20\n",
"num_nodes = 64\n",
"optimizer = 'adam'\n",
"activation = 'relu'"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"def build_model_graph(input_shape=(784,)):\n",
" model = Sequential()\n",
" model.add(Dense(num_nodes, activation='relu', input_shape=(784,)))\n",
" model.add(Dense(256, activation='relu'))\n",
" model.add(Dense(num_classes, activation='softmax'))\n",
" model.compile(loss='categorical_crossentropy',\n",
" optimizer=optimizer,\n",
" metrics=['accuracy'])\n",
" return model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And call it to create the model:"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"model = build_model_graph()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We use the summary method to check the details:"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"dense_4 (Dense) (None, 64) 50240 \n",
"_________________________________________________________________\n",
"dense_5 (Dense) (None, 256) 16640 \n",
"_________________________________________________________________\n",
"dense_6 (Dense) (None, 10) 2570 \n",
"=================================================================\n",
"Total params: 69,450\n",
"Trainable params: 69,450\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Comet Experiment\n",
"\n",
"In order for comet.ml to log your experiment and results, you need to create an Experiment instance. To do this, you'll need two items:\n",
"\n",
"* a Comet `api_key`\n",
"* a `project_name`\n",
"\n",
"You can find your Comet api_key when you log in to https://comet.ml and click on your project. You should see a screen that looks similar to:\n",
"\n",
"![comet_api_key](https://www.comet.ml/docs/img/register_3.png)\n",
"\n",
"Click on the API key to copy the key to your clipboard. \n",
"\n",
"It is recommended that you put your COMET_API_KEY in a `.env` key in the current directory. You can do that using the following code. Put it in a cell, replace the `...` with your key, and then delete the cell. That way your key stays private.\n",
"\n",
"```ipython\n",
"%%writefile .env\n",
"\n",
"COMET_API_KEY=...\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"experiment = Experiment(project_name=\"INSERT_YOUR_PROJECT_NAME\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you get the error that ends with:\n",
"\n",
"```python\n",
"ValueError: Comet.ml requires an API key. Please provide as the first argument to Experiment(api_key) or as an environment variable named COMET_API_KEY \n",
"```\n",
"\n",
"then that means that either you don't have an `.env` file in this directory, or the key is invalid.\n",
"\n",
"Otherwise, you should see the message:\n",
"\n",
"```\n",
"COMET INFO: Experiment is live on comet.ml https://www.comet.ml/...\n",
"```\n",
"\n",
"If you click the URL, then a new page will open up. But, even better, you can execute the following line to see the experiment in the current notebook:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <iframe\n",
" width=\"100%\"\n",
" height=\"800px\"\n",
" src=\"https://www.comet.ml/ceceshao1/jupyter-notebook/e39b6d7e32f24fd5915db13cb8585034\"\n",
" frameborder=\"0\"\n",
" allowfullscreen\n",
" ></iframe>\n",
" "
],
"text/plain": [
"<IPython.lib.display.IFrame at 0xb2bd9c4a8>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"experiment.display()"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [],
"source": [
"# Comet.ml has a method to log a hash of the dataset, so that we can see if it changes:\n",
"experiment.log_dataset_hash(x_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 5. Training\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"params={'batch_size':batch_size,\n",
" 'epochs':epochs,\n",
" 'layer1_type':'Dense',\n",
" 'layer1_num_nodes':num_nodes,\n",
" 'layer1_activation':activation,\n",
" 'optimizer':optimizer\n",
"}\n",
"\n",
"# log parameters in Comet.ml\n",
"experiment.log_parameters(params)"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Train on 60000 samples, validate on 10000 samples\n",
"Epoch 1/24\n",
"60000/60000 [==============================] - 2s 40us/step - loss: 0.0225 - acc: 0.9927 - val_loss: 0.0862 - val_acc: 0.9771\n",
"Epoch 2/24\n",
"60000/60000 [==============================] - 3s 44us/step - loss: 0.0219 - acc: 0.9925 - val_loss: 0.0942 - val_acc: 0.9765\n",
"Epoch 3/24\n",
"60000/60000 [==============================] - 2s 39us/step - loss: 0.0192 - acc: 0.9939 - val_loss: 0.0871 - val_acc: 0.9781\n",
"Epoch 4/24\n",
"60000/60000 [==============================] - 2s 39us/step - loss: 0.0153 - acc: 0.9949 - val_loss: 0.0969 - val_acc: 0.9778\n",
"Epoch 00004: early stopping\n",
"10000/10000 [==============================] - 0s 19us/step\n"
]
}
],
"source": [
"# Once you run this cell, you will be able to see the results live in the embedded Comet UI above \n",
"\n",
"# will log metrics with the prefix 'train_'\n",
"with experiment.train():\n",
" history = model.fit(x_train, y_train,\n",
" batch_size=batch_size,\n",
" epochs=epochs,\n",
" verbose=1,\n",
" validation_data=(x_test, y_test),\n",
" callbacks=[EarlyStopping(monitor='val_loss', min_delta=1e-4,patience=3, verbose=1, mode='auto')])\n",
"\n",
"#will log metrics with the prefix 'test_'\n",
"with experiment.test():\n",
" loss, accuracy = model.evaluate(x_test, y_test)\n",
" metrics = {\n",
" 'loss':loss,\n",
" 'accuracy':accuracy\n",
" }\n",
" experiment.log_metrics(metrics)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 6. Logging\n",
"\n",
"In keras, Comet will automatically log:\n",
"\n",
"* the model description\n",
"* the training loss\n",
"* the training accuracy\n",
"* the training validation loss\n",
"* the training validation accuracy\n",
"* the source code\n",
"\n",
"To log other items manually, you can use any of the following:\n",
"\n",
"* `experiment.log_html(HTML_STRING)`\n",
"* `experiment.html_log_url(URL_STRING)`\n",
"* `experiment.image(FILENAME)`\n",
"* `experiment.log_dataset_hash(DATASET)` (shown above)\n",
"* `experiment.log_other(KEY, VALUE)`\n",
"* `experiment.log_metric(NAME, VALUE)` (shown above, can support multiple metrics)\n",
"* `experiment.log_parameter(PARAMETER, VALUE)` (shown above, can support multiple parameters)\n",
"* `experiment.log_figure(NAME, FIGURE)` (shown below)\n",
"\n",
"For complete details, please see: \n",
"\n",
"https://www.comet.ml/docs/python-sdk/Experiment/#experiment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Example: Logging a figure**\n",
"Let's plot a sample of our model's predictions and log it as a figure in Comet! "
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA6AAAABYCAYAAADr7JpYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xd8Tff/wPHXzZTEHgliBBEhRghFqdWWFlWKqhrV0t3SQXfr26W6UTq0pb/qoK1aVVpVSluldkvsxBajZpBxc35/vO9JXElIIne/n4+HR27OPefm83HOPePz/nzeH4thGCillFJKKaWUUo7m5+oCKKWUUkoppZTyDfoAqpRSSimllFLKKRz+ABoSEnLIYrEYhf0XEhJyyNFlcwStr9bXm+rsa/UFPaa1vlpfT66zr9UX9JjW+mp9fbG+nlxnAIujx4BaLBajKH/DYrFgGIbFAUVyKK1vgbfzqfratvW4OvtafUGP6UJsp/X1AL72Hfa1+oIe04XYTuvrAbS+hdrWI+sM2gVXKaWUUkoppZST6AOoUkoppZRSSimncMkD6NatW4mPj8/+V7p0acaNG+eKojjF3r176dixIw0aNCAuLo7x48e7ukgOdddddxEeHk7Dhg1dXRSnWbhwIfXq1SM6OpqxY8e6ujhOYbVaadq0Kd27d3d1URzOF49pX9q/AOPHj6dhw4bExcV59fXIFBUVRaNGjYiPj6d58+auLo7D+dr+9cVr0okTJ+jTpw+xsbHUr1+fFStWuLpIDqXHtPfyiXsOwzAc+k/+RP4yMzONiIgIIzk52W65bTuHl88Z9T1w4ICxZs0awzAM49SpU0bdunWNTZs2eW19f/vtN2PNmjVGXFxcrve8sb6ZmZlG7dq1jZ07dxppaWlG48aNc+1fT63zpb6/b7/9ttG/f3+jW7dueb7vifXNr86+dkwbhm/t33/++ceIi4szUlNTjYyMDOPaa681tm/f7rX1NQzDqFmzpnHkyJE83/PkOhd1/3pqffOqszdfk/Kqr2nw4MHGxx9/bBiGYaSlpRnHjx/32vr62jmrIMe0N9W3IPccnlxnwzBc3wV38eLF1KlTh5o1a7q6KA5TpUoVmjVrBkCpUqWoX78++/fvd3GpHKddu3aUL1/e1cVwmlWrVhEdHU3t2rUJCgritttuY86cOa4ulkPt27eP+fPnM2zYMFcXxSl87Zj2tf2bmJhIy5YtCQ0NJSAggPbt2/P999+7uliqmPja/vXFa9LJkydZtmwZQ4cOBSAoKIiyZcu6uFSOo8e0dx/TvnDP4fIH0OnTp9O/f39XF8NpkpOTWbduHS1btnR1UVQx2b9/P9WrV8/+vVq1al7dwADwyCOP8MYbb+Dn5/JTiHIAX9u/DRs2ZPny5Rw7doyzZ8/y448/snfvXlcXy6EsFgudO3cmISGByZMnu7o4DuVr+9cXr0lJSUlUqlSJO++8k6ZNmzJs2DBSU1NdXSyH0WPa+49pb+fSu4v09HTmzp1L3759XVkMpzlz5gy9e/dm3LhxlC5d2tXFUapIfvjhB8LDw0lISHB1UZQD+OL+rV+/Pk8++SSdO3fmhhtuID4+Hn9/f1cXy6F+//131q5dy4IFC5g0aRLLli1zdZEcxhf3r6/JzMxk7dq13H///axbt46wsDCvHieox7TydC59AF2wYAHNmjUjIiLClcVwioyMDHr37s2AAQO45ZZbXF0cVYwiIyPtWh737dtHZGSkC0vkWH/88Qdz584lKiqK2267jV9//ZWBAwe6uliqmPjq/h06dChr1qxh2bJllCtXjpiYGFcXyaHMc1R4eDi9evVi1apVLi6RY/nS/vW1axJIRKxatWrZvcv69OnD2rVrXVwqx9Jj2ruPaa/nisG1pn79+hlTpkzxqoG1edU3KyvLGDRokDFixIh8/y+8qb6GYRhJSUk+k7AlIyPDqFWrlrFr167swfH//vuvV9T5Ut9fwzCMJUuW+ESSGsPwrWPa5Ev7NyUlxTAMw9i9e7dRr149r05gcubMGePUqVPZr1u3bm0sWLAg13qeWOei7l9PrW9edfbma1Je9TW1bdvW2LJli2EYhjF69Ghj5MiRXl1fXzpnFeSY9qb6Gsbl7zk8uc6GYRDgkKfaAkhNTWXRokV89NFH+a5zvV9fw4lFKpRFWd9aCrruH3/8wbRp07JT3gOMGTOGrl272q3nLfXt378/S5cu5ejRo1SrVo0XX3wxOzHAhbylvgEBAUycOJEuXbpgtVq56667iIuLy3Ndd61zYepbGO5aX9Bjujh4U3179+7NsWPHCAwMZNKkSXkmMPGW+qakpNCrVy9Aui7efvvt3HDDDXmu6651dsT+BfetLxS8zt5wTYLC7+P33nuPAQMGkJ6eTu3atZk6dWqudbypvr50ziroMe0t9S3oPQe4b50vV1+XPYCGhYVx7NgxV/15p2rbtq3ZyuETvv76a1cXwem6du2aq0HBF3To0IEOHTq4uhgO54vHNPjO/gVYvny5q4vgNLVr12bDhg2uLoZT+dL+Bd+8JsXHx7N69WpXF8Np9Jj2Xr5wz+EbKQ6VUkoppZRSSrmcPoAqpZRSSimllHIKl3XBVQWX/EprAKwlpBtvpbgjAKxoMjN7nTq/3glAqVUhAERM+NOZRVRKKaWU8kh+oaEAJPx5GoDRldYD0HnzLQRdv9tl5VLKW2kEVCmllFJKKaWUU2gE1I0dn18XgH/jJ+b5fsYFeY22dPwEgC+bVwHgm0XtAbAmbndgCV3PkiBZ0ObPnQZAow8fAqD6y54bAfYvWwaArRNrZ+/X5w4nAPDPAJnny7p5m2sKp5RSRRBQWeb7Tq9bNdd7gdv2A7D16doAlN0syRPLJ54HwG/5OmcUUeXh/E1XARCyQObUNJo3IKlHGADXdPoHgOW/NrLbpsoKKwAl5rn/3LJm5HPb5HoAzK40GYAs2/t7N1ShDhoBVd5hx7ut2NnvQwAG724HQErrUy4pi8MfQEuUKJFisVgiCrtd6eAykO6IEjmW1rdgfK2+4Jl19rX6gh7TBaX19Qy+9h32tfqCHtMFpfX1DFrfgvPUOoMTHkDPnTtX2XxdqLlqPPQ/tLjqe3x+Xf6In57nqh+ekFbid1ZcT1RNGQ/6c4PvARhQ6iAArw6pCEDtJx0bAXX1/j3cojQAmUiLa+gBx053U+T6QoHrnFWrGgD/dPgoO8r9SvgaAJr0uhqA6k6KgF5YXyi+fWzt2AyAhyZ/A8AHdaMLXbbT/VpRdv1R+bytOwq9fX5cfUyfGCxjvleO/QCABpMeAKDG66swMjOL549coDjrG1CzOgDhM04A8NuaBgDEvi+/WzdtLVIZ/StVAuDYjXKclJthi8akpRX6s1y9f53NGeesvJwc2AqAY10livlU04UADC79Y651Pz1ZA4BbSs0CoFzfEnbvd49MKPDfddQ5y50V5zHtX7ECANYZkk9iet13AEixBgJQxm8pNQJC7Te6Y5ndr4cHngXgwIQg7h0zAoAKH68ocLEupzjru+vZJgBs7jgBgAG7bgTg2Ku1AKiz8K8il7O46DmrgApRX7NHxsk2Uey/Xv5EUg+JfmcYcj/ZZv1tABzZWw6ABmMPkZm8p+B/pICceY5u02pz9uvPa8r39ppe9wIQOmtl4T7sCukYUKWUUkoppZRSTqFjQN1M5rXS0vtrk0mAtDiOOy7j/pb0ay4rHTgMQMzx1fiVkJbiMStlDMYzFWVMRma54o+UuKPjjaWlal+mREIqfFp8razOFlBdIp+1JhdfNM9d7e4SDEB5/zNF/oxD3dLJGCRtaOW7F0uxXCogUsbGvfzCJ3bLNz/4PgA3TrgG4/Rpp5eroAIqR/DSUsnMXS9QRlB1OiYNu9ZNReuJYUY+B/wuEc9WJSRC9uA/0mLLuk1FLq8jmNGjre9KRK9D3e3sb58BFC1a6yn8mtQHYMvDYSzvPA6ASv5/y3sFaOceWsaMKpS45HrK8baNl2N3a+yntiUS7Qz3l9/ePxHD2tOyzr7Usnbb+lvkez+/3rzsbWY89yYA9yVKfga/39c7rOxFkR5uf6+0cbnk3qi10HPvJVRulmC559j1ovS+mthHrrPtQ85mr5NhyLkqyzYCeHn8V/JGvO1Hhbuo0dcZpXUcM+p5oQPtZMx99CznlkUjoEoppZRSSimlnMJtI6DH7pZxUDUGSTRoy+EI0tMkIhj5tfwM3SfRk6z1m/P4BM90JjIIkFZjM/K5tIdEN627co+f2vFiUwC+Kv+2bYm08lRb6N1tC0YbaZJa3l3Gp7Rf9jAA0XhetsQ9L8i4zoQb5Dh+o8ryfNctebWM+d37vGxTcaO03obMcf9sgwCWQDm+O3W68lbwUutKcOvQ3wBYUlaix9YTJ6/4c13lcJeaAHQOzbBb3mx1PwAqnXHPzMcB1SIBKDPjLI2DJExS75f7AKh7x9or+uzEV6IAuLWkjB9sNu4JAKquc68s14cfku/j6BGfA9At9Ofs93pWvAmAzP0HnF8wJ0mtVQqAbTd+AIQUeDszn8GXu1tccr0yuG+vEL/4BpyvLFlhk3tKJKHPVRL9zTDk+7BkmmSSrfLbSQw3i9pfyGjdhBlXf2T7TW4PF56TCOjYUXcAUGrTUTjyHwB+x/fab+8n9Y15W8atb771PeoElgTg3HOSabPMEBl7l3koxUG1KJzAkjKI7nSW/KyxyHt7KlyKtYNEBgNeSGFevbkABFpkf148JrLCs3IPbknez7GbZJx/+dn/ApDlpr109oyyzSQwaHy+69y5+1oAPq25KM/31189hR5c+lzliaIfdc04Z+9+SlFKKaWUUkop5TbcNgL6xCjpe9077LgsqHPBmx3kR3Km9N0ef6Rjkf7GqsMScQh7W+ZdDFi8pkifU5zKfi7jDvqsHojluLQYZh5Mznf9YV1/AaCkX7Cji+ZW/msgrexV/KV1NvK7QFcW54psvPc9IKeV8VKWNvlSXkjiPmalyryvU073JOBX1x+/l3O6l7SyToiUOtefLeOC6lL47Gtp5QyGl9sCwNJSMgYND4yAmvPQdRn+e57vB0+XDHwYjs3wXFTH20jm29lRk7KX1X9OxqlfyUh0o3UTdnSXaEz7f2TgTfUpsr8v/01xDv8YuTB98riMe4wPkktq1gXrHPxAooNV7pXxsJkHDzmvgMUooFokiU9KT4OIPyXSV/praTn3S5Njc1tGOnszZVxg9QDJfDzkX4mcHU+U8bERf8u6Zf/ci3FGejGVOeG+Ec6Lmb1vdj0ov3/V+mMSbJH/fI2SHirnRqYz+YREjN7fIHN11x2aCEDW+fMOKG3hZJQJuuAYlv00aupdAFSfJb0OLvndy5J3zYhK/aCH2HizRJx+a/QdAG2uk+homS9cHwH1j67FpnZTABhxQKJf/kuurNeGpzDHRJ7uIcfz6Nfk/6F9yNns85eZhf/iMZHNnh8CQJPKfsyJknnqW5SVXmgR77lX7xSjtdwsTbnrvXzXaTx1OAC1XpZ9H/uufLm33Dwp323UlXPbB9AJz0io/4XGEqQtl2hwvL5c9IIay4XtjYYy9ci7VeTmdf7ZknQLzTupyTlDulesTJOuMh1KZIBtu+h+ktAiZnGxV6PIrAWYZiP51dYMLfuW7TdJ3vD4QUl9X+oXuai5y41acbv2AXlQn21LglByqXRP9qT6Bi6Vh0ezm8ulrEuXC0ByhiRl6RUmXaBuLSk3+rdOm1yoqQpcwWgTz6TX5Wbki1PS+BP7nBznRdlvrTv/W1xFc6m0q+Xh+ZXwT+2Wn7V1CSv9leunAciLOeXKkZtzbpybvyU3IZX3Fv0mxLxheO7L/8tedma+PLyFHdtV5M91hMSnpHGg8SUeQFYmyE3bthWyP2+Z9hgAtV+V4QLu8OBxKf5lpYH2qvlJzK4oXfParH7Ibp3gBdLldFS3IdlT7fjXl2Qu5bfulJ9Z9tc0T0mTl9VWbtCT5bmJ+W3kprROgNnV2J9F5+T1M5t7AnBij1yX/u0pN73Pp8h1+Y3Kq2kSshuAd66aAcDTjw4BoNprrr9xt5awZL9u/OcQAGq8WvRy1X1wJT9cJ9e5viWPAXCiRyoAZb4o8scWm63/K3v5lQog7Ubplnm6es4tdaU1EkAw1rhnl+u0DjK069dxE+2WLzlXkhdekUaHwLP2jZ6nasr9eJAtb88TI6dwMku+ySUPut/dl9G6CcYrcq+UYIvRmA/Xs86EAzBlSA+iVkoDkWFrQKn36AYAbpx9PwAvfyjTszQPtnLdv9LF+JeGpRxefkeoM+M+dvb70G7Zjnfl/OTsrrjaBVcppZRSSimllFO4bQQ07LuVtp85y0pftM57lTsA8EqbKHn/tx280SHvCe0Dzkm7R9jGgwBUWDaTRkG2ZEbJntV988QgSdD0x+C3KOMnkc8VadICv/4VSUoUcsozktIUln9cPQDGhH8NwKenPDP5zLmeV3FnlW+BnK63eXXBbbhYkrlUWizNd8EnZZ2nO0jb0T99J2Svu+9pSYTiDi3peTn+9FmqBUhr6WMPdwMg8Hjhuw0HVJFo2NQaC7PTpnuypFvyjp712d7T9so9k9fsHS/JRbZf9RkAzx2OJ3KqtPZfSVv4/g7SS6VNcBYN/5TumzXcrFsXgH+DGH65dpztN4mAvX5MotmrT8g0FTPqLMxeP8aWgOvjAR/IulNuBiArabczilto5hRfad9JBPSZir9S73sJA8bOyns/m9FPAGti0abecSe7vorny9YfA1zQzVb2df+k6wH4e0stYkdIj6NKqVL/SrY170u4DoDDw6XHx6Mf+PNcxFIAlp+TyOD6hyRK2vMLOR4y9+5zTGUKoN7TOdE6/zXFE+F59m85j/XtKD08HoyTaSB+oFyxfP6VeLfljOzXf3wlw0Mqc/lzzc4v5T5rfEu5D2kUJMMnIvxzhkLtyJBr3c3fPQpAnZHu0ZPF7GHy2gcf2S3vv7MrAKdGV6fckrynoCkTXQuA+G+lV0P9ID9i50j9Yr4r/DAaRzvcIoy/Y6VrsdnT7KStZ9Hob6SXZdSK3HU1p8wK/Hk1AAN/kvuwTTdNZFR5qfvHX8u1qVb/DY4qvkNcHP10Jc+/e1NKKaWUUkop5RHcNgJaEGYa77CZ8tMKhH137JLbpAyT6GFcUABv/SfRtKipMq7IU8akHG0m/fLN6CfAHUuHARAz2zsjn6b911ew+33N6Zq2V+ecX5giMCO4r7wzmeZB6eZSu3XMxELPLelN/SdsSVdOnbJbp952maJnVQ85Bq4KPs+C+98AoHMJmaoiaoxEF83WPFcxp1T6ttGbfH6yMQCBvxQ9YdLml2TsYYZh5Y5kiTBYDx+5wlK6TrcW9i2oJ7PkWM74n0xX4OemEVDDkPFiZuR+5bEo/M8dLvTn+JWSSMvWVyU5y+weMrVSFoHU6PtPcRTVIY5eVYGoAEkgdc/edgDsayU5CPzCZJBUwn0PM/LubwAYUEr+b9rZTtvzZu4BYHM390pO5F9OIlNbXpZzzNb67wOwJg1iX5Jr5cXnI2/hFybR9+0vyfi4xPaT8LOdn/+2JVoaMEcSlNR7UaKeMSdW2yWdulCjUvsBWBQgkaPVbyZQ4R2JFPUMO2Fby5LXpk7l1zgWgA5lF7EtQ8YkV9yYcalNCqzcb7YDvmi5Ih3Cv7T0pwvzS+Pnc7LPK7+bd+TTEhhEeke5bj37wVQA2pWQ65cZVVuVJpHPwVv68lgtmYKph+0c8H5PifyOm9ILKFh+D0c6/qxcX8wxkV233AKA/0j5P/Ffl38SphMJck0aHf5N9rLqP+e3tuv5XXcsO4GSmVDpzl09AIh6Pu8ob15i7pf76vfaxvFYebknG9BAxr3/SVBxFdfnaARUKaWUUkoppZRTeHQEtDDMjI0Tn5GMX4EWf74dL9GTCgcL3hLiSumLJNq3IvZt25ISNFkh/dDrPy790t0vD1nxOtXAvlV2/UTJUFgWz9iHWbYU9znRzxx37b4BgNP9ZIxRzL5V+e5PsxX1gc9kbMLqe8dRxV+2WztUxqX1/l6ODWNDYvEUvoj8eh4FoGpAMJ9+JXWsVoBxNhczo8dfXCtjV9KMDPa8I1GasDT3G39SEGldWzAx8mO7ZftsXTH8flvnghIV3Y+xsxm6VMIce05LFD/908r5rn/oGmmS7tpyPQBzq75ve0fG5LdZfxvlcN9xhNbgnKkqNn4kEbPytvNQVqpk+qzy9p98c5NkyOxf6gfZ0JAW+ZQ0ifwa591r4vsDA2Uc69ZeMjZxbqpERD/tfj3WIztdVi5nONHDlhm0r2SX9yOUxeckVDT2ATmfRv8sY/nyOjdbAuT87lfPNj3P7PIAvPm5ZHRuFHQYkKi5v0Xa/xutvB2AyMOu+7/dfodkg72t5BHabhwEQOkf/3ZZeRwt6ZGGALQtsZgGSwYDEI39+dbfNt5x64MRbL7VfgqPxedk/PsDPw0BIHa8XOOCt+1kEnJNem+x3HP+ECuzNbxWQ8ZSB20u1qoUStL0xmxqKlHcfZkSCfV7Vr7fxrqN+W5nTtkS/YgU3s8Wu7pz97WEuGGvu4DIqgA8Xu+XXO/t+layc0dQ+F5TU+Zcx2N3brmywqlsGgFVSimllFJKKeUUPhMB3fJoJAAtgmW8xab0c5TffNaVRSqwgNpRALwcLVlTy9nGfq5Jg5ovSzus9fhxl5TNmdJubMGcztIS+dJRmfOy/ExptctvDI4neCalOQCnhsn4Vuu+gkd9omZKy+vzPVsxtrJ7tVj7V5JckM/FzM9eVm1M0bOZbnlAWumbB8sxP+l4A8Jmembk05TSIncG7pt+eASAurh33cLfk4j7kslyPuoYcp5PaywBwM82ri3rHSPvjS9cB/t1vj4t44wqPBPg1t/rUr0PZr8+2UUinuWn5l7vhZpzba/s23uXr5NxdzHH3SuCcLql/Xj68UnXAhCyzbujnwCGbTj+eSNnXObpLDnOD7WUsV7nbrkKgOi6F+z/8/Id6FtTxs89WHYaAKvTZZs2weaRHJq9zR/nZVnkK/K3XDlW/9Eb5Ry9LeM8QZPMPAveu78tjXPGMAfuDMlzHXOO0C0dJ2WfhwbsuhGAU0/I/WTdFXKOzisavmOXrfdH7JWXt7gMbrAqe0zk7kzbvBJ/XT7yuXWcZM6dU0PmwDX/P3a/WY9QN7xOHW8rWcj7lJyTveyevR0AiLRl8L3SnC8NQyRb9araneTzdiVf4Sf6Hq9/AE3rJt2f1vZ517ZEvlD3jxhByJ/udeHPT51vJJFB0yD7G5j+i+8jZoN7PXQ40r5OATQOkgv9HcnSVSo81TO7Q5jJCwA2NjNvwIvQ3dAiNy8Bfll2nwlw4EX5WbnnxRs5hyVU9lWXUJki56q/B1OZoncHrhj1n93vXyY1pyKuTehwpYKa5jQcJaZLg1jsBGlUcPfu9AG/SiKO8W3lAvzy1VHs6yzH8o6bJNX7qjQ5Pgf+fF+u7et+Ljfc87+dYrf8jc1dAIjc4J4TuJtOz6wCcfJ6SAO5CVvWQh5OjjSVLnpG9/9oGCjXmcQMGT4QZ5uOZdaN0pj2ZKu75UMucSPoTF+3mWx7Jdeb7xp8AUDrdx6n1lwZOuC/NP9EJZ6s3Bw55u4ZPACAL2K/oEeYHNO975cu4lbDvlkkzcgk2HLxrZT8nvPgKTKx0mGjTP9Q/kH5hhu73Oc4/+hYO0r84Bn3RVciNjwl3/csCfKlntX2A9uSQOKW3gNA3aFy/bKcL/jUGy8clnvQEksloZo7N6qZzOEuiQ9Lt+EtN02ye3+JrQtyqT+T3PI6daRZ7sReO8fK0IKQQ8VzfHcPk4Sn7zSXhoaSHvwAGv2oa6YI0i64SimllFJKKaWcwusjoHtulGfskhaJfJqTR4cu3ED+ncPcw/E7ZPqKFyPMpENSB3PqifpP7HDL1idHqdTwcHbrc8Ac109iXRRb75cuWObUFVcq+RbpLvVdpVVk2PqPmZ9ddbSs46oW16z/ZJqBl4/IBN+311nNsiqSnKMwU06YCcT+iJ9uWyLf6XN/VQQPjYCe7y6RstUtPsCchmdrRjgAVg/r6mhOhxX6fQoxkm+Drvc1s1snhtytzubUD2ZX3FeOSmKQmiMkYu7u02JVnpvEtqclIjiqgiTneHK2REgu7Fbcb2c3AM4Nly7pvb5eCsCdpfcCsHO4HM913GOeeq4Klm7h5nnEHPKxpd8kMm6VZQ0XS0S7zN/y3plqUt/Su6DixlS7zzvaWKa5iFgq09C48/Gddfo0AMGd5ec9EbeQ+L8oADonSARr20n5nu7eXxEA/yArPepJ9PqNyqsv+fkNltxDvcelR1NmSuGnLCpu/mUlwlXKb5+LS+Jc1ULl2uSHH1js7wS3DZf7rPqB8j1I+HsgdQZIgqLCXEsDS8q5ITVTPi/r/PkrKXKxmJkUz6gKchw3DZbv6TUbc5frqlA5kXcMkfcurvfjG/oAUC3FfaL3F7KGSon9LoixFVeyJLOnWYa7P0B4AI2AKqWUUkoppZRyCq+NgJqTmw+65ncATmVJS87hMbUBCE5z77GTAZFVuWa4jCsq6Rds996KzdEAxBx37zoUl4BaMv3MW/W+5eOTEg0rP8Uzpl252HPXzLui7QOqVwPgdIKkGf/wzvdzrbMqTaISlnTXxpDMaMLP+yXStTz+Kw7+IC3uyz9qne92JxpI02LJKImEtaqaLJ93UTusxYNbIM9VlFbUC8ftPrFGJgSvhXuMBXS0PaOl7ma08OdX2wFQcq+bhAIvI/PgIe4ZJQmjpr71DgAxgRLtM6daif75bmIfknHqWakSJR37600ADO0pY8xeby7Rhk+adCPLxVMmAdSaJ2NSt3X/MNd75vG69Trb1EHXFfxzVz0lke5HNtvGQHZ3/94L1pTDxNwvkcpk27IgdgNQ1/YT4OdZDYDcEdDkTBnX3fO9J2SbcauwZrpPbH/fUBnvOKCUJA9bmxpV7H8jretJu9/PZgUV+98orCxDYi9ZZIFhP16wSsSJnPeABpVSKEyKR3P6lk3tZGx7u423AlDaDZI6VR64nx6zewHwQ6wk6DEjonm55smHAcjqL+Mdl8d/BUD4x6H5buMOGjdOBnLfMxQHs2eIIz7g488tAAAPR0lEQVTb12gEVCmllFJKKaWUU3htBHT7/6Rl74eKEiG6eXtvAII9ZHLlxGeqM7uyfbSs4z99ARn7Ce6fJbO4bL9Xon2tguHutTLRfXX+dWWRXGbzi5JxbVPnibnem3lGxiR9MFKOkxKJ7pHNsNyLEpFt/7/+zGr4GQCvj84/gr06TaIsVlv7WPOgdNs79i3VNd77x2PbINN6nsh+bWa/rfZJ7ilZvNHReyT6vbGVZFZMtk2IHnIkPd9t3FXJb6WXyp08BsB/t8q+PH/SNo5s1E6sqfZjIus9JZHQa+tKxHtR3EwARo/2I/IWx5f5cuo9KOPdunwrmT8HT5TrUKhfGt1DZfL2izNuF8RVwRLp/r3plwDEvTmcOqM8syfLhZLGtGZtCzPLvn10r88bEvmsOkmmn/LgThtFktkpgelNzWuVfCdmvS7T+pTBPXs6lB0qveVWLpfz8cQa82j9+kgAYiZI1Dtz/4F8t68/Q9ZJscp5rcT48rZ3XB8BzTp9Gq6Vnkmdej0AwOEE+zhUuUSDMl/KvjkyTTKVb7HlX/j0ZBQAoZtkCiL3ieU73+5MuV554nXLXWgEVCmllFJKKaWUU3hdBPTkwFYAbOw3AYCdmTL32pnXZexcMAfz3tDNrOnxLmaLoanMAxLvyTxemBEJni+rek6WtnMnSriwJK4TuLQKAK9VmZnvOp/tvxqAEvPcI/KZbZWMMSnTFQZ1GA7AibrB+a5e4WP7qMj+76U3w5qWn9ktN8eYehL/GMkCvLrFF+YSFpyR7K+Bv6xxUamc6+z1Z+x+77N+GADhSzx3bkkzElryW/vlefVSMY/bU7Nkv5tzib7eeCbvV+kAFC5LdHEzbGMUzePx69iq2e9N6CPjN62B0hvh6pFyrhlbueA9i8zMlNWaeMa1OD8HRsn59qcBbxBisR8TN/645GmoPHU94BlzPxanzE4JAPw3IpXYQDnXP7C/DQBlZ8j33BXRYHN8Zrsyv+a7jhndfP06mUC7ycxd/DtQ7icfaC89sA52k6im9ZjMTX1ikPTqaPvISl6I+AOAhOkSNa2z0D0jvaGz5JwVNSv/dbZ0+gTIGe84aWt7AKru3ezYwrmpYT1/zn5989RRANRY8qerilMkg3e34/Oay+yW7XhXnpucPR+oRkCVUkoppZRSSjmFV0VAAyKr8sjzMwAItkjVbtswCIBKCzxj7OelZERIBtHA9Mh817EeOQqAkSZ99y3B0vroX6lizjqVygKw/fG8s9EZVguxD9vGmZ46dYWlvnLvt/wi+3XkgsKPPXIn/hZpSbxwDNWp21vZrfPiS58COXNwXbh+zvyhuf8fjE77i7OoDuG/VFq/Kywt+DbnkiWjNS3tlxtt4rH8sb54CuYkKR1lDsEL9//EJTI3cV1WuqRMzvZRwjQADlplvGSFce6dUdFRKn0k0cOWN94OwMqErxgxMgqAOo+7LgJ6KWHf2R+j85pI5GfsILm+njXSSVh2PwA1P5Fj/Ohw2c85UX/PltG5OQCzH3oDgBoBOcfvHlvW27lPyjjH4LPufd9ROlmuJ2a23itlCZD7rhOPSpR/dbPpLDoXAsC25yXUH5Rx6blSHcm6IwmA6YdkHuZedRZSs+0eAPxLl5Z1bPc8mbuSAVjT1I92g6TnTvmNMnbfUlF61iVNlKz8m9rJONcU67mcyOdI94x8FoR/XD3bK+kFYY53jJjgGT3QUl+QXhurp/rTPFiO8T3fNgKgRt/8s/5eTouQJFalSe+PqDc3AL7Xu6E4ecUDqHnSa/LDPvqWlHTRX56WG72I5810255v/ndTLrvO1ev6A3A0RU6m5SrJhWBlwleF+lsNnnsIgNpPuC5JxPmb5CLRtoTZpdTzD9exM2QC51uHjstetuxNScaS83CJ7ffc21+8jqnh4vuoi+d2YbwkW+4hv4s6bHjawyfA+fL2iZTWpKVT/3WZBN4XEjrse/pq2gTLcfpXmty4+3tw19srkiXf5Qpvy//D0WnnSLxNzgU3fTUYAGONe070bqrxkzR0Iu28hFqCSGwvDWiDakrDyo9RP9nWtv/+7jlUnrrZk5t4juTu8mAddcGDp9mYMviRxwEIne8ZjUlhM6WcC1+uD0CdEkfYXk26hmfuu3yDZlbbeACSJJ8NvevLOXlM+PTsdcaMvAOAkJ/cZ2jI+WFyf/TOzNjs6UhGLJYuwqs+lEaVkgdyzshHWsgdZIvhuwB4u6pM72dekybbkvN89lZ36njoFHEX2jXaPjjRd50Mk6jsIedqv98kkdqD4x7i7yffA2BRS5n2akhHaUwozHUnaXpjANqUWJN9j10+1f2nkbrQ2V7Sgv95zY9cXJIc2gVXKaWUUkoppZRTeH5ICaCJdBd4OXxa9qJJY2QqirIbPLM16ubNA1jc8LtCb/dn06/zfe+sId0oMgz7eHDXjUMAOLk+p5tu5O+uj8fs6SEhQLM79UtHG1FyjnQJ8dR09rVnSBfpVQNLcFXw+cusnduqNOkCM/mQJAM4/oBMyxKbtMN7p+Wx7WxvmPg5/KJu0nNPNc3uNu8LBvRfTJZthw5dPQSAmkiXKP8KtukKwisAYE3c7vTyuYLZWt/h/0ax+S6JgJ5+VaZwKN1Xup+7a8KtwNWyj1qtlajAX81yrj/TohbZXkk7d5oh3Ra7b5ZERrHDd3rUOcs8PtfdYvZeyUmk1uF36TFUZ5ZnRD7z80DZJFJ+kOjg6v9qXHb9sbUmAxAfZH8ruSZd9uygVUOp8+sWwL2mjbNukylRlt0cR7n5Mk3Su1WXy5svLbdb1w+/fK89DX+/E4Dox+QcXn6/Z95vXsho3YS5Ld+3/Sb3G5bF5VxXoCtQZel/NO80EMgZArCvg9Sp5pLLb5/aW6KG37SUJFQr0oIp/4pndEO+WK0nEl1dhFw0AqqUUkoppZRSyik8OgLq3yAGgHumz8le1mDKgwBETfPcAeAAIV2SiBsjrapGPnupVOx/+Y7tjFsuLXPGnrDsZbW/s01/sMp+EHY5ttv9dDUzGcCTbX60W/7VgnbUzvTsFkbrZhk38MJjw9h7k7Sqbrux4H3yH5hyHwDVXzVTf3v/lDxZJexbn49Y01xUkqIzk4HdXHWD3fJj6SWzE4b5miyrtH8efkims+g2TCIPs3fJlEORt7imXK4SPXkv0/pKj4ZljaT3yw1N7gLA73f3HO9sRmYrPywRkpum9OCZqPkAtLYl/5h5RnrWPPtjPyAn1b87RcQuxb+c1O2RlXJ8lrTYTyH1+rH61L1brp2e2kfjs7e6A3B4xDJerGQ7R1XacIktTHJzkmnbmxukkxUDZ8g4u1pPrXDr/Zy5K5nZHWTM64Q7ZdqV1FoSqf/pBol0d/npkVxdrup9Ir2Xov7eKJ/jjMI6yeEWYdQKkCifGfkNOO+Zfc6yNm4h8lnpITlrlvRgmDvkTQBuqPgYAHUfzOm1YEmQRFkprSXh50ePjwegfpBcq2Ln3UPMX+4zlrkgLjX285oH7wUgepZrnpc0AqqUUkoppZRSyik8OgK65QFbq2tozlQh1ZbamuAMz2yxuVCtZy4f7etOQt7bsrG4i+M0WbaI0Oazkkr7uv2S9r7umE1u3ZpaGCFzVhFjC9y36y9R+8AhKQAsjJOphDr/extZn0k2Z8OWPDVq/RHAc6IHxeGLGz4EIDFdWmP7f/YEADXwoAmgrbLHJie2BeCRq5MBWLo3mkjcO9OpoyS2mwpAVjs5V8ctk2hf9P9kTJYvHeMAmXv38U0vGds96Bc5BxwdJZGW8N9dVqwCyUyWqSzoBMOHS0rU0y1kHGvsczI+Lnq3Z/ZKOtojFoDOoTJozHrRrcWPL3YgLNWzx36Wt2Vu/XtZDO/MlmPusXKX7xEV+5t8Z4P+kYzA1V6Tc3ItPKenkjXlMACRYw/bLX8YyYobQ+6pdDz/7jJ/5ysa2ZHPcf81AKDCx56zPy9m3bQVgP+7oSMAH02Wui3s/g4A31yTwPSvOgHwyT2SMbdpsH1fhhs2y+wFsR+c9theDqY6M+7L7oUS6uKp3zQCqpRSSimllFLKKTwyAmrOD7n4prdtS3xzInNvZY6J2yqBT4LYDXhvRKT017bIgC2BZC/k+A5jF7DLbl1v/T+4lJeSegCQ+n4kADVmelDk08bIlFFCUU9JdK/+azJxomV9KZeVyRV+erY9m5+WMZ4rVkpkKXb8AQDqHJKWauv5wmeH9hZm5t9+uzoDMK/pJwAMbWWbaPEv9+/ZEjFBvp8Rtt89fXxc75G/AGC9KHt89DwZjx8z07Ojnxey7kjil4ZyTvqFZpddvzbuOTZZFd3AnjnpYafMuQ6AKA+KaOcnc1cyAMH9KwFwX9MRAAQ+eYg1D8tYz9h5D9ptU+t7+c4HL5HzblZGujOKWqxCbZm5u8ySOXujcZ+eKBoBVUoppZRSSinlFB4ZAT3Qxh+AGgH2kc8vT4cTeEpaKLy5j75SPuXafQCEsc/FBbly1h1JANTo6+KCuEiJeas4Mk9emy2xnh4hc4SzveQKtvJPGQd/vJ5kMy/nPo3XPqNJiIxv9bdIe/1f56UfSoM3ZMygHr/Km8xMimdUhX8uv6KHsh6RPBqBP8tPfoYetAAghrwz3OrzhGN45APoxV47JgOlV3SJwjjovV8cpZRS3s969BgAk2NqA1DOC7rAeapHvhwKwJa73wfgrikPA1B9l+cNBVDqcozF5XmmmkzdEbHaFwf9KGfRLrhKKaWUUkoppZzCIyOgtZ+S1uCuT108SP6Q8wujlFJKKa9Uc7REOruMliQe1T1p+ielCiliwp/8O0Feh+TTJVWp4qARUKWUUkoppZRSTmExDB1eq5RSSimllFLK8TQCqpRSSimllFLKKfQBVCmllFJKKaWUU+gDqFJKKaWUUkopp9AHUKWUUkoppZRSTqEPoEoppZRSSimlnEIfQJVSSimllFJKOYU+gCqllFJKKaWUcgp9AFVKKaWUUkop5RT6AKqUUkoppZRSyin0AVQppZRSSimllFPoA6hSSimllFJKKafQB1CllFJKKaWUUk6hD6BKKaWUUkoppZxCH0CVUkoppZRSSjmFPoAqpZRSSimllHIKfQBVSimllFJKKeUU+gCqlFJKKaWUUsop9AFUKaWUUkoppZRT6AOoUkoppZRSSimn0AdQpZRSSimllFJOoQ+gSimllFJKKaWcQh9AlVJKKaWUUko5hT6AKqWUUkoppZRyiv8HSViM6s8oS5sAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 1152x576 with 15 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"slice = 15\n",
"predicted = model.predict(x_test[:slice]).argmax(-1)\n",
"\n",
"# remember our copy of x_test? This is where it comes into play again! \n",
"# Since we flatted x_test during pre-processing, we need to preserve an unflattened version\n",
"plt.figure(figsize=(16,8))\n",
"for i in range(slice):\n",
" plt.subplot(1, slice, i+1)\n",
" plt.imshow(x_test_original[i], interpolation='nearest')\n",
" plt.text(0, 0, predicted[i], color='black', \n",
" bbox=dict(facecolor='white', alpha=1))\n",
" plt.axis('off')\n",
"\n",
"# Check the 'Graphics' tab to see this logged figure \n",
"experiment.log_figure('Model Sample Prediction', plt)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Important Note:** Finally, we are ready to tell Comet that our experiment is complete. You don't need to do this is a script that ends. But in Jupyter, we need to indicate that the experiment is finished. We do that with the `experiment.end()` method:"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"COMET INFO: Uploading stats to Comet before program termination (may take several seconds)\n",
"COMET INFO: Experiment is live on comet.ml https://www.comet.ml/ceceshao1/jupyter-experiments/e4baa5e55cb2406e91354bdcca7a890c\n",
"\n"
]
}
],
"source": [
"experiment.end()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 7. Creating another experiment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you would like to create another experiment, simply create a new Experiment(), and then the fit(), and finally, experiment.end()\n",
"\n",
"You could also restart the kernel and clear the cell output, if you wanted a clear session. \n",
"\n",
"In **Section 3: Model Definition**, we defined a starting set of hyperparameters. Feel free to adjust those and log the resulting model results as another experiment and compare the difference! \n",
"\n",
"Comet also offers hyperparameter optimization: https://www.comet.ml/parameter-optimization\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"# try this second set of parameters \n",
"batch_size = 64 # adjusted batch size from 128 to 64 \n",
"num_classes = 10\n",
"epochs = 24 # adjusted number of epochs from 20 to 24\n",
"num_nodes = 64\n",
"optimizer = 'adam'\n",
"activation = 'relu'\n",
"\n",
"# Check out the Comet Hyperparameter tab for the new experiment and note the new epoch value of 24 "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Continue by running **Section 4: Comet Experiment** onwards with these new parameters and see if there's a difference in performance! Don't forget to run `experiment.end()` at the end to mark the completion of your second experiment."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Contact Comet\n",
"Have questions about this notebook or our new Jupyter support? Email us at mail@comet-ml.com or Slack us directly [here](https://join.slack.com/t/cometml/shared_invite/enQtMzM0OTMwNTQ0Mjc5LTM4ZDViODkyYTlmMTVlNWY0NzFjNGQ5Y2Q1Y2EwMjQ5MzQ4YmI2YjhmZTY3YmYxYTYxYTNkYzM4NjgxZmJjMDI)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment