Skip to content

Instantly share code, notes, and snippets.

@rpicard92

rpicard92/2-neural-networks-with-single-neuron.ipynb

Created March 7, 2019 21:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rpicard92/ecbbd196f032c1aba9463c2b2e2e7902 to your computer and use it in GitHub Desktop.
Save rpicard92/ecbbd196f032c1aba9463c2b2e2e7902 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
2-neural-networks-with-single-neuron.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Assignment2\n",
"## CS-5891-01 Special Topics Deep Learning\n",
"## Ronald Picard\n",
"\n",
"In this notebook we will walk through the design, training, and testing of a neural network with one nueron. This nueral network will be used for logistic regression, which is an archaic name for binary classification.\n",
"\n",
"The binary classification will be performed on images of handwritten numerical digits. More specifically, the last numerical digit of my student ID. This digit happens to be 9. Therefore, the goal of our neural network will be to output a the value of 1 when the handwritten numerical digit image input is a 9, and 0 in all other cases.\n",
"\n",
"The data set we will be using is the MNIST data set. This is a very popular data set amoung the machine learning community. The data set contains 60,000 images, and each image contains a handwritten numerical digit. Each of the images have been provided with a truth label that corresponds to the handwritten digit within the image from the set {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}. \n",
"\n",
"For our case, we only care about when the image is 9. Therefore we will need to re-label the truth labels so that all truth labels with the value of 9 are given to the value of 1, and all other truth labels are given the value of 0. \n",
"\n",
"To start we need to import some needed classes."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import numpy as np\n",
"import struct\n",
"from mpl_toolkits.mplot3d import Axes3D\n",
"import matplotlib.pyplot as pyplot\n",
"import csv\n",
"import time"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, we must change our path string to the path of our data file containing the features. (Please note that you must change this string to point to the directory with the data file on your machine data file on your machine.) \n",
"\n",
"Second, we much change the string name of the data files to the names of the MNIST data files. (Please note that you may NOT need to change these. Only change them if your MINST data files are named differently.)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"## path\n",
"path = 'C:/Users/computer/OneDrive - Vanderbilt/Vanderbilt_Spring_2019/CS_5891_01_SpecialTopicsDeepLearning/Assignment2/'\n",
"\n",
"#Train data\n",
"fname_train_images = os.path.join(path, 'train-images.idx3-ubyte') # the training set image file path\n",
"fname_train_labels = os.path.join(path, 'train-labels.idx1-ubyte') # the training set label file path"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we retrieve the data from the data files as follows. This imports the data into a feature tensor (3-D matrix) in which each index is a feature matrix corresponding to an image. The label data comes in the form of a vector where each index corresponds to the index of the feature matrix (image) of the feature tensor. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The training set contains 60000 images\n",
"The shape of the image is (28, 28)\n"
]
}
],
"source": [
"# open the label file and load it to the \"train_labels\"\n",
"with open(fname_train_labels, 'rb') as flbl:\n",
" magic, num = struct.unpack(\">II\", flbl.read(8))\n",
" labels = np.fromfile(flbl, dtype=np.uint8)\n",
"\n",
"# open the image file and load it to the \"train_images\"\n",
"with open(fname_train_images, 'rb') as fimg:\n",
" magic, num, rows, cols = struct.unpack(\">IIII\", fimg.read(16))\n",
" images = np.fromfile(fimg, dtype=np.uint8).reshape(len(labels), rows, cols)\n",
"\n",
"print('The training set contains', len(images), 'images') # print the how many images contained in the training set\n",
"print('The shape of the image is', images[0].shape) # print the shape of the image"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we need to perform both two steps; feature scaling and feature normalization. Feature scaling consists of converting the 28 X 28 image matrices into 784 X 1 feature vectors. In essence we will flatten the images out into vectors so that we can use an input a vector to our single neuron. Feature normalization is a process of normalizing the pixel data to between 0 <= x <= 1 (for logistic regression). Each pixel comes on a scale of 0 <= x <= 255. Since 255 is the maximum for every pixel we shall divide each pixel by that number (elementwise) in order to normalize each pixel to between 0 and 1 (inclusive).\n",
"\n",
"One additional item we need to take care of is relabeling our label (truth) data so that we have a binary classification in which all 9s are converted to 1s and all other labels are converted to 0."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(784, 60000)\n",
"(784, 60000)\n",
"60000\n"
]
}
],
"source": [
"# feature scaling\n",
"matrix_side_length = len(images[0])\n",
"vector_size = matrix_side_length*matrix_side_length\n",
"\n",
"scaled_images_feature_matrix = []\n",
"for image in images:\n",
" reshaped_image = np.array(image).reshape((vector_size))\n",
" scaled_images_feature_matrix.append(reshaped_image)\n",
"\n",
"# convert to numpy array\n",
"scaled_images_feature_matrix = np.transpose(np.array(scaled_images_feature_matrix))\n",
"print(scaled_images_feature_matrix.shape) # scaled_images_feature_matrix is a matrix of 60000 X 784\n",
"#print(scaled_images_feature_matrix[0].shape)\n",
"\n",
"# feature normilization\n",
"normilization_factor = 1/255\n",
"normalized_scaled_images_feature_matrix = np.multiply(normilization_factor, scaled_images_feature_matrix)\n",
"print(normalized_scaled_images_feature_matrix.shape)\n",
"#print(normalized_scaled_images_feature_matrix[0])\n",
"\n",
"# re-label for binary classification\n",
"value_for_1 = 9\n",
"binary_labels = []\n",
"for label in labels:\n",
" if(label == 9):\n",
" binary_labels.append(1)\n",
" else:\n",
" binary_labels.append(0)\n",
"\n",
"# convert to numpy array\n",
"binary_labels = np.array(binary_labels)\n",
"print(len(binary_labels)) # binary_labels is a row vector of 1 X 60000\n",
"#print(binary_labels[0])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In order to test the efficacy of our nueral network, we need to split up the our label data into two data sets; a smaller and a larger one. The larger set will be the training data that we will use to train our nueral network on. The smaller set will be the testing data that we will used to test the accuracy of our nueral net. The MNIST data set contains 60,000 images. Therefore, we will use 50,000 images for our training data set, and 10,000 images for our testing data set. \n",
"\n",
"It is common practice to use a smaller subset of the total data set to debug and tune hyper-parameters before using the entire time-comsuming data set. This smaller subset is known as a validation set. Therefore, we will first use a validation data set of 600 images. 500 of these images will be used for as our training data set, and the other 100 of these images will be used for our test data set. This validation set step is not nessesary with our single-neuron network, but we will do it for completeness sake; however, after our network is working we won't waste time on tuning hyper-parameters with this validation set, because in this case our full data set will run quickly as well. \n",
"\n",
"Thus, we will begin by sifting out a validation set from our total data set."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(784, 500)\n",
"(500,)\n",
"(784, 100)\n",
"(100,)\n"
]
}
],
"source": [
"# create a validation data set\n",
"size = vector_size\n",
"\n",
"number_of_testing_images = 100\n",
"number_of_training_images = 500\n",
"number_of_validation_images = number_of_testing_images + number_of_training_images\n",
"\n",
"validation_training_images = []\n",
"validation_training_labels = []\n",
"validation_testing_images = []\n",
"validation_testing_labels = []\n",
"\n",
"for index in range(0, number_of_validation_images):\n",
" if(index <= number_of_training_images - 1):\n",
" validation_training_images.append(normalized_scaled_images_feature_matrix[:, index]) \n",
" validation_training_labels.append(binary_labels[index])\n",
" else:\n",
" validation_testing_images.append(normalized_scaled_images_feature_matrix[:, index]) \n",
" validation_testing_labels.append(binary_labels[index])\n",
" \n",
"# covert to numpy array\n",
"validation_training_images = np.transpose(np.array(validation_training_images))\n",
"validation_training_labels = np.array(validation_training_labels)\n",
"validation_testing_images = np.transpose(np.array(validation_testing_images))\n",
"validation_testing_labels = np.array(validation_testing_labels)\n",
"\n",
"# logger\n",
"print(validation_training_images.shape) # validation_training_images is a matrix of 784 X 500\n",
"print(validation_training_labels.shape) # validation_testing_labels is a row vector of 1 X 500\n",
"print(validation_testing_images.shape) # validation_training_images is a matrix of 784 X 100\n",
"print(validation_testing_labels.shape) # validation_testing_labels is a row vector of 1 X 100"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we move on to the training of our single-neuron neural network. \n",
"\n",
"Part 1 - Feed Forword:\n",
"\n",
"For this neural network we will stick with a simple model of one nueral. This nueron will have an input of a matrix (784 X number_of_images) of vectorized images of 784 X 1, and will output a row vector of probabilities which we will convert into binary classifications of 0 or 1. (If P(x) >= 0.5 then we will convert it to a 1, otherwise we will convert to 0.) The model of the neuron will be a vecorized linear model Y = W^T * X + B, where W is a vector of parameter weights, X is the input matrix of vectorized images, and B is a row vector of bias's. (Note: in this case, b will be scalar that applied in a broadcasing manner to save on memory.) The output row vector of this linear model will be subject to an elementwise sigmoid activation function. The sigmoid activation function is A = sigma(Z) = 1/(1 - e^-Z), where Z = Y. The resultant row vector will then be used to calculate the cost function values in an elementwise manner. The cost function for this binary classification will be L(Y_Predicted, Y_Label) = -Y_Label * Log(A) - (1-Y_Predicted) * Log(1-A), where Y_Label is the True Label, Y_Predicted is the probability value predicted by the nueral network, and A is the activation function value. The resultant cost row vector will be added up and divided by the number of elements in order to calculate the average cost. \n",
"\n",
"Part 2 - Back Propogation:\n",
"\n",
"The back propogation technique that we will use for training the nueral network, will be gradient descent. This involes utiling the gradient of the cost function to updated our model parameters. In order to calculate the gradient we will utilize the chain rule. The goal of back propogation is the adjust the parameter weights and bias's of our model to accurately perform binary classification. In general the chain rule can be used to find the gradient of these cost function (vecorzied rates of change) with respect to the model parameters. The following is the chain that we will utilize. Note that it is in the form of scalar cases, but we will utilized a vectorized verion.\n",
"\n",
"dl(a, y)/dw = dl(a, y)/da * da/dz * dz/w \n",
"\n",
"dl(a, y)/db = dl(a, y)/da * da/dz * dz/b\n",
"\n",
"\n",
"\n",
"The partial derivative of the cost function with respect to the activation function is found by the following:\n",
"\n",
"dl(a, y)/da = -y/a + (1-y)/(1-a).\n",
"\n",
"\n",
"\n",
"Due to the chain rule, the derivative of the cost function with respect to the linear model z (where z = y) is found by the following:\n",
"\n",
"dl(a, y)/dz = dL(a, y)/da * da/dz.\n",
"\n",
"The derivative of the sigmoid activation function is da/dz is found by the following:\n",
"\n",
"da/dz = sigma(z) = sigma(z) * (1-sigma(z))\n",
"\n",
"Therefore the derivative of the cost function with respect to the output of the linear model is found by the following:\n",
"\n",
"dl(a, y)/da * da/dz = (-y/a + (1-y)/(1-a)) * (sigma(z) * (1-sigma(z))) = a-y. (For convienence we will say dz = a-y.)\n",
"\n",
"\n",
"\n",
"Now we can extrapolate the chain rule to all the way to the paramters of the linear model.\n",
"\n",
"dl(a, y)/dw = dl(a, y)/da * da/dz * dz/w = dL(a, y)/da * da/dz * x = dz * x = x * dz (we will change our notation to dw = x * dz for convienence)\n",
"\n",
"dl(a, y)/db = dl(a, y)/da * da/dz * dz/b = dz * 1 = dz (we will change our notation to db = dz for convienence)\n",
"\n",
"\n",
"\n",
"Now we can set up the vectorized version of our gradient, then take the average of dw and db.\n",
"\n",
"dZ = A - Y^T, where A is a row vector of activation function output, and Y is the vector of truth labels.\n",
"\n",
"dW = 1/m * (X * dZ^T), where X is our feature matrix of vectorized images, and dZ is a vector of the rate of change of our cost function with respect to the activation function, and m is the number of elements in the resultant vector.\n",
"\n",
"dB = 1/m * sum(dZ), where dZ is the rate of change of our cost function with respect to the activation function.\n",
"\n",
"Finally, we will multiple our gradient transposed parameter row vectors by our learning rate and then subtract this from our parameter vectors.\n",
"\n",
"W:= W - alpha * dW^T\n",
"\n",
"W:= B - alpha * dB^T"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The first thing we have have to do is initialize our weights and bias's. The weights and bias's will initialized to 0.01. This was choosen pseudo-randomly. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(784,)\n"
]
}
],
"source": [
"# initialize weights & bias\n",
"initial_value = .01\n",
"W = np.multiply(initial_value, np.ones(size))\n",
"b = initial_value # b will be added in a broadcasting manner\n",
"\n",
"print(W.shape) # W is a vector with a size of 784"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we implement our gradient descent algorithm."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XucHFWd9/HPN5dJQhISSIZbLgQwGBC56MAi3nDBNaxA1BUMXhBWyeoj3i8bUZHlWXe9PYrroisiK7ACRnQ1QhRRREUJZBAIJCEawiVDIEwSciFccuH3/HGqZ2o63T2TyVQ6M/19v1716q6q01WnurrrV+ecqlOKCMzMzAAG1TsDZma2+3BQMDOzDg4KZmbWwUHBzMw6OCiYmVkHBwUzM+vgoLADJP1C0rvrnQ8buCRNkRSShmTjVX9z5Wl7sa4LJF2+M/ndFSR9TtJ/9XVaqyIidvsBeBg4ud75KHgb9wQuAR4FngaWZePj6523HuT9ImBLlu/ScHBu/tHAXcAz2evRuXkCvgSsyYYvA6qynhOBAH5SNv2obPqtuWkB3AcMyk37V+D72fspWZoh2fhE4MfAamB99tlzgFfntmlT9pn8dk4uy8tNwMUV8j4DeKK0vhrfZZd89WHaE4G2XfBbuCD33TwHbMuNL6r3b3UntmtI9l1vyrZlNfBr4IwdWMbJwMM7mY9ZwFJgY/Z7ugEY2Zfrdkkh09uzrT5adxPwG+AlwHRSgDiBdJA8rhfLq8e2/DAiRuWG5VlemoCfAf8D7AVcCfwsmw7pR/4m0oH9SOBU4J9qrKcdOEHSuNy0dwN/qZD2AGBmD/N/NbACOBAYB5wNrIqIP5S2ibR/AMbmtvPRsuV8H3iXJJVNfxfwg4jY2sP89EsR8W+57+t9wO257+ol5enr+b/rpZdk2zaN9Jv+tqTP7IoVSzoJ+BfgzIgYTfo9Xt/nK6p3BO5hdHyYKiUF0kHkHmAd8CfgyNy82cCDpKi6GHhzbt45wB+BrwNrSWeR5wC3AV8FngIeAk7JfeZW4L25z9dKexDw+2zdvwYuBf6nyja8F1gFjKrxHQTwotz494F/zd6fCLQB/0w6e7gaWAKcmks/hHR287Js/Pjs+1oH3AucuBP756Ia2/Z3wGPkzv5JpaHp2fs/AbNy894DzK+yrNJ2/hfwgWza4GzahWxfUvhn4K90lgZqlRSeJleCqbL+Lp+pkmYEqaTxmty0vUhnzUdl428E7gY2kALRRdXWUfabG5z93lYDy4EPlKU9N9vvG7P5/5RNHwk8C7xA51n7AeX7DTgdWJT9Jm4FDiv7D34CWJht3w+B4d18X+cAt5VNK51x/x9SaXhZNv0/s/24AVgAnJD7TH6/vSj7/NlZ+nZgdi/T7kE6sK8jHR9mU+VsOpfvKWXTZ2bf7djcf7m0Dx7M7bsxFfbBPsArgPlZHh4H/gMYWiUPs4Hra3zfw4GvZb+pVcC3smkV111tOf26pCDpZcAVpDPLccB3gLmShmVJHiQV/8eQIuz/SNo/t4i/If159gG+kJu2FBhPqsr4XoWzPnqQ9hrgzixfF5HOFKs5GfhlRDzd/VZXtR+wN+lMdxZwLXBWbv4bgNUR8WdJE4AbSX+gvUl/9h9Lat6J9Z8maa2kRZLen5v+EmBhZL/azEI6z7pfQgpKJffm5lVzFemPDmm7FgErK6T7Cekgc04P8j8fuFTSTEmTe5C+ooh4FpiTyx/AmcADEVHazk3Z/LGkAPF+SW/qweLPI50EHQO0AG8tm/9kNn9PUoD4uqSXRcQm4BRgZXSetXf5viQdSvrNfARoBuYBP8+V6ErbMZ10wnMkPfteqzkdOBZ4aTZ+R7bMvUlnvz/K/Y8rOYF00H8D8C+SpvYi7cWk4Dglm/fOXmzHT4Fh2bZAOhi/kbQPzgO+KenIiFgPnAY8mtsHTwJbgQ+TjiGvJH2/1UrK84E3Svq8pBMqfD9fpXPfTM226zM11l1Rvw4KpC/9OxFxR0Rsi4grgedJZ8FExI8iYmVEvBARPySdNearY1ZGxDcjYmv2ZwZ4JCK+GxHbSFUd+wP7Vll/xbTZQeVY4MKI2BwRtwFza2zHONJZws54Afh8RDyfbcs1wOmS9sjmvz2bBunHPy8i5mXfzc1AK/D3vVz3HOAw0sHkPOBCSaWANIp0Zpm3HhhdZf56YFSNQExE/AnYW9KLSQfXq6olBT6X5afWAQbgDOAPWfqHJN0j6dhuPlPNlcAZkkZk42dn00r5vzUi7su++4Wkg/Fre7DcM4FLImJFRKwF/j0/MyJujIgHI/kd8CvSSVFPvA24MSJujogtpAPMCNIBteQ/sv/TWuDnpLai3vq3iHiq9L+LiKsjYm2k6rUvkw6qL6rx+Ysi4rmI+DPppOCoXqQ9E/hCRKyLiBWk0soOiYjnSDUNe2fjP4+I5dk+uIVULVx1H0TEguz4tTVSletlVPktRMStpBOBY4FfAKslfUXSIEmDSKWUj2Tf6wbS76On1acd+ntQOBD4uKR1pQGYRIr+SDo7+3OX5h1BisglKyos84nSm4h4Jns7qsr6q6U9AFibm1ZtXSVrSAFlZ7RnP9BSfpaRirGnZYHhdDqDwoGkg1b+e3tVpTxIeoekp7PhF5VWHBGLs4PFtuyA/Q06z2KfJv3B8/YkFa8rzd8TeLqsZFHJ1cD5wOuA/62WKCLmkaqrZtVaWPZHmh2p3ntfUpXkT2sFpxrLuo1UVTFD0sGkP3Hpu0fS30j6raR2SetJde/jKy+tiwPo+jt6JD9T0imS5mcltnWkIN+T5ZaW3bG8iHghW9eEXJoncu+fofr/oie6/B8kfUrSA9n38RSpyqtq3iOix3mpkXb/snzU+o9WJGk4KSCszcZPlXRHbh/8HTW2Q9I0STdKekLSBlLppdZ23xgRp5KqJN9COgk7l1RTMAy4N/efvoFUC7JD+ntQWEGK9GNzwx4Rca2kA4Hvkg4c4yJiLHA/6WqXkqK6iH2cdCa7R27apBrpfw28QdLIGmmeIdWBluxXNr/StpSqkGYAi7NAAel7u7rsexsZEV8sX0BE/CBX5DylRv7K81L6nhcBR5YdXI/Mppfm58/yjsrNq+VqUr30vLLgW8lngc/Q9furKiJWk86UDyA7A+yFUhXXu4BfRcSq3LxrSCXHSRExhtRG0pPg8zhdf0cd1VxZSejHWb73zX7v83LL7e63vpJ0slBanrJ1PdaDfPVGR34kvQ74GPAPpCq1vUgnCzsckHfQE6Srzkpq/UereROpdmJBVjK8nnSGXtoHv6L2PvgO6bj0oojYk9Q21u1250r4t5JOdlcBm4EX5/7TY7LfV7V1V9SfgsJQScNzwxDSQf992ZmXJI2U9EZJo0lnGkE6Y0PSuaQvr3AR8QipOuYiSU2SXkGq06umdOXLj7Mzh0GSxmXXkZeqdO4B3i5psKTp9Ky64TrSmcr7yZ2pkhrXTpP0hmx5wyWdKGlixaV0Q9IMSXtl++A44EOkK44g/Wi3AR+SNEzS+dn0W7LXq4CPSZog6QDg46RG9Joi4iHSd9DtlR9Zsfs+0lVK1bbhS5KOkDQk+/28n9QIuqa75VdxFamt6DxyVUeZ0aSS5HPZ9/X2Hi5zDul7nChpL1LDY0kT6UyxHdgq6RTSvi9ZBYyTNIbK5pDqq0+SNJS0H54nXQhQtNGkuvXVwFBSG1ytE6S+Mge4QNLY7Lf/gZ5+MPt/vgv4JvDvEbGO9P03kfbBNkmnAiflPrYKGJ/9vkpGk6pMN0k6jBpX3kl6s6Qzc/+140lVU/OzKuzLgUskNWfzJ0oq/QYqrbui/hQU5pFa0EvDRRHRSvrT/SepyLmMrPErIhYD/w+4nfSFvJR0tdGu8g7SlQVrSA26PyT9ybYTEc+TDiAPADeTGkfvJBUj78iSfZgUWNZly/5pdxmIiMdJ239Ctv7S9BWk0sMFpB/wCuCT9P73MJP03W8kHQy/lLXvEBGbSWdTZ2d5/0fgTdl0SGdKPycdtO8nNYB/pycrjYjboqzBtIbPUvusfw9SNdQ60sUHB5Kq3HolIh4mHVBHsn170v8BLpa0kXRmOKeHi/0u6T6Ie4E/kxrSS+vbSArGc0j/hbfn1xsRD5BKjsuz6oUDyvK7lNTW9E3Swfk04LTcfirSPFJp+a+kq5w2sPNtbD3xedKx4WHSGf0cqvxHcxZJepqU13OBD0bExQBZYPgo6Xe0llSFekPpgxFxP6k093C2D/YhBd93k/473yH3P61gHamqcRnpO7qS1DZT+szHSVWAd5ICza9IDc7V1l2Ruq+6tb4g6YekK1A+X++8mNn2JH2QdMJyUreJB7D+VFLoVyQdK+mQrCpoOunMvNuzezPbNbIqyxOy/+hhdJ7lN7T+djdhf7IfqXg/jnTjzPsj4u76ZsnMcoaRquSmkKrcrqWHVZcDmauPzMysg6uPzMysQ7+rPho/fnxMmTKl3tkwM+tX7rrrrtUR0W1XNv0uKEyZMoXW1tZ6Z8PMrF+R9Ej3qQquPpI0XdJSScskza4wf3J2u//dkhbmbtQyM7M6KCwoSBpM6i76FOBw4CxJh5cl+ywwJyKOId0A9a2i8mNmZt0rsqRwHKmbgOXZXZHXka7Vzws6O0MbQ+Xuj83MbBcpMihMoGuvg2107XERUh8n75TURrrV/YOVFiRplqRWSa3t7e1F5NXMzCg2KFTq6a/8poizSE9Jmkjq5vdqpX7Bu34o4rKIaImIlubmnXkOjJmZ1VJkUGija1e0E9m+eug9ZJ2BRcTtpEfH9bT/dzMz62NFBoUFwFRJByk90m8m2/cW+ShZ17JZ3yPDybq6NjOzXa+woJA9Vu98Ule/S0hXGS2SdLGkUpfEHwfOk3Qvqd+Rc3rwxK3eue02+NznYOvWQhZvZjYQ9Lu+j1paWqJXN6999avwyU/Cxo0wameeImhm1v9IuisiWrpL1zh9HzU1pdfnu3uGhplZ42q8oLB5VzxIysysf2qcoDBsWHp1UDAzq6pxgoKrj8zMutU4QcElBTOzbjVOUHBJwcysW40XFFxSMDOrqnGCgquPzMy61ThBwdVHZmbdapyg4JKCmVm3GicouKRgZtatxgsKLimYmVXVOEGhVH3kkoKZWVWNExRcUjAz61bjBAU3NJuZdatxgoIbms3MutV4QcElBTOzqhovKLikYGZWVeMEhUGDYMgQlxTMzGponKAAqbHZQcHMrKpCg4Kk6ZKWSlomaXaF+V+XdE82/EXSuiLzQ1OTq4/MzGoYUtSCJQ0GLgVeD7QBCyTNjYjFpTQR8dFc+g8CxxSVHyAFBZcUzMyqKrKkcBywLCKWR8Rm4DpgRo30ZwHXFpifVH3kkoKZWVVFBoUJwIrceFs2bTuSDgQOAm4pMD8uKZiZdaPIoKAK06JK2pnA9RGxreKCpFmSWiW1tre39z5Hbmg2M6upyKDQBkzKjU8EVlZJO5MaVUcRcVlEtERES3Nzc+9z5IZmM7OaigwKC4Cpkg6S1EQ68M8tTyTpxcBewO0F5iVxScHMrKbCgkJEbAXOB24ClgBzImKRpIslnZ5LehZwXURUq1rqOy4pmJnVVNglqQARMQ+YVzbtwrLxi4rMQxdNTbBp0y5bnZlZf+M7ms3MrENjBQVXH5mZ1dRYQcElBTOzmhorKLikYGZWU+MFBZcUzMyqaqyg4OojM7OaGisouPrIzKymxgoKLimYmdXUWEGh1KawC26eNjPrjxovKABs2VLffJiZ7aYaKygMG5ZeXYVkZlZRYwWFUknBjc1mZhU1VlBwScHMrKbGCgouKZiZ1dSYQcElBTOzihorKLj6yMyspsYKCq4+MjOrqbGCgksKZmY1NVZQcEnBzKymxgwKLimYmVXUWEHB1UdmZjUVGhQkTZe0VNIySbOrpDlT0mJJiyRdU2R+XH1kZlbbkKIWLGkwcCnweqANWCBpbkQszqWZCnwaeGVEPCVpn6LyA7ikYGbWjSJLCscByyJieURsBq4DZpSlOQ+4NCKeAoiIJwvMj0sKZmbdKDIoTABW5Mbbsml5hwKHSvqjpPmSpldakKRZkloltba3t/c+R25oNjOrqcigoArTyp9uMwSYCpwInAVcLmnsdh+KuCwiWiKipbm5ufc5KlUfuaRgZlZRkUGhDZiUG58IrKyQ5mcRsSUiHgKWkoJEMVxSMDOrqcigsACYKukgSU3ATGBuWZqfAq8DkDSeVJ20vLAcuaHZzKymwoJCRGwFzgduApYAcyJikaSLJZ2eJbsJWCNpMfBb4JMRsaaoPDF0aHp19ZGZWUWFXZIKEBHzgHll0y7MvQ/gY9lQvEGDYMgQlxTMzKporDuaIVUhuaRgZlZR4wWFpiaXFMzMqmi8oDBsmIOCmVkVjRcUmppcfWRmVkVjBgWXFMzMKmq8oOCGZjOzqhovKLikYGZWVeMFBTc0m5lV1XhBwQ3NZmZVNWZQcEnBzKyixgsKbmg2M6uq8YKCSwpmZlU1XlBwQ7OZWVWNFxTc0GxmVlVjBgWXFMzMKmq8oOCGZjOzqhovKLikYGZWVeMFBTc0m5lV1XhBoVRSiKh3TszMdjuNGRQAtmypbz7MzHZDhQYFSdMlLZW0TNLsCvPPkdQu6Z5seG+R+QFS9RG4sdnMrIIhRS1Y0mDgUuD1QBuwQNLciFhclvSHEXF+UfnYTqmk4HYFM7PtFFlSOA5YFhHLI2IzcB0wo8D19UyppOCgYGa2nSKDwgRgRW68LZtW7h8kLZR0vaRJlRYkaZakVkmt7e3tO5erUknB1UdmZtspMiiowrTyS35+DkyJiCOBXwNXVlpQRFwWES0R0dLc3LxzuXL1kZlZVUUGhTYgf+Y/EViZTxARayKidMr+XeDlBeYncUOzmVlVRQaFBcBUSQdJagJmAnPzCSTtnxs9HVhSYH4SlxTMzKoq7OqjiNgq6XzgJmAwcEVELJJ0MdAaEXOBD0k6HdgKrAXOKSo/HdzQbGZWVWFBASAi5gHzyqZdmHv/aeDTReZhO25oNjOrqnHvaHZJwcxsO40XFNzQbGZWVY+CgqSrezKtX3BJwcysqp6WFF6SH8m6sCj+8tEiuKHZzKyqmkFB0qclbQSOlLQhGzYCTwI/2yU57GtuaDYzq6pmUIiIf4+I0cBXImLPbBgdEeOyK4f6H1cfmZlV1dPqoxskjQSQ9E5JX5N0YIH5Ko4bms3MquppUPg28Iyko4BPAY8AVxWWqyK5pGBmVlVPg8LWiAhS19ffiIhvAKOLy1aBXFIwM6uqp3c0b5T0aeBdwKuzq4+GFpetAg3Nsu2SgpnZdnpaUngb8DzwjxHxBOm5CF8pLFdFGjQoBQYHBTOz7fQoKGSB4AfAGEmnAs9FRP9sU4DUruDqIzOz7fT0juYzgTuBM4AzgTskvbXIjBWqqcklBTOzCnrapvAZ4NiIeBJAUjPpSWnXF5WxQg0b5pKCmVkFPW1TGFQKCJk1O/DZ3Y9LCmZmFfW0pPBLSTcB12bjb6PsOQn9yrBhDgpmZhXUDAqSXgTsGxGflPQW4FWAgNtJDc/9kxuazcwq6q4K6BJgI0BE/CQiPhYRHyWVEi4pOnOFcfWRmVlF3QWFKRGxsHxiRLQCUwrJ0a7ghmYzs4q6CwrDa8wb0ZcZ2aVcUjAzq6i7oLBA0nnlEyW9B7iru4VLmi5pqaRlkmbXSPdWSSGppfss9wE3NJuZVdTd1UcfAf5X0jvoDAItQBPw5lofzPpHuhR4PdBGCjBzI2JxWbrRwIeAO3Y8+73khmYzs4pqBoWIWAWcIOl1wBHZ5Bsj4pYeLPs4YFlELAeQdB2pl9XFZen+L/Bl4BM7kvGd4uojM7OKenSfQkT8FvjtDi57ArAiN94G/E0+gaRjgEkRcYOkqkFB0ixgFsDkyZN3MBsVuKHZzKyiIu9KVoVp0TFTGgR8Hfh4dwuKiMsioiUiWpqbm3c+Zy4pmJlVVGRQaAMm5cYnAitz46NJVVK3SnoYOB6Yu0sam93QbGZWUZFBYQEwVdJBkpqAmcDc0syIWB8R4yNiSkRMAeYDp2f3QBTLDc1mZhUVFhQiYitwPnATsASYExGLJF0s6fSi1tsjrj4yM6uopx3i9UpEzKOs47yIuLBK2hOLzEsXbmg2M6uo/3Z/vTOammDLFojoPq2ZWQNpzKAwbFh63bKlvvkwM9vNNGZQaGpKr65CMjProrGDghubzcy6aMygUKo+cknBzKyLxgwKLimYmVXUmEFhRPYoiE2b6psPM7PdTGMGhQkT0utjj9U3H2Zmu5nGDAqTsi6ZHn20vvkwM9vNNGZQmDABJFixovu0ZmYNpDGDwtChsN9+DgpmZmUaMygATJ7soGBmVqZxg8KkSW5TMDMr09hBYcUKd4pnZpbTuEFh8mR49llYu7beOTEz2200blAoXZbqdgUzsw4OCm5XMDPr4KDgkoKZWYfGDQr77pvuV3BQMDPr0LhBYdAgmDjRQcHMLKfQoCBpuqSlkpZJml1h/vsk3SfpHkm3STq8yPxsx/cqmJl1UVhQkDQYuBQ4BTgcOKvCQf+aiHhpRBwNfBn4WlH5qah0r4KZmQHFlhSOA5ZFxPKI2AxcB8zIJ4iIDbnRkcCuvZNs8uTUffa2bbt0tWZmu6sig8IEIH8a3pZN60LSByQ9SCopfKjA/Gxv0iTYuhVWrdqlqzUz210VGRRUYdp2JYGIuDQiDgH+GfhsxQVJsyS1Smptb2/vuxz6XgUzsy6KDAptwKTc+ERgZY301wFvqjQjIi6LiJaIaGlubu67HPpeBTOzLooMCguAqZIOktQEzATm5hNImpobfSPw1wLzs73Jk9Org4KZGQBDilpwRGyVdD5wEzAYuCIiFkm6GGiNiLnA+ZJOBrYATwHvLio/FY0dCyNHOiiYmWUKCwoAETEPmFc27cLc+w8Xuf5uSb5Xwcwsp3HvaC7xvQpmZh0cFPxYTjOzDg4KkybBE0/A88/XOydmZnXnoFC6LPWxx+qbDzOz3YCDwpQp6fWBB+qaDTOz3YGDwgknpEtTr7mm3jkxM6s7B4Xhw+Gss+AnP4H16+udGzOzunJQADj3XHj2WZgzp945MTOrKwcFgJYWOPxw+P73650TM7O6clCAdGfzuefCn/4ES5fWOzdmZnXjoFDyznfC4MFw5ZX1zomZWd04KJTstx+ccgpcdZWfxGZmDctBIe+cc9JNbNdfX++cmJnVhYNC3mmnwctfDrNmuW3BzBqSg0JeU1O6X2HYMJgxw/ctmFnDcVAoN3ky/OhHsGwZvOtd8MIL9c6Rmdku46BQyWtfC1//Ovz853D22bBpU71zZGa2SzgoVHP++XDxxalPpGOPhUWL6p0jM7PCOShUI8HnPgc33wxr1qTAcMklfu6CmQ1oDgrdOekkuOeeVKX00Y/CtGlw9dW+l8HMBiQHhZ7Yf3/4xS9SqWHcuNTOcNhh8I1v+AolMxtQCg0KkqZLWippmaTZFeZ/TNJiSQsl/UbSgUXmZ6edfDLceWe6Omn8ePjIR2DCBDjvPPjDH3ylkpn1e4UFBUmDgUuBU4DDgbMkHV6W7G6gJSKOBK4HvlxUfvrMoEHw1remzvNaW+GMM1Jj9GteA4ccAhdcAHfdBRH1zqmZ2Q4rsqRwHLAsIpZHxGbgOmBGPkFE/DYinslG5wMTC8xP33v5y+G//xtWrUrtDIceCl/+cuqK+6CDUhvELbfA5s31zqmZWY8UGRQmACty423ZtGreA/yi0gxJsyS1Smptb2/vwyz2kVGjUi+rN90ETzwBV1wBRxwB3/pWaqhubk4liu99D9ra6p1bM7OqhhS4bFWYVrFORdI7gRbgtZXmR8RlwGUALS0tu3e9zPjx6dkM554LTz8Nv/kN3HADzJvX2dHeYYel9omTToITT4QxY+qaZTOzkiKDQhswKTc+EVhZnkjSycBngNdGxMC6CWDUqNSH0owZqY1h0aJUmvjVr+Dyy+Gb30xtFC9/eQoOr30tvOpVDhJmVjeKghpEJQ0B/gKcBDwGLADeHhGLcmmOITUwT4+Iv/ZkuS0tLdHa2lpAjnex55+H+fNTSeLWW+GOO1LbgwRHHgmvfnUKECecAJMmdbs4M7NaJN0VES3dpisqKGSZ+HvgEmAwcEVEfEHSxUBrRMyV9GvgpcDj2UcejYjTay1zwASFcs8+C7ffni5t/cMf0vtnsjb4SZPgFa+A449Pr8cck3pyNTProd0iKBRhwAaFclu2wMKF8Mc/pmH+fHj00TRv6FA4+ujU9cZxx6WrnaZNS48TNTOrwEFhIFq5MgWHO+9Mw4IFqTEbYOTIVIJ42cs6h2nTUgAxs4bnoNAItm2Dv/wl3URXGu65p7PaadiwdGnsMcfAUUeltoojj4SxY+ubbzPb5RwUGlUpUNx1F9x7bwoSd9+denotmTQJXvrSNBxxRBqmTYPhw+uXbzMrVE+DQpGXpFo9DB6c7oM47LB0Qx2ky2EffzwFiYUL4b770nDzzantAtKlsYccAocfnobDDkuBYto0GD26fttjZruUg0IjkOCAA9Jwyimd07dsgb/+Fe6/Pw1LlsDixXDjjbB1a2e6CRPgxS/uOhx6KBx4oBu3zQYYVx/Z9rZsgQcfTEFiyRJ44AFYujQN+a7Chw6Fgw+GqVPhRS9KwyGHpOHAA6GpqX7bYGZduPrIem/o0M6qoze/uXN6BLS3pzaL0rBsWSpt3HJLZwM3pOqoyZNT0Dj44NRBYH7YZ59UgjGz3YqDgvWclA7m++yT7rbOi0idAS5blkoZy5en1wcfhLlz4cknu6YfMSKVJioNkyenqq4h/nma7Wr+11nfkNIT6vbfP3XRUW7TJnjoIXj44TSU3j/ySLqUNn91FKSSxgEHpCul8sPEiZ3Dvvs6cJj1Mf+jbNcYObLz8tdKnn4aVqxIQWLFinT39qOPpvd3351KG8891/UzgwbBfvulhvAJEzob0/PD/vunR6i6qsqsRxwUbPcwalTnpbSVRKTSxGOPpWdStLV1vn/ssVRt9bvfwVNPbf/ZoUNT8Nh//66vpWHffTuHkSOL3U6z3ZyDgvUPUnpWxfjx6e7sap59NrVtrFyZhscfT8PKlWn6ww+nzgZXr678yNSRI7tDNBqyAAAJfUlEQVQGiVIbSnPz9q/jxrkbERtwHBRsYBkxovMKp1q2bk1XUj3+eHqc6qpVKWisWpUaxVetSo3kt9+e0r3wQuXl7LVXClTNzdu/jhuX3o8b1znstVeq9jLbTTkoWGMaMqSzYbw727bB2rUpODz5ZHrND6tXp9eHHkqdFK5e3XmneLlBg1LfU/lAMW4c7L1312Gvvbq+HzvWNwraLuGgYNadwYPTmX9zc+oCpDsRsGFDagNZvToNa9akwLJmTddh5cp0N/maNZ093lYzZkwKEPlh7NjO1/L3+WHECDe2W484KJj1NSkdwMeMSTfu9dTmzbBuXWfweOqpNKxd2/X9unXp/ZIl6XXdutSWUsuQISk4lPKVf18a9tyz+uuee6aLAVz1NeA5KJjtLpqaOhu2d9Rzz6UuSEoBo/R+3brK79evT3eir1+fhg0bul+HlDpHLAWJSu9Hj+4cSuOjRnWdXprm6rDdkoOC2UAwfHga9t23d59/4QXYuLFrkNiwYfv3Gzd2jpfeP/ZY5/uNG6s3ypcbMaJrkCgN5ePlw8iRXd+XxkeOTIHV1WQ7xUHBzFK1UKkaaWdEpKqsfJDID08/Xfm19H7t2nTTYn5+vsfe7gwe3DVI5N/vsUfX1/IhP780lI8PHTrgg46Dgpn1HanzANrbUku5zZtTcNi0Kb12977SsG5dKtGUxp95Jr3uaC/Rgwd3DRLlw4gR27/Pv9Z6Xz7U6R6YQoOCpOnAN4DBwOUR8cWy+a8BLgGOBGZGxPVF5sfM+qGmps7Lc/tSRGqLKQWIfLDYtCmVePLvS/OqvW9v7/zMs892vt+2rXf5Gzx4+0Bx0UUwc2affg3lCgsKkgYDlwKvB9qABZLmRsTiXLJHgXOATxSVDzOziqTOg+24ccWtZ8uWziDxzDPdv680PPdcei0yn5kiSwrHAcsiYjmApOuAGUBHUIiIh7N5PWyZMjPrZ4YOTcOee9Y7Jz1S5EXHE4AVufG2bNoOkzRLUquk1vb29j7JnJmZba/IoFCpib5Xz/6MiMsioiUiWpqbm3cyW2ZmVk2RQaENmJQbnwisLHB9Zma2k4oMCguAqZIOktQEzATmFrg+MzPbSYUFhYjYCpwP3AQsAeZExCJJF0s6HUDSsZLagDOA70haVFR+zMyse4XepxAR84B5ZdMuzL1fQKpWMjOz3YC7PDQzsw4OCmZm1kGxo31/1JmkduCRHfjIeGB1QdnZnXm7G0+jbru3u2cOjIhur+nvd0FhR0lqjYiWeudjV/N2N55G3XZvd99y9ZGZmXVwUDAzsw6NEBQuq3cG6sTb3Xgaddu93X1owLcpmJlZzzVCScHMzHrIQcHMzDoM6KAgabqkpZKWSZpd7/wURdIkSb+VtETSIkkfzqbvLelmSX/NXveqd16LIGmwpLsl3ZCNHyTpjmy7f5h1yDigSBor6XpJD2T7/RWNsL8lfTT7jd8v6VpJwwfi/pZ0haQnJd2fm1Zx/yr5j+w4t1DSy3Zm3QM2KOQeB3oKcDhwlqTD65urwmwFPh4RhwHHAx/ItnU28JuImAr8JhsfiD5M6nSx5EvA17Ptfgp4T11yVaxvAL+MiGnAUaTtH9D7W9IE4ENAS0QcQXr2+0wG5v7+PjC9bFq1/XsKMDUbZgHf3pkVD9igQO5xoBGxGSg9DnTAiYjHI+LP2fuNpAPEBNL2XpkluxJ4U31yWBxJE4E3Apdn4wL+Frg+SzLgtlvSnsBrgO8BRMTmiFhHA+xvUieeIyQNAfYAHmcA7u+I+D2wtmxytf07A7gqkvnAWEn793bdAzko9NnjQPsTSVOAY4A7gH0j4nFIgQPYp345K8wlwKeA0nO+xwHrsq7bYWDu94OBduC/s2qzyyWNZIDv74h4DPgq8CgpGKwH7mLg7++Savu3T491Azko9NnjQPsLSaOAHwMfiYgN9c5P0SSdCjwZEXflJ1dIOtD2+xDgZcC3I+IYYBMDrKqokqwOfQZwEHAAMJJUdVJuoO3v7vTpb34gB4WGehyopKGkgPCDiPhJNnlVqRiZvT5Zr/wV5JXA6ZIeJlUP/i2p5DA2q16Agbnf24C2iLgjG7+eFCQG+v4+GXgoItojYgvwE+AEBv7+Lqm2f/v0WDeQg0LDPA40q0f/HrAkIr6WmzUXeHf2/t3Az3Z13ooUEZ+OiIkRMYW0f2+JiHcAvwXemiUbiNv9BLBC0ouzSScBixng+5tUbXS8pD2y33xpuwf0/s6ptn/nAmdnVyEdD6wvVTP1xoC+o1nS35POHAcDV0TEF+qcpUJIehXwB+A+OuvWLyC1K8wBJpP+UGdERHnj1YAg6UTgExFxqqSDSSWHvYG7gXdGxPP1zF9fk3Q0qXG9CVgOnEs6yRvQ+1vSvwBvI11xdzfwXlL9+YDa35KuBU4kdY+9Cvg88FMq7N8sQP4n6WqlZ4BzI6K11+seyEHBzMx2zECuPjIzsx3koGBmZh0cFMzMrIODgpmZdXBQMDOzDg4K1nAkPZ29TpH09j5e9gVl43/qy+WbFc1BwRrZFGCHgkLW+24tXYJCRJywg3kyqysHBWtkXwReLemerJ/+wZK+ImlB1i/9P0G6MS57XsU1pBsEkfRTSXdlffvPyqZ9kdSD5z2SfpBNK5VKlC37fkn3SXpbbtm35p6N8IPsZiQkfVHS4iwvX93l3441pCHdJzEbsGaT3QUNkB3c10fEsZKGAX+U9Kss7XHAERHxUDb+j9ndpCOABZJ+HBGzJZ0fEUdXWNdbgKNJzz4Yn33m99m8Y4CXkPqr+SPwSkmLgTcD0yIiJI3t8603q8AlBbNOf0fqQ+YeUhch40gPLgG4MxcQAD4k6V5gPqkzsqnU9irg2ojYFhGrgN8Bx+aW3RYRLwD3kKq1NgDPAZdLegup+wKzwjkomHUS8MGIODobDoqIUklhU0ei1M/SycArIuIoUn87w3uw7Gry/fRsA4Zkzwc4jtTz7ZuAX+7Qlpj1koOCNbKNwOjc+E3A+7NuyJF0aPbwmnJjgKci4hlJ00iPQC3ZUvp8md8Db8vaLZpJT067s1rGsmdjjImIecBHSFVPZoVzm4I1soXA1qwa6Puk5x5PAf6cNfa2U/nRjr8E3idpIbCUVIVUchmwUNKfs268S/4XeAVwL+kBKJ+KiCeyoFLJaOBnkoaTShkf7d0mmu0Y95JqZmYdXH1kZmYdHBTMzKyDg4KZmXVwUDAzsw4OCmZm1sFBwczMOjgomJlZh/8PQMGGn19LgvEAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# gradient descent\n",
"\n",
"# hyber-parameters\n",
"alpha = 0.1;\n",
"number_of_iterations = 100\n",
"\n",
"# data arrays\n",
"cost_array = []\n",
"interation_array = []\n",
"\n",
"# rename\n",
"X = validation_training_images\n",
"Y = validation_training_labels\n",
"m = size\n",
"\n",
"# main loop\n",
"for interation in range(1, number_of_iterations):\n",
" \n",
" # forward propogation\n",
" Z = np.dot(np.transpose(W), X) + b # Linear Model # Z is 1X500 # broadcasting is used\n",
" A = 1/(1 + np.exp(-Z)) # sigmoid(Z) # A is 1X500 # broadcasting is used \n",
" term1 = np.multiply(-Y, np.log(A)) # -Y * log(A) # broadcasting is used \n",
" term2 = np.multiply((1 - Y), np.log(1 - A)) # (1-Y) * log(1-A) # broadcasting is used \n",
" L = term1 - term2 # loss function # l is 1X500\n",
" cost = np.multiply(1/m, np.sum(L)) # cost # scalar\n",
" \n",
" # storage for plotting\n",
" cost_array.append(cost)\n",
" interation_array.append(interation)\n",
" \n",
" # backpropogation\n",
" dZ = A - Y # dZ is 1X500\n",
" dW = np.multiply(1/m, np.dot(X, np.transpose(dZ))) # dW is 784X1\n",
" db = np.multiply(1/m, np.sum(dZ)) # db is a scalar\n",
" \n",
" # paramter updates\n",
" W = W - np.multiply(alpha, dW)\n",
" b = b - np.multiply(alpha, db)\n",
" \n",
" \n",
"# plot learning curve\n",
"pyplot.figure()\n",
"pyplot.plot(interation_array, cost_array, 'red')\n",
"pyplot.title('Learning Curve - 500 MNIST Validation Training Data Set')\n",
"pyplot.xlabel('Iterations')\n",
"pyplot.ylabel('Cost')\n",
"pyplot.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As shown, our validation set worked, so now we can move on to the full data set.\n",
"\n",
"First, we need to split up our full data set into testing and training data. We will use 50,000 images as the training data set and 10,000 images as the testing data set. "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(784, 50000)\n",
"(50000,)\n",
"(784, 10000)\n",
"(10000,)\n"
]
}
],
"source": [
"# separate the data set\n",
"nbr_of_testing_images = 10000\n",
"nbr_of_training_images = 50000\n",
"nbr_of_images = nbr_of_testing_images + nbr_of_training_images\n",
"\n",
"training_images = []\n",
"training_labels = []\n",
"testing_images = []\n",
"testing_labels = []\n",
"\n",
"for index in range(0, nbr_of_images):\n",
" if(index <= nbr_of_training_images - 1):\n",
" training_images.append(normalized_scaled_images_feature_matrix[:, index]) \n",
" training_labels.append(binary_labels[index])\n",
" else:\n",
" testing_images.append(normalized_scaled_images_feature_matrix[:, index]) \n",
" testing_labels.append(binary_labels[index])\n",
" \n",
"# covert to numpy array\n",
"training_images = np.transpose(np.array(training_images))\n",
"training_labels = np.array(training_labels)\n",
"testing_images = np.transpose(np.array(testing_images))\n",
"testing_labels = np.array(testing_labels)\n",
"\n",
"# logger\n",
"print(training_images.shape) # training_images is a matrix of 784 X 50000\n",
"print(training_labels.shape) # testing_labels is a row vector of 1 X 50000\n",
"print(testing_images.shape) # training_images is a matrix of 784 X 10000\n",
"print(testing_labels.shape) # testing_labels is a row vector of 1 X 10000"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we must reset out weights and bias's."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(784,)\n"
]
}
],
"source": [
"# initialize weights & bias\n",
"initial_value = .1\n",
"W = np.multiply(initial_value, np.ones(size))\n",
"b = initial_value # b will be added in a broadcasting manner\n",
"\n",
"print(W.shape) # W is a vector with a size of 784"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we re-run gradient decent on the full data set.\n",
"\n",
"We will also collected data on the accuracy of our network as a function of training iterations. To do this we will need to find the number of inaccuracate binary classifications (false positives & false negatives). This will be acommplished used our test data set. We will send our test data set through the network and compare the results with the true labels of the test data set. "
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Final Cost: 7.367329980635536\n",
"Final Percent Accuracy: 96.4\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm8VHX9x/HXB5DLduECFxSVRRNcMEVE01yy1MIt6ZeVZonlUlqpafnT/GVmv19Z+TPN+rlUprnvu5akFi4JoiyComBuKKuIXBZB4PP74/sd7zDcmTt3mTlz57yfj8d5zMxZ5nzOnJn5nO/3e873mLsjIiLp1SnpAEREJFlKBCIiKadEICKSckoEIiIpp0QgIpJySgQiIimnRJAgM3vYzMYnHYdIa5jZAWY2q73nlfJLZSIws9fN7KCk43D3Q9z9ulK8t5n1NrNLzexNM1thZnPj6/pSrK89mdkFZvZhjDszbJs1fZSZPWdmq+LjqALv1c/M7jazlWb2hpl9NWf6V+P4lWZ2j5n1K3bZJmJ2MzstZ/wZcfwF8fUB8fXvc+Z70syOj8+PN7Mns6bta2ZPm9n7ZrbUzJ4ysz3M7EdZn88HZrY+6/WsnPffL2vayhhD9uc7JN+25ePu/3D3ke09b0vFz+4DM2sws+VmNsXMzjazrkUu3yV+HsPaEMP+ZvavrH30pJmNLse620MqE0E5mFmXBNfdFXgUGAmMBXoDnwTeBfZsxfslsS23unuvrOHfMZauwL3ADUBf4Drg3gI/+t8Da4HNgWOBK8xsZHyvkcBVwNfj9FXA/xWzbB6vALklvOPi+GwrgeOK+fGbWW/gAeByoB+wFfBTYI27/zzz+QDfBv6V9XltFKe7P5E1b2ZaXdb8b+ast5OZdaT/h2+7ey2wJXA28DXgATOzUq/YzPoC9wGXEL6TWwP/TfjudAzunroBeB04KM+0w4FpwDLgaWCXrGnnAK8CDcCLwBeyph0PPAX8BlhK+CIcDzwJXAy8B7wGHJK1zD+AE7OWLzTvNsDEuO6/E/6kbsizDScCC4FeBT4DB7bLen0t8N/x+QHAPOA/gQXA9cBLwOFZ83cBlgCj4+u94ue1DJgOHNCG/XNBgW37LPA2YFnj3gTGNjFvT8KPcUTWuOuBi+LznwM3ZU37WJy/trll88UcP6eRcdzI+PoG4IKcz/Zy4M9Zyz8JHJ/9XYjPxwDLivjMPlqmiHmHxf3fJWf8k8DPgH8Bq+N8J8ZtaCB890/Mmv8g4PWs1/OAM4EXgPeBm4Gals4bp58bv3tvAyfFeIfl2Z6PPruc38vqzPcC2Bt4Jn4/5wO/BTaL056O778SWAF8EegPPAQsJvwe7we2yrP+vYAlzXzmJwKz43s9DAzOt+7W/m7aMnSkjF9ysSh3DfAtwhfhKuA+M6uJs7wK7Af0IRyV3WBmg7Le4hPAv4GBwP9kjXsZqAd+BfypwFFKoXlvAibHuC4gHMXmcxDwV3df0fxW57UF4Qh0KHAy4Yd6TNb0zxG+/M+b2VbAg4Tk1w/4AXCnmQ1ow/qPiEXsWWZ2Stb4kcAMj7+iaAaNR7nZRgDr3T37iHx61rwj42sA3P1V4p9/Ecvmcz2hFAChdPCXPPP9D/BFM9u+mfd7BVhvZteZ2SHx6LOUvg58k1CKnEc4oDgsvj4JuNzMdimw/JeBg4Ftgd0p/D1tcl4zOxz4HvBpwn74TEs3wt1fA6YSfq8A64DTCb+tfQgl5W/FafvHx5EeSkd3EmpL/gAMIfwGPgQuy7O6l4HOZvZnMxtrZnXZE83sKOCHwJHAAGAS4fecb91lp0SwsZOAq9x9kruv91B/v4aQ8XH32939HXff4O63AnPYuKrlHXe/3N3XufvqOO4Nd/+Du68nVGMMIlQ1NKXJeWP97R7A+e6+1t2fJBRF8+lPOOppiw3AT9x9TdyWm4DPm1mPOP2rNH6ZvwY85O4Pxc9mAjAFOLSV674N2JHwozkJON/MMkmoF+EIMtv7hKP4XM3NW2h6S9aT7QbgGDPbDDg6vt6Euy8ArgQuLPRm7r4c2Jdw1PgHYLGZ3Wdm+b5DbXWNu7/k7h/G7/H97v5vDx4jVDnuV2D5S919gbu/S6jSytt+U2DeLwN/inGsJBx0tcY7hAMT3P3Z+Lte56Ga8WrgU/kWdPfF7n63u6+O++Dn+eZ39/cI+6gT8CfCPron60DoW8DP3f1ld19HOGDaMx5AVQQlgo0NBc4ys2WZARhMqHfEzI4zs2lZ03YmHGFkvNXEey7IPHH3VfFprzzrzzfvlsDSrHH51pXxLiGJtMVid/8gK565hCqCI2Iy+DyNiWAo8KWcz23fpmIws2OzGigfbmrF7v5iTLjr3f1pwpHYUXHyCsLRabbehKqLXM3NW2h6S9aTHfubwFzCH8ccdy+0n34JfM7Mdm3mPV9y9+PdfWvCd25L4NJCy7TBRvGa2eFmNimWzpYRquYKnXCwIOv5KvJ/1wvNu2VOHIU+w0K2IlTTYmY7mNmDZrbAzJYTEnDe7TCznmb2RwsnWywHHis0v7vPcvfx7r4VsAuhJHFJnDwU+H3Wb2MJ4UBr61ZuV7tTItjYW8D/uHtd1tDD3W82s6GEI7LvAv3dvQ6YCWRX85SqK9f5QL+so3EICSqfvxP+YHoWmGcVkP1+W+RMb2pbMtVDRwIvxuQA4XO7Pudz6+nuF+W+gbvf6I0NlIcUiC83lsznPAvYJad6bZc4PtcrQBczG541bteseWfF1wBYODOpJi7X3LKF/AU4i/zVQgDEI+FLCfXyRXH32YT2nJ2LXaaFPtrvZtYduAP4BbB5/M4/wsbf+VKYz8Z/koW+602KDfGjgCfiqKsIv9ft3L03cD6N29HUd/1sQjvDnnH+oqun3P0lwr7P7KO3gBNyfh/d3X1SnnWXXZoTwWZm1i1r6EL4o/+2mX3Cgp5mdpiZZRoPndB4hJl9g9L9GDfi7m8QqlouMLOuZrY3cESBRa4nfPnujEdCncysv4XTDTPVNdOAr5pZZzMbS4FicpZbCEeEp9BYGoBQ/XGEmX0uvl83C6dJtuqIx8yONLO+cR/sCZxGOFMIQgP7euA0M6sxs+/G8Y/FZY83s9cBYrXCXcCFcV/uQ0hi18dlboxx7xeT5oXAXe7eUMSyhdxK+JxuK2LeSwhndO2Y57PYwczOynyWZjaYkIyfKeK926oG6Er4zq+PdfcHlmG9twEnmNn28eDnx8UuGPfVAcA9hJM3/hYn1RKq9laa2Y40tg8Qq2LfJbRVkDX/KuA9M+tPSBz51rmTmZ2ZqeqJVblH07iPrgTOi+vFzOpiu0G+dZddmhPBQ4SzCjLDBe4+hVAn/TtC6/5cwtkYuPuLwP8SzqhYCHyc8EUrl2MJZz68S6hjvJXQfrEJd19DaDCeDUwAlhMamusJDVUQGs6OIJxFcSzhh1OQu88nbP8n4/oz498i/En+iPCn8Rahcay136+jCZ99A+HI6pexvQZ3XwuMIzTILiM0bI6L4yEcPWbvl1OB7sAiQonmFHefFd9rFuG0yxvj9No4f7PLFhLrlf+e1U5UaN7lhBMD+uWZpYFwEsEkM1tJ+HOZSShxlJS7LwO+D9xNqGI5ilCXX+r13g9cQThLbg6N+7PJ73t0pZk1EKqbLiF8Pw9z/+ikgrMIjfcNhNLBrTnL/wS4KVbf/Ed8jz6E39vThDN98mkg/DafjfvoacKB1tlxe26P73d7rGaaQTjZIt+6y87cK6JkIi1kZrcCs939J0nHUknM7BHg9Fg8lypgZh8HniecXroh6XiqkRJBB2FmexCOyl4jVDvcA+zt7lMTDUykBMzsC4RTkmsJpcLV7n5U4aWktdJcNdTRbEGoH19BuBjmFCUBqWLfIZxdMwf4IL6WEilZicDMriFcpbvI3XeO40YRGk66ES7wONXdJ5ckABERKUopSwTXEq7ey/Yr4KfuPorQCv+rEq5fRESKULLOxNx9om3aqZbTeJFOH8KVf82qr6/3YcNy30pERAp57rnnlrh7s129lLtXyTOAv5nZxYTSyCfzzWhmJxP6uGHIkCFMmTKlPBGKiFQJM3ujmPnK3Vh8CvB9dx9MOD/5T/lmdPer3X2Mu48ZMKAtfZeJiEgh5U4E4wlXawLcTiv6xhcRkfZV7kTwDo1dGXyGcGqYiIgkqGRtBGZ2M+EmHPVmNo9wGfVJwGWxX58PiG0AIiKSnFKeNXRMnkm7l2qdIiLScrqyWEQk5ZQIRERSrroTwQMPwEWb3BtFRESyVHci+Nvf4FfqxUJEpJDqTgS1tdDQAOpqW0Qkr+pOBL16wbp1sHZt8/OKiKRU9ScCgBUrko1DRKSCKRGIiKRcdSeC2trwqEQgIpJXdSeCTImgoSHZOEREKlg6EoFKBCIieSkRiIikXHUnArURiIg0q7oTgdoIRESalY5EoBKBiEhe1Z0IevQIj0oEIiJ5VXci6NQplAqUCERE8qruRAAhEaiNQEQkr3QkApUIRETyqv5EUFurRCAiUkD1JwKVCEREClIiEBFJuXQkAjUWi4jkVf2JQG0EIiIFVX8i6NULli9POgoRkYpV/Ymgri5UDW3YkHQkIiIVqfoTQd++4K5SgYhIHtWfCOrqwuN77yUbh4hIhUpPIli2LNk4REQqVPUngr59w6NKBCIiTSpZIjCza8xskZnNzBn/PTN72cxmmdmvSrX+j6hEICJSUClLBNcCY7NHmNmngSOBXdx9JHBxCdcfKBGIiBRUskTg7hOBpTmjTwEucvc1cZ5FpVr/R1Q1JCJSULnbCEYA+5nZJDP7p5ntkW9GMzvZzKaY2ZTFixe3fo29eoUb1KhEICLSpHIngi5AX2Av4IfAbWZmTc3o7le7+xh3HzNgwIDWr7FTJ+jTR4lARCSPcieCecBdHkwGNgD1JV9r376qGhIRyaPcieAe4DMAZjYC6AosKfla6+pUIhARyaNLqd7YzG4GDgDqzWwe8BPgGuCaeErpWmC8u3upYviISgQiInmVLBG4+zF5Jn2tVOvMq64O5s8v+2pFRDqC6r+yGFQ1JCJSQDoSQb9+sHRp6IVUREQ2ko5EUF8PH3wAK1cmHYmISMVJRyLIXIfQlgvTRESqlBKBiEjKpSMR1Mdr1paU/pIFEZGOJh2JQCUCEZG8lAhERFIuHYmgtha6dlUiEBFpQjoSgVkoFSgRiIhsIh2JAJQIRETySFci0FlDIiKbSFciUIlARGQT6UoEi0p/i2QRkY4mPYlgiy2goUH9DYmI5EhPIthyy/Co+xKIiGwkPYlg0KDwqEQgIrKR9CSCTIngnXeSjUNEpMIoEYiIpFx6EkFdHdTUKBGIiORITyIwC6UCtRGIiGwkPYkAQiJQiUBEZCNKBCIiKZeuRDBoELz9dtJRiIhUlHQlgsGDYcUKWLYs6UhERCpGuhLBkCHh8c03k41DRKSCpCsRDB0aHt94I9k4REQqiBKBiEjKpSsRDBwYLipTIhAR+Ui6EkGnTqGdQIlAROQjJUsEZnaNmS0ys5lNTPuBmbmZ1Zdq/XkNHarGYhGRLKUsEVwLjM0daWaDgYOBZP6NVSIQEdlIyRKBu08EljYx6TfA2YCXat0FDRsGCxbA6tWJrF5EpNKUtY3AzD4PvO3u04uY92Qzm2JmUxa3503nt9suPL76avu9p4hIB1a2RGBmPYDzgPOLmd/dr3b3Me4+ZsCAAe0XyPDh4XHu3PZ7TxGRDqycJYKPAdsA083sdWBr4Hkz26KMMcDHPhYelQhERADoUq4VufsLwMDM65gMxrj7knLFAEDfvtC/vxKBiEhUytNHbwb+BWxvZvPM7IRSravFtttOiUBEJCpZicDdj2lm+rBSrbtZw4fDE08ktnoRkUqSriuLM7bbLlxUplNIRURSmgh23BHc4ZVXko5ERCRx6UwEO+0UHl98Mdk4REQqQDoTwfDhoQO6l15KOhIRkcSlMxHU1IR2ApUIRERSmgggVA8pEYiIpDgR7LgjzJkDa9cmHYmISKLSmwhGjYJ162DWrKQjERFJVHoTwejR4fH555ONQ0QkYelNBNtuC717KxGISOqlNxF06gS77aZEICKpl95EAKF6aPr00FYgIpJSSgSrV8Ps2UlHIiKSGCUCUPWQiKRauhPB9ttDjx5KBCKSaulOBJ07h+sJlAhEJMWKSgRmdn0x4zqk0aNh6lTYsCHpSEREElFsiWBk9gsz6wzs3v7hJGCPPWDFCl1hLCKpVTARmNm5ZtYA7GJmy+PQACwC7i1LhKW2//7h8Z//TDYOEZGEFEwE7v4Ld68Ffu3uveNQ6+793f3cMsVYWkOHwuDBMHFi0pGIiCSi2KqhB8ysJ4CZfc3MLjGzoSWMq3zMQqlg4sRw+0oRkZQpNhFcAawys12Bs4E3gL+ULKpy239/WLgwdEstIpIyxSaCde7uwJHAZe5+GVBburDKLNNOoOohEUmhYhNBg5mdC3wdeDCeNbRZ6cIqs+23hwEDlAhEJJWKTQRfAdYA33T3BcBWwK9LFlW5ZbcTiIikTFGJIP753wj0MbPDgQ/cvXraCAA+9Sl44w149dWkIxERKatiryz+MjAZ+BLwZWCSmR1VysDKbuzY8PjQQ8nGISJSZsVWDZ0H7OHu4939OGBP4MelCysBw4eHQYlARFKm2ETQyd0XZb1+twXLdhyHHgqPPw4rVyYdiYhI2RT7Z/5XM/ubmR1vZscDDwLVd+h82GGwZk1IBiIiKdFcX0Pbmdk+7v5D4CpgF2BX4F/A1c0se42ZLTKzmVnjfm1ms81shpndbWZ17bAN7Wf//aFnT3jwwaQjEREpm+ZKBJcCDQDufpe7n+nu3yeUBi5tZtlrgbE54yYAO7v7LsArQGX1V1RTAwcdFNoJ1N2EiKREc4lgmLvPyB3p7lOAYYUWdPeJwNKccY+4e+ZO8c8AWxcfapkcfji8+SZMm5Z0JCIiZdFcIuhWYFr3Nq77m8DD+Saa2clmNsXMpixevLiNq2qBcePCnctuvbV86xQRSVBzieBZMzspd6SZnQA819qVmtl5wDrCRWpNcver3X2Mu48ZMGBAa1fVcvX1oXro1ltVPSQiqdClmelnAHeb2bE0/vGPAboCX2jNCs1sPHA4cGDsyK7yHH00fOMb8OyzsOeeSUcjIlJSzd2YZqG7fxL4KfB6HH7q7nvHbidaxMzGAv8JfN7dV7U83DIZNw66doVbbkk6EhGRkiu2r6HH3f3yODxWzDJmdjPhNNPtzWxerE76HaH76glmNs3Mrmx15KVUVxe6nLjtNt3UXkSqXnNVQ63m7sc0MfpPpVpfuzvmGLjvvnBx2YEHJh2NiEjJVF83Ee1l3Djo1w/+8IekIxERKSklgny6dYOvfx3uvhuWLEk6GhGRklEiKOTEE2HtWvhLdd16QUQkmxJBITvvDHvtFaqHKvRMVxGRtlIiaM7JJ8Ps2fCPfyQdiYhISSgRNOfoo8ON7S+5JOlIRERKQomgOd27w6mnwgMPwEsvJR2NiEi7UyIoxqmnhi6qf/ObpCMREWl3SgTFGDgQjjsunD20cGHS0YiItCslgmL94Afw4Yfw618nHYmISLtSIijWiBFw7LHwf/8HC1rc356ISMVSImiJH/84XGD2y18mHYmISLtRImiJ4cNDW8EVV8DbbycdjYhIu1AiaKkf/zhcZXzeeUlHIiLSLpQIWmqbbeD00+G66+C5Vt+tU0SkYigRtMZ554Wrjc88U30QiUiHp0TQGn36wIUXwsSJ4Sb3IiIdmBJBa510Euy+O5xxBixblnQ0IiKtpkTQWp07w9VXw+LFcO65SUcjItJqSgRtMXp0aDi+8kp44omkoxERaRUlgra68ELYdlsYPx4aGpKORkSkxZQI2qpXr3Aq6euvw1lnJR2NiEiLKRG0h333hR/+MNzS8p57ko5GRKRFlAjay4UXhrOIjj8eXn016WhERIqmRNBeamrgjjugUyf44hdh9eqkIxIRKYoSQXsaNgxuuAGmT4fvfCfpaEREiqJE0N4OPRT+67/gz3+G3/8+6WhERJqlRFAKF1wARxwBp50G99+fdDQiIgUpEZRC585w883hgrOjj4Znn006IhGRvJQISqVnT3jggXDj+8MP15lEIlKxSpYIzOwaM1tkZjOzxvUzswlmNic+9i3V+ivC5pvDww/DunXwmc/AG28kHZGIyCZKWSK4FhibM+4c4FF3Hw48Gl9Xtx12gAkTYPly+PSnYd68pCMSEdlIyRKBu08EluaMPhK4Lj6/DhhXqvVXlNGj4ZFH4N13QzLQ/Y5FpIKUu41gc3efDxAfB+ab0cxONrMpZjZl8eLFZQuwZPbYA/76V1i4EPbZB155JemIRESACm4sdver3X2Mu48ZMGBA0uG0j733hscfh1WrQv9EuuexiFSAcieChWY2CCA+Lirz+pO3++7w5JPQo0eoJpowIemIRCTlyp0I7gPGx+fjgXvLvP7KMGIEPPVU6JLikEPgd78D96SjEpGUKuXpozcD/wK2N7N5ZnYCcBFwsJnNAQ6Or9Npq61CMjjsMPje9+CUU+DDD5OOSkRSqEup3tjdj8kz6cBSrbPDqa2Fu++G886Diy6CGTPglltgyJCkIxORFKnYxuLU6NQJfvGLkABmzoRRo+DedNaYiUgylAgqxVe+As8/H+5/PG4cnHEGrFmTdFQikgJKBJVku+1Cu8Hpp8Nll8Gee+oUUxEpOSWCSlNTA5deGjqsW7IEPvEJ+NGP4IMPko5MRKqUEkGlOuwwmDULjjsutCGMHg1PP510VCJShZQIKlldHVxzTejBdMWK0DXF+PGwYEHSkYlIFVEi6AjGjoUXX4Rzzgk3vBkxAi65RNcdiEi7UCLoKHr1ClVEM2eGforOOgt23hluv11XJYtImygRdDQjRsCDD4Z7IW+2GXz5y+Hsor//PenIRKSDUiLoiMzC7S+nT4drr4VFi+Dgg8Nd0B59VCUEEWkRJYKOrHPn0Hj88svwm9/ASy/BQQeF7q7vv18JQUSKokRQDbp1C1civ/YaXHFFuPnN5z8fuqu46SZYuzbpCEWkgikRVJNu3eDb3w53P7vuupAAjj02dHf9s5+FBCEikkOJoBpttlm4EG3WLHjoIdh1Vzj//NCr6fjxMHmyqo1E5CNKBNWsU6dw45uHH4bZs+Fb34K77grdVuy6a2hXqIb7QYtImygRpMX228Nvfwtvvw1XXRVulXnmmeEGOV/8Yjgldd26pKMUkQQoEaRN795w8snwzDPh4rTTToMnngino269dbhb2lNPwYYNSUcqImWiRJBmI0fCxReHUsJdd4W+jP74x3Dl8tCh4epltSeIVD0lAgmNy1/4Atx5Z7g47YYbYLfd4PLLQ3vCttuGeyQ8+qj6NxKpQuYd4GhvzJgxPmXKlKTDSJ/33oN77gkJ4tFHwz0R+vQJDdBHHBEe+/ZNOkoRycPMnnP3Mc3Op0QgRVm5MvRndN994aY5ixaFK5v32w8++9kw7LZbOFNJRCqCEoGUzoYNoe3g/vtDUpgxI4zv3x8OPDD0e3TwwaGdQUQSo0Qg5bNwYSgtTJgQhnfeCeOHD4dPfQr23z8MSgwiZaVEIMlwD53fTZgQksOTT8KyZWHa4MGNSWG//WCHHUJPqiJSEkoEUhk2bAjXK0yc2Dhk+jwaMAD22ivcT+ETn4A99gi35xSRdqFEIJXJHebODQnhiSdg0qTQ/UXG9tuHpJBJDrvsAl27JhevSAemRCAdx7JlMGVKSAqTJ4fHTKmhpiYkg1GjwrDbbvDxj4dbd4pIQUoE0nG5w1tvNSaGqVPDsHRpmG4WGqIzySGTILbYItm4RSpMsYmgSzmCEWkRs9Bl9pAh8KUvhXHuMG8eTJsWksK0afDss3DbbY3L1deHbjN22ik8Zp4PHJjMdoh0EIkkAjP7PnAi4MALwDfc/YMkYpEOwiycdTR4cLiqOWPZsnDv5qlTQ6P0iy/CjTfC8uWN89TXb5ocdtwRNt9cZy2JkEDVkJltBTwJ7OTuq83sNuAhd7823zKqGpIWcQ/XMsyaFRJD9uP77zfOV1sbqphGjAiP2c/79UsufpF2UulVQ12A7mb2IdADeCehOKQamYX7LGy1Vej6IsMd5s8PCWH2bJgzJ9zWc/LkUMWU3fV2//6bJodttglDfb1KElJVyp4I3P1tM7sYeBNYDTzi7o+UOw5JITPYcsswHHzwxtPWrIHXXguJIZMg5syBxx6D66/feN6ePRuTQlNDbW35tkmkHZQ9EZhZX+BIYBtgGXC7mX3N3W/Ime9k4GSAIUOGlDtMSZuamnCl8w47bDpt5Ur4979DosgdHn8cVqzYeP7+/UPX3dts09iukT0MHKjO+aSiJNFG8CVgrLufEF8fB+zl7qfmW0ZtBFKx3OHdd5tOEq+9Fk6D/SDnPIjNNgt3g2sqSWSGfv1U/SRtVsltBG8Ce5lZD0LV0IGA/uWlYzILbQb19aGLjFyZRPHWW00PTz0VTovNvV909+6N1ViDBm36PPPYu7cShrRZEm0Ek8zsDuB5YB0wFbi63HGIlEV2othtt6bn2bAhXEmdmyTeeSc0bk+dCg8+GKqocmUSRm6CGDQoXGC3+eahKqq+HrrosiFpmq4sFukoGhpCYsgkiHyPDQ2bLmsW2i4GDmxMDtnPc8f17Fn+7ZN2V8lVQyLSGrW1YRgxovB8K1aEpLBwYbiT3KJFjc8zj1OnhufZ11Vk69Fj4+TQv38oVWQes5/37x/aNFTi6LC050SqTa9eIVk0lzAgnDabSRbZiSL7+VtvhS49liyB1avzv1ddXdNJoqkE0rdvGLp3b7/tllZTIhBJs5qaxjOVirFqVWj8XrJk08fs5/PnwwsvhOerVhVefyYp9OvX+LzQkJmvW7f2+QxEiUBEWqBHjzAUmzgglCLefXfjhPHee00P8+aFBPLeexv3F9WUbt2aThR1ddCnT+PQu/fGrzNDz5464ypSIhCR0urePVw3sfXWLVtu3brQhpFJEkuX5k8gS5eGKqwZM8Iyy5eHU3cL6dQpf5IolECyp9fWQufOrf9sKoQSgYhUpi5dQntC//4tX3bDhtBo/v77jYkh8zx3yJ42b14xFHmTAAAHP0lEQVRj54Tvvw/r1ze/ru7dGxvyc4dMsih26NEjkVKKEoGIVJ/M0X7v3i2rxsrmHqq1CiWQhoamhwULQl9VmddNXQOSL+7a2tDgX1MThquugv32a902FEmJQESkKWaNbSKDBrXtvdavDyWUfImjqWHt2jD07t0+21OAEoGISKl17tzYtlCB1AWiiEjKKRGIiKScEoGISMopEYiIpJwSgYhIyikRiIiknBKBiEjKKRGIiKRch7hDmZktBt5oxaL1wJJ2Dicp2pbKUy3bAdqWStXWbRnq7gOam6lDJILWMrMpxdymrSPQtlSeatkO0LZUqnJti6qGRERSTolARCTlqj0RXJ10AO1I21J5qmU7QNtSqcqyLVXdRiAiIs2r9hKBiIg0Q4lARCTlqjYRmNlYM3vZzOaa2TlJx9MSZva6mb1gZtPMbEoc18/MJpjZnPjYN+k4m2Jm15jZIjObmTWuydgt+G3cRzPMbHRykW8qz7ZcYGZvx30zzcwOzZp2btyWl83sc8lEvSkzG2xmj5vZS2Y2y8xOj+M73H4psC0dcb90M7PJZjY9bstP4/htzGxS3C+3mlnXOL4mvp4bpw9rt2DcveoGoDPwKrAt0BWYDuyUdFwtiP91oD5n3K+Ac+Lzc4BfJh1nntj3B0YDM5uLHTgUeBgwYC9gUtLxF7EtFwA/aGLeneL3rAbYJn7/Oie9DTG2QcDo+LwWeCXG2+H2S4Ft6Yj7xYBe8flmwKT4ed8GHB3HXwmcEp+fClwZnx8N3NpesVRriWBPYK67/9vd1wK3AEcmHFNbHQlcF59fB4xLMJa83H0isDRndL7YjwT+4sEzQJ2ZtfHmsO0nz7bkcyRwi7uvcffXgLmE72Hi3H2+uz8fnzcALwFb0QH3S4FtyaeS94u7+4r4crM4OPAZ4I44Pne/ZPbXHcCBZmbtEUu1JoKtgLeyXs+j8Jel0jjwiJk9Z2Ynx3Gbu/t8CD8GYGBi0bVcvtg76n76bqwyuSariq5DbEusTtiNcPTZofdLzrZAB9wvZtbZzKYBi4AJhBLLMndfF2fJjvejbYnT3wf6t0cc1ZoImsqSHek82X3cfTRwCPAdM9s/6YBKpCPupyuAjwGjgPnA/8bxFb8tZtYLuBM4w92XF5q1iXGVvi0dcr+4+3p3HwVsTSip7NjUbPGxZNtSrYlgHjA46/XWwDsJxdJi7v5OfFwE3E34gizMFM/j46LkImyxfLF3uP3k7gvjj3cD8AcaqxkqelvMbDPCH+eN7n5XHN0h90tT29JR90uGuy8D/kFoI6gzsy5xUna8H21LnN6H4qsuC6rWRPAsMDy2vnclNKzcl3BMRTGznmZWm3kOfBaYSYh/fJxtPHBvMhG2Sr7Y7wOOi2ep7AW8n6mqqFQ5deVfIOwbCNtydDyzYxtgODC53PE1JdYj/wl4yd0vyZrU4fZLvm3poPtlgJnVxefdgYMIbR6PA0fF2XL3S2Z/HQU85rHluM2Sbjkv1UA48+EVQp3beUnH04K4tyWc5TAdmJWJnVAX+CgwJz72SzrWPPHfTCiaf0g4gjkhX+yEou7v4z56ARiTdPxFbMv1MdYZ8Yc5KGv+8+K2vAwcknT8WXHtS6hCmAFMi8OhHXG/FNiWjrhfdgGmxphnAufH8dsSktVc4HagJo7vFl/PjdO3ba9Y1MWEiEjKVWvVkIiIFEmJQEQk5ZQIRERSTolARCTllAhERFJOiUBSwcxWxMdhZvbVdn7vH+W8fro931+k1JQIJG2GAS1KBGbWuZlZNkoE7v7JFsYkkiglAkmbi4D9Yp/134+dfv3azJ6NHZZ9C8DMDoj93t9EuFAJM7sndgQ4K9MZoJldBHSP73djHJcpfVh875kW7i/xlaz3/oeZ3WFms83sxkwvkmZ2kZm9GGO5uOyfjqRSl+ZnEakq5xD6rT8cIP6hv+/ue5hZDfCUmT0S590T2NlD98UA33T3pbE7gGfN7E53P8fMvuuh47Bc/0HoBG1XoD4uMzFO2w0YSehH5ilgHzN7kdA9wg7u7pnuB0RKTSUCSbvPEvrVmUbozrg/oT8agMlZSQDgNDObDjxD6PxrOIXtC9zsoTO0hcA/gT2y3nueh07SphGqrJYDHwB/NLP/AFa1eetEiqBEIGlnwPfcfVQctnH3TIlg5UczmR1A6BRsb3ffldBHTLci3jufNVnP1wNdPPQxvyehZ81xwF9btCUiraREIGnTQLjFYcbfgFNi18aY2YjY62uuPsB77r7KzHYgdBec8WFm+RwTga/EdogBhFtf5u35Mvax38fdHwLOIFQriZSc2ggkbWYA62IVz7XAZYRqmedjg+1imr4N6F+Bb5vZDEIvls9kTbsamGFmz7v7sVnj7wb2JvQk68DZ7r4gJpKm1AL3mlk3Qmni+63bRJGWUe+jIiIpp6ohEZGUUyIQEUk5JQIRkZRTIhARSTklAhGRlFMiEBFJOSUCEZGU+3+HfZCVOSvbtAAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcXFWZ//HPNyEhARICIWFNCEtYwxYigkBUcAPZAoqICoiAOCDgMjP81FFEUVDBcVAREBhE9kVgRJawKmjAgAQSEvYlhJA0WchG9uf3xzlFKp1eqpO+Vd1d3/fr1a+quutz+1bd555zzz1XEYGZmdWvbrUOwMzMasuJwMyszjkRmJnVOScCM7M650RgZlbnnAjMzOqcE4GZWZ3r1IlA0muS3pM0T9I0SVdJWq/WcZXLMX6sgum2krRc0m+rEVetSNpU0hWSpkqaK2mSpB9KWrfWsbVG0tGS/i5pgaSHmxi/u6Qn8/gnJe3ewrI2lPQnSfMlvS7p2Ebjj83D50u6XdKGlc7baDnnSApJZzQaflYefk7+/JH8+TeNpntU0gn5/QmSHi0bt1/+f7wraaakxyR9QNJ38m9ynqSFkpaVfZ7QaPn7l42bn2OYV/Y3uLlta4mkXnlZW7QwzamSlpat6xVJv5e0TRvWc4Ok761OjHn+/pL+kI9fc/Lv4RvVWHe5Tp0IskMjYj1gOPABoM3/GElrtXtUbXccMAs4RtLa1VxxtbY/H8z+AfQG9omIPsDHgX5AxT++suVVe7/NBP4bOL+JWHoCdwB/BDYArgbuyMOb8htgMbAx8AXgEkk752XtDFwKfCmPXwD8tpJ5m/ECcHyjYcfl4eXmA8dJGtLCssgx9gX+DFwMbAhsDvwQWBQRP4mI9fLv8lTgH6XPEbFSnBHxt7JpS+P6lU3/RmuxrKGH87rXBz6Zhz0pafuC11vya0DAdqTfwZHAa1Va9woR0Wn/8j/sY2Wffw78Ob9fH7gCmApMAX4MdM/jTgAeA35J+nH/OA8/GZgIzAWeA4bn4ZsBtwINwKvAGWXrPAe4CfhDnm8CMCKPuwZYDrwHzAP+o4VteRn4GjAN+EyjcTsDo3Os04Dv5OHdge/keecCTwKDgCFAAGuVLeNh4KTmtp90IH4QmAG8A1xL+kGW5h8E3Jb/BzNIX+C18/y7lE03MG/vgCa28cfAs0C3Zv4HbY37p8BsYFjZ9APy+gfmz4cAT+fp/g7s2g7fu5NIB5DyYZ/I3zOVDXsD+FQT869LOpBvVzbsGuD8/P4nwHVl47bJ0/dpbd4m1nUOKTlNBHYu+z5NzMPPycM+ArxJOrBfVTb/o8AJZf//R/P7EcDsCv5X789TwbSr7P88fEPS7+ttYDLwg9J3CNghx/hu/m7+IQ9/Ii9rPum3d0QT6zsVuL+J4fcDf8zv1yL99qfl79BDwPZ53BnAEmBRXsfNefj3SceJucB44NMtbPNLTX1HysYPI/0uZ+V9dkRL617dv65QIgBA0iDgYOBfedDVwFJgW2AP0g/1pLJZPgi8QjpwnSfps6QfzXFAX+AwYIakbsD/AeNIZz0HAmdJ+mTZsg4DbiBl9DtJB0ki4kukg8Ghkc5uftZM7PsDW+Rl3JRjKI3rQ/pi3kNKSNsCD+TR3wQ+n7e7L3Ai6eyxEittP+ms5Kd5HTuSDvzn5Bi6k87+Xif9WDcHboiIRTnmL5Yt9/OkH1dDE+v8GHBbRCyvMMbW4j6XlJw+Xzb+aOCRiJguaThwJfBVoD/pLPvOgkpcOwPPRP6VZs+w4iy33HbAsogoPyMfVzbtzvkzABHxMvngX8G8zbmGFd+r40kH1qacBxxVwRnxC8AySVdLOkjSBq1MvyauJR3otwb2Ao4glZYgfWdvJ/32BpP2McDI/Lp9/u3d3ob13QbsX/b5TlIy3gSYRDq2EBH/Q0oSP8rr+Gye/nngQ6ST0QuAGyRt1My6xgAXSDpe0rblI3KpazTphHYj0v67UtK2Lax7tXSFRHC7pNmks4JHgJ9I2hg4CDgrIuZHxHTSWeQxZfO9FREXR8TSiHiPlCR+FhH/jOSliHidVN00ICLOjYjFEfEKcHmjZT0aEX+JiGWkH9xubdyG44G7I2IWcB1wkKSBedwhwNsRcWFELIyIuRHxeB53EvC9iHg+xzwuImZUuM6Vtj9v7+iIWJQP4hcBH87T7kVKEP+e/58LI6JUV3w1cGxOmJB+oNc0s87+pBLammi8365j5URwbB4GqYR3aUQ8HhHLIuJq0hnU3msYQ1PWIx2syr1LOotv67QtjW/Lesr9Efi8pB6k7+4fm5ooIt4GfkdKss2KiDnAfqSz7suBBkl35t9eu5G0Jemg/s2IWBARU4H/YcXvbwnp5GST/D1+rB1W+xapFEL+nl0dEfMiYiGp+msvSb2amzkiboyIqRGxPCKuIZUU92xm8q+SDujfACZJel4rrimOAsZHxLX5+/tP0knpUe2wjSvpCongiIjoFxFbRsS/5YPDlkAPYKqk2TlRXEo6iyyZ3Gg5g0hVLI1tCWxWWk5e1ndI9bMlb5e9XwD0qrT+WlJv4LOksx4i4h+kUkTpAmBzcbU2rjUrbb+kgfni0xRJc0gHitJZzCDg9YhY2nghOSnNBz4saQdSieXOZtY5A9h0NeNtMm5Ssbm3pA/mg8buwJ/yuC2BbzXad4NISW0ljS5w/m414ppHKpWV60uqHmjrtC2Nb8t63heprv0lUrXTixHR+P9Y7gLgk5JaPKGJiIkRcUJEbEGqwtiMdA2lPW0J9CIlmtI+/BUrfn/fANYB/iXpGUlfbGY5bbE5qeoRSWtJ+kW+kDyHVCIQ6aSmSZK+kmMpxbstK35LK8knVudGxO55mf8H3JprArYERjb6/h7Fmv+GVtEVEkFTJpPO/DbKSaJfRPSNlS9UNe52dTJNX7CcDLxatpx+EdEnIg6uMJbWuncdRfoh/1bS25LeJn0RS8X45uJqadz8/LpO2bBNWonrp3nYrhHRl1Tdo7L1DG4huV2dp/8ScEs+c2rK/cCostLDGsedq5luIpUKjiVdIyodFCcD5zXad+tExPWNVxxlFzgj4tRm4mvJBGBXSSobtmse3tgLwFqShpYN261s2gmUlSolbU26HvNCBfO25A/At2i+WgiAXKr8b+BHFSyzNM8k4H9JCaE9TSYlvw0a/ZaH5/VOiYgTSQfHM0hVJ4Np/XfXkiOAv+X3XyZVK3+UVNWzQx5e2s8rrUfSdqTrLKcAG0ZEP1ICLv9eNCki3iU1ROhLquaaDNzX6Pu7XkSc1dS610SXTAS5+HgfcKGkvpK6SdpG0odbmO33wLcl7alk23yG+QQwR9J/SuotqbukYZI+UGE400h1m805nlSPvQvpbHZ3YF9gd0m7kOrmN1Fq7re2pD6SPlgW848kDc0x7yqpf67amQJ8Mcd7Iq23yulD+sHNlrQ58O9l454gVemcL2ldpaZ5+5aNv4aU0L5IyweZi0hf8qvz/xZJm0u6SNKuqxk3pKqgz5Fa0FxXNvxy4NRcWlCO/dP5bKvNcky9SBcQu+X/Q488+mFgGXBG3k+n5+EP5nlPkPQapLNAUj30uTmmfYHDWVGldi1wqFLTynXJ10JytWBr87bkRtJB7aYKpr2IVM+9YzP/ix0kfUu5eabSNbrPk+q8201EvJqX+bP83e+Wv+/75fV+TtJmERGki7kASyNdvypdV2hV3rfbSLqUVBX64zyqD7CQVJpdt2x4SePf93qkBiINpO/IqaQSQXPrPUfScEk9cu3AGaTGGi+Rrn3skbexh6SekvbOyaapda+2LpkIsuOAnqTWP7OAW2ihSBURN5MulF1HKmbfTsroy4BDSQfoV0k76feks4NK/BT4Xi7afbt8RD7gHgj8d0S8Xfb3JOni8PH57PbjOYa3gRdJZyeQfqw3kZLeHNJFpd553Mmkg/kM0oXEv7cS5w9JTXDfBe4iHWxK/5vS/2BbUrXVm6QDb2n8m8BTpDOUv9GMiJhJOrgsAR6XNJd04ftd0hd/deIur57aDLi7bPjYvLxfk74DL5FasayuL5FaJF1Cupj4HinZEBGLSWeSx5EOSCeSqi0X53kHkVo8lfwbaV9NB64HvhYRE/KyJpBatFybx/fJ07c6b0tyHfr9ufq0tWnnAD8j15U3YS7pwv3jkuaTDtbjSSWO9vZ50sXgSaQqmxtZUTW0D6m55zzgZuCUiHgrj/s+cHP+7R3WzLI/kuedQ/ou9iS1+puUx19BOqi/TWrx9mij+S8DPpDXcUNEPEW6xjKWdPK0VX7fnG6katiZpN/VvsDB+VrdLFKT1i/nZb1FSkSlk4+V1t3COlqliHYrXVgdk3Ql6UJuu9zg0tVIug84MyIm1joWs8acCGyNKd2A9DSwRy7Km1kn0pWrhqwKJP2IVCXwcycBs87JJQIzszrnEoGZWZ3rCJ2ttWqjjTaKIUOG1DoMM7NO5cknn3wnIga0Nl2nSARDhgxh7NiWWmCZmVljkl6vZDpXDZmZ1TknAjOzOudEYGZW55wIzMzqnBOBmVmdcyIwM6tzTgRmZnWuU9xHYGZWVdOmweu5CX4EvPACvPgiDBwIhx0GgwfXNr525kRgZh3T0qWwbNmKzwsXwpgxMG9e25e1bBk89VQ6wDclAp59FiZPTu/feaf5ZX3zmzBqVFrWc8/BBhvAXnvBWmWH0ylTYNy4tKw1dcMNcMABa76cFjgRmFkxli+H115Lr+Vmz4bHH4clS9LnOXPS50WLVkwzfz6MHZuSQXvp0QM2beFxv4MHw1FHgQRbbAG77ZbeA2yyCey+O7zyCpx/PoweDf37wxFHpIP+X/+68rL69oVDD4WePdc87o03bn2aNeREYGbtY9YsuOQS+OMf0xn4jBnprxLDhsH6ZQ/969kTzjwzHWxLunWDPfdc/QPjkCHQZ7WeUrrCttvC73+/ZsvogJwIzKwykyfDQw+l6o7Jk+Huu+Hpp1dUfyxcmN4fcECqS19nHdh7b+jde+XlrL12Gl46KK+1Fqy3XnW3xVbiRGDWVUyfDm+9lapi/vWvdHAdOjQdrEt11lOmpGkHDYJddllR9QErLoq+9NKqy46AZ56B98oedzxiBJx88orqjz594KCD0nDrVJwIzDqrJ56A229P1S8LFsDNN69cz97Y5pvDTjul9489lqZvrH9/2GOPVA3T2Be+AF//ejrg9+27crWNdWpOBGadwcyZ6e/RR+G+++CNN9LBvEePFQfko49OrVmkVBKYNw+mToVttoEddkhVMKUSQMTKLXJKundfuZRgdcGJwKzalixJ1TUzZsCDD6YD+tZbp2qZBQtWnX7x4jSuZNNNU9XOueemC6p9+7Y9Bmnl5o5W1/xNMGurWbNS88HSGXVDA0yalFq0rLNO8/PNmQP335/O6OfMScN69EhVMffeCzvumJotNibBccelqp3ddkvNGH3Wbu2o0EQg6UzgZEDA5RHx33n414HTgaXAXRHxH0XGYVaRp56Cyy9PNwlBOvAedVRqI15+c9D48amaplzv3qnpZGs22yxV4XziEzBgAAwfvnpn9GbtqLBEIGkYKQnsBSwG7pF0F7AFcDiwa0QskjSwqBjMVvHWW3DTTfDyy6k9+vrrw1/+ks7oX389ndHvsUeqNrnrLrj+eujVKx2wS61jDjwQzjgjHcghzbPZZvDqq03Xu5f06AFbbeWzeetwiiwR7AiMiYgFAJIeAUYBI4DzI2IRQERMLzAGq0fjxsE//5nuFB02LB14ly+Hq66Cn/401cP36wfvvpvO8rfbDvbdN1W/fPObK25smj49lQ4++MFV28I3Zdtti90us4IUmQjGA+dJ6g+8BxwMjAW2A/aXdB6wEPh2RPyzwDisM3j55ebvQl26FP7xj1SPPmNGqkvfZpump503Dy6+eEX3BY0ddRScdx5sv32q2583L52lN2XgwPRn1sUVlggiYqKkC4DRwDxgHOmawFrABsDewAeAmyRtHbFy70ySTgFOARjcxXr6szL33pvO1G+8sfVpd945neXfd186iDfnU59KyeCFF9IdsCXDhqUz/5IBA1ZU75jVMUV79I5XyYqknwBvAoeRqoYezsNfBvaOiGZ/2SNGjIixY8dWJU5rZ2+9lZo/Dh6c7lh96KEVd6c++yxceWW6Qen002G//Zpfzs47w5ZbVidmsy5C0pMR0eqt3kW3GhoYEdMlDQaOBPYBlgMHAA9L2g7oCbTQ56t1eC+9lC7A9uwJ11yT+pzp0SP12PjAA2madddNPUo2duaZ8LOftU8vjWa2Woq+j+DWfI1gCXBaRMySdCVwpaTxpNZExzeuFrIaikjdBJd3HdyrVzqQL1uW7jx96aV0Nt+rV6qCuf/+FXXyI0akbgzmzk3NLM8+O/X6+OSTqQ38pz614k5YdzZm1iEUmggiYv8mhi0Gvljkeq0FCxemA/1666X3ffqkppMR8OabcNFFqfqmXLdu6W7WhobUguZvf1sxbvPNU1PK005LrXF22snNI806Gd9ZXC+mTk1t4n/5y3TA7949HeA33jh9LunfH374w/TUpZJp09Jj+qR0c9U556SuhqdMSY/ta+luWjPr8JwIOrNFi9LBvEePdAG21JKmf/90oJ8+PZ3t//nP6aLs/PnpkXpf+1pqNjl9eqrm+clPUsmgXz/YZ5/UX7yZ1Q0ngs5q5szUymbu3NQs8qGHVnRB3LNnqn8vdWDWq1c6cz/33NR+3sysjBNBZ/P3v8O3v516r1y2LF2AffXVdJY/bFiaZtKklBR23TXV4X/4w66+MbNmORF0BjNmwJ13poeQ3Hln6tfmpJPg8MNTvzdmZmvAiaCje/XVVG8/bVo6u//ud1OTTDe7NLN24kTQkc2eDZ/+dLoz97HHUkJw00wza2dNPJjUam7JErjsMvjIR1Krnttugw99yEnAzArhRNDRTJ6cmnh+9avpgu/116eEYGZWEFcNdRTTp6f+es49NyWAW29d8SByM7MCORF0BM8+m+4JmDMndZN86aWpt00zsypwIqi1CRPgoINSK6AHH0yPRHQpwMyqyNcIaiUi9fvzwQ+mG8Puvhv23NNJwMyqzomgVn7+8/R83I9+FJ54It0FbGZWA64aqralS+Gzn013CR99dGoV1M352Mxqx0egarvhhpQEfvAD+OMfnQTMrOZcIqim5cvhvPNgl13g+993EjCzDsGJoJruvjv1DHrddU4CZtZh+GhUpOeeS88NgNRf0C9/mXoO/cxnahuXmVkZJ4IiRKRnBuy8c3oU5Pe/D4MHwwMPpOf79uhR6wjNzN7nqqEiXHopXHghnHwyvPIK/OhHKSHcfjscemitozMzW0mhJQJJZ0oaL2mCpLPysHMkTZH0dP47uMgYqu5vf4NvfQs+/vGUEG67DU4/PV0fOPxwXxswsw6nsBKBpGHAycBewGLgHkl35dG/jIhfFLXumnn4YfjkJ2GrreB//zfdJdy3L1x8ca0jMzNrVpFVQzsCYyJiAYCkR4BRBa6vtqZMgSOPhG22SaWC/v1rHZGZWUWKrKcYD4yU1F/SOsDBwKA87nRJz0i6UtIGTc0s6RRJYyWNbWhoKDDMdnLBBTB3Ltxxh5OAmXUqhSWCiJgIXACMBu4BxgFLgUuAbYDdganAhc3Mf1lEjIiIEQMGDCgqzPbxyitw+eVw3HEwdGitozEza5NCr1xGxBURMTwiRgIzgRcjYlpELIuI5cDlpGsIndfcuXDIIdC7N/zXf9U6GjOzNiu0+aikgRExXdJg4EhgH0mbRsTUPMkoUhVS53XeeTBxYrpHYMiQWkdjZtZmRd9HcKuk/sAS4LSImCXpGkm7AwG8Bny14BiK89pr6W7h44+HAw6odTRmZqul0EQQEfs3MexLRa6zqn760/T64x/XNg4zszXgO4tXx+LF6c7hq65Kdw9vsUWtIzIzW22+zXV1XHwxfOc7sPfe8L3v1ToaM7M14hJBWy1YkB4z+bGPwejRtY7GzGyNORG0xdy58OlPw7RpcMsttY7GzKxduGqoLX7xi9R9xPXXw3771ToaM7N24URQqVmz4Fe/glGj4Jhjah2NmVm7cSKoxMyZ6T6BefPSQ2bMzLoQXyOoxFVXwdNPw113we671zoaM7N25RJBJW6/PSWAg7vWM3TMzKCCRCDpVkmfllSfSWP6dHjsMTjiiFpHYmZWiEoO7pcAxwIvSjpf0g4Fx9SxXHhhehj9qK77TB0zq2+tJoKIuD8ivgAMJ3USN1rS3yV9WVKPogOsqaeeSjePnXQS7LprraMxMytERdU9uQfRE4CTgH8BvyIlhq59a+1110GPHun+ATOzLqrVVkOSbgN2AK4BDi17lsCNksYWGVzN3X037L8/rL9+rSMxMytMJc1Hfx0RDzY1IiJGtHM8Hccbb8Bzz8FXvlLrSMzMClVJ1dCOkvqVPkjaQNK/FRhTx1DqS8hNRs2si6skEZwcEbNLHyJiFnBycSF1AMuWwW9+k6qFdqivRlJmVn8qSQTdJKn0QVJ3oGdxIXUA994Lr7wCZ5xR60jMzApXyTWCe4GbJP2O9JzhU4F7Co2q1u64A/r2hcMPr3UkZmaFqyQR/CfpAfNfAwTcB/y+yKBqKgL+8pf04JkeXfs2CTMzqCARRMRy0t3FlxQfTgcwYQK8+Sb84Ae1jsTMrCoq6WtoqKRbJD0n6ZXSXyULl3SmpPGSJkg6q9G4b0sKSRutbvCF+Otf0+vHP17bOMzMqqSSi8VXkUoDS4GPAn8g3VzWIknDSK2L9gJ2Aw6RNDSPGwR8HHhj9cIu0KRJsN56MHhwrSMxM6uKShJB74h4AFBEvB4R5wAHVDDfjsCYiFgQEUuBR4BSz22/BP6DdPG5Y3nhBdhuO1jRUMrMrEurJBEszF1QvyjpdEmjgIEVzDceGCmpv6R1gIOBQZIOA6ZExLiWZpZ0iqSxksY2NDRUsLp2UkoEZmZ1opJEcBawDnAGsCfwReD41maKiInABaSO6e4BxpGql74LtPq8x4i4LCJGRMSIAQMGVBBmO1i4EF57DbbfvjrrMzPrAFpMBPnmsaMjYl5EvBkRX46IoyJiTCULj4grImJ4RIwEZpK6sd4KGCfpNWAL4ClJm6zRVrSXl19OzUddIjCzOtJiIoiIZcCe5XcWt4Wkgfl1MHAk8IeIGBgRQyJiCPAmMDwi3l6d5be7F15Ir04EZlZHKrmh7F/AHZJuBuaXBkbEbRXMe2t+lsES4LTcT1HH9dRT0L27+xcys7pSSSLYEJjByi2FAmg1EUTE/q2MH1LB+qvnkUdg+PDUfNTMrE5Ucmfxl6sRSM0tXAiPP+6O5sys7lTyhLKraKK9f0ScWEhEtfL447B4MYwcWetIzMyqqpKqoT+Xve9FuinsrWLCqaExuSHUvvvWNg4zsyqrpGro1vLPkq4H7i8soloZPx4GDYINN6x1JGZmVVXJDWWNDQW6Xkc848fDsGG1jsLMrOoquUYwl5WvEbxNekZB17F0KUyc6B5HzawuVVI11KcagdTUyy/DokUuEZhZXarkeQSjJK1f9rmfpCOKDavKxo9Pr04EZlaHKrlG8IOIeLf0ISJmA13r8V2TJqVXdzZnZnWokkTQ1DSVNDvtPF59FTbeGNZdt9aRmJlVXSWJYKykiyRtI2lrSb8Eniw6sKp67TXYaqtaR2FmVhOVJIKvA4uBG4GbgPeA04oMqupefRWGDKl1FGZmNVFJq6H5wNlViKU2li2DN96Ao4+udSRmZjVRSauh0ZL6lX3eQNK9xYZVRVOmpPsIXDVkZnWqkqqhjXJLIQDyMwUqeWZx5/Daa+nVVUNmVqcqSQTL8xPGAJC0JU30RtpplRKBSwRmVqcqaQb6XeBRSY/kzyOBrxYXUpVNnpxeBw2qbRxmZjVSycXieyQNB/YGBHwjIt4pPLJqaWiAPn2gV69aR2JmVhMV9T4aEe9ExJ+B54BTJY0vNqwqamiAAQNqHYWZWc1U0mpoU0lnSXoCmAB0Bz5feGTV4kRgZnWu2UQg6WRJDwKPABsBJwFTI+KHEfFsJQuXdKak8ZImSDorD/uRpGckPS3pPkmbtceGrLbp050IzKyutVQi+A3p7P/YiPheRDxDG1oLSRoGnAzsBewGHCJpKPDziNg1InYnPQbz+6sdfXtoaICBXac1rJlZW7WUCDYDbgAukvS8pB8BPdqw7B2BMRGxICKWkkoWoyJiTtk061LLpqgRrhoys7rXbCLIF4gviYiRwIHAu8B0SRMl/aSCZY8HRkrqL2kd4GBgEICk8yRNBr5AMyUCSadIGitpbENDQxs3q0Jz5sCSJU4EZlbXKm019GZE/CIi9gSOABZVMM9E4AJgNHAPMA5Ymsd9NyIGAdcCpzcz/2URMSIiRgwo6kBdSjBOBGZWx9r88PqIeD4ifljhtFdExPBcqpgJvNhokuuAo9oaQ7uZPj29OhGYWR1rcyJoC0kD8+tg4Ejg+nzBuOQwYFKRMbTIJQIzs8KfNHarpP7AEuC0iJgl6feStgeWA68DpxYcQ/NKicCthsysjrWaCCQ9EBEHtjasKRGxfxPDalcV1JhLBGZmzScCSb2AdYCNJG1A6mcIoC+paWnn19CQnlPcu3etIzEzq5mWSgRfBc4iHfSfZEUimEO62azz8z0EZmbNJ4KI+BXwK0lfj4iLqxhT9bh7CTOzirqhvljSh4Ah5dNHxB8KjKs6Ghpgs65Ry2VmtroquVh8DbAN8DSwLA8OoGskgt12q3UUZmY1VUnz0RHAThHRdR5PCe5nyMwsq+SGsvHAJkUHUnXz5sGiRU4EZlb3KikRbAQ8lx9M834fQxFxWGFRVYO7lzAzAypLBOcUHURN+GYyMzOgslZDj0jaEhgaEffnLqW7Fx9awdy9hJkZUNkzi08GbgEuzYM2B24vMqiqcInAzAyo7GLxacC+pDuKiYgXgc5/Gu1EYGYGVJYIFkXE4tIHSWtRy8dLtpcZM2DttWGddWodiZlZTVWSCB6R9B2gt6SPAzcD/1dsWFUwYwb07w9S69OamXVhlSSCs4EG4FlSR3R/Ab5XZFBVUUoEZmZ1rpLmo72BKyPicgBJ3fOwBUUGVriZM50IzMwofGONAAAN6UlEQVSorETwAOnAX9IbuL+YcKpoxgzYcMNaR2FmVnOVJIJeETGv9CG/7/xXWF01ZGYGVJYI5ksaXvogaU/gveJCqoIIJwIzs6ySawRnAjdLeit/3hT4XHEhVcG8ebB0qROBmRmtJAJJ3YCewA7A9qTHVU6KiCVViK04M2akV18jMDNruWooIpYDF0bEkogYHxHPtiUJSDpT0nhJEySdlYf9XNIkSc9I+pOkfmu4DW1XSgQuEZiZVXSN4D5JR0ltu/NK0jDgZGAvYDfgEElDgdHAsIjYFXgB+H9tjHnNzZyZXp0IzMwqukbwTWBdYJmk90jVQxERfVuZb0dgTEQsAJD0CDAqIn5WNs0Y4DNtD3sNuURgZva+VksEEdEnIrpFRI+I6Js/t5YEID3ZbKSk/rnr6oOBQY2mORG4u6mZJZ0iaayksQ2lDuLai68RmJm9r5JuqCXpi5L+K38eJGmv1uaLiInABaSqoHuAccDSsuV+N3++tpn5L4uIERExYkB79xBaqhpyIjAzq+gawW+BfYBj8+d5wG8qWXhEXBERwyNiJDATeBFA0vHAIcAXIqL6PZnOmgXrrQc9elR91WZmHU0l1wg+GBHDJf0LICJmSepZycIlDYyI6ZIGA0cC+0j6FPCfwIdL1w+qbvZs6Ff9xkpmZh1RJYlgSe5oLgAkDQCWV7j8WyX1B5YAp+Uk8mtgbWB0bog0JiJObXvoa2D2bNhgg6qu0syso6okEfwP8CdgoKTzSK18KuqGOiL2b2LYtm2KsAizZrlEYGaWVfLw+mslPQkcSGo6ekS+ENx5zZ4Ngxo3YDIzq0/NJgJJvYBTgW1JD6W5NCKWNjd9pzJ7NuyyS62jMDPrEFpqNXQ1MIKUBA4CflGViKph1ixfIzAzy1qqGtopInYBkHQF8ER1QirY8uUwZ46vEZiZZS2VCN7vXK7LVAlBSgIRTgRmZllLJYLdJM3J7wX0zp8r7WuoY5o1K726asjMDGghEURE92oGUjWzZ6dXlwjMzIDKupjoWpwIzMxW4kRgZlbn6i8R+BqBmdlK6i8RuERgZraS+ksEs2aBBH361DoSM7MOof4SwcyZ6YE03epv083MmlJ/R8NSIjAzM6AeE8GMGU4EZmZl6i8RzJwJ/fvXOgozsw6j/hKBSwRmZiupv0TgEoGZ2UrqKxEsWZJ6H3WJwMzsffWVCEp3FbtEYGb2vkITgaQzJY2XNEHSWXnYZ/Pn5ZJGFLn+VcyYkV5dIjAze19hiUDSMOBkYC9gN+AQSUOB8cCRwF+LWnezZs5Mry4RmJm9r8gSwY7AmIhYkJ9w9ggwKiImRsTzBa63eS4RmJmtoshEMB4YKam/pHWAg4FBlc4s6RRJYyWNbWhoaJ+IXCIwM1tFYYkgIiYCFwCjgXuAcUDFzz6OiMsiYkREjBgwYED7BOUSgZnZKgq9WBwRV0TE8IgYCcwEXixyfa2aORO6d4e+nfNxy2ZmRWjp4fVrTNLAiJguaTDpAvE+Ra6vVaW7iqWahmFm1pEUmgiAWyX1B5YAp0XELEmjgIuBAcBdkp6OiE8WHEfiu4rNzFZRaCKIiP2bGPYn4E9FrrdZ7mfIzGwV9XVnsUsEZmarqK9E4BKBmdkq6isRuERgZraK+kkEixbB/PkuEZiZNVI/icB3FZuZNal+EoHvKjYza1L9JAKXCMzMmlQ/icAlAjOzJtVPIiiVCJwIzMxWUj+JoFQicNWQmdlK6isR9OgB665b60jMzDqU+kkEb78Nm2zinkfNzBqpn0QwdSpsummtozAz63CcCMzM6lx9JYLNNqt1FGZmHU59JIJFi9LFYpcIzMxWUR+J4O2306sTgZnZKuojEUydml6dCMzMVuFEYGZW5+orEfhisZnZKuonEXTrBgMG1DoSM7MOp9BEIOlMSeMlTZB0Vh62oaTRkl7MrxsUGQMAb76ZqoW6dy98VWZmnU1hiUDSMOBkYC9gN+AQSUOBs4EHImIo8ED+XKw33oAttyx8NWZmnVGRJYIdgTERsSAilgKPAKOAw4Gr8zRXA0cUGEPy+usweHDhqzEz64yKTATjgZGS+ktaBzgYGARsHBFTAfLrwKZmlnSKpLGSxjY0NKx+FMuXw+TJTgRmZs0oLBFExETgAmA0cA8wDljahvkvi4gRETFiwJpc5J0+HRYvdtWQmVkzCr1YHBFXRMTwiBgJzAReBKZJ2hQgv04vMgZefz29ukRgZtakolsNDcyvg4EjgeuBO4Hj8yTHA3cUGQNvvJFenQjMzJq0VsHLv1VSf2AJcFpEzJJ0PnCTpK8AbwCfLTSCUonAVUNmZk0qNBFExP5NDJsBHFjkelcybhz06wfrr1+1VZqZdSZd+87id96Bm2+GY46pdSRmZh1W104El1+enkXw9a/XOhIzsw6rayeCTTeFE0+EnXaqdSRmZh1W0ReLa+uEE9KfmZk1q2uXCMzMrFVOBGZmdc6JwMyszjkRmJnVOScCM7M650RgZlbnnAjMzOqcE4GZWZ1TRNQ6hlZJagBeX41ZNwLeaedwasXb0vF0le0Ab0tHtabbsmVEtPpkr06RCFaXpLERMaLWcbQHb0vH01W2A7wtHVW1tsVVQ2Zmdc6JwMysznX1RHBZrQNoR96WjqerbAd4WzqqqmxLl75GYGZmrevqJQIzM2uFE4GZWZ3rsolA0qckPS/pJUln1zqetpD0mqRnJT0taWwetqGk0ZJezK8b1DrOpki6UtJ0SePLhjUZu5L/yfvoGUnDaxf5qprZlnMkTcn75mlJB5eN+395W56X9MnaRL0qSYMkPSRpoqQJks7MwzvdfmlhWzrjfukl6QlJ4/K2/DAP30rS43m/3CipZx6+dv78Uh4/pN2CiYgu9wd0B14GtgZ6AuOAnWodVxvifw3YqNGwnwFn5/dnAxfUOs5mYh8JDAfGtxY7cDBwNyBgb+DxWsdfwbacA3y7iWl3yt+ztYGt8veve623Ice2KTA8v+8DvJDj7XT7pYVt6Yz7RcB6+X0P4PH8/74JOCYP/x3wtfz+34Df5ffHADe2VyxdtUSwF/BSRLwSEYuBG4DDaxzTmjocuDq/vxo4ooaxNCsi/grMbDS4udgPB/4QyRign6RNqxNp65rZluYcDtwQEYsi4lXgJdL3sOYiYmpEPJXfzwUmApvTCfdLC9vSnI68XyIi5uWPPfJfAAcAt+ThjfdLaX/dAhwoSe0RS1dNBJsDk8s+v0nLX5aOJoD7JD0p6ZQ8bOOImArpxwAMrFl0bddc7J11P52eq0yuLKui6xTbkqsT9iCdfXbq/dJoW6AT7hdJ3SU9DUwHRpNKLLMjYmmepDze97clj38X6N8ecXTVRNBUluxM7WT3jYjhwEHAaZJG1jqggnTG/XQJsA2wOzAVuDAP7/DbImk94FbgrIiY09KkTQzr6NvSKfdLRCyLiN2BLUgllR2bmiy/FrYtXTURvAkMKvu8BfBWjWJps4h4K79OB/5E+oJMKxXP8+v02kXYZs3F3un2U0RMyz/e5cDlrKhm6NDbIqkH6cB5bUTclgd3yv3S1LZ01v1SEhGzgYdJ1wj6SVorjyqP9/1tyePXp/KqyxZ11UTwT2Bovvrek3Rh5c4ax1QRSetK6lN6D3wCGE+K//g82fHAHbWJcLU0F/udwHG5lcrewLulqoqOqlFd+SjSvoG0Lcfklh1bAUOBJ6odX1NyPfIVwMSIuKhsVKfbL81tSyfdLwMk9cvvewMfI13zeAj4TJ6s8X4p7a/PAA9GvnK8xmp95byoP1LLhxdIdW7frXU8bYh7a1Irh3HAhFLspLrAB4AX8+uGtY61mfivJxXNl5DOYL7SXOykou5v8j56FhhR6/gr2JZrcqzP5B/mpmXTfzdvy/PAQbWOvyyu/UhVCM8AT+e/gzvjfmlhWzrjftkV+FeOeTzw/Tx8a1Kyegm4GVg7D++VP7+Ux2/dXrG4iwkzszrXVauGzMysQk4EZmZ1zonAzKzOORGYmdU5JwIzszrnRGB1QdK8/DpE0rHtvOzvNPr89/ZcvlnRnAis3gwB2pQIJHVvZZKVEkFEfKiNMZnVlBOB1Zvzgf1zn/XfyJ1+/VzSP3OHZV8FkPSR3O/9daQblZB0e+4IcEKpM0BJ5wO98/KuzcNKpQ/lZY9Xer7E58qW/bCkWyRNknRtqRdJSedLei7H8ouq/3esLq3V+iRmXcrZpH7rDwHIB/R3I+IDktYGHpN0X552L2BYpO6LAU6MiJm5O4B/Sro1Is6WdHqkjsMaO5LUCdpuwEZ5nr/mcXsAO5P6kXkM2FfSc6TuEXaIiCh1P2BWNJcIrN59gtSvztOk7oz7k/qjAXiiLAkAnCFpHDCG1PnXUFq2H3B9pM7QpgGPAB8oW/abkTpJe5pUZTUHWAj8XtKRwII13jqzCjgRWL0T8PWI2D3/bRURpRLB/Pcnkj5C6hRsn4jYjdRHTK8Klt2cRWXvlwFrRepjfi9Sz5pHAPe0aUvMVpMTgdWbuaRHHJbcC3wtd22MpO1yr6+NrQ/MiogFknYgdRdcsqQ0fyN/BT6Xr0MMID36stmeL3Mf++tHxF+As0jVSmaF8zUCqzfPAEtzFc//Ar8iVcs8lS/YNtD0Y0DvAU6V9AypF8sxZeMuA56R9FREfKFs+J+AfUg9yQbwHxHxdk4kTekD3CGpF6k08Y3V20SztnHvo2Zmdc5VQ2Zmdc6JwMyszjkRmJnVOScCM7M650RgZlbnnAjMzOqcE4GZWZ37//ReIEGF6fvVAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# gradient descent\n",
"\n",
"# hyber-parameters\n",
"alpha = 0.01;\n",
"number_of_iterations =300\n",
"\n",
"# data arrays\n",
"cost_array = []\n",
"interation_array = []\n",
"accuracy_array = []\n",
"\n",
"# rename\n",
"X = training_images\n",
"Y = training_labels\n",
"m = size\n",
"\n",
"# main loop\n",
"for interation in range(1, number_of_iterations):\n",
" \n",
" # forward propogation training data set\n",
" Z = np.dot(np.transpose(W), X) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
" A = 1/(1 + np.exp(-Z)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
" term1 = np.multiply(-Y, np.log(A)) # -Y * log(A) # broadcasting is used \n",
" term2 = np.multiply((1 - Y), np.log(1 - A)) # (1-Y) * log(1-A) # broadcasting is used \n",
" L = term1 - term2 # loss function # l is 1X50000\n",
" cost = np.multiply(1/m, np.sum(L)) # cost # scalar \n",
" \n",
" \n",
" # forward propogation test data set\n",
" Z_test = np.dot(np.transpose(W), testing_images) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
" A_test = 1/(1 + np.exp(-Z_test)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
" \n",
" # accuracy\n",
" prediction_labels = np.where(A_test >= 0.5, 1, 0) # if greater than ore equal to 0.5 use 1, else 0\n",
" accuracy = np.sum(np.absolute(testing_labels - prediction_labels)) # Sum of false positives and false negatives # accuracy is scalar \n",
" percent_inaccurate = np.multiply(100/len(testing_labels), accuracy)\n",
" percent_accurate = 100 - percent_inaccurate\n",
" \n",
" # storage for plotting\n",
" cost_array.append(cost)\n",
" accuracy_array.append(percent_accurate)\n",
" interation_array.append(interation)\n",
" \n",
" # backpropogation\n",
" dZ = A - Y # dZ is 1X500\n",
" dW = np.multiply(1/m, np.dot(X, np.transpose(dZ))) # dW is 784X1\n",
" db = np.multiply(1/m, np.sum(dZ)) # db is a scalar\n",
" \n",
" # paramter updates\n",
" W = W - np.multiply(alpha, dW)\n",
" b = b - np.multiply(alpha, db)\n",
"\n",
"# forward propogation training data set\n",
"Z = np.dot(np.transpose(W), X) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
"A = 1/(1 + np.exp(-Z)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
"term1 = np.multiply(-Y, np.log(A)) # -Y * log(A) # broadcasting is used \n",
"term2 = np.multiply((1 - Y), np.log(1 - A)) # (1-Y) * log(1-A) # broadcasting is used \n",
"L = term1 - term2 # loss function # l is 1X50000\n",
"cost = np.multiply(1/m, np.sum(L)) # cost # scalar \n",
"\n",
"# forward propogation test data set\n",
"Z_test = np.dot(np.transpose(W), testing_images) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
"A_test = 1/(1 + np.exp(-Z_test)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
"\n",
"# final accuracy\n",
"prediction_labels = np.where(A_test >= 0.5, 1, 0) # if greater than ore equal to 0.5 use 1, else 0\n",
"accuracy = np.sum(np.absolute(testing_labels - prediction_labels)) # Sum of false positives and false negatives # accuracy is scalar \n",
"percent_inaccurate = np.multiply(100/len(testing_labels), accuracy)\n",
"percent_accurate = 100 - percent_inaccurate\n",
"\n",
"# storage for plotting\n",
"cost_array.append(cost)\n",
"accuracy_array.append(percent_accurate)\n",
"interation_array.append(number_of_iterations + 1)\n",
" \n",
"print('Final Cost: ' + str(cost))\n",
"print('Final Percent Accuracy: ' + str(percent_accurate))\n",
"\n",
"# plot learning curve\n",
"pyplot.figure()\n",
"pyplot.plot(interation_array, cost_array, 'red')\n",
"pyplot.title('Learning Curve - 50,000 MNIST Training Data Set')\n",
"pyplot.xlabel('Iterations')\n",
"pyplot.ylabel('Cost')\n",
"pyplot.show()\n",
"\n",
"# plot percent accuracy curve\n",
"pyplot.figure()\n",
"pyplot.plot(interation_array, accuracy_array, 'red')\n",
"pyplot.title('Percent Accuracy Curve - 10,000 MNIST Test Data Set')\n",
"pyplot.xlabel('Iterations')\n",
"pyplot.ylabel('Percent Accuracy')\n",
"pyplot.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As illustrated, the learning curve flattens out (pesudo-plateau's) at apprximately 8.51270 after 100 training iterations, but adjusts very little after just a few iterations. The percent accuracy initially dramatically increases, then decreases a little bit, then steadly increases assomototically to approximately 95.88% after 100 training iteration. This is reasonable considering that we only have one neuron in our network.\n",
"\n",
"Now we will run through some examples classifications in the network. We will find one true positive, one true negative, one false positive, and one false negative."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"True Negative Found: True Label: 0 Predicted: 0, @ Test Data Index: 0, Size; (1, 28, 28)\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADXNJREFUeJzt3X+IVXUax/HPUxlpSlmSqU3ZjrHsEk1uQ2wUS7UY7SJYgVHQMmuyU1Cw1RYbQ1AUgSzbj6U/DKNBo99pbVK2GhHbryW0H5Rl2Q9cNceZNSOVijCf/WPOxGhzv/fOvefcc8fn/QK5957nnnMeLn7mnHu/596vubsAxHNI2Q0AKAfhB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8Q1GHN3JmZcTkhUDB3t1qe19CR38wuNLOPzexTM7u5kW0BaC6r99p+MztU0kZJcyRtlbRW0uXu/mFiHY78QMGaceQ/U9Kn7v65u38v6XFJ8xrYHoAmaiT8MyRtGfZ4a7ZsP2bWbWbrzGxdA/sCkLNGPvAb6dTiJ6f17r5E0hKJ036glTRy5N8qqW3Y4xMkbWusHQDN0kj410o6xcxONrPDJV0maWU+bQEoWt2n/e6+18yulbRa0qGSet39g9w6A1Couof66toZ7/mBwjXlIh8AYxfhB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0E1dYpu1KejoyNZv/766yvW2tvbk+tOmDAhWe/p6UnWjzrqqGT9hRdeqFjbvXt3cl0UiyM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTV0Cy9ZrZJ0m5JP0ja6+6dVZ7PLL0jmDhxYrK+efPmZP3oo4/Os51cffHFFxVrqesTJGn58uV5txNCrbP05nGRz3nuviOH7QBoIk77gaAaDb9LWmNmb5lZdx4NAWiORk/7z3b3bWZ2nKQXzewjd39l+BOyPwr8YQBaTENHfnfflt0OSHpG0pkjPGeJu3dW+zAQQHPVHX4zO9LMJg3dl3SBpPV5NQagWI2c9k+V9IyZDW3nUXf/Vy5dAShcQ+P8o94Z4/wjmjRpUrK+atWqZP3LL7+sWHvnnXeS686ePTtZP+mkk5L1tra2ZH38+PEVa/39/cl1zzrrrGS92vpR1TrOz1AfEBThB4Ii/EBQhB8IivADQRF+ICiG+tCQKVOmJOs33XRTXTVJWrBgQbK+bNmyZD0qhvoAJBF+ICjCDwRF+IGgCD8QFOEHgiL8QFBM0Y2G7NiR/uHm119/vWKt2jh/ta8bM87fGI78QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4/xoyOTJk5P1np6eurc9ffr0utdFdRz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCoqr/bb2a9kuZKGnD3U7Nlx0h6QtJMSZskXeruX1XdGb/bP+Z0dHQk60899VSyPmvWrIq1jRs3JtedM2dOsr5ly5ZkPao8f7d/qaQLD1h2s6SX3P0USS9ljwGMIVXD7+6vSNp5wOJ5koZ+RmWZpIty7gtAwep9zz/V3fskKbs9Lr+WADRD4df2m1m3pO6i9wNgdOo98veb2TRJym4HKj3R3Ze4e6e7d9a5LwAFqDf8KyV1Zfe7JD2bTzsAmqVq+M3sMUn/kfRzM9tqZgslLZI0x8w+kTQnewxgDKk6zp/rzhjnbzldXV3J+u23356st7W1JevffvttxdrcuXOT67788svJOkaW5zg/gIMQ4QeCIvxAUIQfCIrwA0ERfiAofrr7IDBx4sSKtRtvvDG57i233JKsH3JI+viwc+eB3/na3znnnFOx9tFHHyXXRbE48gNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzHwSWLl1asXbJJZc0tO3ly5cn6/fee2+yzlh+6+LIDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBMc5/EGhvby9s24sXL07W33jjjcL2jWJx5AeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoKqO85tZr6S5kgbc/dRs2W2S/iTpf9nTetx9VVFNIm3NmjUVax0dHYVtW6p+HcCiRYsq1rZt21ZXT8hHLUf+pZIuHGH5Pe5+evaP4ANjTNXwu/srktLTsgAYcxp5z3+tmb1nZr1mNjm3jgA0Rb3hXyypXdLpkvok3VXpiWbWbWbrzGxdnfsCUIC6wu/u/e7+g7vvk/SApDMTz13i7p3u3llvkwDyV1f4zWzasIcXS1qfTzsAmqWWob7HJJ0raYqZbZV0q6Rzzex0SS5pk6SrCuwRQAHM3Zu3M7Pm7SyQ8ePHV6w9/PDDyXXPOOOMZP3EE0+sq6ch27dvr1hbsGBBct3Vq1c3tO+o3N1qeR5X+AFBEX4gKMIPBEX4gaAIPxAU4QeCYqjvIHfEEUck64cdlr7UY9euXXm2s5/vvvsuWb/hhhuS9fvvvz/Pdg4aDPUBSCL8QFCEHwiK8ANBEX4gKMIPBEX4gaAY50fSaaedlqzfc889yfp5551X9743b96crM+cObPubR/MGOcHkET4gaAIPxAU4QeCIvxAUIQfCIrwA0Exzt8CJkyYkKx/8803Tepk9CZPTk/T2NvbW7E2b968hvY9Y8aMZL2vr6+h7Y9VjPMDSCL8QFCEHwiK8ANBEX4gKMIPBEX4gaDSP9ouyczaJD0k6XhJ+yQtcfd/mNkxkp6QNFPSJkmXuvtXxbU6drW3tyfrr732WrL+/PPPJ+vr16+vWKs21r1w4cJkfdy4ccl6tbH2WbNmJespn332WbIedRw/L7Uc+fdK+ou7/0LSryVdY2a/lHSzpJfc/RRJL2WPAYwRVcPv7n3u/nZ2f7ekDZJmSJonaVn2tGWSLiqqSQD5G9V7fjObKWm2pDclTXX3PmnwD4Sk4/JuDkBxqr7nH2JmEyWtkHSdu+8yq+nyYZlZt6Tu+toDUJSajvxmNk6DwX/E3Z/OFveb2bSsPk3SwEjruvsSd+909848GgaQj6rht8FD/IOSNrj73cNKKyV1Zfe7JD2bf3sAilLLaf/Zkv4g6X0zezdb1iNpkaQnzWyhpM2S5hfT4tg3f376pTn++OOT9SuvvDLPdkal2tu7Rr4SvmfPnmT96quvrnvbqK5q+N39NUmV/gf8Nt92ADQLV/gBQRF+ICjCDwRF+IGgCD8QFOEHgqr58l7U79hjjy27hcKsWLEiWb/jjjsq1gYGRrwo9Efbt2+vqyfUhiM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTFFN1NUO3nr88///xk/YorrkjWp0+fXrH29ddfJ9et5r777kvWX3311WR97969De0fo8cU3QCSCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMb5gYMM4/wAkgg/EBThB4Ii/EBQhB8IivADQRF+IKiq4TezNjN72cw2mNkHZvbnbPltZvaFmb2b/ft98e0CyEvVi3zMbJqkae7+tplNkvSWpIskXSppj7v/veadcZEPULhaL/KpOmOPu/dJ6svu7zazDZJmNNYegLKN6j2/mc2UNFvSm9mia83sPTPrNbPJFdbpNrN1ZrauoU4B5Krma/vNbKKkf0u6092fNrOpknZIckl3aPCtwZVVtsFpP1CwWk/7awq/mY2T9Jyk1e5+9wj1mZKec/dTq2yH8AMFy+2LPWZmkh6UtGF48LMPAodcLGn9aJsEUJ5aPu0/R9Krkt6XtC9b3CPpckmna/C0f5Okq7IPB1Pb4sgPFCzX0/68EH6geHyfH0AS4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+IKiqP+CZsx2S/jvs8ZRsWStq1d5atS+J3uqVZ28n1frEpn6f/yc7N1vn7p2lNZDQqr21al8SvdWrrN447QeCIvxAUGWHf0nJ+09p1d5atS+J3upVSm+lvucHUJ6yj/wASlJK+M3sQjP72Mw+NbOby+ihEjPbZGbvZzMPlzrFWDYN2oCZrR+27Bgze9HMPsluR5wmraTeWmLm5sTM0qW+dq0243XTT/vN7FBJGyXNkbRV0lpJl7v7h01tpAIz2ySp091LHxM2s99I2iPpoaHZkMzsb5J2uvui7A/nZHf/a4v0dptGOXNzQb1Vmln6jyrxtctzxus8lHHkP1PSp+7+ubt/L+lxSfNK6KPlufsrknYesHiepGXZ/WUa/M/TdBV6awnu3ufub2f3d0samlm61Ncu0Vcpygj/DElbhj3eqtaa8tslrTGzt8ysu+xmRjB1aGak7Pa4kvs5UNWZm5vpgJmlW+a1q2fG67yVEf6RZhNppSGHs939V5J+J+ma7PQWtVksqV2D07j1SbqrzGaymaVXSLrO3XeV2ctwI/RVyutWRvi3Smob9vgESdtK6GNE7r4tux2Q9IwG36a0kv6hSVKz24GS+/mRu/e7+w/uvk/SAyrxtctmll4h6RF3fzpbXPprN1JfZb1uZYR/raRTzOxkMztc0mWSVpbQx0+Y2ZHZBzEysyMlXaDWm314paSu7H6XpGdL7GU/rTJzc6WZpVXya9dqM16XcpFPNpRxr6RDJfW6+51Nb2IEZvYzDR7tpcFvPD5aZm9m9pikczX4ra9+SbdK+qekJyWdKGmzpPnu3vQP3ir0dq5GOXNzQb1Vmln6TZX42uU543Uu/XCFHxATV/gBQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwjq/03vB3CtDy8wAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"False Negative Found: True Label: 1 Predicted: 0, @ Test Data Index: 3, Size; (1, 28, 28)\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADVNJREFUeJzt3WGsVPWZx/HfT2ij0hokDfSGsgsSorvxhWwIGtls3BjQNSTIC0yN2bBJLSTWuE1IXOKb8sZo1rbQ+IKEplgwrW0TqPJC3RI0uUVXI6KptiyFEBZYCVSpIkYtcp99cQ+7V7zzn8vcM3Pm8nw/CZmZ88yZ82TC754z8z9z/o4IAcjnsqYbANAMwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+IKnJvdyYbU4nBLosIjyW541rz2/7dtv7bR+0vXY8rwWgt9zpuf22J0n6o6TFko5Jek3S3RHxh8I67PmBLuvFnn+hpIMRcSgi/iLpF5KWjeP1APTQeMI/U9LREY+PVcs+x/Yq23ts7xnHtgDUbDxf+I12aPGFw/qI2CRpk8RhP9BPxrPnPyZp1ojH35D0zvjaAdAr4wn/a5Lm2Z5j+8uSvilpRz1tAei2jg/7I+Iz2/dL+g9JkyRtjojf19YZgK7qeKivo43xmR/oup6c5ANg4iL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gqY6n6JYk24clfSjpnKTPImJBHU0B6L5xhb/yjxHxbg2vA6CHOOwHkhpv+EPSb2y/bntVHQ0B6I3xHvYvioh3bE+XtNP2f0XE4MgnVH8U+MMA9BlHRD0vZK+TdCYivl94Tj0bA9BSRHgsz+v4sN/2FNtfPX9f0hJJb3f6egB6azyH/TMk/dr2+df5eUQ8X0tXALqutsP+MW2Mw36g67p+2A9gYiP8QFKEH0iK8ANJEX4gKcIPJFXHr/rQZQMDA8X6fffd11FNks6ePVusHzlypFh/+OGHi/W9e/e2rB09erS4LrqLPT+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJMVPevvANddcU6xv3LixWF+8eHGd7dTq008/bVlbtGhRcd3SOQJojZ/0Aigi/EBShB9IivADSRF+ICnCDyRF+IGkGOfvgZkzZxbrBw4cKNYnTy5fduHxxx/vqCa17+26664r1h977LFiferUqS1rhw4dKq570003Fevvvsvk0KNhnB9AEeEHkiL8QFKEH0iK8ANJEX4gKcIPJNX2uv22N0taKulkRFxfLZsm6ZeSZks6LOmuiPhz99qc2B588MFifdKkScX6vffeW6xv3br1ons67/Dhw8X6Sy+9VKxffvnlxfqGDRta1tpdx2BwcLBYb3cewOnTp4v17May5/+ppNsvWLZW0q6ImCdpV/UYwATSNvwRMSjp1AWLl0naUt3fIunOmvsC0GWdfuafERHHJam6nV5fSwB6oetz9dleJWlVt7cD4OJ0uuc/YXtAkqrbk62eGBGbImJBRCzocFsAuqDT8O+QtLK6v1LSM/W0A6BX2obf9lOS/lPStbaP2f6WpEclLbZ9QNLi6jGACYTf89fgqquuKtbb/V7/iSeeKNbXrp24I6n79+9vWZs3b964Xnv9+vXF+po1a8b1+hMVv+cHUET4gaQIP5AU4QeSIvxAUoQfSIqhvhosXLiwWH/llVeK9XZTbO/ateuie+oXy5cvb1nbvn17cd12/zfff//9Yr00lPjee+8V153IGOoDUET4gaQIP5AU4QeSIvxAUoQfSIrwA0l1/TJeGcyfP39c67/xxhs1ddJ/nn322Za1gwcPFtedO3dusf7JJ58U6x999FGxnh17fiApwg8kRfiBpAg/kBThB5Ii/EBShB9IinH+GuzevbtYHxoaKtZ37txZrC9durRYP378eLHepGuvvbZlrd303rfddluxfuWVVxbr7c4DyI49P5AU4QeSIvxAUoQfSIrwA0kRfiApwg8k1fa6/bY3S1oq6WREXF8tWyfp25L+VD3toYho/cPt/3+tS/K6/e08/fTTxfqyZcuK9Q8++KBYf/7551vWtm3bVlz3hRdeKNZnzpxZrJfG8SVpw4YNLWsDAwPFdW+99dZi/cUXXyzWs6rzuv0/lXT7KMvXR8QN1b+2wQfQX9qGPyIGJZ3qQS8Aemg8n/nvt/0725ttX11bRwB6otPwb5Q0V9INko5L+kGrJ9peZXuP7T0dbgtAF3QU/og4ERHnImJI0o8ltZypMiI2RcSCiFjQaZMA6tdR+G2P/Jp2uaS362kHQK+0/Umv7ack3SLpa7aPSfqepFts3yApJB2WtLqLPQLogrbj/LVuLOk4/xVXXFGsP/LII8X6Aw88UGc7n3PqVHkgZ9q0aV3bdjsrVqwo1tudw5BVneP8AC5BhB9IivADSRF+ICnCDyRF+IGkGOqbAG6++eZi/Z577un4tWfMmNHxupJ09uzZYv3GG29sWZszZ05x3TvuuKNYf+6554r1rBjqA1BE+IGkCD+QFOEHkiL8QFKEH0iK8ANJMUX3BPDyyy+Pq96kJ598smVt9uzZxXXPnTtXczcYiT0/kBThB5Ii/EBShB9IivADSRF+ICnCDyTFOD/61ty5c5tu4ZLGnh9IivADSRF+ICnCDyRF+IGkCD+QFOEHkmo7zm97lqStkr4uaUjSpoj4ke1pkn4pabakw5Luiog/d69VTERnzpxpugW0MJY9/2eS1kTE30i6SdJ3bP+tpLWSdkXEPEm7qscAJoi24Y+I4xGxt7r/oaR9kmZKWiZpS/W0LZLu7FaTAOp3UZ/5bc+WNF/Sq5JmRMRxafgPhKTpdTcHoHvGfG6/7a9I2ibpuxFx2h7TdGCyvUrSqs7aA9AtY9rz2/6ShoP/s4jYXi0+YXugqg9IOjnauhGxKSIWRMSCOhoGUI+24ffwLv4nkvZFxA9HlHZIWlndXynpmfrbA9AtYznsXyTpnyW9ZfvNatlDkh6V9Cvb35J0RNKK7rSIiWxwcLBlbfXq1cV1p0/na6Ruahv+iNgtqdUH/FvrbQdAr3CGH5AU4QeSIvxAUoQfSIrwA0kRfiApLt2Nrrrsstb7l3aniJ88OepJo6gJe34gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIpxfnTV0NBQy1pE9LATXIg9P5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kxTg/+taSJUuK9Y0bN/aok0sTe34gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSKrtOL/tWZK2Svq6pCFJmyLiR7bXSfq2pD9VT30oIp7tVqOYmM6cOdPxupMncxpKN43l3f1M0pqI2Gv7q5Jet72zqq2PiO93rz0A3dI2/BFxXNLx6v6HtvdJmtntxgB010V95rc9W9J8Sa9Wi+63/Tvbm21f3WKdVbb32N4zrk4B1GrM4bf9FUnbJH03Ik5L2ihprqQbNHxk8IPR1ouITRGxICIW1NAvgJqMKfy2v6Th4P8sIrZLUkSciIhzETEk6ceSFnavTQB1axt+D0+l+hNJ+yLihyOWD4x42nJJb9ffHoBucbvLJ9v+e0m/lfSWhof6JOkhSXdr+JA/JB2WtLr6crD0WlyrOZmpU6e2rJ06daq47scff1ysT5kypaOeLnURUZ77vDKWb/t3SxrtxRjTByYwzvADkiL8QFKEH0iK8ANJEX4gKcIPJNV2nL/WjTHOD3TdWMf52fMDSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFK9vjbyu5L+e8Tjr1XL+lG/9tavfUn01qk6e/vrsT6xpyf5fGHj9p5+vbZfv/bWr31J9NappnrjsB9IivADSTUd/k0Nb7+kX3vr174keutUI701+pkfQHOa3vMDaEgj4bd9u+39tg/aXttED63YPmz7LdtvNj3FWDUN2knbb49YNs32TtsHqttRp0lrqLd1tv+neu/etH1HQ73Nsv2i7X22f2/7X6vljb53hb4aed96fthve5KkP0paLOmYpNck3R0Rf+hpIy3YPixpQUQ0PiZs+x8knZG0NSKur5b9u6RTEfFo9Yfz6oj4tz7pbZ2kM03P3FxNKDMwcmZpSXdK+hc1+N4V+rpLDbxvTez5F0o6GBGHIuIvkn4haVkDffS9iBiUdOHMFsskbanub9Hwf56ea9FbX4iI4xGxt7r/oaTzM0s3+t4V+mpEE+GfKenoiMfH1F9Tfoek39h+3faqppsZxYzzMyNVt9Mb7udCbWdu7qULZpbum/eukxmv69ZE+Ee7xFA/DTksioi/k/RPkr5THd5ibMY0c3OvjDKzdF/odMbrujUR/mOSZo14/A1J7zTQx6gi4p3q9qSkX6v/Zh8+cX6S1Or2ZMP9/J9+mrl5tJml1QfvXT/NeN1E+F+TNM/2HNtflvRNSTsa6OMLbE+pvoiR7SmSlqj/Zh/eIWlldX+lpGca7OVz+mXm5lYzS6vh967fZrxu5CSfaihjg6RJkjZHxMM9b2IUtq/R8N5eGv7F48+b7M32U5Ju0fCvvk5I+p6kpyX9StJfSToiaUVE9PyLtxa93aKLnLm5S721mln6VTX43tU543Ut/XCGH5ATZ/gBSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0jqfwFP1/dzjhK1UQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"False Positive Found: True Label: 0 Predicted: 1, @ Test Data Index: 89, Size; (1, 28, 28)\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADcNJREFUeJzt3X+IXXV6x/HPp2YFjQsalqQha501SrHGNFtHKVirRYy2LBjBHytSUrqa/WOVLhSp+s+KZUVqdlv/WohsTBayxoWoE9bSTRCtLRRJImuczXSTKHGdJiYNGnRDMCR5+seclNk493tu7q9zJ8/7BWHuPc899zze8TPn3Ps993wdEQKQz+813QCAZhB+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJzRnkxmxzOiHQZxHhdh7X1Z7f9u22f217r+1Hu3kuAIPlTs/tt32epN2SbpU0KWmbpPsiYldhHfb8QJ8NYs9/vaS9EfF+RByXtFHSHV08H4AB6ib8iyR9OO3+ZLXsd9heZXu77e1dbAtAj3Xzgd9MhxZfOKyPiDWS1kgc9gPDpJs9/6SkS6fd/6qk/d21A2BQugn/NklX2v6a7fMlfVPS5t60BaDfOj7sj4gTth+S9AtJ50laGxG/6llnAPqq46G+jjbGe36g7wZykg+A2YvwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQGOkX3uWrOnPLL+PDDDxfrl19+eVfrA51gzw8kRfiBpAg/kBThB5Ii/EBShB9IivADSXU1zm97n6TPJJ2UdCIiRnvR1GwzOlr+z37mmWeK9bGxsV62gzY88MADxfqdd95ZrNf9Tt94442zbWngenGSz19ExOEePA+AAeKwH0iq2/CHpC22d9he1YuGAAxGt4f9N0TEftvzJW21/d8R8eb0B1R/FPjDAAyZrvb8EbG/+nlI0suSrp/hMWsiYjTrh4HAsOo4/Lbn2v7y6duSlksa71VjAPqrm8P+BZJetn36eX4aEf/Wk64A9F3H4Y+I9yX9cQ97mbVuu+22rtZfvnx5sb5kyZJifXycA66ZzJ8/v2XtscceK6572WWXFet79+4t1mfDOD9DfUBShB9IivADSRF+ICnCDyRF+IGkuHR3D9QN1dW54IILinWG8jpz//33t6zVDeVNTEwU67t37+6op2HCnh9IivADSRF+ICnCDyRF+IGkCD+QFOEHknJEDG5j9uA21mNXXHFFy9qOHTuK686dO7erbddNAY6ZHT7c+qLSF198cXHdJ598sqt6kyLC7TyOPT+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJMUAcptuueWWlrWLLrpogJ3gtBUrVhTr8+bNa1mrO7+lmo/inMaeH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSqh3nt71W0jckHYqIJdWyeZJelDQiaZ+keyLik/612bx77723Za3bayKMjY11tf65qjTFtiQ9++yzxXrp91L3O3vllVeK9XNBO3v+dZJuP2PZo5Jei4grJb1W3Qcwi9SGPyLelPTxGYvvkLS+ur1eUvlUKwBDp9P3/Asi4oAkVT/Lx2cAhk7fz+23vUrSqn5vB8DZ6XTPf9D2Qkmqfh5q9cCIWBMRoxEx2uG2APRBp+HfLGlldXulJD6uBmaZ2vDbfkHSf0n6Q9uTtr8l6WlJt9reI+nW6j6AWaT2PX9E3Nei1PoL7jgr7733XtMtDKVrr722WF+0aFHftv35558X68uWLSvWFy9eXKxv2rTprHvqNc7wA5Ii/EBShB9IivADSRF+ICnCDyTFpbsrddNo103p3I1169b17bmH2fnnn1+sP/LIIwPq5IvGx8eL9SNHjhTrGzZsKNYZ6gPQGMIPJEX4gaQIP5AU4QeSIvxAUoQfSIpx/srIyEixvnTp0o6fe//+/cX6xx+feX3U2aM0DbZUvuT5gw8+WFy3m9e8W9u2bSvWN27cWKzXXVZ8GLDnB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkGOevrFhRnmvUdsfPvXfv3mL9o48+Ktavvvrqjrd93XXXFet1l8e+5pprivWbbrqpWD916lSx3k/Hjx9vWVu9enVx3aeeeqpYP3bsWEc9DRP2/EBShB9IivADSRF+ICnCDyRF+IGkCD+QlCOi/AB7raRvSDoUEUuqZU9IelDS/1YPezwi/rV2Y3Z5Yw06efJksV73OpXs2rWrWN+zZ0+xXncOQje9davu/IcmeytdL+D5558fYCeDFRFtnZTSzp5/naTbZ1j+zxGxrPpXG3wAw6U2/BHxpqTZe6kZADPq5j3/Q7Z32l5r+5KedQRgIDoN/48kLZa0TNIBST9o9UDbq2xvt729w20B6IOOwh8RByPiZESckvScpOsLj10TEaMRMdppkwB6r6Pw21447e6dkspTmgIYOrVf6bX9gqSbJX3F9qSk70m62fYySSFpn6Rv97FHAH1QO87f040lHefvVpNj6Z988kmx/vrrrxfrpe/z33XXXR311K45c3JerqKX4/wAzkGEH0iK8ANJEX4gKcIPJEX4gaRyjoWcY955552Wta1btxbX3b69fNb1li1bivVPP/20WH/uueeK9W7UTX2OMvb8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4/yV8fHy9Ui6mSa7223XTRf96quvtqwdPXq0o57aNW/evGL9xhtv7Nu2x8bG+vbcGbDnB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkuHR3m0rTZE9OThbXrfvO/GxWd/7Dzp07+7btuqnLly9f3rK2efPm4rp1lyQ/ceJEsd4kLt0NoIjwA0kRfiApwg8kRfiBpAg/kBThB5KqHee3famkn0j6fUmnJK2JiGdtz5P0oqQRSfsk3RMRxfmcZ/M4P2ZWN85fmlOg30pTm3/44YfFdZcuXVqs181X0KRejvOfkPT3EXGVpD+V9B3bfyTpUUmvRcSVkl6r7gOYJWrDHxEHIuLt6vZnkiYkLZJ0h6T11cPWSyqfbgVgqJzVe37bI5K+LuktSQsi4oA09QdC0vxeNwegf9q+hp/tiyRtkvTdiPi09H7qjPVWSVrVWXsA+qWtPb/tL2kq+Bsi4qVq8UHbC6v6QkmHZlo3ItZExGhEjPaiYQC9URt+T+3ifyxpIiJ+OK20WdLK6vZKSVxKFZhF2jnsv0HSX0t61/Yvq2WPS3pa0s9sf0vSbyTd3Z8WgZkdO3asWF+9enXL2osvvlhcd5iH8nqlNvwR8Z+SWr3Bv6W37QAYFM7wA5Ii/EBShB9IivADSRF+ICnCDyTFFN3oSt1XYycmJlrWrrrqquK6R44cKdbvvrt8aknd5bezY88PJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kxRTf6amRkpGXtwgsvLK579OjRYv2DDz7opKVzHlN0Aygi/EBShB9IivADSRF+ICnCDyRF+IGkGOcHzjGM8wMoIvxAUoQfSIrwA0kRfiApwg8kRfiBpGrDb/tS26/bnrD9K9t/Vy1/wvb/2P5l9e+v+t8ugF6pPcnH9kJJCyPibdtflrRD0gpJ90j6bUSsbntjnOQD9F27J/nUztgTEQckHahuf2Z7QtKi7toD0LSzes9ve0TS1yW9VS16yPZO22ttX9JinVW2t9ve3lWnAHqq7XP7bV8k6d8lfT8iXrK9QNJhSSHpHzX11uBva56Dw36gz9o97G8r/La/JOnnkn4RET+coT4i6ecRsaTmeQg/0Gc9+2KPbUv6saSJ6cGvPgg87U5J42fbJIDmtPNp/59J+g9J70o6VS1+XNJ9kpZp6rB/n6RvVx8Olp6LPT/QZz097O8Vwg/0H9/nB1BE+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSKr2Ap49dljSB9Puf6VaNoyGtbdh7Uuit071srfL2n3gQL/P/4WN29sjYrSxBgqGtbdh7Uuit0411RuH/UBShB9Iqunwr2l4+yXD2tuw9iXRW6ca6a3R9/wAmtP0nh9AQxoJv+3bbf/a9l7bjzbRQyu299l+t5p5uNEpxqpp0A7ZHp+2bJ7trbb3VD9nnCatod6GYubmwszSjb52wzbj9cAP+22fJ2m3pFslTUraJum+iNg10EZasL1P0mhEND4mbPvPJf1W0k9Oz4Zk+58kfRwRT1d/OC+JiH8Ykt6e0FnO3Nyn3lrNLP03avC16+WM173QxJ7/ekl7I+L9iDguaaOkOxroY+hFxJuSPj5j8R2S1le312vqf56Ba9HbUIiIAxHxdnX7M0mnZ5Zu9LUr9NWIJsK/SNKH0+5Parim/A5JW2zvsL2q6WZmsOD0zEjVz/kN93Om2pmbB+mMmaWH5rXrZMbrXmsi/DPNJjJMQw43RMSfSPpLSd+pDm/Rnh9JWqypadwOSPpBk81UM0tvkvTdiPi0yV6mm6GvRl63JsI/KenSafe/Kml/A33MKCL2Vz8PSXpZU29ThsnB05OkVj8PNdzP/4uIgxFxMiJOSXpODb521czSmyRtiIiXqsWNv3Yz9dXU69ZE+LdJutL212yfL+mbkjY30McX2J5bfRAj23MlLdfwzT68WdLK6vZKSWMN9vI7hmXm5lYzS6vh127YZrxu5CSfaijjXySdJ2ltRHx/4E3MwPblmtrbS1PfePxpk73ZfkHSzZr61tdBSd+T9Iqkn0n6A0m/kXR3RAz8g7cWvd2ss5y5uU+9tZpZ+i01+Nr1csbrnvTDGX5ATpzhByRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gqf8DThY1Rgt59JEAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"True Postitive Found: True Label: 1 Predicted: 1, @ Test Data Index: 304, Size; (1, 28, 28)\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADYZJREFUeJzt3W2MVPUVx/HfgbbRKIkoQVelFRFrjQ9oNlgCUdRYbUMCRjAlUWlSXV9gUg0viiZGEm3SNPSBVyZLJEB8KGgL8kJbiQ/BJo0RkQAtFg2him7YNoCoiRHw9MVemhV2/jM7c592z/eTkJ255869JxN+e+/s/879m7sLQDxjqm4AQDUIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoL5V5s7MjMsJgYK5u7WyXkdHfjO7zcz+ZWYfmNnSTrYFoFzW7rX9ZjZW0h5Jt0jaL+ltSQvd/Z+J13DkBwpWxpF/uqQP3H2vu38l6Y+S5nawPQAl6iT8F0j6aNDz/dmybzCzHjPbamZbO9gXgJx18ge/oU4tTjmtd/deSb0Sp/1AnXRy5N8vadKg5xdK+qSzdgCUpZPwvy1pqplNNrPvSPqppE35tAWgaG2f9rv7MTN7QNJfJY2VtMrd/5FbZwAK1fZQX1s74zM/ULhSLvIBMHIRfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EFTbU3RLkpntk/SZpOOSjrl7dx5NASheR+HP3Oju/81hOwBKxGk/EFSn4XdJr5jZO2bWk0dDAMrR6Wn/THf/xMwmStpsZu+5+5bBK2S/FPjFANSMuXs+GzJbJulzd1+eWCefnQFoyN2tlfXaPu03szPMbNyJx5J+JGlXu9sDUK5OTvvPlbTBzE5s51l3/0suXQEoXG6n/S3tjNP+0nV1dSXrl19+eUmdnGrLli3J+tGjR0vqZHQp/LQfwMhG+IGgCD8QFOEHgiL8QFCEHwgqj2/1oWDz5s1L1qdPn96wduWVVyZfO2fOnLZ6ysM555yTrB88eLCkTmLiyA8ERfiBoAg/EBThB4Ii/EBQhB8IivADQTHOnznttNOS9TFjGv+eXLx4cfK1d9xxR1s9nTB58uRkfeLEiR1tvypTp05N1lesWJGsP/zww8n6G2+80bBW5lfZ64ojPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ENWrG+SdMmJCsz5gxI1lfvrzhREOSpEsvvXTYPZXl9ddfb1j74osvSuxkeK6//vpk/brrrkvWX3vttWT99NNPb1j78ssvk6+NgCM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTVdJzfzFZJmiOp392vyJadLWmdpIsk7ZN0p7sfKq7N5q6++upkfdOmTSV1kr/nn38+We/p6WlYO3z4cN7tfMPMmTOT9fnz5zesPf7443m3g2Fo5ci/WtJtJy1bKulVd58q6dXsOYARpGn43X2LpJOnTpkraU32eI2k9JQyAGqn3c/857p7nyRlP0fmfaSAwAq/tt/MeiQ1/lAKoBLtHvkPmFmXJGU/+xut6O697t7t7t1t7gtAAdoN/yZJi7LHiyS9mE87AMrSNPxm9pykv0v6vpntN7OfS/q1pFvM7H1Jt2TPAYwgTT/zu/vCBqWbc+6l1o4ePdqwduTIkeRr77333mR97969yfrHH3+crBc5ln/VVVcl6+vXr0/Wzz///DzbGZaVK1c2rN19990ldlJPXOEHBEX4gaAIPxAU4QeCIvxAUIQfCGrU3Lp79uzZhW5/586dDWsLFzYaDR2wZ8+evNvJTbPbY7/88svJ+vjx4/NsJ1dTpkypuoVa48gPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0GNmnH+cePGFbr9a6+9tmFt9erVydemrhGo2o033pis13kcH53hyA8ERfiBoAg/EBThB4Ii/EBQhB8IivADQZm7l7czs8J2NmvWrGT9pZdeStaLvk5gpOrr60vWjx8/nqwvW7asYW3p0vTkzpdcckmy3sy2bdsa1m666abkaz/99NOO9l0ld7dW1uPIDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBNR3nN7NVkuZI6nf3K7JlyyTdJ+k/2WqPuHt6IF3FjvM3M3/+/GR97ty5yfpdd92VZzu52rBhQ8Nas+nDm3nooYeS9UOHDrW97WnTpiXr7777btvbbuaJJ55I1h999NHC9l20PMf5V0u6bYjlv3f3adm/psEHUC9Nw+/uWyQdLKEXACXq5DP/A2a2w8xWmRn3egJGmHbD/6SkKZKmSeqT9NtGK5pZj5ltNbOtbe4LQAHaCr+7H3D34+7+taSVkqYn1u1192537263SQD5ayv8ZtY16Ontknbl0w6AsjS9dbeZPSdptqQJZrZf0mOSZpvZNEkuaZ+k+wvsEUABmobf3YeafP6pAnop1AsvvJCsb9y4MVnfvHlznu3kKtV7p+P8GL24wg8IivADQRF+ICjCDwRF+IGgCD8Q1KiZortTx44dS9bXrl1bUidAOTjyA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFBNp+jOdWcVTtGN+jnrrLOS9RUrViTr99xzT9v7fu+995L1BQsWJOu7dtV3npo8p+gGMAoRfiAowg8ERfiBoAg/EBThB4Ii/EBQTe/bb2aTJK2VdJ6kryX1uvsKMztb0jpJF0naJ+lOdz9UXKsYbQ4fPpys79ixo7B9X3bZZcn6jBkzkvU6j/O3qpUj/zFJS9z9B5J+KGmxmV0uaamkV919qqRXs+cARoim4Xf3Pnfflj3+TNJuSRdImitpTbbaGknzimoSQP6G9ZnfzC6SdI2ktySd6+590sAvCEkT824OQHFanqvPzM6U9CdJD7r7EbOWLh+WmfVI6mmvPQBFaenIb2bf1kDwn3H3P2eLD5hZV1bvktQ/1Gvdvdfdu929O4+GAeSjafht4BD/lKTd7v67QaVNkhZljxdJejH/9gAUpelXes1slqQ3Je3UwFCfJD2igc/96yV9V9KHkha4+8Em2+IrvWjZkiVLkvXly5eX1MmpWv3YW4VWv9Lb9DO/u/9NUqON3TycpgDUB1f4AUERfiAowg8ERfiBoAg/EBThB4Li1t2orYsvvjhZX7duXbLe3V3cRaWjYZyfIz8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBNXybbyAsu3duzdZ7+8f8uZRaBFHfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IinF+jFjNvs9/6623NqyNHTs2+dqnn366rZ5GEo78QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBU0/v2m9kkSWslnSfpa0m97r7CzJZJuk/Sf7JVH3H3l5psi/v2ozQ33HBDw9qYMenj3vbt25P1Q4cOtdVTGVq9b38rF/kck7TE3beZ2ThJ75jZ5qz2e3df3m6TAKrTNPzu3iepL3v8mZntlnRB0Y0BKNawPvOb2UWSrpH0VrboATPbYWarzGx8g9f0mNlWM9vaUacActVy+M3sTEl/kvSgux+R9KSkKZKmaeDM4LdDvc7de929292LmzgNwLC1FH4z+7YGgv+Mu/9Zktz9gLsfd/evJa2UNL24NgHkrWn4bWA60qck7Xb33w1a3jVotdsl7cq/PQBFaWWob5akNyXt1MBQnyQ9ImmhBk75XdI+SfdnfxxMbYuhPqBgrQ71NQ1/ngg/ULxWw88VfkBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaDKnqL7v5L+Pej5hGxZHdW1t7r2JdFbu/Ls7Xutrljq9/lP2bnZ1rre26+uvdW1L4ne2lVVb5z2A0ERfiCoqsPfW/H+U+raW137kuitXZX0VulnfgDVqfrID6AilYTfzG4zs3+Z2QdmtrSKHhoxs31mttPMtlc9xVg2DVq/me0atOxsM9tsZu9nP4ecJq2i3paZ2cfZe7fdzH5SUW+TzOx1M9ttZv8ws19kyyt97xJ9VfK+lX7ab2ZjJe2RdIuk/ZLelrTQ3f9ZaiMNmNk+Sd3uXvmYsJldL+lzSWvd/Yps2W8kHXT3X2e/OMe7+y9r0tsySZ9XPXNzNqFM1+CZpSXNk/QzVfjeJfq6UxW8b1Uc+adL+sDd97r7V5L+KGluBX3UnrtvkXTwpMVzJa3JHq/RwH+e0jXorRbcvc/dt2WPP5N0YmbpSt+7RF+VqCL8F0j6aNDz/arXlN8u6RUze8fMeqpuZgjnnpgZKfs5seJ+TtZ05uYynTSzdG3eu3ZmvM5bFeEfajaROg05zHT3ayX9WNLi7PQWrWlp5uayDDGzdC20O+N13qoI/35JkwY9v1DSJxX0MSR3/yT72S9pg+o3+/CBE5OkZj/7K+7n/+o0c/NQM0urBu9dnWa8riL8b0uaamaTzew7kn4qaVMFfZzCzM7I/hAjMztD0o9Uv9mHN0lalD1eJOnFCnv5hrrM3NxoZmlV/N7VbcbrSi7yyYYy/iBprKRV7v6r0psYgpldrIGjvTTwjcdnq+zNzJ6TNFsD3/o6IOkxSRslrZf0XUkfSlrg7qX/4a1Bb7M1zJmbC+qt0czSb6nC9y7PGa9z6Ycr/ICYuMIPCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQ/wN8S+6Q5NK7RgAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"true_positive_found = 0\n",
"true_negative_found = 0\n",
"false_postitive_found = 0\n",
"false_negative_found = 0\n",
"\n",
"true_positive_example = []\n",
"true_negative_example = []\n",
"false_postitive_example = []\n",
"false_negative_example = []\n",
"\n",
"for index in range(0, len(testing_labels)-1):\n",
" true_label = testing_labels[index]\n",
" predicted_label = prediction_labels[index]\n",
" #print('True Label: ' + str(true_label))\n",
" #print('Predicted Label: ' + str(predicted_label))\n",
" if(false_negative_found and false_postitive_found and true_positive_found and true_negative_found):\n",
" break\n",
" elif(not false_negative_found and true_label == 1 and predicted_label == 0):\n",
" false_negative_found = 1\n",
" false_negative_example.append(np.array(images[nbr_of_training_images + index]))\n",
" print('False Negative Found: True Label: ' + str(true_label) + ' Predicted: ' + str(predicted_label) + ', @ Test Data Index: ' + str(index) + ', Size; ' + str(np.array(false_negative_example).shape))\n",
" pyplot.imshow(false_negative_example[0], cmap='gray') # plot the image in \"gray\" colormap\n",
" pyplot.show()\n",
" elif(not false_postitive_found and true_label == 0 and predicted_label == 1):\n",
" false_postitive_found = 1\n",
" false_postitive_example.append(np.array(images[nbr_of_training_images + index]))\n",
" print('False Positive Found: True Label: ' + str(true_label) + ' Predicted: ' + str(predicted_label) + ', @ Test Data Index: ' + str(index) + ', Size; ' + str(np.array(false_postitive_example).shape))\n",
" pyplot.imshow(false_postitive_example[0], cmap='gray') # plot the image in \"gray\" colormap\n",
" pyplot.show()\n",
" elif(not true_positive_found and true_label == 1 and predicted_label == 1):\n",
" true_positive_found = 1\n",
" true_positive_example.append(np.array(images[nbr_of_training_images + index]))\n",
" print('True Postitive Found: True Label: ' + str(true_label) + ' Predicted: ' + str(predicted_label) + ', @ Test Data Index: ' + str(index) + ', Size; ' + str(np.array(true_positive_example).shape))\n",
" pyplot.imshow(true_positive_example[0], cmap='gray') # plot the image in \"gray\" colormap\n",
" pyplot.show()\n",
" elif(not true_negative_example and true_label == 0 and predicted_label == 0):\n",
" true_negative_found = 1\n",
" true_negative_example.append(np.array(images[nbr_of_training_images + index]))\n",
" print('True Negative Found: True Label: ' + str(true_label) + ' Predicted: ' + str(predicted_label) + ', @ Test Data Index: ' + str(index) + ', Size; ' + str(np.array(true_negative_example).shape))\n",
" pyplot.imshow(true_negative_example[0], cmap='gray') # plot the image in \"gray\" colormap\n",
" pyplot.show() "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As illustrated above, we have found one of each possible case; True Negative, True Postitive, False Positive, False Negative. Recall that our truth values are 1 for images with a 9, and 0 in all other cases.\n",
"\n",
"The true negative image shown above is of a 3. Since 3 is not 9, the true label for the image is 0, and the neural network correctly classified it as such.\n",
"\n",
"The true positive image shown above is of a 9. The true label for 9 is 1, and the neuarl network correctly classified it as such.\n",
"\n",
"The false positive image shown above is of a 7. Since 7 is not 9, the true label for the image is 0. However, the nueral network incorrectly classified this image as a 9, thus predicted a 1. This inaccurate classification is relatively understandable since we have a simple single-neuron neural network and the the 7 in this image is bent so that it's features are more similar to a 9 than to a written 7. \n",
"\n",
"The false negative image shown about is an image of a 9. The true label for a 9 is 1. However, our neural network inccorectly classified this image as 'not a 9'. It is not easily possible to know exactly why this occured but our intuition can tell us a couple things. First, we are using a simple single-neuron neural network, so it is generally understandable that it will not be 100% accurate. In fact no nueral network is truly 100% accurate; at least philosophically. Second, we can see in this image that the 9 is round and could potentially be mistaken for a 3 or 0. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have a good model we would like to investigate the impact of the accuracy of our network to lowing the training data size. In order to do some a comparision with our results above we will use 10,000 images for training now, but still use 10,000 images for testing. This will reveal what our results are when we have a much smaller training data set. "
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Final Cost: 2.272081920095334\n",
"Final Percent Accuracy: 92.72\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcXFWd9/HPl3RCNiBbJ0BAAhoHAUWchmFRHxQeBWVzYcSNiDgZfXTclcVxwGdwBsfdGfU1CEgQRJFFeJRBEXHAhWgjyK5kQCASSAPZSEhCkt/zxzmVvl19q7rT6arqdH3fr9d9Vd2l7j1Vt6q+dc69dY8iAjMzs2rbtboAZmY2MjkgzMyslAPCzMxKOSDMzKyUA8LMzEo5IMzMrJQDYgSS9F+S5rW6HGZDIWkvSc8M97LWfA6IAkl/lnRkq8sREUdHxIJGrFvSjpK+IukRSc9IWpTHZzRie8NJ0qsk3SRphaQ/l8yfk+evkXR/vX0paXtJF0paKelxSR+tmn9EXseavM49BvvYqvW8S1JI+lLV9BPy9IsKZQ9JP65a7hJJZ+f7h0taXJi3r6SfSlomabmk2yS9TtLb8759RtKzkjYVxp+pWv/zivNyGVYXxl9R67nVEhEPRsTk4V52S+XXbr2kVXm4S9JnJe24BetYLOnwrSjDiyXdkPfRMkndkl7bjG0PBwdEk0nqaOG2xwE3AvsCRwE7AocCTwEHDWF9zX4uq4ELgU/UmH8ZcDswHfgUcIWkzhrLng3MBfYAXgV8UtJRADksrwI+DUwDuoHvD+axNfwP8Jaq1+tk4E8lyx4s6bA66yr6f8ANwCxgJvBBYGVEXBoRk/MX79HAY5Xx6i/jiHikZN7+hWm3VG9U0phBlm8k+JeI2AHoBE4FXgHcImlCozcsScCPgOtI+2dn4CPAtlNjiggPeQD+DBxZY94xwB3AcuDXwEsK804nfQmsAu4F3lCY9y7gV8CXgaeBc/K0XwJfAJYBDwFHFx7zC+A9hcfXW3ZP4Oa87Z8BXwcuqfEc3gM8AUyu8xoE8ILC+EXAOfn+4cBi4DTgceA7wH3AMYXlO4AngZfl8YPz67Uc+ANw+DDspyOBP1dNeyGwDtihMO0W4L011vEX4DWF8X8Gvpfvzwd+XZg3CXgW2Hugx5Zsp7L/rgden6dNy6/f54GL8rQ5+bU/Dbip8PhLgLOLr3++PyMvP2WA12rzYwb52vbZ/4UyfD0/h9V5nceRPg+rgEeATxeWfwEQhfFfAp/J74NVeT3TtnTZPP+UvL0ngTPz+7H0PVV87QrTdiJ9Bt6bx+cCN5F+JD2Z39M75XmXAZvyvn8G+CjpR/UVef8tJ31WX1Rj+zvn17Pe5+040udieX7u+9Xa9tZ+boYyuAYxCJJeRvrl+vekX6f/CVwrafu8yP+QfpnsRHpzXyJpl8Iq/gZ4kPQr4rOFaX8kfdD/Dbgg/+IoU2/Z7wK/zeU6G3hnnadyJHB9RGzNL5idSV9we5C+SC8D3lqY/1rgyYj4vaTZwI9JoTgN+DhwZZ1f9VtjX+DBiFhVmPaHPL0PSVOBXfP8smX3Lc6LiNWkfbzvIB5by8WkWgPAScA1pECr9nXghYNo6nwKWER6r50gadYAy2+tt5He2zsAvyF9ab2D9J4/FviQpGMGePw8Um1nEunLdouWlfRi4Guk1282qVaw85Y8iYhYQapFV5rORHp/7gLsA+xFqjkSEW8FHiP9IJscEZVmwh+RgmVn4G5SqJRZSvrcXyrpeEkzizMlHQh8i/TDbTrpO+YaSePqbLupHBCD83fAf0bEwojYGOn4wDrSr2Mi4gcR8VhEbIqI7wMP0LfJ5rGI+PeI2BARz+ZpD0fEtyJiI7CA9Aat9SEvXVbS84ADgX+KiPUR8Uvg2jrPYzqwZEivQK9NwFkRsS4/l+8Cx0mamOe/LU+D9AVyXURcl1+bG0jNNa/byjKUmQysqJq2gvSFVrZsZX7ZsvXWNdBja7kaOFzSTqSguLjGcmtJPyLOqbeySD8zX0Wq9X4RWCLpZklzByjHUF0dEb/J+3FdRPw8Iu7O438Avgf8rzqPvyAiHoiINcAPgJcOYdkTgR9GxK8jYh3wj0N8Lo+RfrAQEX+KiBvz52cpqaZf83nk53tRRKyKiLWkH2V/LWlS2bKk2tZf8nqX5ONZz8+LzAe+ERG/y98rF+bpBw7xeQ07B8Tg7AF8LB8IXC5pObA76Zckkk6WdEdh3n6kX/sVj5as8/HKnfxBgN4vn8EuuyvwdGFarW1VPEUKl63Rkz8YlfIsIjUzHZtD4jh6A2IP4MSq1+3lZWWoOrD6X0Mo1zOkYypFO5KaKcqWrcwvW7beugZ6bKkcpj8mfanNiIhf1Vn8W6QfAMcOsM7FEfGBiHg+6bVeTe3g2Vp93leSDpH0C0k9klaQfgXXO9Hh8cL9NdR+r9dbdtdiOXLNbtkgyl5tNqm5F0k7S7pc0l8krSQ1qdZ8HpLGSPo3SQ/m5RflWaWPiYhHI+L/RMRepObg5/I2IO2z06o+H7vk8o0IDojBeRT4bERMKQwTI+IypbNbvgV8AJgeEVNI1c5ic1GjLpm7BJhW+PUOKbhq+Rnw2rJfOwVrgOL6qqvwZc+l0sx0PHBvDg1Ir9t3ql63SRFxbvUKonBgNSKOrlO+Wu4B9pJU/CW/f55eva1lpNdu/xrL3lOcl1+v5wP3DOKx9VwMfIzaTRKV8j1Has75Z/q+j+o95lFS89R+g1l+CKr3+/eAK4HdI2In4HwGWdatsATYrTKS98vULVlBPoPp1aTjUwCfI7UGvDgidiQdM6r32T2ZVAN+Nal57QWVVQ+07Yh4BPgGvfvoUeAzJd8rl9fYdtM5IPobK2l8YeggBcB7Jf2NkkmSXp+/jCaRdmQPgKRTaNyHtI+IeJjUZHO2pHGSDiG1B9fyHdKb8kpJe0vaTtJ0SWdKqjT73AG8Lf9SOor6zQYV3wNeA7yP3toDpIOEx0p6bV7feKVTNXcrXcsAcnnHA2PTqMbnM7OIiD/lsp+Vp78BeAnpS6xyimjxA3cx8I+Spkram9SMeFGedzWwn6Q35e39E3BnRNw/iMfW89/A/wb+fRDLfgfYnnS2WdlrMVXSZyS9IL8uM4B3A7cOYt3DYQdS7XWtpINJxwUa7QfACZIOzvv9/w72gfk90UU69tNDb01rB1LNa4Wk3UnHyYqeIB2XoLD8OlJtfCK9xxTLtjlD0llK//VQPvZ2Cr376Dzg/ZIOzPMnSzq28AOuettN54Do7zrSmQOV4eyI6CZ9CfwHqUq7iPRLg4i4l9QG/BvSDn0x6aylZnk7cAjpDXsO6XTMsoOf5HbbI4H7SadHriQd4J4BLMyLfYgUMsvzun84UAEiYgnp+R9K4XTQ/Kv2eNLZJj2kcPoEQ3/fvZK0T64Dnpfv/7Qw/ySgi7SPzgXeHBE9ed7uuYwVZ5EOPD9M+uL+fERcn8vdA7yJ9OFfRjpJ4KTBPLaeSG6MiKcHsezGvJ1pNRZZTzrz6Wek/Xg3ab+/a6B1D5P3Af8qaRVp/14+wPJbLSLuJJ0m+gPScYSn8lD6fs/OzGV8knT87lbgsEKz7Fmk44UrSMfvrqx6/L8An8lNQB8Gvp23/Rip1vjrOtteR6p53kRqmrwr3747P5+FpNfxm6T32Z9Ix+1qbbvplI512Wgh6fvA/RFxVqvLMpJIOh/4QUT8pNVlseGRm4uWA3vkHyM2zBwQ2zilU+WeJv0/4jWkX/yHRMTtLS2YWQNIOo5Ua9qOdGbQARHR1dpSjV5uYtr27Uz6s84zpHPE3+dwsFHsDaTmncWkJra31l3atoprEGZmVso1CDMzK9WyC8cNhxkzZsScOXNaXQwzs23Kbbfd9mREDHjJm206IObMmUN3d3eri2Fmtk2R9PBglnMTk5mZlXJAmJlZKQeEmZmVckCYmVkpB4SZmZVqWEAodeq+VNLdhWmfV+oI/k5JV0uaUph3hqRFkv6oQXbqbWZmjdPIGsRF9L9U8Q2kPldfQrpy4RkAkvYhXS1z3/yYb2jb6hjdzGzUaVhARMTN5F6bCtN+GhEb8uit9Hb+cTyp0/d1EfEQ6XLaxS47h9fdd8OnPw09PQMva2bWplp5DOLdQKVrydn07dJwMTW63ZM0X1K3pO6eoX7B//GPcM45sGRru2c2Mxu9WhIQkj4FbAAurUwqWaz0KoIRcV5EdEVEV2fngP8ULzdhQrp99tmhPd7MrA00/VIbkuYBxwBHRO+lZBfTty/l3UiX9G2MibnL5TVr6i9nZtbGmlqDyH0cnwYcV+jyD1JXfydJ2l7SnsBcUleYjeGAMDMbUMNqEJIuAw4HZkhaTOr79QxSR+w3SAK4NSLeGxH3SLocuJfU9PT+3CdvY1QCwk1MZmY1NSwgIqKsp6cL6iz/WVIn8Y1XOQbhGoSZWU3t+U9qNzGZmQ2ovQPCTUxmZjW1Z0C4icnMbEDtGRBjx8KYMQ4IM7M62jMgpNTM5CYmM7Oa2jMgIDUzuQZhZlZT+wbExIkOCDOzOhwQZmZWqn0DYsIEH4MwM6ujfQPCNQgzs7ocEGZmVqp9A8JNTGZmdbVvQLgGYWZWlwPCzMxKtW9AuInJzKyu9g0I1yDMzOpq74BYvx42Nq7jOjOzbVn7BkTlkt9uZjIzK9W+AeFe5czM6nJAOCDMzEq1b0C4icnMrK72DQjXIMzM6nJAOCDMzEo5INzEZGZWqn0DonIMwjUIM7NSDQsISRdKWirp7sK0aZJukPRAvp2ap0vS1yQtknSnpJc1qlybuYnJzKyuRtYgLgKOqpp2OnBjRMwFbszjAEcDc/MwH/hmA8uVuInJzKyuhgVERNwMPF01+XhgQb6/ADihMP3iSG4FpkjapVFlA9zEZGY2gGYfg5gVEUsA8u3MPH028GhhucV5Wj+S5kvqltTd09Mz9JK4icnMrK6RcpBaJdOibMGIOC8iuiKiq7Ozc+hbHD8+3bqJycysVLMD4olK01G+XZqnLwZ2Lyy3G/BYQ0uy3XYpJFyDMDMr1eyAuBaYl+/PA64pTD85n810MLCi0hTVUO4Twsyspo5GrVjSZcDhwAxJi4GzgHOByyWdCjwCnJgXvw54HbAIWAOc0qhy9eGAMDOrqWEBERFvrTHriJJlA3h/o8pSk7sdNTOraaQcpG4N1yDMzGpyQDggzMxKtXdAuInJzKym9g4I1yDMzGpyQDggzMxKtXdAuInJzKym9g4I1yDMzGpyQDggzMxKtXdAVJqYovS6gGZmba29A2LixBQO69a1uiRmZiOOAwLczGRmVqK9A6LSq5zPZDIz66e9A8I1CDOzmhwQ4IAwMyvhgAA3MZmZlWjvgKgcg3ANwsysn/YOCDcxmZnV5IAANzGZmZVo74BwE5OZWU3tHRBuYjIzq8kBAQ4IM7MS7R0Q/ie1mVlN7R0QY8dCR4drEGZmJdo7IMB9QpiZ1dCSgJD0EUn3SLpb0mWSxkvaU9JCSQ9I+r6kcU0pjLsdNTMr1fSAkDQb+CDQFRH7AWOAk4DPAV+OiLnAMuDUphTINQgzs1KtamLqACZI6gAmAkuAVwNX5PkLgBOaUhIHhJlZqaYHRET8BfgC8AgpGFYAtwHLI2JDXmwxMLspBXITk5lZqVY0MU0Fjgf2BHYFJgFHlyxa2lG0pPmSuiV19/T0bH2BXIMwMyvViiamI4GHIqInIp4DrgIOBabkJieA3YDHyh4cEedFRFdEdHV2dm59aRwQZmalWhEQjwAHS5ooScARwL3ATcCb8zLzgGuaUho3MZmZlWrFMYiFpIPRvwfuymU4DzgN+KikRcB04IKmFMg1CDOzUh0DLzL8IuIs4KyqyQ8CBzW9MA4IM7NS/ie1m5jMzEo5IFyDMDMr5YCYOBGeew42bBh4WTOzNuKAcLejZmalHBDudtTMrJQDwr3KmZmVckC4icnMrJQDotLEtHp1a8thZjbCOCBcgzAzK+WAmDQp3boGYWbWhwPCB6nNzEo5IBwQZmalHBCVJiYHhJlZHw6ISg3CxyDMzPpwQLiJycyslANi7Ng0OCDMzPpwQECqRbiJycysDwcEuE8IM7MSgwoISd8ZzLRtlgPCzKyfwdYg9i2OSBoD/PXwF6dFJk1yE5OZWZW6ASHpDEmrgJdIWpmHVcBS4JqmlLAZXIMwM+unbkBExL9GxA7A5yNixzzsEBHTI+KMJpWx8RwQZmb9DLaJ6UeSJgFIeoekL0nao4Hlaq5JkxwQZmZVBhsQ3wTWSNof+CTwMHBxw0rVbD7N1cysn8EGxIaICOB44KsR8VVgh8YVq8ncxGRm1s9gA2KVpDOAdwI/zmcxjR3qRiVNkXSFpPsl3SfpEEnTJN0g6YF8O3Wo699ibmIyM+tnsAHxFmAd8O6IeByYDXx+K7b7VeD6iNgb2B+4DzgduDEi5gI35vHmcBOTmVk/gwqIHAqXAjtJOgZYGxFDOgYhaUfglcAFed3rI2I5qflqQV5sAXDCUNY/JBMnwrp1sHFj0zZpZjbSDfaf1H8L/BY4EfhbYKGkNw9xm3sBPcC3Jd0u6fx8htSsiFgCkG9n1ijLfEndkrp7enqGWIQq7pfazKyfwTYxfQo4MCLmRcTJwEHAp4e4zQ7gZcA3I+IAYDVb0JwUEedFRFdEdHV2dg6xCFXcL7WZWT+DDYjtImJpYfypLXhstcXA4ohYmMevIAXGE5J2Aci3S2s8fvi5Twgzs34G+yV/vaSfSHqXpHcBPwauG8oG8/GMRyX9VZ50BHAvcC0wL0+bRzMv5eGAMDPrp6PeTEkvIB0b+ISkNwIvBwT8hnTQeqj+AbhU0jjgQeAUUlhdLulU4BHS8Y7mcL/UZmb91A0I4CvAmQARcRVwFYCkrjzv2KFsNCLuALpKZh0xlPVtNfdLbWbWz0BNTHMi4s7qiRHRDcxpSIlawU1MZmb9DBQQ4+vMmzCcBWkpNzGZmfUzUED8TtLfVU/Mxwlua0yRWsBNTGZm/Qx0DOLDwNWS3k5vIHQB44A3NLJgTeUmJjOzfuoGREQ8ARwq6VXAfnnyjyPi5w0vWTO5icnMrJ+BahAARMRNwE0NLkvrTMiHUxwQZmabDfXf0KNLRweMG+djEGZmBQ6ICncaZGbWhwOiwp0GmZn14YCocKdBZmZ9OCAq3MRkZtaHA6LCTUxmZn04ICrcxGRm1ocDosJNTGZmfTggKtzEZGbWhwOiwjUIM7M+HBAVPgZhZtaHA6LCNQgzsz4cEBWTJsH69bBhQ6tLYmY2IjggKtwnhJlZHw6ICgeEmVkfDogKdxpkZtaHA6LC/VKbmfXhgKhwE5OZWR8tCwhJYyTdLulHeXxPSQslPSDp+5LGNbVAbmIyM+ujlTWIDwH3FcY/B3w5IuYCy4BTm1oa1yDMzPpoSUBI2g14PXB+HhfwauCKvMgC4ISmFsrHIMzM+mhVDeIrwCeBTXl8OrA8Iir/UlsMzG5qidzEZGbWR9MDQtIxwNKIuK04uWTRqPH4+ZK6JXX39PQMX8HcxGRm1kcrahCHAcdJ+jPwPVLT0leAKZI68jK7AY+VPTgizouIrojo6uzsHL5SuYnJzKyPpgdERJwREbtFxBzgJODnEfF24CbgzXmxecA1TS3YhAnp1jUIMzNgZP0P4jTgo5IWkY5JXNDUrY8ZA+PHOyDMzLKOgRdpnIj4BfCLfP9B4KBWlseX/DYz6zWSahCt506DzMw2c0AUuV9qM7PNHBBFbmIyM9vMAVHkJiYzs80cEEVuYjIz28wBUeQmJjOzzRwQRW5iMjPbzAFR5BqEmdlmDogiH4MwM9vMAVHkGoSZ2WYOiKKJE+G559JgZtbmHBBF7jTIzGwzB0SROw0yM9vMAVHkToPMzDZzQBS5icnMbDMHRJGbmMzMNnNAFE2enG5XrmxtOczMRgAHRNHMmel26dLWlsPMbARwQBTNmpVun3iiteUwMxsBHBBFO+4I22/vgDAzwwHRl5RqEQ4IMzMHRD8OCDMzwAHRnwPCzAxwQPQ3c6YDwswMB0R/s2al01w3bWp1SczMWqrpASFpd0k3SbpP0j2SPpSnT5N0g6QH8u3UZpcNSAGxcSM8/XRLNm9mNlK0ogaxAfhYRLwIOBh4v6R9gNOBGyNiLnBjHm8+/xfCzAxoQUBExJKI+H2+vwq4D5gNHA8syIstAE5odtkAB4SZWdbSYxCS5gAHAAuBWRGxBFKIADNrPGa+pG5J3T09PcNfqEpA+HIbZtbmWhYQkiYDVwIfjohBXx0vIs6LiK6I6Ors7Bz+grkGYWYGtCggJI0lhcOlEXFVnvyEpF3y/F2A1vyEnzoVOjocEGbW9lpxFpOAC4D7IuJLhVnXAvPy/XnANc0uGwDbbef/QpiZAR0t2OZhwDuBuyTdkaedCZwLXC7pVOAR4MQWlC3xv6nNzJofEBHxS0A1Zh/RzLLU5IAwM/M/qUs5IMzMHBClKgER0eqSmJm1jAOizMyZsH49rFjR6pKYmbWMA6KM/wthZuaAKOWAMDNzQJRyQJiZOSBKOSDMzBwQpWbMSP+odkCYWRtzQJQZMyaFhAPCzNqYA6IW/1nOzNqcA6KWSt/UZmZtygFRi2sQZtbmHBC1OCDMrM05IGqZNQvWrIFnnml1SczMWsIBUYv/C2Fmbc4BUcvMmen2scdaWw4zsxZxQNSyzz4wdiy86U1w4YWwaVOrS2Rm1lQOiFr22AMWLoS5c+HUU+HQQ+GLX4Sf/xyWLWt16czMGq4VfVJvOw44AG65BS65BM4+Gz7+8d5506alENljD9h1V9hllzTMmgWdnamJasYMmDwZVKuHVTOzkUuxDfea1tXVFd3d3c3bYE8P3H473HEHPPQQPPxwGpYsqV2rGDcOpk9Pw7RpaZg6tXeYMqV32Gkn2HHH3tsddkjNXGZmw0jSbRHRNdByrkFsic5OeM1r0lBt3Tp4/PH07+vK8NRT8OSTaXj66TQsWpTCZNmydBrtQMaPT0ExeXLv7aRJvbfFYeLE3mHChN7b4jB+fO9tZejw28DM+vM3w3DZfvveJqfBWrcudWu6YgUsXw4rV/aOr1qVxleuTP/FWLWq93bVqlRreeaZFDKrV6dhqMaMSeWvNYwb1/+2OIwd2/+2bOjo6HtbPa0yVI8XhzFj+t8W77s5z2zYOCBaafvt07GKyim1WyMC1q5NgbFmDTz7bO/9tWvT+LPP9r2/bl0a1q7tva3cX7++d37l/urVvfefe67v/cr4+vWpLK2y3Xa9oVEdHvWG4uMq98um1Vu2+n7Z+EDTywZp8MsUl92S+8NxW32/3rzB3K8eas0baDrUn2c1OSBGC6m3GWn69NaWZePG3tCoDBs21L5fGS8+buPGvtOL9yvjGzb0HS+7X29acdi0qfb9TZvStteuLV+usky9+xEDT6/c34aPC26TagXScA1bso1ay1amF8v8nvfARz/a0JfGAWHDr/Lrevz4Vpdk2xTRPzgiygOlOL0sbCqBM5j7xYAayvTqaWXjW3q/1noGO734eg7mMcM91Nr+lixbmV58f0Dv1R4aaMQFhKSjgK8CY4DzI+LcFhfJrLmKzSA+gcBaaET9UU7SGODrwNHAPsBbJe3T2lKZmbWnERUQwEHAooh4MCLWA98Djm9xmczM2tJIC4jZwKOF8cV52maS5kvqltTd09PT1MKZmbWTkRYQZeed9TmlIyLOi4iuiOjq7OxsUrHMzNrPSAuIxcDuhfHdAF9v28ysBUZaQPwOmCtpT0njgJOAa1tcJjOztjSizqGLiA2SPgD8hHSa64URcU+Li2Vm1pZGVEAARMR1wHWtLoeZWbvbpi/3LakHeHgLHjIDeLJBxRmp/Jzbg59zexiu57xHRAx4ls82HRBbSlL3YK6BPpr4ObcHP+f20OznPNIOUpuZ2QjhgDAzs1LtFhDntboALeDn3B78nNtDU59zWx2DMDOzwWu3GoSZmQ2SA8LMzEq1TUBIOkrSHyUtknR6q8vTCJJ2l3STpPsk3SPpQ3n6NEk3SHog305tdVmHk6Qxkm6X9KM8vqekhfn5fj9ftmXUkDRF0hWS7s/7+pA22Mcfye/puyVdJmn8aNvPki6UtFTS3YVppftVydfy99mdkl7WiDK1RUC0UUdEG4CPRcSLgIOB9+fneTpwY0TMBW7M46PJh4D7CuOfA76cn+8y4NSWlKpxvgpcHxF7A/uTnvuo3ceSZgMfBLoiYj/SZXhOYvTt54uAo6qm1dqvRwNz8zAf+GYjCtQWAUGbdEQUEUsi4vf5/irSF8ds0nNdkBdbAJzQmhIOP0m7Aa8Hzs/jAl4NXJEXGW3Pd0fglcAFABGxPiKWM4r3cdYBTJDUAUwEljDK9nNE3Aw8XTW51n49Hrg4kluBKZJ2Ge4ytUtADNgR0WgjaQ5wALAQmBURSyCFCDCzdSUbdl8BPgnkHuqZDiyPiA15fLTt672AHuDbuVntfEmTGMX7OCL+AnwBeIQUDCuA2xjd+7mi1n5tyndauwTEgB0RjSaSJgNXAh+OiJWtLk+jSDoGWBoRtxUnlyw6mvZ1B/Ay4JsRcQCwmlHUnFQmt7sfD+wJ7ApMIjWxVBtN+3kgTXmft0tAtE1HRJLGksLh0oi4Kk9+olL9zLdLW1W+YXYYcJykP5OaDV9NqlFMyU0RMPr29WJgcUQszONXkAJjtO5jgCOBhyKiJyKeA64CDmV07+eKWvu1Kd9p7RIQbdERUW5/vwC4LyK+VJh1LTAv358HXNPssjVCRJwREbtFxBzSPv15RLwduAl4c15s1DxfgIh4HHhU0l/lSUcA9zJK93H2CHCwpIn5PV55zqN2PxfU2q/XAifns5kOBlZUmqKGU9v8k1rS60i/LisdEX22xUUadpJeDtwC3EVvm/yZpOMQlwPPI33YToyI6oNh2zRJhwMfj4hjJO1FqlFMA24H3hER61pZvuEk6aWkg/LjgAeBU0g/9kbtPpb0GeAtpDP1bgfeQ2pzHzX7WdJlwOGkS3o/AZwF/JCS/ZqD8j9IZz2tAU6JiO5hL1O7BISZmW2ZdmliMjOzLeSAMDOzUg4IMzMr5YAwM7NSDggzMyvlgLC2JumZfDtH0tuGed1nVo3/ejjOFHxAAAACHElEQVTXb9ZoDgizZA6wRQGRrxJcT5+AiIhDt7BMZi3lgDBLzgVeIemO3PfAGEmfl/S7fL39v4f0h7zc58Z3SX9IRNIPJd2W+yuYn6edS7r66B2SLs3TKrUV5XXfLekuSW8prPsXhb4eLs1/iELSuZLuzWX5QtNfHWtLHQMvYtYWTif/Exsgf9GviIgDJW0P/ErST/OyBwH7RcRDefzd+d+tE4DfSboyIk6X9IGIeGnJtt4IvJTUl8OM/Jib87wDgH1J19X5FXCYpHuBNwB7R0RImjLsz96shGsQZuVeQ7rWzR2kS5VMJ3XOAvDbQjgAfFDSH4BbSRdQm0t9Lwcui4iNEfEE8N/AgYV1L46ITcAdpKavlcBa4HxJbyRdWsGs4RwQZuUE/ENEvDQPe0ZEpQaxevNC6RpQRwKHRMT+pGsCjR/EumspXktoI9CR+zw4iHSV3hOA67fomZgNkQPCLFkF7FAY/wnwvnz5dCS9MHfMU20nYFlErJG0N6mr14rnKo+vcjPwlnyco5PUQ9xvaxUs9++xU0RcB3yY1Dxl1nA+BmGW3AlsyE1FF5H6fZ4D/D4fKO6hvEvL64H3SroT+COpmaniPOBOSb/PlyGvuBo4BPgDqZOXT0bE4zlgyuwAXCNpPKn28ZGhPUWzLeOruZqZWSk3MZmZWSkHhJmZlXJAmJlZKQeEmZmVckCYmVkpB4SZmZVyQJiZWan/D9gis1P1lvVzAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xe8XGWdx/HPN7nphSQkoQUISBQBpQUEsbDiFlQE10oNgiKuStNVYN0VO7gutlVWBN2AEakCsitLURAsgYQOAWFpCaRBekjCvclv/3iekzu5e8vc5M5MMuf7fr3mNXPKnPOcOTPP95znlFFEYGZm5dWv0QUwM7PGchCYmZWcg8DMrOQcBGZmJecgMDMrOQeBmVnJOQjMzEpuiw4CSc9KWiVphaT5kn4maXijy1Upl/GdVYy3i6R1kn5Uj3I1iqTtJF0qaa6k5ZIel/RlScMaXbaeSPqQpD9KekXSHZ0M30fSzDx8pqR9upnWGEm/krRS0nOSjukw/Jjcf6Wk6yWNqfa9HaZznqSQdFqH/mfk/ufl7kNz9w87jHe3pBPz6xMl3V0x7C3581gqaZGkP0g6QNK5+Te5QtJqSWsruh/tMP23VgxbmcuwouKxU1fL1h1Jg/O0JnQzzqmS2irm9bSkSyS9phfz+aWkL25MGfP7t5Z0Wa6/luXfw5n1mHelLToIsiMiYjiwH3AA0OsPRlJLn5eq904AFgMfkTSonjOu1/LnyuxPwBDg4IgYAfw1MAqo+sdXMb16r7dFwHeB8zspy0DgBuDnwGhgKnBD7t+ZHwKvAtsAxwIXSdozT2tP4MfA8Xn4K8CPqnlvF/4CTOnQ74Tcv9JK4ARJE7uZFrmMI4GbgB8AY4AdgC8DayLiGxExPP8uTwX+VHRHxAbljIi7KsYtho2qGP/5nsqyie7I894K+Nvcb6ak19V4voV/BwS8lvQ7+Hvg2TrNu11EbLGP/IG9s6L7X4Gb8uutgEuBucALwNeA/nnYicAfgO+Qftxfy/0/DswClgOPAfvl/tsD1wILgWeA0yrmeR5wFXBZft+jwOQ87HJgHbAKWAF8vptl+V/gk8B84AMdhu0J3JrLOh84N/fvD5yb37scmAnsCEwEAmipmMYdwMe6Wn5SRfxb4GXgJWAa6QdZvH9H4Lr8GbxM+gIPyu9/Q8V44/PyjutkGb8GPAz06+Iz6G25vwksAfaqGH9cnv/43P0e4IE83h+BN/bB9+5jpAqkst/f5O+ZKvo9D/xdJ+8fRqrIX1vR73Lg/Pz6G8AvKoa9Jo8/oqf3djKv80jhNAvYs+L7NCv3Py/3OxSYQ6rYf1bx/ruBEys+/7vz68nAkio+q/XvqWLc/7f+c/8xpN/XPGA28KXiOwTsnsu4NH83L8v978nTWkn67R3VyfxOBW7rpP9twM/z6xbSb39+/g79DnhdHnYa0AqsyfO4Ovf/F1I9sRx4BHh3N8v8VGffkYrhe5F+l4vzOjuqu3lv7KMZ9ggAkLQj8C7g/txrKtAG7AbsS/qhfqziLW8CniZVXF+X9EHSj+YEYCTwXuBlSf2AXwMPkrZ6DgPOkPS3FdN6L/BLUqLfSKokiYjjSZXBEZG2br7VRdnfCkzI07gql6EYNoL0xbyZFEi7AbfnwWcBR+flHgmcRNp6rMYGy0/aKvlmnsfrSRX/ebkM/Ulbf8+Rfqw7AL+MiDW5zMdVTPdo0o9rYSfzfCdwXUSsq7KMPZX7K6RwOrpi+IeAOyNigaT9gJ8CnwC2Jm1l31ijPa49gYci/0qzh2jfyq30WmBtRFRukT9YMe6euRuAiPhfcuVfxXu7cjnt36sppIq1M18H3l/FFvFfgLWSpko6XNLoHsbfFNNIFf2uwIHAUaS9JUjf2etJv72dSOsY4G35+XX5t3d9L+Z3HfDWiu4bSWG8LfA4qW4hIr5PComv5nl8MI//BPBm0sboBcAvJY3tYl5/Bi6QNEXSbpUD8l7XraQN2rGk9fdTSbt1M++N0gxBcL2kJaStgjuBb0jaBjgcOCMiVkbEAtJW5Ecq3vdiRPwgItoiYhUpJL4VEfdG8lREPEdqbhoXEV+JiFcj4mngJx2mdXdE/HdErCX94Pbu5TJMAX4TEYuBXwCHSxqfh70HmBcR/xYRqyNieURMz8M+BnwxIp7IZX4wIl6ucp4bLH9e3lsjYk2uxC8E3p7HPZAUEP+YP8/VEVG0FU8FjsmBCekHenkX89yatIe2KTqut1+wYRAck/tB2sP7cURMj4i1ETGVtAV10CaWoTPDSZVVpaWkrfjejtvd8N7Mp9LPgaMlDSB9d3/e2UgRMQ/4D1LIdikilgFvIW11/wRYKOnG/NvrM5J2JlXqZ0XEKxExF/g+7b+/VtLGybb5e/yHPpjti6S9EPL3bGpErIiI1aTmrwMlDe7qzRFxZUTMjYh1EXE5aU9x/y5G/wSpQj8TeFzSE2o/pvg+4JGImJa/v/eSNkrf3wfLuIFmCIKjImJUROwcEf+QK4edgQHAXElLclD8mLQVWZjdYTo7kppYOtoZ2L6YTp7WuaT22cK8itevAIOrbb+WNAT4IGmrh4j4E2kvojgA2FW5ehrWkw2WX9L4fPDpBUnLSBVFsRWzI/BcRLR1nEgOpZXA2yXtTtpjubGLeb4MbLeR5e203KTd5iGS3pQrjX2AX+VhOwOf7bDudiSF2gY6HOD8j40o1wrSXlmlkaTmgd6O293w3sxnvUht7U+Rmp2ejIiOn2OlC4C/ldTtBk1EzIqIEyNiAqkJY3vSMZS+tDMwmBQ0xTr8Hu2/vzOBocD9kh6SdFwX0+mNHUhNj0hqkfTtfCB5GWmPQKSNmk5JOjmXpSjvbrT/ljaQN6y+EhH75Gn+Grg2twTsDLytw/f3/Wz6b+j/aYYg6Mxs0pbf2BwSoyJiZGx4oKrjbVdn0/kBy9nAMxXTGRURIyLiXVWWpafbu76P9EP+kaR5kuaRvojFbnxX5epu2Mr8PLSi37Y9lOubud8bI2IkqblHFfPZqZtwm5rHPx64Jm85deY24H0Vew+bXO7czHQVaa/gGNIxoqJSnA18vcO6GxoRV3SccVQc4IyIU7soX3ceBd4oSRX93pj7d/QXoEXSpIp+e1eM+ygVe5WSdiUdj/lLFe/tzmXAZ+m6WQiAvFf5XeCrVUyzeM/jwH+SAqEvzSaF3+gOv+X98nxfiIiTSJXjaaSmk53o+XfXnaOAu/Lrj5Kalf+K1NSze+5frOcN5iPptaTjLKcAYyJiFCmAK78XnYqIpaQTEUaSmrlmA7d0+P4Oj4gzOpv3pmjKIMi7j7cA/yZppKR+kl4j6e3dvO0S4HOS9leyW97CvAdYJukLkoZI6i9pL0kHVFmc+aS2za5MIbVjv4G0NbsPcAiwj6Q3kNrmt1U63W+QpBGS3lRR5q9KmpTL/EZJW+emnReA43J5T6Lns3JGkH5wSyTtAPxjxbB7SE0650sapnRq3iEVwy8nBdpxdF/JXEj6kk/Nny2SdpB0oaQ3bmS5ITUFfZh0Bs0vKvr/BDg17y0ol/3deWur13KZBpMOIPbLn8OAPPgOYC1wWl5Pn879f5vfe6KkZyFtBZLaob+Sy3QIcCTtTWrTgCOUTq0cRj4WkpsFe3pvd64kVWpXVTHuhaR27td38VnsLumzyqdnKh2jO5rU5t1nIuKZPM1v5e9+v/x9f0ue74clbR8RQTqYC9AW6fhVcVyhR3ndvkbSj0lNoV/Lg0YAq0l7s8Mq+hc6/r6Hk04QWUj6jpxK2iPoar7nSdpP0oDcOnAa6WSNp0jHPvbNyzhA0kBJB+Ww6WzeG60pgyA7ARhIOvtnMXAN3exSRcTVpANlvyDtZl9PSvS1wBGkCvoZ0kq6hLR1UI1vAl/Mu3afqxyQK9zDgO9GxLyKx0zSweEpeev2r3MZ5gFPkrZOIP1YryKF3jLSQaUhedjHSZX5y6QDiX/soZxfJp2CuxT4L1JlU3w2xWewG6nZag6p4i2GzwHuI22h3EUXImIRqXJpBaZLWk468L2U9MXfmHJXNk9tD/ymov+MPL1/J30HniKdxbKxjiedkXQR6WDiKlLYEBGvkrYkTyBVSCeRmi1fze/dkXTGU+EfSOtqAXAF8MmIeDRP61HSGS3T8vARefwe39ud3IZ+W24+7WncZcC3yG3lnVhOOnA/XdJKUmX9CGmPo68dTToY/DipyeZK2puGDiad7rkCuBo4JSJezMP+Bbg6//be28W0D83vXUb6Lg4knfX3eB5+KalSn0c64+3uDu+/GDggz+OXEXEf6RjLDNLG0y75dVf6kZphF5F+V4cA78rH6haTTmn9aJ7Wi6QgKjY+Nph3N/PokSL6bO/CSkzST0kHcvvkApdmI+kW4PSImNXosph15CCwTaZ0AdIDwL55V97MtiDN3DRkdSDpq6QmgX91CJhtmbxHYGZWct4jMDMruc3hZms9Gjt2bEycOLHRxTAz26LMnDnzpYgY19N4W0QQTJw4kRkzujsDy8zMOpL0XDXjuWnIzKzkHARmZiXnIDAzKzkHgZlZyTkIzMxKzkFgZlZyDgIzs5LbIq4jMDPbYqxbB6tXw6uvpsfq1bBiRXq88kp7/+KxZk16bm3t/HH66TC2q7887hsOAjPbckRAW1uqOCv7tba2V6yrVsHKlanSXb26/REB/fpB//7tFfTq1akiLt5fVL5tbbB8OSxalB4rV7ZPvxje8bFmTfs8+4oExxzjIDCzzVxEeqxZkyrPZcvSY+nS9LxyZdpKXreuvcIsKuHKreKi/8qV8PLL8NJL6XnVqvYKe82a+ixT//4wfDiMGZMew4bBiBEwcCAMGJAeLS1pvOL1gAFpvGHDYNCg9Bg4MD2PGJGmN3Roe/9iWOU0Oz7696/L4joIzJpVRNpCLSrmVavaK93ly1Mlu2hRev3KK+mxcmXqXrGifat61ar2R1F5V24Jb+odjFtaUmU4eHB6DB0KW28N220He+6ZugcPbq9cBw1KlWTl30MXlenAgWn8oUNhyJAN39uvH6xdmwJpwID2+VVWxgMHpspXPf7FcFNxEJhtjpYvhxdegHnz2ivfNWtSBV1saRcV8+rVG1bkL70Ec+fC/PmpGaMaAwakirPY8i22XkePhh12SMOGDEkVZ+UWcdHUUmwZb7UVjByZplG8HjYsDS/GLSrgYqu4Tlu91jUHgVktrFwJs2fDc8/Biy+2Hyxctqy93Xnx4g3bsIuKvDio2JNiK3rQoPat4KFDU3vynnvCNtukLeuiYh46tH3Ld+TI9maPkSNTPystB4FZV1atSlvlc+bAggXtlfmSJal7wYLUvFJsoVc2sbS1dT7NlpZUOY8ZA6NGpcp51Kj2JpFhw9LzttvC9tun52HD2reeiy3t4cPTtMz6gL9J1jzWrk3NIosXb3jAcsmS9Fi6tL3SrnysXLnhGSSVp/11ZfRoGD8+bX1vsw1MmpQq6aIiHzECdtwRdt45VehF5T1wYOnan23z5yCwzdvatakZ5aWXYOHC1Gb+wgupuWX+/LRVPn9+++u1a7uelpSaQSrbsMeMgQkT2tu/K8/kGDkyVeYTJqRKv2g7d1OKNRkHgSXr1qUmjaJ9ujgfujgzpDjbouOjODe7X7/2UwiLreuuzrUuKvaXX97wDJXibJTW1g3PUunM4MGp2WT8+HQwc//9U/e227a3i48cmZpdRo1Klf6IEd4aN+uEg2BLUVSglY8lS9qbN4qLYtra2g88Vl5YU1TsHa9qrLwAp14GD4Zx49or7PHj29vBW1rSozj1rziVcOzY9Nhuu9TUMmqUK3WzPuIgqIe1a9sr4xUrUhPHggXpuaigV61KFXrRnr1oUft53osW9XwWSb9+7Re4FAcei3Opi3br0aM3PGe6swteigtiivOvO54m2K9fqoCL15D2BNatS68rz83ueNFN8TxkiCtxs82Ig6AvrFsHjzwC06fD00/Ds8+m0wYXLkxNIEuWVDedYcPamzHGjIGJE2HffdMW8dZbp4q88vWoUe1NIAMH1nIJzayJOQg21uzZcNNN8JvfwF13tVf2AwbATjuls0UOOCA1Z4wZ034e99ChqVmkOONk+PC0hTxokE8HNLOGcM3TG88/D9OmwZVXwoMPpn677grvfz+8/e1wyCEpAHylpJltQRwE1bj1Vvj61+HOO1P3wQfDBRfAEUfA7ru7vdvMtmgOgu48/zycdRZce21qr//KV+DYY9NegJlZk3AQdOVXv4LjjktnxHzta/C5z6V2fDOzJuMg6MysWXD88bDXXnDVVand38ysSTkIOlqxIh38HTYMrrsuXbVqZtbEHASVIuCUU+CJJ+CWWxwCZlYKDoJK06bBFVekYwKHHdbo0piZ1UW/Rhdgs3L99ensoHPOaXRJzMzqxkFQacYMeNOb2u+hY2ZWAq7xCgsXpvsDHXBAo0tiZlZXDoLCvfem58mTG1sOM7M6cxAUZsxIt4rYb79Gl8TMrK4cBIV77033DRoxotElMTOrKwcBpOsHZszw8QEzK6WaBoGkMyU9KukRSVdIGixpF0nTJT0p6UpJjf9HlRdeSH+K7uMDZlZCNQsCSTsApwGTI2IvoD/wEeAC4DsRMQlYDJxcqzJUrThQ7D0CMyuhWjcNtQBDJLUAQ4G5wDuAa/LwqcBRNS5Dz2bMSP8OtvfejS6JmVnd1SwIIuIF4NvA86QAWArMBJZERFsebQ7Q6Q19JJ0iaYakGQsXLqxVMZN77013Gh0ypLbzMTPbDNWyaWg0cCSwC7A9MAw4vJNRo7P3R8TFETE5IiaPGzeuVsX0gWIzK71aNg29E3gmIhZGRCtwHfBmYFRuKgKYALxYwzL07OmnYfFiHyg2s9KqZRA8DxwkaagkAYcBjwG/Az6Qx5kC3FDDMvRsxoz07D0CMyupWh4jmE46KHwf8HCe18XAF4CzJD0FbA1cWqsyVOW++2DgQNhzz4YWw8ysUWr6fwQR8SXgSx16Pw0cWMv59srSpTBqVAoDM7MS8pXFra0wYECjS2Fm1jAOAgeBmZWcg8BBYGYl5yBwEJhZyTkIHARmVnIOAgeBmZWcg8BBYGYl5yBwEJhZyTkIHARmVnIOAgeBmZWcg6CtzUFgZqXmIPAegZmVnIPAQWBmJecgcBCYWck5CBwEZlZyDgIHgZmVnIPAQWBmJecgcBCYWck5CBwEZlZyDgIHgZmVXI9BIOlaSe+W1HyhEeEri82s9Kqp3C8CjgGelHS+pN1rXKb6aWtLzw4CMyuxHoMgIm6LiGOB/YBngVsl/VHSRyVt2TVoa2t6dhCYWYlV1dwjaWvgROBjwP3A90jBcGvNSlYPDgIzM1p6GkHSdcDuwOXAERExNw+6UtKMWhau5hwEZmY9BwHw7xHx284GRMTkPi5PfRVB0FLNx2Bm1pyqaRp6vaRRRYek0ZL+oYZlqh/vEZiZVRUEH4+IJUVHRCwGPl67ItWRg8DMrKog6CdJRYek/sDA2hWpjhwEZmZVHSP4H+AqSf8BBHAqcHNNS1UvDgIzs6qC4AvAJ4BPAgJuAS6pZaHqxkFgZtZzEETEOtLVxRfVvjh15iAwM6vqOoJJwDeBPYDBRf+I2LWG5aoPB4GZWVUHi39G2htoA/4KuIx0cdmWz0FgZlZVEAyJiNsBRcRzEXEe8I7aFqtOHARmZlUdLF6db0H9pKRPAy8A42tbrDpxEJiZVbVHcAYwFDgN2B84DphSy0LVjYPAzKz7PYJ88diHIuIfgRXAR+tSqnpxEJiZdb9HEBFrgf0rryxuKg4CM7OqjhHcD9wg6WpgZdEzIq7r6Y35ZnWXAHuRrko+CXgCuBKYSPqjmw/l+xfVn4PAzKyqYwRjgJdJZwodkR/vqXL63wNujojdgb2BWcDZwO0RMQm4PXc3hoPAzKyqK4s36riApJHA20j/bEZEvAq8KulI4NA82lTgDtJtLOrPQWBmVtWVxT8jNetsICJO6uGtuwILgZ9J2huYCZwObFP8y1lEzJXU6amokk4BTgHYaaedeirmxnEQmJlV1TR0E/Bf+XE7MJJ0BlFPWkj/a3xRROxLOr5QdTNQRFwcEZMjYvK4ceOqfVvvtLWlZweBmZVYNU1D11Z2S7oCuK2Kac8B5kTE9Nx9DSkI5kvaLu8NbAcs6GWZ+473CMzMqtoj6GgS0GNbTUTMA2ZLel3udRjwGHAj7RekTQFu2Igy9A0HgZlZVccIlrPhMYJ5VH9w9zPANEkDgadJF6T1I/3RzcnA88AHe1XivlQEQf/+DSuCmVmjVdM0NGJjJx4RDwCTOxl02MZOs0+1tqa9gSa9Xs7MrBo9Ng1Jep+krSq6R0k6qrbFqpMiCMzMSqyaYwRfioilRUdELAG+VLsi1ZGDwMysqiDobJxqbk2x+XMQmJlVFQQzJF0o6TWSdpX0HdLFYVs+B4GZWVVB8BngVdKN4q4CVgGfqmWh6sZBYGZW1VlDvboieIviIDAzq+qsoVvz7aSL7tGS/qe2xaqT1lZoaY7DHWZmG6uapqGx+UwhAPJ/BzTPfxZ7j8DMSq6aIFgnaf0tJSTtTCd3I90iOQjMzKo6DfSfgLsl3Zm73wZ8onZFqiMHgZlZVQeLb5a0H3AQIODMiHip5iWrBweBmVl1dx+NiJci4ibS3UNPlfRIbYtVJw4CM7OqzhraTtIZku4BHgX6A0fXvGT14CAwM+s6CCR9XNJvgTuBscDHgLkR8eWIeLheBawpB4GZWbfHCH4I/Ak4JiJmAEhqjrOFCg4CM7Nug2B70p/GXChpG9LtJZqr1nQQmJl13TSUDxBfFBFvI/2RzFJggaRZkr5RtxLWkoPAzKzqs4bmRMS3I2J/4ChgTW2LVScOAjOz3v+vQEQ8AXy5BmWpPweBmVl1ewRNy0FgZuYgcBCYWdlVc0HZ7dX02yI5CMzMuj5GIGkwMBQYK2k06T5DACNJp5Zu+RwEZmbdHiz+BHAGqdKfSXsQLCNdbLblcxCYmXUdBBHxPeB7kj4TET+oY5nqY+1aiHAQmFnpVXMb6h9IejMwsXL8iLishuWqvdbW9OwgMLOS6zEIJF0OvAZ4AFibewfgIDAzawLVXFA2GdgjIprvhnPgIDCz0qvmOoJHgG1rXZC6a2tLzw4CMyu5avYIxgKP5T+mWX+PoYh4b81KVQ/eIzAzA6oLgvNqXYiGcBCYmQHVnTV0p6SdgUkRcZukoaS/q9yyOQjMzIDqbjHxceAa4Me51w7A9bUsVF04CMzMgOoOFn8KOIR0RTER8SQwvpaFqgsHgZkZUF0QrImIV4sOSS2k6wi2bA4CMzOguiC4U9K5wBBJfw1cDfy6tsWqgyIIWnr93zxmZk2lmiA4G1gIPEy6Ed1/A1+sZaHqwnsEZmZAdaePDgF+GhE/AZDUP/d7pZYFqzkHgZkZUN0ewe2kir8wBLitNsWpIweBmRlQXRAMjogVRUd+PbTaGUjqL+l+STfl7l0kTZf0pKQrJQ3sfbH7gIPAzAyoLghWStqv6JC0P7CqF/M4HZhV0X0B8J2ImAQsBk7uxbT6joPAzAyoLghOB66WdJeku4ArgU9XM3FJE4B3A5fkbgHvIF2gBjAVOKq3he4TDgIzM6CHg8WS+gEDgd2B15H+rvLxiGitcvrfBT4PjMjdWwNLIiLf+pM5pCuVO5v3KcApADvttFOVs+sFB4GZGdDDHkFErAP+LSJaI+KRiHi42hCQ9B5gQUTMrOzd2Wy6mPfFETE5IiaPGzeumln2joPAzAyo7vTRWyS9H7iul39OcwjwXknvAgYDI0l7CKMkteS9ggnAi70tdJ9wEJiZAdUdIziLdDXxq5KWSVouaVlPb4qIcyJiQkRMBD4C/DYijgV+B3wgjzYFuGHjir6JHARmZkAVQRARIyKiX0QMiIiRuXvkJszzC8BZkp4iHTO4dBOmtfEcBGZmQHV/Xi/gWGCXiPiqpB2B7SLinmpnEhF3AHfk108DB25UafuSg8DMDKiuaehHwMHAMbl7BfDDmpWoXhwEZmZAdQeL3xQR+0m6HyAiFjfsauC+5CAwMwOq2yNozTeaCwBJ44B1NS1VPbS2Qr9+6WFmVmLV1ILfB34FjJf0deBu4Bs1LVU9tLZ6b8DMjOr+vH6apJnAYaQLwo6KiFk9vG3z5yAwMwO6CQJJg4FTgd1If0rz44pbQ2z5HARmZkD3TUNTgcmkEDgc+HZdSlQvDgIzM6D7pqE9IuINAJIuBaq+bmCL4CAwMwO63yNYf3O5pmoSKjgIzMyA7vcI9q64p5CAIblbQGzibSYaz0FgZgZ0EwQR0b+eBak7B4GZGVDddQTNyUFgZgY4CBpdCjOzhitvELS1OQjMzChzEHiPwMwMcBA0uhRmZg3nIDAzK7lyB0FLNX/HYGbW3ModBN4jMDNzEJiZlZ2DwMys5BwEZmYl5yAwMys5B4GZWck5CMzMSs5BYGZWcg4CM7OSK2cQRPjuo2ZmWTmDoC3/BbODwMyspEHQ2pqeHQRmZg4CM7OycxCYmZWcg8DMrOQcBGZmJecgMDMrOQeBmVnJOQjMzErOQWBmVnIOAjOzkqtZEEjaUdLvJM2S9Kik03P/MZJulfRkfh5dqzJ0yUFgZrZeLfcI2oDPRsTrgYOAT0naAzgbuD0iJgG35+76chCYma1XsyCIiLkRcV9+vRyYBewAHAlMzaNNBY6qVRm65CAwM1uvLscIJE0E9gWmA9tExFxIYQGM7+I9p0iaIWnGwoUL+7ZADgIzs/VqHgSShgPXAmdExLJq3xcRF0fE5IiYPG7cuL4tlIPAzGy9mgaBpAGkEJgWEdfl3vMlbZeHbwcsqGUZOuUgMDNbr5ZnDQm4FJgVERdWDLoRmJJfTwFuqFUZuuQgMDNbr6WG0z4EOB54WNIDud+5wPnAVZJOBp4HPljDMnSu+IeyllouvpnZlqFmNWFE3A2oi8GH1Wq+VfEegZnZer6y2Mys5BwEZmYl5yAwMys5B4GZWck5CMzMSq7cQeDTR83MShwELS2grs5uNTMrj/IGgZuFzMwAB4GZWek5CMzMSs5BYGZWcuUMgnnzYPDgRpfCzGyzUL4g+POf4de/hqOPbnRJzMw2C+UKgnXr4LTTYPvt4dxzG10aM7PVYks+AAAGnElEQVTNQrmuqJo6Fe69Fy6/HIYPb3RpzMw2C+XZI1i2DM45Bw4+GI49ttGlMTPbbDT3HsEnPwm//316vXw5LFgAN93kK4rNzCo0dxDstBPssUd79z//M0ye3LjymJlthpo7CM45p9ElMDPb7JXnGIGZmXXKQWBmVnIOAjOzknMQmJmVnIPAzKzkHARmZiXnIDAzKzkHgZlZySkiGl2GHklaCDzXi7eMBV6qUXE2V17mcijbMpdteaFvl3nniBjX00hbRBD0lqQZEVGqe0l4mcuhbMtctuWFxiyzm4bMzErOQWBmVnLNGgQXN7oADeBlLoeyLXPZlhcasMxNeYzAzMyq16x7BGZmViUHgZlZyTVVEEj6O0lPSHpK0tmNLk8tSNpR0u8kzZL0qKTTc/8xkm6V9GR+Ht3osvY1Sf0l3S/ppty9i6TpeZmvlDSw0WXsS5JGSbpG0uN5fR/c7OtZ0pn5e/2IpCskDW629Szpp5IWSHqkol+n61XJ93Od9pCk/WpRpqYJAkn9gR8ChwN7AEdL2qP7d22R2oDPRsTrgYOAT+XlPBu4PSImAbfn7mZzOjCrovsC4Dt5mRcDJzekVLXzPeDmiNgd2Ju07E27niXtAJwGTI6IvYD+wEdovvX8n8DfdejX1Xo9HJiUH6cAF9WiQE0TBMCBwFMR8XREvAr8EjiywWXqcxExNyLuy6+XkyqHHUjLOjWPNhU4qjElrA1JE4B3A5fkbgHvAK7JozTVMksaCbwNuBQgIl6NiCU0+Xom/X3uEEktwFBgLk22niPi98CiDr27Wq9HApdF8mdglKTt+rpMzRQEOwCzK7rn5H5NS9JEYF9gOrBNRMyFFBbA+MaVrCa+C3weWJe7twaWRERb7m629b0rsBD4WW4Ou0TSMJp4PUfEC8C3gedJAbAUmElzr+dCV+u1LvVaMwWBOunXtOfGShoOXAucERHLGl2eWpL0HmBBRMys7N3JqM20vluA/YCLImJfYCVN1AzUmdwufiSwC7A9MIzUNNJRM63nntTle95MQTAH2LGiewLwYoPKUlOSBpBCYFpEXJd7zy92GfPzgkaVrwYOAd4r6VlSk987SHsIo3ITAjTf+p4DzImI6bn7GlIwNPN6fifwTEQsjIhW4DrgzTT3ei50tV7rUq81UxDcC0zKZxgMJB1kurHBZepzuW38UmBWRFxYMehGYEp+PQW4od5lq5WIOCciJkTERNJ6/W1EHAv8DvhAHq3ZlnkeMFvS63Kvw4DHaOL1TGoSOkjS0Pw9L5a5addzha7W643ACfnsoYOApUUTUp+KiKZ5AO8C/gL8L/BPjS5PjZbxLaRdw4eAB/LjXaQ289uBJ/PzmEaXtUbLfyhwU369K3AP8BRwNTCo0eXr42XdB5iR1/X1wOhmX8/Al4HHgUeAy4FBzbaegStIx0BaSVv8J3e1XklNQz/MddrDpDOq+rxMvsWEmVnJNVPTkJmZbQQHgZlZyTkIzMxKzkFgZlZyDgIzs5JzEFjTk7QiP0+UdEwfT/vcDt1/7Mvpm9WDg8DKZCLQqyDId7XtzgZBEBFv7mWZzBrOQWBlcj7wVkkP5Pve95f0r5Luzfd6/wSApEPzfz78gnQRD5KulzQz3yv/lNzvfNKdMh+QNC33K/Y+lKf9iKSHJX24Ytp3VPzPwLR8FS2Szpf0WC7Lt+v+6VhptfQ8ilnTOBv4XES8ByBX6Esj4gBJg4A/SLolj3sgsFdEPJO7T4qIRZKGAPdKujYizpb06YjYp5N5/T3pyuC9gbH5Pb/Pw/YF9iTdM+YPwCGSHgPeB+weESFpVJ8vvVkXvEdgZfY3pPu4PEC6lffWpD8AAbinIgQATpP0IPBn0k3AJtG9twBXRMTaiJgP3AkcUDHtORGxjnSLkInAMmA1cImkvwde2eSlM6uSg8DKTMBnImKf/NglIoo9gpXrR5IOJd0Z8+CI2Bu4HxhcxbS7sqbi9VqgJdL99g8k3VX2KODmXi2J2SZwEFiZLAdGVHT/D/DJfFtvJL02//lLR1sBiyPiFUm7k/4itNBavL+D3wMfzschxpH+beyergqW/19iq4j4b+AMUrOSWV34GIGVyUNAW27i+U/SfwJPBO7LB2wX0vnfIN4MnCrpIeAJUvNQ4WLgIUn3Rbo1duFXwMHAg6S7xX4+IublIOnMCOAGSYNJexNnbtwimvWe7z5qZlZybhoyMys5B4GZWck5CMzMSs5BYGZWcg4CM7OScxCYmZWcg8DMrOT+D73HKPpi4VOGAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# separate the data set\n",
"nbr_of_testing_images = 10000\n",
"nbr_of_training_images = 10000\n",
"nbr_of_images = nbr_of_testing_images + nbr_of_training_images\n",
"\n",
"training_images = []\n",
"training_labels = []\n",
"testing_images = []\n",
"testing_labels = []\n",
"\n",
"for index in range(0, nbr_of_images):\n",
" if(index <= nbr_of_training_images - 1):\n",
" training_images.append(normalized_scaled_images_feature_matrix[:, index]) \n",
" training_labels.append(binary_labels[index])\n",
" else:\n",
" testing_images.append(normalized_scaled_images_feature_matrix[:, index]) \n",
" testing_labels.append(binary_labels[index])\n",
" \n",
"# covert to numpy array\n",
"training_images = np.transpose(np.array(training_images))\n",
"training_labels = np.array(training_labels)\n",
"testing_images = np.transpose(np.array(testing_images))\n",
"testing_labels = np.array(testing_labels)\n",
"\n",
"# logger\n",
"#print(training_images.shape) # training_images is a matrix of 784 X 50000\n",
"#print(training_labels.shape) # testing_labels is a row vector of 1 X 50000\n",
"#print(testing_images.shape) # training_images is a matrix of 784 X 10000\n",
"#print(testing_labels.shape) # testing_labels is a row vector of 1 X 10000\n",
"\n",
"# initialize weights & bias\n",
"initial_value = .1\n",
"W = np.multiply(initial_value, np.ones(size))\n",
"b = initial_value # b will be added in a broadcasting manner\n",
"\n",
"#print(W.shape) # W is a vector with a size of 784\n",
"\n",
"# gradient descent\n",
"\n",
"# hyber-parameters\n",
"alpha = 0.01;\n",
"number_of_iterations = 100\n",
"\n",
"# data arrays\n",
"cost_array = []\n",
"interation_array = []\n",
"accuracy_array = []\n",
"\n",
"# rename\n",
"X = training_images\n",
"Y = training_labels\n",
"m = size\n",
"\n",
"# main loop\n",
"for interation in range(1, number_of_iterations):\n",
" \n",
" # forward propogation training data set\n",
" Z = np.dot(np.transpose(W), X) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
" A = 1/(1 + np.exp(-Z)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
" term1 = np.multiply(-Y, np.log(A)) # -Y * log(A) # broadcasting is used \n",
" term2 = np.multiply((1 - Y), np.log(1 - A)) # (1-Y) * log(1-A) # broadcasting is used \n",
" L = term1 - term2 # loss function # l is 1X50000\n",
" cost = np.multiply(1/m, np.sum(L)) # cost # scalar \n",
" \n",
" # forward propogation test data set\n",
" Z_test = np.dot(np.transpose(W), testing_images) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
" A_test = 1/(1 + np.exp(-Z_test)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
" \n",
" # accuracy\n",
" prediction_labels = np.where(A_test >= 0.5, 1, 0) # if greater than ore equal to 0.5 use 1, else 0\n",
" accuracy = np.sum(np.absolute(testing_labels - prediction_labels)) # Sum of false positives and false negatives # accuracy is scalar \n",
" percent_inaccurate = np.multiply(100/len(testing_labels), accuracy)\n",
" percent_accurate = 100 - percent_inaccurate\n",
" \n",
" # storage for plotting\n",
" cost_array.append(cost)\n",
" accuracy_array.append(percent_accurate)\n",
" interation_array.append(interation)\n",
" \n",
" # backpropogation\n",
" dZ = A - Y # dZ is 1X500\n",
" dW = np.multiply(1/m, np.dot(X, np.transpose(dZ))) # dW is 784X1\n",
" db = np.multiply(1/m, np.sum(dZ)) # db is a scalar\n",
" \n",
" # paramter updates\n",
" W = W - np.multiply(alpha, dW)\n",
" b = b - np.multiply(alpha, db)\n",
"\n",
"# forward propogation training data set\n",
"Z = np.dot(np.transpose(W), X) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
"A = 1/(1 + np.exp(-Z)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
"term1 = np.multiply(-Y, np.log(A)) # -Y * log(A) # broadcasting is used \n",
"term2 = np.multiply((1 - Y), np.log(1 - A)) # (1-Y) * log(1-A) # broadcasting is used \n",
"L = term1 - term2 # loss function # l is 1X50000\n",
"cost = np.multiply(1/m, np.sum(L)) # cost # scalar \n",
"\n",
"# forward propogation test data set\n",
"Z_test = np.dot(np.transpose(W), testing_images) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
"A_test = 1/(1 + np.exp(-Z_test)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
"\n",
"# final accuracy\n",
"prediction_labels = np.where(A_test >= 0.5, 1, 0) # if greater than ore equal to 0.5 use 1, else 0\n",
"accuracy = np.sum(np.absolute(testing_labels - prediction_labels)) # Sum of false positives and false negatives # accuracy is scalar \n",
"percent_inaccurate = np.multiply(100/len(testing_labels), accuracy)\n",
"percent_accurate = 100 - percent_inaccurate\n",
"\n",
"# storage for plotting\n",
"cost_array.append(cost)\n",
"accuracy_array.append(percent_accurate)\n",
"interation_array.append(number_of_iterations + 1)\n",
" \n",
"print('Final Cost: ' + str(cost))\n",
"print('Final Percent Accuracy: ' + str(percent_accurate))\n",
"\n",
"# plot learning curve\n",
"pyplot.figure()\n",
"pyplot.plot(interation_array, cost_array, 'red')\n",
"pyplot.title('Learning Curve - 10,000 MNIST Training Data Set')\n",
"pyplot.xlabel('Iterations')\n",
"pyplot.ylabel('Cost')\n",
"pyplot.show()\n",
"\n",
"# plot percent accuracy curve\n",
"pyplot.figure()\n",
"pyplot.plot(interation_array, accuracy_array, 'red')\n",
"pyplot.title('Percent Accuracy Curve - 10,000 MNIST Test Data Set')\n",
"pyplot.xlabel('Iterations')\n",
"pyplot.ylabel('Percent Accuracy')\n",
"pyplot.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The results reveal what is intuitively expected. The percent accuracy of the network levels-out at around 92.72 percent after 100 iterations, which is less than the percentage we achieved with a larger training data set. This empiracally leads us to conclude that the more training data we have, the higher accuracy we can achieve with our network. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One last thing we wish to investigate is how the learning rate impacts our results. In order to test this we will decrease our learning rate to a value of 0.00001. This is a very low value, but we are doing this for testing purposes. We will adjust our training data set back to 50,000 images for a direct comparison to the good result we had earlier."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Final Cost: 481.390470076452\n",
"Final Percent Accuracy: 9.61\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XecVNX9//HXGxBURLEgRlDRrxgjRcQVsUbFhh2DPRErUTGJPZZo7NFo7Ird2LsolhCNHfuiiICNGAs/NGAD7AKf3x/nbhjW2QLs7N2dfT8fj33szJ17Zz6zd3Y+95x7z/koIjAzM6uuVd4BmJlZ0+QEYWZmRTlBmJlZUU4QZmZWlBOEmZkV5QRhZmZFOUE0QZL+IWlI3nGYLQhJm0ma0NDrWuNzgigg6X1JW+YdR0QMjIgbS/HckpaUdJGkDyV9JWlSdn+5UrxeQ5J0qqQfs7irflYreLyPpDGSvsl+96nluZaRNELS15I+kLR3tcf3zpZ/Lel+ScvUd9siMYek31dbfkS2/NTs/mbZ/currTda0n7Z7f0kjS54bGNJz0uaLulzSc9JWk/SiQV/n+8kzS64P6Ha829S8NjXWQyFf9+Va3pvNYmIpyKiR0OvO7+yv913kmZKmiGpUtJxktrWc/s22d+j20LEsKmkFwr20WhJfRvjtRuCE0Qjk9Qmx9duCzwO9AC2BZYENgQ+A/otwPPl8V7ujIglCn7ey2JpCzwA3AIsDdwIPFDLl8HlwA9AZ2AfYLikHtlz9QCuAn6TPf4NcEV9tq3BO0D1FuG+2fJCXwP71udLQdKSwEPApcAyQBfgNOD7iDi76u8DHAK8UPD3mifOiHi2YN2qxzoWrP9htddtJak5fW8cEhEdgBWB44BfAw9JUqlfWNLSwEjgAtJnsitwJumz0yw0px2dK0k7SBor6cvsqK13wWPHS/p3dqQyUdKggsf2y47sLpT0OXBq1ZGgpPMlfSHpP5IGFmzzlKSDCravbd1VJT2Tvfa/JF0u6ZYa3sa+wMrAoIiYGBFzImJqRJwREY9kzxeSVi94/r9LOjO7vZmkyZL+KOkT4AZJb0raoWD9NpI+rTpKktQ/+3t9Kel1SZstzH6oxWZAG+CiiPg+Ii4BBGxRfUVJ7YFfASdHxFcRMZr0j/ybbJV9gAcj4pmI+Ao4GdhVUod6bFvMK8Di1RLQYtnyQl8Cfwf+XI/3uwZARNweEbMj4tuIeDQixtVj2/mSff7OkPQCKYmtLOmgbN/PzD77BxWsv6Wk9wvuT5Z0lKQ3siPp2yW1m991s8dPkPSJpP8n6eD6HmVn++oJYGdgE2Cb7Pk2kPRi9vn8WNIlkhbJNnsm+z1BqTX1K0nLSnpE0rTs//FBSV1qeNmfA7Mi4u7sf+2biBgVEeML3s9Bkt7Knusfklaq6bXreo+l4ARRD9mX3fXAb4FlSUeXIws+uP8mfeiWIh3F3SLpZwVPsT7wHrA8cFbBsreB5YC/AtfVclRT27q3AS9ncZ1K7V9UWwKjsi+9BbUC6Yh1FWAocDuwV8Hj2wCfRsSr2T/Ow6SjpmWAY4B7JXVaiNffUampPkHSoQXLewDjYt65Y8Yx96i40BrA7IgoPIJ/vWDdHtl9ACLi36SjvjXqsW1NbiYlaEitiZtqWO8s4FeSfl7H870DzJZ0o6SB2dFqKf0GOIDU6pwM/BfYPrt/MHBp4UFTEbsDWwGrAetS++e06LrZgcjvgM1J++Enyb8uEfEf4DXS/yvALOAPpP+tjUgt699mj22a/e6RtabuJX1nXkM60FoF+BG4uIaXextoLekGSdtK6lj4oKTBwLGkpNUJeIn0/1zTazc6J4j6ORi4KiJeyo7WbgS+B/oDZEcIU7KjhDuBd5m3y2ZKRFwaEbMi4tts2QcRcU1EzCZ1h/yM1GVRTNF1lfqH1wNOiYgfCo5ma7Is8PEC/QXmmgP8OTtK/5b0gd5J0uLZ43sz90P+a+CRiHgk+9s8BlQC2y3ga98F/IL0z3QwcIqkquS0BDC92vrTgQ5FnqeudWt7fH5ep9AtwF7Z0eme2f2fiIhPgCuB02t7soiYAWwMBOkLa5qkkZJq+gwtrOsj4s2I+DH7HD8YEe9F8gSp63KTWra/KCI+iYjPSF1jNZ4fqmXd3YHrsji+Jh2MLYgppAMWIuKV7P96VtZdeTXwy5o2jIhpETEia7HNAM6uaf2I+IK0j1oB15H20f0FB0i/Bc6OiLcjYhbpQKpfLS2SRucEUT+rAEdnzdAvJX0JrETq10TSvprb/fQl0JN0RFLloyLP+UnVjYj4Jru5RA2vX9O6KwKfFyyr6bWqfEZKLgtjWkR8VxDPJOBN0pH94sBOzE0QqwC7Vfu7bVwsBkn7aO6J0X8Ue+GsW2xKlqSfJx25Dc4e/op0NFtoSWBmkaeqa93aHp+f1ymM/UNgEukL5d2IqG0/nQtsI2ntOp7zzYjYLyK6kj5zKwIX1bbNQpgnXqUu15ey1tyXwNbM+5mv7pOC299Q82e9tnVXrBZHbX/D2nQBPgeQtKakh7NuqxmkxFzj+5DUXtK1Shd5zACeqG39iJgQEUMiogvQm9TyuCB7eBXg8oL/jU9JB2BdF/B9NTgniPr5CDgrIjoW/CweEbdLWoV0BHc4sGxEdATGk/q/q5RqytyPgWUKjt4hJa6a/Iv0xdO+lnW+AQqfb4Vqjxd7L1XdTDsDE7OkAenvdnO1v1v7iDin+hNExK0FJ0YHVn+8BsHcv/MEoHe1brre2fLq3gHaSOpesGztgnUnZPcBULpSql22XV3b1uYm4Ghq7l4CIDtyvgg4ox7PWbXNW6TzFz3ru818+t9+l7QYcA/wF6Bz9pl/lHk/86XwMfN+edb2WS8qO1/RB3g2W3QV6f919YhYEjiFue+j2Gf9OGBVoF+2fr27uSLiTdK+r9pHHwEHVvv/WCwiXqrhtRudE8RPLSJp0YKfNqQEcIik9ZW0l7S9pA5Ae9LOnAYgaX9K9086j4j4gNRlc6qktpI2AHasZZObSR/Ke7Mjp1bZSbcTJVV1+4wF9pbUWtK21NLcLnAH6QjyUOa2HiB1o+woaZvs+RZVOtG9QEdIknaWtHS2D/oBvydduQTwFDAb+L2kdpIOz5Y/kW27n7KToVn3xH3A6dm+3IiU3G7Otrk1i3uTLJmeDtwXETPrsW1t7iT9ne6qx7oXkK4w+0UNf4s1JR1d9bfMTm7uBbxYj+deWO2AtqTP/Ozs3MCARnjdu4ADJf08Oyg6ub4bZvtqM+B+4Dngn9lDHUhdhF9L+gVzzz+Qdel+RjoXQsH63wBfSFqWlFBqes21lE64d8nur0zqXqzaR1cCJ2Wvi6SO2XmJml670TlB/NQjwLcFP6dGRCWpz/sy4AtSV8F+kLo9gL8BL5BO3PUifQAbyz7ABqQP05mkL6Hvi60YEd+TTlS/BTwGzCCd4F6OdIIM0gm7HUlX1OxD+oeqVUR8THr/G2avX7X8I9KX54mkL5OPSCflFvRztyfpbz+TdCR2bnY+iIj4AdiFdCL4S9IJ1V2y5ZCONgv3y2GkK4mmklpAh0bEhOy5JpAuD701e7xDtn6d29Ym67f+V8F5qNrWnUG6IGGZGlaZSbp44SVJX5O+dMaTWiglFRFfAkcCI0hdNYNJ5wpK/boPAsNJV/i8y9z9WfTznrlS0kxSt9UFpM/n9hH/u5jhaNJFAzNJrYk7q23/Z+C2rBto1+w5liL9vz0PFO0Ozcwk/W++ku2j50kHYMdl7+fu7PnuzrqrxpFdXVXDazc6hQsGlRVJdwJvRUR9LpVsMSQ9Cvwha+ZbGZDUC3gVaBcRc/KOpxw5QTRzktYjHcX9h9R9cT+wQUS8lmtgZiWgNMboYVKr7ibg24gYXPtWtqDcxdT8rUDqf/8KuITU3eHkYOVqGOlqn3eB77L7ViJuQZiZWVFuQZiZWVG5TRzXEJZbbrno1q1b3mGYmTUrY8aM+TQi6pzyplkniG7dulFZWZl3GGZmzYqkD+qznruYzMysKCcIMzMrygnCzMyKcoIwM7OiSpogssmn7lGqmPSmUvWmZSQ9Jund7PfS2bpSquY0SdI41aNuq5mZlU6pWxAXkyqYrUmaEvlN4Hjg8YjoTioycny27kCge/YzlDQpl5mZ5aRkCUKpqPqmpEpKZBXPviTN7nljttqNpBk4yZbflFWoehHoqHnLdpqZWSMqZQtiNdIUzzdIei2rwtSeVGDkY/jfNNHLZ+t3Yd4KUZOzZfOQNFRSpaTKadOmLVhk06bBkUfC9OqVI83MrEopE0QboC8wPCLWAb5mbndSMcWqUf1koqiIuDoiKiKiolOnOgcCFvf443DJJdCjBzzyyII9h5lZmStlgpgMTM7K50EqUdgX+G9V11H2e2rB+oUlBLuSios3vD33hBdegI4dYfvtYcgQ+PzzkryUmVlzVbIEERGfAB9J+nm2aAAwERhJquBE9ruqZORIYN/saqb+wPSqrqiS6NcPxoyBk0+G226DtdaCESNK9nJmZs1Nqa9i+h1wq6RxpELhZwPnAFtJehfYKrsPqdTne6SSktcwb4nH0mjXDk4/HV55BVZcEXbdFXbfHaZOrXtbM7My16zrQVRUVESDTdb3449w3nlw2mnQoQNcemnqilKxUyNmZs2XpDERUVHXeh5JXWWRReDEE+G112D11WHvvWGXXWBKaU6DmJk1dU4Q1a21Fjz3HJx/Pjz6aLrS6e9/h2bc0jIzWxBOEMW0bg1HHw3jxkGvXrD//jBwIHz4Yd6RmZk1GieI2nTvDk89BZddBqNHp9bElVfCnDl5R2ZmVnJOEHVp1QqGDYM33oD114dDD4Utt4T33ss7MjOzknKCqK9VV4XHHoNrrknjJ3r1gosvhtmz847MzKwknCDmhwQHHQQTJsDmm8MRR8Amm8Bbb+UdmZlZg3OCWBBdu8KDD8Itt8Dbb0OfPnDuuTBrVt6RmZk1GCeIBSXBPvuk1sT228Pxx0P//ulchZlZGXCCWFgrrAD33gt3350ug1133TQa+4cf8o7MzGyhOEE0lMGDYeJE2G03OPVUqKhIJ7PNzJopJ4iGtNxycOut8MAD8Omn6bLYE0+E777LOzIzs/nmBFEKO+2UWhP77gt/+Quss06qP2Fm1ow4QZRKx45w/fUwahR8/TVstBEcdRR8803ekZmZ1YsTRKltsw2MHw+HHAIXXgi9e8PTT+cdlZlZnZwgGsOSS8IVV8CTT6b7m20Ghx0GM2fmGpaZWW2cIBrTZpvB66/DkUemSf969kxTipuZNUFOEI2tfXu44IJUc2LxxVMX1IEHwpdf5h2Zmdk8nCDyssEGqXrdCSfAjTemqcQffDDvqMzM/scJIk+LLgpnnw0vvQTLLpsuj91nnzSGwswsZ04QTcG660JlZRqBfffdqezp3Xe7zKmZ5coJoqlo2xb+/Oc0Pccqq8Duu6fpOz75JO/IzKyFcoJoanr1SqOuzzkHHn44tSZuvtmtCTNrdE4QTVGbNvDHP8LYsbDmmmnKjh13hMmT847MzFqQkiYISe9LekPSWEmV2bI+kl6sWiapX7Zcki6RNEnSOEl9Sxlbs7DmmvDss2kE9hNPpCudrr3WrQkzaxSN0YLYPCL6RERFdv+vwGkR0Qc4JbsPMBDonv0MBYY3QmxNX+vWqbTpG29A375w8MGw9dbw/vt5R2ZmZS6PLqYAlsxuLwVMyW7vDNwUyYtAR0k/yyG+pun//g8efxyGD4cXX0yjsC+7DObMyTsyMytTpU4QATwqaYykodmyI4DzJH0EnA+ckC3vAnxUsO3kbJlVadUqTfo3YQJsvDH87ndp+o533807MjMrQ6VOEBtFRF9S99EwSZsChwJHRsRKwJHAddm6KrL9TzrbJQ3Nzl1UTps2rVRxN20rrwz/+AfccEPqeurdG/72N5g9O+/IzKyMlDRBRMSU7PdUYATQDxgC3Jetcne2DFKLYaWCzbsyt/up8DmvjoiKiKjo1KlTqUJv+iTYb7/Umth6azjmmFRzYuLEvCMzszJRsgQhqb2kDlW3ga2B8aQv/V9mq20BVPWPjAT2za5m6g9Mj4iPSxVf2VhxRbj/frj9dpg0KVWvO+ss+PHHvCMzs2auTQmfuzMwQlLV69wWEaMkfQVcLKkN8B3piiWAR4DtgEnAN8D+JYytvEiw556wxRbpvMSf/gT33JO6oPr0yTs6M2umFM34mvqKioqorKzMO4ymZ8QIOPRQ+OwzOP74lDDatcs7KjNrIiSNKRh6UCOPpC5HgwalcxF77w1nnpnGT7z8ct5RmVkz4wRRrpZZJtWZePhhmDEj1Z849lj49tu8IzOzZsIJotxttx2MH5+q1p1/Pqy9NowenXdUZtYMOEG0BEstBVdfDf/6V7q6adNN08nsr77KOzIza8KcIFqSAQPSwLrDD4fLL09Tiz/+eN5RmVkT5QTR0iyxBFxyCTzzTCpStOWWMHQoTJ+ed2Rm1sQ4QbRUG2+c6k0ceyxcd12a/O+RR/KOysyaECeIlmyxxeCvf00V7JZaCrbfHoYMgc8/zzsyM2sCnCAM+vVLtbBPPhluuy2VOb3vvrq3M7Oy5gRhSbt2cPrp8MoraX6nX/0Kdt8dpk7NOzIzy4kThM2rTx946aU04d8DD6TWxG23ucypWQvkBGE/tcgicOKJ8NprsPrqsM8+sMsuMOUns6+bWRlzgrCarbUWPPdcKkb06KPp/g03uDVh1kI4QVjtWreGo46CcePSNB0HHADbbgsffJB3ZGZWYk4QVj/du8OTT6YR2M89l8ZNDB8Oc+bkHZmZlYgThNVfq1Zw2GFp8r/+/dPtLbaAf/8778jMrAScIGz+deuWzklce206kd2rF1x4IcyenXdkZtaAnCBswUhpCvGJE9MkgEcdlabvePPNvCMzswbiBGELp0sXGDkSbrkF3nknjaP4y19g1qy8IzOzheQEYQtPSmMlJk6EnXZKYyj6909XPplZs+UEYQ2nc2e4++7089FHsO66cOqp8MMPeUdmZgvACcIa3uDBqTWx555w2mkpUbzySt5Rmdl8coKw0lh2Wbj5ZnjwwTR9eP/+cPzx8O23eUdmZvXkBGGltcMOMGFCGoF97rmwzjppoJ2ZNXlOEFZ6HTvCNdeksRPffQebbAJHHAFff513ZGZWi5ImCEnvS3pD0lhJlQXLfyfpbUkTJP21YPkJkiZlj21TytgsB1ttlUZhDxsGF1+cBtg98UTeUZlZDRqjBbF5RPSJiAoASZsDOwO9I6IHcH62fC1gT6AHsC1whaTWjRCfNaYlloBLL4Wnn04TAQ4YAIccAjNm5B2ZmVWTRxfTocA5EfE9QERUlSzbGbgjIr6PiP8Ak4B+OcRnjWHTTeH11+Hoo1P3U8+eMGpU3lGZWYFSJ4gAHpU0RtLQbNkawCaSXpL0tKT1suVdgI8Ktp2cLZuHpKGSKiVVTps2raTBW4ktvjicf346ab3EEjBwIOy/P3zxRd6RmRmlTxAbRURfYCAwTNKmQBtgaaA/cCxwlyQBKrL9TyrTRMTVEVERERWdOnUqYejWaPr3T5P+nXRSujR2rbXg/vvzjsqsxStpgoiIKdnvqcAIUpfRZOC+SF4G5gDLZctXKti8K+Aaly1Fu3Zw5plpQN0KK8CgQbDXXuBWolluSpYgJLWX1KHqNrA1MB64H9giW74G0Bb4FBgJ7CmpnaRVge7Ay6WKz5qoddaBl1+GM86Ae+9NrYk773SZU7MclLIF0RkYLel10hf9wxExCrgeWE3SeOAOYEjWmpgA3AVMBEYBwyLCBQZaokUWgT/9KXU7rbpqmrJj0CD4+OO8IzNrURTN+MisoqIiKisr617Rmq9Zs+Cii+Dkk2HRRVNhoiFD0gyyZrZAJI2pGnpQG4+ktqatTRs45ph0SWzPnukqp4ED4cMP847MrOw5QVjzsMYaaXDdpZfC6NHQowdceSXMmZN3ZGZlywnCmo9WreDww+GNN2D99eHQQ2HLLeHf/847MrOy5ARhzc+qq8Jjj6UR2GPGQO/e6TzFbF/TYNaQnCCseZLgoIPSVOKbbQZHHpmm73jrrbwjMysbThDWvHXtCg89lEZgv/km9OkD55yTrn4ys4XiBGHNnwS//nUqc7r99nDCCWn6jjfeyDsys2bNCcLKxworpNHXd9+dLoNdd91UE/uHH/KOzKxZcoKw8jN4cGpN7LYbnHoqVFSkk9lmNl+cIKw8Lbcc3HorjBwJn32WLos94YRU8tTM6sUJwsrbjjumK52GDEknr/v0geefzzsqs2bBCcLKX8eOcN118OijqQWx8cZwxBHw9dd5R2bWpDlBWMux1VbpyqbDDoOLL04D7J58Mu+ozJosJwhrWTp0gMsuS/M6tWoFW2yRpuyYMSPvyMyaHCcIa5k23TTNEHvUUXDVVWmm2FGj8o7KrElxgrCWa/HF4W9/Syetl1giTSO+//7wxRd5R2bWJDhBmPXvn6rXnXhimrJjrbXggQfyjsosd04QZgDt2sFZZ8Err0DnzrDLLqnU6bRpeUdmlhsnCLNC66yTksTpp8N996XWxJ13QjMuzWu2oOqVICTdXJ9lZmVhkUVSDexXX021J/bcE3bdFT7+OO/IzBpVfVsQPQrvSGoNrNvw4Zg1IT17phPY552XrnBaay248Ua3JqzFqDVBSDpB0kygt6QZ2c9MYCrgs3hW/tq0gWOOSZfE9uwJ++0H222XZos1K3O1JoiI+EtEdADOi4gls58OEbFsRJzQSDGa5W+NNdLguksvhWefTcniqqtgzpy8IzMrmfp2MT0kqT2ApF9LukDSKiWMy6zpadUKDj88TdfRrx8ccggMGAD//nfekZmVRH0TxHDgG0lrA8cBHwA31bWRpPclvSFprKTKao8dIykkLZfdl6RLJE2SNE5S3/l8L2aNY9VV4bHH4Jpr0onsXr3gootg9uy8IzNrUPVNELMiIoCdgYsj4mKgQz233Twi+kRERdUCSSsBWwGFHbkDge7Zz1BSUjJrmiQ46KA0lfgWW8CRR8Imm8Bbb+UdmVmDqW+CmCnpBOA3wMPZVUyLLMTrXkhqiRReDrIzcFMkLwIdJf1sIV7DrPS6doUHH4RbboG33071Js45B2bNyjsys4VW3wSxB/A9cEBEfAJ0Ac6rx3YBPCppjKShAJJ2Av5fRLxebd0uwEcF9ydny8yaNgn22Se1JrbfPlWu698/naswa8bqlSCypHArsJSkHYDvIqLOcxDARhHRl9R9NEzSpsBJwClF1lWxl/7JStJQSZWSKqd5GgRrSlZYAe69F+6+O10Gu+66cNpp8MMPeUdmtkDqO5J6d+BlYDdgd+AlSYPr2i4ipmS/pwIjgF8CqwKvS3of6Aq8KmkFUothpYLNuwJTijzn1RFREREVnTp1qk/4Zo1r8GCYOBF23x1OPRUqKqCyss7NzJqa+nYxnQSsFxFDImJfoB9wcm0bSGovqUPVbWBr4JWIWD4iukVEN1JS6Ju1UEYC+2ZXM/UHpkeE5zaw5mm55dJ5iZEj4bPPUpfT8cenkqdmzUR9E0SrrBVQ5bN6bNsZGC3pdVLr4+GIqK0iyyPAe8Ak4BrgsHrGZtZ07bhjOjex335w7rnpJPbzz+cdlVm9KOoxr4yk84DewO3Zoj2AcRHxxxLGVqeKioqodNPdmovHHoODD07nJ37/+zS9ePv2eUdlLZCkMYVDD2pS11xMq0vaKCKOBa4iJYm1gReAqxskUrOWYqutYPx4GDYMLr4YeveGJ5/MOyqzGtXVTXQRMBMgIu6LiKMi4khSd9BFpQ7OrOwssUSaz+npp9PUHVtsAb/9LUyfnndkZj9RV4LoFhHjqi+MiEqgW0kiMmsJNt00zRB7zDFw7bVp8r9//CPvqMzmUVeCWLSWxxZryEDMWpzFF0+1Jp5/HpZcMk0jPmQIfP553pGZAXUniFckHVx9oaQDgTGlCcmshVl//TTp35/+BLfeCj16wP335x2VWe1XMUnqTBrg9gNzE0IF0BYYlI1fyI2vYrKy89prcMABMHYs7LFHOl/hAaHWwBrkKqaI+G9EbAicBryf/ZwWERvknRzMytI668DLL8MZZ8B996Uyp3fc4TKnlov6zsX0ZERcmv08UeqgzFq0RRZJ3U2vvQarrQZ77QWDBsGUn8w8Y1ZS9R1JbWaNrUcPeO65dCL7n/9MrYkbbnBrwhqNE4RZU9amTboUdty4NLDugANg4MA0GtusxJwgzJqD7t3hqafgsstg9OjUuhg+HObMyTsyK2NOEGbNRatWaZqO8eNhgw3gsMPSSOxJk/KOzMqUE4RZc9OtWzoncd116XLY3r3hwgth9uy8I7My4wRh1hxJ6XzEhAkwYAAcdRRsvDG8+WbekVkZcYIwa866dElFiW69Fd55J9WbOPts+PHHvCOzMuAEYdbcSbD33qnM6c47w0knpek7xo7NOzJr5pwgzMpF585w111wzz1pUN1668Epp8D33+cdmTVTThBm5eZXv0qtib32SlN2rLtumr7DbD45QZiVo2WWgZtugocegi+/TJfFHnccfPtt3pFZM+IEYVbOtt8+Xel04IFpyo61104D7czqwQnCrNwttRRcfTX8618wa1aqZve738FXX+UdmTVxThBmLcWAAfDGGyk5XH459OqVkoZZDZwgzFqS9u3h4ovhmWegbVvYais4+GCYPj3vyKwJcoIwa4k23jiNkzjuOLj++jT538MP5x2VNTFOEGYt1WKLwbnnwosvwtJLww47wG9+A599lndk1kSUNEFIel/SG5LGSqrMlp0n6S1J4ySNkNSxYP0TJE2S9LakbUoZm5ll1lsPxoxJg+ruuCMVJrr33ryjsiagMVoQm0dEn4IC2Y8BPSOiN/AOcAKApLWAPYEewLbAFZJaN0J8Zta2LZx2GlRWQteuMHhw+vnvf/OOzHLU6F1MEfFoRMzK7r4IdM1u7wzcERHfR8R/gElAv8aOz6xFW3tteOmlNOHfQw+l1sQtt7jMaQtV6gQRwKOSxkgaWuTxA4B/ZLe7AB8VPDY5WzYPSUMlVUqqnDZtWoMHbNbitWkDJ5wAr70GP/95Oi+x444weXLekVnpd0reAAAPi0lEQVQjK3WC2Cgi+gIDgWGSNq16QNJJwCzg1qpFRbb/yWFLRFwdERURUdGpU6dSxGxmAL/4BTz7bCpG9MQT6Uqna691a6IFKWmCiIgp2e+pwAiyLiNJQ4AdgH0i/vdpmwysVLB5V2BKKeMzszq0bg1HHJEG2PXtm8ZMbLUV/Oc/eUdmjaBkCUJSe0kdqm4DWwPjJW0L/BHYKSK+KdhkJLCnpHaSVgW6A56C0qwp+L//g8cfh+HD08ywvXrBZZfBnDl5R2YlVMoWRGdgtKTXSV/0D0fEKOAyoAPwWHb565UAETEBuAuYCIwChkWEi+yaNRWtWsEhh8D48bDJJmnKjl/+MlWys7KkaMb9iRUVFVFZWZl3GGYtT0SaTvyII+C77+D00+HII9MJbmvyJI0pGHpQI4+kNrP5J8GQIakw0bbbpik7NtwwtS6sbDhBmNmC+9nP4L774M4704nrvn1TFbsff8w7MmsAThBmtnAk2H331JoYPDhN2bHeevDqq3lHZgvJCcLMGkanTnDbbXD//TB1KvTrByedlM5RWLPkBGFmDWvnnVOZ0333TVN29O0LL7yQd1S2AJwgzKzhLb10qjMxalQqbbrRRnDUUfDNN3Vva02GE4SZlc4226Qrmw45JE3Z0asXPPVU3lFZPTlBmFlpLbkkXHEFPPlkOqG9+eZw6KEwY0bekVkdnCDMrHFsthmMG5e6mq66Cnr2TF1Q1mQ5QZhZ41l8cfjb3+D552GJJWDgQNh/f/jii7wjsyKcIMys8fXvn+pNnHQS3HxzKkx0//15R2XVOEGYWT7atYMzz4RXXoHOnWHQINhrL3AhsCbDCcLM8rXOOilJnHEG3Htvak3ccYcLEzUBThBmlr9FFoE//Sl1O622WmpJDBoEH3+cd2QtmhOEmTUdPXqkE9jnnw///GdqTfz9725N5MQJwsyaltat4eij0yWxvXqlq5wGDoQPPsg7shbHCcLMmqbu3dOo68sug9Gj07iJK65wmdNG5ARhZk1Xq1YwbFiarmODDdLtLbaASZPyjqxFcIIws6avW7d0TuK662DsWOjdGy64AGa7bH0pOUGYWfMgwQEHpKnEt9wynafYaKNUqMhKwgnCzJqXLl3ggQdScaJJk9I4irPPdpnTEnCCMLPmR0pjJSZOTAWKTjopVbAbOzbvyMqKE4SZNV/LLw933QX33ZcG1a23Xhpw9/33eUdWFpwgzKz5GzQotSb22QfOOiuVOX3ppbyjavacIMysPCyzTBp1/cgjqRjRhhvCMce4zOlCKGmCkPS+pDckjZVUmS1bRtJjkt7Nfi+dLZekSyRNkjROUt9SxmZmZWrgwHSl08EHp9oTa68NzzyTd1TNUmO0IDaPiD4RUZHdPx54PCK6A49n9wEGAt2zn6HA8EaIzczK0ZJLwpVXwhNPpJHXv/wlHH44zJyZd2TNSh5dTDsDN2a3bwR2KVh+UyQvAh0l/SyH+MysXGy+eZrT6Q9/SNN09OwJjz6ad1TNRqkTRACPShojaWi2rHNEfAyQ/V4+W94F+Khg28nZsnlIGiqpUlLlNBcWMbO6tG8PF12U5nNabDHYZps04M5lTutU6gSxUUT0JXUfDZO0aS3rqsiyn8zxGxFXR0RFRFR06tSpoeI0s3K34YZpnMTxx8NNN6WpxUeOzDuqJq2kCSIipmS/pwIjgH7Af6u6jrLfU7PVJwMrFWzeFZhSyvjMrIVZdFH4y1/SJbCdOqVBdnvvDZ9+mndkTVLJEoSk9pI6VN0GtgbGAyOBIdlqQ4AHstsjgX2zq5n6A9OruqLMzBrUuuumMqennQb33JMKE911lwsTVVPKFkRnYLSk14GXgYcjYhRwDrCVpHeBrbL7AI8A7wGTgGuAw0oYm5m1dG3bwimnwKuvptli99gDdt3VZU4LKJpxxqyoqIjKysq8wzCz5m7WLLjwwpQwFl00ndTed98051MZkjSmYOhBjTyS2sysTRs49lh4/fV0Kex++8F228GHH+YdWa6cIMzMqqyxBjz9NFx6KTz7bLrS6corW2yZUycIM7NCrVqlUdfjx0P//nDooTBgQIssc+oEYWZWTLduadT1tdfCa6+lMqcXXtiiypw6QZiZ1USCAw9Mk/8NGABHHQUbbwxvvpl3ZI3CCcLMrC5duqRR17fcAu+8A336tIgyp04QZmb1IaWCRIVlTtdfv6zLnDpBmJnNj86d06jre++FKVNSmdNTTinLMqdOEGZmC2LXXVNrYu+94Ywz0vQdZVbm1AnCzGxBLbMM3HgjPPwwTJ+eZow99tiyKXPqBGFmtrC2225umdPzzy+bMqdOEGZmDaFYmdNhw5p1mVMnCDOzhlRV5vSII2D48GZd5tQJwsysobVvn0ZdF5Y5PfBA+PLLvCObL04QZmalUljm9MYbU2GiZlTm1AnCzKyUaipzOm1a3pHVyQnCzKwxVJU5Pf30uWVO77yzSZc5dYIwM2ssbdvCySenMqerrgp77gmDBqUR2U2QE4SZWWPr2ROefx7OOw/++c/UmrjhhibXmnCCMDPLQ5s2cMwxqcxpr15wwAEwcCB88EHekf2PE4SZWZ4Ky5yOHp1aF8OHN4kyp04QZmZ5q17m9LDD0oC7nMucOkGYmTUVhWVOX389lTk9+OB0niKH4kROEGZmTUlVmdOJE2G33eCOO2DbbVMdiv33TzPHNlLtiZInCEmtJb0m6aHs/gBJr0oaK2m0pNWz5e0k3SlpkqSXJHUrdWxmZk3Wiium0dfTpsEDD8COO8KIEbDDDrD88nDBBSUPoTFaEH8ACit8Dwf2iYg+wG3An7LlBwJfRMTqwIXAuY0Qm5lZ07boorDTTilZTJ0KjzwCgwenOtkl1qaUTy6pK7A9cBZwVLY4gCWz20sBVSNEdgZOzW7fA1wmSRFN7MJgM7O8tG2bLoUdOLBRXq6kCQK4CDgO6FCw7CDgEUnfAjOA/tnyLsBHABExS9J0YFng08InlDQUGAqw8sorlzR4M7OWrGRdTJJ2AKZGxJhqDx0JbBcRXYEbgKqONBV5mp+0HiLi6oioiIiKTp06NWjMZmY2VylbEBsBO0naDlgUWFLSw8CaEVFV2ftOYFR2ezKwEjBZUhtS99PnJYzPzMxqUbIWREScEBFdI6IbsCfwBOk8w1KS1shW24q5J7BHAkOy24OBJ3z+wcwsP6U+BzGP7NzCwcC9kuYAXwAHZA9fB9wsaRKp5bBnY8ZmZmbzapQEERFPAU9lt0cAI4qs8x2wW2PEY2ZmdfNIajMzK8oJwszMilJzPg8saRowP5OnL0e1cRUtgN9zy+D33DI01HteJSLqHCfQrBPE/JJUGREVecfRmPyeWwa/55ahsd+zu5jMzKwoJwgzMyuqpSWIq/MOIAd+zy2D33PL0KjvuUWdgzAzs/praS0IMzOrJycIMzMrqsUkCEnbSno7K2l6fN7xlIKklSQ9KelNSRMk/SFbvoykxyS9m/1eOu9YG1KRsrarZmVr383K2LbNO8aGJKmjpHskvZXt6w1awD4+MvtMj5d0u6RFy20/S7pe0lRJ4wuWFd2vSi7Jvs/GSepbiphaRIKQ1Bq4HBgIrAXsJWmtfKMqiVnA0RHxC1IhpmHZ+zweeDwiugOPZ/fLSfWytucCF2bv9wtSOdtycjEwKiLWBNYmvfey3ceSugC/ByoioifQmjSZZ7nt578D21ZbVtN+HQh0z36Gkko5N7gWkSCAfsCkiHgvIn4A7iBNPV5WIuLjiHg1uz2T9MXRhfReb8xWuxHYJZ8IG15BWdtrs/sCtiCVrYXye79LApuSZj8mIn6IiC8p432caQMsltWKWRz4mDLbzxHxDD+tgVPTft0ZuCmSF4GOkn7W0DG1lATxv3KmmcnZsrIlqRuwDvAS0DkiPoaURIDl84uswVWVtZ2T3V8W+DIiZmX3y21frwZMA27IutWuldSeMt7HEfH/gPOBD0mJYTowhvLez1Vq2q+N8p3WUhJEvcqZlgtJSwD3AkdExIy84ymVGsralvu+bgP0BYZHxDrA15RRd1IxWb/7zsCqwIpAe1IXS3XltJ/r0iif85aSIKrKmVbpCkzJKZaSkrQIKTncGhH3ZYv/W9X8zH5PzSu+BlZV1vZ9UrfhFqQWRcesKwLKb19PBiYXlO29h5QwynUfA2wJ/CcipkXEj8B9wIaU936uUtN+bZTvtJaSIF4BumdXPbQlneAamXNMDS7rf78OeDMiLih4qLCc6xDggcaOrRSKlbWNiH2AJ0lla6GM3i9ARHwCfCTp59miAcBEynQfZz4E+ktaPPuMV73nst3PBWraryOBfbOrmfoD06u6ohpSixlJLWk70tFla+D6iDgr55AanKSNgWeBN5jbJ38i6TzEXcDKpH+23SKi+smwZk3SZsAxEbGDpNVILYplgNeAX0fE93nG15Ak9SGdlG8LvAfsTzrYK9t9LOk0YA/SlXqvAQeR+tzLZj9Luh3YjDSl93+BPwP3U2S/ZonyMtJVT98A+0dEZYPH1FIShJmZzZ+W0sVkZmbzyQnCzMyKcoIwM7OinCDMzKwoJwgzMyvKCcJaNElfZb+7Sdq7gZ/7xGr3n2/I5zcrNScIs6QbMF8JIpsluDbzJIiI2HA+YzLLlROEWXIOsImksVntgdaSzpP0Sjbf/m8hDcjLam7cRhqQiKT7JY3J6hUMzZadQ5p9dKykW7NlVa0VZc89XtIbkvYoeO6nCmo93JoNiELSOZImZrGc3+h/HWuR2tS9ilmLcDzZSGyA7It+ekSsJ6kd8JykR7N1+wE9I+I/2f0DstGtiwGvSLo3Io6XdHhE9CnyWrsCfUi1HJbLtnkme2wdoAdpXp3ngI0kTQQGAWtGREjq2ODv3qwItyDMituaNNfNWNJUJcuSirMAvFyQHAB+L+l14EXSBGrdqd3GwO0RMTsi/gs8DaxX8NyTI2IOMJbU9TUD+A64VtKupKkVzErOCcKsOAG/i4g+2c+qEVHVgvj6fyulOaC2BDaIiLVJcwItWo/nrknhXEKzgTZZzYN+pFl6dwFGzdc7MVtAThBmyUygQ8H9fwKHZtOnI2mNrDBPdUsBX0TEN5LWJJV6rfJj1fbVPAPskZ3n6ESqEPdyTYFl9T2WiohHgCNI3VNmJedzEGbJOGBW1lX0d1Ld527Aq9mJ4mkUL2k5CjhE0jjgbVI3U5WrgXGSXs2mIa8yAtgAeJ1U5OW4iPgkSzDFdAAekLQoqfVx5IK9RbP549lczcysKHcxmZlZUU4QZmZWlBOEmZkV5QRhZmZFOUGYmVlRThBmZlaUE4SZmRX1/wHhJg/97Q1YhQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmYHFXZ/vHvTQKGQEIgCcgSCPsqhDAEEEEUFUFkc0EQCSBLFIXgyo+fr6AoiwuiIpuABmRfBERFAgqKyJLIFggYXhAIEDJAICSsCc/7xzlNOk1PV89M90wzc3+ua67pqjpV56murnqqzqmuVkRgZmZWyxK9HYCZmbU+JwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWqM8nC0n/lfSqpHmSnpX0G0nL9nZc5XKMH6mj3JqS3pJ0ek/E1VskrSzpXEnPSHpZ0kOSvidpmd6OrYikz0q6TdIrkm6uMn2MpKl5+lRJY2osawVJv5c0X9LjkvatmL5vHj9f0tWSVqh33orlHCcpJB1RMX5iHn9cHt4hD/+qotytkg7Irw+QdGvZtA/k9+MlSS9I+qekLSUdk/fJeZJek7SwbPiBiuVvVzZtfo5hXtnf6h2tWy2SBuVlrVajzARJC8rqelTSOZLW7kQ9l0j6TldizPMPl3R+Pn7NzfvDUT1Rd7k+nyyyT0bEssBYYEug02+epIENj6rz9gfmAJ+T9J6erLin1j8f8P4FLA1sExFDgI8Cw4C6d9Cy5fX0dnsBOBU4qUosSwHXAL8DlgcmAdfk8dX8CngDWAn4PHCGpI3zsjYGzgK+kKe/Apxez7wd+A8wvmLc/nl8ufnA/pJG11gWOcahwHXAL4EVgFWB7wGvR8QJEbFs3i8nAP8qDUfEYnFGxD/KypamDSsr/0RRLN10c657OWCnPG6qpPWbXG/JaYCA9Uj7wV7Af3uo7kUiok//5Tf1I2XDPwauy6+XA84FngGeAn4ADMjTDgD+CfyMdAD4QR5/CDAdeBl4EBibx68CXAm0A48BR5TVeRxwGXB+nu8BoC1PuwB4C3gVmAd8q8a6/C/wJeBZ4NMV0zYGJudYnwWOyeMHAMfkeV8GpgKjgNFAAAPLlnEzcHBH6086WP8VeB54DriQtNOW5h8FXJXfg+dJH/L35PnfV1Zuxby+I6us4w+A+4ElOngPOhv3icCLwCZl5Ufm+lfMw7sC9+RytwGbNuBzdzDpIFM+7mP5c6aycU8AH68y/zKkg/16ZeMuAE7Kr08ALiqbtnYuP6Ro3ip1HUdKYNOBjcs+T9Pz+OPyuB2AmaSD/2/K5r8VOKDs/b81v24DXqzjvXp7njrKvmP75/ErkPavWcCTwLGlzxCwQY7xpfzZPD+PvzMvaz5p39ujSn0TgBurjL8R+F1+PZC07z+bP0N/A9bP044A3gRez3Vcnsd/l3SceBmYBnyixjo/Uu0zUjZ9E9J+OSdvsz1q1d3Vv/5yZQGApFHALsDdedQkYAGwDrA5aWc+uGyWrYBHSQe3H0r6DGnH2h8YCuwGPC9pCeAPwL2ks6cdgYmSdipb1m7AJaQzg2tJB1Ii4gukA8YnI50l/aiD2LcDVsvLuCzHUJo2hPThvZ6UtNYBbsqTvwbsk9d7KHAQ6Sy0HoutP+ns5sRcx4ak5HBcjmEA6SzycdIOvSpwSUS8nmPer2y5+5B2wPYqdX4EuCoi3qozxqK4v09KYPuUTf8scEtEzJY0FjgPOAwYTjpbv7ZJV24bA/dF3pOz+1h0tlxuPWBhRJSf2d9bVnbjPAxARPwvOUHUMW9HLmDR52o86eBbzQ+BT9VxZv0fYKGkSZJ2lrR8QfnuuJCUDNYCxgF7kK66IH1mrybte6uTtjHA9vn/+nnfu7oT9V0FbFc2fC0pYb8XeIh0bCEifkFKJMfnOj6Tyz8MvJ90wnoycImkER3UdTtwsqTxktYpn5Cv3iaTTnpHkLbfeZLWqVF3l/SXZHG1pBdJZxe3ACdIWgnYGZgYEfMjYjbpbPRzZfM9HRG/jIgFEfEqKZH8KCLuiuSRiHic1LQ1MiK+HxFvRMSjwK8rlnVrRPwpIhaSdsrNOrkO44E/R8Qc4CJgZ0kr5mm7ArMi4qcR8VpEvBwRd+RpBwPfiYiHc8z3RsTzdda52Prn9Z0cEa/nA/0pwAdz2XGkJPLN/H6+FhGltutJwL45qULaiS/ooM7hpCu97qjcbhexeLLYN4+DdKV4VkTcERELI2IS6Uxs627GUM2ypANauZdIVwOdLVtremfqKfc7YB9JS5I+u7+rVigiZgFnkhJxhyJiLvAB0tn7r4F2Sdfmfa9hJK1BOvB/LSJeiYhngF+waP97k3QC8978Of5nA6p9mnQ1Q/6cTYqIeRHxGqmpbZykQR3NHBGXRsQzEfFWRFxAuuLcooPih5EO+kcBD0l6WIv6OPcEpkXEhfnzexfpxPVTDVjHxfSXZLFHRAyLiDUi4sv5ALIGsCTwjKQXczI5i3Q2WvJkxXJGkZpzKq0BrFJaTl7WMaT24pJZZa9fAQbV254uaWngM6SzJyLiX6SrkVKnZUdxFU0rstj6S1oxd5g9JWku6WBSOhsaBTweEQsqF5IT13zgg5I2IF35XNtBnc8DK3cx3qpxky7Rl5a0VT6wjAF+n6etAXy9YtuNIiW+xVR0yp7Zhbjmka7uyg0lNUV0tmyt6Z2p522R2v4fITVxzYiIyvex3MnATpJqnvRExPSIOCAiViM1l6xC6tNppDWAQaRkVNqGP2fR/ncUMBi4W9J9kvbrYDmdsSqpmRNJAyX9JHd+zyVdWYh04lOVpC/mWErxrsOifWkx+eTr+xExJi/zD8CVuUVhDWD7is/vp+j+PvQO/SVZVPMk6QxyRE4kwyJiaCzeuVb5SN4nqd7J+iTwWNlyhkXEkIjYpc5Yih79uydpZz9d0ixJs0gf1lKTQUdx1Zo2P/8fXDbuvQVxnZjHbRoRQ0lNSyqrZ/UaCXBSLv8F4Ip8BlbNjcCeZVch3Y47N2ldRrq62JfUZ1U6cD4J/LBi2w2OiIsrK46yTtmImNBBfLU8AGwqSWXjNs3jK/0HGChp3bJxm5WVfYCyq1NJa5H6h/5Tx7y1nA98nY6boADIV6enAsfXsczSPA8BvyUljUZ6kpQgl6/Yl8fmep+KiINIB9AjSM00q1O839WyB/CP/PpAUhP2h0jNShvk8aXtvFg9ktYj9fscCqwQEcNISbr8c1FVRLxEunliKKlJ7UnghorP77IRMbFa3d3Rb5NFvlS9AfippKGSlpC0tqQP1pjtHOAbkrZQsk4+U70TmCvp25KWljRA0iaStqwznGdJba0dGU9qV38f6ax4DLAtMEbS+0h9Be9VutXxPZKGSNqqLObjJa2bY95U0vDcjPQUsF+O9yCK7zYaQtopX5S0KvDNsml3kpqPTpK0jNJtiduWTb+AlPT2o/aB6BTSjjApv7dIWlXSKZI27WLckJqd9ibdGXRR2fhfAxPyVYdy7J/IZ22dlmMaROr0XCK/D0vmyTcDC4Ej8nb6Sh7/1zzvAZL+C+lsktQu/v0c07bA7ixqvrsQ+KTSbaXLkPtmchNk0by1XEo68F1WR9lTSO3uG3bwXmwg6evKt6Yq9RnuQ2qDb5iIeCwv80f5s79E/rx/INe7t6RVIiJIHdAACyL1p5X6OQrlbbu2pLNIza4/yJOGAK+RroqXKRtfUrl/L0u6qaWd9BmZQLqy6Kje4ySNlbRkbmU4gnSDySOkvpjN8zouKWkpSVvnhFSt7i7rt8ki2x9YinRX0xzgCmpcvkXE5aTOvYtIl/RXk84MFgKfJB3EHyNtyHNIZxn1OBH4Tr6M/Eb5hHxQ3hE4NSJmlf1NJXVoj89nyR/NMcwCZpDOciDt0JeREuNcUkfY0nnaIaQD/vOkzs/bCuL8Hun245eAP5IOSKX3pvQerENqIptJOjiXps8E/k060/kHHYiIF0gHoDeBOyS9TOqsf4m0c3Ql7vKmsFWAP5eNn5KXdxrpM/AI6e6crvoC6U6rM0gdoK+SEhIR8QbpjHR/0kHrIFIT6Rt53lGkO7lKvkzaVrOBi4EvRcQDeVkPkO7UuTBPH5LLF85bS27TvzE31RaVnQv8iNx2X8XLpJsN7pA0n3RAn0a6cmm0fUgd2A+RmocuZVEz1DakW13nAZcDh0bE03nad4HL8763WwfL3iHPO5f0WVyKdDfjQ3n6uaQD/yzSnXy3Vsx/NrBlruOSiPg3qc9nCukEa838uiNLkJp8XyDtV9sCu+S+wzmk23kPzMt6mpSsSicoi9Vdo45CimjYVYpZTZLOI3U+N+RLQn2NpBuAIyNiem/HYlbJycJ6hNKXuO4BNs/NBmb2LtLfm6GsB0g6ntT88GMnCrN3J19ZmJlZIV9ZmJlZoVZ4OF5DjBgxIkaPHt3bYZiZvatMnTr1uYgYWVSuzySL0aNHM2VKrbvPzMyskqTH6ynnZigzMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVmhpiULSedJmi1pWtm4FSRNljQj/1++g3nH5zIzJI1vVoxmZlafZl5Z/Bb4eMW4o4GbImJd4KY8vBhJKwDHAlsB44BjO0oqZmbWM5qWLCLi78ALFaN3Bybl15OAParMuhMwOSJeiIg5wGTemXTMzKwH9XSfxUoR8QxA/r9ilTKrAk+WDc/M495B0qGSpkia0t7e3vBgzcwsacUOblUZF9UKRsTZEdEWEW0jR45sclhmZv1XTyeLZyWtDJD/z65SZiYwqmx4NeDpHojNzMw60NPJ4lqgdHfTeOCaKmX+AnxM0vK5Y/tjeZyZmfWSZt46ezHwL2B9STMlfRE4CfiopBnAR/MwktoknQMQES8AxwN35b/v53FmZtZLFFG1O+Bdp62tLaZMmdLbYZiZvatImhoRbUXlWrGD28zMWoyThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoUKk4WkKyV9QlLDEoukIyVNk/SApIlVpi8n6Q+S7s1lDmxU3WZm1nn1JIAzgH2BGZJOkrRBdyqUtAlwCDAO2AzYVdK6FcUOBx6MiM2AHYCfSlqqO/WamVnXFSaLiLgxIj4PjAX+C0yWdJukAyUt2YU6NwRuj4hXImIBcAuwZ2W1wBBJApYFXgAWdKEuMzNrgLqaliQNBw4ADgbuBn5OSh6Tu1DnNGB7ScMlDQZ2AUZVlDmNlFSeBu4HjoyIt6rEdaikKZKmtLe3dyEUMzOrRz19FlcB/wAGA5+MiN0i4tKI+CrprL9TImI6cDIp0VwP3Ms7rxp2Au4BVgHGAKdJGlplWWdHRFtEtI0cObKzoZiZWZ3qubI4LSI2iogTI+KZ8gkR0daVSiPi3IgYGxHbk5qYZlQUORC4KpJHgMeAbvWVmJlZ19WTLDaUNKw0IGl5SV/uTqWSVsz/Vwf2Ai6uKPIEsGMusxKwPvBod+o0M7OuqydZHBIRL5YGImIO6W6m7rhS0oPAH4DDI2KOpAmSJuTpxwPvl3Q/cBPw7Yh4rpt1mplZFw2so8wSkhQRASBpANCt21gjYrsq484se/008LHu1GFmZo1TT7L4C3CZpDNJt7ROIHVMm5lZP1FPsvg2cBjwJUDADcA5zQzKzMxaS2GyyN9vOCP/mZlZP1SYLPKjOE4ENgIGlcZHxFpNjMvMzFpIPXdD/YZ0VbEA+BBwPnBBM4MyM7PWUk+yWDoibgIUEY9HxHHAh5sblpmZtZJ6Orhfy48nnyHpK8BTwIrNDcvMzFpJPVcWE0nPhToC2ALYDxjfzKDMzKy11LyyyF/A+2xEfBOYR3pmk5mZ9TM1rywiYiGwRf5dCTMz66fq6bO4G7hG0uXA/NLIiLiqaVGZmVlLqSdZrAA8z+J3QAXgZGFm1k/U8w1u91OYmfVz9XyD+zekK4nFRMRBTYnIzMxaTj3NUNeVvR4E7En6bWwzM+sn6mmGurJ8WNLFwI1Ni8jMzFpOPV/Kq7QusHqjAzEzs9ZVT5/FyyzeZzGL9BsXZmbWT9TTDDWkJwIxM7PWVdgMJWlPScuVDQ+TtEdzwzIzs1ZST5/FsRHxUmkgIl4Ejm1eSGZm1mrqSRbVytRzy62ZmfUR9SSLKZJOkbS2pLUk/QyY2uzAzMysddSTLL4KvAFcClwGvAoc3sygzMystdRzN9R84OgeiMXMzFpUPXdDTZY0rGx4eUl/aW5YZmbWSupphhqR74ACICLm4N/gNjPrV+pJFm9JevvxHpLWoMpTaM3MrO+q5xbY/w/cKumWPLw9cFjzQjIzs1ZTTwf39ZLGAlsDAo6KiOeaHpmZmbWMup46GxHPRcR1wIPABEnTmhuWmZm1knruhlpZ0kRJdwIPAAOAfZoemZmZtYwOm6EkHUJKCquRvox3MHBNRHyvu5VKOhI4hNSs9euIOLVKmR2AU4Elgeci4oPdrbdDEyfCPfc0bfFmZk01Zgyc+o7DaEPV6rP4FfAvYN+ImAIgqdt3QUnahJQoxpG+GX69pD9GxIyyMsOA04GPR8QTknyrrplZL6qVLFYBPgOcImkl0tXFkg2oc0Pg9oh4BSDfZbUn8KOyMvsCV0XEEwARMbsB9XasyRnZzOzdrsM+i9ypfUZEbA/sCLwEzJY0XdIJ3ahzGrC9pOGSBgO7AKMqyqwHLC/pZklTJe1fbUGSDpU0RdKU9vb2boRkZma11Hs31MyI+ElEbAHsAbze1QojYjpwMjAZuB64F1hQUWwgsAXwCWAn4H8krVdlWWdHRFtEtI0cObKrIZmZWYG6kkW5iHi4u53cEXFuRIzNVy0vADMqiswEro+I+fk7HX8HNutOnWZm1nWdThaNUOqwzo8R2Qu4uKLINcB2kgbmpqqtgOk9G6WZmZX01i/eXSlpOPAmcHhEzJE0ASAizoyI6ZKuB+4D3gLOiQh/EdDMrJcUJgtJN0XEjkXjOiMitqsy7syK4R8DP+5qHWZm1ji1vpQ3CBgMjJC0POkLdABDSbfVmplZP1HryuIwYCIpMUxlUbKYS/rCnpmZ9RMdJouI+Dnwc0lfjYhf9mBMZmbWYup5RPkvJb0fGF1ePiLOb2JcZmbWQurp4L4AWBu4B1iYRwfgZGFm1k/Uc+tsG7BRRPinVM3M+ql6vpQ3DXhvswMxM7PWVc+VxQjgwfzjR28/EyoidmtaVGZm1lLqSRbHNTsIMzNrbfXcDXWLpDWAdSPixvyspgHND83MzFpFPb/BfQhwBXBWHrUqcHUzgzIzs9ZSTwf34cC2pG9uk3/+1D9zambWj9STLF6PiDdKA5IGkr5nYWZm/UQ9yeIWSccAS0v6KHA58IfmhmVmZq2knmRxNNAO3E96uOCfgO80MygzM2st9dw6uzRwXkT8GkDSgDzulWYGZmZmraOeK4ubSMmhZGngxuaEY2ZmraieZDEoIuaVBvLrwc0LyczMWk09yWK+pLGlAUlbAK82LyQzM2s19fRZHAlcLunpPLwysHfzQjIzs1ZTM1lIWgJYCtgAWJ/006oPRcSbPRCbmZm1iJrJIiLekvTTiNiG9KhyMzPrh+rps7hB0qckqenRmJlZS6qnz+JrwDLAQkmvkpqiIiKGNjUyMzNrGfU8onxITwRiZmatq55HlEvSfpL+Jw+PkjSu+aGZmVmrqKfP4nRgG2DfPDwP+FXTIjIzs5ZTT5/FVhExVtLdABExR9JSTY7LzMxaSD1XFm/mhwcGgKSRwFtNjcrMzFpKPcniF8DvgRUl/RC4FTihqVGZmVlLqeduqAslTQV2JN02u0dETG96ZGZm1jI6TBaSBgETgHVIP3x0VkQs6KnAzMysddRqhpoEtJESxc7ATxpVqaQjJU2T9ICkiTXKbSlpoaRPN6puMzPrvFrNUBtFxPsAJJ0L3NmICiVtAhwCjAPeAK6X9MeImFFRbgBwMvCXRtRrZmZdV+vK4u0nyza4+WlD4PaIeCUv9xZgzyrlvgpcCcxuYN1mZtYFtZLFZpLm5r+XgU1LryXN7Uad04DtJQ2XNBjYBRhVXkDSqqQEcmatBUk6VNIUSVPa29u7EZKZmdXSYTNURAxoRoURMV3SycBk0rfB7wUqr1xOBb4dEQtrPew2Is4GzgZoa2uLZsRrZmb1fYO74SLiXOBcAEknADMrirQBl+REMQLYRdKCiLi6RwM1MzOgl5KFpBUjYrak1YG9SM+eeltErFlW9rfAdU4UZma9p1eSBXClpOGkTvTD8/OmJgBERM1+CjMz63m91Qy1XZVxVZNERBzQ9IDMzKymep4NZWZm/ZyThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoWcLMzMrJCThZmZFXKyMDOzQk4WZmZWyMnCzMwKOVmYmVkhJwszMyvkZGFmZoV6JVlIOlLSNEkPSJpYZfrnJd2X/26TtFlvxGlmZkmPJwtJmwCHAOOAzYBdJa1bUewx4IMRsSlwPHB2z0ZpZmbleuPKYkPg9oh4JSIWALcAe5YXiIjbImJOHrwdWK2HYzQzszK9kSymAdtLGi5pMLALMKpG+S8Cf642QdKhkqZImtLe3t6EUM3MDGBgT1cYEdMlnQxMBuYB9wILqpWV9CFSsvhAB8s6m9xE1dbWFk0J2MzMeqeDOyLOjYixEbE98AIwo7KMpE2Bc4DdI+L5no7RzMwW6fErCwBJK0bEbEmrA3sB21RMXx24CvhCRPynN2I0M7NFeiVZAFdKGg68CRweEXMkTQCIiDOB7wLDgdMlASyIiLZeitXMrN/rlWQREdtVGXdm2euDgYN7NCgzM+uQv8FtZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVSRPR2DA0hqR14vBOzjACea1I4rcrr3D94nfuHRq3zGhExsqhQn0kWnSVpSkS09XYcPcnr3D94nfuHnl5nN0OZmVkhJwszMyvUn5PF2b0dQC/wOvcPXuf+oUfXud/2WZiZWf3685WFmZnVycnCzMwK9ctkIenjkh6W9Iiko3s7nmaQNErS3yRNl/SApCPz+BUkTZY0I/9fvrdjbSRJAyTdLem6PLympDvy+l4qaanejrHRJA2TdIWkh/L23qYvb2dJR+XP9DRJF0sa1Be3s6TzJM2WNK1sXNXtquQX+Zh2n6SxjY6n3yULSQOAXwE7AxsB+0jaqHejaooFwNcjYkNga+DwvJ5HAzdFxLrATXm4LzkSmF42fDLws7y+c4Av9kpUzfVz4PqI2ADYjLT+fXI7S1oVOAJoi4hNgAHA5+ib2/m3wMcrxnW0XXcG1s1/hwJnNDqYfpcsgHHAIxHxaES8AVwC7N7LMTVcRDwTEf/Or18mHUBWJa3rpFxsErBH70TYeJJWAz4BnJOHBXwYuCIX6VPrCyBpKLA9cC5ARLwRES/Sh7czMBBYWtJAYDDwDH1wO0fE34EXKkZ3tF13B86P5HZgmKSVGxlPf0wWqwJPlg3PzOP6LEmjgc2BO4CVIuIZSAkFWLH3Imu4U4FvAW/l4eHAixGxIA/3xW29FtAO/CY3v50jaRn66HaOiKeAnwBPkJLES8BU+v52Lulouzb9uNYfk4WqjOuz9w9LWha4EpgYEXN7O55mkbQrMDsippaPrlK0r23rgcBY4IyI2ByYTx9pcqomt9HvDqwJrAIsQ2qCqdTXtnORpn/W+2OymAmMKhteDXi6l2JpKklLkhLFhRFxVR79bOnyNP+f3VvxNdi2wG6S/ktqWvww6UpjWG6ugL65rWcCMyPijjx8BSl59NXt/BHgsYhoj4g3gauA99P3t3NJR9u16ce1/pgs7gLWzXdPLEXqHLu2l2NquNxefy4wPSJOKZt0LTA+vx4PXNPTsTVDRPy/iFgtIkaTtulfI+LzwN+AT+difWZ9SyJiFvCkpPXzqB2BB+mj25nU/LS1pMH5M15a3z69nct0tF2vBfbPd0VtDbxUaq5qlH75DW5Ju5DOOgcA50XED3s5pIaT9AHgH8D9LGrDP4bUb3EZsDppx/tMRFR2or2rSdoB+EZE7CppLdKVxgrA3cB+EfF6b8bXaJLGkDr1lwIeBQ4knQj2ye0s6XvA3qQ7/u4GDia1z/ep7SzpYmAH0qPInwWOBa6mynbNifM00t1TrwAHRsSUhsbTH5OFmZl1Tn9shjIzs05ysjAzs0JOFmZmVsjJwszMCjlZmJlZIScLs0zSvPx/tKR9G7zsYyqGb2vk8s2azcnC7J1GA51KFvlpxrUsliwi4v2djMmsVzlZmL3TScB2ku7Jv50wQNKPJd2VfyvgMEhf/su/GXIR6cuPSLpa0tT8ewuH5nEnkZ6Seo+kC/O40lWM8rKnSbpf0t5ly7657HcqLsxfvELSSZIezLH8pMffHeuXBhYXMet3jiZ/AxwgH/RfiogtJb0H+KekG3LZccAmEfFYHj4of6N2aeAuSVdGxNGSvhIRY6rUtRcwhvQ7FCPyPH/P0zYHNiY94+efwLaSHgT2BDaIiJA0rOFrb1aFryzMin2M9Nyde0iPSxlO+pEZgDvLEgXAEZLuBW4nPdhtXWr7AHBxRCyMiGfPCmxKAAABIElEQVSBW4Aty5Y9MyLeAu4hNY/NBV4DzpG0F+nRDmZN52RhVkzAVyNiTP5bMyJKVxbz3y6Unkn1EWCbiNiM9IyiQXUsuyPlzzZaCAzMv9kwjvQ04T2A6zu1JmZd5GRh9k4vA0PKhv8CfCk/8h1J6+UfGKq0HDAnIl6RtAHp52xL3izNX+HvwN65X2Qk6Vfv7uwosPz7JMtFxJ+AiaQmLLOmc5+F2TvdByzIzUm/Jf3G9Wjg37mTuZ3qP9t5PTBB0n3Aw6SmqJKzgfsk/Ts/Or3k98A2wL2kH6v5VkTMysmmmiHANZIGka5KjuraKpp1jp86a2ZmhdwMZWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWaH/A3XRRVIhDvtkAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# separate the data set\n",
"nbr_of_testing_images = 10000\n",
"nbr_of_training_images = 50000\n",
"nbr_of_images = nbr_of_testing_images + nbr_of_training_images\n",
"\n",
"training_images = []\n",
"training_labels = []\n",
"testing_images = []\n",
"testing_labels = []\n",
"\n",
"for index in range(0, nbr_of_images):\n",
" if(index <= nbr_of_training_images - 1):\n",
" training_images.append(normalized_scaled_images_feature_matrix[:, index]) \n",
" training_labels.append(binary_labels[index])\n",
" else:\n",
" testing_images.append(normalized_scaled_images_feature_matrix[:, index]) \n",
" testing_labels.append(binary_labels[index])\n",
" \n",
"# covert to numpy array\n",
"training_images = np.transpose(np.array(training_images))\n",
"training_labels = np.array(training_labels)\n",
"testing_images = np.transpose(np.array(testing_images))\n",
"testing_labels = np.array(testing_labels)\n",
"\n",
"# logger\n",
"#print(training_images.shape) # training_images is a matrix of 784 X 50000\n",
"#print(training_labels.shape) # testing_labels is a row vector of 1 X 50000\n",
"#print(testing_images.shape) # training_images is a matrix of 784 X 10000\n",
"#print(testing_labels.shape) # testing_labels is a row vector of 1 X 10000\n",
"\n",
"# initialize weights & bias\n",
"initial_value = .1\n",
"W = np.multiply(initial_value, np.ones(size))\n",
"b = initial_value # b will be added in a broadcasting manner\n",
"\n",
"#print(W.shape) # W is a vector with a size of 784\n",
"\n",
"# gradient descent\n",
"\n",
"# hyber-parameters\n",
"alpha = 0.00001;\n",
"number_of_iterations = 100\n",
"\n",
"# data arrays\n",
"cost_array = []\n",
"interation_array = []\n",
"accuracy_array = []\n",
"\n",
"# rename\n",
"X = training_images\n",
"Y = training_labels\n",
"m = size\n",
"\n",
"# main loop\n",
"for interation in range(1, number_of_iterations):\n",
" \n",
" # forward propogation training data set\n",
" Z = np.dot(np.transpose(W), X) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
" A = 1/(1 + np.exp(-Z)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
" term1 = np.multiply(-Y, np.log(A)) # -Y * log(A) # broadcasting is used \n",
" term2 = np.multiply((1 - Y), np.log(1 - A)) # (1-Y) * log(1-A) # broadcasting is used \n",
" L = term1 - term2 # loss function # l is 1X50000\n",
" cost = np.multiply(1/m, np.sum(L)) # cost # scalar \n",
" \n",
" # forward propogation test data set\n",
" Z_test = np.dot(np.transpose(W), testing_images) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
" A_test = 1/(1 + np.exp(-Z_test)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
" \n",
" # accuracy\n",
" prediction_labels = np.where(A_test >= 0.5, 1, 0) # if greater than ore equal to 0.5 use 1, else 0\n",
" accuracy = np.sum(np.absolute(testing_labels - prediction_labels)) # Sum of false positives and false negatives # accuracy is scalar \n",
" percent_inaccurate = np.multiply(100/len(testing_labels), accuracy)\n",
" percent_accurate = 100 - percent_inaccurate\n",
" \n",
" # storage for plotting\n",
" cost_array.append(cost)\n",
" accuracy_array.append(percent_accurate)\n",
" interation_array.append(interation)\n",
" \n",
" # backpropogation\n",
" dZ = A - Y # dZ is 1X500\n",
" dW = np.multiply(1/m, np.dot(X, np.transpose(dZ))) # dW is 784X1\n",
" db = np.multiply(1/m, np.sum(dZ)) # db is a scalar\n",
" \n",
" # paramter updates\n",
" W = W - np.multiply(alpha, dW)\n",
" b = b - np.multiply(alpha, db)\n",
"\n",
"# forward propogation training data set\n",
"Z = np.dot(np.transpose(W), X) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
"A = 1/(1 + np.exp(-Z)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
"term1 = np.multiply(-Y, np.log(A)) # -Y * log(A) # broadcasting is used \n",
"term2 = np.multiply((1 - Y), np.log(1 - A)) # (1-Y) * log(1-A) # broadcasting is used \n",
"L = term1 - term2 # loss function # l is 1X50000\n",
"cost = np.multiply(1/m, np.sum(L)) # cost # scalar \n",
"\n",
"# forward propogation test data set\n",
"Z_test = np.dot(np.transpose(W), testing_images) + b # Linear Model # Z is 1X50000 # broadcasting is used\n",
"A_test = 1/(1 + np.exp(-Z_test)) # sigmoid(Z) # A is 1X50000 # broadcasting is used \n",
"\n",
"# final accuracy\n",
"prediction_labels = np.where(A_test >= 0.5, 1, 0) # if greater than ore equal to 0.5 use 1, else 0\n",
"accuracy = np.sum(np.absolute(testing_labels - prediction_labels)) # Sum of false positives and false negatives # accuracy is scalar \n",
"percent_inaccurate = np.multiply(100/len(testing_labels), accuracy)\n",
"percent_accurate = 100 - percent_inaccurate\n",
"\n",
"# storage for plotting\n",
"cost_array.append(cost)\n",
"accuracy_array.append(percent_accurate)\n",
"interation_array.append(number_of_iterations + 1)\n",
" \n",
"print('Final Cost: ' + str(cost))\n",
"print('Final Percent Accuracy: ' + str(percent_accurate))\n",
"\n",
"# plot learning curve\n",
"pyplot.figure()\n",
"pyplot.plot(interation_array, cost_array, 'red')\n",
"pyplot.title('Learning Curve - 50,000 MNIST Training Data Set')\n",
"pyplot.xlabel('Iterations')\n",
"pyplot.ylabel('Cost')\n",
"pyplot.show()\n",
"\n",
"# plot percent accuracy curve\n",
"pyplot.figure()\n",
"pyplot.plot(interation_array, accuracy_array, 'red')\n",
"pyplot.title('Percent Accuracy Curve - 10,000 MNIST Test Data Set')\n",
"pyplot.xlabel('Iterations')\n",
"pyplot.ylabel('Percent Accuracy')\n",
"pyplot.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The results line up with what we would intuitively expect. First, the cost does decrease but in 100 iterations it still hasn't gotten below approximately 481. This is because our low learning rate causes our parameters values to be updated at a very slow rate; therefore, the values cannot converge within 100 iterations. Second, we see the same issue being present in our accuracy. The final accuracy is about 9.61 percent; which is extremely low. This again points to the fact that with such a small learning the rate the paramter values are not able to converge within 100 iterations. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we can see from these examples, the single neuron neural network is not perfect, but even with this simple model we can achieve decently accurate results if we set our learning rate appropriately and have large enough of a training data set."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This concludes the assignment."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment