6-neural-networks-with-minibatch-stochastic-gradient-descent-and-momentum.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment6\n",
    "## CS-5891-01 Special Topics Deep Learning\n",
    "## Ronald Picard\n",
    "\n",
    "In this notebook we will walk through the design, training, and testing of neural networks with multiple hidden layers using minibatch stochastic gradient descent with momentum. These neural networks will be used for logistic regression, which is an archaic name for binary classification.\n",
    "\n",
    "The binary classification will be performed on images of handwritten numerical digits. More specifically, the last numerical digit of my student ID. This digit happens to be 9. Therefore, the goal of our neural networks will be to output a the value of 1 when the handwritten numerical digit image input is a 9, and 0 in all other cases.\n",
    "\n",
    "The data set we will be using is the MNIST data set. This is a very popular data set amoung the machine learning community. The data set contains 60,000 images, and each image contains a handwritten numerical digit. Each of the images have been provided with a truth label that corresponds to the handwritten digit within the image from the set {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}. \n",
    "\n",
    "For our case, we only care about when the image is 9. Therefore we will need to re-label the truth labels so that all truth labels with the value of 9 are given to the value of 1, and all other truth labels are given the value of 0. \n",
    "\n",
    "To start we need to import some needed classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import struct\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "import matplotlib.pyplot as pyplot\n",
    "import csv\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we must change our path string to the path of our data file containing the features. (Please note that you must change this string to point to the directory with the data file on your machine data file on your machine.) \n",
    "\n",
    "Second, we much change the string name of the data files to the names of the MNIST data files. (Please note that you may NOT need to change these. Only change them if your MINST data files are named differently.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "## path\n",
    "path = 'C:/Users/computer/OneDrive - Vanderbilt/Vanderbilt_Spring_2019/CS_5891_01_SpecialTopicsDeepLearning/Assignment2/'\n",
    "\n",
    "#Train data\n",
    "fname_train_images = os.path.join(path, 'train-images.idx3-ubyte')  # the training set image file path\n",
    "fname_train_labels = os.path.join(path, 'train-labels.idx1-ubyte')  # the training set label file path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we retrieve the data from the data files as follows. This imports the data into a feature tensor (3-D matrix) in which each index is a feature matrix corresponding to an image. The label data comes in the form of a vector where each index corresponds to the index of the feature matrix (image) of the feature tensor.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The training set contains 60000 images\n",
      "The shape of the image is (28, 28)\n"
     ]
    }
   ],
   "source": [
    "# open the label file and load it to the \"train_labels\"\n",
    "with open(fname_train_labels, 'rb') as flbl:\n",
    "    magic, num = struct.unpack(\">II\", flbl.read(8))\n",
    "    labels = np.fromfile(flbl, dtype=np.uint8)\n",
    "\n",
    "# open the image file and load it to the \"train_images\"\n",
    "with open(fname_train_images, 'rb') as fimg:\n",
    "    magic, num, rows, cols = struct.unpack(\">IIII\", fimg.read(16))\n",
    "    images = np.fromfile(fimg, dtype=np.uint8).reshape(len(labels), rows, cols)\n",
    "\n",
    "print('The training set contains', len(images), 'images')  # print the how many images contained in the training set\n",
    "print('The shape of the image is', images[0].shape)  # print the shape of the image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to perform both two steps; feature scaling and feature normalization. Feature scaling consists of converting the 28 X 28 image matrices into 784 X 1 feature vectors. In essence we will flatten the images out into vectors so that we can use an input a vector to our single neuron. Feature normalization is a process of normalizing the pixel data to between 0 <= x <= 1 (for logistic regression). Each pixel comes on a scale of 0 <= x <= 255. Since 255 is the maximum for every pixel we shall divide each pixel by that number (elementwise) in order to normalize each pixel to between 0 and 1 (inclusive).\n",
    "\n",
    "One additional item we need to take care of is relabeling our label (truth) data so that we have a binary classification in which all 9s are converted to 1s and all other labels are converted to 0s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(784, 60000)\n",
      "(784, 60000)\n",
      "60000\n"
     ]
    }
   ],
   "source": [
    "# feature scaling\n",
    "matrix_side_length = len(images[0])\n",
    "vector_size = matrix_side_length*matrix_side_length\n",
    "\n",
    "scaled_images_feature_matrix = []\n",
    "for image in images:\n",
    "    reshaped_image = np.array(image).reshape((vector_size))\n",
    "    scaled_images_feature_matrix.append(reshaped_image)\n",
    "\n",
    "# convert to numpy array\n",
    "scaled_images_feature_matrix = np.transpose(np.array(scaled_images_feature_matrix))\n",
    "print(scaled_images_feature_matrix.shape) # scaled_images_feature_matrix is a matrix of 60000 X 784\n",
    "#print(scaled_images_feature_matrix[0].shape)\n",
    "\n",
    "# feature normilization\n",
    "normilization_factor = 1/255\n",
    "normalized_scaled_images_feature_matrix = np.multiply(normilization_factor, scaled_images_feature_matrix)\n",
    "print(normalized_scaled_images_feature_matrix.shape)\n",
    "#print(normalized_scaled_images_feature_matrix[0])\n",
    "\n",
    "# re-label for binary classification\n",
    "value_for_1 = 9\n",
    "binary_labels = []\n",
    "for label in labels:\n",
    "    if(label == 9):\n",
    "        binary_labels.append(1)\n",
    "    else:\n",
    "        binary_labels.append(0)\n",
    "\n",
    "# convert to numpy array\n",
    "binary_labels = np.array(binary_labels)\n",
    "print(len(binary_labels)) # binary_labels is a row vector of 1 X 60000\n",
    "#print(binary_labels[0])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to test the efficacy of our neural networks, we need to split up the our label data into two data sets; a smaller and a larger one. The larger set will be the training data that we will use to train our neural networks on. The smaller set will be the testing data that we will used to test the accuracy of our neural nets. The MNIST data set contains 60,000 images. Therefore, we will use 50,000 images for our training data set, and 10,000 images for our testing data set. \n",
    "\n",
    "It is common practice to use a smaller subset of the total data set to debug (ensure it works) and tune hyper-parameters before using the entire time-comsuming data set. This smaller subset is known as a validation set. Therefore, we will first use a validation data set of 600 images. 500 of these images will be used for as our training data set, and the other 100 of these images will be used for our test data set. \n",
    "\n",
    "Thus, we will begin by sifting out a validation set from our total data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(784, 500)\n",
      "(500,)\n",
      "(784, 100)\n",
      "(100,)\n"
     ]
    }
   ],
   "source": [
    "# create a data set\n",
    "size = vector_size\n",
    "\n",
    "number_of_testing_images = 100\n",
    "number_of_training_images = 500\n",
    "number_of_validation_images = number_of_testing_images + number_of_training_images\n",
    "\n",
    "training_images = []\n",
    "training_labels = []\n",
    "testing_images = []\n",
    "testing_labels = []\n",
    "\n",
    "factor = 0\n",
    "for index in range(0, number_of_validation_images):\n",
    "    if(index <= number_of_training_images - 1):\n",
    "        training_images.append(normalized_scaled_images_feature_matrix[:, index + factor])        \n",
    "        training_labels.append(binary_labels[index + factor])\n",
    "    else:\n",
    "        testing_images.append(normalized_scaled_images_feature_matrix[:, index + factor])        \n",
    "        testing_labels.append(binary_labels[index + factor])\n",
    "        \n",
    "# covert to numpy array\n",
    "training_images = np.transpose(np.array(training_images))\n",
    "training_labels = np.array(training_labels)\n",
    "testing_images = np.transpose(np.array(testing_images))\n",
    "testing_labels = np.array(testing_labels)\n",
    "\n",
    "# logger\n",
    "print(training_images.shape) # validation_training_images is a matrix of 784 X 500\n",
    "print(training_labels.shape) # validation_testing_labels is a row vector of 1 X 500\n",
    "print(testing_images.shape) # validation_training_images is a matrix of 784 X 100\n",
    "print(testing_labels.shape)  # validation_testing_labels is a row vector of 1 X 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we move on to the training of our neural network with multiple hidden layers. \n",
    "\n",
    "Part 1 - Feed Forword:\n",
    "\n",
    "For these neural networks we will use multiple hidden layers with between 5-20 units per layer (neurons). The first layer will have an input of a matrix (784 X number_of_images) of vectorized images of 784 X 1, and will output a matrix (# of units X # of images). This matrix will be input into the next hidden layer, which output another matrix (# of units X # of images.) The output layer will take and input matrix that is the output matrix of the last hidden layer and will output a row vector of probabilities which we will convert into binary classifications of 0 or 1. (If P(x) >= 0.5 then we will convert it to a 1, otherwise we will convert to 0.) \n",
    "\n",
    "The model for the units of the hidden layers will be a vecorized linear model Z^[l] = W^[l]^T * A^[l-1] + B^[l], where W is a matrix of 5-10 units (# of units) X # of units or parameter weights, A is the input matrix of vectorized images (784 X # of images) or the output of one of the layers (1 X # of units), and B is a row vector of bias's. (Note: in this case, b will be scalar that applied in a broadcasing manner to save on memory.) The output of this model Z^[l] will be a matrix (5-10 units X # of images). Z^[1] will be subject to an activation function; which for this assignment will be relu (note: we will test tanh once for comparison). \n",
    "\n",
    "Hidden Layer Activation Function: relu activation function is A^[1]  = relu(Z) =  max(Z, 0).\n",
    "\n",
    "The model of the output layer will be a vectorized linear model Z^[1] = W^[l] * A^[1] + b^[l] with a single unit. This linear model will be subjected to a sigmoid activation function.\n",
    "\n",
    "The resultant row vector will then be used to calculate the cost function values in an elementwise manner. The cost function for this binary classification will be L(Y_Predicted, Y_Label) = -Y_Label^[l] * Log(A^[l]) - (1-Y_Predicted^[l]) * Log(1-A^[l]), where Y_Label is the True Label, Y_Predicted is the probability value predicted by the neural network, and A is the activation function value. The resultant cost row vector will be added up and divided by the number of elements in order to calculate the average cost.\n",
    "\n",
    "Part 2 - Back Propogation:\n",
    "\n",
    "The back propogation technique that we will use for training the neural network, will be gradient descent. This involves utilizing the gradient of the cost function to updated the model parameters in our layers. In order to calculate the gradient we will utilize the chain rule. The goal of back propogation is the adjust the parameter weights and bias's of our model to accurately perform binary classification. In general the chain rule can be used to find the gradient of the cost function (vecorzied rates of change) with respect to the model parameters. The following is the chain that we will utilize. \n",
    "\n",
    "\n",
    "Generalized Chain Rule for N layers: \n",
    "\n",
    "dL(A^[n], Y)/dW^[l] = ∏(i = n to l) (dl(A^[i], Y)/dz^[i]) * dZ^[i]/dA^[i-1] * dA^[i-1]/dZ^[i-2] *....* dz^[l]/dW^[l];\n",
    "\n",
    "dL(A^[n], Y)/dB^[l] = ∏(i = n to l) (dl(A^[i], Y)/dz^[i]) * dZ^[i]/dA^[i-1] * dA^[i-1]/dZ^[i-2] *....* dz^[l]/dB^[l];\n",
    "\n",
    "\n",
    "\n",
    "Output Layer - Back Propogation:\n",
    "\n",
    "The partial derivative of the cost function with respect to the output layer sigmoid activation function is found by the following:\n",
    "\n",
    "dL(A^[n], Y)/dA^[n] = -Y/A^[n] + (1-A^[n])/(1-A^[n]).\n",
    "\n",
    "\n",
    "Due to the chain rule, the derivative of the cost function with respect to the linear model Z^[n] is found by the following:\n",
    "\n",
    "dL(A^[n], y)/dz = dL(A^[n], y)/dA^[n] * dA^[n]/dZ^[n].\n",
    "\n",
    "The derivative of the sigmoid activation function is da/dz is found by the following:\n",
    "\n",
    "dA^[n]/dZ^[n] = sigma(Z^[n]) * (1-sigma(Z^[n]))\n",
    "\n",
    "Therefore, the derivative of the cost function with respect to the output of the linear model is found by the following:\n",
    "\n",
    "dL(A^[n], Y)/dA^[n] * dA^[n]/dZ^[n]. = (-Y/A^[n] + (1-Y)/(1-A^[n])) * (sigma(Z^[n]) * (1-sigma(Z^[n]))) = A^[n]-Y. (For convienence we will say dZ^[n] = A^[n]-Y.)\n",
    "\n",
    "Now we can extrapolate the chain rule to all the paramters of the linear model our output layer.\n",
    "\n",
    "dL(A^[n], Y)/dW^[n] = dZ^[n] * dZ^[n]/dW^[n] = dZ^[n] * A^[n-1] = A^[n-1] * dZ^[n] (we will change our notation to dW^[n] = A^[n-1] * dZ^[n] for convienence)\n",
    "\n",
    "dL(A^[n], Y)/dB^[n] = dZ^[n] * dZ^[n]/dB^[n] = dZ^[n] (we will change our notation to dW^[n] = dZ^[n] for convienence)\n",
    "\n",
    "\n",
    "\n",
    "Hidden Layers - Back Propagation:\n",
    "\n",
    "dL(A^[n], Y)/dZ^[l] = ∏(i = n to l) (dl(A^[i], Y)/dz^[i]) * dZ^[i]/dA^[i-1] * dA^[i-1]/dZ^[i-2] *....* dA^[l]/dZ^[l]\n",
    "\n",
    "dL(A^[n], Y)/dZ^[l] = dZ^[l+1] * dZ^[l+1]/dA^[l] = W^[l+1] * dz^[l+1] * (element-wise) dA^[1]/dZ^[1]. The reason this is element-wise is because we are propgating from a single neuron to a layer with multiple neurons (we shall rename this dz^[l] = W^[l+1] * dz^[l+1] * (element-wise) dA^[1]/dZ^[1] for conveinience) \n",
    "\n",
    "dA^[1]/dZ^[1] depends on the activation function we are using in the hidden layer (in this case relu): The derivative of relu activation function is dA^[l]/dZ^[l] = if Z^[l] > 0 then 1 else 0.\n",
    "\n",
    "\n",
    "dL(A^[n], Y)/dW^[l] = dZ^[l] * dZ^[l]/dW^[l] = dZ^[l] * X^T (we will change our notation to dW^[l] = dZ^[l] * A[l-1]^T for convienence)\n",
    "\n",
    "dL(A^[n], Y)/dB^[l] = dZ^[l] * dZ^[l]/dB^[l] = dZ^[l] (we will change our notation to dB^[l] = dZ^[l] for convienence)\n",
    "\n",
    "\n",
    "Find vector averages:\n",
    "\n",
    "m = # number of images\n",
    "\n",
    "dW^[l] = 1/m * (A^[l-1] * dZ^[l])\n",
    "\n",
    "dB^[l] = 1/m * (dZ^[l])\n",
    "\n",
    "\n",
    "Finally, we will update our the weights and bias's of the layers.\n",
    "\n",
    "\n",
    "W^[l]:= W^[l] - alpha * dW^[l]\n",
    "\n",
    "B^[l]:= B^[l] - alpha * dB^[l]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing we have have to do is initialize our weights and bias's. There are multiple ways to initialize weights and bias's. Typically we will set our values based on either a uniform distribution between or a normal distribution with a some reasonable mean and standard deviation. There is some flexibility in the initalization of the weights but in general they need to be small (not to small) and varied. The weights need to be different so that the gradients with respect to each other are different. In other words we don't aways want the relative rates of change to be 0. Additionally, we do not want to reach saturation on our output activation function where the gradients are 0 (vanishing gradiants). For this assignment we wills stick with with a uniform random between -.1 and .1. We will also set a random seed each time so that we fors our random values to be the same (or similar if there are more layers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Size: 784\n",
      "Weights Shape: (20, 784)\n",
      "Bias Shape: (20, 1)\n",
      "Velocity Weights Shape: (20, 784)\n",
      "Velocity Bias Shape: (20, 1)\n"
     ]
    }
   ],
   "source": [
    "# initialize weights & bias\n",
    "np.random.seed(10)\n",
    "print('Feature Size: ' + str(size))\n",
    "\n",
    "lower_bound = -.1\n",
    "upper_bound = .1\n",
    "\n",
    "#mean = 0.015\n",
    "#std = 0.005\n",
    "\n",
    "# hyper-parameters: hidden layers\n",
    "hidden_layers = 2\n",
    "units_array = [20, 10]\n",
    "Weights = []\n",
    "Bias = []\n",
    "V_dW = []\n",
    "V_dB = []\n",
    "for i in range(0, hidden_layers):\n",
    "    if(i == 0):\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], size]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "    else:\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], units_array[i-1]]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "    \n",
    "# output layer\n",
    "_W = np.float64(np.random.uniform(lower_bound, upper_bound, [1, units_array[i]]))\n",
    "_b = np.float64(np.random.uniform(lower_bound, upper_bound)) # b will be added in a broadcasting manner\n",
    "_V_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_V_dB = np.float64(np.zeros(1))\n",
    "Weights.append(_W)\n",
    "Bias.append(_b)\n",
    "V_dW.append(_V_dW)\n",
    "V_dB.append(_V_dB)\n",
    "\n",
    "Weights = np.array(Weights)\n",
    "Bias = np.array(Bias)\n",
    "V_dW = np.array(V_dW)\n",
    "V_dB = np.array(V_dB)\n",
    "\n",
    "for index in range(0, len(Weights) - 1):\n",
    "    Weights[index] = np.where(Weights[index] != 0, Weights[index], np.random.uniform(lower_bound, upper_bound))\n",
    "\n",
    "#print(train_X.shape)\n",
    "#print(np.ravel(train_Y).shape)\n",
    "\n",
    "print('Weights Shape: ' + str(Weights[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Bias Shape: ' + str(Bias[0].shape)) # vector with a size of the # of unit\n",
    "print('Velocity Weights Shape: ' + str(V_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Velocity Bias Shape: ' + str(V_dB[0].shape)) # vector with a size of the # of unit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we implement our minibatch stochastic gradient descent algorithm. The only difference betwen minibatch stochasic gradient descent and general full-batch gradient descent is that during every epoch (run) we split up our training data into minibatches based on the specified minibatch size. If they data does not evenly split, then the last batch will be smaller; so that we utilize all of the trainig data during every epoch. Then during evey epoch we train once on each minibatch. Since we are training on minibatches, our path the extrema of the function we are tryign to find will not as direct. Therefore, our cost and test accuracy will not decrease every epoch; however, the general trend will be decreasing. Minibatch gradient decense will help prevent us from getting stuck on local extrema, as well as increase the speed at which the code runs.\n",
    "\n",
    "We will also collect data on the accuracy of our networks as a function of training iterations. To do this we will need to find the number of inaccuracate binary classifications (false positives & false negatives). This will be acommplished using our test data set. We will send our test data set through the network and compare the results with the true labels of the test data set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Main Loop Epoch: 100\n",
      "Number Of Minibatches: 10\n",
      "Cost: 0.0024360064743517867\n",
      "Main Loop Epoch: 200\n",
      "Number Of Minibatches: 10\n",
      "Cost: 0.0011374617457707798\n",
      "Main Loop Epoch: 300\n",
      "Number Of Minibatches: 10\n",
      "Cost: 0.0002504582126068609\n",
      "Main Loop Epoch: 400\n",
      "Number Of Minibatches: 10\n",
      "Cost: 0.0002903632841629266\n",
      "\n",
      "Results:\n",
      "\n",
      "\n",
      "Run Time: 4.991669654846191 seconds\n",
      "Cost: 0.00019616014290038203\n",
      "Accuracy: 95.0 %\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmYXFWd//H3J91ZSEgCJGHLQgDjILKpkU1ZRBwWEXREAUFBQZSRwVFHBXUYVBxUXNCfDIoKyCYgKkaMht2RESFBNgNGwyIJIaQTkpCQtdPf3x/nXup2pbq7qrsr3Z36vJ6nnnvvuafuPafq1v3WOXdTRGBmZlatQX1dADMzG1gcOMzMrCYOHGZmVhMHDjMzq4kDh5mZ1cSBw8zMauLAkZH0W0mn9nU5rHdIOlXSb3s7r21M0p6S7u+lZR0u6ZneWFYHy+/0u5Z0r6TTOpj3Kkmb7fULkl4v6Q/V5O3zwCHpGUmH93U5IuKoiPhJPZYtaZSkSyQ9K2mlpLnZ9Nh6rK83SbpA0vqs3Plrl8L8fSQ9KGlVNtynME+SviZpSfb6uiRVWMfJhWWvltRWXF93yh0RP4mIo3o7b60kzc/qtFLSMkn/J+nMSp9DB+/v8c5K0rskPSLpJUmLJd0haVIvrvtC4OLC+4p1Xijpx5JG9KQO1ZB0hqR7KqTPl3Qo1Pe77oliGftKRPwZWC2py8+nzwPHpiCpuQ/XPQS4E3gtcCQwCjgQWALs243l9UVdboyILQuvp7KyDAF+BVwLbA38BPhVlg5wJvBOYG9gL+AY4CPlC4+I6/JlA0cBC4rrK8/fl99nNx2V1WMyaQf7OeDyTbFiSf8EXAl8HBgN7Ax8H2jrpeVPAN4M/LpsVl7n1wP7AZ/pjfVZfRR+U9dR4Tdarl8HDknHSHo4+6f2R0l7FeadK+lJSSskPS7pXYV5p2X/7L4t6UXggiztXknfkLRU0tPFyCrpHklnFN7fWd6dJf1vtu47JF0q6doOqvEBYBLwroh4PCLaImJRRHw5IqZnywtJryos/ypJF2bjh2b/Rj4raSFwpaQnJB1TyN+c/ZN8fTa9f/Z5Lcv+aR7ak++hE4cCzcAlEbE2Ir4LCDgsm38q8M2ImB8RzwHfBE7rzoqyz+DTkh4DVmVpX5D0VPY9zJZ0bCH/K/8+s88nJH1EqbW3VNJ3u5m3Sam1uCRb979V2yKIiGURcQtwEnC6pN2yZR6bbecrlFql/1l42/9mefIW2BslTZF0d1aGxZKukTS6g9W+DpgbEfdEsiIibo6I+dlyB0n6XPZbWizpBklbd7TuCsv/Z2BmRKztoM4LgNuAYkt0mKRvSZon6QVJ/yNpWPl7C9/F5ELatZIu6KCuXVJZq0TSkZLmSFou6Tuk7Tef15TtQ5ZIepL0x6+4rK0kXSnp+Wz7/JKkQYX1/D57/7JsW/nnbpR3jKTpklqybfHXksZn805SWRdhtp+4ORvv8HNW1iWYffcLgR9mi7gHeJukwZ2Vq98GjmwneAUp+o0BfgBMkzQ0y/IkcBDpX9QXgWsl7VBYxH7AU8C2wFcKaXOAscDXgR9LHXYZdJb3euCBrFwXAO/vpCqHA7+LiG51uWS2B7YBdiL9i/8paeeTOwJYHBF/zjaq35C6D7YB/gP4uaRxPVj/OyS9mO2czyqkvxZ4NNrft+bRLD2f/0hh3iOFed1xIqlFku8k/wa8KZv+CnC9pO06ef/RwBtIO9NT1HkXaUd5zyJ9p3sBU4F/qbUSEXEfsJC0/QKsBE7J6vEO4OMq/TE4OHtP3gKbSdq5XQjsAOwO7AIUg03Rg8Cekr4p6S3auMvok8Dbs/VMAF4G8kBZad3l9iT9TiqSNJG0w51bSP4GqeWzFzCF1BL7fEfLqBdJ2wI3A+eSfufzSb/73FmkwLg3qXfgvWWLuBZYDexK2hbeDnywMP9A4DHSfuLbwI+7UcxBpJ36JNLvfz3wnWzeLcA/SZpSyH8KcE023tXnPAHYMlv2vwJExD9I21dxmRuLiD59Ac8Ah1dIvwz4clnaHOCQDpbzMHBcNn4a8GzZ/NNI/7zy6eFAANtn0/cAZ3SVN/uQW4HhhfnXAtd2UK7bga928RkE8KrC9FXAhdn4ocA6YFhh/quAFXkZSM3L87PxzwLXlC1/BnBqN7+f3YEdgSbSD+F54KRs3n8CN5Tlvw64IBvfAOxWmDclq6s6Wd+hwPwK6fOBD3RR1r8Ab8/GzwDuycabs/XuX8j7C+A/upH3f4HTC/OOTD+jDss0Hzi0Qvos4LMdvOd7wMWF77rD5Wd5jif96+9o/oHAz4DFwBrSH7J82/k7hd8UMBFYS9phVbPuK/NttazOK7NtNEgtjtHZvEFZGXYq5D8I+Hs2fjjwTNl3Mbnst3ZBB2U5g/TbXFb2asu/g7Lv+kPAvYX3D8q279MK3/UZhflH558HMJ4UNIYW5r8fuL2wnr8W5o3K6jK2lu2kQr6pQEth+ofAF7PxfbLveHCVn/MaYEiFdbwAHNhZOfpti4MUXT+VNfOWSVpG2qh3BJD0AZW6sZYBe5D+NeTmVVjmwnwkIlZloxv1oXeRd0fgxUJaR+vKLSH9M+yJlohYUyjPXOAJUktgOHAsqRUE6XN7T9nn9uZKZVD7g9IVzzSJ1L22ICI2RMQfSf92js9mryT9IIpGkXYYleaPAlbmv7xuaPc5K3UpPlKo52603wbKLSyMr6Lj776zvDuWlaOz774z44EXASQdoNRV2iJpOWmn02E9JG0v6SZJz0l6ifRHo8P8EfHHiHhPRIwltSIOA87LZk8Cfl34DB8j7eC2rbIeS4GRFdKPiYiRwFtJrcxtsvTtgaFA8Xu7tYb1deXeiNiq+AIWdJC33XcZEW2kHXjF+cA/CuM7kerxQqEelwLFFm/5NgSdb3MbkTRC0o+UujBfAu6i/Xf9E+DkbPwU0vHI9VT3Ob8QEesqrHYkKeB2qD8HjnnAV8o2guER8VNJO5Ei7dnAmGzj+AuF/knSxl8PzwPbZDvs3MRO8t8BHFGhi6BoFalVk9u+bH6luuTdVccBj2fBBNLndk3Z5zYiIr5avoAoHJSO6s80CUqf82xgr7Luvr2y9Hz+3oV5exfmdccrn4PSmV2XkboT8m3gr7TfBurheVITP9fZd1+RpP1JO5h7s6QbgJ8DEyNiNPAjSvWo9N1/jdQq2DMiRpFayFXVOyIeIHVx7JElzQfeVra9DIuIhR2su9yjwKs7Wd9dpFZCftbVC6QW9D8V1jc6q3f5e1uzenb22+iJ5yl8f9nxiQkdzScF2dw80u92m0I9RkXEXvSuz5C6m/bNvuvDijMj4t6s7G8i7Q/ybqpqPueNvt9s3wqpJdqh/hI4BmcHcvJXMykwfFTSfkpGSHq7pJHACFKlWwAkfZDSD6GuIvUBziIdcB8i6QBSv3RHriFtZD+XtJvSwcgx2UGpo7M8DwPvUzoYdyRwSBVFuYHU/3oWpdYGpB/pOyQdkS1vmNIB9gkVl9IFScdJ2jr7DvYFziGdSQWpe28DcI6koZLOztLvyoZXA5+UNF7SjsCnSP+Oe8OWlLYBKZ3YsFsvLbszNwH/LmlHpYPIn672jZJGKx3Avx64KiKeyGaNJLVi12RB5cTC2xYBocIp0Fn+l4Hl2TGE/+hknYcoHajdNpt+DWl7/VOW5fvAfys7PVfStiqdZFBp3eVuA96o0pl0lXwbOFrSHhGxgRQYL5E0LtuuJqjjA8ePACdn2/LbSa3n3nIrsE+2jTcDnwCKxwLz73q8pDGkbmAAImIe8HvgG0qn2w9SOn354B6UZ0iF/eBIUoBampXh/Arvu4b0J+rliPhTVr5aP+fcIcAdWaulQ/0lcEwn9RfmrwsiYhbwYVJ/71LSwbXTIHWfkM7QuY8UWfcE/m8Tlvdk4ABSN9SFwI2kf0YbiXS2yeGkf8O3Ay+RDqyPBfIzIj5O+jEvy5Z9S1cFiIjnSfU/MFt/nj6P1Ar5HGmnOo+0c+vud30i6bNfQQoEX4vsepesmftO0pljy0h9xu8sNH9/QDpN8zFSi/A3WVqPRcSjpIO4D5D+Ge5G6fOsp8tIAfMx0oHn35D+2XXmt0rXozxLOhB7Mak7KncWcJGkFaTv7aZ8RkSsAC4C7s+6HKYC/0U6WLscmEZqrXRkKfAu4C9ZGaZny/9mNv9bwO+AO7P1/xF4YyfrbifSWVN/oJM/T1nr5TpKB/A/Rer2eSCrw210fDD2nKz8y4D3ZPXtFRHxAnAC6ftYQmpRFLehy0in0j8GzCQdSC86hfQn9nHS5/wzetYimkH7/eAXSN/P6Kx8fwQqdSlfTfrjfE1Zei2fc+5k0p+JTqn73c2Wk3Qj6UDYf/V1WWzTkvQO0unIu/Z1WfqKpD2BH0bE/n1dlkaUdYMvAvaIiKd7sJzXAf8vIrps1TlwdIPS+ewvAk+TuotuAQ6IiIf6tGBWd9mP9CBS63EH4JfA7yOiw+4is3qS9BnSmak1XyfSXXXtqlLp4pq5ks7tIM97lS7gmy3p+kp5+qHtSd0VK0ndJWc5aDQMka4ZWU7qqnqUdB2R2SYnaT6pq3OT/nGpW4tDUhPpAq23kc7cmEk6///xQp4ppP7WwyJiqaRtI2JRXQpkZma9op4tjn1JF9E9lR0svYF00Lbow8ClEbEUwEHDzKz/q+fN4sbT/uKZ8sv5ITv/W9L/ka5MviAiftfZQseOHRuTJ0/uxWKamW3+HnzwwcUR0ZNbD72inoGj0gVJ5f1izaTTww4lXXjzh+xc73ZXLUo6k3SPJiZNmsSsWbN6v7RmZpsxSf/oOld16tlVNZ/2V11OYONL/+cDv4qI9dlpZHOocJ5xRFweEVMjYuq4cb0SMM3MrJvqGThmAlOUbkE+hHQhWfnFO7cAbwFQeqjRq0l3tDUzs36qboEju8/M2aSrIZ8AboqI2Ur3rM9vaTADWCLpceBu4NMRsaReZTIzs54bcBcATp06NXyMw8ysNpIejIiNbhvTHf3lXlVmZjZAOHCYmVlNHDjMzKwmjRM47r0XPvc5aGvr65KYmQ1ojRM4HngALroIVqzoOq+ZmXWocQLHVlul4bJOH6VrZmZdcOAwM7OaOHCYmVlNHDjMzKwmDhxmZlYTBw4zM6tJ4wSO0aPT0IHDzKxHGidwNDXBqFEOHGZmPdQ4gQNSd9XSpX1dCjOzAa2xAsfw4bB6dV+XwsxsQGuswNHcDK2tfV0KM7MBzYHDzMxq4sBhZmY1ceAwM7OaNFbgGDzYgcPMrIcaK3C4xWFm1mMOHGZmVhMHDjMzq4kDh5mZ1cSBw8zMauLAYWZmNalr4JB0pKQ5kuZKOrfC/NMktUh6OHudUc/yOHCYmfVcc70WLKkJuBR4GzAfmClpWkQ8Xpb1xog4u17laMeBw8ysx+rZ4tgXmBsRT0XEOuAG4Lg6rq9rDhxmZj1Wz8AxHphXmJ6fpZV7t6RHJd0saWKlBUk6U9IsSbNaWlq6X6LmZli/vvvvNzOzugYOVUiLsulfA5MjYi/gDuAnlRYUEZdHxNSImDpu3Ljul8gtDjOzHqtn4JgPFFsQE4AFxQwRsSQi1maTPwTeUMfyOHCYmfWCegaOmcAUSTtLGgKcCEwrZpC0Q2HyWOCJOpbHgcPMrBfU7ayqiGiVdDYwA2gCroiI2ZK+BMyKiGnAOZKOBVqBF4HT6lUewIHDzKwX1C1wAETEdGB6Wdr5hfHzgPPqWYZ2HDjMzHrMV46bmVlNGi9wREBbW1+XxMxswGq8wAFudZiZ9YADh5mZ1cSBw8zMauLAYWZmNXHgMDOzmjhwmJlZTRw4zMysJg4cZmZWEwcOMzOrSWMGDj/Mycys2xorcAwenIZucZiZdVtjBQ53VZmZ9ZgDh5mZ1cSBw8zMauLAYWZmNWnMwOGzqszMuq2xAsfQoWm4dm3flsPMbABrrMCxxRZpuHp135bDzGwAc+AwM7OaOHCYmVlNGitwDBuWhmvW9G05zMwGsMYKHG5xmJn1WGMFjrzF4cBhZtZtjRU4Bg1Kp+Q6cJiZdVtdA4ekIyXNkTRX0rmd5DteUkiaWs/yAKm7ysc4zMy6rW6BQ1ITcClwFLA7cJKk3SvkGwmcA9xfr7K0M2yYWxxmZj1QzxbHvsDciHgqItYBNwDHVcj3ZeDrwKZpBmyxhQOHmVkP1DNwjAfmFabnZ2mvkPQ6YGJE3NrZgiSdKWmWpFktLS09K5UDh5lZj9QzcKhCWrwyUxoEfBv4VFcLiojLI2JqREwdN25cz0pVfoxj4ULfu8rMrAb1DBzzgYmF6QnAgsL0SGAP4B5JzwD7A9PqfoC8/BjHDjvAEUfUdZVmZpuTegaOmcAUSTtLGgKcCEzLZ0bE8ogYGxGTI2Iy8Cfg2IiYVccyte+qyp/L8fvf13WVZmabk7oFjohoBc4GZgBPADdFxGxJX5J0bL3W26Vi4Hj55VL6HXf0TXnMzAaY5nouPCKmA9PL0s7vIO+h9SzLK4rHOFasKKV/5Stw+OGbpAhmZgNZY105DjB8eKmlUQwcPkBuZlaVxgscY8bA4sVpfOXKUnpbW9+Ux8xsgGm8wDFuXDrGsWpVqcWxxRYOHGZmVWq8wDF2bBq2tJRaHKNHO3CYmVWp8QJHfgFhS0upxeHAYWZWNQcOSIFjw4a+K5OZ2QDS2IHDXVVmZjVr7MCRtzhGjnTgMDOrUuMFjpEj03DlyvQaMQKamx04zMyq1HiBY9AgkNIxjTVr0k0PBw1y4DAzq1LjBQ5ILYzWVli/HgYPduAwM6tBYweO1tY07sBhZla1xg4cbnGYmdXMgcOBw8ysJo0dONxVZWZWs8YMHE1NG7c4fOW4mVlVGjNwlHdVNTW5xWFmVqXGDhzuqjIzq1njBo4NG3xw3MysGxo3cPisKjOzbmncwLF8ebrJobuqzMxq0tzXBegTzc0wY0YanzjRgcPMrAaN2+LIuavKzKwmDhzuqjIzq0ljBo6mptK4WxxmZjWpKnBIuqaatAp5jpQ0R9JcSedWmP9RSY9JeljSvZJ2r67YPVSpq8pXjpuZVaXaFsdrixOSmoA3dPaGLM+lwFHA7sBJFQLD9RGxZ0TsA3wd+FaV5emZ8q4qXzluZla1TgOHpPMkrQD2kvRS9loBLAJ+1cWy9wXmRsRTEbEOuAE4rpghIl4qTI4AouYadIcPjpuZdVungSMiLoqIkcDFETEqe42MiDERcV4Xyx4PzCtMz8/S2pH0MUlPkloc51RakKQzJc2SNKulpaWL1VahUuAAiE0Tt8zMBrJqu6pulTQCQNIpkr4laacu3qMKaRvtmSPi0ojYFfgs8IVKC4qIyyNiakRMHTduXJVF7kSls6rArQ4zsypUGzguA1ZJ2hv4DPAP4Oou3jMfmFiYngAs6CT/DcA7qyxPz1Q6qwocOMzMqlBt4GiNiCAdo/hORHwHGNnFe2YCUyTtLGkIcCIwrZhB0pTC5NuBv1dZnt7jFoeZWU2qveXICknnAe8HDsrOmBrc2RsiolXS2cAMoAm4IiJmS/oSMCsipgFnSzocWA8sBU7tbkVqUgwQbW0OHGZmNag2cJwAvA/4UEQslDQJuLirN0XEdGB6Wdr5hfGP11DW3lO8ZsOBw8ysJlV1VUXEQuA6YLSkY4A1EdHVMY7+qxggNmxw4DAzq0G1V46/F3gAeA/wXuB+ScfXs2B11VHg8NXjZmZdqrar6vPAGyNiEYCkccAdwM31KlhdlQeO/CwrtzjMzLpU7VlVg/KgkVlSw3v7Hx8cNzPrtmpbHL+TNAP4aTZ9AmUHvQcUH+MwM+u2TgOHpFcB20XEpyX9C/Bm0hXh95EOlg9MDhxmZt3WVXfTJcAKgIj4RUR8MiI+QWptXFLvwtVN8Z5UDhxmZjXpKnBMjohHyxMjYhYwuS4l2hTc4jAz67aujnEM62TeFr1ZkE2qeNrtwQeDsvsxOnCYmXWpqxbHTEkfLk+UdDrwYH2KtAnkAeKmm+BDH3KLw8ysBl21OP4d+KWkkykFiqnAEOBd9SxYXeUBYuLE1NrwBYBmZlXrNHBExAvAgZLeAuyRJf8mIu6qe8nqKQ8cecBwi8PMrGpVXccREXcDd9e5LJtOeeDwleNmZlUbuFd/94RbHGZm3daYgeMb34DJk2G33dK0A4eZWdWqveXI5uWww+Dpp0vTDhxmZlVrzBZHOQcOM7OqOXCAA4eZWQ0cOMCBw8ysBg4c4MBhZlYDBw7wleNmZjVw4AC3OMzMauDAAb5y3MysBg4c4BaHmVkNHDjAgcPMrAYOHFA5cDz5JAweDE880TdlMjPrp+oaOCQdKWmOpLmSzq0w/5OSHpf0qKQ7Je1Uz/J0qFLguOkmaG2Fn/ykT4pkZtZf1S1wSGoCLgWOAnYHTpK0e1m2h4CpEbEXcDPw9XqVp1N54Fi5Er7//TQ0M7OK6tni2BeYGxFPRcQ64AbguGKGiLg7IlZlk38CJtSxPB3LA8dtt8FZZ8FBB5WeQx7RJ0UyM+uv6hk4xgPzCtPzs7SOnA78ttIMSWdKmiVpVktLSy8WMZMHjgUL0vDhhx04zMw6UM/AoQppFffCkk4hPcv84krzI+LyiJgaEVPHjRvXi0XM5IHjV78qpa1fn6+899dnZjaA1fN5HPOBiYXpCcCC8kySDgc+DxwSEWvrWJ6O5RcAFj3//KYvh5nZAFDPFsdMYIqknSUNAU4EphUzSHod8APg2IhYVMeydG5QhY/huefS0C0OM7N26hY4IqIVOBuYATwB3BQRsyV9SdKxWbaLgS2Bn0l6WNK0DhZXXw4cZmZVq+ujYyNiOjC9LO38wvjh9Vx/1SoFjoULN305zMwGAF85DpUDx6rsLGG3OMzM2nHggMqB4+WX09CBw8ysHQcOaB843ve+NFybneDV2rrpy2Nm1o85cACsW5eGu+0G113XPpDk88zMDHDgSFasSMNRo9JwyJDSvLV9c2mJmVl/5cAB8IY3pC6qq69O04MHl+Y5cJiZtVPX03EHjMGDUxdVcTrnwGFm1o5bHJUUA4ePcZiZtePAUYmPcZiZdciBoxJ3VZmZdciBoxJ3VZmZdciBoxJ3VZmZdciBoxJ3VZmZdciBoxIHDjOzDjlwVFIMHKtXd563ra105bmZWQNw4KikeIwjv0tuRz772XSrkq7ymZltJhw4Kim2OFau7PzW6lddVcpnZtYAHDgqKQaOtrbOj3O0taWhVN8ymZn1Ew4cleSBozm7lVdnrYm8NeLndphZg3DgqCQPHNtvn4Yf/jB8+cuV8+aBY/36+pfLzKwfcOCoJA8CeeC45RY4//zKeR04zKzBOHBUkh/TyANHZ/JjHA4cZtYgHDgqqSVwuMVhZg3GgaMSBw4zsw45cFTiriozsw7VNXBIOlLSHElzJZ1bYf7Bkv4sqVXS8fUsS03c4jAz61DdAoekJuBS4Chgd+AkSbuXZXsWOA24vl7l6JY1a9KwPHBcfPHGed3iMLMGU88Wx77A3Ih4KiLWATcAxxUzRMQzEfEo0FbHctQuf3jTttu2T//MZzbO68BhZg2mnoFjPDCvMD0/S6uZpDMlzZI0q6WlpVcK16m8q2rEiI3n5a2RnLuqzKzB1DNwVLp5Uyd3C+xYRFweEVMjYuq4ceN6WKwqXHkl7Lvvxi0OgOXLywuXhr7liJk1iHoGjvnAxML0BGBBHdfXe444Au6/P92rqvzmheWBw11VZtZg6hk4ZgJTJO0saQhwIjCtjuurj2HD2k8vW1Y5nwOHmTWIugWOiGgFzgZmAE8AN0XEbElfknQsgKQ3SpoPvAf4gaTZ9SpPtzlwmJm101zPhUfEdGB6Wdr5hfGZpC6s/mvo0PbTy5bBc8/BQw/BMceU0h04zKxB1DVwbBYqBY6DD4annoING0rpDhxm1iB8y5GuVOqqeuqpNF58wJMDh5k1CAeOrkyc2H76xRdL4y+9VBp34DCzBuHA0ZXrr4eDDipNz5sHg7KPrXhqrgOHmTUIB46ujBsHX/1qafrJJ6GpKY0XWx8OHGbWIBw4qnHggfDgg3DGGSlwNGfnFCxeXMrjwGFmDcJnVVXr9a+HKVPaB4vifbMcOMysQbjFUYv99ms/Pa9wD0ffq8rMGoQDRy0OOQSOK9wZ/tlnS+NucZhZg3DgqNVOO5XGywPHhg3w9NObvkxmZpuQA0etxo4tjf/jH6Xx9evhggtgl13ap5uZbWYcOGpVDBzF1sX69XDXXWncgcPMNmMOHLXaaquN07bdNgWOLbZI0ytXwsKF8KlPbfzEQDOzAc6Bo1Z5cMiHANttl24/Mnx4ml60CKZNg299C26/fdOX0cysjhw4apUHh332KaW9+tXw17+Wgsn//E/pRogOHGa2mXHgqNUhh8DJJ8NVV5XSdt89XVG+alWanjkTvva1NH7bbZu8iGZm9eQrx2s1dChce20av+YaeOEF2HHH9OzxWbM2zj9nTrpQsPwuu2ZmA5QDR0+cckoa5gFj4cL283fdNbVE7rkn5bnvPrj/fpA2aTHNzHqTu6p6w4TC02/f/W4YNSqNv/Wt6U66c+bAd7+burDuvLNvymhm1kscOHrDttuW7pg7alRpfMoUmDQJbrihlPeBBzZ9+czMepEDR28YNAh22CGNjxpVejLgLrukwPHkkym4QPs76hYdeSTsv3/9y2pm1kMOHL1l663TcOTI0p1yd9klXdMB8KMfwc47p/tbffGL7R87CzBjRjr+8YtfwOOPb7pym5nVyIGjt+SPkT3iiFLazjvDJZek03ePPjo9TfAXv0j3tPrYx+DGG+Gyy9of93j3u+HMMyFikxbfzKxaigG2g5o6dWrMqnTaa1+7916YPRs+8hH40Ifgyis33vkfcwz85jfVLW/y5NK9sF58MV2J/oEPlJ53bmZWA0kPRsTU3liW90K95c1vTkEDUrfUunUb5xk3ruvlbL99Gj7zTLo2BOBzn4MPfhCuu65Ximpm1hMOHPUwaBAMHrxx+ooVaXjhhaVbkhQ99FDp2hCAv/0Nli5NrQ2Aiy9Ow9Wre7e8ZmY1qGvgkHSkpDmS5ko6t8L8oZJuzObfL2lyPcvT5/bcMw3f//50/OOXv0zHPIrzTzihNP2a18A228Dzz8PBB8Ou7CuGAAALSUlEQVRjj6VjKMOHw3nnwVe/CqefnrrIzj8fbr453Zk3Ir1WrICrr4YlS9JdevPnpV9wAVxxxSartpltXup2jENSE/A34G3AfGAmcFJEPF7I86/AXhHxUUknAu+KiBMqLjDTb49xVGP9+nSLkuIFg5DuovumN5Wead7SUjp9F+ATn0iBYf/908WEXdlmm3Rm19Zbp2eDDBuWbpWyfHk6bfj551O+889PZ3Jts026w++4cXDAAen4yrJl8OCDqdUzZkwKdKNGwdy5KW3durSsQYPSzR3XrUv1amtL0w88kIKdlJbV0pLOJHvLW1LdlixJt2EZMSKlz5uXnqA4blw6ffmNb0zD1tZ0Q8lFi9It7VetSoFyjz3SulauTPUbOzbV4+mn03U0EWm948enbr81a1KeMWNSvg0b0mezww7pTLhcW1uat2FDWveGDel7i0jva2oq5d2wIR1/Wrs23Xam/PjT6tUwZEhKX7++dH3PoEFpeWvXpvqMHFm5hZqvY9Wq9Dl15/hWW1sqY/E5Mh1ZuzaV13c22Cz15jGOegaOA4ALIuKIbPo8gIi4qJBnRpbnPknNwEJgXHRSqAEdOGpx331pp/blL8N//3fasT/3XNo577pruiPvbrulmy7OmpV2wl/4QrpuZO+90w7g0UdTK+ahh9IOdPvt4dZb00560KDSMRRIO/tKXWDNzaXTi/u7wYOrf/Z7sV5DhpQCRme/Byl9blJ6tbaW8g8ZUnptsUXaWa9ZU9oRr1mThoMHlwJTcbnNze2XLaUglbcgm5pSwN2wIbUkm5vTspqb07zW1lT39etL9cq/47VrU3DNlztoUFrOsGEp4Dc3p9eSJSmwbtiQ1tvcnFq3a9emfPkp5xFpHa2taZsZOTLlyz/DYvDNg17+eeXduPlrw4a07IiUJyKlr1mT1puvf9iw0vZa/JzyOubfXVtbqcW9dm2qw4gRpTtXF/O0tZW+k+bm9AemublU9vw7GzYsvX/9+tIy8++9+H3lr+L32tsuuiidJNMNvRk46nmvqvHAvML0fGC/jvJERKuk5cAYYHExk6QzgTMBJk2aVK/y9i8HHJCGP/5xKW38ePjDH9IV6dttV0o/7rg0/PCHu95Yv/e99GNeuzYFliVL4Kij0g9j2bIUhJ57Lv1o3vrW1NJYtiz941+yJOUbPTr9yFavToGppaX9D2nhwhTInn027RS22ip1s+23HzzxBMyfn+a3tKT5q1algDh8eBpfvz7lec1rUivn4YfTTmvx4jScNCkte8QI2HLLVLb84Vk775zKke+wn346rX/XXWHBglIrYdmyUqtu6dLSDjgfFsfz1sDixe13UIMHpxZSc3Nq1bS2ps9t9epU7lGj0nRbWyrDmjXpNXRoWu7w4aUgk+88izu+NWtSnlGj0jIXLkzrGjWqtOPOn3WfB5J8JyilZeXrfuml0nLzHfCaNaUd8rp1qVWyYEEq05Zblnb8eXmXLy/tHAcPTjvt4cPTZ//yy2m6qak0bGpKy3r55bS+5ua0rmKAywMFpPc1N6d5Q4aksq1endLWri21uPJ65HXJ11keeAcPTu9bvbrUaszz5MM8CK5fnwJgvm4ofZb5d5rXJw+AxXLkr1z5ePF3WT5di5126t77elk9WxzvAY6IiDOy6fcD+0bEvxXyzM7yzM+mn8zyLOlouQ3T4jAz60UD5XTc+UDxXuITgAUd5cm6qkYDL9axTGZm1kP1DBwzgSmSdpY0BDgRmFaWZxpwajZ+PHBXZ8c3zMys79XtGEd2zOJsYAbQBFwREbMlfQmYFRHTgB8D10iaS2ppnFiv8piZWe+o64OcImI6ML0s7fzC+BrgPfUsg5mZ9S5fOW5mZjVx4DAzs5o4cJiZWU0cOMzMrCYD7nkcklqAf3Tz7WMpuyq9AbjOjcF1bgw9qfNOEVHFsx26NuACR09ImtVbV04OFK5zY3CdG0N/qbO7qszMrCYOHGZmVpNGCxyX93UB+oDr3Bhc58bQL+rcUMc4zMys5xqtxWFmZj3kwGFmZjVpiMAh6UhJcyTNlXRuX5ent0i6QtIiSX8ppG0j6XZJf8+GW2fpkvTd7DN4VNLr+67k3SdpoqS7JT0habakj2fpm229JQ2T9ICkR7I6fzFL31nS/Vmdb8weX4Ckodn03Gz+5L4sf09IapL0kKRbs+lGqPMzkh6T9LCkWVlav9q+N/vAIakJuBQ4CtgdOEnS7n1bql5zFXBkWdq5wJ0RMQW4M5uGVP8p2etM4LJNVMbe1gp8KiJeA+wPfCz7Pjfneq8FDouIvYF9gCMl7Q98Dfh2VuelwOlZ/tOBpRHxKuDbWb6B6uPAE4XpRqgzwFsiYp/CNRv9a/uOiM36BRwAzChMnwec19fl6sX6TQb+UpieA+yQje8AzMnGfwCcVCnfQH4BvwLe1ij1BoYDfwb2I11B3Jylv7Kdk56Bc0A23pzlU1+XvRt1nUDaSR4G3Apoc69zVv5ngLFlaf1q+97sWxzAeGBeYXp+lra52i4ingfIhttm6Zvd55B1R7wOuJ/NvN5Zl83DwCLgduBJYFlEtGZZivV6pc7Z/OXAmE1b4l5xCfAZoC2bHsPmX2eAAG6T9KCkM7O0frV91/VBTv2EKqQ14jnIm9XnIGlL4OfAv0fES1Kl6qWsFdIGXL0jYgOwj6StgF8Cr6mULRsO+DpLOgZYFBEPSjo0T66QdbOpc8GbImKBpG2B2yX9tZO8fVLvRmhxzAcmFqYnAAv6qCybwguSdgDIhouy9M3mc5A0mBQ0rouIX2TJm329ASJiGXAP6fjOVpLyP3/Fer1S52z+aNKjmQeSNwHHSnoGuIHUXXUJm3edAYiIBdlwEelPwr70s+27EQLHTGBKdjbGENJzzaf1cZnqaRpwajZ+KukYQJ7+gewsjP2B5XnTdyBRalr8GHgiIr5VmLXZ1lvSuKylgaQtgMNJB4zvBo7PspXXOf8sjgfuiqwDfKCIiPMiYkJETCb9Zu+KiJPZjOsMIGmEpJH5OPDPwF/ob9t3Xx8I2kQHm44G/kbqF/58X5enF+v1U+B5YD3pn8fppH7dO4G/Z8NtsrwinV32JPAYMLWvy9/NOr+Z1BR/FHg4ex29Odcb2At4KKvzX4Dzs/RdgAeAucDPgKFZ+rBsem42f5e+rkMP638ocGsj1Dmr3yPZa3a+v+pv27dvOWJmZjVphK4qMzPrRQ4cZmZWEwcOMzOriQOHmZnVxIHDzMxq4sBhlpG0Ibsjaf7qtTspS5qswl2MzQayRrjliFm1VkfEPn1dCLP+zi0Osy5kz0f4WvZMjAckvSpL30nSndlzEO6UNClL307SL7PnZzwi6cBsUU2Sfpg9U+O27CpwJJ0j6fFsOTf0UTXNqubAYVayRVlX1QmFeS9FxL7A90j3TCIbvzoi9gKuA76bpX8X+H2k52e8nnQFMKRnJlwaEa8FlgHvztLPBV6XLeej9aqcWW/xleNmGUkrI2LLCunPkB6k9FR2g8WFETFG0mLSsw/WZ+nPR8RYSS3AhIhYW1jGZOD2SA/iQdJngcERcaGk3wErgVuAWyJiZZ2ratYjbnGYVSc6GO8oTyVrC+MbKB1jfDvpfkNvAB4s3P3VrF9y4DCrzgmF4X3Z+B9Jd24FOBm4Nxu/EzgLXnkA06iOFippEDAxIu4mPbRoK2CjVo9Zf+J/NmYlW2RP2cv9LiLyU3KHSrqf9GfrpCztHOAKSZ8GWoAPZukfBy6XdDqpZXEW6S7GlTQB10oaTbrT6bcjPXPDrN/yMQ6zLmTHOKZGxOK+LotZf+CuKjMzq4lbHGZmVhO3OMzMrCYOHGZmVhMHDjMzq4kDh5mZ1cSBw8zMavL/ASKb2hdIvRlAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAEWCAYAAAC0Q+rDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm8HFWZ//HPNxtJWASyyJaQUaKCyBKuCCKI24iICwgqiiwiEUUlv1FHUUdxFBFGRdERBwXHBXcQUUeURZhhFJhEtkDAoIJsIUESMCSELM/vj3M6t9Lpvl333qp0cu/3/Xr1q6trPdVdXU8959SiiMDMzKwOI7pdADMzG7ocZMzMrDYOMmZmVhsHGTMzq42DjJmZ1cZBxszMauMgY2YbjJL/k7RbRfNbIOlFVcyrxbw3k7RU0g5thp8s6co+pr9e0jF1lK3b8u94k6TpncYdVJCRdI+k5fmHeFjSNyVtMZh5Vi2X8eUlxvsHSWskfXVDlKtbJG0v6QJJD0n6u6Q7JX1S0ubdLlsnkt4o6XeSlkm6psXwvSTNycPnSNqrMEySzpL0t/w6W5JazOOteXtemrftNYXPSwdR9udIWtVhnM9KWpl/l8Zv8yVJk/uxnEHt2CRNk3SppEckPSbpVklvqXDZRwIPRMQdeZrGOi+VtETSdZJ6Blr+siSNlRSSdmrq/1lJ3wCIiBURsUVEPFh3efqjWMZuiXSB5TnA6Z3GrSKTeU1EbAHMAJ4PfKy/M5A0qoJyDNaxwGLgzZI225AL3lDrL2lb4PfAOGD/iNgSeAWwNfDMAcxvQ/9ujwJfBD7boixjgJ8B3wW2Ab4F/Cz3B5gJvB7YE9gDOAx4Z/N8IuKivGPZAngV8GDjc+5Xt2/l32UCcBQwDZgtadIGWDbA94G7gCnAROAE4JEK538y8J2mft/K3+0k4HrghxUuzypW+N9fArxa0oQ+J4iIAb+Ae4CXFz7/G/CL3P004ALgIeAB4NPAyDzseOB/SZHwUeDTuf9JwDzg78AdwIzcfwfgYmAR8BfgfYVlng78CPh2nu52oCcP+w6wBlgOLAX+uY91+RPwLuBh4MimYc8FrshlfRj4SO4/EvhInvbvwBzSn3MaEMCowjyuAd7Rbv1JO/mrgb+R/tQXAVsXpp+Sf9RFeZyvAJvl6Z9XGG9yXt9JLdbx08BtwIg230F/y30msATYvTD+pLz8yfnzYcDNebzfAXsMZpvL83wHcE1Tv3/M25kK/f4KHJK7fwfMLAw7Ebi+w3IOBu5v0X8KKaA9AvwZOLkw7ADgJuBxYAFwZu6/MH+3S/Nr7xbz/SzwjaZ+o0n/icZ/ZBLwq7wdPJrLsX0e9nlgNfBkXsbnc//zgPtzmW4E9muzvgJWAs/p4zs5ELgh/55/AA7oa9lN047P85/Ybp1JB6sBbFnodzhwa17m/wC7FYYtAF6Uu38AfKww7BDg7jbrMTYvZ6d2v0HzOKT/1n/l7/H3pO3/ysK0rwbm53J+gRQwjykMfycpgD8K/BLYsWk5J5H2JYuBc/r4DdbbTgrDPk7aR/4dmAu8uvDdPw5ML4y7E7CMvJ8p8T1/gLR/XVbo/z/Am/r6H1XWJiNpCnAo6Q8G6UhyFbALsDdpJ/COwiQvIP1BJwNnSDqKFDCOBbYCXgv8TdII4OfALcCOwMuAWZJeWZjXa0kb2NbAZaQdMBHxNtKO5jWRjkTPblP2A0lf+A9IAevYwrAtgSuBy0nBbhfgqjz4n4Cj83pvBbyd9KOVsc76k/7gZ+Zl7ErakZ2eyzAS+AVwLykQ7Aj8ICJW5DIXqyiOJm34i1os8+XAJRGxpmQZO5X7X0mB7+jC8DcC10bEQkkzgAtJf64JwH8Al9WUKT4XuDXylp/dmvs3ht9SGHZLYVhp+bf4L1LQ2oG0I/uIpBfnUb4CfCYitgKmA5fm/gcBq6M3K7qJEiJiJWn7PzD3GgF8DZgK/EPud04e9/3A/5EOCrbInyHtEJ9H+g1+BvxY0ugWywpSAPmPXDXZXJU0La/PR4FtSbUWl0rapo9lF+0KPB4RLTOjvF28jbRDW5r77Qd8lZRRTSAdOF7apdqP80kB4umkA9K3NwZI2o6073g/6UBgEdBTGP5mYBbwmjz9TaSsu+hVpH3lDOAESQcPoIx3AS8kHeSfBfxA0sSIWAb8hHX3FW8FfhkRS0p+z28i1XwUM5d5pNqB9vqKQJ1epExmKSny3ZsLOY70Ja4AxhXGPRr4be4+Hvhr07x+DZzaYhkvaDHuacA3c/fprHs0sRuwvKmML++wHt8ALs3d+5OOtiYXyn1Tm+nuAl7Xov80OmcEf+1Qptc3lpvLtKg4v6bv5z5ydgLMBt7YZp7zKRx1V1FuUuD6c+Hz/wLH5u7zgE+1+M5ePMjtrlUm8y+kwFvsdxFweu5eTeEInRQAgkLm02I5B9OUyQAvBuY39fskcF7uvpG0E57QNM5zgFUd1qvlESpp53Rbm2n2Ax4qfF7n6LnF+CIdCD27zfCJpBqJeaRagNnkrAv4BPD1pvGvJR/Jllj2y4B7WqzzCtI+ZDUp4zugMPybwEebprkXeEHuHmwm81heduP1JC0ymdy9BphWmMcXyPseUnXsNYVhI/O6HJM//xZ4a2H4aNJ+5umF5fQUhl8GzOrPdtJm3DuBVxa23bsLw24DXtuP7/ktLeb/eeCrfZWhikzm9RGxdUTsHBHvjojlwM75S3woN+YtIR3FFhsw72uazxRSqthsZ2CHxnzyvD5C+nEaFhS6lwFjyx7pSBpHqvu+CCAifk/KfhqNne3K1WlYJ+usv6TJkn4g6QFJj5OOciYWlnNvRKzXcBwRNwBPAC+W9BxSpnVZm2X+Ddh+gOVtWW5SFd84SS+QtDOwF/DTPGxn4P1Nv90UUgawDkkfKTSwf20A5VpKyiaLtiJVG7QavhWwNPI/pR92BqY1rdM/Advl4ceR2nz+KOmGpox7oHYkHUEjaUtJF0r6a95OfkPvdtKSpNMk3SXpMVJVzNh200TEIxHxwYjYNa/TH0nZKqR1P6Zp3Xto8Xu2sRjYskX/70TE1qRt80+kbahhZ1KmWFzmJNJ3UoXn5v3X1rkMX2wz3nakAF3c/u8tdO9QHBYRq0nVtw07A18rrMMiUk1PMVts3o/1uw1Q0on5ZI3Gcnah97f+b2CkpP2VTorZnlT12ihfp++5+b8P6fdc0leZ6jqF+T7S0cnEwg+4VUQUqyea/9z30brx+T7gL8UNISK2jIhDS5al007kcNIO56tKp0MuIH2xjSqzduXqa9gT+X18od92TeM0l+vM3G+PSFUtx5A26sZypvYROL+Vx38b8JOIeLLNeFcCh+cqyFb6Xe5IVW8/ImV8byG1yTV27PcBZzT9duMj4vvNC46Iz0RvVdLJbcrXl9uBPaR1zhjbI/dvDC+m9XsWhvXHfcCdLbbHw/N6zIuIN5EOqM4FLsknH/Q3mAFrG1kPI9V9A3yYtGN6ft5O/pHe7YTm5Uh6BfBe0na+Namaa3nTNC1FxELS0fo0pbMP7yMdQRfXffOIOKfVsluYB2wpqV2AW0iqWj2zMM59wMdbbEOXtJjFE/S97Q7GAtL6TSn0m1rofqg4LP/HmnfQxzetx7iImFNVASU9C/gyKavaNgfNu8m/dT6g+ja9+4ofRKqObZSv0/fc6vfdlXWroddTS5CJiIdIR1ifl7SVpBGSnlmot27lG8AHJO2jZJd8ZHwj8LikD0kaJ2mkpN0lPb9kcR4GntHH8ONI7QbPIx1B7UVqvN1L0vNIbSHbSZqldN78lpJeUCjzpyRNz2XeQ9KESO0hD5CO+kZKejudz97aklz1KGlH4IOFYTeSNuLPStpc6fTLAwrDv0PaiRxD2oja+QIpoH4rf7dI2lHSFyTtMcByA3yPVF/71tzd8HXg5JzlKJf91UrtXP2WyzQWGAWMyN9Do23hGlJ1y/vy7/Se3P/q/P5t4J/y+u5Aqjv/zwEU47pclll5+aPy7z4j9z82bwOrSVUxQapmWUg6ipzads7rrutoSc8lBfAtSQGL3L2MtJ1MZP2zOZu39y1J1TKLgDGkdrSxfSz3c5J2y9/100hng82NiCdIBzNHSXpZHj4udzd25n3+13ItxzWk9ql249xKOuJutOmcD7xXUk/ehraQ9FpJ41tMfjNwmKSt83/ove2W01/5wO3nwCfzeu9B2t4bLgOeL+mwvE1+kBTQG74GfEzSswEkbSPpDYMo0si8/TVeY0iZzxrSbz1C0smkTKbo26R206NZd1/Rn++ZvA6bk/abV7UbB6j27LKmYU+j96yWx0gNXW/Ow44HrmsxzcmkOvulpDMjGnXBO5BOrVxASrmvbyyX1Cbz3cI8plFoVwBeR6r+WgJ8oGl5O5JS1ue1KMt/AZ/L3bvnL3JxLsOHo7fe9WP0ns3xf/SeifKq3H8Jqd7yWtZt27iuaXnPJZ2dtpT0Z3k/hfYA0lHTpfSefXZu0/RX5t+jbRtD4bu8MK/H30l1tp8Axg+k3IX53k2q0hnT1P+Q/L0sIQXKH1M4c6if29vx+bctvv6zMHzv/B0uJ535tHdhmICzcxkfzd2dvquDaX922Y9IO9XFpHaog/KwH+Xf5++kOu9DC9OdRdoBLAH2ajHfz5ICwt9JR+V/JB2Zbt+0HVyXt5M7gXdTaOsh17vncp1Nqrb+DunMogdI7Ttr2zFalOH8PP3SXNafse4ZSQfk5S8mBc7LgB1aLbvN/N8A/LRpnZvPqHtxLu+2+fNr8+/6GPAgqe1lXB5WbJPZnFS19zhpf/MBqj27bDvSCUDtzi57TV7/dmeXnUjKnh8nVbV9rV1ZaGpfalHG5v/B3XnY5/L3vyhvb+u1k+Xf764W8y31PRfGfxvwvU7/W+WRbRMn6ULSNR39vk7JbEPJ1Zk3kKqO7uh2eYYjSd8D7oiITw9iHiIFpDdHxB/7HNdBZtOndGrpzaQj9790tzRmtrGStAspy981Ih7oNH4VfO+yTZykT5GqFv/NAcbM2pF0Nqka8V83VIABZzJmZlYjZzJmZlabjeHGlB1NnDgxpk2b1u1imJltUubMmfNIRGyom6u2tEkEmWnTpjF79uxuF8PMbJMi6d7OY9XL1WVmZlYbBxkzM6uNg4yZmdXGQcbMzGrjIGNmZrWpNchIOlXSXEm3S5qV+/1Q0s35dY+km+ssg5mZdU9tpzBL2p30zOp9gaeAyyX9MtKzNhrjfJ50x08zMxuC6sxkdgWuj4hlkZ7oeC3pmSfA2rt4vpF0C/+N0623wu9+B6tXw4UXpvcI+OY3YcWKbpfOzGyjV+fFmHOBMyRNID3f41DS88IbDgQejoj5rSaWNJP0hDemTi31nKfq7ZkfpPiVr8B73gNPPAE77ghvfzvMnw+f+Ux3ymVmtomoLZOJiHmkh+ZcQXrQzy2kB4Q1HE0fWUxEnB8RPRHRM2lSV++KAI880vu+eHHqfvjh7pXHzGwTUWvDf0RcEBEzIuIg0tMI58Pa55YfAfywzuWbmVl31XrvMkmTI2Jhfq75EcD+edDLgTsj4v46l29mZt1V9w0yL85tMiuBUyIi1zXxZjbmBn8zM6tErUEmIg5s0//4OpdbuVYPdvPD3szMOvIV/2ZmVhsHGTMzq42DjJmZ1cZBpr9Wr+52CczMNhkOMv21cmW3S2BmtslwkOmPCAcZM7N+cJApY1W+G87q1fDUU90ti5nZJsRBpoxGYFm50pmMmVk/OMiU0QgsTz3VG3B8MaaZWUcOMmW0ymSc0ZiZdeQgU0arTMZBxsysIweZMlplMj4BwMysIweZMhoBxZmMmVm/OMiU8eST6d2ZjJlZvzjIlLF8eXp3JmNm1i8OMmU0gowzGTOzfnGQKaNRXVbMZBxkzMw6cpApo1WbjKvLzMw6cpApw5mMmdmA1BpkJJ0qaa6k2yXNKvR/r6S7cv+z6yxDJVq1yTiTMTPraFRdM5a0O3ASsC/wFHC5pF8COwGvA/aIiBWSJtdVhso4kzEzG5DaggywK3B9RCwDkHQtcDjQA3w2IlYARMTCGstQjYW5iEuW9GYwS5fClVfC9OkwciSMHw+33QYvfCGMHt29spqZbUTqrC6bCxwkaYKk8cChwBTgWcCBkm6QdK2k57eaWNJMSbMlzV60aFGNxWyj1V2W77sP7rwzdT/6KLziFTBtGkyZAieeCAcfDN/97oYspZnZRq22IBMR84CzgCuAy4FbgFWk7GkbYD/gg8CPJKnF9OdHRE9E9EyaNKmuYra3Zs26nx98EP7nf9JrwQK47jp4fiE+LliQ3pcs2XBlNDPbyNVZXUZEXABcACDpM8D9pGq0SyIigBslrQEmAl1IV/qwevW6n7ffPr0anv50KAY/X6RpZraeWoOMpMkRsVDSVOAIYH9gDfBS4BpJzwLGAI/UWY4Bac5kWhkzprfbQcbMbD21BhngYkkTgJXAKRGxWNKFwIWS5pLOOjsuZzUbl+ZMppViA3/xgk0zMwPqry47sEW/p4Bj6lxuJcoEmWIms2xZencmY2a2lq/4b6e/QeaJJ9K7Mxkzs7UcZNop0yZTrC5rBBlnMmZmaznItNPfTMbPmTEzW4+DTDv9bfhvcCZjZraWg0w7/c1kGpzJmJmt5SDTTn/bZBqcyZiZreUg044zGTOzQXOQaWegQcaZjJnZWg4y7RSDzIg2X5Ory8zM+uQg006xTWbkyNbjuLrMzKxPDjLtFDOZdkHGmYyZWZ8cZNopU13mTMbMrE8OMu04kzEzGzQHmXbcJmNmNmgOMu04kzEzGzQHmXYG0iYzapQzGTOzAgeZdgaSyWy+uTMZM7MCB5l2ygQZad3Pm2/uTMbMrMBBpp0yDf/NnMmYma3DQaadMplMxLqfncmYma2j1iAj6VRJcyXdLmlW7ne6pAck3Zxfh9ZZhgEr0/DfzJmMmdk6RtU1Y0m7AycB+wJPAZdL+mUefE5EfK6uZQ/KypWpqmzFit5+/akuW706BZpRo8oHp0ZAa17OypXrnlywalUqW6vrc9asSa9Ro1KGtXp16i6KSPMYPXrd7lWr2t91evToNKxYfdiqX9GIEWnZDrhmgzd6dPl9ycYoImp5AUcB3yh8/hfgn4HTgQ/0Z1777LNPbBD//d8Ro0ZFpF1w7+uww1qPf8MN64535JHrTwsRs2al9wcfXH8eq1ZFTJ4cscUWEU88kfodcEDvtJ/4RHo/7bSIsWMjpIjvfa93+gULescdMSLivvsiDj44fb7zzojddot45SvTuMceu37ZzjknzbdVuRuvMWPW7zdiRPvx+xrml19+9e/1q18NeJcGzI6oZx9f9tUxk5F0MXAh8KuIKPG4yLXmAmdImgAsBw4FZgN/A94j6dj8+f0RsbjFcmcCMwGmTp3aj8UOwl/+ko7q3/9+mDABtt4att8eXvKS1uPvuy/8+MfpKOPhh+Hxx+EnP1l/vC9+Mb3ffHOaX9GTT8LChal78WIYPx7+9397h195ZXo/88zefn/8Y2/3rbf2dq9ZA/ffD9dckz7fey/ccUd6QW//oiuvTGWYOROmTVt/2NVXp4zkve9NZb/4YpgzJy3rYx9L5S1asQI++cnUvffecNRR6y/TzMqbPr3bJRiUMtVl5wEnAOdK+jHwnxFxZ6eJImKepLOAK4ClwC3Aqjy/TwGR3z8PvL3F9OcD5wP09PREqbUZrMiLefe74RnPKDfNkUf2dp97bv+XWaxSalW9VLZf2fk1e+KJ9H7CCbDffusOW706BRmAk0+G3XZLgWvOnNTvgx+ErbZad5rly3uDzIwZcNppnctgZkNWx4q+iLgyIt4KzADuAa6Q9DtJJ0hqcV+Vdaa9ICJmRMRBwKPA/Ih4OCJW56zo66Q2m41DI8g0X/9SVqvbzHRSPBut1ZlpZfuVmV+r9WoEmVZlL7b9NIYX+7VqGyrOZyDfh5kNKaVak3KV1/HAO4CbgC+Rgs4VHaabnN+nAkcA35dUrC86nFSttnEYbJBptdPtpFPmUTwBoa/xys6vWSPIdAoYjeGdgsjIkb3f30C+DzMbUsq0yVwCPAf4DvCaiHgoD/qhpNkdJr84B6iVwCkRsVjSdyTtRaouuwd454BLX7WNMZNZtqzvafo7v2Z9BZlWWUvjfcSI1mfdSWmcFSscZMysVJvMVyLi6lYDIqKnrwkj4sAW/d5WsmwbXiPIDPR0wToymUYQaDdNf+YXLZq2+qoua5W1NL+3Mnp0CjKuLjMb9srsTXeVtHXjg6RtJL27xjJ1T92ZTKvrSjplHs1BZuTIdcdrvr6lGFia59fqWpjBZDLttGq/MbNhqUyQOSkiljQ+5NONT6qvSF3UCAJ1tcm0CiKdMpnly9f93HxXgeZpikGpeVhf8+9vJtOXxvfnTMZs2CsTZEZIvXtdSSOBoXmIWncm0+lMsTJtKM33R2uephhkmof1Nf/+ZjJlOJMxG/bKtMn8GviRpK+RGutPBi6vtVTdUvfZZZ2ueSlzNlh/MpnmLKiv+XfKZBqN/P3JTpzJmA17ZYLMh0hngL0LEPAb4Bt1Fqprhlom09yeM5hMpq9+/ZmnmQ0rHYNMvmjyvPwa2jaGTKbVGWBF48eXz2SK3RF9B5lOF2P21a8/8zSzYaXMdTLTgTOB3YCxjf4RUfK+K5uQjSGT6SsQjBoFm202sExm1aq+59tqnTtVoXXiTMZs2CvT8P9NUhazCngJ8G3ShZlDz8aQyXSq0ho9emCZTH/bYxrLK9Ovv/M1s2GjTJAZFxFXAYqIeyPidOCl9RarS+q+GLNMJtMpGIwZM7BMptWdAxralduZjJkNUpmG/ycljQDmS3oP8AAwud5idUnd1WVlMpm+gkyZTGbp0s7dzarOZBrfozMZs2GvzCH7LGA88D5gH+AY4Lg6C9U1G3t1WSOTGUh1Wavb0zQ4kzGzmvSZyeQLL98YER8kPRPmhA1Sqm4Z7BX/VTT8l8lkBlJd1leQcZuMmdWkz0wmIlYD+xSv+B/SNvZMZsyYDZvJ+DoZMxukMm0yNwE/y0/FXLuniohLaitVt2wMpzB3avjfkJnMYKvLnMmYDXtlgsy2wN9Y94yyABxkmg0mk5EGn8lIvcGk2A3OZMysK8pc8T+022GKBhtkRnX4OvvKZMaPH3wms/nmvWeRFbvBmYyZdUWZK/6/Scpc1hERb6+lRN002CDTabp2mczIkTB2bLlTmPvKZPobZEaMSCc7OJMxs5qUqS77RaF7LHA48GA9xemywQaZTtplMo3g0em2MsWLMSNSOYvjb7EFPPzw+t3QOsiMGQNPPln9KcyN789BxmzYK1NddnHxs6TvA1fWVqJuGuwV/520y2Qa1WBlL8aMSE+5HDVq/UymVTe0DjKjR6cg0y5wtPoefAqzmfXDQPam04GpZUaUdKqkuZJulzSradgHJIWkiQMoQz029kymMV5xXs1tMq26oX0mU3wvozFume/ImYzZsFemTebvrNsms4D0jJlO0+1OekzzvsBTwOWSfhkR8yVNAV4B/HVApa7LYC/G7GSwmcyoUb3ZwVNPwbhxg89kGvMtqz/ZSeNBZ2Y2bJWpLttygPPeFbg+IpYBSLqW1J5zNnAO8M/AzwY473rUncn8/vew777r9vvzn1NAGDMGrrwS/vCH9tNLvdnBS16SgsOdd/YOHzu2t3v8+HWnvaTFGeeN8fuzvo2A1FeW4gzGzLIymczhwNUR8Vj+vDVwcERc2mHSucAZkiYAy4FDgdmSXgs8EBG39HUjAUkzgZkAU6eWqp0bvCqCzHnnwTOeAdddlx5/vHJlCiQ77AB/bZG4TZwIL31pCjQ//3nqt99+8LSnwYknwplnwrbbwoIFcPzx8Kxnwete15vBvOhF8MxnwnbbwbOfndpqnvlMOOSQtOxttkkZ2mOPpWW9613wk5+ktphjj4UzzoAT+jhL/ZxzYO+9ez+PHQsf/zgceWT7aa66Ci66CCZM6PfXZ2ZDi6LDkxgl3RwRezX1uyki9m43TWG8E4FTSPc9u4MUbF4I/GNEPCbpHqAnIh7paz49PT0xe/bsTosbvH/9V/jEJ9IDvlzVY2abOElzIqKnm2Uo0/DfapxSlfgRcUFEzIiIg4BHgXuAfwBuyQFmJ+APkrYrV9ya1V1dZmY2zJQJMrMlfUHSMyU9Q9I5wJwyM5c0Ob9PBY4Avh0RkyNiWkRMA+4HZkTEggGWv1oOMmZmlSoTZN5LOjvsh8CPSFVep5Sc/8WS7gB+DpwSEYsHVMoNxUHGzKxSZc4uewL48EBmHhEHdhg+bSDzrU3jKnozM6tEx0xG0hX5jLLG520k/breYnWJg4yZWaXKVJdNjIgljQ+5ymtyfUXqIgcZM7NKlQkya3LDPQCSdqbFXZmHhDVrHGTMzCpU5lTkjwLX5Sv2AQ4C3llfkbrImYyZWaXKNPxfLmkGsB8g4P91unhyk+UgY2ZWqVJ3YY6IRyLiF6Sr9k+WNLfeYnWJg4yZWaXKnF22vaRZkm4EbgdGAkfXXrJucJAxM6tU2yAj6SRJVwPXAhOBdwAPRcQnI+K2DVXADcpBxsysUn21yfw78HvgLRExG0DS0DyrrMFBxsysUn0FmR2Ao4AvSHo66ZYyQ/t5uhH1PXrZzGwYartHzY395+U7KL8MeAxYKGmepM9ssBJuSM5kzMwqVfbssvsj4nMRsQ/wemBFvcXqEgcZM7NK9ePh7klE3AV8soaydJ+v+Dczq5QbIIqcyZiZVcpBpshBxsysUmUuxryqTL8hwUHGzKxSbdtkJI0FxgMTJW1Dum8ZwFak05uHHgcZM7NK9dXw/05gFimgzKE3yDxOulBz6HGQMTOrVNsgExFfAr4k6b0R8eUNWKbucZAxM6tUmVv9f1nSC4FpxfEj4tudppV0KnASKQv6ekR8UdKngNcBa4CFwPER8eDAil8xX/FvZlapMg3/3wE+B7wIeH5+9ZSYbndSgNkX2BM4TNJ04N8iYo+I2Av4BfDxgRe/Ys5kzMwqVeZizB5gt4jo780xdwWuj4hlAPnJmodHxNmFcTZnY3qUsy/GNDOrVJm6obnAdgOY91zgIEkTJI0HDgWmAEg6Q9J9wFtpk8lImilptqTZixYtGsDiB8D/6eC+AAANm0lEQVSZjJlZpcoEmYnAHZJ+LemyxqvTRBExDzgLuAK4HLgFWJWHfTQipgAXAe9pM/35EdETET2TJk0quTqD5CBjZlapMtVlpw905hFxAXABQL5z8/1No3wP+CXwiYEuo1IOMmZmlSpzdtm1knYGpkfElbnqa2SZmUuaHBELJU0FjgD2lzQ9IubnUV4L3DnQwlfOQcbMrFIdg4ykk4CZwLbAM4Edga+RnjHTycWSJgArgVMiYrGkb0h6NukU5nuBkwda+Mo5yJiZVapMddkppNOQbwCIiPmSJpeZeUQc2KLfG/pVwg3JQcbMrFJlGv5XRMRTjQ+SRrExnXZcJV+MaWZWqTJ71GslfQQYJ+kVwI+Bn9dbrC5xJmNmVqkyQebDwCLgNtJNM/8L+FidheoaBxkzs0qVaZMZB1wYEV8HkDQy91tWZ8G6wlf8m5lVqkwmcxUpqDSMA66spzhd5kzGzKxSZYLM2IhY2viQu8fXV6QucpAxM6tUmSDzhKQZjQ+S9gGW11ekLnKQMTOrVJk2mVOBH0tqPPNle+BN9RWpixxkzMwq1WeQkTQCGAM8B3g26eFjd0bEyg1Qtg3PQcbMrFJ9BpmIWCPp8xGxP+nW/UObg4yZWaXKtMn8RtIbpGGw9/UV/2ZmlSrTJvNPpCdYrpa0nFRlFhGxVa0l6wZnMmZmlSpzq/8tN0RBNgoOMmZmlepYN6TkGEn/kj9PkbRv/UXrAl/xb2ZWqTINEF8F9gfekj8vBf69thJ1kzMZM7NKlWmTeUFEzJB0E0B+8NiYmsvVHQ4yZmaVKpPJrMw3xQwASZNIT7UcehxkzMwqVSbInAv8FJgs6QzgOuAztZaqWxxkzMwqVebssoskzQFeRjp9+fURMa/2knWDg4yZWaXaBhlJY4GTgV1IDyz7j4hY1Z+ZSzoVOIkUnL4eEV+U9G/Aa4CngD8BJ0TEkgGWv1oOMmZmleqruuxbQA8pwLwK+Fx/Zixpd1KA2RfYEzhM0nTgCmD3iNgD+CNw2gDKXQ9f8W9mVqm+qst2i4jnAUi6ALixn/PeFbg+IpbleVwLHB4RZxfGuR44sp/zrY8zGTOzSvV12L72Tsv9rSbL5gIHSZogaTxwKDClaZy3A79qNbGkmZJmS5q9aNGiASx+AHwxpplZpfrKZPaU9HjuFjAufy5177KImCfpLFL12FLgFmBtsJL00fz5ojbTnw+cD9DT0xPlVmeQnMmYmVWqbZCJiJGDnXlEXABcACDpM8D9ufs44DDgZRGxYQJIGQ4yZmaVKnPF/4BJmhwRCyVNBY4A9pd0CPAh4MWN9pqNhoOMmVmlag0ywMWSJpDad07Jt6T5CrAZcEV+RM31EXFyzeUox0HGzKxStQaZiDiwRb9d6lzmoDjImJlVyheFFDnImJlVykGmyBdjmplVynvUImcyZmaVcpApcpAxM6uUg0yRr/g3M6uUg0yRMxkzs0o5yBQ5yJiZVcpBpshBxsysUg4yRQ4yZmaVcpApcpAxM6uUg0yRg4yZWaUcZIp8xb+ZWaW8Ry1yJmNmVikHmSIHGTOzSjnIFPmKfzOzSjnIFDmTMTOrlINMkYOMmVmlHGSKHGTMzCrlIFPkIGNmVikHmSIHGTOzStUaZCSdKmmupNslzcr9jsqf10jqqXP5/eYgY2ZWqdqCjKTdgZOAfYE9gcMkTQfmAkcA/13XsgfMV/ybmVWqzj3qrsD1EbEsIlYB1wKHR8S8iLirxuUOnDMZM7NK1Rlk5gIHSZogaTxwKDCl7MSSZkqaLWn2okWLaivkOnwxpplZpWoLMhExDzgLuAK4HLgFWNWP6c+PiJ6I6Jk0aVJNpVxvoQ4yZmYVqrUBIiIuiIgZEXEQ8Cgwv87lDZqDjJlZpUbVOXNJkyNioaSppMb+/etc3qA5yJiZVarWIANcLGkCsBI4JSIWSzoc+DIwCfilpJsj4pU1l6McBxkzs0rVGmQi4sAW/X4K/LTO5Q6Yg4yZWaV8UUiRg4yZWaUcZIp8MaaZWaW8Ry1yJmNmVikHmSIHGTOzSjnIFPmKfzOzSjnIFDmTMTOrlINMkYOMmVmlHGSKHGTMzCrlIFPkIGNmVikHmSIHGTOzSjnIFDnImJlVykGmyFf8m5lVynvUImcyZmaVcpApcpAxM6uUg0yRr/g3M6uUg0yRMxkzs0o5yBQ5yJiZVcpBpshBxsysUg4yRQ4yZmaVqjXISDpV0lxJt0ualfttK+kKSfPz+zZ1lqFfHGTMzCpVW5CRtDtwErAvsCdwmKTpwIeBqyJiOnBV/rxxcJAxM6vUqBrnvStwfUQsA5B0LXA48Drg4DzOt4BrgA/VUoJPfxq+//3y4z/5pIOMmVmF6gwyc4EzJE0AlgOHArOBp0fEQwAR8ZCkya0mljQTmAkwderUgZVgu+1gt93Kj7/77nDUUQNblpmZrUcRUd/MpROBU4ClwB2kYHNCRGxdGGdxRPTZLtPT0xOzZ8+urZxmZkORpDkR0dPNMtTa8B8RF0TEjIg4CHgUmA88LGl7gPy+sM4ymJlZ99R9dtnk/D4VOAL4PnAZcFwe5TjgZ3WWwczMuqfONhmAi3ObzErglIhYLOmzwI9yVdpfATeCmJkNUbUGmYg4sEW/vwEvq3O5Zma2cfAV/2ZmVhsHGTMzq42DjJmZ1cZBxszMalPrxZhVkbQIuHeAk08EHqmwOJsCr/Pw4HUeHgazzjtHxKQqC9Nfm0SQGQxJs7t9xeuG5nUeHrzOw8Omvs6uLjMzs9o4yJiZWW2GQ5A5v9sF6AKv8/DgdR4eNul1HvJtMmZm1j3DIZMxM7MucZAxM7PaDNkgI+kQSXdJulvSh7tdnqpIulDSQklzC/22lXSFpPn5fZvcX5LOzd/BrZJmdK/kAydpiqTfSpon6XZJp+b+Q3a9JY2VdKOkW/I6fzL3/wdJN+R1/qGkMbn/Zvnz3Xn4tG6WfzAkjZR0k6Rf5M/DYZ3vkXSbpJslzc79hsT2PSSDjKSRwL8DrwJ2A46W1I/nMG/U/hM4pKnfh4GrImI6cFX+DGn9p+fXTOC8DVTGqq0C3h8RuwL7Aafk33Mor/cK4KURsSewF3CIpP2As4Bz8jovBk7M458ILI6IXYBz8nibqlOBeYXPw2GdAV4SEXsVrokZGtt3RAy5F7A/8OvC59OA07pdrgrXbxowt/D5LmD73L09cFfu/g/g6Fbjbcov0oPuXjFc1hsYD/wBeAHpyu9Ruf/a7Rz4NbB/7h6Vx1O3yz6Add2JtEN9KfALQEN9nXP57wEmNvUbEtv3kMxkgB2B+wqf78/9hqqnR8RDAPl9cu4/5L6HXCWyN3ADQ3y9c7XRzaRHlF8B/AlYEhGr8ijF9Vq7znn4Y8CEDVviSnwR+GdgTf48gaG/zgAB/EbSHEkzc78hsX3X/WTMblGLfsPxXO0h9T1I2gK4GJgVEY9LrVYvjdqi3ya33hGxGthL0tbAT4FdW42W3zf5dZZ0GLAwIuZIOrjRu8WoQ2adCw6IiAfzI+uvkHRnH+NuUus9VDOZ+4Ephc87AQ92qSwbwsOStgfI7wtz/yHzPUgaTQowF0XEJbn3kF9vgIhYAlxDao/aWlLj4LC4XmvXOQ9/GvDohi3poB0AvFbSPcAPSFVmX2RorzMAEfFgfl9IOqDYlyGyfQ/VIPN/wPR8VsoY4M3AZV0uU50uA47L3ceR2iwa/Y/NZ6PsBzzWSL83JUopywXAvIj4QmHQkF1vSZNyBoOkccDLSY3hvwWOzKM1r3PjuzgSuDpyhf2mIiJOi4idImIa6T97dUS8lSG8zgCSNpe0ZaMb+EdgLkNl++52o1BdL+BQ4I+keuyPdrs8Fa7X94GHgJWkI5oTSfXQVwHz8/u2eVyRzrL7E3Ab0NPt8g9wnV9Eqg64Fbg5vw4dyusN7AHclNd5LvDx3P8ZwI3A3cCPgc1y/7H58915+DO6vQ6DXP+DgV8Mh3XO63dLft3e2F8Nle3bt5UxM7PaDNXqMjMz2wg4yJiZWW0cZMzMrDYOMmZmVhsHGTMzq42DjA1rklbnO982XpXdsVvSNBXulm02HA3V28qYlbU8IvbqdiHMhipnMmYt5Od7nJWf6XKjpF1y/50lXZWf43GVpKm5/9Ml/TQ//+UWSS/Msxop6ev5mTC/yVfvI+l9ku7I8/lBl1bTrHYOMjbcjWuqLntTYdjjEbEv8BXSPbTI3d+OiD2Ai4Bzc/9zgWsjPf9lBunKbUjP/Pj3iHgusAR4Q+7/YWDvPJ+T61o5s27zFf82rElaGhFbtOh/D+mhYX/ON+dcEBETJD1CenbHytz/oYiYKGkRsFNErCjMYxpwRaSHTiHpQ8DoiPi0pMuBpcClwKURsbTmVTXrCmcyZu1Fm+5247SyotC9mt520FeT7j+1DzCncJdhsyHFQcasvTcV3n+fu39HukMwwFuB63L3VcC7YO3DxrZqN1NJI4ApEfFb0gO6tgbWy6bMhgIfPdlwNy4/fbLh8ohonMa8maQbSAdjR+d+7wMulPRBYBFwQu5/KnC+pBNJGcu7SHfLbmUk8F1JTyPdUfecSM+MMRty3CZj1kJuk+mJiEe6XRazTZmry8zMrDbOZMzMrDbOZMzMrDYOMmZmVhsHGTMzq42DjJmZ1cZBxszMavP/AWl2vuUvlU5hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# gradient descent\n",
    "detailed_logger = False\n",
    "main_logger = True\n",
    "main_logger_output_epochs = 100\n",
    "L2 = False\n",
    "Dropout = False\n",
    "momentum = False\n",
    "hidden_layer_relu = True\n",
    "hidden_layer_tanh = False\n",
    "hidden_layer_sigmoid = False\n",
    "\n",
    "# hyber-parameters\n",
    "alpha = .1;\n",
    "epsilon = .85\n",
    "keep_prob = .9\n",
    "number_of_epochs = 500\n",
    "batch_size = 50\n",
    "momentum_coef = .9\n",
    "\n",
    "# copy initalization\n",
    "W = Weights.copy()\n",
    "B = Bias.copy()\n",
    "\n",
    "# data arrays\n",
    "cost_array = []\n",
    "accuracy_array = []\n",
    "interation_array = []\n",
    "\n",
    "# rename\n",
    "X_train = np.float64(training_images).copy()\n",
    "Y_train = np.float64(training_labels).copy()\n",
    "\n",
    "X_test = np.float64(testing_images).copy()\n",
    "Y_test = np.float64(testing_labels).copy()\n",
    "\n",
    "#m = size\n",
    "m = number_of_training_images\n",
    "\n",
    "def model(W, B, A):\n",
    "    return np.dot(W, A) + B\n",
    "\n",
    "def activation_relu(Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    return np.where(Z > 0, Z, 0)\n",
    "\n",
    "def activation_tanh(Z):\n",
    "    return np.tanh(Z)\n",
    "\n",
    "def activation_sigmoid(Z):\n",
    "     return 1/(1 + np.exp(-Z))\n",
    "\n",
    "def loss(A, Y):\n",
    "    epsilon = 1e-20\n",
    "    return np.where((Y == 1), np.multiply(-Y, np.log(A + epsilon)), -np.multiply((1 - Y), np.log(1 - A + epsilon)))\n",
    "    #return np.multiply(-Y, np.log(A)) - np.multiply((1 - Y), np.log(1 - A)) \n",
    "    \n",
    "def cost(L):\n",
    "    return np.multiply(1/L.shape[1], np.sum(L))\n",
    "\n",
    "def cost_L2(L, W, epsilon):\n",
    "    L2 = np.multiply(epsilon/(2*W.shape[1]), np.multiply(W[len(W)-3], W[len(W)-3]).sum() + np.multiply(W[len(W)-2], W[len(W)-2]).sum() + np.multiply(W[len(W)-1], W[len(W)-1]).sum())\n",
    "    J = cost(L)\n",
    "    return L2 + J\n",
    "\n",
    "def prediction(A):\n",
    "    return np.where(A >= 0.5, 1, 0)\n",
    "    \n",
    "def accuracy(prediction, Y):\n",
    "    return 100 - np.multiply(100/Y.shape[0], np.sum(np.absolute(Y - prediction)))  \n",
    "    \n",
    "def forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "            print('Forward Propagation Training Data Complete')\n",
    "    return A_layers, Z_layers, D\n",
    "\n",
    "def forward_propagation(W, B, A, layer):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "        A = forward_propagation(W, B, A, layer)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        A = activation_sigmoid(Z)  \n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "            print('Forward Propagation Testing Data Complete')\n",
    "    return A\n",
    "\n",
    "def dZ(dZ, W, Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    W = np.where(~np.isnan(W), W, 0)\n",
    "    dZ = np.where(~np.isnan(dZ), dZ, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    W = np.where(~np.isinf(W), W, 0)\n",
    "    dZ = np.where(~np.isinf(dZ), dZ, 0)\n",
    "    if(hidden_layer_relu == True):\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.where(Z > 0, 1, 0))\n",
    "    elif(hidden_layer_tanh == True):\n",
    "        A = activation_tanh(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), 1- np.multiply(A, A))\n",
    "    elif(hidden_layer_sigmoid == True): \n",
    "        A = activation_sigmoid(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.multiply(A, (1-A)))\n",
    "\n",
    "def dW(dZ, A):\n",
    "    return np.multiply(1/dZ.shape[1], np.dot(dZ, np.transpose(A)))\n",
    "\n",
    "def dW_L2(dZ, A, W, epsilon):\n",
    "    return np.multiply(epsilon/Z.shape[1], W) + dW(dZ, A)\n",
    "\n",
    "def dB(dZ):\n",
    "    return np.multiply(1/dZ.shape[1], np.sum(dZ))\n",
    "\n",
    "def backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB):\n",
    "    if(layer >= 0):\n",
    "        if(layer == len(W) - 1):\n",
    "            _dZ = A_layers[layer+1] - Y\n",
    "        elif(layer >= 0):\n",
    "            _dZ = dZ(_dZ, W[layer+1], Z_layers[layer])\n",
    "        if(Dropout == True):\n",
    "            _dZ = np.multiply(_dZ, D[layer])\n",
    "        if(L2 == True):\n",
    "            _dW = dW_L2(_dZ, A_layers[layer], W[layer], epsilon)\n",
    "        else:\n",
    "            _dW = dW(_dZ, A_layers[layer])\n",
    "        _dB = dB(_dZ)\n",
    "        if(momentum == True):\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(alpha, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(alpha, _dB)\n",
    "            W[layer] = W[layer] - V_dW[layer]\n",
    "            B[layer] = B[layer] - V_dB[layer]           \n",
    "        else:\n",
    "            W[layer] = W[layer] - np.multiply(alpha, _dW)\n",
    "            B[layer] = B[layer] - np.multiply(alpha, _dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Layer: ' + str(layer))\n",
    "        layer = layer - 1\n",
    "        W, B = backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Propagation Complete')\n",
    "    return W, B\n",
    "    \n",
    "\n",
    "def shuffle(X, Y, number_of_training_images):\n",
    "    random_array = np.random.permutation(np.arange(number_of_training_images))\n",
    "    return X[:, random_array], Y[random_array]\n",
    "    \n",
    "start_time = time.time()   \n",
    "# main loop\n",
    "for epoch in range(1, number_of_epochs):\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Main Loop Epoch: ' + str(epoch))\n",
    "    \n",
    "    # shuffle data\n",
    "    X, Y = shuffle(X_train.copy(), Y_train.copy(), number_of_training_images)\n",
    "    number_of_batches = int(np.floor(number_of_training_images/batch_size))\n",
    "    split_index = number_of_batches*batch_size\n",
    "\n",
    "    # parse into minibatches\n",
    "    X_minibatches = np.split(X[:, 0:split_index], number_of_batches, axis=1)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        X_left_over_portion  = X[:, split_index:number_of_training_images]\n",
    "        X_minibatches.append(X_left_over_portion)\n",
    "    \n",
    "    Y_minibatches = np.split(Y[0:split_index], number_of_batches, axis=0)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        Y_left_over_portion  = Y[split_index:number_of_training_images]\n",
    "        Y_minibatches.append(Y_left_over_portion)\n",
    "    \n",
    "    number_of_minibatches = len(Y_minibatches)\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Number Of Minibatches: ' + str(number_of_minibatches))\n",
    "\n",
    "    for index in range(0, number_of_minibatches-1):\n",
    "        X_minibatch = X_minibatches[index]\n",
    "        Y_minibatch = Y_minibatches[index]\n",
    "\n",
    "        if(hidden_layer_relu + hidden_layer_tanh + hidden_layer_sigmoid != 1):\n",
    "            print(\"ERROR! Please Select Only 1 Hidden Layer Activation Function\")\n",
    "            break\n",
    "\n",
    "        # forward propogation training data set\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, X_minibatch, [X_minibatch], [], 0, [], keep_prob)\n",
    "        L = loss(A_layers[len(A_layers) - 1], Y_minibatch)\n",
    "        if(L2 == True):\n",
    "            C = cost_L2(L, W, epsilon) \n",
    "        else:\n",
    "            C = cost(L) \n",
    "\n",
    "        # backpropogation\n",
    "        W, B = backward_propagation(W, B, Y_minibatch, A_layers, Z_layers, 0, alpha, epsilon, len(W) - 1, D,  V_dW, V_dB)\n",
    "        \n",
    "    if(epoch % main_logger_output_epochs == 0):\n",
    "        print('Cost: ' + str(C))\n",
    "\n",
    "    # forward propogation test data set\n",
    "    A_test = forward_propagation(W, B, X_test, 0)\n",
    "\n",
    "    # accuracy\n",
    "    _prediction = prediction(A_test)    \n",
    "    _accuracy = accuracy(_prediction, Y_test)  \n",
    "\n",
    "    # storage for plotting\n",
    "    cost_array.append(C)\n",
    "    accuracy_array.append(_accuracy)\n",
    "    interation_array.append(epoch)\n",
    "\n",
    "\n",
    "end_time = time.time()\n",
    "run_time = end_time - start_time\n",
    "    \n",
    "print('')\n",
    "print('Results:')\n",
    "print('')\n",
    "    \n",
    "print('')\n",
    "print('Run Time: ' + str(run_time) + ' seconds')\n",
    "print('Cost: ' + str(C))    \n",
    "print('Accuracy: ' + str(_accuracy) + ' %')    \n",
    "print('')\n",
    "print('')\n",
    "\n",
    "\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, cost_array, 'red')\n",
    "pyplot.title('Learning Curve - ' + str(len(X[0])) + ' Training Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Cost')\n",
    "pyplot.show()\n",
    "\n",
    "# plot percent accuracy curve\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, accuracy_array, 'red')\n",
    "pyplot.title('Percent Accuracy Curve - ' + str(len(X_test[0])) + ' Test Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Percent Accuracy')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown, our validation set worked, so now we can move on to the full data set, and begin our evaluation and exploration.\n",
    "\n",
    "First, we need to split up our full data set into testing and training data. We will use 50,000 images as the training data set and 10,000 images as the testing data set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(784, 50000)\n",
      "(50000,)\n",
      "(784, 10000)\n",
      "(10000,)\n"
     ]
    }
   ],
   "source": [
    "# create a data set\n",
    "size = vector_size\n",
    "\n",
    "number_of_testing_images = 10000\n",
    "number_of_training_images = 50000\n",
    "number_of_validation_images = number_of_testing_images + number_of_training_images\n",
    "\n",
    "training_images = []\n",
    "training_labels = []\n",
    "testing_images = []\n",
    "testing_labels = []\n",
    "\n",
    "factor = 0\n",
    "for index in range(0, number_of_validation_images):\n",
    "    if(index <= number_of_training_images - 1):\n",
    "        training_images.append(normalized_scaled_images_feature_matrix[:, index + factor])        \n",
    "        training_labels.append(binary_labels[index + factor])\n",
    "    else:\n",
    "        testing_images.append(normalized_scaled_images_feature_matrix[:, index + factor])        \n",
    "        testing_labels.append(binary_labels[index + factor])\n",
    "        \n",
    "# covert to numpy array\n",
    "training_images = np.transpose(np.array(training_images))\n",
    "training_labels = np.array(training_labels)\n",
    "testing_images = np.transpose(np.array(testing_images))\n",
    "testing_labels = np.array(testing_labels)\n",
    "\n",
    "# logger\n",
    "print(training_images.shape) # validation_training_images is a matrix of 784 X 500\n",
    "print(training_labels.shape) # validation_testing_labels is a row vector of 1 X 500\n",
    "print(testing_images.shape) # validation_training_images is a matrix of 784 X 100\n",
    "print(testing_labels.shape)  # validation_testing_labels is a row vector of 1 X 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we must reset out weights and bias's. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Size: 784\n",
      "Weights Shape: (20, 784)\n",
      "Bias Shape: (20, 1)\n",
      "Velocity Weights Shape: (20, 784)\n",
      "Velocity Bias Shape: (20, 1)\n"
     ]
    }
   ],
   "source": [
    "# initialize weights & bias\n",
    "np.random.seed(10)\n",
    "print('Feature Size: ' + str(size))\n",
    "\n",
    "lower_bound = -.1\n",
    "upper_bound = .1\n",
    "\n",
    "#mean = 0.015\n",
    "#std = 0.005\n",
    "\n",
    "# hyper-parameters: hidden layers\n",
    "hidden_layers = 2\n",
    "units_array = [20, 10]\n",
    "Weights = []\n",
    "Bias = []\n",
    "V_dW = []\n",
    "V_dB = []\n",
    "for i in range(0, hidden_layers):\n",
    "    if(i == 0):\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], size]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "    else:\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], units_array[i-1]]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "    \n",
    "# output layer\n",
    "_W = np.float64(np.random.uniform(lower_bound, upper_bound, [1, units_array[i]]))\n",
    "_b = np.float64(np.random.uniform(lower_bound, upper_bound)) # b will be added in a broadcasting manner\n",
    "_V_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_V_dB = np.float64(np.zeros(1))\n",
    "Weights.append(_W)\n",
    "Bias.append(_b)\n",
    "V_dW.append(_V_dW)\n",
    "V_dB.append(_V_dB)\n",
    "\n",
    "Weights = np.array(Weights)\n",
    "Bias = np.array(Bias)\n",
    "V_dW = np.array(V_dW)\n",
    "V_dB = np.array(V_dB)\n",
    "\n",
    "for index in range(0, len(Weights) - 1):\n",
    "    Weights[index] = np.where(Weights[index] != 0, Weights[index], np.random.uniform(lower_bound, upper_bound))\n",
    "\n",
    "#print(train_X.shape)\n",
    "#print(np.ravel(train_Y).shape)\n",
    "\n",
    "print('Weights Shape: ' + str(Weights[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Bias Shape: ' + str(Bias[0].shape)) # vector with a size of the # of unit\n",
    "print('Velocity Weights Shape: ' + str(V_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Velocity Bias Shape: ' + str(V_dB[0].shape)) # vector with a size of the # of unit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we re-run minibatch stochastic gradient descent on the full data set. We will first utilize minibatches of 500 each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Main Loop Epoch: 100\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.040280227907951584\n",
      "Main Loop Epoch: 200\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.02568970533038529\n",
      "Main Loop Epoch: 300\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.020037735231309108\n",
      "Main Loop Epoch: 400\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.026541878026984538\n",
      "Main Loop Epoch: 500\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.013543108816859831\n",
      "Main Loop Epoch: 600\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.013648362081607569\n",
      "Main Loop Epoch: 700\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.007825852764969\n",
      "Main Loop Epoch: 800\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.009420266622494554\n",
      "Main Loop Epoch: 900\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.004402508996987994\n",
      "Main Loop Epoch: 1000\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.0020197110841280795\n",
      "Main Loop Epoch: 1100\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.0033471696372207216\n",
      "Main Loop Epoch: 1200\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.0036489103960996574\n",
      "Main Loop Epoch: 1300\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.001718017371077936\n",
      "Main Loop Epoch: 1400\n",
      "Number Of Minibatches: 100\n",
      "Cost: 0.00119063126728554\n",
      "\n",
      "Results:\n",
      "\n",
      "\n",
      "Run Time: 2118.1275000572205 seconds\n",
      "Cost: 0.0018593347070249052\n",
      "Accuracy: 99.15 %\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEWCAYAAABFSLFOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XncXOP9//HXJ3ckyEJIiiwkSJFaI7W3jVaJpbH8+DYaWy1BvyktVUKrqosiLfWt2qv2WKKEhtiLItxpY0uEIJskJCQiISHJ5/fHdY0592Rm7m3OzH3f834+HvOYs1znnM85c2Y+57rOMubuiIiIlEq7SgcgIiJtixKLiIiUlBKLiIiUlBKLiIiUlBKLiIiUlBKLiIiUVFUlFjN7yMyOrXQcUnlmdqyZPVTqsrImM9vOzCaWaF77mNmMUsyrwPyLftZm9qyZHVdg3JZm1mbv3zCzgWb2TEPKliWxmNkMM9unHMsqxt33d/eb0pi3mXU1s8vNbJaZLTWz6bG/exrLKyUzu8DMvohxZ16bJ8bvaGaTzOzT+L5jYpyZ2cVm9mF8XWJmVoppE+WGJ+L6zMxWJ2Ntyjq7+03uvn+pyzaWmc2J67TUzBab2b/NbES+7VBg+mb/mJnZoWb2spktMbOFZvaYmW1awmX/Frg0MV1yneeb2Q1m1qk569AQZnaimT2VZ/gcMxsM6X7WzZGMsVLc/T/AZ2ZW7/ZpMzUWM2tfwWV3AB4HvgYMAboCewAfArs0YX6VWJc73b1z4vVOjKUDcD9wK9ANuAm4Pw4HGAEcAuwAbA8cBJzc3GmT3P22TFzA/sDcZKy55Su5LzTR/nE9+hJ+gM8Fri3Hgs1sK+BG4HRgPaAfcDWwukTz7w3sBTyQMyqzzgOBXYGfl2J5ko7Ed+o28nxH1+Duqb+AGcA+BcYdBEwGFgPPAdsnxp0DvA18AkwBDk2MOw74N3AZ8BHhqOg44FlgNLAIeJewA2emeQo4MTF9sbL9gKfjsh8DrgRuLbAOJwLvA52LbAMHtkz0/x34beweDMwBzgbmA7cAU4GDEuXbAwuBgbF/t7i9FgMvA4Ob8flcUGTd9gXeAywxbBYwJHY/B4xIjDsBeKG50xaJdTAwJ8/wOcBZwKvA53HYL4B34mf4OjA05zN7KrFtnfCFmR73hyuaWLYGuJxwUPEO8GPAi6zPnNzPDtid8MO+dewfSviOfBK33y8TZefGeJbG19eB/sCTMYaFcX9ar8DyhwG1ReJrR0h0b8d5jQG6FVp2numPBx4uts7An4D7E/1rx2GzCd+rvwJrx3H7ADNyPou+iWlvBS4o8j19qthnkFuGcKA4DfgY+DPhN+e4xGd9WdzObwMjk581sD4hac+Ly7gQaJdYzr/i9IvjvrJvY/aTOHxDYDywIO6LDwC94rgjgYk55c8G7mnodo6f/Xzgxjh8M2AZsFax72lFayxmNhD4G+FLuiFwDTDOzDrGIm8D3yAcSf0auNXMNknMYlfCB/IV4HeJYdOA7sAlwA1FmhWKlb0deDHGdQFwdJFV2Yfw5WlSs0y0MbAB4YMbAdxB2DEy9gMWuvt/zKwX8E9CMt0A+Bkw1sx6NGP53zOzj8zsdTM7NTH8a8ArHveq6JU4PDP+5cS4l3PGNXXaphhGqNGsF/vfBPaM/b8DbjezjYpMfwCwM7ATcFQ9zbeFyp5K2B+2BwYBhzV2Jdz9ecKX+Rtx0FLgqLge3wNON7OD4rhvxmkyNbiXACPsG5sAA4DNgV8WWNwkYDsz+6OZ7Z2nSeoM4MC4nN6EH5Uriiw713aE71heZtaH8OM9PTF4NOHAbntCkuwLnFdoHmkxs68A9xAOcLsTftx3TRQ5lXDwtAOhZeJ/cmZxK/AZsAVhXzgQ+GFi/B6EA6ENCQnmhiaE2Q64DtiU8NvxBSEBAtwHbGVm/RPljyIcaED927k30DnO+0cA7j6TsH8l57mmYlmnVC8K1FiAq4Df5AybBnyrwHwmAwfH7uOAWTnjjwOmJ/rXJRzRbBz7n6JujSVv2bghVwLr5hwJFTqqfxT4Qz3boL4ay+fEo4U4bEvCEeq6sf824PzEUcctOfOfABzbxM9nANCTcAS2B+EI68g47pfAmJzytxGPCoFVxCPr2N8/rqs1Z9oisQ6mcI3lmHrW8zXgwNidrxayW6LsvcDPmlD2aeCExLghNLLGEofXAmcXmOYvwKWJ/aTg/GOZw4GXiozfA7ibUCNZTjjYy+x3b5H4PgJ9gBWEH7SGLPvGzH6es85L4/7twCPEGlWc73Jgs0T5bwBvxe7m1lhWEmoIyddq8tRYCLWtZxPTtyN8N45LfNYnJsYfkNkeQC9CUumYGH808GhiOW8kxnWN69K9MftJnnKDgAWJ/uuAX8fuHeNnvFYDt/NyoEOeZbwP7FEsjkqfY9kMODOetFxsZosJO25PADM7xswmJ8ZtSzhyyJidZ57zMx3u/mnsXKMdvp6yPYGPEsMKLSvjQ8LRYXMscPfliXimE5rDvmdm6xKaQ26PozcDjsjZbnvliyHnxHfeq13cfYq7z3X3Ve7+HOGI5/A4eilhp0/qSvhRyDe+K7A0fruaM21T1PmMzOy4eFI6s422pu7+k2t+ovtTCu83xcr2zImj2H5TTC9CEy9mtruZPWVmC8zsY8KPUsH1MLONzewuM3vPzJYQDmIKlnf359z9CHfvTqiFfBsYFUdvCjyQ2IavEn4Av9LA9VgEdMkz/CB37wJ8h1BL3SAO3xjoCCQ/twcbsbz6POvu6ydfhCa9fOp8lu6+mvADn3c8MDPRvRlhPd5PrMeVQLLGnLsPQfF9bg1m1snMrrdw0dAS4AnqftY3AcNj91GEc6lf0LDt/L67f55nsV0ICbmgSieW2cDvcj7odd39DjPbjJBtRwIbxh3gNcKRcEZTf4DqMw/YIP6gZ/QpUv4xYL96rmz5lFArytg4Z3y+dck0hx0MTInJBsJ2uyVnu3Vy9z/kzsATJ7694Ve7ZGocEM5NbJ/TnLh9HJ4Zv0Ni3A4545o6bVN8uQ0tXNV2FaG5IrP/vEHd/ScN8whNCBnF9pu8zGw3wg/Qs3HQGGAs0Mfd1wOuJ7se+fabiwm1iu3cvSuhdt6g9Xb3FwlNKNvGQXOA7+bsa2u7+/wCy871CvDVIst7glDLyFw19j6h9r5VYnnrxfXOnXZlXM9i36vmmEfi8zOzdtT9bOuMJyThjNmE7/wGifXo6u7blzA+CBc99AN2iZ/1t5Mj3f3ZGPuehN+STDNYQ7bzGp9v/F2GUJMtqJyJZS0zWzvxak9IHKeY2a4WdDKzA82sC9CJsGILAMzsh2R39lR5aEesBS4wsw5mtjuhbbuQWwg70lgz29rM2pnZhmZ2rpkdEMtMBn5gZjVmNgT4VgNCGUNowz2VbG0Fwhfxe2a2X5zf2mY2OF6B02hmdrCZdYufwS7AaYSruSA0H64CTjOzjmY2Mg5/Ir7fDJxhZr3MrCdwJuEIubnTNldnsvuPmdmJhBpL2u4CfmJmPc2sG+GCggYxs/XMLFMz/bu7T42juhBq0Mtj0hmWmOwDwC1xeXgsvwz4OJ7D+FmRZX7LwmW4X4n92xD29RdikauB31u8/NjMvhJjLLTsXI8AX7fslYD5XAYcYGbbuvsqQuK83Mx6xH2yt5ntW2Dal4Hh8XtwIKHmXioPAjvG70d74KdA8jxm5rPuZWYbEpqoAXD32YST86Mt3IrQzsLl2d9sRjwd8vyGdiEksEUxhvPzTHcL4SBrmbu/EONr7HbO+BbwWKz1FFTOxDKe0OaYeV3g7rXASYQ240WEE3jHQWieAf4IPE/IrtsRrsgol+GEq3M+JJwIvZNwdLQGd19BaJN8g3C+ZQnhxH93IHNj2OmEL+ziOO/76gvA3ecR1n+PuPzM8NmEWsy5hB/O2YQfsKZ+nsMI2/4Two/9xR7v94lV4UOAY2LsxwOHJKrI1xCuRHmVUKP8ZxzWrGmby91fIZxkfpFwZLk12c8iTVcREuqrhBPj/yQcGRbzkIX7cWYRThRfSmjuyjgVuMjMPiF85ndlRrj7J8BFwMTYpDEI+BXhZPLHwDhCbaeQRcChwGsxhvFx/n+M4/8EPAw8Hpf/HOHKs0LLrsPd5wLPUOTALNZ+biN7gcGZhGalF+M6PELhk8WnxfgXA0fE9S0Jd38f+D7h8/iQUCNJ7kNXEW4zeBV4iXCiP+kowgHyFMJ2vpvm1agmUPc39BeEz2e9GN9zQL7m7psJB+W35AxvzHbOGE442CjKmt6cXV3M7E7CybZfVToWaT3M7HvA5e6+RaVjqRQz2w64zt13q3Qs1Sg20X8AbOvu7zZjPjsB/+fu9dYKlVgKMLOvE06evktojroP2N3d/1vRwKRFi1/ibxBqrpsA/wD+5e4Fm6NE0mRmPydclVtfM1fJtLY7lMtpY8JlpBsSTmCeqqQiDWCEe2buIZzneJBwD5ZI2ZnZHMK9LQeXdbmqsYiISClV+nJjERFpY1pdU1j37t29b9++lQ5DRKRVmTRp0kJ3b85jnxqs1SWWvn37UltbW+kwRERaFTObWX+p0lBTmIiIlJQSi4iIlJQSi4iIlJQSi4iIlJQSi4iIlJQSi4iIlJQSi4iIlFT1JJZnn4Vf/hI+r+8J5iIi0hzVk1iefx5++1v4ouj/04iISDNVT2LJ/Dvu6tWVjUNEpI2rvsSipzmLiKSqehJLu7iqSiwiIqmqnsSipjARkbKonsSiGouISFlUT2JRjUVEpCyqJ7GoxiIiUhbVk1hUYxERKYvqSyyqsYiIpKp6EouawkREyqJ6EouawkREyqJ6EotqLCIiZVE9iUU1FhGRsqiexKIai4hIWVRPYlGNRUSkLFJNLGY2xMymmdl0Mzsnz/jjzGyBmU2OrxNTDCa8q8YiIpKq9mnN2MxqgCuB7wJzgJfMbJy7T8kpeqe7j0wrji+pKUxEpCzSrLHsAkx393fc/XNgDHBwissrTk1hIiJlkWZi6QXMTvTPicNy/T8ze8XM7jGzPvlmZGYjzKzWzGoXLFjQtGhUYxERKYs0E4vlGZb7q/4A0NfdtwceA27KNyN3v9bdB7n7oB49ejQxGtVYRETKIc3EMgdI1kB6A3OTBdz9Q3dfEXuvA3ZOLRrVWEREyiLNxPIS0N/M+plZB2AYMC5ZwMw2SfQOBaamFo1qLCIiZZHaVWHuvtLMRgITgBrgb+7+upldCNS6+zjgNDMbCqwEPgKOSyseXW4sIlIeqSUWAHcfD4zPGXZ+onsUMCrNGL6kpjARkbLQnfciIlJS1ZNYVGMRESmL6kksqrGIiJRF9SUW1VhERFJVPYlFTWEiImVRPYlFTWEiImVRPYlFNRYRkbKonsSiGouISFlUT2JRjUVEpCyqJ7GoxiIiUhbVl1hUYxERSVX1JBY1hYmIlEX1JBY1hYmIlEX1JBbVWEREyqJ6EotqLCIiZVE9iUU1FhGRsqiexKIai4hIWVRfYlGNRUQkVdWTWNQUJiJSFtWTWNQUJiJSFtWTWFRjEREpi+pJLKqxiIiURfUlFtVYRERSVT2JJdMUphqLiEiqqiexqMYiIlIW1ZNYdPJeRKQsUk0sZjbEzKaZ2XQzO6dIucPNzM1sUIrBhHc1hYmIpCq1xGJmNcCVwP7AAOBIMxuQp1wX4DRgYlqxAKqxiIiUSZo1ll2A6e7+jrt/DowBDs5T7jfAJcDyFGNRjUVEpEzSTCy9gNmJ/jlx2JfMbCegj7s/mGIcmYWFd9VYRERSlWZisTzDvvxVN7N2wGXAmfXOyGyEmdWaWe2CBQuaFo2awkREyiLNxDIH6JPo7w3MTfR3AbYFnjKzGcBuwLh8J/Dd/Vp3H+Tug3r06NG0aNQUJiJSFmkmlpeA/mbWz8w6AMOAcZmR7v6xu3d3977u3hd4ARjq7rWpRKMai4hIWaSWWNx9JTASmABMBe5y99fN7EIzG5rWcgtSjUVEpCzapzlzdx8PjM8Zdn6BsoPTjEU1FhGR8qieO+9VYxERKYvqSyyqsYiIpKp6EouawkREyqJ6EouawkREyqJ6EotqLCIiZVE9iUU1FhGRsqiexKIai4hIWVRPYlGNRUSkLKovsajGIiKSqupJLGoKExEpi+pJLGoKExEpi+pJLKqxiIiURfUkFtVYRETKonoSS01NeF+1qrJxiIi0cdWTWDp2DO9z58LKlZWNRUSkDauexNI+/vXM5ZfDaadVNhYRkTasehJL5hwLwNixlYtDRKSNq57EktShQ6UjEBFps6ozsay1VqUjEBFps6ozsWTOt4iISMlVZ2JRjUVEJDVKLCIiUlLVmVh08l5EJDXVmVhUYxERSU11JhadvBcRSU11JhbVWEREUlOdiUXnWEREUpNqYjGzIWY2zcymm9k5ecafYmavmtlkM3vWzAakGc+XVGMREUlNaonFzGqAK4H9gQHAkXkSx+3uvp277whcAvwprXjq0DkWEZHUpFlj2QWY7u7vuPvnwBjg4GQBd1+S6O0ElOfvHVVjERFJTZqH7r2A2Yn+OcCuuYXM7H+BM4AOwLdTjCe50LIsRkSkGjWoxmJmtzRkWG6RPMPWqJG4+5XuvgVwNvCLAssfYWa1Zla7YMGChoRcnP5FUkQkNQ1tCvtasieeP9m5nmnmAH0S/b2BuUXKjwEOyTfC3a9190HuPqhHjx4NCLceSiwiIqkpmljMbJSZfQJsb2ZL4usT4APg/nrm/RLQ38z6mVkHYBgwLmf+/RO9BwJvNXoNmkJ/TSwikpqiicXdL3L3LsCl7t41vrq4+4buPqqeaVcCI4EJwFTgLnd/3cwuNLOhsdhIM3vdzCYTzrMc2/xVKqJfv/CuGouISGoaevL+QTPr5O7LzOwoYCDwZ3efWWwidx8PjM8Zdn6i+/TGBtwsr74KO+6oxCIikqKGnmO5CvjUzHYAfg7MBG5OLaq0dOoE3bsrsYiIpKihiWWluzvhPpQ/u/ufgS7phZWimholFhGRFDW0KewTMxsFHA18I14V1jrvMmzfXifvRURS1NAay/eBFcDx7j6fcPPjpalFlSbVWEREUtWgxBKTyW3AemZ2ELDc3VvfORZQYhERSVlD77z/H+BF4Ajgf4CJZnZ4moGlRolFRCRVDT3Hch7wdXf/AMDMegCPAfekFVhqlFhERFLV0HMs7TJJJfqwEdO2LDp5LyKSqobWWB42swnAHbH/++Tc+NhqqMYiIpKqoonFzLYENnL3s8zsMGAvwlOLnyeczG99lFhERFJVX3PW5cAnAO5+r7uf4e4/JdRWLk87uFQosYiIpKq+xNLX3V/JHejutUDfVCJKW02NzrGIiKSovsSydpFx65QykLLp2RPeew9WrKh0JCIibVJ9ieUlMzspd6CZnQBMSieklPXrB59/DosXVzoSEZE2qb6rwn4C/MPMhpNNJIMI/09/aJqBpaZjx/C+fHll4xARaaOKJhZ3fx/Yw8z2BraNg//p7k+kHllaMolFTWEiIqlo0H0s7v4k8GTKsZSHEouISKpa593zzaHEIiKSKiUWEREpqepLLGvHK6hfeKGycYiItFHVl1i23jq8L1tW2ThERNqo6kss3buHd/fKxiEi0kZVX2JpF1dZzwsTEUlF9SUW0IMoRURSpMQiIiIlpcQiIiIlVb2JZfXqSkchItImpZpYzGyImU0zs+lmdk6e8WeY2RQze8XMHjezzdKM50vt2qnGIiKSktQSi5nVAFcC+wMDgCPNbEBOsf8Cg9x9e+Ae4JK04qmjpgamTNElxyIiKUizxrILMN3d33H3z4ExwMHJAu7+pLt/GntfAHqnGE/WokXw6KNw001lWZyISDVJM7H0AmYn+ufEYYWcADyUb4SZjTCzWjOrXbBgQekinDy5dPMSEREg3cRieYblbXsys6MIfyB2ab7x7n6tuw9y90E9evQoXYQdOpRuXiIiAjTw/1iaaA7QJ9HfG5ibW8jM9gHOA77l7uV95HD7NFdfRKQ6pVljeQnob2b9zKwDMAwYlyxgZjsB1wBD3f2DFGPJ76KLyr5IEZG2LrXE4u4rgZHABGAqcJe7v25mF5rZ0FjsUqAzcLeZTTazcQVmJyIirUSqbUHuPh4YnzPs/ET3PmkuX0REyq8677wXEZHUKLHoJkkRkZKqzsTSuXO2e8yYysUhItIGVWdiSd5k+YMfVC4OEZE2qDoTi+5fERFJTfUmli23rHQUIiJtUnUmFoCrrqp0BCIibVL1Jpbttqt0BCIibVL1JpaNNqp0BCIibVL1JhaAb30rvOteFhGRkqnuxPKvf4X32trKxiEi0oZUd2LJWLWq0hGIiLQZSiwAu+8OW29d6ShERNoEJZaMadNg8WI491xYubLS0YiItFpKLElnnx3+/OvuuysdiYhIq6XEkrRsWXhXjUVEpMmqO7Fcdlnd/tWrw/tNN+kSZBGRJqruxDJoUN3+TGJ5/HG4447yxyMi0gZUd2JZa626/ZnEAjBhQnljERFpI6o7sXToULc/2fx1883w6KPljUdEpA2o7sSSW2PJvVHynXfKF4uISBuhxJKUbAoDWHfd8sUiItJGKLEk5V4JNmkSmMHzz5cvJhGRVk6JJSm3xvLnP4f3hx8uTzwiIm1AdSeW3JP3uYklQ/e0iIg0WHUnltway5Il+cu5w4oV8JOfwKJF6cclItKKta90ABWVm1h69sxfbvVqGDs2NI0tWwbXXZd+bCIirVSqNRYzG2Jm08xsupmdk2f8N83sP2a20swOTzOWvHITS6FnhL3/fvZS5Hnz0o1JRKSVSy2xmFkNcCWwPzAAONLMBuQUmwUcB9yeVhxF5SaWe+/NX+6GG+CYY0L34sXpxiQi0sql2RS2CzDd3d8BMLMxwMHAlEwBd58RxxU4a56ympqKLFZEpC1LsymsFzA70T8nDms0MxthZrVmVrtgwYKSBCciIulIM7FYnmFNum7X3a9190HuPqhHjx7NDKvEnnoKZs6sdBQiIi1GmollDtAn0d8bmJvi8prmoYegU6eGl//3v8P7+PFw6qmw996wxRbpxCYi0gqlmVheAvqbWT8z6wAMA8aluLymGTIEhg9v3DTTpsGBB8LVV4f+3IdXiohUsdQSi7uvBEYCE4CpwF3u/rqZXWhmQwHM7OtmNgc4ArjGzF5PK56i/vjHxpX/7LN04hARaQNSvUHS3ccD43OGnZ/ofonQRFZZnTtnu6+5Bk4+uXj5v/wl3XhERFqx6n6kS9Kxx4b3hlyCfMMN6cYiItKKKbFkZB40qXtbRESaRYklI3MCXolFRKRZlFgyMomlfTNOO33xBSxfXpp4RERaKSWWjHPPhS23hP32g5dfbto8tt8e1lkndK9cGa42U6IRkSqjxJKx3Xbw1luwwQYhQTTFG2+E9222CTdO/uxnMGIEvPdew6ZfvTr8+diVV9Yd/uqr4S+SX321aXGJiJSReSv7d8RBgwZ5bW1t+guyfE+kKcK9+DQ/+AHcdlvxeaxYAWuvDe3a1b3p8gc/gDvuyC5HRKSRzGySuw8qx7JUYymX228Pr0xiGD0azjqrbplMMsn9i+RMUhERaQWUWMpp+HAYPBg++CAkldGjw138v/hFOBfzxRfNm/8779R9IOYzz8App6xZ7q23Qu0q89wzEZESUmJpqFGjio/fa6+Gzefpp8N5l4zLLoPf/Q7+7/8K/4NlQ22xBfTtm+3/5jfDkwRya0CPPRbeb721ecsTEclDiaWQww4LV3XNmhVqGmefXbx8Y47+778/2/3QQ+F92bLmJ5ZC9JBMESmjVJ8V1qqNHZvtTvPI/tlnw/vKlfmbwkpxsv7DD2H99cNfMQ8Zkq2xiIikQDWWptpkk9LOb9WqujWWa66BqVNh+vS65V58MZwfeeWVhs97k01gjz2gX7/CSeWQQ+Cmmxoft4hIDiWWpth3X9h999LO8w9/gOTfLp9yCmy7LXz1q3XLPfBAeL/vvsbN/7//hdmzC4+//3447rjGzbMhXnsNPvmk+fOZN0//1CnSSiixNMZHH8GSJTBhwponxEvhiSfq9udbxm9/G96XLQvv7qHZrpTnZ556Ct5+u+HlN910zUunIcS23XYwdGjzY+rZs+6FCSLSYimxNEa3btClS+geMKD08//ww4aXXbo0vI8dC4cfHi5dbgr3cEVa0t57w9e+1vB5zJ6df/mffx7e//WvpsUmIq2SEktT/frX4f2AA0JtYeutmz/PSy9teNlMYnn//fA+a1bTljlhApx2WrZ/yZLwvmJF0+aXlHlOmp4YLVJVlFiaqn37cO7gvvvCD+fUqXDmmeVb/s03h/tUfvKT0H/VVU2bz8cf1+1fvLjpMV1xRd3+zF8419SE8yOPPNL0eYtIq6HE0hydO4dLeDPOOAMGDizf8p95Jv+5lZkzQ+JpiNx7XBr6NOYPPoCJE+ueBzr99GxNKjmvFSvC+ZH99gsXHyQv5RaRNkeJpZR69oRJk8KNlQC77Vb/NEOGlD6Ovn2zf7VczKxZayaWTC0jH/fseaA99wzrd/LJdcvsums4+f/GG/DPf645j6FDwzmhXDNnwn/+U3/My5Zl7+3RAzlFWiQlljRkajH77AMbbVS8bObO+0oYPx6OOabusB13zHZ/+GF4/tivfx1+xEeNgu7dww2jmftrrr++7vRTpoSnMW+zDYwc2fBYNt8cdt452//EE9mkl7xMunNnOO+8kMDbtQtJLNeqVfDpp2sOdw8XEighiaRKd96nYcQImD8//BD37Ak/+hH8+MfhaL5bt/CfK5Mn5282a9cu/Mjm3hgJYdpFi9KPP6N792z39tvDxReH7qOPLj7dvHn1z3vs2HA/0MyZ4QKETJPaddeF81fHHw8XXAC/+lW4nDnpoovCdgW4887wYM/ksjPjVq+u+1cGd90Fw4aFZHjCCXXnOW1aWOZDD0HXrvXHLyKFuXureu28887e6qxaVXx8OIYOr2nT3D/6qO6w668P79261R3e1l99+7ovXJh/3O23Z7uTbr01O/yjj9xPPNH9pJPCuN//Pgz/+c/X/AwOOyyMu/JK96VLw7AVK9xHjw7v7u5vvhnKPPyw+6RJ7i++GF651lvPfd99G7ePiKQMqPUy/U6rKayPEcx1AAAOmUlEQVQc2jViM7dvH2omDz8cjrY/+ih7933mHpojjwwnxItdCXb22WEezY2nkmbMCFe+5dOxY7Z75MiwzaBujW7ixFA7ue660J+pvVxySXhlPPMM3Htv6P7f/80+qfqvfw3/Apq52i3zoNFLLw3NdrvsEl6Z571lfPxx9gq4mTPDcq+9NtSYCqmtzcaQz8KFocY4YULhMs21eHGI9Z570luGVIdyZbBSvVpljaU+o0e7r722e7t27p9+uub4p58OR8p77eX+1FPuy5Zlxy1a5H7bbe5du7oPGBDK9ehRd9qjjnIfNiyM69IlHMm3a1f5GklzXscfv+awV15xP+SQ/OXd3S+6aM1h7oXLX3BB6D7vvNB/4435y952W93PKzmPm2/Ov8xc+cYvWxZqYKtXuw8cWP88MpYudf/kk/rL5Zo4Mcx/0KDGT5s0b577tdc2bx5ScpSxxlKWhZTy1SYTS32SiaWQVavCa9Ei98WL85eZNs196tRs/29+U/dH73e/K30CaMmvzz8P2yHfuD33dD/77NC9337uRx/tfsQR+cuOHOk+a1aY38qV2eGHHeZ+7LF1y/7xj+7PPZf9DCZMqDs+44UXssNGjMhfJmPJkrCsmTNDf6dO7maF95XNNnP/7nfXHP7ii2H+AwcWnrYh9twzzGfGjObNR0pKiaXIqyoTy4wZ4aP6/e9LP+/Zs92vuMJ98uTssEWLwo9l587ZH7Mf/cj9pz8N3TU19f9oP/NMaZNAW3ttuGH+4atXuz/7bPFpH3647mc4dmwYvsce4XxQptycOXXLffGFe/fu2fEZK1a4P/BANpn17Oneq5f7+PF1p586NXv+qZi+fcN8Bg9u2D64alWomX3xRcPK12fZMve993Z/+eXSzK+NaDOJBRgCTAOmA+fkGd8RuDOOnwj0rW+eVZlY3N3ff7/+iwBKbYstwi7y7rth2VOmhP477ljzx+7cc7Pdy5eH6e+6Kzvsvvvcf/Ur97POCk0t4L7ppu5PPpkt05CEdcUV5fvxbwuv0aPXrJlmXjNnuj/4oPtXvxr6zz9/zTIff+y+5Zbut9wS+r/73WxN7777QlIbPNj9hhvCPvrxx3Wnv/320Cy2666hqfahh0INbMstw7Jffz1bdscd3d9+2/2JJ/LvjwsWhMT79NPhPSM3IWVqgXvvXeIvRAM8+WRo/ixk/vw1E767+2efpRZSRptILEAN8DawOdABeBkYkFPmR8DVsXsYcGd9863axFIJb77pfuGFdb/EmfM7jz0WXu+9F2pUq1a5n3yy+2uv1Z3HN77hdY6O3UNzUbK5bsWKuucErrvOff/93bfZJky71Vbhx2zhwuwPy+OPr/kjOHy4+1tvhaahQw91/+Uvw7mlzBE0uJ9zjvv3vpft32ij8Kp0AtCr7ivzGe2wQ0h8W23VtPlsskm2u0ePcGBzxRVhv9h1V/d+/dzXWcf9ssvcr746W/b3v3cfMiTU3HLnueuu7qecEuZx/PHu3/lO2IdOOy1bZtYs94svDt2nn+5+5pnhasTM+PHj3Y85JpzXypzv/Mtfwn4L7h07uh98sPvf/x4S8Pjx7j/7WdPOnUVtJbHsDkxI9I8CRuWUmQDsHrvbAwsBKzZfJZZW5rPPQkJIw6pVIfn99a91L2jI5/rrw5c0M91ee4Wj7IwPPwzJ66OPQvJ65BH3e+/NHl3W1oaj6U8/db/77pAYJ00KP0q1te4vvRSa/4YOrfsj9I9/ZLvHjAkJ8dJLw/C5c8Owrl3rTjN4sPujj4Za4OjRDfsBPeus0v+469XyXhdf3OSvSzkTi4XllZ6ZHQ4McfcTY//RwK7uPjJR5rVYZk7sfzuWWZgzrxHACIBNN91055n6wydpDdzr3qBZn5Urw6Xg9V0OPndueADqVlutuZwVK7KXYi9bFh7b06cPdOoUHtfTvn24QXfhwvCUhG22gR494M03w02rJ50UytTUhPeLLw6P4Zk/P5SdPh169QrLXr4c3n0X1l47XAq/wQbh0uyTTgrjFi8Of173j3/AoYeGJ2cPHhzinTs3/Llc//5h2XvvHf4++5lnwjPlFi0Kj/jZcMPwD6hmYdisWeEvHd57Lzx1YffdYcstQ9mDDw5/Vjd0aIjlvfdgvfXCk8cffTT89feUKaH8178e+leuDOMGDoQttgjr06lTWM/Fi8N6b7xxiDmznTp1gp12gnXXDeuxZEn4i4iJE8P4k08O2+7TT8M8NtoInnsuTPPCC2H7b755WJdu3cLl3TfeGKa/+moYPjz7+XTqFC5Z33ff8N9GF1zQ5KeFm9kkdx/UpIkbu6wUE8sRwH45iWUXd/9xoszrsUwysezi7gX/mGTQoEFeW1ubSswiIm1VORNLmnfKzQH6JPp7A3MLlTGz9sB6wEcpxiQiIilLM7G8BPQ3s35m1oFwcn5cTplxwLGx+3DgCU+rCiUiImWR2kMo3X2lmY0knKCvAf7m7q+b2YWEk0jjgBuAW8xsOqGmMiyteEREpDxSfbqxu48HxucMOz/RvRw4Is0YRESkvFrJ0whFRKS1UGIREZGSUmIREZGSUmIREZGSSu0GybSY2QKgqbfedyc8NqYla+kxtvT4QDGWQkuPDxRjY23m7j3KsaBWl1iaw8xqy3XnaVO19BhbenygGEuhpccHirElU1OYiIiUlBKLiIiUVLUllmsrHUADtPQYW3p8oBhLoaXHB4qxxaqqcywiIpK+aquxiIhIypRYRESkpKomsZjZEDObZmbTzeycCsXQx8yeNLOpZva6mZ0eh29gZo+a2VvxvVscbmZ2RYz5FTMbWKY4a8zsv2b2YOzvZ2YTY3x3xr9BwMw6xv7pcXzfMsW3vpndY2ZvxG25ewvchj+Nn/FrZnaHma1d6e1oZn8zsw/iP7dmhjV6u5nZsbH8W2Z2bL5llTC+S+Pn/IqZ/cPM1k+MGxXjm2Zm+yWGp/ZdzxdjYtzPzMzNrHvsL/s2bDHK9R/IlXwRHtv/NrA50AF4GRhQgTg2AQbG7i7Am8AA4BLgnDj8HODi2H0A8BBgwG7AxDLFeQZwO/Bg7L8LGBa7rwZOjd0/Aq6O3cOAO8sU303AibG7A7B+S9qGQC/gXWCdxPY7rtLbEfgmMBB4LTGsUdsN2AB4J753i93dUoxvX6B97L44Ed+A+D3uCPSL3++atL/r+WKMw/sQ/iJkJtC9UtuwpbwqHkBZVhJ2ByYk+kcBo1pAXPcD3wWmAZvEYZsA02L3NcCRifJflksxpt7A48C3gQfjl2Jh4sv95baMX6TdY3f7WM5Sjq9r/NG2nOEtaRv2AmbHH472cTvu1xK2I9A354e7UdsNOBK4JjG8TrlSx5cz7lDgtthd5zuc2Ybl+K7nixG4B9gBmEE2sVRkG7aEV7U0hWW+6Blz4rCKic0dOwETgY3cfR5AfP9KLFaJuC8Hfg6sjv0bAovdfWWeGL6ML47/OJZP0+bAAuDG2Fx3vZl1ogVtQ3d/DxgNzALmEbbLJFrWdsxo7Har5HfpeEINgCJxlD0+MxsKvOfuL+eMajExllu1JBbLM6xi11mbWWdgLPATd19SrGieYanFbWYHAR+4+6QGxlCJ7dqe0BRxlbvvBCwjNOEUUvYY43mKgwlNND2BTsD+ReJoUftnVCimisRqZucBK4HbMoMKxFHu78y6wHnA+flGF4ilJX7eJVUtiWUOoQ00ozcwtxKBmNlahKRym7vfGwe/b2abxPGbAB/E4eWOe09gqJnNAMYQmsMuB9Y3s8y/jSZj+DK+OH49wl9Mp2kOMMfdJ8b+ewiJpqVsQ4B9gHfdfYG7fwHcC+xBy9qOGY3dbmXfnvHk9kHAcI9tRy0ovi0IBxAvx+9Nb+A/ZrZxC4qx7KolsbwE9I9X5XQgnCAdV+4gzMyAG4Cp7v6nxKhxQObKkGMJ514yw4+JV5fsBnycabZIg7uPcvfe7t6XsI2ecPfhwJPA4QXiy8R9eCyf6pGXu88HZpvZVnHQd4AptJBtGM0CdjOzdeNnnomxxWzHhMZutwnAvmbWLdbM9o3DUmFmQ4CzgaHu/mlO3MPiFXX9gP7Ai5T5u+7ur7r7V9y9b/zezCFcoDOfFrINK6LSJ3nK9SJcofEm4YqR8yoUw16EKu8rwOT4OoDQnv448FZ83yCWN+DKGPOrwKAyxjqY7FVhmxO+tNOBu4GOcfjasX96HL95mWLbEaiN2/E+wpU1LWobAr8G3gBeA24hXL1U0e0I3EE45/MF4QfwhKZsN8K5junx9cOU45tOOB+R+b5cnSh/XoxvGrB/Ynhq3/V8MeaMn0H25H3Zt2FLeemRLiIiUlLV0hQmIiJlosQiIiIlpcQiIiIlpcQiIiIlpcQiIiIlpcQiEpnZKjObnHiV7Mm4ZtY33xNxRdqi9vUXEakan7n7jpUOQqS1U41FpB5mNsPMLjazF+Nryzh8MzN7PP7XxuNmtmkcvlH875CX42uPOKsaM7vOwv+0PGJm68Typ5nZlDifMRVaTZGSUWIRyVonpyns+4lxS9x9F+AvhOenEbtvdvftCQ9HvCIOvwL4l7vvQHiO2etxeH/gSnf/GrAY+H9x+DnATnE+p6S1ciLlojvvRSIzW+runfMMnwF8293fiQ8Rne/uG5rZQsJ/mXwRh89z9+5mtgDo7e4rEvPoCzzq7v1j/9nAWu7+WzN7GFhKeDzNfe6+NOVVFUmVaiwiDeMFuguVyWdFonsV2XOcBxKeKbUzMCnxBGSRVkmJRaRhvp94fz52P0d4ei7AcODZ2P04cCqAmdWYWddCMzWzdkAfd3+S8Adr6wNr1JpEWhMdGYlkrWNmkxP9D7t75pLjjmY2kXAwdmQcdhrwNzM7i/Cvlj+Mw08HrjWzEwg1k1MJT8TNpwa41czWIzwN9zJ3X1yyNRKpAJ1jEalHPMcyyN0XVjoWkdZATWEiIlJSqrGIiEhJqcYiIiIlpcQiIiIlpcQiIiIlpcQiIiIlpcQiIiIl9f8B0OEB196u86oAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAEWCAYAAADPZygPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcXFWZ//HPN52QBUISsrBmYYkQQJYQWUSWEURARBBUFhUVQZRRGEfHBX8jjo4C7rgAjqDoMCAKAqLsShSVJRECgRAIsgVZEshiICQheX5/nFN0panuru707Vvd9X2/XvWqu9/n3qp7n3vOPXVLEYGZmVkZBpQdgJmZNS8nITMzK42TkJmZlcZJyMzMSuMkZGZmpXESMjOz0jgJmVnDUHKXpO17aHnPSHpTTyyrxrIHS1omabN2xp8i6eYO5r9d0nuLiK1s+XO8W9LkzqbtNAlJekzS8ryzn5X0E0kb9EyoPSPHeGAd020paY2kH/ZGXGWRtKmkCyU9Lemfkh6U9CVJ65cdW2ckvVvSXyS9JOnWGuN3kTQzj58paZeqcZJ0tqTn8+scSeqJeaumOz4fC8vycbGmqn/ZOmz3dpJe6WSasyStyp9p5XP9rqRxXVjPOp34JE2SdJWkhZKWSLpX0nE9uO6jgaci4oE8T2Wbl0laLOk2SdO6G3+9JA2RFJK2aDP8LEk/BoiIFRGxQUT8o+h4uqI6xrJE+gHqt4EzO5u23pLQ2yNiA2Aq8AbgC10NStLArs5TgPcDi4BjJA3uzRX31vZL2gj4KzAU2CsihgNvAUYCW3djeb39ub0AfAc4q0Ys6wFXA/8LjAIuBq7OwwFOBo4AdgZ2Ag4DPrKu81aLiEvyiWcD4BDgH5X+PKxoF+fPdDTwLmASMEPS2F5YN8ClwFxgPDAG+CCwsAeXfwrw8zbDLs77dixwO/CLHlyf9bCqc8aVwNskje5whojo8AU8BhxY1f914NrcPQK4EHgaeAr4CtCSx30A+DMpG74AfCUPPwmYA/wTeACYmodvBlwBLAAeBT5Rtc4zgcuBn+X57gem5XE/B9YAy4FlwH90sC2PAB8FngWObjNuB+CmHOuzwOfz8Bbg83nefwIzSQfgJCCAgVXLuBX4cHvbT0oCvweeJx24lwAjq+Yfnz+4BXma7wOD8/yvr5puXN7esTW28SvAfcCAdvZBV+P+GrAY2LFq+rF5/eNy/2HAPXm6vwA7dfa9quN792Hg1jbDDsrfM1UNewI4OHf/BTi5atyJwO3rOm8HMe4PzK8xfDwp4S0E/g6cUjVub+BuYCnwDPC1PPy5/Lksy69dayz3LODHbYYNIh1PleNrLHBd/g69kOPYNI/7JrAaeDmv45t5+HnA/BzTncCe7WyvgFXAdh3sk32AO/J34W/A3h2tu828w/Lyx7S3zaQL4QCGVw07Erg3r/NPwPZV454B3pS7LwO+UDXuYGBeO9sxJK9ni/Y+g7bTkI7L3+X9+FfSsXNz1bxvAx7OcX6LlFDfWzX+I6QE/wLwW2DzNus5iXQeWgR8u4PP4DXfk6px/0k6v/4TmA28rWrfLwUmV027BfAS+RxVx37+FOnc/FLV8D8B7+noOOrSPSFJ44FDSQcRpKvJV4BtgF1JB/qHq2bZg3QQjgP+W9K7SAnl/cCGwOHA85IGAL8BZgGbAwcAp0t6a9WyDid9iUYC15BO0ETE+0gnk7dHuho9p53Y9yHt1MtICe39VeOGAzcD15OS4TbALXn0J4Fj83ZvCHyI9MHUY63tJx3EX8vrmEI6WZ2ZY2gBrgUeJyWKzYHLImJFjrm6GuNY0pd7QY11HghcGRFr6oyxs7j/i5QYj60a/25gekQ8J2kqcBHpABoNXABcU1BJcwfg3sjf7uzePLwyflbVuFltxnV33rrlz/F3pKS2GelE93lJ++VJvg98NSI2BCYDV+Xh+wKro7VUdTd1iIhVpGNnnzxoAHA+MAHYMg/7dp7234G7SBccG+R+SCfM15M+v6uBX0oaVGNdQUowF+Rq07ZVVZPy9pwBbESqMblK0qgO1l1tCrA0ImqWrPJ36n2kE96yPGxP4IekEtlo0kXpVSXVvPyIlEA2Jl3sfqgyQtImpPPOv5MuFBYA06rGHwOcDrw9z383qdRe7RDSeXYq8EFJ+3cjxrnAG0kFiLOByySNiYiXgF+x9nnmeOC3EbG4zv38HlKtS3XJZw6pdqF9HWWofLw+RvrAF5NOkD8kVfVsDKwAhlZNeyzwh9z9AeCJNsu6ATitxjr2qDHt54Cf5O4zWfuKYntgeZsYD+xkO34MXJW79yJdcY2rivvuduabC7yjxvBJdF6ieKKTmI6orDfHtKB6eW32z5Pk0g0wA3h3O8t8mKor756Im5TY/l7V/2fg/bn7PODLNfbZfp19tzrZN7VKQv+PlJirh10CnJm7V1N1lU46yQcp+Xd73g5i3J82JSFgP+DhNsO+BJyXu+8knaRHt5lmO+CVTvZJzStc0snrvnbm2RN4uqp/ravvGtOLdJG1bTvjx5BqQ+aQaiBmkEttwBeB/2kz/XTylXAd6z4AeKzGNq8gnX9Wk0qMe1eN/wlwRpt5Hgf2yN3rWhJaktddeb1MjZJQ7l4DTKpaxrfI5y1Sde+tVeNa8ra8N/f/ATi+avwg0jlq46r1TKsafw1wele+J+1M+yDw1qrv7ryqcfcBh3dhPx9XY/nfBH7YUQz1loSOiIiRETExIj4WEcuBiXlHPZ1vGC4mXQVX3yR9ss1yxpOKk21NBDarLCcv6/P5A6h4pqr7JWBIvVc7koaS6s8vAYiIv5JKT5Ubqu3F1dm4zqy1/ZLGSbpM0lOSlpKudMZUrefxiHjNzemIuAN4EdhP0nakkto17azzeWDTbsZbM25SFeJQSXtImgjsAvw6j5sI/Hubz248qRSwFkmfr7qJf3434lpGKo1W25BUtVBr/IbAskhHw7rM2xUTgUlt9scngU3y+BNI95weknRHm9J+d21OugJH0nBJF0l6In/HbqT1O1aTpM9JmitpCamqZ0h780TEwoj4dERMydv0EKmkDGnb39tm26dR47vQjkXA8BrDfx4RI0nf60dI37+KiaSSZvU6x5L2SU/YIZ/7RuYYvtPOdJuQEnj1sfN4Vfdm1eMiYjWperhiInB+1TYsINUyVZc2254Du3wPUtKJuTFJZT3b0PpZ/xFokbSXUqOdTUlVu5X4OtvPbc8bkD7PxR3FtC5NtJ8kXaGMqfqQNoyI6iqMtgfwk9S+Of4k8Gj1hx0RwyPi0Dpj6exEcSTppPJDpSabz5B2XqVKrr24Ohr3Yn4fVjVskzbTtI3ra3nYTpGqY95L+uJW1jOhg8R6cZ7+fcCvIuLldqa7GTgyV3HW0uW4I1XtXU4qMR5HuidYOXk/Cfx3m89uWERc2nbFEfHVaK1uOqWd+DpyP7CTtFartZ3y8Mr46qL/zm3GdXferngSeLDGd/lIgIiYExHvIV2snQtcmRtHdDXZAa/eBD6MVPcO8FnSiesN+Tt2EK3fMdquR9JbgI+TjpGRpGq05W3mqSkiniNd7U9Sann5JOkKvHrb14+Ib9dadw1zgOGS2kuAz5Gqfb9WNc2TwH/W+P5dWWMRL9Lx935dPEPavvFVwyZUdT9dPS4fn21P4B9osx1DI2JmTwUo6XXA90ilso1yUp1H/qzzBdfPaD3PXBapurcSX2f7udbnO4W1q7lfo9tJKCKeJl1lfVPShpIGSNq6qu67lh8Dn5K0m5Jt8pX1ncBSSZ+RNFRSi6QdJb2hznCeBbbqYPwJpPsWryddRe1CukG8i6TXk+7FbCLpdKW2/8Ml7VEV85clTc4x7yRpdKT7MU+RrvxaJH2IzlufDSdXbUraHPh01bg7SV/UsyStr9REdO+q8T8nnSjeS/qitOdbpIR7cd63SNpc0rck7dTNuAH+j1Tne3zurvgf4JRcSlKO/W1K99m6LMc0BBgIDMj7oXJ/4lZSlcwn8uf0r3n47/P7z4BP5u3djFT//tMemLcrbsvbcXqOfWD+zkzNw9+fvz+rSVU9QarGeY50FTqh3SVXkTRI0g6ki4PhpIRG7n6J9B0bw2tbsrY9VoaTqn0WAOuR7gEO6WC935C0ff6cRpBas82OiBdJF0rvknRAHj80d1dO9h0ep7mG5VbS/bH2prmXdMVeuaf0I+Djkqbl798Gkg6XNKzG7PcAh0kamY+/j7e3nq7KF4W/Ab6Ut3sn0rFScQ3wBkmH5e/zp0kJv+J84AuStgWQNErSUesQUkv+/lVe65FKTmtIn/UASaeQSkLVfka653ssa59nurKfyduwPumce0t70wBdbx3XZtwIWlvWLCHdTDsmj/sAcFuNeU4h3TNYRmqdUalP3ozU/PMZUrH89sp6SfeE/rdqGZOouq8BvINUvbYY+FSb9W1OKta+vkYsvwO+kbt3zDtrUY7hs9Fad/sFWluU3EVra5hD8vDFpLrP6ax9b+W2NuvbgdS6bhnpgPh3qu4pkK6crqK19dy5bea/OX8e7d6nqNqXF+Xt+Cep3veLwLDuxF213Hmkap/12gw/OO+XxaRE+kuqWi915ZXXH21eP60av2veh8tJra92rRon4Jwc4wu5Wz0xbzux7k/7reMuJ510F5Huoe2bx12eP9t/kurcD62a72zSCWIxsEuN5Z5FShj/JF3VP0S6st20zXfotvwdexD4GFX3msj1/jmuc0hV6j8ntYx6inR/6dX7KDVi+FGef1mO9WrWblG1d17/IlJivQbYrNa621n+UcCv22xz2xaB++V4N8r9h+fPdQnwD9K9n6F5XPU9ofVJVYdLSeeqT9GzreM2ITVuaq913Nvz9rfXOu5EUul7Kakq7/z2YqHN/a0aMbY9hublcd/I+39B/r695j5d/vzm1lhuXfu5avr3Af/X2TGvPLH1AZIuIv0upcu/0zLrC3J16R2kqqkHyo6nGUn6P+CBiPjKOixDpIR1TEQ81OG0TkJ9g1Lz13tIV++PlhuNmfVHkrYh1RJMiYinOpu+J/jZcX2ApC+Tqi6/7gRkZkWQdA6pmvK/eisBgUtCZmZWIpeEzMysNI3wUNG6jBkzJiZNmlR2GGZmfcbMmTMXRkRvPdy2W/pMEpo0aRIzZswoOwwzsz5D0uOdT1UuV8eZmVlpnITMzKw0TkJmZlYaJyEzMyuNk5CZmZXGScjMzErjJGRmZqXpM78TMms4a9ak9/vvhy23hGefhXHj4LnnYMIEWLEC1lsPImD1apg/HwYOBAnmzYONN4YBA2DiRLjjDth2W7jrLrjySnjXu2D33eH552HHHdO6fve79L7zzmmeSgwD6riWbG+6NWtSnIMHw6pVMCj/dZOUXsuXw5AhabqVK1P8EWm7OlqmlKYbMABeeSV1V7zyCrzwQho2dizcc09a3oIFsPXW8Mc/wiGHwL33pv224YbpPQJGj4YRI2DGjLSPd87/Q/jkk7DBBjByZOofMCBNL6XtamlJ27ZmTes2Dx2apo2Al16CYcPS8IH5tDhoUOofNCgtZ+XKtJ8q2zJnTvoMhw9Py58zB173utZ9CGn+efNghx3SfJVlrFrVup9Wr06vYcNSHEOGpP5K7BFpH/RTTkLW81asgJdfTgfTyy/DokXpZLP++q3TLF2a3gcMSAfeqFFrH7z1uPVW2GILeOIJOPdc+N734Npr4brrYOHCdPLeay+47DK48cY0T0sL7LMP3HlnOiGMH59OBkcfnU5Kf/oT/PnPMHkyTJmSTpAjR6YTzezZaZr581u3rSiXXFL/tIMHp/gWLkz9o0alfd7WoEFpf69YkfbDzjvD3/7WvfhaWtKJsi/ZaKOU/PqaUaP6Ztx16jMPMJ02bVr4iQklWbIkXREPHgxf+QrcdBP813+lE/vQoekkfcEF6Up25cr2l7PRRrB4cWsJois23jiVNPqKMWNSopo/Hw44ICXdu+6qPe26btsGG6RlPPJI59OOGpU+Lwn+/vfurW/KlHTV35ExY1qTIsAee6QLkRUr0mv27HRiHTgwlYwGDEhJceON4frrW+frarI78MB0cXDbba3DNtoolZomToSrr07D3vSmtac56ij4zW86/v5KabsWLGgd9ta3wg03tPYffDDMnQuPPpq2pfI9mDcv7fcpU9J+33VXeOaZ9Jkdckg6js49F97ylnR8HXZY2jeve12K+2Mfq38frBWyZkbEtG7N3EuchJpRpZQC6eTX0pK6V6+GBx5IB+xPf5pKEPPm9ey6e+pqtHKS22YbOO44+M53WktX73tfqhYbNChVcbS0pOnuuy9V5Wy5JRxxRNq2F15IVTtPPZW2f5dd0kl99Oh08D/ySOrfdNOUhB95JJ1QX3gh7cOWljR8XS1cmEp0U6em/jVrUly77JLiqlRrDRqUqq5efjmdpAZWVWasWJFKblOmpOmGDEn7ZMiQzmN88cUUw6pV6eTZ0pJOyCNGpOrFUaPSulavTu+LF6dhFUuWpHkGDkzrrnynekulSmu99dZ9WZWquX7ASagHOQl14tFH08n1739PVVJf/3o6Gey6K9xySzqRSunKfF0ceGA64Rx2GGyyCVx+eXr/8IfTiXnNmlRqOu64tL7qewaVewhPPpmupA86KA2//HLYc890Aqmc5O+8M63noIPSyXjMGNhqq9ZlSeu2HWZNwEmoBzkJZfPnp5P8eeelm9lbbAGzZqUqgJ50+OFwzTWphHHPPfDpT6f7DHvv3bPrMbPC9IUk5IYJje7ll+HBB1N10PTp8J731DffBz6Qqpsq9d4XXphKJ7NmpWVNmJDGbbppqo8eNCiVNubOTfXQ9bS4MjNbR05CjaBSGq1UMS1enG5SfvGL9c2///5w6qkwcyZ84Qtrt0Jra489Wrvf9KbXjt9uu/rWaWbWA5yEyrZ0abpZv9VWsNlmcPLJcPzxr51u/fXTzeM3vjE1Ia7l6KOLjdXMrIc5CZXl0UdTU9RK08uHH06v6dPXnq5SPWZm1g85CfWmV15Jrdd22SU1LmjPBRek0lBH1WpmZv2Ak1DRKj9sO+II+Mtfak9z0EHwi1+0PnLEzKxJOAkV6a670vO/2tpqq/SImXHjYLfd/JsXM2taTkJFWb26dgJ66qnUAMHMzPxXDj3uxRfhq19d+3EqkB4lE+EEZGZWxSWhnrbBBmv3f+EL8OUvlxOLmVmDcxLqKWvWpOeqVRx9NHz+8+nZbWZmVpOTUE+ZNg3uvru1/+yz137gppmZvYbvCa2rCPj4x1sT0OTJaZgTkJlZp5yE1sUNN6Tf+Hz/+/Av/5KeePDgg2VHZWbWZ7g6rjtWr4Yzz0z/MgrpD9NuvPG1LeLMzKxDPmt2x1ZbpX/BhPSUg+nTnYDMzLrB1XFd9be/tSagQw9Nf4/t3/6YmXWLL9+74tln02N2AEaMgN/+ttx4zMz6uMJLQpJOkzRb0v2STs/DdpF0u6R7JM2QVOP5Ng3m8cdhk01a+59/vrxYzMz6iUKTkKQdgZOA3YGdgcMkTQbOAb4UEbsA/5n7G9ukSa3dS5dCS0tpoZiZ9RdFV8dNAW6PiJcAJE0HjgQC2DBPMwL4R8FxrJvqUs+ll8Lw4eXFYmbWjxSdhGYD/y1pNLAcOBSYAZwO3CDpG6TS2BsLjqP7Vq6EY45p7X/DG8qLxcysnym0Oi4i5gBnAzcB1wOzgFeAjwL/FhHjgX8DLqw1v6ST8z2jGQsqfw7X237wA7j55vRk7OXLYeuty4nDzKwfUkT03sqkrwLzga8BIyMiJAlYEhEbdjTvtGnTYsaMGb0RZqtFi1JjhP33h+uv95/PmVmfImlmREwrO46O9EbruHH5fQLwTuBS0j2g/fIkbwYeLjqObnnggVQd94lPOAGZmRWgN34ndEW+J7QKODUiFkk6CfiupIHAy8DJvRBH182dm963267cOMzM+qnCk1BE7FNj2G3AbkWve5099BAMGgQTJ5YdiZlZv+TH9nTk3nvTw0n9XDgzs0I4CbXnnnvguuvg4IPLjsTMrN9yEmrP7ben9+OOKzcOM7N+zEmoPX/8I2y4IUydWnYkZmb9lpNQLQsXwuWXw4knwgDvIjOzovgMW8v996d/T/X9IDOzQjkJ1VL5fdC225Ybh5lZP+ckVMt3vpOelD1+fNmRmJn1a05CbS1ZAnPmwH77+X6QmVnBfJZt66qr0vvpp5cbh5lZE3ASauumm2DcOHjzm8uOxMys33MSqvbyy3DttalVnJ+abWZWOCeharNmpXtCRxxRdiRmZk3BSajannum9x13LDcOM7Mm4SRU8eyzrd3bbFNeHGZmTcRJqOLmm9P7nXf6fpCZWS9xEqq45BIYPdoPLDUz60VOQhXXXZeaZbe0lB2JmVnTcBIC+MMf0vs115Qbh5lZk3ESArjxxvR+4YXlxmFm1mSchCA9K26HHeD448uOxMysqTgJRcDdd/tvG8zMSuAkdMcd8MQT8I53lB2JmVnTcRK64Yb0u6DDDy87EjOzpuMkdNNN6TE9I0eWHYmZWdOpKwlJukLS2yT1r6T16KPw5z/DUUeVHYmZWVOqN6mcBxwHPCzpLEnbFRhT73noofR+wAHlxmFm1qTqSkIRcXNEHA9MBR4DbpL0F0kflDSoyAAL9cQT6X2TTcqNw8ysSdVdvSZpNPAB4MPA3cB3SUnppkIi6w233gojRsD48WVHYmbWlAbWM5GkK4HtgJ8Db4+Ip/OoX0iaUVRwhVq+HH79a9hvPxg8uOxozMyaUl1JCPh+RPy+1oiImNaD8fSeP/85JaJ99y07EjOzplVvddwUSa+2YZY0StLHCoqpdzzzTHp3yzgzs9LUm4ROiojFlZ6IWAScVExIveTxx9P7ppuWG4eZWROrNwkNkFr/blRSC7BeMSH1kt/9DnbaCYYPLzsSM7OmVW8SugG4XNIBkt4MXApcX1xYBVu1CmbMgLe+texIzMyaWr0NEz4DfAT4KCDgRuDHRQVVuIcegpUrYeedy47EzKyp1ftj1TURcV5EHB0RR0XEBRGxurP5JJ0mabak+yWdXjX845Lm5uHnrMsGdMuTT6b3Lbfs9VWbmVmren8nNBn4GrA9MKQyPCK26mCeHUmNF3YHVgLXS/otsAXwDmCniFghaVz3w++mJUvS+4gRvb5qMzNrVW913E+ALwLfBv4F+CCpWq4jU4DbI+IlAEnTgSOBacBZEbECICKe60bc62ZxbujnJ2ebmZWq3oYJQyPiFkAR8XhEnAm8uZN5ZgP7ShotaRhwKDAeeB2wj6Q7JE2X9IbuBt9tTkJmZg2h3pLQy/lvHB6W9K/AU0CH1WgRMUfS2aRnyy0DZgGv5HWOAvYE3kBqdbdVRETbZUg6GTgZYMKECXWGWoeFC2HIEBg2rOeWaWZmXVZvSeh0YBjwCWA34L3ACZ3NFBEXRsTUiNgXeAF4GJgPXBnJncAaYEw78/8oIqZFxLSxY8fWGWodFi6EMWPSP6qamVlpOi0J5R+mvjsiPk0q0Xyw3oVLGhcRz0maALwT2IuUdN4M3CrpdaQfvS7sTvDdVklCZmZWqk6TUESslrSbJNWqMuvEFfkvIFYBp0bEIkkXARdJmk1qNXdCN5a7bpyEzMwaQr33hO4Grpb0S+DFysCIuLKjmSJinxrDVpKq88rz/PPQk/eYzMysW+pNQhsBz7N2i7gAOkxCDevll90owcysAdSVhCKi7vtAfcLKlbBe337+qplZf1DvExN+Qir5rCUiPtTjEfUGJyEzs4ZQb3XctVXdQ0hPPvhHz4fTS5yEzMwaQr3VcVdU90u6FLi5kIh6w8qVMHhw2VGYmTW9en+s2tZkoG82L4tI/yfkkpCZWenqvSf0T9a+J/QM6T+G+p5Vq9K7k5CZWenqrY7rP/+BvWJFencSMjMrXV3VcZKOlDSiqn+kpCOKC6tAK1emdychM7PS1XtP6IsRsaTSExGLSf8v1Pc4CZmZNYx6k1Ct6ept3t1YKknIrePMzEpXbxKaIelbkraWtJWkbwMziwysMC4JmZk1jHqT0MdJT7z+BXA5sBw4taigCuUkZGbWMOptHfci8NmCY+kdTkJmZg2j3tZxN0kaWdU/StINxYVVIDfRNjNrGPVWx43JLeIAiIhFwLhiQiqYS0JmZg2j3iS0Jv9FNwCSJlLjqdp9QuWJCYMGlRuHmZnV3cz6DOA2SdNz/77AR4oJqWBr1qT3Ad19bJ6ZmfWUehsmXC9pKrAnIODfImJhoZEVTSo7AjOzpld3cSAiFkbEtcADwCmSZhcXVoGib9Yimpn1R/W2jttU0umS7gTuB1qAYwuNrCiVJOSSkJlZ6TpMQpJOkvR7YDowBvgw8HREfCki7uuNAHuck5CZWcPo7J7QD4C/AsdFxAwASf2jPstJyMysdJ0loc2AdwHfkrQx6ZE9fbtts+8JmZk1jA6r43JjhPMiYl/gAGAJ8JykOZK+2isR9jRXx5mZNYyutI6bHxHfiIjdgCOAFcWFVSAnITOzhtGt/wSKiLnAl3o4lt7lJGRmVrrme2yA7wmZmTWM5k1CLgmZmZWu3h+r3lLPsD7BScjMrGF0eE9I0hBgGDBG0ijSc+MANiQ13+67nITMzErXWcOEjwCnkxLOTFqT0FLSD1n7Ht8TMjNrGB0moYj4LvBdSR+PiO/1UkzFcnWcmVnDqPevHL4n6Y3ApOp5IuJnBcVVHCchM7OGUVcSkvRzYGvgHmB1HhxA30tCFU5CZmalq/fHqtOA7SP6wQ2VfrAJZmb9Rb2/E5oNbNKdFUg6TdJsSfdLOr3NuE9JCkljurPsbnF1nJlZw6i3JDQGeCD/qd2rz4yLiMM7mknSjsBJwO7ASuB6Sb+NiIcljQfeAjzRrci7y0nIzKxh1JuEzuzm8qcAt0fESwCSpgNHAucA3wb+A7i6m8s2M7M+rq7quIiYDjwGDMrddwF/q2PW2cC+kkZLGgYcCoyXdDjwVETM6l7Y68AlITOzhlFv67iTgJOBjUit5DYHzif9x1C7ImKOpLOBm4BlwCzgFeAM4KA61ntyXi8TJkyoJ9TOOQmZmTWMehsmnArsTXpSAhHxMDCunhkj4sKImJr/GO8FUolqS2CWpMeALYC/SXpNw4dgEbASAAANh0lEQVSI+FFETIuIaWPHjq0z1Do5CZmZla7eJLQiIlZWeiQNJP1OqFOSxuX3CcA7gZ9FxLiImBQRk4D5wNSIeKZLkXeXm2ibmTWMehsmTJf0eWCopLcAHwN+U+e8V0gaDawCTo2IRd2Is+e4Os7MrGHUm4Q+C5wI3Ed6qOnvgB/XM2NE7NPJ+El1xtAznITMzBpGvUloKHBRRPwPgKSWPOylogIrnJOQmVnp6r0ndAsp6VQMBW7u+XB6ge8JmZk1jHqT0JCIWFbpyd3DigmpYK6OMzNrGPUmoRclTa30SNoNWF5MSAVzEjIzaxj13hM6DfilpH/k/k2B9xQTUi9xEjIzK12nSUjSAGA9YDtgW9JffD8YEasKjq0YvidkZtYwOk1CEbFG0jcjYi/Ss+D6NlfHmZk1jHrvCd0o6SipH5y5nYTMzBpGvfeEPgmsD6yWtJxUJRcRsWFhkRXNScjMrHR1JaGIGF50IL3G94TMzBpGXdVxSt4r6f/l/vGSdi82tIK4Os7MrGHUe0/oh8BewHG5fxnwg0IiKpqTkJlZw6j3ntAeETFV0t0AEbFI0noFxlU8JyEzs9LVWxJalR9aGgCSxgJrCouqSL4nZGbWMOpNQucCvwbGSfpv4Dbgq4VFVSRXx5mZNYx6W8ddImkmcACpefYRETGn0MiK4iRkZtYwOkxCkoYApwDbkP7Q7oKIeKU3Aiuck5CZWek6q467GJhGSkCHAN8oPKKi+Z6QmVnD6Kw6bvuIeD2ApAuBO4sPqWCujjMzaxidlYRefVJ2v6mGcxIyM2sYnZWEdpa0NHcLGJr7/ew4MzNbZx0moYho6a1Aeo3vCZmZNYx6fyfUf7g6zsysYTgJmZlZaZovCVU4CZmZla75kpDvCZmZNYzmTUIuCZmZlc5JyMzMStN8ScjMzBpG8yUhl4TMzBqGk5CZmZWm+ZJQhZOQmVnpmi8JuYm2mVnDaN4k5JKQmVnpnITMzKw0zZeEKpyEzMxK13xJyPeEzMwaRqFJSNJpkmZLul/S6XnY1yU9KOleSb+WNLLIGF7D1XFmZg2jsCQkaUfgJGB3YGfgMEmTgZuAHSNiJ+Ah4HNFxVCTk5CZWcMosiQ0Bbg9Il6KiFeA6cCREXFj7ge4HdiiwBja5yRkZla6IpPQbGBfSaMlDQMOBca3meZDwHXtLUDSyZJmSJqxYMGCnonK94TMzBpGYUkoIuYAZ5Oq364HZgGVEhCSzsj9l3SwjB9FxLSImDZ27NieCqwSQM8sz8zMuq3QhgkRcWFETI2IfYEXgIcBJJ0AHAYcH9HLRZPK6gY0X8NAM7NGM7DIhUsaFxHPSZoAvBPYS9LBwGeA/SLipSLXX9OaNZXgen3VZma2tkKTEHCFpNHAKuDUiFgk6fvAYOAmpURwe0ScUnAcrVwSMjNrGIUmoYjYp8awbYpcZ6dcEjIzaxjNVxxwScjMrGE035nYJSEzs4bRfEkowgnIzKxBNF8SWrPGScjMrEE0XxKK8P0gM7MG0XxnY5eEzMwaRnMmIZeEzMwaQvOdjV0dZ2bWMJrvbOzqODOzhtF8ScglITOzhtF8Z2OXhMzMGkbzJSGXhMzMGkbznY1dEjIzaxjNl4RcEjIzaxjNdzZ2ScjMrGE0XxJyScjMrGE039nYJSEzs4bRfEnIJSEzs4bRfGdjl4TMzBpG8yUhl4TMzBpG852NXRIyM2sYzZeEXBIyM2sYzXc2dknIzKxhNF8ScknIzKxhNN/Z2CUhM7OG0XxJyCUhM7OG0XxnY5eEzMwaRvMlIZeEzMwaRvOdjV0SMjNrGAPLDqBwe+0FS5e29j/wAOy0U3nxmJnZq/p/Etp2W3jxxdb+7beHE04oLx4zM3tV/09CP/1p2RGYmVk7mu+ekJmZNQwnITMzK42TkJmZlcZJyMzMSlN4EpJ0mqTZku6XdHoetpGkmyQ9nN9HFR2HmZk1nkKTkKQdgZOA3YGdgcMkTQY+C9wSEZOBW3K/mZk1maJLQlOA2yPipYh4BZgOHAm8A7g4T3MxcETBcZiZWQMqOgnNBvaVNFrSMOBQYDywcUQ8DZDfx9WaWdLJkmZImrFgwYKCQzUzs96miCh2BdKJwKnAMuABYDnwwYgYWTXNoojo8L6QpAXA490MYwywsJvz9oZGjw8cY09o9Pig8WNs9PigsWKcGBFjyw6iI4UnobVWJn0VmA+cBuwfEU9L2hS4NSK2LXC9MyJiWlHLX1eNHh84xp7Q6PFB48fY6PFB34ixkfRG67hx+X0C8E7gUuAaoPIAtxOAq4uOw8zMGk9vPDvuCkmjgVXAqRGxSNJZwOW5qu4J4F29EIeZmTWYwpNQROxTY9jzwAFFr7vKj3pxXd3R6PGBY+wJjR4fNH6MjR4f9I0YG0av3hMyMzOr5sf2mJlZaZyEzMysNP06CUk6WNJcSfMklfZoIEnjJf1B0pz8DL3T8vCaz9BTcm6O+15JU3spzhZJd0u6NvdvKemOHN8vJK2Xhw/O/fPy+Em9FN9ISb+S9GDel3s10j6U9G/5850t6VJJQ8reh5IukvScpNlVw7q8zySdkKd/WFKP/jVxOzF+PX/O90r6taTq3xV+Lsc4V9Jbq4YXdrzXirFq3KckhaQxub+U/dhnRUS/fAEtwCPAVsB6wCxg+5Ji2RSYmruHAw8B2wPnAJ/Nwz8LnJ27DwWuAwTsCdzRS3F+Evg/4NrcfzlwTO4+H/ho7v4YcH7uPgb4RS/FdzHw4dy9HjCyUfYhsDnwKDC0at99oOx9COwLTAVmVw3r0j4DNgL+nt9H5e5RBcd4EDAwd59dFeP2+VgeDGyZj/GWoo/3WjHm4eOBG0g/pB9T5n7sq6/SAyhsw2Av4Iaq/s8Bnys7rhzL1cBbgLnApnnYpsDc3H0BcGzV9K9OV2BMW5AeJvtm4Np8AC2sOhG8uj/zQbdX7h6Yp1PB8W2YT/JqM7wh9iEpCT2ZTzAD8z58ayPsQ2BSmxN8l/YZcCxwQdXwtaYrIsY2444ELsndax3Hlf3YG8d7rRiBX5EezvwYrUmotP3YF1/9uTquclKomJ+HlSpXu+wK3EH7z9ArI/bvAP8BrMn9o4HFkR482zaGV+PL45fk6Yu0FbAA+EmuMvyxpPVpkH0YEU8B3yD97u1p0j6ZSWPtw4qu7rOyj6UPkUoWdBBLr8co6XDgqYiY1WZUw8TYF/TnJKQaw0ptjy5pA+AK4PSIWNrRpDWGFRa7pMOA5yJiZp0xlLFvB5KqQ86LiF2BF+n4L0B6ex+OIj0dfktgM2B94JAOYmi47yftx1RarJLOAF4BLqkMaieW3v68hwFnAP9Za3Q7sTTiZ166/pyE5pPqayu2AP5RUixIGkRKQJdExJV58LNKz84jvz+Xh/d27HsDh0t6DLiMVCX3HWCkpMoPmqtjeDW+PH4E8EKB8VXWOT8i7sj9vyIlpUbZhwcCj0bEgohYBVwJvJHG2ocVXd1npRxL+cb9YcDxkeuvGijGrUkXHLPycbMF8DdJmzRQjH1Cf05CdwGTc+uk9Ug3f68pIxBJAi4E5kTEt6pGtfcMvWuA9+dWNnsCSyrVJ0WIiM9FxBYRMYm0n34fEccDfwCObie+StxH5+kLvaKLiGeAJyVVHnR7AOmp7A2xD0nVcHtKGpY/70p8DbMPq3R1n90AHCRpVC7xHZSHFUbSwcBngMMj4qU2sR+TWxduCUwG7qSXj/eIuC8ixkXEpHzczCc1PnqGBtqPfULZN6WKfJFaqTxEajVzRolxvIlU7L4XuCe/DiXdA7gFeDi/b5SnF/CDHPd9wLRejHV/WlvHbUU6wOcBvwQG5+FDcv+8PH6rXoptF2BG3o9XkVoYNcw+BL4EPEj6H62fk1pwlboPSQ8Mfpr07Mb5wInd2Wek+zLz8uuDvRDjPNL9k8rxcn7V9GfkGOcCh1QNL+x4rxVjm/GP0dowoZT92FdffmyPmZmVpj9Xx5mZWYNzEjIzs9I4CZmZWWmchMzMrDROQmZmVhonIWtqklZLuqfq1WNPX5Y0qdZTl82sVeF/723W4JZHxC5lB2HWrFwSMqtB0mOSzpZ0Z35tk4dPlHRL/p+YWyRNyMM3zv97Myu/3pgX1SLpf5T+Z+hGSUPz9J+Q9EBezmUlbaZZ6ZyErNkNbVMd956qcUsjYnfg+6Rn6ZG7fxYRO5EeqnluHn4uMD0idiY90+7+PHwy8IOI2AFYDByVh38W2DUv55SiNs6s0fmJCdbUJC2LiA1qDH8MeHNE/D0/fPaZiBgtaSHpv3hW5eFPR8QYSQuALSJiRdUyJgE3RcTk3P8ZYFBEfEXS9cAy0uOHroqIZQVvqllDcknIrH3RTnd709Syoqp7Na33Yd9Ger7YbsDMqidtmzUVJyGz9r2n6v2vufsvpCc0AxwP3Ja7bwE+CiCpRdKG7S1U0gBgfET8gfRHgiOB15TGzJqBr76s2Q2VdE9V//URUWmmPVjSHaSLtWPzsE8AF0n6NOmfXj+Yh58G/EjSiaQSz0dJT12upQX4X0kjSE9c/nZELO6xLTLrQ3xPyKyGfE9oWkQsLDsWs/7M1XFmZlYal4TMzKw0LgmZmVlpnITMzKw0TkJmZlYaJyEzMyuNk5CZmZXm/wNv8E5AbpNolgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# gradient descent\n",
    "detailed_logger = False\n",
    "main_logger = True\n",
    "main_logger_output_epochs = 100\n",
    "L2 = False\n",
    "Dropout = False\n",
    "momentum = False\n",
    "hidden_layer_relu = True\n",
    "hidden_layer_tanh = False\n",
    "hidden_layer_sigmoid = False\n",
    "\n",
    "# hyber-parameters\n",
    "alpha = .01;\n",
    "epsilon = .85\n",
    "keep_prob = .9\n",
    "number_of_epochs = 1500\n",
    "batch_size = 500\n",
    "momentum_coef = .9\n",
    "\n",
    "# copy initalization\n",
    "W = Weights.copy()\n",
    "B = Bias.copy()\n",
    "\n",
    "# data arrays\n",
    "cost_array = []\n",
    "accuracy_array = []\n",
    "interation_array = []\n",
    "\n",
    "# rename\n",
    "X_train = np.float64(training_images).copy()\n",
    "Y_train = np.float64(training_labels).copy()\n",
    "\n",
    "X_test = np.float64(testing_images).copy()\n",
    "Y_test = np.float64(testing_labels).copy()\n",
    "\n",
    "#m = size\n",
    "m = number_of_training_images\n",
    "\n",
    "def model(W, B, A):\n",
    "    return np.dot(W, A) + B\n",
    "\n",
    "def activation_relu(Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    return np.where(Z > 0, Z, 0)\n",
    "\n",
    "def activation_tanh(Z):\n",
    "    return np.tanh(Z)\n",
    "\n",
    "def activation_sigmoid(Z):\n",
    "     return 1/(1 + np.exp(-Z))\n",
    "\n",
    "def loss(A, Y):\n",
    "    epsilon = 1e-20\n",
    "    return np.where((Y == 1), np.multiply(-Y, np.log(A + epsilon)), -np.multiply((1 - Y), np.log(1 - A + epsilon)))\n",
    "    #return np.multiply(-Y, np.log(A)) - np.multiply((1 - Y), np.log(1 - A)) \n",
    "    \n",
    "def cost(L):\n",
    "    return np.multiply(1/L.shape[1], np.sum(L))\n",
    "\n",
    "def cost_L2(L, W, epsilon):\n",
    "    L2 = np.multiply(epsilon/(2*W.shape[1]), np.multiply(W[len(W)-3], W[len(W)-3]).sum() + np.multiply(W[len(W)-2], W[len(W)-2]).sum() + np.multiply(W[len(W)-1], W[len(W)-1]).sum())\n",
    "    J = cost(L)\n",
    "    return L2 + J\n",
    "\n",
    "def prediction(A):\n",
    "    return np.where(A >= 0.5, 1, 0)\n",
    "    \n",
    "def accuracy(prediction, Y):\n",
    "    return 100 - np.multiply(100/Y.shape[0], np.sum(np.absolute(Y - prediction)))  \n",
    "    \n",
    "def forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "            print('Forward Propagation Training Data Complete')\n",
    "    return A_layers, Z_layers, D\n",
    "\n",
    "def forward_propagation(W, B, A, layer):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "        A = forward_propagation(W, B, A, layer)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        A = activation_sigmoid(Z)  \n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "            print('Forward Propagation Testing Data Complete')\n",
    "    return A\n",
    "\n",
    "def dZ(dZ, W, Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    W = np.where(~np.isnan(W), W, 0)\n",
    "    dZ = np.where(~np.isnan(dZ), dZ, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    W = np.where(~np.isinf(W), W, 0)\n",
    "    dZ = np.where(~np.isinf(dZ), dZ, 0)\n",
    "    if(hidden_layer_relu == True):\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.where(Z > 0, 1, 0))\n",
    "    elif(hidden_layer_tanh == True):\n",
    "        A = activation_tanh(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), 1- np.multiply(A, A))\n",
    "    elif(hidden_layer_sigmoid == True): \n",
    "        A = activation_sigmoid(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.multiply(A, (1-A)))\n",
    "\n",
    "def dW(dZ, A):\n",
    "    return np.multiply(1/dZ.shape[1], np.dot(dZ, np.transpose(A)))\n",
    "\n",
    "def dW_L2(dZ, A, W, epsilon):\n",
    "    return np.multiply(epsilon/Z.shape[1], W) + dW(dZ, A)\n",
    "\n",
    "def dB(dZ):\n",
    "    return np.multiply(1/dZ.shape[1], np.sum(dZ))\n",
    "\n",
    "def backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB):\n",
    "    if(layer >= 0):\n",
    "        if(layer == len(W) - 1):\n",
    "            _dZ = A_layers[layer+1] - Y\n",
    "        elif(layer >= 0):\n",
    "            _dZ = dZ(_dZ, W[layer+1], Z_layers[layer])\n",
    "        if(Dropout == True):\n",
    "            _dZ = np.multiply(_dZ, D[layer])\n",
    "        if(L2 == True):\n",
    "            _dW = dW_L2(_dZ, A_layers[layer], W[layer], epsilon)\n",
    "        else:\n",
    "            _dW = dW(_dZ, A_layers[layer])\n",
    "        _dB = dB(_dZ)\n",
    "        if(momentum == True):\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(alpha, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(alpha, _dB)\n",
    "            W[layer] = W[layer] - V_dW[layer]\n",
    "            B[layer] = B[layer] - V_dB[layer]           \n",
    "        else:\n",
    "            W[layer] = W[layer] - np.multiply(alpha, _dW)\n",
    "            B[layer] = B[layer] - np.multiply(alpha, _dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Layer: ' + str(layer))\n",
    "        layer = layer - 1\n",
    "        W, B = backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Propagation Complete')\n",
    "    return W, B\n",
    "    \n",
    "\n",
    "def shuffle(X, Y, number_of_training_images):\n",
    "    random_array = np.random.permutation(np.arange(number_of_training_images))\n",
    "    return X[:, random_array], Y[random_array]\n",
    "    \n",
    "start_time = time.time()   \n",
    "# main loop\n",
    "for epoch in range(1, number_of_epochs):\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Main Loop Epoch: ' + str(epoch))\n",
    "    \n",
    "    # shuffle data\n",
    "    X, Y = shuffle(X_train.copy(), Y_train.copy(), number_of_training_images)\n",
    "    number_of_batches = int(np.floor(number_of_training_images/batch_size))\n",
    "    split_index = number_of_batches*batch_size\n",
    "\n",
    "    # parse into minibatches\n",
    "    X_minibatches = np.split(X[:, 0:split_index], number_of_batches, axis=1)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        X_left_over_portion  = X[:, split_index:number_of_training_images]\n",
    "        X_minibatches.append(X_left_over_portion)\n",
    "    \n",
    "    Y_minibatches = np.split(Y[0:split_index], number_of_batches, axis=0)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        Y_left_over_portion  = Y[split_index:number_of_training_images]\n",
    "        Y_minibatches.append(Y_left_over_portion)\n",
    "    \n",
    "    number_of_minibatches = len(Y_minibatches)\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Number Of Minibatches: ' + str(number_of_minibatches))\n",
    "\n",
    "    for index in range(0, number_of_minibatches-1):\n",
    "        X_minibatch = X_minibatches[index]\n",
    "        Y_minibatch = Y_minibatches[index]\n",
    "\n",
    "        if(hidden_layer_relu + hidden_layer_tanh + hidden_layer_sigmoid != 1):\n",
    "            print(\"ERROR! Please Select Only 1 Hidden Layer Activation Function\")\n",
    "            break\n",
    "\n",
    "        # forward propogation training data set\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, X_minibatch, [X_minibatch], [], 0, [], keep_prob)\n",
    "        L = loss(A_layers[len(A_layers) - 1], Y_minibatch)\n",
    "        if(L2 == True):\n",
    "            C = cost_L2(L, W, epsilon) \n",
    "        else:\n",
    "            C = cost(L) \n",
    "\n",
    "        # backpropogation\n",
    "        W, B = backward_propagation(W, B, Y_minibatch, A_layers, Z_layers, 0, alpha, epsilon, len(W) - 1, D,  V_dW, V_dB)\n",
    "        \n",
    "    if(epoch % main_logger_output_epochs == 0):\n",
    "        print('Cost: ' + str(C))\n",
    "\n",
    "    # forward propogation test data set\n",
    "    A_test = forward_propagation(W, B, X_test, 0)\n",
    "\n",
    "    # accuracy\n",
    "    _prediction = prediction(A_test)    \n",
    "    _accuracy = accuracy(_prediction, Y_test)  \n",
    "\n",
    "    # storage for plotting\n",
    "    cost_array.append(C)\n",
    "    accuracy_array.append(_accuracy)\n",
    "    interation_array.append(epoch)\n",
    "\n",
    "\n",
    "end_time = time.time()\n",
    "run_time = end_time - start_time\n",
    "    \n",
    "print('')\n",
    "print('Results:')\n",
    "print('')\n",
    "    \n",
    "print('')\n",
    "print('Run Time: ' + str(run_time) + ' seconds')\n",
    "print('Cost: ' + str(C))    \n",
    "print('Accuracy: ' + str(_accuracy) + ' %')    \n",
    "print('')\n",
    "print('')\n",
    "\n",
    "\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, cost_array, 'red')\n",
    "pyplot.title('Learning Curve - ' + str(len(X[0])) + ' Training Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Cost')\n",
    "pyplot.show()\n",
    "\n",
    "# plot percent accuracy curve\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, accuracy_array, 'red')\n",
    "pyplot.title('Percent Accuracy Curve - ' + str(len(X_test[0])) + ' Test Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Percent Accuracy')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated the after 1400 epochs with minibatches of 500 the cost became approximately 0.001859 and the test data accuracy reached 99.15%. These results are very good. The test accuracy is high because minibatch stochastic gradient descent inately provides a form of regularization.\n",
    "\n",
    "We now wish to explore the impact of adjusting the minibatch size. We will re-run the algorithm with a smaller minibatch size of 27 and see what the results we achieve.\n",
    "\n",
    "First we reinitialize our weights and bias's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Size: 784\n",
      "Weights Shape: (20, 784)\n",
      "Bias Shape: (20, 1)\n",
      "Velocity Weights Shape: (20, 784)\n",
      "Velocity Bias Shape: (20, 1)\n"
     ]
    }
   ],
   "source": [
    "# initialize weights & bias\n",
    "np.random.seed(10)\n",
    "print('Feature Size: ' + str(size))\n",
    "\n",
    "lower_bound = -.1\n",
    "upper_bound = .1\n",
    "\n",
    "#mean = 0.015\n",
    "#std = 0.005\n",
    "\n",
    "# hyper-parameters: hidden layers\n",
    "hidden_layers = 2\n",
    "units_array = [20, 10]\n",
    "Weights = []\n",
    "Bias = []\n",
    "V_dW = []\n",
    "V_dB = []\n",
    "for i in range(0, hidden_layers):\n",
    "    if(i == 0):\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], size]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "    else:\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], units_array[i-1]]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "    \n",
    "# output layer\n",
    "_W = np.float64(np.random.uniform(lower_bound, upper_bound, [1, units_array[i]]))\n",
    "_b = np.float64(np.random.uniform(lower_bound, upper_bound)) # b will be added in a broadcasting manner\n",
    "_V_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_V_dB = np.float64(np.zeros(1))\n",
    "Weights.append(_W)\n",
    "Bias.append(_b)\n",
    "V_dW.append(_V_dW)\n",
    "V_dB.append(_V_dB)\n",
    "\n",
    "Weights = np.array(Weights)\n",
    "Bias = np.array(Bias)\n",
    "V_dW = np.array(V_dW)\n",
    "V_dB = np.array(V_dB)\n",
    "\n",
    "for index in range(0, len(Weights) - 1):\n",
    "    Weights[index] = np.where(Weights[index] != 0, Weights[index], np.random.uniform(lower_bound, upper_bound))\n",
    "\n",
    "#print(train_X.shape)\n",
    "#print(np.ravel(train_Y).shape)\n",
    "\n",
    "print('Weights Shape: ' + str(Weights[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Bias Shape: ' + str(Bias[0].shape)) # vector with a size of the # of unit\n",
    "print('Velocity Weights Shape: ' + str(V_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Velocity Bias Shape: ' + str(V_dB[0].shape)) # vector with a size of the # of unit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we re-run our minibatch stochastic gradient descent algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Main Loop Epoch: 100\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.00040557499288796804\n",
      "Main Loop Epoch: 200\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.00022533569347979733\n",
      "Main Loop Epoch: 300\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 8.647075623703261e-09\n",
      "Main Loop Epoch: 400\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 4.046850915373965e-05\n",
      "\n",
      "Results:\n",
      "\n",
      "\n",
      "Run Time: 744.6950306892395 seconds\n",
      "Cost: 1.6063560949634133e-05\n",
      "Accuracy: 99.18 %\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEWCAYAAABSaiGHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X28HVV97/HPNydPEAgEElQSQoLEhyAI9BgerIoWJQgS2mINVQstXipXrnq1raAWFWutj1BbqtIrakEaUCtGjCKCcOXylBOeA6aEiMkxgQQS8kCeOOR3/1hrOHN29j7n5Jwz+5Bzvu/Xa7/2zJo1M2tmz57fXmvWnlFEYGZmVrURg10AMzMbHhxwzMysKRxwzMysKRxwzMysKRxwzMysKRxwzMysKRxwAEk/k3TWYJfDBp+ksyT9bKDz2s4kHS7prgFa1omSHh+IZTVYfreftaTbJJ3dYNqhkobs/08kHS3p173JO6gBR9Ljkk4czDIARMTJEfHdKpYtabykSyUtl7RJ0tI8PrGK9Q0kSZ+W9Fwud/E6pDT9SEmLJG3O70eWpknSFyQ9nV9flKSBmLeU792lcm2RtKNc1r5sc0R8NyJOHui8u0pSe96mTZKekfT/JJ1bbz80mL/fJzlJfyzpfkkbJD0l6ZeSpg7guv8B+FJpvvI2PyHpW5LG9WcbekPS+yTdUie9XdIJUO1n3R/lMg6WiLgH2CKpx/0z5Gs4kkYO4rpHAzcBhwGzgfHA8cDTwKw+LG8wtuWaiNir9FqWyzIa+DFwFTAB+C7w45wOcC5wOvBa4AjgVOCv+ztvWUR8rygXcDKwslzW2vyDeSz00cl5O6aRTswfBy5vxoolvRL4NvAhYB9gOvANYMcALX8K8IfAT2omFdt8NHAM8HcDsT6rRuk79T3qfEd3EhGD9gIeB05sMO1U4D7gGeB24IjStAuAx4CNwMPAH5emnQ38P+ASYC3pV9TZwG3Al4F1wG9JB3Yxzy3A+0rzd5d3OvB/87p/CVwGXNVgG94HPAns1c0+CODQ0vh3gH/IwycA7cDHgCeAK4FHgFNL+UcCTwFH5/Fj8/56BrgfOKEfn8+nu9m2twG/B1RKWw7MzsO3A+eWpp0D3Nnfebsp6wlAe530duBvgQeB7Tntk8Cy/BkuBk6r+cxuKe3bIH2Rlubj4Wt9zNsCXEr6sbEM+F9AdLM97bWfHXAc6YT/qjx+Guk7sjHvv78v5V2Zy7Mpv14HzAB+lcvwVD6e9mmw/rlAWzflG0EKgI/lZc0DJjRad535/wr4eXfbDHwV+HFpfGxOW0H6Xv0bMDZPOxF4vOazmFaa9yrg0918T2/p7jOozUP6AbkEWA/8M+mcc3bps74k7+fHgPPLnzWwLymYr8rruBgYUVrPrXn+Z/Kx8rZdOU5y+v7AAmBNPhZ/AkzO084E7qrJ/zHgB73dz/mzfwL4dk4/GHgWGNXd9/RFWcORdDRwBenLuz/wTWC+pDE5y2PAG0i/vD4DXCXpZaVFHEP6oA4APldKWwJMBL4IfKub5onu8l4N3J3L9Wngvd1syomkL1WfmneylwL7kT7Qc4H/JB0whZOApyLiHkmTgZ+Sgux+wN8AP5Q0qR/rf4ektZIWSzqvlH4Y8EDkoy17IKcX0+8vTbu/Zlpf5+2LuaQa0D55/L+B1+fxzwFXS3pJN/O/HfgD4CjgPT00AzfKex7peDgCaAX+ZFc3IiLuIH3J35CTNgHvydvxDuBDkk7N096Y5ylqfAsBkY6NlwEzgUOAv2+wukXA4ZK+IunNdZq2PgKcktczhXSy+Vo36651OOk7Vpekg0gn9aWl5C+TfvAdQQqe04BPNFpGVSQdAPyA9MN3Iumkf0wpy3mkH1WvJbVk/FnNIq4CtgAvJx0LpwB/WZp+POkH0v6kwPOtPhRzBPDvwFTSueM5UmAEuA54paQZpfzvIf0AgZ738xRgr7zs/wkQEb8jHV/lZe6su2hU9YsGNRzg68Bna9KWAG9qsJz7gDl5+Gxgec30s4GlpfE9Sb+AXprHb6FrDadu3ryDO4A9a345NaoF3Aj8Uw/7oKcaznbyr4ucdijpF+2eefx7wEWlXylX1iz/BuCsPn4+M4EDSb/Yjif9IjszT/t7YF5N/u+Rf0UCz5N/iefxGXlb1Z95uynrCTSu4fxFD9v5EHBKHq5Xazm2lPe/gL/pQ97/C5xTmjabXazh5PQ24GMN5vlX4Eul46Th8nOeM4CF3Uw/Hvg+qQazlfQjsDjuHqX0fQQOAraRTnS9Wfe3i+O8Zps35eM7gF+Qa2B5uVuBg0v53wA8mof7W8PpINUoyq8d1KnhkGpnt5XmH0H6bpxd+qzfV5r+9mJ/AJNJwWZMafp7gRtL6/lNadr4vC0Td+U4qZOvFVhTGv934DN5+Mj8GY/q5X7eCoyus44ngeO7K8eLsoZDisgfzRdLn5H0DOmAPhBA0l9Iuq807TWkXxqFFXWW+UQxEBGb8+BO7fw95D0QWFtKa7SuwtOkX5P9sSYitpbKs5TUrPYOSXuSmlWuzpMPBt5Zs9/+sF4Zai641+19ExEPR8TKiHg+Im4n/UI6I0/eRPoylI0nnSzqTR8PbMrfuv7M2xddPiNJZ+eL4cU+ehVdj59aT5SGN9P4uOku74E15ejuuOnOZFJTMZKOk3SLpDWS1pNOVg23Q9JLJV0r6feSNpB+3DTMHxG3R8Q7I2IiqdbyFuDCPHkq8JPSPnyQdGI8oJfbsQ7Yu076qRGxN/BHpFrtfjn9pcAYoPy5Xb8L6+vJbRGxb/lFahqsp8tnGRE7SCf+utOB35WGDyZtx5Ol7bgMKNewa48h6P6Y24mkcZL+j1JnpQ3AzXT9rL8LvDsPv4d0rfY5erefn4yI7XVWuzcpUDf0Yg04K4DP1RwAe0bEf0o6mBSdzwf2zwfGQ6RfzoW+nph6sgrYL5/oCwd1k/+XwEk99LTZTKpFFV5aM73ethTNanOAh3MQgrTfrqzZb+Mi4p9qFxClC+7R+943RQ0F0rWPI2qaJY/I6cX015amvbZmWl/n7YsX9qFSL7uvk5o9iuPnN3Q9fqqwitQUUejuuKlL0rGkE9NtOWke8EPgoIjYB/g/dG5HvePmC6RayOERMZ5Um+/VdkfE3aSmmNfkpHbgrTXH2tiIeKLBums9ALyim/XdTKqVFL3YniTV9l9ZWt8+ebtr5+3I29nd96o/VlH6/CSNoOtn22U6KTgXVpC+8/uVtmN8RBwxgOWD1NliOjArf9ZvKU+MiNty2V9POpcUzWm92c87fb75vAyp5tvQiyHgjJI0tvQaSQoo75d0jJJxkk6RtDcwjrTBawAk/SWdX4JKRWqnbAM+LWm0pONIbeeNXEk6wH4o6VWSRkjaX9LHJb0957kP+HNJLZJmA2/qRVHmkdqIz6OzdgPpC/oOSSfl5Y2VdELuEbTLJM2RNCF/BrOAD5J6l0Fqhnwe+KCkMZLOz+k35/f/AD4iabKkA4GPkn5R93fe/tqLzuNHkt5HquFU7Vrgw5IOlDSB1JGhVyTtI6moyX4nIh7Jk/Ym1bi35mA0tzTbaiBU6sae8z8LrM/XSP6mm3W+Sam78AF5/NWkY/3OnOUbwD8qd5OWdEAuY6N11/oF8Dp19kys5xLg7ZJeExHPkwLqpZIm5WNyiqS3NZj3fuDd+XtwCqmmP1CuB47M34+RwP8GytdJi896sqT9SU3dAETEClKngC8r/WVihFI38jf2ozyj65xD9yYFtnW5DBfVme9K0o+vZyPizly+Xd3PhTcBv8y1pIZeDAFnAalNs3h9OiLagP9BapNeR7pweDakZh7gK8AdpGh8OKmHSLO8m9Rb6GnSBdhrSL+mdhIR20htnr8hXc/ZQOpwMBEo/vD2IdIX+Zm87Ot6KkBErCJt//F5/UX6ClKt5+OkE+oK0omtr5/zXNK+30gKAl+I/H+lXKU+HfiLXPa/Ak4vVbW/SeoZ8yCpBvrTnNavefsrIh4gXdy+m/RL9FV0fhZV+jop0D5IuiD/U9Ivye78TOn/RMtJF6i/RGo2K5wHfF7SRtJnfm0xISI2Ap8H7spNI63Ap0gXsdcD80m1o0bWAX8MPJTLsCAv/yt5+leBnwM35fXfTuoJ12jdXUTESuDXdPODLdeWvkdnx4aPkpqn7s7b8AsaX6T+YC7/M8A78/YOiIh4EngX6fN4mlSDKR9DXyf9HeJBYCGpg0HZe0g/nB8m7efv078a2A10PYd+kvT57JPLdztQr9n8P0g/1q+sSd+V/Vx4N+lHSLfU92ZxA5B0Deki36cGuyy2+5D0DuDSiHj5YJdlsEg6HPj3iDh2sMsyHOWm/tXAayLit/1YzlHAv0REj7VIB5xdJOl1pIu2vyU1a10HHBcR9w5qwexFLX+530Cq6b4M+BFwa0Q0bNYyq5KkvyP1Eu6puWzA7G7/vH4xeCmpu+v+pAun5znYWC+I9J+fH5Cuo1xP+g+ZWdNJaif9N2dOU9frGo6ZmTXDi6HTgJmZDQNDpklt4sSJMW3atMEuhpnZbmXRokVPRUR/bn/Va0Mm4EybNo22trbBLoaZ2W5F0u96zjUw3KRmZmZNUWnAkTRb0hKlh45d0E2+MyRF+Q9iki7M8y2RdFKV5TQzs+pV1qQmqYV0U7q3kroPL5Q0P98poJxvb9K/gu8qpc0k/cv9MNKN8H4p6RX5tgtmZrYbqrKGM4t0m/9l+ZYl86jf5/uzpGfObC2lzSHdvn5b/gfsUvrwhEwzM3vxqDLgTKbrLbrbc9oL8i0RDoqI63d1XjMz271UGXDq3fa8fJv4EaS7wX50V+ctLeNcSW2S2tasWdPngpqZWfWqDDjtdH0mxBS6PtBob9KdSm+R9DhwLOkx0q29mBeAiLg8IlojonXSpKZ0Izczsz6qMuAsBGZImp6feTGX0i3CI2J9REyMiGkRMY30nI3T8qMJ5gNz87NSppNujX13hWWFW2+F3/ym0lWYmQ1nlfVSi4iO/GCtG4AW4IqIWCzpYqAtIho+nyLnu5b0vIgO4AOV91A74YRi5ZWuxsxsuBoyN+9sbW2Nft1poHja8RDZH2ZmvSFpUUTs9JC8KvhOA2Zm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hQOOGZm1hSVBhxJsyUtkbRU0gV1pr9f0oOS7pN0m6SZOX2apC05/T5J36iynGZmVr2RVS1YUgtwGfBWoB1YKGl+RDxcynZ1RHwj5z8N+CowO097LCKOrKp8ZmbWXFXWcGYBSyNiWURsB+YBc8oZImJDaXQcEBWWx8zMBlGVAWcysKI03p7TupD0AUmPAV8EPliaNF3SvZJulfSGeiuQdK6kNklta9asGciym5nZAKsy4KhO2k41mIi4LCJeDnwM+GROXgVMjYijgI8AV0saX2feyyOiNSJaJ02aNIBFNzOzgVZlwGkHDiqNTwFWdpN/HnA6QERsi4in8/Ai4DHgFRWV08zMmqDKgLMQmCFpuqTRwFxgfjmDpBml0VOAR3P6pNzpAEmHADOAZRWW1czMKlZZL7WI6JB0PnAD0AJcERGLJV0MtEXEfOB8SScCzwHrgLPy7G8ELpbUATwPvD8i1lZVVjMzq54ihkbHsNbW1mhra+v7ApQvOQ2R/WFm1huSFkVEazPW5TsNmJlZUzjgmJlZUzjgmJlZUzjgmJlZUzjgmJlZUzjgmJlZUzjgmJlZUzjgmJlZUzjggP/saWbWBA444IBjZtYEDjjggGNm1gQOOOCAY2bWBA444IBjZtYEDjjggGNm1gQOOOCAY2bWBA444IBjZtYElQYcSbMlLZG0VNIFdaa/X9KDku6TdJukmaVpF+b5lkg6qcpyOuCYmVWvsoAjqQW4DDgZmAmcWQ4o2dURcXhEHAl8EfhqnncmMBc4DJgN/FteXjUccMzMKldlDWcWsDQilkXEdmAeMKecISI2lEbHAcWZfw4wLyK2RcRvgaV5edVwwDEzq9zICpc9GVhRGm8HjqnNJOkDwEeA0cBbSvPeWTPv5DrzngucCzB16tS+l9QBx8ysclXWcFQnbacze0RcFhEvBz4GfHIX5708IlojonXSpEl9L6kDjplZ5aoMOO3AQaXxKcDKbvLPA07v47z944BjZla5KgPOQmCGpOmSRpM6AcwvZ5A0ozR6CvBoHp4PzJU0RtJ0YAZwd2UldcAxM6tcZddwIqJD0vnADUALcEVELJZ0MdAWEfOB8yWdCDwHrAPOyvMulnQt8DDQAXwgIp6vqqwOOGZm1VMMkZNta2trtLW19W3mdetgv/3S8BDZH2ZmvSFpUUS0NmNdvtMAOMiYmTWBAw444JiZNYEDDjjgmJk1gQMOOOCYmTWBAw444JiZNYEDDjjgmJk1gQMOOOCYmTWBAw444JiZNYEDDjjgmJk1gQMOOOCYmTWBA46ZmTWFAw64hmNm1gQOOOCAY2bWBA444IBjZtYEDjjggGNm1gQOOOCAY2bWBJUGHEmzJS2RtFTSBXWmf0TSw5IekHSTpINL056XdF9+za+dd0A54JiZVa6yR0xLagEuA94KtAMLJc2PiIdL2e4FWiNis6TzgC8C78rTtkTEkVWVrwsHHDOzylVZw5kFLI2IZRGxHZgHzClniIhfRcTmPHonMKXC8jTmgGNmVrkqA85kYEVpvD2nNXIO8LPS+FhJbZLulHR6vRkknZvztK1Zs6bvJXXAMTOrXGVNaoDqpNU9s0t6D9AKvKmUPDUiVko6BLhZ0oMR8ViXhUVcDlwO0Nra2veo4YBjZla5Kms47cBBpfEpwMraTJJOBD4BnBYR24r0iFiZ35cBtwBHVVZSBxwzs8pVGXAWAjMkTZc0GpgLdOltJuko4JukYLO6lD5B0pg8PBF4PVDubDCwHHDMzCpXWZNaRHRIOh+4AWgBroiIxZIuBtoiYj7wJWAv4PuSAJZHxGnAq4FvStpBCor/VNO7baALW9mizcwsqfIaDhGxAFhQk3ZRafjEBvPdDhxeZdlqVti0VZmZDVe+0wA44JiZNYEDDjjgmJk1gQMONA44mzfDvHnNLYuZ2RDlgAONA84HPwhnngm3397c8piZDUEOONA44Cxfnt43bmxeWczMhigHHGgccIp01btpgpmZ7QoHHOi504ADjplZvznggHupmZk1gQMO9NykZmZm/eaAA25SMzNrAgcccA3HzKwJHHDANRwzsyZwwAHXcMzMmqBXAUfSlb1J2235fzhmZpXrbQ3nsPKIpBbgDwa+OIPETWpmZpXrNuBIulDSRuAISRvyayOwGvhxU0rYDG5SMzOrXLcBJyI+HxF7A1+KiPH5tXdE7B8RF/a0cEmzJS2RtFTSBXWmf0TSw5IekHSTpINL086S9Gh+ndWnrest13DMzCrX2ya16yWNA5D0HklfLQeHenKz22XAycBM4ExJM2uy3Qu0RsQRwA+AL+Z59wM+BRwDzAI+JWlCL8u661yTMTOrXG8DzteBzZJeC/wd8DvgP3qYZxawNCKWRcR2YB4wp5whIn4VEZvz6J3AlDx8EnBjRKyNiHXAjcDsXpZ117lJzcyscr0NOB0REaSA8c8R8c/A3j3MMxlYURpvz2mNnAP8bFfmlXSupDZJbWvWrOmhON1wk5qZWeV6G3A2SroQeC/w09xcNqqHeeqdpeue2SW9B2gFvrQr80bE5RHRGhGtkyZN6qE43XANx8yscr0NOO8CtgF/FRFPkGobX+p+FtqBg0rjU4CVtZkknQh8AjgtIrbtyrwDxjUcM7PK9Srg5CDzPWAfSacCWyOip2s4C4EZkqZLGg3MBeaXM0g6CvgmKdisLk26AXibpAm5s8Dbclo1XMMxM6tcb+808GfA3cA7gT8D7pJ0RnfzREQHcD4pUDwCXBsRiyVdLOm0nO1LwF7A9yXdJ2l+nnct8FlS0FoIXJzTqtFTwNmxo7JVm5kNFyN7me8TwOuKWoikScAvSV2ZG4qIBcCCmrSLSsMndjPvFcAVvSxf//RUk3FNx8ys33p7DWdETZPX07sw74tfTwHFNRwzs37rbQ3n55JuAP4zj7+LmprLbq2nJjXXcMzM+q3bgCPpUOAlEfG3kv4E+ENSl+U7SJ0IhgY3qZmZVa6nZrFLgY0AEfFfEfGRiPjfpNrNpVUXrmncacDMrHI9BZxpEfFAbWJEtAHTKinRYHANx8yscj0FnLHdTNtjIAsyqHwNx8yscj0FnIWS/kdtoqRzgEXVFGkQuEnNzKxyPfVS+zDwI0nvpjPAtAKjgT+usmBN5SY1M7PKdRtwIuJJ4HhJbwZek5N/GhE3V16yZvL/cMzMKter/+FExK+AX1VclsFTDjgRO9+s0zUcM7N+Gzp3C+iP2oBTO+yAY2bWbw440HPAcZOamVm/OeBA44DTXZqZme0SBxxwDcfMrAkccMDdos3MmsABB9xpwMysCSoNOJJmS1oiaamkC+pMf6OkeyR11D5BVNLz+SmgLzwJtDJuUjMzq1xvn4ezyyS1AJcBbwXaSbfJmR8RD5eyLQfOBv6mziK2RMSRVZWvC3caMDOrXGUBB5gFLI2IZQCS5gFzgBcCTkQ8nqcNbhXCAcfMrHJVNqlNBlaUxttzWm+NldQm6U5Jp9fLIOncnKdtzZo1fSvlc8/B/fd3jtcLLm5SMzPrtyoDjuqk7UpVYWpEtAJ/Dlwq6eU7LSzi8ohojYjWSZMm9a2U69bB5z5Xf5o7DZiZDZgqA047cFBpfAqwsrczR8TK/L4MuAU4aiAL94JRo2pXvPOwazhmZv1WZcBZCMyQNF3SaGAu0KveZpImSBqThycCr6d07WdAjay5jOVrOGZmlags4EREB3A+cAPwCHBtRCyWdLGk0wAkvU5SO/BO4JuSFufZXw20SbqfdJfqf6rp3TZwelPDccAxM+u3KnupERELgAU1aReVhheSmtpq57sdOLzKsr2guxqOm9TMzAaM7zTQ0tJ13E1qZmaVcMCpfdhaPQ44Zmb95oBTy//DMTOrhANOLXcaMDOrhANOLXcaMDOrhANOLXcaMDOrhANOLTepmZlVwgGnN9ykZmbWbw44tVzDMTOrhANOLXeLNjOrhANOLXcaMDOrhANOLTepmZlVwgGnlpvUzMwq4YDTHddwzMwGjANOLTepmZlVwgGnlpvUzMwqUWnAkTRb0hJJSyVdUGf6GyXdI6lD0hk1086S9Gh+nVVlObtwDcfMrBKVBRxJLcBlwMnATOBMSTNrsi0Hzgaurpl3P+BTwDHALOBTkiZUVdYufPNOM7NKVFnDmQUsjYhlEbEdmAfMKWeIiMcj4gGg9ox+EnBjRKyNiHXAjcDsCsvaPddwzMz6rcqAMxlYURpvz2lVz9s//uOnmVklqgw49Z7d3Nszd6/mlXSupDZJbWvWrNmlwjXkJjUzs0pUGXDagYNK41OAlQM5b0RcHhGtEdE6adKkPhe0ZqGdw0WgcQ3HzKzfqgw4C4EZkqZLGg3MBeb3ct4bgLdJmpA7C7wtp1WvXsBxDcfMrN8qCzgR0QGcTwoUjwDXRsRiSRdLOg1A0usktQPvBL4paXGedy3wWVLQWghcnNOq5xqOmVklRla58IhYACyoSbuoNLyQ1FxWb94rgCuqLF+PHHDMzAaM7zRQy01qZmaVcMABUKlTnJvUzMwq4YADMLLUsuiAY2ZWCQccgJaWzmH/D8fMrBIOOOAajplZEzjgAHz3u/XT3WnAzGzAOOAA/MmfwNX5htWvehV8+ctp2DUcM7MB44BTKPdU++xn07sDjpnZgHHAKajO/ULdpGZmNmAccArdBRzXcMzM+s0Bp1Dvz58OOGZmA8YBp1CvhtPRkd7dpGZm1m8OOI08/3x6gWs4ZmYDwAGnUFvD2b69c9g1HDOzfnPAKXQXcFzDMTPrNwecggOOmVmlHHAKtQFn27bOYTepmZn1W6UBR9JsSUskLZV0QZ3pYyRdk6ffJWlaTp8maYuk+/LrG1WWMxemc3jjRpg+vXPcNRwzs36rLOBIagEuA04GZgJnSppZk+0cYF1EHApcAnyhNO2xiDgyv95fVTkbKrpEQ/0azmOPNZ73zjvh97/vmhbRtZnOzGyYqbKGMwtYGhHLImI7MA+YU5NnDlDcqvkHwB9J9f4Q0wTdrba2hnPddXDoofCTn9TPf9xx8MpXdk378pdhzBhYu7Z/5TQz201VGXAmAytK4+05rW6eiOgA1gP752nTJd0r6VZJb6i3AknnSmqT1LZmzZr+lXZXAs4dd6T3hx9uPM+zz3Yd//a30/uqVbteNjOzIaDKgFPvDF57MaRRnlXA1Ig4CvgIcLWk8TtljLg8IlojonXSpEn9LG03Aae2SW3z5vS+xx79W6eZ2TBSZcBpBw4qjU8BVjbKI2kksA+wNiK2RcTTABGxCHgMeEWFZd21Gk4RcPbcs+e8ZmYGVBtwFgIzJE2XNBqYC8yvyTMfOCsPnwHcHBEhaVLudICkQ4AZwLIKy9q3gDNmzM55y50N6ulpupnZEDWyqgVHRIek84EbgBbgiohYLOlioC0i5gPfAq6UtBRYSwpKAG8ELpbUATwPvD8iBu9qe22T2pYt6b2411pZo55oRdByTzUzG6YqCzgAEbEAWFCTdlFpeCvwzjrz/RD4YZVl20lfajj1gkf5D6P1ltFoupnZEOc7DRQaBZxRoxrXcOoFnJ5qMK7hmNkw5YBTaBRwxo7dtRqOA46ZWV0OOIXuAs6WLV3vHNBdDaenJjM3qZnZMOWAU+gu4PzsZzBlSmfTWlHDqRc83GnAzKwuB5yelLs+r16d3vvTpOYajpkNUw44he5qOIXly9P7pk3pfft2ePpp2LAhBaFrr+05oLiGY2bDlANOoVHAGT26c3jFivTfm/I1nIkT4aCD4Mwz4V3vgnvv7X4927fDggVpfU8+mdKee86ByMyGPAecQqOAM6K0i5Yv73pTziJIbNgA8/NNFH772/rLKf8P56tfTcP335/ejzwSJkzoW7nNzHYTlf7xc7fSm4Dzy1/CjBmd4/Waz5aV7sAYPOszAAAOV0lEQVQTUf/R1cUdClpa0nt3d502MxsiXMMpNLrbdDlgLFgA73hH5/jWrTvnLwecere+2bYtNaFBCj4DebPPRYv8vB0ze9FywCm8ouZm1H/+5/CjH3Wt4dR66qmd08pNauUaUBF8tm/vvIHnxo2p00GhP8EnAlpb4Y1v7PsyzMwq5IBTKHcOAPjTP4XTT+/+HmtFN+lCSws880zn+JYtsH59Gi6Cz7ZtnQFnw4auNaKiu3VfbNyY3hcv7vsyzMwq5IBT9rKXdQ6/IT9ktLuAU/QyK7zudV3HP/Qh2Hff1PRWdDCoreEUXa2ha7DaVW5KM7MXOQecsiVLUhCJ6Lym06hJbcKEnQPO4Yd3Hb/66vS+alVnDWf79s5rOBs3whNPdOYvakN9MZABx39ONbMKOOCU7b03HHBA7/Lut1/Xay6jRqX/49TT3t75Z9Ft21JTGsCaNXDPPZ35+lPDKV8L6k/guv329GfXX/+678swM6vDAacntdd2Cvvv33X8JS+ByZPr5z3xxM77sG3b1lkb+dd/hW9/uzPfQNVwli7t+3J++tP0/vOfd01fsaJr85+Z2S6qNOBImi1piaSlki6oM32MpGvy9LskTStNuzCnL5F0UpXl7FZtM1mhqLEUugs45bsIPPpo484B5RrOpk3w0Y/Cf//3zo+lXrt25x5t5RrOX/91qm2V59u+vfF6d+yAu+5Kw8UfW8tl3rEDpk6F6dPrz29m1guVBRxJLcBlwMnATOBMSTNrsp0DrIuIQ4FLgC/keWeSHjd9GDAb+Le8vOZrba2fXvxZ8zOfSe8veUk6KReuu65r/q9/Pd0Gp9yEVuvXv4YHH4S774YLL0x3JHjlK9OdCIoazC23pNrVV76S7lRw/fUpvVzDWbQoNeMtXJgCx1VXwYEHwmGHwS9+ka4bbd2aalu//z1ccgkce2wq85IlaRkPPgh33JECW1Hb2bEj3b6nuAYVkXrZlf9vtGoV/Mu/dL0OtGlTmrejo/5/l1avhh//GObNS82NmzZ17XK+dWvX+Vau7HrHh97avDkF/Oef7wzYjz4KH/5w6s5e+6C98vprr2t1dKT98Pzzqft8cdeIWjt21N/mWuU/BO+qzZvT59VTt/qIrj9Cnn22a214x470+TW6W0Z/7dgxsP87q8LGjZ3Htw04RUUHgKTjgE9HxEl5/EKAiPh8Kc8NOc8dkkYCTwCTgAvKecv5Gq2vtbU12traBn5DnnsOLroonaS/852UduqpcOed6aR4zz1w9NHw3e/Ce98Ln/tc6kzwyU/CS1+a8h9wQEr7x3+ET3wCRo5MQaRc3lNO6WzOqmfSJNhrr3SyrT35HXpouh7U0ZEeo1AEjQMOSOVft67+MqWuJ4Bx43Y+kU+dmpr6apv7Jk5M161WrUrXfCZNSie+9etTOSQ45JA0vHx5CsjbtqWu4lOnpvVGpJPQ4493nuz33LPzJP3yl6d9tXRpatqcOjVNW7YslXXixDTPli0wfnwa3rQplWHKlK49DJ97Lp1Ii5P6HnukILxyZee98fbeO23/nnumfAcfnJbxu9+lchQ12I6ONN/WrakMTz6Z9sXBB6f1FCes0aPTPlm7Nq2r0NKSXlu2pOUXeSZMSPup2De1302pc5t27EjbOn58Ctjr1qXhAw9MeXbs6DzBF8PF53jggSnvb3+bynDIIemu6KtXd9aUi309alSaVtt5Zv36tJ1jx6Z9VnSy2bEjpT/5ZPoMoHM7Vq5M+3jDhrSP99orlXXbtrR/i/1e/CAYOTK9b9mSXlu3pu/UqFH1A1exb8rvRQ/RceM6y1Lsjx07UoAZPz6Vady4VMY99kjfnfKd4iPSj7P99kvlLNbfm/ddyQuppUNKZRg1qmsZysrjr30tXHPNzvukFyQtiogGv6wHVpW3tpkMrCiNtwPHNMoTER2S1gP75/Q7a+bdqb1K0rnAuQBTy7WLgTRqFHz+8+mAP/poeM1rUm3gySfTSf6oo9Jw0dngk5/snHfTps4DGeDjH4cjjkjzveUtqUaxeTOccEIKQDfckGomY8emE9nEifDII6lGc9116QAbPz79P+iqq9Iyy1/OWbNS3uuvh332SSeh0aPh+ONhzpx0Ynj00XSLnnHjUhCdMSOt81WvSjceHTcOXv/6VONYty6VpaUl1QJuvTVt6/r1aZ3PPtt5UnnqqZR3jz3SOp9+Om3biBHpi7ttWxoeMSKdBIqTp5T+ZPv2t6dtufLKtIz99oOHHkrrmTOns9Yjpe7nO3akckV0ntghpY0Zs3PglGDu3FTG1avT/lq5MpVn333TiaelJXWNL/5fVXTuOProtJ4ikLS0wOzZaVnLl8Mxx6Qaxtat6XgpThJFs2QRyArFSbV4fPn48en4efzxlK+8b4qTZ70TV3HybmlJwW7TplT2iM59XX4VPwx+//uU94/+KJVhRf6a7rlnakJevjx9fqNHd9ZKa092Y8akeTs60vumTZ3lHTEiBZP167tux557pu/RHnt01hCL46OjI32GLS3ps4E0vZivOK5WreosS/kHRaMT/dixqXzFfi3vDykd7xs2pH25ZUv6PEaM6Lyhbnkdb35zOk6KH3z1Alyj913JO25cel+zZudab+3fNIrxQw9ld1BlDeedwEkR8b48/l5gVkT8r1KexTlPex5/DJgFXAzcERFX5fRvAQsi4oeN1ldZDcfMbAhrZg2nyk4D7UC5n/AUYGWjPLlJbR9gbS/nNTOz3UiVAWchMEPSdEmjSZ0A5tfkmQ+clYfPAG6OVOWaD8zNvdimAzOAuyssq5mZVayyazj5msz5wA1AC3BFRCyWdDHQFhHzgW8BV0paSqrZzM3zLpZ0LfAw0AF8ICL62IXHzMxeDCq7htNsvoZjZrbrhso1HDMzsxc44JiZWVM44JiZWVM44JiZWVMMmU4DktYAv+vj7BOBOs+LHtK8zcPDcNxmGJ7b3ddtPjgiJg10YeoZMgGnPyS1NauXxouFt3l4GI7bDMNzu3eHbXaTmpmZNYUDjpmZNYUDTnL5YBdgEHibh4fhuM0wPLf7Rb/NvoZjZmZN4RqOmZk1hQOOmZk1xbAPOJJmS1oiaamkCwa7PANF0hWSVkt6qJS2n6QbJT2a3yfkdEn6Wt4HD0g6evBK3neSDpL0K0mPSFos6UM5fchut6Sxku6WdH/e5s/k9OmS7srbfE1+RAj5kR/X5G2+S9K0wSx/f0hqkXSvpOvz+JDeZkmPS3pQ0n2S2nLabnVsD+uAI6kFuAw4GZgJnClp5uCWasB8B5hdk3YBcFNEzABuyuOQtn9Gfp0LfL1JZRxoHcBHI+LVwLHAB/LnOZS3exvwloh4LXAkMFvSscAXgEvyNq8Dzsn5zwHWRcShwCU53+7qQ8AjpfHhsM1vjogjS/+32b2O7YgYti/gOOCG0viFwIWDXa4B3L5pwEOl8SXAy/Lwy4AlefibwJn18u3OL+DHwFuHy3YDewL3AMeQ/nE+Mqe/cJyTnk91XB4emfNpsMveh22dQjrBvgW4HtAw2ObHgYk1abvVsT2sazjAZGBFabw9pw1VL4mIVQD5/YCcPuT2Q242OQq4iyG+3blp6T5gNXAj8BjwTER05Czl7Xphm/P09cD+zS3xgLgU+DtgRx7fn6G/zQH8QtIiSefmtN3q2K7siZ+7CdVJG479xIfUfpC0F/BD4MMRsUGqt3kpa5203W67Iz0N90hJ+wI/Al5dL1t+3+23WdKpwOqIWCTphCK5TtYhs83Z6yNipaQDgBsl/aabvC/KbR7uNZx24KDS+BRg5SCVpRmelPQygPy+OqcPmf0gaRQp2HwvIv4rJw/57QaIiGeAW0jXr/aVVPygLG/XC9ucp+9Derz77uT1wGmSHgfmkZrVLmVobzMRsTK/ryb9sJjFbnZsD/eAsxCYkXu3jAbmAvMHuUxVmg+clYfPIl3jKNL/IvdsORZYX1TTdydKVZlvAY9ExFdLk4bsdkualGs2SNoDOJF0If1XwBk5W+02F/viDODmyI38u4uIuDAipkTENNJ39uaIeDdDeJsljZO0dzEMvA14iN3t2B7si0iD/QLeDvw3qd37E4NdngHcrv8EVgHPkX7tnENqt74JeDS/75fzitRb7zHgQaB1sMvfx23+Q1KzwQPAffn19qG83cARwL15mx8CLsrphwB3A0uB7wNjcvrYPL40Tz9ksLehn9t/AnD9UN/mvG3359fi4ly1ux3bvrWNmZk1xXBvUjMzsyZxwDEzs6ZwwDEzs6ZwwDEzs6ZwwDEzs6ZwwDHrgaTn8x16i9eA3VVc0jSV7uhtNpQN91vbmPXGlog4crALYba7cw3HrI/y80m+kJ9Hc7ekQ3P6wZJuys8huUnS1Jz+Ekk/ys+uuV/S8XlRLZL+PT/P5hf5jgFI+qCkh/Ny5g3SZpoNGAccs57tUdOk9q7StA0RMQv4V9L9vMjD/xERRwDfA76W078G3Brp2TVHk/4xDumZJZdFxGHAM8Cf5vQLgKPyct5f1caZNYvvNGDWA0mbImKvOumPkx5+tizfNPSJiNhf0lOkZ488l9NXRcRESWuAKRGxrbSMacCNkR6ghaSPAaMi4h8k/RzYBFwHXBcRmyreVLNKuYZj1j/RYLhRnnq2lYafp/Pa6imk+2H9AbCodCdks92SA45Z/7yr9H5HHr6ddBdjgHcDt+Xhm4Dz4IWHpo1vtFBJI4CDIuJXpAeN7QvsVMsy2534F5NZz/bIT9Qs/Dwiiq7RYyTdRfrxdmZO+yBwhaS/BdYAf5nTPwRcLukcUk3mPNIdvetpAa6StA/pzr+XRHrejdluy9dwzPooX8NpjYinBrssZrsDN6mZmVlTuIZjZmZN4RqOmZk1hQOOmZk1hQOOmZk1hQOOmZk1hQOOmZk1xf8Ht2bJ3+AfMHMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAEWCAYAAADPZygPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm8HHWZ7/HPl4SQhUB2CZAQkCggawybCHIVHUBEcEcZdxb1IijuznX0jgugwuioKAqKiOuAyOAIAgJeBgGDEAgECCgYICEJyQEC2fPcP55fezonZ+mcnE5Xzvm+X69+dXetz6+qup76/aq6ShGBmZlZK2zR6gDMzGzgchIyM7OWcRIyM7OWcRIyM7OWcRIyM7OWcRIyM7OWcRIys8pQ+rOkPfpoevMlvbwvptXJtLeStFTS9l30P1XSdd2Mf6ukE5sRW6uV9XinpKk9DdtjEpL0iKRlZWE/KemHkrbum1D7RonxiAaG21nSWknf2RRxtYqkiZIulDRP0rOS7pf0BUkjWh1bTyS9RdItkp6XdGMn/feVdEfpf4ekfev6SdLZkp4qr3MkqS/GrRvuHeW3sLT8LtbWfV+6EeXeTdLqHoY5S9Kqsk5r6/UbkiZswHw2ascnaYqkKyQtkvS0pLslvb0P5/0m4PGIuK+MUyvzUkltkm6WNL238TdK0lBJIWnHDt3PkvQDgIhYERFbR8QTzY5nQ9TH2CqRf0A9D/h8T8M2WhN6XURsDUwD9gf+ZUODkjR4Q8dpgncCS4C3SdpqU854U5Vf0hjgT8Aw4OCIGAm8GhgFvLAX09vU620x8O/AWZ3EMgT4DfATYDRwMfCb0h3gZOA4YB9gb+AY4JSNHbdeRFxadjxbA0cBT9S+l27NdnFZp2OBNwNTgBmSxm+CeQP8DHgAmASMA94DLOrD6Z8KXNKh28Vl2Y4HbgV+0Yfzsz5Wt8+4HHitpLHdjhAR3b6AR4Aj6r5/FbiqfN4WuBCYBzwOfBEYVPq9G/gfMhsuBr5Yup8EzAaeBe4DppXu2wOXAQuBvwEfrpvn54FfAj8u490LTC/9LgHWAsuApcAnuinLw8AHgCeBN3Xo9xLg2hLrk8BnSvdBwGfKuM8Cd5A/wClAAIPrpnEj8P6uyk8mgT8AT5E/3EuBUXXjTyorbmEZ5lvAVmX8veqGm1DKO76TMn4RuAfYootlsKFxfwVoA/asG358mf+E8v0Y4K4y3C3A3j1tVw1sd+8HbuzQ7TVlO1Ndt78DR5bPtwAn1/V7H3Drxo7bTYyHA4910n0SmfAWAX8FTq3rdwhwJ/AMMB/4Sum+oKyXpeW1XyfTPQv4QYduW5K/p9rvazzwu7INLS5xTCz9vg6sAZaXeXy9dD8feKzEdDtwUBflFbAK2K2bZXIocFvZFv4CHNLdvDuMO7xMf1xXZSYPhAMYWdfteODuMs//B+xR128+8PLy+efAv9T1OxJ4qItyDC3z2bGrddBxGPJ3+d9lOf6J/O1cVzfua4E5Jc5zyYR6Yl3/U8gEvxj4LbBDh/mcRO6HlgDndbMO1ttO6vp9jty/PgvMAl5bt+yfAabWDbsj8DxlH9XAcv4YuW9+vq77/wPe2t3vaIPOCUmaBBxN/oggjyZXA7sC+5E/9PfXjXIg+SOcAHxJ0pvJhPJOYBvgWOApSVsA/wXMBHYAXgWcIemf6qZ1LLkRjQKuJHfQRMQ/kzuT10UejZ7TReyHkgv152RCe2ddv5HAdcDVZDLcFbi+9P4ocEIp9zbAe8kV04h1yk/+iL9S5rE7ubP6fIlhEHAV8CiZKHYAfh4RK0rM9c0YJ5Ab98JO5nkEcHlErG0wxp7i/r9kYjyhrv9bgJsiYoGkacBF5A9oLPA94Mom1TRfAtwdZesu7i7da/1n1vWb2aFfb8dtWFmP/00mte3JHd1nJL2iDPIt4MsRsQ0wFbiidD8MWBPttao7aUBErCJ/O4eWTlsA3wUmAzuXbueVYc8E/kwecGxdvkPuMPci199vgF9J2rKTeQWZYL5Xmk07NlVNKeX5LDCGbDG5QtLobuZdb3fgmYjotGZVtql/Jnd4S0u3g4DvkDWyseRB6RUtanm5gEwgLyAPdt9b6yFpO3K/cyZ5oLAQmF7X/23AGcDryvh3krX2ekeR+9lpwHskHd6LGB8AXkZWIM4Gfi5pXEQ8D/wn6+5n3gH8NiLaGlzObyVbXeprPrPJ1oWudZehyu/1EXKFt5E7yO+QTT0vAFYAw+qGPQG4oXx+N/D3DtO6Bji9k3kc2MmwnwZ+WD5/nnWPKPYAlnWI8YgeyvED4Iry+WDyiGtCXdx3djHeA8DrO+k+hZ5rFH/vIabjavMtMS2sn16H5TOXUrsBZgBv6WKac6g78u6LuMnE9te67/8DvLN8Ph/4t06W2St62rZ6WDad1YT+D5mY67tdCny+fF5D3VE6uZMPMvn3etxuYjycDjUh4BXAnA7dvgCcXz7fTu6kx3YYZjdgdQ/LpNMjXHLndU8X4xwEzKv7vs7RdyfDizzIenEX/ceRrSGzyRaIGZRaG/CvwPc7DH8T5Ui4gXm/CnikkzKvIPc/a8ga4yF1/X8IfLbDOI8CB5bPG1sTerrMu/ZaTic1ofJ5LTClbhrnUvZbZHPvjXX9BpWynFi+3wC8o67/luQ+6gV185le1/9K4IwN2U66GPZ+4J/qtt2H6vrdAxy7Acv57Z1M/+vAd7qLodGa0HERMSoidoqID0bEMmCnsqDmlROGbeRRcP1J0rkdpjOJrE52tBOwfW06ZVqfKSugZn7d5+eBoY0e7UgaRrafXwoQEX8ia0+1E6pdxdVTv56sU35JEyT9XNLjkp4hj3TG1c3n0YhY7+R0RNwGPAe8QtJuZE3tyi7m+RQwsZfxdho32YQ4TNKBknYC9gV+XfrtBJzZYd1NImsB65D0mbqT+N/tRVxLydpovW3IpoXO+m8DLI38NWzMuBtiJ2BKh+XxUWC70v9d5DmnByXd1qG231s7kEfgSBop6SJJfy/b2O9p38Y6JenTkh6Q9DTZ1DO0q3EiYlFEfDwidi9lepCsKUOW/cQOZZ9OJ9tCF5YAIzvpfklEjCK364fJ7a9mJ7KmWT/P8eQy6QsvKfu+USWGf+9iuO3IBF7/23m07vP29f0iYg3ZPFyzE/DdujIsJFuZ6mubHfeBG3wOUtL7ysUktfnsSvu6/iMwSNLByot2JpJNu7X4elrOHfcbkOuzrbuYNuYS7bnkEcq4upW0TUTUN2F0/AHPpfOT43OBv9Wv7IgYGRFHNxhLTzuK48mdyneUl2zOJxderUmuq7i66/dceR9e1227DsN0jOsrpdvekc0xJ5Ibbm0+k7tJrBeX4f8Z+M+IWN7FcNcBx5cmzs5scNyRTXu/JGuMbyfPCdZ23nOBL3VYd8Mj4mcdZxwRX4725qZTu4ivO/cCe0vrXLW2d+le619f9d+nQ7/ejrsh5gL3d7ItHw8QEbMj4q3kwdo3gcvLxREbmuyAf5wEPoZsewf4FLnj2r9sY6+hfRuj43wkvRo4jfyNjCKb0ZZ1GKdTEbGAPNqforzyci55BF5f9hERcV5n8+7EbGCkpK4S4AKy2fcrdcPMBT7XyfZ3eSeTeI7ut/uNMZ8s36S6bpPrPs+r71d+nx134O/uUI5hEXFHXwUo6UXAf5C1sjElqT5EWdflgOvHtO9nfh7Z3FuLr6fl3Nn63Z11m7nX0+skFBHzyKOsr0vaRtIWkl5Y1/bdmR8AH5P0UqVdy5H17cAzkj4paZikQZL2lLR/g+E8CezSTf93kect9iKPovYlTxDvK2kv8lzMdpLOUF77P1LSgXUx/5ukqSXmvSWNjTwf8zh55DdI0nvp+eqzkZSmTUk7AB+v63c7uaGeJWmE8hLRQ+r6X0LuKE4kN5SunEsm3IvLskXSDpLOlbR3L+MG+CnZ5vuO8rnm+8CppZakEvtrlefZNliJaSgwGNiiLIfa+YkbySaZD5f19L9L9z+U9x8DHy3l3Z5sf/9RH4y7IW4u5TijxD64bDPTSvd3lu1nDdnUE2QzzgLyKHRyl1OuI2lLSS8hDw5GkgmN8vl5chsbx/pXsnb8rYwkm30WAkPIc4BDu5nv1yTtUdbTtuTVbLMi4jnyQOnNkl5V+g8rn2s7+25/p6WF5Uby/FhXw9xNHrHXzildAJwmaXrZ/raWdKyk4Z2MfhdwjKRR5fd3Wlfz2VDloPC/gC+Ucu9N/lZqrgT2l3RM2Z4/Tib8mu8C/yLpxQCSRkt640aENKhsf7XXELLmtJZc11tIOpWsCdX7MXnO9wTW3c9syHKmlGEEuc+9vqthgA2/Oq5Dv21pv7LmafJk2ttKv3cDN3cyzqnkOYOl5NUZtfbk7cnLP+eT1fJba/Mlzwn9pG4aU6g7rwG8nmxeawM+1mF+O5DV2r06ieW/ga+Vz3uWhbWkxPCpaG+7/Rfaryj5M+1XwxxVureRbZ83se65lZs7zO8l5NV1S8kfxJnUnVMgj5yuoP3quW92GP+6sj66PE9RtywvKuV4lmz3/VdgeG/irpvuQ2Szz5AO3Y8sy6WNTKS/ou7qpQ15lflHh9eP6vrvV5bhMvLqq/3q+gk4p8S4uHxWX4zbRayH0/XVcb8kd7pLyHNoh5V+vyzr9lmyzf3ouvHOJncQbcC+nUz3LDJhPEse1T9IHtlO7LAN3Vy2sfuBD1J3ronS7l/iOodsUr+EvDLqcfL80j/Oo3QSwwVl/KUl1t+w7hVVh5T5LyET65XA9p3Nu4vpvxH4dYcyd7wi8BUl3jHl+7FlvT4NPEGe+xlW+tWfExpBNh0+Q+6rPkbfXh23HXlxU1dXx72ulL+rq+PeR9a+nyGb8r7bVSx0OL/VSYwdf0MPlX5fK8t/Ydne1jtPV9bfA51Mt6HlXDf8PwM/7ek3rzKwbQYkXUT+L2WD/6dltjkozaW3kU1T97U6noFI0k+B+yLiixsxDZEJ620R8WC3wzoJbR6Ul7/eRR69/6210ZhZfyRpV7KVYPeIeLyn4fuC7x23GZD0b2TT5VedgMysGSSdQzZT/t9NlYDANSEzM2sh14TMzKxlqnBT0YaMGzcupkyZ0uowzMw2G3fccceiiNhUN7ftlc0mCU2ZMoUZM2a0Ogwzs82GpEd7Hqq13BxnZmYt4yRkZmYt4yRkZmYt4yRkZmYt4yRkZmYt4yRkZmYt4yRkZmYt4yTULGvWwKxZPQ/39NPwyCPrd1+7tuvp3n03RMDq1euPs3w5zJyZw9UsWwarVrGeCLj8cvjqV+HSS/N7BNx7LyxZAj/6ETz1VM9l6MwNN8Cf/9x5vxUr8tWT+mXQ1gaPPQZz58KiRVm+Wry15dBxedSG6Wx6kP3uumvdYXpSv1zNbKNtNn9WrZzvfhf23x9e+tLO+590Evzwh/DXv8LOO2e3+++HK6+EF70IzjsPDjgAbr0Vbr4ZPvpROOYYuO8+mDYNjj8e3vte+MAHYNKkTApf/CL89Kcwfz6MHp3d9tkHpk/PZLJkSfv8x4/P+S5dCg88AKNG5fyk7DdnDmy5Jdx0U/s4X/oSbLttxlRz6KEZz29/C1ttBS97WQ6zaFFOY9q0TA4rV8KRR+b8fvpTuOeeHP/EE2Hx4pzvs8/mMliwIKf16lfD0KE5zstfDs89ByNGwNVXw7x58PDDMHIknHYa/OAHWb4ttshxli+HXXeF7baDW26ByZNzWX/kI/DiF2cCvesuGDIkh5k7F555Bo4+OvvffjsMGgR//COccw6MHZvfAf7nf2DrrTMBz50Lhx0Gr3tdlukjH4GjjoIpU3K91JJ0W1vO7/DD4dFHYZttsgxPPpnrcvx4eP759iTW1gZXXZXTnjw5DxTuvx8GD85XRB6g1EyalN2GD8/lt3YtbL99roPRo2HCBJg9O7v/7W/w+OOw++65POfPz3lMnJjbxKxZuZ1NmJDLJiKXzTbb5LpYvDjfd9gh1xvkeDfcAOPG5bLae+/sd889WZZHHoGpU3P91Js6FRYuzLLtsENurw8/nAchy5fndvn447k9TZqUy3u33XL4iRMzJshyTJ2ay/DRR3N5rVyZ/XbZJct94425PHfYIee5Zk2W75JL4FWvyml3PBCR8vc4Zsy63efOzXW3994Zw1Zb5bjPPJPzP+SQXE4zZ+bvaPLk3H5GjIA998zl0daWy0PKg8D7788DpbFjc3vYY49cN3fdlb+F0aNz3n/5S247Q4bAgw/Chz+cy6Sf2mxuYDp9+vSozB0T7r47f0w77ZQ7vosvzh3Z97+fO683vSl/aJAb98qVueEtWZIbMeQGuyFH1bWks8UW+WM45hi45prOazif/CQ88UTuaMaPh5e8JHe6Dz2UG/e4cfnjfPbZTBJTp8KMGfCtb+XO5vWvz53C9dfn8DUvelHu9CJyBzF2bP5IxozJH+mj5c/ZI0fmtPfdN6cztDyoc+ut84c7aBC85jWZrJYuze8dazFdOfhg+NOf8vPw4blT2muv/JHecUeuD8gkMXhwLqtaN8idXltbxt5VLW/bbdsTwNZbZ3Ls6XdSSx61HWtbW2Pl6c7227ev73nz2mMYMiS7L+/whPfx43M9PPtse/wTJ+Y6GDIkh1+zJpfb5Mm5A6ztKLfbLr+PHJnjjhvXvt5WrcodMuQOdcSIXG+Qn8eMyWXWscyrV+c0hwzJ7bc2DYBhw3Ln/uSTWc4FC7Lb4MG5nQ8dmjEsXtxe263f1keOzHmuWdO+jY4YkeuqK2PGZNnrPf10Lq/ObLVVYzX2Zhs9OpNaLSFvAEl3RMT0vg+q7zgJbahf/zp33M8/v2HjTZ2aO72Xvxz22y9rONuVpx6fdloeBc6fD7/4Re4gPv3p/JGdckoeQR1xBHziE1kTGT48dwbLl2eT12GHwVvfCmeemUeVxx3XdRyrVuUPvXaEW2/p0tzp1kTAbbflj2DSpNxJPPRQTmOPPXKYZctyJyPlzmDLLfMIe8WK/BF3tGZNvoYMyRpUW1smjDvvzB3P+PHZff/9cwe0bFnOd9y4HG/o0Ixh2bJMPG1tOU4t3ltuyW7/9E9ZzlqMCxbkznbSpPYd6HXX5bRnz84a0ooVuUM66KBcv0OGZHkefzxrh8OG5XRnzsx4Bg3KbpDD1WopEyfmznDIkIxpwYKsnYwb174jWbAgd5rbbJPlknKHOHRojr9mzbrrYsmSHGblyhxviy3ygObvf2+vsWy3Xb5HZJkHD84Y1q7N6S1bluPUEsyqVe3ro7YDHzYsd/hDhqy73p58MpOrlK9aQpkwYf1h69XKOWJE+/ocMyaX3ZAh7dvc2rX5Gjw4fwfjxuXnWm1n5cr2hDl6dPbbsjz1fdGi7D9xYvs8arXlOXPyN/PUU7l8OtbUVq7MbbxWk6ltL4sXZyvH/Pn5WrIk491ll5zvLbfk9v3yl2et65lnclnMm5cJY8yYjHPVqizXsGFZc1+7NmuPY8fmQeKgQbmt3357e5IdPrx9mbz4xbntveENXS/jbjgJ9aGmJ6HnnsuNadGi3BFtuy2ccUZuKCNGZG1i/Hj4cXnsev2R9CGHwDe+Ab/7XVbjL7hg3WmvWbP+xg+54W25ZSalrixbljvEsWM77x+RifGII3p1pGRm/ZeTUB9qehL64Afh/PPX7bbDDpl47ror29dnz86jwBtvzKOgo4/OxHPAAeuON2JEJo6HH84jtsmTmxe3mVkXNock5AsTam68cd3vBx6YNYzx47PJY5ddMqksX561Iuj6nMKsWe3jmJlZl5yEbroJLrssazk1l122bhtsLZm88IWNTXPnnduviDMzsy4N7CS0dGmeaK5dAfPxj+e5oSOPbG1cZmYDxMBOQn/5Syagn/0sr7g65ZT2K63MzKzpBnYS+sUv8v2Vr4S3va21sZiZDUAD97Y9t94K3/lO/hdgwoRWR2NmNiAN3CT0k5/kf3d+97tWR2JmNmANzCR05pnw7W/n/3x8DsjMrGUGXhJ68kk491w49ti8waiZmbXMwEtCtccmnHRS3p/KzMxaZuAmoSlTWhmFmZkxkJPQTju1NAwzMxuoSWjs2HweiZmZtdTASkIRea+4vfZqdSRmZsZAS0L33Zc3Kn3LW1odiZmZMdCS0B//mO9HHdXaOMzMDBhoSej22/MWPb4owcysEgZeEtp//3w6qpmZtdzASUKrV8ODD/qiBDOzChk4SejRRzMRTZ3a6kjMzKxoehKSdLqkWZLulXRG6bavpFsl3SVphqQDmh0Hc+bk+667Nn1WZmbWmKYmIUl7AicBBwD7AMdImgqcA3whIvYFPle+N1ctCbkmZGZWGc1+suruwK0R8TyApJuA44EAtinDbAs80eQ4YN48GDQIttuu6bMyM7PGNDsJzQK+JGkssAw4GpgBnAFcI+lrZG3sZZ2NLOlk4GSAyZMnb1wkS5bA6NG+Ms7MrEKa2hwXEbOBs4FrgauBmcBq4APARyJiEvAR4MIuxr8gIqZHxPTxG/vwubY2GDVq46ZhZmZ9qukXJkTEhRExLSIOAxYDc4B3AZeXQX5FnjNqLichM7PK2RRXx00o75OBNwA/I88BvaIM8koyMTVXW1s2x5mZWWU0+5wQwGXlnNAq4EMRsUTSScA3JA0GllPO+zTVkiUwaVLTZ2NmZo1rehKKiEM76XYz8NJmz3sdbo4zM6ucgXPHBDfHmZlVzsBIQsuXw4oVrgmZmVXMwEhCS5bku5OQmVmlDIwk1NaW726OMzOrlIGVhFwTMjOrFCchMzNrmYGRhGrnhNwcZ2ZWKQMjCbkmZGZWSU5CZmbWMgMnCQ0bBltt1epIzMyszsBIQkuWuBZkZlZBAyMJ+b5xZmaVNDCS0NNPw7bbtjoKMzPrYGAkoeeegxEjWh2FmZl14CRkZmYt4yRkZmYt4yRkZmYt4yRkZmYt0/+TUISTkJlZRfX/JLRyJaxZ4yRkZlZB/T8JPfdcvjsJmZlVjpOQmZm1jJOQmZm1TENJSNJlkl4rafNLWk5CZmaV1WhSOR94OzBH0lmSdmtiTH2rloSGD29tHGZmtp6GklBEXBcR7wCmAY8A10q6RdJ7JG3ZzAA3mmtCZmaV1XDzmqSxwLuB9wN3At8gk9K1TYmsryxblu+uCZmZVc7gRgaSdDmwG3AJ8LqImFd6/ULSjGYF1yfWrMn3QYNaG4eZma2noSQEfCsi/tBZj4iY3ofx9L2IfJdaG4eZma2n0ea43SX949GkkkZL+mCTYupbTkJmZpXVaBI6KSLaal8iYglwUnNC6mO1JLTF5nd1uZlZf9fonnkLqb0qIWkQMKQ5IfWxtWvz3TUhM7PKafSc0DXALyV9FwjgVODqpkXVl9wcZ2ZWWY0moU8CpwAfAAT8HvhBs4LqU05CZmaV1VASioi15F0Tzm9uOE3gJGRmVlmN3jtuqqT/lHSfpL/WXg2Md7qkWZLulXRGXffTJD1Qup+zMQXokZOQmVllNdoc90PgX4HzgP8FvIdsluuSpD3JK+gOAFYCV0v6LbAj8Hpg74hYIWlCL2NvjJOQmVllNXp13LCIuB5QRDwaEZ8HXtnDOLsDt0bE8xGxGrgJOJ48r3RWRKwAiIgFvQu9QU5CZmaV1WgSWl4e4zBH0v+WdDzQUw1mFnCYpLGShgNHA5OAFwGHSrpN0k2S9u919I2oXaLt/wmZmVVOo81xZwDDgQ8D/0Y2yb2ruxEiYraks8kbnC4FZgKryzxHAwcB+5OXfu8SUauytJN0MnAywOTJkxsMdb1AahPr3fhmZtY0PVYPyh9T3xIRSyPisYh4T0S8MSJu7WnciLgwIqZFxGHAYmAO8BhweaTbgbXAuC7GvyAipkfE9PHjx29QweomUitI78Y3M7Om6bEmFBFrJL1UkjqrrXRH0oSIWCBpMvAG4GAy6bwSuFHSi8g7LyzqReyNcRIyM6usRpvj7gR+I+lXwHO1jhFxeQ/jXVaeQ7QK+FBELJF0EXCRpFnkVXPv2tDktkGchMzMKqvRJDQGeIp1r4gLoNskFBGHdtJtJXBiowFuNCchM7PKavSOCe9pdiBN4yRkZlZZjT5Z9YdkzWcdEfHePo+or/lRDmZmldVoc9xVdZ+Hkn86faLvw2kCP8rBzKyyGm2Ou6z+u6SfAdc1JaK+5uY4M7PK6m0b1VSgl/8e3cSchMzMKqvRc0LPsu45ofnkM4aqz0nIzKyyGm2OG9nsQJrGScjMrLIafZ7Q8ZK2rfs+StJxzQurDzkJmZlVVqPnhP41Ip6ufYmINvL5QtXnS7TNzCqr0T1zZ8M1enl3a/kSbTOzymo0Cc2QdK6kF0raRdJ5wB3NDKzPuDnOzKyyGk1Cp5E3G/0F8EtgGfChZgXVp5yEzMwqq9Gr454DPtXkWJrDScjMrLIavTruWkmj6r6PlnRN88LqQ05CZmaV1Whz3LhyRRwAEbEEmNCckPqYk5CZWWU1moTWlqejAiBpJzq5q3Yl+RJtM7PKavQy688CN0u6qXw/DDilOSH1MV+ibWZWWY1emHC1pGnAQYCAj0TEoqZG1lfcHGdmVlkNt1FFxKKIuAq4DzhV0qzmhdWHnITMzCqr0avjJko6Q9LtwL3AIOCEpkbWV5yEzMwqq9skJOkkSX8AbgLGAe8H5kXEFyLink0R4EZzEjIzq6yezgl9G/gT8PaImAEgafO4Kq7GScjMrLJ6SkLbA28GzpX0AvKWPVs2Paq+5CRkZlZZ3TbHlYsRzo+Iw4BXAU8DCyTNlvTlTRLhxopwAjIzq6gNuTrusYj4WkS8FDgOWNG8sPrQ2rVOQmZmFdWrZwJFxAPAF/o4luZwTcjMrLL6/71snITMzCrLScjMzFqm0T+rXt9It0pyEjIzq6xuzwlJGgoMB8ZJGk3eNw5gG/Ly7epzEjIzq6yeLkw4BTiDTDh30J6EniH/yFp9EX6Mg5lZRXWbhCLiG8A3JJ0WEf+xiWLqW75E28ysshp9lMN/SHoZMKV+nIj4cZPi6jtujjMzq6yGkpCkS4AXAncBa0rnAJyEzMys1xr9s+p0YI+I2LxuXgpOQmZmFdboGftZwHbNDKRpnITMzCqr0SQ0DrhP0jWSrqy9GhlR0umSZkm6V9IZHfp9TFJIGrehgTfMScjMrLIrxO8PAAAMqklEQVQabY77fG8mLmlP4CTgAGAlcLWk30bEHEmTgFcDf+/NtBvmS7TNzCqrob1zRNwEPAJsWT7/GfhLA6PuDtwaEc9HxGryCa3Hl37nAZ8gL3BoHl+ibWZWWY3etuck4D+B75VOOwBXNDDqLOAwSWMlDQeOBiZJOhZ4PCJm9jDfkyXNkDRj4cKFjYS6PjfHmZlVVqPtVB8CDiHvlEBEzAEm9DRSRMwGzgauBa4GZgKrgc8Cn2tg/AsiYnpETB8/fnyDoa43ESchM7OKajQJrYiIlbUvkgbTYDNaRFwYEdPK01kXk816OwMzJT0C7Aj8RVJzrr5zEjIzq6xGk9BNkj4DDJP0auBXwH81MqKkCeV9MvAG4McRMSEipkTEFOAxYFpEzN/g6BvhJGRmVlmNXh33KeB9wD3kTU3/G/hBg+NeJmkssAr4UEQs2eAoN4aTkJlZZTWahIYBF0XE9wEkDSrdnu9pxIg4tIf+UxqMoXechMzMKqvR5rjryaRTMwy4ru/DaYK1a/0/ITOzimp07zw0IpbWvpTPw5sTUh9zTcjMrLIaTULPSZpW+yLppcCy5oTUx5yEzMwqq9FzQqcDv5L0RPk+EXhrc0LqY05CZmaV1WMSkrQFMATYDXgx+Yjv+yNiVZNj6xtOQmZmldVjEoqItZK+HhEHk7fh2bw4CZmZVVaj54R+L+mN0ma4N3cSMjOrrEbPCX0UGAGskbSMbJKLiNimaZH1FT/KwcysshpKQhExstmBNI0f5WBmVlmNPspBkk6U9H/K90mSDmhuaH3EzXFmZpXVaDvVd4CDgbeX70uBbzclor7mJGRmVlmNnhM6MCKmSboTICKWSBrSxLj6jpOQmVllNVoTWlVuWhoAksYDa5sWVV9yEjIzq6xGk9A3gV8DEyR9CbgZ+HLToupLTkJmZpXV6NVxl0q6A3gVeXn2ceXR3dXnS7TNzCqr2yQkaShwKrAr+UC770XE6k0RWJ/xJdpmZpXVUxXhYmA6mYCOAr7W9Ij6mpvjzMwqq6fmuD0iYi8ASRcCtzc/pD7mJGRmVlk91YT+cafsza4ZrsZJyMyssnqqCe0j6ZnyWcCw8n3zuneck5CZWSV1m4QiYtCmCqRpnITMzCqr/1+77CRkZlZZ/T8JrV3r/wmZmVVU/987uyZkZlZZTkJmZtYyTkJmZtYyTkJmZtYyTkJmZtYyTkJmZtYyAyMJ+RJtM7NK6v97Zz/Kwcyssvp/EnJznJlZZTkJmZlZyzgJmZlZyzgJmZlZyzgJmZlZyzQ1CUk6XdIsSfdKOqN0+6qk+yXdLenXkkY1MwZfom1mVl1N2ztL2hM4CTgA2Ac4RtJU4Fpgz4jYG3gQ+HSzYgB8ibaZWYU1s4qwO3BrRDwfEauBm4DjI+L35TvArcCOTYzBzXFmZhXWzCQ0CzhM0lhJw4GjgUkdhnkv8LuuJiDpZEkzJM1YuHBh76JwEjIzq6ymJaGImA2cTTa/XQ3MBGo1ICR9tny/tJtpXBAR0yNi+vjx43sbiJOQmVlFNfWMfURcGBHTIuIwYDEwB0DSu4BjgHdERDQzBichM7PqGtzMiUuaEBELJE0G3gAcLOlI4JPAKyLi+WbOH3ASMjOrsKYmIeAySWOBVcCHImKJpG8BWwHXKpPDrRFxatMicBIyM6uspiahiDi0k267NnOe61m71v8TMjOrqP6/d3ZNyMysspyEzMysZZyEzMysZZyEzMysZZyEzMysZZyEzMysZQZGEvIl2mZmldT/985+lIOZWWX1/yTk5jgzs8pyEjIzs5ZxEjIzs5ZxEjIzs5ZxEjIzs5YZGEnIl2ibmVVS/987+xJtM7PK6v9JyM1xZmaV5SRkZmYt4yRkZmYt4yRkZmYt4yRkZmYtMzCSkC/RNjOrpP6/d/Yl2mZmldX/k5Cb48zMKstJyMzMWsZJyMzMWsZJyMzMWsZJyMzMWsZJyMzMWmZgJCH/T8jMrJL6/97Z/xMyM6us/p+EjjsO9tuv1VGYmVknBrc6gKb70Y9aHYGZmXWh/9eEzMysspyEzMysZZyEzMysZZqehCSdLmmWpHslnVG6jZF0raQ55X10s+MwM7PqaWoSkrQncBJwALAPcIykqcCngOsjYipwffluZmYDTLNrQrsDt0bE8xGxGrgJOB54PXBxGeZi4Lgmx2FmZhXU7CQ0CzhM0lhJw4GjgUnACyJiHkB5n9DkOMzMrIKa+j+hiJgt6WzgWmApMBNY3ej4kk4GTgaYPHlyU2I0M7PWUURsuplJXwYeA04HDo+IeZImAjdGxIt7GHch8GgvZjsOWNSL8TZnLvPAMRDL7TI3bqeIGN/XwfSlpt8xQdKEiFggaTLwBuBgYGfgXcBZ5f03PU2ntwtS0oyImN6bcTdXLvPAMRDL7TL3L5vitj2XSRoLrAI+FBFLJJ0F/FLS+4C/A2/eBHGYmVnFND0JRcShnXR7CnhVs+dtZmbVNhDumHBBqwNoAZd54BiI5XaZ+5FNemGCmZlZvYFQEzIzs4pyEjIzs5bp10lI0pGSHpD0kKR+c386SRdJWiBpVl23Tm8Kq/TNsgzuljStdZH3nqRJkm6QNLvcDPf00r3fllvSUEm3S5pZyvyF0n1nSbeVMv9C0pDSfavy/aHSf0or498YkgZJulPSVeV7vy6zpEck3SPpLkkzSrd+u23X67dJSNIg4NvAUcAewAmS9mhtVH3mR8CRHbp1dVPYo4Cp5XUycP4mirGvrQbOjIjdgYOAD5X12Z/LvQJ4ZUTsA+wLHCnpIOBs4LxS5iXA+8rw7wOWRMSuwHlluM3V6cDsuu8Docz/KyL2rfs/UH/etttFRL98kX+Kvabu+6eBT7c6rj4s3xRgVt33B4CJ5fNE4IHy+XvACZ0Ntzm/yD84v3qglBsYDvwFOJD85/zg0v0f2zlwDXBw+Ty4DKdWx96Lsu5I7nRfCVwFaACU+RFgXIduA2Lb7rc1IWAHYG7d98dKt/6qq5vC9rvlUJpc9gNuo5+XuzRL3QUsIO/B+DDQFnlXeli3XP8oc+n/NDB200bcJ/4d+ASwtnwfS/8vcwC/l3RHuWcm9PNtu2ZT3DGhVdRJt4F4PXq/Wg6StgYuA86IiGekzoqXg3bSbbMrd0SsAfaVNAr4Nfl4lPUGK++bfZklHQMsiIg7JB1e69zJoP2mzMUhEfGEpAnAtZLu72bY/lJmoB+fEyKPDibVfd8ReKJFsWwKT5abwVLeF5Tu/WY5SNqSTECXRsTlpXO/LzdARLQBN5Lnw0ZJqh1A1pfrH2Uu/bcFFm/aSDfaIcCxkh4Bfk42yf07/bvMRMQT5X0BebBxAANk2+7PSejPwNRyVc0Q4G3AlS2OqZmuJG8GC+veFPZK4J3lipqDgKdrVfzNibLKcyEwOyLOrevVb8staXypASFpGHAEebL+BuBNZbCOZa4tizcBf4hy0mBzERGfjogdI2IK+Zv9Q0S8g35cZkkjJI2sfQZeQz6Lrd9u2+to9UmpZr7Ih+g9SLajf7bV8fRhuX4GzCNvCvsYeYXQWPJk7pzyPqYMK/IqwYeBe4DprY6/l2V+OdnkcDdwV3kd3Z/LDewN3FnKPAv4XOm+C3A78BDwK2Cr0n1o+f5Q6b9Lq8uwkeU/HLiqv5e5lG1med1b21f15227/uXb9piZWcv05+Y4MzOrOCchMzNrGSchMzNrGSchMzNrGSchMzNrGSchG9AkrSl3Lq69+uxu65KmqO5O52a2vv582x6zRiyLiH1bHYTZQOWakFknyvNdzi7P87ld0q6l+06Sri/Pcble0uTS/QWSfl2e/TNT0svKpAZJ+n55HtDvy50PkPRhSfeV6fy8RcU0azknIRvohnVojntrXb9nIuIA4Fvk/cson38cEXsDlwLfLN2/CdwU+eyfaeQ/3yGf+fLtiHgJ0Aa8sXT/FLBfmc6pzSqcWdX5jgk2oElaGhFbd9L9EfKBcn8tN06dHxFjJS0in92yqnSfFxHjJC0EdoyIFXXTmAJcG/lQMiR9EtgyIr4o6WpgKXAFcEVELG1yUc0qyTUhs65FF5+7GqYzK+o+r6H9POxryft/vRS4o+4O0WYDipOQWdfeWvf+p/L5FvLuzgDvAG4un68HPgD/eBDdNl1NVNIWwKSIuIF8eNsoYL3amNlA4KMvG+iGlSeX1lwdEbXLtLeSdBt5sHZC6fZh4CJJHwcWAu8p3U8HLpD0PrLG8wHyTuedGQT8RNK25B2Rz4t8XpDZgONzQmadKOeEpkfEolbHYtafuTnOzMxaxjUhMzNrGdeEzMysZZyEzMysZZyEzMysZZyEzMysZZyEzMysZf4/GsVBSA9MC+gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# gradient descent\n",
    "detailed_logger = False\n",
    "main_logger = True\n",
    "main_logger_output_epochs = 100\n",
    "L2 = False\n",
    "Dropout = False\n",
    "momentum = False\n",
    "hidden_layer_relu = True\n",
    "hidden_layer_tanh = False\n",
    "hidden_layer_sigmoid = False\n",
    "\n",
    "# hyber-parameters\n",
    "alpha = .01;\n",
    "epsilon = .85\n",
    "keep_prob = .9\n",
    "number_of_epochs = 500\n",
    "batch_size = 27\n",
    "momentum_coef = .9\n",
    "\n",
    "# copy initalization\n",
    "W = Weights.copy()\n",
    "B = Bias.copy()\n",
    "\n",
    "# data arrays\n",
    "cost_array = []\n",
    "accuracy_array = []\n",
    "interation_array = []\n",
    "\n",
    "# rename\n",
    "X_train = np.float64(training_images).copy()\n",
    "Y_train = np.float64(training_labels).copy()\n",
    "\n",
    "X_test = np.float64(testing_images).copy()\n",
    "Y_test = np.float64(testing_labels).copy()\n",
    "\n",
    "#m = size\n",
    "m = number_of_training_images\n",
    "\n",
    "def model(W, B, A):\n",
    "    return np.dot(W, A) + B\n",
    "\n",
    "def activation_relu(Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    return np.where(Z > 0, Z, 0)\n",
    "\n",
    "def activation_tanh(Z):\n",
    "    return np.tanh(Z)\n",
    "\n",
    "def activation_sigmoid(Z):\n",
    "     return 1/(1 + np.exp(-Z))\n",
    "\n",
    "def loss(A, Y):\n",
    "    epsilon = 1e-20\n",
    "    return np.where((Y == 1), np.multiply(-Y, np.log(A + epsilon)), -np.multiply((1 - Y), np.log(1 - A + epsilon)))\n",
    "    #return np.multiply(-Y, np.log(A)) - np.multiply((1 - Y), np.log(1 - A)) \n",
    "    \n",
    "def cost(L):\n",
    "    return np.multiply(1/L.shape[1], np.sum(L))\n",
    "\n",
    "def cost_L2(L, W, epsilon):\n",
    "    L2 = np.multiply(epsilon/(2*W.shape[1]), np.multiply(W[len(W)-3], W[len(W)-3]).sum() + np.multiply(W[len(W)-2], W[len(W)-2]).sum() + np.multiply(W[len(W)-1], W[len(W)-1]).sum())\n",
    "    J = cost(L)\n",
    "    return L2 + J\n",
    "\n",
    "def prediction(A):\n",
    "    return np.where(A >= 0.5, 1, 0)\n",
    "    \n",
    "def accuracy(prediction, Y):\n",
    "    return 100 - np.multiply(100/Y.shape[0], np.sum(np.absolute(Y - prediction)))  \n",
    "    \n",
    "def forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "            print('Forward Propagation Training Data Complete')\n",
    "    return A_layers, Z_layers, D\n",
    "\n",
    "def forward_propagation(W, B, A, layer):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "        A = forward_propagation(W, B, A, layer)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        A = activation_sigmoid(Z)  \n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "            print('Forward Propagation Testing Data Complete')\n",
    "    return A\n",
    "\n",
    "def dZ(dZ, W, Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    W = np.where(~np.isnan(W), W, 0)\n",
    "    dZ = np.where(~np.isnan(dZ), dZ, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    W = np.where(~np.isinf(W), W, 0)\n",
    "    dZ = np.where(~np.isinf(dZ), dZ, 0)\n",
    "    if(hidden_layer_relu == True):\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.where(Z > 0, 1, 0))\n",
    "    elif(hidden_layer_tanh == True):\n",
    "        A = activation_tanh(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), 1- np.multiply(A, A))\n",
    "    elif(hidden_layer_sigmoid == True): \n",
    "        A = activation_sigmoid(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.multiply(A, (1-A)))\n",
    "\n",
    "def dW(dZ, A):\n",
    "    return np.multiply(1/dZ.shape[1], np.dot(dZ, np.transpose(A)))\n",
    "\n",
    "def dW_L2(dZ, A, W, epsilon):\n",
    "    return np.multiply(epsilon/Z.shape[1], W) + dW(dZ, A)\n",
    "\n",
    "def dB(dZ):\n",
    "    return np.multiply(1/dZ.shape[1], np.sum(dZ))\n",
    "\n",
    "def backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB):\n",
    "    if(layer >= 0):\n",
    "        if(layer == len(W) - 1):\n",
    "            _dZ = A_layers[layer+1] - Y\n",
    "        elif(layer >= 0):\n",
    "            _dZ = dZ(_dZ, W[layer+1], Z_layers[layer])\n",
    "        if(Dropout == True):\n",
    "            _dZ = np.multiply(_dZ, D[layer])\n",
    "        if(L2 == True):\n",
    "            _dW = dW_L2(_dZ, A_layers[layer], W[layer], epsilon)\n",
    "        else:\n",
    "            _dW = dW(_dZ, A_layers[layer])\n",
    "        _dB = dB(_dZ)\n",
    "        if(momentum == True):\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(alpha, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(alpha, _dB)\n",
    "            W[layer] = W[layer] - V_dW[layer]\n",
    "            B[layer] = B[layer] - V_dB[layer]           \n",
    "        else:\n",
    "            W[layer] = W[layer] - np.multiply(alpha, _dW)\n",
    "            B[layer] = B[layer] - np.multiply(alpha, _dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Layer: ' + str(layer))\n",
    "        layer = layer - 1\n",
    "        W, B = backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Propagation Complete')\n",
    "    return W, B\n",
    "    \n",
    "\n",
    "def shuffle(X, Y, number_of_training_images):\n",
    "    random_array = np.random.permutation(np.arange(number_of_training_images))\n",
    "    return X[:, random_array], Y[random_array]\n",
    "    \n",
    "start_time = time.time()   \n",
    "# main loop\n",
    "for epoch in range(1, number_of_epochs):\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Main Loop Epoch: ' + str(epoch))\n",
    "    \n",
    "    # shuffle data\n",
    "    X, Y = shuffle(X_train.copy(), Y_train.copy(), number_of_training_images)\n",
    "    number_of_batches = int(np.floor(number_of_training_images/batch_size))\n",
    "    split_index = number_of_batches*batch_size\n",
    "\n",
    "    # parse into minibatches\n",
    "    X_minibatches = np.split(X[:, 0:split_index], number_of_batches, axis=1)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        X_left_over_portion  = X[:, split_index:number_of_training_images]\n",
    "        X_minibatches.append(X_left_over_portion)\n",
    "    \n",
    "    Y_minibatches = np.split(Y[0:split_index], number_of_batches, axis=0)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        Y_left_over_portion  = Y[split_index:number_of_training_images]\n",
    "        Y_minibatches.append(Y_left_over_portion)\n",
    "    \n",
    "    number_of_minibatches = len(Y_minibatches)\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Number Of Minibatches: ' + str(number_of_minibatches))\n",
    "\n",
    "    for index in range(0, number_of_minibatches-1):\n",
    "        X_minibatch = X_minibatches[index]\n",
    "        Y_minibatch = Y_minibatches[index]\n",
    "\n",
    "        if(hidden_layer_relu + hidden_layer_tanh + hidden_layer_sigmoid != 1):\n",
    "            print(\"ERROR! Please Select Only 1 Hidden Layer Activation Function\")\n",
    "            break\n",
    "\n",
    "        # forward propogation training data set\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, X_minibatch, [X_minibatch], [], 0, [], keep_prob)\n",
    "        L = loss(A_layers[len(A_layers) - 1], Y_minibatch)\n",
    "        if(L2 == True):\n",
    "            C = cost_L2(L, W, epsilon) \n",
    "        else:\n",
    "            C = cost(L) \n",
    "\n",
    "        # backpropogation\n",
    "        W, B = backward_propagation(W, B, Y_minibatch, A_layers, Z_layers, 0, alpha, epsilon, len(W) - 1, D,  V_dW, V_dB)\n",
    "        \n",
    "    if(epoch % main_logger_output_epochs == 0):\n",
    "        print('Cost: ' + str(C))\n",
    "\n",
    "    # forward propogation test data set\n",
    "    A_test = forward_propagation(W, B, X_test, 0)\n",
    "\n",
    "    # accuracy\n",
    "    _prediction = prediction(A_test)    \n",
    "    _accuracy = accuracy(_prediction, Y_test)  \n",
    "\n",
    "    # storage for plotting\n",
    "    cost_array.append(C)\n",
    "    accuracy_array.append(_accuracy)\n",
    "    interation_array.append(epoch)\n",
    "\n",
    "\n",
    "end_time = time.time()\n",
    "run_time = end_time - start_time\n",
    "    \n",
    "print('')\n",
    "print('Results:')\n",
    "print('')\n",
    "    \n",
    "print('')\n",
    "print('Run Time: ' + str(run_time) + ' seconds')\n",
    "print('Cost: ' + str(C))    \n",
    "print('Accuracy: ' + str(_accuracy) + ' %')    \n",
    "print('')\n",
    "print('')\n",
    "\n",
    "\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, cost_array, 'red')\n",
    "pyplot.title('Learning Curve - ' + str(len(X[0])) + ' Training Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Cost')\n",
    "pyplot.show()\n",
    "\n",
    "# plot percent accuracy curve\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, accuracy_array, 'red')\n",
    "pyplot.title('Percent Accuracy Curve - ' + str(len(X_test[0])) + ' Test Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Percent Accuracy')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated, after 500 epochs with minibatches of 27 the cost became approximately 1.6e-05 and the test data accuracy reached 99.18%. These results are very good. The test accuracy is high because minibatch stochastic gradient descent inately provides a form of regularization.\n",
    "\n",
    "Additionally, it should be noted that this algorithm reached convergence in approximatley 120 epochs, which was much less than with larger minibatches. \n",
    "\n",
    "Now we will run minibatch stochastic gradient descent with momentum. Momentum is a tehcnique that takes into account the historical rate of change (velocity) of the gradients when adjusting the weights and bias's rather than strictly adjusting them based on the current gradient. This momentum intuitively reflects a ball heading a down a hill to the bottom. When the hill (gradient) is steep for several steps, momentum picks up and the ball heads to its destination faster. Implementing momentum should helps use arrive at convergence in a quicker manner. With this in mind, we implement a new hyper-parameter for momentum which controls the amoung of focus we put on the gradient momentum versus the current gradient. We will start by setting this momentum coefficient to 0.9 in order to heavely focus on the momentum of the gradient. We will leave our minibatch size at 27 since we have achieved good results with it. \n",
    "\n",
    "First we reinitialize our weights and bias's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Size: 784\n",
      "Weights Shape: (20, 784)\n",
      "Bias Shape: (20, 1)\n",
      "Velocity Weights Shape: (20, 784)\n",
      "Velocity Bias Shape: (20, 1)\n"
     ]
    }
   ],
   "source": [
    "# initialize weights & bias\n",
    "np.random.seed(10)\n",
    "print('Feature Size: ' + str(size))\n",
    "\n",
    "lower_bound = -.1\n",
    "upper_bound = .1\n",
    "\n",
    "#mean = 0.015\n",
    "#std = 0.005\n",
    "\n",
    "# hyper-parameters: hidden layers\n",
    "hidden_layers = 2\n",
    "units_array = [20, 10]\n",
    "Weights = []\n",
    "Bias = []\n",
    "V_dW = []\n",
    "V_dB = []\n",
    "for i in range(0, hidden_layers):\n",
    "    if(i == 0):\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], size]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "    else:\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], units_array[i-1]]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "    \n",
    "# output layer\n",
    "_W = np.float64(np.random.uniform(lower_bound, upper_bound, [1, units_array[i]]))\n",
    "_b = np.float64(np.random.uniform(lower_bound, upper_bound)) # b will be added in a broadcasting manner\n",
    "_V_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_V_dB = np.float64(np.zeros(1))\n",
    "Weights.append(_W)\n",
    "Bias.append(_b)\n",
    "V_dW.append(_V_dW)\n",
    "V_dB.append(_V_dB)\n",
    "\n",
    "Weights = np.array(Weights)\n",
    "Bias = np.array(Bias)\n",
    "V_dW = np.array(V_dW)\n",
    "V_dB = np.array(V_dB)\n",
    "\n",
    "for index in range(0, len(Weights) - 1):\n",
    "    Weights[index] = np.where(Weights[index] != 0, Weights[index], np.random.uniform(lower_bound, upper_bound))\n",
    "\n",
    "#print(train_X.shape)\n",
    "#print(np.ravel(train_Y).shape)\n",
    "\n",
    "print('Weights Shape: ' + str(Weights[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Bias Shape: ' + str(Bias[0].shape)) # vector with a size of the # of unit\n",
    "print('Velocity Weights Shape: ' + str(V_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Velocity Bias Shape: ' + str(V_dB[0].shape)) # vector with a size of the # of unit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we run our minibatch stochastic gradient descent algorithm with momentum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Main Loop Epoch: 10\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.003075740130681305\n",
      "Main Loop Epoch: 20\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.0002421045249690861\n",
      "Main Loop Epoch: 30\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.0001461015181742864\n",
      "Main Loop Epoch: 40\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.001100090690208689\n",
      "Main Loop Epoch: 50\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 4.661676444755369e-07\n",
      "Main Loop Epoch: 60\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 2.430669273114621e-07\n",
      "Main Loop Epoch: 70\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 1.0348610490080223e-06\n",
      "Main Loop Epoch: 80\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 6.290041172420984e-08\n",
      "Main Loop Epoch: 90\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 4.956932166353874e-09\n",
      "\n",
      "Results:\n",
      "\n",
      "\n",
      "Run Time: 229.67204904556274 seconds\n",
      "Cost: 6.6395106245545e-08\n",
      "Accuracy: 99.22 %\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEWCAYAAABSaiGHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm4XFWZ7/HvLwkJc8IQRBIgoYlKlEEMwaFFHFrDoKHvBQmigkLjhKiIin1tQLSvoiJoy6XBBkVmRMUokRlUBCEBGQwxEhDJEUhCQgIhhuTkvPePtcuzzz5Vdaba51Sd/D7PU0/VnteuXbXfWu9atbciAjMzs7KNGOoCmJnZxsEBx8zMBoUDjpmZDQoHHDMzGxQOOGZmNigccMzMbFA44ACSfiXpmKEuhw09ScdI+lWj57XuJO0p6Z4Gresdkp5oxLpqrL/usZZ0p6Rja0zbXdKw/f+JpH0l/bY38w5pwJH0hKR3DGUZACLioIi4pIx1S9pa0rmSnpS0WtKibHj7MrbXSJLOkLQ+K3flsVtu+j6S7pO0JnveJzdNks6StDx7fEOSGrFsbr6jc+X6u6SOfFn7s88RcUlEHNToeftKUlu2T6slrZT0O0knVHsfaiw/4JOcpH+V9KCk5yU9K+kWSbs0cNtfBb6ZWy6/z89IukjSFgPZh96QdLykO6qMb5N0IJR7rAciX8ahEhH3A3+X1OP7M+xrOJJGDeG2RwO3Aq8GZgBbA28ElgPT+7G+odiXqyNiy9zj8awso4GfA5cB2wCXAD/PxgOcABwG7A3sBRwKfGSgy+ZFxOWVcgEHAU/ly1qcfyg/C/10ULYfk0gn5n8HLhyMDUt6JfAD4FPAWGAy8N9AR4PWPxH4Z+AXhUmVfd4X2B/4fCO2Z+XIfacup8p3tJuIGLIH8ATwjhrTDgUeAFYCdwF75aadCjwGvAA8AvxrbtqxwO+Ac4AVpF9RxwJ3At8CngP+QvpgV5a5Azg+t3y9eScDv8m2fQtwHnBZjX04HlgCbFnnPQhg99zwD4GvZq8PBNqALwDPAJcCC4BDc/OPAp4F9s2GX5+9XyuBB4EDB3B8zqizb+8E/gYoN+5JYEb2+i7ghNy044DfD3TZOmU9EGirMr4N+BzwMLAuG/cl4PHsGM4H3lM4Znfk3tsgfZEWZZ+H7/Zz3pHAuaQfG48DnwSizv60FY8d8AbSCf9V2fB7SN+RF7L37z9y8z6VlWd19tgPmALcnpXh2ezzNLbG9mcB8+qUbwQpAD6WresqYJta266y/IeBG+rtM/Bt4Oe54U2zcYtJ36v/B2yaTXsH8EThWEzKLXsZcEad7+kd9Y5BcR7SD8iFwCrgO6RzzrG5Y31O9j4/BpyYP9bAOFIwfzrbxpnAiNx2fp0tvzL7rLyzL5+TbPx2wBxgWfZZ/AUwIZt2FHBPYf4vANf29n3Ojv0zwA+y8bsCLwKb1PueNmUNR9K+wMWkL+92wAXAbEljslkeA95M+uX1ZeAySS/PrWJ/0oHaAfjP3LiFwPbAN4CL6qQn6s17BXBvVq4zgA/U2ZV3kL5U/UrvZHYEtiUd0BOAK0kfmIp3Ac9GxP2SJgDXk4LstsApwE8kjR/A9t8taYWk+ZI+lhv/auChyD5tmYey8ZXpD+amPViY1t9l+2MWqQY0Nhv+M/CmbPg/gSskvazO8gcDrwNeC7y/hzRwrXk/Rvo87AVMA/5XX3ciIu4mfcnfnI1aDbw/2493A5+SdGg27YBsmUqNby4g0mfj5cBUYDfgP2ps7j5gT0lnS3prldTWycAh2XYmkk42362z7aI9Sd+xqiTtTDqpL8qN/hbpB99epOA5Cfg/tdZRFkk7ANeSfvhuTzrp75+b5WOkH1V7kzIZ7y2s4jLg78A/kT4LhwAfyk1/I+kH0nakwHNRP4o5Avg+sAvp3LGeFBgBrgNeKWlKbv73k36AQM/v80Rgy2zdHweIiL+SPl/5dXZXLxqV/aBGDQc4H/hKYdxC4C011vMAMDN7fSzwZGH6scCi3PDmpF9AO2bDd9C1hlN13uwNbgc2L/xyqlULuBn4eg/vQU81nHVkvy6ycbuTftFung1fDpyW+5VyaWH9NwLH9PP4TAV2Iv1ieyPpF9lR2bT/AK4qzH852a9IYAPZL/FseEq2rxrIsnXKeiC1azgf7GE//wgckr2uVmt5fW7enwKn9GPe3wDH5abNoI81nGz8POALNZb5HvDN3Oek5vqzeQ4H5taZ/kbgx6QazFrSj8DK5+5Rct9HYGfgJdKJrjfb/kHlc17Y59XZ5zuAm8hqYNl61wK75uZ/M/Bo9nqgNZx2Uo0i/+igSg2HVDu7M7f8CNJ349jcsT4+N/3gyvsBTCAFmzG56R8Abs5t50+5aVtn+7J9Xz4nVeabBizLDX8f+HL2ep/sGG/Sy/d5LTC6yjaWAG+sV46mrOGQIvJns8bSlZJWkj7QOwFI+qCkB3LTXkP6pVGxuMo6n6m8iIg12ctuef4e5t0JWJEbV2tbFctJvyYHYllErM2VZxEprfZuSZuT0ipXZJN3BY4ovG//XK0MhQb3qr1vIuKRiHgqIjZExF2kX0iHZ5NXk74MeVuTThbVpm8NrM6+dQNZtj+6HCNJx2aN4ZX36FV0/fwUPZN7vYban5t68+5UKEe9z009E0ipYiS9QdIdkpZJWkU6WdXcD0k7SrpG0t8kPU/6cVNz/oi4KyKOiIjtSbWWtwFfzCbvAvwi9x4+TDox7tDL/XgO2KrK+EMjYivg7aRa7bbZ+B2BMUD+uP2yD9vryZ0RMS7/IKUGq+lyLCOig3Tirzod+Gvu9a6k/ViS24/zgHwNu/gZgvqfuW4kbSHpf5Q6Kz0P3EbXY30JcHT2+v2kttr19O59XhIR66psditSoK6pWQPOYuA/Cx+AzSPiSkm7kqLzicB22Qfjj6RfzhX9PTH15Glg2+xEX7FznflvAd7VQ0+bNaRaVMWOhenV9qWSVpsJPJIFIUjv26WF922LiPh6cQWRa3CP3ve+qdRQILV97FVIS+6Vja9M3zs3be/CtP4u2x//eA+VetmdT0p7VD4/f6Lr56cMT5NSERX1PjdVSXo96cR0ZzbqKuAnwM4RMRb4Hzr3o9rn5ixSLWTPiNiaVJvv1X5HxL2kVMxrslFtwL8UPmubRsQzNbZd9BDwijrbu41UK6n0YltCqu2/Mre9sdl+F5dtz/az3vdqIJ4md/wkjaDrse0ynRScKxaTvvPb5vZj64jYq4Hlg9TZYjIwPTvWb8tPjIg7s7K/iXQuqaTTevM+dzu+2XkZUs23pmYIOJtI2jT3GEUKKB+VtL+SLSQdImkrYAvSDi8DkPQhOr8EpYqUp5wHnCFptKQ3kHLntVxK+oD9RNKrJI2QtJ2kf5d0cDbPA8D7JI2UNAN4Sy+KchUpR/wxOms3kL6g75b0rmx9m0o6MOsR1GeSZkraJjsG04GTSL3LIKUhNwAnSRoj6cRs/G3Z84+AkyVNkLQT8FnSL+qBLjtQW9L5+ZGk40k1nLJdA3xa0k6StiF1ZOgVSWMlVWqyP4yIBdmkrUg17rVZMJqVW2wpEMp1Y8/mfxFYlbWRnFJnm29R6i68Qza8B+mz/vtslv8G/q+ybtKSdsjKWGvbRTcB+6mzZ2I15wAHS3pNRGwgBdRzJY3PPpMTJb2zxrIPAkdn34NDSDX9RvklsE/2/RgFfAbIt5NWjvUESduRUt0ARMRiUqeAbyn9ZWKEUjfyAwZQntFVzqFbkQLbc1kZTquy3KWkH18vRsTvs/L19X2ueAtwS1ZLqqkZAs4cUk6z8jgjIuYB/0bKST9Hajg8FlKaBzgbuJsUjfck9RAZLEeTegstJzXAXk36NdVNRLxEynn+idSe8zypw8H2QOUPb58ifZFXZuu+rqcCRMTTpP1/Y7b9yvjFpFrPv5NOqItJJ7b+HudZpPf+BVIQOCuy/ytlVerDgA9mZf8wcFiuqn0BqWfMw6Qa6PXZuAEtO1AR8RCpcfte0i/RV9F5LMp0PinQPkxqkL+e9Euynl8p/Z/oSVID9TdJabOKjwFfk/QC6ZhfU5kQES8AXwPuyVIj04DTSY3Yq4DZpNpRLc8B/wr8MSvDnGz9Z2fTvw3cANyabf8uUk+4WtvuIiKeAn5LnR9sWW3pcjo7NnyWlJ66N9uHm6jdSH1SVv6VwBHZ/jZERCwBjiQdj+WkGkz+M3Q+6e8QDwNzSR0M8t5P+uH8COl9/jEDq4HdSNdz6JdIx2dsVr67gGpp8x+RfqxfWhjfl/e54mjSj5C61P+0uAFIuprUyHf6UJfFWoekdwPnRsQ/DXVZhoqkPYHvR8Trh7osG6Ms1b8UeE1E/GUA63kt8F8R0WMt0gGnjyTtR2q0/QsprXUd8IaI+MOQFsyaWvblfjOppvty4GfAryOiZlrLrEySPk/qJdxTuqxhWu2f181gR1J31+1IDacfc7CxXhDpPz/XktpRfkn6D5nZoJPURvpvzsxB3a5rOGZmNhiaodOAmZltBIZNSm377bePSZMmDXUxzMxayn333fdsRAzk8le9NmwCzqRJk5g3b95QF8PMrKVI+mvPczWGU2pmZjYoHHDMzGxQlBpwJM2QtFDpLpenVpl+gKT7JbVLOrwwbRdJN0laIOkRSZPKLKuZmZWrtIAjaSTpKqgHkS5zf5SkqYXZniRdsuYKuvsR6VLre5Aux7G0rLKamVn5yuw0MJ10X5nKLYmvIru6cWWGiHgim9bltrVZYBoVETdn8w3kBmZmZtYEykypTaDrPSHasnG98QpgpaSfSvqDpG9mNaYuJJ0gaZ6kecuWLWtAkc3MrCxlBpxq99no7WUNRpGuO3UK6Qq0u5FdLbrLyiIujIhpETFt/PhB6UZuZmb9VGbAaaPrTYgmUvsOetWW/UNEPJ7dTOk6YN8Gl69nd9wBC2vedt3MzPqgzIAzF5giaXJ2k6VZ9P6eFHOBbSRVqi1vI9f2M2iOOw6+3u1mmWZm1g+lBZysZnIi6eZAC4BrImK+pDMrdwaUtF921dIjgAskzc+W3UBKp90q6WFSeu77ZZW1prVr4aWq91YzM7M+KvXSNhExh3SnwPy403Kv59L1XuD5+W4m3ed+6LS3p4eZmQ2YrzRQz/r16WFmZgPmgFOPazhmZg3jgFOPA46ZWcM44NSzfr0DjplZgzjg1BKRgo3bcMzMGsIBp5aO7PJuruGYmTWEA04tlZqNA46ZWUM44NRSCTQOOGZmDeGAU0sl0LgNx8ysIRxwanFKzcysoRxwanFKzcysoRxwanHAMTNrKAecWiopNbfhmJk1hANOLa7hmJk1lANOLQ44ZmYN5YBTi3upmZk1VKkBR9IMSQslLZJ0apXpB0i6X1K7pMOrTN9a0t8kfa/Mclbl/+GYmTVUaQFH0kjgPOAgYCpwlKSphdmeBI4Frqixmq8Avy6rjHU5pWZm1lBl1nCmA4si4vGIWAdcBczMzxART0TEQ0BHcWFJrwNeBtxUYhlrc0rNzKyhygw4E4DFueG2bFyPJI0AzgY+18N8J0iaJ2nesmXL+l3QqvI1nIjGrtvMbCNUZsBRlXG9PXN/HJgTEYvrzRQRF0bEtIiYNn78+D4XsK58zWbDhsau28xsIzSqxHW3ATvnhicCT/Vy2TcAb5b0cWBLYLSk1RHRreNBafKdBdrbYVSZb5WZ2fBX5ll0LjBF0mTgb8As4H29WTAijq68lnQsMG1Qgw10reG4HcfMbMBKS6lFRDtwInAjsAC4JiLmSzpT0nsAJO0nqQ04ArhA0vyyytNnDjhmZg1Vap4oIuYAcwrjTsu9nktKtdVbxw+BH5ZQvPryKTX/F8fMbMB8pYFaXMMxM2soB5xaHHDMzBrKAaeWYi81MzMbEAecWvJBxm04ZmYD5oBTi1NqZmYN5YBTiwOOmVlDOeDU4jYcM7OGcsCpxW04ZmYN5YBTi1NqZmYN5YBTi1NqZmYN5YBTi2s4ZmYN5YBTi9twzMwaygGnFqfUzMwaygGnFqfUzMwaygGnFgccM7OGcsCpxffDMTNrqFIDjqQZkhZKWiSp2y2iJR0g6X5J7ZIOz43fR9LdkuZLekjSkWWWsyrXcMzMGqq0gCNpJHAecBAwFThK0tTCbE8CxwJXFMavAT4YEa8GZgDnShpXVlmram+HkSM7X5uZ2YCUeYvp6cCiiHgcQNJVwEzgkcoMEfFENq0jv2BE/Dn3+ilJS4HxwMoSy9vV+vWw2WawerUDjplZA5SZUpsALM4Nt2Xj+kTSdGA08FiVaSdImidp3rJly/pd0Kra22HTTdNrt+GYmQ1YmQFHVcZFn1YgvRy4FPhQRHQUp0fEhRExLSKmjR8/vp/FrCEfcFzDMTMbsDIDThuwc254IvBUbxeWtDVwPfCliPh9g8vWs0pKDRxwzMwaoMyAMxeYImmypNHALGB2bxbM5v8Z8KOI+HGJZaytvd0Bx8ysgUoLOBHRDpwI3AgsAK6JiPmSzpT0HgBJ+0lqA44ALpA0P1v8vcABwLGSHsge+5RV1qrchmNm1lBl9lIjIuYAcwrjTsu9nktKtRWXuwy4rMyy9Wj9erfhmJk1kK80UEt7O4weDSNGOOCYmTWAA04t7e0walR6OOCYmQ2YA04t69fDJpukh9twzMwGzAGnFtdwzMwaygGnFgccM7OGcsCppZJSc8AxM2sIB5xaKjUct+GYmTWEA04tTqmZmTWUA04t69c74JiZNZADTi3t7W7DMTNrIAecWtyGY2bWUA44tbgNx8ysoRxwanG3aDOzhnLAqcU1HDOzhnLAqaajAyLchmNm1kAOONVUAoxTamZmDVNqwJE0Q9JCSYsknVpl+gGS7pfULunwwrRjJD2aPY4ps5zdVAKMU2pmZg1TWsCRNBI4DzgImAocJWlqYbYngWOBKwrLbgucDuwPTAdOl7RNWWXtxgHHzKzhyqzhTAcWRcTjEbEOuAqYmZ8hIp6IiIeAjsKy7wJujogVEfEccDMwo8SydpVPqbkNx8ysIcoMOBOAxbnhtmxcw5aVdIKkeZLmLVu2rN8F7cY1HDOzhisz4KjKuGjkshFxYURMi4hp48eP71Ph6nLAMTNruDIDThuwc254IvDUICw7cO6lZmbWcGUGnLnAFEmTJY0GZgGze7nsjcA7JW2TdRZ4ZzZucORrOG7DMTNriNICTkS0AyeSAsUC4JqImC/pTEnvAZC0n6Q24AjgAknzs2VXAF8hBa25wJnZuMHhlJqZWcONKnPlETEHmFMYd1ru9VxSuqzashcDF5dZvpqcUjMzazhfaaAa13DMzBrOAacat+GYmTWcA041TqmZmTWcA041TqmZmTWcA041DjhmZg3ngFNN8VpqEbBhw9CWycysxTngVFOs4eTHmZlZvzjgVOOAY2bWcA441RR7qYEDjpnZADngVFP8Hw74vzhmZgPkgFONU2pmZg3ngFONU2pmZg3ngFNNtRqOU2pmZgPSq4Aj6dLejBs2qrXhuIZjZjYgva3hvDo/IGkk8LrGF6dJOKVmZtZwdQOOpC9KegHYS9Lz2eMFYCnw80Ep4VBwpwEzs4arG3Ai4msRsRXwzYjYOntsFRHbRcQXe1q5pBmSFkpaJOnUKtPHSLo6m36PpEnZ+E0kXSLpYUkLJPW4rYZyG46ZWcP1NqX2S0lbAEh6v6RvS9q13gJZ2u084CBgKnCUpKmF2Y4DnouI3YFzgLOy8UcAYyJiT1Lq7iOVYDQoKgGnci21/DgzM+uX3gac84E1kvYGPg/8FfhRD8tMBxZFxOMRsQ64CphZmGcmcEn2+lrg7ZIEBLCFpFHAZsA64PlelnXgKrWZkSOdUjMza5DeBpz2iAhSgPhORHwH2KqHZSYAi3PDbdm4qvNERDuwCtiOFHxeBJ4GngS+FRErihuQdIKkeZLmLVu2rJe70gvt7TBiRHo44JiZNURvA84LWTvKB4Drs3TZJj0soyrjopfzTAc2ADsBk4HPStqt24wRF0bEtIiYNn78+J72offa2ztTaW7DMTNriN4GnCOBl4APR8QzpJrJN3tYpg3YOTc8EXiq1jxZ+mwssAJ4H3BDRKyPiKXA74BpvSzrwK1f3xloXMMxM2uIXgWcLMhcDoyVdCiwNiJ6asOZC0yRNFnSaGAWMLswz2zgmOz14cBtWeruSeBtSrYAXg/8qVd71Ajt7Z2Bxp0GzMwaordXGngvcC+p99h7gXskHV5vmaxN5kTgRmABcE1EzJd0pqT3ZLNdBGwnaRFwMlDpOn0esCXwR1Lg+kFEPNSnPRuIaik1BxwzswEZ1cv5/g+wX5beQtJ44BZS435NETEHmFMYd1ru9VpSECsut7ra+EFTLaXmNhwzswHpbRvOiEqwySzvw7KtJ59Scw3HzKwhelvDuUHSjcCV2fCRFGouw0o+peY2HDOzhqgbcCTtDrwsIj4n6X8B/0zqynw3qRPB8OReamZmDddTWuxc4AWAiPhpRJwcEZ8h1W7OLbtwQ6ZaSs1tOGZmA9JTwJlUrXdYRMwDJpVSombgNhwzs4brKeBsWmfaZo0sSFNZv95tOGZmDdZTwJkr6d+KIyUdB9xXTpGagGs4ZmYN11MvtU8DP5N0NJ0BZhowGvjXMgs2pNyGY2bWcHUDTkQsAd4o6a3Aa7LR10fEbaWXbCjlU2rNXMNZuxaeeQYmTRrqkpiZ9ahX/8OJiNuB20suS/Nob4fNsiaqZm7DOe88+PKXYeXKdCsFM7Mm5rNUNfmU2ogRIDVnwFm8GF54Af7+96EuiZlZjxxwqsmn1CAFn2Zsw1m1Kj074JhZC3DAqSZfw4H0uhlrOCtXpmcHHDNrAQ441RQDziabNHfAWbNmaMthZtYLDjjVVEupNXPAcQ3HzFqAA0411VJqzdiG44BjZi2k1IAjaYakhZIWSTq1yvQxkq7Opt8jaVJu2l6S7pY0X9LDkupdZqexWq0Nxyk1M2sBpQUcSSNJt4o+CJgKHCVpamG244DnImJ34BzgrGzZUcBlwEcj4tXAgcDgVTGKKbVmbMPp6IDnn0+vXcMxsxZQZg1nOrAoIh6PiHXAVcDMwjwzgUuy19cCb5ck4J3AQxHxIEBELI+IDSWWtatWqOGsXp2CDjjgmFlLKDPgTAAW54bbsnFV54mIdmAVsB3wCiAk3SjpfkmfL7Gc3bVCG04lnQYOOGbWEnp7i+n+UJVx0ct5RpHuLrofsAa4VdJ9EXFrl4WlE4ATAHbZZZcBF/gfWqGXWj7guA3HzFpAmTWcNmDn3PBE4Kla82TtNmOBFdn4X0fEsxGxhnSH0X2LG4iICyNiWkRMGz9+fONK3gr/w3ENx8xaTJkBZy4wRdJkSaOBWcDswjyzgWOy14cDt0VEADcCe0naPAtEbwEeKbGsXbVCG07lsjbggGNmLaG0lFpEtEs6kRQ8RgIXR8R8SWcC8yJiNnARcKmkRaSazaxs2eckfZsUtAKYExHXl1XWLjo60qPZr6XmlJqZtZgy23CIiDmkdFh+3Gm512uBI2osexmpa/TgqtRkmr2G45SambUYX2mgqFrAaeY2nHHjHHDMrCU44BRVAksr9FLbckvYaiun1MysJTjgFFXaalrhfzhjx8Lmm7uGY2YtodQ2nJbUKm04q1aldNommzjgmFlLcMApqpZSa9Y2nHHjUo86p9TMrAU4pVZUK6XWrAFns81cwzGzluCAU1QrpdaMbTjjxrkNx8xahlNqRa3US23sWFi3zik1M2sJruEUVUupNVsbToRTambWchxwilqhl9qaNbBhg1NqZtZSHHCKaqXUmqkNJ3+VAddwzKxFOOAUtUIvtWLAWbMmpdnMzJqYA05RK1xLrRJwKlca6OhorhqYmVkVDjhFrdBLrVjDAafVzKzpOeAUtcK11Co3X8sHHHeNNrMm54BTVKuXWkRKXTUD13DMrAU54BTVSqnlpw21YhsOOOCYWdMrNeBImiFpoaRFkk6tMn2MpKuz6fdImlSYvouk1ZJOKbOcXdT64yc0V8DZdFMYM8YpNTNrGaUFHEkjgfOAg4CpwFGSphZmOw54LiJ2B84BzipMPwf4VVllrKpWSg2apx2ncpUBcErNzFpGmTWc6cCiiHg8ItYBVwEzC/PMBC7JXl8LvF2SACQdBjwOzC+xjN21SkqtEnCcUjOzFlFmwJkALM4Nt2Xjqs4TEe3AKmA7SVsAXwC+XG8Dkk6QNE/SvGXLljWm1LV6qUHzBJzKzdfAKTUzaxllBhxVGVf8O3yteb4MnBMRq+ttICIujIhpETFt/Pjx/SxmQa0/fuanDTWn1MysBZV5e4I2YOfc8ETgqRrztEkaBYwFVgD7A4dL+gYwDuiQtDYivldieZNWacOZPDm9dkrNzFpEmQFnLjBF0mTgb8As4H2FeWYDxwB3A4cDt0VEAG+uzCDpDGD1oAQb6AwqrdKG4xqOmbWI0gJORLRLOhG4ERgJXBwR8yWdCcyLiNnARcClkhaRajazyipPr9Wr4TRDwMnfCwfchmNmLaPUO35GxBxgTmHcabnXa4EjeljHGaUUrpZmb8NZuzbd5dM1HDNrMb7SQFG9lFoztOHkr6MGMHIkjB7tgGNmTc8Bp6jZU2r5y9pUVO6JY2bWxBxwitrbQYIRubemGQNOpYYDvuunmbUEB5yi9eu7ptOgudpwqgWczTd3wDGzpueAU9Te3jWdBs3VhlOrhuOUmpk1OQeconoBpxlqOMVOA+CUmpm1BAecovb27im1Zgo41ToNOKVmZi3AAado/fruNZxma8PZZJPO/9+AU2pm1hIccIpaoQ1n7NjUk67CKTUzawEOOEVlpNQ2bIDLLmtMDWnpUiheGdspNTNrAQ44RdVSagMNOLfdBh/4ANx888DKBingvOxlXcc5pWZmLcABp6haSm2gbTgLF6bnxYvrz9cbS5ZUDziu4ZhZk3PAKaqXUutvG86jj6bnp4q3A+qHpUthhx26jnNKzcxagANOURkptUrA+dvf+l8uSFeKXrWqeg3npZego2Ng6zczK5EDTlEZf/xsVMBZujQ9Vws44FqOmTU1B5yiaim1gbThrF8Pf/lLej3QlFol4BRTag44ZtYCSg04kmZIWihpkaRTq0wfI+lNdq2TAAASD0lEQVTqbPo9kiZl4/9F0n2SHs6e31ZmObuol1LrTxvOE0+kbtFbbjnwGs6SJem5WMPZfPP07IBjZk2stIAjaSRwHnAQMBU4StLUwmzHAc9FxO7AOcBZ2fhngXdHxJ7AMcClZZWzm0an1CrptDe9CZ59NrW19Fcl4NSq4bhrtJk1sTJrONOBRRHxeESsA64CZhbmmQlckr2+Fni7JEXEHyKikn+aD2wqaUyJZe3U6D9+VgLOgQem56ef7nfR3IZjZq2szIAzAcj/8aQtG1d1nohoB1YB2xXm+d/AHyKiW9VA0gmS5kmat2zZssaUulpKbeTI9NzfgLP11rDPPml4IGm1JUtSaq6SQqtwSs3MWkCZAUdVxkVf5pH0alKa7SPVNhARF0bEtIiYNr54uZf+qpZSk9K4/rThPPooTJkCE7JYO5CAU+0/OOCUmpm1hDIDThuwc254IlDspvWPeSSNAsYCK7LhicDPgA9GxGMllrOraik1SAGnNzWcDRu6Di9alALOTjul4YH0VKt2lQFwSs3MWkKZAWcuMEXSZEmjgVnA7MI8s0mdAgAOB26LiJA0Drge+GJE/K7EMnZXLaUGvQs4V1wBO+7Yec+adetSL7UpU2DbbWHMmIGn1KrVcJxSM7MWUFrAydpkTgRuBBYA10TEfElnSnpPNttFwHaSFgEnA5Wu0ycCuwP/IemB7FHlTFuCaik1SLWengLOb36TeqJdd10a/stf0r//d989peUmTBh4Sq1eDccpNTNrYlXOrI0TEXOAOYVxp+VerwWOqLLcV4Gvllm2muql1Hpqw1mwID1ffTUce2xnD7UpU9LzhAn9T6lt2JCCmVNqZtaifKWBooGk1BYsgBEj4JZbYPny7gFnp536X8N59tlUW3JKzcxalANOUa2UWk8BZ/lyWLYM3ve+NN9Pf5oCzrhxsF3W07uSUotiZ71eqPUfHHBKzcxaQqkptZZUK6XWUxtOJZ121FFwzz0prSal2k3ldtATJqRayKpVKRD1Ra3L2gCMHp224RqOmTUx13CK6qXU6rXhVALOHnvAkUfC7bfD/fd3ptOgs2t0f9JqtS7cCSnY+CZsZtbkHHDyIvrfhrNgQTrp77prCjgdHbBiRdeAM5A/f9ar4YBvwmZmTc8BJ+/009N/Z175yu7TehNwXvnK1GngNa+Bqdl1SqsFnFo91R59tPM/PEVLlqS0Xq1U3GabuQ3HzJqaA07F+efDV74CH/4wfOhD3af3pg1njz06h488Mj3vvnvnuHoptXXrYPp0+Pznq6+/clkbVbsaEE6pmVnTc6cBgJ/9DD7xCTj0ULjgguon9XptOC++CH/9Kxx3XOe4T34yXWhzv/06x226abriQLWA89vfptrNzTdX30aty9pUOKVmZk3ONZyFC1PPsv33Tz3LqrXfQOravHBh92ulVdYBXWs422wDJ5+cUmx5tf78ef316fmJJ9KjqNaFOyucUjOzJueA84pXwNe/Dr/4RffL/ud9+MOpFvPzn3eflu+h1pNal7eZMwcmT06vb7+9+/SeajhOqZlZk3PAkeDTn4btt68/32GHwaRJcM453actWJDumZPvIFBLtasNPPZYqiV96lMwfnz3gBPhlJqZtTwHnN4aORJOOgnuvBPmzes6bcEC+Kd/Sn/A7MmECSl45DsgzMkuN3fIIenOoLff3vVqBM8/nzoVNDKltnYt7L03HH98/+7zY2bWRw44fXHccbDVVt1rOcUeavVMmJD+o1P5Xw2kgDNlSurR9ta3QltbqvVU9PQfHOh7Su3yy+Ghh+Cii+Dd74YXXuj9smZm/eCA0xdbb52CzjXXdKbF1q9P/5/pS8CBzuVffDHVaA45JA0feGB6zqfVKgGnpxpObwNORwecfXa67fX3v58uNvrWt3YNgsPZQw8N7EZ4ZtYvDjh9ddJJ6YRdqeU89lhKj/U24FT+i1NJy91+O7z0Ehx8cBp+1avSTdzuuKNzmXoX7qzYfPPep9RuuCHVyk45JaXUrrsOHnkEZs7s3V1NB1N/LnRazy9+AdOmpcD+4ouNXbeZ1eWA01eTJ6c/dZ59NsyYAddem8b3NuBMnZquRPCJT8BHPpK6Ym+xBRxwQJoudW/H6UtKrTcn6G99CyZOhPe+Nw0feij88IfpoqNfLdyG6Cc/gbPOSm1Ig6m9HY4+OqUa//znxqxzzhw4/PDU3rZoUQq4ZjZ4IqK0BzADWAgsAk6tMn0McHU2/R5gUm7aF7PxC4F39bSt173udTFo/v73iLPPjthmm4h0io94/vneL79mTcQpp0RIadmZM7tOv+CCNP5Pf0rDp5+e5l2/vvY6v/KVtMxLL9Xf9rx5ab5vfrP7tA98IGLEiIi77oro6Ij4xjc692+ffSL++Mfe72PFc8+ldfVFe3vEUUel7W61VcQOO0Q88EDft533q19FjBkTse++EStWRHzuc2n9v/zlwNZr1uKAeVFiHMg/ygw2I4HHgN2A0cCDwNTCPB8H/jt7PQu4Ons9NZt/DDA5W8/Ietsb1IBTsWJFxKmnRpx4Yv+Wv+uuiLe+NeKmm7qO//Of06E5//w0/NGPRmy/ff11nX12Wua++yIuvjjiM5+J+K//ivjNbyKWL08n/qeeijj88HQSX7my+zpWrYqYNClit90iPvnJtL4jj4y49tqI8ePTCfuzn03B6jvfibjoorQPq1Z1Xc/atRFXXhlx4IFpHZMnR3zpSxELFqRgUi8AbdgQceyxabmvfS0F3YkTI8aNi7jlllTu9vae39uItJ2bboqYMaMzaC5f3lnGvfZKwez3v48488yIPfZI2/rc5/oXXBth/fqItraIRx6JWLKk9/tqfbdhQ/rx2NcfRGUYwjIMZsBR2l7jSXoDcEZEvCsb/mJWo/pabp4bs3nuljQKeAYYD5yanzc/X63tTZs2LeYVuyu3qgjYZZfUW23LLVN6abfdYP782sucfz58/OOdw6NH106DnXxySglW87vfpfReR0f6X9C3v52ulrB0aUoBXndd9eV23DFdpaGjI3XjXr06pR9nzYL77ksdEzo6ui6z2WZp/7bcsrNL+dq16Q+2Z5yRLqYKafhf/qXzDqqQ0pCbb56eN920832rfJ6l1EbT1pbK9olPpMsNjR3buY6HH07tOZX36YAD0vRf/Sq955Mmdd5rqLLOyuu+fm9eeim9Jy++mNaR3++OjrS+F1+Ep5/u+j5J6aoVm24KY8akR2/KUOuae3md9deuy1VbNv8eVFtPbxW3V1x/rW1AuspH5TFiRPqrwsiR3a/m0ZN16+C559KjoyMdg3HjUqegESPSo3isI9K8HR1pfH6+WvuZf105xpX9k9Lw88+nx5o16fswdmzqCVvriie17L03XHll35ahUhzdFxHT+rVwH5V5LbUJwOLccBuwf615IqJd0ipgu2z87wvLTihuQNIJwAkAu+yyS8MKPuQkuOKK1I6zcmX6YrzjHfWXOfhg+OhHYa+94C1vSZ0PnnoKHnyw84+plRP8zJm11/OmN8EPfpBO/P/2b51fqB12SNecW78+fWHXrUu3X3jkkXTirnTjltJJ8bDD4O1v7zwZPP10Wn758s6Txtq16SS8enXX/wKdckoKEBW77gp3352u8rBqVdcv6Zo1nb3zKieAyglCSu1sRx2VylS0557w4x+n9pwjjoCdd07jly5N7/+993ae/PMnysq6e3NSr8w/Zkx677fYIg1Xgs9LL3WWe7PNUtvaxInp5Ld8eSrLihXpvXrppfTI6ykAVMpaS2U/8u9bcZn8ftdbT2/eh/z2apW31rKVADNyZBpub0+fo74G/002SUF8223Te/7CC+k79vzznUGluM5igOnoqH6Zq7z8PuaXzX+Wtt46BZlKp5/KZzv/uevNe1u5SkmTK7OGcwSp7eX4bPgDwPSI+GRunvnZPG3Z8GPAdOBM4O6IuCwbfxEwJyJ+Umt7w6qGY2Y2SAazhlNmL7U2YOfc8ESg+OeHf8yTpdTGAit6uayZmbWQMgPOXGCKpMmSRpM6BcwuzDMbOCZ7fThwW9aINRuYJWmMpMnAFODeEstqZmYlK60NJ2uTORG4kdRj7eKImC/pTFKviNnARcClkhaRajazsmXnS7oGeARoBz4RET0kTM3MrJmV1oYz2NyGY2bWd8OlDcfMzOwfHHDMzGxQOOCYmdmgcMAxM7NBMWw6DUhaBvy1j4ttDzxbQnGanfd74+L93rj0db93jYjxZRUmb9gEnP6QNG+wemc0E+/3xsX7vXFp5v12Ss3MzAaFA46ZmQ2KjT3gXDjUBRgi3u+Ni/d749K0+71Rt+GYmdng2dhrOGZmNkgccMzMbFBslAFH0gxJCyUtknTqUJenLJJ2lnS7pAWS5kv6VDZ+W0k3S3o0e95mqMtaBkkjJf1B0i+z4cmS7sn2++rsthnDiqRxkq6V9KfsuL9hYzjekj6Tfcb/KOlKSZsO1+Mt6WJJSyX9MTeu6jFW8t3sXPeQpH2HruQbYcCRNBI4DzgImAocJWnq0JaqNO3AZyNiD+D1wCeyfT0VuDUipgC3ZsPD0aeABbnhs4Bzsv1+DjhuSEpVru8AN0TEq4C9Sfs/rI+3pAnAScC0iHgN6XYosxi+x/uHwIzCuFrH+CDS/cSmACcA5w9SGava6AIO6RbWiyLi8YhYB1wFzBziMpUiIp6OiPuz1y+QTj4TSPt7STbbJcBhQ1PC8kiaCBwC/E82LOBtwLXZLMNuvyVtDRxAus8UEbEuIlayERxv0r29NsvuHLw58DTD9HhHxG9I9w/Lq3WMZwI/iuT3wDhJLx+ckna3MQacCcDi3HBbNm5YkzQJeC1wD/CyiHgaUlACdhi6kpXmXODzQEc2vB2wMiLas+HheNx3A5YBP8hSif8jaQuG+fGOiL8B3wKeJAWaVcB9DP/jnVfrGDfV+W5jDDiqMm5Y9w2XtCXwE+DTEfH8UJenbJIOBZZGxH350VVmHW7HfRSwL3B+RLwWeJFhlj6rJmuvmAlMBnYCtiClkoqG2/Hujab63G+MAacN2Dk3PBF4aojKUjpJm5CCzeUR8dNs9JJKtTp7XjpU5SvJm4D3SHqClDJ9G6nGMy5LucDwPO5tQFtE3JMNX0sKQMP9eL8D+EtELIuI9cBPgTcy/I93Xq1j3FTnu40x4MwFpmQ9WEaTGhdnD3GZSpG1W1wELIiIb+cmzQaOyV4fA/x8sMtWpoj4YkRMjIhJpON7W0QcDdwOHJ7NNhz3+xlgsaRXZqPeDjzCMD/epFTa6yVtnn3mK/s9rI93Qa1jPBv4YNZb7fXAqkrqbShslFcakHQw6RfvSODiiPjPIS5SKST9M/Bb4GE62zL+ndSOcw2wC+nLekREFBshhwVJBwKnRMShknYj1Xi2Bf4AvD8iXhrK8jWapH1IHSVGA48DHyL9sBzWx1vSl4EjST0z/wAcT2qrGHbHW9KVwIGk2xAsAU4HrqPKMc4C8PdIvdrWAB+KiHlDUW7YSAOOmZkNvo0xpWZmZkPAAcfMzAaFA46ZmQ0KBxwzMxsUDjhmZjYoHHDMeiBpg6QHco+G/Xtf0qT8VX/NhrNRPc9ittH7e0TsM9SFMGt1ruGY9ZOkJySdJene7LF7Nn5XSbdm9x+5VdIu2fiXSfqZpAezxxuzVY2U9P3sfi43Sdosm/8kSY9k67lqiHbTrGEccMx6tlkhpXZkbtrzETGd9G/uc7Nx3yNdEn4v4HLgu9n47wK/joi9Sdc4m5+NnwKcFxGvBlYC/zsbfyrw2mw9Hy1r58wGi680YNYDSasjYssq458A3hYRj2cXSX0mIraT9Czw8ohYn41/OiK2l7QMmJi/vEp224ibsxtnIekLwCYR8VVJNwCrSZctuS4iVpe8q2alcg3HbGCixuta81STv77XBjrbVg8h3Z32dcB9uSsfm7UkBxyzgTky93x39vou0lWqAY4G7sxe3wp8DNKtzrM7dFYlaQSwc0TcTrqR3DigWy3LrJX4F5NZzzaT9EBu+IaIqHSNHiPpHtKPt6OycScBF0v6HOkOnB/Kxn8KuFDScaSazMdId6isZiRwmaSxpJtonZPdLtqsZbkNx6yfsjacaRHx7FCXxawVOKVmZmaDwjUcMzMbFK7hmJnZoHDAMTOzQeGAY2Zmg8IBx8zMBoUDjpmZDYr/D6hwyhArJrfKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAAEWCAYAAAA3h9P4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXm4HFW1t9+ViSQQSMhgZhIgYpAhJIEEkVFURESioiAyKSCKA4rei36O9+pVEEG4KorMgyCjonJRhogiMiQyJQRMIDMJCSQQQgaSk/X9sarsOn26z+k+pzrdOfm9z9NPd+0a9qqu7vrVWnvtvc3dEUIIIRqVLvU2QAghhGgNCZUQQoiGRkIlhBCioZFQCSGEaGgkVEIIIRoaCZUQQoiGRkIlhNiisOAxM9s9p+MtNbN35nGsEsfexsxWm9nQMuvPNLN7W9n/YTP7RC1sqzfJdXzczMa0tW0uQmVm88xsbXJBXjKzq8xsuzyOnReJjYdXsN1oM9tkZj/fHHbVCzMbYmZXmNkSM3vdzJ41s++a2bb1tq0tzOyjZvaQma0xs7+UWD/OzKYn66eb2bjMOjOz88zsleR1vplZHvtmtjsh+S+sTv4XmzLLqztw3m8zs41tbPNDM9uQXNP0ul5sZoOqqKdDN0czG2VmvzWzl83sNTN7ysw+nmPdHwEWu/szyT7pOa82s1fN7EEzm9he+yvFzHqamZvZ8KLyH5rZ5QDuvt7dt3P3F2ttTzVkbawXHp14LwK+09a2eXpUH3D37YDxwL7AN6o9gJl1y9Ge9nISsBI4zsy22ZwVb67zN7MdgX8AvYD93b0P8G6gL7BLO463ua/bCuAnwA9L2NID+B1wPdAPuAb4XVIOcAZwDLA3sBdwFPDpju6bxd1vSG5O2wHvA15Ml5OyWnNNck37A8cCo4BpZjZwM9QNcCPwHDACGACcCryc4/HPBK4rKrsm+W4HAg8Dv8mxPpEzmXvG7cD7zax/qzu4e4dfwDzg8Mzyj4A/JJ93AK4AlgCLge8BXZN1pwB/J1R1BfC9pPx0YBbwOvAMMD4pHwrcBiwH5gJfyNT5HeBm4Npkv5nAxGTddcAmYC2wGviPVs7leeAzwEvAR4rWvR24J7H1JeDrSXlX4OvJvq8D04k/6SjAgW6ZY/wFOK3c+RNCcT/wCvHnvgHom9l/RHJxlyfb/BTYJtl/z8x2g5LzHVjiHL8HPA10KfMdVGv3D4BXgT0y2w9M6h+ULB8FPJFs9xCwVw6/u9OAvxSVvSf5nVmmbAFwRPL5IeCMzLpPAQ93dN9WbDwEWFSifAQhii8DLwBnZtYdADwOrAKWAj9Iypcl12V18tqnxHF/CFxeVNad+D+l/6+BwP8lv6EViR1DknU/BpqAdUkdP07KLwUWJTY9Ckwuc74GbADe1sp3ciDwSPJb+CdwQGt1F+3bOzn+gHLnTDwsO9AnUzYFeCqp82/A7pl1S4F3Jp9vAr6RWXcEMKfMefRM6hle7hoUb0P8L+9Kvsd/EP+dezP7vh+Yndh5ISG6n8is/zTxELAC+CMwrKie04n70ErgolauQYvfSWbdt4j76+vADOD9me9+FTAms+1wYA3JPaqC7/krxL15Tab8b8DHWvsf5d5GZWYjgCOJPxrEU+lGYFdgH+JmcFpml0nEH3UQ8H0zO5YQnZOA7YGjgVfMrAvwe+BJYBjwLuBsM3tv5lhHEz+0vsCdxE0cdz+RuOF8wOOp9vwyth9IfPE3EaJ3UmZdH+Be4G5CMHcF7ktWfxk4Pjnv7YFPEhevEpqdP/FH/0FSx1jihvadxIauwB+A+YSYDANucvf1ic3ZkMnxxB9geYk6Dwdud/dNFdrYlt3/RYjn8Zn1HwUecPdlZjYeuJL4k/UHfgncWSOP9e3AU578AxKeSsrT9U9m1j1ZtK69+1ZMch3vIoRvKHEz/LqZHZxs8lPgf9x9e2AM8Nuk/CCgyQve2eNUgLtvIP47ByZFXYBfACOB0UnZRcm25wCPEQ8l2yXLEDfVPYnr9zvgFjPrXqIuJ0Tol0mItjgsNio5n/8H7EhEXn5rZv1aqTvLWGCVu5f00JLf1InETXF1UjYZ+Dnh2fUnHlx/W6cIzmWEyLyFeCD+ZLrCzAYT951ziIeJ5cDEzPrjgLOBDyT7P054/1neR9xnxwOnmtkh7bDxOeAdhJNxHnCTmQ1w9zXArTS/z5wA/NHdX63we/4YEb3JelCziChFeVpTsUpfhEe1mlDR+YmxvYgvcz3QK7Pt8cDU5PMpwIKiY/0J+GKJOiaV2PZrwFXJ5+/Q/Mlkd2BtkY2Ht3EelwO/TT7vTzy5DcrY/XiZ/Z4DPliifBRteyYL2rDpmLTexKbl2eMVfT8LSbwkYBrw0TLHnE3mCT4PuwnxeyGz/HfgpOTzpcB/l/jODu7g766UR/VNQryzZTcA30k+N5F52ieEwIkHhHbv24qNh1DkUQEHA7OLyr4LXJp8fpS4kfcv2uZtwMY2vpOST8rEDe7pMvtMBpZklps9xZfY3ogHsd3KrB9ARFVmEZGMaSTeH/Bt4FdF2z9A8kRdQd3vAuaVOOf1xP2nifA8D8isvwr4f0X7zAcmJZ876lG9ltSdvtZRwqNKPm8CRmWOcSHJfYsILf8ls65rci6fSJanAidk1ncn7lFvydQzMbP+TuDsan4nZbZ9Fnhv5rc7J7PuaeDoKr7nj5c4/o+Bn7dmQ54e1THu3tfdd3L3z7r7WmCn5MtckjRyvko8TWcbdhcWHWcE4boWsxMwND1OcqyvExcpZWnm8xqgZ6VPTWbWi4jn3wDg7v8gvLC0EbicXW2ta4tm529mg8zsJjNbbGariCemAZl65rt7iwZ1d38EeAM42MzeRnh8d5ap8xVgSDvtLWk3Ea7sZWaTzGwnYBxwR7JuJ+Ccoms3gvAmmmFmX88kHvyiHXatJrzaLNsTYYxS67cHVnv8YzqybzXsBIwq+j6+DAxO1p9MtIH9y8weKYoatJdhxJM8ZtbHzK40swXJb+zPFH5jJTGzr5nZc2b2GhFW6lluH3d/2d2/6u5jk3P6F+FxQ5z7J4rOfSIlfgtlWAn0KVF+nbv3JX7XzxO/v5SdCI81W+dA4jvJg7cn976+iQ0/KbPdYELks/+d+ZnPQ7Pr3L2JCEWn7AT8InMOy4loVdZrLb4HVt0mamafShJg0np2pXCt/wp0NbP9LRKNhhBh5NS+tr7n4vsGxPV8tTWbap2evpB40hmQuZDbu3s2XFL8J19I6Qb9hcDc7A/C3fu4+5EV2tLWzWQKceP5uUW66lLiC07Df+Xsam3dG8l770zZ4KJtiu36QVK2l0fo5xPEjzutZ2Qr4ntNsv2JwK3uvq7MdvcCU5JwaimqttsjjHgz4Xl+nGijTG/wC4HvF1273u5+Y3HF7v4/XghtnVnGvtaYCexl1iwbb6+kPF2fDTPsXbSuvftWw0Lg2RK/5SkA7j7L3T9GPNBdAtyeJHRUK4jAvxuujyLaAgDOJW5u+ya/sfdQ+I1RXI+ZvRv4PPEf6UuE7NYW7VMSd19GeA2jLDJKFxJP8tlz39bdLypVdwlmAX3MrJxILiNCzD/IbLMQ+FaJ39/tJQ7xBq3/7jvCUuL8RmTKRmY+L8muS/6fxTf5U4rOo5e7T8/LQDN7K/C/hHe3YyK8c0iudfJQdi2F+8xNHqHl1L62vudS13cszUPqLaipULn7EuJp7cdmtr2ZdTGzXTKx+FJcDnzFzCZYsGvyhP4osMrM/tPMeplZVzPbw8z2rdCcl4CdW1l/MtGOsifxNDaOaNQeZ2Z7Em1Dg83sbIu+EX3MbFLG5v82szGJzXuZWX+P9qHFxBNkVzP7JG1n1fUhCaOa2TDgq5l1jxI/5h+a2bYW6bEHZNZfR9xMPkH8mMpxISHK1yTfLWY2zMwuNLO92mk3wK+JGPQJyeeUXwFnJt6WJba/36Ldr2oSm3oC3YAuyfeQtpf8hQj/fCG5Tp9Lyu9P3q8Fvpyc71CiPeDqHPathgeT8zg7sb1b8psZn5SflPx+moiwkhMho2XE0+zIskfOYGbdzeztxANEH0L0SD6vIX5jA2iZoVv8X+lDhJiWAz2INsmerdR7gZntnlynHYgsvRnu/gbxMHWsmb0rWd8r+ZwKQqv/0yRS8xeiva7cNk8RT/5pG9dlwOfNbGLy+9vOzI42s94ldn8COMrM+ib/v8+Xq6dakgfH3wPfTc57L+K/knInsK+ZHZX8nr9KPBSk/AL4hpntBmBm/czswx0wqWvy+0tfPQgPbBNxrbuY2ZmER5XlWqIN+nia32eq+Z5JzmFb4p57X7ltgNpk/RWt24FCxtBrRAPgccm6U4AHS+xzJtGGsZrIOknj20OJ1NelRAjg4bReoo3q+swxRpFpZwE+SITyXgW+UlTfMMKF3rOELXcBFySf90i+0JWJDed6IZb8DQqZMo9RyPJ5X1L+KhGLfYDmbT0PFtX3diJrcDXxpzmHTBsH8QT2WwpZgZcU7X9vcj3Ktptkvssrk/N4nYhDfxvo3R67M8edQ4SYehSVH5F8L68SYnsLmaysKn9vpyTXNvu6OrN+n+Q7XEtkle2TWWfA+YmNK5LPlse+ZWw9hPJZfzcTN+aVRJveQcm6m5Nr+zrRBnBkZr/ziJvIq8C4Esf9ISEqrxPewb+IJ+QhRb+hB5Pf2LPAZ8m0fZG0QyR2nU+E768jMr4WE+1d/27XKWHDZcn+qxNbf0fzTLEDkvpXEuJ7JzC0VN1ljv9h4I6icy7OdDw4sXfHZPno5Lq+BrxItEX1StZl26i2JcKUq4h71VfIN+tvMJGQVS7r7wPJ+ZfL+vsU4cWvIsKGvyhnC0XtbSVsLP4PzUnWXZB8/8uT31uLdsPk+j1X4rgVfc+Z7U8Eft3Wf96SjUUnwcyuJPrtVN2PTYgtgSQ0+wgRBnum3vZsjZjZr4Fn3P17HTiGEaJ2nLv/q9VtJVSdB4vU3ycIL2Bufa0RQnRGzGxXItow1t0Xt7V9Hmisv06Cmf03ESb9kURKCFELzOx8IiT6X5tLpEAelRBCiAZHHpUQQoiGphEGgc2NAQMG+KhRo+pthhBCbDFMnz79ZXffXAMWt4tOJVSjRo1i2rRp9TZDCCG2GMxsfttb1ReF/oQQQjQ0EiohhBANjYRKCCFEQyOhEkII0dBIqIQQQjQ0EiohhBANjYRKCCFEQyOhEvnz61/DqyUm7Lz3XvhXq4Mkd5xbboFly2pbhxBisyKhEvnyr3/BCSfArbe2XPfJT8I3v1m7ul95BT76UbjsstrVIYTY7EioRL48+2y8r1vXct3atTBjRu3qXriw+Xs9+fOfYXpuM4QLsVUjoRL58txz8b5hQ8t1GzaEx/Xmm7Wpe9Gi5u/1YupUeN/7YL/94CtfgTVr6muPEFs4nWqsP9EAtCVUGzeGWO2xR3XHXb0arr++IHLbbQcnnQTdMj/hVKAW13CanDVr4Pe/h6OPhl69Wq5fsgSOPx7e+lY46CD48Y/hd7+Dz3ymYOvkySFi1bJ+Pdx+Oxx+OAxs6DFEhcgVCZXIl7aECmDmzOqF6ic/adm+NXJk3LRTUoGqlUf117/Cpz4Fc+bAiSfCNdeAWWH9xo1w3HHw+utw//2w++6xfNppcM45he222QYeegjGj6+87kcfjTa+mTNhwAD46U+jPS5bvxCdFIX+tmZmzsy/PaecULk3F6pq2LQJrroKDj44EiaeeirKX3ih+XapQL3ySrSH5cWGDfC5z0X9mzbBKafAddfB5ZcXtnGHr30txOyXvwyRAjj00PAgX3klXi+8EN7Qscc2z4xcuhT++Mc4fpa1a+E//gP23x9eey3qHD06BPDDHw4Pbktg/vyw/Ve/itfvftfyXIUoh7t3mteECRNcVMHuu7t/7GP5HW/FCve4Zbt//evN123YUFg3ZUp1x506Nfa77rpY3rjRvVs393PPbb7d4YcX6pg9u92n0YJbboljfvaz7qtXuzc1ub/nPe7bbOM+fbr7/PnuRxwR23z6020f76GHwv5jjoljXXONe79+sf+hh7rPmRPb/e1v7mPGRPkZZ7i/+mqUb9jg/qMfuffs6d63r/vVV7tv2pTf+eZJU5P7xRe79+5duDbp6+CD871Ool0A07wB7t+tvRT625pZuhS23z6/46XeFEQYLEt2uVqP6qqrws4PfSiWu3aNsN+8ec23W7QI+vWDlSvj8667VldPOZ58Erp0ifamnj2j7PrrYZ99oq1q1SpoaoJLLoGzzmr7ePvvD+efD1/+coRAZ82Cd7wjzu+//gv22guOOALuuAN22in6n73rXYX9u3WLJI2jj45w4CmnwE03RbeAakKBw4aFl5jdZ+1auOuu0lmb1bJpU3iXf/87HHkk/OhHsMMOse7uuyMcutde4YnuvHPH6wPYc884Zpbly+M77GweXKlz7azUWynzfMmjqoKmJncz95Ej8zvm1VcXnpa/9KXm61ativLttnPv0sV97drKjvnaa+69eoVHkeVd73KfNKmwvGmT+7bbFjyb1PvKgylT3HfbrWX5gw+69+jhfthh7s8/X90xN21y/8hHwtO4+OLwEt3dFy50P/LIuDZf+IL766+3fpymJvdLLintsVTymjLFfcmSwvm89a3tO065V79+7tdeW9rjW7TI/aij8q2vSxf3c85xf+ONqPOGG9z798+3jkZ5deni/uUvx7l2AORRiapYuBC6d4fBg2tf16pV8XNfsiSeNLvk0Fz53HPxtN+rV8s2qnR5773jCfvZZ2HcuJbHeO01ePFFGDs2lm++OZ7yTz21+XajRsEf/tD8fN54AyZNiqf1PBMqZsyIp9diDjggntb79Kk+qcEsvKA1a2L/lOHD47xWroQdd2z7OF26wOc/H8kdy5dXZ8Mdd8C3vhXtaUccEfaMHBlZjbvtVt2xyjF4cPPzyzJsGNx5JyxYkE+XhaamSLpJMy3HjIH/+7/IsvzDH6B//47X0SikHvyFF8a5XnFFeMedlXorZZ6vLd6j2n//8Ag2B88/X3gyW7Ysn2N+6EPheQwa1LKtZunSqOuss+L9+utLH+Pss71Zm8z++7uPHdvyifx734vt0qfJGTNi+cYbo93mc5/L55zWro0n1299K5/jNRrPPut+wAHx3X3+8217cFsC99/vvvPO4YlfdFHBW+2MTJ0a5zpgQLSftgO2do/KzL4InA4Y8Ct3/4mZ7Q38AtgOmAec4O6rivYbAVwLDAY2AZe5+8W1tLUhmD07+so0NUU7TFs0NcU+b3tb9XWtWFH4/OKL+fTLee65eBKfPr28R7X77uE1lhuhYsaMeAK//PJ4sl+yJNpzij2WUaPiff788L5SD2r48Hjl5VE9+2x4nG9/ez7HazR22y0yFV96CYYMqbc1+XDoodEOunp1pPJ3Zg45JLJgZ82CbbettzU1o2bp6Wa2ByFS+wF7A0eZ2RjgcuBcd98TuAP4aondNwLnuPtYYDJwlpntXitbG4J16+Dll6MPzqxZle1zyy1xA12woPr6ioWqozQ1Rf+i3XYLISonVL17R2fYcgkVc+bAUUdFP6N+/SKMeOKJLbcbPTre586N97QPVd5ClQpqZxUqiPBhZxGplJ49O79IpWy7LUycWG8rakot+1GNBR529zXuvhF4AJgC7Ab8NdnmHuDDxTu6+xJ3/2fy+XVgFjCshrbWn6xYPPxwZfs880w87VcqbFlWrixddzHr1lU20sP8+eENtiVU3bvHTb+UUL35ZojurrtGW9Pjj8Pzz5dus0s9qjTzLxWmoUPzFaqZM8PmMWPyOZ4QompqKVQzgIPMrL+Z9QaOBEYk5Ucn2xyblJXFzEYB+wCP1MzSRiB7Y32kwlNNvYnnn6++vko9qh//OEJrpabtyJKmprclVN26RUr2Cy9E8kOWefNCeHfZJZZ79Cj/pD94cIzwkH4HixbBoEGxz7BhEcrKo4F+5szwAHv06PixhBDtomZC5e6zgPMIr+lu4EkipPdJIpQ3HegDlL2bmNl2wG3A2cXtWJltzjCzaWY2bXm1WU+NRCpUo0dX7lGl3sScOdXXlwpVnz6tC9XcuRGO/M1vWj9esVCV60eVelTQ0hNMBbeS/k9dukQfo6xHNXx4fB4+vJDR2FFmzKh+uCchRK7UdAgld7/C3ce7+0HACmC2uz/r7u9x9wnAjUBJd8DMuhMidYO7395KHZe5+0R3nzhwSx6oMxWqD384nuJXldTl5nTUo+rdO4SxNaFKxf/KK1s/3nPPRZvSgAHhNbUW+ktv/MUJFangph5VW4we3dyjygoVdHxw2jfeiON35vYpIbYAaipUZjYoeR8JfAi4MVPWBfgGkQFYvJ8BVwCz3P3CWtrYMCxaFKMvHH54eAOPPdb69uvXFwSmPR5V2k9n6NDWherll+P90UdbH1Eizfgza7uNapddImxXfLznn4+G4be8pbJzGD26vEeVllWDe3iPKanHJ6ESoq7UelDa28zsGeD3wFnuvhI43sz+BTwLvAhcBWBmQ83srmS/A4ATgcPM7InkdWSNba0v6Y02nf6hrXaqBQvixjpoULT3VDs8zIoV4QG1JVTLl8Nhh4WXdNVV5bdLhQraFqquXaPdKx1cNmXOnBCxSjvPjhoVA70uXRrC2xGhco9pQ0aNKniRqcen0J8QdaXWob8D3X13d9/b3e9Lyi5297cmr3OTDme4+4vufmTy+UF3N3ffy93HJa+7Wqur4Vm9uvX1ixfHDbZfv+gX1VY7VepJHH54ZOZVm2K+YkXBo1q6NNLLS/Hyy9H36QMfiBHDS03f8frrUX+lQgWRTvvYYyEQKc8/X934fGmK+t//Hu+pQO2wQ3hmWaFat65lu1mWX/wixu9bsSLGpIPw+LbZpvJQpBCiJmiaj83BkiXRofZ//qf8NtnQ1eTJIVTZm3gxadtMOlhpte1U2dDfpk2wbFnLbTZsiO0GDIjBT5ctiwFLi5k9O96rEarJk+PY6b5NTeEZViMKaYr6gw/Ge/r9mTVPUXeP4Y4++9nSx5k2Dc4+OwZO/fjHY66nl14KoRo7trLO10KImiGh2hzcd1880X/jG/G5mA0bQszSG+2kSRF+SsWoFHPnxk3/wANjudp2qqxHBaU9sjQzcODAGAtu8ODSSRXz58d76uFUKlRQ8BwXL4508vZ4VMVClX5Oherhh+Gf/4Q//7nlMVaujLmhBg+Ga6+F73wn7DjvvAj9qX1KiLojodocTJ0aIb2xY+OJvVgUli6Np/6sRwWth//mzYsBREePjvajaj2qbBsVlBaqtK0mzeQ78cSY3K84IzEdGWPkyHhvqx8VxHex/faFc0ztr8ajGjAgMhcffzyWh2X6hA8bVsj6S8V1/vz4rrN861ux3c03x6ClY8bEef785zFIsIRKiLojocqD1kJ0EEJ18MFw662R8nzccc3bS7Lj1EE03vfu3XpCxdy5Efrq1i3eq/Go1q4ND68tjyoVqjTtf/LkCNEVi+KCBWFvOtp3W/2oIPpB7btv4RxT+6vxqMxCqJuaQnR79y6sGz48zintA5bOuFv8nd51V4T8Jk0qlH3zmwV7lUghRN2RUHWUG2+Mm/2aNaXXz5sXonLooeFFXHYZ/O1vcOmlhW1SoUo9gm7d4ib+0EPl6503rxD62nXX0h7Vpk3ws5+F53HHHYXydPikHXeMVHCz0kKVpqanY6albULFIckFC8KbSrP12upHlTJ5ckxKuGZN2N+9e/PwXSWkNhXvN3x4iM3Pfx5ideGFcfysl7pgQbSLHXpo83133rkwrYiESoi6I6HqKBddFOGkhQtLr586Nd7Tm+HHPx7hpfvvL2yTHVA15cADI6RVquPvmjXR2J/epHfZJTySrGc3e3aMrPy5z0UKd1b00ranHXcMUXnLWyrzqFJhLJ5ZNxWqlEraqKDgoU2fHvbvvHP1iQupTaWECuCCC0LI3/OemP8q61EVX5ssF1wAt99eOL4Qom5IqDrCjBmFjrnFbR8pU6eGR5Jt6yjO6lu0KEYJ79evsM2hh8ZN/G9/a3nM4uSFXXcNQUs9oPnzY5r0p56K9pkxY5qPsJ4KVVpfub5U6fHSCef69o12pXIeVUqlQpWG2x55JDyq9qSBt+ZRpedwyinh7U2aFB2X01T8qVPj3Ep5TTvsAFOmVG+PECJ3JFQdIdsBttS4cu5xMzzkkOYz6E6eHMKWikeamp7t6Lr//jEQavrUnyUViqxHBYXw35VXhtf12GMRwtppp9JClbYplROq5ctDnFJxSduEsh7V+vVxLu0RqoEDw4v6xz/Co6qmfSqlLY/KDE4+OT5PnhxthDNnlr82QoiGQ//Q9rJhQ3SAPeSQWC7lUT3/fIhQcWgp9STS9pJsH6qUXr1CrEoJVSoUWY8K4ma/aRNcfTW8+92FqSlGjmwuVNk2Kmjdoyqe02fUqOYeVdq+1h6hghCPe+6JDtHt8ajKCdWAATEn0XvfWzqbcu7c+E5Khf2EEA2FhKq9/PGP4XGcc07cfEt5VOXaQPbaK26irQlVut/jjzefOwriJrvNNoUx8UaPDs/h+eej7WvBguigmzJyZNiXTntRyqNatqyluCxf3nLm39SjSsOWxanpUL1QpePrtcejGjcOfvUr+OhHm5ebwa9/DRdnJobeeecQsEceab19SgjRUEio2suVV8ZcSWlH2FIe1dSpsa54qvju3WMIoUceCQ9o8eLmfYBSDj00BOGvf21ePm9eeDZpyKpnzxC6OXPCrr594YMfLGw/cmQcJ03aWLEikhb69InlNEW9+BzKeVRvvFFov8pDqFLa41GZwWmnwXbbtVw3ZUrMJZXddtKkeECYOjWEfuzY6usUQmxWJFTtYenS6H9z0kmRNVdKqLJtIKUGWZ08OUZLWLQo0qhLeVSTJoUIFYf/0j5UWXbZJY53xx1wwgmxX0oqIqmopJ19U7vK9aUq51FBIfyYHjNrf2v9qNIOvyl77x3eYZcuLc+pFkyeHDMj/+lP5a+NEKKhkFC1h+uui8yxtK/NkCEtQ3+zZ4d4lQstTZoUiQh//GMslxKqbbaJMeqKhSrbhypl113jBrxuXcGulGKhSsf5SyklVO7lPSootFPnv9zwAAAds0lEQVQtWFCYbTelW7fYPzvQbepRFaef9+gB48fDiBHNj1ErUg/u5ZcV9hNiC0FCVS3uke23//6FQVhLeVTpXEb77FP6OOkN89Zb471cR9dDD4008zTU9vrr0S+qlEcF0f41fnzzdemxsx5VW0L1+uvRplXsUaX1Zj2qbNgPCuG9bPhvw4YoL+XB/PCH8JOftCyvBfvuW7BBQiXEFoGEqloeeSREKJusMGRICEn2xpzeyMuFs4YPj3apv/ylsFyK9Gb6wAOtHzdNRDj11JZi0KtXzFtVHPpLGTgwPJ2sUBWPSpGy/fYhclmPqhqhKsVBB8Exx5Relzc77BDtUkOHFrIihRANjYSqWq66KsaUy2aZDR4cnlZ2qoy5c2NOpOIbfZbJkyOZonv3lp5Lyr77xnHS8F8qEMWhv/e+F849Fz71qdLHyaaoF3tUXbqE2GaFqnhUiizZzL/58zsuVJub886DSy5R+5QQWwjd2t5E/Js1a2Jsv2OPDc8iZfDgeF+6tJC9l2bmtXYznDQJbrst9inX6bR7d3jnO6Nv1AMPFFLViz2qPn3gBz8oX9fIkfDss/G5uI0KwsNIswKh4FGVEqpRo2JUjldeiQFutzShOuqoelsghKgCCVU13H57tN0UJysMGRLv2YSKuXPbHicubacqlZqe5T//M9Kv075Lu+xS3gMrx8iRMR9TUxO8+mpLodp11+Zp8NkpPooZPTqSQNKhnLY0oRJCbFFIqKrhyitDJA46qHl51qOCEJR582Jqj9aYMCHahtoaMfzQQzve8D9yZIz+kIYOs21UEEkfv/51IdOvLY9q3boYNy89dhYJlRAiR9RGVSlz50Y7UTrAaZZioVq5MgaJbatfUO/eEa477bS8rW1JKiZPPBHvxR7VuHHxnk5CuHx5pI6X6kibeoqpB1ZOqLJ9qTZubNmHSgghKkBCVSnXXNN8gNMs22wTHkoa+isei681vvpVOPzw3MwsSyomTz4Z78VClabRp0L28svhTZVqY0sF+IEHomNxcXgwFSR5VEKIHJBQVcq0abDnntExtRTZvlTFo5s3Am15VP37x7llPapyGYvpeS1ZEvsUi5lCf0KIHJFQVcqiRS1DXFmGDCkIVTUe1eZi4MDw/FKhKm6jgvCqUqFKPapS9O4d/bKg9HcioRJC5IiEqlIWL2496WHw4ELob+7c6Fjat+/msa0SunQJ7yedlqPYo4IQqueei0FnW/OooCDCEiohRI2RUFXCunXhYbQlVEuXFjL+GsmbSsmKSimPaty4sP/pp0sPSJslDf9JqIQQNUZCVQlpR9jWhGrIkOj8umpV6dHNG4FUVLbbrrRopAkVjz4Kr70mj0oI0RBIqCohDZe15VFBhP8a3aMqFfZL1/frB/feG8vyqIQQDYCEqhJSoWptBIl0dIqnn46hlhrZoyonVGbhVaXjCrbmUR1+eHRCnjCh5Tr1oxJC5IiEqhIqEarUo/rHP+J9S/SoIIRq9er43JpHtcsuMe19qbYu9aMSQuSIhKoSFi+OLL506vZSFAtVI3tUpcQlJR2hAlr3qFpDoT8hRI5IqCph0aK2x+Pbcce4Ef/zn7HciEKVdlZuy6NKqXbg2xQJlRAiRyRUlVCJUJmFV/XmmzHKQ2veV73o3Rs+8IHWB8vdbbcYFgniPNqDhEoIkSM1FSoz+6KZzTCzmWZ2dlK2t5n9w8yeNrPfm9n2ZfY9wsyeM7M5ZnZuLe1sk0qECgrhv0Zsn0q580444YTy67t1i+ns+/Vrf/KDhEoIkSM1Eyoz2wM4HdgP2Bs4yszGAJcD57r7nsAdwFdL7NsV+BnwPmB34Hgz271WtrbKhg3RkbcSoUoz/xox7FcNH/kIvPvd7d9fQiWEyJFa5guPBR529zUAZvYAMAXYDUhn6LsH+BPwzaJ99wPmuPsLyb43AR8EnqmhvaVZsiRGa2hrckPYMjyqSvhqi2eH6pBQCSFypJahvxnAQWbW38x6A0cCI5Lyo5Ntjk3KihkGLMwsL0rKWmBmZ5jZNDObtjydlTZPKhmVIqWzeFQdRf2ohBA5UjOhcvdZwHmE13Q38CSwEfgkcJaZTQf6AG+W2L3EJEh4mXouc/eJ7j5xYHuz1FqjklEpUjqLR9VR1I9KCJEjbQqVmd1mZu83s6pFzd2vcPfx7n4QsAKY7e7Puvt73H0CcCPwfIldF9Hc0xoOvFht/blQjVBNmBBJCHvvXVubGh2F/oQQOVKJ+FwKfByYbWY/NLO3VXpwMxuUvI8EPgTcmCnrAnwD+EWJXR8DxpjZaDPrARwH3FlpvbmyaFGkdVcyZce++8KKFTB0aO3tamS6do33VKjcoalJQiWEaBdtCpW73+vuJwDjgXnAPWb2kJmdamZt3XluM7NngN8DZ7n7SiKD71/As4SXdBWAmQ01s7uSOjcCnyMSLWYBN7v7zHadYUdJU9NLTckuSmMWopQKVfouoRJCtIOKWrfNrD/wCeBE4HHgBuCdwMnAIeX2c/cDS5RdDFxcovxFIuEiXb4LuKsS+2pKpX2oRHMkVEKInKikjep24G9Ab+AD7n60u//G3T8PbFdrA+vOokWVpaaL5kiohBA5UYlH9VN3v7/UCnefmLM9jcWmTfDii/Ko2oOESgiRE5UkU4w1s39nEphZPzP7bA1tahyWLYv+PxKq6unevdCPKn1XPyohRDuoRKhOd/dX04UkIeL02pnUQFSTmi6a062bPCohRC5UIlRdzAopb8k4fD1qZ1IDIaFqPwr9CSFyopJYzJ+Am83sF8ToEGcSI010fiRU7UdCJYTIiUqE6j+BTwOfIYY2+jMxAnrnZ9GiuLm2d6bbrRkJlRAiJ9oUKnffRIxOcWntzWkwFi+O1PQuml+yaiRUQoicaFOokjmkfkDMC9UzLXf3nWtoV2OwcKHCfu1FQiWEyIlKXIWrCG9qI3AocC1wXS2NahjmzdOUHe1FQiWEyIlKhKqXu98HmLvPd/fvAIfV1qwGYMOG8KgkVO1D/aiEEDlRyZ1jXTLS+Wwz+xywGBhUW7MagEWLYmSKrX1uqfbSrRu88UZ8lkclhOgAlXhUZxPj/H0BmEAMTntyLY1qCObOjXd5VO1DoT8hRE606lElnXs/6u5fBVYDp24WqxqBefPiXR5V+5BQCSFyolWPyt2bgAnZkSm2GubOjbR0Zf21DwmVECInKmmjehz4nZndAryRFrr77TWzqhGYNw9GjNDNtb1IqIQQOVGJUO0IvELzTD8HOrdQzZ2r9qmOIKESQuREJSNTbD3tUlnmzYN3v7veVmy5SKiEEDlRycgUVxEeVDPc/ZM1sagRWL8+JkxUIkX7UT8qIUROVHLn+EPmc09gCvBibcxpEBYsAHeF/jqC5qMSQuREJaG/27LLZnYjcG/NLGoE0j5U8qjaj0J/QoicaM+w4GOAkXkb0lCkfajkUbUfCZUQIicqaaN6neZtVEuJOao6L3Pnxk116NB6W7LlIqESQuREJaG/PpvDkIZi3jwYORK6dq23JVsu3btDU1O09UmohBAdoM3Qn5lNMbMdMst9zeyY2ppVZ+bOVftUR0lFacOGeHXtClvhACdCiI5TSRvVt939tXTB3V8Fvl07kxoAzUPVcYqFSt6UEKKdVCJUpbbpvB1i1qyBl16SR9VRUmHauDFe6kMlhGgnlQjVNDO70Mx2MbOdzewiYHqtDasb8+fHuzyqjpEKkzwqIUQHqUSoPg+8CfwGuBlYC5xVS6PqivpQ5YNCf0KInKgk6+8N4NzNYEtjoAkT80FCJYTIiUqy/u4xs76Z5X5m9qfamlVH5s2Dnj1h8OB6W7JlI6ESQuREJaG/AUmmHwDuvhIYVDuT6szcubDTTkql7igSKiFETlQiVJvM7N9DJpnZTpQYTb0UZvZFM5thZjPN7OykbJyZPWxmT5jZNDPbr8y+5yf7zTKzSzbbLMPz5ql9Kg8kVEKInKgkZ/j/AQ+a2QPJ8kHAp9vaycz2AE4H9iOSMe42sz8C5wPfdff/M7Mjk+VDivZ9B3AAsFdS9CBwMPCXCuztGK+8AmPH1ryaTo+ESgiRE5UkU9xtZuOByYABX3L3lys49ljgYXdfA5AI3RTCG9s+2WYHSk8Z4sSUIj2SOrsDL1VQZ8dZvz7aqETHUD8qIUROVDR6uru/7O5/AJ4BzjSzGRXsNgM4yMz6m1lv4EhgBHA28CMzWwhcAHytRH3/AKYCS5LXn9x9VqlKzOyMJIQ4bfny5ZWcTuusWyehygP1oxJC5EQlWX9DzOxsM3sUmAl0BY5va79EWM4D7gHuBp4ENgKfIbyyEcCXgCtK1Lkr4ZENB4YBh5nZQWXquczdJ7r7xIEDB7ZlVttIqPJBoT8hRE6UFSozO93M7gceAAYApwFL3P277v50JQd39yvcfby7HwSsAGYDJwO3J5vcQrRhFTOFCBuudvfVwP8Rocfa4h6hv222qXlVnR4JlRAiJ1rzqH5GeE8fd/dvuPtTVJjtl2Jmg5L3kcCHgBuJNqmDk00OI8SrmAXAwWbWzcy6J9uXDP3lysaNsGmTPKo8kFAJIXKitRbuocCxwIVm9hZi+KRq7za3mVl/YANwlruvNLPTgYvNrBuwDjgDwMwmAme6+2nArYSIPU2I493u/vsq666edeviXULVcSRUQoicKCtUSWbfpcClZjYcOA5YZmazgDvc/ettHdzdDyxR9iAwoUT5NCK8iLs3UUEKfO6sXx/vCv11HAmVECInKs36W+TuF7j7BOAYYH1tzaoT8qjyQ0IlhMiJqju3uPtzwHdrYEv9kVDlR3E/KgmVEKKdVORRbTUo9Jcfxf2o1OFXCNFOJFRZ5FHlh0J/QoicqKTD732VlHUKJFT5IaESQuRE2XiMmfUEegMDzKwfMeYexDh9QzeDbZsfhf7yQ0IlhMiJ1hoOPk2MyzcUmE5BqFYRnYE7H/Ko8kNCJYTIidb6UV1MdMz9vLv/72a0qX5IqPJDQiWEyIlKpvn432R+qFHZ7d392hraVR8U+ssPCZUQIifaFCozuw7YBXgCaEqKHeh8QiWPKj/MoGvXwncqoRJCtJNKOrdMBHZ396oGpN0ikVDlS7dusHZt4bMQQrSDSvpRzQAG19qQhkChv3zp3r0gVPKohBDtpJLH3AHAM8nEif8e48/dj66ZVfVCHlW+dO8Oa9YUPgshRDuoRKi+U2sjGgZ5VPkioRJC5EAlWX8PmNlOwBh3v9fMehMTKnY+1q2LG2oXjSyVCxIqIUQOVDKE0unERIa/TIqGAb+tpVF1Y906hf3yREIlhMiBSlyHs4ADiBEpcPfZwKBaGlU31q+XUOWJhEoIkQOVCNV6d38zXUimkO+cqerr1ql9Kk8kVEKIHKhEqB4ws68Dvczs3cAtwO9ra1adUOgvX9SPSgiRA5UI1bnAcuBpYqDau4Bv1NKourF+vTyqPFE/KiFEDlTymNsLuNLdfwVgZl2TsjW1NKwuyKPKF4X+hBA5UIlHdR8hTCm9gHtrY06dkVDli4RKCJEDlQhVT3dfnS4kn3vXzqQ6otBfvnTvDk1Nhc9CCNEOKhGqN8xsfLpgZhOAtbUzqY7Io8qXrDhJqIQQ7aSSNqovAreY2YvJ8hDgY7UzqY5IqPJFQiWEyIFWhcrMugA9gLcBuxHT0T/r7hs2g22bH4X+8kVCJYTIgVaFyt03mdmP3X1/YrqPzo08qnzJ9p1SPyohRDuppI3qz2b2YTOzmltTbyRU+SKPSgiRA5U85n4Z2BZoMrO1RPjP3X37mlpWDxT6yxcJlRAiByqZ5qPP5jCkIZBHlS8SKiFEDlQyzYeZ2SfM7JvJ8ggz26/2pm1mNm2CDRskVHkioRJC5EAlbVQ/B/YHPp4srwZ+VjOL6oVm980fCZUQIgcqEapJ7n4WsA7A3VcSKettYmZfNLMZZjbTzM5OysaZ2cNm9oSZTSvnnZnZSDP7s5nNMrNnzGxURWfUXtati3d5VPkhoRJC5EAlQrUhGYjWAcxsILCprZ3MbA/gdGA/YG/gKDMbA5wPfNfdxwHfSpZLcS3wI3cfmxxjWQW2tp/Uo5JQ5YeESgiRA5UI1SXAHcAgM/s+8CDwPxXsNxZ42N3XuPtG4AFgCiF4acbgDsCLxTua2e5AN3e/B2J8QXev7WjtqUel0F9+qB+VECIHKsn6u8HMpgPvIlLTj3H3WRUcewbwfTPrT4wNeCQwDTgb+JOZXUAI5TtK7PtW4FUzux0YTYzWfq67NxVvaGZnAGcAjBw5sgKzyqDQX/5kvaiuXetnhxBii6asUJlZT+BMYFdi0sRfJp5RRbj7LDM7D7iHSMB4EtgIfAb4krvfZmYfBa4ADi9h14HAPsAC4DfAKcm2xfVcBlwGMHHiRK/UvhYo9Jc/qVB17w5bQX9xIURtaC30dw0wkRCp9wEXVHtwd7/C3ce7+0HACmA2cDJwe7LJLUT7UzGLgMfd/YVEHH8LjC+xXX4o9Jc/WaESQoh20ppQ7e7un3D3XwIfAQ6q9uBmNih5Hwl8CLiRaJM6ONnkMEK8inkM6JckbqTbPVNt/VWh0F/+SKiEEDnQWhvVv0dId/eN7Rzq77akjWoDcJa7rzSz04GLzawbkfJ+BoCZTQTOdPfT3L3JzL4C3JeMMTgd+FV7DKgYhf7yR0IlhMiB1oRqbzNblXw2oFeyXPFYf+5+YImyB4EJJcqnAadllu8B9mqrjtxQ6C9/JFRCiBwoK1TuvnWlaSn0lz8SKiFEDlTSj2rrQKG//En7TqkPlRCiA0ioUhT6yx95VEKIHJBQpSj0lz8SKiFEDkioUhT6yx8JlRAiByRUKQr95Y+ESgiRAxKqlHXroEsXNfzniYRKCJEDEqqU9evDm9KYdPkhoRJC5ICEKmXdOrVP5Y2ESgiRAxKqFAlV/qgflRAiByRUKWnoT+SHPCohRA5IqFLkUeWPhEoIkQMSqhQJVf5IqIQQOSChSlHoL38kVEKIHJBQpcijyh8JlRAiByRUKevXS6jyRkIlhMgBCVXKunUK/eWNhEoIkQMSqhSF/vJH/aiEEDkgoUpR6C9/5FEJIXJAQpWi0F/+SKiEEDkgoUpR6C9/unaFffaBPfaotyVCiC0YNR6kKPSXP2bwz3/W2wohxBaOPCoAd4X+hBCiQZFQAWzYEGIlj0oIIRoOCRVE2A8kVEII0YBIqCDCfqDQnxBCNCASKigIlTwqIYRoOCRUoNCfEEI0MBIqUOhPCCEaGAkVKPQnhBANjIQKFPoTQogGpqZCZWZfNLMZZjbTzM5OysaZ2cNm9oSZTTOz/VrZf3szW2xmP62lnQr9CSFE41IzoTKzPYDTgf2AvYGjzGwMcD7wXXcfB3wrWS7HfwMP1MrGf6PQnxBCNCy19KjGAg+7+xp330gIzhTAge2TbXYAXiy1s5lNAN4C/LmGNgZp6E8elRBCNBy1HJR2BvB9M+sPrAWOBKYBZwN/MrMLCKF8R/GOZtYF+DFwIvCuGtoYyKMSQoiGpWYelbvPAs4D7gHuBp4ENgKfAb7k7iOALwFXlNj9s8Bd7r6wrXrM7IykrWva8uXL22eshEoIIRqWmiZTuPsV7j7e3Q8CVgCzgZOB25NNbiHasIrZH/icmc0DLgBOMrMflqnjMnef6O4TBw4c2D5DFfoTQoiGpdZZf4OS95HAh4AbiTapg5NNDiPEqxnufoK7j3T3UcBXgGvd/dyaGSqPSgghGpZaT5x4W9JGtQE4y91XmtnpwMVm1g1YB5wBYGYTgTPd/bQa29QS9aMSQoiGpaZC5e4Hlih7EJhQonwa0EKk3P1q4OoamFcg9ah69KhpNUIIIapHI1NACFWPHtBFX4cQQjQaujNDhP4U9hNCiIZEQgXhUSnjTwghGhIJFYRQyaMSQoiGREIFCv0JIUQDI6EChf6EEKKBkVCBQn9CCNHASKhAoT8hhGhgJFSg0J8QQjQwEipQ6E8IIRoYCRUo9CeEEA2MhAoU+hNCiAZGQgUK/QkhRAMjoQKF/oQQooGRUIFCf0II0cBIqAA++EHYZ596WyGEEKIEtZ7hd8vg+uvrbYEQQogyyKMSQgjR0EiohBBCNDQSKiGEEA2NhEoIIURDI6ESQgjR0EiohBBCNDQSKiGEEA2NhEoIIURDY+5ebxtyw8yWA/Or2GUA8HKNzGlkdN5bFzrvrYtqz3sndx9YK2PyoFMJVbWY2TR3n1hvOzY3Ou+tC5331kVnPG+F/oQQQjQ0EiohhBANzdYuVJfV24A6ofPeutB5b110uvPeqtuohBBCND5bu0clhBCiwZFQCSGEaGi2SqEysyPM7Dkzm2Nm59bbnlphZiPMbKqZzTKzmWb2xaR8RzO7x8xmJ+/96m1rLTCzrmb2uJn9IVkebWaPJOf9GzPrUW8b88bM+prZrWb2bHLd99+KrveXkt/5DDO70cx6dsZrbmZXmtkyM5uRKSt5jS24JLnXPWVm4+tnefvZ6oTKzLoCPwPeB+wOHG9mu9fXqpqxETjH3ccCk4GzknM9F7jP3ccA9yXLnZEvArMyy+cBFyXnvRL4VF2sqi0XA3e7+9uAvYnz7/TX28yGAV8AJrr7HkBX4Dg65zW/GjiiqKzcNX4fMCZ5nQFcuplszJWtTqiA/YA57v6Cu78J3AR8sM421QR3X+Lu/0w+v07ctIYR53tNstk1wDH1sbB2mNlw4P3A5cmyAYcBtyabdLrzNrPtgYOAKwDc/U13f5Wt4HondAN6mVk3oDewhE54zd39r8CKouJy1/iDwLUePAz0NbMhm8fS/NgahWoYsDCzvCgp69SY2ShgH+AR4C3uvgRCzIBB9bOsZvwE+A9gU7LcH3jV3Tcmy53xuu8MLAeuSkKel5vZtmwF19vdFwMXAAsIgXoNmE7nv+Yp5a5xp7jfbY1CZSXKOnWOvpltB9wGnO3uq+ptT60xs6OAZe4+PVtcYtPOdt27AeOBS919H+ANOmGYrxRJm8wHgdHAUGBbIuxVTGe75m3RKX73W6NQLQJGZJaHAy/WyZaaY2bdCZG6wd1vT4pfSt3/5H1ZveyrEQcAR5vZPCK0exjhYfVNwkLQOa/7ImCRuz+SLN9KCFdnv94AhwNz3X25u28AbgfeQee/5inlrnGnuN9tjUL1GDAmyQbqQTS43llnm2pC0i5zBTDL3S/MrLoTODn5fDLwu81tWy1x96+5+3B3H0Vc3/vd/QRgKvCRZLPOeN5LgYVmtltS9C7gGTr59U5YAEw2s97J7z499059zTOUu8Z3Aicl2X+TgdfSEOGWxFY5MoWZHUk8YXcFrnT379fZpJpgZu8E/gY8TaGt5utEO9XNwEjiD36suxc3znYKzOwQ4CvufpSZ7Ux4WDsCjwOfcPf19bQvb8xsHJFA0gN4ATiVeCDt9NfbzL4LfIzIdn0cOI1oj+lU19zMbgQOIabzeAn4NvBbSlzjRLR/SmQJrgFOdfdp9bC7I2yVQiWEEGLLYWsM/QkhhNiCkFAJIYRoaCRUQgghGhoJlRBCiIZGQiWEEKKhkVAJ0QZm1mRmT2ReuY32YGajsqNgCyFa0q3tTYTY6lnr7uPqbYQQWyvyqIRoJ2Y2z8zOM7NHk9euSflOZnZfMv/PfWY2Mil/i5ndYWZPJq93JIfqama/SuZS+rOZ9Uq2/4KZPZMc56Y6naYQdUdCJUTb9CoK/X0ss26Vu+9H9P7/SVL2U2Jqhb2AG4BLkvJLgAfcfW9iDL6ZSfkY4Gfu/nbgVeDDSfm5wD7Jcc6s1ckJ0ehoZAoh2sDMVrv7diXK5wGHufsLyeC/S929v5m9DAxx9w1J+RJ3H2Bmy4Hh2SF8kulX7kkmvMPM/hPo7u7fM7O7gdXE8Di/dffVNT5VIRoSeVRCdAwv87ncNqXIjj3XRKHt+P3EbNQTgOmZUcCF2KqQUAnRMT6Wef9H8vkhYtR2gBOAB5PP9wGfATCzrsmMvCUxsy7ACHefSkwA2Rdo4dUJsTWgJzQh2qaXmT2RWb7b3dMU9W3M7BHioe/4pOwLwJVm9lVixt1Tk/IvApeZ2acIz+kzxGy0pegKXG9mOxCT312UTCsvxFaH2qiEaCdJG9VEd3+53rYI0ZlR6E8IIURDI49KCCFEQyOPSgghREMjoRJCCNHQSKiEEEI0NBIqIYQQDY2ESgghREPz/wEhlzxzDVThlwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# gradient descent\n",
    "detailed_logger = False\n",
    "main_logger = True\n",
    "main_logger_output_epochs = 10\n",
    "L2 = False\n",
    "Dropout = False\n",
    "momentum = True\n",
    "hidden_layer_relu = True\n",
    "hidden_layer_tanh = False\n",
    "hidden_layer_sigmoid = False\n",
    "\n",
    "# hyber-parameters\n",
    "alpha = .01;\n",
    "epsilon = .85\n",
    "keep_prob = .9\n",
    "number_of_epochs = 100\n",
    "batch_size = 27\n",
    "momentum_coef = .9\n",
    "\n",
    "# copy initalization\n",
    "W = Weights.copy()\n",
    "B = Bias.copy()\n",
    "\n",
    "# data arrays\n",
    "cost_array = []\n",
    "accuracy_array = []\n",
    "interation_array = []\n",
    "\n",
    "# rename\n",
    "X_train = np.float64(training_images).copy()\n",
    "Y_train = np.float64(training_labels).copy()\n",
    "\n",
    "X_test = np.float64(testing_images).copy()\n",
    "Y_test = np.float64(testing_labels).copy()\n",
    "\n",
    "#m = size\n",
    "m = number_of_training_images\n",
    "\n",
    "def model(W, B, A):\n",
    "    return np.dot(W, A) + B\n",
    "\n",
    "def activation_relu(Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    return np.where(Z > 0, Z, 0)\n",
    "\n",
    "def activation_tanh(Z):\n",
    "    return np.tanh(Z)\n",
    "\n",
    "def activation_sigmoid(Z):\n",
    "     return 1/(1 + np.exp(-Z))\n",
    "\n",
    "def loss(A, Y):\n",
    "    epsilon = 1e-20\n",
    "    return np.where((Y == 1), np.multiply(-Y, np.log(A + epsilon)), -np.multiply((1 - Y), np.log(1 - A + epsilon)))\n",
    "    #return np.multiply(-Y, np.log(A)) - np.multiply((1 - Y), np.log(1 - A)) \n",
    "    \n",
    "def cost(L):\n",
    "    return np.multiply(1/L.shape[1], np.sum(L))\n",
    "\n",
    "def cost_L2(L, W, epsilon):\n",
    "    L2 = np.multiply(epsilon/(2*W.shape[1]), np.multiply(W[len(W)-3], W[len(W)-3]).sum() + np.multiply(W[len(W)-2], W[len(W)-2]).sum() + np.multiply(W[len(W)-1], W[len(W)-1]).sum())\n",
    "    J = cost(L)\n",
    "    return L2 + J\n",
    "\n",
    "def prediction(A):\n",
    "    return np.where(A >= 0.5, 1, 0)\n",
    "    \n",
    "def accuracy(prediction, Y):\n",
    "    return 100 - np.multiply(100/Y.shape[0], np.sum(np.absolute(Y - prediction)))  \n",
    "    \n",
    "def forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "            print('Forward Propagation Training Data Complete')\n",
    "    return A_layers, Z_layers, D\n",
    "\n",
    "def forward_propagation(W, B, A, layer):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "        A = forward_propagation(W, B, A, layer)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        A = activation_sigmoid(Z)  \n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "            print('Forward Propagation Testing Data Complete')\n",
    "    return A\n",
    "\n",
    "def dZ(dZ, W, Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    W = np.where(~np.isnan(W), W, 0)\n",
    "    dZ = np.where(~np.isnan(dZ), dZ, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    W = np.where(~np.isinf(W), W, 0)\n",
    "    dZ = np.where(~np.isinf(dZ), dZ, 0)\n",
    "    if(hidden_layer_relu == True):\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.where(Z > 0, 1, 0))\n",
    "    elif(hidden_layer_tanh == True):\n",
    "        A = activation_tanh(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), 1- np.multiply(A, A))\n",
    "    elif(hidden_layer_sigmoid == True): \n",
    "        A = activation_sigmoid(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.multiply(A, (1-A)))\n",
    "\n",
    "def dW(dZ, A):\n",
    "    return np.multiply(1/dZ.shape[1], np.dot(dZ, np.transpose(A)))\n",
    "\n",
    "def dW_L2(dZ, A, W, epsilon):\n",
    "    return np.multiply(epsilon/Z.shape[1], W) + dW(dZ, A)\n",
    "\n",
    "def dB(dZ):\n",
    "    return np.multiply(1/dZ.shape[1], np.sum(dZ))\n",
    "\n",
    "def backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB):\n",
    "    if(layer >= 0):\n",
    "        if(layer == len(W) - 1):\n",
    "            _dZ = A_layers[layer+1] - Y\n",
    "        elif(layer >= 0):\n",
    "            _dZ = dZ(_dZ, W[layer+1], Z_layers[layer])\n",
    "        if(Dropout == True):\n",
    "            _dZ = np.multiply(_dZ, D[layer])\n",
    "        if(L2 == True):\n",
    "            _dW = dW_L2(_dZ, A_layers[layer], W[layer], epsilon)\n",
    "        else:\n",
    "            _dW = dW(_dZ, A_layers[layer])\n",
    "        _dB = dB(_dZ)\n",
    "        if(momentum == True):\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(alpha, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(alpha, _dB)\n",
    "            W[layer] = W[layer] - V_dW[layer]\n",
    "            B[layer] = B[layer] - V_dB[layer]           \n",
    "        else:\n",
    "            W[layer] = W[layer] - np.multiply(alpha, _dW)\n",
    "            B[layer] = B[layer] - np.multiply(alpha, _dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Layer: ' + str(layer))\n",
    "        layer = layer - 1\n",
    "        W, B = backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Propagation Complete')\n",
    "    return W, B\n",
    "    \n",
    "\n",
    "def shuffle(X, Y, number_of_training_images):\n",
    "    random_array = np.random.permutation(np.arange(number_of_training_images))\n",
    "    return X[:, random_array], Y[random_array]\n",
    "    \n",
    "start_time = time.time()   \n",
    "# main loop\n",
    "for epoch in range(1, number_of_epochs):\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Main Loop Epoch: ' + str(epoch))\n",
    "    \n",
    "    # shuffle data\n",
    "    X, Y = shuffle(X_train.copy(), Y_train.copy(), number_of_training_images)\n",
    "    number_of_batches = int(np.floor(number_of_training_images/batch_size))\n",
    "    split_index = number_of_batches*batch_size\n",
    "\n",
    "    # parse into minibatches\n",
    "    X_minibatches = np.split(X[:, 0:split_index], number_of_batches, axis=1)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        X_left_over_portion  = X[:, split_index:number_of_training_images]\n",
    "        X_minibatches.append(X_left_over_portion)\n",
    "    \n",
    "    Y_minibatches = np.split(Y[0:split_index], number_of_batches, axis=0)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        Y_left_over_portion  = Y[split_index:number_of_training_images]\n",
    "        Y_minibatches.append(Y_left_over_portion)\n",
    "    \n",
    "    number_of_minibatches = len(Y_minibatches)\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Number Of Minibatches: ' + str(number_of_minibatches))\n",
    "\n",
    "    for index in range(0, number_of_minibatches-1):\n",
    "        X_minibatch = X_minibatches[index]\n",
    "        Y_minibatch = Y_minibatches[index]\n",
    "\n",
    "        if(hidden_layer_relu + hidden_layer_tanh + hidden_layer_sigmoid != 1):\n",
    "            print(\"ERROR! Please Select Only 1 Hidden Layer Activation Function\")\n",
    "            break\n",
    "\n",
    "        # forward propogation training data set\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, X_minibatch, [X_minibatch], [], 0, [], keep_prob)\n",
    "        L = loss(A_layers[len(A_layers) - 1], Y_minibatch)\n",
    "        if(L2 == True):\n",
    "            C = cost_L2(L, W, epsilon) \n",
    "        else:\n",
    "            C = cost(L) \n",
    "\n",
    "        # backpropogation\n",
    "        W, B = backward_propagation(W, B, Y_minibatch, A_layers, Z_layers, 0, alpha, epsilon, len(W) - 1, D,  V_dW, V_dB)\n",
    "        \n",
    "    if(epoch % main_logger_output_epochs == 0):\n",
    "        print('Cost: ' + str(C))\n",
    "\n",
    "    # forward propogation test data set\n",
    "    A_test = forward_propagation(W, B, X_test, 0)\n",
    "\n",
    "    # accuracy\n",
    "    _prediction = prediction(A_test)    \n",
    "    _accuracy = accuracy(_prediction, Y_test)  \n",
    "\n",
    "    # storage for plotting\n",
    "    cost_array.append(C)\n",
    "    accuracy_array.append(_accuracy)\n",
    "    interation_array.append(epoch)\n",
    "\n",
    "\n",
    "end_time = time.time()\n",
    "run_time = end_time - start_time\n",
    "    \n",
    "print('')\n",
    "print('Results:')\n",
    "print('')\n",
    "    \n",
    "print('')\n",
    "print('Run Time: ' + str(run_time) + ' seconds')\n",
    "print('Cost: ' + str(C))    \n",
    "print('Accuracy: ' + str(_accuracy) + ' %')    \n",
    "print('')\n",
    "print('')\n",
    "\n",
    "\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, cost_array, 'red')\n",
    "pyplot.title('Learning Curve - ' + str(len(X[0])) + ' Training Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Cost')\n",
    "pyplot.show()\n",
    "\n",
    "# plot percent accuracy curve\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, accuracy_array, 'red')\n",
    "pyplot.title('Percent Accuracy Curve - ' + str(len(X_test[0])) + ' Test Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Percent Accuracy')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated, after 100 epochs with minibatches of 27 the cost became approximately 6.6395e-08 and the test data accuracy reached 99.22%. These results are excellent. The test accuracy is high because minibatch stochastic gradient descent inately provides a form of regularization. In fact, we converged in approximatley 45 iterations to a very high accuracy and low cost. This lines up with what we would intuitively expect by adding in momentum. Our algorithm focused highly on the historical trend of the gradient rather than each step, and as a result converged faster.\n",
    "\n",
    "Now we wish to explore the impact of adjusting the momentum hyper-paramter. Therefore, we will adjust its value to .1 and re-run our algorithm. \n",
    "\n",
    "First we reinitialize our weights and bias's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Size: 784\n",
      "Weights Shape: (20, 784)\n",
      "Bias Shape: (20, 1)\n",
      "Velocity Weights Shape: (20, 784)\n",
      "Velocity Bias Shape: (20, 1)\n"
     ]
    }
   ],
   "source": [
    "# initialize weights & bias\n",
    "np.random.seed(10)\n",
    "print('Feature Size: ' + str(size))\n",
    "\n",
    "lower_bound = -.1\n",
    "upper_bound = .1\n",
    "\n",
    "#mean = 0.015\n",
    "#std = 0.005\n",
    "\n",
    "# hyper-parameters: hidden layers\n",
    "hidden_layers = 2\n",
    "units_array = [20, 10]\n",
    "Weights = []\n",
    "Bias = []\n",
    "V_dW = []\n",
    "V_dB = []\n",
    "for i in range(0, hidden_layers):\n",
    "    if(i == 0):\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], size]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "    else:\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], units_array[i-1]]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "    \n",
    "# output layer\n",
    "_W = np.float64(np.random.uniform(lower_bound, upper_bound, [1, units_array[i]]))\n",
    "_b = np.float64(np.random.uniform(lower_bound, upper_bound)) # b will be added in a broadcasting manner\n",
    "_V_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_V_dB = np.float64(np.zeros(1))\n",
    "Weights.append(_W)\n",
    "Bias.append(_b)\n",
    "V_dW.append(_V_dW)\n",
    "V_dB.append(_V_dB)\n",
    "\n",
    "Weights = np.array(Weights)\n",
    "Bias = np.array(Bias)\n",
    "V_dW = np.array(V_dW)\n",
    "V_dB = np.array(V_dB)\n",
    "\n",
    "for index in range(0, len(Weights) - 1):\n",
    "    Weights[index] = np.where(Weights[index] != 0, Weights[index], np.random.uniform(lower_bound, upper_bound))\n",
    "\n",
    "#print(train_X.shape)\n",
    "#print(np.ravel(train_Y).shape)\n",
    "\n",
    "print('Weights Shape: ' + str(Weights[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Bias Shape: ' + str(Bias[0].shape)) # vector with a size of the # of unit\n",
    "print('Velocity Weights Shape: ' + str(V_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Velocity Bias Shape: ' + str(V_dB[0].shape)) # vector with a size of the # of unit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we re-run our minibatch stochastic gradient descent algorithm with momentum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Main Loop Epoch: 10\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.01529208269507389\n",
      "Main Loop Epoch: 20\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.004077579948707226\n",
      "Main Loop Epoch: 30\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.028170922925060595\n",
      "Main Loop Epoch: 40\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.0042440696729762\n",
      "Main Loop Epoch: 50\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.0001764298629206007\n",
      "Main Loop Epoch: 60\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 0.00039509593526748134\n",
      "Main Loop Epoch: 70\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 2.1314596748085606e-06\n",
      "Main Loop Epoch: 80\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 8.821269877158751e-06\n",
      "Main Loop Epoch: 90\n",
      "Number Of Minibatches: 1852\n",
      "Cost: 2.0576062738186753e-06\n",
      "\n",
      "Results:\n",
      "\n",
      "\n",
      "Run Time: 261.67622208595276 seconds\n",
      "Cost: 0.0002186226582897106\n",
      "Accuracy: 99.16 %\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEWCAYAAABSaiGHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXmYHFXV/z/fTEhCCAmEhC0LCSEJRnaGsLsAQjBK1Bdeo4CA8EZQfuAuuIBGXHlVXFBAQVHQgCwaJMDLqkaELKwmGAlhyZiVJUCALJOc3x+nyqnp6e7pmemaDp3zeZ5+quoude+tqq5vnXtP3ZKZEQRBEAR506PWFQiCIAg2D0JwgiAIgm4hBCcIgiDoFkJwgiAIgm4hBCcIgiDoFkJwgiAIgm4hBAeQdJukU2pdj6D2SDpF0m3VThu0RdKekh6s0r6OkvRMNfZVYv9lz7WkmZJOLRG3m6S6ff9E0n6S/lpJ2poKjqRnJB1VyzoAmNmxZnZ1HvuW1F/SJZKek7Ra0sJke1Ae5VUTSV+VtD6pd/rbNRO/j6S5kl5Plvtk4iTpO5JeSH7flaRq5M2kOzFTrzckbczWtTNtNrOrzezYaqftKJKakjatlrRK0t8kTSl2HErk7/JNTtL7JT0q6RVJz0u6S9LwKpZ9EXBxJl+2zcskXSlpq660oRIknSHpviLhTZLeAfme666QrWOtMLOHgDcktXt86t7CkdSzhmX3Au4G3gpMAPoDhwAvAOM7sb9atOU6M+uX+S1K6tIL+CNwDbAtcDXwxyQcYArwPmBvYC/gPcDHupo3i5ldm9YLOBZYkq1rYfpaXgud5NikHSPwG/MXgSu6o2BJY4FfAucCA4CRwGXAxirtfyhwGHBLQVTa5v2AA4HPV6O8IB8y/6lrKfIfbYOZ1ewHPAMcVSLuPcAjwCrgfmCvTNx5wFPAq8B84P2ZuFOBvwE/AF7En6JOBWYC/wu8BDyNX9hpnvuAMzL5y6UdCfwlKfsu4FLgmhJtOANYDvQrcwwM2C2z/SvgomT9HUAT8AVgGfAb4AngPZn0PYHngf2S7YOS47UKeBR4RxfOz1fLtO1o4N+AMmHPAROS9fuBKZm404EHupq3TF3fATQVCW8CPgc8DqxLwr4MLErO4TzguIJzdl/m2Br+R1qYXA8/6mTaBuAS/GFjEfD/ACvTnqbCcwccjN/wd0+2j8P/I68mx+8rmbRLkvqsTn4HAKOBe5M6PJ9cTwNKlD8ZmFOmfj1wAXwq2dc0YNtSZRfJ/1Hg9nJtBr4P/DGz3ScJW4z/r34K9EnijgKeKTgXIzJ5rwG+WuZ/el+5c1CYBn+AXAC8DPwQv+ecmjnXP0iO81PA2dlzDWyDi/nSpIypQI9MOX9O8q9KrpWjO3KdJOHbATOAlcm1eAswJIn7EPBgQfovADdUepyTc78M+GUSvgvwGrBFuf/pJmnhSNoPuAr/824HXA5Ml9Q7SfIUcDj+5PU14BpJO2V2cSB+orYHvpEJWwAMAr4LXFmme6Jc2t8Cs5J6fRU4uUxTjsL/VJ3q3knYERiIn9ApwO/wCyblGOB5M3tI0hDgVlxkBwKfBW6UNLgL5b9X0ouS5kk6KxP+VuAxS662hMeS8DT+0UzcowVxnc3bGSbjFtCAZPtfwKHJ9jeA30raoUz+dwP7A/sCJ7XTDVwq7Vn49bAX0Ah8oKONMLO/43/yw5Og1cBJSTveC5wr6T1J3NuSPKnFNxsQfm3sBIwDdgW+UqK4ucCekr4n6Z1FurY+DUxMyhmK32x+VKbsQvbE/2NFkTQMv6kvzAT/L/7AtxcuniOAL5XaR15I2h64AX/wHYTf9A/MJDkLf6jaG+/J+O+CXVwDvAGMwq+FicBpmfhD8Aek7XDhubIT1ewB/BwYjt871uPCCPAHYKyk0Zn0J+EPIND+cR4K9Ev2/XEAM3sWv76y+2xLOTXK+0cJCwf4GfD1grAFwNtL7OcRYFKyfirwXEH8qcDCzHZf/Alox2T7PlpbOEXTJge4Gehb8ORUygq4E/h2O8egPQtnHcnTRRK2G/5E2zfZvha4IPOU8puC/d8BnNLJ8zMO2Bl/YjsEfyL7UBL3FWBaQfprSZ4igQ0kT+LJ9uikrepK3jJ1fQelLZyPtNPOfwATk/ViVstBmbQ3AZ/tRNq/AKdn4ibQQQsnCZ8DfKFEnp8AF2euk5L7T9IcD8wuE38I8HvcglmDPwSm192TZP6PwDBgLX6jq6TsX6bXeUGbVyfXtwH/R2KBJftdA+ySSX848GSy3lULpxm3KLK/jRSxcHDrbGYmfw/8v3Fq5lyfkYl/d3o8gCG42PTOxJ8M3Jkp55+ZuP5JWwZ15Dopkq4RWJnZ/jnwtWR9n+Qcb1HhcV4D9CpSxnLgkHL12CQtHFyRP5MMlq6StAq/oHcGkPQRSY9k4vbAnzRSFhfZ57J0xcxeT1bb9PO3k3Zn4MVMWKmyUl7Anya7wkozW5Opz0K8W+29kvri3Sq/TaJ3AU4oOG6HFatDwYB7Ue8bM5tvZkvMbIOZ3Y8/IR2fRK/G/wxZ+uM3i2Lx/YHVyb+uK3k7Q6tzJOnUZDA8PUa70/r6KWRZZv11Sl835dLuXFCPctdNOYbgXcVIOljSfZJWSnoZv1mVbIekHSVdL+nfkl7BH25Kpjez+83sBDMbhFstRwDnJ9HDgVsyx/Bx/Ma4fYXteAnYukj4e8xsa+BI3KodmITvCPQGsuftTx0orz1mmtk22R/eNViMVufSzDbiN/6i8cCzmfVd8HYsz7TjUiBrYRdeQ1D+mmuDpK0k/ULurPQKcA+tz/XVwInJ+kn4WO16KjvOy81sXZFit8aFuiSbquAsBr5RcAH0NbPfSdoFV+ezge2SC+Mf+JNzSmdvTO2xFBiY3OhThpVJfxdwTDueNq/jVlTKjgXxxdqSdqtNAuYnIgR+3H5TcNy2MrNvF+7AMgPuVrn3TWqhgI997FXQLblXEp7G752J27sgrrN5O8N/jqHcy+5neLdHev38k9bXTx4sxbsiUspdN0WRdBB+Y5qZBE0DbgSGmdkA4Be0tKPYdfMd3ArZ08z649Z8Re02s1l4V8weSVAT8K6Ca62PmS0rUXYhjwFjypR3D26VpF5sy3Frf2ymvAFJuwvzNiftLPe/6gpLyZw/ST1ofW5bxePinLIY/88PzLSjv5ntVcX6gTtbjATGJ+f6iGykmc1M6n4ofi9Ju9MqOc5tzm9yXwa3fEuyKQjOFpL6ZH49cUE5U9KBcraSNFHS1sBWeINXAkg6jZY/Qa6Y91POAb4qqZekg/G+81L8Br/AbpS0u6QekraT9EVJ707SPAJ8WFKDpAnA2yuoyjS8j/gsWqwb8D/oeyUdk+yvj6R3JB5BHUbSJEnbJudgPHAO7l0G3g25AThHUm9JZyfh9yTLXwOfljRE0s7AZ/An6q7m7Sr9aLl+JOkM3MLJm+uBT0raWdK2uCNDRUgaICm1ZH9lZk8kUVvjFveaRIwmZ7KtAEwZN/Yk/WvAy8kYyWfLlPl2ubvw9sn2W/Br/YEkyWXAN5W4SUvaPqljqbIL+T/gALV4JhbjB8C7Je1hZhtwQb1E0uDkmhwq6egSeR8FTkz+BxNxS79a/AnYJ/l/9AQ+BWTHSdNzPUTSdnhXNwBmthh3Cvhf+SsTPeRu5G/rQn16FbmHbo0L20tJHS4oku83+MPXa2b2QFK/jh7nlLcDdyVWUkk2BcGZgfdppr+vmtkc4H/wPumX8IHDU8G7eYDvAX/H1XhP3EOkuzgR9xZ6AR+AvQ5/mmqDma3F+zz/iY/nvII7HAwC0hfezsX/yKuSff+hvQqY2VK8/Yck5afhi3Gr54v4DXUxfmPr7HmejB/7V3ER+I4l7yslJvX7gI8kdf8o8L6MqX057hnzOG6B3pqEdSlvVzGzx/DB7Vn4k+jutJyLPPkZLrSP4wPyt+JPkuW4Tf4+0XP4APXFeLdZylnAtyS9ip/z69MIM3sV+BbwYNI10ghciA9ivwxMx62jUrwEvB/4R1KHGcn+v5fEfx+4Hbg7Kf9+3BOuVNmtMLMlwF8p88CWWEvX0uLY8Bm8e2pW0ob/o/Qg9TlJ/VcBJyTtrQpmthz4IH4+XsAtmOw19DP8dYjHgdm4g0GWk/AH5/n4cf49XbPA7qD1PfTL+PkZkNTvfqBYt/mv8Yf13xSEd+Q4p5yIP4SURZ3vFg8AJF2HD/JdWOu6BG8eJL0XuMTMRtW6LrVC0p7Az83soFrXZXMk6epfAexhZk93YT/7Aj82s3atyBCcDiLpAHzQ9mm8W+sPwMFm9nBNKxZs0iR/7sNxS3cn4Gbgz2ZWslsrCPJE0udxL+H2usuqxpvtzetNgR1xd9ft8IHTs0JsggoQ/s7PDfg4yp/wd8iCoNuR1IS/mzOpW8sNCycIgiDoDjYFp4EgCIJgM6BuutQGDRpkI0aMqHU1giAI3lTMnTv3eTPryvRXFZOr4CTvlfwQnxrlF8VeQEzSHY+7Bh6QuEQj6Xx80sYNwDlmdke5skaMGMGcOXOqWf0gCIK6R9Kz7aeqDrkJjqQGfMqGd+GD67MlTU/eo8mm2xr3mX8wEzYOfwfkrfg0EXdJGpO8lBQEQRC8CclzDGc8PgnmouSFvmkU94j4Oj4j85pM2CR8cse1iX/4Qjrx/ZggCIJg0yFPwRlC6wnsmpKw/5C8MDTMzP7U0bxJ/imS5kias3LlyurUOgiCIMiFPAWn2KSA2UkUe+BzJX2mo3n/E2B2hZk1mlnj4MHdMuYVBEEQdJI8nQaaaD1j6lBaT/e9NT6Pz33JpME74h9ZO66CvEEQBMGbjDwtnNnAaEkjkxlhJ5OZQM/MXjazQWY2wsxG4LPQHpd4qU0HJiczCY/EJ46blWNdgyAIgpzJzcIxs+Zk2vk7cLfoq8xsnqSp+LfSS87emqS7Hp9NtRn4RHioBUEQvLmpm6ltGhsbrerv4dx7L+y8M4wdW939BkEQbCJImmtmbT4hkQcxtU05zjgDvl30XdUgCIKgg4TglOONN2Bt0W+rBUEQBB0kBKcc69dDc3OtaxEEQVAXhOCUIwQnCIKgaoTglGP9ev8FQRAEXSYEpxxh4QRBEFSNEJxSmIXgBEEQVJEQnFJsSN4zDcEJgiCoCiE4pUjHbkJwgiAIqkIITilSwQmngSAIgqoQglOKsHCCIAiqSghOKUJwgiAIqkoITilCcIIgCKpKCE4pQnCCIAiqSghOKcJpIAiCoKqE4JQiLJwgCIKqkqvgSJogaYGkhZLOKxJ/pqTHJT0iaaakcUn4CElvJOGPSLosz3oWJQQnCIKgquT2iWlJDcClwLuAJmC2pOlmNj+T7LdmdlmS/jjg+8CEJO4pM9snr/q1SwhOEARBVcnTwhkPLDSzRWa2DpgGTMomMLNXMptbAZvO965DcIIgCKpKnoIzBFic2W5Kwloh6ROSngK+C5yTiRop6WFJf5Z0eI71LE44DQRBEFSVPAVHRcLaWDBmdqmZjQK+AHw5CV4KDDezfYFPA7+V1L9NAdIUSXMkzVm5cmUVq05YOEEQBFUmT8FpAoZltocCS8qknwa8D8DM1prZC8n6XOApYExhBjO7wswazaxx8ODBVas4EIITBEFQZfIUnNnAaEkjJfUCJgPTswkkjc5sTgSeTMIHJ04HSNoVGA0syrGubQnBCYIgqCq5eamZWbOks4E7gAbgKjObJ2kqMMfMpgNnSzoKWA+8BJySZH8bMFVSM7ABONPMXsyrrkVJhcbMv43T0NCtxQdBENQbuQkOgJnNAGYUhF2QWT+3RL4bgRvzrFu7ZJ0FmptDcIIgCLpIzDRQikLBCYIgCLpECE4pQnCCIAiqSghOKUJwgiAIqkoITimyghMvfwZBEHSZEJxShIUTBEFQVUJwShGCEwRBUFVCcEoRghMEQVBVQnBKEYITBEFQVUJwShFOA0EQBFUlBKcUYeEEQRBUlRCcUoTgBEEQVJUQnFKE4ARBEFSVEJxShOAEQRBUlRCcUoTTQBAEQVUJwSlFWDhBEARVJQSnFCE4QRAEVSUEpxQhOEEQBFUlV8GRNEHSAkkLJZ1XJP5MSY9LekTSTEnjMnHnJ/kWSDomz3oWZf166JEcnhCcIAiCLpOb4EhqAC4FjgXGAR/KCkrCb81sTzPbB/gu8P0k7zhgMvBWYALw02R/3cf69bDlli3rQRAEQZfI08IZDyw0s0Vmtg6YBkzKJjCzVzKbWwGWrE8CppnZWjN7GliY7K/7WL8e+vb19bBwgiAIukzPHPc9BFic2W4CDixMJOkTwKeBXsARmbwPFOQdUiTvFGAKwPDhw6tS6f+QtXBCcIIgCLpMnhaOioRZmwCzS81sFPAF4MsdzHuFmTWaWePgwYO7VNk2hOAEQRBUlTwFpwkYltkeCiwpk34a8L5O5q0+MYYTBEFQVfIUnNnAaEkjJfXCnQCmZxNIGp3ZnAg8maxPByZL6i1pJDAamJVjXdsSFk4QBEFVyW0Mx8yaJZ0N3AE0AFeZ2TxJU4E5ZjYdOFvSUcB64CXglCTvPEnXA/OBZuATZrYhr7oWJZwGgiAIqkqeTgOY2QxgRkHYBZn1c8vk/Qbwjfxq1w5h4QRBEFSVmGmgFM3NIThBEARVJASnFOE0EARBUFVCcEqxfj306ePrYeEEQRB0mRCcUqxfD1tsAQ0NIThBEARVIASnFKngbLFFCE4QBEEVCMEpRSo4PXuG4ARBEFSBEJxSZAUnnAaCIAi6TAhOMcxgw4awcIIgCKpICE4xUosmxnCCIAiqRghOMbKCExZOEARBVQjBKUYIThAEQdUJwSlGoeCE00AQBEGXCcEpRlg4QRAEVScEpxjhNBAEQVB1QnCKERZOEARB1QnBKUYIThAEQdXJVXAkTZC0QNJCSecVif+0pPmSHpN0t6RdMnEbJD2S/KYX5s2VcBoIgiCoOrl98VNSA3Ap8C6gCZgtabqZzc8kexhoNLPXJZ0FfBf4YBL3hpntk1f9yhIWThAEQdXJ08IZDyw0s0Vmtg6YBkzKJjCze83s9WTzAWBojvWpnHAaCIIgqDp5Cs4QYHFmuykJK8XpwG2Z7T6S5kh6QNL7imWQNCVJM2flypVdr3FKWDhBEARVJ7cuNUBFwqxoQukkoBF4eyZ4uJktkbQrcI+kx83sqVY7M7sCuAKgsbGx6L47RYzhBEEQVJ08LZwmYFhmeyiwpDCRpKOALwHHmdnaNNzMliTLRcB9wL451rU1YeEEQRBUnTwFZzYwWtJISb2AyUArbzNJ+wKX42KzIhO+raTeyfog4FAg62yQLyE4QRAEVSe3LjUza5Z0NnAH0ABcZWbzJE0F5pjZdOBioB/we0kAz5nZccBbgMslbcRF8dsF3m35Ek4DQRAEVSfPMRzMbAYwoyDsgsz6USXy3Q/smWfdyhIWThAEQdWJmQaKEU4DQRAEVScEpxhh4QRBEFSdEJxipALTs2cIThAEQZUIwSlGOA0EQRBUnRCcYkSXWhAEQdUJwSlGOA0EQRBUnRCcYoSFEwRBUHVCcIpRTHCselO1BUEQbI6E4BQjFZyGBhcdgI0ba1efIAiCOiAEpxjr17vQSG7hQHSrBUEQdJEQnGKkggMtghOOA0EQBF0iBKcYxQQnLJwgCIIuEYJTjKzgpMvuEJypU+Guu/IvJwiCoAaE4BSjVhbO974Hv/99/uUEQRDUgBCcYtRCcMzgtddg7dr20wZBELwJqUhwJP2mkrC6oRZOA+vXw4YNsGZNvuUEQRDUiEotnLdmNyQ1APtXvzqbCLWwcF5/3ZchOEEQ1CllBUfS+ZJeBfaS9EryexVYAfyxvZ1LmiBpgaSFks4rEv9pSfMlPSbpbkm7ZOJOkfRk8julE23rPLVwGnjtNV9Gl1oQBHVKWcExs2+Z2dbAxWbWP/ltbWbbmdn55fImVtClwLHAOOBDksYVJHsYaDSzvYAbgO8meQcCFwIHAuOBCyVt24n2dY6wcIIgCKpOpV1qf5K0FYCkkyR9P2uNlGA8sNDMFpnZOmAaMCmbwMzuNbPkTssDwNBk/RjgTjN70cxeAu4EJlRY165TS8EJCycIgjqlUsH5GfC6pL2BzwPPAr9uJ88QYHFmuykJK8XpwG0dyStpiqQ5kuasXLmynep0gFo4DaRdamHhBEFQp1QqOM1mZriF8kMz+yGwdTt5VCSs6JTLkk4CGoGLO5LXzK4ws0Yzaxw8eHA71ekA0aUWBEFQdSoVnFclnQ+cDNyajM9s0U6eJmBYZnsosKQwkaSjgC8Bx5nZ2o7kzY1aOA1El1oQBHVOpYLzQWAt8FEzW4Z3b11cPguzgdGSRkrqBUwGpmcTSNoXuBwXmxWZqDuAoyVtmzgLHJ2EdQ+1sHCiSy0IgjqnIsFJROZaYICk9wBrzKzsGI6ZNQNn40LxBHC9mc2TNFXScUmyi4F+wO8lPSJpepL3ReDruGjNBqYmYd1DLcZwwsIJgqDO6VlJIkn/jYvDffj4yo8lfc7MbiiXz8xmADMKwi7IrB9VJu9VwFWV1K/qxBhOEARB1alIcPAxlgPSbi9Jg4G78Hdn6o/oUguCIKg6lY7h9CgYY3mhA3nffNTSaWDDhvj2ThAEdUmlFs7tku4Afpdsf5CCrrK6Yv36Fsumu7vUwMdxelZ6aoIgCN4clL2rSdoN2MHMPifpA8Bh+BjO33Engvqkli9+gnerbbVVvuUFQRB0M+11i10CvApgZjeZ2afN7FO4dXNJ3pWrGc3NtXMagPBUC4KgLmlPcEaY2WOFgWY2BxiRS402BWrpNADhOBAEQV3SnuD0KRO3ZTUrsklRS6cBCAsnCIK6pD3BmS3pfwoDJZ0OzM2nSpsAtXwPB8LCCYKgLmnPFeqTwM2STqRFYBqBXsD786xYzdiwAcxq4zTQowds3BiCEwRBXVJWcMxsOXCIpHcCeyTBt5rZPbnXrFakwlILC2fbbeGFF6JLLQiCuqSilz3M7F7g3pzrsmlQKDjdOYYzcKALTlg4QRDUIfU7W0BnqZWF89prLjgQFk4QBHVJCE4hhYLTIzlE3WXhQFg4QRDUJSE4hRQKjuRWTp5OA2YhOEEQ1D0hOIUUCg644ORp4axZ46ITXWpBENQxITiFFBOcLbbIV3DSd3C23daXYeEEQVCH5Co4kiZIWiBpoaTzisS/TdJDkpolHV8QtyH5Cuh/vgTaLdTCwkkFJyycIAjqmNzmwJfUAFwKvAtowmctmG5m8zPJngNOBT5bZBdvmNk+edWvJLUQnHQetRjDCYKgjsnzoyvjgYVmtghA0jRgEvAfwTGzZ5K4jTnWo2OUEpw8nQZSC6d/f2hoCMEJgqAuybNLbQiwOLPdlIRVSh9JcyQ9IOl9xRJImpKkmbNy5cqu1LWFWnap9e0LffpEl1oQBHVJnoKjImHWgfzDzawR+DBwiaRRbXZmdoWZNZpZ4+DBgztbz9bUwmkg7VLbaivo3TssnCAI6pI8BacJGJbZHgosqTSzmS1JlouA+4B9q1m5koSFEwRBkAt5Cs5sYLSkkZJ6AZOBirzNJG0rqXeyPgg4lMzYT67UcgwnFZywcIIgqENyExwzawbOBu4AngCuN7N5kqZKOg5A0gGSmoATgMslzUuyvwWYI+lRfNLQbxd4t+VHLb3UokstCII6Jk8vNcxsBjCjIOyCzPpsvKutMN/9wJ551u0/vPQSTJ0K//VfcNhh0aUWBEGQEzHTQI8ecMkl8OCDvl3LmQb69g0LJwiCuiUEp39/tyqWLfPtWnWpNTRAr14xhhMEQd0SgiPBjju2Lzh5Ow307et1iS61IAjqlBAccMFZvtzXU2HpmRne6o4xnK228vXoUguCoE4JwYHKLJy8u9T69vX1sHCCIKhTQnCgfcHpDqeBVHDCwgmCoE4JwQEXnOefd7FJhaW7LZy0Sy2cBoIgqFNCcMAFxwxWrqyt0wBEl1oQBHVLCA644IB3q9Xqxc/oUguCoM4JwYG2gtOjh/9SurtLbe1at7iCIAjqiBAcgB128GUqOFnrBrrXaaBPHxebPLvwgiAIakAIDrQvON3dpQbRrRYEQd2R6+Sdbxq23BIGDHDBMSsuOHlaHIVdahCOA0EQ1B1h4aSk7+J0t4WzcaNbM2HhBEFQ54TgpJQTnDzHcN54w5fZMRwICycIgrojBCelVhZO9uNr0CI4YeEEQVBnhOCkpBN4drfgZL+FA9GlFgRB3ZKr4EiaIGmBpIWSzisS/zZJD0lqlnR8Qdwpkp5MfqfkWU/ABeeVV/xXSnDyeDcmFZxwGgiCoM7JTXAkNQCXAscC44APSRpXkOw54FTgtwV5BwIXAgcC44ELJW2bV12Blpc/Fy8uLjgAGzZUv9y0Sy0snCAI6pw8LZzxwEIzW2Rm64BpwKRsAjN7xsweAzYW5D0GuNPMXjSzl4A7gQk51rW84KTbeXSrFXapxRhOEAR1Sp6CMwRYnNluSsKqllfSFElzJM1ZuXJlpysKtAjOqlWlLZw8BSe61IIgqHPyFBwVCat0EKSivGZ2hZk1mlnj4MGDO1S5NqSCA6UFJ4+XP6NLLQiCzYQ8BacJGJbZHgos6Ya8nWPQIFCic7WwcOI9nCAI6pw8BWc2MFrSSEm9gMnA9Arz3gEcLWnbxFng6CQsP3r2hNRKqmWXWlg4QRDUKbkJjpk1A2fjQvEEcL2ZzZM0VdJxAJIOkNQEnABcLmlekvdF4Ou4aM0GpiZh+ZJ2q3Wn00Bhl1o4DQRBUKfkOnmnmc0AZhSEXZBZn413lxXLexVwVZ71a8OOO8Jjj0WXWhAEQQ7ETANZSlk4eToNvP469OrVUkavXr4MCycIgjojBCdLe4KTV5daat2Af2m0V6+wcIIgqDtCcLLUQnCyH19L6d07LJwgCOqOEJwsqeD0LBjayttpIPVQS+nTJwQnCIK6IwQny6Zi4fTpE11qQRDUHSE4WWrlNFBo4USXWhAEdUgITpZNwWkAwsIJgqAuCcHJss02sNNOMLTg1aBwGgiCIOgyub74+aZDggULYMstW4fn/XmCcBoIgmAzIASnkK23bhtWiy61EJwgCOqM6FKrhHJOA2vWwLJlnd93dKkFQbCZEIJTCeUsnG99C/bCqh4GAAAZjUlEQVTbD6zST/0UEF1qQRBsJoTgVEK5MZz582HpUnj55Y7vt7kZ1q1rO2YUXmpBENQhITiVUM7CWby49bIjLEm+KZf92ihEl1oQBHVJCE4llBOcpiZfdkZwnn7alyNHtg4PCycIgjokBKcSSjkNNDd7dxpUV3DCwgmCoA7JVXAkTZC0QNJCSecVie8t6bok/kFJI5LwEZLekPRI8rssz3q2SykLZ+lS2LjR1zsjOM884+/+DBvWOjycBoIgqENyew9HUgNwKfAuoAmYLWm6mc3PJDsdeMnMdpM0GfgO8MEk7ikz2yev+nWIUk4DaXcadN7CGTLELZosffq4NbVxo38fJwiCoA7I8242HlhoZovMbB0wDZhUkGYScHWyfgNwpCTlWKfOUcrCSUVmwAB47rmO7/fpp9t2p0GLAMU4ThAEdUSegjMEyD72NyVhRdOYWTPwMrBdEjdS0sOS/izp8GIFSJoiaY6kOStXrqxu7bOUGsNJLZyDDy5v4SxbBkcc0doigtKC06ePL0NwgiCoI/IUnGKWSuHbkaXSLAWGm9m+wKeB30rq3yah2RVm1mhmjYMHD+5yhUtSzsLZaivYc08Xk1Ivf95xB9x7ry9T1q6Ff/+7vIUT4zhBENQReQpOE5AdDR8KLCmVRlJPYADwopmtNbMXAMxsLvAUMCbHupanlOA0NfnM0sOGuYCUsrIeesiXjz3WErZ4sQvUiBFt06cWTncKzimnwKWXdl95QRBsduQpOLOB0ZJGSuoFTAamF6SZDpySrB8P3GNmJmlw4nSApF2B0cCiHOtanoYG9yYrZuEMG9biZVaqW62Y4JRyiYbu71J75BH49a/h5pu7p7xNlXvugauuqnUtgqBuyU1wkjGZs4E7gCeA681snqSpko5Lkl0JbCdpId51lrpOvw14TNKjuDPBmWb2Yl51rYiePctbOFBccDZsgIcf9vXHHmvpdisnON3dpXb55b586qnuKW9T5eKL4bOf7fy8eEEQlCXXzxOY2QxgRkHYBZn1NcAJRfLdCNyYZ906TM+erZ0G0pc+27NwnnzSP0HQ2Ahz5vh0NkOGuOD07OnrhXSnhfPqq3DNNW7FPfecz+3Wq1f+5VbC2rXwxhv+YbzuYMECeOkleOEFGDSoe8oMgs2IeMmjUgotnPSlz6FDYfBgt0qKCU7anXbqqb5Mu9WeeQaGD/cbfSHdaeH87newejWccYa3pzPu3Xnx5S/D/vt3j8WxZo2fE4B//Sv/8oJgMyQEp1IKBSd1cR461Md3hg4tLTi9e8N//7dvp4JTyiUaus9pwAwuuwz23hs+/GEP25S61R58EBYtcm++vFm4sEXYFizIv7wg2AwJwamULbZoLTipuKTdacOHlxacvfZyK2j48I4JTt5darNn+/jSmWfCqFEetqkIjhnMm+frc+fmX15WZMLCCYJcCMGplHIWDrjwFAqOmQvO/vv79l57ueC89hqsWFFacMp1qS1aBHff3fl2ZLnsMujXD048EXbayYVuUxGcFSvgxcRPpDsFZ+jQsHCCICdCcCql0GkgfekzHdAeNsy7fjZsaEmzaJF/mG2//Xx7r73gn/9seYIu9g4OlO9S++IX4ZhjYNasLjWHdevguutg8mTYemufs23XXb3OebFiRfFPPBRjfjLlntR9grPzzn6uwsIJglwIwamUYhZOOn4DLjgbNrR8rgBaHAZSwdl7b9/Hbbf5dnsWTrEutQcf9HJOOcU9uDrLQw/5562PPbYlbNdd87Nw5s2DXXaBiy6qLH0qOEce6YKTt+PAv/4FY8f6b+HC1g8OlbB6NRx1lHsiBkFQlBCcSikmONnPChRzjX7oIc+3xx6+vddevvzjH33ZUaeBlSvdk+rd73ZL6ctf7lRTAPjb33x56KEtYaNGuYVT7Zv7+vXwkY94ey6/vO2cdMWYP98nRX3ve2H58tZCXm3M3MIZOxbGjHGh76i33r33elfntGn51DEI6oAQnEop5jSQjt9AacHZY48Wi2W33VxMZs3y5Q47FC+rlNPA7Nm+/Pzn4eMfhx/8AP7yl861Z+ZMr0+2DqNG+fjS8uWd22cpLrrIj8UZZ/hEprfe2n6e+fNh3LiW8a88u9Wef97fv0kFBzrerXbXXb68//7q1i0I6ogQnErJWjjZlz5TCgXHzG+S6Q0z3cdb3+rrI0a0dMcVUsppYNYsz7PffvCd73gX2Ec/2vHuHzMXnMMOax2eeqpVcxxnzhz4xjfg5JPhZz/zcZIrrmg/Xyo4e++d/zhO6iQwZoyLDnRccFJHjrlzu+5deNVVsPvuMXlrUHeE4FRK1mkg+9JnyoAB7vGVCs7ixf7Gejp+k5J2q5XqTgO3pqS2N5zZs/0mvPXWXtbXvuZjLoU3Y7PyYwn/+pc/1We708AFDCobx/nzn2HixPI313XrvCttxx3hRz/yY3j66XD77fDss6XzPf+8OxiMG+ft3H337hGcsWNh++2hf/+OeaotW+ZjVAcd5G1Ox+46w9q1cMEFXv6993Z+P0GwCRKCUylZwSl0iYaWT0WngnPttb7sjOBIbuVkb+ZmLjgHHNASdswxnjZ1Qkj5/e89XdrNU0g6flNo4Ywc6furRHCmToUZM8q7aN98MzzxBPzkJy3efKef7ssrryyd74knfDlunC/33z9/wenVq8XqHDOmYxZOegy+8hVfdqVb7Zpr3NtRgltu6fx+gmATJASnUsaM8SfOW25p+9JnSio4l17q7suTJsH48a3T7L23L8sJDvg4TtbCefZZdxrI7m/QIN++/fbWedOB69/9rvi+Z86E7bZr6T5K6d3bRbQ9wXnySZ9ZGeCmm0qnu/xyv4kfd1xL2C67wIQJLjilXKRTD7Ws4Cxdmp/jwIIFPp6VTjM0dmzHLJy774aBA71dI0fC3//euXps2ADf/S7su69fO7fcEhOJBnVFCE6l/OQnfiM4/vgW6yVr4YALzsMPw9ln+w3j+uv9/ZYsBxzgrr5HH12+vD59Wls4qcNA1sIBv8k9+KB334G75952m5d7003exVNIOn5TbAxp1Kj2BefnP/eb8xFHuMddMeFIu4SmTGl7DKZM8UlMZ8xomw9ccPr1axH0dBysK11V5Ug91FLGjHEvtUrczs3cknznO72dhxziFmRnhOLmm92yOv98F+mmJv90RJAP69fnd00FRQnBqZT+/d2SGDsWpk9v/dJnyvDhfvNNxabYrMv9+vkNKu1aK0Xv3q0tnFmzfH+F+Y491m9ud97p27fe6vk++1lYtaptt9ry5W6hFI7fpLT38ufatfDLX3obP/5xH2+ZObNtuiuu8G7I005rGzdxos+SfdFFPhZWyPz58Ja3tAjivvvm5zjQ3OwCWyg44O/jtMfChW7VHnWUbx9yiI/plBujKoYZfOtbMHo0fOADfoyiWy1f0slh4xh3GyE4HWHgQL+xjxnjv0IL4aST4JvfLC02HaGwS232bNhnn7b7bWz07rF0HOf3v/dB+q9+1R0Zrruudfp0fKFw/CZl1CgXpdWri8fffLOLzJQpbl316dO2W23NGvjVr+D97/e6FLLFFvDtb3ubio3lpB5qKf36uSBUS3BWrHD3b/A57ZqbWwtOul5Jt1o6fnPkkb48+GBfdnQc5667/Gn7859363H77d0JYXrhNwuDqvDii/DTn/r6mWf6jCBB7oTgdJQddnAPsMKBenDr4Pzzq/M9mUGD/Ka1YoX37c+Z03Y8CPzmdPTRcMcd/m2bGTP8CXnLLf2G/4c/tO6amznTRaLQmSGlPdfoyy/3cYp3vcutvAkTXHCylsqNN/of+mMfK92+E0+Eww/345XOmQZulS1Z0lpwwJ9EZ80q/47Q4sVwwgmlp/0xg1/8wuvf2OiD81kPtZTRo31ZiePAXXe5Zbvbbr69555+XDoyjrN2LXzqU95Fe/LJLeHvfa+LbHfMlr258eMf+0PVVVe5Rfq5z9W6RpsHZlYXv/3339/qilmzzPr0MTv0ULO5c83A7Oqri6e9+mqPP+88X957r4ffdptvT5/eknb8eLPDDy9d7uzZnuemm3z7rrvMLrjAy7jpJo/75jdb0v/mNx72wAMtYYcdZrbbbmYbNpRv46OPmjU0mJ15ZkvY/ff7/m65pXXam24yk/yYnH222bPPto5ftcpsjz0873bbmT3xROv45583+8AHPP7QQ8369TMbNcrsnHM87IUXWqffeWezU04pX/8NG8wGDjQ77bTW4UccYbbffsXz/OMfZgsXtg77whe8DjNmtE0LZpddVr4eQcd45RWzbbc1O+443/7c5/w43313betVI4A51k336Xx3DhOABcBC4Lwi8b2B65L4B4ERmbjzk/AFwDHtlVV3gmNmNm2an6IRI3w5f37xdMuWeXzPnmbbb2/W3Ozh69b5H+vEE802bjS79lpPc/75pct84QXf18UXm33jG36Td9ugpYylS1vSv/SSh33+82ZvvGF20UUt+Svh3HO9jFmzfPsXv/D8Tz3VNu2CBWYf/aiX16uXt2P1arO1a82OPNLDf/5zPwbDh5s1NXncpZea7bCD2RZbeL02bHCB3GYbL2vQoLZlvfOdZgcdVLre69a1CPy117aO+9KXXEhXr24J27jR7Mc/9jr27Wt2ww0e/re/mfXoYXbGGW3L2LjRbORIs4kTyx/DvHj6abMrrvDrq564+OLWD0mvv242erQf62LXXcrq1X593n9/9eqycaPZb3/r1+/UqWbPPVe9fVdIXQgO0AA8BewK9AIeBcYVpPk4cFmyPhm4Llkfl6TvDYxM9tNQrry6FBwzswsv9NO09dblLYb99vN0WWvBzOz00/1p/pBDPH6//dpaB4Vss41Z//6e/sMfduvhn/90q+Mvf2mb/phjzHbc0WyXXTzP+9/vT5GVsGqVi0GPHn6DHz/ebMstW0SzGM8+a3byyV7WsGFmxx7r67/6lcfPnettHjvWbNddPe7ww80eeqj1fh56yK2hd76zbRkf+5gL1MSJZl/5itmNN5o9/rjfnBYsMGts9P2edpqLT5Zbb/W4227zc/baa2YnneRhEyd6O8Hsi1/0G90uu5i9/HLxtp5zjlnv3i5i06ebLV5stmKFi8CSJWbLl5u9+KLZq6/6sVy+3NMsWeJW3auv+r5XrHABXrbMHwzMPH7aNG/DhAlm3/2ut/G557z9PXt6Pfv08Xo0NVV0Ssvy8st+3G+7zexPfzL74x/dsnviCbM1a9rP/8YbZs884w9GGzd2vPw33vBr9cgjW4fPnOkPAr16ucWzalVL3Pr1ZpdfbrbTTi0PXpMmlX4ALMXGjd7G9Np+/HGzt7/d9zd0qC8ls6OPNvvUp8y+/nWzn/7Uj9HDD/t57kyb26FeBOdg4I7M9vnA+QVp7gAOTtZ7As8DKkybTVfqV7eCs2GDP/2edVb5dF/8op/Ou+5qHX7nnR6+ww5mV17ZfjeXmd/0Jb8BVXKBp1bJXnt1rlti4UIX1gMO8HIPO6yyfH/9q9nee3vZF17YOu6uu/zmsffefkMr1Y4lS/xXyLx5Lmp77OFiWGjlDRzYYqUU8sILrfP06uXt+vrX/fivWeOWWhp/zz2l2/ivf7m4NTS0rkNXf717t1iv225r9pa3tI7fYgu/5mbOdEFqaPA2bbONdzeOGmU2Zoz/xo71LtQRI/zGOWyYC/2YMWa77+7xY8eaDR5cvk6SX6fDhvm+Ro1yQR471veVWqTZNuyyi9c9+9t999blpvnHjHHLt1T3WVOT2amnej222cZs3Dj/pUJz6KF+ri66yB8Ae/TwfaflFf7SckeM8GOcPYfpsR840MWsudmtqy9/2fP061f6GPXr56K5885+TAcMKP7QVCHdKTjy8qqPpOOBCWZ2RrJ9MnCgmZ2dSfOPJE1Tsv0UcCDwVeABM7smCb8SuM3MbigoYwowBWD48OH7P9tRV9R6Ytkyf0v9U59qeYERWt4TOfBAd+2uhMce83dQDjywsvQbN7qDw8EHty67M6xc6U4XAwZUlr652eubuk5nWbXK21z4HlBHef1195x78kn/vfoqfPKT7tpdijvu8OluXnnF00+c6O8tpZi5J9+6deWdK7J1eOghePxxP949enh7m5v9fZL1690NvXdv9wI0c2eEtWs9be/eflzXr3ePrJdfdu+/d73L3+1qaPD3ftJph844w1/STXn6abj6ap/k9PXX/bdxY8utsGdP/zU0+HZar/T+Ivm52G03d0zZeeeW9GvWuJPKokXu+NHc7I4yGzZ4/rScwYP9Q4E77OAD/kuXuoNJ6hRj1nINFC6z66NH+7RQpeYyfOghn4op9WRsaPDvRk2a1JJn5Ur4/vdLO9ekdZH8fGyzjV/Tffu2HJu+fd3bc7vtiu9j7Vp3qGlq8vfCnn3Wr+nVq/2aMvN99+rlDkvnnlv6+imDpLlm1tipzB0tK0fBOQEfe8kKzngz+3+ZNPOSNFnBGQ9MBf5eIDgzzOzGUuU1NjbanPgWSRAEQYfoTsHJ0y26CcjO/TIUWFIqjaSewADgxQrzBkEQBG8i8hSc2cBoSSMl9cKdAgrfYpsOnJKsHw/ck/QpTgcmS+otaSQwGujiN5WDIAiCWtIzrx2bWbOks/EB/wbgKjObJ2kqPkg1HbgS+I2khbhlMznJO0/S9cB8oBn4hJl18KMvQRAEwaZEbmM43U2M4QRBEHScehnDCYIgCIL/EIITBEEQdAshOEEQBEG3EIITBEEQdAt14zQgaSXQ0akGBuHT6WxuRLs3L6LdmxcdbfcuZjY4r8pkqRvB6QyS5nSXd8amRLR78yLavXmxKbc7utSCIAiCbiEEJwiCIOgWNnfBuaLWFagR0e7Ni2j35sUm2+7NegwnCIIg6D42dwsnCIIg6CZCcIIgCIJuYbMUHEkTJC2QtFDSebWuT15IGibpXklPSJon6dwkfKCkOyU9mSy3rXVd80BSg6SHJf0p2R4p6cGk3dcln82oKyRtI+kGSf9MzvvBm8P5lvSp5Br/h6TfSepTr+db0lWSViRfTE7Dip5jOT9K7nWPSdqvdjXfDAVHUgNwKXAsMA74kKRxta1VbjQDnzGztwAHAZ9I2noecLeZjQbuTrbrkXOBJzLb3wF+kLT7JeD0mtQqX34I3G5muwN74+2v6/MtaQhwDtBoZnvgn0OZTP2e718BEwrCSp3jY/HviY0GpgA/66Y6FmWzExz8E9YLzWyRma0DpgGTalynXDCzpWb2ULL+Kn7zGYK39+ok2dXA+2pTw/yQNBSYCPwi2RZwBHBDkqTu2i2pP/A2/DtTmNk6M1vFZnC+8W97bZl8ObgvsJQ6Pd9m9hf8+2FZSp3jScCvzXkA2EbSTt1T07ZsjoIzBFic2W5KwuoaSSOAfYEHgR3MbCm4KAHb165muXEJ8HlgY7K9HbDKzJqT7Xo877sCK4FfJl2Jv5C0FXV+vs3s38D/As/hQvMyMJf6P99ZSp3jTep+tzkKjoqE1bVvuKR+wI3AJ83slVrXJ28kvQdYYWZzs8FFktbbee8J7Af8zMz2BV6jzrrPipGMV0wCRgI7A1vhXUmF1Nv5roRN6rrfHAWnCRiW2R4KLKlRXXJH0ha42FxrZjclwctTszpZrqhV/XLiUOA4Sc/gXaZH4BbPNkmXC9TneW8CmszswWT7BlyA6v18HwU8bWYrzWw9cBNwCPV/vrOUOseb1P1ucxSc2cDoxIOlFz64OL3GdcqFZNziSuAJM/t+Jmo6cEqyfgrwx+6uW56Y2flmNtTMRuDn9x4zOxG4Fzg+SVaP7V4GLJY0Ngk6EphPnZ9vvCvtIEl9k2s+bXddn+8CSp3j6cBHEm+1g4CX0663WrBZzjQg6d34E28DcJWZfaPGVcoFSYcBfwUep2Us44v4OM71wHD8z3qCmRUOQtYFkt4BfNbM3iNpV9ziGQg8DJxkZmtrWb9qI2kf3FGiF7AIOA1/sKzr8y3pa8AHcc/Mh4Ez8LGKujvfkn4HvAP/DMFy4ELgDxQ5x4kA/wT3ansdOM3M5tSi3rCZCk4QBEHQ/WyOXWpBEARBDQjBCYIgCLqFEJwgCIKgWwjBCYIgCLqFEJwgCIKgWwjBCYJ2kLRB0iOZX9Xe3pc0IjvrbxDUMz3bTxIEmz1vmNk+ta5EELzZCQsnCDqJpGckfUfSrOS3WxK+i6S7k++P3C1peBK+g6SbJT2a/A5JdtUg6efJ91z+T9KWSfpzJM1P9jOtRs0MgqoRghME7bNlQZfaBzNxr5jZePxt7kuSsJ/gU8LvBVwL/CgJ/xHwZzPbG5/jbF4SPhq41MzeCqwC/isJPw/YN9nPmXk1Lgi6i5hpIAjaQdJqM+tXJPwZ4AgzW5RMkrrMzLaT9Dywk5mtT8KXmtkgSSuBodnpVZLPRtyZfDgLSV8AtjCziyTdDqzGpy35g5mtzrmpQZArYeEEQdewEuul0hQjO7/XBlrGVifiX6fdH5ibmfk4CN6UhOAEQdf4YGb592T9fnyWaoATgZnJ+t3AWeCfOk++0FkUST2AYWZ2L/4huW2ANlZWELyZiCemIGifLSU9ktm+3cxS1+jekh7EH94+lISdA1wl6XP4FzhPS8LPBa6QdDpuyZyFf6GyGA3ANZIG4B/R+kHyueggeNMSYzhB0EmSMZxGM3u+1nUJgjcD0aUWBEEQdAth4QRBEATdQlg4QRAEQbcQghMEQRB0CyE4QRAEQbcQghMEQRB0CyE4QRAEQbfw/wGLYqW8D6CnXwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAEWCAYAAADPZygPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcHFW5//HPNwuQQEIgC0tICELQsBPCJrIIomyy6FXZERHEy0W4rij+VO51AURQXEAu4MIqm4CIyB5BhZAIwbAGIawJCSQBQoCZJM/vj3Pa6Qw9Mz3JdLqm+/t+veY13bV0PbX0eeqcOl2liMDMzKwe+tQ7ADMza15OQmZmVjdOQmZmVjdOQmZmVjdOQmZmVjdOQmZmVjdOQmZWGEoekLRJD33eLEkf6InPqvDZK0taIGndDsYfL+n2Tua/T9LhtYit3vJ+fFDS2K6m7TIJSZoh6a28sV+W9CtJq/VMqD0jx/ihKqbbQNISSb9YEXHVi6R1JF0kaaakNyQ9Luk0SavWO7auSPqkpL9JWijp7grjt5I0JY+fImmrsnGSdIakV/PfmZLUE/OWTXdY/i4syN+LJWXvFyzHer9P0qIupjldUmvep6X9+hNJI7qxnOUq+CSNkXS9pFckvSbpYUmH9uCy/wN4MSIezfOU1nmBpPmS7pU0YVnjr5akVSSFpPXaDT9d0oUAEfFORKwWES/VOp7uKI+xXiL9APUc4DtdTVttTeijEbEaMB7YFvhmd4OS1K+789TAkcA84GBJK6/IBa+o9Ze0JvB3YACwY0QMAvYEhgAbLsPnrej9Nhf4MXB6hVhWAm4ALgXWAH4D3JCHAxwHHAhsCWwB7Ad8bnnnLRcRl+WCZzVgb+Cl0vs8rNZ+k/fpUOATwBhgsqThK2DZAFcATwCjgGHA0cArPfj5xwOXtBv2m7xthwP3Ab/rweVZDysrM64D9pU0tNMZIqLTP2AG8KGy9z8EbsqvVwcuAmYCLwLfBfrmcZ8G/krKhnOB7+bhxwKPAW8AjwLj8/B1gWuBOcAzwBfKlvkd4Crgt3m+R4AJedwlwBLgLWAB8NVO1uVfwOeBl4H/aDduU+C2HOvLwDfy8L7AN/K8bwBTSF/AMUAA/co+427gsx2tPykJ3Am8SvriXgYMKZt/VN5xc/I0PwNWzvNvXjbdiLy+wyus43eBfwJ9OtgG3Y37B8B8YLOy6Yfn5Y/I7/cDHsrT/Q3Yoqvjqorj7rPA3e2GfTgfZyob9hywV379N+C4snHHAPct77ydxLgb8EKF4aNICe8V4Gng+LJxOwEPAq8Ds4Af5OGz835ZkP+2rvC5pwMXthvWn/R9Kn2/hgN/ysfQ3BzHOnncj4DFwNt5GT/Kw88DXsgxTQJ26GB9BbQC7+tkm+wM3J+PhX8AO3W27HbzDsyfP6yjdSadCAcwqGzYQcDDeZn3AJuUjZsFfCC/vhL4Ztm4vYCnOliPVfJy1utoH7SfhvS9vDlvx7+Tvju3l827LzA9x3k2KaEeXjb+c6QEPxf4IzCy3XKOJZVD84BzOtkH7zpOysZ9i1S+vgFMA/Yt2/avA2PLpl0PWEguo6rYzl8mlc0Ly4bfA3yqs+9Rt64JSRoF7EP6EkE6m1wEbARsTfqif7Zslu1JX8IRwPckfYKUUI4EBgP7A69K6gP8AZgKjAT2AE6W9JGyz9qfdBANAW4kFdBExBGkwuSjkc5Gz+wg9p1JG/VKUkI7smzcIOB24BZSMtwIuCOP/iJwSF7vwcBnSDumGkutP+lL/IO8jHGkwuo7OYa+wE3As6REMRK4MiLeyTGXN2McQjq451RY5oeA6yJiSZUxdhX3/5AS4yFl4z8JTIyI2ZLGAxeTvkBDgV8CN9aoprkp8HDkozt7OA8vjZ9aNm5qu3HLOm/V8n68mZTU1iUVdN+QtGue5GfA9yNiMDAWuD4P3wVYHG21qgepQkS0kr47O+dBfYDzgdHABnnYOXnaLwEPkE44VsvvIRWYm5P23w3A1ZL6V1hWkBLML3OzafumqjF5fU4F1iS1mFwvaY1Oll1uHPB6RFSsWeVj6ghSgbcgD9sB+AWpRjaUdFJ6fZ1aXi4gJZC1SCe7nymNkLQ2qdz5EulEYQ4woWz8wcDJwEfz/A+Sau3l9iaVs+OBoyXttgwxPgG8n1SBOAO4UtKwiFgIXMPS5cxhwB8jYn6V2/lTpFaX8prPY6TWhY51lqHy93UGaYfPJxWQvyA19awFvAMMKJv2EOCu/PrTwHPtPuvPwEkVlrF9hWm/Dvwqv/4OS59RbAK81S7GD3WxHhcC1+fXO5LOuEaUxf1gB/M9ARxQYfgYuq5RPNdFTAeWlptjmlP+ee22z/Pk2g0wGfhkB585nbIz756Im5TYni57/1fgyPz6POB/K2yzXbs6trrYNpVqQv+PlJjLh10GfCe/XkzZWTqpkA9S8l/meTuJcTfa1YSAXYHp7YadBpyXX08iFdJD203zPmBRF9uk4hkuqfD6Zwfz7ADMLHu/1Nl3helFOsl6bwfjh5FaQx4jtUBMJtfagG8D/9du+onkM+Eqlr0HMKPCOr9DKn8Wk2qMO5WN/xVwart5ngW2z6+Xtyb0Wl526e9tKtSE8uslwJiyzzibXG6RmnvvLhvXN6/L4fn9XcBhZeP7k8qotcqWM6Fs/I3Ayd05TjqY9nHgI2XH7lNl4/4J7N+N7Xxohc//EfCLzmKotiZ0YEQMiYj1I+I/I+ItYP28oWbmC4bzSWfB5RdJn2/3OaNI1cn21gfWLX1O/qxv5B1QMqvs9UJglWrPdiQNILWfXwYQEX8n1Z5KF1Q7iqurcV1Zav0ljZB0paQXJb1OOtMZVracZyPiXRenI+J+4E1gV0nvI9XUbuxgma8C6yxjvBXjJjUhDpC0vaT1ga2A3+dx6wNfarfvRpFqAUuR9I2yi/jnL0NcC0i10XKDSU0LlcYPBhZE+jYsz7zdsT4wpt32+CKwdh5/FOma05OS7m9X219WI0ln4EgaJOliSc/lY+xW2o6xiiR9XdITkl4jNfWs0tE8EfFKRHwlIsbldXqSVFOGtO6Ht1v3CVQ4FjowDxhUYfglETGEdFz/i3T8laxPqmmWL3M4aZv0hE1z2Tckx/DjDqZbm5TAy787z5a9Xrd8XEQsJjUPl6wPnF+2DnNIrUzltc32ZWC3r0FKOiZ3JiktZyPa9vVfgL6SdlTqtLMOqWm3FF9X27l9uQFpf87vLKbl6aL9POkMZVjZThocEeVNGO2/wM9T+eL488Az5Ts7IgZFxD5VxtJVQXEQqVD5hVKXzVmkjVdqkusors7GvZn/Dywbtna7adrH9YM8bItIzTGHkw7c0nJGd5JYf5OnPwK4JiLe7mC624GDchNnJd2OO1LT3lWkGuOhpGuCpcL7eeB77fbdwIi4ov2CI+L70dbcdHwH8XXmEWALaalea1vk4aXx5VX/LduNW9Z5u+N54PEKx/JBABHxWER8inSydi5wXe4c0d1kB/z7IvB+pLZ3gFNIBde2+Rj7MG3HGO2XI2lP4ETSd2QIqRntrXbzVBQRs0ln+2OUel4+TzoDL1/3VSPinErLruAxYJCkjhLgbFKz7w/Kpnke+FaF4++6Ch/xJp0f98tjFmn9RpUNG132emb5uPz9bF+Af7rdegyIiCk9FaCkjYGfkmpla+ak+hR5X+cTrt/SVs5cGam5txRfV9u50v4dx9LN3O+yzEkoImaSzrJ+JGmwpD6SNixr+67kQuDLkrZRslE+s54EvC7pa5IGSOoraTNJ21YZzsvAezoZfxTpusXmpLOorUgXiLeStDnpWszakk5W6vs/SNL2ZTH/r6SxOeYtJA2NdD3mRdKZX19Jn6Hr3meDyE2bkkYCXykbN4l0oJ4uaVWlLqI7lY2/hFRQHE46UDpyNinh/iZvWySNlHS2pC2WMW6Ay0ltvofl1yX/Bxyfa0nKse+rdJ2t23JMqwD9gD55O5SuT9xNapL5Qt5P/5WH35n//xb4Yl7fdUnt77/ugXm74968Hifn2PvlY2Z8Hn5kPn4Wk5p6gtSMM5t0Fjq6w08uI6m/pE1JJweDSAmN/Hoh6Rgbxrt7srb/rgwiNfvMAVYiXQNcpZPlniVpk7yfVif1ZpsWEW+STpQ+IWmPPH5Afl0q7Dv9nuYWlrtJ18c6muZh0hl76ZrSBcCJkibk4281SftLGlhh9oeA/SQNyd+/EztaTnflk8I/AKfl9d6C9F0puRHYVtJ++Xj+Cinhl5wPfFPSewEkrSHp48sRUt98/JX+ViLVnJaQ9nUfSceTakLlfku65nsIS5cz3dnO5HVYlVTm3tHRNED3e8e1G7c6bT1rXiNdTDs4j/s0cG+FeY4nXTNYQOqdUWpPXpfU/XMWqVp+X2m5pGtCl5Z9xhjKrmsAB5Ca1+YDX263vJGkau3mFWK5GTgrv94sb6x5OYZToq3t9pu09Sh5gLbeMHvn4fNJbZ8TWfrayr3tlrcpqXfdAtIX4kuUXVMgnTldT1vvuXPbzX973h8dXqco25YX5/V4g9Tu+21g4LLEXfa5T5GafVZqN3yvvF3mkxLp1ZT1XurOX15+tPv7ddn4rfM2fIvU+2rrsnECzswxzs2v1RPzdhDrbnTcO+4qUqE7j3QNbZc87qq8b98gtbnvUzbfGaQCYj6wVYXPPZ2UMN4gndU/STqzXafdMXRvPsYeB/6TsmtN5Hb/HNeZpCb1S0g9o14kXV/693WUCjFckOdfkGO9gaV7VO2Ulz+PlFhvBNattOwOPv/jwO/brXP7HoG75njXzO/3z/v1NeAl0rWfAXlc+TWhVUlNh6+Tyqov07O949YmdW7qqHfcR/P6d9Q77hhS7ft1UlPe+R3FQrvrWxVibP8deiqPOytv/zn5eHvXdbq8/56o8LlVbeey6Y8ALu/qO688sfUCki4m/S6l27/TMusNcnPp/aSmqUfrHU8zknQ58GhEfHc5PkOkhHVwRDzZ6bROQr2DUvfXh0hn78/UNxoza0SSNiK1EoyLiBe7mr4n+N5xvYCk/yU1Xf7QCcjMakHSmaRmyv9ZUQkIXBMyM7M6ck3IzMzqpgg3Fa3KsGHDYsyYMfUOw8ys15gyZcorEbGibm67THpNEhozZgyTJ0+udxhmZr2GpGe7nqq+3BxnZmZ14yRkZmZ14yRkZmZ14yRkZmZ14yRkZmZ14yRkZmZ14yRkZmZ102t+J2RmK8CSJXDFFdDaCttvD+99L/TxuarVjpOQNbcIePZZWGstGDCg4+leew1WXRX61fErs2gR3H03tLTAdtvBsA6e2r1wIUyZAvffn/5mzIAttkhJZccdYbPNQBUenPraa3DUUXDDDW3DVl8dNtwQhg5Nf+9/P3zuc7DSSpWX/fbb8I9/tC37scdg0KA077rrwte/DqPLntsXkdapTx/YZhtYrYMnVi9YkNZ/yJDOt88//gH33ZeW/dRTsMcecMQRMG5cWtb06fDggymWbbaBgRWeyRaR5l95Zdh666XHPfccPPQQfOADsOaa7563K7NmpW1XmnfxYpg0Cf7853T8HXwwrL9+9z+3F+s1NzCdMGFC+I4JDWbRoqUL9blz4eqr4Q9/SF/WuXPhjTdgl13g8MNhn31SwQCpIO7ff+nC9I034Ec/ghdegI98BPbcMyWOv/0Nbr4ZnnkmFSrbbw/Dh6dlXXYZPP10mn+99dKZ/667pmVtvTX85S/wk5/AjTfCJpvAhRem+QFmzoSzz4bnn4eNNoKxY1OhUiqAX3sNPvSh9Fk77ABvvpnWae5cePXV9H/+/FT7KG2PZ59NhefTT6d4tt8eJkyAhx9ONZSXX25b3w03TDGVEsSbb6blPvxwigNggw3S39SpaZkAG2+ctufBB8Pa+aGnTz8Nn/xk+n/WWWnbldbj+edTrLNnp/Ebb5y28777pkK9NN3996fltOYnQo8eDZtvnpLi3Llp2sGD4aabUgJ4+2044QS4+OI0fZ8+KUFuv33b37PPwqWXpsTY2gp77ZVi33VXeP31tE5PPAF/+hPcemvanpCSzJgxKSEtWZKS0KxZMG9e2/br2zfFt8MOaVnbbgsPPADnnpsSFaRtdMYZKWmcfnraNu+8k2LdYYd0bK5S4UG0ffqk9R87FtZZJyWZSy+Fv/41jV9zzbT//vWvtG369Gk7DnbdNZ1kzJ+fxknpWF0GkqZExIRlmnkFcRKyzi1Zkgr1dddtSxitrXDNNXDRRen92LHpr1QQv+c96WyvVOguXpy+dIMHp4L7iivSF/Khh2DkyDTPgAFw++3pszfeuO3su3//lEBefjmdlQ8alD5z4cK0vMMOg0MOgb//PZ1lz5qVlvP666mQGTAgnUX365cK9Rkz2tatT590prz//qlwmj4dHnkkFUARKYG9+WaK47DD4Lrr4MUX4aST0vznn5/iXX/9VFiWCv7VV0+FyGqrwR13pFiqtd56aXuMGZM+c9KkFP9KK8F++6U4hg5tK/RLhdirr6Z13G67pQvxESPS50akBHLnnXD55an20d6IEamw26XDp2unffHFL6aCf7XVUmyQXm+77dLLXmedped99NGUkOfMSYn9l7+EyZPhm99MNbT7709JY9KktmQCaX0PPjjVWi6/PO2D9tZeG/beO5187LRT2o6Qjpvf/S7FPXp0imv8+HRMl7bhpElL76NNN4UTT4SXXoIzz0xJYPXV07F12GHw6U/DPfekz+xOmbTJJnDooSlpTZ+e9t3IkSnuD384HYOXX56+G888k74zpRrkrbdWv5wyTkI9yEmozLRp6SyydJCuvvq7p4lIf5Xa81tbU6H9pz+lL1HprPztt1PhN3Zsap6aOrXtCzpgQDojHzcuncm+9FJbopg+fekzzD59UoHY0rL0cvv2TUktIhWWe+yRCpTp09Py9903NZ1sueXSNZxFi1Jhfs01Sye0v/wlFaqlY3j77VPhts02qXD505/S5+65Z1rW4MHp/aRJabl7752+4O3Nnp3OXP/yl3S2e+ihaf1ffx1OOQXOOy+tyxFHpAJ0ww3TNn3mmRTL2LFt2721NdXEpk5NTUlrrtm234YOTcP69m1bdvtmssWL4cknUyG7xhodHhLd9txz8Mc/pmQOaX994hOVt0d7ra0pAT/ySDomdtghHRfl69GRl1+Gj3401TgGDYJLLoEDDlh6miVL0jExaVJa5w9/uK35b/FimDgxLbu0LddbLyWOZb12tWQJPP54imnUKPjgB9v2w7PPpn388stw2mkpWZbrqPxsbU0nPNOnp229447vPq5XACehHuQklJ17btuZeMk++6Rmofe+N30prr4avva19MXZcMNUKK62WltT0KOPpqaifv1Sk9Naa6Uvc//+qSCdPj2d9ZWaRjbfPA27777U1POBD6QY9tqr7Yv/6qupGWn69PT39tupkF1zzVQ4zZuXphkwIDX7bLxxz2yPF19MyWnttVMhuiIuoj/6aFqPDTao/bIa0cKF8NOfwoEHpmPWasZJqAc5CZGq6Ucckc4cjzkmFer/+ldKTAsXwvHHpzPue+5JZ1177NGWGErNSkOHptrO3nun6xWDB1deVsQKP2szs57VG5KQe8cVxaJFqbbw5JOpGenmm9MZ9957pwuxkNqid98drrxy6YuhJ56Ymgx+/vPUY+qCC+Azn6mueaQjTkBmtgK4JlQvpa6p554Ld92VmsdK+vZNXWHHjUu9smbNSsMnTEjXQAYNqvyZL72UxnU03syaimtCljz+OPzgB6l3Uumi9JQp6fpKqefPOuuk1yNHwm67tV2E/sUvUuKZOBFOPrnzBFPNRWUzswJxEqqViNSL7YwzUpfkUu+yGTPSD+rWWit1cT7kkM5/JNm3b+rdteeeKyx0M7MVxUmop7z9duok8MQTqTvxzTenhDNwIHzpS/CVr6QfSJqZ2b85CS2vF19MPyKcOrXtNwMDB6aeZ6ecAgcd1PaDQTMzW4qT0PJobU2/eZk+Hb71rfTbl7Fj0326SreXMTOzDjkJLY+vfjX9Gv6KK1LnAjMz6xbfo31ZXXUV/PjH8IUvOAGZmS0jJ6HuWrIk3fX3mGPS/aB++MN6R2Rm1ms5CXXHPfekOwUfc0y6n9rVV3f8XBUzM+uSk1C1fvazdIv72bPT7db/+tf0w1IzM1tm7phQjUsvTfdnO+CAlIAqPY3RzMy6zUmoKzfd1PGNQ83MbLm4Oa4zd96ZnlGz9dZw/fVOQGZmPcxJqCM33JAeFrfRRunRCr4ztZlZj3MSquSSS+DjH4ettkp3rx42rN4RmZk1JCeh9q6+Go48Mj1O4fbb06MXzMysJtwxodzixXDqqakGdNNNvgZkZlZjNa8JSTpJ0jRJj0g6OQ/bStJ9kh6SNFnSdrWOoyrXX59uRnrqqU5AZmYrQE2TkKTNgGOB7YAtgf0kjQXOBE6LiK2Ab+X39RWRHkC30Ubp8QtmZlZztW6OGwfcFxELASRNBA4CAhicp1kdeKnGcXRt4kR44AE4//z0NFMzM6u5WiehacD3JA0F3gL2ASYDJwN/lnQWqTb2/kozSzoOOA5g9OjRtY30jDPSw+eOPLK2yzEzs3+raXNcRDwGnAHcBtwCTAUWAZ8H/jsiRgH/DVzUwfwXRMSEiJgwvJaPxn74YbjlFjjpJBgwoHbLMTOzpdS8Y0JEXBQR4yNiF2AuMB04CrguT3I16ZpR/fz857DqqvD5z9c1DDOzZrMieseNyP9HAx8DriBdA9o1T7I7KTHVz623wkc+AmusUdcwzMyazYr4ndC1+ZpQK3BCRMyTdCzwE0n9gLfJ133q4plnYMYM+PKX6xaCmVmzqnkSioidKwy7F9im1suuyp13pv8f/GB94zAza0K+bc9dd8Faa8G4cfWOxMys6TR3EopINaHddwep3tGYmTWd5k5CTzwBM2emJGRmZitccyeh0vUgJyEzs7pwElp/fdhgg3pHYmbWlJo3CS1Zkjol+HqQmVndNG8SevhhmDvXTXFmZnXUvEnIvw8yM6u75k1Cd98NG28MI0fWOxIzs6bVnEkoIj07aMcd6x2JmVlTa84k9NJLMGsWTJhQ70jMzJpacyahyZPT/22Kcfs6M7Nm1ZxJaMqU9AjvLbesdyRmZk2tOZPQ5MmwySYwcGC9IzEza2rNl4QiUk3I14PMzOqu+ZLQCy/A7NlOQmZmBdB8ScidEszMCqP5ktCUKdCvH2yxRb0jMTNres2XhCZPhk03hQED6h2JmVnTa64k5E4JZmaF0lxJ6Lnn4JVXnITMzAqiuZKQOyWYmRVKcyWhKVOgf393SjAzK4iqkpCkayXtK6l3J63Jk2GzzWDllesdiZmZUX1N6DzgUGC6pNMlva+GMdXO88/DRhvVOwozM8uqSkIRcXtEHAaMB2YAt0n6m6SjJfWvZYA9qqXFtSAzswKpunlN0lDg08BngQeBn5CS0m01iawWWlthpZXqHYWZmWX9qplI0nXA+4BLgI9GxMw86neSJtcquB7X0pI6JpiZWSFUlYSAn0XEnZVGRETv+dFNS4trQmZmBVJtc9w4SUNKbyStIek/axRT7TgJmZkVSrVJ6NiImF96ExHzgGNrE1INtba6Oc7MrECqTUJ9JKn0RlJfoHdVKSJcEzIzK5hqrwn9GbhK0vlAAMcDt9QsqlpYtCj9dxIyMyuMapPQ14DPAZ8HBNwKXFiroGqitTX9d3OcmVlhVJWEImIJ6a4J59U2nBpqaUn/XRMyMyuMau8dN1bSNZIelfR06a+K+U6SNE3SI5JOLht+oqQn8vAzl2cFquYkZGZWONU2x/0K+DZwDvBB4GhSs1yHJG1G6kG3HdAC3CLpj8B6wAHAFhHxjqQRyxh797g5zsyscKrtHTcgIu4AFBHPRsR3gN27mGcccF9ELIyIRcBE4CDSdaXTI+IdgIiYvWyhd5NrQmZmhVNtEno7P8ZhuqT/knQQ0FUNZhqwi6ShkgYC+wCjgI2BnSXdL2mipG2XOfrucBIyMyucapvjTgYGAl8A/pfUJHdUZzNExGOSziDd4HQBMBVYlJe5BrADsC2p6/d7IiLaf4ak44DjAEaPHl1lqB1wc5yZWeF0WRPKP0z9ZEQsiIgXIuLoiPh4RNzX1bwRcVFEjI+IXYC5wHTgBeC6SCYBS4BhHcx/QURMiIgJw4cP79aKvYtrQmZmhdNlTSgiFkvaRpIq1VY6I2lERMyWNBr4GLAjKensDtwtaWPSnRdeWYbYu8dJyMyscKptjnsQuEHS1cCbpYERcV0X812bn0PUCpwQEfMkXQxcLGkaqdfcUd1Nbsuk1BznJGRmVhjVJqE1gVdZukdcAJ0moYjYucKwFuDwagPsMaWakK8JmZkVRrV3TDi61oHUnJvjzMwKp9onq/6KVPNZSkR8pscjqhUnITOzwqm2Oe6msterkH50+lLPh1ND7qJtZlY41TbHXVv+XtIVwO01iahWXBMyMyucau+Y0N5YYDl/PbqCOQmZmRVOtdeE3mDpa0KzSM8Y6j3cHGdmVjjVNscNqnUgNeeakJlZ4VT7PKGDJK1e9n6IpANrF1YNOAmZmRVOtdeEvh0Rr5XeRMR80vOFeg83x5mZFU61SajSdNV27y4G14TMzAqn2iQ0WdLZkjaU9B5J5wBTahlYj2tpAQn69q13JGZmllWbhE4k3Wz0d8BVwFvACbUKqiZaW1NTnDp9KrmZma1A1faOexM4pcax1FZLi5vizMwKptrecbdJGlL2fg1Jf65dWDXgJGRmVjjVNscNyz3iAIiIecCI2oRUI05CZmaFU20SWpKfjgqApPWpcFftQitdEzIzs8Kotpv1qcC9kibm97sAn6tNSDXimpCZWeFU2zHhFknjgR0AAf8dEa/UNLKe5iRkZlY4Vd9FOyJeiYibgEeB4yVNq11YNeDmODOzwqm2d9w6kk6WNAl4BOgLHFLTyHqaa0JmZoXTaRKSdKykO4GJwDDgs8DMiDgtIv65IgLsMU5CZmaF09U1oZ8DfwcOjYjJAJJ6V6+4EjfHmZkVTldJaF3gE8DZktYi3bKnd5bkLS2w2mr1jsLMzMp02hyXOyOcFxG7AHsArwGzJT0m6fsrJMKe4uY4M7PC6U7vuBe0rJcZAAAOCklEQVQi4qyI2AY4EHindmHVgJvjzMwKZ5meCRQRTwCn9XAsteWakJlZ4VRdE+r1nITMzAqnuZKQm+PMzAql2h+r3lHNsEJrbXVNyMysYDq9JiRpFWAgMEzSGqT7xgEMJnXf7j3cHGdmVjhddUz4HHAyKeFMoS0JvU76IWvv4SRkZlY4nSahiPgJ8BNJJ0bET1dQTLXhLtpmZoVT7aMcfirp/cCY8nki4rc1iqtnRbgmZGZWQFUlIUmXABsCDwGL8+AAekcSWrQo/XcSMjMrlGp/rDoB2CQieu/NS8HNcWZmBVPt74SmAWvXMpCaamlJ/10TMjMrlGqT0DDgUUl/lnRj6a+aGSWdJGmapEckndxu3JclhaRh3Q28W5yEzMwKqdrmuO8sy4dL2gw4FtgOaAFukfTHiJguaRSwJ/Dcsnx2t7g5zsyskKqqCUXERGAG0D+/fgD4RxWzjgPui4iFEbGI9ITWg/K4c4Cvkjo41JZrQmZmhVTtbXuOBa4BfpkHjQSur2LWacAukoZKGgjsA4yStD/wYkRM7WK5x0maLGnynDlzqgm1MichM7NCqvaa0AnATqQ7JRAR04ERXc0UEY8BZwC3AbcAU4FFwKnAt6qY/4KImBARE4YPH15lqBWUkpCb48zMCqXaJPRORLSU3kjqR5XNaBFxUUSMz09nnUtq1tsAmCppBrAe8A9Jtet9V7om5JqQmVmhVJuEJkr6BjBA0p7A1cAfqplR0oj8fzTwMeC3ETEiIsZExBjgBWB8RMzqdvTVcnOcmVkhVds77hTgGOCfpJua3gxcWOW810oaCrQCJ0TEvG5HubzcHGdmVkjVJqEBwMUR8X8AkvrmYQu7mjEidu5i/JgqY1h2bo4zMyukapvj7iAlnZIBwO09H06NuDnOzKyQqk1Cq0TEgtKb/HpgbUKqATfHmZkVUrVJ6E1J40tvJG0DvFWbkGrAzXFmZoVU7TWhk4CrJb2U368DfKo2IdWAm+PMzAqpyyQkqQ+wEvA+4L2kR3w/HhGtNY6t5zgJmZkVUpdJKCKWSPpRROxIug1P7+MbmJqZFVK114RulfRxSappNLXimpCZWSFVe03oi8CqwGJJb5Ga5CIiBtcssp7kJGRmVkhVJaGIGFTrQGrKXbTNzAqp2kc5SNLhkv5ffj9K0na1Da0HuYu2mVkhVXtN6BfAjsCh+f0C4Oc1iagWWlpAgr596x2JmZmVqfaa0PYRMV7SgwARMU9S76lWtLSkprhe2q/CzKxRVVsTas03LQ0AScOBJTWLqqe1tropzsysgKpNQucCvwdGSPoecC/w/ZpF1dNaWpyEzMwKqNrecZdJmgLsQeqefWB+dHfvUGqOMzOzQuk0CUlaBTge2Ij0QLtfRsSiFRFYj3JznJlZIXXVHPcbYAIpAe0NnFXziGrBzXFmZoXUVXPcJhGxOYCki4BJtQ+pBtwcZ2ZWSF3VhP59p+xe2QxX4uY4M7NC6qomtKWk1/NrAQPy+9537zgnITOzwuk0CUVEY9xiwEnIzKyQqv2dUO/W2uprQmZmBdQcScg1ITOzQnISMjOzummeJOTmODOzwmmOJOQu2mZmhdQcScjNcWZmhdQ8ScjNcWZmhdMcScjNcWZmhdQcScjNcWZmhdQ8ScjNcWZmhdMcScjNcWZmhdT4SSjCzXFmZgXV+EloUX4ChZvjzMwKp/GTUGt+JJJrQmZmhdP4SailJf13EjIzK5yaJiFJJ0maJukRSSfnYT+U9LikhyX9XtKQWsbgJGRmVlw1S0KSNgOOBbYDtgT2kzQWuA3YLCK2AJ4Evl6rGIC2JORrQmZmhVPLmtA44L6IWBgRi4CJwEERcWt+D3AfsF4NY/A1ITOzAqtlEpoG7CJpqKSBwD7AqHbTfAb4U0cfIOk4SZMlTZ4zZ86yReHmODOzwqpZEoqIx4AzSM1vtwBTgVINCEmn5veXdfIZF0TEhIiYMHz48GULxM1xZmaFVdOOCRFxUUSMj4hdgLnAdABJRwH7AYdFRNQyBjfHmZkVV79afrikERExW9Jo4GPAjpL2Ar4G7BoRC2u5fMDNcWZmBVbTJARcK2ko0AqcEBHzJP0MWBm4TRKkzgvH1ywCN8eZmRVWTZNQROxcYdhGtVzmu7g5zsyssHzHBDMzq5vmSUJujjMzK5zGT0JujjMzK6zGT0JujjMzK6zmSUJujjMzK5zmSUKuCZmZFU7jJyFfEzIzK6zGT0KuCZmZFVbzJCFfEzIzK5zGT0JujjMzK6zGT0ItLSBB3771jsTMzNppjiTUv39KRGZmViiNn4RaW90UZ2ZWUI2fhFpanITMzAqqOZKQe8aZmRVS4ychN8eZmRVW4ychN8eZmRVWcyQhN8eZmRVScyQh14TMzAqp8ZOQrwmZmRVW4ychN8eZmRVWcyQh14TMzAqp8ZOQm+PMzAqr8ZOQm+PMzAqrOZKQa0JmZoXU+EnIzXFmZoXV+EnINSEzs8JqjiTka0JmZoXU+EnIzXFmZoXV+EnIzXFmZoXV+Elo//1h/Ph6R2FmZhX0q3cANffrX9c7AjMz60Dj14TMzKywnITMzKxunITMzKxuap6EJJ0kaZqkRySdnIetKek2SdPz/zVqHYeZmRVPTZOQpM2AY4HtgC2B/SSNBU4B7oiIscAd+b2ZmTWZWteExgH3RcTCiFgETAQOAg4AfpOn+Q1wYI3jMDOzAqp1EpoG7CJpqKSBwD7AKGCtiJgJkP+PqHEcZmZWQDX9nVBEPCbpDOA2YAEwFVhU7fySjgOOAxg9enRNYjQzs/pRRKy4hUnfB14ATgJ2i4iZktYB7o6I93Yx7xzg2W4sbhjwyjIH23t5vZuL17u5dHe914+I4bUKpifU/I4JkkZExGxJo4GPATsCGwBHAafn/zd09Tnd3ZCSJkfEhGUIuVfzejcXr3dzacT1XhG37blW0lCgFTghIuZJOh24StIxwHPAJ1ZAHGZmVjA1T0IRsXOFYa8Ce9R62WZmVmyNfMeEC+odQJ14vZuL17u5NNx6r9COCWZmZuUauSZkZmYF5yRkZmZ103BJSNJekp6Q9JSkhr0nnaRRku6S9Fi+OexJeXhT3BxWUl9JD0q6Kb/fQNL9eb1/J6nhnukuaYikayQ9nvf7jk20v/87H+fTJF0haZVG3OeSLpY0W9K0smEV97GSc3NZ97CkXvkI6YZKQpL6Aj8H9gY2AQ6RtEl9o6qZRcCXImIcsANwQl7XZrk57EnAY2XvzwDOyes9DzimLlHV1k+AWyLifaQbAj9GE+xvSSOBLwATImIzoC9wMI25z38N7NVuWEf7eG9gbP47DjhvBcXYoxoqCZHu1v1URDwdES3AlaSbpTaciJgZEf/Ir98gFUgjaYKbw0paD9gXuDC/F7A7cE2epOHWW9JgYBfgIoCIaImI+TTB/s76AQMk9QMGAjNpwH0eEX8B5rYb3NE+PgD4bST3AUPyHWh6lUZLQiOB58vev5CHNTRJY4CtgftpjpvD/hj4KrAkvx8KzM93aofG3O/vAeYAv8rNkBdKWpUm2N8R8SJwFumH7TOB14ApNP4+L+loHzdEeddoSUgVhjV0H3RJqwHXAidHxOv1jqfWJO0HzI6IKeWDK0zaaPu9HzAeOC8itgbepAGb3irJ10AOIN3ua11gVVJTVHuNts+70hDHfaMloRdIj4ooWQ94qU6x1Jyk/qQEdFlEXJcHv1yqkuf/s+sVX43sBOwvaQapuXV3Us1oSG6qgcbc7y8AL0TE/fn9NaSk1Oj7G+BDwDMRMSciWoHrgPfT+Pu8pKN93BDlXaMloQeAsbnXzEqki5c31jmmmsjXQS4CHouIs8tG3Ui6KSxUeXPY3iQivh4R60XEGNL+vTMiDgPuAv4jT9aI6z0LeF5S6W7zewCP0uD7O3sO2EHSwHzcl9a9ofd5mY728Y3AkbmX3A7Aa6Vmu96k4e6YIGkf0plxX+DiiPhenUOqCUkfAO4B/knbtZFvkK4LXQWMJt8cNiLaX+hsCJJ2A74cEftJeg+pZrQm8CBweES8U8/4epqkrUidMVYCngaOJp1INvz+lnQa8ClSr9AHgc+Srn801D6XdAWwG+mRDS8D3waup8I+zgn5Z6TedAuBoyNicj3iXh4Nl4TMzKz3aLTmODMz60WchMzMrG6chMzMrG6chMzMrG6chMzMrG6chKypSVos6aGyvx67C4GkMeV3Qzazd+vX9SRmDe2tiNiq3kGYNSvXhMwqkDRD0hmSJuW/jfLw9SXdkZ/fcoek0Xn4WpJ+L2lq/nt//qi+kv4vPwvnVkkD8vRfkPRo/pwr67SaZnXnJGTNbkC75rhPlY17PSK2I/0q/cd52M9It8/fArgMODcPPxeYGBFbku7p9kgePhb4eURsCswHPp6HnwJsnT/n+FqtnFnR+Y4J1tQkLYiI1SoMnwHsHhFP5xvFzoqIoZJeAdaJiNY8fGZEDJM0B1iv/LYx+REbt+WHkSHpa0D/iPiupFuABaRbslwfEQtqvKpmheSakFnHooPXHU1TSfm9zBbTdh12X9JTgLcBppTdDdqsqTgJmXXsU2X//55f/410926Aw4B78+s7gM9Desx8fhJqRZL6AKMi4i7Sw/mGAO+qjZk1A599WbMbIOmhsve3RESpm/bKku4nnawdkod9AbhY0ldITzo9Og8/CbhA0jGkGs/nSU8BraQvcKmk1UkPJjsnP6rbrOn4mpBZBfma0ISIeKXesZg1MjfHmZlZ3bgmZGZmdeOakJmZ1Y2TkJmZ1Y2TkJmZ1Y2TkJmZ1Y2TkJmZ1c3/B7aEH79RmsVcAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# gradient descent\n",
    "detailed_logger = False\n",
    "main_logger = True\n",
    "main_logger_output_epochs = 10\n",
    "L2 = False\n",
    "Dropout = False\n",
    "momentum = True\n",
    "hidden_layer_relu = True\n",
    "hidden_layer_tanh = False\n",
    "hidden_layer_sigmoid = False\n",
    "\n",
    "# hyber-parameters\n",
    "alpha = .01;\n",
    "epsilon = .85\n",
    "keep_prob = .9\n",
    "number_of_epochs = 100\n",
    "batch_size = 27\n",
    "momentum_coef = .1\n",
    "\n",
    "# copy initalization\n",
    "W = Weights.copy()\n",
    "B = Bias.copy()\n",
    "\n",
    "# data arrays\n",
    "cost_array = []\n",
    "accuracy_array = []\n",
    "interation_array = []\n",
    "\n",
    "# rename\n",
    "X_train = np.float64(training_images).copy()\n",
    "Y_train = np.float64(training_labels).copy()\n",
    "\n",
    "X_test = np.float64(testing_images).copy()\n",
    "Y_test = np.float64(testing_labels).copy()\n",
    "\n",
    "#m = size\n",
    "m = number_of_training_images\n",
    "\n",
    "def model(W, B, A):\n",
    "    return np.dot(W, A) + B\n",
    "\n",
    "def activation_relu(Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    return np.where(Z > 0, Z, 0)\n",
    "\n",
    "def activation_tanh(Z):\n",
    "    return np.tanh(Z)\n",
    "\n",
    "def activation_sigmoid(Z):\n",
    "     return 1/(1 + np.exp(-Z))\n",
    "\n",
    "def loss(A, Y):\n",
    "    epsilon = 1e-20\n",
    "    return np.where((Y == 1), np.multiply(-Y, np.log(A + epsilon)), -np.multiply((1 - Y), np.log(1 - A + epsilon)))\n",
    "    #return np.multiply(-Y, np.log(A)) - np.multiply((1 - Y), np.log(1 - A)) \n",
    "    \n",
    "def cost(L):\n",
    "    return np.multiply(1/L.shape[1], np.sum(L))\n",
    "\n",
    "def cost_L2(L, W, epsilon):\n",
    "    L2 = np.multiply(epsilon/(2*W.shape[1]), np.multiply(W[len(W)-3], W[len(W)-3]).sum() + np.multiply(W[len(W)-2], W[len(W)-2]).sum() + np.multiply(W[len(W)-1], W[len(W)-1]).sum())\n",
    "    J = cost(L)\n",
    "    return L2 + J\n",
    "\n",
    "def prediction(A):\n",
    "    return np.where(A >= 0.5, 1, 0)\n",
    "    \n",
    "def accuracy(prediction, Y):\n",
    "    return 100 - np.multiply(100/Y.shape[0], np.sum(np.absolute(Y - prediction)))  \n",
    "    \n",
    "def forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "            print('Forward Propagation Training Data Complete')\n",
    "    return A_layers, Z_layers, D\n",
    "\n",
    "def forward_propagation(W, B, A, layer):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "        A = forward_propagation(W, B, A, layer)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        A = activation_sigmoid(Z)  \n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "            print('Forward Propagation Testing Data Complete')\n",
    "    return A\n",
    "\n",
    "def dZ(dZ, W, Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    W = np.where(~np.isnan(W), W, 0)\n",
    "    dZ = np.where(~np.isnan(dZ), dZ, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    W = np.where(~np.isinf(W), W, 0)\n",
    "    dZ = np.where(~np.isinf(dZ), dZ, 0)\n",
    "    if(hidden_layer_relu == True):\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.where(Z > 0, 1, 0))\n",
    "    elif(hidden_layer_tanh == True):\n",
    "        A = activation_tanh(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), 1- np.multiply(A, A))\n",
    "    elif(hidden_layer_sigmoid == True): \n",
    "        A = activation_sigmoid(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.multiply(A, (1-A)))\n",
    "\n",
    "def dW(dZ, A):\n",
    "    return np.multiply(1/dZ.shape[1], np.dot(dZ, np.transpose(A)))\n",
    "\n",
    "def dW_L2(dZ, A, W, epsilon):\n",
    "    return np.multiply(epsilon/Z.shape[1], W) + dW(dZ, A)\n",
    "\n",
    "def dB(dZ):\n",
    "    return np.multiply(1/dZ.shape[1], np.sum(dZ))\n",
    "\n",
    "def backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB):\n",
    "    if(layer >= 0):\n",
    "        if(layer == len(W) - 1):\n",
    "            _dZ = A_layers[layer+1] - Y\n",
    "        elif(layer >= 0):\n",
    "            _dZ = dZ(_dZ, W[layer+1], Z_layers[layer])\n",
    "        if(Dropout == True):\n",
    "            _dZ = np.multiply(_dZ, D[layer])\n",
    "        if(L2 == True):\n",
    "            _dW = dW_L2(_dZ, A_layers[layer], W[layer], epsilon)\n",
    "        else:\n",
    "            _dW = dW(_dZ, A_layers[layer])\n",
    "        _dB = dB(_dZ)\n",
    "        if(momentum == True):\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(alpha, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(alpha, _dB)\n",
    "            W[layer] = W[layer] - V_dW[layer]\n",
    "            B[layer] = B[layer] - V_dB[layer]           \n",
    "        else:\n",
    "            W[layer] = W[layer] - np.multiply(alpha, _dW)\n",
    "            B[layer] = B[layer] - np.multiply(alpha, _dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Layer: ' + str(layer))\n",
    "        layer = layer - 1\n",
    "        W, B = backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Propagation Complete')\n",
    "    return W, B\n",
    "    \n",
    "\n",
    "def shuffle(X, Y, number_of_training_images):\n",
    "    random_array = np.random.permutation(np.arange(number_of_training_images))\n",
    "    return X[:, random_array], Y[random_array]\n",
    "    \n",
    "start_time = time.time()   \n",
    "# main loop\n",
    "for epoch in range(1, number_of_epochs):\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Main Loop Epoch: ' + str(epoch))\n",
    "    \n",
    "    # shuffle data\n",
    "    X, Y = shuffle(X_train.copy(), Y_train.copy(), number_of_training_images)\n",
    "    number_of_batches = int(np.floor(number_of_training_images/batch_size))\n",
    "    split_index = number_of_batches*batch_size\n",
    "\n",
    "    # parse into minibatches\n",
    "    X_minibatches = np.split(X[:, 0:split_index], number_of_batches, axis=1)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        X_left_over_portion  = X[:, split_index:number_of_training_images]\n",
    "        X_minibatches.append(X_left_over_portion)\n",
    "    \n",
    "    Y_minibatches = np.split(Y[0:split_index], number_of_batches, axis=0)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        Y_left_over_portion  = Y[split_index:number_of_training_images]\n",
    "        Y_minibatches.append(Y_left_over_portion)\n",
    "    \n",
    "    number_of_minibatches = len(Y_minibatches)\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Number Of Minibatches: ' + str(number_of_minibatches))\n",
    "\n",
    "    for index in range(0, number_of_minibatches-1):\n",
    "        X_minibatch = X_minibatches[index]\n",
    "        Y_minibatch = Y_minibatches[index]\n",
    "\n",
    "        if(hidden_layer_relu + hidden_layer_tanh + hidden_layer_sigmoid != 1):\n",
    "            print(\"ERROR! Please Select Only 1 Hidden Layer Activation Function\")\n",
    "            break\n",
    "\n",
    "        # forward propogation training data set\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, X_minibatch, [X_minibatch], [], 0, [], keep_prob)\n",
    "        L = loss(A_layers[len(A_layers) - 1], Y_minibatch)\n",
    "        if(L2 == True):\n",
    "            C = cost_L2(L, W, epsilon) \n",
    "        else:\n",
    "            C = cost(L) \n",
    "\n",
    "        # backpropogation\n",
    "        W, B = backward_propagation(W, B, Y_minibatch, A_layers, Z_layers, 0, alpha, epsilon, len(W) - 1, D,  V_dW, V_dB)\n",
    "        \n",
    "    if(epoch % main_logger_output_epochs == 0):\n",
    "        print('Cost: ' + str(C))\n",
    "\n",
    "    # forward propogation test data set\n",
    "    A_test = forward_propagation(W, B, X_test, 0)\n",
    "\n",
    "    # accuracy\n",
    "    _prediction = prediction(A_test)    \n",
    "    _accuracy = accuracy(_prediction, Y_test)  \n",
    "\n",
    "    # storage for plotting\n",
    "    cost_array.append(C)\n",
    "    accuracy_array.append(_accuracy)\n",
    "    interation_array.append(epoch)\n",
    "\n",
    "\n",
    "end_time = time.time()\n",
    "run_time = end_time - start_time\n",
    "    \n",
    "print('')\n",
    "print('Results:')\n",
    "print('')\n",
    "    \n",
    "print('')\n",
    "print('Run Time: ' + str(run_time) + ' seconds')\n",
    "print('Cost: ' + str(C))    \n",
    "print('Accuracy: ' + str(_accuracy) + ' %')    \n",
    "print('')\n",
    "print('')\n",
    "\n",
    "\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, cost_array, 'red')\n",
    "pyplot.title('Learning Curve - ' + str(len(X[0])) + ' Training Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Cost')\n",
    "pyplot.show()\n",
    "\n",
    "# plot percent accuracy curve\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, accuracy_array, 'red')\n",
    "pyplot.title('Percent Accuracy Curve - ' + str(len(X_test[0])) + ' Test Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Percent Accuracy')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated, after 100 epochs with minibatches of 27 the cost became approximately 0.0002186 and the test data accuracy reached 99.16%. These results are excellent. The test accuracy is high because minibatch stochastic gradient descent inately provides a form of regularization. This time, we converged in approximatley 55 iterations to a very high accuracy and low cost. This lines up with what we would intuitively expect by making the momentum coefficient low. Our algorithm focused less on historical trend of the gradient and more on each step. As a results we converged slower than with a higher momentum coefficient."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have illustrated how the minibatch stochastic gradient provies many benefits for training neural networks. This technique provides a form of regularization by preventing the network from getting stuck local minima. The networks also converge faster. We have also explored how taking into account the momentum of the gradient can speed up the convergence of the network."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}